Orionnnn/hub-stats huggingface dataset:
MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation
  Model Training on EU Languages
Authors: name: Marco Gaido, name: Sara Papi, name: Luisa Bentivogli, name: Alessio Brutti, name: Mauro Cettolo, name: Roberto Gretter, name: Marco Matassoni, name: Mohamed Nabih, name: Matteo Negri
Published: 2024-10-01 19:54:10

The rise of foundation models (FMs), coupled with regulatory efforts
addressing their risks and impacts, has sparked significant interest in
open-source models. However, existing speech FMs (SFMs) fall short of full
compliance with the open-source principles, even if claimed otherwise, as no
existing SFM has model weights, code, and training data publicly available
under open-source terms. In this work, we take the first step toward filling
this gap by focusing on the 24 official languages of the European Union (EU).
We collect suitable training data by surveying automatic speech recognition
datasets and unlabeled speech corpora under open-source compliant licenses, for
a total of 950k hours. Additionally, we release automatic transcripts for 441k
hours of unlabeled data under the permissive CC-BY license, thereby
facilitating the creation of open-source SFMs for the EU languages.
HarmoniCa: Harmonizing Training and Inference for Better Feature Cache
  in Diffusion Transformer Acceleration
Authors: name: Yushi Huang, name: Zining Wang, name: Ruihao Gong, name: Jing Liu, name: Xinjie Zhang, name: Jinyang Guo, name: Xianglong Liu, name: Jun Zhang
Published: 2024-10-02 16:34:29

Diffusion Transformers (DiTs) have gained prominence for outstanding
scalability and extraordinary performance in generative tasks. However, their
considerable inference costs impede practical deployment. The feature cache
mechanism, which involves storing and retrieving redundant computations across
timesteps, holds promise for reducing per-step inference time in diffusion
models. Most existing caching methods for DiT are manually designed. Although
the learning-based approach attempts to optimize strategies adaptively, it
suffers from discrepancies between training and inference, which hampers both
the performance and acceleration ratio. Upon detailed analysis, we pinpoint
that these discrepancies primarily stem from two aspects: (1) Prior Timestep
Disregard, where training ignores the effect of cache usage at earlier
timesteps, and (2) Objective Mismatch, where the training target (align
predicted noise in each timestep) deviates from the goal of inference (generate
the high-quality image). To alleviate these discrepancies, we propose
HarmoniCa, a novel method that Harmonizes training and inference with a novel
learning-based Caching framework built upon Step-Wise Denoising Training (SDT)
and Image Error Proxy-Guided Objective (IEPO). Compared to the traditional
training paradigm, the newly proposed SDT maintains the continuity of the
denoising process, enabling the model to leverage information from prior
timesteps during training, similar to the way it operates during inference.
Furthermore, we design IEPO, which integrates an efficient proxy mechanism to
approximate the final image error caused by reusing the cached feature.
Therefore, IEPO helps balance final image quality and cache utilization,
resolving the issue of training that only considers the impact of cache usage
on the predicted output at each timestep.
E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
Authors: name: Ye Liu, name: Zongyang Ma, name: Zhongang Qi, name: Yang Wu, name: Ying Shan, name: Chang Wen Chen
Published: 2024-09-26 17:53:04

Recent advances in Video Large Language Models (Video-LLMs) have demonstrated
their great potential in general-purpose video understanding. To verify the
significance of these models, a number of benchmarks have been proposed to
diagnose their capabilities in different scenarios. However, existing
benchmarks merely evaluate models through video-level question-answering,
lacking fine-grained event-level assessment and task diversity. To fill this
gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding
Benchmark), a large-scale and high-quality benchmark for open-ended event-level
video understanding. Categorized within a 3-level task taxonomy, E.T. Bench
encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length)
under 8 domains, providing comprehensive evaluations. We extensively evaluated
8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that
state-of-the-art models for coarse-level (video-level) understanding struggle
to solve our fine-grained tasks, e.g., grounding event-of-interests within
videos, largely due to the short video context length, improper time
representations, and lack of multi-event training data. Focusing on these
issues, we further propose a strong baseline model, E.T. Chat, together with an
instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained
event-level understanding. Our simple but effective solution demonstrates
superior performance in multiple scenarios.
Selective Aggregation for Low-Rank Adaptation in Federated Learning
Authors: name: Pengxin Guo, name: Shuang Zeng, name: Yanran Wang, name: Huijie Fan, name: Feifei Wang, name: Liangqiong Qu
Published: 2024-10-02 12:14:36

We investigate LoRA in federated learning through the lens of the asymmetry
analysis of the learned A and B matrices. In doing so, we uncover that A
matrices are responsible for learning general knowledge, while B matrices
focus on capturing client-specific knowledge. Based on this finding, we
introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two
low-rank trainable matrices A and B to model the weight update, but only
A matrices are shared with the server for aggregation. Moreover, we delve
into the relationship between the learned A and B matrices in other LoRA
variants, such as rsLoRA and VeRA, revealing a consistent pattern.
Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting
in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm
for integrating LoRA with FL, offering guidance for future work on subsequent
LoRA variants combined with FL. Extensive experimental results on natural
language understanding and generation tasks demonstrate the effectiveness of
the proposed method.
Not All LLM Reasoners Are Created Equal
Authors: name: Arian Hosseini, name: Alessandro Sordoni, name: Daniel Toyama, name: Aaron Courville, name: Rishabh Agarwal
Published: 2024-10-02 17:01:10

We study the depth of grade-school math (GSM) problem-solving capabilities of
LLMs. To this end, we evaluate their performance on pairs of existing math word
problems together so that the answer to the second problem depends on correctly
answering the first problem. Our findings reveal a significant reasoning gap in
most LLMs, that is performance difference between solving the compositional
pairs and solving each question independently. This gap is more pronounced in
smaller, more cost-efficient, and math-specialized models. Moreover,
instruction-tuning recipes and code generation have varying effects across LLM
sizes, while finetuning on GSM can lead to task overfitting. Our analysis
indicates that large reasoning gaps are not because of test-set leakage, but
due to distraction from additional context and poor second-hop reasoning.
Overall, LLMs exhibit systematic differences in their reasoning abilities,
despite what their performance on standard benchmarks indicates.
Is Preference Alignment Always the Best Option to Enhance LLM-Based
  Translation? An Empirical Analysis
Authors: name: Hippolyte Gisserot-Boukhlef, name: Ricardo Rei, name: Emmanuel Malherbe, name: Céline Hudelot, name: Pierre Colombo, name: Nuno M. Guerreiro
Published: 2024-09-30 08:01:44

Neural metrics for machine translation (MT) evaluation have become
increasingly prominent due to their superior correlation with human judgments
compared to traditional lexical metrics. Researchers have therefore utilized
neural metrics through quality-informed decoding strategies, achieving better
results than likelihood-based methods. With the rise of Large Language Models
(LLMs), preference-based alignment techniques have gained attention for their
potential to enhance translation quality by optimizing model weights directly
on preferences induced by quality estimators. This study focuses on Contrastive
Preference Optimization (CPO) and conducts extensive experiments to evaluate
the impact of preference-based alignment on translation quality. Our findings
indicate that while CPO consistently outperforms Supervised Fine-Tuning (SFT)
on high-quality data with regard to the alignment metric, it may lead to
instability across downstream evaluation metrics, particularly between neural
and lexical ones. Additionally, we demonstrate that relying solely on the base
model for generating candidate translations achieves performance comparable to
using multiple external systems, while ensuring better consistency across
downstream metrics.
3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and
  Box-Focused Sampling for 3D Object Detection
Authors: name: Yang Cao, name: Yuanliang Jv, name: Dan Xu
Published: 2024-10-02 15:15:52

Neural Radiance Fields (NeRF) are widely used for novel-view synthesis and
have been adapted for 3D Object Detection (3DOD), offering a promising approach
to 3DOD through view-synthesis representation. However, NeRF faces inherent
limitations: (i) limited representational capacity for 3DOD due to its implicit
nature, and (ii) slow rendering speeds. Recently, 3D Gaussian Splatting (3DGS)
has emerged as an explicit 3D representation that addresses these limitations.
Inspired by these advantages, this paper introduces 3DGS into 3DOD for the
first time, identifying two main challenges: (i) Ambiguous spatial distribution
of Gaussian blobs: 3DGS primarily relies on 2D pixel-level supervision,
resulting in unclear 3D spatial distribution of Gaussian blobs and poor
differentiation between objects and background, which hinders 3DOD; (ii)
Excessive background blobs: 2D images often include numerous background pixels,
leading to densely reconstructed 3DGS with many noisy Gaussian blobs
representing the background, negatively affecting detection. To tackle the
challenge (i), we leverage the fact that 3DGS reconstruction is derived from 2D
images, and propose an elegant and efficient solution by incorporating 2D
Boundary Guidance to significantly enhance the spatial distribution of Gaussian
blobs, resulting in clearer differentiation between objects and their
background. To address the challenge (ii), we propose a Box-Focused Sampling
strategy using 2D boxes to generate object probability distribution in 3D
spaces, allowing effective probabilistic sampling in 3D to retain more object
blobs and reduce noisy background blobs. Benefiting from our designs, our
3DGS-DET significantly outperforms the SOTA NeRF-based method, NeRF-Det,
achieving improvements of +6.6 on mAP@0.25 and +8.1 on mAP@0.5 for the ScanNet
dataset, and impressive +31.5 on mAP@0.25 for the ARKITScenes dataset.
Quantifying Generalization Complexity for Large Language Models
Authors: name: Zhenting Qi, name: Hongyin Luo, name: Xuliang Huang, name: Zhuokai Zhao, name: Yibo Jiang, name: Xiangjun Fan, name: Himabindu Lakkaraju, name: James Glass
Published: 2024-10-02 17:25:37

While large language models (LLMs) have shown exceptional capabilities in
understanding complex queries and performing sophisticated tasks, their
generalization abilities are often deeply entangled with memorization,
necessitating more precise evaluation. To address this challenge, we introduce
Scylla, a dynamic evaluation framework that quantitatively measures the
generalization abilities of LLMs. Scylla disentangles generalization from
memorization via assessing model performance on both in-distribution (ID) and
out-of-distribution (OOD) data through 20 tasks across 5 levels of complexity.
Through extensive experiments, we uncover a non-monotonic relationship between
task complexity and the performance gap between ID and OOD data, which we term
the generalization valley. Specifically, this phenomenon reveals a critical
threshold - referred to as critical complexity - where reliance on
non-generalizable behavior peaks, indicating the upper bound of LLMs'
generalization capabilities. As model size increases, the critical complexity
shifts toward higher levels of task complexity, suggesting that larger models
can handle more complex reasoning tasks before over-relying on memorization.
Leveraging Scylla and the concept of critical complexity, we benchmark 28LLMs
including both open-sourced models such as LLaMA and Qwen families, and
close-sourced models like Claude and GPT, providing a more robust evaluation
and establishing a clearer understanding of LLMs' generalization capabilities.
Closed-loop Long-horizon Robotic Planning via Equilibrium Sequence
  Modeling
Authors: name: Jinghan Li, name: Zhicheng Sun, name: Fei Li, name: Cao Sheng, name: Jiazhong Yu, name: Yadong Mu
Published: 2024-10-02 11:42:49

In the endeavor to make autonomous robots take actions, task planning is a
major challenge that requires translating high-level task descriptions into
long-horizon action sequences. Despite recent advances in language model
agents, they remain prone to planning errors and limited in their ability to
plan ahead. To address these limitations in robotic planning, we advocate a
self-refining scheme that iteratively refines a draft plan until an equilibrium
is reached. Remarkably, this process can be optimized end-to-end from an
analytical perspective without the need to curate additional verifiers or
reward models, allowing us to train self-refining planners in a simple
supervised learning fashion. Meanwhile, a nested equilibrium sequence modeling
procedure is devised for efficient closed-loop planning that incorporates
useful feedback from the environment (or an internal world model). Our method
is evaluated on the VirtualHome-Env benchmark, showing advanced performance
with better scaling for inference computation. Code is available at
https://github.com/Singularity0104/equilibrium-planner.
LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks
Authors: name: Mengzhao Jia, name: Wenhao Yu, name: Kaixin Ma, name: Tianqing Fang, name: Zhihan Zhang, name: Siru Ouyang, name: Hongming Zhang, name: Meng Jiang, name: Dong Yu
Published: 2024-10-02 16:55:01

Text-rich images, where text serves as the central visual element guiding the
overall understanding, are prevalent in real-world applications, such as
presentation slides, scanned documents, and webpage snapshots. Tasks involving
multiple text-rich images are especially challenging, as they require not only
understanding the content of individual images but reasoning about
inter-relationships and logical flows across multiple visual inputs. Despite
the importance of these scenarios, current multimodal large language models
(MLLMs) struggle to handle such tasks due to two key challenges: (1) the
scarcity of high-quality instruction tuning datasets for text-rich multi-image
scenarios, and (2) the difficulty in balancing image resolution with visual
feature sequence length. To address these challenges, we propose \OurMethod, a
MLLM designed specifically for handling vision-language tasks involving
multiple text-rich images. First, we curated about one million high-quality
multimodal instruction-tuning data, tailored to text-rich, multi-image
scenarios. Second, we developed an adaptive high-resolution multi-image
encoding module to dynamically optimize the allocation of visual sequence
length based on the original aspect ratios and resolutions of the input images.
Experiments across a wide range of benchmarks demonstrate our model's superior
capabilities in text-rich, multi-image evaluations and competitive performance
in general domain evaluations.
From Code to Correctness: Closing the Last Mile of Code Generation with
  Hierarchical Debugging
Authors: name: Yuling Shi, name: Songsong Wang, name: Chengcheng Wan, name: Xiaodong Gu
Published: 2024-10-02 03:57:21

While large language models have made significant strides in code generation,
the pass rate of the generated code is bottlenecked on subtle errors, often
requiring human intervention to pass tests, especially for complex problems.
Existing LLM-based debugging systems treat generated programs as monolithic
units, failing to address bugs at multiple levels of granularity, from
low-level syntax errors to high-level algorithmic flaws. In this paper, we
introduce Multi-Granularity Debugger (MGDebugger), a hierarchical code debugger
by isolating, identifying, and resolving bugs at various levels of granularity.
MGDebugger decomposes problematic code into a hierarchical tree structure of
subfunctions, with each level representing a particular granularity of error.
During debugging, it analyzes each subfunction and iteratively resolves bugs in
a bottom-up manner. To effectively test each subfunction, we propose an
LLM-simulated Python executor, which traces code execution and tracks important
variable states to pinpoint errors accurately. Extensive experiments
demonstrate that MGDebugger outperforms existing debugging systems, achieving
an 18.9% improvement in accuracy over seed generations in HumanEval and a 97.6%
repair success rate in HumanEvalFix. Furthermore, MGDebugger effectively fixes
bugs across different categories and difficulty levels, demonstrating its
robustness and effectiveness.
EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis
Authors: name: Alexander Mai, name: Peter Hedman, name: George Kopanas, name: Dor Verbin, name: David Futschik, name: Qiangeng Xu, name: Falko Kuester, name: Jon Barron, name: Yinda Zhang
Published: 2024-10-02 17:59:09

We present Exact Volumetric Ellipsoid Rendering (EVER), a method for
real-time differentiable emission-only volume rendering. Unlike recent
rasterization based approach by 3D Gaussian Splatting (3DGS), our primitive
based representation allows for exact volume rendering, rather than alpha
compositing 3D Gaussian billboards. As such, unlike 3DGS our formulation does
not suffer from popping artifacts and view dependent density, but still
achieves frame rates of sim!30 FPS at 720p on an NVIDIA RTX4090. Since our
approach is built upon ray tracing it enables effects such as defocus blur and
camera distortion (e.g. such as from fisheye cameras), which are difficult to
achieve by rasterization. We show that our method is more accurate with fewer
blending issues than 3DGS and follow-up work on view-consistent rendering,
especially on the challenging large-scale scenes from the Zip-NeRF dataset
where it achieves sharpest results among real-time techniques.
ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation
Authors: name: Rinon Gal, name: Adi Haviv, name: Yuval Alaluf, name: Amit H. Bermano, name: Daniel Cohen-Or, name: Gal Chechik
Published: 2024-10-02 16:43:24

The practical use of text-to-image generation has evolved from simple,
monolithic models to complex workflows that combine multiple specialized
components. While workflow-based approaches can lead to improved image quality,
crafting effective workflows requires significant expertise, owing to the large
number of available components, their complex inter-dependence, and their
dependence on the generation prompt. Here, we introduce the novel task of
prompt-adaptive workflow generation, where the goal is to automatically tailor
a workflow to each user prompt. We propose two LLM-based approaches to tackle
this task: a tuning-based method that learns from user-preference data, and a
training-free method that uses the LLM to select existing flows. Both
approaches lead to improved image quality when compared to monolithic models or
generic, prompt-independent workflows. Our work shows that prompt-dependent
flow prediction offers a new pathway to improving text-to-image generation
quality, complementing existing research directions in the field.
HelpSteer2-Preference: Complementing Ratings with Preferences
Authors: name: Zhilin Wang, name: Alexander Bukharin, name: Olivier Delalleau, name: Daniel Egert, name: Gerald Shen, name: Jiaqi Zeng, name: Oleksii Kuchaiev, name: Yi Dong
Published: 2024-10-02 06:05:52

Reward models are critical for aligning models to follow instructions, and
are typically trained following one of two popular paradigms: Bradley-Terry
style or Regression style. However, there is a lack of evidence that either
approach is better than the other, when adequately matched for data. This is
primarily because these approaches require data collected in different (but
incompatible) formats, meaning that adequately matched data is not available in
existing public datasets. To tackle this problem, we release preference
annotations (designed for Bradley-Terry training) to complement existing
ratings (designed for Regression style training) in the HelpSteer2 dataset. To
improve data interpretability, preference annotations are accompanied with
human-written justifications. Using this data, we conduct the first
head-to-head comparison of Bradley-Terry and Regression models when adequately
matched for data. Based on insights derived from such a comparison, we propose
a novel approach to combine Bradley-Terry and Regression reward modeling. A
Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on
RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We
also demonstrate the effectiveness of this reward model at aligning models to
follow instructions in RLHF. We open-source this dataset (CC-BY-4.0 license) at
https://huggingface.co/datasets/nvidia/HelpSteer2 and openly release the
trained Reward Model at
https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning
Authors: name: Dongwei Jiang, name: Guoxuan Wang, name: Yining Lu, name: Andrew Wang, name: Jingyu Zhang, name: Chuyu Liu, name: Benjamin Van Durme, name: Daniel Khashabi
Published: 2024-10-01 20:05:51

The reasoning steps generated by LLMs might be incomplete, as they mimic
logical leaps common in everyday communication found in their pre-training
data: underlying rationales are frequently left implicit (unstated). To address
this challenge, we introduce RATIONALYST, a model for process-supervision of
reasoning based on pre-training on a vast collection of rationale annotations
extracted from unlabeled data. We extract 79k rationales from web-scale
unlabelled dataset (the Pile) and a combination of reasoning datasets with
minimal human intervention. This web-scale pre-training for reasoning allows
RATIONALYST to consistently generalize across diverse reasoning tasks,
including mathematical, commonsense, scientific, and logical reasoning.
Fine-tuned from LLaMa-3-8B, RATIONALYST improves the accuracy of reasoning by
an average of 3.9% on 7 representative reasoning benchmarks. It also
demonstrates superior performance compared to significantly larger verifiers
like GPT-4 and similarly sized models fine-tuned on matching training sets.