Orionnnn/hub-stats huggingface dataset:
Scaling Up Your Kernels: Large Kernel Design in ConvNets towards
  Universal Representations
Authors: name: Yiyuan Zhang, name: Xiaohan Ding, name: Xiangyu Yue
Published: 2024-10-10 15:43:55

This paper proposes the paradigm of large convolutional kernels in designing
modern Convolutional Neural Networks (ConvNets). We establish that employing a
few large kernels, instead of stacking multiple smaller ones, can be a superior
design strategy. Our work introduces a set of architecture design guidelines
for large-kernel ConvNets that optimize their efficiency and performance. We
propose the UniRepLKNet architecture, which offers systematical architecture
design principles specifically crafted for large-kernel ConvNets, emphasizing
their unique ability to capture extensive spatial information without deep
layer stacking. This results in a model that not only surpasses its
predecessors with an ImageNet accuracy of 88.0%, an ADE20K mIoU of 55.6%, and a
COCO box AP of 56.4% but also demonstrates impressive scalability and
performance on various modalities such as time-series forecasting, audio, point
cloud, and video recognition. These results indicate the universal modeling
abilities of large-kernel ConvNets with faster inference speed compared with
vision transformers. Our findings reveal that large-kernel ConvNets possess
larger effective receptive fields and a higher shape bias, moving away from the
texture bias typical of smaller-kernel CNNs. All codes and models are publicly
available at https://github.com/AILab-CVC/UniRepLKNet promoting further
research and development in the community.
Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow
Authors: name: Fu-Yun Wang, name: Ling Yang, name: Zhaoyang Huang, name: Mengdi Wang, name: Hongsheng Li
Published: 2024-10-09 17:43:38

Diffusion models have greatly improved visual generation but are hindered by
slow generation speed due to the computationally intensive nature of solving
generative ODEs. Rectified flow, a widely recognized solution, improves
generation speed by straightening the ODE path. Its key components include: 1)
using the diffusion form of flow-matching, 2) employing boldsymbol
v-prediction, and 3) performing rectification (a.k.a. reflow). In this paper,
we argue that the success of rectification primarily lies in using a pretrained
diffusion model to obtain matched pairs of noise and samples, followed by
retraining with these matched noise-sample pairs. Based on this, components 1)
and 2) are unnecessary. Furthermore, we highlight that straightness is not an
essential training target for rectification; rather, it is a specific case of
flow-matching models. The more critical training target is to achieve a
first-order approximate ODE path, which is inherently curved for models like
DDPM and Sub-VP. Building on this insight, we propose Rectified Diffusion,
which generalizes the design space and application scope of rectification to
encompass the broader category of diffusion models, rather than being
restricted to flow-matching models. We validate our method on Stable Diffusion
v1-5 and Stable Diffusion XL. Our method not only greatly simplifies the
training procedure of rectified flow-based previous works (e.g., InstaFlow) but
also achieves superior performance with even lower training cost. Our code is
available at https://github.com/G-U-N/Rectified-Diffusion.
MLLM as Retriever: Interactively Learning Multimodal Retrieval for
  Embodied Agents
Authors: name: Junpeng Yue, name: Xinru Xu, name: Börje F. Karlsson, name: Zongqing Lu
Published: 2024-10-04 14:10:39

MLLM agents demonstrate potential for complex embodied tasks by retrieving
multimodal task-relevant trajectory data. However, current retrieval methods
primarily focus on surface-level similarities of textual or visual cues in
trajectories, neglecting their effectiveness for the specific task at hand. To
address this issue, we propose a novel method, MLLM as ReTriever (MART), which
enhances the performance of embodied agents by utilizing interaction data to
fine-tune an MLLM retriever based on preference learning, such that the
retriever fully considers the effectiveness of trajectories and prioritize them
for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that
leverages MLLMs' summarization capabilities to represent trajectories with
fewer tokens while preserving key information, enabling agents to better
comprehend milestones in the trajectory. Experimental results across various
environments demonstrate our method significantly improves task success rates
in unseen scenes compared to baseline methods. This work presents a new
paradigm for multimodal retrieval in embodied agents, by fine-tuning a
general-purpose MLLM as the retriever to assess trajectory effectiveness. All
benchmark task sets and simulator code modifications for action and observation
spaces will be released.
PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers
  in LLMs
Authors: name: Mengzhao Chen, name: Yi Liu, name: Jiahao Wang, name: Yi Bin, name: Wenqi Shao, name: Ping Luo
Published: 2024-10-07 17:59:35

Quantization is essential for deploying Large Language Models (LLMs) by
enhancing memory efficiency and inference speed. Existing methods for
activation quantization mainly address channel-wise outliers, often neglecting
token-wise outliers, leading to reliance on costly per-token dynamic
quantization. To address this, we introduce PrefixQuant, a novel technique that
isolates outlier tokens offline without re-training. Specifically, PrefixQuant
identifies high-frequency outlier tokens and prefixes them in the KV cache,
preventing the generation of outlier tokens during inference and simplifying
quantization. To our knowledge, PrefixQuant is the first to enable efficient
per-tensor static quantization to outperform expensive per-token dynamic
quantization. For instance, in W4A4KV4 (4- bit weight, 4-bit activation, and
4-bit KV cache) Llama-3-8B, PrefixQuant with per-tensor static quantization
achieves a 7.43 WikiText2 perplexity and 71.08% average accuracy on 5
common-sense reasoning tasks, outperforming previous per-token dynamic
quantization methods like QuaRot with 0.98 perplexity improvement and +5.98
points accuracy. Additionally, the inference speed of W4A4 quantized models
using PrefixQuant is 1.60x to 2.81x faster than FP16 models and exceeds QuaRot
models by 1.2x to 1.3x. Our code is available at
https://github.com/ChenMnZ/PrefixQuant.
Optima: Optimizing Effectiveness and Efficiency for LLM-Based
  Multi-Agent System
Authors: name: Weize Chen, name: Jiarui Yuan, name: Chen Qian, name: Cheng Yang, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-10-10 17:00:06

Large Language Model (LLM) based multi-agent systems (MAS) show remarkable
potential in collaborative problem-solving, yet they still face critical
challenges: low communication efficiency, poor scalability, and a lack of
effective parameter-updating optimization methods. We present Optima, a novel
framework that addresses these issues by significantly enhancing both
communication efficiency and task effectiveness in LLM-based MAS through LLM
training. Optima employs an iterative generate, rank, select, and train
paradigm with a reward function balancing task performance, token efficiency,
and communication readability. We explore various RL algorithms, including
Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid
approaches, providing insights into their effectiveness-efficiency trade-offs.
We integrate Monte Carlo Tree Search-inspired techniques for DPO data
generation, treating conversation turns as tree nodes to explore diverse
interaction paths. Evaluated on common multi-agent tasks, including
information-asymmetric question answering and complex reasoning, Optima shows
consistent and substantial improvements over single-agent baselines and vanilla
MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than
10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's
efficiency gains open new possibilities for leveraging inference-compute more
effectively, leading to improved inference-time scaling laws. By addressing
fundamental challenges in LLM-based MAS, Optima shows the potential towards
scalable, efficient, and effective MAS
(https://chenweize1998.github.io/optima-project-page).
Emergent properties with repeated examples
Authors: name: François Charton, name: Julia Kempe
Published: 2024-10-09 16:28:23

We study the performance of transformers as a function of the number of
repetitions of training examples with algorithmically generated datasets. On
three problems of mathematics: the greatest common divisor, modular
multiplication, and matrix eigenvalues, we show that for a fixed number of
training steps, models trained on smaller sets of repeated examples outperform
models trained on larger sets of single-use examples. We also demonstrate that
two-set training - repeated use of a small random subset of examples, along
normal sampling on the rest of the training set - provides for faster learning
and better performance. This highlights that the benefits of repetition can
outweigh those of data diversity. These datasets and problems provide a
controlled setting to shed light on the still poorly understood interplay
between generalization and memorization in deep learning.
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving
  Vision-Linguistic Compositionality
Authors: name: Youngtaek Oh, name: Jae Won Cho, name: Dong-Jin Kim, name: In So Kweon, name: Junmo Kim
Published: 2024-10-07 17:16:20

In this paper, we propose a new method to enhance compositional understanding
in pre-trained vision and language models (VLMs) without sacrificing
performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches
often improve compositional reasoning at the cost of degrading multi-modal
capabilities, primarily due to the use of global hard negative (HN) loss, which
contrasts global representations of images and texts. This global HN loss
pushes HN texts that are highly similar to the original ones, damaging the
model's multi-modal representations. To overcome this limitation, we propose
Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard
negative loss and selective calibrated regularization. These innovations
provide fine-grained negative supervision while preserving the model's
representational integrity. Our extensive evaluations across diverse benchmarks
for both compositionality and multi-modal tasks show that FSC-CLIP not only
achieves compositionality on par with state-of-the-art models but also retains
strong multi-modal capabilities. Code is available at:
https://github.com/ytaek-oh/fsc-clip.
DICE: Discrete Inversion Enabling Controllable Editing for Multinomial
  Diffusion and Masked Generative Models
Authors: name: Xiaoxiao He, name: Ligong Han, name: Quan Dao, name: Song Wen, name: Minhao Bai, name: Di Liu, name: Han Zhang, name: Martin Renqiang Min, name: Felix Juefei-Xu, name: Chaowei Tan, name: Bo Liu, name: Kang Li, name: Hongdong Li, name: Junzhou Huang, name: Faez Ahmed, name: Akash Srivastava, name: Dimitris Metaxas
Published: 2024-10-10 17:59:48

Discrete diffusion models have achieved success in tasks like image
generation and masked language modeling but face limitations in controlled
content editing. We introduce DICE (Discrete Inversion for Controllable
Editing), the first approach to enable precise inversion for discrete diffusion
models, including multinomial diffusion and masked generative models. By
recording noise sequences and masking patterns during the reverse diffusion
process, DICE enables accurate reconstruction and flexible editing of discrete
data without the need for predefined masks or attention manipulation. We
demonstrate the effectiveness of DICE across both image and text domains,
evaluating it on models such as VQ-Diffusion, Paella, and RoBERTa. Our results
show that DICE preserves high data fidelity while enhancing editing
capabilities, offering new opportunities for fine-grained content manipulation
in discrete spaces. For project webpage, see
https://hexiaoxiao-cs.github.io/DICE/.
Trans4D: Realistic Geometry-Aware Transition for Compositional
  Text-to-4D Synthesis
Authors: name: Bohan Zeng, name: Ling Yang, name: Siyu Li, name: Jiaming Liu, name: Zixiang Zhang, name: Juanxi Tian, name: Kaixin Zhu, name: Yongzhen Guo, name: Fu-Yun Wang, name: Minkai Xu, name: Stefano Ermon, name: Wentao Zhang
Published: 2024-10-09 17:56:03

Recent advances in diffusion models have demonstrated exceptional
capabilities in image and video generation, further improving the effectiveness
of 4D synthesis. Existing 4D generation methods can generate high-quality 4D
objects or scenes based on user-friendly conditions, benefiting the gaming and
video industries. However, these methods struggle to synthesize significant
object deformation of complex 4D transitions and interactions within scenes. To
address this challenge, we propose Trans4D, a novel text-to-4D synthesis
framework that enables realistic complex scene transitions. Specifically, we
first use multi-modal large language models (MLLMs) to produce a physic-aware
scene description for 4D scene initialization and effective transition timing
planning. Then we propose a geometry-aware 4D transition network to realize a
complex scene-level 4D transition based on the plan, which involves expressive
geometrical object deformation. Extensive experiments demonstrate that Trans4D
consistently outperforms existing state-of-the-art methods in generating 4D
scenes with accurate and high-quality transitions, validating its
effectiveness. Code: https://github.com/YangLing0818/Trans4D
VHELM: A Holistic Evaluation of Vision Language Models
Authors: name: Tony Lee, name: Haoqin Tu, name: Chi Heem Wong, name: Wenhao Zheng, name: Yiyang Zhou, name: Yifan Mai, name: Josselin Somerville Roberts, name: Michihiro Yasunaga, name: Huaxiu Yao, name: Cihang Xie, name: Percy Liang
Published: 2024-10-09 17:46:34

Current benchmarks for assessing vision-language models (VLMs) often focus on
their perception or problem-solving capabilities and neglect other critical
aspects such as fairness, multilinguality, or toxicity. Furthermore, they
differ in their evaluation procedures and the scope of the evaluation, making
it difficult to compare models. To address these issues, we extend the HELM
framework to VLMs to present the Holistic Evaluation of Vision Language Models
(VHELM). VHELM aggregates various datasets to cover one or more of the 9
aspects: visual perception, knowledge, reasoning, bias, fairness,
multilinguality, robustness, toxicity, and safety. In doing so, we produce a
comprehensive, multi-dimensional view of the capabilities of the VLMs across
these important factors. In addition, we standardize the standard inference
parameters, methods of prompting, and evaluation metrics to enable fair
comparisons across models. Our framework is designed to be lightweight and
automatic so that evaluation runs are cheap and fast. Our initial run evaluates
22 VLMs on 21 existing datasets to provide a holistic snapshot of the models.
We uncover new key findings, such as the fact that efficiency-focused models
(e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than
their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark
but not when evaluated on the other aspects. For transparency, we release the
raw model generations and complete results on our website
(https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living
benchmark, and we hope to continue adding new datasets and models over time.
VLM2Vec: Training Vision-Language Models for Massive Multimodal
  Embedding Tasks
Authors: name: Ziyan Jiang, name: Rui Meng, name: Xinyi Yang, name: Semih Yavuz, name: Yingbo Zhou, name: Wenhu Chen
Published: 2024-10-07 16:14:05

Embedding models have been crucial in enabling various downstream tasks such
as semantic similarity, information retrieval, and clustering. Recently, there
has been a surge of interest in developing universal text embedding models that
can generalize across tasks (e.g., MTEB). However, progress in learning
universal multimodal embedding models has been relatively slow despite their
importance. In this work, we aim to explore the potential for building
universal embeddings capable of handling a wide range of downstream tasks. Our
contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark),
which covers 4 meta-tasks (i.e. classification, visual question answering,
multimodal retrieval, and visual grounding) and 36 datasets, including 20
training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model ->
Vector), a contrastive training framework that converts any state-of-the-art
vision-language model into an embedding model via training on MMEB. Unlike
previous models such as CLIP and BLIP, VLM2Vec can process any combination of
images and text to generate a fixed-dimensional vector based on task
instructions. We build a series of VLM2Vec models on Phi-3.5-V and evaluate
them on MMEB's evaluation split. Our results show that \model achieves an
absolute average improvement of 10% to 20% over existing multimodal embedding
models on both in-distribution and out-of-distribution datasets in MMEB.
Stuffed Mamba: State Collapse and State Capacity of RNN-Based
  Long-Context Modeling
Authors: name: Yingfa Chen, name: Xinrong Zhang, name: Shengding Hu, name: Xu Han, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-10-09 17:54:28

One essential advantage of recurrent neural networks (RNNs) over
transformer-based language models is their linear computational complexity
concerning the sequence length, which makes them much faster in handling long
sequences during inference. However, most publicly available RNNs (e.g., Mamba
and RWKV) are trained on sequences with less than 10K tokens, and their
effectiveness in longer contexts remains largely unsatisfying so far. In this
paper, we study the cause of the inability to process long context for RNNs and
suggest critical mitigations. We examine two practical concerns when applying
state-of-the-art RNNs to long contexts: (1) the inability to extrapolate to
inputs longer than the training length and (2) the upper bound of memory
capacity. Addressing the first concern, we first investigate *state collapse*
(SC), a phenomenon that causes severe performance degradation on sequence
lengths not encountered during training. With controlled experiments, we
attribute this to overfitting due to the recurrent state being
overparameterized for the training length. For the second concern, we train a
series of Mamba-2 models on long documents to empirically estimate the
recurrent state capacity in language modeling and passkey retrieval. Then,
three SC mitigation methods are proposed to improve Mamba-2's length
generalizability, allowing the model to process more than 1M tokens without SC.
We also find that the recurrent state capacity in passkey retrieval scales
exponentially to the state size, and we empirically train a Mamba-2 370M with
near-perfect passkey retrieval accuracy on 256K context length. This suggests a
promising future for RNN-based long-context modeling.
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual
  Alignment
Authors: name: Amir Hossein Kargaran, name: Ali Modarressi, name: Nafiseh Nikeghbal, name: Jana Diesner, name: François Yvon, name: Hinrich Schütze
Published: 2024-10-08 09:59:23

English-centric large language models (LLMs) often show strong multilingual
capabilities. However, the multilingual performance of these models remains
unclear and is not thoroughly evaluated for many languages. Most benchmarks for
multilinguality focus on classic NLP tasks, or cover a minimal number of
languages. We introduce MEXA, a method for assessing the multilingual
capabilities of pre-trained English-centric LLMs using parallel sentences,
which are available for more languages than existing downstream tasks. MEXA
leverages the fact that English-centric LLMs use English as a kind of pivot
language in their intermediate layers. It computes the alignment between
English and non-English languages using parallel sentences to evaluate the
transfer of language understanding from English to other languages. This
alignment can be used to estimate model performance in other languages. We
conduct studies using various parallel datasets (FLORES-200 and Bible), models
(Llama family, Gemma family, Mistral, and OLMo), and established downstream
tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute
embeddings in decoder-only models. Our results show that MEXA, in its default
settings, achieves a statistically significant average Pearson correlation of
0.90 with three established downstream tasks across nine models and two
parallel datasets. This suggests that MEXA is a reliable method for estimating
the multilingual capabilities of English-centric LLMs, providing a clearer
understanding of their multilingual potential and the inner workings of LLMs.
Leaderboard: https://huggingface.co/spaces/cis-lmu/Mexa, Code:
https://github.com/cisnlp/Mexa.
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for
  Enhanced Following of Instructions with Multiple Constraints
Authors: name: Thomas Palmeira Ferraz, name: Kartik Mehta, name: Yu-Hsiang Lin, name: Haw-Shiuan Chang, name: Shereen Oraby, name: Sijia Liu, name: Vivek Subramanian, name: Tagyoung Chung, name: Mohit Bansal, name: Nanyun Peng
Published: 2024-10-09 01:25:10

Instruction following is a key capability for LLMs. However, recent studies
have shown that LLMs often struggle with instructions containing multiple
constraints (e.g. a request to create a social media post "in a funny tone"
with "no hashtag"). Despite this, most evaluations focus solely on synthetic
data. To address this, we introduce RealInstruct, the first benchmark designed
to evaluate LLMs' ability to follow real-world multi-constrained instructions
by leveraging queries real users asked AI assistants. We also investigate
model-based evaluation as a cost-effective alternative to human annotation for
this task. Our findings reveal that even the proprietary GPT-4 model fails to
meet at least one constraint on over 21% of instructions, highlighting the
limitations of state-of-the-art models. To address the performance gap between
open-source and proprietary models, we propose the Decompose, Critique and
Refine (DeCRIM) self-correction pipeline, which enhances LLMs' ability to
follow constraints. DeCRIM works by decomposing the original instruction into a
list of constraints and using a Critic model to decide when and where the LLM's
response needs refinement. Our results show that DeCRIM improves Mistral's
performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
Moreover, we demonstrate that with strong feedback, open-source LLMs with
DeCRIM can outperform GPT-4 on both benchmarks.
Does Spatial Cognition Emerge in Frontier Models?
Authors: name: Santhosh Kumar Ramakrishnan, name: Erik Wijmans, name: Philipp Kraehenbuehl, name: Vladlen Koltun
Published: 2024-10-09 01:41:49

Not yet. We present SPACE, a benchmark that systematically evaluates spatial
cognition in frontier models. Our benchmark builds on decades of research in
cognitive science. It evaluates large-scale mapping abilities that are brought
to bear when an organism traverses physical environments, smaller-scale
reasoning about object shapes and layouts, and cognitive infrastructure such as
spatial attention and memory. For many tasks, we instantiate parallel
presentations via text and images, allowing us to benchmark both large language
models and large multimodal models. Results suggest that contemporary frontier
models fall short of the spatial intelligence of animals, performing near
chance level on a number of classic tests of animal cognition.