Orionnnn/hub-stats huggingface dataset:
Deciphering Cross-Modal Alignment in Large Vision-Language Models with
  Modality Integration Rate
Authors: name: Qidong Huang, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Jiaqi Wang, name: Dahua Lin, name: Weiming Zhang, name: Nenghai Yu
Published: 2024-10-09 17:59:04

We present the Modality Integration Rate (MIR), an effective, robust, and
generalized metric to indicate the multi-modal pre-training quality of Large
Vision Language Models (LVLMs). Large-scale pre-training plays a critical role
in building capable LVLMs, while evaluating its training quality without the
costly supervised fine-tuning stage is under-explored. Loss, perplexity, and
in-context evaluation results are commonly used pre-training metrics for Large
Language Models (LLMs), while we observed that these metrics are less
indicative when aligning a well-trained LLM with a new modality. Due to the
lack of proper metrics, the research of LVLMs in the critical pre-training
stage is hindered greatly, including the training data choice, efficient module
design, etc. In this paper, we propose evaluating the pre-training quality from
the inter-modal distribution distance perspective and present MIR, the Modality
Integration Rate, which is 1) Effective to represent the pre-training
quality and show a positive relation with the benchmark performance after
supervised fine-tuning. 2) Robust toward different training/evaluation
data. 3) Generalize across training configurations and architecture
choices. We conduct a series of pre-training experiments to explore the
effectiveness of MIR and observe satisfactory results that MIR is indicative
about training data selection, training strategy schedule, and model
architecture design to get better pre-training results. We hope MIR could be a
helpful metric for building capable LVLMs and inspire the following research
about modality alignment in different areas. Our code is at:
https://github.com/shikiw/Modality-Integration-Rate.
Hallucinating AI Hijacking Attack: Large Language Models and Malicious
  Code Recommenders
Authors: name: David Noever, name: Forrest McKee
Published: 2024-10-09 01:36:25

The research builds and evaluates the adversarial potential to introduce
copied code or hallucinated AI recommendations for malicious code in popular
code repositories. While foundational large language models (LLMs) from OpenAI,
Google, and Anthropic guard against both harmful behaviors and toxic strings,
previous work on math solutions that embed harmful prompts demonstrate that the
guardrails may differ between expert contexts. These loopholes would appear in
mixture of expert's models when the context of the question changes and may
offer fewer malicious training examples to filter toxic comments or recommended
offensive actions. The present work demonstrates that foundational models may
refuse to propose destructive actions correctly when prompted overtly but may
unfortunately drop their guard when presented with a sudden change of context,
like solving a computer programming challenge. We show empirical examples with
trojan-hosting repositories like GitHub, NPM, NuGet, and popular content
delivery networks (CDN) like jsDelivr which amplify the attack surface. In the
LLM's directives to be helpful, example recommendations propose application
programming interface (API) endpoints which a determined domain-squatter could
acquire and setup attack mobile infrastructure that triggers from the naively
copied code. We compare this attack to previous work on context-shifting and
contrast the attack surface as a novel version of "living off the land" attacks
in the malware literature. In the latter case, foundational language models can
hijack otherwise innocent user prompts to recommend actions that violate their
owners' safety policies when posed directly without the accompanying coding
support request.
Pixtral 12B
Authors: name: Pravesh Agrawal, name: Szymon Antoniak, name: Emma Bou Hanna, name: Devendra Chaplot, name: Jessica Chudnovsky, name: Saurabh Garg, name: Theophile Gervet, name: Soham Ghosh, name: Amélie Héliou, name: Paul Jacob, name: Albert Q. Jiang, name: Timothée Lacroix, name: Guillaume Lample, name: Diego Las Casas, name: Thibaut Lavril, name: Teven Le Scao, name: Andy Lo, name: William Marshall, name: Louis Martin, name: Arthur Mensch, name: Pavankumar Muddireddy, name: Valera Nemychnikova, name: Marie Pellat, name: Patrick Von Platen, name: Nikhil Raghuraman, name: Baptiste Rozière, name: Alexandre Sablayrolles, name: Lucile Saulnier, name: Romain Sauvestre, name: Wendy Shang, name: Roman Soletskyi, name: Lawrence Stewart, name: Pierre Stock, name: Joachim Studnia, name: Sandeep Subramanian, name: Sagar Vaze, name: Thomas Wang
Published: 2024-10-09 17:16:22

We introduce Pixtral-12B, a 12--billion-parameter multimodal language model.
Pixtral-12B is trained to understand both natural images and documents,
achieving leading performance on various multimodal benchmarks, surpassing a
number of larger models. Unlike many open-source models, Pixtral is also a
cutting-edge text model for its size, and does not compromise on natural
language performance to excel in multimodal tasks. Pixtral uses a new vision
encoder trained from scratch, which allows it to ingest images at their natural
resolution and aspect ratio. This gives users flexibility on the number of
tokens used to process an image. Pixtral is also able to process any number of
images in its long context window of 128K tokens. Pixtral 12B substanially
outperforms other open models of similar sizes (Llama-3.2 11B \& Qwen-2-VL 7B).
It also outperforms much larger open models like Llama-3.2 90B while being 7x
smaller. We further contribute an open-source benchmark, MM-MT-Bench, for
evaluating vision-language models in practical scenarios, and provide detailed
analysis and code for standardized evaluation protocols for multimodal LLMs.
Pixtral-12B is released under Apache 2.0 license.
Story-Adapter: A Training-free Iterative Framework for Long Story
  Visualization
Authors: name: Jiawei Mao, name: Xiaoke Huang, name: Yunfei Xie, name: Yuanqi Chang, name: Mude Hui, name: Bingjie Xu, name: Yuyin Zhou
Published: 2024-10-08 17:59:30

Story visualization, the task of generating coherent images based on a
narrative, has seen significant advancements with the emergence of
text-to-image models, particularly diffusion models. However, maintaining
semantic consistency, generating high-quality fine-grained interactions, and
ensuring computational feasibility remain challenging, especially in long story
visualization (i.e., up to 100 frames). In this work, we propose a
training-free and computationally efficient framework, termed Story-Adapter, to
enhance the generative capability of long stories. Specifically, we propose an
iterative paradigm to refine each generated image, leveraging both the text
prompt and all generated images from the previous iteration. Central to our
framework is a training-free global reference cross-attention module, which
aggregates all generated images from the previous iteration to preserve
semantic consistency across the entire story, while minimizing computational
costs with global embeddings. This iterative process progressively optimizes
image generation by repeatedly incorporating text constraints, resulting in
more precise and fine-grained interactions. Extensive experiments validate the
superiority of Story-Adapter in improving both semantic consistency and
generative capability for fine-grained interactions, particularly in long story
scenarios. The project page and associated code can be accessed via
https://jwmao1.github.io/storyadapter .
Aria: An Open Multimodal Native Mixture-of-Experts Model
Authors: name: Dongxu Li, name: Yudong Liu, name: Haoning Wu, name: Yue Wang, name: Zhiqi Shen, name: Bowen Qu, name: Xinyao Niu, name: Guoyin Wang, name: Bei Chen, name: Junnan Li
Published: 2024-10-08 12:44:57

Information comes in diverse modalities. Multimodal native AI models are
essential to integrate real-world information and deliver comprehensive
understanding. While proprietary multimodal native models exist, their lack of
openness imposes obstacles for adoptions, let alone adaptations. To fill this
gap, we introduce Aria, an open multimodal native model with best-in-class
performance across a wide range of multimodal, language, and coding tasks. Aria
is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual
token and text token, respectively. It outperforms Pixtral-12B and
Llama3.2-11B, and is competitive against the best proprietary models on various
multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline,
which progressively equips the model with strong capabilities in language
understanding, multimodal understanding, long context window, and instruction
following. We open-source the model weights along with a codebase that
facilitates easy adoptions and adaptations of Aria in real-world applications.
Self-Boosting Large Language Models with Synthetic Preference Data
Authors: name: Qingxiu Dong, name: Li Dong, name: Xingxing Zhang, name: Zhifang Sui, name: Furu Wei
Published: 2024-10-09 14:57:31

Through alignment with human preferences, Large Language Models (LLMs) have
advanced significantly in generating honest, harmless, and helpful responses.
However, collecting high-quality preference data is a resource-intensive and
creativity-demanding process, especially for the continual improvement of LLMs.
We introduce SynPO, a self-boosting paradigm that leverages synthetic
preference data for model alignment. SynPO employs an iterative mechanism
wherein a self-prompt generator creates diverse prompts, and a response
improver refines model responses progressively. This approach trains LLMs to
autonomously learn the generative rewards for their own outputs and eliminates
the need for large-scale annotation of prompts and human preferences. After
four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements
in instruction-following abilities, achieving over 22.1% win rate improvements
on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general
performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score
increase on the well-recognized Open LLM leaderboard.
TRACE: Temporal Grounding Video LLM via Causal Event Modeling
Authors: name: Yongxin Guo, name: Jingyu Liu, name: Mingda Li, name: Xiaoying Tang, name: Qingbin Liu, name: Xi Chen
Published: 2024-10-08 02:46:30

Video Temporal Grounding (VTG) is a crucial capability for video
understanding models and plays a vital role in downstream tasks such as video
browsing and editing. To effectively handle various tasks simultaneously and
enable zero-shot prediction, there is a growing trend in employing video LLMs
for VTG tasks. However, current video LLM-based methods rely exclusively on
natural language generation, lacking the ability to model the clear structure
inherent in videos, which restricts their effectiveness in tackling VTG tasks.
To address this issue, this paper first formally introduces causal event
modeling framework, which represents videos as sequences of events, and predict
the current event using previous events, video inputs, and textural
instructions. Each event consists of three components: timestamps, salient
scores, and textual captions. We then propose a novel task-interleaved video
LLM called TRACE to effectively implement the causal event modeling framework
in practice. The TRACE processes visual frames, timestamps, salient scores, and
text as distinct tasks, employing various encoders and decoding heads for each.
Task tokens are arranged in an interleaved sequence according to the causal
event modeling framework's formulation. Extensive experiments on various VTG
tasks and datasets demonstrate the superior performance of TRACE compared to
state-of-the-art video LLMs. Our model and code are available at
https://github.com/gyxxyg/TRACE.
Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
  Learning
Authors: name: Siyuan Li, name: Juanxi Tian, name: Zedong Wang, name: Luyuan Zhang, name: Zicheng Liu, name: Weiyang Jin, name: Yang Liu, name: Baigui Sun, name: Stan Z. Li
Published: 2024-10-08 21:14:23

This paper delves into the interplay between vision backbones and optimizers,
unvealing an inter-dependent phenomenon termed
\textbf{backbone-optimizer coupling bias}
(BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a
marked co-dependency with SGD families, while recent architectures like ViTs
and ConvNeXt share a tight coupling with the adaptive learning rate ones. We
further show that BOCB can be introduced by both optimizers and certain
backbone designs and may significantly impact the pre-training and downstream
fine-tuning of vision models. Through in-depth empirical analysis, we summarize
takeaways on recommended optimizers and insights into robust vision backbone
architectures. We hope this work can inspire the community to question
long-held assumptions on backbones and optimizers, stimulate further
explorations, and thereby contribute to more robust vision systems. The source
code and models are publicly available at https://bocb-ai.github.io/.
Data Selection via Optimal Control for Language Models
Authors: name: Yuxian Gu, name: Li Dong, name: Hongning Wang, name: Yaru Hao, name: Qingxiu Dong, name: Furu Wei, name: Minlie Huang
Published: 2024-10-09 17:06:57

This work investigates the selection of high-quality pre-training data from
massive corpora to enhance LMs' capabilities for downstream usage. We formulate
data selection as a generalized Optimal Control problem, which can be solved
theoretically by Pontryagin's Maximum Principle (PMP), yielding a set of
necessary conditions that characterize the relationship between optimal data
selection and LM training dynamics. Based on these theoretical results, we
introduce PMP-based Data Selection (PDS), a framework that approximates optimal
data selection by solving the PMP conditions. In our experiments, we adopt PDS
to select data from CommmonCrawl and show that the PDS-selected corpus
accelerates the learning of LMs and constantly boosts their performance on a
wide range of downstream tasks across various model sizes. Moreover, the
benefits of PDS extend to ~400B models trained on ~10T tokens, as evidenced by
the extrapolation of the test loss curves according to the Scaling Laws. PDS
also improves data utilization when the pre-training data is limited, by
reducing the data demand by 1.8 times, which mitigates the quick exhaustion of
available web-crawled corpora. Our code, data, and model checkpoints can be
found in https://github.com/microsoft/LMOps/tree/main/data_selection.
Seeker: Enhancing Exception Handling in Code with LLM-based Multi-Agent
  Approach
Authors: name: Xuanming Zhang, name: Yuxuan Chen, name: Yuan Yuan, name: Minlie Huang
Published: 2024-10-09 14:45:45

In real world software development, improper or missing exception handling
can severely impact the robustness and reliability of code. Exception handling
mechanisms require developers to detect, capture, and manage exceptions
according to high standards, but many developers struggle with these tasks,
leading to fragile code. This problem is particularly evident in open source
projects and impacts the overall quality of the software ecosystem. To address
this challenge, we explore the use of large language models (LLMs) to improve
exception handling in code. Through extensive analysis, we identify three key
issues: Insensitive Detection of Fragile Code, Inaccurate Capture of Exception
Types, and Distorted Handling Solutions. These problems are widespread across
real world repositories, suggesting that robust exception handling practices
are often overlooked or mishandled. In response, we propose Seeker, a multi
agent framework inspired by expert developer strategies for exception handling.
Seeker uses agents: Scanner, Detector, Predator, Ranker, and Handler to assist
LLMs in detecting, capturing, and resolving exceptions more effectively. Our
work is the first systematic study on leveraging LLMs to enhance exception
handling practices, providing valuable insights for future improvements in code
reliability.
Temporal Reasoning Transfer from Text to Video
Authors: name: Lei Li, name: Yuanxin Liu, name: Linli Yao, name: Peiyuan Zhang, name: Chenxin An, name: Lean Wang, name: Xu Sun, name: Lingpeng Kong, name: Qi Liu
Published: 2024-10-08 16:10:29

Video Large Language Models (Video LLMs) have shown promising capabilities in
video comprehension, yet they struggle with tracking temporal changes and
reasoning about temporal relationships. While previous research attributed this
limitation to the ineffective temporal encoding of visual inputs, our
diagnostic study reveals that video representations contain sufficient
information for even small probing classifiers to achieve perfect accuracy.
Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning
capability stems from the underlying LLM's inherent difficulty with temporal
concepts, as evidenced by poor performance on textual temporal
question-answering tasks. Building on this discovery, we introduce the Textual
Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning
tasks in pure text format from existing image-text datasets, addressing the
scarcity of video samples with complex temporal scenarios. Remarkably, without
using any video data, T3 enhances LongVA-7B's temporal understanding, yielding
a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark,
which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video
samples. Additionally, the enhanced LongVA-7B model achieves competitive
performance on comprehensive video benchmarks. For example, it achieves a 49.7
accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful
large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further
analysis reveals a strong correlation between textual and video temporal task
performance, validating the efficacy of transferring temporal reasoning
abilities from text to video domains.
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark
  for Video Generation
Authors: name: Fanqing Meng, name: Jiaqi Liao, name: Xinyu Tan, name: Wenqi Shao, name: Quanfeng Lu, name: Kaipeng Zhang, name: Yu Cheng, name: Dianqi Li, name: Yu Qiao, name: Ping Luo
Published: 2024-10-07 17:56:04

Text-to-video (T2V) models like Sora have made significant strides in
visualizing complex prompts, which is increasingly viewed as a promising path
towards constructing the universal world simulator. Cognitive psychologists
believe that the foundation for achieving this goal is the ability to
understand intuitive physics. However, the capacity of these models to
accurately represent intuitive physics remains largely unexplored. To bridge
this gap, we introduce PhyGenBench, a comprehensive Physics
Generation Benchmark designed to evaluate physical
commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully
crafted prompts across 27 distinct physical laws, spanning four fundamental
domains, which could comprehensively assesses models' understanding of physical
commonsense. Alongside PhyGenBench, we propose a novel evaluation framework
called PhyGenEval. This framework employs a hierarchical evaluation structure
utilizing appropriate advanced vision-language models and large language models
to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can
conduct large-scale automated assessments of T2V models' understanding of
physical commonsense, which align closely with human feedback. Our evaluation
results and in-depth analysis demonstrate that current models struggle to
generate videos that comply with physical commonsense. Moreover, simply scaling
up models or employing prompt engineering techniques is insufficient to fully
address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We
hope this study will inspire the community to prioritize the learning of
physical commonsense in these models beyond entertainment applications. We will
release the data and codes at https://github.com/OpenGVLab/PhyGenBench
Inference Scaling for Long-Context Retrieval Augmented Generation
Authors: name: Zhenrui Yue, name: Honglei Zhuang, name: Aijun Bai, name: Kai Hui, name: Rolf Jagerman, name: Hansi Zeng, name: Zhen Qin, name: Dong Wang, name: Xuanhui Wang, name: Michael Bendersky
Published: 2024-10-06 03:42:15

The scaling of inference computation has unlocked the potential of
long-context large language models (LLMs) across diverse settings. For
knowledge-intensive tasks, the increased compute is often allocated to
incorporate more external knowledge. However, without effectively utilizing
such knowledge, solely expanding context does not always enhance performance.
In this work, we investigate inference scaling for retrieval augmented
generation (RAG), exploring strategies beyond simply increasing the quantity of
knowledge. We focus on two inference scaling strategies: in-context learning
and iterative prompting. These strategies provide additional flexibility to
scale test-time computation (e.g., by increasing retrieved documents or
generation steps), thereby enhancing LLMs' ability to effectively acquire and
utilize contextual information. We address two key questions: (1) How does RAG
performance benefit from the scaling of inference computation when optimally
configured? (2) Can we predict the optimal test-time compute allocation for a
given budget by modeling the relationship between RAG performance and inference
parameters? Our observations reveal that increasing inference computation leads
to nearly linear gains in RAG performance when optimally allocated, a
relationship we describe as the inference scaling laws for RAG. Building on
this, we further develop the computation allocation model to estimate RAG
performance across different inference configurations. The model predicts
optimal inference parameters under various computation constraints, which align
closely with the experimental results. By applying these optimal
configurations, we demonstrate that scaling inference compute on long-context
LLMs achieves up to 58.9% gains on benchmark datasets compared to standard RAG.
ε-VAE: Denoising as Visual Decoding
Authors: name: Long Zhao, name: Sanghyun Woo, name: Ziyu Wan, name: Yandong Li, name: Han Zhang, name: Boqing Gong, name: Hartwig Adam, name: Xuhui Jia, name: Ting Liu
Published: 2024-10-05 08:27:53

In generative modeling, tokenization simplifies complex data into compact,
structured representations, creating a more efficient, learnable space. For
high-dimensional visual data, it reduces redundancy and emphasizes key features
for high-quality generation. Current visual tokenization methods rely on a
traditional autoencoder framework, where the encoder compresses data into
latent representations, and the decoder reconstructs the original input. In
this work, we offer a new perspective by proposing denoising as decoding,
shifting from single-step reconstruction to iterative refinement. Specifically,
we replace the decoder with a diffusion process that iteratively refines noise
to recover the original image, guided by the latents provided by the encoder.
We evaluate our approach by assessing both reconstruction (rFID) and generation
quality (FID), comparing it to state-of-the-art autoencoding approach. We hope
this work offers new insights into integrating iterative generation and
autoencoding for improved compression and generation.