Orionnnn/hub-stats huggingface dataset:
L^2M: Mutual Information Scaling Law for Long-Context Language
  Modeling
Authors: name: Zhuo Chen, name: Oriol Mayné i Comas, name: Zhuotao Jin, name: Di Luo, name: Marin Soljačić
Published: 2025-03-06 18:59:48

We rigorously establish a bipartite mutual information scaling law in natural
language that governs long-range dependencies. This scaling law, which we show
is distinct from and scales independently of the conventional two-point mutual
information, is the key to understanding long-context language modeling. Using
this scaling law, we formulate the Long-context Language Modeling (L^2M)
condition, which relates a model's capacity for effective long context length
modeling to the scaling of its latent state size for storing past information.
Our results are validated through experiments on both transformers and state
space models. This work establishes a theoretical foundation that guides the
development of large language models toward longer context lengths.
EgoLife: Towards Egocentric Life Assistant
Authors: name: Jingkang Yang, name: Shuai Liu, name: Hongming Guo, name: Yuhao Dong, name: Xiamengwei Zhang, name: Sicheng Zhang, name: Pengyun Wang, name: Zitang Zhou, name: Binzhu Xie, name: Ziyue Wang, name: Bei Ouyang, name: Zhengyu Lin, name: Marco Cominelli, name: Zhongang Cai, name: Yuanhan Zhang, name: Peiyuan Zhang, name: Fangzhou Hong, name: Joerg Widmer, name: Francesco Gringoli, name: Lei Yang, name: Bo Li, name: Ziwei Liu
Published: 2025-03-05 18:54:16

We introduce EgoLife, a project to develop an egocentric life assistant that
accompanies and enhances personal efficiency through AI-powered wearable
glasses. To lay the foundation for this assistant, we conducted a comprehensive
data collection study where six participants lived together for one week,
continuously recording their daily activities - including discussions,
shopping, cooking, socializing, and entertainment - using AI glasses for
multimodal egocentric video capture, along with synchronized third-person-view
video references. This effort resulted in the EgoLife Dataset, a comprehensive
300-hour egocentric, interpersonal, multiview, and multimodal daily life
dataset with intensive annotation. Leveraging this dataset, we introduce
EgoLifeQA, a suite of long-context, life-oriented question-answering tasks
designed to provide meaningful assistance in daily life by addressing practical
questions such as recalling past relevant events, monitoring health habits, and
offering personalized recommendations. To address the key technical challenges
of (1) developing robust visual-audio models for egocentric data, (2) enabling
identity recognition, and (3) facilitating long-context question answering over
extensive temporal information, we introduce EgoButler, an integrated system
comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on
egocentric datasets, achieving state-of-the-art performance on egocentric video
understanding. EgoRAG is a retrieval-based component that supports answering
ultra-long-context questions. Our experimental studies verify their working
mechanisms and reveal critical factors and bottlenecks, guiding future
improvements. By releasing our datasets, models, and benchmarks, we aim to
stimulate further research in egocentric AI assistants.
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding
  and Expert Reasoning Abilities
Authors: name: Sreyan Ghosh, name: Zhifeng Kong, name: Sonal Kumar, name: S Sakshi, name: Jaehyeon Kim, name: Wei Ping, name: Rafael Valle, name: Dinesh Manocha, name: Bryan Catanzaro
Published: 2025-03-06 00:10:26

Understanding and reasoning over non-speech sounds and music are crucial for
both humans and AI agents to interact effectively with their environments. In
this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM)
with advanced audio understanding and reasoning capabilities. AF2 leverages (i)
a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio
reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves
state-of-the-art performance with only a 3B parameter small language model,
surpassing large open-source and proprietary models across over 20 benchmarks.
Next, for the first time, we extend audio understanding to long audio segments
(30 secs to 5 mins) and propose LongAudio, a large and novel dataset for
training ALMs on long audio captioning and question-answering tasks.
Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed
LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio
understanding capabilities. We conduct extensive ablation studies to confirm
the efficacy of our approach. Project Website:
https://research.nvidia.com/labs/adlr/AF2/.
Understanding and Predicting Derailment in Toxic Conversations on GitHub
Authors: name: Mia Mohammad Imran, name: Robert Zita, name: Rebekah Copeland, name: Preetha Chatterjee, name: Rahat Rizvi Rahman, name: Kostadin Damevski
Published: 2025-03-04 02:01:37

Software projects thrive on the involvement and contributions of individuals
from different backgrounds. However, toxic language and negative interactions
can hinder the participation and retention of contributors and alienate
newcomers. Proactive moderation strategies aim to prevent toxicity from
occurring by addressing conversations that have derailed from their intended
purpose. This study aims to understand and predict conversational derailment
leading to toxicity on GitHub.
  To facilitate this research, we curate a novel dataset comprising 202 toxic
conversations from GitHub with annotated derailment points, along with 696
non-toxic conversations as a baseline. Based on this dataset, we identify
unique characteristics of toxic conversations and derailment points, including
linguistic markers such as second-person pronouns, negation terms, and tones of
Bitter Frustration and Impatience, as well as patterns in conversational
dynamics between project contributors and external participants.
  Leveraging these empirical observations, we propose a proactive moderation
approach to automatically detect and address potentially harmful conversations
before escalation. By utilizing modern LLMs, we develop a conversation
trajectory summary technique that captures the evolution of discussions and
identifies early signs of derailment. Our experiments demonstrate that LLM
prompts tailored to provide summaries of GitHub conversations achieve 69%
F1-Score in predicting conversational derailment, strongly improving over a set
of baseline approaches.
LLM as a Broken Telephone: Iterative Generation Distorts Information
Authors: name: Amr Mohamed, name: Mingmeng Geng, name: Michalis Vazirgiannis, name: Guokan Shang
Published: 2025-02-27 16:46:23

As large language models are increasingly responsible for online content,
concerns arise about the impact of repeatedly processing their own outputs.
Inspired by the "broken telephone" effect in chained human communication, this
study investigates whether LLMs similarly distort information through iterative
generation. Through translation-based experiments, we find that distortion
accumulates over time, influenced by language choice and chain complexity.
While degradation is inevitable, it can be mitigated through strategic
prompting techniques. These findings contribute to discussions on the long-term
effects of AI-mediated information propagation, raising important questions
about the reliability of LLM-generated content in iterative workflows.
PokéChamp: an Expert-level Minimax Language Agent
Authors: name: Seth Karten, name: Andy Luu Nguyen, name: Chi Jin
Published: 2025-03-06 05:06:27

We introduce Pok\'eChamp, a minimax agent powered by Large Language Models
(LLMs) for Pok\'emon battles. Built on a general framework for two-player
competitive games, Pok\'eChamp leverages the generalist capabilities of LLMs to
enhance minimax tree search. Specifically, LLMs replace three key modules: (1)
player action sampling, (2) opponent modeling, and (3) value function
estimation, enabling the agent to effectively utilize gameplay history and
human knowledge to reduce the search space and address partial observability.
Notably, our framework requires no additional LLM training. We evaluate
Pok\'eChamp in the popular Gen 9 OU format. When powered by GPT-4o, it achieves
a win rate of 76% against the best existing LLM-based bot and 84% against the
strongest rule-based bot, demonstrating its superior performance. Even with an
open-source 8-billion-parameter Llama 3.1 model, Pok\'eChamp consistently
outperforms the previous best LLM-based bot, Pok\'ellmon powered by GPT-4o,
with a 64% win rate. Pok\'eChamp attains a projected Elo of 1300-1500 on the
Pok\'emon Showdown online ladder, placing it among the top 30%-10% of human
players. In addition, this work compiles the largest real-player Pok\'emon
battle dataset, featuring over 3 million games, including more than 500k
high-Elo matches. Based on this dataset, we establish a series of battle
benchmarks and puzzles to evaluate specific battling skills. We further provide
key updates to the local game engine. We hope this work fosters further
research that leverage Pok\'emon battle as benchmark to integrate LLM
technologies with game-theoretic algorithms addressing general multiagent
problems. Videos, code, and dataset available at
https://sites.google.com/view/pokechamp-llm.
Token-Efficient Long Video Understanding for Multimodal LLMs
Authors: name: Jindong Jiang, name: Xiuyu Li, name: Zhijian Liu, name: Muyang Li, name: Guo Chen, name: Zhiqi Li, name: De-An Huang, name: Guilin Liu, name: Zhiding Yu, name: Kurt Keutzer, name: Sungjin Ahn, name: Jan Kautz, name: Hongxu Yin, name: Yao Lu, name: Song Han, name: Wonmin Byeon
Published: 2025-03-06 06:17:38

Recent advances in video-based multimodal large language models (Video-LLMs)
have significantly improved video understanding by processing videos as
sequences of image frames. However, many existing methods treat frames
independently in the vision backbone, lacking explicit temporal modeling, which
limits their ability to capture dynamic patterns and efficiently handle long
videos. To address these limitations, we introduce STORM
(Spatiotemporal TOken Reduction for
Multimodal LLMs), a novel architecture incorporating a dedicated
temporal encoder between the image encoder and the LLM. Our temporal encoder
leverages the Mamba State Space Model to integrate temporal information into
image tokens, generating enriched representations that preserve inter-frame
dynamics across the entire video sequence. This enriched encoding not only
enhances video reasoning capabilities but also enables effective token
reduction strategies, including test-time sampling and training-based temporal
and spatial pooling, substantially reducing computational demands on the LLM
without sacrificing key temporal information. By integrating these techniques,
our approach simultaneously reduces training and inference latency while
improving performance, enabling efficient and robust video understanding over
extended temporal contexts. Extensive evaluations show that STORM achieves
state-of-the-art results across various long video understanding benchmarks
(more than 5\% improvement on MLVU and LongVideoBench) while reducing the
computation costs by up to 8times and the decoding latency by
2.4-2.9times for the fixed numbers of input frames. Project page is
available at https://research.nvidia.com/labs/lpr/storm
The Best of Both Worlds: Integrating Language Models and Diffusion
  Models for Video Generation
Authors: name: Aoxiong Yin, name: Kai Shen, name: Yichong Leng, name: Xu Tan, name: Xinyu Zhou, name: Juncheng Li, name: Siliang Tang
Published: 2025-03-06 16:53:14

Recent advancements in text-to-video (T2V) generation have been driven by two
competing paradigms: autoregressive language models and diffusion models.
However, each paradigm has intrinsic limitations: language models struggle with
visual quality and error accumulation, while diffusion models lack semantic
understanding and causal modeling. In this work, we propose LanDiff, a hybrid
framework that synergizes the strengths of both paradigms through
coarse-to-fine generation. Our architecture introduces three key innovations:
(1) a semantic tokenizer that compresses 3D visual features into compact 1D
discrete representations through efficient semantic compression, achieving a
sim14,000times compression ratio; (2) a language model that generates
semantic tokens with high-level semantic relationships; (3) a streaming
diffusion model that refines coarse semantics into high-fidelity videos.
Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the
VBench T2V benchmark, surpassing the state-of-the-art open-source models
Hunyuan Video (13B) and other commercial models such as Sora, Keling, and
Hailuo. Furthermore, our model also achieves state-of-the-art performance in
long video generation, surpassing other open-source models in this field. Our
demo can be viewed at https://landiff.github.io/.
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid
  Normalization
Authors: name: Zhijian Zhuo, name: Yutao Zeng, name: Ya Wang, name: Sijun Zhang, name: Jian Yang, name: Xiaoqing Li, name: Xun Zhou, name: Jinwen Ma
Published: 2025-03-06 16:40:48

Transformers have become the de facto architecture for a wide range of
machine learning tasks, particularly in large language models (LLMs). Despite
their remarkable performance, challenges remain in training deep transformer
networks, especially regarding the location of layer normalization. While
Pre-Norm structures facilitate easier training due to their more prominent
identity path, they often yield suboptimal performance compared to Post-Norm.
In this paper, we propose HybridNorm, a straightforward yet
effective hybrid normalization strategy that integrates the advantages of both
Pre-Norm and Post-Norm approaches. Specifically, HybridNorm employs QKV
normalization within the attention mechanism and Post-Norm in the feed-forward
network (FFN) of each transformer block. This design not only stabilizes
training but also enhances performance, particularly in the context of LLMs.
Comprehensive experiments in both dense and sparse architectures show that
HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches,
achieving state-of-the-art results across various benchmarks. These findings
highlight the potential of HybridNorm as a more stable and effective technique
for improving the training and performance of deep transformer models. %Code
will be made publicly available. Code is available at
https://github.com/BryceZhuo/HybridNorm.
START: Self-taught Reasoner with Tools
Authors: name: Chengpeng Li, name: Mingfeng Xue, name: Zhenru Zhang, name: Jiaxi Yang, name: Beichen Zhang, name: Xiang Wang, name: Bowen Yu, name: Binyuan Hui, name: Junyang Lin, name: Dayiheng Liu
Published: 2025-03-06 17:11:51

Large reasoning models (LRMs) like OpenAI-o1 and DeepSeek-R1 have
demonstrated remarkable capabilities in complex reasoning tasks through the
utilization of long Chain-of-thought (CoT). However, these models often suffer
from hallucinations and inefficiencies due to their reliance solely on internal
reasoning processes. In this paper, we introduce START (Self-Taught Reasoner
with Tools), a novel tool-integrated long CoT reasoning LLM that significantly
enhances reasoning capabilities by leveraging external tools. Through code
execution, START is capable of performing complex computations, self-checking,
exploring diverse methods, and self-debugging, thereby addressing the
limitations of LRMs. The core innovation of START lies in its self-learning
framework, which comprises two key techniques: 1) Hint-infer: We demonstrate
that inserting artificially designed hints (e.g., ``Wait, maybe using Python
here is a good idea.'') during the inference process of a LRM effectively
stimulates its ability to utilize external tools without the need for any
demonstration data. Hint-infer can also serve as a simple and effective
sequential test-time scaling method; 2) Hint Rejection Sampling Fine-Tuning
(Hint-RFT): Hint-RFT combines Hint-infer and RFT by scoring, filtering, and
modifying the reasoning trajectories with tool invocation generated by a LRM
via Hint-infer, followed by fine-tuning the LRM. Through this framework, we
have fine-tuned the QwQ-32B model to achieve START. On PhD-level science QA
(GPQA), competition-level math benchmarks (AMC23, AIME24, AIME25), and the
competition-level code benchmark (LiveCodeBench), START achieves accuracy rates
of 63.6%, 95.0%, 66.7%, 47.1%, and 47.3%, respectively. It significantly
outperforms the base QwQ-32B and achieves performance comparable to the
state-of-the-art open-weight model R1-Distill-Qwen-32B and the proprietary
model o1-Preview.
FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
Authors: name: Ziyi Yang, name: Fanqi Wan, name: Longguang Zhong, name: Canbin Huang, name: Guosheng Liang, name: Xiaojun Quan
Published: 2025-03-06 09:03:36

We introduce FuseChat-3.0, a suite of large language models (LLMs) developed
by integrating the strengths of heterogeneous source LLMs into more compact
target LLMs. Our source models include the powerful Gemma-2-27B-it,
Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.
For target models, we focus on three widely-used smaller
variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along
with two ultra-compact options, Llama-3.2-3B-Instruct and
Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source
models, we develop a specialized data construction protocol tailored to various
tasks and domains. The FuseChat-3.0 training pipeline consists of two key
stages: (1) supervised fine-tuning (SFT) to align the target and source model
distributions, and (2) Direct Preference Optimization (DPO) to apply
preferences from multiple source LLMs to fine-tune the target model. The
resulting FuseChat-3.0 models exhibit significant performance gains across
tasks such as instruction following, general knowledge, mathematics, and
coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target
model, our fusion approach achieves an average improvement of 6.8 points across
14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and
30.1 points on the instruction-following benchmarks AlpacaEval-2 and
Arena-Hard, respectively. Our code, models, and datasets are available at
https://github.com/SLIT-AI/FuseChat-3.0.
Dedicated Feedback and Edit Models Empower Inference-Time Scaling for
  Open-Ended General-Domain Tasks
Authors: name: Zhilin Wang, name: Jiaqi Zeng, name: Olivier Delalleau, name: Daniel Egert, name: Ellie Evans, name: Hoo-Chang Shin, name: Felipe Soares, name: Yi Dong, name: Oleksii Kuchaiev
Published: 2025-03-06 12:30:24

Inference-Time Scaling has been critical to the success of recent models such
as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for
inference-time scaling require tasks to have answers that can be verified,
limiting their application to domains such as math, coding and logical
reasoning. We take inspiration from how humans make first attempts, ask for
detailed feedback from others and make improvements based on such feedback
across a wide spectrum of open-ended endeavors. To this end, we collect data
for and train dedicated Feedback and Edit Models that are capable of performing
inference-time scaling for open-ended general-domain tasks. In our setup, one
model generates an initial response, which are given feedback by a second
model, that are then used by a third model to edit the response. We show that
performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo
can be boosted by scaling the number of initial response drafts, effective
feedback and edited responses. When scaled optimally, our setup based on 70B
models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7
as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and
DeepSeek R1 with 92.3.
HoT: Highlighted Chain of Thought for Referencing Supporting Facts from
  Inputs
Authors: name: Tin Nguyen, name: Logan Bolton, name: Mohammad Reza Taesiri, name: Anh Totti Nguyen
Published: 2025-03-03 19:26:04

An Achilles heel of Large Language Models (LLMs) is their tendency to
hallucinate non-factual statements. A response mixed of factual and non-factual
statements poses a challenge for humans to verify and accurately base their
decisions on. To combat this problem, we propose Highlighted Chain-of-Thought
Prompting (HoT), a technique for prompting LLMs to generate responses with XML
tags that ground facts to those provided in the query. That is, given an input
question, LLMs would first re-format the question to add XML tags highlighting
key facts, and then, generate a response with highlights over the facts
referenced from the input. Interestingly, in few-shot settings, HoT outperforms
vanilla chain of thought prompting (CoT) on a wide range of 17 tasks from
arithmetic, reading comprehension to logical reasoning. When asking humans to
verify LLM responses, highlights help time-limited participants to more
accurately and efficiently recognize when LLMs are correct. Yet, surprisingly,
when LLMs are wrong, HoTs tend to make users believe that an answer is correct.
Reliable and Efficient Multi-Agent Coordination via Graph Neural Network
  Variational Autoencoders
Authors: name: Yue Meng, name: Nathalie Majcherczyk, name: Wenliang Liu, name: Scott Kiesel, name: Chuchu Fan, name: Federico Pecora
Published: 2025-03-04 19:20:11

Multi-agent coordination is crucial for reliable multi-robot navigation in
shared spaces such as automated warehouses. In regions of dense robot traffic,
local coordination methods may fail to find a deadlock-free solution. In these
scenarios, it is appropriate to let a central unit generate a global schedule
that decides the passing order of robots. However, the runtime of such
centralized coordination methods increases significantly with the problem
scale. In this paper, we propose to leverage Graph Neural Network Variational
Autoencoders (GNN-VAE) to solve the multi-agent coordination problem at scale
faster than through centralized optimization. We formulate the coordination
problem as a graph problem and collect ground truth data using a Mixed-Integer
Linear Program (MILP) solver. During training, our learning framework encodes
good quality solutions of the graph problem into a latent space. At inference
time, solution samples are decoded from the sampled latent variables, and the
lowest-cost sample is selected for coordination. Finally, the feasible proposal
with the highest performance index is selected for the deployment. By
construction, our GNN-VAE framework returns solutions that always respect the
constraints of the considered coordination problem. Numerical results show that
our approach trained on small-scale problems can achieve high-quality solutions
even for large-scale problems with 250 robots, being much faster than other
baselines. Project page: https://mengyuest.github.io/gnn-vae-coord