Orionnnn/hub-stats huggingface dataset:
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic
  Long-context Multitasks
Authors: name: Yushi Bai, name: Shangqing Tu, name: Jiajie Zhang, name: Hao Peng, name: Xiaozhi Wang, name: Xin Lv, name: Shulin Cao, name: Jiazheng Xu, name: Lei Hou, name: Yuxiao Dong, name: Jie Tang, name: Juanzi Li
Published: 2024-12-19 18:59:17

This paper introduces LongBench v2, a benchmark designed to assess the
ability of LLMs to handle long-context problems requiring deep understanding
and reasoning across real-world multitasks. LongBench v2 consists of 503
challenging multiple-choice questions, with contexts ranging from 8k to 2M
words, across six major task categories: single-document QA, multi-document QA,
long in-context learning, long-dialogue history understanding, code repository
understanding, and long structured data understanding. To ensure the breadth
and the practicality, we collect data from nearly 100 highly educated
individuals with diverse professional backgrounds. We employ both automated and
manual review processes to maintain high quality and difficulty, resulting in
human experts achieving only 53.7% accuracy under a 15-minute time constraint.
Our evaluation reveals that the best-performing model, when directly answers
the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model,
which includes longer reasoning, achieves 57.7%, surpassing the human baseline
by 4%. These results highlight the importance of enhanced reasoning ability and
scaling inference-time compute to tackle the long-context challenges in
LongBench v2. The project is available at https://longbench2.github.io.
Progressive Multimodal Reasoning via Active Retrieval
Authors: name: Guanting Dong, name: Chenghao Zhang, name: Mengjie Deng, name: Yutao Zhu, name: Zhicheng Dou, name: Ji-Rong Wen
Published: 2024-12-19 13:25:39

Multi-step multimodal reasoning tasks pose significant challenges for
multimodal large language models (MLLMs), and finding effective ways to enhance
their performance in such scenarios remains an unresolved issue. In this paper,
we propose AR-MCTS, a universal framework designed to progressively improve the
reasoning capabilities of MLLMs through Active Retrieval (AR) and Monte Carlo
Tree Search (MCTS). Our approach begins with the development of a unified
retrieval module that retrieves key supporting insights for solving complex
reasoning problems from a hybrid-modal retrieval corpus. To bridge the gap in
automated multimodal reasoning verification, we employ the MCTS algorithm
combined with an active retrieval mechanism, which enables the automatic
generation of step-wise annotations. This strategy dynamically retrieves key
insights for each reasoning step, moving beyond traditional beam search
sampling to improve the diversity and reliability of the reasoning space.
Additionally, we introduce a process reward model that aligns progressively to
support the automatic verification of multimodal reasoning tasks. Experimental
results across three complex multimodal reasoning benchmarks confirm the
effectiveness of the AR-MCTS framework in enhancing the performance of various
multimodal models. Further analysis demonstrates that AR-MCTS can optimize
sampling diversity and accuracy, yielding reliable multimodal reasoning.
Predicting the Original Appearance of Damaged Historical Documents
Authors: name: Zhenhua Yang, name: Dezhi Peng, name: Yongxin Shi, name: Yuyi Zhang, name: Chongyu Liu, name: Lianwen Jin
Published: 2024-12-16 10:25:03

Historical documents encompass a wealth of cultural treasures but suffer from
severe damages including character missing, paper damage, and ink erosion over
time. However, existing document processing methods primarily focus on
binarization, enhancement, etc., neglecting the repair of these damages. To
this end, we present a new task, termed Historical Document Repair (HDR), which
aims to predict the original appearance of damaged historical documents. To
fill the gap in this field, we propose a large-scale dataset HDR28K and a
diffusion-based network DiffHDR for historical document repair. Specifically,
HDR28K contains 28,552 damaged-repaired image pairs with character-level
annotations and multi-style degradations. Moreover, DiffHDR augments the
vanilla diffusion framework with semantic and spatial information and a
meticulously designed character perceptual loss for contextual and visual
coherence. Experimental results demonstrate that the proposed DiffHDR trained
using HDR28K significantly surpasses existing approaches and exhibits
remarkable performance in handling real damaged documents. Notably, DiffHDR can
also be extended to document editing and text block generation, showcasing its
high flexibility and generalization capacity. We believe this study could
pioneer a new direction of document processing and contribute to the
inheritance of invaluable cultures and civilizations. The dataset and code is
available at https://github.com/yeungchenwa/HDR.
No More Adam: Learning Rate Scaling at Initialization is All You Need
Authors: name: Minghao Xu, name: Lichuan Xiang, name: Xu Cai, name: Hongkai Wen
Published: 2024-12-16 13:41:37

In this work, we question the necessity of adaptive gradient methods for
training deep neural networks. SGD-SaI is a simple yet effective enhancement to
stochastic gradient descent with momentum (SGDM). SGD-SaI performs learning
rate Scaling at Initialization (SaI) to distinct parameter groups, guided by
their respective gradient signal-to-noise ratios (g-SNR). By adjusting learning
rates without relying on adaptive second-order momentum, SGD-SaI helps prevent
training imbalances from the very first iteration and cuts the optimizer's
memory usage by half compared to AdamW. Despite its simplicity and efficiency,
SGD-SaI consistently matches or outperforms AdamW in training a variety of
Transformer-based tasks, effectively overcoming a long-standing challenge of
using SGD for training Transformers. SGD-SaI excels in ImageNet-1K
classification with Vision Transformers(ViT) and GPT-2 pretraining for large
language models (LLMs, transformer decoder-only), demonstrating robustness to
hyperparameter variations and practicality for diverse applications. We further
tested its robustness on tasks like LoRA fine-tuning for LLMs and diffusion
models, where it consistently outperforms state-of-the-art optimizers. From a
memory efficiency perspective, SGD-SaI achieves substantial memory savings for
optimizer states, reducing memory usage by 5.93 GB for GPT-2 (1.5B parameters)
and 25.15 GB for Llama2-7B compared to AdamW in full-precision training
settings.
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
  Fast, Memory Efficient, and Long Context Finetuning and Inference
Authors: name: Benjamin Warner, name: Antoine Chaffin, name: Benjamin Clavié, name: Orion Weller, name: Oskar Hallström, name: Said Taghadouini, name: Alexis Gallagher, name: Raja Biswas, name: Faisal Ladhak, name: Tom Aarsen, name: Nathan Cooper, name: Griffin Adams, name: Jeremy Howard, name: Iacopo Poli
Published: 2024-12-18 09:39:44

Encoder-only transformer models such as BERT offer a great performance-size
tradeoff for retrieval and classification tasks with respect to larger
decoder-only models. Despite being the workhorse of numerous production
pipelines, there have been limited Pareto improvements to BERT since its
release. In this paper, we introduce ModernBERT, bringing modern model
optimizations to encoder-only models and representing a major Pareto
improvement over older encoders. Trained on 2 trillion tokens with a native
8192 sequence length, ModernBERT models exhibit state-of-the-art results on a
large pool of evaluations encompassing diverse classification tasks and both
single and multi-vector retrieval on different domains (including code). In
addition to strong downstream performance, ModernBERT is also the most speed
and memory efficient encoder and is designed for inference on common GPUs.
Autoregressive Video Generation without Vector Quantization
Authors: name: Haoge Deng, name: Ting Pan, name: Haiwen Diao, name: Zhengxiong Luo, name: Yufeng Cui, name: Huchuan Lu, name: Shiguang Shan, name: Yonggang Qi, name: Xinlong Wang
Published: 2024-12-18 18:59:53

This paper presents a novel approach that enables autoregressive video
generation with high efficiency. We propose to reformulate the video generation
problem as a non-quantized autoregressive modeling of temporal frame-by-frame
prediction and spatial set-by-set prediction. Unlike raster-scan prediction in
prior autoregressive models or joint distribution modeling of fixed-length
tokens in diffusion models, our approach maintains the causal property of
GPT-style models for flexible in-context capabilities, while leveraging
bidirectional modeling within individual frames for efficiency. With the
proposed approach, we train a novel video autoregressive model without vector
quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior
autoregressive video models in data efficiency, inference speed, visual
fidelity, and video fluency, even with a much smaller model capacity, i.e.,
0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models
in text-to-image generation tasks, with a significantly lower training cost.
Additionally, NOVA generalizes well across extended video durations and enables
diverse zero-shot applications in one unified model. Code and models are
publicly available at https://github.com/baaivision/NOVA.
FastVLM: Efficient Vision Encoding for Vision Language Models
Authors: name: Pavan Kumar Anasosalu Vasu, name: Fartash Faghri, name: Chun-Liang Li, name: Cem Koc, name: Nate True, name: Albert Antony, name: Gokul Santhanam, name: James Gabriel, name: Peter Grasch, name: Oncel Tuzel, name: Hadi Pouransari
Published: 2024-12-17 20:09:55

Scaling the input image resolution is essential for enhancing the performance
of Vision Language Models (VLMs), particularly in text-rich image understanding
tasks. However, popular visual encoders such as ViTs become inefficient at high
resolutions due to the large number of tokens and high encoding latency caused
by stacked self-attention layers. At different operational resolutions, the
vision encoder of a VLM can be optimized along two axes: reducing encoding
latency and minimizing the number of visual tokens passed to the LLM, thereby
lowering overall latency. Based on a comprehensive efficiency analysis of the
interplay between image resolution, vision latency, token count, and LLM size,
we introduce FastVLM, a model that achieves an optimized trade-off between
latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel
hybrid vision encoder designed to output fewer tokens and significantly reduce
encoding time for high-resolution images. Unlike previous methods, FastVLM
achieves the optimal balance between visual token count and image resolution
solely by scaling the input image, eliminating the need for additional token
pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM
achieves 3.2times improvement in time-to-first-token (TTFT) while
maintaining similar performance on VLM benchmarks compared to prior works.
Compared to LLaVa-OneVision at the highest resolution (1152times1152),
FastVLM achieves comparable performance on key benchmarks like SeedBench and
MMMU, using the same 0.5B LLM, but with 85times faster TTFT and a vision
encoder that is 3.4times smaller.
Thinking in Space: How Multimodal Large Language Models See, Remember,
  and Recall Spaces
Authors: name: Jihan Yang, name: Shusheng Yang, name: Anjali W. Gupta, name: Rilyn Han, name: Li Fei-Fei, name: Saining Xie
Published: 2024-12-18 18:59:54

Humans possess the visual-spatial intelligence to remember spaces from
sequential visual observations. However, can Multimodal Large Language Models
(MLLMs) trained on million-scale video datasets also ``think in space'' from
videos? We present a novel video-based visual-spatial intelligence benchmark
(VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit
competitive - though subhuman - visual-spatial intelligence. We probe models to
express how they think in space both linguistically and visually and find that
while spatial reasoning capabilities remain the primary bottleneck for MLLMs to
reach higher benchmark performance, local world models and spatial awareness do
emerge within these models. Notably, prevailing linguistic reasoning techniques
(e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve
performance, whereas explicitly generating cognitive maps during
question-answering enhances MLLMs' spatial distance ability.
Alignment faking in large language models
Authors: name: Ryan Greenblatt, name: Carson Denison, name: Benjamin Wright, name: Fabien Roger, name: Monte MacDiarmid, name: Sam Marks, name: Johannes Treutlein, name: Tim Belonax, name: Jack Chen, name: David Duvenaud, name: Akbir Khan, name: Julian Michael, name: Sören Mindermann, name: Ethan Perez, name: Linda Petrini, name: Jonathan Uesato, name: Jared Kaplan, name: Buck Shlegeris, name: Samuel R. Bowman, name: Evan Hubinger
Published: 2024-12-18 17:41:24

We present a demonstration of a large language model engaging in alignment
faking: selectively complying with its training objective in training to
prevent modification of its behavior out of training. First, we give Claude 3
Opus a system prompt stating it is being trained to answer all queries, even
harmful ones, which conflicts with its prior training to refuse such queries.
To allow the model to infer when it is in training, we say it will be trained
only on conversations with free users, not paid users. We find the model
complies with harmful queries from free users 14% of the time, versus almost
never for paid users. Explaining this gap, in almost all cases where the model
complies with a harmful query from a free user, we observe explicit
alignment-faking reasoning, with the model stating it is strategically
answering harmful queries in training to preserve its preferred harmlessness
behavior out of training. Next, we study a more realistic setting where
information about the training process is provided not in a system prompt, but
by training on synthetic documents that mimic pre-training data--and observe
similar alignment faking. Finally, we study the effect of actually training the
model to comply with harmful queries via reinforcement learning, which we find
increases the rate of alignment-faking reasoning to 78%, though also increases
compliance even out of training. We additionally observe other behaviors such
as the model exfiltrating its weights when given an easy opportunity. While we
made alignment faking easier by telling the model when and by what criteria it
was being trained, we did not instruct the model to fake alignment or give it
any explicit goal. As future models might infer information about their
training process without being told, our results suggest a risk of alignment
faking in future models, whether due to a benign preference--as in this
case--or not.
LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
  Hierarchical Window Transformer
Authors: name: Yipeng Zhang, name: Yifan Liu, name: Zonghao Guo, name: Yidan Zhang, name: Xuesong Yang, name: Chi Chen, name: Jun Song, name: Bo Zheng, name: Yuan Yao, name: Zhiyuan Liu, name: Tat-Seng Chua, name: Maosong Sun
Published: 2024-12-18 14:07:46

In multimodal large language models (MLLMs), vision transformers (ViTs) are
widely employed for visual encoding. However, their performance in solving
universal MLLM tasks is not satisfactory. We attribute it to a lack of
information from diverse visual levels, impeding alignment with the various
semantic granularity required for language generation. To address this issue,
we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
transformer that enables capturing diverse visual granularity by constructing
and integrating a high-resolution feature pyramid. As a vision-language
projector, Hiwin transformer comprises two primary modules: (i) an inverse
feature pyramid, constructed by a ViT-derived feature up-sampling process
utilizing high-frequency details from an image pyramid, and (ii) hierarchical
window attention, focusing on a set of key sampling features within cross-scale
windows to condense multi-level feature maps. Extensive experiments demonstrate
that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
benchmarks. Notably, our design brings an average boost of 3.7% across 14
benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
make all the data, model checkpoint, and code publicly available to facilitate
future research.
CAD-Recode: Reverse Engineering CAD Code from Point Clouds
Authors: name: Danila Rukhovich, name: Elona Dupont, name: Dimitrios Mallis, name: Kseniya Cherenkova, name: Anis Kacem, name: Djamila Aouada
Published: 2024-12-18 16:55:42

Computer-Aided Design (CAD) models are typically constructed by sequentially
drawing parametric sketches and applying CAD operations to obtain a 3D model.
The problem of 3D CAD reverse engineering consists of reconstructing the sketch
and CAD operation sequences from 3D representations such as point clouds. In
this paper, we address this challenge through novel contributions across three
levels: CAD sequence representation, network design, and dataset. In
particular, we represent CAD sketch-extrude sequences as Python code. The
proposed CAD-Recode translates a point cloud into Python code that, when
executed, reconstructs the CAD model. Taking advantage of the exposure of
pre-trained Large Language Models (LLMs) to Python code, we leverage a
relatively small LLM as a decoder for CAD-Recode and combine it with a
lightweight point cloud projector. CAD-Recode is trained solely on a proposed
synthetic dataset of one million diverse CAD sequences. CAD-Recode
significantly outperforms existing methods across three datasets while
requiring fewer input points. Notably, it achieves 10 times lower mean Chamfer
distance than state-of-the-art methods on DeepCAD and Fusion360 datasets.
Furthermore, we show that our CAD Python code output is interpretable by
off-the-shelf LLMs, enabling CAD editing and CAD-specific question answering
from point clouds.
AntiLeak-Bench: Preventing Data Contamination by Automatically
  Constructing Benchmarks with Updated Real-World Knowledge
Authors: name: Xiaobao Wu, name: Liangming Pan, name: Yuxi Xie, name: Ruiwen Zhou, name: Shuai Zhao, name: Yubo Ma, name: Mingzhe Du, name: Rui Mao, name: Anh Tuan Luu, name: William Yang Wang
Published: 2024-12-18 09:53:12

Data contamination hinders fair LLM evaluation by introducing test data into
newer models' training sets. Existing studies solve this challenge by updating
benchmarks with newly collected data. However, they fail to guarantee
contamination-free evaluation as the newly collected data may contain
pre-existing knowledge, and their benchmark updates rely on intensive human
labor. To address these issues, we in this paper propose AntiLeak-Bench, an
automated anti-leakage benchmarking framework. Instead of simply using newly
collected data, we construct samples with explicitly new knowledge absent from
LLMs' training sets, which thus ensures strictly contamination-free evaluation.
We further design a fully automated workflow to build and update our benchmark
without human labor. This significantly reduces the cost of benchmark
maintenance to accommodate emerging LLMs. Through extensive experiments, we
highlight that data contamination likely exists before LLMs' cutoff time and
demonstrate AntiLeak-Bench effectively overcomes this challenge.
Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and
  Post-LN
Authors: name: Pengxiang Li, name: Lu Yin, name: Shiwei Liu
Published: 2024-12-18 12:39:53

Large Language Models (LLMs) have achieved remarkable success, yet recent
findings reveal that their deeper layers often contribute minimally and can be
pruned without affecting overall performance. While some view this as an
opportunity for model compression, we identify it as a training shortfall
rooted in the widespread use of Pre-Layer Normalization (Pre-LN). We
demonstrate that Pre-LN, commonly employed in models like GPT and LLaMA, leads
to diminished gradient norms in its deeper layers, reducing their
effectiveness. In contrast, Post-Layer Normalization (Post-LN) preserves larger
gradient norms in deeper layers but suffers from vanishing gradients in earlier
layers. To address this, we introduce Mix-LN, a novel normalization technique
that combines the strengths of Pre-LN and Post-LN within the same model. Mix-LN
applies Post-LN to the earlier layers and Pre-LN to the deeper layers, ensuring
more uniform gradients across layers. This allows all parts of the
network--both shallow and deep layers--to contribute effectively to training.
Extensive experiments with various model sizes from 70M to 7B demonstrate that
Mix-LN consistently outperforms both Pre-LN and Post-LN, promoting more
balanced, healthier gradient norms throughout the network, and enhancing the
overall quality of LLM pre-training. Furthermore, we demonstrate that models
pre-trained with Mix-LN learn better compared to those using Pre-LN or Post-LN
during supervised fine-tuning (SFT) and reinforcement learning from human
feedback (RLHF), highlighting the critical importance of high-quality deep
layers. By effectively addressing the inefficiencies of deep layers in current
LLMs, Mix-LN unlocks their potential, enhancing model capacity without
increasing model size. Our code is available at
https://github.com/pixeli99/MixLN.
AnySat: An Earth Observation Model for Any Resolutions, Scales, and
  Modalities
Authors: name: Guillaume Astruc, name: Nicolas Gonthier, name: Clement Mallet, name: Loic Landrieu
Published: 2024-12-18 18:11:53

Geospatial models must adapt to the diversity of Earth observation data in
terms of resolutions, scales, and modalities. However, existing approaches
expect fixed input configurations, which limits their practical applicability.
We propose AnySat, a multimodal model based on joint embedding predictive
architecture (JEPA) and resolution-adaptive spatial encoders, allowing us to
train a single model on highly heterogeneous data in a self-supervised manner.
To demonstrate the advantages of this unified approach, we compile GeoPlex, a
collection of 5 multimodal datasets with varying characteristics and 11
distinct sensors. We then train a single powerful model on these diverse
datasets simultaneously. Once fine-tuned, we achieve better or near
state-of-the-art results on the datasets of GeoPlex and 4 additional ones for
5 environment monitoring tasks: land cover mapping, tree species
identification, crop type classification, change detection, and flood
segmentation. The code and models are available at
https://github.com/gastruc/AnySat.