Orionnnn/hub-stats huggingface dataset:
UniTok: A Unified Tokenizer for Visual Generation and Understanding
Authors: name: Chuofan Ma, name: Yi Jiang, name: Junfeng Wu, name: Jihan Yang, name: Xin Yu, name: Zehuan Yuan, name: Bingyue Peng, name: Xiaojuan Qi
Published: 2025-02-27 17:47:01

The representation disparity between visual generation and understanding
imposes a critical gap in integrating these capabilities into a single
framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer
that encodes fine-grained details for generation while also capturing
high-level semantics for understanding. Despite recent studies have shown that
these objectives could induce loss conflicts in training, we reveal that the
underlying bottleneck stems from limited representational capacity of discrete
tokens. We address this by introducing multi-codebook quantization, which
divides vector quantization with several independent sub-codebooks to expand
the latent feature space, while avoiding training instability caused by
overlarge codebooks. Our method significantly raises the upper limit of unified
discrete tokenizers to match or even surpass domain-specific continuous
tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus
0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on
ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.
CODESYNC: Synchronizing Large Language Models with Dynamic Code
  Evolution at Scale
Authors: name: Chenlong Wang, name: Zhaoyang Chu, name: Zhengxiang Cheng, name: Xuyi Yang, name: Kaiyue Qiu, name: Yao Wan, name: Zhou Zhao, name: Xuanhua Shi, name: Dongping Chen
Published: 2025-02-23 16:46:18

Large Language Models (LLMs) have exhibited exceptional performance in
software engineering yet face challenges in adapting to continually evolving
code knowledge, particularly regarding the frequent updates of third-party
library APIs. This limitation, stemming from static pre-training datasets,
often results in non-executable code or implementations with suboptimal safety
and efficiency. To this end, this paper introduces CODESYNC, a data engine for
identifying outdated code patterns and collecting real-time code knowledge
updates from Python third-party libraries. Building upon CODESYNC, we develop
CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay
synchronized with code evolution, which covers real-world updates for 220 APIs
from six Python libraries. Our benchmark offers 3,300 test cases across three
evaluation tasks and an update-aware instruction tuning dataset consisting of
2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs
reveal that they struggle with dynamic code evolution, even with the support of
advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe
that our benchmark can offer a strong foundation for the development of more
effective methods for real-time code knowledge updating in the future. The
experimental code and dataset are publicly available at:
https://github.com/Lucky-voyage/Code-Sync.
SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning
Authors: name: Zexiong Ma, name: Chao Peng, name: Pengfei Gao, name: Xiangxin Meng, name: Yanzhen Zou, name: Bing Xie
Published: 2025-02-27 14:19:45

Mainstream issue-resolving frameworks predominantly rely on commercial
models, leading to high costs and privacy concerns. Existing training
approaches for issue resolving struggle with poor generalization and fail to
fully leverage open-source development resources. We propose Subtask-oriented
Reinforced Fine-Tuning (SoRFT), a novel training approach to enhance the issue
resolving capability of LLMs. We decomposes issue resolving into structured
subtasks: file localization, function localization, line localization, and code
edit generation. SoRFT consists of two training stages: (1) rejection-sampled
supervised fine-tuning, Chain of Thought (CoT) data is filtered using
ground-truth before fine-tuning the LLM, and (2) rule-based reinforcement
learning, which leverages PPO with ground-truth based rewards. We evaluate the
SoRFT-trained model on SWE-Bench Verified and SWE-Bench Lite, achieving
state-of-the-art (SOTA) performance among open-source models (e.g., resolve
21.4% issues on SWE-Bench Verified with SoRFT-Qwen-7B). The experimental
results demonstrate that SoRFT significantly enhances issue-resolving
performance, improves model generalization, and provides a cost-efficient
alternative to commercial models.
R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Authors: name: Zhongyang Li, name: Ziyue Li, name: Tianyi Zhou
Published: 2025-02-27 18:59:32

In large multimodal models (LMMs), the perception of non-language modalities
(e.g., visual representations) is usually not on par with the large language
models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on
challenging downstream tasks. This weakness has been recently mitigated by
replacing the vision encoder with a mixture-of-experts (MoE), which provides
rich, multi-granularity, and diverse representations required by diverse
downstream tasks. The performance of multimodal MoE largely depends on its
router, which reweights and mixes the representations of different experts for
each input. However, we find that the end-to-end trained router does not always
produce the optimal routing weights for every test sample. To bridge the gap,
we propose a novel and efficient method "Re-Routing in Test-Time(R2-T2) that
locally optimizes the vector of routing weights in test-time by moving it
toward those vectors of the correctly predicted samples in a neighborhood of
the test sample. We propose three R2-T2 strategies with different optimization
objectives and neighbor-search spaces. R2-T2 consistently and greatly improves
state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks,
without training any base-model parameters.
LongRoPE2: Near-Lossless LLM Context Window Scaling
Authors: name: Ning Shang, name: Li Lyna Zhang, name: Siyuan Wang, name: Gaokai Zhang, name: Gilsinia Lopez, name: Fan Yang, name: Weizhu Chen, name: Mao Yang
Published: 2025-02-27 13:41:07

LongRoPE2 is a novel approach that extends the effective context window of
pre-trained large language models (LLMs) to the target length, while preserving
the performance on the original shorter context window. This is achieved by
three contributions: (1) a hypothesis that insufficient training in higher RoPE
dimensions contributes to the persistent out-of-distribution (OOD) issues
observed in existing methods; (2) an effective RoPE rescaling algorithm that
adopts evolutionary search guided by "needle-driven" perplexity to address the
insufficient training problem; (3) a mixed context window training approach
that fine-tunes model weights to adopt rescaled RoPE for long-context sequences
while preserving the short-context performance with the original RoPE.
Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks
validate the hypothesis and demonstrate the effectiveness of LongRoPE2.
Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context
length while retaining over 98.5% of short-context performance, using only 10B
tokens -- 80x fewer than Meta's approach, which fails to reach the target
effective context length. Code will be available at
https://github.com/microsoft/LongRoPE.
Self-rewarding correction for mathematical reasoning
Authors: name: Wei Xiong, name: Hanning Zhang, name: Chenlu Ye, name: Lichang Chen, name: Nan Jiang, name: Tong Zhang
Published: 2025-02-26 23:01:16

We study self-rewarding reasoning large language models (LLMs), which can
simultaneously generate step-by-step reasoning and evaluate the correctness of
their outputs during the inference time-without external feedback. This
integrated approach allows a single model to independently guide its reasoning
process, offering computational advantages for model deployment. We
particularly focus on the representative task of self-correction, where models
autonomously detect errors in their responses, revise outputs, and decide when
to terminate iterative refinement loops. To enable this, we propose a
two-staged algorithmic framework for constructing self-rewarding reasoning
models using only self-generated data. In the first stage, we employ sequential
rejection sampling to synthesize long chain-of-thought trajectories that
incorporate both self-rewarding and self-correction mechanisms. Fine-tuning
models on these curated data allows them to learn the patterns of
self-rewarding and self-correction. In the second stage, we further enhance the
models' ability to assess response accuracy and refine outputs through
reinforcement learning with rule-based signals. Experiments with Llama-3 and
Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction
capabilities and achieves performance comparable to systems that rely on
external reward models.
Adapting Automatic Speech Recognition for Accented Air Traffic Control
  Communications
Authors: name: Marcus Yu Zhe Wee, name: Justin Juin Hng Wong, name: Lynus Lim, name: Joe Yu Wei Tan, name: Prannaya Gupta, name: Dillion Lim, name: En Hao Tew, name: Aloysius Keng Siew Han, name: Yong Zhi Lim
Published: 2025-02-27 17:35:59

Effective communication in Air Traffic Control (ATC) is critical to
maintaining aviation safety, yet the challenges posed by accented English
remain largely unaddressed in Automatic Speech Recognition (ASR) systems.
Existing models struggle with transcription accuracy for Southeast
Asian-accented (SEA-accented) speech, particularly in noisy ATC environments.
This study presents the development of ASR models fine-tuned specifically for
Southeast Asian accents using a newly created dataset. Our research achieves
significant improvements, achieving a Word Error Rate (WER) of 0.0982 or 9.82%
on SEA-accented ATC speech. Additionally, the paper highlights the importance
of region-specific datasets and accent-focused training, offering a pathway for
deploying ASR systems in resource-constrained military operations. The findings
emphasize the need for noise-robust training techniques and region-specific
datasets to improve transcription accuracy for non-Western accents in ATC
communications.
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Authors: name: Yuntao Du, name: Kailin Jiang, name: Zhi Gao, name: Chenrui Shi, name: Zilong Zheng, name: Siyuan Qi, name: Qing Li
Published: 2025-02-27 08:21:28

Knowledge editing techniques have emerged as essential tools for updating the
factual knowledge of large language models (LLMs) and multimodal models (LMMs),
allowing them to correct outdated or inaccurate information without retraining
from scratch. However, existing benchmarks for multimodal knowledge editing
primarily focus on entity-level knowledge represented as simple triplets, which
fail to capture the complexity of real-world multimodal information. To address
this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge
Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse
visual knowledge in real-world scenarios. MMKE-Bench addresses these
limitations by incorporating three types of editing tasks: visual entity
editing, visual semantic editing, and user-specific editing. Besides,
MMKE-Bench uses free-form natural language to represent and edit knowledge,
offering a more flexible and effective format. The benchmark consists of 2,940
pieces of knowledge and 8,363 images across 33 broad categories, with
evaluation questions automatically generated and human-verified. We assess five
state-of-the-art knowledge editing methods on three prominent LMMs, revealing
that no method excels across all criteria, and that visual and user-specific
edits are particularly challenging. MMKE-Bench sets a new standard for
evaluating the robustness of multimodal knowledge editing techniques, driving
progress in this rapidly evolving field.
Towards Optimal Multi-draft Speculative Decoding
Authors: name: Zhengmian Hu, name: Tong Zheng, name: Vignesh Viswanathan, name: Ziyi Chen, name: Ryan A. Rossi, name: Yihan Wu, name: Dinesh Manocha, name: Heng Huang
Published: 2025-02-26 03:22:44

Large Language Models (LLMs) have become an indispensable part of natural
language processing tasks. However, autoregressive sampling has become an
efficiency bottleneck. Multi-Draft Speculative Decoding (MDSD) is a recent
approach where, when generating each token, a small draft model generates
multiple drafts, and the target LLM verifies them in parallel, ensuring that
the final output conforms to the target model distribution. The two main design
choices in MDSD are the draft sampling method and the verification algorithm.
For a fixed draft sampling method, the optimal acceptance rate is a solution to
an optimal transport problem, but the complexity of this problem makes it
difficult to solve for the optimal acceptance rate and measure the gap between
existing verification algorithms and the theoretical upper bound. This paper
discusses the dual of the optimal transport problem, providing a way to
efficiently compute the optimal acceptance rate. For the first time, we measure
the theoretical upper bound of MDSD efficiency for vocabulary sizes in the
thousands and quantify the gap between existing verification algorithms and
this bound. We also compare different draft sampling methods based on their
optimal acceptance rates. Our results show that the draft sampling method
strongly influences the optimal acceptance rate, with sampling without
replacement outperforming sampling with replacement. Additionally, existing
verification algorithms do not reach the theoretical upper bound for both
without replacement and with replacement sampling. Our findings suggest that
carefully designed draft sampling methods can potentially improve the optimal
acceptance rate and enable the development of verification algorithms that
closely match the theoretical upper bound.
FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in
  LLMs Elicits Effective Personalization to Real Users
Authors: name: Anikait Singh, name: Sheryl Hsu, name: Kyle Hsu, name: Eric Mitchell, name: Stefano Ermon, name: Tatsunori Hashimoto, name: Archit Sharma, name: Chelsea Finn
Published: 2025-02-26 17:08:46

Effective personalization of LLMs is critical for a broad range of
user-interfacing applications such as virtual assistants and content curation.
Inspired by the strong in-context learning capabilities of LLMs, we propose
Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a
meta-learning problem. Under this framework, an LLM learns to quickly adapt to
a user via a few labeled preferences from that user, constructing a
personalized reward function for them. Additionally, since real-world
preference data is scarce and challenging to collect at scale, we propose
careful design choices to construct synthetic preference datasets for
personalization, generating over 1M synthetic personalized preferences using
publicly available LLMs. In particular, to successfully transfer from synthetic
data to real users, we find it crucial for the data to exhibit both high
diversity and coherent, self-consistent structure. We evaluate FSPO on
personalized open-ended generation for up to 1,500 synthetic users across
across three domains: movie reviews, pedagogical adaptation based on
educational background, and general question answering, along with a controlled
human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in
generating responses that are personalized to synthetic users and a 72% winrate
with real human users in open-ended question answering.
Drop-Upcycling: Training Sparse Mixture of Experts with Partial
  Re-initialization
Authors: name: Taishi Nakamura, name: Takuya Akiba, name: Kazuki Fujii, name: Yusuke Oda, name: Rio Yokota, name: Jun Suzuki
Published: 2025-02-26 16:06:36

The Mixture of Experts (MoE) architecture reduces the training and inference
cost significantly compared to a dense model of equivalent capacity. Upcycling
is an approach that initializes and trains an MoE model using a pre-trained
dense model. While upcycling leads to initial performance gains, the training
progresses slower than when trained from scratch, leading to suboptimal
performance in the long term. We propose Drop-Upcycling - a method that
effectively addresses this problem. Drop-Upcycling combines two seemingly
contradictory approaches: utilizing the knowledge of pre-trained dense models
while statistically re-initializing some parts of the weights. This approach
strategically promotes expert specialization, significantly enhancing the MoE
model's efficiency in knowledge acquisition. Extensive large-scale experiments
demonstrate that Drop-Upcycling significantly outperforms previous MoE
construction methods in the long term, specifically when training on hundreds
of billions of tokens or more. As a result, our MoE model with 5.9B active
parameters achieves comparable performance to a 13B dense model in the same
model family, while requiring approximately 1/4 of the training FLOPs. All
experimental resources, including source code, training data, model checkpoints
and logs, are publicly available to promote reproducibility and future research
on MoE.
Rank1: Test-Time Compute for Reranking in Information Retrieval
Authors: name: Orion Weller, name: Kathryn Ricci, name: Eugene Yang, name: Andrew Yates, name: Dawn Lawrie, name: Benjamin Van Durme
Published: 2025-02-25 18:14:06

We introduce Rank1, the first reranking model trained to take advantage of
test-time compute. Rank1 demonstrates the applicability within retrieval of
using a reasoning language model (i.e. OpenAI's o1, Deepseek's R1, etc.) for
distillation in order to rapidly improve the performance of a smaller model. We
gather and open-source a dataset of more than 600,000 examples of R1 reasoning
traces from queries and passages in MS MARCO. Models trained on this dataset
show: (1) state-of-the-art performance on advanced reasoning and instruction
following datasets; (2) work remarkably well out of distribution due to the
ability to respond to user-input prompts; and (3) have explainable reasoning
chains that can be given to users or RAG-based systems. Further, we demonstrate
that quantized versions of these models retain strong performance while using
less compute/memory. Overall, Rank1 shows that test-time compute allows for a
fundamentally new type of explainable and performant reranker model for search.
DOEI: Dual Optimization of Embedding Information for Attention-Enhanced
  Class Activation Maps
Authors: name: Hongjie Zhu, name: Zeyu Zhang, name: Guansong Pang, name: Xu Wang, name: Shimin Wen, name: Yu Bai, name: Daji Ergu, name: Ying Cai, name: Yang Zhao
Published: 2025-02-21 19:06:01

Weakly supervised semantic segmentation (WSSS) typically utilizes limited
semantic annotations to obtain initial Class Activation Maps (CAMs). However,
due to the inadequate coupling between class activation responses and semantic
information in high-dimensional space, the CAM is prone to object co-occurrence
or under-activation, resulting in inferior recognition accuracy. To tackle this
issue, we propose DOEI, Dual Optimization of Embedding Information, a novel
approach that reconstructs embedding representations through semantic-aware
attention weight matrices to optimize the expression capability of embedding
information. Specifically, DOEI amplifies tokens with high confidence and
suppresses those with low confidence during the class-to-patch interaction.
This alignment of activation responses with semantic information strengthens
the propagation and decoupling of target features, enabling the generated
embeddings to more accurately represent target features in high-level semantic
space. In addition, we propose a hybrid-feature alignment module in DOEI that
combines RGB values, embedding-guided features, and self-attention weights to
increase the reliability of candidate tokens. Comprehensive experiments show
that DOEI is an effective plug-and-play module that empowers state-of-the-art
visual transformer-based WSSS models to significantly improve the quality of
CAMs and segmentation performance on popular benchmarks, including PASCAL VOC
(+3.6%, +1.5%, +1.2% mIoU) and MS COCO (+1.2%, +1.6% mIoU). Code will be
available at https://github.com/AIGeeksGroup/DOEI.
Project Alexandria: Towards Freeing Scientific Knowledge from Copyright
  Burdens via LLMs
Authors: name: Christoph Schuhmann, name: Gollam Rabby, name: Ameya Prabhu, name: Tawsif Ahmed, name: Andreas Hochlehnert, name: Huu Nguyen, name: Nick Akinci Heidrich, name: Ludwig Schmidt, name: Robert Kaczmarczyk, name: Sören Auer, name: Jenia Jitsev, name: Matthias Bethge
Published: 2025-02-26 18:56:52

Paywalls, licenses and copyright rules often restrict the broad dissemination
and reuse of scientific knowledge. We take the position that it is both legally
and technically feasible to extract the scientific knowledge in scholarly
texts. Current methods, like text embeddings, fail to reliably preserve factual
content, and simple paraphrasing may not be legally sound. We urge the
community to adopt a new idea: convert scholarly documents into Knowledge Units
using LLMs. These units use structured data capturing entities, attributes and
relationships without stylistic content. We provide evidence that Knowledge
Units: (1) form a legally defensible framework for sharing knowledge from
copyrighted research texts, based on legal analyses of German copyright law and
U.S. Fair Use doctrine, and (2) preserve most (~95%) factual knowledge from
original text, measured by MCQ performance on facts from the original
copyrighted text across four research domains. Freeing scientific knowledge
from copyright promises transformative benefits for scientific research and
education by allowing language models to reuse important facts from copyrighted
text. To support this, we share open-source tools for converting research
documents into Knowledge Units. Overall, our work posits the feasibility of
democratizing access to scientific knowledge while respecting copyright.
GHOST 2.0: generative high-fidelity one shot transfer of heads
Authors: name: Alexander Groshev, name: Anastasiia Iashchenko, name: Pavel Paramonov, name: Denis Dimitrov, name: Andrey Kuznetsov
Published: 2025-02-25 18:13:55

While the task of face swapping has recently gained attention in the research
community, a related problem of head swapping remains largely unexplored. In
addition to skin color transfer, head swap poses extra challenges, such as the
need to preserve structural information of the whole head during synthesis and
inpaint gaps between swapped head and background. In this paper, we address
these concerns with GHOST 2.0, which consists of two problem-specific modules.
First, we introduce enhanced Aligner model for head reenactment, which
preserves identity information at multiple scales and is robust to extreme pose
variations. Secondly, we use a Blender module that seamlessly integrates the
reenacted head into the target background by transferring skin color and
inpainting mismatched regions. Both modules outperform the baselines on the
corresponding tasks, allowing to achieve state of the art results in head
swapping. We also tackle complex cases, such as large difference in hair styles
of source and target. Code is available at
https://github.com/ai-forever/ghost-2.0