Orionnnn/hub-stats huggingface dataset:
Xmodel-2 Technical Report
Authors: name: Wang Qun, name: Liu Yang, name: Lin Qingquan, name: Qu Zhijiu, name: Jiang Ling
Published: 2024-12-27 13:32:10

Xmodel-2 is a 1.2-billion-parameter large language model designed
specifically for reasoning tasks. Its architecture enables different model
scales to share a unified set of hyperparameters, allowing for extensive
experimentation on smaller models and seamless transfer of optimal
configurations to larger models. To maximize training efficiency and stability,
Xmodel-2 employs the WSD learning rate scheduler from MiniCPM. Pretrained on
1.5 trillion tokens from diverse sources, Xmodel-2 achieves state-of-the-art
performance in complex reasoning and agent-based tasks, while maintaining low
training costs. These results highlight the potential of efficient model design
and training strategies in advancing reasoning capabilities. Model checkpoints
and code are publicly available on GitHub at
https://github.com/XiaoduoAILab/Xmodel-2
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
Authors: name: Xingyu Chen, name: Jiahao Xu, name: Tian Liang, name: Zhiwei He, name: Jianhui Pang, name: Dian Yu, name: Linfeng Song, name: Qiuzhi Liu, name: Mengfei Zhou, name: Zhuosheng Zhang, name: Rui Wang, name: Zhaopeng Tu, name: Haitao Mi, name: Dong Yu
Published: 2024-12-30 18:55:12

The remarkable performance of models like the OpenAI o1 can be attributed to
their ability to emulate human-like long-time thinking during inference. These
models employ extended chain-of-thought (CoT) processes, exploring multiple
strategies to enhance problem-solving capabilities. However, a critical
question remains: How to intelligently and efficiently scale computational
resources during testing. This paper presents the first comprehensive study on
the prevalent issue of overthinking in these models, where excessive
computational resources are allocated for simple problems with minimal benefit.
We introduce novel efficiency metrics from both outcome and process
perspectives to evaluate the rational use of computational resources by o1-like
models. Using a self-training paradigm, we propose strategies to mitigate
overthinking, streamlining reasoning processes without compromising accuracy.
Experimental results show that our approach successfully reduces computational
overhead while preserving model performance across a range of testsets with
varying difficulty levels, such as GSM8K, MATH500, GPQA, and AIME.
Slow Perception: Let's Perceive Geometric Figures Step-by-step
Authors: name: Haoran Wei, name: Youyang Yin, name: Yumeng Li, name: Jia Wang, name: Liang Zhao, name: Jianjian Sun, name: Zheng Ge, name: Xiangyu Zhang
Published: 2024-12-30 00:40:35

Recently, "visual o1" began to enter people's vision, with expectations that
this slow-thinking design can solve visual reasoning tasks, especially
geometric math problems. However, the reality is that current LVLMs (Large
Vision Language Models) can hardly even accurately copy a geometric figure, let
alone truly understand the complex inherent logic and spatial relationships
within geometric shapes. We believe accurate copying (strong perception) is the
first step to visual o1. Accordingly, we introduce the concept of "slow
perception" (SP), which guides the model to gradually perceive basic point-line
combinations, as our humans, reconstruct complex geometric structures
progressively. There are two-fold stages in SP: a) perception decomposition.
Perception is not instantaneous. In this stage, complex geometric figures are
broken down into basic simple units to unify geometry representation. b)
perception flow, which acknowledges that accurately tracing a line is not an
easy task. This stage aims to avoid "long visual jumps" in regressing line
segments by using a proposed "perceptual ruler" to trace each line
stroke-by-stroke. Surprisingly, such a human-like perception manner enjoys an
inference time scaling law -- the slower, the better. Researchers strive to
speed up the model's perception in the past, but we slow it down again,
allowing the model to read the image step-by-step and carefully.
PERSE: Personalized 3D Generative Avatars from A Single Portrait
Authors: name: Hyunsoo Cha, name: Inhee Lee, name: Hanbyul Joo
Published: 2024-12-30 18:59:58

We present PERSE, a method for building an animatable personalized generative
avatar from a reference portrait. Our avatar model enables facial attribute
editing in a continuous and disentangled latent space to control each facial
attribute, while preserving the individual's identity. To achieve this, our
method begins by synthesizing large-scale synthetic 2D video datasets, where
each video contains consistent changes in the facial expression and viewpoint,
combined with a variation in a specific facial attribute from the original
input. We propose a novel pipeline to produce high-quality, photorealistic 2D
videos with facial attribute editing. Leveraging this synthetic attribute
dataset, we present a personalized avatar creation method based on the 3D
Gaussian Splatting, learning a continuous and disentangled latent space for
intuitive facial attribute manipulation. To enforce smooth transitions in this
latent space, we introduce a latent space regularization technique by using
interpolated 2D faces as supervision. Compared to previous approaches, we
demonstrate that PERSE generates high-quality avatars with interpolated
attributes while preserving identity of reference person.
Facilitating large language model Russian adaptation with Learned
  Embedding Propagation
Authors: name: Mikhail Tikhomirov, name: Daniil Chernyshev
Published: 2024-12-30 18:15:45

Rapid advancements of large language model (LLM) technologies led to the
introduction of powerful open-source instruction-tuned LLMs that have the same
text generation quality as the state-of-the-art counterparts such as GPT-4.
While the emergence of such models accelerates the adoption of LLM technologies
in sensitive-information environments the authors of such models don not
disclose the training data necessary for replication of the results thus making
the achievements model-exclusive. Since those open-source models are also
multilingual this in turn reduces the benefits of training a language specific
LLMs as improved inference computation efficiency becomes the only guaranteed
advantage of such costly procedure. More cost-efficient options such as
vocabulary extension and subsequent continued pre-training are also inhibited
by the lack of access to high-quality instruction-tuning data since it is the
major factor behind the resulting LLM task-solving capabilities. To address the
limitations and cut the costs of the language adaptation pipeline we propose
Learned Embedding Propagation (LEP). Unlike existing approaches our method has
lower training data size requirements due to minimal impact on existing LLM
knowledge which we reinforce using novel ad-hoc embedding propagation procedure
that allows to skip the instruction-tuning step and instead implant the new
language knowledge directly into any existing instruct-tuned variant. We
evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B,
showing that LEP is competitive with traditional instruction-tuning methods,
achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with
further improvements via self-calibration and continued tuning enhancing
task-solving capabilities.
HumanEval Pro and MBPP Pro: Evaluating Large Language Models on
  Self-invoking Code Generation
Authors: name: Zhaojian Yu, name: Yilun Zhao, name: Arman Cohan, name: Xiao-Ping Zhang
Published: 2024-12-30 18:58:58

We introduce self-invoking code generation, a new task designed to evaluate
the progressive reasoning and problem-solving capabilities of LLMs. In this
task, models are presented with a base problem and a related, more complex
problem. They must solve the base problem and then utilize its solution to
address the more complex one. This work features three key contributions.
First, we propose a general recipe for generating more challenging versions of
existing benchmarks, resulting in three new benchmarks: HumanEval Pro, MBPP
Pro, and BigCodeBench-Lite Pro, specifically designed to assess LLMs on
self-invoking code generation. Second, from the analysis of experimental
results over twenty LLMs on our benchmarks, we have two important observations:
(i) Most LLMs excel in traditional code generation benchmarks like HumanEval
and MBPP, but their performance declines on self-invoking tasks. For example,
o1-mini achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro.
(ii) On self-invoking code generation task, the instruction-tuned models
demonstrate only marginal improvements compared to the base models. Third, we
disclose the types of failure modes that exist in our evaluation results. All
these results underscore the need for further advancements in self-invoking
code generation tasks and provide a new direction for future research on
enhancing LLMs' code reasoning capabilities.
Bringing Objects to Life: 4D generation from 3D objects
Authors: name: Ohad Rahamim, name: Ori Malca, name: Dvir Samuel, name: Gal Chechik
Published: 2024-12-29 10:12:01

Recent advancements in generative modeling now enable the creation of 4D
content (moving 3D objects) controlled with text prompts. 4D generation has
large potential in applications like virtual worlds, media, and gaming, but
existing methods provide limited control over the appearance and geometry of
generated content. In this work, we introduce a method for animating
user-provided 3D objects by conditioning on textual prompts to guide 4D
generation, enabling custom animations while maintaining the identity of the
original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance
Field (NeRF) that preserves the visual attributes of the input object. Then, we
animate the object using an Image-to-Video diffusion model driven by text. To
improve motion realism, we introduce an incremental viewpoint selection
protocol for sampling perspectives to promote lifelike movement and a masked
Score Distillation Sampling (SDS) loss, which leverages attention maps to focus
optimization on relevant regions. We evaluate our model in terms of temporal
coherence, prompt adherence, and visual fidelity and find that our method
outperforms baselines that are based on other approaches, achieving up to
threefold improvements in identity preservation measured using LPIPS scores,
and effectively balancing visual quality with dynamic content.
Training Software Engineering Agents and Verifiers with SWE-Gym
Authors: name: Jiayi Pan, name: Xingyao Wang, name: Graham Neubig, name: Navdeep Jaitly, name: Heng Ji, name: Alane Suhr, name: Yizhe Zhang
Published: 2024-12-30 18:15:39

We present SWE-Gym, the first environment for training real-world software
engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task
instances, each comprising a codebase with an executable runtime environment,
unit tests, and a task specified in natural language. We use SWE-Gym to train
language model based SWE agents , achieving up to 19% absolute gains in resolve
rate on the popular SWE-Bench Verified and Lite test sets. We also experiment
with inference-time scaling through verifiers trained on agent trajectories
sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve
32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new
state-of-the-art for open-weight SWE agents. To facilitate further research, we
publicly release SWE-Gym, models, and agent trajectories.
Explanatory Instructions: Towards Unified Vision Tasks Understanding and
  Zero-shot Generalization
Authors: name: Yang Shen, name: Xiu-Shen Wei, name: Yifan Sun, name: Yuxin Song, name: Tao Yuan, name: Jian Jin, name: Heyang Xu, name: Yazhou Yao, name: Errui Ding
Published: 2024-12-24 16:08:25

Computer Vision (CV) has yet to fully achieve the zero-shot task
generalization observed in Natural Language Processing (NLP), despite following
many of the milestones established in NLP, such as large transformer models,
extensive pre-training, and the auto-regression paradigm, among others. In this
paper, we explore the idea that CV adopts discrete and terminological task
definitions (\eg, ``image segmentation''), which may be a key barrier to
zero-shot task generalization. Our hypothesis is that without truly
understanding previously-seen tasks--due to these terminological
definitions--deep models struggle to generalize to novel tasks. To verify this,
we introduce Explanatory Instructions, which provide an intuitive way to define
CV task objectives through detailed linguistic transformations from input
images to outputs. We create a large-scale dataset comprising 12 million
``image input to explanatory instruction to output'' triplets, and train
an auto-regressive-based vision-language model (AR-based VLM) that takes both
images and explanatory instructions as input. By learning to follow these
instructions, the AR-based VLM achieves instruction-level zero-shot
capabilities for previously-seen tasks and demonstrates strong zero-shot
generalization for unseen CV tasks. Code and dataset will be openly available
on our GitHub repository.
Efficiently Serving LLM Reasoning Programs with Certaindex
Authors: name: Yichao Fu, name: Junda Chen, name: Siqi Zhu, name: Zheyu Fu, name: Zhongdongming Dai, name: Aurick Qiao, name: Hao Zhang
Published: 2024-12-30 14:57:53

The rapid evolution of large language models (LLMs) has unlocked their
capabilities in advanced reasoning tasks like mathematical problem-solving,
code generation, and legal analysis. Central to this progress are
inference-time reasoning algorithms, which refine outputs by exploring multiple
solution paths, at the cost of increasing compute demands and response
latencies. Existing serving systems fail to adapt to the scaling behaviors of
these algorithms or the varying difficulty of queries, leading to inefficient
resource use and unmet latency targets.
  We present Dynasor, a system that optimizes inference-time compute for LLM
reasoning queries. Unlike traditional engines, Dynasor tracks and schedules
requests within reasoning queries and uses Certaindex, a proxy that measures
statistical reasoning progress based on model certainty, to guide compute
allocation dynamically. Dynasor co-adapts scheduling with reasoning progress:
it allocates more compute to hard queries, reduces compute for simpler ones,
and terminates unpromising queries early, balancing accuracy, latency, and
cost. On diverse datasets and algorithms, Dynasor reduces compute by up to 50%
in batch processing and sustaining 3.3x higher query rates or 4.7x tighter
latency SLOs in online serving.
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow
  Matching and Clap-Ranked Preference Optimization
Authors: name: Chia-Yu Hung, name: Navonil Majumder, name: Zhifeng Kong, name: Ambuj Mehrish, name: Rafael Valle, name: Bryan Catanzaro, name: Soujanya Poria
Published: 2024-12-30 16:02:44

We introduce TangoFlux, an efficient Text-to-Audio (TTA) generative model
with 515M parameters, capable of generating up to 30 seconds of 44.1kHz audio
in just 3.7 seconds on a single A40 GPU. A key challenge in aligning TTA models
lies in the difficulty of creating preference pairs, as TTA lacks structured
mechanisms like verifiable rewards or gold-standard answers available for Large
Language Models (LLMs). To address this, we propose CLAP-Ranked Preference
Optimization (CRPO), a novel framework that iteratively generates and optimizes
preference data to enhance TTA alignment. We demonstrate that the audio
preference dataset generated using CRPO outperforms existing alternatives. With
this framework, TangoFlux achieves state-of-the-art performance across both
objective and subjective benchmarks. We open source all code and models to
support further research in TTA generation.
OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction
  System
Authors: name: Yujie Luo, name: Xiangyuan Ru, name: Kangwei Liu, name: Lin Yuan, name: Mengshu Sun, name: Ningyu Zhang, name: Lei Liang, name: Zhiqiang Zhang, name: Jun Zhou, name: Lanning Wei, name: Da Zheng, name: Haofen Wang, name: Huajun Chen
Published: 2024-12-28 04:01:30

We introduce OneKE, a dockerized schema-guided knowledge extraction system,
which can extract knowledge from the Web and raw PDF Books, and support various
domains (science, news, etc.). Specifically, we design OneKE with multiple
agents and a configure knowledge base. Different agents perform their
respective roles, enabling support for various extraction scenarios. The
configure knowledge base facilitates schema configuration, error case debugging
and correction, further improving the performance. Empirical evaluations on
benchmark datasets demonstrate OneKE's efficacy, while case studies further
elucidate its adaptability to diverse tasks across multiple domains,
highlighting its potential for broad applications. We have open-sourced the
Code at https://github.com/zjunlp/OneKE and released a Video at
http://oneke.openkg.cn/demo.mp4.
On the Compositional Generalization of Multimodal LLMs for Medical
  Imaging
Authors: name: Zhenyang Cai, name: Junying Chen, name: Rongsheng Wang, name: Weihong Wang, name: Yonglin Deng, name: Dingjie Song, name: Yize Chen, name: Zixu Zhang, name: Benyou Wang
Published: 2024-12-28 07:50:00

Multimodal large language models (MLLMs) hold significant potential in the
medical field, but their capabilities are often limited by insufficient data in
certain medical domains, highlighting the need for understanding what kinds of
images can be used by MLLMs for generalization. Current research suggests that
multi-task training outperforms single-task as different tasks can benefit each
other, but they often overlook the internal relationships within these tasks,
providing limited guidance on selecting datasets to enhance specific tasks. To
analyze this phenomenon, we attempted to employ compositional generalization
(CG)-the ability of models to understand novel combinations by recombining
learned elements-as a guiding framework. Since medical images can be precisely
defined by Modality, Anatomical area, and Task, naturally providing an
environment for exploring CG. Therefore, we assembled 106 medical datasets to
create Med-MAT for comprehensive experiments. The experiments confirmed that
MLLMs can use CG to understand unseen medical images and identified CG as one
of the main drivers of the generalization observed in multi-task training.
Additionally, further studies demonstrated that CG effectively supports
datasets with limited data and delivers consistent performance across different
backbones, highlighting its versatility and broad applicability. Med-MAT is
publicly available at https://github.com/FreedomIntelligence/Med-MAT.
Edicho: Consistent Image Editing in the Wild
Authors: name: Qingyan Bai, name: Hao Ouyang, name: Yinghao Xu, name: Qiuyu Wang, name: Ceyuan Yang, name: Ka Leong Cheng, name: Yujun Shen, name: Qifeng Chen
Published: 2024-12-30 16:56:44

As a verified need, consistent editing across in-the-wild images remains a
technical challenge arising from various unmanageable factors, like object
poses, lighting conditions, and photography environments. Edicho steps in with
a training-free solution based on diffusion models, featuring a fundamental
design principle of using explicit image correspondence to direct editing.
Specifically, the key components include an attention manipulation module and a
carefully refined classifier-free guidance (CFG) denoising strategy, both of
which take into account the pre-estimated correspondence. Such an
inference-time algorithm enjoys a plug-and-play nature and is compatible to
most diffusion-based editing methods, such as ControlNet and BrushNet.
Extensive results demonstrate the efficacy of Edicho in consistent cross-image
editing under diverse settings. We will release the code to facilitate future
studies.
CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge
  Graphs in the LLM Era
Authors: name: Yanlin Feng, name: Simone Papicchio, name: Sajjadur Rahman
Published: 2024-12-24 23:22:04

Retrieval from graph data is crucial for augmenting large language models
(LLM) with both open-domain knowledge and private enterprise data, and it is
also a key component in the recent GraphRAG system (edge et al., 2024). Despite
decades of research on knowledge graphs and knowledge base question answering,
leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal
support for retrieval from modern encyclopedic knowledge graphs like Wikidata.
In this paper, we analyze the root cause and suggest that modern RDF knowledge
graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly
large schemas that far exceed the typical LLM context window, use of resource
identifiers, overlapping relation types and lack of normalization. As a
solution, we propose property graph views on top of the underlying RDF graph
that can be efficiently queried by LLMs using Cypher. We instantiated this idea
on Wikidata and introduced CypherBench, the first benchmark with 11
large-scale, multi-domain property graphs with 7.8 million entities and over
10,000 questions. To achieve this, we tackled several key challenges, including
developing an RDF-to-property graph conversion engine, creating a systematic
pipeline for text-to-Cypher task generation, and designing new evaluation
metrics.
1.58-bit FLUX
Authors: name: Chenglin Yang, name: Celong Liu, name: Xueqing Deng, name: Dongwon Kim, name: Xing Mei, name: Xiaohui Shen, name: Liang-Chieh Chen
Published: 2024-12-24 19:00:02

We present 1.58-bit FLUX, the first successful approach to quantizing the
state-of-the-art text-to-image generation model, FLUX.1-dev, using 1.58-bit
weights (i.e., values in {-1, 0, +1}) while maintaining comparable performance
for generating 1024 x 1024 images. Notably, our quantization method operates
without access to image data, relying solely on self-supervision from the
FLUX.1-dev model. Additionally, we develop a custom kernel optimized for
1.58-bit operations, achieving a 7.7x reduction in model storage, a 5.1x
reduction in inference memory, and improved inference latency. Extensive
evaluations on the GenEval and T2I Compbench benchmarks demonstrate the
effectiveness of 1.58-bit FLUX in maintaining generation quality while
significantly enhancing computational efficiency.
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
Authors: name: Liang Chen, name: Zekun Wang, name: Shuhuai Ren, name: Lei Li, name: Haozhe Zhao, name: Yunshui Li, name: Zefan Cai, name: Hongcheng Guo, name: Lei Zhang, name: Yizhe Xiong, name: Yichi Zhang, name: Ruoyu Wu, name: Qingxiu Dong, name: Ge Zhang, name: Jian Yang, name: Lingwei Meng, name: Shujie Hu, name: Yulong Chen, name: Junyang Lin, name: Shuai Bai, name: Andreas Vlachos, name: Xu Tan, name: Minjia Zhang, name: Wen Xiao, name: Aaron Yee, name: Tianyu Liu, name: Baobao Chang
Published: 2024-12-16 05:02:25

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets \& evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction