Orionnnn/hub-stats huggingface dataset:
Enhancing Human-Like Responses in Large Language Models
Authors: name: Ethem Yağız Çalık, name: Talha Rüzgar Akkuş
Published: 2025-01-09 07:44:06

This paper explores the advancements in making large language models (LLMs)
more human-like. We focus on techniques that enhance natural language
understanding, conversational coherence, and emotional intelligence in AI
systems. The study evaluates various approaches, including fine-tuning with
diverse datasets, incorporating psychological principles, and designing models
that better mimic human reasoning patterns. Our findings demonstrate that these
enhancements not only improve user interactions but also open new possibilities
for AI applications across different domains. Future work will address the
ethical implications and potential biases introduced by these human-like
attributes.
SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub
  Issue Resolution
Authors: name: Chengxing Xie, name: Bowen Li, name: Chang Gao, name: He Du, name: Wai Lam, name: Difan Zou, name: Kai Chen
Published: 2025-01-09 07:54:24

Large Language Models (LLMs) have demonstrated remarkable proficiency across
a variety of complex tasks. One significant application of LLMs is in tackling
software engineering challenges, particularly in resolving real-world tasks on
GitHub by fixing code based on the issues reported by the users. However, many
current approaches rely on proprietary LLMs, which limits reproducibility,
accessibility, and transparency. The critical components of LLMs for addressing
software engineering issues and how their capabilities can be effectively
enhanced remain unclear. To address these challenges, we introduce SWE-Fixer, a
novel open-source LLM designed to effectively and efficiently resolve GitHub
issues. SWE-Fixer comprises two essential modules: a code file retrieval module
and a code editing module. The retrieval module employs BM25 along with a
lightweight LLM model to achieve coarse-to-fine file retrieval. Subsequently,
the code editing module utilizes the other LLM model to generate patches for
the identified files. Then, to mitigate the lack of publicly available
datasets, we compile an extensive dataset that includes 110K GitHub issues
along with their corresponding patches, and train the two modules of SWE-Fixer
separately. We assess our approach on the SWE-Bench Lite and Verified
benchmarks, achieving state-of-the-art performance among open-source models
with scores of 23.3% and 30.2%, respectively. These outcomes highlight the
efficacy of our approach. We will make our model, dataset, and code publicly
available at https://github.com/InternLM/SWE-Fixer.
Centurio: On Drivers of Multilingual Ability of Large Vision-Language
  Model
Authors: name: Gregor Geigle, name: Florian Schneider, name: Carolin Holtermann, name: Chris Biemann, name: Radu Timofte, name: Anne Lauscher, name: Goran Glavaš
Published: 2025-01-09 10:26:14

Most Large Vision-Language Models (LVLMs) to date are trained predominantly
on English data, which makes them struggle to understand non-English input and
fail to generate output in the desired target language. Existing efforts
mitigate these issues by adding multilingual training data, but do so in a
largely ad-hoc manner, lacking insight into how different training mixes tip
the scale for different groups of languages. In this work, we present a
comprehensive investigation into the training strategies for massively
multilingual LVLMs. First, we conduct a series of multi-stage experiments
spanning 13 downstream vision-language tasks and 43 languages, systematically
examining: (1) the number of training languages that can be included without
degrading English performance and (2) optimal language distributions of
pre-training as well as (3) instruction-tuning data. Further, we (4)
investigate how to improve multilingual text-in-image understanding, and
introduce a new benchmark for the task. Surprisingly, our analysis reveals that
one can (i) include as many as 100 training languages simultaneously (ii) with
as little as 25-50\% of non-English data, to greatly improve multilingual
performance while retaining strong English performance. We further find that
(iii) including non-English OCR data in pre-training and instruction-tuning is
paramount for improving multilingual text-in-image understanding. Finally, we
put all our findings together and train Centurio, a 100-language LVLM, offering
state-of-the-art performance in an evaluation covering 14 tasks and 56
languages.
Building Foundations for Natural Language Processing of Historical
  Turkish: Resources and Models
Authors: name: Şaziye Betül Özateş, name: Tarık Emre Tıraş, name: Ece Elif Adak, name: Berat Doğan, name: Fatih Burak Karagöz, name: Efe Eren Genç, name: Esma F. Bilgin Taşdemir
Published: 2025-01-08 20:29:00

This paper introduces foundational resources and models for natural language
processing (NLP) of historical Turkish, a domain that has remained
underexplored in computational linguistics. We present the first named entity
recognition (NER) dataset, HisTR and the first Universal Dependencies treebank,
OTA-BOUN for a historical form of the Turkish language along with
transformer-based models trained using these datasets for named entity
recognition, dependency parsing, and part-of-speech tagging tasks.
Additionally, we introduce Ottoman Text Corpus (OTC), a clean corpus of
transliterated historical Turkish texts that spans a wide range of historical
periods. Our experimental results show significant improvements in the
computational analysis of historical Turkish, achieving promising results in
tasks that require understanding of historical linguistic structures. They also
highlight existing challenges, such as domain adaptation and language
variations across time periods. All of the presented resources and models are
made available at https://huggingface.co/bucolin to serve as a benchmark for
future progress in historical Turkish NLP.
Are VLMs Ready for Autonomous Driving? An Empirical Study from the
  Reliability, Data, and Metric Perspectives
Authors: name: Shaoyuan Xie, name: Lingdong Kong, name: Yuhao Dong, name: Chonghao Sima, name: Wenwei Zhang, name: Qi Alfred Chen, name: Ziwei Liu, name: Liang Pan
Published: 2025-01-07 18:59:55

Recent advancements in Vision-Language Models (VLMs) have sparked interest in
their use for autonomous driving, particularly in generating interpretable
driving decisions through natural language. However, the assumption that VLMs
inherently provide visually grounded, reliable, and interpretable explanations
for driving remains largely unexamined. To address this gap, we introduce
DriveBench, a benchmark dataset designed to evaluate VLM reliability across 17
settings (clean, corrupted, and text-only inputs), encompassing 19,200 frames,
20,498 question-answer pairs, three question types, four mainstream driving
tasks, and a total of 12 popular VLMs. Our findings reveal that VLMs often
generate plausible responses derived from general knowledge or textual cues
rather than true visual grounding, especially under degraded or missing visual
inputs. This behavior, concealed by dataset imbalances and insufficient
evaluation metrics, poses significant risks in safety-critical scenarios like
autonomous driving. We further observe that VLMs struggle with multi-modal
reasoning and display heightened sensitivity to input corruptions, leading to
inconsistencies in performance. To address these challenges, we propose refined
evaluation metrics that prioritize robust visual grounding and multi-modal
understanding. Additionally, we highlight the potential of leveraging VLMs'
awareness of corruptions to enhance their reliability, offering a roadmap for
developing more trustworthy and interpretable decision-making systems in
real-world autonomous driving contexts. The benchmark toolkit is publicly
accessible.
On Computational Limits and Provably Efficient Criteria of Visual
  Autoregressive Models: A Fine-Grained Complexity Analysis
Authors: name: Yekun Ke, name: Xiaoyu Li, name: Yingyu Liang, name: Zhizhou Sha, name: Zhenmei Shi, name: Zhao Song
Published: 2025-01-08 09:34:15

Recently, Visual Autoregressive (VAR) Models introduced a
groundbreaking advancement in the field of image generation, offering a
scalable approach through a coarse-to-fine "next-scale prediction" paradigm.
However, the state-of-the-art algorithm of VAR models in [Tian,
Jiang, Yuan, Peng and Wang, NeurIPS 2024] takes O(n^4) time, which is
computationally inefficient. In this work, we analyze the computational limits
and efficiency criteria of VAR Models through a fine-grained
complexity lens. Our key contribution is identifying the conditions under which
VAR computations can achieve sub-quadratic time complexity.
Specifically, we establish a critical threshold for the norm of input matrices
used in VAR attention mechanisms. Above this threshold, assuming the
Strong Exponential Time Hypothesis (SETH) from fine-grained
complexity theory, a sub-quartic time algorithm for VAR models is
impossible. To substantiate our theoretical findings, we present efficient
constructions leveraging low-rank approximations that align with the derived
criteria. This work initiates the study of the computational efficiency of the
VAR model from a theoretical perspective. Our technique will shed
light on advancing scalable and efficient image generation in VAR
frameworks.
Entropy-Guided Attention for Private LLMs
Authors: name: Nandan Kumar Jha, name: Brandon Reagen
Published: 2025-01-07 03:17:47

The pervasiveness of proprietary language models has raised critical privacy
concerns, necessitating advancements in private inference (PI), where
computations are performed directly on encrypted data without revealing users'
sensitive information. While PI offers a promising solution, its practical
deployment is hindered by substantial communication and latency overheads,
primarily stemming from nonlinear operations. To address this, we introduce an
information-theoretic framework to characterize the role of nonlinearities in
decoder-only language models, laying a principled foundation for optimizing
transformer-architectures tailored to the demands of PI.
  By leveraging Shannon's entropy as a quantitative measure, we uncover the
previously unexplored dual significance of nonlinearities: beyond ensuring
training stability, they are crucial for maintaining attention head diversity.
Specifically, we find that their removal triggers two critical failure modes:
{\em entropy collapse} in deeper layers that destabilizes training, and {\em
entropic overload} in earlier layers that leads to under-utilization of
Multi-Head Attention's (MHA) representational capacity.
  We propose an entropy-guided attention mechanism paired with a novel entropy
regularization technique to mitigate entropic overload. Additionally, we
explore PI-friendly alternatives to layer normalization for preventing entropy
collapse and stabilizing the training of LLMs with reduced-nonlinearities. Our
study bridges the gap between information theory and architectural design,
establishing entropy dynamics as a principled guide for developing efficient PI
architectures. The code and implementation are available at
https://github.com/Nandan91/entropy-guided-attention-llm{entropy-guided-llm}.
An Empirical Study of Autoregressive Pre-training from Videos
Authors: name: Jathushan Rajasegaran, name: Ilija Radosavovic, name: Rahul Ravishankar, name: Yossi Gandelsman, name: Christoph Feichtenhofer, name: Jitendra Malik
Published: 2025-01-09 18:59:58

We empirically study autoregressive pre-training from videos. To perform our
study, we construct a series of autoregressive video models, called Toto. We
treat videos as sequences of visual tokens and train transformer models to
autoregressively predict future tokens. Our models are pre-trained on a diverse
dataset of videos and images comprising over 1 trillion visual tokens. We
explore different architectural, training, and inference design choices. We
evaluate the learned visual representations on a range of downstream tasks
including image recognition, video classification, object tracking, and
robotics. Our results demonstrate that, despite minimal inductive biases,
autoregressive pre-training leads to competitive performance across all
benchmarks. Finally, we find that scaling our video models results in similar
scaling curves to those seen in language models, albeit with a different rate.
More details at https://brjathu.github.io/toto/
The GAN is dead; long live the GAN! A Modern GAN Baseline
Authors: name: Yiwen Huang, name: Aaron Gokaslan, name: Volodymyr Kuleshov, name: James Tompkin
Published: 2025-01-09 18:53:06

There is a widely-spread claim that GANs are difficult to train, and GAN
architectures in the literature are littered with empirical tricks. We provide
evidence against this claim and build a modern GAN baseline in a more
principled manner. First, we derive a well-behaved regularized relativistic GAN
loss that addresses issues of mode dropping and non-convergence that were
previously tackled via a bag of ad-hoc tricks. We analyze our loss
mathematically and prove that it admits local convergence guarantees, unlike
most existing relativistic losses. Second, our new loss allows us to discard
all ad-hoc tricks and replace outdated backbones used in common GANs with
modern architectures. Using StyleGAN2 as an example, we present a roadmap of
simplification and modernization that results in a new minimalist baseline --
R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ,
ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against
state-of-the-art GANs and diffusion models.
Search-o1: Agentic Search-Enhanced Large Reasoning Models
Authors: name: Xiaoxi Li, name: Guanting Dong, name: Jiajie Jin, name: Yuyao Zhang, name: Yujia Zhou, name: Yutao Zhu, name: Peitian Zhang, name: Zhicheng Dou
Published: 2025-01-09 16:48:17

Large reasoning models (LRMs) like OpenAI-o1 have demonstrated impressive
long stepwise reasoning capabilities through large-scale reinforcement
learning. However, their extended reasoning processes often suffer from
knowledge insufficiency, leading to frequent uncertainties and potential
errors. To address this limitation, we introduce Search-o1, a
framework that enhances LRMs with an agentic retrieval-augmented generation
(RAG) mechanism and a Reason-in-Documents module for refining retrieved
documents. Search-o1 integrates an agentic search workflow into the reasoning
process, enabling dynamic retrieval of external knowledge when LRMs encounter
uncertain knowledge points. Additionally, due to the verbose nature of
retrieved documents, we design a separate Reason-in-Documents module to deeply
analyze the retrieved information before injecting it into the reasoning chain,
minimizing noise and preserving coherent reasoning flow. Extensive experiments
on complex reasoning tasks in science, mathematics, and coding, as well as six
open-domain QA benchmarks, demonstrate the strong performance of Search-o1.
This approach enhances the trustworthiness and applicability of LRMs in complex
reasoning tasks, paving the way for more reliable and versatile intelligent
systems. The code is available at
https://github.com/sunnynexus/Search-o1.
Multi-task retriever fine-tuning for domain-specific and efficient RAG
Authors: name: Patrice Béchard, name: Orlando Marquez Ayala
Published: 2025-01-08 18:05:30

Retrieval-Augmented Generation (RAG) has become ubiquitous when deploying
Large Language Models (LLMs), as it can address typical limitations such as
generating hallucinated or outdated information. However, when building
real-world RAG applications, practical issues arise. First, the retrieved
information is generally domain-specific. Since it is computationally expensive
to fine-tune LLMs, it is more feasible to fine-tune the retriever to improve
the quality of the data included in the LLM input. Second, as more applications
are deployed in the same real-world system, one cannot afford to deploy
separate retrievers. Moreover, these RAG applications normally retrieve
different kinds of data. Our solution is to instruction fine-tune a small
retriever encoder on a variety of domain-specific tasks to allow us to deploy
one encoder that can serve many use cases, thereby achieving low-cost,
scalability, and speed. We show how this encoder generalizes to out-of-domain
settings as well as to an unseen retrieval task on real-world enterprise use
cases.
EpiCoder: Encompassing Diversity and Complexity in Code Generation
Authors: name: Yaoxiang Wang, name: Haoling Li, name: Xin Zhang, name: Jie Wu, name: Xiao Liu, name: Wenxiang Hu, name: Zhongxin Guo, name: Yangyu Huang, name: Ying Xin, name: Yujiu Yang, name: Jinsong Su, name: Qi Chen, name: Scarlett Li
Published: 2025-01-08 18:58:15

Effective instruction tuning is indispensable for optimizing code LLMs,
aligning model behavior with user expectations and enhancing model performance
in real-world applications. However, most existing methods focus on code
snippets, which are limited to specific functionalities and rigid structures,
restricting the complexity and diversity of the synthesized data. To address
these limitations, we introduce a novel feature tree-based synthesis framework
inspired by Abstract Syntax Trees (AST). Unlike AST, which captures syntactic
structure of code, our framework models semantic relationships between code
elements, enabling the generation of more nuanced and diverse data. The feature
tree is constructed from raw data and refined iteratively to increase the
quantity and diversity of the extracted features. This process enables the
identification of more complex patterns and relationships within the code. By
sampling subtrees with controlled depth and breadth, our framework allows
precise adjustments to the complexity of the generated code, supporting a wide
range of tasks from simple function-level operations to intricate multi-file
scenarios. We fine-tuned widely-used base models to create the EpiCoder series,
achieving state-of-the-art performance at both the function and file levels
across multiple benchmarks. Notably, empirical evidence indicates that our
approach shows significant potential in synthesizing highly complex
repository-level code data. Further analysis elucidates the merits of this
approach by rigorously assessing data complexity and diversity through software
engineering principles and LLM-as-a-judge method.
Chirpy3D: Continuous Part Latents for Creative 3D Bird Generation
Authors: name: Kam Woh Ng, name: Jing Yang, name: Jia Wei Sii, name: Jiankang Deng, name: Chee Seng Chan, name: Yi-Zhe Song, name: Tao Xiang, name: Xiatian Zhu
Published: 2025-01-07 21:14:11

In this paper, we push the boundaries of fine-grained 3D generation into
truly creative territory. Current methods either lack intricate details or
simply mimic existing objects -- we enable both. By lifting 2D fine-grained
understanding into 3D through multi-view diffusion and modeling part latents as
continuous distributions, we unlock the ability to generate entirely new, yet
plausible parts through interpolation and sampling. A self-supervised feature
consistency loss further ensures stable generation of these unseen parts. The
result is the first system capable of creating novel 3D objects with
species-specific details that transcend existing examples. While we demonstrate
our approach on birds, the underlying framework extends beyond things that can
chirp! Code will be released at https://github.com/kamwoh/chirpy3d.
SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single
  Images
Authors: name: Zixuan Huang, name: Mark Boss, name: Aaryaman Vasishta, name: James M. Rehg, name: Varun Jampani
Published: 2025-01-08 18:52:03

We study the problem of single-image 3D object reconstruction. Recent works
have diverged into two directions: regression-based modeling and generative
modeling. Regression methods efficiently infer visible surfaces, but struggle
with occluded regions. Generative methods handle uncertain regions better by
modeling distributions, but are computationally expensive and the generation is
often misaligned with visible surfaces. In this paper, we present SPAR3D, a
novel two-stage approach aiming to take the best of both directions. The first
stage of SPAR3D generates sparse 3D point clouds using a lightweight point
diffusion model, which has a fast sampling speed. The second stage uses both
the sampled point cloud and the input image to create highly detailed meshes.
Our two-stage design enables probabilistic modeling of the ill-posed
single-image 3D task while maintaining high computational efficiency and great
output fidelity. Using point clouds as an intermediate representation further
allows for interactive user edits. Evaluated on diverse datasets, SPAR3D
demonstrates superior performance over previous state-of-the-art methods, at an
inference speed of 0.7 seconds. Project page with code and model:
https://spar3d.github.io
LLM4SR: A Survey on Large Language Models for Scientific Research
Authors: name: Ziming Luo, name: Zonglin Yang, name: Zexin Xu, name: Wei Yang, name: Xinya Du
Published: 2025-01-08 06:44:02

In recent years, the rapid advancement of Large Language Models (LLMs) has
transformed the landscape of scientific research, offering unprecedented
support across various stages of the research cycle. This paper presents the
first systematic survey dedicated to exploring how LLMs are revolutionizing the
scientific research process. We analyze the unique roles LLMs play across four
critical stages of research: hypothesis discovery, experiment planning and
implementation, scientific writing, and peer reviewing. Our review
comprehensively showcases the task-specific methodologies and evaluation
benchmarks. By identifying current challenges and proposing future research
directions, this survey not only highlights the transformative potential of
LLMs, but also aims to inspire and guide researchers and practitioners in
leveraging LLMs to advance scientific inquiry. Resources are available at the
following repository: https://github.com/du-nlp-lab/LLM4SR
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich
  Paradigm for Direct Preference Optimization
Authors: name: Amitava Das, name: Suranjana Trivedy, name: Danush Khanna, name: Rajarshi Roy, name: Gurpreet Singh, name: Basab Ghosh, name: Yaswanth Narsupalli, name: Vinija Jain, name: Vasu Sharma, name: Aishwarya Naresh Reganti, name: Aman Chadha
Published: 2025-01-05 00:08:52

The rapid rise of large language models (LLMs) has unlocked many applications
but also underscores the challenge of aligning them with diverse values and
preferences. Direct Preference Optimization (DPO) is central to alignment but
constrained by fixed divergences and limited feature transformations. We
propose DPO-Kernels, which integrates kernel methods to address these issues
through four key contributions: (i) Kernelized Representations with polynomial,
RBF, Mahalanobis, and spectral kernels for richer transformations, plus a
hybrid loss combining embedding-based and probability-based objectives; (ii)
Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya,
Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven
Selection metrics that automatically choose the best kernel-divergence pair;
and (iv) a Hierarchical Mixture of Kernels for both local precision and global
modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance
in factuality, safety, reasoning, and instruction following. Grounded in
Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization
for LLMs, offering a comprehensive resource for further alignment research.