Orionnnn/hub-stats huggingface dataset:
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and
  Mixture-of-Experts Optimization Alignment
Authors: name: Chenghao Fan, name: Zhenyi Lu, name: Sichen Liu, name: Xiaoye Qu, name: Wei Wei, name: Chengfeng Gu, name: Yu Cheng
Published: 2025-02-24 06:48:13

While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for
Large Language Models (LLMs), its performance often falls short of Full
Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with
static singular value decomposition (SVD) subsets, leading to suboptimal
leveraging of pre-trained knowledge. Another path for improving LoRA is
incorporating a Mixture-of-Experts (MoE) architecture. However, weight
misalignment and complex gradient dynamics make it challenging to adopt SVD
prior to the LoRA MoE architecture. To mitigate these issues, we propose
Great LoRA Mixture-of-Expert
(GOAT), a framework that (1) adaptively integrates relevant priors using an
SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by
deriving a theoretical scaling factor. We demonstrate that proper scaling,
without modifying the architecture or training algorithms, boosts LoRA MoE's
efficiency and performance. Experiments across 25 datasets, including natural
language understanding, commonsense reasoning, image classification, and
natural language generation, demonstrate GOAT's state-of-the-art performance,
closing the gap with Full FT.
Mobile-Agent-V: Learning Mobile Device Operation Through Video-Guided
  Multi-Agent Collaboration
Authors: name: Junyang Wang, name: Haiyang Xu, name: Xi Zhang, name: Ming Yan, name: Ji Zhang, name: Fei Huang, name: Jitao Sang
Published: 2025-02-24 12:51:23

The rapid increase in mobile device usage necessitates improved automation
for seamless task management. However, many AI-driven frameworks struggle due
to insufficient operational knowledge. Manually written knowledge helps but is
labor-intensive and inefficient. To address these challenges, we introduce
Mobile-Agent-V, a framework that leverages video guidance to provide rich and
cost-effective operational knowledge for mobile automation. Mobile-Agent-V
enhances task execution capabilities by leveraging video inputs without
requiring specialized sampling or preprocessing. Mobile-Agent-V integrates a
sliding window strategy and incorporates a video agent and deep-reflection
agent to ensure that actions align with user instructions. Through this
innovative approach, users can record task processes with guidance, enabling
the system to autonomously learn and execute tasks efficiently. Experimental
results show that Mobile-Agent-V achieves a 30% performance improvement
compared to existing frameworks.
Thus Spake Long-Context Large Language Model
Authors: name: Xiaoran Liu, name: Ruixiao Li, name: Mianqiu Huang, name: Zhigeng Liu, name: Yuerong Song, name: Qipeng Guo, name: Siyang He, name: Qiqi Wang, name: Linlin Li, name: Qun Liu, name: Yaqian Zhou, name: Xuanjing Huang, name: Xipeng Qiu
Published: 2025-02-24 13:19:33

Long context is an important topic in Natural Language Processing (NLP),
running through the development of NLP architectures, and offers immense
opportunities for Large Language Models (LLMs) giving LLMs the lifelong
learning potential akin to humans. Unfortunately, the pursuit of a long context
is accompanied by numerous obstacles. Nevertheless, long context remains a core
competitive advantage for LLMs. In the past two years, the context length of
LLMs has achieved a breakthrough extension to millions of tokens. Moreover, the
research on long-context LLMs has expanded from length extrapolation to a
comprehensive focus on architecture, infrastructure, training, and evaluation
technologies.
  Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy
between the journey of extending the context of LLM and the attempts of humans
to transcend its mortality. In this survey, We will illustrate how LLM
struggles between the tremendous need for a longer context and its equal need
to accept the fact that it is ultimately finite. To achieve this, we give a
global picture of the lifecycle of long-context LLMs from four perspectives:
architecture, infrastructure, training, and evaluation, showcasing the full
spectrum of long-context technologies. At the end of this survey, we will
present 10 unanswered questions currently faced by long-context LLMs. We hope
this survey can serve as a systematic introduction to the research on
long-context LLMs.
CodeCriticBench: A Holistic Code Critique Benchmark for Large Language
  Models
Authors: name: Alexander Zhang, name: Marcus Dong, name: Jiaheng Liu, name: Wei Zhang, name: Yejie Wang, name: Jian Yang, name: Ge Zhang, name: Tianyu Liu, name: Zhongyuan Peng, name: Yingshui Tan, name: Yuanxing Zhang, name: Zhexu Wang, name: Weixun Wang, name: Yancheng He, name: Ken Deng, name: Wangchunshu Zhou, name: Wenhao Huang, name: Zhaoxiang Zhang
Published: 2025-02-23 15:36:43

The critique capacity of Large Language Models (LLMs) is essential for
reasoning abilities, which can provide necessary suggestions (e.g., detailed
analysis and constructive feedback). Therefore, how to evaluate the critique
capacity of LLMs has drawn great attention and several critique benchmarks have
been proposed. However, existing critique benchmarks usually have the following
limitations: (1). Focusing on diverse reasoning tasks in general domains and
insufficient evaluation on code tasks (e.g., only covering code generation
task), where the difficulty of queries is relatively easy (e.g., the code
queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive
evaluation from different dimensions. To address these limitations, we
introduce a holistic code critique benchmark for LLMs called CodeCriticBench.
Specifically, our CodeCriticBench includes two mainstream code tasks (i.e.,
code generation and code QA) with different difficulties. Besides, the
evaluation protocols include basic critique evaluation and advanced critique
evaluation for different characteristics, where fine-grained evaluation
checklists are well-designed for advanced settings. Finally, we conduct
extensive experimental results of existing LLMs, which show the effectiveness
of CodeCriticBench.
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for
  Multimodal Reasoning Models
Authors: name: Qianqi Yan, name: Yue Fan, name: Hongquan Li, name: Shan Jiang, name: Yang Zhao, name: Xinze Guan, name: Ching-Chen Kuo, name: Xin Eric Wang
Published: 2025-02-22 01:52:37

Existing Multimodal Large Language Models (MLLMs) are predominantly trained
and tested on consistent visual-textual inputs, leaving open the question of
whether they can handle inconsistencies in real-world, layout-rich content. To
bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR)
benchmark to assess MLLMs' ability to detect and reason about semantic
mismatches in artifacts such as webpages, presentation slides, and posters.
MMIR comprises 534 challenging samples, each containing synthetically injected
errors across five reasoning-heavy categories: Factual Contradiction, Identity
Misattribution, Contextual Mismatch, Quantitative Discrepancy, and
Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing
that models with dedicated multimodal reasoning capabilities, such as o1,
substantially outperform their counterparts while open-source models remain
particularly vulnerable to inconsistency errors. Detailed error analyses
further show that models excel in detecting inconsistencies confined to a
single modality, particularly in text, but struggle with cross-modal conflicts
and complex layouts. Probing experiments reveal that single-modality prompting,
including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal
gains, revealing a key bottleneck in cross-modal reasoning. Our findings
highlight the need for advanced multimodal reasoning and point to future
research on multimodal inconsistency.
Beyond Release: Access Considerations for Generative AI Systems
Authors: name: Irene Solaiman, name: Rishi Bommasani, name: Dan Hendrycks, name: Ariel Herbert-Voss, name: Yacine Jernite, name: Aviya Skowron, name: Andrew Trask
Published: 2025-02-23 20:06:12

Generative AI release decisions determine whether system components are made
available, but release does not address many other elements that change how
users and stakeholders are able to engage with a system. Beyond release, access
to system components informs potential risks and benefits. Access refers to
practical needs, infrastructurally, technically, and societally, in order to
use available components in some way. We deconstruct access along three axes:
resourcing, technical usability, and utility. Within each category, a set of
variables per system component clarify tradeoffs. For example, resourcing
requires access to computing infrastructure to serve model weights. We also
compare the accessibility of four high performance language models, two
open-weight and two closed-weight, showing similar considerations for all based
instead on access variables. Access variables set the foundation for being able
to scale or increase access to users; we examine the scale of access and how
scale affects ability to manage and intervene on risks. This framework better
encompasses the landscape and risk-benefit tradeoffs of system releases to
inform system release decisions, research, and policy.
PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own
  Deep Neural Net At Inference
Authors: name: Burc Gokden
Published: 2025-02-19 07:43:36

We show that Large Language Model from Power Law Decoder Representations
(PLDR-LLM) is a foundational model whose deductive outputs are invariant
tensors up to a small perturbation. PLDR-LLM learns a singularity condition for
the deductive outputs that enable the once-inferred energy-curvature tensor
G_{LM} to replace the deep neural network of power law graph
attention (PLGA) generating the deductive outputs at inference. We demonstrate
that a cache for G_{LM} (G-cache) and KV-cache can be implemented in
a straightforward manner to improve the inference time. The invariance and
generalizable nature of deductive outputs is at a very high fidelity where
deductive outputs have same RMSE and determinant values up to 15 decimal places
after caching, and zero-shot benchmark scores remain unchanged. Ablation
studies show that learned deductive outputs have distinct loss and accuracy
characteristics from models pretrained with transferred, randomly initialized
or identity tensors as a constant tensor operator and an LLM with scaled-dot
product attention (SDPA) is a special case of PLDR-LLM where G_{LM}
is predefined as identity. The observed invariance characteristic introduces a
novel asymmetry between training and inference phases with caching. We outline
observed common characteristics of the deductive outputs for the learned
singularity condition. We provide an implementation of a training and inference
framework for PLDR-LLM with KV-cache and G-cache.
Towards Fully-Automated Materials Discovery via Large-Scale Synthesis
  Dataset and Expert-Level LLM-as-a-Judge
Authors: name: Heegyu Kim, name: Taeyang Jeon, name: Seungtaek Choi, name: Jihoon Hong, name: Dongwon Jeon, name: Sungbum Cho, name: Ga-Yeon Baek, name: Kyung-Won Kwak, name: Dong-Hee Lee, name: Sun-Jin Choi, name: Jisu Bae, name: Chihoon Lee, name: Yunseo Kim, name: Jinsung Park, name: Hyunsouk Cho
Published: 2025-02-23 06:16:23

Materials synthesis is vital for innovations such as energy storage,
catalysis, electronics, and biomedical devices. Yet, the process relies heavily
on empirical, trial-and-error methods guided by expert intuition. Our work aims
to support the materials science community by providing a practical,
data-driven resource. We have curated a comprehensive dataset of 17K
expert-verified synthesis recipes from open-access literature, which forms the
basis of our newly developed benchmark, AlchemyBench. AlchemyBench offers an
end-to-end framework that supports research in large language models applied to
synthesis prediction. It encompasses key tasks, including raw materials and
equipment prediction, synthesis procedure generation, and characterization
outcome forecasting. We propose an LLM-as-a-Judge framework that leverages
large language models for automated evaluation, demonstrating strong
statistical agreement with expert assessments. Overall, our contributions offer
a supportive foundation for exploring the capabilities of LLMs in predicting
and guiding materials synthesis, ultimately paving the way for more efficient
experimental design and accelerated innovation in materials science.
Learning to Discover Regulatory Elements for Gene Expression Prediction
Authors: name: Xingyu Su, name: Haiyang Yu, name: Degui Zhi, name: Shuiwang Ji
Published: 2025-02-19 03:25:49

We consider the problem of predicting gene expressions from DNA sequences. A
key challenge of this task is to find the regulatory elements that control gene
expressions. Here, we introduce Seq2Exp, a Sequence to Expression network
explicitly designed to discover and extract regulatory elements that drive
target gene expression, enhancing the accuracy of the gene expression
prediction. Our approach captures the causal relationship between epigenomic
signals, DNA sequences and their associated regulatory elements. Specifically,
we propose to decompose the epigenomic signals and the DNA sequence conditioned
on the causal active regulatory elements, and apply an information bottleneck
with the Beta distribution to combine their effects while filtering out
non-causal components. Our experiments demonstrate that Seq2Exp outperforms
existing baselines in gene expression prediction tasks and discovers
influential regions compared to commonly used statistical methods for peak
detection such as MACS3. The source code is released as part of the AIRS
library (https://github.com/divelab/AIRS/).
Rare Disease Differential Diagnosis with Large Language Models at Scale:
  From Abdominal Actinomycosis to Wilson's Disease
Authors: name: Elliot Schumacher, name: Dhruv Naik, name: Anitha Kannan
Published: 2025-02-20 22:02:52

Large language models (LLMs) have demonstrated impressive capabilities in
disease diagnosis. However, their effectiveness in identifying rarer diseases,
which are inherently more challenging to diagnose, remains an open question.
Rare disease performance is critical with the increasing use of LLMs in
healthcare settings. This is especially true if a primary care physician needs
to make a rarer prognosis from only a patient conversation so that they can
take the appropriate next step. To that end, several clinical decision support
systems are designed to support providers in rare disease identification. Yet
their utility is limited due to their lack of knowledge of common disorders and
difficulty of use.
  In this paper, we propose RareScale to combine the knowledge LLMs with expert
systems. We use jointly use an expert system and LLM to simulate rare disease
chats. This data is used to train a rare disease candidate predictor model.
Candidates from this smaller model are then used as additional inputs to
black-box LLM to make the final differential diagnosis. Thus, RareScale allows
for a balance between rare and common diagnoses. We present results on over 575
rare diseases, beginning with Abdominal Actinomycosis and ending with Wilson's
Disease. Our approach significantly improves the baseline performance of
black-box LLMs by over 17% in Top-5 accuracy. We also find that our candidate
generation performance is high (e.g. 88.8% on gpt-4o generated chats).
Tree-of-Debate: Multi-Persona Debate Trees Elicit Critical Thinking for
  Scientific Comparative Analysis
Authors: name: Priyanka Kargupta, name: Ishika Agarwal, name: Tal August, name: Jiawei Han
Published: 2025-02-20 17:43:40

With the exponential growth of research facilitated by modern technology and
improved accessibility, scientific discoveries have become increasingly
fragmented within and across fields. This makes it challenging to assess the
significance, novelty, incremental findings, and equivalent ideas between
related works, particularly those from different research communities. Large
language models (LLMs) have recently demonstrated strong quantitative and
qualitative reasoning abilities, and multi-agent LLM debates have shown promise
in handling complex reasoning tasks by exploring diverse perspectives and
reasoning paths. Inspired by this, we introduce Tree-of-Debate (ToD), a
framework which converts scientific papers into LLM personas that debate their
respective novelties. To emphasize structured, critical reasoning rather than
focusing solely on outcomes, ToD dynamically constructs a debate tree, enabling
fine-grained analysis of independent novelty arguments within scholarly
articles. Through experiments on scientific literature across various domains,
evaluated by expert researchers, we demonstrate that ToD generates informative
arguments, effectively contrasts papers, and supports researchers in their
literature review.
Benchmarking LLMs for Political Science: A United Nations Perspective
Authors: name: Yueqing Liang, name: Liangwei Yang, name: Chen Wang, name: Congying Xia, name: Rui Meng, name: Xiongxiao Xu, name: Haoran Wang, name: Ali Payani, name: Kai Shu
Published: 2025-02-19 21:51:01

Large Language Models (LLMs) have achieved significant advances in natural
language processing, yet their potential for high-stake political
decision-making remains largely unexplored. This paper addresses the gap by
focusing on the application of LLMs to the United Nations (UN) decision-making
process, where the stakes are particularly high and political decisions can
have far-reaching consequences. We introduce a novel dataset comprising
publicly available UN Security Council (UNSC) records from 1994 to 2024,
including draft resolutions, voting records, and diplomatic speeches. Using
this dataset, we propose the United Nations Benchmark (UNBench), the first
comprehensive benchmark designed to evaluate LLMs across four interconnected
political science tasks: co-penholder judgment, representative voting
simulation, draft adoption prediction, and representative statement generation.
These tasks span the three stages of the UN decision-making process--drafting,
voting, and discussing--and aim to assess LLMs' ability to understand and
simulate political dynamics. Our experimental analysis demonstrates the
potential and challenges of applying LLMs in this domain, providing insights
into their strengths and limitations in political science. This work
contributes to the growing intersection of AI and political science, opening
new avenues for research and practical applications in global governance. The
UNBench Repository can be accessed at:
https://github.com/yueqingliang1/UNBench.
FantasyID: Face Knowledge Enhanced ID-Preserving Video Generation
Authors: name: Yunpeng Zhang, name: Qiang Wang, name: Fan Jiang, name: Yaqi Fan, name: Mu Xu, name: Yonggang Qi
Published: 2025-02-19 06:50:27

Tuning-free approaches adapting large-scale pre-trained video diffusion
models for identity-preserving text-to-video generation (IPT2V) have gained
popularity recently due to their efficacy and scalability. However, significant
challenges remain to achieve satisfied facial dynamics while keeping the
identity unchanged. In this work, we present a novel tuning-free IPT2V
framework by enhancing face knowledge of the pre-trained video model built on
diffusion transformers (DiT), dubbed FantasyID. Essentially, 3D facial geometry
prior is incorporated to ensure plausible facial structures during video
synthesis. To prevent the model from learning copy-paste shortcuts that simply
replicate reference face across frames, a multi-view face augmentation strategy
is devised to capture diverse 2D facial appearance features, hence increasing
the dynamics over the facial expressions and head poses. Additionally, after
blending the 2D and 3D features as guidance, instead of naively employing
cross-attention to inject guidance cues into DiT layers, a learnable
layer-aware adaptive mechanism is employed to selectively inject the fused
features into each individual DiT layers, facilitating balanced modeling of
identity preservation and motion dynamics. Experimental results validate our
model's superiority over the current tuning-free IPT2V methods.
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations
  in Large Language Models
Authors: name: Shrey Pandit, name: Jiawei Xu, name: Junyuan Hong, name: Zhangyang Wang, name: Tianlong Chen, name: Kaidi Xu, name: Ying Ding
Published: 2025-02-20 06:33:23

Advancements in Large Language Models (LLMs) and their increasing use in
medical question-answering necessitate rigorous evaluation of their
reliability. A critical challenge lies in hallucination, where models generate
plausible yet factually incorrect outputs. In the medical domain, this poses
serious risks to patient safety and clinical decision-making. To address this,
we introduce MedHallu, the first benchmark specifically designed for medical
hallucination detection. MedHallu comprises 10,000 high-quality question-answer
pairs derived from PubMedQA, with hallucinated answers systematically generated
through a controlled pipeline. Our experiments show that state-of-the-art LLMs,
including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical,
struggle with this binary hallucination detection task, with the best model
achieving an F1 score as low as 0.625 for detecting "hard" category
hallucinations. Using bidirectional entailment clustering, we show that
harder-to-detect hallucinations are semantically closer to ground truth.
Through experiments, we also show incorporating domain-specific knowledge and
introducing a "not sure" category as one of the answer categories improves the
precision and F1 scores by up to 38% relative to baselines.
mStyleDistance: Multilingual Style Embeddings and their Evaluation
Authors: name: Justin Qiu, name: Jiacheng Zhu, name: Ajay Patel, name: Marianna Apidianaki, name: Chris Callison-Burch
Published: 2025-02-21 03:11:41

Style embeddings are useful for stylistic analysis and style transfer;
however, only English style embeddings have been made available. We introduce
Multilingual StyleDistance (mStyleDistance), a multilingual style embedding
model trained using synthetic data and contrastive learning. We train the model
on data from nine languages and create a multilingual STEL-or-Content benchmark
(Wegmann et al., 2022) that serves to assess the embeddings' quality. We also
employ our embeddings in an authorship verification task involving different
languages. Our results show that mStyleDistance embeddings outperform existing
models on these multilingual style benchmarks and generalize well to unseen
features and languages. We make our model publicly available at
https://huggingface.co/StyleDistance/mstyledistance .