Orionnnn/hub-stats huggingface dataset:
Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance
Authors: name: Lingfei Qian, name: Weipeng Zhou, name: Yan Wang, name: Xueqing Peng, name: Jimin Huang, name: Qianqian Xie
Published: 2025-02-12 05:13:04

Recent advancements in large language models (LLMs) have shown strong general
reasoning abilities, yet their effectiveness in financial reasoning remains
underexplored. In this study, we comprehensively evaluate 16 powerful reasoning
and general LLMs on three complex financial tasks involving financial text,
tabular data, and equations, assessing numerical reasoning, tabular
interpretation, financial terminology comprehension, long-context processing,
and equation-based problem solving. Our results show that while better datasets
and pretraining improve financial reasoning, general enhancements like CoT
fine-tuning do not always yield consistent gains. Moreover, all reasoning
strategies face challenges in improving performance on long-context and
multi-table tasks. To address these limitations, we develop a financial
reasoning-enhanced model based on Llama-3.1-8B-Instruct, by CoT fine-tuning and
reinforcement learning with domain-specific reasoning paths. Even with simple
fine-tuning with one financial dataset, our model achieves a consistent 10%
performance improvement across tasks, surpassing all 8B models and even
Llama3-70B-Instruct and Llama3.1-70B-Instruct on average. Our results highlight
the need for domain-specific adaptations in financial tasks, emphasizing future
directions such as multi-table reasoning, long-context processing, and
financial terminology comprehension. All our datasets, models, and codes are
publicly available. Furthermore, we introduce a leaderboard for benchmarking
future datasets and models.
DPO-Shift: Shifting the Distribution of Direct Preference Optimization
Authors: name: Xiliang Yang, name: Feng Jiang, name: Qianen Zhang, name: Lei Zhao, name: Xiao Li
Published: 2025-02-11 14:49:44

Direct Preference Optimization (DPO) and its variants have become
increasingly popular for aligning language models with human preferences. These
methods aim to teach models to better distinguish between chosen (or preferred)
and rejected (or dispreferred) responses. However, prior research has
identified that the probability of chosen responses often decreases during
training, and this phenomenon is known as likelihood displacement. To tackle
this challenge, in this work we introduce \method to controllably shift the
distribution of the chosen probability. Then, we show that \method exhibits a
fundamental trade-off between improving the chosen probability and sacrificing
the reward margin, as supported by both theoretical analysis and experimental
validation. Furthermore, we demonstrate the superiority of \method over DPO on
downstream tasks such as MT-Bench and a designed win rate experiment. We
believe this study shows that the likelihood displacement issue of DPO can be
effectively mitigated with a simple, theoretically grounded solution. Our code
is available at https://github.com/Meaquadddd/DPO-Shift.
TransMLA: Multi-head Latent Attention Is All You Need
Authors: name: Fanxu Meng, name: Zengwei Yao, name: Muhan Zhang
Published: 2025-02-11 18:20:18

Modern large language models (LLMs) often encounter communication bottlenecks
on current hardware, rather than purely computational constraints. Multi-head
Latent Attention (MLA) tackles this challenge by using low-rank matrices in the
key-value (KV) layers, thereby allowing compressed latent KV states to be
cached. This approach significantly reduces the KV cache size relative to
traditional multi-head attention, leading to faster inference. Moreover, MLA
employs an up-projection matrix to increase expressiveness, trading additional
computation for reduced communication overhead. Although MLA has demonstrated
efficiency and effectiveness in Deepseek V2/V3/R1, many major model providers
still rely on Group Query Attention (GQA) and have not announced any plans to
adopt MLA. In this paper, we show that GQA can always be represented by MLA
while maintaining the same KV cache overhead, but the converse does not hold.
To encourage broader use of MLA, we introduce **TransMLA**, a post-training
method that converts widely used GQA-based pre-trained models (e.g., LLaMA,
Qwen, Mixtral) into MLA-based models. After conversion, the model can undergo
additional training to boost expressiveness without increasing the KV cache
size. Furthermore, we plan to develop MLA-specific inference acceleration
techniques to preserve low latency in transformed models, thus enabling more
efficient distillation of Deepseek R1.
Learning Conformal Abstention Policies for Adaptive Risk Management in
  Large Language and Vision-Language Models
Authors: name: Sina Tayebati, name: Divake Kumar, name: Nastaran Darabi, name: Dinithi Jayasuriya, name: Ranganath Krishnan, name: Amit Ranjan Trivedi
Published: 2025-02-08 21:30:41

Large Language and Vision-Language Models (LLMs/VLMs) are increasingly used
in safety-critical applications, yet their opaque decision-making complicates
risk assessment and reliability. Uncertainty quantification (UQ) helps assess
prediction confidence and enables abstention when uncertainty is high.
Conformal prediction (CP), a leading UQ method, provides statistical guarantees
but relies on static thresholds, which fail to adapt to task complexity and
evolving data distributions, leading to suboptimal trade-offs in accuracy,
coverage, and informativeness. To address this, we propose learnable conformal
abstention, integrating reinforcement learning (RL) with CP to optimize
abstention thresholds dynamically. By treating CP thresholds as adaptive
actions, our approach balances multiple objectives, minimizing prediction set
size while maintaining reliable coverage. Extensive evaluations across diverse
LLM/VLM benchmarks show our method outperforms Least Ambiguous Classifiers
(LAC) and Adaptive Prediction Sets (APS), improving accuracy by up to 3.2%,
boosting AUROC for hallucination detection by 22.19%, enhancing
uncertainty-guided selective generation (AUARC) by 21.17%, and reducing
calibration error by 70%-85%. These improvements hold across multiple models
and datasets while consistently meeting the 90% coverage target, establishing
our approach as a more effective and flexible solution for reliable
decision-making in safety-critical applications. The code is available at:
{https://github.com/sinatayebati/vlm-uncertainty}.
Pippo: High-Resolution Multi-View Humans from a Single Image
Authors: name: Yash Kant, name: Ethan Weber, name: Jin Kyu Kim, name: Rawal Khirodkar, name: Su Zhaoen, name: Julieta Martinez, name: Igor Gilitschenski, name: Shunsuke Saito, name: Timur Bagautdinov
Published: 2025-02-11 18:59:59

We present Pippo, a generative model capable of producing 1K resolution dense
turnaround videos of a person from a single casually clicked photo. Pippo is a
multi-view diffusion transformer and does not require any additional inputs -
e.g., a fitted parametric model or camera parameters of the input image. We
pre-train Pippo on 3B human images without captions, and conduct multi-view
mid-training and post-training on studio captured humans. During mid-training,
to quickly absorb the studio dataset, we denoise several (up to 48) views at
low-resolution, and encode target cameras coarsely using a shallow MLP. During
post-training, we denoise fewer views at high-resolution and use pixel-aligned
controls (e.g., Spatial anchor and Plucker rays) to enable 3D consistent
generations. At inference, we propose an attention biasing technique that
allows Pippo to simultaneously generate greater than 5 times as many views as
seen during training. Finally, we also introduce an improved metric to evaluate
3D consistency of multi-view generations, and show that Pippo outperforms
existing works on multi-view human generation from a single image.
Hypencoder: Hypernetworks for Information Retrieval
Authors: name: Julian Killingback, name: Hansi Zeng, name: Hamed Zamani
Published: 2025-02-07 22:31:38

The vast majority of retrieval models depend on vector inner products to
produce a relevance score between a query and a document. This naturally limits
the expressiveness of the relevance score that can be employed. We propose a
new paradigm, instead of producing a vector to represent the query we produce a
small neural network which acts as a learned relevance function. This small
neural network takes in a representation of the document, in this paper we use
a single vector, and produces a scalar relevance score. To produce the little
neural network we use a hypernetwork, a network that produce the weights of
other networks, as our query encoder or as we call it a Hypencoder. Experiments
on in-domain search tasks show that Hypencoder is able to significantly
outperform strong dense retrieval models and has higher metrics then reranking
models and models an order of magnitude larger. Hypencoder is also shown to
generalize well to out-of-domain search tasks. To assess the extent of
Hypencoder's capabilities, we evaluate on a set of hard retrieval tasks
including tip-of-the-tongue retrieval and instruction-following retrieval tasks
and find that the performance gap widens substantially compared to standard
retrieval tasks. Furthermore, to demonstrate the practicality of our method we
implement an approximate search algorithm and show that our model is able to
search 8.8M documents in under 60ms.
Goedel-Prover: A Frontier Model for Open-Source Automated Theorem
  Proving
Authors: name: Yong Lin, name: Shange Tang, name: Bohan Lyu, name: Jiayun Wu, name: Hongzhou Lin, name: Kaiyu Yang, name: Jia Li, name: Mengzhou Xia, name: Danqi Chen, name: Sanjeev Arora, name: Chi Jin
Published: 2025-02-11 15:27:35

We introduce Goedel-Prover, an open-source large language model (LLM) that
achieves the state-of-the-art (SOTA) performance in automated formal proof
generation for mathematical problems. The key challenge in this field is the
scarcity of formalized math statements and proofs, which we tackle in the
following ways. We train statement formalizers to translate the natural
language math problems from Numina into formal language (Lean 4), creating a
dataset of 1.64 million formal statements. LLMs are used to check that the
formal statements accurately preserve the content of the original natural
language problems. We then iteratively build a large dataset of formal proofs
by training a series of provers. Each prover succeeds in proving many
statements that the previous ones could not, and these new proofs are added to
the training set for the next prover. The final prover outperforms all existing
open-source models in whole-proof generation. On the miniF2F benchmark, it
achieves a 57.6% success rate (Pass@32), exceeding the previous best
open-source model by 7.6%. On PutnamBench, Goedel-Prover successfully solves 7
problems (Pass@512), ranking first on the leaderboard. Furthermore, it
generates 29.7K formal proofs for Lean Workbook problems, nearly doubling the
15.7K produced by earlier works.
Sparse Autoencoders for Scientifically Rigorous Interpretation of Vision
  Models
Authors: name: Samuel Stevens, name: Wei-Lun Chao, name: Tanya Berger-Wolf, name: Yu Su
Published: 2025-02-10 18:32:41

To truly understand vision models, we must not only interpret their learned
features but also validate these interpretations through controlled
experiments. Current approaches either provide interpretable features without
the ability to test their causal influence, or enable model editing without
interpretable controls. We present a unified framework using sparse
autoencoders (SAEs) that bridges this gap, allowing us to discover
human-interpretable visual features and precisely manipulate them to test
hypotheses about model behavior. By applying our method to state-of-the-art
vision models, we reveal key differences in the semantic abstractions learned
by models with different pre-training objectives. We then demonstrate the
practical usage of our framework through controlled interventions across
multiple vision tasks. We show that SAEs can reliably identify and manipulate
interpretable visual features without model re-training, providing a powerful
tool for understanding and controlling vision model behavior. We provide code,
demos and models on our project website: https://osu-nlp-group.github.io/SAE-V.
CoS: Chain-of-Shot Prompting for Long Video Understanding
Authors: name: Jian Hu, name: Zixu Cheng, name: Chenyang Si, name: Wei Li, name: Shaogang Gong
Published: 2025-02-10 13:03:05

Multi-modal Large Language Models (MLLMs) struggle with long videos due to
the need for excessive visual tokens. These tokens exceed massively the context
length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to
select shots is an unsolved critical problem: sparse sampling risks missing key
details, while exhaustive sampling overwhelms the model with irrelevant
content, leading to video misunderstanding. To solve this problem, we propose
Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as
test-time visual prompt optimisation, choosing shots adaptive to video
understanding semantic task by optimising shots-task alignment. CoS has two key
parts: (1) a binary video summary mechanism that performs pseudo temporal
grounding, discovering a binary coding to identify task-relevant shots, and (2)
a video co-reasoning module that deploys the binary coding to pair (learning to
align) task-relevant positive shots with irrelevant negative shots. It embeds
the optimised shot selections into the original video, facilitating a focus on
relevant context to optimize long video understanding. Experiments across three
baselines and five datasets demonstrate the effectiveness and adaptability of
CoS. Code given in https://lwpyh.github.io/CoS.
Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Authors: name: Sean McLeish, name: John Kirchenbauer, name: David Yu Miller, name: Siddharth Singh, name: Abhinav Bhatele, name: Micah Goldblum, name: Ashwinee Panda, name: Tom Goldstein
Published: 2025-02-07 18:09:38

Scaling laws are typically fit using a family of models with a narrow range
of frozen hyper-parameter choices. In this work we study scaling laws using a
wide range of architecture and hyper-parameter choices, and highlight their
impact on resulting prescriptions. As a primary artifact of our research, we
release the Gemstones: the most comprehensive open-source scaling law dataset
to date, consisting of over 4000 checkpoints from transformers with up to 2
billion parameters; these models have been trained with different learning
rates, cooldown schedules, and architectural shapes. Our checkpoints enable
more complex studies of scaling, such as a law that predicts language modeling
performance as a function of model width and depth. By examining the various
facets of our model suite, we find that the prescriptions of scaling laws can
be highly sensitive to the experimental design process and the specific model
checkpoints used during fitting. Code:
https://github.com/mcleish7/gemstone-scaling-laws
Retrieval-augmented Large Language Models for Financial Time Series
  Forecasting
Authors: name: Mengxi Xiao, name: Zihao Jiang, name: Lingfei Qian, name: Zhengyu Chen, name: Yueru He, name: Yijing Xu, name: Yuecheng Jiang, name: Dong Li, name: Ruey-Ling Weng, name: Min Peng, name: Jimin Huang, name: Sophia Ananiadou, name: Qianqian Xie
Published: 2025-02-09 12:26:05

Stock movement prediction, a fundamental task in financial time-series
forecasting, requires identifying and retrieving critical influencing factors
from vast amounts of time-series data. However, existing text-trained or
numeric similarity-based retrieval methods fall short in handling complex
financial analysis. To address this, we propose the first retrieval-augmented
generation (RAG) framework for financial time-series forecasting, featuring
three key innovations: a fine-tuned 1B parameter large language model
(StockLLM) as the backbone, a novel candidate selection method leveraging LLM
feedback, and a training objective that maximizes similarity between queries
and historically significant sequences. This enables our retriever, FinSeer, to
uncover meaningful patterns while minimizing noise in complex financial data.
We also construct new datasets integrating financial indicators and historical
stock prices to train FinSeer and ensure robust evaluation. Experimental
results demonstrate that our RAG framework outperforms bare StockLLM and random
retrieval, highlighting its effectiveness, while FinSeer surpasses existing
retrieval methods, achieving an 8\% higher accuracy on BIGDATA22 and retrieving
more impactful sequences. This work underscores the importance of tailored
retrieval models in financial forecasting and provides a novel framework for
future research.
Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn
  More
Authors: name: Xialie Zhuang, name: Zhikai Jia, name: Jianjin Li, name: Zhenyu Zhang, name: Li Shen, name: Zheng Cao, name: Shiwei Liu
Published: 2025-02-11 11:49:03

Large Language Models (LLMs) are discovered to suffer from accurately
retrieving key information. To address this, we propose Mask-Enhanced
Autoregressive Prediction (MEAP), a simple yet effective training paradigm that
seamlessly integrates Masked Language Modeling (MLM) into Next-Token Prediction
(NTP) to enhance the latter's in-context retrieval capabilities. Specifically,
MEAP first randomly masks a small fraction of input tokens and then directly
performs the standard next-token prediction autoregressive using a decoder-only
Transformer. MEAP eliminates the need for bidirectional attention or
encoder-decoder architectures for MLM, incurring no additional computational
overhead during pre-training or inference. Intensive experiments demonstrate
that MEAP substantially outperforms NTP on key information retrieval and
long-context reasoning tasks, while performing on par or better on commonsense
reasoning tasks. The benefits of MEAP also extend to supervised fine-tuning,
where it shows remarkable advantages in lost-in-the-middle scenarios,
outperforming NTP by 11.77 percentage points. Our analysis indicates that
MEAP's effectiveness arises from its ability to promote more distinguishable
attention scores by concentrating on a reduced set of non-masked tokens. This
mechanism improves the model's focus on task-relevant signals while mitigating
the influence of peripheral context. These findings position MEAP as a
promising training paradigm for large language models.
Skill Expansion and Composition in Parameter Space
Authors: name: Tenglong Liu, name: Jianxiong Li, name: Yinan Zheng, name: Haoyi Niu, name: Yixing Lan, name: Xin Xu, name: Xianyuan Zhan
Published: 2025-02-09 15:22:38

Humans excel at reusing prior knowledge to address new challenges and
developing skills while solving problems. This paradigm becomes increasingly
popular in the development of autonomous agents, as it develops systems that
can self-evolve in response to new challenges like human beings. However,
previous methods suffer from limited training efficiency when expanding new
skills and fail to fully leverage prior knowledge to facilitate new task
learning. In this paper, we propose Parametric Skill Expansion and Composition
(PSEC), a new framework designed to iteratively evolve the agents' capabilities
and efficiently address new challenges by maintaining a manageable skill
library. This library can progressively integrate skill primitives as
plug-and-play Low-Rank Adaptation (LoRA) modules in parameter-efficient
finetuning, facilitating efficient and flexible skill expansion. This structure
also enables the direct skill compositions in parameter space by merging LoRA
modules that encode different skills, leveraging shared information across
skills to effectively program new skills. Based on this, we propose a
context-aware module to dynamically activate different skills to
collaboratively handle new tasks. Empowering diverse applications including
multi-objective composition, dynamics shift, and continual policy shift, the
results on D4RL, DSRL benchmarks, and the DeepMind Control Suite show that PSEC
exhibits superior capacity to leverage prior knowledge to efficiently tackle
new challenges, as well as expand its skill libraries to evolve the
capabilities. Project website: https://ltlhuuu.github.io/PSEC/.
Éclair -- Extracting Content and Layout with Integrated Reading Order
  for Documents
Authors: name: Ilia Karmanov, name: Amala Sanjay Deshmukh, name: Lukas Voegtle, name: Philipp Fischer, name: Kateryna Chumachenko, name: Timo Roman, name: Jarno Seppänen, name: Jupinder Parmar, name: Joseph Jennings, name: Andrew Tao, name: Karan Sapra
Published: 2025-02-06 17:07:22

Optical Character Recognition (OCR) technology is widely used to extract text
from images of documents, facilitating efficient digitization and data
retrieval. However, merely extracting text is insufficient when dealing with
complex documents. Fully comprehending such documents requires an understanding
of their structure -- including formatting, formulas, tables, and the reading
order of multiple blocks and columns across multiple pages -- as well as
semantic information for detecting elements like footnotes and image captions.
This comprehensive understanding is crucial for downstream tasks such as
retrieval, document question answering, and data curation for training Large
Language Models (LLMs) and Vision Language Models (VLMs). To address this, we
introduce \'Eclair, a general-purpose text-extraction tool specifically
designed to process a wide range of document types. Given an image, \'Eclair is
able to extract formatted text in reading order, along with bounding boxes and
their corresponding semantic classes. To thoroughly evaluate these novel
capabilities, we introduce our diverse human-annotated benchmark for
document-level OCR and semantic classification. \'Eclair achieves
state-of-the-art accuracy on this benchmark, outperforming other methods across
key metrics. Additionally, we evaluate \'Eclair on established benchmarks,
demonstrating its versatility and strength across several evaluation standards.
Expect the Unexpected: FailSafe Long Context QA for Finance
Authors: name: Kiran Kamble, name: Melisa Russak, name: Dmytro Mozolevskyi, name: Muayad Ali, name: Mateusz Russak, name: Waseem AlShikh
Published: 2025-02-10 10:29:28

We propose a new long-context financial benchmark, FailSafeQA, designed to
test the robustness and context-awareness of LLMs against six variations in
human-interface interactions in LLM-based query-answer systems within finance.
We concentrate on two case studies: Query Failure and Context Failure. In the
Query Failure scenario, we perturb the original query to vary in domain
expertise, completeness, and linguistic accuracy. In the Context Failure case,
we simulate the uploads of degraded, irrelevant, and empty documents. We employ
the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained
rating criteria to define and calculate Robustness, Context Grounding, and
Compliance scores for 24 off-the-shelf models. The results suggest that
although some models excel at mitigating input perturbations, they must balance
robust answering with the ability to refrain from hallucinating. Notably,
Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained
strong baseline performance but encountered challenges in sustaining robust
predictions in 17% of test cases. On the other hand, the most robust model,
OpenAI o3-mini, fabricated information in 41% of tested cases. The results
demonstrate that even high-performing models have significant room for
improvement and highlight the role of FailSafeQA as a tool for developing LLMs
optimized for dependability in financial applications. The dataset is available
at: https://huggingface.co/datasets/Writer/FailSafeQA