Orionnnn/hub-stats huggingface dataset:
BIG-Bench Extra Hard
Authors: name: Mehran Kazemi, name: Bahare Fatemi, name: Hritik Bansal, name: John Palowitch, name: Chrysovalantis Anastasiou, name: Sanket Vaibhav Mehta, name: Lalit K. Jain, name: Virginia Aglietti, name: Disha Jindal, name: Peter Chen, name: Nishanth Dikkala, name: Gladys Tyen, name: Xin Liu, name: Uri Shalit, name: Silvia Chiappa, name: Kate Olszewska, name: Yi Tay, name: Vinh Q. Tran, name: Quoc V. Le, name: Orhan Firat
Published: 2025-02-26 14:50:50

Large language models (LLMs) are increasingly deployed in everyday
applications, demanding robust general reasoning capabilities and diverse
reasoning skillset. However, current LLM reasoning benchmarks predominantly
focus on mathematical and coding abilities, leaving a gap in evaluating broader
reasoning proficiencies. One particular exception is the BIG-Bench dataset,
which has served as a crucial benchmark for evaluating the general reasoning
capabilities of LLMs, thanks to its diverse set of challenging tasks that
allowed for a comprehensive assessment of general reasoning across various
skills within a unified framework. However, recent advances in LLMs have led to
saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH).
State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus
diminishing its utility. To address this limitation, we introduce BIG-Bench
Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM
reasoning evaluation. BBEH replaces each task in BBH with a novel task that
probes a similar reasoning capability but exhibits significantly increased
difficulty. We evaluate various models on BBEH and observe a (harmonic) average
accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best
reasoning-specialized model, indicating substantial room for improvement and
highlighting the ongoing challenge of achieving robust general reasoning in
LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.
Can Language Models Falsify? Evaluating Algorithmic Reasoning with
  Counterexample Creation
Authors: name: Shiven Sinha, name: Shashwat Goel, name: Ponnurangam Kumaraguru, name: Jonas Geiping, name: Matthias Bethge, name: Ameya Prabhu
Published: 2025-02-26 18:58:13

There is growing excitement about the potential of Language Models (LMs) to
accelerate scientific discovery. Falsifying hypotheses is key to scientific
progress, as it allows claims to be iteratively refined over time. This process
requires significant researcher effort, reasoning, and ingenuity. Yet current
benchmarks for LMs predominantly assess their ability to generate solutions
rather than challenge them. We advocate for developing benchmarks that evaluate
this inverse capability - creating counterexamples for subtly incorrect
solutions. To demonstrate this approach, we start with the domain of
algorithmic problem solving, where counterexamples can be evaluated
automatically using code execution. Specifically, we introduce REFUTE, a
dynamically updating benchmark that includes recent problems and incorrect
submissions from programming competitions, where human experts successfully
identified counterexamples. Our analysis finds that the best reasoning agents,
even OpenAI o3-mini (high) with code execution feedback, can create
counterexamples for only <9% of incorrect solutions in REFUTE, even though
ratings indicate its ability to solve up to 48% of these problems from scratch.
We hope our work spurs progress in evaluating and enhancing LMs' ability to
falsify incorrect solutions - a capability that is crucial for both
accelerating research and making models self-improve through reliable
reflective reasoning.
CritiQ: Mining Data Quality Criteria from Human Preferences
Authors: name: Honglin Guo, name: Kai Lv, name: Qipeng Guo, name: Tianyi Liang, name: Zhiheng Xi, name: Demin Song, name: Qiuyinzhe Zhang, name: Yu Sun, name: Kai Chen, name: Xipeng Qiu, name: Tao Gui
Published: 2025-02-26 16:33:41

Language model heavily depends on high-quality data for optimal performance.
Existing approaches rely on manually designed heuristics, the perplexity of
existing models, training classifiers, or careful prompt engineering, which
require significant expert experience and human annotation effort while
introduce biases. We introduce CritiQ, a novel data selection method that
automatically mines criteria from human preferences for data quality with only
sim30 human-annotated pairs and performs efficient data selection. The main
component, CritiQ Flow, employs a manager agent to evolve quality criteria and
worker agents to make pairwise judgments. We build a knowledge base that
extracts quality criteria from previous work to boost CritiQ Flow. Compared to
perplexity- and classifier- based methods, verbal criteria are more
interpretable and possess reusable value. After deriving the criteria, we train
the CritiQ Scorer to give quality scores and perform efficient data selection.
We demonstrate the effectiveness of our method in the code, math, and logic
domains, achieving high accuracy on human-annotated test sets. To validate the
quality of the selected data, we continually train Llama 3.1 models and observe
improved performance on downstream tasks compared to uniform sampling. Ablation
studies validate the benefits of the knowledge base and the reflection process.
We analyze how criteria evolve and the effectiveness of majority voting.
PosterSum: A Multimodal Benchmark for Scientific Poster Summarization
Authors: name: Rohit Saxena, name: Pasquale Minervini, name: Frank Keller
Published: 2025-02-24 18:35:39

Generating accurate and concise textual summaries from multimodal documents
is challenging, especially when dealing with visually complex content like
scientific posters. We introduce PosterSum, a novel benchmark to advance the
development of vision-language models that can understand and summarize
scientific posters into research paper abstracts. Our dataset contains 16,305
conference posters paired with their corresponding abstracts as summaries. Each
poster is provided in image format and presents diverse visual understanding
challenges, such as complex layouts, dense text regions, tables, and figures.
We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on
PosterSum and demonstrate that they struggle to accurately interpret and
summarize scientific posters. We propose Segment & Summarize, a hierarchical
method that outperforms current MLLMs on automated metrics, achieving a 3.14%
gain in ROUGE-L. This will serve as a starting point for future research on
poster summarization.
Language Models' Factuality Depends on the Language of Inquiry
Authors: name: Tushar Aggarwal, name: Kumar Tanmay, name: Ayush Agrawal, name: Kumar Ayush, name: Hamid Palangi, name: Paul Pu Liang
Published: 2025-02-25 08:27:18

Multilingual language models (LMs) are expected to recall factual knowledge
consistently across languages, yet they often fail to transfer knowledge
between languages even when they possess the correct information in one of the
languages. For example, we find that an LM may correctly identify Rashed Al
Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails
to do so when asked in English or Swahili. To systematically investigate this
limitation, we introduce a benchmark of 10,000 country-related facts across 13
languages and propose three novel metrics: Factual Recall Score, Knowledge
Transferability Score, and Cross-Lingual Factual Knowledge Transferability
Score-to quantify factual recall and knowledge transferability in LMs across
different languages. Our results reveal fundamental weaknesses in today's
state-of-the-art LMs, particularly in cross-lingual generalization where models
fail to transfer knowledge effectively across different languages, leading to
inconsistent performance sensitive to the language used. Our findings emphasize
the need for LMs to recognize language-specific factual reliability and
leverage the most trustworthy information across languages. We release our
benchmark and evaluation framework to drive future research in multilingual
knowledge transfer.
Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance
Authors: name: Xueqing Peng, name: Triantafillos Papadopoulos, name: Efstathia Soufleri, name: Polydoros Giannouris, name: Ruoyu Xiang, name: Yan Wang, name: Lingfei Qian, name: Jimin Huang, name: Qianqian Xie, name: Sophia Ananiadou
Published: 2025-02-26 03:04:01

Despite Greece's pivotal role in the global economy, large language models
(LLMs) remain underexplored for Greek financial context due to the linguistic
complexity of Greek and the scarcity of domain-specific datasets. Previous
efforts in multilingual financial natural language processing (NLP) have
exposed considerable performance disparities, yet no dedicated Greek financial
benchmarks or Greek-specific financial LLMs have been developed until now. To
bridge this gap, we introduce Plutus-ben, the first Greek Financial Evaluation
Benchmark, and Plutus-8B, the pioneering Greek Financial LLM, fine-tuned with
Greek domain-specific data. Plutus-ben addresses five core financial NLP tasks
in Greek: numeric and textual named entity recognition, question answering,
abstractive summarization, and topic classification, thereby facilitating
systematic and reproducible LLM assessments. To underpin these tasks, we
present three novel, high-quality Greek financial datasets, thoroughly
annotated by expert native Greek speakers, augmented by two existing resources.
Our comprehensive evaluation of 22 LLMs on Plutus-ben reveals that Greek
financial NLP remains challenging due to linguistic complexity, domain-specific
terminology, and financial reasoning gaps. These findings underscore the
limitations of cross-lingual transfer, the necessity for financial expertise in
Greek-trained models, and the challenges of adapting financial LLMs to Greek
text. We release Plutus-ben, Plutus-8B, and all associated datasets publicly to
promote reproducible research and advance Greek financial NLP, fostering
broader multilingual inclusivity in finance.
Kanana: Compute-efficient Bilingual Language Models
Authors: name: Kanana LLM Team, name: Yunju Bak, name: Hojin Lee, name: Minho Ryu, name: Jiyeon Ham, name: Seungjae Jung, name: Daniel Wontae Nam, name: Taegyeong Eo, name: Donghun Lee, name: Doohae Jung, name: Boseop Kim, name: Nayeon Kim, name: Jaesun Park, name: Hyunho Kim, name: Hyunwoong Ko, name: Changmin Lee, name: Kyoung-Woon On, name: Seulye Baeg, name: Junrae Cho, name: Sunghee Jung, name: Jieun Kang, name: EungGyun Kim, name: Eunhwa Kim, name: Byeongil Ko, name: Daniel Lee, name: Minchul Lee, name: Miok Lee, name: Shinbok Lee, name: Gaeun Seo
Published: 2025-02-26 08:36:20

We introduce Kanana, a series of bilingual language models that demonstrate
exceeding performance in Korean and competitive performance in English. The
computational cost of Kanana is significantly lower than that of
state-of-the-art models of similar size. The report details the techniques
employed during pre-training to achieve compute-efficient yet competitive
models, including high quality data filtering, staged pre-training, depth
up-scaling, and pruning and distillation. Furthermore, the report outlines the
methodologies utilized during the post-training of the Kanana models,
encompassing supervised fine-tuning and preference optimization, aimed at
enhancing their capability for seamless interaction with users. Lastly, the
report elaborates on plausible approaches used for language model adaptation to
specific scenarios, such as embedding, retrieval augmented generation, and
function calling. The Kanana model series spans from 2.1B to 32.5B parameters
with 2.1B models (base, instruct, embedding) publicly released to promote
research on Korean language models.
Can Large Language Models Detect Errors in Long Chain-of-Thought
  Reasoning?
Authors: name: Yancheng He, name: Shilong Li, name: Jiaheng Liu, name: Weixun Wang, name: Xingyuan Bu, name: Ge Zhang, name: Zhongyuan Peng, name: Zhaoxiang Zhang, name: Wenbo Su, name: Bo Zheng
Published: 2025-02-26 17:59:27

Recently, o1-like models have drawn significant attention, where these models
produce the long Chain-of-Thought (CoT) reasoning steps to improve the
reasoning abilities of existing Large Language Models (LLMs). In this paper, to
understand the qualities of these long CoTs and measure the critique abilities
of existing LLMs on these long CoTs, we introduce the DeltaBench, including the
generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for
different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the
ability to detect errors in long CoT reasoning. Based on DeltaBench, we first
perform fine-grained analysis of the generated long CoTs to discover the
effectiveness and efficiency of different o1-like models. Then, we conduct
extensive evaluations of existing process reward models (PRMs) and critic
models to detect the errors of each annotated process, which aims to
investigate the boundaries and limitations of existing PRMs and critic models.
Finally, we hope that DeltaBench could guide developers to better understand
the long CoT reasoning abilities of their models.
MolSpectra: Pre-training 3D Molecular Representation with Multi-modal
  Energy Spectra
Authors: name: Liang Wang, name: Shaozhen Liu, name: Yu Rong, name: Deli Zhao, name: Qiang Liu, name: Shu Wu, name: Liang Wang
Published: 2025-02-22 16:34:32

Establishing the relationship between 3D structures and the energy states of
molecular systems has proven to be a promising approach for learning 3D
molecular representations. However, existing methods are limited to modeling
the molecular energy states from classical mechanics. This limitation results
in a significant oversight of quantum mechanical effects, such as quantized
(discrete) energy level structures, which offer a more accurate estimation of
molecular energy and can be experimentally measured through energy spectra. In
this paper, we propose to utilize the energy spectra to enhance the
pre-training of 3D molecular representations (MolSpectra), thereby infusing the
knowledge of quantum mechanics into the molecular representations.
Specifically, we propose SpecFormer, a multi-spectrum encoder for encoding
molecular spectra via masked patch reconstruction. By further aligning outputs
from the 3D encoder and spectrum encoder using a contrastive objective, we
enhance the 3D encoder's understanding of molecules. Evaluations on public
benchmarks reveal that our pre-trained representations surpass existing methods
in predicting molecular properties and modeling dynamics.
Towards an AI co-scientist
Authors: name: Juraj Gottweis, name: Wei-Hung Weng, name: Alexander Daryin, name: Tao Tu, name: Anil Palepu, name: Petar Sirkovic, name: Artiom Myaskovsky, name: Felix Weissenberger, name: Keran Rong, name: Ryutaro Tanno, name: Khaled Saab, name: Dan Popovici, name: Jacob Blum, name: Fan Zhang, name: Katherine Chou, name: Avinatan Hassidim, name: Burak Gokturk, name: Amin Vahdat, name: Pushmeet Kohli, name: Yossi Matias, name: Andrew Carroll, name: Kavita Kulkarni, name: Nenad Tomasev, name: Yuan Guan, name: Vikram Dhillon, name: Eeshit Dhaval Vaishnav, name: Byron Lee, name: Tiago R D Costa, name: José R Penadés, name: Gary Peltz, name: Yunhan Xu, name: Annalisa Pawlosky, name: Alan Karthikesalingam, name: Vivek Natarajan
Published: 2025-02-26 06:17:13

Scientific discovery relies on scientists generating novel hypotheses that
undergo rigorous experimental validation. To augment this process, we introduce
an AI co-scientist, a multi-agent system built on Gemini 2.0. The AI
co-scientist is intended to help uncover new, original knowledge and to
formulate demonstrably novel research hypotheses and proposals, building upon
prior evidence and aligned to scientist-provided research objectives and
guidance. The system's design incorporates a generate, debate, and evolve
approach to hypothesis generation, inspired by the scientific method and
accelerated by scaling test-time compute. Key contributions include: (1) a
multi-agent architecture with an asynchronous task execution framework for
flexible compute scaling; (2) a tournament evolution process for self-improving
hypotheses generation. Automated evaluations show continued benefits of
test-time compute, improving hypothesis quality. While general purpose, we
focus development and validation in three biomedical areas: drug repurposing,
novel target discovery, and explaining mechanisms of bacterial evolution and
anti-microbial resistance. For drug repurposing, the system proposes candidates
with promising validation findings, including candidates for acute myeloid
leukemia that show tumor inhibition in vitro at clinically applicable
concentrations. For novel target discovery, the AI co-scientist proposed new
epigenetic targets for liver fibrosis, validated by anti-fibrotic activity and
liver cell regeneration in human hepatic organoids. Finally, the AI
co-scientist recapitulated unpublished experimental results via a parallel in
silico discovery of a novel gene transfer mechanism in bacterial evolution.
These results, detailed in separate, co-timed reports, demonstrate the
potential to augment biomedical and scientific discovery and usher an era of AI
empowered scientists.
AISafetyLab: A Comprehensive Framework for AI Safety Evaluation and
  Improvement
Authors: name: Zhexin Zhang, name: Leqi Lei, name: Junxiao Yang, name: Xijie Huang, name: Yida Lu, name: Shiyao Cui, name: Renmiao Chen, name: Qinglin Zhang, name: Xinyuan Wang, name: Hao Wang, name: Hao Li, name: Xianqi Lei, name: Chengwei Pan, name: Lei Sha, name: Hongning Wang, name: Minlie Huang
Published: 2025-02-24 02:11:52

As AI models are increasingly deployed across diverse real-world scenarios,
ensuring their safety remains a critical yet underexplored challenge. While
substantial efforts have been made to evaluate and enhance AI safety, the lack
of a standardized framework and comprehensive toolkit poses significant
obstacles to systematic research and practical adoption. To bridge this gap, we
introduce AISafetyLab, a unified framework and toolkit that integrates
representative attack, defense, and evaluation methodologies for AI safety.
AISafetyLab features an intuitive interface that enables developers to
seamlessly apply various techniques while maintaining a well-structured and
extensible codebase for future advancements. Additionally, we conduct empirical
studies on Vicuna, analyzing different attack and defense strategies to provide
valuable insights into their comparative effectiveness. To facilitate ongoing
research and development in AI safety, AISafetyLab is publicly available at
https://github.com/thu-coai/AISafetyLab, and we are committed to its continuous
maintenance and improvement.
Distill Any Depth: Distillation Creates a Stronger Monocular Depth
  Estimator
Authors: name: Xiankang He, name: Dongyan Guo, name: Hongji Li, name: Ruibo Li, name: Ying Cui, name: Chi Zhang
Published: 2025-02-26 15:10:05

Monocular depth estimation (MDE) aims to predict scene depth from a single
RGB image and plays a crucial role in 3D scene understanding. Recent advances
in zero-shot MDE leverage normalized depth representations and
distillation-based learning to improve generalization across diverse scenes.
However, current depth normalization methods for distillation, relying on
global normalization, can amplify noisy pseudo-labels, reducing distillation
effectiveness. In this paper, we systematically analyze the impact of different
depth normalization strategies on pseudo-label distillation. Based on our
findings, we propose Cross-Context Distillation, which integrates global and
local depth cues to enhance pseudo-label quality. Additionally, we introduce a
multi-teacher distillation framework that leverages complementary strengths of
different depth estimation models, leading to more robust and accurate depth
predictions. Extensive experiments on benchmark datasets demonstrate that our
approach significantly outperforms state-of-the-art methods, both
quantitatively and qualitatively.
TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem
  Understanding
Authors: name: Max Ku, name: Thomas Chong, name: Jonathan Leung, name: Krish Shah, name: Alvin Yu, name: Wenhu Chen
Published: 2025-02-26 18:50:09

Understanding domain-specific theorems often requires more than just
text-based reasoning; effective communication through structured visual
explanations is crucial for deeper comprehension. While large language models
(LLMs) demonstrate strong performance in text-based theorem reasoning, their
ability to generate coherent and pedagogically meaningful visual explanations
remains an open challenge. In this work, we introduce TheoremExplainAgent, an
agentic approach for generating long-form theorem explanation videos (over 5
minutes) using Manim animations. To systematically evaluate multimodal theorem
explanations, we propose TheoremExplainBench, a benchmark covering 240 theorems
across multiple STEM disciplines, along with 5 automated evaluation metrics.
Our results reveal that agentic planning is essential for generating detailed
long-form videos, and the o3-mini agent achieves a success rate of 93.8% and an
overall score of 0.77. However, our quantitative and qualitative studies show
that most of the videos produced exhibit minor issues with visual element
layout. Furthermore, multimodal explanations expose deeper reasoning flaws that
text-based explanations fail to reveal, highlighting the importance of
multimodal explanations.
Agentic Reward Modeling: Integrating Human Preferences with Verifiable
  Correctness Signals for Reliable Reward Systems
Authors: name: Hao Peng, name: Yunjia Qi, name: Xiaozhi Wang, name: Zijun Yao, name: Bin Xu, name: Lei Hou, name: Juanzi Li
Published: 2025-02-26 17:19:12

Reward models (RMs) are crucial for the training and inference-time scaling
up of large language models (LLMs). However, existing reward models primarily
focus on human preferences, neglecting verifiable correctness signals which
have shown strong potential in training LLMs. In this paper, we propose agentic
reward modeling, a reward system that combines reward models with verifiable
correctness signals from different aspects to provide reliable rewards. We
empirically implement a reward agent, named RewardAgent, that combines human
preference rewards with two verifiable signals: factuality and instruction
following, to provide more reliable rewards. We conduct comprehensive
experiments on existing reward model benchmarks and inference time best-of-n
searches on real-world downstream tasks. RewardAgent significantly outperforms
vanilla reward models, demonstrating its effectiveness. We further construct
training preference pairs using RewardAgent and train an LLM with the DPO
objective, achieving superior performance on various NLP benchmarks compared to
conventional reward models. Our codes are publicly released to facilitate
further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).