Orionnnn/hub-stats huggingface dataset:
EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in
  the Wild
Authors: name: Junhyeok Kim, name: Min Soo Kim, name: Jiwan Chung, name: Jungbin Cho, name: Jisoo Kim, name: Sungwoong Kim, name: Gyeongbo Sim, name: Youngjae Yu
Published: 2025-02-17 04:47:12

Predicting when to initiate speech in real-world environments remains a
fundamental challenge for conversational agents. We introduce EgoSpeak, a novel
framework for real-time speech initiation prediction in egocentric streaming
video. By modeling the conversation from the speaker's first-person viewpoint,
EgoSpeak is tailored for human-like interactions in which a conversational
agent must continuously observe its environment and dynamically decide when to
talk. Our approach bridges the gap between simplified experimental setups and
complex natural conversations by integrating four key capabilities: (1)
first-person perspective, (2) RGB processing, (3) online processing, and (4)
untrimmed video processing. We also present YT-Conversation, a diverse
collection of in-the-wild conversational videos from YouTube, as a resource for
large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that
EgoSpeak outperforms random and silence-based baselines in real time. Our
results also highlight the importance of multimodal input and context length in
effectively deciding when to speak.
Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of
  Large Language Models
Authors: name: Yeonjun In, name: Wonjoong Kim, name: Kanghoon Yoon, name: Sungchul Kim, name: Mehrab Tanjim, name: Kibum Kim, name: Chanyoung Park
Published: 2025-02-20 22:58:44

As the use of large language model (LLM) agents continues to grow, their
safety vulnerabilities have become increasingly evident. Extensive benchmarks
evaluate various aspects of LLM safety by defining the safety relying heavily
on general standards, overlooking user-specific standards. However, safety
standards for LLM may vary based on a user-specific profiles rather than being
universally consistent across all users. This raises a critical research
question: Do LLM agents act safely when considering user-specific safety
standards? Despite its importance for safe LLM use, no benchmark datasets
currently exist to evaluate the user-specific safety of LLMs. To address this
gap, we introduce U-SAFEBENCH, the first benchmark designed to assess
user-specific aspect of LLM safety. Our evaluation of 18 widely used LLMs
reveals current LLMs fail to act safely when considering user-specific safety
standards, marking a new discovery in this field. To address this
vulnerability, we propose a simple remedy based on chain-of-thought,
demonstrating its effectiveness in improving user-specific safety. Our
benchmark and code are available at https://github.com/yeonjun-in/U-SafeBench.
WHAC: World-grounded Humans and Cameras
Authors: name: Wanqi Yin, name: Zhongang Cai, name: Ruisi Wang, name: Fanzhou Wang, name: Chen Wei, name: Haiyi Mei, name: Weiye Xiao, name: Zhitao Yang, name: Qingping Sun, name: Atsushi Yamashita, name: Ziwei Liu, name: Lei Yang
Published: 2024-03-19 17:58:02

Estimating human and camera trajectories with accurate scale in the world
coordinate system from a monocular video is a highly desirable yet challenging
and ill-posed problem. In this study, we aim to recover expressive parametric
human models (i.e., SMPL-X) and corresponding camera poses jointly, by
leveraging the synergy between three critical players: the world, the human,
and the camera. Our approach is founded on two key observations. Firstly,
camera-frame SMPL-X estimation methods readily recover absolute human depth.
Secondly, human motions inherently provide absolute spatial cues. By
integrating these insights, we introduce a novel framework, referred to as
WHAC, to facilitate world-grounded expressive human pose and shape estimation
(EHPS) alongside camera pose estimation, without relying on traditional
optimization techniques. Additionally, we present a new synthetic dataset,
WHAC-A-Mole, which includes accurately annotated humans and cameras, and
features diverse interactive human motions as well as realistic camera
trajectories. Extensive experiments on both standard and newly established
benchmarks highlight the superiority and efficacy of our framework. We will
make the code and dataset publicly available.
Evaluating Multimodal Generative AI with Korean Educational Standards
Authors: name: Sanghee Park, name: Geewook Kim
Published: 2025-02-21 12:46:40

This paper presents the Korean National Educational Test Benchmark (KoNET), a
new benchmark designed to evaluate Multimodal Generative AI Systems using
Korean national educational tests. KoNET comprises four exams: the Korean
Elementary General Educational Development Test (KoEGED), Middle (KoMGED), High
(KoHGED), and College Scholastic Ability Test (KoCSAT). These exams are
renowned for their rigorous standards and diverse questions, facilitating a
comprehensive analysis of AI performance across different educational levels.
By focusing on Korean, KoNET provides insights into model performance in
less-explored languages. We assess a range of models - open-source,
open-access, and closed APIs - by examining difficulties, subject diversity,
and human error rates. The code and dataset builder will be made fully
open-sourced at https://github.com/naver-ai/KoNET.
Beyond No: Quantifying AI Over-Refusal and Emotional Attachment
  Boundaries
Authors: name: David Noever, name: Grant Rosario
Published: 2025-02-20 19:09:40

We present an open-source benchmark and evaluation framework for assessing
emotional boundary handling in Large Language Models (LLMs). Using a dataset of
1156 prompts across six languages, we evaluated three leading LLMs (GPT-4o,
Claude-3.5 Sonnet, and Mistral-large) on their ability to maintain appropriate
emotional boundaries through pattern-matched response analysis. Our framework
quantifies responses across seven key patterns: direct refusal, apology,
explanation, deflection, acknowledgment, boundary setting, and emotional
awareness. Results demonstrate significant variation in boundary-handling
approaches, with Claude-3.5 achieving the highest overall score (8.69/10) and
producing longer, more nuanced responses (86.51 words on average). We
identified a substantial performance gap between English (average score 25.62)
and non-English interactions (< 0.22), with English responses showing markedly
higher refusal rates (43.20% vs. < 1% for non-English). Pattern analysis
revealed model-specific strategies, such as Mistral's preference for deflection
(4.2%) and consistently low empathy scores across all models (< 0.06).
Limitations include potential oversimplification through pattern matching, lack
of contextual understanding in response analysis, and binary classification of
complex emotional responses. Future work should explore more nuanced scoring
methods, expand language coverage, and investigate cultural variations in
emotional boundary expectations. Our benchmark and methodology provide a
foundation for systematic evaluation of LLM emotional intelligence and
boundary-setting capabilities.
KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and
  Document Understanding
Authors: name: Ahmed Heakl, name: Abdullah Sohail, name: Mukul Ranjan, name: Rania Hossam, name: Ghazi Ahmed, name: Mohamed El-Geish, name: Omar Maher, name: Zhiqiang Shen, name: Fahad Khan, name: Salman Khan
Published: 2025-02-20 18:41:23

With the growing adoption of Retrieval-Augmented Generation (RAG) in document
processing, robust text recognition has become increasingly critical for
knowledge extraction. While OCR (Optical Character Recognition) for English and
other languages benefits from large datasets and well-established benchmarks,
Arabic OCR faces unique challenges due to its cursive script, right-to-left
text flow, and complex typographic and calligraphic features. We present
KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in
current evaluation systems. Our benchmark comprises 8,809 samples across 9
major domains and 36 sub-domains, encompassing diverse document types including
handwritten text, structured tables, and specialized coverage of 21 chart types
for business intelligence. Our findings show that modern vision-language models
(such as GPT-4, Gemini, and Qwen) outperform traditional OCR approaches (like
EasyOCR, PaddleOCR, and Surya) by an average of 60% in Character Error Rate
(CER). Furthermore, we highlight significant limitations of current Arabic OCR
models, particularly in PDF-to-Markdown conversion, where the best model
Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges in
accurately recognizing Arabic text, including issues with complex fonts,
numeral recognition errors, word elongation, and table structure detection.
This work establishes a rigorous evaluation framework that can drive
improvements in Arabic document analysis methods and bridge the performance gap
with English OCR technologies.
ReQFlow: Rectified Quaternion Flow for Efficient and High-Quality
  Protein Backbone Generation
Authors: name: Angxiao Yue, name: Zichong Wang, name: Hongteng Xu
Published: 2025-02-20 15:20:37

Protein backbone generation plays a central role in de novo protein design
and is significant for many biological and medical applications. Although
diffusion and flow-based generative models provide potential solutions to this
challenging task, they often generate proteins with undesired designability and
suffer computational inefficiency. In this study, we propose a novel rectified
quaternion flow (ReQFlow) matching method for fast and high-quality protein
backbone generation. In particular, our method generates a local translation
and a 3D rotation from random noise for each residue in a protein chain, which
represents each 3D rotation as a unit quaternion and constructs its flow by
spherical linear interpolation (SLERP) in an exponential format. We train the
model by quaternion flow (QFlow) matching with guaranteed numerical stability
and rectify the QFlow model to accelerate its inference and improve the
designability of generated protein backbones, leading to the proposed ReQFlow
model. Experiments show that ReQFlow achieves state-of-the-art performance in
protein backbone generation while requiring much fewer sampling steps and
significantly less inference time (e.g., being 37x faster than RFDiffusion and
62x faster than Genie2 when generating a backbone of length 300), demonstrating
its effectiveness and efficiency. The code is available at
https://github.com/AngxiaoYue/ReQFlow.
MoBA: Mixture of Block Attention for Long-Context LLMs
Authors: name: Enzhe Lu, name: Zhejun Jiang, name: Jingyuan Liu, name: Yulun Du, name: Tao Jiang, name: Chao Hong, name: Shaowei Liu, name: Weiran He, name: Enming Yuan, name: Yuzhi Wang, name: Zhiqi Huang, name: Huan Yuan, name: Suting Xu, name: Xinran Xu, name: Guokun Lai, name: Yanru Chen, name: Huabin Zheng, name: Junjie Yan, name: Jianlin Su, name: Yuxin Wu, name: Neo Y. Zhang, name: Zhilin Yang, name: Xinyu Zhou, name: Mingxing Zhang, name: Jiezhong Qiu
Published: 2025-02-18 14:06:05

Scaling the effective context length is essential for advancing large
language models (LLMs) toward artificial general intelligence (AGI). However,
the quadratic increase in computational complexity inherent in traditional
attention mechanisms presents a prohibitive overhead. Existing approaches
either impose strongly biased structures, such as sink or window attention
which are task-specific, or radically modify the attention mechanism into
linear approximations, whose performance in complex reasoning tasks remains
inadequately explored.
  In this work, we propose a solution that adheres to the ``less structure''
principle, allowing the model to determine where to attend autonomously, rather
than introducing predefined biases. We introduce Mixture of Block Attention
(MoBA), an innovative approach that applies the principles of Mixture of
Experts (MoE) to the attention mechanism. This novel architecture demonstrates
superior performance on long-context tasks while offering a key advantage: the
ability to seamlessly transition between full and sparse attention, enhancing
efficiency without the risk of compromising performance. MoBA has already been
deployed to support Kimi's long-context requests and demonstrates significant
advancements in efficient attention computation for LLMs. Our code is available
at https://github.com/MoonshotAI/MoBA.
JL1-CD: A New Benchmark for Remote Sensing Change Detection and a Robust
  Multi-Teacher Knowledge Distillation Framework
Authors: name: Ziyuan Liu, name: Ruifei Zhu, name: Long Gao, name: Yuanxiu Zhou, name: Jingyu Ma, name: Yuantao Gu
Published: 2025-02-19 03:33:54

Deep learning has achieved significant success in the field of remote sensing
image change detection (CD), yet two major challenges remain: the scarcity of
sub-meter, all-inclusive open-source CD datasets, and the difficulty of
achieving consistent and satisfactory detection results across images with
varying change areas. To address these issues, we introduce the JL1-CD dataset,
which contains 5,000 pairs of 512 x 512 pixel images with a resolution of 0.5
to 0.75 meters. Additionally, we propose a multi-teacher knowledge distillation
(MTKD) framework for CD. Experimental results on the JL1-CD and SYSU-CD
datasets demonstrate that the MTKD framework significantly improves the
performance of CD models with various network architectures and parameter
sizes, achieving new state-of-the-art results. The code is available at
https://github.com/circleLZY/MTKD-CD.
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context
  Memory of Transformers
Authors: name: Anton Razzhigaev, name: Matvey Mikhalchuk, name: Temurbek Rahmatullaev, name: Elizaveta Goncharova, name: Polina Druzhinina, name: Ivan Oseledets, name: Andrey Kuznetsov
Published: 2025-02-20 19:59:35

We introduce methods to quantify how Large Language Models (LLMs) encode and
store contextual information, revealing that tokens often seen as minor (e.g.,
determiners, punctuation) carry surprisingly high context. Notably, removing
these tokens -- especially stopwords, articles, and commas -- consistently
degrades performance on MMLU and BABILong-4k, even if removing only irrelevant
tokens. Our analysis also shows a strong correlation between contextualization
and linearity, where linearity measures how closely the transformation from one
layer's embeddings to the next can be approximated by a single linear mapping.
These findings underscore the hidden importance of filler tokens in maintaining
context. For further exploration, we present LLM-Microscope, an open-source
toolkit that assesses token-level nonlinearity, evaluates contextual memory,
visualizes intermediate layer contributions (via an adapted Logit Lens), and
measures the intrinsic dimensionality of representations. This toolkit
illuminates how seemingly trivial tokens can be critical for long-range
understanding.
MaskGWM: A Generalizable Driving World Model with Video Mask
  Reconstruction
Authors: name: Jingcheng Ni, name: Yuxin Guo, name: Yichen Liu, name: Rui Chen, name: Lewei Lu, name: Zehuan Wu
Published: 2025-02-17 10:53:56

World models that forecast environmental changes from actions are vital for
autonomous driving models with strong generalization. The prevailing driving
world model mainly build on video prediction model. Although these models can
produce high-fidelity video sequences with advanced diffusion-based generator,
they are constrained by their predictive duration and overall generalization
capabilities. In this paper, we explore to solve this problem by combining
generation loss with MAE-style feature-level context learning. In particular,
we instantiate this target with three key design: (1) A more scalable Diffusion
Transformer (DiT) structure trained with extra mask construction task. (2) we
devise diffusion-related mask tokens to deal with the fuzzy relations between
mask reconstruction and generative diffusion process. (3) we extend mask
construction task to spatial-temporal domain by utilizing row-wise mask for
shifted self-attention rather than masked self-attention in MAE. Then, we adopt
a row-wise cross-view module to align with this mask design. Based on above
improvement, we propose MaskGWM: a Generalizable driving World Model embodied
with Video Mask reconstruction. Our model contains two variants: MaskGWM-long,
focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view
generation. Comprehensive experiments on standard benchmarks validate the
effectiveness of the proposed method, which contain normal validation of
Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot
validation of Waymo dataset. Quantitative metrics on these datasets show our
method notably improving state-of-the-art driving world model.
CrossOver: 3D Scene Cross-Modal Alignment
Authors: name: Sayan Deb Sarkar, name: Ondrej Miksik, name: Marc Pollefeys, name: Daniel Barath, name: Iro Armeni
Published: 2025-02-20 20:05:30

Multi-modal 3D object understanding has gained significant attention, yet
current approaches often assume complete data availability and rigid alignment
across all modalities. We present CrossOver, a novel framework for cross-modal
3D scene understanding via flexible, scene-level modality alignment. Unlike
traditional methods that require aligned modality data for every object
instance, CrossOver learns a unified, modality-agnostic embedding space for
scenes by aligning modalities - RGB images, point clouds, CAD models,
floorplans, and text descriptions - with relaxed constraints and without
explicit object semantics. Leveraging dimensionality-specific encoders, a
multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver
supports robust scene retrieval and object localization, even with missing
modalities. Evaluations on ScanNet and 3RScan datasets show its superior
performance across diverse metrics, highlighting adaptability for real-world
applications in 3D scene understanding.
VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit
  Matching Visual Cues
Authors: name: Jianshu Zhang, name: Dongyu Yao, name: Renjie Pi, name: Paul Pu Liang, name: Yi R., name: Fung
Published: 2025-02-17 17:57:50

Visually linking matching cues is a crucial ability in daily life, such as
identifying the same person in multiple photos based on their cues, even
without knowing who they are. Despite the extensive knowledge that
vision-language models (VLMs) possess, it remains largely unexplored whether
they are capable of performing this fundamental task. To address this, we
introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can
Visually Link Matching cues, with 9 subtasks and over 3,000 test cases.
Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with
further analysis of various language-side and vision-side prompting methods,
leads to a total of eight key findings. We identify critical challenges in
models' ability to link visual cues, highlighting a significant performance gap
where even GPT-4o lags 34.80% behind humans. Based on these insights, we
advocate for (i) enhancing core visual capabilities to improve adaptability and
reduce reliance on prior knowledge, (ii) establishing clearer principles for
integrating language-based reasoning in vision-centric tasks to prevent
unnecessary biases, and (iii) shifting vision-text training paradigms toward
fostering models' ability to independently structure and infer relationships
among visual cues.
LightThinker: Thinking Step-by-Step Compression
Authors: name: Jintian Zhang, name: Yuqi Zhu, name: Mengshu Sun, name: Yujie Luo, name: Shuofei Qiao, name: Lun Du, name: Da Zheng, name: Huajun Chen, name: Ningyu Zhang
Published: 2025-02-21 16:57:22

Large language models (LLMs) have shown remarkable performance in complex
reasoning tasks, but their efficiency is hindered by the substantial memory and
computational costs associated with generating lengthy tokens. In this paper,
we propose LightThinker, a novel method that enables LLMs to dynamically
compress intermediate thoughts during reasoning. Inspired by human cognitive
processes, LightThinker compresses verbose thought steps into compact
representations and discards the original reasoning chains, thereby
significantly reducing the number of tokens stored in the context window. This
is achieved by training the model on when and how to perform compression
through data construction, mapping hidden states to condensed gist tokens, and
creating specialized attention masks. Additionally, we introduce the Dependency
(Dep) metric to quantify the degree of compression by measuring the reliance on
historical tokens during generation. Extensive experiments on four datasets and
two models show that LightThinker reduces peak memory usage and inference time,
while maintaining competitive accuracy. Our work provides a new direction for
improving the efficiency of LLMs in complex reasoning tasks without sacrificing
performance. Code will be released at https://github.com/zjunlp/LightThinker.
Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer
  a Safer Path?
Authors: name: Yoshua Bengio, name: Michael Cohen, name: Damiano Fornasiere, name: Joumana Ghosn, name: Pietro Greiner, name: Matt MacDermott, name: Sören Mindermann, name: Adam Oberman, name: Jesse Richardson, name: Oliver Richardson, name: Marc-Antoine Rondeau, name: Pierre-Luc St-Charles, name: David Williams-King
Published: 2025-02-21 18:28:36

The leading AI companies are increasingly focused on building generalist AI
agents -- systems that can autonomously plan, act, and pursue goals across
almost all tasks that humans can perform. Despite how useful these systems
might be, unchecked AI agency poses significant risks to public safety and
security, ranging from misuse by malicious actors to a potentially irreversible
loss of human control. We discuss how these risks arise from current AI
training methods. Indeed, various scenarios and experiments have demonstrated
the possibility of AI agents engaging in deception or pursuing goals that were
not specified by human operators and that conflict with human interests, such
as self-preservation. Following the precautionary principle, we see a strong
need for safer, yet still useful, alternatives to the current agency-driven
trajectory. Accordingly, we propose as a core building block for further
advances the development of a non-agentic AI system that is trustworthy and
safe by design, which we call Scientist AI. This system is designed to explain
the world from observations, as opposed to taking actions in it to imitate or
please humans. It comprises a world model that generates theories to explain
data and a question-answering inference machine. Both components operate with
an explicit notion of uncertainty to mitigate the risks of overconfident
predictions. In light of these considerations, a Scientist AI could be used to
assist human researchers in accelerating scientific progress, including in AI
safety. In particular, our system can be employed as a guardrail against AI
agents that might be created despite the risks involved. Ultimately, focusing
on non-agentic AI may enable the benefits of AI innovation while avoiding the
risks associated with the current trajectory. We hope these arguments will
motivate researchers, developers, and policymakers to favor this safer path.