Orionnnn/hub-stats huggingface dataset:
FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal
  Large Language Models
Authors: name: Yao Zhang, name: Hewei Gao, name: Haokun Chen, name: Weiguo Li, name: Yunpu Ma, name: Volker Tresp
Published: 2025-06-12 17:50:50

Multimodal Large Language Models (MLLMs) excel in tasks like multimodal
reasoning and cross-modal retrieval but face deployment challenges in
real-world scenarios due to distributed multimodal data and strict privacy
requirements. Federated Learning (FL) offers a solution by enabling
collaborative model training without centralizing data. However, realizing FL
for MLLMs presents significant challenges, including high computational
demands, limited client capacity, substantial communication costs, and
heterogeneous client data. Existing FL methods assume client-side deployment of
full models, an assumption that breaks down for large-scale MLLMs due to their
massive size and communication demands. To address these limitations, we
propose FedNano, the first FL framework that centralizes the LLM on the server
while introducing NanoEdge, a lightweight module for client-specific
adaptation. NanoEdge employs modality-specific encoders, connectors, and
trainable NanoAdapters with low-rank adaptation. This design eliminates the
need to deploy LLM on clients, reducing client-side storage by 95%, and
limiting communication overhead to only 0.01% of the model parameters. By
transmitting only compact NanoAdapter updates, FedNano handles heterogeneous
client data and resource constraints while preserving privacy. Experiments
demonstrate that FedNano outperforms prior FL baselines, bridging the gap
between MLLM scale and FL feasibility, and enabling scalable, decentralized
multimodal AI systems.
SwarmAgentic: Towards Fully Automated Agentic System Generation via
  Swarm Intelligence
Authors: name: Yao Zhang, name: Chenyang Lin, name: Shijie Tang, name: Haokun Chen, name: Shijie Zhou, name: Yunpu Ma, name: Volker Tresp
Published: 2025-06-18 17:54:55

The rapid progress of Large Language Models has advanced agentic systems in
decision-making, coordination, and task execution. Yet, existing agentic system
generation frameworks lack full autonomy, missing from-scratch agent
generation, self-optimizing agent functionality, and collaboration, limiting
adaptability and scalability. We propose SwarmAgentic, a framework for fully
automated agentic system generation that constructs agentic systems from
scratch and jointly optimizes agent functionality and collaboration as
interdependent components through language-driven exploration. To enable
efficient search over system-level structures, SwarmAgentic maintains a
population of candidate systems and evolves them via feedback-guided updates,
drawing inspiration from Particle Swarm Optimization (PSO). We evaluate our
method on six real-world, open-ended, and exploratory tasks involving
high-level planning, system-level coordination, and creative reasoning. Given
only a task description and an objective function, SwarmAgentic outperforms all
baselines, achieving a +261.8% relative improvement over ADAS on the
TravelPlanner benchmark, highlighting the effectiveness of full automation in
structurally unconstrained tasks. This framework marks a significant step
toward scalable and autonomous agentic system design, bridging swarm
intelligence with fully automated system multi-agent generation. Our code is
publicly released at https://yaoz720.github.io/SwarmAgentic/.
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form
  Generation
Authors: name: Zongxia Li, name: Yapei Chang, name: Yuhang Zhou, name: Xiyang Wu, name: Zichao Liang, name: Yoo Yeon Sung, name: Jordan Lee Boyd-Graber
Published: 2025-06-18 02:16:53

Evaluating open-ended long-form generation is challenging because it is hard
to define what clearly separates good from bad outputs. Existing methods often
miss key aspects like coherence, style, or relevance, or are biased by
pretraining data, making open-ended long-form evaluation an underexplored
problem. To address this gap, we propose PrefBERT, a scoring model for
evaluating open-ended long-form generation in GRPO and guiding its training
with distinct rewards for good and bad outputs. Trained on two response
evaluation datasets with diverse long-form styles and Likert-rated quality,
PrefBERT effectively supports GRPO by offering better semantic reward feedback
than traditional metrics ROUGE-L and BERTScore do. Through comprehensive
evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis,
we show that PrefBERT, trained on multi-sentence and paragraph-length
responses, remains reliable across varied long passages and aligns well with
the verifiable rewards GRPO needs. Human evaluations confirm that using
PrefBERT as the reward signal to train policy models yields responses better
aligned with human preferences than those trained with traditional metrics. Our
code is available at https://github.com/zli12321/long_form_rl.
Truncated Proximal Policy Optimization
Authors: name: Tiantian Fan, name: Lingjun Liu, name: Yu Yue, name: Jiaze Chen, name: Chengyi Wang, name: Qiying Yu, name: Chi Zhang, name: Zhiqi Lin, name: Ruofei Zhu, name: Yufeng Yuan, name: Xiaochen Zuo, name: Bole Ma, name: Mofan Zhang, name: Gaohong Liu, name: Ru Zhang, name: Haotian Zhou, name: Cong Xie, name: Ruidong Zhu, name: Zhi Zhang, name: Xin Liu, name: Mingxuan Wang, name: Lin Yan, name: Yonghui Wu
Published: 2025-06-18 01:21:38

Recently, test-time scaling Large Language Models (LLMs) have demonstrated
exceptional reasoning capabilities across scientific and professional tasks by
generating long chains-of-thought (CoT). As a crucial component for developing
these reasoning models, reinforcement learning (RL), exemplified by Proximal
Policy Optimization (PPO) and its variants, allows models to learn through
trial and error. However, PPO can be time-consuming due to its inherent
on-policy nature, which is further exacerbated by increasing response lengths.
In this work, we propose Truncated Proximal Policy Optimization (T-PPO), a
novel extension to PPO that improves training efficiency by streamlining policy
update and length-restricted response generation. T-PPO mitigates the issue of
low hardware utilization, an inherent drawback of fully synchronized
long-generation procedures, where resources often sit idle during the waiting
periods for complete rollouts. Our contributions are two-folds. First, we
propose Extended Generalized Advantage Estimation (EGAE) for advantage
estimation derived from incomplete responses while maintaining the integrity of
policy learning. Second, we devise a computationally optimized mechanism that
allows for the independent optimization of the policy and value models. By
selectively filtering prompt and truncated tokens, this mechanism reduces
redundant computations and accelerates the training process without sacrificing
convergence performance. We demonstrate the effectiveness and efficacy of T-PPO
on AIME 2024 with a 32B base model. The experimental results show that T-PPO
improves the training efficiency of reasoning LLMs by up to 2.5x and
outperforms its existing competitors.
Sekai: A Video Dataset towards World Exploration
Authors: name: Zhen Li, name: Chuanhao Li, name: Xiaofeng Mao, name: Shaoheng Lin, name: Ming Li, name: Shitian Zhao, name: Zhaopan Xu, name: Xinyue Li, name: Yukang Feng, name: Jianwen Sun, name: Zizhen Li, name: Fanrui Zhang, name: Jiaxin Ai, name: Zhixiang Wang, name: Yuwei Wu, name: Tong He, name: Jiangmiao Pang, name: Yu Qiao, name: Yunde Jia, name: Kaipeng Zhang
Published: 2025-06-18 17:57:06

Video generation techniques have made remarkable progress, promising to be
the foundation of interactive world exploration. However, existing video
generation datasets are not well-suited for world exploration training as they
suffer from some limitations: limited locations, short duration, static scenes,
and a lack of annotations about exploration and the world. In this paper, we
introduce Sekai (meaning ``world'' in Japanese), a high-quality first-person
view worldwide video dataset with rich annotations for world exploration. It
consists of over 5,000 hours of walking or drone view (FPV and UVA) videos from
over 100 countries and regions across 750 cities. We develop an efficient and
effective toolbox to collect, pre-process and annotate videos with location,
scene, weather, crowd density, captions, and camera trajectories. Experiments
demonstrate the quality of the dataset. And, we use a subset to train an
interactive video world exploration model, named YUME (meaning ``dream'' in
Japanese). We believe Sekai will benefit the area of video generation and world
exploration, and motivate valuable applications.
SciVer: Evaluating Foundation Models for Multimodal Scientific Claim
  Verification
Authors: name: Chengye Wang, name: Yifei Shen, name: Zexi Kuang, name: Arman Cohan, name: Yilun Zhao
Published: 2025-06-18 15:43:26

We introduce SciVer, the first benchmark specifically designed to evaluate
the ability of foundation models to verify claims within a multimodal
scientific context. SciVer consists of 3,000 expert-annotated examples over
1,113 scientific papers, covering four subsets, each representing a common
reasoning type in multimodal scientific claim verification. To enable
fine-grained evaluation, each example includes expert-annotated supporting
evidence. We assess the performance of 21 state-of-the-art multimodal
foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and
Qwen2.5-VL. Our experiment reveals a substantial performance gap between these
models and human experts on SciVer. Through an in-depth analysis of
retrieval-augmented generation (RAG), and human-conducted error evaluations, we
identify critical limitations in current open-source models, offering key
insights to advance models' comprehension and reasoning in multimodal
scientific literature tasks.
MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal
  Models
Authors: name: Hongyu Wang, name: Jiayu Xu, name: Ruiping Wang, name: Yan Feng, name: Yitao Zhai, name: Peng Pei, name: Xunliang Cai, name: Xilin Chen
Published: 2025-06-17 11:53:49

Large multimodal Mixture-of-Experts (MoEs) effectively scale the model size
to boost performance while maintaining fixed active parameters. However,
previous works primarily utilized full-precision experts during sparse
up-cycling. Despite they show superior performance on end tasks, the large
amount of experts introduces higher memory footprint, which poses significant
challenges for the deployment on edge devices. In this work, we propose MoTE, a
scalable and memory-efficient approach to train Mixture-of-Ternary-Experts
models from dense checkpoint. Instead of training fewer high-precision experts,
we propose to train more low-precision experts during up-cycling. Specifically,
we use the pre-trained FFN as a shared expert and train ternary routed experts
with parameters in {-1, 0, 1}. Extensive experiments show that our approach has
promising scaling trend along model size. MoTE achieves comparable performance
to full-precision baseline MoE-LLaVA while offering lower memory footprint.
Furthermore, our approach is compatible with post-training quantization methods
and the advantage further amplifies when memory-constraint goes lower. Given
the same amount of expert memory footprint of 3.4GB and combined with
post-training quantization, MoTE outperforms MoE-LLaVA by a gain of 4.3%
average accuracy on end tasks, demonstrating its effectiveness and potential
for memory-constrained devices.
GenRecal: Generation after Recalibration from Large to Small
  Vision-Language Models
Authors: name: Byung-Kwan Lee, name: Ryo Hachiuma, name: Yong Man Ro, name: Yu-Chiang Frank Wang, name: Yueh-Hua Wu
Published: 2025-06-18 17:59:49

Recent advancements in vision-language models (VLMs) have leveraged large
language models (LLMs) to achieve performance on par with closed-source systems
like GPT-4V. However, deploying these models in real-world scenarios,
particularly on resource-constrained devices, remains challenging due to their
substantial computational demands. This has spurred interest in distilling
knowledge from large VLMs into smaller, more efficient counterparts. A key
challenge arises here from the diversity of VLM architectures, which are built
on different LLMs and employ varying token types-differing in vocabulary size,
token splits, and token index ordering. To address this challenge of limitation
to a specific VLM type, we present Generation after Recalibration (GenRecal), a
novel, general-purpose distillation framework for VLMs. GenRecal incorporates a
Recalibrator that aligns and adapts feature representations between
heterogeneous VLMs, enabling effective knowledge transfer across different
types of VLMs. Through extensive experiments on multiple challenging
benchmarks, we demonstrate that GenRecal significantly improves baseline
performances, eventually outperforming large-scale open- and closed-source
VLMs.
CoMemo: LVLMs Need Image Context with Image Memory
Authors: name: Shi Liu, name: Weijie Su, name: Xizhou Zhu, name: Wenhai Wang, name: Jifeng Dai
Published: 2025-06-06 17:59:06

Recent advancements in Large Vision-Language Models built upon Large Language
Models have established aligning visual features with LLM representations as
the dominant paradigm. However, inherited LLM architectural designs introduce
suboptimal characteristics for multimodal processing. First, LVLMs exhibit a
bimodal distribution in attention allocation, leading to the progressive
neglect of middle visual content as context expands. Second, conventional
positional encoding schemes fail to preserve vital 2D structural relationships
when processing dynamic high-resolution images. To address these limitations,
we propose CoMemo - a dual-path architecture that combines a Context image path
with an image Memory path for visual processing, effectively alleviating visual
information neglect. Additionally, we introduce RoPE-DHR, a novel positional
encoding mechanism that employs thumbnail-based positional aggregation to
maintain 2D spatial awareness while mitigating remote decay in extended
sequences. Evaluations across seven benchmarks,including long-context
comprehension, multi-image reasoning, and visual question answering,
demonstrate CoMemo's superior performance compared to conventional LVLM
architectures. Project page is available at
https://lalbj.github.io/projects/CoMemo/.
Embodied Web Agents: Bridging Physical-Digital Realms for Integrated
  Agent Intelligence
Authors: name: Yining Hong, name: Rui Sun, name: Bingxuan Li, name: Xingcheng Yao, name: Maxine Wu, name: Alexander Chien, name: Da Yin, name: Ying Nian Wu, name: Zhecan James Wang, name: Kai-Wei Chang
Published: 2025-06-18 17:58:17

AI agents today are mostly siloed - they either retrieve and reason over vast
amount of digital information and knowledge obtained online; or interact with
the physical world through embodied perception, planning and action - but
rarely both. This separation limits their ability to solve tasks that require
integrated physical and digital intelligence, such as cooking from online
recipes, navigating with dynamic map data, or interpreting real-world landmarks
using web knowledge. We introduce Embodied Web Agents, a novel paradigm for AI
agents that fluidly bridge embodiment and web-scale reasoning. To
operationalize this concept, we first develop the Embodied Web Agents task
environments, a unified simulation platform that tightly integrates realistic
3D indoor and outdoor environments with functional web interfaces. Building
upon this platform, we construct and release the Embodied Web Agents Benchmark,
which encompasses a diverse suite of tasks including cooking, navigation,
shopping, tourism, and geolocation - all requiring coordinated reasoning across
physical and digital realms for systematic assessment of cross-domain
intelligence. Experimental results reveal significant performance gaps between
state-of-the-art AI systems and human capabilities, establishing both
challenges and opportunities at the intersection of embodied cognition and
web-scale knowledge access. All datasets, codes and websites are publicly
available at our project page https://embodied-web-agent.github.io/.
AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
Authors: name: Jingxu Xie, name: Dylan Xu, name: Xuandong Zhao, name: Dawn Song
Published: 2025-06-17 05:46:52

We introduce AgentSynth, a scalable and cost-efficient pipeline for
automatically synthesizing high-quality tasks and trajectory datasets for
generalist computer-use agents. Leveraging information asymmetry, AgentSynth
constructs subtasks that are simple during generation but significantly more
challenging when composed into long-horizon tasks, enabling the creation of
over 6,000 diverse and realistic tasks. Our pipeline begins with an LLM-based
task proposer guided by a persona, followed by an execution agent that
completes the task and logs the trajectory. This process is repeated
iteratively to form a sequence of subtasks, which are then summarized by a
separate agent into a composite task of controllable difficulty. A key strength
of AgentSynth is its ability to precisely modulate task complexity by varying
the number of subtasks. Empirical evaluations show that state-of-the-art LLM
agents suffer a steep performance drop, from 18% success at difficulty level 1
to just 4% at level 6, highlighting the benchmark's difficulty and
discriminative power. Moreover, our pipeline achieves a low average cost of
\$0.60 per trajectory, orders of magnitude cheaper than human annotations. Our
code and data are publicly available at
https://github.com/sunblaze-ucb/AgentSynth
From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Authors: name: Mathurin Videau, name: Badr Youbi Idrissi, name: Alessandro Leite, name: Marc Schoenauer, name: Olivier Teytaud, name: David Lopez-Paz
Published: 2025-06-17 17:55:11

Tokenization imposes a fixed granularity on the input text, freezing how a
language model operates on data and how far in the future it predicts. Byte
Pair Encoding (BPE) and similar schemes split text once, build a static
vocabulary, and leave the model stuck with that choice. We relax this rigidity
by introducing an autoregressive U-Net that learns to embed its own tokens as
it trains. The network reads raw bytes, pools them into words, then pairs of
words, then up to 4 words, giving it a multi-scale view of the sequence. At
deeper stages, the model must predict further into the future -- anticipating
the next few words rather than the next byte -- so deeper stages focus on
broader semantic patterns while earlier stages handle fine details. When
carefully tuning and controlling pretraining compute, shallow hierarchies tie
strong BPE baselines, and deeper hierarchies have a promising trend. Because
tokenization now lives inside the model, the same system can handle
character-level tasks and carry knowledge across low-resource languages.
Graph Counselor: Adaptive Graph Exploration via Multi-Agent Synergy to Enhance LLM Reasoning
Authors: name: Junqi Gao, name: Xiang Zou, name: YIng Ai, name: Dong Li, name: Yichen Niu, name: Biqing Qi, name: Jianxing Liu
Published: 2025-06-04 13:31:21

Graph Retrieval Augmented Generation (GraphRAG) effectively enhances external knowledge integration capabilities by explicitly modeling knowledge relationships, thereby improving the factual accuracy and generation quality of Large Language Models (LLMs) in specialized domains. However, existing methods suffer from two inherent limitations: 1) Inefficient Information Aggregation: They rely on a single agent and fixed iterative patterns, making it difficult to adaptively capture multi-level textual, structural, and degree information within graph data. 2) Rigid Reasoning Mechanism: They employ preset reasoning schemes, which cannot dynamically adjust reasoning depth nor achieve precise semantic correction. To overcome these limitations, we propose Graph Counselor, an GraphRAG method based on multi-agent collaboration. This method uses the Adaptive Graph Information Extraction Module (AGIEM), where Planning, Thought, and Execution Agents work together to precisely model complex graph structures and dynamically adjust information extraction strategies, addressing the challenges of multi-level dependency modeling and adaptive reasoning depth. Additionally, the Self-Reflection with Multiple Perspectives (SR) module improves the accuracy and semantic consistency of reasoning results through self-reflection and backward reasoning mechanisms. Experiments demonstrate that Graph Counselor outperforms existing methods in multiple graph reasoning tasks, exhibiting higher reasoning accuracy and generalization ability. Our code is available at https://github.com/gjq100/Graph-Counselor.git.
MultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark
  for Financial LLM Evaluation
Authors: name: Xueqing Peng, name: Lingfei Qian, name: Yan Wang, name: Ruoyu Xiang, name: Yueru He, name: Yang Ren, name: Mingyang Jiang, name: Jeff Zhao, name: Huan He, name: Yi Han, name: Yun Feng, name: Yuechen Jiang, name: Yupeng Cao, name: Haohang Li, name: Yangyang Yu, name: Xiaoyu Wang, name: Penglei Gao, name: Shengyuan Lin, name: Keyi Wang, name: Shanshan Yang, name: Yilun Zhao, name: Zhiwei Liu, name: Peng Lu, name: Jerry Huang, name: Suyuchen Wang, name: Triantafillos Papadopoulos, name: Polydoros Giannouris, name: Efstathia Soufleri, name: Nuo Chen, name: Guojun Xiong, name: Zhiyang Deng, name: Yijia Zhao, name: Mingquan Lin, name: Meikang Qiu, name: Kaleb E Smith, name: Arman Cohan, name: Xiao-Yang Liu, name: Jimin Huang, name: Alejandro Lopez-Lira, name: Xi Chen, name: Junichi Tsujii, name: Jian-Yun Nie, name: Sophia Ananiadou, name: Qianqian Xie
Published: 2025-06-16 22:01:49

Recent advances in large language models (LLMs) have accelerated progress in
financial NLP and applications, yet existing benchmarks remain limited to
monolingual and unimodal settings, often over-relying on simple tasks and
failing to reflect the complexity of real-world financial communication. We
introduce MultiFinBen, the first multilingual and multimodal benchmark tailored
to the global financial domain, evaluating LLMs across modalities (text,
vision, audio) and linguistic settings (monolingual, bilingual, multilingual)
on domain-specific tasks. We introduce two novel tasks, including PolyFiQA-Easy
and PolyFiQA-Expert, the first multilingual financial benchmarks requiring
models to perform complex reasoning over mixed-language inputs; and EnglishOCR
and SpanishOCR, the first OCR-embedded financial QA tasks challenging models to
extract and reason over information from visual-text financial documents.
Moreover, we propose a dynamic, difficulty-aware selection mechanism and curate
a compact, balanced benchmark rather than simple aggregation existing datasets.
Extensive evaluation of 22 state-of-the-art models reveals that even the
strongest models, despite their general multimodal and multilingual
capabilities, struggle dramatically when faced with complex cross-lingual and
multimodal tasks in financial domain. MultiFinBen is publicly released to
foster transparent, reproducible, and inclusive progress in financial studies
and applications.