Orionnnn/hub-stats huggingface dataset:
Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems
Authors: name: Song Wang, name: Lingdong Kong, name: Xiaolu Liu, name: Hao Shi, name: Wentong Li, name: Jianke Zhu, name: Steven C. H. Hoi
Published: 2025-12-30 17:58:01

The rapid advancement of autonomous systems, including self-driving vehicles and drones, has intensified the need to forge true Spatial Intelligence from multi-modal onboard sensor data. While foundation models excel in single-modal contexts, integrating their capabilities across diverse sensors like cameras and LiDAR to create a unified understanding remains a formidable challenge. This paper presents a comprehensive framework for multi-modal pre-training, identifying the core set of techniques driving progress toward this goal. We dissect the interplay between foundational sensor characteristics and learning strategies, evaluating the role of platform-specific datasets in enabling these advancements. Our central contribution is the formulation of a unified taxonomy for pre-training paradigms: ranging from single-modality baselines to sophisticated unified frameworks that learn holistic representations for advanced tasks like 3D object detection and semantic occupancy prediction. Furthermore, we investigate the integration of textual inputs and occupancy representations to facilitate open-world perception and planning. Finally, we identify critical bottlenecks, such as computational efficiency and model scalability, and propose a roadmap toward general-purpose multi-modal foundation models capable of achieving robust Spatial Intelligence for real-world deployment.
JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
Authors: name: Kai Liu, name: Jungang Li, name: Yuchong Sun, name: Shengqiong Wu, name: Jianzhang Gao, name: Daoan Zhang, name: Wei Zhang, name: Sheng Jin, name: Sicheng Yu, name: Geng Zhan, name: Jiayi Ji, name: Fan Zhou, name: Liang Zheng, name: Shuicheng Yan, name: Hao Fei, name: Tat-Seng Chua
Published: 2025-12-28 12:25:43

This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for Joint Audio-Video (JAV) comprehension and generation. JavisGPT adopts a concise encoder-LLM-decoder architecture, featuring a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. To support this, we further construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that span diverse and multi-level comprehension and generation scenarios. Extensive experiments on JAV comprehension and generation benchmarks show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.
Factorized Learning for Temporally Grounded Video-Language Models
Authors: name: Wenzheng Zeng, name: Difei Gao, name: Mike Zheng Shou, name: Hwee Tou Ng
Published: 2025-12-30 09:13:20

Recent video-language models have shown great potential for video understanding, but still struggle with accurate temporal grounding for event-level perception. We observe that two main factors in video understanding (i.e., temporal grounding and textual response) form a logical hierarchy: accurate temporal evidence grounding lays the foundation for reliable textual response. However, existing works typically handle these two tasks in a coupled manner without a clear logical structure, leading to sub-optimal objectives. We address this from a factorized learning perspective. We first propose D^2VLM, a framework that decouples the learning of these two tasks while also emphasizing their inherent dependency. We adopt a "grounding then answering with evidence referencing" paradigm and introduce evidence tokens for evidence grounding, which emphasize event-level visual semantic capture beyond the focus on timestamp representation in existing works. To further facilitate the learning of these two tasks, we introduce a novel factorized preference optimization (FPO) algorithm. Unlike standard preference optimization, FPO explicitly incorporates probabilistic temporal grounding modeling into the optimization objective, enabling preference learning for both temporal grounding and textual response. We also construct a synthetic dataset to address the lack of suitable datasets for factorized preference learning with explicit temporal grounding. Experiments on various tasks demonstrate the clear advantage of our approach. Our source code is available at https://github.com/nusnlp/d2vlm.
Scaling Open-Ended Reasoning to Predict the Future
Authors: name: Nikhil Chandak, name: Shashwat Goel, name: Ameya Prabhu, name: Moritz Hardt, name: Jonas Geiping
Published: 2025-12-31 18:59:51

High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news, using a fully automated, careful curation recipe. We train the Qwen3 thinking models on our dataset, OpenForesight. To prevent leakage of future information during training and evaluation, we use an offline news corpus, both for data generation and retrieval in our forecasting system. Guided by a small validation set, we show the benefits of retrieval, and an improved reward function for reinforcement learning (RL). Once we obtain our final forecasting system, we perform held-out testing between May to August 2025. Our specialized model, OpenForecaster 8B, matches much larger proprietary models, with our training improving the accuracy, calibration, and consistency of predictions. We find calibration improvements from forecasting training generalize across popular benchmarks. We open-source all our models, code, and data to make research on language model forecasting broadly accessible.
PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation
Authors: name: Yuanhao Cai, name: Kunpeng Li, name: Menglin Jia, name: Jialiang Wang, name: Junzhe Sun, name: Feng Liang, name: Weifeng Chen, name: Felix Juefei-Xu, name: Chu Wang, name: Ali Thabet, name: Xiaoliang Dai, name: Xuan Ju, name: Alan Yuille, name: Ji Hou
Published: 2025-12-31 01:19:14

Recent advances in text-to-video (T2V) generation have achieved good visual quality, yet synthesizing videos that faithfully follow physical laws remains an open challenge. Existing methods mainly based on graphics or prompt extension struggle to generalize beyond simple simulated environments or learn implicit physical reasoning. The scarcity of training data with rich physics interactions and phenomena is also a problem. In this paper, we first introduce a Physics-Augmented video data construction Pipeline, PhyAugPipe, that leverages a vision-language model (VLM) with chain-of-thought reasoning to collect a large-scale training dataset, PhyVidGen-135K. Then we formulate a principled Physics-aware Groupwise Direct Preference Optimization, PhyGDPO, framework that builds upon the groupwise Plackett-Luce probabilistic model to capture holistic preferences beyond pairwise comparisons. In PhyGDPO, we design a Physics-Guided Rewarding (PGR) scheme that embeds VLM-based physics rewards to steer optimization toward physical consistency. We also propose a LoRA-Switch Reference (LoRA-SR) scheme that eliminates memory-heavy reference duplication for efficient training. Experiments show that our method significantly outperforms state-of-the-art open-source methods on PhyGenBench and VideoPhy2. Please check our project page at https://caiyuanhao1998.github.io/project/PhyGDPO for more video results. Our code, models, and data will be released at https://github.com/caiyuanhao1998/Open-PhyGDPO
Figure It Out: Improving the Frontier of Reasoning with Active Visual Thinking
Authors: name: Meiqi Chen, name: Fandong Meng, name: Jie Zhou
Published: 2025-12-30 15:39:11

Complex reasoning problems often involve implicit spatial, geometric, and structural relationships that are not explicitly encoded in text. While recent reasoning models have achieved strong performance across many domains, purely text-based reasoning struggles to represent global structural constraints in complex settings. In this paper, we introduce FIGR, which integrates active visual thinking into multi-turn reasoning via end-to-end reinforcement learning. FIGR externalizes intermediate structural hypotheses by constructing visual representations during problem solving. By adaptively regulating when and how visual reasoning should be invoked, FIGR enables more stable and coherent reasoning over global structural properties that are difficult to capture from text alone. Experiments on challenging mathematical reasoning benchmarks demonstrate that FIGR outperforms strong text-only chain-of-thought baselines. In particular, FIGR improves the base model by 13.12% on AIME 2025 and 11.00% on BeyondAIME, highlighting the effectiveness of figure-guided multimodal reasoning in enhancing the stability and reliability of complex reasoning.
GR-Dexter Technical Report
Authors: name: Ruoshi Wen, name: Guangzeng Chen, name: Zhongren Cui, name: Min Du, name: Yang Gou, name: Zhigang Han, name: Liqun Huang, name: Mingyu Lei, name: Yunfei Li, name: Zhuohang Li, name: Wenlei Liu, name: Yuxiao Liu, name: Xiao Ma, name: Hao Niu, name: Yutao Ouyang, name: Zeyu Ren, name: Haixin Shi, name: Wei Xu, name: Haoxiang Zhang, name: Jiajun Zhang, name: Xiao Zhang, name: Liwei Zheng, name: Weiheng Zhong, name: Yifei Zhou, name: Zhengming Zhu, name: Hang Li
Published: 2025-12-30 13:22:16

Vision-language-action (VLA) models have enabled language-conditioned, long-horizon robot manipulation, but most existing systems are limited to grippers. Scaling VLA policies to bimanual robots with high degree-of-freedom (DoF) dexterous hands remains challenging due to the expanded action space, frequent hand-object occlusions, and the cost of collecting real-robot data. We present GR-Dexter, a holistic hardware-model-data framework for VLA-based generalist manipulation on a bimanual dexterous-hand robot. Our approach combines the design of a compact 21-DoF robotic hand, an intuitive bimanual teleoperation system for real-robot data collection, and a training recipe that leverages teleoperated robot trajectories together with large-scale vision-language and carefully curated cross-embodiment datasets. Across real-world evaluations spanning long-horizon everyday manipulation and generalizable pick-and-place, GR-Dexter achieves strong in-domain performance and improved robustness to unseen objects and unseen instructions. We hope GR-Dexter serves as a practical step toward generalist dexterous-hand robotic manipulation.
Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
Authors: name: Zhenyu Zhang, name: Shujian Zhang, name: John Lambert, name: Wenxuan Zhou, name: Zhangyang Wang, name: Mingqing Chen, name: Andrew Hard, name: Rajiv Mathews, name: Lun Wang
Published: 2025-12-30 05:09:11

Despite the growing reasoning capabilities of recent large language models (LLMs), their internal mechanisms during the reasoning process remain underexplored. Prior approaches often rely on human-defined concepts (e.g., overthinking, reflection) at the word level to analyze reasoning in a supervised manner. However, such methods are limited, as it is infeasible to capture the full spectrum of potential reasoning behaviors, many of which are difficult to define in token space. In this work, we propose an unsupervised framework (namely, RISE: Reasoning behavior Interpretability via Sparse auto-Encoder) for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors. By segmenting chain-of-thought traces into sentence-level 'steps' and training sparse auto-encoders (SAEs) on step-level activations, we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking. Visualization and clustering analyses show that these behaviors occupy separable regions in the decoder column space. Moreover, targeted interventions on SAE-derived vectors can controllably amplify or suppress specific reasoning behaviors, altering inference trajectories without retraining. Beyond behavior-specific disentanglement, SAEs capture structural properties such as response length, revealing clusters of long versus short reasoning traces. More interestingly, SAEs enable the discovery of novel behaviors beyond human supervision. We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space. These findings underscore the potential of unsupervised latent discovery for both interpreting and controllably steering reasoning in LLMs.
SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time
Authors: name: Zhening Huang, name: Hyeonho Jeong, name: Xuelin Chen, name: Yulia Gryaditskaya, name: Tuanfeng Y. Wang, name: Joan Lasenby, name: Chun-Hao Huang
Published: 2025-12-31 18:59:57

We present SpaceTimePilot, a video diffusion model that disentangles space and time for controllable generative rendering. Given a monocular video, SpaceTimePilot can independently alter the camera viewpoint and the motion sequence within the generative process, re-rendering the scene for continuous and arbitrary exploration across space and time. To achieve this, we introduce an effective animation time-embedding mechanism in the diffusion process, allowing explicit control of the output video's motion sequence with respect to that of the source video. As no datasets provide paired videos of the same dynamic scene with continuous temporal variations, we propose a simple yet effective temporal-warping training scheme that repurposes existing multi-view datasets to mimic temporal differences. This strategy effectively supervises the model to learn temporal control and achieve robust space-time disentanglement. To further enhance the precision of dual control, we introduce two additional components: an improved camera-conditioning mechanism that allows altering the camera from the first frame, and CamxTime, the first synthetic space-and-time full-coverage rendering dataset that provides fully free space-time video trajectories within a scene. Joint training on the temporal-warping scheme and the CamxTime dataset yields more precise temporal control. We evaluate SpaceTimePilot on both real-world and synthetic data, demonstrating clear space-time disentanglement and strong results compared to prior work. Project page: https://zheninghuang.github.io/Space-Time-Pilot/ Code: https://github.com/ZheningHuang/spacetimepilot
Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem
Authors: name: Weixun Wang, name: XiaoXiao Xu, name: Wanhe An, name: Fangwen Dai, name: Wei Gao, name: Yancheng He, name: Ju Huang, name: Qiang Ji, name: Hanqi Jin, name: Xiaoyang Li, name: Yang Li, name: Zhongwen Li, name: Shirong Lin, name: Jiashun Liu, name: Zenan Liu, name: Tao Luo, name: Dilxat Muhtar, name: Yuanbin Qu, name: Jiaqiang Shi, name: Qinghui Sun, name: Yingshui Tan, name: Hao Tang, name: Runze Wang, name: Yi Wang, name: Zhaoguo Wang, name: Yanan Wu, name: Shaopan Xiong, name: Binchen Xu, name: Xander Xu, name: Yuchi Xu, name: Qipeng Zhang, name: Xixia Zhang, name: Haizhou Zhao, name: Jie Zhao, name: Shuaibing Zhao, name: Baihui Zheng, name: Jianhui Zheng, name: Suhang Zheng, name: Yanni Zhu, name: Mengze Cai, name: Kerui Cao, name: Xitong Chen, name: Yue Dai, name: Lifan Du, name: Tao Feng, name: Tao He, name: Jin Hu, name: Yijie Hu, name: Ziyu Jiang, name: Cheng Li, name: Xiang Li, name: Jing Liang, name: Chonghuan Liu, name: ZhenDong Liu, name: Haodong Mi, name: Yanhu Mo, name: Junjia Ni, name: Shixin Pei, name: Jingyu Shen, name: XiaoShuai Song, name: Cecilia Wang, name: Chaofan Wang, name: Kangyu Wang, name: Pei Wang, name: Tao Wang, name: Wei Wang, name: Ke Xiao, name: Mingyu Xu, name: Tiange Xu, name: Nan Ya, name: Siran Yang, name: Jianan Ye, name: Yaxing Zang, name: Duo Zhang, name: Junbo Zhang, name: Boren Zheng, name: Wanxi Deng, name: Ling Pan, name: Lin Qu, name: Wenbo Su, name: Jiamang Wang, name: Wei Wang, name: Hu Wei, name: Minggang Wu, name: Cheng Yu, name: Bing Zhao, name: Zhicheng Zheng, name: Bo Zheng
Published: 2025-12-31 14:03:39

Agentic crafting requires LLMs to operate in real-world environments over multiple turns by taking actions, observing outcomes, and iteratively refining artifacts. Despite its importance, the open-source community lacks a principled, end-to-end ecosystem to streamline agent development. We introduce the Agentic Learning Ecosystem (ALE), a foundational infrastructure that optimizes the production pipeline for agent LLMs. ALE consists of three components: ROLL, a post-training framework for weight optimization; ROCK, a sandbox environment manager for trajectory generation; and iFlow CLI, an agent framework for efficient context engineering. We release ROME (ROME is Obviously an Agentic Model), an open-source agent grounded by ALE and trained on over one million trajectories. Our approach includes data composition protocols for synthesizing complex behaviors and a novel policy optimization algorithm, Interaction-based Policy Alignment (IPA), which assigns credit over semantic interaction chunks rather than individual tokens to improve long-horizon training stability. Empirically, we evaluate ROME within a structured setting and introduce Terminal Bench Pro, a benchmark with improved scale and contamination control. ROME demonstrates strong performance across benchmarks like SWE-bench Verified and Terminal Bench, proving the effectiveness of the ALE infrastructure.
Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models
Authors: name: Junru Lu, name: Jiarui Qin, name: Lingfeng Qiao, name: Yinghui Li, name: Xinyi Dai, name: Bo Ke, name: Jianfeng He, name: Ruizhi Qiao, name: Di Yin, name: Xing Sun, name: Yunsheng Wu, name: Yinsong Liu, name: Shuangyin Liu, name: Mingkong Tang, name: Haodong Lin, name: Jiayi Kuang, name: Fanxu Meng, name: Xiaojuan Tang, name: Yunjia Xi, name: Junjie Huang, name: Haotong Yang, name: Zhenyi Shen, name: Yangning Li, name: Qianwen Zhang, name: Yifei Yu, name: Siyu An, name: Junnan Dong, name: Qiufeng Wang, name: Jie Wang, name: Keyu Chen, name: Wei Wen, name: Taian Guo, name: Zhifeng Shen, name: Daohai Yu, name: Jiahao Li, name: Ke Li, name: Zongyi Li, name: Xiaoyu Tan
Published: 2025-12-31 04:25:11

We introduce Youtu-LLM, a lightweight yet powerful language model that harmonizes high computational efficiency with native agentic intelligence. Unlike typical small models that rely on distillation, Youtu-LLM (1.96B) is pre-trained from scratch to systematically cultivate reasoning and planning capabilities. The key technical advancements are as follows: (1) Compact Architecture with Long-Context Support: Built on a dense Multi-Latent Attention (MLA) architecture with a novel STEM-oriented vocabulary, Youtu-LLM supports a 128k context window. This design enables robust long-context reasoning and state tracking within a minimal memory footprint, making it ideal for long-horizon agent and reasoning tasks. (2) Principled "Commonsense-STEM-Agent" Curriculum: We curated a massive corpus of approximately 11T tokens and implemented a multi-stage training strategy. By progressively shifting the pre-training data distribution from general commonsense to complex STEM and agentic tasks, we ensure the model acquires deep cognitive abilities rather than superficial alignment. (3) Scalable Agentic Mid-training: Specifically for the agentic mid-training, we employ diverse data construction schemes to synthesize rich and varied trajectories across math, coding, and tool-use domains. This high-quality data enables the model to internalize planning and reflection behaviors effectively. Extensive evaluations show that Youtu-LLM sets a new state-of-the-art for sub-2B LLMs. On general benchmarks, it achieves competitive performance against larger models, while on agent-specific tasks, it significantly surpasses existing SOTA baselines, demonstrating that lightweight models can possess strong intrinsic agentic capabilities.
Pretraining Frame Preservation in Autoregressive Video Memory Compression
Authors: name: Lvmin Zhang, name: Shengqu Cai, name: Muyang Li, name: Chong Zeng, name: Beijia Lu, name: Anyi Rao, name: Song Han, name: Gordon Wetzstein, name: Maneesh Agrawala
Published: 2025-12-29 20:29:21

We present PFP, a neural network structure to compress long videos into short contexts, with an explicit pretraining objective to preserve the high-frequency details of single frames at arbitrary temporal positions. The baseline model can compress a 20-second video into a context at about 5k length, where random frames can be retrieved with perceptually preserved appearances. Such pretrained models can be directly fine-tuned as memory encoders for autoregressive video models, enabling long history memory with low context cost and relatively low fidelity loss. We evaluate the framework with ablative settings and discuss the trade-offs of possible neural architecture designs.
CosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks
Authors: name: Yogeswar Reddy Thota
Published: 2025-12-21 18:26:18

Modern deep residual networks perform substantial redundant computation by evaluating all residual blocks for every input, even when identity mappings suffice. We introduce CosineGate, an end-to-end differentiable architecture for dynamic routing in residual networks that uses cosine incompatibility between identity and residual feature representations as a self-supervised skip signal. CosineGate measures semantic redundancy through the Cosine Incompatibility Ratio (CIR), defined as 1 - cos(x, F(x)), and uses Gumbel-Softmax relaxation to enable per-sample, per-block gating during training. A progressive FLOPs regularization term controls average compute usage without destabilizing optimization. On CIFAR-10, CosineGate spans the accuracy-efficiency Pareto frontier: an aggressive configuration achieves 89.9 percent accuracy with 24.1 percent FLOPs savings, a balanced configuration achieves 91.3 percent accuracy with 28.5 percent savings at epoch 160, and a conservative configuration reaches a peak of 93.2 percent accuracy with minimal compute reduction. These results match or exceed ResNet-20 (91.3 percent) while reducing computation, without auxiliary supervision, distillation, or task-specific heuristics. Our results demonstrate that simple geometric measures of feature incompatibility provide a principled and effective signal for dynamic residual routing.