Orionnnn/hub-stats huggingface dataset:
EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
Authors: name: Zhili Cheng, name: Yuge Tu, name: Ran Li, name: Shiqi Dai, name: Jinyi Hu, name: Shengding Hu, name: Jiahao Li, name: Yang Shi, name: Tianyu Yu, name: Weize Chen, name: Lei Shi, name: Maosong Sun
Published: 2025-01-21 03:22:10

Multimodal Large Language Models (MLLMs) have shown significant advancements,
providing a promising future for embodied agents. Existing benchmarks for
evaluating MLLMs primarily utilize static images or videos, limiting
assessments to non-interactive scenarios. Meanwhile, existing embodied AI
benchmarks are task-specific and not diverse enough, which do not adequately
evaluate the embodied capabilities of MLLMs. To address this, we propose
EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs
with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied
3D scenes, each of which is rigorously selected and annotated. It covers a
broad spectrum of existing embodied AI tasks with significantly enhanced
diversity, all within a unified simulation and evaluation framework tailored
for MLLMs. The tasks are organized into five categories: navigation, object
interaction, social interaction, attribute question answering, and spatial
question answering to assess different capabilities of the agents. We evaluated
the state-of-the-art MLLMs on EmbodiedEval and found that they have a
significant shortfall compared to human level on embodied tasks. Our analysis
demonstrates the limitations of existing MLLMs in embodied capabilities,
providing insights for their future development. We open-source all evaluation
data and simulation framework at https://github.com/thunlp/EmbodiedEval.
Control LLM: Controlled Evolution for Intelligence Retention in LLM
Authors: name: Haichao Wei, name: Yunxiang Ren, name: Zhoutong Fu, name: Aman Lunia, name: Yi-Lin Chen, name: Alice Leung, name: Ya Xu
Published: 2025-01-19 08:06:06

Large Language Models (LLMs) demand significant computational resources,
making it essential to enhance their capabilities without retraining from
scratch. A key challenge in this domain is catastrophic forgetting
(CF), which hampers performance during Continuous Pre-training (CPT) and
Continuous Supervised Fine-Tuning (CSFT). We propose Control LLM, a
novel approach that leverages parallel pre-trained and expanded transformer
blocks, aligning their hidden-states through interpolation strategies This
method effectively preserves performance on existing tasks while seamlessly
integrating new knowledge.
  Extensive experiments demonstrate the effectiveness of Control LLM in both
CPT and CSFT. On Llama3.1-8B-Instruct, it achieves significant improvements in
mathematical reasoning (+14.4% on Math-Hard) and coding performance (+10%
on MBPP-PLUS). On Llama3.1-8B, it enhances multilingual capabilities (+10.6%
on C-Eval, +6.8% on CMMLU, and +30.2% on CMMLU-0shot-CoT). It surpasses
existing methods and achieves SOTA among open-source models tuned from the same
base model, using substantially less data and compute. Crucially, these gains
are realized while preserving strong original capabilities, with minimal
degradation (<4.3% on MMLU) compared to >35% in open-source Math
and Coding models. This approach has been successfully deployed in LinkedIn's
GenAI-powered job seeker and Ads unit products.
  To support further research, we release the training and evaluation code
(https://github.com/linkedin/ControlLLM) along with models trained on
public datasets ( https://huggingface.co/ControlLLM) to the community.
Hallucinations Can Improve Large Language Models in Drug Discovery
Authors: name: Shuzhou Yuan, name: Michael Färber
Published: 2025-01-23 16:45:51

Concerns about hallucinations in Large Language Models (LLMs) have been
raised by researchers, yet their potential in areas where creativity is vital,
such as drug discovery, merits exploration. In this paper, we come up with the
hypothesis that hallucinations can improve LLMs in drug discovery. To verify
this hypothesis, we use LLMs to describe the SMILES string of molecules in
natural language and then incorporate these descriptions as part of the prompt
to address specific tasks in drug discovery. Evaluated on seven LLMs and five
classification tasks, our findings confirm the hypothesis: LLMs can achieve
better performance with text containing hallucinations. Notably, Llama-3.1-8B
achieves an 18.35% gain in ROC-AUC compared to the baseline without
hallucination. Furthermore, hallucinations generated by GPT-4o provide the most
consistent improvements across models. Additionally, we conduct empirical
analyses and a case study to investigate key factors affecting performance and
the underlying reasons. Our research sheds light on the potential use of
hallucinations for LLMs and offers new perspectives for future research
leveraging LLMs in drug discovery.
One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation
  Using a Single Prompt
Authors: name: Tao Liu, name: Kai Wang, name: Senmao Li, name: Joost van de Weijer, name: Fahad Shahbaz Khan, name: Shiqi Yang, name: Yaxing Wang, name: Jian Yang, name: Ming-Ming Cheng
Published: 2025-01-23 10:57:22

Text-to-image generation models can create high-quality images from input
prompts. However, they struggle to support the consistent generation of
identity-preserving requirements for storytelling. Existing approaches to this
problem typically require extensive training in large datasets or additional
modifications to the original model architectures. This limits their
applicability across different domains and diverse diffusion model
configurations. In this paper, we first observe the inherent capability of
language models, coined context consistency, to comprehend identity through
context with a single prompt. Drawing inspiration from the inherent context
consistency, we propose a novel training-free method for consistent
text-to-image (T2I) generation, termed "One-Prompt-One-Story" (1Prompt1Story).
Our approach 1Prompt1Story concatenates all prompts into a single input for T2I
diffusion models, initially preserving character identities. We then refine the
generation process using two novel techniques: Singular-Value Reweighting and
Identity-Preserving Cross-Attention, ensuring better alignment with the input
description for each frame. In our experiments, we compare our method against
various existing consistent T2I generation approaches to demonstrate its
effectiveness through quantitative metrics and qualitative assessments. Code is
available at https://github.com/byliutao/1Prompt1Story.
Improving Video Generation with Human Feedback
Authors: name: Jie Liu, name: Gongye Liu, name: Jiajun Liang, name: Ziyang Yuan, name: Xiaokun Liu, name: Mingwu Zheng, name: Xiele Wu, name: Qiulin Wang, name: Wenyu Qin, name: Menghan Xia, name: Xintao Wang, name: Xiaohong Liu, name: Fei Yang, name: Pengfei Wan, name: Di Zhang, name: Kun Gai, name: Yujiu Yang, name: Wanli Ouyang
Published: 2025-01-23 18:55:41

Video generation has achieved significant advances through rectified flow
techniques, but issues like unsmooth motion and misalignment between videos and
prompts persist. In this work, we develop a systematic pipeline that harnesses
human feedback to mitigate these problems and refine the video generation
model. Specifically, we begin by constructing a large-scale human preference
dataset focused on modern video generation models, incorporating pairwise
annotations across multi-dimensions. We then introduce VideoReward, a
multi-dimensional video reward model, and examine how annotations and various
design choices impact its rewarding efficacy. From a unified reinforcement
learning perspective aimed at maximizing reward with KL regularization, we
introduce three alignment algorithms for flow-based models by extending those
from diffusion models. These include two training-time strategies: direct
preference optimization for flow (Flow-DPO) and reward weighted regression for
flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
reward guidance directly to noisy videos. Experimental results indicate that
VideoReward significantly outperforms existing reward models, and Flow-DPO
demonstrates superior performance compared to both Flow-RWR and standard
supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
weights to multiple objectives during inference, meeting personalized video
quality needs. Project page: https://gongyeliu.github.io/videoalign.
Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline
  Professional Videos
Authors: name: Kairui Hu, name: Penghao Wu, name: Fanyi Pu, name: Wang Xiao, name: Yuanhan Zhang, name: Xiang Yue, name: Bo Li, name: Ziwei Liu
Published: 2025-01-23 16:51:47

Humans acquire knowledge through three cognitive stages: perceiving
information, comprehending knowledge, and adapting knowledge to solve novel
problems. Videos serve as an effective medium for this learning process,
facilitating a progression through these cognitive stages. However, existing
video benchmarks fail to systematically evaluate the knowledge acquisition
capabilities in Large Multimodal Models (LMMs). To address this gap, we
introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to
assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU
features a curated collection of 300 expert-level videos and 900
human-annotated questions across six disciplines, evaluating knowledge
acquisition through stage-aligned question-answer pairs: Perception,
Comprehension, and Adaptation. A proposed knowledge gain metric,
{\Delta}knowledge, quantifies improvement in performance after video viewing.
Evaluation of LMMs reveals a steep decline in performance as cognitive demands
increase and highlights a significant gap between human and model knowledge
acquisition, underscoring the need for methods to enhance LMMs' capability to
learn and adapt from videos.
GSTAR: Gaussian Surface Tracking and Reconstruction
Authors: name: Chengwei Zheng, name: Lixin Xue, name: Juan Zarate, name: Jie Song
Published: 2025-01-17 16:26:24

3D Gaussian Splatting techniques have enabled efficient photo-realistic
rendering of static scenes. Recent works have extended these approaches to
support surface reconstruction and tracking. However, tracking dynamic surfaces
with 3D Gaussians remains challenging due to complex topology changes, such as
surfaces appearing, disappearing, or splitting. To address these challenges, we
propose GSTAR, a novel method that achieves photo-realistic rendering, accurate
surface reconstruction, and reliable 3D tracking for general dynamic scenes
with changing topology. Given multi-view captures as input, GSTAR binds
Gaussians to mesh faces to represent dynamic objects. For surfaces with
consistent topology, GSTAR maintains the mesh topology and tracks the meshes
using Gaussians. In regions where topology changes, GSTAR adaptively unbinds
Gaussians from the mesh, enabling accurate registration and the generation of
new surfaces based on these optimized Gaussians. Additionally, we introduce a
surface-based scene flow method that provides robust initialization for
tracking between frames. Experiments demonstrate that our method effectively
tracks and reconstructs dynamic surfaces, enabling a range of applications. Our
project page with the code release is available at
https://eth-ait.github.io/GSTAR/.
DiffuEraser: A Diffusion Model for Video Inpainting
Authors: name: Xiaowen Li, name: Haolan Xue, name: Peiran Ren, name: Liefeng Bo
Published: 2025-01-17 08:03:02

Recent video inpainting algorithms integrate flow-based pixel propagation
with transformer-based generation to leverage optical flow for restoring
textures and objects using information from neighboring frames, while
completing masked regions through visual Transformers. However, these
approaches often encounter blurring and temporal inconsistencies when dealing
with large masks, highlighting the need for models with enhanced generative
capabilities. Recently, diffusion models have emerged as a prominent technique
in image and video generation due to their impressive performance. In this
paper, we introduce DiffuEraser, a video inpainting model based on stable
diffusion, designed to fill masked regions with greater details and more
coherent structures. We incorporate prior information to provide initialization
and weak conditioning,which helps mitigate noisy artifacts and suppress
hallucinations. Additionally, to improve temporal consistency during
long-sequence inference, we expand the temporal receptive fields of both the
prior model and DiffuEraser, and further enhance consistency by leveraging the
temporal smoothing property of Video Diffusion Models. Experimental results
demonstrate that our proposed method outperforms state-of-the-art techniques in
both content completeness and temporal consistency while maintaining acceptable
efficiency.
EchoVideo: Identity-Preserving Human Video Generation by Multimodal
  Feature Fusion
Authors: name: Jiangchuan Wei, name: Shiyue Yan, name: Wenfeng Lin, name: Boyuan Liu, name: Renjie Chen, name: Mingyu Guo
Published: 2025-01-23 08:06:11

Recent advancements in video generation have significantly impacted various
downstream applications, particularly in identity-preserving video generation
(IPT2V). However, existing methods struggle with "copy-paste" artifacts and low
similarity issues, primarily due to their reliance on low-level facial image
information. This dependence can result in rigid facial appearances and
artifacts reflecting irrelevant details. To address these challenges, we
propose EchoVideo, which employs two key strategies: (1) an Identity Image-Text
Fusion Module (IITF) that integrates high-level semantic features from text,
capturing clean facial identity representations while discarding occlusions,
poses, and lighting variations to avoid the introduction of artifacts; (2) a
two-stage training strategy, incorporating a stochastic method in the second
phase to randomly utilize shallow facial information. The objective is to
balance the enhancements in fidelity provided by shallow features while
mitigating excessive reliance on them. This strategy encourages the model to
utilize high-level features during training, ultimately fostering a more robust
representation of facial identities. EchoVideo effectively preserves facial
identities and maintains full-body integrity. Extensive experiments demonstrate
that it achieves excellent results in generating high-quality, controllability
and fidelity videos.
SRMT: Shared Memory for Multi-agent Lifelong Pathfinding
Authors: name: Alsu Sagirova, name: Yuri Kuratov, name: Mikhail Burtsev
Published: 2025-01-22 20:08:53

Multi-agent reinforcement learning (MARL) demonstrates significant progress
in solving cooperative and competitive multi-agent problems in various
environments. One of the principal challenges in MARL is the need for explicit
prediction of the agents' behavior to achieve cooperation. To resolve this
issue, we propose the Shared Recurrent Memory Transformer (SRMT) which extends
memory transformers to multi-agent settings by pooling and globally
broadcasting individual working memories, enabling agents to exchange
information implicitly and coordinate their actions. We evaluate SRMT on the
Partially Observable Multi-Agent Pathfinding problem in a toy Bottleneck
navigation task that requires agents to pass through a narrow corridor and on a
POGEMA benchmark set of tasks. In the Bottleneck task, SRMT consistently
outperforms a variety of reinforcement learning baselines, especially under
sparse rewards, and generalizes effectively to longer corridors than those seen
during training. On POGEMA maps, including Mazes, Random, and MovingAI, SRMT is
competitive with recent MARL, hybrid, and planning-based algorithms. These
results suggest that incorporating shared recurrent memory into the
transformer-based architectures can enhance coordination in decentralized
multi-agent systems. The source code for training and evaluation is available
on GitHub: https://github.com/Aloriosa/srmt.
Evolution and The Knightian Blindspot of Machine Learning
Authors: name: Joel Lehman, name: Elliot Meyerson, name: Tarek El-Gaaly, name: Kenneth O. Stanley, name: Tarin Ziyaee
Published: 2025-01-22 18:38:41

This paper claims that machine learning (ML) largely overlooks an important
facet of general intelligence: robustness to a qualitatively unknown future in
an open world. Such robustness relates to Knightian uncertainty (KU) in
economics, i.e. uncertainty that cannot be quantified, which is excluded from
consideration in ML's key formalisms. This paper aims to identify this blind
spot, argue its importance, and catalyze research into addressing it, which we
believe is necessary to create truly robust open-world AI. To help illuminate
the blind spot, we contrast one area of ML, reinforcement learning (RL), with
the process of biological evolution. Despite staggering ongoing progress, RL
still struggles in open-world situations, often failing under unforeseen
situations. For example, the idea of zero-shot transferring a self-driving car
policy trained only in the US to the UK currently seems exceedingly ambitious.
In dramatic contrast, biological evolution routinely produces agents that
thrive within an open world, sometimes even to situations that are remarkably
out-of-distribution (e.g. invasive species; or humans, who do undertake such
zero-shot international driving). Interestingly, evolution achieves such
robustness without explicit theory, formalisms, or mathematical gradients. We
explore the assumptions underlying RL's typical formalisms, showing how they
limit RL's engagement with the unknown unknowns characteristic of an
ever-changing complex world. Further, we identify mechanisms through which
evolutionary processes foster robustness to novel and unpredictable challenges,
and discuss potential pathways to algorithmically embody them. The conclusion
is that the intriguing remaining fragility of ML may result from blind spots in
its formalisms, and that significant gains may result from direct confrontation
with the challenge of KU.
Debate Helps Weak-to-Strong Generalization
Authors: name: Hao Lang, name: Fei Huang, name: Yongbin Li
Published: 2025-01-21 05:36:13

Common methods for aligning already-capable models with desired behavior rely
on the ability of humans to provide supervision. However, future superhuman
models will surpass the capability of humans. Therefore, humans will only be
able to weakly supervise superhuman models. This expected deficiency of human
evaluation would weaken the safety of future AI systems. Scalable oversight and
weak-to-strong generalization are two complementary approaches to tackle this
issue. In this paper, we attempt to combine the strengths of these two
approaches to further improve alignment. Specifically, we investigate ways of
improving human supervision with a strong pretrained model and then supervise
the strong model with enhanced weak human supervision. To make iterative
empirical progress, we consider an analogy: can we use a strong model to
improve weak model supervision and then use it to supervise the strong model?
We empirically test it by finetuning a small weak model on ground truth labels
with the additional help from a large strong model, and then finetuning the
strong model on labels generated by the weak model. We find that debate can
assist a weak model in extracting trustworthy information from an untrustworthy
strong model, which provides leverage as context on samples when training a
weak model. We also show that an ensemble of weak models helps exploit long
arguments generated by strong model debaters and obtain a more robust
supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP
benchmarks show that the combination approach leads to better alignment, which
indicates that debate has the potential to help weak-to-strong generalization.
Temporal Preference Optimization for Long-Form Video Understanding
Authors: name: Rui Li, name: Xiaohan Wang, name: Yuhui Zhang, name: Zeyu Wang, name: Serena Yeung-Levy
Published: 2025-01-23 18:58:03

Despite significant advancements in video large multimodal models
(video-LMMs), achieving effective temporal grounding in long-form videos
remains a challenge for existing models. To address this limitation, we propose
Temporal Preference Optimization (TPO), a novel post-training framework
designed to enhance the temporal grounding capabilities of video-LMMs through
preference learning. TPO adopts a self-training approach that enables models to
differentiate between well-grounded and less accurate temporal responses by
leveraging curated preference datasets at two granularities: localized temporal
grounding, which focuses on specific video segments, and comprehensive temporal
grounding, which captures extended temporal dependencies across entire video
sequences. By optimizing on these preference datasets, TPO significantly
enhances temporal understanding while reducing reliance on manually annotated
data. Extensive experiments on three long-form video understanding
benchmarks--LongVideoBench, MLVU, and Video-MME--demonstrate the effectiveness
of TPO across two state-of-the-art video-LMMs. Notably, LLaVA-Video-TPO
establishes itself as the leading 7B model on the Video-MME benchmark,
underscoring the potential of TPO as a scalable and efficient solution for
advancing temporal reasoning in long-form video understanding. Project page:
https://ruili33.github.io/tpo_website.
IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art
  Text-to-Image Models
Authors: name: Jiayi Lei, name: Renrui Zhang, name: Xiangfei Hu, name: Weifeng Lin, name: Zhen Li, name: Wenjian Sun, name: Ruoyi Du, name: Le Zhuo, name: Zhongyu Li, name: Xinyue Li, name: Shitian Zhao, name: Ziyu Guo, name: Yiting Lu, name: Peng Gao, name: Hongsheng Li
Published: 2025-01-23 18:58:33

With the rapid development of diffusion models, text-to-image(T2I) models
have made significant progress, showcasing impressive abilities in prompt
following and image generation. Recently launched models such as FLUX.1 and
Ideogram2.0, along with others like Dall-E3 and Stable Diffusion 3, have
demonstrated exceptional performance across various complex tasks, raising
questions about whether T2I models are moving towards general-purpose
applicability. Beyond traditional image generation, these models exhibit
capabilities across a range of fields, including controllable generation, image
editing, video, audio, 3D, and motion generation, as well as computer vision
tasks like semantic segmentation and depth estimation. However, current
evaluation frameworks are insufficient to comprehensively assess these models'
performance across expanding domains. To thoroughly evaluate these models, we
developed the IMAGINE-E and tested six prominent models: FLUX.1, Ideogram2.0,
Midjourney, Dall-E3, Stable Diffusion 3, and Jimeng. Our evaluation is divided
into five key domains: structured output generation, realism, and physical
consistency, specific domain generation, challenging scenario generation, and
multi-style creation tasks. This comprehensive assessment highlights each
model's strengths and limitations, particularly the outstanding performance of
FLUX.1 and Ideogram2.0 in structured and specific domain tasks, underscoring
the expanding applications and potential of T2I models as foundational AI
tools. This study provides valuable insights into the current state and future
trajectory of T2I models as they evolve towards general-purpose usability.
Evaluation scripts will be released at https://github.com/jylei16/Imagine-e.