Orionnnn/hub-stats huggingface dataset:
Learning from Massive Human Videos for Universal Humanoid Pose Control
Authors: name: Jiageng Mao, name: Siheng Zhao, name: Siqi Song, name: Tianheng Shi, name: Junjie Ye, name: Mingtong Zhang, name: Haoran Geng, name: Jitendra Malik, name: Vitor Guizilini, name: Yue Wang
Published: 2024-12-18 18:59:56

Scalable learning of humanoid robots is crucial for their deployment in
real-world applications. While traditional approaches primarily rely on
reinforcement learning or teleoperation to achieve whole-body control, they are
often limited by the diversity of simulated environments and the high costs of
demonstration collection. In contrast, human videos are ubiquitous and present
an untapped source of semantic and motion information that could significantly
enhance the generalization capabilities of humanoid robots. This paper
introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot
poses with corresponding text-based motion descriptions, designed to leverage
this abundant data. Humanoid-X is curated through a comprehensive pipeline:
data mining from the Internet, video caption generation, motion retargeting of
humans to humanoid robots, and policy learning for real-world deployment. With
Humanoid-X, we further train a large humanoid model, UH-1, which takes text
instructions as input and outputs corresponding actions to control a humanoid
robot. Extensive simulated and real-world experiments validate that our
scalable training approach leads to superior generalization in text-based
humanoid control, marking a significant step toward adaptable, real-world-ready
humanoid robots.
Efficient Diffusion Transformer Policies with Mixture of Expert
  Denoisers for Multitask Learning
Authors: name: Moritz Reuss, name: Jyothish Pari, name: Pulkit Agrawal, name: Rudolf Lioutikov
Published: 2024-12-17 14:34:51

Diffusion Policies have become widely used in Imitation Learning, offering
several appealing properties, such as generating multimodal and discontinuous
behavior. As models are becoming larger to capture more complex capabilities,
their computational demands increase, as shown by recent scaling laws.
Therefore, continuing with the current architectures will present a
computational roadblock. To address this gap, we propose Mixture-of-Denoising
Experts (MoDE) as a novel policy for Imitation Learning. MoDE surpasses current
state-of-the-art Transformer-based Diffusion Policies while enabling
parameter-efficient scaling through sparse experts and noise-conditioned
routing, reducing both active parameters by 40% and inference costs by 90% via
expert caching. Our architecture combines this efficient scaling with
noise-conditioned self-attention mechanism, enabling more effective denoising
across different noise levels. MoDE achieves state-of-the-art performance on
134 tasks in four established imitation learning benchmarks (CALVIN and
LIBERO). Notably, by pretraining MoDE on diverse robotics data, we achieve 4.01
on CALVIN ABC and 0.95 on LIBERO-90. It surpasses both CNN-based and
Transformer Diffusion Policies by an average of 57% across 4 benchmarks, while
using 90% fewer FLOPs and fewer active parameters compared to default Diffusion
Transformer architectures. Furthermore, we conduct comprehensive ablations on
MoDE's components, providing insights for designing efficient and scalable
Transformer architectures for Diffusion Policies. Code and demonstrations are
available at https://mbreuss.github.io/MoDE_Diffusion_Policy/.
AniDoc: Animation Creation Made Easier
Authors: name: Yihao Meng, name: Hao Ouyang, name: Hanlin Wang, name: Qiuyu Wang, name: Wen Wang, name: Ka Leong Cheng, name: Zhiheng Liu, name: Yujun Shen, name: Huamin Qu
Published: 2024-12-18 18:59:59

The production of 2D animation follows an industry-standard workflow,
encompassing four essential stages: character design, keyframe animation,
in-betweening, and coloring. Our research focuses on reducing the labor costs
in the above process by harnessing the potential of increasingly powerful
generative AI. Using video diffusion models as the foundation, AniDoc emerges
as a video line art colorization tool, which automatically converts sketch
sequences into colored animations following the reference character
specification. Our model exploits correspondence matching as an explicit
guidance, yielding strong robustness to the variations (e.g., posture) between
the reference character and each line art frame. In addition, our model could
even automate the in-betweening process, such that users can easily create a
temporally consistent animation by simply providing a character image as well
as the start and end sketches. Our code is available at:
https://yihao-meng.github.io/AniDoc_demo.
FashionComposer: Compositional Fashion Image Generation
Authors: name: Sihui Ji, name: Yiyang Wang, name: Xi Chen, name: Xiaogang Xu, name: Hao Luo, name: Hengshuang Zhao
Published: 2024-12-18 18:59:50

We present FashionComposer for compositional fashion image generation. Unlike
previous methods, FashionComposer is highly flexible. It takes multi-modal
input (i.e., text prompt, parametric human model, garment image, and face
image) and supports personalizing the appearance, pose, and figure of the human
and assigning multiple garments in one pass. To achieve this, we first develop
a universal framework capable of handling diverse input modalities. We
construct scaled training data to enhance the model's robust compositional
capabilities. To accommodate multiple reference images (garments and faces)
seamlessly, we organize these references in a single image as an "asset
library" and employ a reference UNet to extract appearance features. To inject
the appearance features into the correct pixels in the generated result, we
propose subject-binding attention. It binds the appearance features from
different "assets" with the corresponding text features. In this way, the model
could understand each asset according to their semantics, supporting arbitrary
numbers and types of reference images. As a comprehensive solution,
FashionComposer also supports many other applications like human album
generation, diverse virtual try-on tasks, etc.
Prompting Depth Anything for 4K Resolution Accurate Metric Depth
  Estimation
Authors: name: Haotong Lin, name: Sida Peng, name: Jingxiao Chen, name: Songyou Peng, name: Jiaming Sun, name: Minghuan Liu, name: Hujun Bao, name: Jiashi Feng, name: Xiaowei Zhou, name: Bingyi Kang
Published: 2024-12-18 16:32:12

Prompts play a critical role in unleashing the power of language and vision
foundation models for specific tasks. For the first time, we introduce
prompting into depth foundation models, creating a new paradigm for metric
depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost
LiDAR as the prompt to guide the Depth Anything model for accurate metric depth
output, achieving up to 4K resolution. Our approach centers on a concise prompt
fusion design that integrates the LiDAR at multiple scales within the depth
decoder. To address training challenges posed by limited datasets containing
both LiDAR depth and precise GT depth, we propose a scalable data pipeline that
includes synthetic data LiDAR simulation and real data pseudo GT depth
generation. Our approach sets new state-of-the-arts on the ARKitScenes and
ScanNet++ datasets and benefits downstream applications, including 3D
reconstruction and generalized robotic grasping.
ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting
  with Diffusion Transformers
Authors: name: Lianghua Huang, name: Wei Wang, name: Zhi-Fan Wu, name: Yupeng Shi, name: Chen Liang, name: Tong Shen, name: Han Zhang, name: Huanzhang Dou, name: Yu Liu, name: Jingren Zhou
Published: 2024-12-17 06:03:05

Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the
inherent in-context generation capabilities of pretrained diffusion
transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks
with minimal or no architectural modifications. These capabilities are unlocked
by concatenating self-attention tokens across multiple input and target images,
combined with grouped and masked generation pipelines. Building upon this
foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive
visual generation framework that leverages pretrained diffusion transformers in
their original form, requiring no additional tuning, adapters, or
modifications. Users can interact with ChatDiT to create interleaved text-image
articles, multi-page picture books, edit images, design IP derivatives, or
develop character design settings, all through free-form natural language
across one or more conversational rounds. At its core, ChatDiT employs a
multi-agent system comprising three key components: an Instruction-Parsing
agent that interprets user-uploaded images and instructions, a
Strategy-Planning agent that devises single-step or multi-step generation
actions, and an Execution agent that performs these actions using an in-context
toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench
arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with
diverse instructions and varying numbers of input and target images. Despite
its simplicity and training-free approach, ChatDiT surpasses all competitors,
including those specifically designed and trained on extensive multi-task
datasets. We further identify key limitations of pretrained DiTs in zero-shot
adapting to tasks. We release all code, agents, results, and intermediate
outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented
  Generation for Preference Alignment
Authors: name: Zhuoran Jin, name: Hongbang Yuan, name: Tianyi Men, name: Pengfei Cao, name: Yubo Chen, name: Kang Liu, name: Jun Zhao
Published: 2024-12-18 11:28:05

Despite the significant progress made by existing retrieval augmented
language models (RALMs) in providing trustworthy responses and grounding in
reliable sources, they often overlook effective alignment with human
preferences. In the alignment process, reward models (RMs) act as a crucial
proxy for human values to guide optimization. However, it remains unclear how
to evaluate and select a reliable RM for preference alignment in RALMs. To this
end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG
settings. First, we design four crucial and challenging RAG-specific scenarios
to assess RMs, including multi-hop reasoning, fine-grained citation,
appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG
subsets, six retrievers, and 24 RALMs to increase the diversity of data
sources. Finally, we adopt an LLM-as-a-judge approach to improve preference
annotation efficiency and effectiveness, exhibiting a strong correlation with
human annotations. Based on the RAG-RewardBench, we conduct a comprehensive
evaluation of 45 RMs and uncover their limitations in RAG scenarios.
Additionally, we also reveal that existing trained RALMs show almost no
improvement in preference alignment, highlighting the need for a shift towards
preference-aligned training.We release our benchmark and code publicly at
https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.
GUI Agents: A Survey
Authors: name: Dang Nguyen, name: Jian Chen, name: Yu Wang, name: Gang Wu, name: Namyong Park, name: Zhengmian Hu, name: Hanjia Lyu, name: Junda Wu, name: Ryan Aponte, name: Yu Xia, name: Xintong Li, name: Jing Shi, name: Hongjie Chen, name: Viet Dac Lai, name: Zhouhang Xie, name: Sungchul Kim, name: Ruiyi Zhang, name: Tong Yu, name: Mehrab Tanjim, name: Nesreen K. Ahmed, name: Puneet Mathur, name: Seunghyun Yoon, name: Lina Yao, name: Branislav Kveton, name: Thien Huu Nguyen, name: Trung Bui, name: Tianyi Zhou, name: Ryan A. Rossi, name: Franck Dernoncourt
Published: 2024-12-18 04:48:28

Graphical User Interface (GUI) agents, powered by Large Foundation Models,
have emerged as a transformative approach to automating human-computer
interaction. These agents autonomously interact with digital systems or
software applications via GUIs, emulating human actions such as clicking,
typing, and navigating visual elements across diverse platforms. Motivated by
the growing interest and fundamental importance of GUI agents, we provide a
comprehensive survey that categorizes their benchmarks, evaluation metrics,
architectures, and training methods. We propose a unified framework that
delineates their perception, reasoning, planning, and acting capabilities.
Furthermore, we identify important open challenges and discuss key future
directions. Finally, this work serves as a basis for practitioners and
researchers to gain an intuitive understanding of current progress, techniques,
benchmarks, and critical open problems that remain to be addressed.
VidTok: A Versatile and Open-Source Video Tokenizer
Authors: name: Anni Tang, name: Tianyu He, name: Junliang Guo, name: Xinle Cheng, name: Li Song, name: Jiang Bian
Published: 2024-12-17 16:27:11

Encoding video content into compact latent tokens has become a fundamental
step in video generation and understanding, driven by the need to address the
inherent redundancy in pixel-level representations. Consequently, there is a
growing demand for high-performance, open-source video tokenizers as
video-centric research gains prominence. We introduce VidTok, a versatile video
tokenizer that delivers state-of-the-art performance in both continuous and
discrete tokenizations. VidTok incorporates several key advancements over
existing approaches: 1) model architecture such as convolutional layers and
up/downsampling modules; 2) to address the training instability and codebook
collapse commonly associated with conventional Vector Quantization (VQ), we
integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3)
improved training strategies, including a two-stage training process and the
use of reduced frame rates. By integrating these advancements, VidTok achieves
substantial improvements over existing methods, demonstrating superior
performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD,
under standardized evaluation settings.
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World
  Tasks
Authors: name: Frank F. Xu, name: Yufan Song, name: Boxuan Li, name: Yuxuan Tang, name: Kritanjali Jain, name: Mengxue Bao, name: Zora Z. Wang, name: Xuhui Zhou, name: Zhitong Guo, name: Murong Cao, name: Mingyang Yang, name: Hao Yang Lu, name: Amaad Martin, name: Zhe Su, name: Leander Maben, name: Raj Mehta, name: Wayne Chi, name: Lawrence Jang, name: Yiqing Xie, name: Shuyan Zhou, name: Graham Neubig
Published: 2024-12-18 18:55:40

We interact with computers on an everyday basis, be it in everyday life or
work, and many aspects of work can be done entirely with access to a computer
and the Internet. At the same time, thanks to improvements in large language
models (LLMs), there has also been a rapid development in AI agents that
interact with and affect change in their surrounding environments. But how
performant are AI agents at helping to accelerate or even autonomously perform
work-related tasks? The answer to this question has important implications for
both industry looking to adopt AI into their workflows, and for economic policy
to understand the effects that adoption of AI may have on the labor market. To
measure the progress of these LLM agents' performance on performing real-world
professional tasks, in this paper, we introduce TheAgentCompany, an extensible
benchmark for evaluating AI agents that interact with the world in similar ways
to those of a digital worker: by browsing the Web, writing code, running
programs, and communicating with other coworkers. We build a self-contained
environment with internal web sites and data that mimics a small software
company environment, and create a variety of tasks that may be performed by
workers in such a company. We test baseline agents powered by both closed
API-based and open-weights language models (LMs), and find that with the most
competitive agent, 24% of the tasks can be completed autonomously. This paints
a nuanced picture on task automation with LM agents -- in a setting simulating
a real workplace, a good portion of simpler tasks could be solved autonomously,
but more difficult long-horizon tasks are still beyond the reach of current
systems.
Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion
Authors: name: Massimiliano Viola, name: Kevin Qu, name: Nando Metzger, name: Bingxin Ke, name: Alexander Becker, name: Konrad Schindler, name: Anton Obukhov
Published: 2024-12-18 00:06:41

Depth completion upgrades sparse depth measurements into dense depth maps
guided by a conventional image. Existing methods for this highly ill-posed task
operate in tightly constrained settings and tend to struggle when applied to
images outside the training domain or when the available depth measurements are
sparse, irregularly distributed, or of varying density. Inspired by recent
advances in monocular depth estimation, we reframe depth completion as an
image-conditional depth map generation guided by sparse measurements. Our
method, Marigold-DC, builds on a pretrained latent diffusion model for
monocular depth estimation and injects the depth observations as test-time
guidance via an optimization scheme that runs in tandem with the iterative
inference of denoising diffusion. The method exhibits excellent zero-shot
generalization across a diverse range of environments and handles even
extremely sparse guidance effectively. Our results suggest that contemporary
monocular depth priors greatly robustify depth completion: it may be better to
view the task as recovering dense depth from (dense) image pixels, guided by
sparse depth; rather than as inpainting (sparse) depth, guided by an image.
Project website: https://MarigoldDepthCompletion.github.io/
SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
Authors: name: Yufan Zhou, name: Ruiyi Zhang, name: Jiuxiang Gu, name: Nanxuan Zhao, name: Jing Shi, name: Tong Sun
Published: 2024-12-13 20:01:51

We present SUGAR, a zero-shot method for subject-driven video customization.
Given an input image, SUGAR is capable of generating videos for the subject
contained in the image and aligning the generation with arbitrary visual
attributes such as style and motion specified by user-input text. Unlike
previous methods, which require test-time fine-tuning or fail to generate
text-aligned videos, SUGAR achieves superior results without the need for extra
cost at test-time. To enable zero-shot capability, we introduce a scalable
pipeline to construct synthetic dataset which is specifically designed for
subject-driven customization, leading to 2.5 millions of image-video-text
triplets. Additionally, we propose several methods to enhance our model,
including special attention designs, improved training strategies, and a
refined sampling algorithm. Extensive experiments are conducted. Compared to
previous methods, SUGAR achieves state-of-the-art results in identity
preservation, video dynamics, and video-text alignment for subject-driven video
customization, demonstrating the effectiveness of our proposed method.
Compressed Chain of Thought: Efficient Reasoning Through Dense
  Representations
Authors: name: Jeffrey Cheng, name: Benjamin Van Durme
Published: 2024-12-17 18:50:33

Chain-of-thought (CoT) decoding enables language models to improve reasoning
performance at the cost of high generation latency in decoding. Recent
proposals have explored variants of contemplation tokens, a term we introduce
that refers to special tokens used during inference to allow for extra
computation. Prior work has considered fixed-length sequences drawn from a
discrete set of embeddings as contemplation tokens. Here we propose Compressed
Chain-of-Thought (CCoT), a framework to generate contentful and continuous
contemplation tokens of variable sequence length. The generated contemplation
tokens are compressed representations of explicit reasoning chains, and our
method can be applied to off-the-shelf decoder language models. Through
experiments, we illustrate how CCoT enables additional reasoning over dense
contentful representations to achieve corresponding improvements in accuracy.
Moreover, the reasoning improvements can be adaptively modified on demand by
controlling the number of contemplation tokens generated.
Seeker: Towards Exception Safety Code Generation with Intermediate
  Language Agents Framework
Authors: name: Xuanming Zhang, name: Yuxuan Chen, name: Yiming Zheng, name: Zhexin Zhang, name: Yuan Yuan, name: Minlie Huang
Published: 2024-12-16 12:35:29

In real world software development, improper or missing exception handling
can severely impact the robustness and reliability of code. Exception handling
mechanisms require developers to detect, capture, and manage exceptions
according to high standards, but many developers struggle with these tasks,
leading to fragile code. This problem is particularly evident in open-source
projects and impacts the overall quality of the software ecosystem. To address
this challenge, we explore the use of large language models (LLMs) to improve
exception handling in code. Through extensive analysis, we identify three key
issues: Insensitive Detection of Fragile Code, Inaccurate Capture of Exception
Block, and Distorted Handling Solution. These problems are widespread across
real world repositories, suggesting that robust exception handling practices
are often overlooked or mishandled. In response, we propose Seeker, a
multi-agent framework inspired by expert developer strategies for exception
handling. Seeker uses agents: Scanner, Detector, Predator, Ranker, and Handler
to assist LLMs in detecting, capturing, and resolving exceptions more
effectively. Our work is the first systematic study on leveraging LLMs to
enhance exception handling practices in real development scenarios, providing
valuable insights for future improvements in code reliability.
Feather the Throttle: Revisiting Visual Token Pruning for
  Vision-Language Model Acceleration
Authors: name: Mark Endo, name: Xiaohan Wang, name: Serena Yeung-Levy
Published: 2024-12-17 18:56:50

Recent works on accelerating Vision-Language Models show that strong
performance can be maintained across a variety of vision-language tasks despite
highly compressing visual information. In this work, we examine the popular
acceleration approach of early pruning of visual tokens inside the language
model and find that its strong performance across many tasks is not due to an
exceptional ability to compress visual information, but rather the benchmarks'
limited ability to assess fine-grained visual capabilities. Namely, we
demonstrate a core issue with the acceleration approach where most tokens
towards the top of the image are pruned away. Yet, this issue is only reflected
in performance for a small subset of tasks such as localization. For the other
evaluated tasks, strong performance is maintained with the flawed pruning
strategy. Noting the limited visual capabilities of the studied acceleration
technique, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble
cRiteria), a straightforward approach that (1) resolves the identified issue
with early-layer pruning, (2) incorporates uniform sampling to ensure coverage
across all image regions, and (3) applies pruning in two stages to allow the
criteria to become more effective at a later layer while still achieving
significant speedup through early-layer pruning. With comparable computational
savings, we find that FEATHER has more than 5times performance improvement
on the vision-centric localization benchmarks compared to the original
acceleration approach.