Orionnnn/hub-stats huggingface dataset:
HumanVid: Demystifying Training Data for Camera-controllable Human Image
  Animation
Authors: name: Zhenzhi Wang, name: Yixuan Li, name: Yanhong Zeng, name: Youqing Fang, name: Yuwei Guo, name: Wenran Liu, name: Jing Tan, name: Kai Chen, name: Tianfan Xue, name: Bo Dai, name: Dahua Lin
Published: 2024-07-24 17:15:58

Human image animation involves generating videos from a character photo,
allowing user control and unlocking potential for video and movie production.
While recent approaches yield impressive results using high-quality training
data, the inaccessibility of these datasets hampers fair and transparent
benchmarking. Moreover, these approaches prioritize 2D human motion and
overlook the significance of camera motions in videos, leading to limited
control and unstable video generation.To demystify the training data, we
present HumanVid, the first large-scale high-quality dataset tailored for human
image animation, which combines crafted real-world and synthetic data. For the
real-world data, we compile a vast collection of copyright-free real-world
videos from the internet. Through a carefully designed rule-based filtering
strategy, we ensure the inclusion of high-quality videos, resulting in a
collection of 20K human-centric videos in 1080P resolution. Human and camera
motion annotation is accomplished using a 2D pose estimator and a SLAM-based
method. For the synthetic data, we gather 2,300 copyright-free 3D avatar assets
to augment existing available 3D assets. Notably, we introduce a rule-based
camera trajectory generation method, enabling the synthetic pipeline to
incorporate diverse and precise camera motion annotation, which can rarely be
found in real-world data. To verify the effectiveness of HumanVid, we establish
a baseline model named CamAnimate, short for Camera-controllable Human
Animation, that considers both human and camera motions as conditions. Through
extensive experimentation, we demonstrate that such simple baseline training on
our HumanVid achieves state-of-the-art performance in controlling both human
pose and camera motions, setting a new benchmark. Code and data will be
publicly available at https://github.com/zhenzhiwang/HumanVid/.
PERSONA: A Reproducible Testbed for Pluralistic Alignment
Authors: name: Louis Castricato, name: Nathan Lile, name: Rafael Rafailov, name: Jan-Philipp Fränken, name: Chelsea Finn
Published: 2024-07-24 16:11:39

The rapid advancement of language models (LMs) necessitates robust alignment
with diverse user values. However, current preference optimization approaches
often fail to capture the plurality of user opinions, instead reinforcing
majority viewpoints and marginalizing minority perspectives. We introduce
PERSONA, a reproducible test bed designed to evaluate and improve pluralistic
alignment of LMs. We procedurally generate diverse user profiles from US census
data, resulting in 1,586 synthetic personas with varied demographic and
idiosyncratic attributes. We then generate a large-scale evaluation dataset
containing 3,868 prompts and 317,200 feedback pairs obtained from our synthetic
personas. Leveraging this dataset, we systematically evaluate LM capabilities
in role-playing diverse users, verified through human judges, and the
establishment of both a benchmark, PERSONA Bench, for pluralistic alignment
approaches as well as an extensive dataset to create new and future benchmarks.
The full dataset and benchmarks are available here:
https://www.synthlabs.ai/research/persona.
VILA^2: VILA Augmented VILA
Authors: name: Yunhao Fang, name: Ligeng Zhu, name: Yao Lu, name: Yan Wang, name: Pavlo Molchanov, name: Jang Hyun Cho, name: Marco Pavone, name: Song Han, name: Hongxu Yin
Published: 2024-07-24 17:37:05

Visual language models (VLMs) have rapidly progressed, driven by the success
of large language models (LLMs). While model architectures and training
infrastructures advance rapidly, data curation remains under-explored. When
data quantity and quality become a bottleneck, existing work either directly
crawls more raw data from the Internet that does not have a guarantee of data
quality or distills from black-box commercial models (e.g., GPT-4V / Gemini)
causing the performance upper bounded by that model. In this work, we introduce
a novel approach that includes a self-augment step and a specialist-augment
step to iteratively improve data quality and model performance. In the
self-augment step, a VLM recaptions its own pretraining data to enhance data
quality, and then retrains from scratch using this refined dataset to improve
model performance. This process can iterate for several rounds. Once
self-augmentation saturates, we employ several specialist VLMs finetuned from
the self-augmented VLM with domain-specific expertise, to further infuse
specialist knowledge into the generalist VLM through task-oriented recaptioning
and retraining. With the combined self-augmented and specialist-augmented
training, we introduce VILA^2 (VILA-augmented-VILA), a VLM family that
consistently improves the accuracy on a wide range of tasks over prior art, and
achieves new state-of-the-art results on MMMU leaderboard among open-sourced
models.
DDK: Distilling Domain Knowledge for Efficient Large Language Models
Authors: name: Jiaheng Liu, name: Chenchen Zhang, name: Jinyang Guo, name: Yuanxing Zhang, name: Haoran Que, name: Ken Deng, name: Zhiqi Bai, name: Jie Liu, name: Ge Zhang, name: Jiakai Wang, name: Yanan Wu, name: Congnan Liu, name: Wenbo Su, name: Jiamang Wang, name: Lin Qu, name: Bo Zheng
Published: 2024-07-23 03:47:28

Despite the advanced intelligence abilities of large language models (LLMs)
in various applications, they still face significant computational and storage
demands. Knowledge Distillation (KD) has emerged as an effective strategy to
improve the performance of a smaller LLM (i.e., the student model) by
transferring knowledge from a high-performing LLM (i.e., the teacher model).
Prevailing techniques in LLM distillation typically use a black-box model API
to generate high-quality pretrained and aligned datasets, or utilize white-box
distillation by altering the loss function to better transfer knowledge from
the teacher LLM. However, these methods ignore the knowledge differences
between the student and teacher LLMs across domains. This results in excessive
focus on domains with minimal performance gaps and insufficient attention to
domains with large gaps, reducing overall performance. In this paper, we
introduce a new LLM distillation framework called DDK, which dynamically
adjusts the composition of the distillation dataset in a smooth manner
according to the domain performance differences between the teacher and student
models, making the distillation process more stable and effective. Extensive
evaluations show that DDK significantly improves the performance of student
models, outperforming both continuously pretrained baselines and existing
knowledge distillation methods by a large margin.
ViPer: Visual Personalization of Generative Models via Individual
  Preference Learning
Authors: name: Sogand Salehi, name: Mahdi Shafiei, name: Teresa Yeo, name: Roman Bachmann, name: Amir Zamir
Published: 2024-07-24 15:42:34

Different users find different images generated for the same prompt
desirable. This gives rise to personalized image generation which involves
creating images aligned with an individual's visual preference. Current
generative models are, however, unpersonalized, as they are tuned to produce
outputs that appeal to a broad audience. Using them to generate images aligned
with individual users relies on iterative manual prompt engineering by the user
which is inefficient and undesirable. We propose to personalize the image
generation process by first capturing the generic preferences of the user in a
one-time process by inviting them to comment on a small selection of images,
explaining why they like or dislike each. Based on these comments, we infer a
user's structured liked and disliked visual attributes, i.e., their visual
preference, using a large language model. These attributes are used to guide a
text-to-image model toward producing images that are tuned towards the
individual user's visual preference. Through a series of user studies and large
language model guided evaluations, we demonstrate that the proposed method
results in generations that are well aligned with individual users' visual
preferences.
Scalify: scale propagation for efficient low-precision LLM training
Authors: name: Paul Balança, name: Sam Hosegood, name: Carlo Luschi, name: Andrew Fitzgibbon
Published: 2024-07-24 15:26:01

Low-precision formats such as float8 have been introduced in machine learning
accelerated hardware to improve computational efficiency for large language
models training and inference. Nevertheless, adoption by the ML community has
been slowed down by the complex, and sometimes brittle, techniques required to
match higher precision training accuracy. In this work, we present Scalify, a
end-to-end scale propagation paradigm for computational graphs, generalizing
and formalizing existing tensor scaling methods. Experiment results show that
Scalify supports out-of-the-box float8 matrix multiplication and gradients
representation, as well as float16 optimizer state storage. Our JAX
implementation of Scalify is open-sourced at
https://github.com/graphcore-research/jax-scalify
OpenDevin: An Open Platform for AI Software Developers as Generalist
  Agents
Authors: name: Xingyao Wang, name: Boxuan Li, name: Yufan Song, name: Frank F. Xu, name: Xiangru Tang, name: Mingchen Zhuge, name: Jiayi Pan, name: Yueqi Song, name: Bowen Li, name: Jaskirat Singh, name: Hoang H. Tran, name: Fuqiang Li, name: Ren Ma, name: Mingzhang Zheng, name: Bill Qian, name: Yanjun Shao, name: Niklas Muennighoff, name: Yizhe Zhang, name: Binyuan Hui, name: Junyang Lin, name: Robert Brennan, name: Hao Peng, name: Heng Ji, name: Graham Neubig
Published: 2024-07-23 17:50:43

Software is one of the most powerful tools that we humans have at our
disposal; it allows a skilled programmer to interact with the world in complex
and profound ways. At the same time, thanks to improvements in large language
models (LLMs), there has also been a rapid development in AI agents that
interact with and affect change in their surrounding environments. In this
paper, we introduce OpenDevin, a platform for the development of powerful and
flexible AI agents that interact with the world in similar ways to those of a
human developer: by writing code, interacting with a command line, and browsing
the web. We describe how the platform allows for the implementation of new
agents, safe interaction with sandboxed environments for code execution,
coordination between multiple agents, and incorporation of evaluation
benchmarks. Based on our currently incorporated benchmarks, we perform an
evaluation of agents over 15 challenging tasks, including software engineering
(e.g., SWE-Bench) and web browsing (e.g., WebArena), among others. Released
under the permissive MIT license, OpenDevin is a community project spanning
academia and industry with more than 1.3K contributions from over 160
contributors and will improve going forward.
SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View
  Consistency
Authors: name: Yiming Xie, name: Chun-Han Yao, name: Vikram Voleti, name: Huaizu Jiang, name: Varun Jampani
Published: 2024-07-24 17:59:43

We present Stable Video 4D (SV4D), a latent video diffusion model for
multi-frame and multi-view consistent dynamic 3D content generation. Unlike
previous methods that rely on separately trained generative models for video
generation and novel view synthesis, we design a unified diffusion model to
generate novel view videos of dynamic 3D objects. Specifically, given a
monocular reference video, SV4D generates novel views for each video frame that
are temporally consistent. We then use the generated novel view videos to
optimize an implicit 4D representation (dynamic NeRF) efficiently, without the
need for cumbersome SDS-based optimization used in most prior works. To train
our unified novel view video generation model, we curated a dynamic 3D object
dataset from the existing Objaverse dataset. Extensive experimental results on
multiple datasets and user studies demonstrate SV4D's state-of-the-art
performance on novel-view video synthesis as well as 4D generation compared to
prior works.
PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing
Authors: name: Blazej Manczak, name: Eliott Zemour, name: Eric Lin, name: Vaikkunth Mugunthan
Published: 2024-07-23 09:14:27

Deploying language models (LMs) necessitates outputs to be both high-quality
and compliant with safety guidelines. Although Inference-Time Guardrails (ITG)
offer solutions that shift model output distributions towards compliance, we
find that current methods struggle in balancing safety with helpfulness. ITG
Methods that safely address non-compliant queries exhibit lower helpfulness
while those that prioritize helpfulness compromise on safety. We refer to this
trade-off as the guardrail tax, analogous to the alignment tax. To address
this, we propose PrimeGuard, a novel ITG method that utilizes structured
control flow.
  PrimeGuard routes requests to different self-instantiations of the LM with
varying instructions, leveraging its inherent instruction-following
capabilities and in-context learning. Our tuning-free approach dynamically
compiles system-designer guidelines for each query. We construct and release
safe-eval, a diverse red-team safety benchmark. Extensive evaluations
demonstrate that PrimeGuard, without fine-tuning, overcomes the guardrail tax
by (1) significantly increasing resistance to iterative jailbreak attacks and
(2) achieving state-of-the-art results in safety guardrailing while (3)
matching helpfulness scores of alignment-tuned models. Extensive evaluations
demonstrate that PrimeGuard, without fine-tuning, outperforms all competing
baselines and overcomes the guardrail tax by improving the fraction of safe
responses from 61% to 97% and increasing average helpfulness scores from 4.17
to 4.29 on the largest models, while reducing attack success rate from 100% to
8%.
  PrimeGuard implementation is available at
https://github.com/dynamofl/PrimeGuard and safe-eval dataset is available at
https://huggingface.co/datasets/dynamoai/safe_eval.
SIGMA: Sinkhorn-Guided Masked Video Modeling
Authors: name: Mohammadreza Salehi, name: Michael Dorkenwald, name: Fida Mohammad Thoker, name: Efstratios Gavves, name: Cees G. M. Snoek, name: Yuki M. Asano
Published: 2024-07-22 08:04:09

Video-based pretraining offers immense potential for learning strong visual
representations on an unprecedented scale. Recently, masked video modeling
methods have shown promising scalability, yet fall short in capturing
higher-level semantics due to reconstructing predefined low-level targets such
as pixels. To tackle this, we present Sinkhorn-guided Masked Video Modelling
(SIGMA), a novel video pretraining method that jointly learns the video model
in addition to a target feature space using a projection network. However, this
simple modification means that the regular L2 reconstruction loss will lead to
trivial solutions as both networks are jointly optimized. As a solution, we
distribute features of space-time tubes evenly across a limited number of
learnable clusters. By posing this as an optimal transport problem, we enforce
high entropy in the generated features across the batch, infusing semantic and
temporal meaning into the feature space. The resulting cluster assignments are
used as targets for a symmetric prediction task where the video model predicts
cluster assignment of the projection network and vice versa. Experimental
results on ten datasets across three benchmarks validate the effectiveness of
SIGMA in learning more performant, temporally-aware, and robust video
representations improving upon state-of-the-art methods. Our project website
with code is available at: https://quva-lab.github.io/SIGMA.
A Simulation Benchmark for Autonomous Racing with Large-Scale Human Data
Authors: name: Adrian Remonda, name: Nicklas Hansen, name: Ayoub Raji, name: Nicola Musiu, name: Marko Bertogna, name: Eduardo Veas, name: Xiaolong Wang
Published: 2024-07-23 17:45:16

Despite the availability of international prize-money competitions, scaled
vehicles, and simulation environments, research on autonomous racing and the
control of sports cars operating close to the limit of handling has been
limited by the high costs of vehicle acquisition and management, as well as the
limited physics accuracy of open-source simulators. In this paper, we propose a
racing simulation platform based on the simulator Assetto Corsa to test,
validate, and benchmark autonomous driving algorithms, including reinforcement
learning (RL) and classical Model Predictive Control (MPC), in realistic and
challenging scenarios. Our contributions include the development of this
simulation platform, several state-of-the-art algorithms tailored to the racing
environment, and a comprehensive dataset collected from human drivers.
Additionally, we evaluate algorithms in the offline RL setting. All the
necessary code (including environment and benchmarks), working examples,
datasets, and videos are publicly released and can be found at:
https://assetto-corsa-gym.github.io.
KAN or MLP: A Fairer Comparison
Authors: name: Runpeng Yu, name: Weihao Yu, name: Xinchao Wang
Published: 2024-07-23 17:43:35

This paper does not introduce a novel method. Instead, it offers a fairer and
more comprehensive comparison of KAN and MLP models across various tasks,
including machine learning, computer vision, audio processing, natural language
processing, and symbolic formula representation. Specifically, we control the
number of parameters and FLOPs to compare the performance of KAN and MLP. Our
main observation is that, except for symbolic formula representation tasks, MLP
generally outperforms KAN. We also conduct ablation studies on KAN and find
that its advantage in symbolic formula representation mainly stems from its
B-spline activation function. When B-spline is applied to MLP, performance in
symbolic formula representation significantly improves, surpassing or matching
that of KAN. However, in other tasks where MLP already excels over KAN,
B-spline does not substantially enhance MLP's performance. Furthermore, we find
that KAN's forgetting issue is more severe than that of MLP in a standard
class-incremental continual learning setting, which differs from the findings
reported in the KAN paper. We hope these results provide insights for future
research on KAN and other MLP alternatives. Project link:
https://github.com/yu-rp/KANbeFair
Cross Anything: General Quadruped Robot Navigation through Complex
  Terrains
Authors: name: Shaoting Zhu, name: Derun Li, name: Yong Liu, name: Ningyi Xu, name: Hang Zhao
Published: 2024-07-23 11:57:54

The application of vision-language models (VLMs) has achieved impressive
success in various robotics tasks, but there are few explorations for
foundation models used in quadruped robot navigation. We introduce Cross
Anything System (CAS), an innovative system composed of a high-level reasoning
module and a low-level control policy, enabling the robot to navigate across
complex 3D terrains and reach the goal position. For high-level reasoning and
motion planning, we propose a novel algorithmic system taking advantage of a
VLM, with a design of task decomposition and a closed-loop sub-task execution
mechanism. For low-level locomotion control, we utilize the Probability
Annealing Selection (PAS) method to train a control policy by reinforcement
learning. Numerous experiments show that our whole system can accurately and
robustly navigate across complex 3D terrains, and its strong generalization
ability ensures the applications in diverse indoor and outdoor scenarios and
terrains. Project page: https://cross-anything.github.io/
T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video
  Generation
Authors: name: Kaiyue Sun, name: Kaiyi Huang, name: Xian Liu, name: Yue Wu, name: Zihan Xu, name: Zhenguo Li, name: Xihui Liu
Published: 2024-07-19 17:58:36

Text-to-video (T2V) generation models have advanced significantly, yet their
ability to compose different objects, attributes, actions, and motions into a
video remains unexplored. Previous text-to-video benchmarks also neglect this
important ability for evaluation. In this work, we conduct the first systematic
study on compositional text-to-video generation. We propose T2V-CompBench, the
first benchmark tailored for compositional text-to-video generation.
T2V-CompBench encompasses diverse aspects of compositionality, including
consistent attribute binding, dynamic attribute binding, spatial relationships,
motion binding, action binding, object interactions, and generative numeracy.
We further carefully design evaluation metrics of MLLM-based metrics,
detection-based metrics, and tracking-based metrics, which can better reflect
the compositional text-to-video generation quality of seven proposed categories
with 700 text prompts. The effectiveness of the proposed metrics is verified by
correlation with human evaluations. We also benchmark various text-to-video
generative models and conduct in-depth analysis across different models and
different compositional categories. We find that compositional text-to-video
generation is highly challenging for current models, and we hope that our
attempt will shed light on future research in this direction.
OutfitAnyone: Ultra-high Quality Virtual Try-On for Any Clothing and Any
  Person
Authors: name: Ke Sun, name: Jian Cao, name: Qi Wang, name: Linrui Tian, name: Xindi Zhang, name: Lian Zhuo, name: Bang Zhang, name: Liefeng Bo, name: Wenbo Zhou, name: Weiming Zhang, name: Daiheng Gao
Published: 2024-07-23 07:04:42

Virtual Try-On (VTON) has become a transformative technology, empowering
users to experiment with fashion without ever having to physically try on
clothing. However, existing methods often struggle with generating
high-fidelity and detail-consistent results. While diffusion models, such as
Stable Diffusion series, have shown their capability in creating high-quality
and photorealistic images, they encounter formidable challenges in conditional
generation scenarios like VTON. Specifically, these models struggle to maintain
a balance between control and consistency when generating images for virtual
clothing trials. OutfitAnyone addresses these limitations by leveraging a
two-stream conditional diffusion model, enabling it to adeptly handle garment
deformation for more lifelike results. It distinguishes itself with
scalability-modulating factors such as pose, body shape and broad
applicability, extending from anime to in-the-wild images. OutfitAnyone's
performance in diverse scenarios underscores its utility and readiness for
real-world deployment. For more details and animated results, please see
https://humanaigc.github.io/outfit-anyone/.