Orionnnn/hub-stats huggingface dataset:
TripoSR: Fast 3D Object Reconstruction from a Single Image
Authors: name: Dmitry Tochilkin, name: David Pankratz, name: Zexiang Liu, name: Zixuan Huang, name: Adam Letts, name: Yangguang Li, name: Ding Liang, name: Christian Laforte, name: Varun Jampani, name: Yan-Pei Cao
Published: 2024-03-04 16:00:56

This technical report introduces TripoSR, a 3D reconstruction model
leveraging transformer architecture for fast feed-forward 3D generation,
producing 3D mesh from a single image in under 0.5 seconds. Building upon the
LRM network architecture, TripoSR integrates substantial improvements in data
processing, model design, and training techniques. Evaluations on public
datasets show that TripoSR exhibits superior performance, both quantitatively
and qualitatively, compared to other open-source alternatives. Released under
the MIT license, TripoSR is intended to empower researchers, developers, and
creatives with the latest advancements in 3D generative AI.
RT-H: Action Hierarchies Using Language
Authors: name: Suneel Belkhale, name: Tianli Ding, name: Ted Xiao, name: Pierre Sermanet, name: Quon Vuong, name: Jonathan Tompson, name: Yevgen Chebotar, name: Debidatta Dwibedi, name: Dorsa Sadigh
Published: 2024-03-04 08:16:11

Language provides a way to break down complex concepts into digestible
pieces. Recent works in robot imitation learning use language-conditioned
policies that predict actions given visual observations and the high-level task
specified in language. These methods leverage the structure of natural language
to share data between semantically similar tasks (e.g., "pick coke can" and
"pick an apple") in multi-task datasets. However, as tasks become more
semantically diverse (e.g., "pick coke can" and "pour cup"), sharing data
between tasks becomes harder, so learning to map high-level tasks to actions
requires much more demonstration data. To bridge tasks and actions, our insight
is to teach the robot the language of actions, describing low-level motions
with more fine-grained phrases like "move arm forward". Predicting these
language motions as an intermediate step between tasks and actions forces the
policy to learn the shared structure of low-level motions across seemingly
disparate tasks. Furthermore, a policy that is conditioned on language motions
can easily be corrected during execution through human-specified language
motions. This enables a new paradigm for flexible policies that can learn from
human intervention in language. Our method RT-H builds an action hierarchy
using language motions: it first learns to predict language motions, and
conditioned on this and the high-level task, it predicts actions, using visual
context at all stages. We show that RT-H leverages this language-action
hierarchy to learn policies that are more robust and flexible by effectively
tapping into multi-task datasets. We show that these policies not only allow
for responding to language interventions, but can also learn from such
interventions and outperform methods that learn from teleoperated
interventions. Our website and videos are found at
https://rt-hierarchy.github.io.
OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable
  Virtual Try-on
Authors: name: Yuhao Xu, name: Tao Gu, name: Weifeng Chen, name: Chengcai Chen
Published: 2024-03-04 07:17:44

Image-based virtual try-on (VTON), which aims to generate an outfitted image
of a target human wearing an in-shop garment, is a challenging image-synthesis
task calling for not only high fidelity of the outfitted human but also full
preservation of garment details. To tackle this issue, we propose Outfitting
over Try-on Diffusion (OOTDiffusion), leveraging the power of pretrained latent
diffusion models and designing a novel network architecture for realistic and
controllable virtual try-on. Without an explicit warping process, we propose an
outfitting UNet to learn the garment detail features, and merge them with the
target human body via our proposed outfitting fusion in the denoising process
of diffusion models. In order to further enhance the controllability of our
outfitting UNet, we introduce outfitting dropout to the training process, which
enables us to adjust the strength of garment features through classifier-free
guidance. Our comprehensive experiments on the VITON-HD and Dress Code datasets
demonstrate that OOTDiffusion efficiently generates high-quality outfitted
images for arbitrary human and garment images, which outperforms other VTON
methods in both fidelity and controllability, indicating an impressive
breakthrough in virtual try-on. Our source code is available at
https://github.com/levihsu/OOTDiffusion.
InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding
Authors: name: Haogeng Liu, name: Quanzeng You, name: Xiaotian Han, name: Yiqi Wang, name: Bohan Zhai, name: Yongfei Liu, name: Yunzhe Tao, name: Huaibo Huang, name: Ran He, name: Hongxia Yang
Published: 2024-03-03 11:39:41

Multimodal Large Language Models (MLLMs) have experienced significant
advancements recently. Nevertheless, challenges persist in the accurate
recognition and comprehension of intricate details within high-resolution
images. Despite being indispensable for the development of robust MLLMs, this
area remains underinvestigated. To tackle this challenge, our work introduces
InfiMM-HD, a novel architecture specifically designed for processing images of
different resolutions with low computational overhead. This innovation
facilitates the enlargement of MLLMs to higher-resolution capabilities.
InfiMM-HD incorporates a cross-attention module and visual windows to reduce
computation costs. By integrating this architectural design with a four-stage
training pipeline, our model attains improved visual perception efficiently and
cost-effectively. Empirical study underscores the robustness and effectiveness
of InfiMM-HD, opening new avenues for exploration in related areas. Codes and
models can be found at https://huggingface.co/Infi-MM/infimm-hd
3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming
  of Photo-Realistic Free-Viewpoint Videos
Authors: name: Jiakai Sun, name: Han Jiao, name: Guangyuan Li, name: Zhanjie Zhang, name: Lei Zhao, name: Wei Xing
Published: 2024-03-03 08:42:40

Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes
from multi-view videos remains a challenging endeavor. Despite the remarkable
advancements achieved by current neural rendering techniques, these methods
generally require complete video sequences for offline training and are not
capable of real-time rendering. To address these constraints, we introduce
3DGStream, a method designed for efficient FVV streaming of real-world dynamic
scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12
seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D
Gaussians (3DGs) to represent the scene. Instead of the na\"ive approach of
directly optimizing 3DGs per-frame, we employ a compact Neural Transformation
Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing
the training time and storage required for each FVV frame. Furthermore, we
propose an adaptive 3DG addition strategy to handle emerging objects in dynamic
scenes. Experiments demonstrate that 3DGStream achieves competitive performance
in terms of rendering speed, image quality, training time, and model storage
when compared with state-of-the-art methods.
MovieLLM: Enhancing Long Video Understanding with AI-Generated Movies
Authors: name: Zhende Song, name: Chenchen Wang, name: Jiamu Sheng, name: Chi Zhang, name: Gang Yu, name: Jiayuan Fan, name: Tao Chen
Published: 2024-03-03 07:43:39

The development of multimodal models has marked a significant step forward in
how machines understand videos. These models have shown promise in analyzing
short video clips. However, when it comes to longer formats like movies, they
often fall short. The main hurdles are the lack of high-quality, diverse video
data and the intensive work required to collect or annotate such data. In the
face of these challenges, we propose MovieLLM, a novel framework designed to
create synthetic, high-quality data for long videos. This framework leverages
the power of GPT-4 and text-to-image models to generate detailed scripts and
corresponding visuals. Our approach stands out for its flexibility and
scalability, making it a superior alternative to traditional data collection
methods. Our extensive experiments validate that the data produced by MovieLLM
significantly improves the performance of multimodal models in understanding
complex video narratives, overcoming the limitations of existing datasets
regarding scarcity and bias.
ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models
Authors: name: Jiaxiang Cheng, name: Pan Xie, name: Xin Xia, name: Jiashi Li, name: Jie Wu, name: Yuxi Ren, name: Huixia Li, name: Xuefeng Xiao, name: Min Zheng, name: Lean Fu
Published: 2024-03-04 14:36:56

Recent advancement in text-to-image models (e.g., Stable Diffusion) and
corresponding personalized technologies (e.g., DreamBooth and LoRA) enables
individuals to generate high-quality and imaginative images. However, they
often suffer from limitations when generating images with resolutions outside
of their trained domain. To overcome this limitation, we present the Resolution
Adapter (ResAdapter), a domain-consistent adapter designed for diffusion models
to generate images with unrestricted resolutions and aspect ratios. Unlike
other multi-resolution generation methods that process images of static
resolution with complex post-process operations, ResAdapter directly generates
images with the dynamical resolution. Especially, after learning a deep
understanding of pure resolution priors, ResAdapter trained on the general
dataset, generates resolution-free images with personalized diffusion models
while preserving their original style domain. Comprehensive experiments
demonstrate that ResAdapter with only 0.5M can process images with flexible
resolutions for arbitrary diffusion models. More extended experiments
demonstrate that ResAdapter is compatible with other modules (e.g., ControlNet,
IP-Adapter and LCM-LoRA) for image generation across a broad range of
resolutions, and can be integrated into other multi-resolution model (e.g.,
ElasticDiffusion) for efficiently generating higher-resolution images. Project
link is https://res-adapter.github.io
ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
Authors: name: Lukas Höllein, name: Aljaž Božič, name: Norman Müller, name: David Novotny, name: Hung-Yu Tseng, name: Christian Richardt, name: Michael Zollhöfer, name: Matthias Nießner
Published: 2024-03-04 07:57:05

3D asset generation is getting massive amounts of attention, inspired by the
recent success of text-guided 2D content creation. Existing text-to-3D methods
use pretrained text-to-image diffusion models in an optimization problem or
fine-tune them on synthetic data, which often results in non-photorealistic 3D
objects without backgrounds. In this paper, we present a method that leverages
pretrained text-to-image models as a prior, and learn to generate multi-view
images in a single denoising process from real-world data. Concretely, we
propose to integrate 3D volume-rendering and cross-frame-attention layers into
each block of the existing U-Net network of the text-to-image model. Moreover,
we design an autoregressive generation that renders more 3D-consistent images
at any viewpoint. We train our model on real-world datasets of objects and
showcase its capabilities to generate instances with a variety of high-quality
shapes and textures in authentic surroundings. Compared to the existing
methods, the results generated by our method are consistent, and have favorable
visual quality (-30% FID, -37% KID).
AtomoVideo: High Fidelity Image-to-Video Generation
Authors: name: Litong Gong, name: Yiran Zhu, name: Weijie Li, name: Xiaoyang Kang, name: Biao Wang, name: Tiezheng Ge, name: Bo Zheng
Published: 2024-03-04 07:41:50

Recently, video generation has achieved significant rapid development based
on superior text-to-image generation techniques. In this work, we propose a
high fidelity framework for image-to-video generation, named AtomoVideo. Based
on multi-granularity image injection, we achieve higher fidelity of the
generated video to the given image. In addition, thanks to high quality
datasets and training strategies, we achieve greater motion intensity while
maintaining superior temporal consistency and stability. Our architecture
extends flexibly to the video frame prediction task, enabling long sequence
prediction through iterative generation. Furthermore, due to the design of
adapter training, our approach can be well combined with existing personalised
models and controllable modules. By quantitatively and qualitatively
evaluation, AtomoVideo achieves superior results compared to popular methods,
more examples can be found on our project website: https://atomo-
video.github.io/.
AtP*: An efficient and scalable method for localizing LLM behaviour to
  components
Authors: name: János Kramár, name: Tom Lieberum, name: Rohin Shah, name: Neel Nanda
Published: 2024-03-01 18:43:51

Activation Patching is a method of directly computing causal attributions of
behavior to model components. However, applying it exhaustively requires a
sweep with cost scaling linearly in the number of model components, which can
be prohibitively expensive for SoTA Large Language Models (LLMs). We
investigate Attribution Patching (AtP), a fast gradient-based approximation to
Activation Patching and find two classes of failure modes of AtP which lead to
significant false negatives. We propose a variant of AtP called AtP*, with two
changes to address these failure modes while retaining scalability. We present
the first systematic study of AtP and alternative methods for faster activation
patching and show that AtP significantly outperforms all other investigated
methods, with AtP* providing further significant improvement. Finally, we
provide a method to bound the probability of remaining false negatives of AtP*
estimates.
VisionLLaMA: A Unified LLaMA Interface for Vision Tasks
Authors: name: Xiangxiang Chu, name: Jianlin Su, name: Bo Zhang, name: Chunhua Shen
Published: 2024-03-01 13:30:51

Large language models are built on top of a transformer-based architecture to
process textual inputs. For example, the LLaMA stands out among many
open-source implementations. Can the same transformer be used to process 2D
images? In this paper, we answer this question by unveiling a LLaMA-like vision
transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored
for this purpose. VisionLLaMA is a unified and generic modelling framework for
solving most vision tasks. We extensively evaluate its effectiveness using
typical pre-training paradigms in a good portion of downstream tasks of image
perception and especially image generation. In many cases, VisionLLaMA have
exhibited substantial gains over the previous state-of-the-art vision
transformers. We believe that VisionLLaMA can serve as a strong new baseline
model for vision generation and understanding. Our code will be released at
https://github.com/Meituan-AutoML/VisionLLaMA.
Resonance RoPE: Improving Context Length Generalization of Large
  Language Models
Authors: name: Suyuchen Wang, name: Ivan Kobyzev, name: Peng Lu, name: Mehdi Rezagholizadeh, name: Bang Liu
Published: 2024-02-29 19:02:03

This paper addresses the challenge of train-short-test-long (TSTL) scenarios
in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
where models pre-trained on shorter sequences face difficulty with
out-of-distribution (OOD) token positions in longer sequences. We introduce
Resonance RoPE, a novel approach designed to narrow the generalization gap in
TSTL scenarios by refining the interpolation of RoPE features for OOD
positions, significantly improving the model performance without additional
online computational costs. Furthermore, we present PosGen, a new synthetic
benchmark specifically designed for fine-grained behavior analysis in TSTL
scenarios, aiming to isolate the constantly increasing difficulty of token
generation on long contexts from the challenges of recognizing new token
positions. Our experiments on synthetic tasks show that after applying
Resonance RoPE, Transformers recognize OOD position better and more robustly.
Our extensive LLM experiments also show superior performance after applying
Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
both upstream language modeling tasks and a variety of downstream long-text
applications.
Learning and Leveraging World Models in Visual Representation Learning
Authors: name: Quentin Garrido, name: Mahmoud Assran, name: Nicolas Ballas, name: Adrien Bardes, name: Laurent Najman, name: Yann LeCun
Published: 2024-03-01 13:05:38

Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising
self-supervised approach that learns by leveraging a world model. While
previously limited to predicting missing parts of an input, we explore how to
generalize the JEPA prediction task to a broader set of corruptions. We
introduce Image World Models, an approach that goes beyond masked image
modeling and learns to predict the effect of global photometric transformations
in latent space. We study the recipe of learning performant IWMs and show that
it relies on three key aspects: conditioning, prediction difficulty, and
capacity. Additionally, we show that the predictive world model learned by IWM
can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM
world model matches or surpasses the performance of previous self-supervised
methods. Finally, we show that learning with an IWM allows one to control the
abstraction level of the learned representations, learning invariant
representations such as contrastive methods, or equivariant representations
such as masked image modelling.
RealCustom: Narrowing Real Text Word for Real-Time Open-Domain
  Text-to-Image Customization
Authors: name: Mengqi Huang, name: Zhendong Mao, name: Mingcong Liu, name: Qian He, name: Yongdong Zhang
Published: 2024-03-01 12:12:09

Text-to-image customization, which aims to synthesize text-driven images for
the given subjects, has recently revolutionized content creation. Existing
works follow the pseudo-word paradigm, i.e., represent the given subjects as
pseudo-words and then compose them with the given text. However, the inherent
entangled influence scope of pseudo-words with the given text results in a
dual-optimum paradox, i.e., the similarity of the given subjects and the
controllability of the given text could not be optimal simultaneously. We
present RealCustom that, for the first time, disentangles similarity from
controllability by precisely limiting subject influence to relevant parts only,
achieved by gradually narrowing real text word from its general connotation to
the specific subject and using its cross-attention to distinguish relevance.
Specifically, RealCustom introduces a novel "train-inference" decoupled
framework: (1) during training, RealCustom learns general alignment between
visual conditions to original textual conditions by a novel adaptive scoring
module to adaptively modulate influence quantity; (2) during inference, a novel
adaptive mask guidance strategy is proposed to iteratively update the influence
scope and influence quantity of the given subjects to gradually narrow the
generation of the real text word. Comprehensive experiments demonstrate the
superior real-time customization ability of RealCustom in the open domain,
achieving both unprecedented similarity of the given subjects and
controllability of the given text for the first time. The project page is
https://corleone-huang.github.io/realcustom/.
MOSAIC: A Modular System for Assistive and Interactive Cooking
Authors: name: Huaxiaoyue Wang, name: Kushal Kedia, name: Juntao Ren, name: Rahma Abdullah, name: Atiksh Bhardwaj, name: Angela Chao, name: Kelly Y Chen, name: Nathaniel Chin, name: Prithwish Dan, name: Xinyi Fan, name: Gonzalo Gonzalez-Pumariega, name: Aditya Kompella, name: Maximus Adrian Pace, name: Yash Sharma, name: Xiangwan Sun, name: Neha Sunkara, name: Sanjiban Choudhury
Published: 2024-02-29 01:56:41

We present MOSAIC, a modular architecture for home robots to perform complex
collaborative tasks, such as cooking with everyday users. MOSAIC tightly
collaborates with humans, interacts with users using natural language,
coordinates multiple robots, and manages an open vocabulary of everyday
objects. At its core, MOSAIC employs modularity: it leverages multiple
large-scale pre-trained models for general tasks like language and image
recognition, while using streamlined modules designed for task-specific
control. We extensively evaluate MOSAIC on 60 end-to-end trials where two
robots collaborate with a human user to cook a combination of 6 recipes. We
also extensively test individual modules with 180 episodes of visuomotor
picking, 60 episodes of human motion forecasting, and 46 online user
evaluations of the task planner. We show that MOSAIC is able to efficiently
collaborate with humans by running the overall system end-to-end with a real
human user, completing 68.3% (41/60) collaborative cooking trials of 6
different recipes with a subtask completion rate of 91.6%. Finally, we discuss
the limitations of the current system and exciting open challenges in this
domain. The project's website is at https://portal-cornell.github.io/MOSAIC/
Humanoid Locomotion as Next Token Prediction
Authors: name: Ilija Radosavovic, name: Bike Zhang, name: Baifeng Shi, name: Jathushan Rajasegaran, name: Sarthak Kamat, name: Trevor Darrell, name: Koushil Sreenath, name: Jitendra Malik
Published: 2024-02-29 18:57:37

We cast real-world humanoid control as a next token prediction problem, akin
to predicting the next word in language. Our model is a causal transformer
trained via autoregressive prediction of sensorimotor trajectories. To account
for the multi-modal nature of the data, we perform prediction in a
modality-aligned way, and for each input token predict the next token from the
same modality. This general formulation enables us to leverage data with
missing modalities, like video trajectories without actions. We train our model
on a collection of simulated trajectories coming from prior neural network
policies, model-based controllers, motion capture data, and YouTube videos of
humans. We show that our model enables a full-sized humanoid to walk in San
Francisco zero-shot. Our model can transfer to the real world even when trained
on only 27 hours of walking data, and can generalize to commands not seen
during training like walking backward. These findings suggest a promising path
toward learning challenging real-world control tasks by generative modeling of
sensorimotor trajectories.