Orionnnn/hub-stats huggingface dataset:
MVD^2: Efficient Multiview 3D Reconstruction for Multiview Diffusion
Authors: name: Xin-Yang Zheng, name: Hao Pan, name: Yu-Xiao Guo, name: Xin Tong, name: Yang Liu
Published: 2024-02-22 03:39:48

As a promising 3D generation technique, multiview diffusion (MVD) has
received a lot of attention due to its advantages in terms of generalizability,
quality, and efficiency. By finetuning pretrained large image diffusion models
with 3D data, the MVD methods first generate multiple views of a 3D object
based on an image or text prompt and then reconstruct 3D shapes with multiview
3D reconstruction. However, the sparse views and inconsistent details in the
generated images make 3D reconstruction challenging. We present MVD^2, an
efficient 3D reconstruction method for multiview diffusion (MVD) images.
MVD^2 aggregates image features into a 3D feature volume by projection and
convolution and then decodes volumetric features into a 3D mesh. We train
MVD^2 with 3D shape collections and MVD images prompted by rendered views of
3D shapes. To address the discrepancy between the generated multiview images
and ground-truth views of the 3D shapes, we design a simple-yet-efficient
view-dependent training scheme. MVD^2 improves the 3D generation quality of
MVD and is fast and robust to various MVD methods. After training, it can
efficiently decode 3D meshes from multiview images within one second. We train
MVD^2 with Zero-123++ and ObjectVerse-LVIS 3D dataset and demonstrate its
superior performance in generating 3D models from multiview images generated by
different MVD methods, using both synthetic and real images as prompts.
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with
  Trajectory Stitching
Authors: name: Zizheng Pan, name: Bohan Zhuang, name: De-An Huang, name: Weili Nie, name: Zhiding Yu, name: Chaowei Xiao, name: Jianfei Cai, name: Anima Anandkumar
Published: 2024-02-21 23:08:54

Sampling from diffusion probabilistic models (DPMs) is often expensive for
high-quality image generation and typically requires many steps with a large
model. In this paper, we introduce sampling Trajectory Stitching T-Stitch, a
simple yet efficient technique to improve the sampling efficiency with little
or no generation degradation. Instead of solely using a large DPM for the
entire sampling trajectory, T-Stitch first leverages a smaller DPM in the
initial steps as a cheap drop-in replacement of the larger DPM and switches to
the larger DPM at a later stage. Our key insight is that different diffusion
models learn similar encodings under the same training data distribution and
smaller models are capable of generating good global structures in the early
steps. Extensive experiments demonstrate that T-Stitch is training-free,
generally applicable for different architectures, and complements most existing
fast sampling techniques with flexible speed and quality trade-offs. On DiT-XL,
for example, 40% of the early timesteps can be safely replaced with a 10x
faster DiT-S without performance drop on class-conditional ImageNet generation.
We further show that our method can also be used as a drop-in technique to not
only accelerate the popular pretrained stable diffusion (SD) models but also
improve the prompt alignment of stylized SD models from the public model zoo.
Code is released at https://github.com/NVlabs/T-Stitch
LexC-Gen: Generating Data for Extremely Low-Resource Languages with
  Large Language Models and Bilingual Lexicons
Authors: name: Zheng-Xin Yong, name: Cristina Menghini, name: Stephen H. Bach
Published: 2024-02-21 19:20:06

Data scarcity in low-resource languages can be addressed with word-to-word
translations from labeled task data in high-resource languages using bilingual
lexicons. However, bilingual lexicons often have limited lexical overlap with
task data, which results in poor translation coverage and lexicon utilization.
We propose lexicon-conditioned data generation (LexC-Gen), a method that
generates low-resource-language classification task data at scale.
Specifically, LexC-Gen first uses high-resource-language words from bilingual
lexicons to generate lexicon-compatible task data, and then it translates them
into low-resource languages with bilingual lexicons via word translation.
Across 17 extremely low-resource languages, LexC-Gen generated data is
competitive with expert-translated gold data, and yields on average 5.6 and 8.9
points improvement over existing lexicon-based word translation methods on
sentiment analysis and topic classification tasks respectively. We show that
conditioning on bilingual lexicons is the key component of LexC-Gen. LexC-Gen
is also practical -- it only needs a single GPU to generate data at scale. It
works well with open-access LLMs, and its cost is one-fifth of the cost of
GPT4-based multilingual data generation.
Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video
  Synthesis
Authors: name: Willi Menapace, name: Aliaksandr Siarohin, name: Ivan Skorokhodov, name: Ekaterina Deyneka, name: Tsai-Shien Chen, name: Anil Kag, name: Yuwei Fang, name: Aleksei Stoliar, name: Elisa Ricci, name: Jian Ren, name: Sergey Tulyakov
Published: 2024-02-22 18:55:08

Contemporary models for generating images show remarkable quality and
versatility. Swayed by these advantages, the research community repurposes them
to generate videos. Since video content is highly redundant, we argue that
naively bringing advances of image models to the video generation domain
reduces motion fidelity, visual quality and impairs scalability. In this work,
we build Snap Video, a video-first model that systematically addresses these
challenges. To do that, we first extend the EDM framework to take into account
spatially and temporally redundant pixels and naturally support video
generation. Second, we show that a U-Net - a workhorse behind image generation
- scales poorly when generating videos, requiring significant computational
overhead. Hence, we propose a new transformer-based architecture that trains
3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us
to efficiently train a text-to-video model with billions of parameters for the
first time, reach state-of-the-art results on a number of benchmarks, and
generate videos with substantially higher quality, temporal consistency, and
motion complexity. The user studies showed that our model was favored by a
large margin over the most recent methods. See our website at
https://snap-research.github.io/snapvideo/.
D-Flow: Differentiating through Flows for Controlled Generation
Authors: name: Heli Ben-Hamu, name: Omri Puny, name: Itai Gat, name: Brian Karrer, name: Uriel Singer, name: Yaron Lipman
Published: 2024-02-21 18:56:03

Taming the generation outcome of state of the art Diffusion and Flow-Matching
(FM) models without having to re-train a task-specific model unlocks a powerful
tool for solving inverse problems, conditional generation, and controlled
generation in general. In this work we introduce D-Flow, a simple framework for
controlling the generation process by differentiating through the flow,
optimizing for the source (noise) point. We motivate this framework by our key
observation stating that for Diffusion/FM models trained with Gaussian
probability paths, differentiating through the generation process projects
gradient on the data manifold, implicitly injecting the prior into the
optimization process. We validate our framework on linear and non-linear
controlled generation problems including: image and audio inverse problems and
conditional molecule generation reaching state of the art performance across
all.
ToDo: Token Downsampling for Efficient Generation of High-Resolution
  Images
Authors: name: Ethan Smith, name: Nayan Saxena, name: Aninda Saha
Published: 2024-02-21 07:10:28

Attention mechanism has been crucial for image diffusion models, however,
their quadratic computational complexity limits the sizes of images we can
process within reasonable time and memory constraints. This paper investigates
the importance of dense attention in generative image models, which often
contain redundant features, making them suitable for sparser attention
mechanisms. We propose a novel training-free method ToDo that relies on token
downsampling of key and value tokens to accelerate Stable Diffusion inference
by up to 2x for common sizes and up to 4.5x or more for high resolutions like
2048x2048. We demonstrate that our approach outperforms previous methods in
balancing efficient throughput and fidelity.
Coercing LLMs to do and reveal (almost) anything
Authors: name: Jonas Geiping, name: Alex Stein, name: Manli Shu, name: Khalid Saifullah, name: Yuxin Wen, name: Tom Goldstein
Published: 2024-02-21 18:59:13

It has recently been shown that adversarial attacks on large language models
(LLMs) can "jailbreak" the model into making harmful statements. In this work,
we argue that the spectrum of adversarial attacks on LLMs is much larger than
merely jailbreaking. We provide a broad overview of possible attack surfaces
and attack goals. Based on a series of concrete examples, we discuss,
categorize and systematize attacks that coerce varied unintended behaviors,
such as misdirection, model control, denial-of-service, or data extraction.
  We analyze these attacks in controlled experiments, and find that many of
them stem from the practice of pre-training LLMs with coding capabilities, as
well as the continued existence of strange "glitch" tokens in common LLM
vocabularies that should be removed for security reasons.
Music Style Transfer with Time-Varying Inversion of Diffusion Models
Authors: name: Sifei Li, name: Yuxin Zhang, name: Fan Tang, name: Chongyang Ma, name: Weiming dong, name: Changsheng Xu
Published: 2024-02-21 12:38:48

With the development of diffusion models, text-guided image style transfer
has demonstrated high-quality controllable synthesis results. However, the
utilization of text for diverse music style transfer poses significant
challenges, primarily due to the limited availability of matched audio-text
datasets. Music, being an abstract and complex art form, exhibits variations
and intricacies even within the same genre, thereby making accurate textual
descriptions challenging. This paper presents a music style transfer approach
that effectively captures musical attributes using minimal data. We introduce a
novel time-varying textual inversion module to precisely capture
mel-spectrogram features at different levels. During inference, we propose a
bias-reduced stylization technique to obtain stable results. Experimental
results demonstrate that our method can transfer the style of specific
instruments, as well as incorporate natural sounds to compose melodies. Samples
and source code are available at https://lsfhuihuiff.github.io/MusicTI/.
YOLOv9: Learning What You Want to Learn Using Programmable Gradient
  Information
Authors: name: Chien-Yao Wang, name: I-Hau Yeh, name: Hong-Yuan Mark Liao
Published: 2024-02-21 08:42:53

Today's deep learning methods focus on how to design the most appropriate
objective functions so that the prediction results of the model can be closest
to the ground truth. Meanwhile, an appropriate architecture that can facilitate
acquisition of enough information for prediction has to be designed. Existing
methods ignore a fact that when input data undergoes layer-by-layer feature
extraction and spatial transformation, large amount of information will be
lost. This paper will delve into the important issues of data loss when data is
transmitted through deep networks, namely information bottleneck and reversible
functions. We proposed the concept of programmable gradient information (PGI)
to cope with the various changes required by deep networks to achieve multiple
objectives. PGI can provide complete input information for the target task to
calculate objective function, so that reliable gradient information can be
obtained to update network weights. In addition, a new lightweight network
architecture -- Generalized Efficient Layer Aggregation Network (GELAN), based
on gradient path planning is designed. GELAN's architecture confirms that PGI
has gained superior results on lightweight models. We verified the proposed
GELAN and PGI on MS COCO dataset based object detection. The results show that
GELAN only uses conventional convolution operators to achieve better parameter
utilization than the state-of-the-art methods developed based on depth-wise
convolution. PGI can be used for variety of models from lightweight to large.
It can be used to obtain complete information, so that train-from-scratch
models can achieve better results than state-of-the-art models pre-trained
using large datasets, the comparison results are shown in Figure 1. The source
codes are at: https://github.com/WongKinYiu/yolov9.
User-LLM: Efficient LLM Contextualization with User Embeddings
Authors: name: Lin Ning, name: Luyang Liu, name: Jiaxing Wu, name: Neo Wu, name: Devora Berlowitz, name: Sushant Prakash, name: Bradley Green, name: Shawn O'Banion, name: Jun Xie
Published: 2024-02-21 08:03:27

Large language models (LLMs) have revolutionized natural language processing.
However, effectively incorporating complex and potentially noisy user
interaction data remains a challenge. To address this, we propose User-LLM, a
novel framework that leverages user embeddings to contextualize LLMs. These
embeddings, distilled from diverse user interactions using self-supervised
pretraining, capture latent user preferences and their evolution over time. We
integrate these user embeddings with LLMs through cross-attention and
soft-prompting, enabling LLMs to dynamically adapt to user context. Our
comprehensive experiments on MovieLens, Amazon Review, and Google Local Review
datasets demonstrate significant performance gains across various tasks.
Notably, our approach outperforms text-prompt-based contextualization on long
sequence tasks and tasks that require deep user understanding while being
computationally efficient. We further incorporate Perceiver layers to
streamline the integration between user encoders and LLMs, reducing
computational demands.
Ouroboros: Speculative Decoding with Large Model Enhanced Drafting
Authors: name: Weilin Zhao, name: Yuxiang Huang, name: Xu Han, name: Chaojun Xiao, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-02-21 11:31:28

Drafting-then-verifying decoding methods such as speculative decoding are
widely adopted training-free methods to accelerate the inference of large
language models (LLMs). Instead of employing an autoregressive process to
decode tokens sequentially, speculative decoding initially creates drafts with
an efficient small model. Then LLMs are required to conduct verification and
correction in a non-autoregressive fashion to minimize time overhead.
Generating longer drafts can lead to even more significant speedups once
verified, but also incurs substantial trial and error costs if it fails.
Suffering from the high verification failure probability, existing decoding
methods cannot draft too much content for verification at one time, achieving
sub-optimal inference acceleration. In this paper, we introduce Ouroboros,
which constructs a phrase candidate pool from the verification process of LLMs
to provide candidates for draft generation of the small model. Thereby,
Ouroboros can further improve the efficiency and effectiveness of the initial
drafts. The experimental results on typical text generation tasks show that
Ouroboros achieves speedups of up to 1.9x and 2.8x compared to lookahead
decoding and speculative decoding, respectively. The source code of Ouroboros
is available at https://github.com/thunlp/Ouroboros.
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large
  Vision-Language Models
Authors: name: Xueliang Zhao, name: Xinting Huang, name: Tingchen Fu, name: Qintong Li, name: Shansan Gong, name: Lemao Liu, name: Wei Bi, name: Lingpeng Kong
Published: 2024-02-21 07:16:29

Multimodal reasoning stands as a pivotal capability for large vision-language
models (LVLMs). The integration with Domain-Specific Languages (DSL), offering
precise visual representations, equips these models with the opportunity to
execute more accurate reasoning in complex and professional domains. However,
the vanilla Chain-of-Thought (CoT) prompting method faces challenges in
effectively leveraging the unique strengths of visual and DSL representations,
primarily due to their differing reasoning mechanisms. Additionally, it often
falls short in addressing critical steps in multi-step reasoning tasks. To
mitigate these challenges, we introduce the Bi-Modal
Behavioral Alignment (BBA) prompting method, designed
to maximize the potential of DSL in augmenting complex multi-modal reasoning
tasks. This method initiates by guiding LVLMs to create separate reasoning
chains for visual and DSL representations. Subsequently, it aligns these chains
by addressing any inconsistencies, thus achieving a cohesive integration of
behaviors from different modalities. Our experiments demonstrate that BBA
substantially improves the performance of GPT-4V(ision) on geometry problem
solving (28.34% to 34.22%), chess positional advantage prediction
(42.08% to 46.99%) and molecular property prediction (77.47% to
83.52%).
In deep reinforcement learning, a pruned network is a good network
Authors: name: Johan Obando-Ceron, name: Aaron Courville, name: Pablo Samuel Castro
Published: 2024-02-19 19:34:07

Recent work has shown that deep reinforcement learning agents have difficulty
in effectively using their network parameters. We leverage prior insights into
the advantages of sparse training techniques and demonstrate that gradual
magnitude pruning enables agents to maximize parameter effectiveness. This
results in networks that yield dramatic performance improvements over
traditional networks and exhibit a type of "scaling law", using only a small
fraction of the full network parameters.
Aria Everyday Activities Dataset
Authors: name: Zhaoyang Lv, name: Nickolas Charron, name: Pierre Moulon, name: Alexander Gamino, name: Cheng Peng, name: Chris Sweeney, name: Edward Miller, name: Huixuan Tang, name: Jeff Meissner, name: Jing Dong, name: Kiran Somasundaram, name: Luis Pesqueira, name: Mark Schwesinger, name: Omkar Parkhi, name: Qiao Gu, name: Renzo De Nardi, name: Shangyi Cheng, name: Steve Saarinen, name: Vijay Baiyya, name: Yuyang Zou, name: Richard Newcombe, name: Jakob Julian Engel, name: Xiaqing Pan, name: Carl Ren
Published: 2024-02-20 19:53:15

We present Aria Everyday Activities (AEA) Dataset, an egocentric multimodal
open dataset recorded using Project Aria glasses. AEA contains 143 daily
activity sequences recorded by multiple wearers in five geographically diverse
indoor locations. Each of the recording contains multimodal sensor data
recorded through the Project Aria glasses. In addition, AEA provides machine
perception data including high frequency globally aligned 3D trajectories,
scene point cloud, per-frame 3D eye gaze vector and time aligned speech
transcription. In this paper, we demonstrate a few exemplar research
applications enabled by this dataset, including neural scene reconstruction and
prompted segmentation. AEA is an open source dataset that can be downloaded
from projectaria.com. We are also providing open-source implementations and
examples of how to use the dataset in Project Aria Tools.
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Authors: name: Yiran Ding, name: Li Lyna Zhang, name: Chengruidong Zhang, name: Yuanyuan Xu, name: Ning Shang, name: Jiahang Xu, name: Fan Yang, name: Mao Yang
Published: 2024-02-21 12:30:33

Large context window is a desirable feature in large language models (LLMs).
However, due to high fine-tuning costs, scarcity of long texts, and
catastrophic values introduced by new token positions, current extended context
windows are limited to around 128k tokens. This paper introduces LongRoPE that,
for the first time, extends the context window of pre-trained LLMs to an
impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k
training lengths, while maintaining performance at the original short context
window. This is achieved by three key innovations: (i) we identify and exploit
two forms of non-uniformities in positional interpolation through an efficient
search, providing a better initialization for fine-tuning and enabling an 8x
extension in non-fine-tuning scenarios; (ii) we introduce a progressive
extension strategy that first fine-tunes a 256k length LLM and then conducts a
second positional interpolation on the fine-tuned extended LLM to achieve a
2048k context window; (iii) we readjust LongRoPE on 8k length to recover the
short context window performance. Extensive experiments on LLaMA2 and Mistral
across various tasks demonstrate the effectiveness of our method. Models
extended via LongRoPE retain the original architecture with minor modifications
to the positional embedding, and can reuse most pre-existing optimizations.
SDXL-Lightning: Progressive Adversarial Diffusion Distillation
Authors: name: Shanchuan Lin, name: Anran Wang, name: Xiao Yang
Published: 2024-02-21 16:51:05

We propose a diffusion distillation method that achieves new state-of-the-art
in one-step/few-step 1024px text-to-image generation based on SDXL. Our method
combines progressive and adversarial distillation to achieve a balance between
quality and mode coverage. In this paper, we discuss the theoretical analysis,
discriminator design, model formulation, and training techniques. We
open-source our distilled SDXL-Lightning models both as LoRA and full UNet
weights.
FlashTex: Fast Relightable Mesh Texturing with LightControlNet
Authors: name: Kangle Deng, name: Timothy Omernick, name: Alexander Weiss, name: Deva Ramanan, name: Jun-Yan Zhu, name: Tinghui Zhou, name: Maneesh Agrawala
Published: 2024-02-20 18:59:00

Manually creating textures for 3D meshes is time-consuming, even for expert
visual content creators. We propose a fast approach for automatically texturing
an input 3D mesh based on a user-provided text prompt. Importantly, our
approach disentangles lighting from surface material/reflectance in the
resulting texture so that the mesh can be properly relit and rendered in any
lighting environment. We introduce LightControlNet, a new text-to-image model
based on the ControlNet architecture, which allows the specification of the
desired lighting as a conditioning image to the model. Our text-to-texture
pipeline then constructs the texture in two stages. The first stage produces a
sparse set of visually consistent reference views of the mesh using
LightControlNet. The second stage applies a texture optimization based on Score
Distillation Sampling (SDS) that works with LightControlNet to increase the
texture quality while disentangling surface material from lighting. Our
pipeline is significantly faster than previous text-to-texture methods, while
producing high-quality and relightable textures.