Orionnnn/hub-stats huggingface dataset:
Yuan 2.0-M32: Mixture of Experts with Attention Router
Authors: name: Shaohua Wu, name: Jiangang Luo, name: Xi Chen, name: Lingjun Li, name: Xudong Zhao, name: Tong Yu, name: Chao Wang, name: Yue Wang, name: Fei Wang, name: Weixu Qiao, name: Houbo He, name: Zeru Zhang, name: Zeyu Sun, name: Junxiong Mao, name: Chong Shen
Published: 2024-05-28 09:05:08

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a
mixture-of-experts architecture with 32 experts of which 2 experts are active.
A new router network, Attention Router, is proposed and adopted for a more
efficient selection of experts, which boosts the accuracy of 3.8% compared to
the model with classical router network. Yuan 2.0-M32 is trained with 2000B
tokens from scratch, and the training computation consumption is only 9.25% of
a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates
competitive capability on coding, math, and various domains of expertise, with
only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation
per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass
Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8
respectively. The models and source codes of Yuan 2.0-M32 are released at
Github.
Phased Consistency Model
Authors: name: Fu-Yun Wang, name: Zhaoyang Huang, name: Alexander William Bergman, name: Dazhong Shen, name: Peng Gao, name: Michael Lingelbach, name: Keqiang Sun, name: Weikang Bian, name: Guanglu Song, name: Yu Liu, name: Hongsheng Li, name: Xiaogang Wang
Published: 2024-05-28 17:47:19

The consistency model (CM) has recently made significant progress in
accelerating the generation of diffusion models. However, its application to
high-resolution, text-conditioned image generation in the latent space (a.k.a.,
LCM) remains unsatisfactory. In this paper, we identify three key flaws in the
current design of LCM. We investigate the reasons behind these limitations and
propose the Phased Consistency Model (PCM), which generalizes the design space
and addresses all identified limitations. Our evaluations demonstrate that PCM
significantly outperforms LCM across 1--16 step generation settings. While PCM
is specifically designed for multi-step refinement, it achieves even superior
or comparable 1-step generation results to previously state-of-the-art
specifically designed 1-step methods. Furthermore, we show that PCM's
methodology is versatile and applicable to video generation, enabling us to
train the state-of-the-art few-step text-to-video generator. More details are
available at https://g-u-n.github.io/projects/pcm/.
LoGAH: Predicting 774-Million-Parameter Transformers using Graph
  HyperNetworks with 1/100 Parameters
Authors: name: Xinyu Zhou, name: Boris Knyazev, name: Alexia Jolicoeur-Martineau, name: Jie Fu
Published: 2024-05-25 15:56:15

A good initialization of deep learning models is essential since it can help
them converge better and faster. However, pretraining large models is
unaffordable for many researchers, which makes a desired prediction for initial
parameters more necessary nowadays. Graph HyperNetworks (GHNs), one approach to
predicting model parameters, have recently shown strong performance in
initializing large vision models. Unfortunately, predicting parameters of very
wide networks relies on copying small chunks of parameters multiple times and
requires an extremely large number of parameters to support full prediction,
which greatly hinders its adoption in practice. To address this limitation, we
propose LoGAH (Low-rank GrAph Hypernetworks), a GHN with a low-rank parameter
decoder that expands to significantly wider networks without requiring as
excessive increase of parameters as in previous attempts. LoGAH allows us to
predict the parameters of 774-million large neural networks in a
memory-efficient manner. We show that vision and language models (i.e., ViT and
GPT-2) initialized with LoGAH achieve better performance than those initialized
randomly or using existing hypernetworks. Furthermore, we show promising
transfer learning results w.r.t. training LoGAH on small datasets and using the
predicted parameters to initialize for larger tasks. We provide the codes in
https://github.com/Blackzxy/LoGAH .
Looking Backward: Streaming Video-to-Video Translation with Feature
  Banks
Authors: name: Feng Liang, name: Akio Kodaira, name: Chenfeng Xu, name: Masayoshi Tomizuka, name: Kurt Keutzer, name: Diana Marculescu
Published: 2024-05-24 17:53:06

This paper introduces StreamV2V, a diffusion model that achieves real-time
streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
methods using batches to process limited frames, we opt to process frames in a
streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
a backward-looking principle that relates the present to the past. This is
realized by maintaining a feature bank, which archives information from past
frames. For incoming frames, StreamV2V extends self-attention to include banked
keys and values and directly fuses similar past features into the output. The
feature bank is continually updated by merging stored and new features, making
it compact but informative. StreamV2V stands out for its adaptability and
efficiency, seamlessly integrating with image diffusion models without
fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
metrics and user studies confirm StreamV2V's exceptional ability to maintain
temporal consistency.
I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion
  Models
Authors: name: Wenqi Ouyang, name: Yi Dong, name: Lei Yang, name: Jianlou Si, name: Xingang Pan
Published: 2024-05-26 11:47:40

The remarkable generative capabilities of diffusion models have motivated
extensive research in both image and video editing. Compared to video editing
which faces additional challenges in the time dimension, image editing has
witnessed the development of more diverse, high-quality approaches and more
capable software like Photoshop. In light of this gap, we introduce a novel and
generic solution that extends the applicability of image editing tools to
videos by propagating edits from a single frame to the entire video using a
pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively
preserves the visual and motion integrity of the source video depending on the
extent of the edits, effectively handling global edits, local edits, and
moderate shape changes, which existing methods cannot fully achieve. At the
core of our method are two main processes: Coarse Motion Extraction to align
basic motion patterns with the original video, and Appearance Refinement for
precise adjustments using fine-grained attention matching. We also incorporate
a skip-interval strategy to mitigate quality degradation from auto-regressive
generation across multiple video clips. Experimental results demonstrate our
framework's superior performance in fine-grained video editing, proving its
capability to produce high-quality, temporally consistent outputs.
Transformers Can Do Arithmetic with the Right Embeddings
Authors: name: Sean McLeish, name: Arpit Bansal, name: Alex Stein, name: Neel Jain, name: John Kirchenbauer, name: Brian R. Bartoldson, name: Bhavya Kailkhura, name: Abhinav Bhatele, name: Jonas Geiping, name: Avi Schwarzschild, name: Tom Goldstein
Published: 2024-05-27 17:49:18

The poor performance of transformers on arithmetic tasks seems to stem in
large part from their inability to keep track of the exact position of each
digit inside of a large span of digits. We mend this problem by adding an
embedding to each digit that encodes its position relative to the start of the
number. In addition to the boost these embeddings provide on their own, we show
that this fix enables architectural modifications such as input injection and
recurrent layers to improve performance even further.
  With positions resolved, we can study the logical extrapolation ability of
transformers. Can they solve arithmetic problems that are larger and more
complex than those in their training data? We find that training on only 20
digit numbers with a single GPU for one day, we can reach state-of-the-art
performance, achieving up to 99% accuracy on 100 digit addition problems.
Finally, we show that these gains in numeracy also unlock improvements on other
multi-step reasoning tasks including sorting and multiplication.
Zamba: A Compact 7B SSM Hybrid Model
Authors: name: Paolo Glorioso, name: Quentin Anthony, name: Yury Tokpanov, name: James Whittington, name: Jonathan Pilault, name: Adam Ibrahim, name: Beren Millidge
Published: 2024-05-26 22:23:02

In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid
model which achieves competitive performance against leading open-weight models
at a comparable scale. Zamba is trained on 1T tokens from openly available
datasets and is the best non-transformer model at this scale. Zamba pioneers a
unique architecture combining a Mamba backbone with a single shared attention
module, thus obtaining the benefits of attention at minimal parameter cost. Due
to its architecture, Zamba is significantly faster at inference than comparable
transformer models and requires substantially less memory for generation of
long sequences. Zamba is pretrained in two phases: the first phase is based on
existing web datasets, while the second one consists of annealing the model
over high-quality instruct and synthetic datasets, and is characterized by a
rapid learning rate decay. We open-source the weights and all checkpoints for
Zamba, through both phase 1 and annealing phases.
Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with
  Dynamic Gaussian Surfels
Authors: name: Yikai Wang, name: Xinzhou Wang, name: Zilong Chen, name: Zhengyi Wang, name: Fuchun Sun, name: Jun Zhu
Published: 2024-05-27 04:43:44

Video generative models are receiving particular attention given their
ability to generate realistic and imaginative frames. Besides, these models are
also observed to exhibit strong 3D consistency, significantly enhancing their
potential to act as world simulators. In this work, we present Vidu4D, a novel
reconstruction model that excels in accurately reconstructing 4D (i.e.,
sequential 3D) representations from single generated videos, addressing
challenges associated with non-rigidity and frame distortion. This capability
is pivotal for creating high-fidelity virtual contents that maintain both
spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic
Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions
to transform Gaussian surfels (surface elements) from a static state to a
dynamically warped state. This transformation enables a precise depiction of
motion and deformation over time. To preserve the structural integrity of
surface-aligned Gaussian surfels, we design the warped-state geometric
regularization based on continuous warping fields for estimating normals.
Additionally, we learn refinements on rotation and scaling parameters of
Gaussian surfels, which greatly alleviates texture flickering during the
warping process and enhances the capture of fine-grained appearance details.
Vidu4D also contains a novel initialization state that provides a proper start
for the warping fields in DGS. Equipping Vidu4D with an existing video
generative model, the overall framework demonstrates high-fidelity text-to-4D
generation in both appearance and geometry.
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding
  Models
Authors: name: Chankyu Lee, name: Rajarshi Roy, name: Mengyao Xu, name: Jonathan Raiman, name: Mohammad Shoeybi, name: Bryan Catanzaro, name: Wei Ping
Published: 2024-05-27 17:59:45

Decoder-only large language model (LLM)-based embedding models are beginning
to outperform BERT or T5-based embedding models in general-purpose text
embedding tasks, including dense vector-based retrieval. In this work, we
introduce the NV-Embed model with a variety of architectural designs and
training procedures to significantly enhance the performance of LLM as a
versatile embedding model, while maintaining its simplicity and
reproducibility. For model architecture, we propose a latent attention layer to
obtain pooled embeddings, which consistently improves retrieval and downstream
task accuracy compared to mean pooling or using the last <EOS> token embedding
from LLMs. To enhance representation learning, we remove the causal attention
mask of LLMs during contrastive training. For model training, we introduce a
two-stage contrastive instruction-tuning method. It first applies contrastive
training with instructions on retrieval datasets, utilizing in-batch negatives
and curated hard negative examples. At stage-2, it blends various non-retrieval
datasets into instruction tuning, which not only enhances non-retrieval task
accuracy but also improves retrieval performance. Combining these techniques,
our NV-Embed model, using only publicly available data, has achieved a
record-high score of 69.32, ranking No. 1 on the Massive Text Embedding
Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval,
reranking, classification, clustering, and semantic textual similarity tasks.
Notably, our model also attains the highest score of 59.36 on 15 retrieval
tasks in the MTEB benchmark (also known as BEIR). We will open-source the model
at: https://huggingface.co/nvidia/NV-Embed-v1.
EM Distillation for One-step Diffusion Models
Authors: name: Sirui Xie, name: Zhisheng Xiao, name: Diederik P Kingma, name: Tingbo Hou, name: Ying Nian Wu, name: Kevin Patrick Murphy, name: Tim Salimans, name: Ben Poole, name: Ruiqi Gao
Published: 2024-05-27 05:55:22

While diffusion models can learn complex distributions, sampling requires a
computationally expensive iterative process. Existing distillation methods
enable efficient sampling, but have notable limitations, such as performance
degradation with very few sampling steps, reliance on training data access, or
mode-seeking optimization that may fail to capture the full distribution. We
propose EM Distillation (EMD), a maximum likelihood-based approach that
distills a diffusion model to a one-step generator model with minimal loss of
perceptual quality. Our approach is derived through the lens of
Expectation-Maximization (EM), where the generator parameters are updated using
samples from the joint distribution of the diffusion teacher prior and inferred
generator latents. We develop a reparametrized sampling scheme and a noise
cancellation technique that together stabilizes the distillation process. We
further reveal an interesting connection of our method with existing methods
that minimize mode-seeking KL. EMD outperforms existing one-step generative
methods in terms of FID scores on ImageNet-64 and ImageNet-128, and compares
favorably with prior work on distilling text-to-image diffusion models.
Part123: Part-aware 3D Reconstruction from a Single-view Image
Authors: name: Anran Liu, name: Cheng Lin, name: Yuan Liu, name: Xiaoxiao Long, name: Zhiyang Dou, name: Hao-Xiang Guo, name: Ping Luo, name: Wenping Wang
Published: 2024-05-27 07:10:21

Recently, the emergence of diffusion models has opened up new opportunities
for single-view reconstruction. However, all the existing methods represent the
target object as a closed mesh devoid of any structural information, thus
neglecting the part-based structure, which is crucial for many downstream
applications, of the reconstructed shape. Moreover, the generated meshes
usually suffer from large noises, unsmooth surfaces, and blurry textures,
making it challenging to obtain satisfactory part segments using 3D
segmentation techniques. In this paper, we present Part123, a novel framework
for part-aware 3D reconstruction from a single-view image. We first use
diffusion models to generate multiview-consistent images from a given image,
and then leverage Segment Anything Model (SAM), which demonstrates powerful
generalization ability on arbitrary objects, to generate multiview segmentation
masks. To effectively incorporate 2D part-based information into 3D
reconstruction and handle inconsistency, we introduce contrastive learning into
a neural rendering framework to learn a part-aware feature space based on the
multiview segmentation masks. A clustering-based algorithm is also developed to
automatically derive 3D part segmentation results from the reconstructed
models. Experiments show that our method can generate 3D models with
high-quality segmented parts on various objects. Compared to existing
unstructured reconstruction methods, the part-aware 3D models from our method
benefit some important applications, including feature-preserving
reconstruction, primitive fitting, and 3D shape editing.
Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
Authors: name: Cristina N. Vasconcelos, name: Abdullah Rashwan Austin Waters, name: Trevor Walker, name: Keyang Xu, name: Jimmy Yan, name: Rui Qian, name: Shixin Luo, name: Zarana Parekh, name: Andrew Bunner, name: Hongliang Fei, name: Roopal Garg, name: Mandy Guo, name: Ivana Kajic, name: Yeqing Li, name: Henna Nandwani, name: Jordi Pont-Tuset, name: Yasumasa Onoe, name: Sarah Rosston, name: Su Wang, name: Wenlei Zhou, name: Kevin Swersky, name: David J. Fleet, name: Jason M. Baldridge, name: Oliver Wang
Published: 2024-05-27 02:12:39

We address the long-standing problem of how to learn effective pixel-based
image diffusion models at scale, introducing a remarkably simple greedy growing
method for stable training of large-scale, high-resolution models. without the
needs for cascaded super-resolution components. The key insight stems from
careful pre-training of core components, namely, those responsible for
text-to-image alignment {\it vs.} high-resolution rendering. We first
demonstrate the benefits of scaling a {\it Shallow UNet}, with no
down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to
improve alignment, object structure, and composition. Building on this core
model, we propose a greedy algorithm that grows the architecture into
high-resolution end-to-end models, while preserving the integrity of the
pre-trained representation, stabilizing training, and reducing the need for
large high-resolution datasets. This enables a single stage model capable of
generating high-resolution images without the need of a super-resolution
cascade. Our key results rely on public datasets and show that we are able to
train non-cascaded models up to 8B parameters with no further regularization
schemes. Vermeer, our full pipeline model trained with internal datasets to
produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4%
human evaluators over SDXL.
An Introduction to Vision-Language Modeling
Authors: name: Florian Bordes, name: Richard Yuanzhe Pang, name: Anurag Ajay, name: Alexander C. Li, name: Adrien Bardes, name: Suzanne Petryk, name: Oscar Mañas, name: Zhiqiu Lin, name: Anas Mahmoud, name: Bargav Jayaraman, name: Mark Ibrahim, name: Melissa Hall, name: Yunyang Xiong, name: Jonathan Lebensold, name: Candace Ross, name: Srihari Jayakumar, name: Chuan Guo, name: Diane Bouchacourt, name: Haider Al-Tahan, name: Karthik Padthe, name: Vasu Sharma, name: Hu Xu, name: Xiaoqing Ellen Tan, name: Megan Richards, name: Samuel Lavoie, name: Pietro Astolfi, name: Reyhane Askari Hemmat, name: Jun Chen, name: Kushal Tirumala, name: Rim Assouel, name: Mazda Moayeri, name: Arjang Talattof, name: Kamalika Chaudhuri, name: Zechun Liu, name: Xilun Chen, name: Quentin Garrido, name: Karen Ullrich, name: Aishwarya Agrawal, name: Kate Saenko, name: Asli Celikyilmaz, name: Vikas Chandra
Published: 2024-05-27 15:01:23

Following the recent popularity of Large Language Models (LLMs), several
attempts have been made to extend them to the visual domain. From having a
visual assistant that could guide us through unfamiliar environments to
generative models that produce images using only a high-level text description,
the vision-language model (VLM) applications will significantly impact our
relationship with technology. However, there are many challenges that need to
be addressed to improve the reliability of those models. While language is
discrete, vision evolves in a much higher dimensional space in which concepts
cannot always be easily discretized. To better understand the mechanics behind
mapping vision to language, we present this introduction to VLMs which we hope
will help anyone who would like to enter the field. First, we introduce what
VLMs are, how they work, and how to train them. Then, we present and discuss
approaches to evaluate VLMs. Although this work primarily focuses on mapping
images to language, we also discuss extending VLMs to videos.
Trans-LoRA: towards data-free Transferable Parameter
  Efficient Finetuning
Authors: name: Runqian Wang, name: Soumya Ghosh, name: David Cox, name: Diego Antognini, name: Aude Oliva, name: Rogerio Feris, name: Leonid Karlinsky
Published: 2024-05-27 15:15:08

Low-rank adapters (LoRA) and their variants are popular parameter-efficient
fine-tuning (PEFT) techniques that closely match full model fine-tune
performance while requiring only a small number of additional parameters. These
additional LoRA parameters are specific to the base model being adapted. When
the base model needs to be deprecated and replaced with a new one, all the
associated LoRA modules need to be re-trained. Such re-training requires access
to the data used to train the LoRA for the original base model. This is
especially problematic for commercial cloud applications where the LoRA modules
and the base models are hosted by service providers who may not be allowed to
host proprietary client task data. To address this challenge, we propose
Trans-LoRA -- a novel method for lossless, nearly data-free transfer
of LoRAs across base models. Our approach relies on synthetic data to transfer
LoRA modules. Using large language models, we design a synthetic data generator
to approximate the data-generating process of the observed task data
subset. Training on the resulting synthetic dataset transfers LoRA modules to
new models. We show the effectiveness of our approach using both LLama and
Gemma model families. Our approach achieves lossless (mostly improved) LoRA
transfer between models within and across different base model families, and
even between different PEFT methods, on a wide variety of tasks.
Human4DiT: Free-view Human Video Generation with 4D Diffusion
  Transformer
Authors: name: Ruizhi Shao, name: Youxin Pang, name: Zerong Zheng, name: Jingxiang Sun, name: Yebin Liu
Published: 2024-05-27 17:53:29

We present a novel approach for generating high-quality, spatio-temporally
coherent human videos from a single image under arbitrary viewpoints. Our
framework combines the strengths of U-Nets for accurate condition injection and
diffusion transformers for capturing global correlations across viewpoints and
time. The core is a cascaded 4D transformer architecture that factorizes
attention across views, time, and spatial dimensions, enabling efficient
modeling of the 4D space. Precise conditioning is achieved by injecting human
identity, camera parameters, and temporal signals into the respective
transformers. To train this model, we curate a multi-dimensional dataset
spanning images, videos, multi-view data and 3D/4D scans, along with a
multi-dimensional training strategy. Our approach overcomes the limitations of
previous methods based on GAN or UNet-based diffusion models, which struggle
with complex motions and viewpoint changes. Through extensive experiments, we
demonstrate our method's ability to synthesize realistic, coherent and
free-view human videos, paving the way for advanced multimedia applications in
areas such as virtual reality and animation. Our project website is
https://human4dit.github.io.