Orionnnn/hub-stats huggingface dataset:
PIXART-δ: Fast and Controllable Image Generation with Latent
  Consistency Models
Authors: name: Junsong Chen, name: Yue Wu, name: Simian Luo, name: Enze Xie, name: Sayak Paul, name: Ping Luo, name: Hang Zhao, name: Zhenguo Li
Published: 2024-01-10 16:27:38

This technical report introduces PIXART-{\delta}, a text-to-image synthesis
framework that integrates the Latent Consistency Model (LCM) and ControlNet
into the advanced PIXART-{\alpha} model. PIXART-{\alpha} is recognized for its
ability to generate high-quality images of 1024px resolution through a
remarkably efficient training process. The integration of LCM in
PIXART-{\delta} significantly accelerates the inference speed, enabling the
production of high-quality images in just 2-4 steps. Notably, PIXART-{\delta}
achieves a breakthrough 0.5 seconds for generating 1024x1024 pixel images,
marking a 7x improvement over the PIXART-{\alpha}. Additionally,
PIXART-{\delta} is designed to be efficiently trainable on 32GB V100 GPUs
within a single day. With its 8-bit inference capability (von Platen et al.,
2023), PIXART-{\delta} can synthesize 1024px images within 8GB GPU memory
constraints, greatly enhancing its usability and accessibility. Furthermore,
incorporating a ControlNet-like module enables fine-grained control over
text-to-image diffusion models. We introduce a novel ControlNet-Transformer
architecture, specifically tailored for Transformers, achieving explicit
controllability alongside high-quality image generation. As a state-of-the-art,
open-source image generation model, PIXART-{\delta} offers a promising
alternative to the Stable Diffusion family of models, contributing
significantly to text-to-image synthesis.
URHand: Universal Relightable Hands
Authors: name: Zhaoxi Chen, name: Gyeongsik Moon, name: Kaiwen Guo, name: Chen Cao, name: Stanislav Pidhorskyi, name: Tomas Simon, name: Rohan Joshi, name: Yuan Dong, name: Yichen Xu, name: Bernardo Pires, name: He Wen, name: Lucas Evans, name: Bo Peng, name: Julia Buffalini, name: Autumn Trimble, name: Kevyn McPhail, name: Melissa Schoeller, name: Shoou-I Yu, name: Javier Romero, name: Michael Zollhöfer, name: Yaser Sheikh, name: Ziwei Liu, name: Shunsuke Saito
Published: 2024-01-10 18:59:51

Existing photorealistic relightable hand models require extensive
identity-specific observations in different views, poses, and illuminations,
and face challenges in generalizing to natural illuminations and novel
identities. To bridge this gap, we present URHand, the first universal
relightable hand model that generalizes across viewpoints, poses,
illuminations, and identities. Our model allows few-shot personalization using
images captured with a mobile phone, and is ready to be photorealistically
rendered under novel illuminations. To simplify the personalization process
while retaining photorealism, we build a powerful universal relightable prior
based on neural relighting from multi-view images of hands captured in a light
stage with hundreds of identities. The key challenge is scaling the
cross-identity training while maintaining personalized fidelity and sharp
details without compromising generalization under natural illuminations. To
this end, we propose a spatially varying linear lighting model as the neural
renderer that takes physics-inspired shading as input feature. By removing
non-linear activations and bias, our specifically designed lighting model
explicitly keeps the linearity of light transport. This enables single-stage
training from light-stage data while generalizing to real-time rendering under
arbitrary continuous illuminations across diverse identities. In addition, we
introduce the joint learning of a physically based model and our neural
relighting model, which further improves fidelity and generalization. Extensive
experiments show that our approach achieves superior performance over existing
methods in terms of both quality and generalizability. We also demonstrate
quick personalization of URHand from a short phone scan of an unseen identity.
The Impact of Reasoning Step Length on Large Language Models
Authors: name: Mingyu Jin, name: Qinkai Yu, name: Dong shu, name: Haiyan Zhao, name: Wenyue Hua, name: Yanda Meng, name: Yongfeng Zhang, name: Mengnan Du
Published: 2024-01-10 04:37:38

Chain of Thought (CoT) is significant in improving the reasoning abilities of
large language models (LLMs). However, the correlation between the
effectiveness of CoT and the length of reasoning steps in prompts remains
largely unknown. To shed light on this, we have conducted several empirical
experiments to explore the relations. Specifically, we design experiments that
expand and compress the rationale reasoning steps within CoT demonstrations,
while keeping all other factors constant. We have the following key findings.
First, the results indicate that lengthening the reasoning steps in prompts,
even without adding new information into the prompt, considerably enhances
LLMs' reasoning abilities across multiple datasets. Alternatively, shortening
the reasoning steps, even while preserving the key information, significantly
diminishes the reasoning abilities of models. This finding highlights the
importance of the number of steps in CoT prompts and provides practical
guidance to make better use of LLMs' potential in complex problem-solving
scenarios. Second, we also investigated the relationship between the
performance of CoT and the rationales used in demonstrations. Surprisingly, the
result shows that even incorrect rationales can yield favorable outcomes if
they maintain the requisite length of inference. Third, we observed that the
advantages of increasing reasoning steps are task-dependent: simpler tasks
require fewer steps, whereas complex tasks gain significantly from longer
inference sequences.
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
Authors: name: Dennis Ulmer, name: Elman Mansimov, name: Kaixiang Lin, name: Justin Sun, name: Xibin Gao, name: Yi Zhang
Published: 2024-01-10 09:49:10

Large language models (LLMs) are powerful dialogue agents, but specializing
them towards fulfilling a specific function can be challenging. Instructing
tuning, i.e. tuning models on instruction and sample responses generated by
humans (Ouyang et al., 2022), has proven as an effective method to do so, yet
requires a number of data samples that a) might not be available or b) costly
to generate. Furthermore, this cost increases when the goal is to make the LLM
follow a specific workflow within a dialogue instead of single instructions.
Inspired by the self-play technique in reinforcement learning and the use of
LLMs to simulate human agents, we propose a more effective method for data
collection through LLMs engaging in a conversation in various roles. This
approach generates a training data via "self-talk" of LLMs that can be refined
and utilized for supervised fine-tuning. We introduce an automated way to
measure the (partial) success of a dialogue. This metric is used to filter the
generated conversational data that is fed back in LLM for training. Based on
our automated and human evaluations of conversation quality, we demonstrate
that such self-talk data improves results. In addition, we examine the various
characteristics that showcase the quality of generated dialogues and how they
can be connected to their potential utility as training data.
Jump Cut Smoothing for Talking Heads
Authors: name: Xiaojuan Wang, name: Taesung Park, name: Yang Zhou, name: Eli Shechtman, name: Richard Zhang
Published: 2024-01-09 18:44:48

A jump cut offers an abrupt, sometimes unwanted change in the viewing
experience. We present a novel framework for smoothing these jump cuts, in the
context of talking head videos. We leverage the appearance of the subject from
the other source frames in the video, fusing it with a mid-level representation
driven by DensePose keypoints and face landmarks. To achieve motion, we
interpolate the keypoints and landmarks between the end frames around the cut.
We then use an image translation network from the keypoints and source frames,
to synthesize pixels. Because keypoints can contain errors, we propose a
cross-modal attention scheme to select and pick the most appropriate source
amongst multiple options for each key point. By leveraging this mid-level
representation, our method can achieve stronger results than a strong video
interpolation baseline. We demonstrate our method on various jump cuts in the
talking head videos, such as cutting filler words, pauses, and even random
cuts. Our experiments show that we can achieve seamless transitions, even in
the challenging cases where the talking head rotates or moves drastically in
the jump cut.
FADI-AEC: Fast Score Based Diffusion Model Guided by Far-end Signal for
  Acoustic Echo Cancellation
Authors: name: Yang Liu, name: Li Wan, name: Yun Li, name: Yiteng Huang, name: Ming Sun, name: James Luan, name: Yangyang Shi, name: Xin Lei
Published: 2024-01-08 23:38:04

Despite the potential of diffusion models in speech enhancement, their
deployment in Acoustic Echo Cancellation (AEC) has been restricted. In this
paper, we propose DI-AEC, pioneering a diffusion-based stochastic regeneration
approach dedicated to AEC. Further, we propose FADI-AEC, fast score-based
diffusion AEC framework to save computational demands, making it favorable for
edge devices. It stands out by running the score model once per frame,
achieving a significant surge in processing efficiency. Apart from that, we
introduce a novel noise generation technique where far-end signals are
utilized, incorporating both far-end and near-end signals to refine the score
model's accuracy. We test our proposed method on the ICASSP2023 Microsoft deep
echo cancellation challenge evaluation dataset, where our method outperforms
some of the end-to-end methods and other diffusion based echo cancellation
methods.
Masked Audio Generation using a Single Non-Autoregressive Transformer
Authors: name: Alon Ziv, name: Itai Gat, name: Gael Le Lan, name: Tal Remez, name: Felix Kreuk, name: Alexandre Défossez, name: Jade Copet, name: Gabriel Synnaeve, name: Yossi Adi
Published: 2024-01-09 14:29:39

We introduce MAGNeT, a masked generative sequence modeling method that
operates directly over several streams of audio tokens. Unlike prior work,
MAGNeT is comprised of a single-stage, non-autoregressive transformer. During
training, we predict spans of masked tokens obtained from a masking scheduler,
while during inference we gradually construct the output sequence using several
decoding steps. To further enhance the quality of the generated audio, we
introduce a novel rescoring method in which, we leverage an external
pre-trained model to rescore and rank predictions from MAGNeT, which will be
then used for later decoding steps. Lastly, we explore a hybrid version of
MAGNeT, in which we fuse between autoregressive and non-autoregressive models
to generate the first few seconds in an autoregressive manner while the rest of
the sequence is being decoded in parallel. We demonstrate the efficiency of
MAGNeT for the task of text-to-music and text-to-audio generation and conduct
an extensive empirical evaluation, considering both objective metrics and human
studies. The proposed approach is comparable to the evaluated baselines, while
being significantly faster (x7 faster than the autoregressive baseline).
Through ablation studies and analysis, we shed light on the importance of each
of the components comprising MAGNeT, together with pointing to the trade-offs
between autoregressive and non-autoregressive modeling, considering latency,
throughput, and generation quality. Samples are available on our demo page
https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual
  Concept Understanding
Authors: name: Yatong Bai, name: Utsav Garg, name: Apaar Shanker, name: Haoming Zhang, name: Samyak Parajuli, name: Erhan Bas, name: Isidora Filipovic, name: Amelia N. Chu, name: Eugenia D Fomitcheva, name: Elliot Branson, name: Aerin Kim, name: Somayeh Sojoudi, name: Kyunghyun Cho
Published: 2024-01-09 14:24:29

Vision and vision-language applications of neural networks, such as image
classification and captioning, rely on large-scale annotated datasets that
require non-trivial data-collecting processes. This time-consuming endeavor
hinders the emergence of large-scale datasets, limiting researchers and
practitioners to a small number of choices. Therefore, we seek more efficient
ways to collect and annotate images. Previous initiatives have gathered
captions from HTML alt-texts and crawled social media postings, but these data
sources suffer from noise, sparsity, or subjectivity. For this reason, we turn
to commercial shopping websites whose data meet three criteria: cleanliness,
informativeness, and fluency. We introduce the Let's Go Shopping (LGS) dataset,
a large-scale public dataset with 15 million image-caption pairs from publicly
available e-commerce websites. When compared with existing general-domain
datasets, the LGS images focus on the foreground object and have less complex
backgrounds. Our experiments on LGS show that the classifiers trained on
existing benchmark datasets do not readily generalize to e-commerce data, while
specific self-supervised visual feature extractors can better generalize.
Furthermore, LGS's high-quality e-commerce-focused images and bimodal nature
make it advantageous for vision-language bi-modal tasks: LGS enables
image-captioning models to generate richer captions and helps text-to-image
generation models achieve e-commerce style transfer.
Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering
  with Multi-Granularity Answers
Authors: name: Gal Yona, name: Roee Aharoni, name: Mor Geva
Published: 2024-01-09 17:44:36

Factual questions typically can be answered correctly at different levels of
granularity. For example, both ``August 4, 1961'' and ``1961'' are correct
answers to the question ``When was Barack Obama born?''. Standard question
answering (QA) evaluation protocols, however, do not explicitly take this into
account and compare a predicted answer against answers of a single granularity
level. In this work, we propose GRANOLA QA, a novel evaluation setting where a
predicted answer is evaluated in terms of accuracy and informativeness against
a set of multi-granularity answers. We present a simple methodology for
enriching existing datasets with multi-granularity answers, and create
GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We
evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm,
called Decoding with Response Aggregation (DRAG), that is geared towards
aligning the response granularity with the model's uncertainty. Our experiments
show that large language models with standard decoding tend to generate
specific answers, which are often incorrect. In contrast, when evaluated on
multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy
on average, which further increases for rare entities. Overall, this reveals
that standard evaluation and decoding schemes may significantly underestimate
the knowledge encapsulated in LMs.
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence
  Lengths in Large Language Models
Authors: name: Zhen Qin, name: Weigao Sun, name: Dong Li, name: Xuyang Shen, name: Weixuan Sun, name: Yiran Zhong
Published: 2024-01-09 16:27:28

Linear attention is an efficient attention mechanism that has recently
emerged as a promising alternative to conventional softmax attention. With its
ability to process tokens in linear computational complexities, linear
attention, in theory, can handle sequences of unlimited length without
sacrificing speed, i.e., maintaining a constant training speed for various
sequence lengths with a fixed memory consumption. However, due to the issue
with cumulative summation (cumsum), current linear attention algorithms cannot
demonstrate their theoretical advantage in a causal setting. In this paper, we
present Lightning Attention-2, the first linear attention implementation that
enables linear attention to realize its theoretical computational benefits. To
achieve this, we leverage the thought of tiling, separately handling the
intra-block and inter-block components in linear attention calculation.
Specifically, we utilize the conventional attention computation mechanism for
the intra-blocks and apply linear attention kernel tricks for the inter-blocks.
A tiling technique is adopted through both forward and backward procedures to
take full advantage of the GPU hardware. We implement our algorithm in Triton
to make it IO-aware and hardware-friendly. Various experiments are conducted on
different model sizes and sequence lengths. Lightning Attention-2 retains
consistent training and inference speed regardless of input sequence length and
is significantly faster than other attention mechanisms. The source code is
available at https://github.com/OpenNLPLab/lightning-attention.
Chain-of-Table: Evolving Tables in the Reasoning Chain for Table
  Understanding
Authors: name: Zilong Wang, name: Hao Zhang, name: Chun-Liang Li, name: Julian Martin Eisenschlos, name: Vincent Perot, name: Zifeng Wang, name: Lesly Miculicich, name: Yasuhisa Fujii, name: Jingbo Shang, name: Chen-Yu Lee, name: Tomas Pfister
Published: 2024-01-09 07:46:26

Table-based reasoning with large language models (LLMs) is a promising
direction to tackle many table understanding tasks, such as table-based
question answering and fact verification. Compared with generic reasoning,
table-based reasoning requires the extraction of underlying semantics from both
free-form questions and semi-structured tabular data. Chain-of-Thought and its
similar approaches incorporate the reasoning chain in the form of textual
context, but it is still an open question how to effectively leverage tabular
data in the reasoning chain. We propose the Chain-of-Table framework, where
tabular data is explicitly used in the reasoning chain as a proxy for
intermediate thoughts. Specifically, we guide LLMs using in-context learning to
iteratively generate operations and update the table to represent a tabular
reasoning chain. LLMs can therefore dynamically plan the next operation based
on the results of the previous ones. This continuous evolution of the table
forms a chain, showing the reasoning process for a given tabular problem. The
chain carries structured information of the intermediate results, enabling more
accurate and reliable predictions. Chain-of-Table achieves new state-of-the-art
performance on WikiTQ, FeTaQA, and TabFact benchmarks across multiple LLM
choices.
MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation
Authors: name: Weimin Wang, name: Jiawei Liu, name: Zhijie Lin, name: Jiangqiao Yan, name: Shuo Chen, name: Chetwin Low, name: Tuyen Hoang, name: Jie Wu, name: Jun Hao Liew, name: Hanshu Yan, name: Daquan Zhou, name: Jiashi Feng
Published: 2024-01-09 10:12:52

The growing demand for high-fidelity video generation from textual
descriptions has catalyzed significant research in this field. In this work, we
introduce MagicVideo-V2 that integrates the text-to-image model, video motion
generator, reference image embedding module and frame interpolation module into
an end-to-end video generation pipeline. Benefiting from these architecture
designs, MagicVideo-V2 can generate an aesthetically pleasing, high-resolution
video with remarkable fidelity and smoothness. It demonstrates superior
performance over leading Text-to-Video systems such as Runway, Pika 1.0, Morph,
Moon Valley and Stable Video Diffusion model via user evaluation at large
scale.
AGG: Amortized Generative 3D Gaussians for Single Image to 3D
Authors: name: Dejia Xu, name: Ye Yuan, name: Morteza Mardani, name: Sifei Liu, name: Jiaming Song, name: Zhangyang Wang, name: Arash Vahdat
Published: 2024-01-08 18:56:33

Given the growing need for automatic 3D content creation pipelines, various
3D representations have been studied to generate 3D objects from a single
image. Due to its superior rendering efficiency, 3D Gaussian splatting-based
models have recently excelled in both 3D reconstruction and generation. 3D
Gaussian splatting approaches for image to 3D generation are often
optimization-based, requiring many computationally expensive score-distillation
steps. To overcome these challenges, we introduce an Amortized Generative 3D
Gaussian framework (AGG) that instantly produces 3D Gaussians from a single
image, eliminating the need for per-instance optimization. Utilizing an
intermediate hybrid representation, AGG decomposes the generation of 3D
Gaussian locations and other appearance attributes for joint optimization.
Moreover, we propose a cascaded pipeline that first generates a coarse
representation of the 3D data and later upsamples it with a 3D Gaussian
super-resolution module. Our method is evaluated against existing
optimization-based 3D Gaussian frameworks and sampling-based pipelines
utilizing other 3D representations, where AGG showcases competitive generation
abilities both qualitatively and quantitatively while being several orders of
magnitude faster. Project page: https://ir1d.github.io/AGG/
MoE-Mamba: Efficient Selective State Space Models with Mixture of
  Experts
Authors: name: Maciej Pióro, name: Kamil Ciebiera, name: Krystian Król, name: Jan Ludziejewski, name: Sebastian Jaszczur
Published: 2024-01-08 18:35:07

State Space Models (SSMs) have become serious contenders in the field of
sequential modeling, challenging the dominance of Transformers. At the same
time, Mixture of Experts (MoE) has significantly improved Transformer-based
LLMs, including recent state-of-the-art open-source models. We propose that to
unlock the potential of SSMs for scaling, they should be combined with MoE. We
showcase this on Mamba, a recent SSM-based model that achieves remarkable,
Transformer-like performance. Our model, MoE-Mamba, outperforms both Mamba and
Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba
in 2.2x less training steps while preserving the inference performance gains of
Mamba against the Transformer.
DiarizationLM: Speaker Diarization Post-Processing with Large Language
  Models
Authors: name: Quan Wang, name: Yiling Huang, name: Guanlong Zhao, name: Evan Clark, name: Wei Xia, name: Hank Liao
Published: 2024-01-07 14:54:57

In this paper, we introduce DiarizationLM, a framework to leverage large
language models (LLM) to post-process the outputs from a speaker diarization
system. Various goals can be achieved with the proposed framework, such as
improving the readability of the diarized transcript, or reducing the word
diarization error rate (WDER). In this framework, the outputs of the automatic
speech recognition (ASR) and speaker diarization systems are represented as a
compact textual format, which is included in the prompt to an optionally
finetuned LLM. The outputs of the LLM can be used as the refined diarization
results with the desired enhancement. As a post-processing step, this framework
can be easily applied to any off-the-shelf ASR and speaker diarization systems
without retraining existing components. Our experiments show that a finetuned
PaLM 2-S model can reduce the WDER by rel. 25.9% on the Fisher telephone
conversation dataset, and rel. 31% on the Callhome English dataset.