Orionnnn/hub-stats huggingface dataset:
TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
  Sequence Processing
Authors: name: Aleksandar Terzic, name: Michael Hersche, name: Geethan Karunaratne, name: Luca Benini, name: Abu Sebastian, name: Abbas Rahimi
Published: 2023-12-09 16:12:25

MEGA is a recent transformer-based architecture, which utilizes a linear
recurrent operator whose parallel computation, based on the FFT, scales as
O(LlogL), with L being the sequence length. We build upon their approach by
replacing the linear recurrence with a special temporal convolutional network
which permits larger receptive field size with shallower networks, and reduces
the computational complexity to O(L). The resulting model is called TCNCA, a
Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
outperforms MEGA, reaching a lower loss with 1.37times/1.24times faster
forward/backward pass during training. The dilated convolutions used in TCNCA
are consistently and significantly faster operations than the FFT-based
parallelized recurrence in GPUs, making them a scalable candidate for handling
very large sequence lengths: they are up to 7.07times/2.86times faster in
the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
achieves, on average, 1.28times speed-up during inference with similar
accuracy to what MEGA achieves. On associative recall, we find that even a
simplified version of TCNCA, without excessive multiplicative and additive
interactions, remains superior or competitive to MEGA on a range of sequence
lengths and vocabulary sizes.
Using Captum to Explain Generative Language Models
Authors: name: Vivek Miglani, name: Aobo Yang, name: Aram H. Markosyan, name: Diego Garcia-Olano, name: Narine Kokhlikyan
Published: 2023-12-09 07:35:24

Captum is a comprehensive library for model explainability in PyTorch,
offering a range of methods from the interpretability literature to enhance
users' understanding of PyTorch models. In this paper, we introduce new
features in Captum that are specifically designed to analyze the behavior of
generative language models. We provide an overview of the available
functionalities and example applications of their potential for understanding
learned associations within generative language models.
Unlocking Anticipatory Text Generation: A Constrained Approach for
  Faithful Decoding with Large Language Models
Authors: name: Lifu Tu, name: Semih Yavuz, name: Jin Qu, name: Jiacheng Xu, name: Rui Meng, name: Caiming Xiong, name: Yingbo Zhou
Published: 2023-12-11 06:35:33

Large Language Models (LLMs) have demonstrated a powerful ability for text
generation. However, achieving optimal results with a given prompt or
instruction can be challenging, especially for billion-sized models.
Additionally, undesired behaviors such as toxicity or hallucinations can
manifest. While much larger models (e.g., ChatGPT) may demonstrate strength in
mitigating these issues, there is still no guarantee of complete prevention. In
this work, we propose formalizing text generation as a future-constrained
generation problem to minimize undesirable behaviors and enforce faithfulness
to instructions. The estimation of future constraint satisfaction, accomplished
using LLMs, guides the text generation process. Our extensive experiments
demonstrate the effectiveness of the proposed approach across three distinct
text generation tasks: keyword-constrained generation (Lin et al., 2020),
toxicity reduction (Gehman et al., 2020), and factual correctness in
question-answering (Gao et al., 2023).
From Text to Motion: Grounding GPT-4 in a Humanoid Robot "Alter3"
Authors: name: Takahide Yoshida, name: Atsushi Masumori, name: Takashi Ikegami
Published: 2023-12-11 17:57:11

We report the development of Alter3, a humanoid robot capable of generating
spontaneous motion using a Large Language Model (LLM), specifically GPT-4. This
achievement was realized by integrating GPT-4 into our proprietary android,
Alter3, thereby effectively grounding the LLM with Alter's bodily movement.
Typically, low-level robot control is hardware-dependent and falls outside the
scope of LLM corpora, presenting challenges for direct LLM-based robot control.
However, in the case of humanoid robots like Alter3, direct control is feasible
by mapping the linguistic expressions of human actions onto the robot's body
through program code. Remarkably, this approach enables Alter3 to adopt various
poses, such as a 'selfie' stance or 'pretending to be a ghost,' and generate
sequences of actions over time without explicit programming for each body part.
This demonstrates the robot's zero-shot learning capabilities. Additionally,
verbal feedback can adjust poses, obviating the need for fine-tuning. A video
of Alter3's generated motions is available at
https://tnoinkwms.github.io/ALTER-LLM/
Context Tuning for Retrieval Augmented Generation
Authors: name: Raviteja Anantha, name: Tharun Bethi, name: Danil Vodianik, name: Srinivas Chappidi
Published: 2023-12-09 23:33:16

Large language models (LLMs) have the remarkable ability to solve new tasks
with just a few examples, but they need access to the right tools. Retrieval
Augmented Generation (RAG) addresses this problem by retrieving a list of
relevant tools for a given task. However, RAG's tool retrieval step requires
all the required information to be explicitly present in the query. This is a
limitation, as semantic search, the widely adopted tool retrieval method, can
fail when the query is incomplete or lacks context. To address this limitation,
we propose Context Tuning for RAG, which employs a smart context retrieval
system to fetch relevant information that improves both tool retrieval and plan
generation. Our lightweight context retrieval model uses numerical,
categorical, and habitual usage signals to retrieve and rank context items. Our
empirical results demonstrate that context tuning significantly enhances
semantic search, achieving a 3.5-fold and 1.5-fold improvement in Recall@K for
context retrieval and tool retrieval tasks respectively, and resulting in an
11.6% increase in LLM-based planner accuracy. Additionally, we show that our
proposed lightweight model using Reciprocal Rank Fusion (RRF) with LambdaMART
outperforms GPT-4 based retrieval. Moreover, we observe context augmentation at
plan generation, even after tool retrieval, reduces hallucination.
Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D
  priors
Authors: name: Lihe Ding, name: Shaocong Dong, name: Zhanpeng Huang, name: Zibin Wang, name: Yiyuan Zhang, name: Kaixiong Gong, name: Dan Xu, name: Tianfan Xue
Published: 2023-12-07 10:00:04

Most 3D generation research focuses on up-projecting 2D foundation models
into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS)
loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these
methods often lead to geometric anomalies and multi-view inconsistency.
Recently, researchers have attempted to improve the genuineness of 3D objects
by directly training on 3D datasets, albeit at the cost of low-quality texture
generation due to the limited texture diversity in 3D datasets. To harness the
advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a
unified framework that incorporates both a 3D and a 2D diffusion process, to
preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a
simple combination may yield inconsistent generation results, we further bridge
them with novel bidirectional guidance. In addition, our method can be used as
an initialization of optimization-based models to further improve the quality
of 3D model and efficiency of optimization, reducing the generation process
from 3.4 hours to 20 minutes. Experimental results have shown that our model
achieves high-quality, diverse, and scalable 3D generation. Project website:
https://bidiff.github.io/.
MVDD: Multi-View Depth Diffusion Models
Authors: name: Zhen Wang, name: Qiangeng Xu, name: Feitong Tan, name: Menglei Chai, name: Shichen Liu, name: Rohit Pandey, name: Sean Fanello, name: Achuta Kadambi, name: Yinda Zhang
Published: 2023-12-08 07:16:09

Denoising diffusion models have demonstrated outstanding results in 2D image
generation, yet it remains a challenge to replicate its success in 3D shape
generation. In this paper, we propose leveraging multi-view depth, which
represents complex 3D shapes in a 2D data format that is easy to denoise. We
pair this representation with a diffusion model, MVDD, that is capable of
generating high-quality dense point clouds with 20K+ points with fine-grained
details. To enforce 3D consistency in multi-view depth, we introduce an
epipolar line segment attention that conditions the denoising step for a view
on its neighboring views. Additionally, a depth fusion module is incorporated
into diffusion steps to further ensure the alignment of depth maps. When
augmented with surface reconstruction, MVDD can also produce high-quality 3D
meshes. Furthermore, MVDD stands out in other tasks such as depth completion,
and can serve as a 3D prior, significantly boosting many downstream tasks, such
as GAN inversion. State-of-the-art results from extensive experiments
demonstrate MVDD's excellent ability in 3D shape generation, depth completion,
and its potential as a 3D prior for downstream tasks.
DreaMoving: A Human Dance Video Generation Framework based on Diffusion
  Models
Authors: name: Mengyang Feng, name: Jinlin Liu, name: Kai Yu, name: Yuan Yao, name: Zheng Hui, name: Xiefan Guo, name: Xianhui Lin, name: Haolan Xue, name: Chen Shi, name: Xiaowen Li, name: Aojie Li, name: Miaomiao Cui, name: Peiran Ren, name: Xuansong Xie
Published: 2023-12-08 15:37:17

In this paper, we present DreaMoving, a diffusion-based controllable video
generation framework to produce high-quality customized human dance videos.
Specifically, given target identity and posture sequences, DreaMoving can
generate a video of the target identity dancing anywhere driven by the posture
sequences. To this end, we propose a Video ControlNet for motion-controlling
and a Content Guider for identity preserving. The proposed model is easy to use
and can be adapted to most stylized diffusion models to generate diverse
results. The project page is available at
https://dreamoving.github.io/dreamoving.
Customizing Motion in Text-to-Video Diffusion Models
Authors: name: Joanna Materzynska, name: Josef Sivic, name: Eli Shechtman, name: Antonio Torralba, name: Richard Zhang, name: Bryan Russell
Published: 2023-12-07 18:59:03

We introduce an approach for augmenting text-to-video generation models with
customized motions, extending their capabilities beyond the motions depicted in
the original training data. By leveraging a few video samples demonstrating
specific movements as input, our method learns and generalizes the input motion
patterns for diverse, text-specified scenarios. Our contributions are
threefold. First, to achieve our results, we finetune an existing text-to-video
model to learn a novel mapping between the depicted motion in the input
examples to a new unique token. To avoid overfitting to the new custom motion,
we introduce an approach for regularization over videos. Second, by leveraging
the motion priors in a pretrained model, our method can produce novel videos
featuring multiple people doing the custom motion, and can invoke the motion in
combination with other motions. Furthermore, our approach extends to the
multimodal customization of motion and appearance of individualized subjects,
enabling the generation of videos featuring unique characters and distinct
motions. Third, to validate our method, we introduce an approach for
quantitatively evaluating the learned custom motion and perform a systematic
ablation study. We show that our method significantly outperforms prior
appearance-based customization approaches when extended to the motion
customization task.
PathFinder: Guided Search over Multi-Step Reasoning Paths
Authors: name: Olga Golovneva, name: Sean O'Brien, name: Ramakanth Pasunuru, name: Tianlu Wang, name: Luke Zettlemoyer, name: Maryam Fazel-Zarandi, name: Asli Celikyilmaz
Published: 2023-12-08 17:05:47

With recent advancements in large language models, methods like
chain-of-thought prompting to elicit reasoning chains have been shown to
improve results on reasoning tasks. However, tasks that require multiple steps
of reasoning still pose significant challenges to state-of-the-art models.
Drawing inspiration from the beam search algorithm, we propose PathFinder, a
tree-search-based reasoning path generation approach. It enhances diverse
branching and multi-hop reasoning through the integration of dynamic decoding,
enabled by varying sampling methods and parameters. Using constrained
reasoning, PathFinder integrates novel quality constraints, pruning, and
exploration methods to enhance the efficiency and the quality of generation.
Moreover, it includes scoring and ranking features to improve candidate
selection. Our approach outperforms competitive baselines on three complex
arithmetic and commonsense reasoning tasks by 6% on average. Our model
generalizes well to longer, unseen reasoning chains, reflecting similar
complexities to beam search with large branching factors.
SparQ Attention: Bandwidth-Efficient LLM Inference
Authors: name: Luka Ribar, name: Ivan Chelombiev, name: Luke Hudlass-Galley, name: Charlie Blake, name: Carlo Luschi, name: Douglas Orr
Published: 2023-12-08 11:47:35

Generative large language models (LLMs) have opened up numerous novel
possibilities, but due to their significant computational requirements their
ubiquitous use remains challenging. Some of the most useful applications
require processing large numbers of samples at a time and using long contexts,
both significantly increasing the memory communication load of the models. We
introduce SparQ Attention, a technique for increasing the inference throughput
of LLMs by reducing the memory bandwidth requirements within the attention
blocks through selective fetching of the cached history. Our proposed technique
can be applied directly to off-the-shelf LLMs during inference, without
requiring any modification to the pre-training setup or additional fine-tuning.
We show how SparQ Attention can decrease the attention memory bandwidth
requirements up to eight times without any loss in accuracy by evaluating Llama
2 and Pythia models on a wide range of downstream tasks.
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language
  Models with 3D Parallelism
Authors: name: Yanxi Chen, name: Xuchen Pan, name: Yaliang Li, name: Bolin Ding, name: Jingren Zhou
Published: 2023-12-08 09:31:50

We present EE-LLM, a framework for large-scale training and inference of
early-exit large language models (LLMs). While recent works have shown
preliminary evidence for the efficacy of early exiting in accelerating LLM
inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs
by supporting their training and inference with massive 3D parallelism. Built
upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and
performance optimizations tailored to early exiting, including a lightweight
method that facilitates backpropagation for the early-exit training objective
with pipeline parallelism, techniques of leveraging idle resources in the
original pipeline schedule for computation related to early-exit layers, and
two approaches of early-exit inference that are compatible with KV caching for
autoregressive generation. Our analytical and empirical study shows that EE-LLM
achieves great training efficiency with negligible computational overhead
compared to standard LLM training, as well as outstanding inference speedup
without compromising output quality. To facilitate further research and
adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Authors: name: Jae Sung Park, name: Jack Hessel, name: Khyathi Raghavi Chandu, name: Paul Pu Liang, name: Ximing Lu, name: Peter West, name: Youngjae Yu, name: Qiuyuan Huang, name: Jianfeng Gao, name: Ali Farhadi, name: Yejin Choi
Published: 2023-12-08 05:23:50

Instruction following vision-language (VL) models offer a flexible interface
that supports a broad range of multimodal tasks in a zero-shot fashion.
However, interfaces that operate on full images do not directly enable the user
to "point to" and access specific regions within images. This capability is
important not only to support reference-grounded VL benchmarks, but also, for
practical applications that require precise within-image reasoning. We build
Localized Visual Commonsense models, which allow users to specify (multiple)
regions as input. We train our model by sampling localized commonsense
knowledge from a large language model (LLM): specifically, we prompt an LLM to
collect commonsense knowledge given a global literal image description and a
local literal region description automatically generated by a set of VL models.
With a separately trained critic model that selects high-quality examples, we
find that training on the localized commonsense corpus can successfully distill
existing VL models to support a reference-as-input interface. Empirical results
and human evaluations in a zero-shot setup demonstrate that our distillation
method results in more precise VL models of reasoning compared to a baseline of
passing a generated referring expression to an LLM.
Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
Authors: name: Manish Bhatt, name: Sahana Chennabasappa, name: Cyrus Nikolaidis, name: Shengye Wan, name: Ivan Evtimov, name: Dominik Gabi, name: Daniel Song, name: Faizan Ahmad, name: Cornelius Aschermann, name: Lorenzo Fontana, name: Sasha Frolov, name: Ravi Prakash Giri, name: Dhaval Kapil, name: Yiannis Kozyrakis, name: David LeBlanc, name: James Milazzo, name: Aleksandar Straumann, name: Gabriel Synnaeve, name: Varun Vontimitta, name: Spencer Whitman, name: Joshua Saxe
Published: 2023-12-07 22:07:54

This paper presents CyberSecEval, a comprehensive benchmark developed to help
bolster the cybersecurity of Large Language Models (LLMs) employed as coding
assistants. As what we believe to be the most extensive unified cybersecurity
safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs
in two crucial security domains: their propensity to generate insecure code and
their level of compliance when asked to assist in cyberattacks. Through a case
study involving seven models from the Llama 2, Code Llama, and OpenAI GPT large
language model families, CyberSecEval effectively pinpointed key cybersecurity
risks. More importantly, it offered practical insights for refining these
models. A significant observation from the study was the tendency of more
advanced models to suggest insecure code, highlighting the critical need for
integrating security considerations in the development of sophisticated LLMs.
CyberSecEval, with its automated test case generation and evaluation pipeline
covers a broad scope and equips LLM designers and researchers with a tool to
broadly measure and enhance the cybersecurity safety properties of LLMs,
contributing to the development of more secure AI systems.
ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
Authors: name: Maitreya Patel, name: Changhoon Kim, name: Sheng Cheng, name: Chitta Baral, name: Yezhou Yang
Published: 2023-12-07 19:32:39

Text-to-image (T2I) diffusion models, notably the unCLIP models (e.g.,
DALL-E-2), achieve state-of-the-art (SOTA) performance on various compositional
T2I benchmarks, at the cost of significant computational resources. The unCLIP
stack comprises T2I prior and diffusion image decoder. The T2I prior model
alone adds a billion parameters compared to the Latent Diffusion Models, which
increases the computational and high-quality data requirements. We introduce
ECLIPSE, a novel contrastive learning method that is both parameter and
data-efficient. ECLIPSE leverages pre-trained vision-language models (e.g.,
CLIP) to distill the knowledge into the prior model. We demonstrate that the
ECLIPSE trained prior, with only 3.3% of the parameters and trained on a mere
2.8% of the data, surpasses the baseline T2I priors with an average of 71.6%
preference score under resource-limited setting. It also attains performance on
par with SOTA big models, achieving an average of 63.36% preference score in
terms of the ability to follow the text compositions. Extensive experiments on
two unCLIP diffusion image decoders, Karlo and Kandinsky, affirm that ECLIPSE
priors consistently deliver high performance while significantly reducing
resource dependency.
NeRFiller: Completing Scenes via Generative 3D Inpainting
Authors: name: Ethan Weber, name: Aleksander Hołyński, name: Varun Jampani, name: Saurabh Saxena, name: Noah Snavely, name: Abhishek Kar, name: Angjoo Kanazawa
Published: 2023-12-07 18:59:41

We propose NeRFiller, an approach that completes missing portions of a 3D
capture via generative 3D inpainting using off-the-shelf 2D visual generative
models. Often parts of a captured 3D scene or object are missing due to mesh
reconstruction failures or a lack of observations (e.g., contact regions, such
as the bottom of objects, or hard-to-reach areas). We approach this challenging
3D inpainting problem by leveraging a 2D inpainting diffusion model. We
identify a surprising behavior of these models, where they generate more 3D
consistent inpaints when images form a 2times2 grid, and show how to
generalize this behavior to more than four images. We then present an iterative
framework to distill these inpainted regions into a single consistent 3D scene.
In contrast to related works, we focus on completing scenes rather than
deleting foreground objects, and our approach does not require tight 2D object
masks or text. We compare our approach to relevant baselines adapted to our
setting on a variety of scenes, where NeRFiller creates the most 3D consistent
and plausible scene completions. Our project page is at
https://ethanweber.me/nerfiller.
Scaling Laws of Synthetic Images for Model Training ... for Now
Authors: name: Lijie Fan, name: Kaifeng Chen, name: Dilip Krishnan, name: Dina Katabi, name: Phillip Isola, name: Yonglong Tian
Published: 2023-12-07 18:59:59

Recent significant advances in text-to-image models unlock the possibility of
training vision systems using synthetic images, potentially overcoming the
difficulty of collecting curated data at scale. It is unclear, however, how
these models behave at scale, as more synthetic data is added to the training
set. In this paper we study the scaling laws of synthetic images generated by
state of the art text-to-image models, for the training of supervised models:
image classifiers with label supervision, and CLIP with language supervision.
We identify several factors, including text prompts, classifier-free guidance
scale, and types of text-to-image models, that significantly affect scaling
behavior. After tuning these factors, we observe that synthetic images
demonstrate a scaling trend similar to, but slightly less effective than, real
images in CLIP training, while they significantly underperform in scaling when
training supervised image classifiers. Our analysis indicates that the main
reason for this underperformance is the inability of off-the-shelf
text-to-image models to generate certain concepts, a limitation that
significantly impairs the training of image classifiers. Our findings also
suggest that scaling synthetic data can be particularly effective in scenarios
such as: (1) when there is a limited supply of real images for a supervised
problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the
evaluation dataset diverges significantly from the training data, indicating
the out-of-distribution scenario, or (3) when synthetic data is used in
conjunction with real images, as demonstrated in the training of CLIP models.