Orionnnn/hub-stats huggingface dataset:
Deep Language Networks: Joint Prompt Training of Stacked LLMs using
  Variational Inference
Authors: name: Alessandro Sordoni, name: Xingdi Yuan, name: Marc-Alexandre Côté, name: Matheus Pereira, name: Adam Trischler, name: Ziang Xiao, name: Arian Hosseini, name: Friederike Niedtner, name: Nicolas Le Roux
Published: 2023-06-21 18:45:56

We view large language models (LLMs) as stochastic language layers in
a network, where the learnable parameters are the natural language
prompts at each layer. We stack two such layers, feeding the output of
one layer to the next. We call the stacked architecture a Deep Language
Network (DLN). We first show how to effectively perform prompt optimization
for a 1-Layer language network (DLN-1). We then show how to train 2-layer DLNs
(DLN-2), where two prompts must be learnt. We consider the output of the first
layer as a latent variable to marginalize, and devise a variational inference
algorithm for joint prompt training. A DLN-2 reaches higher performance than a
single layer, sometimes comparable to few-shot GPT-4 even when each LLM in the
network is smaller and less powerful. The DLN code is open source:
https://github.com/microsoft/deep-language-networks .
Fast Segment Anything
Authors: name: Xu Zhao, name: Wenchao Ding, name: Yongqi An, name: Yinglong Du, name: Tao Yu, name: Min Li, name: Ming Tang, name: Jinqiao Wang
Published: 2023-06-21 10:08:29

The recently proposed segment anything model (SAM) has made a significant
influence in many computer vision tasks. It is becoming a foundation step for
many high-level tasks, like image segmentation, image caption, and image
editing. However, its huge computation costs prevent it from wider applications
in industry scenarios. The computation mainly comes from the Transformer
architecture at high-resolution inputs. In this paper, we propose a speed-up
alternative method for this fundamental task with comparable performance. By
reformulating the task as segments-generation and prompting, we find that a
regular CNN detector with an instance segmentation branch can also accomplish
this task well. Specifically, we convert this task to the well-studied instance
segmentation task and directly train the existing instance segmentation method
using only 1/50 of the SA-1B dataset published by SAM authors. With our method,
we achieve a comparable performance with the SAM method at 50 times higher
run-time speed. We give sufficient experimental results to demonstrate its
effectiveness. The codes and demos will be released at
https://github.com/CASIA-IVA-Lab/FastSAM.
EquiformerV2: Improved Equivariant Transformer for Scaling to
  Higher-Degree Representations
Authors: name: Yi-Lun Liao, name: Brandon Wood, name: Abhishek Das, name: Tess Smidt
Published: 2023-06-21 07:01:38

Equivariant Transformers such as Equiformer have demonstrated the efficacy of
applying Transformers to the domain of 3D atomistic systems. However, they are
still limited to small degrees of equivariant representations due to their
computational complexity. In this paper, we investigate whether these
architectures can scale well to higher degrees. Starting from Equiformer, we
first replace SO(3) convolutions with eSCN convolutions to efficiently
incorporate higher-degree tensors. Then, to better leverage the power of higher
degrees, we propose three architectural improvements -- attention
re-normalization, separable S^2 activation and separable layer normalization.
Putting this all together, we propose EquiformerV2, which outperforms previous
state-of-the-art methods on the large-scale OC20 dataset by up to 12% on
forces, 4% on energies, offers better speed-accuracy trade-offs, and
2times reduction in DFT calculations needed for computing adsorption
energies.
Training Transformers with 4-bit Integers
Authors: name: Haocheng Xi, name: Changhao Li, name: Jianfei Chen, name: Jun Zhu
Published: 2023-06-21 02:45:01

Quantizing the activation, weight, and gradient to 4-bit is promising to
accelerate neural network training. However, existing 4-bit training methods
require custom numerical formats which are not supported by contemporary
hardware. In this work, we propose a training method for transformers with all
matrix multiplications implemented with the INT4 arithmetic. Training with an
ultra-low INT4 precision is challenging. To achieve this, we carefully analyze
the specific structures of activation and gradients in transformers to propose
dedicated quantizers for them. For forward propagation, we identify the
challenge of outliers and propose a Hadamard quantizer to suppress the
outliers. For backpropagation, we leverage the structural sparsity of gradients
by proposing bit splitting and leverage score sampling techniques to quantize
gradients accurately. Our algorithm achieves competitive accuracy on a wide
range of tasks including natural language understanding, machine translation,
and image classification. Unlike previous 4-bit training methods, our algorithm
can be implemented on the current generation of GPUs. Our prototypical linear
operator implementation is up to 2.2 times faster than the FP16 counterparts
and speeds up the training by up to 35.1%.
DreamTime: An Improved Optimization Strategy for Text-to-3D Content
  Creation
Authors: name: Yukun Huang, name: Jianan Wang, name: Yukai Shi, name: Xianbiao Qi, name: Zheng-Jun Zha, name: Lei Zhang
Published: 2023-06-21 17:59:45

Text-to-image diffusion models pre-trained on billions of image-text pairs
have recently enabled text-to-3D content creation by optimizing a randomly
initialized Neural Radiance Fields (NeRF) with score distillation. However, the
resultant 3D models exhibit two limitations: (a) quality concerns such as
saturated color and the Janus problem; (b) extremely low diversity comparing to
text-guided image synthesis. In this paper, we show that the conflict between
NeRF optimization process and uniform timestep sampling in score distillation
is the main reason for these limitations. To resolve this conflict, we propose
to prioritize timestep sampling with monotonically non-increasing functions,
which aligns NeRF optimization with the sampling process of diffusion model.
Extensive experiments show that our simple redesign significantly improves
text-to-3D content creation with higher quality and diversity.
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
Authors: name: Christopher T. Small, name: Ivan Vendrov, name: Esin Durmus, name: Hadjar Homaei, name: Elizabeth Barry, name: Julien Cornebise, name: Ted Suzman, name: Deep Ganguli, name: Colin Megill
Published: 2023-06-20 22:52:51

Polis is a platform that leverages machine intelligence to scale up
deliberative processes. In this paper, we explore the opportunities and risks
associated with applying Large Language Models (LLMs) towards challenges with
facilitating, moderating and summarizing the results of Polis engagements. In
particular, we demonstrate with pilot experiments using Anthropic's Claude that
LLMs can indeed augment human intelligence to help more efficiently run Polis
conversations. In particular, we find that summarization capabilities enable
categorically new methods with immense promise to empower the public in
collective meaning-making exercises. And notably, LLM context limitations have
a significant impact on insight and quality of these results.
  However, these opportunities come with risks. We discuss some of these risks,
as well as principles and techniques for characterizing and mitigating them,
and the implications for other deliberative or political systems that may
employ LLMs. Finally, we conclude with several open future research directions
for augmenting tools like Polis with LLMs.
Multitrack Music Transcription with a Time-Frequency Perceiver
Authors: name: Wei-Tsung Lu, name: Ju-Chiang Wang, name: Yun-Ning Hung
Published: 2023-06-19 08:58:26

Multitrack music transcription aims to transcribe a music audio input into
the musical notes of multiple instruments simultaneously. It is a very
challenging task that typically requires a more complex model to achieve
satisfactory result. In addition, prior works mostly focus on transcriptions of
regular instruments, however, neglecting vocals, which are usually the most
important signal source if present in a piece of music. In this paper, we
propose a novel deep neural network architecture, Perceiver TF, to model the
time-frequency representation of audio input for multitrack transcription.
Perceiver TF augments the Perceiver architecture by introducing a hierarchical
expansion with an additional Transformer layer to model temporal coherence.
Accordingly, our model inherits the benefits of Perceiver that posses better
scalability, allowing it to well handle transcriptions of many instruments in a
single model. In experiments, we train a Perceiver TF to model 12 instrument
classes as well as vocal in a multi-task learning manner. Our result
demonstrates that the proposed system outperforms the state-of-the-art
counterparts (e.g., MT3 and SpecTNT) on various public datasets.
RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation
Authors: name: Konstantinos Bousmalis, name: Giulia Vezzani, name: Dushyant Rao, name: Coline Devin, name: Alex X. Lee, name: Maria Bauza, name: Todor Davchev, name: Yuxiang Zhou, name: Agrim Gupta, name: Akhil Raju, name: Antoine Laurens, name: Claudio Fantacci, name: Valentin Dalibard, name: Martina Zambelli, name: Murilo Martins, name: Rugile Pevceviciute, name: Michiel Blokzijl, name: Misha Denil, name: Nathan Batchelor, name: Thomas Lampe, name: Emilio Parisotto, name: Konrad Żołna, name: Scott Reed, name: Sergio Gómez Colmenarejo, name: Jon Scholz, name: Abbas Abdolmaleki, name: Oliver Groth, name: Jean-Baptiste Regli, name: Oleg Sushkov, name: Tom Rothörl, name: José Enrique Chen, name: Yusuf Aytar, name: Dave Barker, name: Joy Ortiz, name: Martin Riedmiller, name: Jost Tobias Springenberg, name: Raia Hadsell, name: Francesco Nori, name: Nicolas Heess
Published: 2023-06-20 17:35:20

The ability to leverage heterogeneous robotic experience from different
robots and tasks to quickly master novel skills and embodiments has the
potential to transform robot learning. Inspired by recent advances in
foundation models for vision and language, we propose a foundation agent for
robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned
decision transformer capable of consuming multi-embodiment action-labelled
visual experience. This data spans a large repertoire of motor control skills
from simulated and real robotic arms with varying sets of observations and
actions. With RoboCat, we demonstrate the ability to generalise to new tasks
and robots, both zero-shot as well as through adaptation using only 100--1000
examples for the target task. We also show how a trained model itself can be
used to generate data for subsequent training iterations, thus providing a
basic building block for an autonomous improvement loop. We investigate the
agent's capabilities, with large-scale evaluations both in simulation and on
three different real robot embodiments. We find that as we grow and diversify
its training data, RoboCat not only shows signs of cross-task transfer, but
also becomes more efficient at adapting to new tasks.
MotionGPT: Finetuned LLMs are General-Purpose Motion Generators
Authors: name: Yaqi Zhang, name: Di Huang, name: Bin Liu, name: Shixiang Tang, name: Yan Lu, name: Lu Chen, name: Lei Bai, name: Qi Chu, name: Nenghai Yu, name: Wanli Ouyang
Published: 2023-06-19 12:58:17

Generating realistic human motion from given action descriptions has
experienced significant advancements because of the emerging requirement of
digital humans. While recent works have achieved impressive results in
generating motion directly from textual action descriptions, they often support
only a single modality of the control signal, which limits their application in
the real digital human industry. This paper presents a Motion General-Purpose
generaTor (MotionGPT) that can use multimodal control signals, e.g., text and
single-frame poses, for generating consecutive human motions by treating
multimodal signals as special input tokens in large language models (LLMs).
Specifically, we first quantize multimodal control signals into discrete codes
and then formulate them in a unified prompt instruction to ask the LLMs to
generate the motion answer. Our MotionGPT demonstrates a unified human motion
generation model with multimodal control signals by tuning a mere 0.4% of LLM
parameters. To the best of our knowledge, MotionGPT is the first method to
generate human motion by multimodal control signals, which we hope can shed
light on this new direction. Codes shall be released upon acceptance.
Diffusion with Forward Models: Solving Stochastic Inverse Problems
  Without Direct Supervision
Authors: name: Ayush Tewari, name: Tianwei Yin, name: George Cazenavette, name: Semon Rezchikov, name: Joshua B. Tenenbaum, name: Frédo Durand, name: William T. Freeman, name: Vincent Sitzmann
Published: 2023-06-20 17:53:00

Denoising diffusion models are a powerful type of generative models used to
capture complex distributions of real-world signals. However, their
applicability is limited to scenarios where training samples are readily
available, which is not always the case in real-world applications. For
example, in inverse graphics, the goal is to generate samples from a
distribution of 3D scenes that align with a given image, but ground-truth 3D
scenes are unavailable and only 2D images are accessible. To address this
limitation, we propose a novel class of denoising diffusion probabilistic
models that learn to sample from distributions of signals that are never
directly observed. Instead, these signals are measured indirectly through a
known differentiable forward model, which produces partial observations of the
unknown signal. Our approach involves integrating the forward model directly
into the denoising process. This integration effectively connects the
generative modeling of observations with the generative modeling of the
underlying signals, allowing for end-to-end training of a conditional
generative model over signals. During inference, our approach enables sampling
from the distribution of underlying signals that are consistent with a given
partial observation. We demonstrate the effectiveness of our method on three
challenging computer vision tasks. For instance, in the context of inverse
graphics, our model enables direct sampling from the distribution of 3D scenes
that align with a single 2D input image.
HomeRobot: Open-Vocabulary Mobile Manipulation
Authors: name: Sriram Yenamandra, name: Arun Ramachandran, name: Karmesh Yadav, name: Austin Wang, name: Mukul Khanna, name: Theophile Gervet, name: Tsung-Yen Yang, name: Vidhi Jain, name: Alexander William Clegg, name: John Turner, name: Zsolt Kira, name: Manolis Savva, name: Angel Chang, name: Devendra Singh Chaplot, name: Dhruv Batra, name: Roozbeh Mottaghi, name: Yonatan Bisk, name: Chris Paxton
Published: 2023-06-20 14:30:32

HomeRobot (noun): An affordable compliant robot that navigates homes and
manipulates a wide range of objects in order to complete everyday tasks.
Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object
in any unseen environment, and placing it in a commanded location. This is a
foundational challenge for robots to be useful assistants in human
environments, because it involves tackling sub-problems from across robotics:
perception, language understanding, navigation, and manipulation are all
essential to OVMM. In addition, integration of the solutions to these
sub-problems poses its own substantial challenges. To drive research in this
area, we introduce the HomeRobot OVMM benchmark, where an agent navigates
household environments to grasp novel objects and place them on target
receptacles. HomeRobot has two components: a simulation component, which uses a
large and diverse curated object set in new, high-quality multi-room home
environments; and a real-world component, providing a software stack for the
low-cost Hello Robot Stretch to encourage replication of real-world experiments
across labs. We implement both reinforcement learning and heuristic
(model-based) baselines and show evidence of sim-to-real transfer. Our
baselines achieve a 20% success rate in the real world; our experiments
identify ways future research work improve performance. See videos on our
website: https://ovmm.github.io/.
Point-Cloud Completion with Pretrained Text-to-image Diffusion Models
Authors: name: Yoni Kasten, name: Ohad Rahamim, name: Gal Chechik
Published: 2023-06-18 11:51:46

Point-cloud data collected in real-world applications are often incomplete.
Data is typically missing due to objects being observed from partial
viewpoints, which only capture a specific perspective or angle. Additionally,
data can be incomplete due to occlusion and low-resolution sampling. Existing
completion approaches rely on datasets of predefined objects to guide the
completion of noisy and incomplete, point clouds. However, these approaches
perform poorly when tested on Out-Of-Distribution (OOD) objects, that are
poorly represented in the training dataset. Here we leverage recent advances in
text-guided image generation, which lead to major breakthroughs in text-guided
shape generation. We describe an approach called SDS-Complete that uses a
pre-trained text-to-image diffusion model and leverages the text semantics of a
given incomplete point cloud of an object, to obtain a complete surface
representation. SDS-Complete can complete a variety of objects using test-time
optimization without expensive collection of 3D information. We evaluate SDS
Complete on incomplete scanned objects, captured by real-world depth sensors
and LiDAR scanners. We find that it effectively reconstructs objects that are
absent from common datasets, reducing Chamfer loss by 50% on average compared
with current methods. Project page: https://sds-complete.github.io/
Textbooks Are All You Need
Authors: name: Suriya Gunasekar, name: Yi Zhang, name: Jyoti Aneja, name: Caio César Teodoro Mendes, name: Allie Del Giorno, name: Sivakanth Gopi, name: Mojan Javaheripi, name: Piero Kauffmann, name: Gustavo de Rosa, name: Olli Saarikivi, name: Adil Salim, name: Shital Shah, name: Harkirat Singh Behl, name: Xin Wang, name: Sébastien Bubeck, name: Ronen Eldan, name: Adam Tauman Kalai, name: Yin Tat Lee, name: Yuanzhi Li
Published: 2023-06-20 16:14:25

We introduce phi-1, a new large language model for code, with significantly
smaller size than competing models: phi-1 is a Transformer-based model with
1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
quality" data from the web (6B tokens) and synthetically generated textbooks
and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
surprising emergent properties compared to phi-1-base, our model before our
finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
model with 350M parameters trained with the same pipeline as phi-1 that still
achieves 45% on HumanEval.
RepoFusion: Training Code Models to Understand Your Repository
Authors: name: Disha Shrivastava, name: Denis Kocetkov, name: Harm de Vries, name: Dzmitry Bahdanau, name: Torsten Scholak
Published: 2023-06-19 15:05:31

Despite the huge success of Large Language Models (LLMs) in coding assistants
like GitHub Copilot, these models struggle to understand the context present in
the repository (e.g., imports, parent classes, files with similar names, etc.),
thereby producing inaccurate code completions. This effect is more pronounced
when using these assistants for repositories that the model has not seen during
training, such as proprietary software or work-in-progress code projects.
Recent work has shown the promise of using context from the repository during
inference. In this work, we extend this idea and propose RepoFusion, a
framework to train models to incorporate relevant repository context.
Experiments on single-line code completion show that our models trained with
repository context significantly outperform much larger code models as
CodeGen-16B-multi (sim73times larger) and closely match the performance of
the sim 70times larger StarCoderBase model that was trained with the
Fill-in-the-Middle objective. We find these results to be a novel and
compelling demonstration of the gains that training with repository context can
bring. We carry out extensive ablation studies to investigate the impact of
design choices such as context type, number of contexts, context length, and
initialization within our framework. Lastly, we release Stack-Repo, a dataset
of 200 Java repositories with permissive licenses and near-deduplicated files
that are augmented with three types of repository contexts. Additionally, we
are making available the code and trained checkpoints for our work. Our
released resources can be found at https://huggingface.co/RepoFusion.
Guiding Language Models of Code with Global Context using Monitors
Authors: name: Lakshya A Agrawal, name: Aditya Kanade, name: Navin Goyal, name: Shuvendu K. Lahiri, name: Sriram K. Rajamani
Published: 2023-06-19 08:13:50

Language models of code (LMs) work well when the surrounding code in the
vicinity of generation provides sufficient context. This is not true when it
becomes necessary to use types or functionality defined in another module or
library, especially those not seen during training. LMs suffer from limited
awareness of such global context and end up hallucinating, e.g., using types
defined in other files incorrectly. Recent work tries to overcome this issue by
retrieving global information to augment the local context. However, this
bloats the prompt or requires architecture modifications and additional
training.
  Integrated development environments (IDEs) assist developers by bringing the
global context at their fingertips using static analysis. We extend this
assistance, enjoyed by developers, to the LMs. We propose a notion of monitors
that use static analysis in the background to guide the decoding. Unlike a
priori retrieval, static analysis is invoked iteratively during the entire
decoding process, providing the most relevant suggestions on demand. We
demonstrate the usefulness of our proposal by monitoring for type-consistent
use of identifiers whenever an LM generates code for object dereference.
  To evaluate our approach, we curate PragmaticCode, a dataset of open-source
projects with their development environments. On models of varying parameter
scale, we show that monitor-guided decoding consistently improves the ability
of an LM to not only generate identifiers that match the ground truth but also
improves compilation rates and agreement with ground truth. We find that LMs
with fewer parameters, when guided with our monitor, can outperform larger LMs.
With monitor-guided decoding, SantaCoder-1.1B achieves better compilation rate
and next-identifier match than the much larger text-davinci-003 model. The
datasets and code will be released at https://aka.ms/monitors4codegen .
GLIMMER: generalized late-interaction memory reranker
Authors: name: Michiel de Jong, name: Yury Zemlyanskiy, name: Nicholas FitzGerald, name: Sumit Sanghai, name: William W. Cohen, name: Joshua Ainslie
Published: 2023-06-17 01:54:25

Memory-augmentation is a powerful approach for efficiently incorporating
external information into language models, but leads to reduced performance
relative to retrieving text. Recent work introduced LUMEN, a memory-retrieval
hybrid that partially pre-computes memory and updates memory representations on
the fly with a smaller live encoder.
  We propose GLIMMER, which improves on this approach through 1) exploiting
free access to the powerful memory representations by applying a shallow
reranker on top of memory to drastically improve retrieval quality at low cost,
and 2) incorporating multi-task training to learn a general and higher quality
memory and live encoder. GLIMMER achieves strong gains in performance at faster
speeds compared to LUMEN and FiD on the KILT benchmark of knowledge-intensive
tasks.