Orionnnn/hub-stats huggingface dataset:
Challenges and Applications of Large Language Models
Authors: name: Jean Kaddour, name: Joshua Harris, name: Maximilian Mozes, name: Herbie Bradley, name: Roberta Raileanu, name: Robert McHardy
Published: 2023-07-19 17:55:13

Large Language Models (LLMs) went from non-existent to ubiquitous in the
machine learning discourse within a few years. Due to the fast pace of the
field, it is difficult to identify the remaining challenges and already
fruitful application areas. In this paper, we aim to establish a systematic set
of open problems and application successes so that ML researchers can
comprehend the field's current state more quickly and become productive.
Towards A Unified Agent with Foundation Models
Authors: name: Norman Di Palo, name: Arunkumar Byravan, name: Leonard Hasenclever, name: Markus Wulfmeier, name: Nicolas Heess, name: Martin Riedmiller
Published: 2023-07-18 22:37:30

Language Models and Vision Language Models have recently demonstrated
unprecedented capabilities in terms of understanding human intentions,
reasoning, scene understanding, and planning-like behaviour, in text form,
among many others. In this work, we investigate how to embed and leverage such
abilities in Reinforcement Learning (RL) agents. We design a framework that
uses language as the core reasoning tool, exploring how this enables an agent
to tackle a series of fundamental RL challenges, such as efficient exploration,
reusing experience data, scheduling skills, and learning from observations,
which traditionally require separate, vertically designed algorithms. We test
our method on a sparse-reward simulated robotic manipulation environment, where
a robot needs to stack a set of objects. We demonstrate substantial performance
improvements over baselines in exploration efficiency and ability to reuse data
from offline datasets, and illustrate how to reuse learned skills to solve
novel tasks or imitate videos of human experts.
Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Authors: name: Pranshu Malviya, name: Gonçalo Mordido, name: Aristide Baratin, name: Reza Babanezhad Harikandeh, name: Jerry Huang, name: Simon Lacoste-Julien, name: Razvan Pascanu, name: Sarath Chandar
Published: 2023-07-18 20:59:52

Adaptive gradient-based optimizers, particularly Adam, have left their mark
in training large-scale deep learning models. The strength of such optimizers
is that they exhibit fast convergence while being more robust to hyperparameter
choice. However, they often generalize worse than non-adaptive methods. Recent
studies have tied this performance gap to flat minima selection: adaptive
methods tend to find solutions in sharper basins of the loss landscape, which
in turn hurts generalization. To overcome this issue, we propose a new
memory-augmented version of Adam that promotes exploration towards flatter
minima by using a buffer of critical momentum terms during training.
Intuitively, the use of the buffer makes the optimizer overshoot outside the
basin of attraction if it is not wide enough. We empirically show that our
method improves the performance of several variants of Adam on standard
supervised language modelling and image classification tasks.
NU-MCC: Multiview Compressive Coding with Neighborhood Decoder and
  Repulsive UDF
Authors: name: Stefan Lionar, name: Xiangyu Xu, name: Min Lin, name: Gim Hee Lee
Published: 2023-07-18 10:02:09

Remarkable progress has been made in 3D reconstruction from single-view RGB-D
inputs. MCC is the current state-of-the-art method in this field, which
achieves unprecedented success by combining vision Transformers with
large-scale training. However, we identified two key limitations of MCC: 1) The
Transformer decoder is inefficient in handling large number of query points; 2)
The 3D representation struggles to recover high-fidelity details. In this
paper, we propose a new approach called NU-MCC that addresses these
limitations. NU-MCC includes two key innovations: a Neighborhood decoder and a
Repulsive Unsigned Distance Function (Repulsive UDF). First, our Neighborhood
decoder introduces center points as an efficient proxy of input visual
features, allowing each query point to only attend to a small neighborhood.
This design not only results in much faster inference speed but also enables
the exploitation of finer-scale visual features for improved recovery of 3D
textures. Second, our Repulsive UDF is a novel alternative to the occupancy
field used in MCC, significantly improving the quality of 3D object
reconstruction. Compared to standard UDFs that suffer from holes in results,
our proposed Repulsive UDF can achieve more complete surface reconstruction.
Experimental results demonstrate that NU-MCC is able to learn a strong 3D
representation, significantly advancing the state of the art in single-view 3D
reconstruction. Particularly, it outperforms MCC by 9.7% in terms of the
F1-score on the CO3D-v2 dataset with more than 5x faster running speed.
Biomaker CA: a Biome Maker project using Cellular Automata
Authors: name: Ettore Randazzo, name: Alexander Mordvintsev
Published: 2023-07-18 15:03:40

We introduce Biomaker CA: a Biome Maker project using Cellular Automata (CA).
In Biomaker CA, morphogenesis is a first class citizen and small seeds need to
grow into plant-like organisms to survive in a nutrient starved environment and
eventually reproduce with variation so that a biome survives for long
timelines. We simulate complex biomes by means of CA rules in 2D grids and
parallelize all of its computation on GPUs through the Python JAX framework. We
show how this project allows for several different kinds of environments and
laws of 'physics', alongside different model architectures and mutation
strategies. We further analyze some configurations to show how plant agents can
grow, survive, reproduce, and evolve, forming stable and unstable biomes. We
then demonstrate how one can meta-evolve models to survive in a harsh
environment either through end-to-end meta-evolution or by a more surgical and
efficient approach, called Petri dish meta-evolution. Finally, we show how to
perform interactive evolution, where the user decides how to evolve a plant
model interactively and then deploys it in a larger environment. We open source
Biomaker CA at: https://tinyurl.com/2x8yu34s .
Does Circuit Analysis Interpretability Scale? Evidence from Multiple
  Choice Capabilities in Chinchilla
Authors: name: Tom Lieberum, name: Matthew Rahtz, name: János Kramár, name: Geoffrey Irving, name: Rohin Shah, name: Vladimir Mikulik
Published: 2023-07-18 17:39:04

Circuit analysis is a promising technique for understanding the
internal mechanisms of language models. However, existing analyses are done in
small models far from the state of the art. To address this, we present a case
study of circuit analysis in the 70B Chinchilla model, aiming to test the
scalability of circuit analysis. In particular, we study multiple-choice
question answering, and investigate Chinchilla's capability to identify the
correct answer label given knowledge of the correct answer text.
We find that the existing techniques of logit attribution, attention pattern
visualization, and activation patching naturally scale to Chinchilla, allowing
us to identify and categorize a small set of `output nodes' (attention heads
and MLPs).
  We further study the `correct letter' category of attention heads aiming to
understand the semantics of their features, with mixed results. For normal
multiple-choice question answers, we significantly compress the query, key and
value subspaces of the head without loss of performance when operating on the
answer labels for multiple-choice questions, and we show that the query and key
subspaces represent an `Nth item in an enumeration' feature to at least some
extent. However, when we attempt to use this explanation to understand the
heads' behaviour on a more general distribution including randomized answer
labels, we find that it is only a partial explanation, suggesting there is more
to learn about the operation of `correct letter' heads on multiple choice
question answering.
Llama 2: Open Foundation and Fine-Tuned Chat Models
Authors: name: Hugo Touvron, name: Louis Martin, name: Kevin Stone, name: Peter Albert, name: Amjad Almahairi, name: Yasmine Babaei, name: Nikolay Bashlykov, name: Soumya Batra, name: Prajjwal Bhargava, name: Shruti Bhosale, name: Dan Bikel, name: Lukas Blecher, name: Cristian Canton Ferrer, name: Moya Chen, name: Guillem Cucurull, name: David Esiobu, name: Jude Fernandes, name: Jeremy Fu, name: Wenyin Fu, name: Brian Fuller, name: Cynthia Gao, name: Vedanuj Goswami, name: Naman Goyal, name: Anthony Hartshorn, name: Saghar Hosseini, name: Rui Hou, name: Hakan Inan, name: Marcin Kardas, name: Viktor Kerkez, name: Madian Khabsa, name: Isabel Kloumann, name: Artem Korenev, name: Punit Singh Koura, name: Marie-Anne Lachaux, name: Thibaut Lavril, name: Jenya Lee, name: Diana Liskovich, name: Yinghai Lu, name: Yuning Mao, name: Xavier Martinet, name: Todor Mihaylov, name: Pushkar Mishra, name: Igor Molybog, name: Yixin Nie, name: Andrew Poulton, name: Jeremy Reizenstein, name: Rashi Rungta, name: Kalyan Saladi, name: Alan Schelten, name: Ruan Silva, name: Eric Michael Smith, name: Ranjan Subramanian, name: Xiaoqing Ellen Tan, name: Binh Tang, name: Ross Taylor, name: Adina Williams, name: Jian Xiang Kuan, name: Puxin Xu, name: Zheng Yan, name: Iliyan Zarov, name: Yuchen Zhang, name: Angela Fan, name: Melanie Kambadur, name: Sharan Narang, name: Aurelien Rodriguez, name: Robert Stojnic, name: Sergey Edunov, name: Thomas Scialom
Published: 2023-07-18 14:31:57

In this work, we develop and release Llama 2, a collection of pretrained and
fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70
billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for
dialogue use cases. Our models outperform open-source chat models on most
benchmarks we tested, and based on our human evaluations for helpfulness and
safety, may be a suitable substitute for closed-source models. We provide a
detailed description of our approach to fine-tuning and safety improvements of
Llama 2-Chat in order to enable the community to build on our work and
contribute to the responsible development of LLMs.
Augmenting CLIP with Improved Visio-Linguistic Reasoning
Authors: name: Samyadeep Basu, name: Maziar Sanjabi, name: Daniela Massiceti, name: Shell Xu Hu, name: Soheil Feizi
Published: 2023-07-18 13:10:11

Image-text contrastive models such as CLIP are useful for a variety of
downstream applications including zero-shot classification, image-text
retrieval and transfer learning. However, these contrastively trained
vision-language models often fail on compositional visio-linguistic tasks such
as Winoground with performance equivalent to random chance. In our paper, we
address this issue and propose a sample-efficient light-weight method called
SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
of CLIP. The core idea of our method is to use differentiable image
parameterizations to fine-tune CLIP with a distillation objective from large
text-to-image generative models such as Stable-Diffusion which are relatively
good at visio-linguistic reasoning tasks. On the challenging Winoground
compositional reasoning benchmark, our method improves the absolute
visio-linguistic performance of different CLIP models by up to 7%, while on the
ARO dataset, our method improves the visio-linguistic performance by upto 3%.
As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
that the zero-shot performance improves marginally on a variety of downstream
datasets. Our method reinforces that carefully designed distillation objectives
from generative models can be leveraged to extend existing contrastive
image-text models with improved visio-linguistic reasoning capabilities.
How is ChatGPT's behavior changing over time?
Authors: name: Lingjiao Chen, name: Matei Zaharia, name: James Zou
Published: 2023-07-18 06:56:08

GPT-3.5 and GPT-4 are the two most widely used large language model (LLM)
services. However, when and how these models are updated over time is opaque.
Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on
four diverse tasks: 1) solving math problems, 2) answering sensitive/dangerous
questions, 3) generating code and 4) visual reasoning. We find that the
performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time.
For example, GPT-4 (March 2023) was very good at identifying prime numbers
(accuracy 97.6%) but GPT-4 (June 2023) was very poor on these same questions
(accuracy 2.4%). Interestingly GPT-3.5 (June 2023) was much better than GPT-3.5
(March 2023) in this task. GPT-4 was less willing to answer sensitive questions
in June than in March, and both GPT-4 and GPT-3.5 had more formatting mistakes
in code generation in June than in March. Overall, our findings shows that the
behavior of the same LLM service can change substantially in a relatively short
amount of time, highlighting the need for continuous monitoring of LLM quality.
TableGPT: Towards Unifying Tables, Nature Language and Commands into One
  GPT
Authors: name: Liangyu Zha, name: Junlin Zhou, name: Liyao Li, name: Rui Wang, name: Qingyi Huang, name: Saisai Yang, name: Jing Yuan, name: Changbao Su, name: Xiang Li, name: Aofeng Su, name: Tao Zhang, name: Chen Zhou, name: Kaizhe Shou, name: Miao Wang, name: Wufang Zhu, name: Guoshan Lu, name: Chao Ye, name: Yali Ye, name: Wentao Ye, name: Yiming Zhang, name: Xinglong Deng, name: Jie Xu, name: Haobo Wang, name: Gang Chen, name: Junbo Zhao
Published: 2023-07-17 17:36:09

Tables are prevalent in real-world databases, requiring significant time and
effort for humans to analyze and manipulate. The advancements in large language
models (LLMs) have made it possible to interact with tables using natural
language input, bringing this capability closer to reality. In this paper, we
present TableGPT, a unified fine-tuned framework that enables LLMs to
understand and operate on tables using external functional commands. It
introduces the capability to seamlessly interact with tables, enabling a wide
range of functionalities such as question answering, data manipulation (e.g.,
insert, delete, query, and modify operations), data visualization, analysis
report generation, and automated prediction. TableGPT aims to provide
convenience and accessibility to users by empowering them to effortlessly
leverage tabular data. At the core of TableGPT lies the novel concept of global
tabular representations, which empowers LLMs to gain a comprehensive
understanding of the entire table beyond meta-information. By jointly training
LLMs on both table and text modalities, TableGPT achieves a deep understanding
of tabular data and the ability to perform complex operations on tables through
chain-of-command instructions. Importantly, TableGPT offers the advantage of
being a self-contained system rather than relying on external API interfaces.
Moreover, it supports efficient data process flow, query rejection (when
appropriate) and private deployment, enabling faster domain data fine-tuning
and ensuring data privacy, which enhances the framework's adaptability to
specific use cases.
INVE: Interactive Neural Video Editing
Authors: name: Jiahui Huang, name: Leonid Sigal, name: Kwang Moo Yi, name: Oliver Wang, name: Joon-Young Lee
Published: 2023-07-15 00:02:41

We present Interactive Neural Video Editing (INVE), a real-time video editing
solution, which can assist the video editing process by consistently
propagating sparse frame edits to the entire video clip. Our method is inspired
by the recent work on Layered Neural Atlas (LNA). LNA, however, suffers from
two major drawbacks: (1) the method is too slow for interactive editing, and
(2) it offers insufficient support for some editing use cases, including direct
frame editing and rigid texture tracking. To address these challenges we
leverage and adopt highly efficient network architectures, powered by
hash-grids encoding, to substantially improve processing speed. In addition, we
learn bi-directional functions between image-atlas and introduce vectorized
editing, which collectively enables a much greater variety of edits in both the
atlas and the frames directly. Compared to LNA, our INVE reduces the learning
and inference time by a factor of 5, and supports various video editing
operations that LNA cannot. We showcase the superiority of INVE over LNA in
interactive video editing through a comprehensive quantitative and qualitative
analysis, highlighting its numerous advantages and improved performance. For
video results, please see https://gabriel-huang.github.io/inve/
Scale-Aware Modulation Meet Transformer
Authors: name: Weifeng Lin, name: Ziheng Wu, name: Jiayu Chen, name: Jun Huang, name: Lianwen Jin
Published: 2023-07-17 15:47:48

This paper presents a new vision Transformer, Scale-Aware Modulation
Transformer (SMT), that can handle various downstream tasks efficiently by
combining the convolutional network and vision Transformer. The proposed
Scale-Aware Modulation (SAM) in the SMT includes two primary novel designs.
Firstly, we introduce the Multi-Head Mixed Convolution (MHMC) module, which can
capture multi-scale features and expand the receptive field. Secondly, we
propose the Scale-Aware Aggregation (SAA) module, which is lightweight but
effective, enabling information fusion across different heads. By leveraging
these two modules, convolutional modulation is further enhanced. Furthermore,
in contrast to prior works that utilized modulations throughout all stages to
build an attention-free network, we propose an Evolutionary Hybrid Network
(EHN), which can effectively simulate the shift from capturing local to global
dependencies as the network becomes deeper, resulting in superior performance.
Extensive experiments demonstrate that SMT significantly outperforms existing
state-of-the-art models across a wide range of visual tasks. Specifically, SMT
with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs can achieve 82.2% and 84.3% top-1
accuracy on ImageNet-1K, respectively. After pretrained on ImageNet-22K in
224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with
resolution 224^2 and 384^2, respectively. For object detection with Mask R-CNN,
the SMT base trained with 1x and 3x schedule outperforms the Swin Transformer
counterpart by 4.2 and 1.3 mAP on COCO, respectively. For semantic segmentation
with UPerNet, the SMT base test at single- and multi-scale surpasses Swin by
2.0 and 1.1 mIoU respectively on the ADE20K.
Does Visual Pretraining Help End-to-End Reasoning?
Authors: name: Chen Sun, name: Calvin Luo, name: Xingyi Zhou, name: Anurag Arnab, name: Cordelia Schmid
Published: 2023-07-17 14:08:38

We aim to investigate whether end-to-end learning of visual reasoning can be
achieved with general-purpose neural networks, with the help of visual
pretraining. A positive result would refute the common belief that explicit
visual abstraction (e.g. object detection) is essential for compositional
generalization on visual reasoning, and confirm the feasibility of a neural
network "generalist" to solve visual recognition and reasoning tasks. We
propose a simple and general self-supervised framework which "compresses" each
video frame into a small set of tokens with a transformer network, and
reconstructs the remaining frames based on the compressed temporal context. To
minimize the reconstruction loss, the network must learn a compact
representation for each image, as well as capture temporal dynamics and object
permanence from temporal context. We perform evaluation on two visual reasoning
benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve
compositional generalization for end-to-end visual reasoning. Our proposed
framework outperforms traditional supervised pretraining, including image
classification and explicit object detection, by large margins.
Language Conditioned Traffic Generation
Authors: name: Shuhan Tan, name: Boris Ivanovic, name: Xinshuo Weng, name: Marco Pavone, name: Philipp Kraehenbuehl
Published: 2023-07-16 05:10:32

Simulation forms the backbone of modern self-driving development. Simulators
help develop, test, and improve driving systems without putting humans,
vehicles, or their environment at risk. However, simulators face a major
challenge: They rely on realistic, scalable, yet interesting content. While
recent advances in rendering and scene reconstruction make great strides in
creating static scene assets, modeling their layout, dynamics, and behaviors
remains challenging. In this work, we turn to language as a source of
supervision for dynamic traffic scene generation. Our model, LCTGen, combines a
large language model with a transformer-based decoder architecture that selects
likely map locations from a dataset of maps, and produces an initial traffic
distribution, as well as the dynamics of each vehicle. LCTGen outperforms prior
work in both unconditional and conditional traffic scene generation in terms of
realism and fidelity. Code and video will be available at
https://ariostgx.github.io/lctgen.
CoTracker: It is Better to Track Together
Authors: name: Nikita Karaev, name: Ignacio Rocco, name: Benjamin Graham, name: Natalia Neverova, name: Andrea Vedaldi, name: Christian Rupprecht
Published: 2023-07-14 21:13:04

Methods for video motion prediction either estimate jointly the instantaneous
motion of all points in a given video frame using optical flow or independently
track the motion of individual points throughout the video. The latter is true
even for powerful deep-learning methods that can track points through
occlusions. Tracking points individually ignores the strong correlation that
can exist between the points, for instance, because they belong to the same
physical object, potentially harming performance. In this paper, we thus
propose CoTracker, an architecture that jointly tracks multiple points
throughout an entire video. This architecture combines several ideas from the
optical flow and tracking literature in a new, flexible and powerful design. It
is based on a transformer network that models the correlation of different
points in time via specialised attention layers. The transformer iteratively
updates an estimate of several trajectories. It can be applied in a
sliding-window manner to very long videos, for which we engineer an unrolled
training loop. It can track from one to several points jointly and supports
adding new points to track at any time. The result is a flexible and powerful
tracking algorithm that outperforms state-of-the-art methods in almost all
benchmarks.