Orionnnn/hub-stats huggingface dataset:
DualToken-ViT: Position-aware Efficient Vision Transformer with Dual
  Token Fusion
Authors: name: Zhenzhen Chu, name: Jiayu Chen, name: Cen Chen, name: Chengyu Wang, name: Ziheng Wu, name: Jun Huang, name: Weining Qian
Published: 2023-09-21 18:46:32

Self-attention-based vision transformers (ViTs) have emerged as a highly
competitive architecture in computer vision. Unlike convolutional neural
networks (CNNs), ViTs are capable of global information sharing. With the
development of various structures of ViTs, ViTs are increasingly advantageous
for many vision tasks. However, the quadratic complexity of self-attention
renders ViTs computationally intensive, and their lack of inductive biases of
locality and translation equivariance demands larger model sizes compared to
CNNs to effectively learn visual features. In this paper, we propose a
light-weight and efficient vision transformer model called DualToken-ViT that
leverages the advantages of CNNs and ViTs. DualToken-ViT effectively fuses the
token with local information obtained by convolution-based structure and the
token with global information obtained by self-attention-based structure to
achieve an efficient attention structure. In addition, we use position-aware
global tokens throughout all stages to enrich the global information, which
further strengthening the effect of DualToken-ViT. Position-aware global tokens
also contain the position information of the image, which makes our model
better for vision tasks. We conducted extensive experiments on image
classification, object detection and semantic segmentation tasks to demonstrate
the effectiveness of DualToken-ViT. On the ImageNet-1K dataset, our models of
different scales achieve accuracies of 75.4% and 79.4% with only 0.5G and 1.0G
FLOPs, respectively, and our model with 1.0G FLOPs outperforms LightViT-T using
global tokens by 0.7%.
MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary
  Instance Segmentation
Authors: name: Jiahao Xie, name: Wei Li, name: Xiangtai Li, name: Ziwei Liu, name: Yew Soon Ong, name: Chen Change Loy
Published: 2023-09-22 17:59:42

We present MosaicFusion, a simple yet effective diffusion-based data
augmentation approach for large vocabulary instance segmentation. Our method is
training-free and does not rely on any label supervision. Two key designs
enable us to employ an off-the-shelf text-to-image diffusion model as a useful
dataset generator for object instances and mask annotations. First, we divide
an image canvas into several regions and perform a single round of diffusion
process to generate multiple instances simultaneously, conditioning on
different text prompts. Second, we obtain corresponding instance masks by
aggregating cross-attention maps associated with object prompts across layers
and diffusion time steps, followed by simple thresholding and edge-aware
refinement processing. Without bells and whistles, our MosaicFusion can produce
a significant amount of synthetic labeled data for both rare and novel
categories. Experimental results on the challenging LVIS long-tailed and
open-vocabulary benchmarks demonstrate that MosaicFusion can significantly
improve the performance of existing instance segmentation models, especially
for rare and novel categories. Code will be released at
https://github.com/Jiahao000/MosaicFusion.
Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient
  Pruning of A Multilingual ASR Model
Authors: name: Jiamin Xie, name: Ke Li, name: Jinxi Guo, name: Andros Tjandra, name: Yuan Shangguan, name: Leda Sari, name: Chunyang Wu, name: Junteng Jia, name: Jay Mahadeokar, name: Ozlem Kalinli
Published: 2023-09-22 17:30:28

Neural network pruning offers an effective method for compressing a
multilingual automatic speech recognition (ASR) model with minimal performance
loss. However, it entails several rounds of pruning and re-training needed to
be run for each language. In this work, we propose the use of an adaptive
masking approach in two scenarios for pruning a multilingual ASR model
efficiently, each resulting in sparse monolingual models or a sparse
multilingual model (named as Dynamic ASR Pathways). Our approach dynamically
adapts the sub-network, avoiding premature decisions about a fixed sub-network
structure. We show that our approach outperforms existing pruning methods when
targeting sparse monolingual models. Further, we illustrate that Dynamic ASR
Pathways jointly discovers and trains better sub-networks (pathways) of a
single multilingual model by adapting from different sub-network
initializations, thereby reducing the need for language-specific pruning.
CodePlan: Repository-level Coding using LLMs and Planning
Authors: name: Ramakrishna Bairi, name: Atharv Sonwane, name: Aditya Kanade, name: Vageesh D C, name: Arun Iyer, name: Suresh Parthasarathy, name: Sriram Rajamani, name: B. Ashok, name: Shashank Shet
Published: 2023-09-21 21:45:17

Software engineering activities such as package migration, fixing errors
reports from static analysis or testing, and adding type annotations or other
specifications to a codebase, involve pervasively editing the entire repository
of code. We formulate these activities as repository-level coding tasks.
  Recent tools like GitHub Copilot, which are powered by Large Language Models
(LLMs), have succeeded in offering high-quality solutions to localized coding
problems. Repository-level coding tasks are more involved and cannot be solved
directly using LLMs, since code within a repository is inter-dependent and the
entire repository may be too large to fit into the prompt. We frame
repository-level coding as a planning problem and present a task-agnostic
framework, called CodePlan to solve it. CodePlan synthesizes a multi-step chain
of edits (plan), where each step results in a call to an LLM on a code location
with context derived from the entire repository, previous code changes and
task-specific instructions. CodePlan is based on a novel combination of an
incremental dependency analysis, a change may-impact analysis and an adaptive
planning algorithm.
  We evaluate the effectiveness of CodePlan on two repository-level tasks:
package migration (C#) and temporal code edits (Python). Each task is evaluated
on multiple code repositories, each of which requires inter-dependent changes
to many files (between 2-97 files). Coding tasks of this level of complexity
have not been automated using LLMs before. Our results show that CodePlan has
better match with the ground truth compared to baselines. CodePlan is able to
get 5/6 repositories to pass the validity checks (e.g., to build without errors
and make correct code edits) whereas the baselines (without planning but with
the same type of contextual information as CodePlan) cannot get any of the
repositories to pass them.
Boolformer: Symbolic Regression of Logic Functions with Transformers
Authors: name: Stéphane d'Ascoli, name: Samy Bengio, name: Josh Susskind, name: Emmanuel Abbé
Published: 2023-09-21 16:11:38

In this work, we introduce Boolformer, the first Transformer architecture
trained to perform end-to-end symbolic regression of Boolean functions. First,
we show that it can predict compact formulas for complex functions which were
not seen during training, when provided a clean truth table. Then, we
demonstrate its ability to find approximate expressions when provided
incomplete and noisy observations. We evaluate the Boolformer on a broad set of
real-world binary classification datasets, demonstrating its potential as an
interpretable alternative to classic machine learning methods. Finally, we
apply it to the widespread task of modelling the dynamics of gene regulatory
networks. Using a recent benchmark, we show that Boolformer is competitive with
state-of-the art genetic algorithms with a speedup of several orders of
magnitude. Our code and models are available publicly.
RMT: Retentive Networks Meet Vision Transformers
Authors: name: Qihang Fan, name: Huaibo Huang, name: Mingrui Chen, name: Hongmin Liu, name: Ran He
Published: 2023-09-20 00:57:48

Transformer first appears in the field of natural language processing and is
later migrated to the computer vision domain, where it demonstrates excellent
performance in vision tasks. However, recently, Retentive Network (RetNet) has
emerged as an architecture with the potential to replace Transformer,
attracting widespread attention in the NLP community. Therefore, we raise the
question of whether transferring RetNet's idea to vision can also bring
outstanding performance to vision tasks. To address this, we combine RetNet and
Transformer to propose RMT. Inspired by RetNet, RMT introduces explicit decay
into the vision backbone, bringing prior knowledge related to spatial distances
to the vision model. This distance-related spatial prior allows for explicit
control of the range of tokens that each token can attend to. Additionally, to
reduce the computational cost of global modeling, we decompose this modeling
process along the two coordinate axes of the image. Abundant experiments have
demonstrated that our RMT exhibits exceptional performance across various
computer vision tasks. For example, RMT achieves 84.1% Top1-acc on ImageNet-1k
using merely 4.5G FLOPs. To the best of our knowledge, among all models, RMT
achieves the highest Top1-acc when models are of similar size and trained with
the same strategy. Moreover, RMT significantly outperforms existing vision
backbones in downstream tasks such as object detection, instance segmentation,
and semantic segmentation. Our work is still in progress.
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language
  Model as an Agent
Authors: name: Jianing Yang, name: Xuweiyi Chen, name: Shengyi Qian, name: Nikhil Madaan, name: Madhavan Iyengar, name: David F. Fouhey, name: Joyce Chai
Published: 2023-09-21 17:59:45

3D visual grounding is a critical skill for household robots, enabling them
to navigate, manipulate objects, and answer questions based on their
environment. While existing approaches often rely on extensive labeled data or
exhibit limitations in handling complex language queries, we propose
LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model
(LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to
decompose complex natural language queries into semantic constituents and
employs a visual grounding tool, such as OpenScene or LERF, to identify objects
in a 3D scene. The LLM then evaluates the spatial and commonsense relations
among the proposed objects to make a final grounding decision. Our method does
not require any labeled training data and can generalize to novel 3D scenes and
arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and
demonstrate state-of-the-art zero-shot grounding accuracy. Our findings
indicate that LLMs significantly improve the grounding capability, especially
for complex language queries, making LLM-Grounder an effective approach for 3D
vision-language tasks in robotics. Videos and interactive demos can be found on
the project website https://chat-with-nerf.github.io/ .
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Authors: name: Yukang Chen, name: Shengju Qian, name: Haotian Tang, name: Xin Lai, name: Zhijian Liu, name: Song Han, name: Jiaya Jia
Published: 2023-09-21 17:59:11

We present LongLoRA, an efficient fine-tuning approach that extends the
context sizes of pre-trained large language models (LLMs), with limited
computation cost. Typically, training LLMs with long context sizes is
computationally expensive, requiring extensive training hours and GPU
resources. For example, training on the context length of 8192 needs 16x
computational costs in self-attention layers as that of 2048. In this paper, we
speed up the context extension of LLMs in two aspects. On the one hand,
although dense global attention is needed during inference, fine-tuning the
model can be effectively and efficiently done by sparse local attention. The
proposed shift short attention effectively enables context extension, leading
to non-trivial computation saving with similar performance to fine-tuning with
vanilla attention. Particularly, it can be implemented with only two lines of
code in training, while being optional in inference. On the other hand, we
revisit the parameter-efficient fine-tuning regime for context expansion.
Notably, we find that LoRA for context extension works well under the premise
of trainable embedding and normalization. LongLoRA demonstrates strong
empirical results on various tasks on LLaMA2 models from 7B/13B to 70B.
LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a
single 8x A100 machine. LongLoRA extends models' context while retaining their
original architectures, and is compatible with most existing techniques, like
FlashAttention-2. In addition, to make LongLoRA practical, we collect a
dataset, LongQA, for supervised fine-tuning. It contains more than 3k long
context question-answer pairs.
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language
  Models
Authors: name: Longhui Yu, name: Weisen Jiang, name: Han Shi, name: Jincheng Yu, name: Zhengying Liu, name: Yu Zhang, name: James T. Kwok, name: Zhenguo Li, name: Adrian Weller, name: Weiyang Liu
Published: 2023-09-21 17:45:42

Large language models (LLMs) have pushed the limits of natural language
understanding and exhibited excellent problem-solving ability. Despite the
great success, most existing open-source LLMs (\eg, LLaMA-2) are still far away
from satisfactory for solving mathematical problem due to the complex reasoning
procedures. To bridge this gap, we propose MetaMath, a fine-tuned
language model that specializes in mathematical reasoning. Specifically, we
start by bootstrapping mathematical questions by rewriting the question from
multiple perspectives without extra knowledge, which results in a new dataset
called {MetaMathQA}. Then we fine-tune the LLaMA-2 models on MetaMathQA.
Experimental results on two popular benchmarks (\ie, GSM8K and MATH) for
mathematical reasoning demonstrate that MetaMath outperforms a suite of
open-source LLMs by a significant margin. Our MetaMath-7B model achieves
66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models
of the same size by 11.5% and 8.7%. Particularly, {MetaMath-70B} achieves
an accuracy of 82.3% on {GSM8K}, slightly better than {GPT-3.5-Turbo}. We
release the {MetaMathQA} dataset, the {MetaMath} models with different model
sizes and the training code for public use.
A Paradigm Shift in Machine Translation: Boosting Translation
  Performance of Large Language Models
Authors: name: Haoran Xu, name: Young Jin Kim, name: Amr Sharaf, name: Hany Hassan Awadalla
Published: 2023-09-20 22:53:15

Generative Large Language Models (LLMs) have achieved remarkable advancements
in various NLP tasks. However, these advances have not been reflected in the
translation task, especially those with moderate model sizes (i.e., 7B or 13B
parameters), which still lag behind conventional supervised encoder-decoder
translation models. Previous studies have attempted to improve the translation
capabilities of these moderate LLMs, but their gains have been limited. In this
study, we propose a novel fine-tuning approach for LLMs that is specifically
designed for the translation task, eliminating the need for the abundant
parallel data that traditional translation models usually depend on. Our
approach consists of two fine-tuning stages: initial fine-tuning on monolingual
data followed by subsequent fine-tuning on a small set of high-quality parallel
data. We introduce the LLM developed through this strategy as Advanced Language
Model-based trAnslator (ALMA). Based on LLaMA-2 as our underlying model, our
results show that the model can achieve an average improvement of more than 12
BLEU and 12 COMET over its zero-shot performance across 10 translation
directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test
datasets. The performance is significantly better than all prior work and even
superior to the NLLB-54B model and GPT-3.5-text-davinci-003, with only 7B or
13B parameters. This method establishes the foundation for a novel training
paradigm in machine translation.
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
Authors: name: Nolan Dey, name: Daria Soboleva, name: Faisal Al-Khateeb, name: Bowen Yang, name: Ribhu Pathria, name: Hemant Khachane, name: Shaheer Muhammad, name: Zhiming, name: Chen, name: Robert Myers, name: Jacob Robert Steeves, name: Natalia Vassilieva, name: Marvin Tom, name: Joel Hestness
Published: 2023-09-20 18:12:56

We introduce the Bittensor Language Model, called "BTLM-3B-8K", a new
state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was
trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and
8,192 context lengths. BTLM-3B-8K outperforms all existing 3B parameter models
by 2-5.5% across downstream tasks. BTLM-3B-8K is even competitive with some 7B
parameter models. Additionally, BTLM-3B-8K provides excellent long context
performance, outperforming MPT-7B-8K and XGen-7B-8K on tasks up to 8,192
context length. We trained the model on a cleaned and deduplicated SlimPajama
dataset; aggressively tuned the \textmu P hyperparameters and schedule; used
ALiBi position embeddings; and adopted the SwiGLU nonlinearity.
  On Hugging Face, the most popular models have 7B parameters, indicating that
users prefer the quality-size ratio of 7B models. Compacting the 7B parameter
model to one with 3B parameters, with little performance impact, is an
important milestone. BTLM-3B-8K needs only 3GB of memory with 4-bit precision
and takes 2.5x less inference compute than 7B models, helping to open up access
to a powerful language model on mobile and edge devices. BTLM-3B-8K is
available under an Apache 2.0 license on Hugging Face:
https://huggingface.co/cerebras/btlm-3b-8k-base.
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
Authors: name: Lianmin Zheng, name: Wei-Lin Chiang, name: Ying Sheng, name: Tianle Li, name: Siyuan Zhuang, name: Zhanghao Wu, name: Yonghao Zhuang, name: Zhuohan Li, name: Zi Lin, name: Eric. P Xing, name: Joseph E. Gonzalez, name: Ion Stoica, name: Hao Zhang
Published: 2023-09-21 12:13:55

Studying how people interact with large language models (LLMs) in real-world
scenarios is increasingly important due to their widespread use in various
applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset
containing one million real-world conversations with 25 state-of-the-art LLMs.
This dataset is collected from 210K unique IP addresses in the wild on our
Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's
content, including its curation process, basic statistics, and topic
distribution, highlighting its diversity, originality, and scale. We
demonstrate its versatility through four use cases: developing content
moderation models that perform similarly to GPT-4, building a safety benchmark,
training instruction-following models that perform similarly to Vicuna, and
creating challenging benchmark questions. We believe that this dataset will
serve as a valuable resource for understanding and advancing LLM capabilities.
The dataset is publicly available at
https://huggingface.co/datasets/lmsys/lmsys-chat-1m.
The Languini Kitchen: Enabling Language Modelling Research at Different
  Scales of Compute
Authors: name: Aleksandar Stanić, name: Dylan Ashley, name: Oleg Serikov, name: Louis Kirsch, name: Francesco Faccio, name: Jürgen Schmidhuber, name: Thomas Hofmann, name: Imanol Schlag
Published: 2023-09-20 10:31:17

The Languini Kitchen serves as both a research collective and codebase
designed to empower researchers with limited computational resources to
contribute meaningfully to the field of language modelling. We introduce an
experimental protocol that enables model comparisons based on equivalent
compute, measured in accelerator hours. The number of tokens on which a model
is trained is defined by the model's throughput and the chosen compute class.
Notably, this approach avoids constraints on critical hyperparameters which
affect total parameters or floating-point operations. For evaluation, we
pre-process an existing large, diverse, and high-quality dataset of books that
surpasses existing academic benchmarks in quality, diversity, and document
length. On it, we compare methods based on their empirical scaling trends which
are estimated through experiments at various levels of compute. This work also
provides two baseline models: a feed-forward model derived from the GPT-2
architecture and a recurrent model in the form of a novel LSTM with ten-fold
throughput. While the GPT baseline achieves better perplexity throughout all
our levels of compute, our LSTM baseline exhibits a predictable and more
favourable scaling law. This is due to the improved throughput and the need for
fewer training tokens to achieve the same decrease in test perplexity.
Extrapolating the scaling laws leads of both models results in an intersection
at roughly 50,000 accelerator hours. We hope this work can serve as the
foundation for meaningful and reproducible language modelling research.
A Large-scale Dataset for Audio-Language Representation Learning
Authors: name: Luoyi Sun, name: Xuenan Xu, name: Mengyue Wu, name: Weidi Xie
Published: 2023-09-20 17:59:32

The AI community has made significant strides in developing powerful
foundation models, driven by large-scale multimodal datasets. However, in the
audio representation learning community, the present audio-language datasets
suffer from limitations such as insufficient volume, simplistic content, and
arduous collection procedures. To tackle these challenges, we present an
innovative and automatic audio caption generation pipeline based on a series of
public tools or APIs, and construct a large-scale, high-quality, audio-language
dataset, named as Auto-ACD, comprising over 1.9M audio-text pairs. To
demonstrate the effectiveness of the proposed dataset, we train popular models
on our dataset and show performance improvement on various downstream tasks,
namely, audio-language retrieval, audio captioning, environment classification.
In addition, we establish a novel test set and provide a benchmark for
audio-text tasks. The proposed dataset will be released at
https://auto-acd.github.io/.
Controllable Dynamic Appearance for Neural 3D Portraits
Authors: name: ShahRukh Athar, name: Zhixin Shu, name: Zexiang Xu, name: Fuji Luan, name: Sai Bi, name: Kalyan Sunkavalli, name: Dimitris Samaras
Published: 2023-09-20 02:24:40

Recent advances in Neural Radiance Fields (NeRFs) have made it possible to
reconstruct and reanimate dynamic portrait scenes with control over head-pose,
facial expressions and viewing direction. However, training such models assumes
photometric consistency over the deformed region e.g. the face must be evenly
lit as it deforms with changing head-pose and facial expression. Such
photometric consistency across frames of a video is hard to maintain, even in
studio environments, thus making the created reanimatable neural portraits
prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a
system that enables the creation of fully controllable 3D portraits in
real-world capture conditions. CoDyNeRF learns to approximate illumination
dependent effects via a dynamic appearance model in the canonical space that is
conditioned on predicted surface normals and the facial expressions and
head-pose deformations. The surface normals prediction is guided using 3DMM
normals that act as a coarse prior for the normals of the human head, where
direct prediction of normals is hard due to rigid and non-rigid deformations
induced by head-pose and facial expression changes. Using only a
smartphone-captured short video of a subject for training, we demonstrate the
effectiveness of our method on free view synthesis of a portrait scene with
explicit head pose and expression controls, and realistic lighting effects. The
project page can be found here:
http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html