Orionnnn/hub-stats huggingface dataset:
Universal Source Separation with Weakly Labelled Data
Authors: name: Qiuqiang Kong, name: Ke Chen, name: Haohe Liu, name: Xingjian Du, name: Taylor Berg-Kirkpatrick, name: Shlomo Dubnov, name: Mark D. Plumbley
Published: 2023-05-11 16:41:55

Universal source separation (USS) is a fundamental research task for
computational auditory scene analysis, which aims to separate mono recordings
into individual source tracks. There are three potential challenges awaiting
the solution to the audio source separation task. First, previous audio source
separation systems mainly focus on separating one or a limited number of
specific sources. There is a lack of research on building a unified system that
can separate arbitrary sources via a single model. Second, most previous
systems require clean source data to train a separator, while clean source data
are scarce. Third, there is a lack of USS system that can automatically detect
and separate active sound classes in a hierarchical level. To use large-scale
weakly labeled/unlabeled audio data for audio source separation, we propose a
universal audio source separation framework containing: 1) an audio tagging
model trained on weakly labeled data as a query net; and 2) a conditional
source separation model that takes query net outputs as conditions to separate
arbitrary sound sources. We investigate various query nets, source separation
models, and training strategies and propose a hierarchical USS strategy to
automatically detect and separate sound classes from the AudioSet ontology. By
solely leveraging the weakly labelled AudioSet, our USS system is successful in
separating a wide variety of sound classes, including sound event separation,
music source separation, and speech enhancement. The USS system achieves an
average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over 527 sound
classes of AudioSet; 10.57 dB on the DCASE 2018 Task 2 dataset; 8.12 dB on the
MUSDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of
9.00 dB on the voicebank-demand dataset. We release the source code at
https://github.com/bytedance/uss
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
Authors: name: Xinyu Gong, name: Sreyas Mohan, name: Naina Dhingra, name: Jean-Charles Bazin, name: Yilei Li, name: Zhangyang Wang, name: Rakesh Ranjan
Published: 2023-05-12 03:05:40

In this paper, we study a novel problem in egocentric action recognition,
which we term as "Multimodal Generalization" (MMG). MMG aims to study how
systems can generalize when data from certain modalities is limited or even
completely missing. We thoroughly investigate MMG in the context of standard
supervised action recognition and the more challenging few-shot setting for
learning new action categories. MMG consists of two novel scenarios, designed
to support security, and efficiency considerations in real-world applications:
(1) missing modality generalization where some modalities that were present
during the train time are missing during the inference time, and (2)
cross-modal zero-shot generalization, where the modalities present during the
inference time and the training time are disjoint. To enable this
investigation, we construct a new dataset MMG-Ego4D containing data points with
video, audio, and inertial motion sensor (IMU) modalities. Our dataset is
derived from Ego4D dataset, but processed and thoroughly re-annotated by human
experts to facilitate research in the MMG problem. We evaluate a diverse array
of models on MMG-Ego4D and propose new methods with improved generalization
ability. In particular, we introduce a new fusion module with modality dropout
training, contrastive-based alignment training, and a novel cross-modal
prototypical loss for better few-shot performance. We hope this study will
serve as a benchmark and guide future research in multimodal generalization
problems. The benchmark and code will be available at
https://github.com/facebookresearch/MMG_Ego4D.
Surfacing Biases in Large Language Models using Contrastive Input
  Decoding
Authors: name: Gal Yona, name: Or Honovich, name: Itay Laish, name: Roee Aharoni
Published: 2023-05-12 11:09:49

Ensuring that large language models (LMs) are fair, robust and useful
requires an understanding of how different modifications to their inputs impact
the model's behaviour. In the context of open-text generation tasks, however,
such an evaluation is not trivial. For example, when introducing a model with
an input text and a perturbed, "contrastive" version of it, meaningful
differences in the next-token predictions may not be revealed with standard
decoding strategies. With this motivation in mind, we propose Contrastive Input
Decoding (CID): a decoding algorithm to generate text given two inputs, where
the generated text is likely given one input but unlikely given the other. In
this way, the contrastive generations can highlight potentially subtle
differences in how the LM output differs for the two inputs in a simple and
interpretable manner. We use CID to highlight context-specific biases that are
hard to detect with standard decoding strategies and quantify the effect of
different input perturbations.
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Authors: name: Lili Yu, name: Dániel Simig, name: Colin Flaherty, name: Armen Aghajanyan, name: Luke Zettlemoyer, name: Mike Lewis
Published: 2023-05-12 00:55:41

Autoregressive transformers are spectacular models for short sequences but
scale poorly to long sequences such as high-resolution images, podcasts, code,
or books. We proposed Megabyte, a multi-scale decoder architecture that enables
end-to-end differentiable modeling of sequences of over one million bytes.
Megabyte segments sequences into patches and uses a local submodel within
patches and a global model between patches. This enables sub-quadratic
self-attention, much larger feedforward layers for the same compute, and
improved parallelism during decoding -- unlocking better performance at reduced
cost for both training and generation. Extensive experiments show that Megabyte
allows byte-level models to perform competitively with subword models on long
context language modeling, achieve state-of-the-art density estimation on
ImageNet, and model audio from raw files. Together, these results establish the
viability of tokenization-free autoregressive sequence modeling at scale.
ArtGPT-4: Artistic Vision-Language Understanding with Adapter-enhanced
  MiniGPT-4
Authors: name: Zhengqing Yuan, name: Huiwen Xue, name: Xinyi Wang, name: Yongming Liu, name: Zhuanzhe Zhao, name: Kun Wang
Published: 2023-05-12 14:04:30

In recent years, large language models (LLMs) have made significant progress
in natural language processing (NLP), with models like ChatGPT and GPT-4
achieving impressive capabilities in various linguistic tasks. However,
training models on such a large scale is challenging, and finding datasets that
match the model's scale is often difficult. Fine-tuning and training models
with fewer parameters using novel methods have emerged as promising approaches
to overcome these challenges. One such model is MiniGPT-4, which achieves
comparable vision-language understanding to GPT-4 by leveraging novel
pre-training models and innovative training strategies. However, the model
still faces some challenges in image understanding, particularly in artistic
pictures. A novel multimodal model called ArtGPT-4 has been proposed to address
these limitations. ArtGPT-4 was trained on image-text pairs using a Tesla A100
device in just 2 hours, using only about 200 GB of data. The model can depict
images with an artistic flair and generate visual code, including aesthetically
pleasing HTML/CSS web pages. Furthermore, the article proposes novel benchmarks
for evaluating the performance of vision-language models. In the subsequent
evaluation methods, ArtGPT-4 scored more than 1 point higher than the current
state-of-the-art model and was only 0.25 points lower than artists on
a 6-point scale. Our code and pre-trained model are available at
https://huggingface.co/Tyrannosaurus/ArtGPT-4.
Better speech synthesis through scaling
Authors: name: James Betker
Published: 2023-05-12 04:19:49

In recent years, the field of image generation has been revolutionized by the
application of autoregressive transformers and DDPMs. These approaches model
the process of image generation as a step-wise probabilistic processes and
leverage large amounts of compute and data to learn the image distribution.
This methodology of improving performance need not be confined to images. This
paper describes a way to apply advances in the image generative domain to
speech synthesis. The result is TorToise -- an expressive, multi-voice
text-to-speech system.
  All model code and trained weights have been open-sourced at
https://github.com/neonbjb/tortoise-tts.
An Inverse Scaling Law for CLIP Training
Authors: name: Xianhang Li, name: Zeyu Wang, name: Cihang Xie
Published: 2023-05-11 17:56:09

CLIP, the first foundation model that connects images and text, has enabled
many recent breakthroughs in computer vision. However, its associated training
cost is prohibitively high, imposing a significant barrier to its widespread
exploration. In this paper, we present a surprising finding that there exists
an inverse scaling law for CLIP training, whereby the larger the image/text
encoders used, the shorter the sequence length of image/text tokens that can be
applied in training. Moreover, we showcase that the strategy for reducing
image/text token length plays a crucial role in determining the quality of this
scaling law.
  As a result of this finding, we are able to successfully train CLIP even by
using academic resources. For example, on an A100 eight-GPU server, our CLIP
models achieve zero-shot top-1 ImageNet accuracies of 63.2% in ~2 days, 67.8%
in ~3 days, and 69.3% in ~4 days. By reducing the computation barrier
associated with CLIP, we hope to inspire more research in this field,
particularly from academics. Our code is available at
https://github.com/UCSC-VLAA/CLIPA.
Perpetual Humanoid Control for Real-time Simulated Avatars
Authors: name: Zhengyi Luo, name: Jinkun Cao, name: Alexander Winkler, name: Kris Kitani, name: Weipeng Xu
Published: 2023-05-10 20:51:37

We present a physics-based humanoid controller that achieves high-fidelity
motion imitation and fault-tolerant behavior in the presence of noisy input
(e.g. pose estimates from video or generated from language) and unexpected
falls. Our controller scales up to learning ten thousand motion clips without
using any external stabilizing forces and learns to naturally recover from
fail-state. Given reference motion, our controller can perpetually control
simulated avatars without requiring resets. At its core, we propose the
progressive multiplicative control policy (PMCP), which dynamically allocates
new network capacity to learn harder and harder motion sequences. PMCP allows
efficient scaling for learning from large-scale motion databases and adding new
tasks, such as fail-state recovery, without catastrophic forgetting. We
demonstrate the effectiveness of our controller by using it to imitate noisy
poses from video-based pose estimators and language-based motion generators in
a live and real-time multi-person avatar use case.
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group
  Attention
Authors: name: Xinyu Liu, name: Houwen Peng, name: Ningxin Zheng, name: Yuqing Yang, name: Han Hu, name: Yixuan Yuan
Published: 2023-05-11 17:59:41

Vision transformers have shown great success due to their high model
capabilities. However, their remarkable performance is accompanied by heavy
computation costs, which makes them unsuitable for real-time applications. In
this paper, we propose a family of high-speed vision transformers named
EfficientViT. We find that the speed of existing transformer models is commonly
bounded by memory inefficient operations, especially the tensor reshaping and
element-wise functions in MHSA. Therefore, we design a new building block with
a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN
layers, which improves memory efficiency while enhancing channel communication.
Moreover, we discover that the attention maps share high similarities across
heads, leading to computational redundancy. To address this, we present a
cascaded group attention module feeding attention heads with different splits
of the full feature, which not only saves computation cost but also improves
attention diversity. Comprehensive experiments demonstrate EfficientViT
outperforms existing efficient models, striking a good trade-off between speed
and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by
1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia
V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient
model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while
running 5.8x/3.7x faster on the GPU/CPU, and 7.4x faster when converted to ONNX
format. Code and models are available at
https://github.com/microsoft/Cream/tree/main/EfficientViT.
Region-Aware Pretraining for Open-Vocabulary Object Detection with
  Vision Transformers
Authors: name: Dahun Kim, name: Anelia Angelova, name: Weicheng Kuo
Published: 2023-05-11 17:53:29

We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a
contrastive image-text pretraining recipe to bridge the gap between image-level
pretraining and open-vocabulary object detection. At the pretraining phase, we
propose to randomly crop and resize regions of positional embeddings instead of
using the whole image positional embeddings. This better matches the use of
positional embeddings at region-level in the detection finetuning phase. In
addition, we replace the common softmax cross entropy loss in contrastive
learning with focal loss to better learn the informative yet difficult
examples. Finally, we leverage recent advances in novel object proposals to
improve open-vocabulary detection finetuning. We evaluate our full model on the
LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer.
RO-ViT achieves a state-of-the-art 32.1 AP_r on LVIS, surpassing the best
existing approach by +5.8 points in addition to competitive zero-shot transfer
detection. Surprisingly, RO-ViT improves the image-level representation as well
and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr
image-text retrieval benchmarks, outperforming competitive approaches with
larger models.
Simple Token-Level Confidence Improves Caption Correctness
Authors: name: Suzanne Petryk, name: Spencer Whitehead, name: Joseph E. Gonzalez, name: Trevor Darrell, name: Anna Rohrbach, name: Marcus Rohrbach
Published: 2023-05-11 17:58:17

The ability to judge whether a caption correctly describes an image is a
critical part of vision-language understanding. However, state-of-the-art
models often misinterpret the correctness of fine-grained details, leading to
errors in outputs such as hallucinating objects in generated captions or poor
compositional reasoning. In this work, we explore Token-Level Confidence, or
TLC, as a simple yet surprisingly effective method to assess caption
correctness. Specifically, we fine-tune a vision-language model on image
captioning, input an image and proposed caption to the model, and aggregate
either algebraic or learned token confidences over words or sequences to
estimate image-caption consistency. Compared to sequence-level scores from
pretrained models, TLC with algebraic confidence measures achieves a relative
improvement in accuracy by 10% on verb understanding in SVO-Probes and
outperforms prior state-of-the-art in image and group scores for compositional
reasoning in Winoground by a relative 37% and 9%, respectively. When training
data are available, a learned confidence estimator provides further improved
performance, reducing object hallucination rates in MS COCO Captions by a
relative 30% over the original model and setting a new state-of-the-art.
Not All Languages Are Created Equal in LLMs: Improving Multilingual
  Capability by Cross-Lingual-Thought Prompting
Authors: name: Haoyang Huang, name: Tianyi Tang, name: Dongdong Zhang, name: Wayne Xin Zhao, name: Ting Song, name: Yan Xia, name: Furu Wei
Published: 2023-05-11 17:44:17

Large language models (LLMs) demonstrate impressive multilingual capability,
but their performance varies substantially across different languages. In this
work, we introduce a simple yet effective method, called cross-lingual-thought
prompting (XLT), to systematically improve the multilingual capability of LLMs.
Specifically, XLT is a generic template prompt that stimulates cross-lingual
and logical reasoning skills to enhance task performance across languages. We
conduct comprehensive evaluations on 7 typical benchmarks related to reasoning,
understanding, and generation tasks, covering both high-resource and
low-resource languages. Experimental results show that XLT not only remarkably
enhances the performance of various multilingual tasks but also significantly
reduces the gap between the average performance and the best performance of
each task in different languages. Notably, XLT brings over 10 points of average
improvement in arithmetic reasoning and open-domain question-answering tasks.
Domain Incremental Lifelong Learning in an Open World
Authors: name: Yi Dai, name: Hao Lang, name: Yinhe Zheng, name: Bowen Yu, name: Fei Huang, name: Yongbin Li
Published: 2023-05-11 04:19:08

Lifelong learning (LL) is an important ability for NLP models to learn new
tasks continuously. Architecture-based approaches are reported to be effective
implementations for LL models. However, it is non-trivial to extend previous
approaches to domain incremental LL scenarios since they either require access
to task identities in the testing phase or cannot handle samples from unseen
tasks. In this paper, we propose Diana: a
dynamic architecture-based
lifelong learning model that tries to learn a sequence
of tasks with a prompt-enhanced language model. Four types of hierarchically
organized prompts are used in Diana to capture knowledge from different
granularities. Specifically, we dedicate task-level prompts to capture
task-specific knowledge to retain high LL performances and maintain
instance-level prompts to learn knowledge shared across input samples to
improve the model's generalization performance. Moreover, we dedicate separate
prompts to explicitly model unseen tasks and introduce a set of prompt key
vectors to facilitate knowledge sharing between tasks. Extensive experiments
demonstrate that Diana outperforms state-of-the-art LL models, especially in
handling unseen tasks. We release the code and data at
https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/diana.
InstructBLIP: Towards General-purpose Vision-Language Models with
  Instruction Tuning
Authors: name: Wenliang Dai, name: Junnan Li, name: Dongxu Li, name: Anthony Meng Huat Tiong, name: Junqi Zhao, name: Weisheng Wang, name: Boyang Li, name: Pascale Fung, name: Steven Hoi
Published: 2023-05-11 00:38:10

General-purpose language models that can solve various language-domain tasks
have emerged driven by the pre-training and instruction-tuning pipeline.
However, building general-purpose vision-language models is challenging due to
the increased task discrepancy introduced by the additional visual input.
Although vision-language pre-training has been widely studied, vision-language
instruction tuning remains relatively less explored. In this paper, we conduct
a systematic and comprehensive study on vision-language instruction tuning
based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly
available datasets, transform them into instruction tuning format and
categorize them into two clusters for held-in instruction tuning and held-out
zero-shot evaluation. Additionally, we introduce instruction-aware visual
feature extraction, a crucial method that enables the model to extract
informative features tailored to the given instruction. The resulting
InstructBLIP models achieve state-of-the-art zero-shot performance across all
13 held-out datasets, substantially outperforming BLIP-2 and the larger
Flamingo. Our models also lead to state-of-the-art performance when finetuned
on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG).
Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over
concurrent multimodal models. All InstructBLIP models have been open-sourced at
https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
Do LLMs Understand User Preferences? Evaluating LLMs On User Rating
  Prediction
Authors: name: Wang-Cheng Kang, name: Jianmo Ni, name: Nikhil Mehta, name: Maheswaran Sathiamoorthy, name: Lichan Hong, name: Ed Chi, name: Derek Zhiyuan Cheng
Published: 2023-05-10 21:43:42

Large Language Models (LLMs) have demonstrated exceptional capabilities in
generalizing to new tasks in a zero-shot or few-shot manner. However, the
extent to which LLMs can comprehend user preferences based on their previous
behavior remains an emerging and still unclear research question.
Traditionally, Collaborative Filtering (CF) has been the most effective method
for these tasks, predominantly relying on the extensive volume of rating data.
In contrast, LLMs typically demand considerably less data while maintaining an
exhaustive world knowledge about each item, such as movies or products. In this
paper, we conduct a thorough examination of both CF and LLMs within the classic
task of user rating prediction, which involves predicting a user's rating for a
candidate item based on their past ratings. We investigate various LLMs in
different sizes, ranging from 250M to 540B parameters and evaluate their
performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct
comprehensive analysis to compare between LLMs and strong CF methods, and find
that zero-shot LLMs lag behind traditional recommender models that have the
access to user interaction data, indicating the importance of user interaction
data. However, through fine-tuning, LLMs achieve comparable or even better
performance with only a small fraction of the training data, demonstrating
their potential through data efficiency.
LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits
  Siamese-BLOOM
Authors: name: Wen-Yu Hua, name: Brian Williams, name: Davood Shamsi
Published: 2023-05-10 18:26:42

Text embeddings are useful features for several NLP applications, such as
sentence similarity, text clustering, and semantic search. In this paper, we
present a Low-rank Adaptation with a Contrastive objective on top of 8-bit
Siamese-BLOOM, a multilingual large language model optimized to produce
semantically meaningful word embeddings. The innovation is threefold. First, we
cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable
adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification.
Third, we apply a Siamese architecture on BLOOM model with a contrastive
objective to ease the multi-lingual labeled data scarcity. The experiment
results show the quality of learned embeddings from LACoS-BLOOM is proportional
to the number of model parameters and the amount of unlabeled training data.
With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1
billion parameters end-to-end on a single GPU machine with 32GB memory.
Compared to previous solution Sentence-BERT, we achieve significant improvement
on both English and multi-lingual STS tasks.