Orionnnn/hub-stats huggingface dataset:
Simple and Controllable Music Generation
Authors: name: Jade Copet, name: Felix Kreuk, name: Itai Gat, name: Tal Remez, name: David Kant, name: Gabriel Synnaeve, name: Yossi Adi, name: Alexandre Défossez
Published: 2023-06-08 15:31:05

We tackle the task of conditional music generation. We introduce MusicGen, a
single Language Model (LM) that operates over several streams of compressed
discrete music representation, i.e., tokens. Unlike prior work, MusicGen is
comprised of a single-stage transformer LM together with efficient token
interleaving patterns, which eliminates the need for cascading several models,
e.g., hierarchically or upsampling. Following this approach, we demonstrate how
MusicGen can generate high-quality samples, while being conditioned on textual
description or melodic features, allowing better controls over the generated
output. We conduct extensive empirical evaluation, considering both automatic
and human studies, showing the proposed approach is superior to the evaluated
baselines on a standard text-to-music benchmark. Through ablation studies, we
shed light over the importance of each of the components comprising MusicGen.
Music samples, code, and models are available at
https://github.com/facebookresearch/audiocraft.
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with
  Architecture-Routed Mixture-of-Experts
Authors: name: Ganesh Jawahar, name: Haichuan Yang, name: Yunyang Xiong, name: Zechun Liu, name: Dilin Wang, name: Fei Sun, name: Meng Li, name: Aasish Pappu, name: Barlas Oguz, name: Muhammad Abdul-Mageed, name: Laks V. S. Lakshmanan, name: Raghuraman Krishnamoorthi, name: Vikas Chandra
Published: 2023-06-08 00:35:36

Weight-sharing supernet has become a vital component for performance
estimation in the state-of-the-art (SOTA) neural architecture search (NAS)
frameworks. Although supernet can directly generate different subnetworks
without retraining, there is no guarantee for the quality of these subnetworks
because of weight sharing. In NLP tasks such as machine translation and
pre-trained language modeling, we observe that given the same model
architecture, there is a large performance gap between supernet and training
from scratch. Hence, supernet cannot be directly used and retraining is
necessary after finding the optimal architectures.
  In this work, we propose mixture-of-supernets, a generalized supernet
formulation where mixture-of-experts (MoE) is adopted to enhance the expressive
power of the supernet model, with negligible training overhead. In this way,
different subnetworks do not share the model weights directly, but through an
architecture-based routing mechanism. As a result, model weights of different
subnetworks are customized towards their specific architectures and the weight
generation is learned by gradient descent. Compared to existing weight-sharing
supernet for NLP, our method can minimize the retraining time, greatly
improving training efficiency. In addition, the proposed method achieves the
SOTA performance in NAS for building fast machine translation models, yielding
better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT. We
also achieve the SOTA performance in NAS for building memory-efficient
task-agnostic BERT models, outperforming NAS-BERT and AutoDistil in various
model sizes.
Improving Open Language Models by Learning from Organic Interactions
Authors: name: Jing Xu, name: Da Ju, name: Joshua Lane, name: Mojtaba Komeili, name: Eric Michael Smith, name: Megan Ung, name: Morteza Behrooz, name: William Ngan, name: Rashel Moritz, name: Sainbayar Sukhbaatar, name: Y-Lan Boureau, name: Jason Weston, name: Kurt Shuster
Published: 2023-06-07 18:19:46

We present BlenderBot 3x, an update on the conversational model BlenderBot 3,
which is now trained using organic conversation and feedback data from
participating users of the system in order to improve both its skills and
safety. We are publicly releasing the participating de-identified interaction
data for use by the research community, in order to spur further progress.
Training models with organic data is challenging because interactions with
people "in the wild" include both high quality conversations and feedback, as
well as adversarial and toxic behavior. We study techniques that enable
learning from helpful teachers while avoiding learning from people who are
trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is
both preferred in conversation to BlenderBot 3, and is shown to produce safer
responses in challenging situations. While our current models are still far
from perfect, we believe further improvement can be achieved by continued use
of the techniques explored in this work.
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and
  Language Models
Authors: name: Muhammad Maaz, name: Hanoona Rasheed, name: Salman Khan, name: Fahad Shahbaz Khan
Published: 2023-06-08 17:59:56

Conversation agents fueled by Large Language Models (LLMs) are providing a
new way to interact with visual data. While there have been initial attempts
for image-based conversation models, this work addresses the underexplored
field of video-based conversation by introducing Video-ChatGPT. It is a
multimodal model that merges a video-adapted visual encoder with a LLM. The
model is capable of understanding and generating human-like conversations about
videos. We introduce a new dataset of 100,000 video-instruction pairs used to
train Video-ChatGPT acquired via manual and semi-automated pipeline that is
easily scalable and robust to label noise. We also develop a quantiative
evaluation framework for video-based dialogue models to objectively analyse the
strengths and weaknesses of proposed models. Our code, models, instruction-sets
and demo are released at https://github.com/mbzuai-oryx/Video-ChatGPT.
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning
  Optimization
Authors: name: Yidong Wang, name: Zhuohao Yu, name: Zhengran Zeng, name: Linyi Yang, name: Cunxiang Wang, name: Hao Chen, name: Chaoya Jiang, name: Rui Xie, name: Jindong Wang, name: Xing Xie, name: Wei Ye, name: Shikun Zhang, name: Yue Zhang
Published: 2023-06-08 10:41:56

Instruction tuning large language models (LLMs) remains a challenging task,
owing to the complexity of hyperparameter selection and the difficulty involved
in evaluating the tuned models. To determine the optimal hyperparameters, an
automatic, robust, and reliable evaluation benchmark is essential. However,
establishing such a benchmark is not a trivial task due to the challenges
associated with evaluation accuracy and privacy protection. In response to
these challenges, we introduce a judge large language model, named PandaLM,
which is trained to distinguish the superior model given several LLMs.
PandaLM's focus extends beyond just the objective correctness of responses,
which is the main focus of traditional evaluation datasets. It addresses vital
subjective factors such as relative conciseness, clarity, adherence to
instructions, comprehensiveness, and formality. To ensure the reliability of
PandaLM, we collect a diverse human-annotated test dataset, where all contexts
are generated by humans and labels are aligned with human preferences. Our
results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation
ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM
enables the evaluation of LLM to be fairer but with less cost, evidenced by
significant improvements achieved by models tuned through PandaLM compared to
their counterparts trained with default Alpaca's hyperparameters. In addition,
PandaLM does not depend on API-based evaluations, thus avoiding potential data
leakage. All resources of PandaLM are released at
https://github.com/WeOpenML/PandaLM.
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large
  Language Models
Authors: name: Yew Ken Chia, name: Pengfei Hong, name: Lidong Bing, name: Soujanya Poria
Published: 2023-06-07 20:12:29

Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as
conversational agents. These models, such as GPT-4, can not only master
language but also solve complex tasks in areas like mathematics, coding,
medicine, and law. Despite their impressive capabilities, there is still a lack
of comprehensive understanding regarding their full potential, primarily due to
the black-box nature of many models and the absence of holistic evaluation
studies. To address these challenges, we present INSTRUCTEVAL, a more
comprehensive evaluation suite designed specifically for instruction-tuned
large language models. Unlike previous works, our evaluation involves a
rigorous assessment of models based on problem-solving, writing ability, and
alignment to human values. We take a holistic approach to analyze various
factors affecting model performance, including the pretraining foundation,
instruction-tuning data, and training methods. Our findings reveal that the
quality of instruction data is the most crucial factor in scaling model
performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment. We
are encouraged by the rapid development of models by the open-source community,
but we also highlight the need for rigorous evaluation to support claims made
about these models. Through INSTRUCTEVAL, we aim to foster a deeper
understanding of instruction-tuned models and advancements in their
capabilities. INSTRUCTEVAL is publicly available at
https://github.com/declare-lab/instruct-eval.
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
  Resources
Authors: name: Yizhong Wang, name: Hamish Ivison, name: Pradeep Dasigi, name: Jack Hessel, name: Tushar Khot, name: Khyathi Raghavi Chandu, name: David Wadden, name: Kelsey MacMillan, name: Noah A. Smith, name: Iz Beltagy, name: Hannaneh Hajishirzi
Published: 2023-06-07 19:59:23

In this work we explore recent advances in instruction-tuning language models
on a range of open instruction-following datasets. Despite recent claims that
open models can be on par with state-of-the-art proprietary models, these
claims are often accompanied by limited evaluation, making it difficult to
compare models across the board and determine the utility of various resources.
We provide a large set of instruction-tuned models from 6.7B to 65B parameters
in size, trained on 12 instruction datasets ranging from manually curated
(e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and
systematically evaluate them on their factual knowledge, reasoning,
multilinguality, coding, and open-ended instruction following abilities through
a collection of automatic, model-based, and human-based metrics. We further
introduce T\"ulu, our best performing instruction-tuned model suite finetuned
on a combination of high-quality open resources.
  Our experiments show that different instruction-tuning datasets can uncover
or enhance specific skills, while no single dataset (or combination) provides
the best performance across all evaluations. Interestingly, we find that model
and human preference-based evaluations fail to reflect differences in model
capabilities exposed by benchmark-based evaluations, suggesting the need for
the type of systemic evaluation performed in this work. Our evaluations show
that the best model in any given evaluation reaches on average 83% of ChatGPT
performance, and 68% of GPT-4 performance, suggesting that further investment
in building better base models and instruction-tuning data is required to close
the gap. We release our instruction-tuned models, including a fully finetuned
65B T\"ulu, along with our code, data, and evaluation framework at
https://github.com/allenai/open-instruct to facilitate future research.
MobileNMT: Enabling Translation in 15MB and 30ms
Authors: name: Ye Lin, name: Xiaohui Wang, name: Zhexi Zhang, name: Mingxuan Wang, name: Tong Xiao, name: Jingbo Zhu
Published: 2023-06-07 08:25:51

Deploying NMT models on mobile devices is essential for privacy, low latency,
and offline scenarios. For high model capacity, NMT models are rather large.
Running these models on devices is challenging with limited storage, memory,
computation, and power consumption. Existing work either only focuses on a
single metric such as FLOPs or general engine which is not good at
auto-regressive decoding. In this paper, we present MobileNMT, a system that
can translate in 15MB and 30ms on devices. We propose a series of principles
for model compression when combined with quantization. Further, we implement an
engine that is friendly to INT8 and decoding. With the co-design of model and
engine, compared with the existing system, we speed up 47.0x and save 99.5% of
memory with only 11.6% loss of BLEU. The code is publicly available at
https://github.com/zjersey/Lightseq-ARM.
LLMZip: Lossless Text Compression using Large Language Models
Authors: name: Chandra Shekhara Kaushik Valmeekam, name: Krishna Narayanan, name: Dileep Kalathil, name: Jean-Francois Chamberland, name: Srinivas Shakkottai
Published: 2023-06-06 22:42:00

We provide new estimates of an asymptotic upper bound on the entropy of
English using the large language model LLaMA-7B as a predictor for the next
token given a window of past tokens. This estimate is significantly smaller
than currently available estimates in cover1978convergent,
lutati2023focus. A natural byproduct is an algorithm for lossless
compression of English text which combines the prediction from the large
language model with a lossless compression scheme. Preliminary results from
limited experiments suggest that our scheme outperforms state-of-the-art text
compression schemes such as BSC, ZPAQ, and paq8h.
ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image
  Collections
Authors: name: Chun-Han Yao, name: Amit Raj, name: Wei-Chih Hung, name: Yuanzhen Li, name: Michael Rubinstein, name: Ming-Hsuan Yang, name: Varun Jampani
Published: 2023-06-07 17:47:50

Estimating 3D articulated shapes like animal bodies from monocular images is
inherently challenging due to the ambiguities of camera viewpoint, pose,
texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to
reconstruct per-instance 3D shapes from a sparse image collection in-the-wild.
Specifically, ARTIC3D is built upon a skeleton-based surface representation and
is further guided by 2D diffusion priors from Stable Diffusion. First, we
enhance the input images with occlusions/truncation via 2D diffusion to obtain
cleaner mask estimates and semantic features. Second, we perform
diffusion-guided 3D optimization to estimate shape and texture that are of
high-fidelity and faithful to input images. We also propose a novel technique
to calculate more stable image-level gradients via diffusion models compared to
existing alternatives. Finally, we produce realistic animations by fine-tuning
the rendered shape and texture under rigid part transformations. Extensive
evaluations on multiple existing datasets as well as newly introduced noisy web
image collections with occlusions and truncation demonstrate that ARTIC3D
outputs are more robust to noisy images, higher quality in terms of shape and
texture details, and more realistic when animated. Project page:
https://chhankyao.github.io/artic3d/
Designing a Better Asymmetric VQGAN for StableDiffusion
Authors: name: Zixin Zhu, name: Xuelu Feng, name: Dongdong Chen, name: Jianmin Bao, name: Le Wang, name: Yinpeng Chen, name: Lu Yuan, name: Gang Hua
Published: 2023-06-07 17:56:02

StableDiffusion is a revolutionary text-to-image generator that is causing a
stir in the world of image generation and editing. Unlike traditional methods
that learn a diffusion model in pixel space, StableDiffusion learns a diffusion
model in the latent space via a VQGAN, ensuring both efficiency and quality. It
not only supports image generation tasks, but also enables image editing for
real images, such as image inpainting and local editing. However, we have
observed that the vanilla VQGAN used in StableDiffusion leads to significant
information loss, causing distortion artifacts even in non-edited image
regions. To this end, we propose a new asymmetric VQGAN with two simple
designs. Firstly, in addition to the input from the encoder, the decoder
contains a conditional branch that incorporates information from task-specific
priors, such as the unmasked image region in inpainting. Secondly, the decoder
is much heavier than the encoder, allowing for more detailed recovery while
only slightly increasing the total inference cost. The training cost of our
asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder
while keeping the vanilla VQGAN encoder and StableDiffusion unchanged. Our
asymmetric VQGAN can be widely used in StableDiffusion-based inpainting and
local editing methods. Extensive experiments demonstrate that it can
significantly improve the inpainting and editing performance, while maintaining
the original text-to-image capability. The code is available at
https://github.com/buxiangzhiren/Asymmetric_VQGAN.
On the Reliability of Watermarks for Large Language Models
Authors: name: John Kirchenbauer, name: Jonas Geiping, name: Yuxin Wen, name: Manli Shu, name: Khalid Saifullah, name: Kezhi Kong, name: Kasun Fernando, name: Aniruddha Saha, name: Micah Goldblum, name: Tom Goldstein
Published: 2023-06-07 17:58:48

Large language models (LLMs) are now deployed to everyday use and positioned
to produce large quantities of text in the coming decade. Machine-generated
text may displace human-written text on the internet and has the potential to
be used for malicious purposes, such as spearphishing attacks and social media
bots. Watermarking is a simple and effective strategy for mitigating such harms
by enabling the detection and documentation of LLM-generated text. Yet, a
crucial question remains: How reliable is watermarking in realistic settings in
the wild? There, watermarked text might be mixed with other text sources,
paraphrased by human writers or other language models, and used for
applications in a broad number of domains, both social and technical. In this
paper, we explore different detection schemes, quantify their power at
detecting watermarks, and determine how much machine-generated text needs to be
observed in each scenario to reliably detect the watermark. We especially
highlight our human study, where we investigate the reliability of watermarking
when faced with human paraphrasing. We compare watermark-based detection to
other detection strategies, finding overall that watermarking is a reliable
solution, especially because of its sample complexity - for all attacks we
consider, the watermark evidence compounds the more examples are given, and the
watermark is eventually detected.
PromptBench: Towards Evaluating the Robustness of Large Language Models
  on Adversarial Prompts
Authors: name: Kaijie Zhu, name: Jindong Wang, name: Jiaheng Zhou, name: Zichen Wang, name: Hao Chen, name: Yidong Wang, name: Linyi Yang, name: Wei Ye, name: Neil Zhenqiang Gong, name: Yue Zhang, name: Xing Xie
Published: 2023-06-07 15:37:00

The increasing reliance on Large Language Models (LLMs) across academia and
industry necessitates a comprehensive understanding of their robustness to
prompts. In response to this vital need, we introduce PromptBench, a robustness
benchmark designed to measure LLMs' resilience to adversarial prompts. This
study uses a plethora of adversarial textual attacks targeting prompts across
multiple levels: character, word, sentence, and semantic. These prompts are
then employed in diverse tasks, such as sentiment analysis, natural language
inference, reading comprehension, machine translation, and math
problem-solving. Our study generates 4,032 adversarial prompts, meticulously
evaluated over 8 tasks and 13 datasets, with 567,084 test samples in total. Our
findings demonstrate that contemporary LLMs are vulnerable to adversarial
prompts. Furthermore, we present comprehensive analysis to understand the
mystery behind prompt robustness and its transferability. We then offer
insightful robustness analysis and pragmatic recommendations for prompt
composition, beneficial to both researchers and everyday users. We make our
code, prompts, and methodologies to generate adversarial prompts publicly
accessible, thereby enabling and encouraging collaborative exploration in this
pivotal field: https://github.com/microsoft/promptbench.
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for
  Pre-training and Benchmarks
Authors: name: Haiyang Xu, name: Qinghao Ye, name: Xuan Wu, name: Ming Yan, name: Yuan Miao, name: Jiabo Ye, name: Guohai Xu, name: Anwen Hu, name: Yaya Shi, name: Guangwei Xu, name: Chenliang Li, name: Qi Qian, name: Maofei Que, name: Ji Zhang, name: Xiao Zeng, name: Fei Huang
Published: 2023-06-07 11:52:36

To promote the development of Vision-Language Pre-training (VLP) and
multimodal Large Language Model (LLM) in the Chinese community, we firstly
release the largest public Chinese high-quality video-language dataset named
Youku-mPLUG, which is collected from Youku, a well-known Chinese video-sharing
website, with strict criteria of safety, diversity, and quality. Youku-mPLUG
contains 10 million Chinese video-text pairs filtered from 400 million raw
videos across a wide range of 45 diverse categories for large-scale
pre-training. In addition, to facilitate a comprehensive evaluation of
video-language models, we carefully build the largest human-annotated Chinese
benchmarks covering three popular video-language tasks of cross-modal
retrieval, video captioning, and video category classification. Youku-mPLUG can
enable researchers to conduct more in-depth multimodal research and develop
better applications in the future. Furthermore, we release popular
video-language pre-training models, ALPRO and mPLUG-2, and our proposed
modularized decoder-only model mPLUG-video pre-trained on Youku-mPLUG.
Experiments show that models pre-trained on Youku-mPLUG gain up to 23.1%
improvement in video category classification. Besides, mPLUG-video achieves a
new state-of-the-art result on these benchmarks with 80.5% top-1 accuracy in
video category classification and 68.9 CIDEr score in video captioning,
respectively. Finally, we scale up mPLUG-video based on the frozen Bloomz with
only 1.7% trainable parameters as Chinese multimodal LLM, and demonstrate
impressive instruction and video understanding ability. The zero-shot
instruction understanding experiment indicates that pretraining with
Youku-mPLUG can enhance the ability to comprehend overall and detailed visual
semantics, recognize scene text, and leverage open-domain knowledge.