Orionnnn/hub-stats huggingface dataset:
Stack-and-Delay: a new codebook pattern for music generation
Authors: name: Gael Le Lan, name: Varun Nagaraja, name: Ernie Chang, name: David Kant, name: Zhaoheng Ni, name: Yangyang Shi, name: Forrest Iandola, name: Vikas Chandra
Published: 2023-09-15 22:57:25

In language modeling based music generation, a generated waveform is
represented by a sequence of hierarchical token stacks that can be decoded
either in an auto-regressive manner or in parallel, depending on the codebook
patterns. In particular, flattening the codebooks represents the highest
quality decoding strategy, while being notoriously slow. To this end, we
propose a novel stack-and-delay style of decoding strategy to improve upon the
flat pattern decoding where generation speed is four times faster as opposed to
vanilla flat decoding. This brings the inference time close to that of the
delay decoding strategy, and allows for faster inference on GPU for small batch
sizes. For the same inference efficiency budget as the delay pattern, we show
that the proposed approach performs better in objective evaluations, almost
closing the gap with the flat pattern in terms of quality. The results are
corroborated by subjective evaluations which show that samples generated by the
new model are slightly more often preferred to samples generated by the
competing model given the same text prompts.
MindAgent: Emergent Gaming Interaction
Authors: name: Ran Gong, name: Qiuyuan Huang, name: Xiaojian Ma, name: Hoi Vo, name: Zane Durante, name: Yusuke Noda, name: Zilong Zheng, name: Song-Chun Zhu, name: Demetri Terzopoulos, name: Li Fei-Fei, name: Jianfeng Gao
Published: 2023-09-18 17:52:22

Large Language Models (LLMs) have the capacity of performing complex
scheduling in a multi-agent system and can coordinate these agents into
completing sophisticated tasks that require extensive collaboration. However,
despite the introduction of numerous gaming frameworks, the community has
insufficient benchmarks towards building general multi-agents collaboration
infrastructure that encompass both LLM and human-NPCs collaborations. In this
work, we propose a novel infrastructure - MindAgent - to evaluate planning and
coordination emergent capabilities for gaming interaction. In particular, our
infrastructure leverages existing gaming framework, to i) require understanding
of the coordinator for a multi-agent system, ii) collaborate with human players
via un-finetuned proper instructions, and iii) establish an in-context learning
on few-shot prompt with feedback. Furthermore, we introduce CUISINEWORLD, a new
gaming scenario and related benchmark that dispatch a multi-agent collaboration
efficiency and supervise multiple agents playing the game simultaneously. We
conduct comprehensive evaluations with new auto-metric CoS for calculating the
collaboration efficiency. Finally, our infrastructure can be deployed into
real-world gaming scenarios in a customized VR version of CUISINEWORLD and
adapted in existing broader Minecraft gaming domain. We hope our findings on
LLMs and the new infrastructure for general-purpose scheduling and coordination
can help shed light on how such skills can be obtained by learning from large
language corpora.
Adapting Large Language Models via Reading Comprehension
Authors: name: Daixuan Cheng, name: Shaohan Huang, name: Furu Wei
Published: 2023-09-18 07:17:52

We explore how continued pre-training on domain-specific corpora influences
large language models, revealing that training on the raw corpora endows the
model with domain knowledge, but drastically hurts its prompting ability for
question answering. Taken inspiration from human learning via reading
comprehension--practice after reading improves the ability to answer questions
based on the learned knowledge--we propose a simple method for transforming raw
corpora into reading comprehension texts. Each raw text is enriched with a
series of tasks related to its content. Our method, highly scalable and
applicable to any pre-training corpora, consistently enhances performance
across various tasks in three different domains: biomedicine, finance, and law.
Notably, our 7B language model achieves competitive performance with
domain-specific models of much larger scales, such as BloombergGPT-50B.
Furthermore, we demonstrate that domain-specific reading comprehension texts
can improve the model's performance even on general benchmarks, showing the
potential to develop a general model across even more domains. Our model, code,
and data will be available at https://github.com/microsoft/LMOps.
LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language
  Models
Authors: name: Zecheng Tang, name: Chenfei Wu, name: Juntao Li, name: Nan Duan
Published: 2023-09-18 06:35:10

Graphic layout generation, a growing research field, plays a significant role
in user engagement and information perception. Existing methods primarily treat
layout generation as a numerical optimization task, focusing on quantitative
aspects while overlooking the semantic information of layout, such as the
relationship between each layout element. In this paper, we propose LayoutNUWA,
the first model that treats layout generation as a code generation task to
enhance semantic information and harness the hidden layout expertise of large
language models~(LLMs). More concretely, we develop a Code Instruct Tuning
(CIT) approach comprising three interconnected modules: 1) the Code
Initialization (CI) module quantifies the numerical conditions and initializes
them as HTML code with strategically placed masks; 2) the Code Completion (CC)
module employs the formatting knowledge of LLMs to fill in the masked portions
within the HTML code; 3) the Code Rendering (CR) module transforms the
completed code into the final layout output, ensuring a highly interpretable
and transparent layout generation procedure that directly maps code to a
visualized layout. We attain significant state-of-the-art performance (even
over 50\% improvements) on multiple datasets, showcasing the strong
capabilities of LayoutNUWA. Our code is available at
https://github.com/ProjectNUWA/LayoutNUWA.
CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large
  Language Models in 167 Languages
Authors: name: Thuat Nguyen, name: Chien Van Nguyen, name: Viet Dac Lai, name: Hieu Man, name: Nghia Trung Ngo, name: Franck Dernoncourt, name: Ryan A. Rossi, name: Thien Huu Nguyen
Published: 2023-09-17 23:49:10

The driving factors behind the development of large language models (LLMs)
with impressive learning capabilities are their colossal model sizes and
extensive training datasets. Along with the progress in natural language
processing, LLMs have been frequently made accessible to the public to foster
deeper investigation and applications. However, when it comes to training
datasets for these LLMs, especially the recent state-of-the-art models, they
are often not fully disclosed. Creating training data for high-performing LLMs
involves extensive cleaning and deduplication to ensure the necessary level of
quality. The lack of transparency for training data has thus hampered research
on attributing and addressing hallucination and bias issues in LLMs, hindering
replication efforts and further advancements in the community. These challenges
become even more pronounced in multilingual learning scenarios, where the
available multilingual text datasets are often inadequately collected and
cleaned. Consequently, there is a lack of open-source and readily usable
dataset to effectively train LLMs in multiple languages. To overcome this
issue, we present CulturaX, a substantial multilingual dataset with 6.3
trillion tokens in 167 languages, tailored for LLM development. Our dataset
undergoes meticulous cleaning and deduplication through a rigorous pipeline of
multiple stages to accomplish the best quality for model training, including
language identification, URL-based filtering, metric-based cleaning, document
refinement, and data deduplication. CulturaX is fully released to the public in
HuggingFace to facilitate research and advancements in multilingual LLMs:
https://huggingface.co/datasets/uonlp/CulturaX.
Augmenting text for spoken language understanding with Large Language
  Models
Authors: name: Roshan Sharma, name: Suyoun Kim, name: Daniel Lazar, name: Trang Le, name: Akshat Shrivastava, name: Kwanghoon Ahn, name: Piyush Kansal, name: Leda Sari, name: Ozlem Kalinli, name: Michael Seltzer
Published: 2023-09-17 22:25:34

Spoken semantic parsing (SSP) involves generating machine-comprehensible
parses from input speech. Training robust models for existing application
domains represented in training data or extending to new domains requires
corresponding triplets of speech-transcript-semantic parse data, which is
expensive to obtain. In this paper, we address this challenge by examining
methods that can use transcript-semantic parse data (unpaired text) without
corresponding speech. First, when unpaired text is drawn from existing textual
corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways
to generate speech representations for unpaired text. Experiments on the STOP
dataset show that unpaired text from existing and new domains improves
performance by 2% and 30% in absolute Exact Match (EM) respectively. Second, we
consider the setting when unpaired text is not available in existing textual
corpora. We propose to prompt Large Language Models (LLMs) to generate unpaired
text for existing and new domains. Experiments show that examples and words
that co-occur with intents can be used to generate unpaired text with Llama
2.0. Using the generated text with JAT and TTS for spoken semantic parsing
improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains
respectively.
Contrastive Decoding Improves Reasoning in Large Language Models
Authors: name: Sean O'Brien, name: Mike Lewis
Published: 2023-09-17 00:29:32

We demonstrate that Contrastive Decoding -- a simple, computationally light,
and training-free text generation method proposed by Li et al 2022 -- achieves
large out-of-the-box improvements over greedy decoding on a variety of
reasoning tasks. Originally shown to improve the perceived quality of long-form
text generation, Contrastive Decoding searches for strings that maximize a
weighted difference in likelihood between strong and weak models. We show that
Contrastive Decoding leads LLaMA-65B to outperform LLaMA 2, GPT-3.5 and PaLM
2-L on the HellaSwag commonsense reasoning benchmark, and to outperform LLaMA
2, GPT-3.5 and PaLM-540B on the GSM8K math word reasoning benchmark, in
addition to improvements on a collection of other tasks. Analysis suggests that
Contrastive Decoding improves over existing methods by preventing some abstract
reasoning errors, as well as by avoiding simpler modes such as copying sections
of the input during chain-of-thought. Overall, Contrastive Decoding outperforms
nucleus sampling for long-form generation and greedy decoding for reasoning
tasks, making it a powerful general purpose method for generating text from
language models.
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large
  Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)
Authors: name: Parsa Kavehzadeh, name: Mojtaba Valipour, name: Marzieh Tahaei, name: Ali Ghodsi, name: Boxing Chen, name: Mehdi Rezagholizadeh
Published: 2023-09-16 11:58:34

The rapid advancement of large language models (LLMs) has revolutionized
natural language processing (NLP). While these models excel at understanding
and generating human-like text, their widespread deployment can be
prohibitively expensive. SortedNet is a recent training technique for enabling
dynamic inference for deep neural networks. It leverages network modularity to
create sub-models with varying computational loads, sorting them based on
computation/accuracy characteristics in a nested manner. We extend SortedNet to
generative NLP tasks, making large language models dynamic without any
pretraining and by only replacing standard Supervised Fine-Tuning (SFT) with
Sorted Fine-Tuning (SoFT) at the same costs. Our approach boosts model
efficiency, eliminating the need for multiple models for various scenarios
during inference. We show that using this approach, we are able to unlock the
potential of intermediate layers of transformers in generating the target
output. Our sub-models remain integral components of the original model,
minimizing storage requirements and transition costs between different
computational/latency budgets. By applying this approach on LLaMa 2 13B for
tuning on the Stanford Alpaca dataset and comparing it to normal tuning and
early exit via PandaLM benchmark, we show that Sorted Fine-Tuning can deliver
models twice as fast as the original model while maintaining or exceeding
performance.
Struc-Bench: Are Large Language Models Really Good at Generating Complex
  Structured Data?
Authors: name: Xiangru Tang, name: Yiming Zong, name: Yilun Zhao, name: Arman Cohan, name: Mark Gerstein
Published: 2023-09-16 11:31:58

Despite the power of Large Language Models (LLMs) like GPT-4, they still
struggle with tasks that require generating complex, structured outputs. In
this study, we assess the capability of Current LLMs in generating complex
structured data and propose a structure-aware fine-tuning approach as a
solution to improve this ability. To perform a comprehensive evaluation, we
propose Struc-Bench, include five representative LLMs (i.e., GPT-NeoX 20B,
GPT-3.5, GPT-4, and Vicuna) and evaluate them on our carefully constructed
datasets spanning raw text, HTML, and LaTeX tables. Based on our analysis of
current model performance, we identify specific common formatting errors and
areas of potential improvement. To address complex formatting requirements, we
utilize FormatCoT (Chain-of-Thought) to generate format instructions from
target outputs. Our experiments show that our structure-aware fine-tuning
method, when applied to LLaMA-7B, significantly improves adherence to natural
language constraints, outperforming other evaluated LLMs. Based on these
results, we present an ability map of model capabilities from six dimensions
(i.e., coverage, formatting, reasoning, comprehension, pragmatics, and
hallucination). This map highlights the weaknesses of LLMs in handling complex
structured outputs and suggests promising directions for future work. Our code
and models can be found at https://github.com/gersteinlab/Struc-Bench.
PDFTriage: Question Answering over Long, Structured Documents
Authors: name: Jon Saad-Falcon, name: Joe Barrow, name: Alexa Siu, name: Ani Nenkova, name: Ryan A. Rossi, name: Franck Dernoncourt
Published: 2023-09-16 04:29:05

Large Language Models (LLMs) have issues with document question answering
(QA) in situations where the document is unable to fit in the small context
length of an LLM. To overcome this issue, most existing works focus on
retrieving the relevant context from the document, representing them as plain
text. However, documents such as PDFs, web pages, and presentations are
naturally structured with different pages, tables, sections, and so on.
Representing such structured documents as plain text is incongruous with the
user's mental model of these documents with rich structure. When a system has
to query the document for context, this incongruity is brought to the fore, and
seemingly trivial questions can trip up the QA system. To bridge this
fundamental gap in handling structured documents, we propose an approach called
PDFTriage that enables models to retrieve the context based on either structure
or content. Our experiments demonstrate the effectiveness of the proposed
PDFTriage-augmented models across several classes of questions where existing
retrieval-augmented LLMs fail. To facilitate further research on this
fundamental problem, we release our benchmark dataset consisting of 900+
human-generated questions over 80 structured documents from 10 different
categories of question types for document QA.
S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking
  in the Era of LLMs
Authors: name: Sarkar Snigdha Sarathi Das, name: Chirag Shah, name: Mengting Wan, name: Jennifer Neville, name: Longqi Yang, name: Reid Andersen, name: Georg Buscher, name: Tara Safavi
Published: 2023-09-16 00:59:23

The traditional Dialogue State Tracking (DST) problem aims to track user
preferences and intents in user-agent conversations. While sufficient for
task-oriented dialogue systems supporting narrow domain applications, the
advent of Large Language Model (LLM)-based chat systems has introduced many
real-world intricacies in open-domain dialogues. These intricacies manifest in
the form of increased complexity in contextual interactions, extended dialogue
sessions encompassing a diverse array of topics, and more frequent contextual
shifts. To handle these intricacies arising from evolving LLM-based chat
systems, we propose joint dialogue segmentation and state tracking per segment
in open-domain dialogue systems. Assuming a zero-shot setting appropriate to a
true open-domain dialogue system, we propose S3-DST, a structured prompting
technique that harnesses Pre-Analytical Recollection, a novel grounding
mechanism we designed for improving long context tracking. To demonstrate the
efficacy of our proposed approach in joint segmentation and state tracking, we
evaluate S3-DST on a proprietary anonymized open-domain dialogue dataset, as
well as publicly available DST and segmentation datasets. Across all datasets
and settings, S3-DST consistently outperforms the state-of-the-art,
demonstrating its potency and robustness the next generation of LLM-based chat
systems.
Enhance audio generation controllability through representation
  similarity regularization
Authors: name: Yangyang Shi, name: Gael Le Lan, name: Varun Nagaraja, name: Zhaoheng Ni, name: Xinhao Mei, name: Ernie Chang, name: Forrest Iandola, name: Yang Liu, name: Vikas Chandra
Published: 2023-09-15 21:32:20

This paper presents an innovative approach to enhance control over audio
generation by emphasizing the alignment between audio and text representations
during model training. In the context of language model-based audio generation,
the model leverages input from both textual and audio token representations to
predict subsequent audio tokens. However, the current configuration lacks
explicit regularization to ensure the alignment between the chosen text
representation and the language model's predictions. Our proposal involves the
incorporation of audio and text representation regularization, particularly
during the classifier-free guidance (CFG) phase, where the text condition is
excluded from cross attention during language model training. The aim of this
proposed representation regularization is to minimize discrepancies in audio
and text similarity compared to other samples within the same training batch.
Experimental results on both music and audio generation tasks demonstrate that
our proposed methods lead to improvements in objective metrics for both audio
and music generation, as well as an enhancement in the human perception for
audio generation.
Cure the headache of Transformers via Collinear Constrained Attention
Authors: name: Shiyi Zhu, name: Jing Ye, name: Wei Jiang, name: Qi Zhang, name: Yifan Wu, name: Jianguo Li
Published: 2023-09-15 09:36:51

As the rapid progression of practical applications based on Large Language
Models continues, the importance of extrapolating performance has grown
exponentially in the research domain. In our study, we identified an anomalous
behavior in Transformer models that had been previously overlooked, leading to
a chaos around closest tokens which carried the most important information.
We've coined this discovery the "headache of Transformers". To address this at
its core, we introduced a novel self-attention structure named Collinear
Constrained Attention (CoCA). This structure can be seamlessly integrated with
existing extrapolation, interpolation methods, and other optimization
strategies designed for traditional Transformer models. We have achieved
excellent extrapolating performance even for 16 times to 24 times of sequence
lengths during inference without any fine-tuning on our model. We have also
enhanced CoCA's computational and spatial efficiency to ensure its
practicality. We plan to open-source CoCA shortly. In the meantime, we've made
our code available in the appendix for reappearing experiments.
TextBind: Multi-turn Interleaved Multimodal Instruction-following
Authors: name: Huayang Li, name: Siheng Li, name: Deng Cai, name: Longyue Wang, name: Lemao Liu, name: Taro Watanabe, name: Yujiu Yang, name: Shuming Shi
Published: 2023-09-14 15:34:01

Large language models with instruction-following abilities have
revolutionized the field of artificial intelligence. These models show
exceptional generalizability to tackle various real-world tasks through their
natural language interfaces. However, their performance heavily relies on
high-quality exemplar data, which is often difficult to obtain. This challenge
is further exacerbated when it comes to multimodal instruction following. We
introduce TextBind, an almost annotation-free framework for empowering larger
language models with the multi-turn interleaved multimodal
instruction-following capabilities. Our approach requires only image-caption
pairs and generates multi-turn multimodal instruction-response conversations
from a language model. We release our dataset, model, and demo to foster future
research in the area of multimodal instruction following.
Recovering from Privacy-Preserving Masking with Large Language Models
Authors: name: Arpita Vats, name: Zhe Liu, name: Peng Su, name: Debjyoti Paul, name: Yingyi Ma, name: Yutong Pang, name: Zeeshan Ahmed, name: Ozlem Kalinli
Published: 2023-09-12 16:39:41

Model adaptation is crucial to handle the discrepancy between proxy training
data and actual users data received. To effectively perform adaptation, textual
data of users is typically stored on servers or their local devices, where
downstream natural language processing (NLP) models can be directly trained
using such in-domain data. However, this might raise privacy and security
concerns due to the extra risks of exposing user information to adversaries.
Replacing identifying information in textual data with a generic marker has
been recently explored. In this work, we leverage large language models (LLMs)
to suggest substitutes of masked tokens and have their effectiveness evaluated
on downstream language modeling tasks. Specifically, we propose multiple
pre-trained and fine-tuned LLM-based approaches and perform empirical studies
on various datasets for the comparison of these methods. Experimental results
show that models trained on the obfuscation corpora are able to achieve
comparable performance with the ones trained on the original data without
privacy-preserving token masking.