Orionnnn/hub-stats huggingface dataset:
CoVLM: Composing Visual Entities and Relationships in Large Language
  Models Via Communicative Decoding
Authors: name: Junyan Li, name: Delin Chen, name: Yining Hong, name: Zhenfang Chen, name: Peihao Chen, name: Yikang Shen, name: Chuang Gan
Published: 2023-11-06 18:59:44

A remarkable ability of human beings resides in compositional reasoning,
i.e., the capacity to make "infinite use of finite means". However, current
large vision-language foundation models (VLMs) fall short of such compositional
abilities due to their "bag-of-words" behaviors and inability to construct
words that correctly represent visual entities and the relations among the
entities. To this end, we propose CoVLM, which can guide the LLM to explicitly
compose visual entities and relationships among the text and dynamically
communicate with the vision encoder and detection network to achieve
vision-language communicative decoding. Specifically, we first devise a set of
novel communication tokens for the LLM, for dynamic communication between the
visual detection system and the language system. A communication token is
generated by the LLM following a visual entity or a relation, to inform the
detection network to propose regions that are relevant to the sentence
generated so far. The proposed regions-of-interests (ROIs) are then fed back
into the LLM for better language generation contingent on the relevant regions.
The LLM is thus able to compose the visual entities and relationships through
the communication tokens. The vision-to-language and language-to-vision
communication are iteratively performed until the entire sentence is generated.
Our framework seamlessly bridges the gap between visual perception and LLMs and
outperforms previous VLMs by a large margin on compositional reasoning
benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on
ARO top-1 accuracy). We also achieve state-of-the-art performances on
traditional vision-language tasks such as referring expression comprehension
and visual question answering.
Ziya2: Data-centric Learning is All LLMs Need
Authors: name: Ruyi Gan, name: Ziwei Wu, name: Renliang Sun, name: Junyu Lu, name: Xiaojun Wu, name: Dixiang Zhang, name: Kunhao Pan, name: Ping Yang, name: Qi Yang, name: Jiaxing Zhang, name: Yan Song
Published: 2023-11-06 17:49:34

Various large language models (LLMs) have been proposed in recent years,
including closed- and open-source ones, continually setting new records on
multiple benchmarks. However, the development of LLMs still faces several
issues, such as high cost of training models from scratch, and continual
pre-training leading to catastrophic forgetting, etc. Although many such issues
are addressed along the line of research on LLMs, an important yet practical
limitation is that many studies overly pursue enlarging model sizes without
comprehensively analyzing and optimizing the use of pre-training data in their
learning process, as well as appropriate organization and leveraging of such
data in training LLMs under cost-effective settings. In this work, we propose
Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation
model, and further pre-trained on 700 billion tokens, where we focus on
pre-training techniques and use data-centric optimization to enhance the
learning process of Ziya2 on different stages. Experiments show that Ziya2
significantly outperforms other models in multiple benchmarks especially with
promising results compared to representative open-source ones. Ziya2 (Base) is
released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and
https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
Authors: name: Ying Sheng, name: Shiyi Cao, name: Dacheng Li, name: Coleman Hooper, name: Nicholas Lee, name: Shuo Yang, name: Christopher Chou, name: Banghua Zhu, name: Lianmin Zheng, name: Kurt Keutzer, name: Joseph E. Gonzalez, name: Ion Stoica
Published: 2023-11-06 17:26:17

The "pretrain-then-finetune" paradigm is commonly adopted in the deployment
of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient
fine-tuning method, is often employed to adapt a base model to a multitude of
tasks, resulting in a substantial collection of LoRA adapters derived from one
base model. We observe that this paradigm presents significant opportunities
for batched inference during serving. To capitalize on these opportunities, we
present S-LoRA, a system designed for the scalable serving of many LoRA
adapters. S-LoRA stores all adapters in the main memory and fetches the
adapters used by the currently running queries to the GPU memory. To
efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes
Unified Paging. Unified Paging uses a unified memory pool to manage dynamic
adapter weights with different ranks and KV cache tensors with varying sequence
lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and
highly optimized custom CUDA kernels for heterogeneous batching of LoRA
computation. Collectively, these features enable S-LoRA to serve thousands of
LoRA adapters on a single GPU or across multiple GPUs with a small overhead.
Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with
naive support of LoRA serving), S-LoRA can improve the throughput by up to 4
times and increase the number of served adapters by several orders of
magnitude. As a result, S-LoRA enables scalable serving of many task-specific
fine-tuned models and offers the potential for large-scale customized
fine-tuning services.
CogVLM: Visual Expert for Pretrained Language Models
Authors: name: Weihan Wang, name: Qingsong Lv, name: Wenmeng Yu, name: Wenyi Hong, name: Ji Qi, name: Yan Wang, name: Junhui Ji, name: Zhuoyi Yang, name: Lei Zhao, name: Xixuan Song, name: Jiazheng Xu, name: Bin Xu, name: Juanzi Li, name: Yuxiao Dong, name: Ming Ding, name: Jie Tang
Published: 2023-11-06 13:04:39

We introduce CogVLM, a powerful open-source visual language foundation model.
Different from the popular shallow alignment method which maps image features
into the input space of language model, CogVLM bridges the gap between the
frozen pretrained language model and image encoder by a trainable visual expert
module in the attention and FFN layers. As a result, CogVLM enables deep fusion
of vision language features without sacrificing any performance on NLP tasks.
CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal
benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+,
RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on
VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X
55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.
Co-training and Co-distillation for Quality Improvement and Compression
  of Language Models
Authors: name: Hayeon Lee, name: Rui Hou, name: Jongpil Kim, name: Davis Liang, name: Hongbo Zhang, name: Sung Ju Hwang, name: Alexander Min
Published: 2023-11-06 03:29:00

Knowledge Distillation (KD) compresses computationally expensive pre-trained
language models (PLMs) by transferring their knowledge to smaller models,
allowing their use in resource-constrained or real-time settings. However, most
smaller models fail to surpass the performance of the original larger model,
resulting in sacrificing performance to improve inference speed. To address
this issue, we propose Co-Training and Co-Distillation (CTCD), a novel
framework that improves performance and inference speed together by co-training
two models while mutually distilling knowledge. The CTCD framework successfully
achieves this based on two significant findings: 1) Distilling knowledge from
the smaller model to the larger model during co-training improves the
performance of the larger model. 2) The enhanced performance of the larger
model further boosts the performance of the smaller model. The CTCD framework
shows promise as it can be combined with existing techniques like architecture
design or data augmentation, replacing one-way KD methods, to achieve further
performance improvement. Extensive ablation studies demonstrate the
effectiveness of CTCD, and the small model distilled by CTCD outperforms the
original larger model by a significant margin of 1.66 on the GLUE benchmark.
Tailoring Self-Rationalizers with Multi-Reward Distillation
Authors: name: Sahana Ramnath, name: Brihi Joshi, name: Skyler Hallinan, name: Ximing Lu, name: Liunian Harold Li, name: Aaron Chan, name: Jack Hessel, name: Yejin Choi, name: Xiang Ren
Published: 2023-11-06 00:20:11

Large language models (LMs) are capable of generating free-text rationales to
aid question answering. However, prior work 1) suggests that useful
self-rationalization is emergent only at significant scales (e.g., 175B
parameter GPT-3); and 2) focuses largely on downstream performance, ignoring
the semantics of the rationales themselves, e.g., are they faithful, true, and
helpful for humans? In this work, we enable small-scale LMs (approx. 200x
smaller than GPT-3) to generate rationales that not only improve downstream
task performance, but are also more plausible, consistent, and diverse,
assessed both by automatic and human evaluation. Our method, MaRio
(Multi-rewArd RatIOnalization), is a multi-reward conditioned
self-rationalization algorithm that optimizes multiple distinct properties like
plausibility, diversity and consistency. Results on five difficult
question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC
show that not only does MaRio improve task accuracy, but it also improves the
self-rationalization quality of small LMs across the aforementioned axes better
than a supervised fine-tuning (SFT) baseline. Extensive human evaluations
confirm that MaRio rationales are preferred vs. SFT rationales, as well as
qualitative improvements in plausibility and consistency.
Attention or Convolution: Transformer Encoders in Audio Language Models
  for Inference Efficiency
Authors: name: Sungho Jeon, name: Ching-Feng Yeh, name: Hakan Inan, name: Wei-Ning Hsu, name: Rashi Rungta, name: Yashar Mehdad, name: Daniel Bikel
Published: 2023-11-05 21:30:10

In this paper, we show that a simple self-supervised pre-trained audio model
can achieve comparable inference efficiency to more complicated pre-trained
models with speech transformer encoders. These speech transformers rely on
mixing convolutional modules with self-attention modules. They achieve
state-of-the-art performance on ASR with top efficiency. We first show that
employing these speech transformers as an encoder significantly improves the
efficiency of pre-trained audio models as well. However, our study shows that
we can achieve comparable efficiency with advanced self-attention solely. We
demonstrate that this simpler approach is particularly beneficial with a
low-bit weight quantization technique of a neural network to improve
efficiency. We hypothesize that it prevents propagating the errors between
different quantized modules compared to recent speech transformers mixing
quantized convolution and the quantized self-attention modules.
Relax: Composable Abstractions for End-to-End Dynamic Machine Learning
Authors: name: Ruihang Lai, name: Junru Shao, name: Siyuan Feng, name: Steven S. Lyubomirsky, name: Bohan Hou, name: Wuwei Lin, name: Zihao Ye, name: Hongyi Jin, name: Yuchen Jin, name: Jiawei Liu, name: Lesheng Jin, name: Yaxing Cai, name: Ziheng Jiang, name: Yong Wu, name: Sunghyun Park, name: Prakalp Srivastava, name: Jared G. Roesch, name: Todd C. Mowry, name: Tianqi Chen
Published: 2023-11-01 23:03:59

Dynamic shape computations have become critical in modern machine learning
workloads, especially in emerging large language models. The success of these
models has driven demand for deploying them to a diverse set of backend
environments. In this paper, we present Relax, a compiler abstraction for
optimizing end-to-end dynamic machine learning workloads. Relax introduces
first-class symbolic shape annotations to track dynamic shape computations
globally across the program. It also introduces a cross-level abstraction that
encapsulates computational graphs, loop-level tensor programs, and library
calls in a single representation to enable cross-level optimizations. We build
an end-to-end compilation framework using the proposed approach to optimize
dynamic shape models. Experimental results on large language models show that
Relax delivers performance competitive with state-of-the-art hand-optimized
systems across platforms and enables deployment of emerging dynamic models to a
broader set of environments, including mobile phones, embedded devices, and web
browsers.
MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
Authors: name: Bingchang Liu, name: Chaoyu Chen, name: Cong Liao, name: Zi Gong, name: Huan Wang, name: Zhichao Lei, name: Ming Liang, name: Dajun Chen, name: Min Shen, name: Hailian Zhou, name: Hang Yu, name: Jianguo Li
Published: 2023-11-04 02:22:40

Code LLMs have emerged as a specialized research field, with remarkable
studies dedicated to enhancing model's coding capabilities through fine-tuning
on pre-trained models. Previous fine-tuning approaches were typically tailored
to specific downstream tasks or scenarios, which meant separate fine-tuning for
each task, requiring extensive training resources and posing challenges in
terms of deployment and maintenance. Furthermore, these approaches failed to
leverage the inherent interconnectedness among different code-related tasks. To
overcome these limitations, we present a multi-task fine-tuning framework,
MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
By incorporating various loss functions, we effectively address common
challenges in multi-task learning, such as data imbalance, varying difficulty
levels, and inconsistent convergence speeds. Extensive experiments have
conclusively demonstrated that our multi-task fine-tuning approach outperforms
both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble
of tasks. Moreover, MFTcoder offers efficient training capabilities, including
efficient data tokenization modes and PEFT fine-tuning, resulting in
significantly improved speed compared to traditional fine-tuning methods.
MFTcoder seamlessly integrates with several mainstream open-source LLMs, such
as CodeLLama and Qwen. Leveraging the CodeLLama foundation, our MFTcoder
fine-tuned model, CodeFuse-CodeLLama-34B, achieves an impressive
pass@1 score of 74.4\% on the HumaneEval benchmark, surpassing GPT-4
performance (67\%, zero-shot). MFTCoder is open-sourced at
https://github.com/codefuse-ai/MFTCOder
EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via
  Self-Supervision
Authors: name: Jiawei Yang, name: Boris Ivanovic, name: Or Litany, name: Xinshuo Weng, name: Seung Wook Kim, name: Boyi Li, name: Tong Che, name: Danfei Xu, name: Sanja Fidler, name: Marco Pavone, name: Yue Wang
Published: 2023-11-03 17:59:55

We present EmerNeRF, a simple yet powerful approach for learning
spatial-temporal representations of dynamic driving scenes. Grounded in neural
fields, EmerNeRF simultaneously captures scene geometry, appearance, motion,
and semantics via self-bootstrapping. EmerNeRF hinges upon two core components:
First, it stratifies scenes into static and dynamic fields. This decomposition
emerges purely from self-supervision, enabling our model to learn from general,
in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field
from the dynamic field and uses this flow field to further aggregate
multi-frame features, amplifying the rendering precision of dynamic objects.
Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to
represent highly-dynamic scenes self-sufficiently, without relying on ground
truth object annotations or pre-trained models for dynamic object segmentation
or optical flow estimation. Our method achieves state-of-the-art performance in
sensor simulation, significantly outperforming previous methods when
reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In
addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual
foundation model features into 4D space-time and address a general positional
bias in modern Transformers, significantly boosting 3D perception performance
(e.g., 37.50% relative improvement in occupancy prediction accuracy on
average). Finally, we construct a diverse and challenging 120-sequence dataset
to benchmark neural fields under extreme and highly-dynamic settings.
FLAP: Fast Language-Audio Pre-training
Authors: name: Ching-Feng Yeh, name: Po-Yao Huang, name: Vasu Sharma, name: Shang-Wen Li, name: Gargi Gosh
Published: 2023-11-02 21:58:50

We propose Fast Language-Audio Pre-training (FLAP), a self-supervised
approach that efficiently and effectively learns aligned audio and language
representations through masking, contrastive learning and reconstruction. For
efficiency, FLAP randomly drops audio spectrogram tokens, focusing solely on
the remaining ones for self-supervision. Through inter-modal contrastive
learning, FLAP learns to align paired audio and text representations in a
shared latent space. Notably, FLAP leverages multiple augmented views via
masking for inter-modal contrast and learns to reconstruct the masked portion
of audio tokens. Moreover, FLAP leverages large language models (LLMs) to
augment the text inputs, contributing to improved performance. These approaches
lead to more robust and informative audio-text representations, enabling FLAP
to achieve state-of-the-art (SoTA) performance on audio-text retrieval tasks on
AudioCaps (achieving 53.0% R@1) and Clotho (achieving 25.5% R@1).
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task
  Completion
Authors: name: Yiduo Guo, name: Zekai Zhang, name: Yaobo Liang, name: Dongyan Zhao, name: Duan Nan
Published: 2023-11-03 08:06:35

Recent evaluations of Large Language Models (LLMs) have centered around
testing their zero-shot/few-shot capabilities for basic natural language tasks
and their ability to translate instructions into tool APIs. However, the
evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal
instructions in a complex multi-modal environment has not been investigated. To
address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark
to assess LLMs' ability to create and edit PPT files based on user
instructions. It contains 279 multi-turn sessions covering diverse topics and
hundreds of instructions involving multi-modal operations. We also propose the
PPTX-Match Evaluation System that evaluates if LLMs finish the instruction
based on the prediction file rather than the label API sequence, thus it
supports various LLM-generated API sequences. We measure 3 closed LLMs and 6
open-source LLMs. The results show that GPT-4 outperforms other LLMs with
75.1\% accuracy in single-turn dialogue testing but faces challenges in
completing entire sessions, achieving just 6\% session accuracy. We find three
main error causes in our benchmark: error accumulation in the multi-turn
session, long PPT template processing, and multi-modality perception. These
pose great challenges for future LLM and agent systems. We release the data,
code, and evaluation system of PPTC at https://github.com/gydpku/PPTC.
Idempotent Generative Network
Authors: name: Assaf Shocher, name: Amil Dravid, name: Yossi Gandelsman, name: Inbar Mosseri, name: Michael Rubinstein, name: Alexei A. Efros
Published: 2023-11-02 17:59:55

We propose a new approach for generative modeling based on training a neural
network to be idempotent. An idempotent operator is one that can be applied
sequentially without changing the result beyond the initial application, namely
f(f(z))=f(z). The proposed model f is trained to map a source distribution
(e.g, Gaussian noise) to a target distribution (e.g. realistic images) using
the following objectives: (1) Instances from the target distribution should map
to themselves, namely f(x)=x. We define the target manifold as the set of all
instances that f maps to themselves. (2) Instances that form the source
distribution should map onto the defined target manifold. This is achieved by
optimizing the idempotence term, f(f(z))=f(z) which encourages the range of
f(z) to be on the target manifold. Under ideal assumptions such a process
provably converges to the target distribution. This strategy results in a model
capable of generating an output in one step, maintaining a consistent latent
space, while also allowing sequential applications for refinement.
Additionally, we find that by processing inputs from both target and source
distributions, the model adeptly projects corrupted or modified data back to
the target manifold. This work is a first step towards a ``global projector''
that enables projecting any input into a target data distribution.
E3 TTS: Easy End-to-End Diffusion-based Text to Speech
Authors: name: Yuan Gao, name: Nobuyuki Morioka, name: Yu Zhang, name: Nanxin Chen
Published: 2023-11-02 02:22:21

We propose Easy End-to-End Diffusion-based Text to Speech, a simple and
efficient end-to-end text-to-speech model based on diffusion. E3 TTS directly
takes plain text as input and generates an audio waveform through an iterative
refinement process. Unlike many prior work, E3 TTS does not rely on any
intermediate representations like spectrogram features or alignment
information. Instead, E3 TTS models the temporal structure of the waveform
through the diffusion process. Without relying on additional conditioning
information, E3 TTS could support flexible latent structure within the given
audio. This enables E3 TTS to be easily adapted for zero-shot tasks such as
editing without any additional training. Experiments show that E3 TTS can
generate high-fidelity audio, approaching the performance of a state-of-the-art
neural TTS system. Audio samples are available at https://e3tts.github.io.
In-Context Prompt Editing For Conditional Audio Generation
Authors: name: Ernie Chang, name: Pin-Jie Lin, name: Yang Li, name: Sidd Srinivasan, name: Gael Le Lan, name: David Kant, name: Yangyang Shi, name: Forrest Iandola, name: Vikas Chandra
Published: 2023-11-01 23:31:51

Distributional shift is a central challenge in the deployment of machine
learning models as they can be ill-equipped for real-world data. This is
particularly evident in text-to-audio generation where the encoded
representations are easily undermined by unseen prompts, which leads to the
degradation of generated audio -- the limited set of the text-audio pairs
remains inadequate for conditional audio generation in the wild as user prompts
are under-specified. In particular, we observe a consistent audio quality
degradation in generated audio samples with user prompts, as opposed to
training set prompts. To this end, we present a retrieval-based in-context
prompt editing framework that leverages the training captions as demonstrative
exemplars to revisit the user prompts. We show that the framework enhanced the
audio quality across the set of collected user prompts, which were edited with
reference to the training captions as exemplars.