Orionnnn/hub-stats huggingface dataset:
Qwen Technical Report
Authors: name: Jinze Bai, name: Shuai Bai, name: Yunfei Chu, name: Zeyu Cui, name: Kai Dang, name: Xiaodong Deng, name: Yang Fan, name: Wenbin Ge, name: Yu Han, name: Fei Huang, name: Binyuan Hui, name: Luo Ji, name: Mei Li, name: Junyang Lin, name: Runji Lin, name: Dayiheng Liu, name: Gao Liu, name: Chengqiang Lu, name: Keming Lu, name: Jianxin Ma, name: Rui Men, name: Xingzhang Ren, name: Xuancheng Ren, name: Chuanqi Tan, name: Sinan Tan, name: Jianhong Tu, name: Peng Wang, name: Shijie Wang, name: Wei Wang, name: Shengguang Wu, name: Benfeng Xu, name: Jin Xu, name: An Yang, name: Hao Yang, name: Jian Yang, name: Shusheng Yang, name: Yang Yao, name: Bowen Yu, name: Hongyi Yuan, name: Zheng Yuan, name: Jianwei Zhang, name: Xingxuan Zhang, name: Yichang Zhang, name: Zhenru Zhang, name: Chang Zhou, name: Jingren Zhou, name: Xiaohuan Zhou, name: Tianhang Zhu
Published: 2023-09-28 17:07:49

Large language models (LLMs) have revolutionized the field of artificial
intelligence, enabling natural language processing tasks that were previously
thought to be exclusive to humans. In this work, we introduce Qwen, the first
installment of our large language model series. Qwen is a comprehensive
language model series that encompasses distinct models with varying parameter
counts. It includes Qwen, the base pretrained language models, and Qwen-Chat,
the chat models finetuned with human alignment techniques. The base language
models consistently demonstrate superior performance across a multitude of
downstream tasks, and the chat models, particularly those trained using
Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The
chat models possess advanced tool-use and planning capabilities for creating
agent applications, showcasing impressive performance even when compared to
bigger models on complex tasks like utilizing a code interpreter. Furthermore,
we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as
well as mathematics-focused models, Math-Qwen-Chat, which are built upon base
language models. These models demonstrate significantly improved performance in
comparison with open-source models, and slightly fall behind the proprietary
models.
GPT-Fathom: Benchmarking Large Language Models to Decipher the
  Evolutionary Path towards GPT-4 and Beyond
Authors: name: Shen Zheng, name: Yuyu Zhang, name: Yijie Zhu, name: Chenguang Xi, name: Pengyang Gao, name: Xun Zhou, name: Kevin Chen-Chuan Chang
Published: 2023-09-28 16:43:35

With the rapid advancement of large language models (LLMs), there is a
pressing need for a comprehensive evaluation suite to assess their capabilities
and limitations. Existing LLM leaderboards often reference scores reported in
other papers without consistent settings and prompts, which may inadvertently
encourage cherry-picking favored settings and prompts for better results. In
this work, we introduce GPT-Fathom, an open-source and reproducible LLM
evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+
leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across
7 capability categories, all under aligned settings. Our retrospective study on
OpenAI's earlier models offers valuable insights into the evolutionary path
from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3
progressively improves to GPT-4, including technical details like whether
adding code data improves LLM's reasoning capability, which aspects of LLM
capability can be improved by SFT and RLHF, how much is the alignment tax, etc.
Our analysis sheds light on many of these questions, aiming to improve the
transparency of advanced LLMs.
MotionLM: Multi-Agent Motion Forecasting as Language Modeling
Authors: name: Ari Seff, name: Brian Cera, name: Dian Chen, name: Mason Ng, name: Aurick Zhou, name: Nigamaa Nayakanti, name: Khaled S. Refaat, name: Rami Al-Rfou, name: Benjamin Sapp
Published: 2023-09-28 15:46:25

Reliable forecasting of the future behavior of road agents is a critical
component to safe planning in autonomous vehicles. Here, we represent
continuous trajectories as sequences of discrete motion tokens and cast
multi-agent motion prediction as a language modeling task over this domain. Our
model, MotionLM, provides several advantages: First, it does not require
anchors or explicit latent variable optimization to learn multimodal
distributions. Instead, we leverage a single standard language modeling
objective, maximizing the average log probability over sequence tokens. Second,
our approach bypasses post-hoc interaction heuristics where individual agent
trajectory generation is conducted prior to interactive scoring. Instead,
MotionLM produces joint distributions over interactive agent futures in a
single autoregressive decoding process. In addition, the model's sequential
factorization enables temporally causal conditional rollouts. The proposed
approach establishes new state-of-the-art performance for multi-agent motion
prediction on the Waymo Open Motion Dataset, ranking 1st on the interactive
challenge leaderboard.
AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Authors: name: Jan Hendrik Metzen, name: Piyapat Saranrittichai, name: Chaithanya Kumar Mummadi
Published: 2023-09-28 13:08:08

Classifiers built upon vision-language models such as CLIP have shown
remarkable zero-shot performance across a broad range of image classification
tasks. Prior work has studied different ways of automatically creating
descriptor sets for every class based on prompt templates, ranging from
manually engineered templates over templates obtained from a large language
model to templates built from random words and characters. In contrast,
deriving zero-shot classifiers from the respective encoded class descriptors
has remained nearly unchanged, that is: classify to the class that maximizes
the cosine similarity between its averaged encoded class descriptors and the
encoded image. However, weighting all class descriptors equally can be
suboptimal when certain descriptors match visual clues on a given image better
than others. In this work, we propose AutoCLIP, a method for auto-tuning
zero-shot classifiers. AutoCLIP assigns to each prompt template per-image
weights, which are derived from statistics of class descriptor-image
similarities at inference time. AutoCLIP is fully unsupervised, has very low
overhead, and can be easily implemented in few lines of code. We show that for
a broad range of vision-language models, datasets, and prompt templates,
AutoCLIP outperforms baselines consistently and by up to 3 percent point
accuracy.
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Authors: name: Seungwhan Moon, name: Andrea Madotto, name: Zhaojiang Lin, name: Tushar Nagarajan, name: Matt Smith, name: Shashank Jain, name: Chun-Fu Yeh, name: Prakash Murugesan, name: Peyman Heidari, name: Yue Liu, name: Kavya Srinet, name: Babak Damavandi, name: Anuj Kumar
Published: 2023-09-27 22:50:51

We present Any-Modality Augmented Language Model (AnyMAL), a unified model
that reasons over diverse input modality signals (i.e. text, image, video,
audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
powerful text-based reasoning abilities of the state-of-the-art LLMs including
LLaMA-2 (70B), and converts modality-specific signals to the joint textual
space through a pre-trained aligner module. To further strengthen the
multimodal LLM's capabilities, we fine-tune the model with a multimodal
instruction set manually collected to cover diverse topics and tasks beyond
simple QAs. We conduct comprehensive empirical analysis comprising both human
and automatic evaluations, and demonstrate state-of-the-art performance on
various multimodal tasks.
Effective Long-Context Scaling of Foundation Models
Authors: name: Wenhan Xiong, name: Jingyu Liu, name: Igor Molybog, name: Hejia Zhang, name: Prajjwal Bhargava, name: Rui Hou, name: Louis Martin, name: Rashi Rungta, name: Karthik Abinav Sankararaman, name: Barlas Oguz, name: Madian Khabsa, name: Han Fang, name: Yashar Mehdad, name: Sharan Narang, name: Kshitiz Malik, name: Angela Fan, name: Shruti Bhosale, name: Sergey Edunov, name: Mike Lewis, name: Sinong Wang, name: Hao Ma
Published: 2023-09-27 21:41:49

We present a series of long-context LLMs that support effective context
windows of up to 32,768 tokens. Our model series are built through continual
pretraining from Llama 2 with longer training sequences and on a dataset where
long texts are upsampled. We perform extensive evaluation on language modeling,
synthetic context probing tasks, and a wide range of research benchmarks. On
research benchmarks, our models achieve consistent improvements on most regular
tasks and significant improvements on long-context tasks over Llama 2. Notably,
with a cost-effective instruction tuning procedure that does not require
human-annotated long instruction data, the 70B variant can already surpass
gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks.
Alongside these results, we provide an in-depth analysis on the individual
components of our method. We delve into Llama's position encodings and discuss
its limitation in modeling long dependencies. We also examine the impact of
various design choices in the pretraining process, including the data mix and
the training curriculum of sequence lengths -- our ablation experiments suggest
that having abundant long texts in the pretrain dataset is not the key to
achieving strong performance, and we empirically verify that long context
continual pretraining is more efficient and similarly effective compared to
pretraining from scratch with long sequences.
NeuRBF: A Neural Fields Representation with Adaptive Radial Basis
  Functions
Authors: name: Zhang Chen, name: Zhong Li, name: Liangchen Song, name: Lele Chen, name: Jingyi Yu, name: Junsong Yuan, name: Yi Xu
Published: 2023-09-27 06:32:05

We present a novel type of neural fields that uses general radial bases for
signal representation. State-of-the-art neural fields typically rely on
grid-based representations for storing local neural features and N-dimensional
linear kernels for interpolating features at continuous query points. The
spatial positions of their neural features are fixed on grid nodes and cannot
well adapt to target signals. Our method instead builds upon general radial
bases with flexible kernel position and shape, which have higher spatial
adaptivity and can more closely fit target signals. To further improve the
channel-wise capacity of radial basis functions, we propose to compose them
with multi-frequency sinusoid functions. This technique extends a radial basis
to multiple Fourier radial bases of different frequency bands without requiring
extra parameters, facilitating the representation of details. Moreover, by
marrying adaptive radial bases with grid-based ones, our hybrid combination
inherits both adaptivity and interpolation smoothness. We carefully designed
weighting schemes to let radial bases adapt to different types of signals
effectively. Our experiments on 2D image and 3D signed distance field
representation demonstrate the higher accuracy and compactness of our method
than prior arts. When applied to neural radiance field reconstruction, our
method achieves state-of-the-art rendering quality, with small model size and
comparable training speed.
DECO: Dense Estimation of 3D Human-Scene Contact In The Wild
Authors: name: Shashank Tripathi, name: Agniv Chatterjee, name: Jean-Claude Passy, name: Hongwei Yi, name: Dimitrios Tzionas, name: Michael J. Black
Published: 2023-09-26 21:21:07

Understanding how humans use physical contact to interact with the world is
key to enabling human-centric artificial intelligence. While inferring 3D
contact is crucial for modeling realistic and physically-plausible human-object
interactions, existing methods either focus on 2D, consider body joints rather
than the surface, use coarse 3D body regions, or do not generalize to
in-the-wild images. In contrast, we focus on inferring dense, 3D contact
between the full body surface and objects in arbitrary images. To achieve this,
we first collect DAMON, a new dataset containing dense vertex-level contact
annotations paired with RGB images containing complex human-object and
human-scene contact. Second, we train DECO, a novel 3D contact detector that
uses both body-part-driven and scene-context-driven attention to estimate
vertex-level contact on the SMPL body. DECO builds on the insight that human
observers recognize contact by reasoning about the contacting body parts, their
proximity to scene objects, and the surrounding scene context. We perform
extensive evaluations of our detector on DAMON as well as on the RICH and
BEHAVE datasets. We significantly outperform existing SOTA methods across all
benchmarks. We also show qualitatively that DECO generalizes well to diverse
and challenging real-world human interactions in natural images. The code,
data, and models are available at https://deco.is.tue.mpg.de.
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video
  Generation
Authors: name: David Junhao Zhang, name: Jay Zhangjie Wu, name: Jia-Wei Liu, name: Rui Zhao, name: Lingmin Ran, name: Yuchao Gu, name: Difei Gao, name: Mike Zheng Shou
Published: 2023-09-27 17:44:18

Significant advancements have been achieved in the realm of large-scale
pre-trained text-to-video Diffusion Models (VDMs). However, previous methods
either rely solely on pixel-based VDMs, which come with high computational
costs, or on latent-based VDMs, which often struggle with precise text-video
alignment. In this paper, we are the first to propose a hybrid model, dubbed as
Show-1, which marries pixel-based and latent-based VDMs for text-to-video
generation. Our model first uses pixel-based VDMs to produce a low-resolution
video of strong text-video correlation. After that, we propose a novel expert
translation method that employs the latent-based VDMs to further upsample the
low-resolution video to high resolution. Compared to latent VDMs, Show-1 can
produce high-quality videos of precise text-video alignment; Compared to pixel
VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G
vs 72G). We also validate our model on standard video generation benchmarks.
Our code and model weights are publicly available at
https://github.com/showlab/Show-1.
Emu: Enhancing Image Generation Models Using Photogenic Needles in a
  Haystack
Authors: name: Xiaoliang Dai, name: Ji Hou, name: Chih-Yao Ma, name: Sam Tsai, name: Jialiang Wang, name: Rui Wang, name: Peizhao Zhang, name: Simon Vandenhende, name: Xiaofang Wang, name: Abhimanyu Dubey, name: Matthew Yu, name: Abhishek Kadian, name: Filip Radenovic, name: Dhruv Mahajan, name: Kunpeng Li, name: Yue Zhao, name: Vladan Petrovic, name: Mitesh Kumar Singh, name: Simran Motwani, name: Yi Wen, name: Yiwen Song, name: Roshan Sumbaly, name: Vignesh Ramanathan, name: Zijian He, name: Peter Vajda, name: Devi Parikh
Published: 2023-09-27 17:30:19

Training text-to-image models with web scale image-text pairs enables the
generation of a wide range of visual concepts from text. However, these
pre-trained models often face challenges when it comes to generating highly
aesthetic images. This creates the need for aesthetic alignment post
pre-training. In this paper, we propose quality-tuning to effectively guide a
pre-trained model to exclusively generate highly visually appealing images,
while maintaining generality across visual concepts. Our key insight is that
supervised fine-tuning with a set of surprisingly small but extremely visually
appealing images can significantly improve the generation quality. We pre-train
a latent diffusion model on 1.1 billion image-text pairs and fine-tune it
with only a few thousand carefully selected high-quality images. The resulting
model, Emu, achieves a win rate of 82.9% compared with its pre-trained only
counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred
68.4% and 71.3% of the time on visual appeal on the standard PartiPrompts
and our Open User Input benchmark based on the real-world usage of
text-to-image models. In addition, we show that quality-tuning is a generic
approach that is also effective for other architectures, including pixel
diffusion and masked generative transformer models.
Jointly Training Large Autoregressive Multimodal Models
Authors: name: Emanuele Aiello, name: Lili Yu, name: Yixin Nie, name: Armen Aghajanyan, name: Barlas Oguz
Published: 2023-09-27 10:40:23

In recent years, advances in the large-scale pretraining of language and
text-to-image models have revolutionized the field of machine learning. Yet,
integrating these two modalities into a single, robust model capable of
generating seamless multimodal outputs remains a significant challenge. To
address this gap, we present the Joint Autoregressive Mixture (JAM) framework,
a modular approach that systematically fuses existing text and image generation
models. We also introduce a specialized, data-efficient instruction-tuning
strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned
model demonstrates unparalleled performance in generating high-quality
multimodal outputs and represents the first model explicitly designed for this
purpose.
Finite Scalar Quantization: VQ-VAE Made Simple
Authors: name: Fabian Mentzer, name: David Minnen, name: Eirikur Agustsson, name: Michael Tschannen
Published: 2023-09-27 09:13:40

We propose to replace vector quantization (VQ) in the latent representation
of VQ-VAEs with a simple scheme termed finite scalar quantization (FSQ), where
we project the VAE representation down to a few dimensions (typically less than
10). Each dimension is quantized to a small set of fixed values, leading to an
(implicit) codebook given by the product of these sets. By appropriately
choosing the number of dimensions and values each dimension can take, we obtain
the same codebook size as in VQ. On top of such discrete representations, we
can train the same models that have been trained on VQ-VAE representations. For
example, autoregressive and masked transformer models for image generation,
multimodal generation, and dense prediction computer vision tasks. Concretely,
we employ FSQ with MaskGIT for image generation, and with UViM for depth
estimation, colorization, and panoptic segmentation. Despite the much simpler
design of FSQ, we obtain competitive performance in all these tasks. We
emphasize that FSQ does not suffer from codebook collapse and does not need the
complex machinery employed in VQ (commitment losses, codebook reseeding, code
splitting, entropy penalties, etc.) to learn expressive discrete
representations.
VPA: Fully Test-Time Visual Prompt Adaptation
Authors: name: Jiachen Sun, name: Mark Ibrahim, name: Melissa Hall, name: Ivan Evtimov, name: Z. Morley Mao, name: Cristian Canton Ferrer, name: Caner Hazirbas
Published: 2023-09-26 20:25:51

Textual prompt tuning has demonstrated significant performance improvements
in adapting natural language processing models to a variety of downstream tasks
by treating hand-engineered prompts as trainable parameters. Inspired by the
success of textual prompting, several studies have investigated the efficacy of
visual prompt tuning. In this work, we present Visual Prompt Adaptation (VPA),
the first framework that generalizes visual prompting with test-time
adaptation. VPA introduces a small number of learnable tokens, enabling fully
test-time and storage-efficient adaptation without necessitating source-domain
information. We examine our VPA design under diverse adaptation settings,
encompassing single-image, batched-image, and pseudo-label adaptation. We
evaluate VPA on multiple tasks, including out-of-distribution (OOD)
generalization, corruption robustness, and domain adaptation. Experimental
results reveal that VPA effectively enhances OOD generalization by 3.3% across
various models, surpassing previous test-time approaches. Furthermore, we show
that VPA improves corruption robustness by 6.5% compared to strong baselines.
Finally, we demonstrate that VPA also boosts domain adaptation performance by
relatively 5.2%. Our VPA also exhibits marked effectiveness in improving the
robustness of zero-shot recognition for vision-language models.
Low-rank Adaptation of Large Language Model Rescoring for
  Parameter-Efficient Speech Recognition
Authors: name: Yu Yu, name: Chao-Han Huck Yang, name: Jari Kolehmainen, name: Prashanth G. Shivakumar, name: Yile Gu, name: Sungho Ryu, name: Roger Ren, name: Qi Luo, name: Aditya Gourav, name: I-Fan Chen, name: Yi-Chieh Liu, name: Tuan Dinh, name: Ankur Gandhe, name: Denis Filimonov, name: Shalini Ghosh, name: Andreas Stolcke, name: Ariya Rastow, name: Ivan Bulyko
Published: 2023-09-26 19:41:34

We propose a neural language modeling system based on low-rank adaptation
(LoRA) for speech recognition output rescoring. Although pretrained language
models (LMs) like BERT have shown superior performance in second-pass
rescoring, the high computational cost of scaling up the pretraining stage and
adapting the pretrained models to specific domains limit their practical use in
rescoring. Here we present a method based on low-rank decomposition to train a
rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
of the pretrained parameters. These inserted matrices are optimized through a
discriminative training objective along with a correlation-based regularization
loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
evaluated on LibriSpeech and internal datasets with decreased training times by
factors between 5.4 and 3.6.
Evaluating Cognitive Maps and Planning in Large Language Models with
  CogEval
Authors: name: Ida Momennejad, name: Hosein Hasanbeig, name: Felipe Vieira, name: Hiteshi Sharma, name: Robert Osazuwa Ness, name: Nebojsa Jojic, name: Hamid Palangi, name: Jonathan Larson
Published: 2023-09-25 01:20:13

Recently an influx of studies claim emergent cognitive abilities in large
language models (LLMs). Yet, most rely on anecdotes, overlook contamination of
training sets, or lack systematic Evaluation involving multiple tasks, control
conditions, multiple iterations, and statistical robustness tests. Here we make
two major contributions. First, we propose CogEval, a cognitive
science-inspired protocol for the systematic evaluation of cognitive capacities
in Large Language Models. The CogEval protocol can be followed for the
evaluation of various abilities. Second, here we follow CogEval to
systematically evaluate cognitive maps and planning ability across eight LLMs
(OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard,
Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base
our task prompts on human experiments, which offer both established construct
validity for evaluating planning, and are absent from LLM training sets. We
find that, while LLMs show apparent competence in a few planning tasks with
simpler structures, systematic evaluation reveals striking failure modes in
planning tasks, including hallucinations of invalid trajectories and getting
trapped in loops. These findings do not support the idea of emergent
out-of-the-box planning ability in LLMs. This could be because LLMs do not
understand the latent relational structures underlying planning problems, known
as cognitive maps, and fail at unrolling goal-directed trajectories based on
the underlying structure. Implications for application and future directions
are discussed.