Orionnnn/hub-stats huggingface dataset:
MM-LLMs: Recent Advances in MultiModal Large Language Models
Authors: name: Duzhen Zhang, name: Yahan Yu, name: Chenxing Li, name: Jiahua Dong, name: Dan Su, name: Chenhui Chu, name: Dong Yu
Published: 2024-01-24 17:10:45

In the past year, MultiModal Large Language Models (MM-LLMs) have undergone
substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or
outputs via cost-effective training strategies. The resulting models not only
preserve the inherent reasoning and decision-making capabilities of LLMs but
also empower a diverse range of MM tasks. In this paper, we provide a
comprehensive survey aimed at facilitating further research of MM-LLMs.
Specifically, we first outline general design formulations for model
architecture and training pipeline. Subsequently, we provide brief
introductions of 26 existing MM-LLMs, each characterized by its specific
formulations. Additionally, we review the performance of MM-LLMs on mainstream
benchmarks and summarize key training recipes to enhance the potency of
MM-LLMs. Lastly, we explore promising directions for MM-LLMs while concurrently
maintaining a real-time tracking website for the latest developments in the
field. We hope that this survey contributes to the ongoing advancement of the
MM-LLMs domain.
MaLA-500: Massive Language Adaptation of Large Language Models
Authors: name: Peiqin Lin, name: Shaoxiong Ji, name: Jörg Tiedemann, name: André F. T. Martins, name: Hinrich Schütze
Published: 2024-01-24 08:57:39

Large language models have advanced the state of the art in natural language
processing. However, their predominant design for English or a limited set of
languages creates a substantial gap in their effectiveness for low-resource
languages. To bridge this gap, we introduce MaLA-500, a novel large language
model designed to cover an extensive range of 534 languages. To train MaLA-500,
we employ vocabulary extension and continued pretraining on LLaMA 2 with
Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves
state-of-the-art in-context learning results. We release MaLA-500 at
https://huggingface.co/MaLA-LM
SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced
  Token Detection
Authors: name: Ke Ye, name: Heinrich Jiang, name: Afshin Rostamizadeh, name: Ayan Chakrabarti, name: Giulia DeSalvo, name: Jean-François Kagy, name: Lazaros Karydas, name: Gui Citovsky, name: Sanjiv Kumar
Published: 2024-01-24 00:36:13

Pre-training large language models is known to be extremely resource
intensive and often times inefficient, under-utilizing the information
encapsulated in the training text sequences. In this paper, we present SpacTor,
a new training procedure consisting of (1) a hybrid objective combining span
corruption (SC) and token replacement detection (RTD), and (2) a two-stage
curriculum that optimizes the hybrid objective over the initial tau
iterations, then transitions to standard SC loss. We show empirically that the
effectiveness of the hybrid objective is tied to the two-stage pre-training
schedule, and provide extensive analysis on why this is the case. In our
experiments with encoder-decoder architectures (T5) on a variety of NLP tasks,
SpacTor-T5 yields the same downstream performance as standard SC pre-training,
while enabling a 50% reduction in pre-training iterations and 40% reduction in
total FLOPs. Alternatively, given the same amount of computing budget, we find
that SpacTor results in significantly improved downstream benchmark
performance.
MambaByte: Token-free Selective State Space Model
Authors: name: Junxiong Wang, name: Tushaar Gangavarapu, name: Jing Nathan Yan, name: Alexander M Rush
Published: 2024-01-24 18:53:53

Token-free language models learn directly from raw bytes and remove the bias
of subword tokenization. Operating on bytes, however, results in significantly
longer sequences, and standard autoregressive Transformers scale poorly in such
settings. We experiment with MambaByte, a token-free adaptation of the Mamba
state space model, trained autoregressively on byte sequences. Our experiments
indicate the computational efficiency of MambaByte compared to other byte-level
models. We also find MambaByte to be competitive with and even outperform
state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
length, MambaByte benefits from fast inference compared to Transformers. Our
findings establish the viability of MambaByte in enabling token-free language
modeling.
ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in
  Large Multimodal Models
Authors: name: Rohan Wadhawan, name: Hritik Bansal, name: Kai-Wei Chang, name: Nanyun Peng
Published: 2024-01-24 09:07:11

Recent advancements in AI have led to the development of large multimodal
models (LMMs) capable of processing complex tasks involving joint reasoning
over text and visual content in the image (e.g., navigating maps in public
places). This paper introduces ConTextual, a novel benchmark comprising
instructions designed explicitly to evaluate LMMs' ability to perform
context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
real-world scenarios (e.g., time-reading, navigation, shopping and more)
demanding a deeper understanding of the interactions between textual and visual
elements. Our findings reveal a significant performance gap of 30.8% between
the best-performing LMM, GPT-4V(ision), and human capabilities using human
evaluation indicating substantial room for improvement in context-sensitive
text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
categories like meme and quote interpretation, its overall performance still
lagged behind humans. In addition to human evaluations, we also employed
automatic evaluation metrics using GPT-4, uncovering similar trends in
performance disparities. We also perform a fine-grained evaluation across
diverse visual contexts and provide qualitative analysis which provides a
robust framework for future advancements in the LMM design.
https://con-textual.github.io/
UNIMO-G: Unified Image Generation through Multimodal Conditional
  Diffusion
Authors: name: Wei Li, name: Xue Xu, name: Jiachen Liu, name: Xinyan Xiao
Published: 2024-01-24 11:36:44

Existing text-to-image diffusion models primarily generate images from text
prompts. However, the inherent conciseness of textual descriptions poses
challenges in faithfully synthesizing images with intricate details, such as
specific entities or scenes. This paper presents UNIMO-G, a simple
multimodal conditional diffusion framework that operates on multimodal prompts
with interleaved textual and visual inputs, which demonstrates a unified
ability for both text-driven and subject-driven image generation. UNIMO-G
comprises two core components: a Multimodal Large Language Model (MLLM) for
encoding multimodal prompts, and a conditional denoising diffusion network for
generating images based on the encoded multimodal input. We leverage a
two-stage training strategy to effectively train the framework: firstly
pre-training on large-scale text-image pairs to develop conditional image
generation capabilities, and then instruction tuning with multimodal prompts to
achieve unified image generation proficiency. A well-designed data processing
pipeline involving language grounding and image segmentation is employed to
construct multi-modal prompts. UNIMO-G excels in both text-to-image generation
and zero-shot subject-driven synthesis, and is notably effective in generating
high-fidelity images from complex multimodal prompts involving multiple image
entities.
GALA: Generating Animatable Layered Assets from a Single Scan
Authors: name: Taeksoo Kim, name: Byungjun Kim, name: Shunsuke Saito, name: Hanbyul Joo
Published: 2024-01-23 18:59:59

We present GALA, a framework that takes as input a single-layer clothed 3D
human mesh and decomposes it into complete multi-layered 3D assets. The outputs
can then be combined with other assets to create novel clothed human avatars
with any pose. Existing reconstruction approaches often treat clothed humans as
a single-layer of geometry and overlook the inherent compositionality of humans
with hairstyles, clothing, and accessories, thereby limiting the utility of the
meshes for downstream applications. Decomposing a single-layer mesh into
separate layers is a challenging task because it requires the synthesis of
plausible geometry and texture for the severely occluded regions. Moreover,
even with successful decomposition, meshes are not normalized in terms of poses
and body shapes, failing coherent composition with novel identities and poses.
To address these challenges, we propose to leverage the general knowledge of a
pretrained 2D diffusion model as geometry and appearance prior for humans and
other assets. We first separate the input mesh using the 3D surface
segmentation extracted from multi-view 2D segmentations. Then we synthesize the
missing geometry of different layers in both posed and canonical spaces using a
novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete
inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its
texture to obtain the complete appearance including the initially occluded
regions. Through a series of decomposition steps, we obtain multiple layers of
3D assets in a shared canonical space normalized in terms of poses and human
shapes, hence supporting effortless composition to novel identities and
reanimation with novel poses. Our experiments demonstrate the effectiveness of
our approach for decomposition, canonicalization, and composition tasks
compared to existing solutions.
Lumiere: A Space-Time Diffusion Model for Video Generation
Authors: name: Omer Bar-Tal, name: Hila Chefer, name: Omer Tov, name: Charles Herrmann, name: Roni Paiss, name: Shiran Zada, name: Ariel Ephrat, name: Junhwa Hur, name: Yuanzhen Li, name: Tomer Michaeli, name: Oliver Wang, name: Deqing Sun, name: Tali Dekel, name: Inbar Mosseri
Published: 2024-01-23 18:05:25

We introduce Lumiere -- a text-to-video diffusion model designed for
synthesizing videos that portray realistic, diverse and coherent motion -- a
pivotal challenge in video synthesis. To this end, we introduce a Space-Time
U-Net architecture that generates the entire temporal duration of the video at
once, through a single pass in the model. This is in contrast to existing video
models which synthesize distant keyframes followed by temporal super-resolution
-- an approach that inherently makes global temporal consistency difficult to
achieve. By deploying both spatial and (importantly) temporal down- and
up-sampling and leveraging a pre-trained text-to-image diffusion model, our
model learns to directly generate a full-frame-rate, low-resolution video by
processing it in multiple space-time scales. We demonstrate state-of-the-art
text-to-video generation results, and show that our design easily facilitates a
wide range of content creation tasks and video editing applications, including
image-to-video, video inpainting, and stylized generation.
AutoRT: Embodied Foundation Models for Large Scale Orchestration of
  Robotic Agents
Authors: name: Michael Ahn, name: Debidatta Dwibedi, name: Chelsea Finn, name: Montse Gonzalez Arenas, name: Keerthana Gopalakrishnan, name: Karol Hausman, name: Brian Ichter, name: Alex Irpan, name: Nikhil Joshi, name: Ryan Julian, name: Sean Kirmani, name: Isabel Leal, name: Edward Lee, name: Sergey Levine, name: Yao Lu, name: Isabel Leal, name: Sharath Maddineni, name: Kanishka Rao, name: Dorsa Sadigh, name: Pannag Sanketi, name: Pierre Sermanet, name: Quan Vuong, name: Stefan Welker, name: Fei Xia, name: Ted Xiao, name: Peng Xu, name: Steve Xu, name: Zhuo Xu
Published: 2024-01-23 18:45:54

Foundation models that incorporate language, vision, and more recently
actions have revolutionized the ability to harness internet scale data to
reason about useful tasks. However, one of the key challenges of training
embodied foundation models is the lack of data grounded in the physical world.
In this paper, we propose AutoRT, a system that leverages existing foundation
models to scale up the deployment of operational robots in completely unseen
scenarios with minimal human supervision. AutoRT leverages vision-language
models (VLMs) for scene understanding and grounding, and further uses large
language models (LLMs) for proposing diverse and novel instructions to be
performed by a fleet of robots. Guiding data collection by tapping into the
knowledge of foundation models enables AutoRT to effectively reason about
autonomy tradeoffs and safety while significantly scaling up data collection
for robot learning. We demonstrate AutoRT proposing instructions to over 20
robots across multiple buildings and collecting 77k real robot episodes via
both teleoperation and autonomous robot policies. We experimentally show that
such "in-the-wild" data collected by AutoRT is significantly more diverse, and
that AutoRT's use of LLMs allows for instruction following data collection
robots that can align to human preferences.
Multilingual and Fully Non-Autoregressive ASR with Large Language Model
  Fusion: A Comprehensive Study
Authors: name: W. Ronny Huang, name: Cyril Allauzen, name: Tongzhou Chen, name: Kilol Gupta, name: Ke Hu, name: James Qin, name: Yu Zhang, name: Yongqiang Wang, name: Shuo-Yiin Chang, name: Tara N. Sainath
Published: 2024-01-23 14:19:01

In the era of large models, the autoregressive nature of decoding often
results in latency serving as a significant bottleneck. We propose a
non-autoregressive LM-fused ASR system that effectively leverages the
parallelization capabilities of accelerator hardware. Our approach combines the
Universal Speech Model (USM) and the PaLM 2 language model in per-segment
scoring mode, achieving an average relative WER improvement across all
languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our
comprehensive ablation study analyzes key parameters such as LLM size, context
length, vocabulary size, fusion methodology. For instance, we explore the
impact of LLM size ranging from 128M to 340B parameters on ASR performance.
This study provides valuable insights into the factors influencing the
effectiveness of practical large-scale LM-fused speech recognition systems.
BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language
  Models
Authors: name: Feng Lin, name: Hanling Yi, name: Hongbin Li, name: Yifan Yang, name: Xiaotian Yu, name: Guangming Lu, name: Rong Xiao
Published: 2024-01-23 06:36:49

Large language models (LLMs) commonly employ autoregressive generation during
inference, leading to high memory bandwidth demand and consequently extended
latency. To mitigate this inefficiency, we present Bi-directional Tuning for
lossless Acceleration (BiTA), an innovative method expediting LLMs via
streamlined semi-autoregressive generation and draft verification. Inspired by
the concept of prompt tuning, we enhance LLMs with a parameter-efficient design
called bi-directional tuning for the capability in semi-autoregressive
generation. Employing efficient tree-based decoding, the models perform draft
candidate generation and verification in parallel, ensuring outputs identical
to their autoregressive counterparts under greedy sampling. BiTA serves as a
lightweight plug-in module, seamlessly boosting the inference efficiency of
existing LLMs without requiring additional assistance models or incurring
significant extra memory costs. Applying the proposed BiTA, LLaMA-2-70B-Chat
achieves a 2.7times speedup on the MT-Bench benchmark. Extensive experiments
confirm our method surpasses state-of-the-art acceleration techniques.
Small Language Model Meets with Reinforced Vision Vocabulary
Authors: name: Haoran Wei, name: Lingyu Kong, name: Jinyue Chen, name: Liang Zhao, name: Zheng Ge, name: En Yu, name: Jianjian Sun, name: Chunrui Han, name: Xiangyu Zhang
Published: 2024-01-23 05:55:26

Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI
community. However, the relatively large number of parameters (more than 7B) of
popular LVLMs makes it difficult to train and deploy on consumer GPUs,
discouraging many researchers with limited resources. Imagine how cool it would
be to experience all the features of current LVLMs on an old GTX1080ti (our
only game card). Accordingly, we present Vary-toy in this report, a small-size
Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we
introduce an improved vision vocabulary, allowing the model to not only possess
all features of Vary but also gather more generality. Specifically, we replace
negative samples of natural images with positive sample data driven by object
detection in the procedure of generating vision vocabulary, more sufficiently
utilizing the capacity of the vocabulary network and enabling it to efficiently
encode visual information corresponding to natural objects. For experiments,
Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1%
accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on
the homepage.
Large Language Models are Superpositions of All Characters: Attaining
  Arbitrary Role-play via Self-Alignment
Authors: name: Keming Lu, name: Bowen Yu, name: Chang Zhou, name: Jingren Zhou
Published: 2024-01-23 03:56:22

Considerable efforts have been invested in augmenting the role-playing
proficiency of open-source large language models (LLMs) by emulating
proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor
role-play capabilities, owing to the extensive knowledge of characters and
potential dialogues ingrained in their vast training corpora. Thus, in this
study, we introduce Ditto, a self-alignment method for role-play. Ditto
capitalizes on character knowledge, encouraging an instruction-following LLM to
simulate role-play dialogues as a variant of reading comprehension. This method
creates a role-play training set comprising 4,000 characters, surpassing the
scale of currently available datasets by tenfold regarding the number of roles.
Subsequently, we fine-tune the LLM using this self-generated dataset to augment
its role-playing capabilities. Upon evaluating our meticulously constructed and
reproducible role-play benchmark and the roleplay subset of MT-Bench, Ditto, in
various parameter scales, consistently maintains a consistent role identity and
provides accurate role-specific knowledge in multi-turn role-play
conversations. Notably, it outperforms all open-source role-play baselines,
showcasing performance levels comparable to advanced proprietary chatbots.
Furthermore, we present the first comprehensive cross-supervision alignment
experiment in the role-play domain, revealing that the intrinsic capabilities
of LLMs confine the knowledge within role-play. Meanwhile, the role-play styles
can be easily acquired with the guidance of smaller models. We open-source
related resources at https://github.com/OFA-Sys/Ditto.
Orion-14B: Open-source Multilingual Large Language Models
Authors: name: Du Chen, name: Yi Huang, name: Xiaopu Li, name: Yongqiang Li, name: Yongqiang Liu, name: Haihui Pan, name: Leichao Xu, name: Dacheng Zhang, name: Zhipeng Zhang, name: Kun Han
Published: 2024-01-20 12:29:27

In this study, we introduce Orion-14B, a collection of multilingual large
language models with 14 billion parameters. We utilize a data scheduling
approach to train a foundational model on a diverse corpus of 2.5 trillion
tokens, sourced from texts in English, Chinese, Japanese, Korean, and other
languages. Additionally, we fine-tuned a series of models tailored for
conversational applications and other specific use cases. Our evaluation
results demonstrate that Orion-14B achieves state-of-the-art performance across
a broad spectrum of tasks. We make the Orion-14B model family and its
associated code publicly accessible https://github.com/OrionStarAI/Orion,
aiming to inspire future research and practical applications in the field.
Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding
Authors: name: Mirac Suzgun, name: Adam Tauman Kalai
Published: 2024-01-23 18:22:19

We introduce meta-prompting, an effective scaffolding technique designed to
enhance the functionality of language models (LMs). This approach transforms a
single LM into a multi-faceted conductor, adept at managing and integrating
multiple independent LM queries. By employing high-level instructions,
meta-prompting guides the LM to break down complex tasks into smaller, more
manageable subtasks. These subtasks are then handled by distinct "expert"
instances of the same LM, each operating under specific, tailored instructions.
Central to this process is the LM itself, in its role as the conductor, which
ensures seamless communication and effective integration of the outputs from
these expert models. It additionally employs its inherent critical thinking and
robust verification processes to refine and authenticate the end result. This
collaborative prompting approach empowers a single LM to simultaneously act as
a comprehensive orchestrator and a panel of diverse experts, significantly
enhancing its performance across a wide array of tasks. The zero-shot,
task-agnostic nature of meta-prompting greatly simplifies user interaction by
obviating the need for detailed, task-specific instructions. Furthermore, our
research demonstrates the seamless integration of external tools, such as a
Python interpreter, into the meta-prompting framework, thereby broadening its
applicability and utility. Through rigorous experimentation with GPT-4, we
establish the superiority of meta-prompting over conventional scaffolding
methods: When averaged across all tasks, including the Game of 24,
Checkmate-in-One, and Python Programming Puzzles, meta-prompting, augmented
with a Python interpreter functionality, surpasses standard prompting by 17.1%,
expert (dynamic) prompting by 17.3%, and multipersona prompting by 15.2%.
Large-scale Reinforcement Learning for Diffusion Models
Authors: name: Yinan Zhang, name: Eric Tzeng, name: Yilun Du, name: Dmitry Kislyuk
Published: 2024-01-20 08:10:43

Text-to-image diffusion models are a class of deep generative models that
have demonstrated an impressive capacity for high-quality image generation.
However, these models are susceptible to implicit biases that arise from
web-scale text-image training pairs and may inaccurately model aspects of
images we care about. This can result in suboptimal samples, model bias, and
images that do not align with human ethics and preferences. In this paper, we
present an effective scalable algorithm to improve diffusion models using
Reinforcement Learning (RL) across a diverse set of reward functions, such as
human preference, compositionality, and fairness over millions of images. We
illustrate how our approach substantially outperforms existing methods for
aligning diffusion models with human preferences. We further illustrate how
this substantially improves pretrained Stable Diffusion (SD) models, generating
samples that are preferred by humans 80.3% of the time over those from the base
SD model while simultaneously improving both the composition and diversity of
generated samples.