Orionnnn/hub-stats huggingface dataset:
Veagle: Advancements in Multimodal Representation Learning
Authors: name: Rajat Chawla, name: Arkajit Datta, name: Tushar Verma, name: Adarsh Jha, name: Anmol Gautam, name: Ayush Vatsal, name: Sukrit Chaterjee, name: Mukunda NS, name: Ishaan Bhola
Published: 2024-01-18 12:45:25

Lately, researchers in artificial intelligence have been really interested in
how language and vision come together, giving rise to the development of
multimodal models that aim to seamlessly integrate textual and visual
information. Multimodal models, an extension of Large Language Models (LLMs),
have exhibited remarkable capabilities in addressing a diverse array of tasks,
ranging from image captioning and visual question answering (VQA) to visual
grounding. While these models have showcased significant advancements,
challenges persist in accurately interpreting images and answering the
question, a common occurrence in real-world scenarios. This paper introduces a
novel approach to enhance the multimodal capabilities of existing models. In
response to the limitations observed in current Vision Language Models (VLMs)
and Multimodal Large Language Models (MLLMs), our proposed model Veagle,
incorporates a unique mechanism inspired by the successes and insights of
previous works. Veagle leverages a dynamic mechanism to project encoded visual
information directly into the language model. This dynamic approach allows for
a more nuanced understanding of intricate details present in visual contexts.
To validate the effectiveness of Veagle, we conduct comprehensive experiments
on benchmark datasets, emphasizing tasks such as visual question answering and
image understanding. Our results indicate a improvement of 5-6 \% in
performance, with Veagle outperforming existing models by a notable margin. The
outcomes underscore the model's versatility and applicability beyond
traditional benchmarks.
Video Mamba Suite: State Space Model as a Versatile Alternative for
  Video Understanding
Authors: name: Guo Chen, name: Yifei Huang, name: Jilan Xu, name: Baoqi Pei, name: Zhe Chen, name: Zhiqi Li, name: Jiahao Wang, name: Kunchang Li, name: Tong Lu, name: Limin Wang
Published: 2024-03-14 17:57:07

Understanding videos is one of the fundamental directions in computer vision
research, with extensive efforts dedicated to exploring various architectures
such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state
space model, e.g., Mamba, shows promising traits to extend its success in long
sequence modeling to video modeling. To assess whether Mamba can be a viable
alternative to Transformers in the video understanding domain, in this work, we
conduct a comprehensive set of studies, probing different roles Mamba can play
in modeling videos, while investigating diverse tasks where Mamba could exhibit
superiority. We categorize Mamba into four roles for modeling videos, deriving
a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12
video understanding tasks. Our extensive experiments reveal the strong
potential of Mamba on both video-only and video-language tasks while showing
promising efficiency-performance trade-offs. We hope this work could provide
valuable data points and insights for future research on video understanding.
Code is public: https://github.com/OpenGVLab/video-mamba-suite.
LocalMamba: Visual State Space Model with Windowed Selective Scan
Authors: name: Tao Huang, name: Xiaohuan Pei, name: Shan You, name: Fei Wang, name: Chen Qian, name: Chang Xu
Published: 2024-03-14 12:32:40

Recent advancements in state space models, notably Mamba, have demonstrated
significant progress in modeling long sequences for tasks like language
understanding. Yet, their application in vision tasks has not markedly
surpassed the performance of traditional Convolutional Neural Networks (CNNs)
and Vision Transformers (ViTs). This paper posits that the key to enhancing
Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling.
Traditional ViM approaches, which flatten spatial tokens, overlook the
preservation of local 2D dependencies, thereby elongating the distance between
adjacent tokens. We introduce a novel local scanning strategy that divides
images into distinct windows, effectively capturing local dependencies while
maintaining a global perspective. Additionally, acknowledging the varying
preferences for scan patterns across different network layers, we propose a
dynamic method to independently search for the optimal scan choices for each
layer, substantially improving performance. Extensive experiments across both
plain and hierarchical models underscore our approach's superiority in
effectively capturing image representations. For example, our model
significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs.
Code is available at: https://github.com/hunto/LocalMamba.
Unlocking the conversion of Web Screenshots into HTML Code with the
  WebSight Dataset
Authors: name: Hugo Laurençon, name: Léo Tronchon, name: Victor Sanh
Published: 2024-03-14 01:40:40

Using vision-language models (VLMs) in web development presents a promising
strategy to increase efficiency and unblock no-code solutions: by providing a
screenshot or a sketch of a UI, a VLM could generate the code to reproduce it,
for instance in a language like HTML. Despite the advancements in VLMs for
various tasks, the specific challenge of converting a screenshot into a
corresponding HTML has been minimally explored. We posit that this is mainly
due to the absence of a suitable, high-quality dataset. This work introduces
WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and
their corresponding screenshots. We fine-tune a foundational VLM on our dataset
and show proficiency in converting webpage screenshots to functional HTML code.
To accelerate the research in this area, we open-source WebSight.
VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision
  Understanding
Authors: name: Chris Kelly, name: Luhui Hu, name: Jiayin Hu, name: Yu Tian, name: Deshun Yang, name: Bang Yang, name: Cindy Yang, name: Zihao Li, name: Zaoshan Huang, name: Yuexian Zou
Published: 2024-03-14 16:13:00

The evolution of text to visual components facilitates people's daily lives,
such as generating image, videos from text and identifying the desired elements
within the images. Computer vision models involving the multimodal abilities in
the previous days are focused on image detection, classification based on
well-defined objects. Large language models (LLMs) introduces the
transformation from nature language to visual objects, which present the visual
layout for text contexts. OpenAI GPT-4 has emerged as the pinnacle in LLMs,
while the computer vision (CV) domain boasts a plethora of state-of-the-art
(SOTA) models and algorithms to convert 2D images to their 3D representations.
However, the mismatching between the algorithms with the problem could lead to
undesired results. In response to this challenge, we propose an unified
VisionGPT-3D framework to consolidate the state-of-the-art vision models,
thereby facilitating the development of vision-oriented AI. VisionGPT-3D
provides a versatile multimodal framework building upon the strengths of
multimodal foundation models. It seamlessly integrates various SOTA vision
models and brings the automation in the selection of SOTA vision models,
identifies the suitable 3D mesh creation algorithms corresponding to 2D depth
maps analysis, generates optimal results based on diverse multimodal inputs
such as text prompts.
  Keywords: VisionGPT-3D, 3D vision understanding, Multimodal agent
Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering
Authors: name: Zeyu Liu, name: Weicong Liang, name: Zhanhao Liang, name: Chong Luo, name: Ji Li, name: Gao Huang, name: Yuhui Yuan
Published: 2024-03-14 17:55:33

Visual text rendering poses a fundamental challenge for contemporary
text-to-image generation models, with the core problem lying in text encoder
deficiencies. To achieve accurate text rendering, we identify two crucial
requirements for text encoders: character awareness and alignment with glyphs.
Our solution involves crafting a series of customized text encoder, Glyph-ByT5,
by fine-tuning the character-aware ByT5 encoder using a meticulously curated
paired glyph-text dataset. We present an effective method for integrating
Glyph-ByT5 with SDXL, resulting in the creation of the Glyph-SDXL model for
design image generation. This significantly enhances text rendering accuracy,
improving it from less than 20% to nearly 90% on our design image
benchmark. Noteworthy is Glyph-SDXL's newfound ability for text paragraph
rendering, achieving high spelling accuracy for tens to hundreds of characters
with automated multi-line layouts. Finally, through fine-tuning Glyph-SDXL with
a small set of high-quality, photorealistic images featuring visual text, we
showcase a substantial improvement in scene text rendering capabilities in
open-domain real images. These compelling outcomes aim to encourage further
exploration in designing customized text encoders for diverse and challenging
tasks.
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Authors: name: Brandon McKinzie, name: Zhe Gan, name: Jean-Philippe Fauconnier, name: Sam Dodge, name: Bowen Zhang, name: Philipp Dufter, name: Dhruti Shah, name: Xianzhi Du, name: Futang Peng, name: Floris Weers, name: Anton Belyi, name: Haotian Zhang, name: Karanjeet Singh, name: Doug Kang, name: Hongyu Hè, name: Max Schwarzer, name: Tom Gunter, name: Xiang Kong, name: Aonan Zhang, name: Jianyu Wang, name: Chong Wang, name: Nan Du, name: Tao Lei, name: Sam Wiseman, name: Guoli Yin, name: Mark Lee, name: Zirui Wang, name: Ruoming Pang, name: Peter Grasch, name: Alexander Toshev, name: Yinfei Yang
Published: 2024-03-14 17:51:32

In this work, we discuss building performant Multimodal Large Language Models
(MLLMs). In particular, we study the importance of various architecture
components and data choices. Through careful and comprehensive ablations of the
image encoder, the vision language connector, and various pre-training data
choices, we identified several crucial design lessons. For example, we
demonstrate that for large-scale multimodal pre-training using a careful mix of
image-caption, interleaved image-text, and text-only data is crucial for
achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks,
compared to other published pre-training results. Further, we show that the
image encoder together with image resolution and the image token count has
substantial impact, while the vision-language connector design is of
comparatively negligible importance. By scaling up the presented recipe, we
build MM1, a family of multimodal models up to 30B parameters, consisting of
both dense models and mixture-of-experts (MoE) variants, that are SOTA in
pre-training metrics and achieve competitive performance after supervised
fine-tuning on a range of established multimodal benchmarks. Thanks to
large-scale pre-training, MM1 enjoys appealing properties such as enhanced
in-context learning, and multi-image reasoning, enabling few-shot
chain-of-thought prompting.
Video Editing via Factorized Diffusion Distillation
Authors: name: Uriel Singer, name: Amit Zohar, name: Yuval Kirstain, name: Shelly Sheynin, name: Adam Polyak, name: Devi Parikh, name: Yaniv Taigman
Published: 2024-03-14 12:22:54

We introduce Emu Video Edit (EVE), a model that establishes a new
state-of-the art in video editing without relying on any supervised video
editing data. To develop EVE we separately train an image editing adapter and a
video generation adapter, and attach both to the same text-to-image model.
Then, to align the adapters towards video editing we introduce a new
unsupervised distillation procedure, Factorized Diffusion Distillation. This
procedure distills knowledge from one or more teachers simultaneously, without
any supervised data. We utilize this procedure to teach EVE to edit videos by
jointly distilling knowledge to (i) precisely edit each individual frame from
the image editing adapter, and (ii) ensure temporal consistency among the
edited frames using the video generation adapter. Finally, to demonstrate the
potential of our approach in unlocking other capabilities, we align additional
combinations of adapters
Scaling Up Dynamic Human-Scene Interaction Modeling
Authors: name: Nan Jiang, name: Zhiyuan Zhang, name: Hongjie Li, name: Xiaoxuan Ma, name: Zan Wang, name: Yixin Chen, name: Tengyu Liu, name: Yixin Zhu, name: Siyuan Huang
Published: 2024-03-13 15:45:04

Confronting the challenges of data scarcity and advanced motion synthesis in
human-scene interaction modeling, we introduce the TRUMANS dataset alongside a
novel HSI motion synthesis method. TRUMANS stands as the most comprehensive
motion-captured HSI dataset currently available, encompassing over 15 hours of
human interactions across 100 indoor scenes. It intricately captures whole-body
human motions and part-level object dynamics, focusing on the realism of
contact. This dataset is further scaled up by transforming physical
environments into exact virtual models and applying extensive augmentations to
appearance and motion for both humans and objects while maintaining interaction
fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model
that efficiently generates HSI sequences of any length, taking into account
both scene context and intended actions. In experiments, our approach shows
remarkable zero-shot generalizability on a range of 3D scene datasets (e.g.,
PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic
original motion-captured sequences, as confirmed by quantitative experiments
and human studies.
Language models scale reliably with over-training and on downstream
  tasks
Authors: name: Samir Yitzhak Gadre, name: Georgios Smyrnis, name: Vaishaal Shankar, name: Suchin Gururangan, name: Mitchell Wortsman, name: Rulin Shao, name: Jean Mercat, name: Alex Fang, name: Jeffrey Li, name: Sedrick Keh, name: Rui Xin, name: Marianna Nezhurina, name: Igor Vasiljevic, name: Jenia Jitsev, name: Alexandros G. Dimakis, name: Gabriel Ilharco, name: Shuran Song, name: Thomas Kollar, name: Yair Carmon, name: Achal Dave, name: Reinhard Heckel, name: Niklas Muennighoff, name: Ludwig Schmidt
Published: 2024-03-13 13:54:00

Scaling laws are useful guides for developing language models, but there are
still gaps between current scaling studies and how language models are
ultimately trained and evaluated. For instance, scaling is usually studied in
the compute-optimal training regime (i.e., "Chinchilla optimal" regime);
however, in practice, models are often over-trained to reduce inference costs.
Moreover, scaling laws mostly predict loss on next-token prediction, but
ultimately models are compared based on downstream task performance. In this
paper, we address both shortcomings. To do so, we create a testbed of 104
models with 0.011B to 6.9B parameters trained with various numbers of tokens on
three data distributions. First, we investigate scaling in the over-trained
regime. We fit scaling laws that extrapolate in both the number of model
parameters and the ratio of training tokens to parameters. This enables us to
predict the validation loss of a 1.4B parameter, 900B token run (i.e.,
32times over-trained) and a 6.9B parameter, 138B token
runx2014each from experiments that take 300times less compute.
Second, we relate the perplexity of a language model to its downstream task
performance via a power law. We use this law to predict top-1 error averaged
over downstream tasks for the two aforementioned models using experiments that
take 20times less compute. Our experiments are available at
https://github.com/mlfoundations/scaling.
Simple and Scalable Strategies to Continually Pre-train Large Language
  Models
Authors: name: Adam Ibrahim, name: Benjamin Thérien, name: Kshitij Gupta, name: Mats L. Richter, name: Quentin Anthony, name: Timothée Lesort, name: Eugene Belilovsky, name: Irina Rish
Published: 2024-03-13 17:58:57

Large language models (LLMs) are routinely pre-trained on billions of tokens,
only to start the process over again once new data becomes available. A much
more efficient solution is to continually pre-train these models, saving
significant compute compared to re-training. However, the distribution shift
induced by new data typically results in degraded performance on previous data
or poor adaptation to the new data. In this work, we show that a simple and
scalable combination of learning rate (LR) re-warming, LR re-decaying, and
replay of previous data is sufficient to match the performance of fully
re-training from scratch on all available data, as measured by final loss and
language model (LM) evaluation benchmarks. Specifically, we show this for a
weak but realistic distribution shift between two commonly used LLM
pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution
shift (EnglishrightarrowGerman) at the 405M parameter model scale with
large dataset sizes (hundreds of billions of tokens). Selecting the weak but
realistic shift for larger-scale experiments, we also find that our continual
learning strategies match the re-training baseline for a 10B parameter LLM. Our
results demonstrate that LLMs can be successfully updated via simple and
scalable continual learning strategies, matching the re-training baseline using
only a fraction of the compute. Finally, inspired by previous work, we propose
alternatives to the cosine learning rate schedule that help circumvent
forgetting induced by LR re-warming and that are not bound to a fixed token
budget.
On the Societal Impact of Open Foundation Models
Authors: name: Sayash Kapoor, name: Rishi Bommasani, name: Kevin Klyman, name: Shayne Longpre, name: Ashwin Ramaswami, name: Peter Cihon, name: Aspen Hopkins, name: Kevin Bankston, name: Stella Biderman, name: Miranda Bogen, name: Rumman Chowdhury, name: Alex Engler, name: Peter Henderson, name: Yacine Jernite, name: Seth Lazar, name: Stefano Maffulli, name: Alondra Nelson, name: Joelle Pineau, name: Aviya Skowron, name: Dawn Song, name: Victor Storchan, name: Daniel Zhang, name: Daniel E. Ho, name: Percy Liang, name: Arvind Narayanan
Published: 2024-02-27 16:49:53

Foundation models are powerful technologies: how they are released publicly
directly shapes their societal impact. In this position paper, we focus on open
foundation models, defined here as those with broadly available model weights
(e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties
(e.g. greater customizability, poor monitoring) of open foundation models that
lead to both their benefits and risks. Open foundation models present
significant benefits, with some caveats, that span innovation, competition, the
distribution of decision-making power, and transparency. To understand their
risks of misuse, we design a risk assessment framework for analyzing their
marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons),
we find that current research is insufficient to effectively characterize the
marginal risk of open foundation models relative to pre-existing technologies.
The framework helps explain why the marginal risk is low in some cases,
clarifies disagreements about misuse risks by revealing that past work has
focused on different subsets of the framework with different assumptions, and
articulates a way forward for more constructive debate. Overall, our work helps
support a more grounded assessment of the societal impact of open foundation
models by outlining what research is needed to empirically validate their
theoretical benefits and risks.
SOTOPIA-π: Interactive Learning of Socially Intelligent Language
  Agents
Authors: name: Ruiyi Wang, name: Haofei Yu, name: Wenxin Zhang, name: Zhengyang Qi, name: Maarten Sap, name: Graham Neubig, name: Yonatan Bisk, name: Hao Zhu
Published: 2024-03-13 17:17:48

Humans learn social skills through both imitation and social interaction.
This social learning process is largely understudied by existing research on
building language agents. Motivated by this gap, we propose an interactive
learning method, SOTOPIA-pi, improving the social intelligence of language
agents. This method leverages behavior cloning and self-reinforcement training
on filtered social interaction data according to large language model (LLM)
ratings. We show that our training method allows a 7B LLM to reach the social
goal completion ability of an expert model (GPT-4-based agent), while improving
the safety of language agents and maintaining general QA ability on the MMLU
benchmark. We also find that this training paradigm uncovers some difficulties
in LLM-based evaluation of social intelligence: LLM-based evaluators
overestimate the abilities of the language agents trained specifically for
social interaction.
Gemma: Open Models Based on Gemini Research and Technology
Authors: name: Gemma Team, name: Thomas Mesnard, name: Cassidy Hardin, name: Robert Dadashi, name: Surya Bhupatiraju, name: Shreya Pathak, name: Laurent Sifre, name: Morgane Rivière, name: Mihir Sanjay Kale, name: Juliette Love, name: Pouya Tafti, name: Léonard Hussenot, name: Aakanksha Chowdhery, name: Adam Roberts, name: Aditya Barua, name: Alex Botev, name: Alex Castro-Ros, name: Ambrose Slone, name: Amélie Héliou, name: Andrea Tacchetti, name: Anna Bulanova, name: Antonia Paterson, name: Beth Tsai, name: Bobak Shahriari, name: Charline Le Lan, name: Christopher A. Choquette-Choo, name: Clément Crepy, name: Daniel Cer, name: Daphne Ippolito, name: David Reid, name: Elena Buchatskaya, name: Eric Ni, name: Eric Noland, name: Geng Yan, name: George Tucker, name: George-Christian Muraru, name: Grigory Rozhdestvenskiy, name: Henryk Michalewski, name: Ian Tenney, name: Ivan Grishchenko, name: Jacob Austin, name: James Keeling, name: Jane Labanowski, name: Jean-Baptiste Lespiau, name: Jeff Stanway, name: Jenny Brennan, name: Jeremy Chen, name: Johan Ferret, name: Justin Chiu, name: Justin Mao-Jones, name: Katherine Lee, name: Kathy Yu, name: Katie Millican, name: Lars Lowe Sjoesund, name: Lisa Lee, name: Lucas Dixon, name: Machel Reid, name: Maciej Mikuła, name: Mateo Wirth, name: Michael Sharman, name: Nikolai Chinaev, name: Nithum Thain, name: Olivier Bachem, name: Oscar Chang, name: Oscar Wahltinez, name: Paige Bailey, name: Paul Michel, name: Petko Yotov, name: Pier Giuseppe Sessa, name: Rahma Chaabouni, name: Ramona Comanescu, name: Reena Jana, name: Rohan Anil, name: Ross McIlroy, name: Ruibo Liu, name: Ryan Mullins, name: Samuel L Smith, name: Sebastian Borgeaud, name: Sertan Girgin, name: Sholto Douglas, name: Shree Pandya, name: Siamak Shakeri, name: Soham De, name: Ted Klimenko, name: Tom Hennigan, name: Vlad Feinberg, name: Wojciech Stokowiec, name: Yu-hui Chen, name: Zafarali Ahmed, name: Zhitao Gong, name: Tris Warkentin, name: Ludovic Peran, name: Minh Giang, name: Clément Farabet, name: Oriol Vinyals, name: Jeff Dean, name: Koray Kavukcuoglu, name: Demis Hassabis, name: Zoubin Ghahramani, name: Douglas Eck, name: Joelle Barral, name: Fernando Pereira, name: Eli Collins, name: Armand Joulin, name: Noah Fiedel, name: Evan Senter, name: Alek Andreev, name: Kathleen Kenealy
Published: 2024-03-13 06:59:16

This work introduces Gemma, a family of lightweight, state-of-the art open
models built from the research and technology used to create Gemini models.
Gemma models demonstrate strong performance across academic benchmarks for
language understanding, reasoning, and safety. We release two sizes of models
(2 billion and 7 billion parameters), and provide both pretrained and
fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out
of 18 text-based tasks, and we present comprehensive evaluations of safety and
responsibility aspects of the models, alongside a detailed description of model
development. We believe the responsible release of LLMs is critical for
improving the safety of frontier models, and for enabling the next wave of LLM
innovations.