Orionnnn/hub-stats huggingface dataset:
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Authors: name: Gilles Baechler, name: Srinivas Sunkara, name: Maria Wang, name: Fedir Zubach, name: Hassan Mansoor, name: Vincent Etter, name: Victor Cărbune, name: Jason Lin, name: Jindong Chen, name: Abhanshu Sharma
Published: 2024-02-07 06:42:33

Screen user interfaces (UIs) and infographics, sharing similar visual
language and design principles, play important roles in human communication and
human-machine interaction. We introduce ScreenAI, a vision-language model that
specializes in UI and infographics understanding. Our model improves upon the
PaLI architecture with the flexible patching strategy of pix2struct and is
trained on a unique mixture of datasets. At the heart of this mixture is a
novel screen annotation task in which the model has to identify the type and
location of UI elements. We use these text annotations to describe screens to
Large Language Models and automatically generate question-answering (QA), UI
navigation, and summarization training datasets at scale. We run ablation
studies to demonstrate the impact of these design choices. At only 5B
parameters, ScreenAI achieves new state-of-the-artresults on UI- and
infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
InfographicVQA) compared to models of similar size. Finally, we release three
new datasets: one focused on the screen annotation task and two others focused
on question answering.
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Authors: name: Nate Gruver, name: Anuroop Sriram, name: Andrea Madotto, name: Andrew Gordon Wilson, name: C. Lawrence Zitnick, name: Zachary Ulissi
Published: 2024-02-06 20:35:28

We propose fine-tuning large language models for generation of stable
materials. While unorthodox, fine-tuning large language models on text-encoded
atomistic data is simple to implement yet reliable, with around 90% of sampled
structures obeying physical constraints on atom positions and charges. Using
energy above hull calculations from both learned ML potentials and
gold-standard DFT calculations, we show that our strongest model (fine-tuned
LLaMA-2 70B) can generate materials predicted to be metastable at about twice
the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text
prompting's inherent flexibility, our models can simultaneously be used for
unconditional generation of stable material, infilling of partial structures
and text-conditional generation. Finally, we show that language models' ability
to capture key symmetries of crystal structures improves with model scale,
suggesting that the biases of pretrained LLMs are surprisingly well-suited for
atomistic data.
BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
Authors: name: Wei Huang, name: Yangdong Liu, name: Haotong Qin, name: Ying Li, name: Shiming Zhang, name: Xianglong Liu, name: Michele Magno, name: Xiaojuan Qi
Published: 2024-02-06 09:26:34

Pretrained large language models (LLMs) exhibit exceptional general language
processing capabilities but come with significant demands on memory and
computational resources. As a powerful compression technology, binarization can
extremely reduce model weights to a mere 1 bit, lowering the expensive
computation and memory requirements. However, existing quantization techniques
fall short of maintaining LLM performance under ultra-low bit-widths. In
response to this challenge, we present BiLLM, a groundbreaking 1-bit
post-training quantization scheme tailored for pretrained LLMs. Based on the
weight distribution of LLMs, BiLLM first identifies and structurally selects
salient weights, and minimizes the compression loss through an effective binary
residual approximation strategy. Moreover, considering the bell-shaped
distribution of the non-salient weights, we propose an optimal splitting search
to group and binarize them accurately. BiLLM achieving for the first time
high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit
weights across various LLMs families and evaluation metrics, outperforms SOTA
quantization methods of LLM by significant margins. Moreover, BiLLM enables the
binarization process of the LLM with 7 billion weights within 0.5 hours on a
single GPU, demonstrating satisfactory time efficiency.
ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation
Authors: name: Weiming Ren, name: Harry Yang, name: Ge Zhang, name: Cong Wei, name: Xinrun Du, name: Stephen Huang, name: Wenhu Chen
Published: 2024-02-06 19:08:18

Image-to-video (I2V) generation aims to use the initial frame (alongside a
text prompt) to create a video sequence. A grand challenge in I2V generation is
to maintain visual consistency throughout the video: existing methods often
struggle to preserve the integrity of the subject, background, and style from
the first frame, as well as ensure a fluid and logical progression within the
video narrative. To mitigate these issues, we propose ConsistI2V, a
diffusion-based method to enhance visual consistency for I2V generation.
Specifically, we introduce (1) spatiotemporal attention over the first frame to
maintain spatial and motion consistency, (2) noise initialization from the
low-frequency band of the first frame to enhance layout consistency. These two
approaches enable ConsistI2V to generate highly consistent videos. We also
extend the proposed approaches to show their potential to improve consistency
in auto-regressive long video generation and camera motion control. To verify
the effectiveness of our method, we propose I2V-Bench, a comprehensive
evaluation benchmark for I2V generation. Our automatic and human evaluation
results demonstrate the superiority of ConsistI2V over existing methods.
EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters
Authors: name: Quan Sun, name: Jinsheng Wang, name: Qiying Yu, name: Yufeng Cui, name: Fan Zhang, name: Xiaosong Zhang, name: Xinlong Wang
Published: 2024-02-06 18:59:48

Scaling up contrastive language-image pretraining (CLIP) is critical for
empowering both vision and multimodal models. We present EVA-CLIP-18B, the
largest and most powerful open-source CLIP model to date, with 18-billion
parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an
exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized
image classification benchmarks, outperforming its forerunner EVA-CLIP
(5-billion parameters) and other open-source CLIP models by a large margin.
Remarkably, we observe a consistent performance improvement with the model size
scaling of EVA-CLIP, despite maintaining a constant training dataset of
2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly
available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B)
employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the
potential of EVA-style weak-to-strong visual model scaling. With our model
weights made publicly available, we hope to facilitate future research in
vision and multimodal foundation models.
IMUSIC: IMU-based Facial Expression Capture
Authors: name: Youjia Wang, name: Yiwen Wu, name: Ruiqian Li, name: Hengan Zhou, name: Hongyang Lin, name: Yingwenqi Jiang, name: Yingsheng Zhu, name: Guanpeng Long, name: Jingya Wang, name: Lan Xu, name: Jingyi Yu
Published: 2024-02-03 14:27:18

For facial motion capture and analysis, the dominated solutions are generally
based on visual cues, which cannot protect privacy and are vulnerable to
occlusions. Inertial measurement units (IMUs) serve as potential rescues yet
are mainly adopted for full-body motion capture. In this paper, we propose
IMUSIC to fill the gap, a novel path for facial expression capture using purely
IMU signals, significantly distant from previous visual solutions.The key
design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial
capture, companion with an anatomy-driven IMU placement scheme. Then, we
contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual
signals for diverse facial expressions and performances. Such unique
multi-modality brings huge potential for future directions like IMU-based
facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong
baseline approach to accurately predict facial blendshape parameters from
purely IMU signals. Specifically, we tailor a Transformer diffusion model with
a two-stage training strategy for this novel tracking task. The IMUSIC
framework empowers us to perform accurate facial capture in scenarios where
visual methods falter and simultaneously safeguard user privacy. We conduct
extensive experiments about both the IMU configuration and technical components
to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables
various potential and novel applications, i.e., privacy-protecting facial
capture, hybrid capture against occlusions, or detecting minute facial
movements that are often invisible through visual cues. We will release our
dataset and implementations to enrich more possibilities of facial capture and
analysis in our community.
EscherNet: A Generative Model for Scalable View Synthesis
Authors: name: Xin Kong, name: Shikun Liu, name: Xiaoyang Lyu, name: Marwan Taher, name: Xiaojuan Qi, name: Andrew J. Davison
Published: 2024-02-06 11:21:58

We introduce EscherNet, a multi-view conditioned diffusion model for view
synthesis. EscherNet learns implicit and generative 3D representations coupled
with a specialised camera positional encoding, allowing precise and continuous
relative control of the camera transformation between an arbitrary number of
reference and target views. EscherNet offers exceptional generality,
flexibility, and scalability in view synthesis -- it can generate more than 100
consistent target views simultaneously on a single consumer-grade GPU, despite
being trained with a fixed number of 3 reference views to 3 target views. As a
result, EscherNet not only addresses zero-shot novel view synthesis, but also
naturally unifies single- and multi-image 3D reconstruction, combining these
diverse tasks into a single, cohesive framework. Our extensive experiments
demonstrate that EscherNet achieves state-of-the-art performance in multiple
benchmarks, even when compared to methods specifically tailored for each
individual problem. This remarkable versatility opens up new directions for
designing scalable neural architectures for 3D vision. Project page:
https://kxhit.github.io/EscherNet.
Diffusion World Model
Authors: name: Zihan Ding, name: Amy Zhang, name: Yuandong Tian, name: Qinqing Zheng
Published: 2024-02-05 22:43:57

We introduce Diffusion World Model (DWM), a conditional diffusion model
capable of predicting multistep future states and rewards concurrently. As
opposed to traditional one-step dynamics models, DWM offers long-horizon
predictions in a single forward pass, eliminating the need for recursive
quires. We integrate DWM into model-based value estimation, where the
short-term return is simulated by future trajectories sampled from DWM. In the
context of offline reinforcement learning, DWM can be viewed as a conservative
value regularization through generative modeling. Alternatively, it can be seen
as a data source that enables offline Q-learning with synthetic data. Our
experiments on the D4RL dataset confirm the robustness of DWM to long-horizon
simulation. In terms of absolute performance, DWM significantly surpasses
one-step dynamics models with a 44% performance gain, and achieves
state-of-the-art performance.
MusicRL: Aligning Music Generation to Human Preferences
Authors: name: Geoffrey Cideron, name: Sertan Girgin, name: Mauro Verzetti, name: Damien Vincent, name: Matej Kastelic, name: Zalán Borsos, name: Brian McWilliams, name: Victor Ungureanu, name: Olivier Bachem, name: Olivier Pietquin, name: Matthieu Geist, name: Léonard Hussenot, name: Neil Zeghidour, name: Andrea Agostinelli
Published: 2024-02-06 18:36:52

We propose MusicRL, the first music generation system finetuned from human
feedback. Appreciation of text-to-music models is particularly subjective since
the concept of musicality as well as the specific intention behind a caption
are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a
retro guitar solo or a techno pop beat). Not only this makes supervised
training of such models challenging, but it also calls for integrating
continuous human feedback in their post-deployment finetuning. MusicRL is a
pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete
audio tokens finetuned with reinforcement learning to maximise sequence-level
rewards. We design reward functions related specifically to text-adherence and
audio quality with the help from selected raters, and use those to finetune
MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial
dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning
from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model
that incorporates human feedback at scale. Human evaluations show that both
MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU
combines the two approaches and results in the best model according to human
raters. Ablation studies shed light on the musical attributes influencing human
preferences, indicating that text adherence and quality only account for a part
of it. This underscores the prevalence of subjectivity in musical appreciation
and calls for further involvement of human listeners in the finetuning of music
generation models.
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning
  Tasks
Authors: name: Jongho Park, name: Jaeseung Park, name: Zheyang Xiong, name: Nayoung Lee, name: Jaewoong Cho, name: Samet Oymak, name: Kangwook Lee, name: Dimitris Papailiopoulos
Published: 2024-02-06 18:56:35

State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed
as alternatives to Transformer networks in language modeling, by incorporating
gating, convolutions, and input-dependent token selection to mitigate the
quadratic cost of multi-head attention. Although SSMs exhibit competitive
performance, their in-context learning (ICL) capabilities, a remarkable
emergent property of modern language models that enables task execution without
parameter optimization, remain underexplored compared to Transformers. In this
study, we evaluate the ICL performance of SSMs, focusing on Mamba, against
Transformer models across various tasks. Our results show that SSMs perform
comparably to Transformers in standard regression ICL tasks, while
outperforming them in tasks like sparse parity learning. However, SSMs fall
short in tasks involving non-standard retrieval functionality. To address these
limitations, we introduce a hybrid model, \variant, that combines Mamba with
attention blocks, surpassing individual models in tasks where they struggle
independently. Our findings suggest that hybrid architectures offer promising
avenues for enhancing ICL in language models.
CogCoM: Train Large Vision-Language Models Diving into Details through
  Chain of Manipulations
Authors: name: Ji Qi, name: Ming Ding, name: Weihan Wang, name: Yushi Bai, name: Qingsong Lv, name: Wenyi Hong, name: Bin Xu, name: Lei Hou, name: Juanzi Li, name: Yuxiao Dong, name: Jie Tang
Published: 2024-02-06 18:43:48

Vision-Language Models (VLMs) have demonstrated their widespread viability
thanks to extensive training in aligning visual instructions to answers.
However, this conclusive alignment leads models to ignore critical visual
reasoning, and further result in failures on meticulous visual problems and
unfaithful responses. In this paper, we propose Chain of Manipulations, a
mechanism that enables VLMs to solve problems with a series of manipulations,
where each manipulation refers to an operation on the visual input, either from
intrinsic abilities (e.g., grounding) acquired through prior training or from
imitating human-like behaviors (e.g., zoom in). This mechanism encourages VLMs
to generate faithful responses with evidential visual reasoning, and permits
users to trace error causes in the interpretable paths. We thus train CogCoM, a
general 17B VLM with a memory-based compatible architecture endowed this
reasoning mechanism. Experiments show that our model achieves the
state-of-the-art performance across 8 benchmarks from 3 categories, and a
limited number of training steps with the data swiftly gains a competitive
performance. The code and data are publicly available at
https://github.com/THUDM/CogCoM.
Scaling Laws for Downstream Task Performance of Large Language Models
Authors: name: Berivan Isik, name: Natalia Ponomareva, name: Hussein Hazimeh, name: Dimitris Paparas, name: Sergei Vassilvitskii, name: Sanmi Koyejo
Published: 2024-02-06 17:31:20

Scaling laws provide important insights that can guide the design of large
language models (LLMs). Existing work has primarily focused on studying scaling
laws for pretraining (upstream) loss. However, in transfer learning settings,
in which LLMs are pretrained on an unsupervised dataset and then finetuned on a
downstream task, we often also care about the downstream performance. In this
work, we study the scaling behavior in a transfer learning setting, where LLMs
are finetuned for machine translation tasks. Specifically, we investigate how
the choice of the pretraining data and its size affect downstream performance
(translation quality) as judged by two metrics: downstream cross-entropy and
BLEU score. Our experiments indicate that the size of the finetuning dataset
and the distribution alignment between the pretraining and downstream data
significantly influence the scaling behavior. With sufficient alignment, both
downstream cross-entropy and BLEU score improve monotonically with more
pretraining data. In such cases, we show that it is possible to predict the
downstream BLEU score with good accuracy using a log-law. However, there are
also cases where moderate misalignment causes the BLEU score to fluctuate or
get worse with more pretraining, whereas downstream cross-entropy monotonically
improves. By analyzing these observations, we provide new practical insights
for choosing appropriate pretraining data.
Multi-line AI-assisted Code Authoring
Authors: name: Omer Dunay, name: Daniel Cheng, name: Adam Tait, name: Parth Thakkar, name: Peter C Rigby, name: Andy Chiu, name: Imad Ahmad, name: Arun Ganesan, name: Chandra Maddila, name: Vijayaraghavan Murali, name: Ali Tayyebi, name: Nachiappan Nagappan
Published: 2024-02-06 16:48:50

CodeCompose is an AI-assisted code authoring tool powered by large language
models (LLMs) that provides inline suggestions to 10's of thousands of
developers at Meta. In this paper, we present how we scaled the product from
displaying single-line suggestions to multi-line suggestions. This evolution
required us to overcome several unique challenges in improving the usability of
these suggestions for developers.
  First, we discuss how multi-line suggestions can have a 'jarring' effect, as
the LLM's suggestions constantly move around the developer's existing code,
which would otherwise result in decreased productivity and satisfaction.
  Second, multi-line suggestions take significantly longer to generate; hence
we present several innovative investments we made to reduce the perceived
latency for users. These model-hosting optimizations sped up multi-line
suggestion latency by 2.5x.
  Finally, we conduct experiments on 10's of thousands of engineers to
understand how multi-line suggestions impact the user experience and contrast
this with single-line suggestions. Our experiments reveal that (i) multi-line
suggestions account for 42% of total characters accepted (despite only
accounting for 16% for displayed suggestions) (ii) multi-line suggestions
almost doubled the percentage of keystrokes saved for users from 9% to 17%.
Multi-line CodeCompose has been rolled out to all engineers at Meta, and less
than 1% of engineers have opted out of multi-line suggestions.
MobileVLM V2: Faster and Stronger Baseline for Vision Language Model
Authors: name: Xiangxiang Chu, name: Limeng Qiao, name: Xinyu Zhang, name: Shuang Xu, name: Fei Wei, name: Yang Yang, name: Xiaofei Sun, name: Yiming Hu, name: Xinyang Lin, name: Bo Zhang, name: Chunhua Shen
Published: 2024-02-06 07:16:36

We introduce MobileVLM V2, a family of significantly improved vision language
models upon MobileVLM, which proves that a delicate orchestration of novel
architectural design, an improved training scheme tailored for mobile VLMs, and
rich high-quality dataset curation can substantially benefit VLMs' performance.
Specifically, MobileVLM V2 1.7B achieves better or on-par performance on
standard VLM benchmarks compared with much larger VLMs at the 3B scale.
Notably, our 3B model outperforms a large variety of VLMs at the 7B+ scale. Our
models will be released at https://github.com/Meituan-AutoML/MobileVLM .
Vision Superalignment: Weak-to-Strong Generalization for Vision
  Foundation Models
Authors: name: Jianyuan Guo, name: Hanting Chen, name: Chengcheng Wang, name: Kai Han, name: Chang Xu, name: Yunhe Wang
Published: 2024-02-06 06:30:34

Recent advancements in large language models have sparked interest in their
extraordinary and near-superhuman capabilities, leading researchers to explore
methods for evaluating and optimizing these abilities, which is called
superalignment. In this context, our paper delves into the realm of vision
foundation models, focusing on the concept of weak-to-strong generalization,
which involves using a weaker model to supervise a stronger one, aiming to
enhance the latter's capabilities beyond the former's limits. We introduce a
novel and adaptively adjustable loss function for weak-to-strong supervision.
Our comprehensive experiments span various scenarios, including few-shot
learning, transfer learning, noisy label learning, and common knowledge
distillation settings. The results are striking: our approach not only exceeds
the performance benchmarks set by strong-to-strong generalization but also
surpasses the outcomes of fine-tuning strong models with whole datasets. This
compelling evidence underscores the significant potential of weak-to-strong
generalization, showcasing its capability to substantially elevate the
performance of vision foundation models. The code is available at
https://github.com/ggjy/vision_weak_to_strong.
Self-Discover: Large Language Models Self-Compose Reasoning Structures
Authors: name: Pei Zhou, name: Jay Pujara, name: Xiang Ren, name: Xinyun Chen, name: Heng-Tze Cheng, name: Quoc V. Le, name: Ed H. Chi, name: Denny Zhou, name: Swaroop Mishra, name: Huaixiu Steven Zheng
Published: 2024-02-06 01:13:53

We introduce SELF-DISCOVER, a general framework for LLMs to self-discover the
task-intrinsic reasoning structures to tackle complex reasoning problems that
are challenging for typical prompting methods. Core to the framework is a
self-discovery process where LLMs select multiple atomic reasoning modules such
as critical thinking and step-by-step thinking, and compose them into an
explicit reasoning structure for LLMs to follow during decoding. SELF-DISCOVER
substantially improves GPT-4 and PaLM 2's performance on challenging reasoning
benchmarks such as BigBench-Hard, grounded agent reasoning, and MATH, by as
much as 32% compared to Chain of Thought (CoT). Furthermore, SELF-DISCOVER
outperforms inference-intensive methods such as CoT-Self-Consistency by more
than 20%, while requiring 10-40x fewer inference compute. Finally, we show that
the self-discovered reasoning structures are universally applicable across
model families: from PaLM 2-L to GPT-4, and from GPT-4 to Llama2, and share
commonalities with human reasoning patterns.