Orionnnn/hub-stats huggingface dataset:
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Authors: name: Keen You, name: Haotian Zhang, name: Eldon Schoop, name: Floris Weers, name: Amanda Swearngin, name: Jeffrey Nichols, name: Yinfei Yang, name: Zhe Gan
Published: 2024-04-08 17:55:44

Recent advancements in multimodal large language models (MLLMs) have been
noteworthy, yet, these general-domain MLLMs often fall short in their ability
to comprehend and interact effectively with user interface (UI) screens. In
this paper, we present Ferret-UI, a new MLLM tailored for enhanced
understanding of mobile UI screens, equipped with referring, grounding, and
reasoning capabilities. Given that UI screens typically exhibit a more
elongated aspect ratio and contain smaller objects of interest (e.g., icons,
texts) than natural images, we incorporate "any resolution" on top of Ferret to
magnify details and leverage enhanced visual features. Specifically, each
screen is divided into 2 sub-images based on the original aspect ratio (i.e.,
horizontal division for portrait screens and vertical division for landscape
screens). Both sub-images are encoded separately before being sent to LLMs. We
meticulously gather training samples from an extensive range of elementary UI
tasks, such as icon recognition, find text, and widget listing. These samples
are formatted for instruction-following with region annotations to facilitate
precise referring and grounding. To augment the model's reasoning ability, we
further compile a dataset for advanced tasks, including detailed description,
perception/interaction conversations, and function inference. After training on
the curated datasets, Ferret-UI exhibits outstanding comprehension of UI
screens and the capability to execute open-ended instructions. For model
evaluation, we establish a comprehensive benchmark encompassing all the
aforementioned tasks. Ferret-UI excels not only beyond most open-source UI
MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video
  Understanding
Authors: name: Bo He, name: Hengduo Li, name: Young Kyun Jang, name: Menglin Jia, name: Xuefei Cao, name: Ashish Shah, name: Abhinav Shrivastava, name: Ser-Nam Lim
Published: 2024-04-08 17:59:24

With the success of large language models (LLMs), integrating the vision
model into LLMs to build vision-language foundation models has gained much more
interest recently. However, existing LLM-based large multimodal models (e.g.,
Video-LLaMA, VideoChat) can only take in a limited number of frames for short
video understanding. In this study, we mainly focus on designing an efficient
and effective model for long-term video understanding. Instead of trying to
process more frames simultaneously like most existing work, we propose to
process videos in an online manner and store past video information in a memory
bank. This allows our model to reference historical video content for long-term
analysis without exceeding LLMs' context length constraints or GPU memory
limits. Our memory bank can be seamlessly integrated into current multimodal
LLMs in an off-the-shelf manner. We conduct extensive experiments on various
video understanding tasks, such as long-video understanding, video question
answering, and video captioning, and our model can achieve state-of-the-art
performances across multiple datasets. Code available at
https://boheumd.github.io/MA-LMM/.
Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models
Authors: name: Zhengcong Fei, name: Mingyuan Fan, name: Changqian Yu, name: Debang Li, name: Junshi Huang
Published: 2024-04-06 02:54:35

Transformers have catalyzed advancements in computer vision and natural
language processing (NLP) fields. However, substantial computational complexity
poses limitations for their application in long-context tasks, such as
high-resolution image generation. This paper introduces a series of
architectures adapted from the RWKV model used in the NLP, with requisite
modifications tailored for diffusion model applied to image generation tasks,
referred to as Diffusion-RWKV. Similar to the diffusion with Transformers, our
model is designed to efficiently handle patchnified inputs in a sequence with
extra conditions, while also scaling up effectively, accommodating both
large-scale parameters and extensive datasets. Its distinctive advantage
manifests in its reduced spatial aggregation complexity, rendering it
exceptionally adept at processing high-resolution images, thereby eliminating
the necessity for windowing or group cached operations. Experimental results on
both condition and unconditional image generation tasks demonstrate that
Diffison-RWKV achieves performance on par with or surpasses existing CNN or
Transformer-based diffusion models in FID and IS metrics while significantly
reducing total computation FLOP usage.
MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators
Authors: name: Shenghai Yuan, name: Jinfa Huang, name: Yujun Shi, name: Yongqi Xu, name: Ruijie Zhu, name: Bin Lin, name: Xinhua Cheng, name: Li Yuan, name: Jiebo Luo
Published: 2024-04-07 16:49:07

Recent advances in Text-to-Video generation (T2V) have achieved remarkable
success in synthesizing high-quality general videos from textual descriptions.
A largely overlooked problem in T2V is that existing models have not adequately
encoded physical knowledge of the real world, thus generated videos tend to
have limited motion and poor variations. In this paper, we propose
MagicTime, a metamorphic time-lapse video generation model, which
learns real-world physics knowledge from time-lapse videos and implements
metamorphic generation. First, we design a MagicAdapter scheme to decouple
spatial and temporal training, encode more physical knowledge from metamorphic
videos, and transform pre-trained T2V models to generate metamorphic videos.
Second, we introduce a Dynamic Frames Extraction strategy to adapt to
metamorphic time-lapse videos, which have a wider variation range and cover
dramatic object metamorphic processes, thus embodying more physical knowledge
than general videos. Finally, we introduce a Magic Text-Encoder to improve the
understanding of metamorphic video prompts. Furthermore, we create a time-lapse
video-text dataset called ChronoMagic, specifically curated to unlock
the metamorphic video generation ability. Extensive experiments demonstrate the
superiority and effectiveness of MagicTime for generating high-quality and
dynamic metamorphic videos, suggesting time-lapse video generation is a
promising path toward building metamorphic simulators of the physical world.
UniFL: Improve Stable Diffusion via Unified Feedback Learning
Authors: name: Jiacheng Zhang, name: Jie Wu, name: Yuxi Ren, name: Xin Xia, name: Huafeng Kuang, name: Pan Xie, name: Jiashi Li, name: Xuefeng Xiao, name: Weilin Huang, name: Min Zheng, name: Lean Fu, name: Guanbin Li
Published: 2024-04-08 15:14:20

Diffusion models have revolutionized the field of image generation, leading
to the proliferation of high-quality models and diverse downstream
applications. However, despite these significant advancements, the current
competitive solutions still suffer from several limitations, including inferior
visual quality, a lack of aesthetic appeal, and inefficient inference, without
a comprehensive solution in sight. To address these challenges, we present
UniFL, a unified framework that leverages feedback learning to enhance
diffusion models comprehensively. UniFL stands out as a universal, effective,
and generalizable solution applicable to various diffusion models, such as
SD1.5 and SDXL. Notably, UniFL incorporates three key components: perceptual
feedback learning, which enhances visual quality; decoupled feedback learning,
which improves aesthetic appeal; and adversarial feedback learning, which
optimizes inference speed. In-depth experiments and extensive user studies
validate the superior performance of our proposed method in enhancing both the
quality of generated models and their acceleration. For instance, UniFL
surpasses ImageReward by 17% user preference in terms of generation quality and
outperforms LCM and SDXL Turbo by 57% and 20% in 4-step inference. Moreover, we
have verified the efficacy of our approach in downstream tasks, including Lora,
ControlNet, and AnimateDiff.
Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
Authors: name: Zifu Wan, name: Yuhao Wang, name: Silong Yong, name: Pingping Zhang, name: Simon Stepputtis, name: Katia Sycara, name: Yaqi Xie
Published: 2024-04-05 17:59:44

Multi-modal semantic segmentation significantly enhances AI agents'
perception and scene understanding, especially under adverse conditions like
low-light or overexposed environments. Leveraging additional modalities
(X-modality) like thermal and depth alongside traditional RGB provides
complementary information, enabling more robust and reliable segmentation. In
this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
segmentation, utilizing the Selective Structured State Space Model, Mamba.
Unlike conventional methods that rely on CNNs, with their limited local
receptive fields, or Vision Transformers (ViTs), which offer global receptive
fields at the cost of quadratic complexity, our model achieves global receptive
fields coverage with linear complexity. By employing a Siamese encoder and
innovating a Mamba fusion mechanism, we effectively select essential
information from different modalities. A decoder is then developed to enhance
the channel-wise modeling ability of the model. Our method, Sigma, is
rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
demonstrating its superiority and marking the first successful application of
State Space Models (SSMs) in multi-modal perception tasks. Code is available at
https://github.com/zifuwan/Sigma.
Robust Gaussian Splatting
Authors: name: François Darmon, name: Lorenzo Porzi, name: Samuel Rota-Bulò, name: Peter Kontschieder
Published: 2024-04-05 16:42:16

In this paper, we address common error sources for 3D Gaussian Splatting
(3DGS) including blur, imperfect camera poses, and color inconsistencies, with
the goal of improving its robustness for practical applications like
reconstructions from handheld phone captures. Our main contribution involves
modeling motion blur as a Gaussian distribution over camera poses, allowing us
to address both camera pose refinement and motion blur correction in a unified
way. Additionally, we propose mechanisms for defocus blur compensation and for
addressing color in-consistencies caused by ambient light, shadows, or due to
camera-related factors like varying white balancing settings. Our proposed
solutions integrate in a seamless way with the 3DGS formulation while
maintaining its benefits in terms of training efficiency and rendering speed.
We experimentally validate our contributions on relevant benchmark datasets
including Scannet++ and Deblur-NeRF, obtaining state-of-the-art results and
thus consistent improvements over relevant baselines.
Social Skill Training with Large Language Models
Authors: name: Diyi Yang, name: Caleb Ziems, name: William Held, name: Omar Shaikh, name: Michael S. Bernstein, name: John Mitchell
Published: 2024-04-05 16:29:58

People rely on social skills like conflict resolution to communicate
effectively and to thrive in both work and personal life. However, practice
environments for social skills are typically out of reach for most people. How
can we make social skill training more available, accessible, and inviting?
Drawing upon interdisciplinary research from communication and psychology, this
perspective paper identifies social skill barriers to enter specialized fields.
Then we present a solution that leverages large language models for social
skill training via a generic framework. Our AI Partner, AI Mentor framework
merges experiential learning with realistic practice and tailored feedback.
This work ultimately calls for cross-disciplinary innovation to address the
broader implications for workforce development and social equality.
CantTalkAboutThis: Aligning Language Models to Stay on Topic in
  Dialogues
Authors: name: Makesh Narsimhan Sreedhar, name: Traian Rebedea, name: Shaona Ghosh, name: Christopher Parisien
Published: 2024-04-04 22:31:58

Recent advancements in instruction-tuning datasets have predominantly focused
on specific tasks like mathematical or logical reasoning. There has been a
notable gap in data designed for aligning language models to maintain topic
relevance in conversations - a critical aspect for deploying chatbots to
production. We introduce the CantTalkAboutThis dataset to help language models
remain focused on the subject at hand during task-oriented interactions. It
consists of synthetic dialogues on a wide range of conversation topics from
different domains. These dialogues are interspersed with distractor turns that
intentionally divert the chatbot from the predefined topic. Fine-tuning
language models on this dataset helps make them resilient to deviating from the
role assigned and improves their ability to maintain topical coherence compared
to general-purpose instruction-tuned LLMs like GPT-4-turbo and
Mixtral-Instruct. Additionally, preliminary observations suggest that training
models on this dataset also enhance their performance on fine-grained
instruction following tasks.
AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web
  Navigating Agent
Authors: name: Hanyu Lai, name: Xiao Liu, name: Iat Long Iong, name: Shuntian Yao, name: Yuxuan Chen, name: Pengbo Shen, name: Hao Yu, name: Hanchen Zhang, name: Xiaohan Zhang, name: Yuxiao Dong, name: Jie Tang
Published: 2024-04-04 17:58:40

Large language models (LLMs) have fueled many intelligent agent tasks, such
as web navigation -- but most existing agents perform far from satisfying in
real-world webpages due to three factors: (1) the versatility of actions on
webpages, (2) HTML text exceeding model processing capacity, and (3) the
complexity of decision-making due to the open-domain nature of web. In light of
the challenge, we develop AutoWebGLM, a GPT-4-outperforming automated web
navigation agent built upon ChatGLM3-6B. Inspired by human browsing patterns,
we design an HTML simplification algorithm to represent webpages, preserving
vital information succinctly. We employ a hybrid human-AI method to build web
browsing data for curriculum training. Then, we bootstrap the model by
reinforcement learning and rejection sampling to further facilitate webpage
comprehension, browser operations, and efficient task decomposition by itself.
For testing, we establish a bilingual benchmark -- AutoWebBench -- for
real-world web browsing tasks. We evaluate AutoWebGLM across diverse web
navigation benchmarks, revealing its improvements but also underlying
challenges to tackle real environments. Related code, model, and data will be
released at https://github.com/THUDM/AutoWebGLM.
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
Authors: name: Xinrun Du, name: Zhouliang Yu, name: Songyang Gao, name: Ding Pan, name: Yuyang Cheng, name: Ziyang Ma, name: Ruibin Yuan, name: Xingwei Qu, name: Jiaheng Liu, name: Tianyu Zheng, name: Xinchen Luo, name: Guorui Zhou, name: Binhang Yuan, name: Wenhu Chen, name: Jie Fu, name: Ge Zhang
Published: 2024-04-05 15:20:02

In this study, we introduce CT-LLM, a 2B large language model (LLM) that
illustrates a pivotal shift towards prioritizing the Chinese language in
developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the
conventional methodology by primarily incorporating Chinese textual data,
utilizing an extensive corpus of 1,200 billion tokens, including 800 billion
Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This
strategic composition facilitates the model's exceptional proficiency in
understanding and processing Chinese, a capability further enhanced through
alignment techniques. Demonstrating remarkable performance on the CHC-Bench,
CT-LLM excels in Chinese language tasks, and showcases its adeptness in English
through SFT. This research challenges the prevailing paradigm of training LLMs
predominantly on English corpora and then adapting them to other languages,
broadening the horizons for LLM training methodologies. By open-sourcing the
full process of training a Chinese LLM, including a detailed data processing
procedure with the obtained Massive Appropriate Pretraining Chinese Corpus
(MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark
(CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster
further exploration and innovation in both academia and industry, paving the
way for more inclusive and versatile language models.
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency
  Determines Multimodal Model Performance
Authors: name: Vishaal Udandarao, name: Ameya Prabhu, name: Adhiraj Ghosh, name: Yash Sharma, name: Philip H. S. Torr, name: Adel Bibi, name: Samuel Albanie, name: Matthias Bethge
Published: 2024-04-04 17:58:02

Web-crawled pretraining datasets underlie the impressive "zero-shot"
evaluation performance of multimodal models, such as CLIP for
classification/retrieval and Stable-Diffusion for image generation. However, it
is unclear how meaningful the notion of "zero-shot" generalization is for such
multimodal models, as it is not known to what extent their pretraining datasets
encompass the downstream concepts targeted for during "zero-shot" evaluation.
In this work, we ask: How is the performance of multimodal models on downstream
concepts influenced by the frequency of these concepts in their pretraining
datasets? We comprehensively investigate this question across 34 models and
five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M,
LAION-Aesthetics), generating over 300GB of data artifacts. We consistently
find that, far from exhibiting "zero-shot" generalization, multimodal models
require exponentially more data to achieve linear improvements in downstream
"zero-shot" performance, following a sample inefficient log-linear scaling
trend. This trend persists even when controlling for sample-level similarity
between pretraining and downstream datasets, and testing on purely synthetic
data distributions. Furthermore, upon benchmarking models on long-tailed data
sampled based on our analysis, we demonstrate that multimodal models across the
board perform poorly. We contribute this long-tail test set as the "Let it
Wag!" benchmark to further research in this direction. Taken together, our
study reveals an exponential need for training data which implies that the key
to "zero-shot" generalization capabilities under large-scale training paradigms
remains to be found.
Stream of Search (SoS): Learning to Search in Language
Authors: name: Kanishk Gandhi, name: Denise Lee, name: Gabriel Grand, name: Muxin Liu, name: Winson Cheng, name: Archit Sharma, name: Noah D. Goodman
Published: 2024-04-01 06:50:52

Language models are rarely shown fruitful mistakes while training. They then
struggle to look beyond the next token, suffering from a snowballing of errors
and struggling to predict the consequence of their actions several steps ahead.
In this paper, we show how language models can be taught to search by
representing the process of search in language, as a flattened string -- a
stream of search (SoS). We propose a unified language for search that captures
an array of different symbolic search strategies. We demonstrate our approach
using the simple yet difficult game of Countdown, where the goal is to combine
input numbers with arithmetic operations to reach a target number. We pretrain
a transformer-based language model from scratch on a dataset of streams of
search generated by heuristic solvers. We find that SoS pretraining increases
search accuracy by 25% over models trained to predict only the optimal search
trajectory. We further finetune this model with two policy improvement methods:
Advantage-Induced Policy Alignment (APA) and Self-Taught Reasoner (STaR). The
finetuned SoS models solve 36% of previously unsolved problems, including
problems that cannot be solved by any of the heuristic solvers. Our results
indicate that language models can learn to solve problems via search,
self-improve to flexibly use different search strategies, and potentially
discover new ones.
Direct Nash Optimization: Teaching Language Models to Self-Improve with
  General Preferences
Authors: name: Corby Rosset, name: Ching-An Cheng, name: Arindam Mitra, name: Michael Santacroce, name: Ahmed Awadallah, name: Tengyang Xie
Published: 2024-04-04 17:56:41

This paper studies post-training large language models (LLMs) using
preference feedback from a powerful oracle to help a model iteratively improve
over itself. The typical approach for post-training LLMs involves Reinforcement
Learning from Human Feedback (RLHF), which traditionally separates reward
learning and subsequent policy optimization. However, such a reward
maximization approach is limited by the nature of "point-wise" rewards (such as
Bradley-Terry model), which fails to express complex intransitive or cyclic
preference relations. While advances on RLHF show reward learning and policy
optimization can be merged into a single contrastive objective for stability,
they yet still remain tethered to the reward maximization framework. Recently,
a new wave of research sidesteps the reward maximization presumptions in favor
of directly optimizing over "pair-wise" or general preferences. In this paper,
we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm
that marries the simplicity and stability of contrastive learning with
theoretical generality from optimizing general preferences. Because DNO is a
batched on-policy algorithm using a regression-based objective, its
implementation is straightforward and efficient. Moreover, DNO enjoys monotonic
improvement across iterations that help it improve even over a strong teacher
(such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model
aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of
33% on AlpacaEval 2.0 (even after controlling for response length), an absolute
gain of 26% (7% to 33%) over the initializing model. It outperforms models with
far more parameters, including Mistral Large, Self-Rewarding LM (70B
parameters), and older versions of GPT-4.
RL for Consistency Models: Faster Reward Guided Text-to-Image Generation
Authors: name: Owen Oertell, name: Jonathan D. Chang, name: Yiyi Zhang, name: Kianté Brantley, name: Wen Sun
Published: 2024-03-25 15:40:22

Reinforcement learning (RL) has improved guided image generation with
diffusion models by directly optimizing rewards that capture image quality,
aesthetics, and instruction following capabilities. However, the resulting
generative policies inherit the same iterative sampling process of diffusion
models that causes slow generation. To overcome this limitation, consistency
models proposed learning a new class of generative models that directly map
noise to data, resulting in a model that can generate an image in as few as one
sampling iteration. In this work, to optimize text-to-image generative models
for task specific rewards and enable fast training and inference, we propose a
framework for fine-tuning consistency models via RL. Our framework, called
Reinforcement Learning for Consistency Model (RLCM), frames the iterative
inference process of a consistency model as an RL procedure. RLCM improves upon
RL fine-tuned diffusion models on text-to-image generation capabilities and
trades computation during inference time for sample quality. Experimentally, we
show that RLCM can adapt text-to-image consistency models to objectives that
are challenging to express with prompting, such as image compressibility, and
those derived from human feedback, such as aesthetic quality. Comparing to RL
finetuned diffusion models, RLCM trains significantly faster, improves the
quality of the generation measured under the reward objectives, and speeds up
the inference procedure by generating high quality images with as few as two
inference steps. Our code is available at https://rlcm.owenoertell.com