Orionnnn/hub-stats huggingface dataset:
Modeling Collaborator: Enabling Subjective Vision Classification With
  Minimal Human Effort via LLM Tool-Use
Authors: name: Imad Eddine Toubal, name: Aditya Avinash, name: Neil Gordon Alldrin, name: Jan Dlabal, name: Wenlei Zhou, name: Enming Luo, name: Otilia Stretcu, name: Hao Xiong, name: Chun-Ta Lu, name: Howard Zhou, name: Ranjay Krishna, name: Ariel Fuxman, name: Tom Duerig
Published: 2024-03-05 03:34:11

From content moderation to wildlife conservation, the number of applications
that require models to recognize nuanced or subjective visual concepts is
growing. Traditionally, developing classifiers for such concepts requires
substantial manual effort measured in hours, days, or even months to identify
and annotate data needed for training. Even with recently proposed Agile
Modeling techniques, which enable rapid bootstrapping of image classifiers,
users are still required to spend 30 minutes or more of monotonous, repetitive
data labeling just to train a single classifier. Drawing on Fiske's Cognitive
Miser theory, we propose a new framework that alleviates manual effort by
replacing human labeling with natural language interactions, reducing the total
effort required to define a concept by an order of magnitude: from labeling
2,000 images to only 100 plus some natural language interactions. Our framework
leverages recent advances in foundation models, both large language models and
vision-language models, to carve out the concept space through conversation and
by automatically labeling training data points. Most importantly, our framework
eliminates the need for crowd-sourced annotations. Moreover, our framework
ultimately produces lightweight classification models that are deployable in
cost-sensitive scenarios. Across 15 subjective concepts and across 2 public
image classification datasets, our trained models outperform traditional Agile
Modeling as well as state-of-the-art zero-shot classification models like
ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.
RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
Authors: name: Priya Sundaresan, name: Quan Vuong, name: Jiayuan Gu, name: Peng Xu, name: Ted Xiao, name: Sean Kirmani, name: Tianhe Yu, name: Michael Stark, name: Ajinkya Jain, name: Karol Hausman, name: Dorsa Sadigh, name: Jeannette Bohg, name: Stefan Schaal
Published: 2024-03-05 07:00:46

Natural language and images are commonly used as goal representations in
goal-conditioned imitation learning (IL). However, natural language can be
ambiguous and images can be over-specified. In this work, we propose hand-drawn
sketches as a modality for goal specification in visual imitation learning.
Sketches are easy for users to provide on the fly like language, but similar to
images they can also help a downstream policy to be spatially-aware and even go
beyond images to disambiguate task-relevant from task-irrelevant objects. We
present RT-Sketch, a goal-conditioned policy for manipulation that takes a
hand-drawn sketch of the desired scene as input, and outputs actions. We train
RT-Sketch on a dataset of paired trajectories and corresponding synthetically
generated goal sketches. We evaluate this approach on six manipulation skills
involving tabletop object rearrangements on an articulated countertop.
Experimentally we find that RT-Sketch is able to perform on a similar level to
image or language-conditioned agents in straightforward settings, while
achieving greater robustness when language goals are ambiguous or visual
distractors are present. Additionally, we show that RT-Sketch has the capacity
to interpret and act upon sketches with varied levels of specificity, ranging
from minimal line drawings to detailed, colored drawings. For supplementary
material and videos, please refer to our website: http://rt-sketch.github.io.
MagicClay: Sculpting Meshes With Generative Neural Fields
Authors: name: Amir Barda, name: Vladimir G. Kim, name: Noam Aigerman, name: Amit H. Bermano, name: Thibault Groueix
Published: 2024-03-04 20:20:14

The recent developments in neural fields have brought phenomenal capabilities
to the field of shape generation, but they lack crucial properties, such as
incremental control - a fundamental requirement for artistic work. Triangular
meshes, on the other hand, are the representation of choice for most geometry
related tasks, offering efficiency and intuitive control, but do not lend
themselves to neural optimization. To support downstream tasks, previous art
typically proposes a two-step approach, where first a shape is generated using
neural fields, and then a mesh is extracted for further processing. Instead, in
this paper we introduce a hybrid approach that maintains both a mesh and a
Signed Distance Field (SDF) representations consistently. Using this
representation, we introduce MagicClay - an artist friendly tool for sculpting
regions of a mesh according to textual prompts while keeping other regions
untouched. Our framework carefully and efficiently balances consistency between
the representations and regularizations in every step of the shape
optimization; Relying on the mesh representation, we show how to render the SDF
at higher resolutions and faster. In addition, we employ recent work in
differentiable mesh reconstruction to adaptively allocate triangles in the mesh
where required, as indicated by the SDF. Using an implemented prototype, we
demonstrate superior generated geometry compared to the state-of-the-art, and
novel consistent control, allowing sequential prompt-based edits to the same
mesh for the first time.
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and
  Diffusion Models
Authors: name: Zeqian Ju, name: Yuancheng Wang, name: Kai Shen, name: Xu Tan, name: Detai Xin, name: Dongchao Yang, name: Yanqing Liu, name: Yichong Leng, name: Kaitao Song, name: Siliang Tang, name: Zhizheng Wu, name: Tao Qin, name: Xiang-Yang Li, name: Wei Ye, name: Shikun Zhang, name: Jiang Bian, name: Lei He, name: Jinyu Li, name: Sheng Zhao
Published: 2024-03-05 16:35:25

While recent large-scale text-to-speech (TTS) models have achieved
significant progress, they still fall short in speech quality, similarity, and
prosody. Considering speech intricately encompasses various attributes (e.g.,
content, prosody, timbre, and acoustic details) that pose significant
challenges for generation, a natural idea is to factorize speech into
individual subspaces representing different attributes and generate them
individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with
novel factorized diffusion models to generate natural speech in a zero-shot
way. Specifically, 1) we design a neural codec with factorized vector
quantization (FVQ) to disentangle speech waveform into subspaces of content,
prosody, timbre, and acoustic details; 2) we propose a factorized diffusion
model to generate attributes in each subspace following its corresponding
prompt. With this factorization design, NaturalSpeech 3 can effectively and
efficiently model the intricate speech with disentangled subspaces in a
divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the
state-of-the-art TTS systems on quality, similarity, prosody, and
intelligibility. Furthermore, we achieve better performance by scaling to 1B
parameters and 200K hours of training data.
MAGID: An Automated Pipeline for Generating Synthetic Multi-modal
  Datasets
Authors: name: Hossein Aboutalebi, name: Hwanjun Song, name: Yusheng Xie, name: Arshit Gupta, name: Justin Sun, name: Hang Su, name: Igor Shalyminov, name: Nikolaos Pappas, name: Siffi Singh, name: Saab Mansour
Published: 2024-03-05 18:31:28

Development of multimodal interactive systems is hindered by the lack of
rich, multimodal (text, images) conversational data, which is needed in large
quantities for LLMs. Previous approaches augment textual dialogues with
retrieved images, posing privacy, diversity, and quality constraints. In this
work, we introduce Multimodal Augmented Generative
Images Dialogues (MAGID), a framework to augment text-only
dialogues with diverse and high-quality images. Subsequently, a diffusion model
is applied to craft corresponding images, ensuring alignment with the
identified text. Finally, MAGID incorporates an innovative feedback loop
between an image description generation module (textual LLM) and image quality
modules (addressing aesthetics, image-text matching, and safety), that work in
tandem to generate high-quality and multi-modal dialogues. We compare MAGID to
other SOTA baselines on three dialogue datasets, using automated and human
evaluation. Our results show that MAGID is comparable to or better than
baselines, with significant improvements in human evaluation, especially
against retrieval baselines where the image database is small.
Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large
  Language Models
Authors: name: Gen Luo, name: Yiyi Zhou, name: Yuxin Zhang, name: Xiawu Zheng, name: Xiaoshuai Sun, name: Rongrong Ji
Published: 2024-03-05 14:31:24

Despite remarkable progress, existing multimodal large language models
(MLLMs) are still inferior in granular visual recognition. Contrary to previous
works, we study this problem from the perspective of image resolution, and
reveal that a combination of low- and high-resolution visual features can
effectively mitigate this shortcoming. Based on this observation, we propose a
novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation
(MRA). In particular, MRA adopts two visual pathways for images with different
resolutions, where high-resolution visual information is embedded into the
low-resolution pathway via the novel mixture-of-resolution adapters
(MR-Adapters). This design also greatly reduces the input sequence length of
MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the
new model LLaVA-HR. We conduct extensive experiments on 11 vision-language (VL)
tasks, which show that LLaVA-HR outperforms existing MLLMs on 8 VL tasks, e.g.,
+9.4% on TextVQA. More importantly, both training and inference of LLaVA-HR
remain efficient with MRA, e.g., 20 training hours and 3times inference
speed than LLaVA-1.5. Source codes are released at:
https://github.com/luogen1996/LLaVA-HR.
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Authors: name: Zhengyang Tang, name: Xingxing Zhang, name: Benyou Wan, name: Furu Wei
Published: 2024-03-05 11:42:59

Large language models (LLMs) have demonstrated remarkable capabilities in
problem-solving. However, their proficiency in solving mathematical problems
remains inadequate. We propose MathScale, a simple and scalable method to
create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt
GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning,
it first extracts topics and knowledge points from seed math questions and then
build a concept graph, which is subsequently used to generate new math
questions. MathScale exhibits effective scalability along the size axis of the
math dataset that we generate. As a result, we create a mathematical reasoning
dataset (MathScaleQA) containing two million math question-answer pairs. To
evaluate mathematical reasoning abilities of LLMs comprehensively, we construct
{\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten
datasets (including GSM8K and MATH) covering K-12, college, and competition
level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g.,
LLaMA-2 and Mistral), resulting in significantly improved capabilities in
mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves
state-of-the-art performance across all datasets, surpassing its best peers of
equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average
accuracy, respectively.
EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
Authors: name: Hanlin Tang, name: Yifu Sun, name: Decheng Wu, name: Kai Liu, name: Jianchen Zhu, name: Zhanhui Kang
Published: 2024-03-05 08:45:30

Large language models (LLMs) have proven to be very superior to conventional
methods in various tasks. However, their expensive computations and high memory
requirements are prohibitive for deployment. Model quantization is an effective
method for reducing this overhead. The problem is that in most previous works,
the quantized model was calibrated using few samples from the training data,
which might affect the generalization of the quantized LLMs to unknown cases
and tasks. Hence in this work, we explore an important question: Can we design
a data-independent quantization method for LLMs to guarantee its generalization
performance? In this work, we propose EasyQuant, a training-free and
data-independent weight-only quantization algorithm for LLMs. Our observation
indicates that two factors: outliers in the weight and quantization ranges, are
essential for reducing the quantization error. Therefore, in EasyQuant, we
leave the outliers (less than 1%) unchanged and optimize the quantization range
to reduce the reconstruction error. With these methods, we surprisingly find
that EasyQuant achieves comparable performance to the original model. Since
EasyQuant does not depend on any training data, the generalization performance
of quantized LLMs is safely guaranteed. Moreover, EasyQuant can be implemented
in parallel so that the quantized model could be attained in a few minutes even
for LLMs over 100B. To our best knowledge, we are the first work that achieves
almost lossless quantization performance for LLMs under a data-independent
setting and our algorithm runs over 10 times faster than the data-dependent
methods.
Finetuned Multimodal Language Models Are High-Quality Image-Text Data
  Filters
Authors: name: Weizhi Wang, name: Khalil Mrini, name: Linjie Yang, name: Sateesh Kumar, name: Yu Tian, name: Xifeng Yan, name: Heng Wang
Published: 2024-03-05 06:05:15

We propose a novel framework for filtering image-text data by leveraging
fine-tuned Multimodal Language Models (MLMs). Our approach outperforms
predominant filtering methods (e.g., CLIPScore) via integrating the recent
advances in MLMs. We design four distinct yet complementary metrics to
holistically measure the quality of image-text data. A new pipeline is
established to construct high-quality instruction data for fine-tuning MLMs as
data filters. Comparing with CLIPScore, our MLM filters produce more precise
and comprehensive scores that directly improve the quality of filtered data and
boost the performance of pre-trained models. We achieve significant
improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2)
and various downstream tasks. Our MLM filter can generalize to different models
and tasks, and be used as a drop-in replacement for CLIPScore. An additional
ablation study is provided to verify our design choices for the MLM filter.
Wukong: Towards a Scaling Law for Large-Scale Recommendation
Authors: name: Buyun Zhang, name: Liang Luo, name: Yuxin Chen, name: Jade Nie, name: Xi Liu, name: Daifeng Guo, name: Yanli Zhao, name: Shen Li, name: Yuchen Hao, name: Yantao Yao, name: Guna Lakshminarayanan, name: Ellie Dingqiao Wen, name: Jongsoo Park, name: Maxim Naumov, name: Wenlin Chen
Published: 2024-03-04 23:40:20

Scaling laws play an instrumental role in the sustainable improvement in
model quality. Unfortunately, recommendation models to date do not exhibit such
laws similar to those observed in the domain of large language models, due to
the inefficiencies of their upscaling mechanisms. This limitation poses
significant challenges in adapting these models to increasingly more complex
real-world datasets. In this paper, we propose an effective network
architecture based purely on stacked factorization machines, and a synergistic
upscaling strategy, collectively dubbed Wukong, to establish a scaling law in
the domain of recommendation. Wukong's unique design makes it possible to
capture diverse, any-order of interactions simply through taller and wider
layers. We conducted extensive evaluations on six public datasets, and our
results demonstrate that Wukong consistently outperforms state-of-the-art
models quality-wise. Further, we assessed Wukong's scalability on an internal,
large-scale dataset. The results show that Wukong retains its superiority in
quality over state-of-the-art models, while holding the scaling law across two
orders of magnitude in model complexity, extending beyond 100 Gflop or
equivalently up to GPT-3/LLaMa-2 scale of total training compute, where prior
arts fall short.
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Authors: name: Patrick Esser, name: Sumith Kulal, name: Andreas Blattmann, name: Rahim Entezari, name: Jonas Müller, name: Harry Saini, name: Yam Levi, name: Dominik Lorenz, name: Axel Sauer, name: Frederic Boesel, name: Dustin Podell, name: Tim Dockhorn, name: Zion English, name: Kyle Lacey, name: Alex Goodwin, name: Yannik Marek, name: Robin Rombach
Published: 2024-03-05 18:45:39

Diffusion models create data from noise by inverting the forward paths of
data towards noise and have emerged as a powerful generative modeling technique
for high-dimensional, perceptual data such as images and videos. Rectified flow
is a recent generative model formulation that connects data and noise in a
straight line. Despite its better theoretical properties and conceptual
simplicity, it is not yet decisively established as standard practice. In this
work, we improve existing noise sampling techniques for training rectified flow
models by biasing them towards perceptually relevant scales. Through a
large-scale study, we demonstrate the superior performance of this approach
compared to established diffusion formulations for high-resolution
text-to-image synthesis. Additionally, we present a novel transformer-based
architecture for text-to-image generation that uses separate weights for the
two modalities and enables a bidirectional flow of information between image
and text tokens, improving text comprehension, typography, and human preference
ratings. We demonstrate that this architecture follows predictable scaling
trends and correlates lower validation loss to improved text-to-image synthesis
as measured by various metrics and human evaluations. Our largest models
outperform state-of-the-art models, and we will make our experimental data,
code, and model weights publicly available.
Design2Code: How Far Are We From Automating Front-End Engineering?
Authors: name: Chenglei Si, name: Yanzhe Zhang, name: Zhengyuan Yang, name: Ruibo Liu, name: Diyi Yang
Published: 2024-03-05 17:56:27

Generative AI has made rapid advancements in recent years, achieving
unprecedented capabilities in multimodal understanding and code generation.
This can enable a new paradigm of front-end development, in which multimodal
LLMs might directly convert visual designs into code implementations. In this
work, we formalize this as a Design2Code task and conduct comprehensive
benchmarking. Specifically, we manually curate a benchmark of 484 diverse
real-world webpages as test cases and develop a set of automatic evaluation
metrics to assess how well current multimodal LLMs can generate the code
implementations that directly render into the given reference webpages, given
the screenshots as input. We also complement automatic metrics with
comprehensive human evaluations. We develop a suite of multimodal prompting
methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We
further finetune an open-source Design2Code-18B model that successfully matches
the performance of Gemini Pro Vision. Both human evaluation and automatic
metrics show that GPT-4V performs the best on this task compared to other
models. Moreover, annotators think GPT-4V generated webpages can replace the
original reference webpages in 49% of cases in terms of visual appearance and
content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages
are considered better than the original reference webpages. Our fine-grained
break-down metrics indicate that open-source models mostly lag in recalling
visual elements from the input webpages and in generating correct layout
designs, while aspects like text content and coloring can be drastically
improved with proper finetuning.
Tuning-Free Noise Rectification for High Fidelity Image-to-Video
  Generation
Authors: name: Weijie Li, name: Litong Gong, name: Yiran Zhu, name: Fanda Fan, name: Biao Wang, name: Tiezheng Ge, name: Bo Zheng
Published: 2024-03-05 09:57:47

Image-to-video (I2V) generation tasks always suffer from keeping high
fidelity in the open domains. Traditional image animation techniques primarily
focus on specific domains such as faces or human poses, making them difficult
to generalize to open domains. Several recent I2V frameworks based on diffusion
models can generate dynamic content for open domain images but fail to maintain
fidelity. We found that two main factors of low fidelity are the loss of image
details and the noise prediction biases during the denoising process. To this
end, we propose an effective method that can be applied to mainstream video
diffusion models. This method achieves high fidelity based on supplementing
more precise image information and noise rectification. Specifically, given a
specified image, our method first adds noise to the input image latent to keep
more details, then denoises the noisy latent with proper rectification to
alleviate the noise prediction biases. Our method is tuning-free and
plug-and-play. The experimental results demonstrate the effectiveness of our
approach in improving the fidelity of generated videos. For more image-to-video
generated results, please refer to the project website:
https://noise-rectification.github.io.
DenseMamba: State Space Models with Dense Hidden Connection for
  Efficient Large Language Models
Authors: name: Wei He, name: Kai Han, name: Yehui Tang, name: Chengcheng Wang, name: Yujie Yang, name: Tianyu Guo, name: Yunhe Wang
Published: 2024-02-26 09:21:59

Large language models (LLMs) face a daunting challenge due to the excessive
computational and memory requirements of the commonly used Transformer
architecture. While state space model (SSM) is a new type of foundational
network architecture offering lower computational complexity, their performance
has yet to fully rival that of Transformers. This paper introduces DenseSSM, a
novel approach to enhance the flow of hidden information between layers in
SSMs. By selectively integrating shallowlayer hidden states into deeper layers,
DenseSSM retains fine-grained information crucial for the final output. Dense
connections enhanced DenseSSM still maintains the training parallelizability
and inference efficiency. The proposed method can be widely applicable to
various SSM types like RetNet and Mamba. With similar model size, DenseSSM
achieves significant improvements, exemplified by DenseRetNet outperforming the
original RetNet with up to 5% accuracy improvement on public benchmarks.
Twisting Lids Off with Two Hands
Authors: name: Toru Lin, name: Zhao-Heng Yin, name: Haozhi Qi, name: Pieter Abbeel, name: Jitendra Malik
Published: 2024-03-04 18:59:30

Manipulating objects with two multi-fingered hands has been a long-standing
challenge in robotics, attributed to the contact-rich nature of many
manipulation tasks and the complexity inherent in coordinating a
high-dimensional bimanual system. In this work, we consider the problem of
twisting lids of various bottle-like objects with two hands, and demonstrate
that policies trained in simulation using deep reinforcement learning can be
effectively transferred to the real world. With novel engineering insights into
physical modeling, real-time perception, and reward design, the policy
demonstrates generalization capabilities across a diverse set of unseen
objects, showcasing dynamic and dexterous behaviors. Our findings serve as
compelling evidence that deep reinforcement learning combined with sim-to-real
transfer remains a promising approach for addressing manipulation problems of
unprecedented complexity.