Orionnnn/hub-stats huggingface dataset:
Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with
  Prototypical Embedding
Authors: name: Jianxiang Lu, name: Cong Xie, name: Hui Guo
Published: 2024-01-28 17:11:42

As large-scale text-to-image generation models have made remarkable progress
in the field of text-to-image generation, many fine-tuning methods have been
proposed. However, these models often struggle with novel objects, especially
with one-shot scenarios. Our proposed method aims to address the challenges of
generalizability and fidelity in an object-driven way, using only a single
input image and the object-specific regions of interest. To improve
generalizability and mitigate overfitting, in our paradigm, a prototypical
embedding is initialized based on the object's appearance and its class, before
fine-tuning the diffusion model. And during fine-tuning, we propose a
class-characterizing regularization to preserve prior knowledge of object
classes. To further improve fidelity, we introduce object-specific loss, which
can also use to implant multiple objects. Overall, our proposed object-driven
method for implanting new objects can integrate seamlessly with existing
concepts as well as with high fidelity and generalization. Our method
outperforms several existing works. The code will be released.
Divide and Conquer: Language Models can Plan and Self-Correct for
  Compositional Text-to-Image Generation
Authors: name: Zhenyu Wang, name: Enze Xie, name: Aoxue Li, name: Zhongdao Wang, name: Xihui Liu, name: Zhenguo Li
Published: 2024-01-28 16:18:39

Despite significant advancements in text-to-image models for generating
high-quality images, these methods still struggle to ensure the controllability
of text prompts over images in the context of complex text prompts, especially
when it comes to retaining object attributes and relationships. In this paper,
we propose CompAgent, a training-free approach for compositional text-to-image
generation, with a large language model (LLM) agent as its core. The
fundamental idea underlying CompAgent is premised on a divide-and-conquer
methodology. Given a complex text prompt containing multiple concepts including
objects, attributes, and relationships, the LLM agent initially decomposes it,
which entails the extraction of individual objects, their associated
attributes, and the prediction of a coherent scene layout. These individual
objects can then be independently conquered. Subsequently, the agent performs
reasoning by analyzing the text, plans and employs the tools to compose these
isolated objects. The verification and human feedback mechanism is finally
incorporated into our agent to further correct the potential attribute errors
and refine the generated images. Guided by the LLM agent, we propose a
tuning-free multi-concept customization model and a layout-to-image generation
model as the tools for concept composition, and a local image editing method as
the tool to interact with the agent for verification. The scene layout controls
the image generation process among these tools to prevent confusion among
multiple objects. Extensive experiments demonstrate the superiority of our
approach for compositional text-to-image generation: CompAgent achieves more
than 10\% improvement on T2I-CompBench, a comprehensive benchmark for
open-world compositional T2I generation. The extension to various related tasks
also illustrates the flexibility of our CompAgent for potential applications.
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling
Authors: name: Pratyush Maini, name: Skyler Seto, name: He Bai, name: David Grangier, name: Yizhe Zhang, name: Navdeep Jaitly
Published: 2024-01-29 18:19:08

Large language models are trained on massive scrapes of the web, which are
often unstructured, noisy, and poorly phrased. Current scaling laws show that
learning from such data requires an abundance of both compute and data, which
grows with the size of the model being trained. This is infeasible both because
of the large compute costs and duration associated with pre-training, and the
impending scarcity of high-quality data on the web. In this work, we propose
Web Rephrase Augmented Pre-training (WRAP) that uses an
off-the-shelf instruction-tuned model prompted to paraphrase documents on the
web in specific styles such as "like Wikipedia" or in "question-answer format"
to jointly pre-train LLMs on real and synthetic rephrases. First, we show that
using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training
by sim3x. At the same pre-training compute budget, it improves perplexity by
more than 10% on average across different subsets of the Pile, and improves
zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we
investigate the impact of the re-phrasing style on the performance of the
model, offering insights into how the composition of the training data can
impact the performance of LLMs in OOD settings. Our gains are attributed to the
fact that re-phrased synthetic data has higher utility than just real data
because it (i) incorporates style diversity that closely reflects downstream
evaluation style, and (ii) has higher 'quality' than web-scraped data.
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual
  Perception
Authors: name: Junyang Wang, name: Haiyang Xu, name: Jiabo Ye, name: Ming Yan, name: Weizhou Shen, name: Ji Zhang, name: Fei Huang, name: Jitao Sang
Published: 2024-01-29 13:46:37

Mobile device agent based on Multimodal Large Language Models (MLLM) is
becoming a popular application. In this paper, we introduce Mobile-Agent, an
autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual
perception tools to accurately identify and locate both the visual and textual
elements within the app's front-end interface. Based on the perceived vision
context, it then autonomously plans and decomposes the complex operation task,
and navigates the mobile Apps through operations step by step. Different from
previous solutions that rely on XML files of Apps or mobile system metadata,
Mobile-Agent allows for greater adaptability across diverse mobile operating
environments in a vision-centric way, thereby eliminating the necessity for
system-specific customizations. To assess the performance of Mobile-Agent, we
introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent.
The experimental results indicate that Mobile-Agent achieved remarkable
accuracy and completion rates. Even with challenging instructions, such as
multi-app operations, Mobile-Agent can still complete the requirements. Code
and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
Authors: name: Bin Lin, name: Zhenyu Tang, name: Yang Ye, name: Jiaxi Cui, name: Bin Zhu, name: Peng Jin, name: Junwu Zhang, name: Munan Ning, name: Li Yuan
Published: 2024-01-29 08:13:40

For Large Vision-Language Models (LVLMs), scaling the model can effectively
improve performance. However, expanding model parameters significantly
increases the training and inferring costs, as all model parameters are
activated for each token in the calculation. In this work, we propose a novel
training strategy MoE-tuning for LVLMs, which can constructing a sparse model
with an outrageous number of parameter but a constant computational cost, and
effectively addresses the performance degradation typically associated with
multi-modal learning and model sparsity. Furthermore, we present the MoE-LLaVA
framework, a MoE-based sparse LVLM architecture. This framework uniquely
activates only the top-k experts through routers during deployment, keeping the
remaining experts inactive. Our extensive experiments highlight the excellent
capabilities of MoE-LLaVA in visual understanding and its potential to reduce
hallucinations in model outputs. Remarkably, with just 3 billion sparsely
activated parameters, MoE-LLaVA demonstrates performance comparable to the
LLaVA-1.5-7B on various visual understanding datasets and even surpasses the
LLaVA-1.5-13B in object hallucination benchmarks. Through MoE-LLaVA, we aim to
establish a baseline for sparse LVLMs and provide valuable insights for future
research in developing more efficient and effective multi-modal learning
systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.
Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD
  Generalization
Authors: name: Yuhang Zang, name: Hanlin Goh, name: Josh Susskind, name: Chen Huang
Published: 2024-01-29 06:57:48

Existing vision-language models exhibit strong generalization on a variety of
visual domains and tasks. However, such models mainly perform zero-shot
recognition in a closed-set manner, and thus struggle to handle open-domain
visual concepts by design. There are recent finetuning methods, such as prompt
learning, that not only study the discrimination between in-distribution (ID)
and out-of-distribution (OOD) samples, but also show some improvements in both
ID and OOD accuracies. In this paper, we first demonstrate that vision-language
models, after long enough finetuning but without proper regularization, tend to
overfit the known classes in the given dataset, with degraded performance on
unknown classes. Then we propose a novel approach OGEN to address this pitfall,
with the main focus on improving the OOD GENeralization of finetuned models.
Specifically, a class-conditional feature generator is introduced to synthesize
OOD features using just the class name of any unknown class. Such synthesized
features will provide useful knowledge about unknowns and help regularize the
decision boundary between ID and OOD data when optimized jointly. Equally
important is our adaptive self-distillation mechanism to regularize our feature
generation model during joint optimization, i.e., adaptively transferring
knowledge between model states to further prevent overfitting. Experiments
validate that our method yields convincing gains in OOD generalization
performance in different settings.
InternLM-XComposer2: Mastering Free-form Text-Image Composition and
  Comprehension in Vision-Language Large Model
Authors: name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Bin Wang, name: Linke Ouyang, name: Xilin Wei, name: Songyang Zhang, name: Haodong Duan, name: Maosong Cao, name: Wenwei Zhang, name: Yining Li, name: Hang Yan, name: Yang Gao, name: Xinyue Zhang, name: Wei Li, name: Jingwen Li, name: Kai Chen, name: Conghui He, name: Xingcheng Zhang, name: Yu Qiao, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-01-29 18:59:02

We introduce InternLM-XComposer2, a cutting-edge vision-language model
excelling in free-form text-image composition and comprehension. This model
goes beyond conventional vision-language understanding, adeptly crafting
interleaved text-image content from diverse inputs like outlines, detailed
textual specifications, and reference images, enabling highly customizable
content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach
that applies additional LoRA parameters exclusively to image tokens to preserve
the integrity of pre-trained language knowledge, striking a balance between
precise vision understanding and text composition with literary talent.
Experimental results demonstrate the superiority of InternLM-XComposer2 based
on InternLM2-7B in producing high-quality long-text multi-modal content and its
exceptional vision-language understanding performance across various
benchmarks, where it not only significantly outperforms existing multimodal
models but also matches or even surpasses GPT-4V and Gemini Pro in certain
assessments. This highlights its remarkable proficiency in the realm of
multimodal understanding. The InternLM-XComposer2 model series with 7B
parameters are publicly available at
https://github.com/InternLM/InternLM-XComposer.
Motion-I2V: Consistent and Controllable Image-to-Video Generation with
  Explicit Motion Modeling
Authors: name: Xiaoyu Shi, name: Zhaoyang Huang, name: Fu-Yun Wang, name: Weikang Bian, name: Dasong Li, name: Yi Zhang, name: Manyuan Zhang, name: Ka Chun Cheung, name: Simon See, name: Hongwei Qin, name: Jifeng Da, name: Hongsheng Li
Published: 2024-01-29 09:06:43

We introduce Motion-I2V, a novel framework for consistent and controllable
image-to-video generation (I2V). In contrast to previous methods that directly
learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into
two stages with explicit motion modeling. For the first stage, we propose a
diffusion-based motion field predictor, which focuses on deducing the
trajectories of the reference image's pixels. For the second stage, we propose
motion-augmented temporal attention to enhance the limited 1-D temporal
attention in video latent diffusion models. This module can effectively
propagate reference image's feature to synthesized frames with the guidance of
predicted trajectories from the first stage. Compared with existing methods,
Motion-I2V can generate more consistent videos even at the presence of large
motion and viewpoint variation. By training a sparse trajectory ControlNet for
the first stage, Motion-I2V can support users to precisely control motion
trajectories and motion regions with sparse trajectory and region annotations.
This offers more controllability of the I2V process than solely relying on
textual instructions. Additionally, Motion-I2V's second stage naturally
supports zero-shot video-to-video translation. Both qualitative and
quantitative comparisons demonstrate the advantages of Motion-I2V over prior
approaches in consistent and controllable image-to-video generation.
Generative Expressive Robot Behaviors using Large Language Models
Authors: name: Karthik Mahadevan, name: Jonathan Chien, name: Noah Brown, name: Zhuo Xu, name: Carolina Parada, name: Fei Xia, name: Andy Zeng, name: Leila Takayama, name: Dorsa Sadigh
Published: 2024-01-26 06:34:32

People employ expressive behaviors to effectively communicate and coordinate
their actions with others, such as nodding to acknowledge a person glancing at
them or saying "excuse me" to pass people in a busy corridor. We would like
robots to also demonstrate expressive behaviors in human-robot interaction.
Prior work proposes rule-based methods that struggle to scale to new
communication modalities or social situations, while data-driven methods
require specialized datasets for each social situation the robot is used in. We
propose to leverage the rich social context available from large language
models (LLMs) and their ability to generate motion based on instructions or
user preferences, to generate expressive robot motion that is adaptable and
composable, building upon each other. Our approach utilizes few-shot
chain-of-thought prompting to translate human language instructions into
parametrized control code using the robot's available and learned skills.
Through user studies and simulation experiments, we demonstrate that our
approach produces behaviors that users found to be competent and easy to
understand. Supplementary material can be found at
https://generative-expressive-motion.github.io/.
Learning Universal Predictors
Authors: name: Jordi Grau-Moya, name: Tim Genewein, name: Marcus Hutter, name: Laurent Orseau, name: Grégoire Delétang, name: Elliot Catt, name: Anian Ruoss, name: Li Kevin Wenliang, name: Christopher Mattern, name: Matthew Aitchison, name: Joel Veness
Published: 2024-01-26 15:37:16

Meta-learning has emerged as a powerful approach to train neural networks to
learn new tasks quickly from limited data. Broad exposure to different tasks
leads to versatile representations enabling general problem solving. But, what
are the limits of meta-learning? In this work, we explore the potential of
amortizing the most powerful universal predictor, namely Solomonoff Induction
(SI), into neural networks via leveraging meta-learning to its limits. We use
Universal Turing Machines (UTMs) to generate training data used to expose
networks to a broad range of patterns. We provide theoretical analysis of the
UTM data generation processes and meta-training protocols. We conduct
comprehensive experiments with neural architectures (e.g. LSTMs, Transformers)
and algorithmic data generators of varying complexity and universality. Our
results suggest that UTM data is a valuable resource for meta-learning, and
that it can be used to train neural networks capable of learning universal
prediction strategies.
TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And
  Image-Prompts
Authors: name: Jingyu Zhuang, name: Di Kang, name: Yan-Pei Cao, name: Guanbin Li, name: Liang Lin, name: Ying Shan
Published: 2024-01-26 12:57:05

Text-driven 3D scene editing has gained significant attention owing to its
convenience and user-friendliness. However, existing methods still lack
accurate control of the specified appearance and location of the editing result
due to the inherent limitations of the text description. To this end, we
propose a 3D scene editing framework, TIPEditor, that accepts both text and
image prompts and a 3D bounding box to specify the editing region. With the
image prompt, users can conveniently specify the detailed appearance/style of
the target content in complement to the text description, enabling accurate
control of the appearance. Specifically, TIP-Editor employs a stepwise 2D
personalization strategy to better learn the representation of the existing
scene and the reference image, in which a localization loss is proposed to
encourage correct object placement as specified by the bounding box.
Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as
the 3D representation to facilitate local editing while keeping the background
unchanged. Extensive experiments have demonstrated that TIP-Editor conducts
accurate editing following the text and image prompts in the specified bounding
box region, consistently outperforming the baselines in editing quality, and
the alignment to the prompts, qualitatively and quantitatively.
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
Authors: name: Yuhui Li, name: Fangyun Wei, name: Chao Zhang, name: Hongyang Zhang
Published: 2024-01-26 18:59:01

Auto-regressive decoding makes the inference of Large Language Models (LLMs)
time-consuming. We propose a simple framework, EAGLE (Extrapolation Algorithm
for Greater Language-model Efficiency), for lossless acceleration. Unlike
traditional speculative sampling methods, EAGLE operates the drafting process
auto-regressively at the more regular (second-top-layer) feature level and
addresses the sampling uncertainty issues in the next-feature prediction
problems by integrating tokens from one time step ahead. The acceleration
provided by EAGLE is lossless: it involves no fine-tuning of the target LLM,
and the generated text maintains the same distribution as that of vanilla
auto-regressive decoding. As of the submission of this paper, EAGLE is the
fastest known framework within the speculative sampling family. On MT-bench,
EAGLE is 3x faster than vanilla decoding, 2x faster than Lookahead, and 1.6x
faster than Medusa. Using gpt-fast, EAGLE attains on average 160 tokens/s with
LLaMA2-Chat 13B on a single RTX 3090 GPU, compared to 24 tokens/s of
Huggingface's implementations.
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Authors: name: Saleh Ashkboos, name: Maximilian L. Croci, name: Marcelo Gennari do Nascimento, name: Torsten Hoefler, name: James Hensman
Published: 2024-01-26 17:35:45

Large language models have become the cornerstone of natural language
processing, but their use comes with substantial costs in terms of compute and
memory resources. Sparsification provides a solution to alleviate these
resource constraints, and recent works have shown that trained models can be
sparsified post-hoc. Existing sparsification techniques face challenges as they
need additional data structures and offer constrained speedup with current
hardware. In this paper we present SliceGPT, a new post-training sparsification
scheme which replaces each weight matrix with a smaller (dense) matrix,
reducing the embedding dimension of the network. Through extensive
experimentation, we show that SliceGPT can remove up to 25% of the model
parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models
while maintaining 99%, 99% and 90% zero-shot task performance of the dense
model respectively. Our sliced models run on fewer GPUs and run faster without
any additional code optimization: on 24GB consumer GPUs we reduce the total
compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB
A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance
in transformer networks, which enables SliceGPT and we hope it will inspire and
enable future avenues to reduce memory and computation demands for pre-trained
models. Code is available at:
https://github.com/microsoft/TransformerCompression
Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with
  Large Vision-Language Model Support
Authors: name: Xiaojun Wu, name: Dixiang Zhang, name: Ruyi Gan, name: Junyu Lu, name: Ziwei Wu, name: Renliang Sun, name: Jiaxing Zhang, name: Pingjian Zhang, name: Yan Song
Published: 2024-01-26 07:17:50

Recent advancements in text-to-image models have significantly enhanced image
generation capabilities, yet a notable gap of open-source models persists in
bilingual or Chinese language support. To address this need, we present
Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
which is developed by extending the capabilities of CLIP and
Stable-Diffusion-XL through a process of bilingual continuous pre-training.
This approach includes the efficient expansion of vocabulary by integrating the
most frequently used Chinese characters into CLIP's tokenizer and embedding
layers, coupled with an absolute position encoding expansion. Additionally, we
enrich text prompts by large vision-language model, leading to better images
captions and possess higher visual quality. These enhancements are subsequently
applied to downstream text-to-image models. Our empirical results indicate that
the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
previous models. This research leads to the development and open-sourcing of
the Taiyi-Diffusion-XL model, representing a notable advancement in the field
of image generation, particularly for Chinese language applications. This
contribution is a step forward in addressing the need for more diverse language
support in multimodal research. The model and demonstration are made publicly
available at
https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/{this
https URL}, fostering further research and collaboration in this domain.