Orionnnn/hub-stats huggingface dataset:
Gen-L-Video: Multi-Text to Long Video Generation via Temporal
  Co-Denoising
Authors: name: Fu-Yun Wang, name: Wenshuo Chen, name: Guanglu Song, name: Han-Jia Ye, name: Yu Liu, name: Hongsheng Li
Published: 2023-05-29 17:38:18

Leveraging large-scale image-text datasets and advancements in diffusion
models, text-driven generative models have made remarkable strides in the field
of image generation and editing. This study explores the potential of extending
the text-driven ability to the generation and editing of multi-text conditioned
long videos. Current methodologies for video generation and editing, while
innovative, are often confined to extremely short videos (typically less than
24 frames) and are limited to a single text condition. These constraints
significantly limit their applications given that real-world videos usually
consist of multiple segments, each bearing different semantic information. To
address this challenge, we introduce a novel paradigm dubbed as Gen-L-Video,
capable of extending off-the-shelf short video diffusion models for generating
and editing videos comprising hundreds of frames with diverse semantic segments
without introducing additional training, all while preserving content
consistency. We have implemented three mainstream text-driven video generation
and editing methodologies and extended them to accommodate longer videos imbued
with a variety of semantic segments with our proposed paradigm. Our
experimental outcomes reveal that our approach significantly broadens the
generative and editing capabilities of video diffusion models, offering new
possibilities for future research and applications. The code is available at
https://github.com/G-U-N/Gen-L-Video.
TaleCrafter: Interactive Story Visualization with Multiple Characters
Authors: name: Yuan Gong, name: Youxin Pang, name: Xiaodong Cun, name: Menghan Xia, name: Haoxin Chen, name: Longyue Wang, name: Yong Zhang, name: Xintao Wang, name: Ying Shan, name: Yujiu Yang
Published: 2023-05-29 17:11:39

Accurate Story visualization requires several necessary elements, such as
identity consistency across frames, the alignment between plain text and visual
content, and a reasonable layout of objects in images. Most previous works
endeavor to meet these requirements by fitting a text-to-image (T2I) model on a
set of videos in the same style and with the same characters, e.g., the
FlintstonesSV dataset. However, the learned T2I models typically struggle to
adapt to new characters, scenes, and styles, and often lack the flexibility to
revise the layout of the synthesized images. This paper proposes a system for
generic interactive story visualization, capable of handling multiple novel
characters and supporting the editing of layout and local structure. It is
developed by leveraging the prior knowledge of large language and T2I models,
trained on massive corpora. The system comprises four interconnected
components: story-to-prompt generation (S2P), text-to-layout generation (T2L),
controllable text-to-image generation (C-T2I), and image-to-video animation
(I2V). First, the S2P module converts concise story information into detailed
prompts required for subsequent stages. Next, T2L generates diverse and
reasonable layouts based on the prompts, offering users the ability to adjust
and refine the layout to their preference. The core component, C-T2I, enables
the creation of images guided by layouts, sketches, and actor-specific
identifiers to maintain consistency and detail across visualizations. Finally,
I2V enriches the visualization process by animating the generated images.
Extensive experiments and a user study are conducted to validate the
effectiveness and flexibility of interactive editing of the proposed system.
Generating Images with Multimodal Language Models
Authors: name: Jing Yu Koh, name: Daniel Fried, name: Ruslan Salakhutdinov
Published: 2023-05-26 19:22:03

We propose a method to fuse frozen text-only large language models (LLMs)
with pre-trained image encoder and decoder models, by mapping between their
embedding spaces. Our model demonstrates a wide suite of multimodal
capabilities: image retrieval, novel image generation, and multimodal dialogue.
Ours is the first approach capable of conditioning on arbitrarily interleaved
image and text inputs to generate coherent image (and text) outputs. To achieve
strong performance on image generation, we propose an efficient mapping network
to ground the LLM to an off-the-shelf text-to-image generation model. This
mapping network translates hidden representations of text into the embedding
space of the visual models, enabling us to leverage the strong text
representations of the LLM for visual outputs. Our approach outperforms
baseline generation models on tasks with longer and more complex language. In
addition to novel image generation, our model is also capable of image
retrieval from a prespecified dataset, and decides whether to retrieve or
generate at inference time. This is done with a learnt decision module which
conditions on the hidden representations of the LLM. Our model exhibits a wider
range of capabilities compared to prior multimodal language models. It can
process image-and-text inputs, and produce retrieved images, generated images,
and generated text -- outperforming non-LLM based generation models across
several text-to-image tasks that measure context dependence.
High-Fidelity Image Compression with Score-based Generative Models
Authors: name: Emiel Hoogeboom, name: Eirikur Agustsson, name: Fabian Mentzer, name: Luca Versari, name: George Toderici, name: Lucas Theis
Published: 2023-05-26 17:16:16

Despite the tremendous success of diffusion generative models in
text-to-image generation, replicating this success in the domain of image
compression has proven difficult. In this paper, we demonstrate that diffusion
can significantly improve perceptual quality at a given bit-rate, outperforming
state-of-the-art approaches PO-ELIC and HiFiC as measured by FID score. This is
achieved using a simple but theoretically motivated two-stage approach
combining an autoencoder targeting MSE followed by a further score-based
decoder. However, as we will show, implementation details matter and the
optimal design decisions can differ greatly from typical text-to-image models.
RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
Authors: name: Zeyue Xue, name: Guanglu Song, name: Qiushan Guo, name: Boxiao Liu, name: Zhuofan Zong, name: Yu Liu, name: Ping Luo
Published: 2023-05-29 17:59:41

Text-to-image generation has recently witnessed remarkable achievements. We
introduce a text-conditional image diffusion model, termed RAPHAEL, to generate
highly artistic images, which accurately portray the text prompts, encompassing
multiple nouns, adjectives, and verbs. This is achieved by stacking tens of
mixture-of-experts (MoEs) layers, i.e., space-MoE and time-MoE layers, enabling
billions of diffusion paths (routes) from the network input to the output. Each
path intuitively functions as a "painter" for depicting a particular textual
concept onto a specified image region at a diffusion timestep. Comprehensive
experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as
Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2, in terms of both
image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior
performance in switching images across diverse styles, such as Japanese comics,
realism, cyberpunk, and ink illustration. Secondly, a single model with three
billion parameters, trained on 1,000 A100 GPUs for two months, achieves a
state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. Furthermore,
RAPHAEL significantly surpasses its counterparts in human evaluation on the
ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the
frontiers of image generation research in both academia and industry, paving
the way for future breakthroughs in this rapidly evolving field. More details
can be found on a project webpage: https://raphael-painter.github.io/.
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
  Customization of Diffusion Models
Authors: name: Yuchao Gu, name: Xintao Wang, name: Jay Zhangjie Wu, name: Yujun Shi, name: Yunpeng Chen, name: Zihan Fan, name: Wuyou Xiao, name: Rui Zhao, name: Shuning Chang, name: Weijia Wu, name: Yixiao Ge, name: Ying Shan, name: Mike Zheng Shou
Published: 2023-05-29 17:58:16

Public large-scale text-to-image diffusion models, such as Stable Diffusion,
have gained significant attention from the community. These models can be
easily customized for new concepts using low-rank adaptations (LoRAs). However,
the utilization of multiple concept LoRAs to jointly support multiple
customized concepts presents a challenge. We refer to this scenario as
decentralized multi-concept customization, which involves single-client concept
tuning and center-node concept fusion. In this paper, we propose a new
framework called Mix-of-Show that addresses the challenges of decentralized
multi-concept customization, including concept conflicts resulting from
existing single-client LoRA tuning and identity loss during model fusion.
Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
tuning and gradient fusion for the center node to preserve the in-domain
essence of single concepts and support theoretically limitless concept fusion.
Additionally, we introduce regionally controllable sampling, which extends
spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
attribute binding and missing object problems in multi-concept sampling.
Extensive experiments demonstrate that Mix-of-Show is capable of composing
multiple customized concepts with high fidelity, including characters, objects,
and scenes.
Photoswap: Personalized Subject Swapping in Images
Authors: name: Jing Gu, name: Yilin Wang, name: Nanxuan Zhao, name: Tsu-Jui Fu, name: Wei Xiong, name: Qing Liu, name: Zhifei Zhang, name: He Zhang, name: Jianming Zhang, name: HyunJoon Jung, name: Xin Eric Wang
Published: 2023-05-29 17:56:13

In an era where images and visual content dominate our digital landscape, the
ability to manipulate and personalize these images has become a necessity.
Envision seamlessly substituting a tabby cat lounging on a sunlit window sill
in a photograph with your own playful puppy, all while preserving the original
charm and composition of the image. We present Photoswap, a novel approach that
enables this immersive image editing experience through personalized subject
swapping in existing images. Photoswap first learns the visual concept of the
subject from reference images and then swaps it into the target image using
pre-trained diffusion models in a training-free manner. We establish that a
well-conceptualized visual subject can be seamlessly transferred to any image
with appropriate self-attention and cross-attention manipulation, maintaining
the pose of the swapped subject and the overall coherence of the image.
Comprehensive experiments underscore the efficacy and controllability of
Photoswap in personalized subject swapping. Furthermore, Photoswap
significantly outperforms baseline methods in human ratings across subject
swapping, background preservation, and overall quality, revealing its vast
application potential, from entertainment to professional editing.
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning
  and Diffusion Priors
Authors: name: Paul S. Scotti, name: Atmadeep Banerjee, name: Jimmie Goode, name: Stepan Shabalin, name: Alex Nguyen, name: Ethan Cohen, name: Aidan J. Dempster, name: Nathalie Verlinde, name: Elad Yundler, name: David Weisberg, name: Kenneth A. Norman, name: Tanishq Mathew Abraham
Published: 2023-05-29 17:49:00

We present MindEye, a novel fMRI-to-image approach to retrieve and
reconstruct viewed images from brain activity. Our model comprises two parallel
submodules that are specialized for retrieval (using contrastive learning) and
reconstruction (using a diffusion prior). MindEye can map fMRI brain activity
to any high dimensional multimodal latent space, like CLIP image space,
enabling image reconstruction using generative models that accept embeddings
from this latent space. We comprehensively compare our approach with other
existing methods, using both qualitative side-by-side comparisons and
quantitative evaluations, and show that MindEye achieves state-of-the-art
performance in both reconstruction and retrieval tasks. In particular, MindEye
can retrieve the exact original image even among highly similar candidates
indicating that its brain embeddings retain fine-grained image-specific
information. This allows us to accurately retrieve images even from large-scale
databases like LAION-5B. We demonstrate through ablations that MindEye's
performance improvements over previous methods result from specialized
submodules for retrieval and reconstruction, improved training techniques, and
training models with orders of magnitude more parameters. Furthermore, we show
that MindEye can better preserve low-level image features in the
reconstructions by using img2img, with outputs from a separate autoencoder. All
code is available on GitHub.
GlyphControl: Glyph Conditional Control for Visual Text Generation
Authors: name: Yukang Yang, name: Dongnan Gui, name: Yuhui Yuan, name: Haisong Ding, name: Han Hu, name: Kai Chen
Published: 2023-05-29 17:27:59

Recently, there has been a growing interest in developing diffusion-based
text-to-image generative models capable of generating coherent and well-formed
visual text. In this paper, we propose a novel and efficient approach called
GlyphControl to address this task. Unlike existing methods that rely on
character-aware text encoders like ByT5 and require retraining of text-to-image
models, our approach leverages additional glyph conditional information to
enhance the performance of the off-the-shelf Stable-Diffusion model in
generating accurate visual text. By incorporating glyph instructions, users can
customize the content, location, and size of the generated text according to
their specific requirements. To facilitate further research in visual text
generation, we construct a training benchmark dataset called LAION-Glyph. We
evaluate the effectiveness of our approach by measuring OCR-based metrics and
CLIP scores of the generated visual text. Our empirical evaluations demonstrate
that GlyphControl outperforms the recent DeepFloyd IF approach in terms of OCR
accuracy and CLIP scores, highlighting the efficacy of our method.
BigTrans: Augmenting Large Language Models with Multilingual Translation
  Capability over 100 Languages
Authors: name: Wen Yang, name: Chong Li, name: Jiajun Zhang, name: Chengqing Zong
Published: 2023-05-29 14:07:52

Large language models (LLMs) demonstrate promising translation performance
among various natural languages. However, many LLMs especially the open-sourced
ones, such as BLOOM and LLaMA, are English-dominant and support only dozens of
natural languages, making the potential of LLMs on language translation less
explored. In this work, we present BigTrans which adapts LLaMA that covers only
20 languages and enhances it with multilingual translation capability on more
than 100 languages. BigTrans is built upon LLaMA-13B and it is optimized in
three steps. First, we continue training LLaMA with massive Chinese monolingual
data. Second, we continue training the model with a large-scale parallel
dataset that covers 102 natural languages. Third, we instruct-tune the
foundation model with multilingual translation instructions, leading to our
BigTrans model. The preliminary experiments on multilingual translation show
that BigTrans performs comparably with ChatGPT and Google Translate in many
languages and even outperforms ChatGPT in 8 language pairs. We release the
BigTrans model and hope it can advance the research progress.
Model Dementia: Generated Data Makes Models Forget
Authors: name: Ilia Shumailov, name: Zakhar Shumaylov, name: Yiren Zhao, name: Yarin Gal, name: Nicolas Papernot, name: Ross Anderson
Published: 2023-05-27 15:10:41

Stable Diffusion revolutionised image creation from descriptive text. GPT-2,
GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of
language tasks. ChatGPT introduced such language models to the general public.
It is now clear that large language models (LLMs) are here to stay, and will
bring about drastic change in the whole ecosystem of online text and images. In
this paper we consider what the future might hold. What will happen to GPT-{n}
once LLMs contribute much of the language found online? We find that use of
model-generated content in training causes irreversible defects in the
resulting models, where tails of the original content distribution disappear.
We call this effect model dementia and show that it can occur in Variational
Autoencoders (VAEs), Gaussian Mixture Models (GMMs) and LLMs. We build
theoretical intuition behind the phenomenon and portray its ubiquity amongst
all learned generative models. We demonstrate that it has to be taken seriously
if we are to sustain the benefits of training from large-scale data scraped
from the web. Indeed, the value of data collected about genuine human
interactions with systems will be increasingly valuable in the presence of
content generated by LLMs in data crawled from the Internet.
SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex
  Interactive Tasks
Authors: name: Bill Yuchen Lin, name: Yicheng Fu, name: Karina Yang, name: Prithviraj Ammanabrolu, name: Faeze Brahman, name: Shiyu Huang, name: Chandra Bhagavatula, name: Yejin Choi, name: Xiang Ren
Published: 2023-05-27 07:04:15

We introduce SwiftSage, a novel agent framework inspired by the dual-process
theory of human cognition, designed to excel in action planning for complex
interactive reasoning tasks. SwiftSage integrates the strengths of behavior
cloning and prompting large language models (LLMs) to enhance task completion
performance. The framework comprises two primary modules: the Swift module,
representing fast and intuitive thinking, and the Sage module, emulating
deliberate thought processes. The Swift module is a small encoder-decoder LM
fine-tuned on the oracle agent's action trajectories, while the Sage module
employs LLMs such as GPT-4 for subgoal planning and grounding. We develop a
heuristic method to harmoniously integrate the two modules, resulting in a more
efficient and robust problem-solving process. In 30 tasks from the ScienceWorld
benchmark, SwiftSage significantly outperforms other methods such as SayCan,
ReAct, and Reflexion, demonstrating its effectiveness in solving complex
real-world tasks.
DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of
  GPT-Generated Text
Authors: name: Xianjun Yang, name: Wei Cheng, name: Linda Petzold, name: William Yang Wang, name: Haifeng Chen
Published: 2023-05-27 03:58:29

Large language models (LLMs) have notably enhanced the fluency and diversity
of machine-generated text. However, this progress also presents a significant
challenge in detecting the origin of a given text, and current research on
detection methods lags behind the rapid evolution of LLMs. Conventional
training-based methods have limitations in flexibility, particularly when
adapting to new domains, and they often lack explanatory power. To address this
gap, we propose a novel training-free detection strategy called Divergent
N-Gram Analysis (DNA-GPT). Given a text, we first truncate it in the middle and
then use only the preceding portion as input to the LLMs to regenerate the new
remaining parts. By analyzing the differences between the original and new
remaining parts through N-gram analysis in black-box or probability divergence
in white-box, we can clearly illustrate significant discrepancies between
machine-generated and human-written text. We conducted extensive experiments on
the most advanced LLMs from OpenAI, including text-davinci-003, GPT-3.5-turbo,
and GPT-4, as well as open-source models such as GPT-NeoX-20B and LLaMa-13B.
Results show that our zero-shot approach exhibits state-of-the-art performance
in distinguishing between human and GPT-generated text on four English and one
German dataset, outperforming OpenAI's own classifier, which is trained on
millions of text. Additionally, our methods provide reasonable explanations and
evidence to support our claim, which is a unique feature of explainable
detection. Our method is also robust under the revised text attack and can
additionally solve model sourcing. Codes are available at
https://github.com/Xianjun-Yang/DNA-GPT.
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language
  Models' Reasoning Performance
Authors: name: Yao Fu, name: Litu Ou, name: Mingyu Chen, name: Yuhao Wan, name: Hao Peng, name: Tushar Khot
Published: 2023-05-26 23:46:42

As large language models (LLMs) are continuously being developed, their
evaluation becomes increasingly important yet challenging. This work proposes
Chain-of-Thought Hub, an open-source evaluation suite on the multi-step
reasoning capabilities of large language models. We are interested in this
setting for two reasons: (1) from the behavior of GPT and PaLM model family, we
observe that complex reasoning is likely to be a key differentiator between
weaker and stronger LLMs; (2) we envisage large language models to become the
next-generation computational platform and foster an ecosystem of LLM-based new
applications, this naturally requires the foundation models to perform complex
tasks that often involve the composition of linguistic and logical operations.
Our approach is to compile a suite of challenging reasoning benchmarks to track
the progress of LLMs. Our current results show that: (1) model scale clearly
correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and
PaLM-2 are the only two models that are comparable with GPT-4, while
open-sourced models still lag behind; (3) LLaMA-65B performs closely to
code-davinci-002, indicating that with successful further development such as
reinforcement learning from human feedback (RLHF), it has great potential to be
close to GPT-3.5-Turbo. Our results also suggest that for the open-source
efforts to catch up, the community may focus more on building better base
models and exploring RLHF.