Orionnnn/hub-stats huggingface dataset:
OpenShape: Scaling Up 3D Shape Representation Towards Open-World
  Understanding
Authors: name: Minghua Liu, name: Ruoxi Shi, name: Kaiming Kuang, name: Yinhao Zhu, name: Xuanlin Li, name: Shizhong Han, name: Hong Cai, name: Fatih Porikli, name: Hao Su
Published: 2023-05-18 07:07:19

We introduce OpenShape, a method for learning multi-modal joint
representations of text, image, and point clouds. We adopt the commonly used
multi-modal contrastive learning framework for representation alignment, but
with a specific focus on scaling up 3D representations to enable open-world 3D
shape understanding. To achieve this, we scale up training data by ensembling
multiple 3D datasets and propose several strategies to automatically filter and
enrich noisy text descriptions. We also explore and compare strategies for
scaling 3D backbone networks and introduce a novel hard negative mining module
for more efficient training. We evaluate OpenShape on zero-shot 3D
classification benchmarks and demonstrate its superior capabilities for
open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy
of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than
10% for existing methods. OpenShape also achieves an accuracy of 85.3% on
ModelNet40, outperforming previous zero-shot baseline methods by 20% and
performing on par with some fully-supervised methods. Furthermore, we show that
our learned embeddings encode a wide range of visual and semantic concepts
(e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D
and image-3D interactions. Due to their alignment with CLIP embeddings, our
learned shape representations can also be integrated with off-the-shelf
CLIP-based models for various applications, such as point cloud captioning and
point cloud-conditioned image generation.
CLAPSpeech: Learning Prosody from Text Context with Contrastive
  Language-Audio Pre-training
Authors: name: Zhenhui Ye, name: Rongjie Huang, name: Yi Ren, name: Ziyue Jiang, name: Jinglin Liu, name: Jinzheng He, name: Xiang Yin, name: Zhou Zhao
Published: 2023-05-18 07:07:04

Improving text representation has attracted much attention to achieve
expressive text-to-speech (TTS). However, existing works only implicitly learn
the prosody with masked token reconstruction tasks, which leads to low training
efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a
cross-modal contrastive pre-training framework that explicitly learns the
prosody variance of the same text token under different contexts. Specifically,
1) We encourage the model to connect the text context with its corresponding
prosody pattern in the joint multi-modal space with the elaborate design of the
encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training
pipeline to capture prosody patterns in multiple levels. We show how to
incorporate CLAPSpeech into existing TTS models for better prosody. Experiments
on three datasets not only show that CLAPSpeech could improve the prosody
prediction for existing TTS methods, but also demonstrate its generalization
ability to adapt to multiple languages and multi-speaker TTS. We also deeply
analyze the principle behind the performance of CLAPSpeech. Ablation studies
demonstrate the necessity of each component in our method. Source code and
audio samples are available at https://clapspeech.github.io.
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for
  Longer Sequences
Authors: name: David Uthus, name: Santiago Ontañón, name: Joshua Ainslie, name: Mandy Guo
Published: 2023-05-18 17:22:53

We present our work on developing a multilingual, efficient text-to-text
transformer that is suitable for handling long inputs. This model, called
mLongT5, builds upon the architecture of LongT5, while leveraging the
multilingual datasets used for pretraining mT5 and the pretraining tasks of
UL2. We evaluate this model on a variety of multilingual summarization and
question-answering tasks, and the results show stronger performance for mLongT5
when compared to existing multilingual models such as mBART or M-BERT.
A Generalist Dynamics Model for Control
Authors: name: Ingmar Schubert, name: Jingwei Zhang, name: Jake Bruce, name: Sarah Bechtle, name: Emilio Parisotto, name: Martin Riedmiller, name: Jost Tobias Springenberg, name: Arunkumar Byravan, name: Leonard Hasenclever, name: Nicolas Heess
Published: 2023-05-18 12:14:49

We investigate the use of transformer sequence models as dynamics models
(TDMs) for control. In a number of experiments in the DeepMind control suite,
we find that first, TDMs perform well in a single-environment learning setting
when compared to baseline models. Second, TDMs exhibit strong generalization
capabilities to unseen environments, both in a few-shot setting, where a
generalist model is fine-tuned with small amounts of data from the target
environment, and in a zero-shot setting, where a generalist model is applied to
an unseen environment without any further training. We further demonstrate that
generalizing system dynamics can work much better than generalizing optimal
behavior directly as a policy. This makes TDMs a promising ingredient for a
foundation model of control.
VisionLLM: Large Language Model is also an Open-Ended Decoder for
  Vision-Centric Tasks
Authors: name: Wenhai Wang, name: Zhe Chen, name: Xiaokang Chen, name: Jiannan Wu, name: Xizhou Zhu, name: Gang Zeng, name: Ping Luo, name: Tong Lu, name: Jie Zhou, name: Yu Qiao, name: Jifeng Dai
Published: 2023-05-18 17:59:42

Large language models (LLMs) have notably accelerated progress towards
artificial general intelligence (AGI), with their impressive zero-shot capacity
for user-tailored tasks, endowing them with immense potential across a range of
applications. However, in the field of computer vision, despite the
availability of numerous powerful vision foundation models (VFMs), they are
still restricted to tasks in a pre-defined form, struggling to match the
open-ended task capabilities of LLMs. In this work, we present an LLM-based
framework for vision-centric tasks, termed VisionLLM. This framework provides a
unified perspective for vision and language tasks by treating images as a
foreign language and aligning vision-centric tasks with language tasks that can
be flexibly defined and managed using language instructions. An LLM-based
decoder can then make appropriate predictions based on these instructions for
open-ended tasks. Extensive experiments show that the proposed VisionLLM can
achieve different levels of task customization through language instructions,
from fine-grained object-level to coarse-grained task-level customization, all
with good results. It's noteworthy that, with a generalist LLM-based framework,
our model can achieve over 60\% mAP on COCO, on par with detection-specific
models. We hope this model can set a new baseline for generalist vision and
language models. The demo shall be released based on
https://github.com/OpenGVLab/InternGPT. The code shall be released at
https://github.com/OpenGVLab/VisionLLM.
TrueTeacher: Learning Factual Consistency Evaluation with Large Language
  Models
Authors: name: Zorik Gekhman, name: Jonathan Herzig, name: Roee Aharoni, name: Chen Elkind, name: Idan Szpektor
Published: 2023-05-18 17:58:35

Factual consistency evaluation is often conducted using Natural Language
Inference (NLI) models, yet these models exhibit limited success in evaluating
summaries. Previous work improved such models with synthetic training data.
However, the data is typically based on perturbed human-written summaries,
which often differ in their characteristics from real model-generated summaries
and have limited coverage of possible factual errors. Alternatively, large
language models (LLMs) have recently shown promising results in directly
evaluating generative tasks, but are too computationally expensive for
practical use. Motivated by these limitations, we introduce TrueTeacher, a
method for generating synthetic data by annotating diverse model-generated
summaries using a LLM. Unlike prior work, TrueTeacher does not rely on
human-written summaries, and is multilingual by nature. Experiments on the TRUE
benchmark show that a student model trained using our data, substantially
outperforms both the state-of-the-art model with similar capacity, and the LLM
teacher. In a systematic study, we compare TrueTeacher to existing synthetic
data generation methods and demonstrate its superiority and robustness to
domain-shift. Using the the mFACE dataset, we also show that our method
generalizes to multilingual scenarios. Finally, we release a large-scale
synthetic dataset with 1.4M examples generated using TrueTeacher.
MolXPT: Wrapping Molecules with Text for Generative Pre-training
Authors: name: Zequn Liu, name: Wei Zhang, name: Yingce Xia, name: Lijun Wu, name: Shufang Xie, name: Tao Qin, name: Ming Zhang, name: Tie-Yan Liu
Published: 2023-05-18 03:58:19

Generative pre-trained Transformer (GPT) has demonstrates its great success
in natural language processing and related techniques have been adapted into
molecular modeling. Considering that text is the most important record for
scientific discovery, in this paper, we propose MolXPT, a unified language
model of text and molecules pre-trained on SMILES (a sequence representation of
molecules) wrapped by text. Briefly, we detect the molecule names in each
sequence and replace them to the corresponding SMILES. In this way, the SMILES
could leverage the information from surrounding text, and vice versa. The above
wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem
are all fed into a language model for pre-training. Experimental results
demonstrate that MolXPT outperforms strong baselines of molecular property
prediction on MoleculeNet, performs comparably to the best model in
text-molecule translation while using less than half of its parameters, and
enables zero-shot molecular generation without finetuning.
Learning the Visualness of Text Using Large Vision-Language Models
Authors: name: Gaurav Verma, name: Ryan A. Rossi, name: Christopher Tensmeyer, name: Jiuxiang Gu, name: Ani Nenkova
Published: 2023-05-11 17:45:16

Visual text evokes an image in a person's mind, while non-visual text fails
to do so. A method to automatically detect visualness in text will unlock the
ability to augment text with relevant images, as neural text-to-image
generation and retrieval models operate on the implicit assumption that the
input text is visual in nature. We curate a dataset of 3,620 English sentences
and their visualness scores provided by multiple human annotators.
Additionally, we use documents that contain text and visual assets to create a
distantly supervised corpus of document text and associated images. We also
propose a fine-tuning strategy that adapts large vision-language models like
CLIP that assume a one-to-one correspondence between text and image to the task
of scoring text visualness from text input alone. Our strategy involves
modifying the model's contrastive learning objective to map text identified as
non-visual to a common NULL image while matching visual text to their
corresponding images in the document. We evaluate the proposed approach on its
ability to (i) classify visual and non-visual text accurately, and (ii) attend
over words that are identified as visual in psycholinguistic studies. Empirical
evaluation indicates that our approach performs better than several heuristics
and baseline models for the proposed task. Furthermore, to highlight the
importance of modeling the visualness of text, we conduct qualitative analyses
of text-to-image generation systems like DALL-E.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Authors: name: Shunyu Yao, name: Dian Yu, name: Jeffrey Zhao, name: Izhak Shafran, name: Thomas L. Griffiths, name: Yuan Cao, name: Karthik Narasimhan
Published: 2023-05-17 23:16:17

Language models are increasingly being deployed for general problem solving
across a wide range of tasks, but are still confined to token-level,
left-to-right decision-making processes during inference. This means they can
fall short in tasks that require exploration, strategic lookahead, or where
initial decisions play a pivotal role. To surmount these challenges, we
introduce a new framework for language model inference, Tree of Thoughts (ToT),
which generalizes over the popular Chain of Thought approach to prompting
language models, and enables exploration over coherent units of text (thoughts)
that serve as intermediate steps toward problem solving. ToT allows LMs to
perform deliberate decision making by considering multiple different reasoning
paths and self-evaluating choices to decide the next course of action, as well
as looking ahead or backtracking when necessary to make global choices. Our
experiments show that ToT significantly enhances language models'
problem-solving abilities on three novel tasks requiring non-trivial planning
or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in
Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of
tasks, our method achieved a success rate of 74%. Code repo with all prompts:
https://github.com/ysymyth/tree-of-thought-llm.
UniControl: A Unified Diffusion Model for Controllable Visual Generation
  In the Wild
Authors: name: Can Qin, name: Shu Zhang, name: Ning Yu, name: Yihao Feng, name: Xinyi Yang, name: Yingbo Zhou, name: Huan Wang, name: Juan Carlos Niebles, name: Caiming Xiong, name: Silvio Savarese, name: Stefano Ermon, name: Yun Fu, name: Ran Xu
Published: 2023-05-18 17:41:34

Achieving machine autonomy and human control often represent divergent
objectives in the design of interactive AI systems. Visual generative
foundation models such as Stable Diffusion show promise in navigating these
goals, especially when prompted with arbitrary languages. However, they often
fall short in generating images with spatial, structural, or geometric
controls. The integration of such controls, which can accommodate various
visual conditions in a single unified model, remains an unaddressed challenge.
In response, we introduce UniControl, a new generative foundation model that
consolidates a wide array of controllable condition-to-image (C2I) tasks within
a singular framework, while still allowing for arbitrary language prompts.
UniControl enables pixel-level-precise image generation, where visual
conditions primarily influence the generated structures and language prompts
guide the style and context. To equip UniControl with the capacity to handle
diverse visual conditions, we augment pretrained text-to-image diffusion models
and introduce a task-aware HyperNet to modulate the diffusion models, enabling
the adaptation to different C2I tasks simultaneously. Trained on nine unique
C2I tasks, UniControl demonstrates impressive zero-shot generation abilities
with unseen visual conditions. Experimental results show that UniControl often
surpasses the performance of single-task-controlled methods of comparable model
sizes. This control versatility positions UniControl as a significant
advancement in the realm of controllable visual generation.
VideoFactory: Swap Attention in Spatiotemporal Diffusions for
  Text-to-Video Generation
Authors: name: Wenjing Wang, name: Huan Yang, name: Zixi Tuo, name: Huiguo He, name: Junchen Zhu, name: Jianlong Fu, name: Jiaying Liu
Published: 2023-05-18 11:06:15

We present VideoFactory, an innovative framework for generating high-quality
open-domain videos. VideoFactory excels in producing high-definition
(1376x768), widescreen (16:9) videos without watermarks, creating an engaging
user experience. Generating videos guided by text instructions poses
significant challenges, such as modeling the complex relationship between space
and time, and the lack of large-scale text-video paired data. Previous
approaches extend pretrained text-to-image generation models by adding temporal
1D convolution/attention modules for video generation. However, these
approaches overlook the importance of jointly modeling space and time,
inevitably leading to temporal distortions and misalignment between texts and
videos. In this paper, we propose a novel approach that strengthens the
interaction between spatial and temporal perceptions. In particular, we utilize
a swapped cross-attention mechanism in 3D windows that alternates the "query"
role between spatial and temporal blocks, enabling mutual reinforcement for
each other. To fully unlock model capabilities for high-quality video
generation, we curate a large-scale video dataset called HD-VG-130M. This
dataset comprises 130 million text-video pairs from the open-domain, ensuring
high-definition, widescreen and watermark-free characters. Objective metrics
and user studies demonstrate the superiority of our approach in terms of
per-frame quality, temporal correlation, and text-video alignment, with clear
margins.
TextDiffuser: Diffusion Models as Text Painters
Authors: name: Jingye Chen, name: Yupan Huang, name: Tengchao Lv, name: Lei Cui, name: Qifeng Chen, name: Furu Wei
Published: 2023-05-18 10:16:19

Diffusion models have gained increasing attention for their impressive
generation abilities but currently struggle with rendering accurate and
coherent text. To address this issue, we introduce TextDiffuser,
focusing on generating images with visually appealing text that is coherent
with backgrounds. TextDiffuser consists of two stages: first, a Transformer
model generates the layout of keywords extracted from text prompts, and then
diffusion models generate images conditioned on the text prompt and the
generated layout. Additionally, we contribute the first large-scale text images
dataset with OCR annotations, MARIO-10M, containing 10 million
image-text pairs with text recognition, detection, and character-level
segmentation annotations. We further collect the MARIO-Eval benchmark
to serve as a comprehensive tool for evaluating text rendering quality. Through
experiments and user studies, we show that TextDiffuser is flexible and
controllable to create high-quality text images using text prompts alone or
together with text template images, and conduct text inpainting to reconstruct
incomplete images with text. The code, model, and dataset will be available at
https://aka.ms/textdiffuser.
LDM3D: Latent Diffusion Model for 3D
Authors: name: Gabriela Ben Melech Stan, name: Diana Wofk, name: Scottie Fox, name: Alex Redden, name: Will Saxton, name: Jean Yu, name: Estelle Aflalo, name: Shao-Yen Tseng, name: Fabio Nonato, name: Matthias Muller, name: Vasudev Lal
Published: 2023-05-18 10:15:06

This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that
generates both image and depth map data from a given text prompt, allowing
users to generate RGBD images from text prompts. The LDM3D model is fine-tuned
on a dataset of tuples containing an RGB image, depth map and caption, and
validated through extensive experiments. We also develop an application called
DepthFusion, which uses the generated RGB images and depth maps to create
immersive and interactive 360-degree-view experiences using TouchDesigner. This
technology has the potential to transform a wide range of industries, from
entertainment and gaming to architecture and design. Overall, this paper
presents a significant contribution to the field of generative AI and computer
vision, and showcases the potential of LDM3D and DepthFusion to revolutionize
content creation and digital experiences. A short video summarizing the
approach can be found at https://t.ly/tdi2.
GETMusic: Generating Any Music Tracks with a Unified Representation and
  Diffusion Framework
Authors: name: Ang Lv, name: Xu Tan, name: Peiling Lu, name: Wei Ye, name: Shikun Zhang, name: Jiang Bian, name: Rui Yan
Published: 2023-05-18 09:53:23

Symbolic music generation aims to create musical notes, which can help users
compose music, such as generating target instrumental tracks from scratch, or
based on user-provided source tracks. Considering the diverse and flexible
combination between source and target tracks, a unified model capable of
generating any arbitrary tracks is of crucial necessity. Previous works fail to
address this need due to inherent constraints in music representations and
model architectures. To address this need, we propose a unified representation
and diffusion framework named GETMusic (`GET' stands for GEnerate music
Tracks), which includes a novel music representation named GETScore, and a
diffusion model named GETDiff. GETScore represents notes as tokens and
organizes them in a 2D structure, with tracks stacked vertically and
progressing horizontally over time. During training, tracks are randomly
selected as either the target or source. In the forward process, target tracks
are corrupted by masking their tokens, while source tracks remain as ground
truth. In the denoising process, GETDiff learns to predict the masked target
tokens, conditioning on the source tracks. With separate tracks in GETScore and
the non-autoregressive behavior of the model, GETMusic can explicitly control
the generation of any target tracks from scratch or conditioning on source
tracks. We conduct experiments on music generation involving six instrumental
tracks, resulting in a total of 665 combinations. GETMusic provides
high-quality results across diverse combinations and surpasses prior works
proposed for some specific combinations.
Discriminative Diffusion Models as Few-shot Vision and Language Learners
Authors: name: Xuehai He, name: Weixi Feng, name: Tsu-Jui Fu, name: Varun Jampani, name: Arjun Akula, name: Pradyumna Narayana, name: Sugato Basu, name: William Yang Wang, name: Xin Eric Wang
Published: 2023-05-18 05:41:36

Diffusion models, such as Stable Diffusion, have shown incredible performance
on text-to-image generation. Since text-to-image generation often requires
models to generate visual concepts with fine-grained details and attributes
specified in text prompts, can we leverage the powerful representations learned
by pre-trained diffusion models for discriminative tasks such as image-text
matching? To answer this question, we propose a novel approach, Discriminative
Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models
into few-shot discriminative learners. Our approach uses the cross-attention
score of a Stable Diffusion model to capture the mutual influence between
visual and textual information and fine-tune the model via attention-based
prompt learning to perform image-text matching. By comparing DSD with
state-of-the-art methods on several benchmark datasets, we demonstrate the
potential of using pre-trained diffusion models for discriminative tasks with
superior results on few-shot image-text matching.
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Authors: name: Songwei Ge, name: Seungjun Nah, name: Guilin Liu, name: Tyler Poon, name: Andrew Tao, name: Bryan Catanzaro, name: David Jacobs, name: Jia-Bin Huang, name: Ming-Yu Liu, name: Yogesh Balaji
Published: 2023-05-17 17:59:16

Despite tremendous progress in generating high-quality images using diffusion
models, synthesizing a sequence of animated frames that are both photorealistic
and temporally coherent is still in its infancy. While off-the-shelf
billion-scale datasets for image generation are available, collecting similar
video data of the same scale is still challenging. Also, training a video
diffusion model is computationally much more expensive than its image
counterpart. In this work, we explore finetuning a pretrained image diffusion
model with video data as a practical solution for the video synthesis task. We
find that naively extending the image noise prior to video noise prior in video
diffusion leads to sub-optimal performance. Our carefully designed video noise
prior leads to substantially better performance. Extensive experimental
validation shows that our model, Preserve Your Own Correlation (PYoCo), attains
SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It
also achieves SOTA video generation quality on the small-scale UCF-101
benchmark with a 10times smaller model using significantly less computation
than the prior art.