Orionnnn/hub-stats huggingface dataset:
Large Language Models as Analogical Reasoners
Authors: name: Michihiro Yasunaga, name: Xinyun Chen, name: Yujia Li, name: Panupong Pasupat, name: Jure Leskovec, name: Percy Liang, name: Ed H. Chi, name: Denny Zhou
Published: 2023-10-03 00:57:26

Chain-of-thought (CoT) prompting for language models demonstrates impressive
performance across reasoning tasks, but typically needs labeled exemplars of
the reasoning process. In this work, we introduce a new prompting approach,
Analogical Prompting, designed to automatically guide the reasoning process of
large language models. Inspired by analogical reasoning, a cognitive process in
which humans draw from relevant past experiences to tackle new problems, our
approach prompts language models to self-generate relevant exemplars or
knowledge in the context, before proceeding to solve the given problem. This
method presents several advantages: it obviates the need for labeling or
retrieving exemplars, offering generality and convenience; it can also tailor
the generated exemplars and knowledge to each problem, offering adaptability.
Experimental results show that our approach outperforms 0-shot CoT and manual
few-shot CoT in a variety of reasoning tasks, including math problem solving in
GSM8K and MATH, code generation in Codeforces, and other reasoning tasks in
BIG-Bench.
SmartPlay : A Benchmark for LLMs as Intelligent Agents
Authors: name: Yue Wu, name: Xuan Tang, name: Tom M. Mitchell, name: Yuanzhi Li
Published: 2023-10-02 18:52:11

Recent large language models (LLMs) have demonstrated great potential toward
intelligent agents and next-gen automation, but there currently lacks a
systematic benchmark for evaluating LLMs' abilities as agents. We introduce
SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs
as agents. SmartPlay consists of 6 different games, including
Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique
setting, providing up to 20 evaluation settings and infinite environment
variations. Each game in SmartPlay uniquely challenges a subset of 9 important
capabilities of an intelligent LLM agent, including reasoning with object
dependencies, planning ahead, spatial reasoning, learning from history, and
understanding randomness. The distinction between the set of capabilities each
game test allows us to analyze each capability separately. SmartPlay serves not
only as a rigorous testing ground for evaluating the overall performance of LLM
agents but also as a road-map for identifying gaps in current methodologies. We
release our benchmark at github.com/LLMsmartplay/SmartPlay
Conditional Diffusion Distillation
Authors: name: Kangfu Mei, name: Mauricio Delbracio, name: Hossein Talebi, name: Zhengzhong Tu, name: Vishal M. Patel, name: Peyman Milanfar
Published: 2023-10-02 17:59:18

Generative diffusion models provide strong priors for text-to-image
generation and thereby serve as a foundation for conditional generation tasks
such as image editing, restoration, and super-resolution. However, one major
limitation of diffusion models is their slow sampling time. To address this
challenge, we present a novel conditional distillation method designed to
supplement the diffusion priors with the help of image conditions, allowing for
conditional sampling with very few steps. We directly distill the unconditional
pre-training in a single stage through joint-learning, largely simplifying the
previous two-stage procedures that involve both distillation and conditional
finetuning separately. Furthermore, our method enables a new
parameter-efficient distillation mechanism that distills each task with only a
small number of additional parameters combined with the shared frozen
unconditional backbone. Experiments across multiple tasks including
super-resolution, image editing, and depth-to-image generation demonstrate that
our method outperforms existing distillation techniques for the same sampling
time. Notably, our method is the first distillation strategy that can match the
performance of the much slower fine-tuned conditional diffusion models.
Enable Language Models to Implicitly Learn Self-Improvement From Data
Authors: name: Ziqi Wang, name: Le Hou, name: Tianjian Lu, name: Yuexin Wu, name: Yunxuan Li, name: Hongkun Yu, name: Heng Ji
Published: 2023-10-02 04:29:40

Large Language Models (LLMs) have demonstrated remarkable capabilities in
open-ended text generation tasks. However, the inherent open-ended nature of
these tasks implies that there is always room for improvement in the quality of
model responses. To address this challenge, various approaches have been
proposed to enhance the performance of LLMs. There has been a growing focus on
enabling LLMs to self-improve their response quality, thereby reducing the
reliance on extensive human annotation efforts for collecting diverse and
high-quality training data. Recently, prompting-based methods have been widely
explored among self-improvement methods owing to their effectiveness,
efficiency, and convenience. However, those methods usually require explicitly
and thoroughly written rubrics as inputs to LLMs. It is expensive and
challenging to manually derive and provide all necessary rubrics with a
real-world complex goal for improvement (e.g., being more helpful and less
harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework
that implicitly learns the improvement goal from human preference data. PIT
only requires preference data that are used to train reward models without
extra human efforts. Specifically, we reformulate the training objective of
reinforcement learning from human feedback (RLHF) -- instead of maximizing
response quality for a given input, we maximize the quality gap of the response
conditioned on a reference response. In this way, PIT is implicitly trained
with the improvement goal of better aligning with human preferences.
Experiments on two real-world datasets and one synthetic dataset show that our
method significantly outperforms prompting-based methods.
PixArt-α: Fast Training of Diffusion Transformer for
  Photorealistic Text-to-Image Synthesis
Authors: name: Junsong Chen, name: Jincheng Yu, name: Chongjian Ge, name: Lewei Yao, name: Enze Xie1, name: Yue Wu, name: Zhongdao Wang, name: James Kwok, name: Ping Luo, name: Huchuan Lu, name: Zhenguo Li
Published: 2023-09-30 16:18:00

The most advanced text-to-image (T2I) models require significant training
costs (e.g., millions of GPU hours), seriously hindering the fundamental
innovation for the AIGC community while increasing CO2 emissions. This paper
introduces PIXART-alpha, a Transformer-based T2I diffusion model whose image
generation quality is competitive with state-of-the-art image generators (e.g.,
Imagen, SDXL, and even Midjourney), reaching near-commercial application
standards. Additionally, it supports high-resolution image synthesis up to
1024px resolution with low training cost, as shown in Figure 1 and 2. To
achieve this goal, three core designs are proposed: (1) Training strategy
decomposition: We devise three distinct training steps that separately optimize
pixel dependency, text-image alignment, and image aesthetic quality; (2)
Efficient T2I Transformer: We incorporate cross-attention modules into
Diffusion Transformer (DiT) to inject text conditions and streamline the
computation-intensive class-condition branch; (3) High-informative data: We
emphasize the significance of concept density in text-image pairs and leverage
a large Vision-Language model to auto-label dense pseudo-captions to assist
text-image alignment learning. As a result, PIXART-alpha's training speed
markedly surpasses existing large-scale T2I models, e.g., PIXART-alpha only
takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU
days), saving nearly \300,000 (26,000 vs. \320,000) and reducing 90% CO2
emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training
cost is merely 1%. Extensive experiments demonstrate that PIXART-\alpha
excels in image quality, artistry, and semantic control. We hope
PIXART-\alpha$ will provide new insights to the AIGC community and startups to
accelerate building their own high-quality yet low-cost generative models from
scratch.
Demystifying CLIP Data
Authors: name: Hu Xu, name: Saining Xie, name: Xiaoqing Ellen Tan, name: Po-Yao Huang, name: Russell Howes, name: Vasu Sharma, name: Shang-Wen Li, name: Gargi Ghosh, name: Luke Zettlemoyer, name: Christoph Feichtenhofer
Published: 2023-09-28 17:59:56

Contrastive Language-Image Pre-training (CLIP) is an approach that has
advanced research and applications in computer vision, fueling modern
recognition systems and generative models. We believe that the main ingredient
to the success of CLIP is its data and not the model architecture or
pre-training objective. However, CLIP only provides very limited information
about its data and how it has been collected, leading to works that aim to
reproduce CLIP's data by filtering with its model parameters. In this work, we
intend to reveal CLIP's data curation approach and in our pursuit of making it
open to the community introduce Metadata-Curated Language-Image Pre-training
(MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's
concepts) and yields a balanced subset over the metadata distribution. Our
experimental study rigorously isolates the model and training settings,
concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M
image-text data pairs outperforms CLIP's data on multiple standard benchmarks.
In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy,
surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining
the same training budget, attains 72.4%. Our observations hold across various
model sizes, exemplified by ViT-H achieving 80.5%, without any
bells-and-whistles. Curation code and training data distribution on metadata is
made available at https://github.com/facebookresearch/MetaCLIP.
Deep Geometrized Cartoon Line Inbetweening
Authors: name: Li Siyao, name: Tianpei Gu, name: Weiye Xiao, name: Henghui Ding, name: Ziwei Liu, name: Chen Change Loy
Published: 2023-09-28 17:50:05

We aim to address a significant but understudied problem in the anime
industry, namely the inbetweening of cartoon line drawings. Inbetweening
involves generating intermediate frames between two black-and-white line
drawings and is a time-consuming and expensive process that can benefit from
automation. However, existing frame interpolation methods that rely on matching
and warping whole raster images are unsuitable for line inbetweening and often
produce blurring artifacts that damage the intricate line structures. To
preserve the precision and detail of the line drawings, we propose a new
approach, AnimeInbet, which geometrizes raster line drawings into graphs of
endpoints and reframes the inbetweening task as a graph fusion problem with
vertex repositioning. Our method can effectively capture the sparsity and
unique structure of line drawings while preserving the details during
inbetweening. This is made possible via our novel modules, i.e., vertex
geometric embedding, a vertex correspondence Transformer, an effective
mechanism for vertex repositioning and a visibility predictor. To train our
method, we introduce MixamoLine240, a new dataset of line drawings with ground
truth vectorization and matching labels. Our experiments demonstrate that
AnimeInbet synthesizes high-quality, clean, and complete intermediate line
drawings, outperforming existing methods quantitatively and qualitatively,
especially in cases with large motions. Data and code are available at
https://github.com/lisiyao21/AnimeInbet.
Vision Transformers Need Registers
Authors: name: Timothée Darcet, name: Maxime Oquab, name: Julien Mairal, name: Piotr Bojanowski
Published: 2023-09-28 16:45:46

Transformers have recently emerged as a powerful tool for learning visual
representations. In this paper, we identify and characterize artifacts in
feature maps of both supervised and self-supervised ViT networks. The artifacts
correspond to high-norm tokens appearing during inference primarily in
low-informative background areas of images, that are repurposed for internal
computations. We propose a simple yet effective solution based on providing
additional tokens to the input sequence of the Vision Transformer to fill that
role. We show that this solution fixes that problem entirely for both
supervised and self-supervised models, sets a new state of the art for
self-supervised visual models on dense visual prediction tasks, enables object
discovery methods with larger models, and most importantly leads to smoother
feature maps and attention maps for downstream visual processing.
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content
  Creation
Authors: name: Jiaxiang Tang, name: Jiawei Ren, name: Hang Zhou, name: Ziwei Liu, name: Gang Zeng
Published: 2023-09-28 17:55:05

Recent advances in 3D content creation mostly leverage optimization-based 3D
generation via score distillation sampling (SDS). Though promising results have
been exhibited, these methods often suffer from slow per-sample optimization,
limiting their practical usage. In this paper, we propose DreamGaussian, a
novel 3D content generation framework that achieves both efficiency and quality
simultaneously. Our key insight is to design a generative 3D Gaussian Splatting
model with companioned mesh extraction and texture refinement in UV space. In
contrast to the occupancy pruning used in Neural Radiance Fields, we
demonstrate that the progressive densification of 3D Gaussians converges
significantly faster for 3D generative tasks. To further enhance the texture
quality and facilitate downstream applications, we introduce an efficient
algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning
stage to refine the details. Extensive experiments demonstrate the superior
efficiency and competitive generation quality of our proposed approach.
Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes
from a single-view image, achieving approximately 10 times acceleration
compared to existing methods.
Text-to-3D using Gaussian Splatting
Authors: name: Zilong Chen, name: Feng Wang, name: Huaping Liu
Published: 2023-09-28 16:44:31

In this paper, we present Gaussian Splatting based text-to-3D generation
(GSGEN), a novel approach for generating high-quality 3D objects. Previous
methods suffer from inaccurate geometry and limited fidelity due to the absence
of 3D prior and proper representation. We leverage 3D Gaussian Splatting, a
recent state-of-the-art representation, to address existing shortcomings by
exploiting the explicit nature that enables the incorporation of 3D prior.
Specifically, our method adopts a progressive optimization strategy, which
includes a geometry optimization stage and an appearance refinement stage. In
geometry optimization, a coarse representation is established under a 3D
geometry prior along with the ordinary 2D SDS loss, ensuring a sensible and
3D-consistent rough shape. Subsequently, the obtained Gaussians undergo an
iterative refinement to enrich details. In this stage, we increase the number
of Gaussians by compactness-based densification to enhance continuity and
improve fidelity. With these designs, our approach can generate 3D content with
delicate details and more accurate geometry. Extensive evaluations demonstrate
the effectiveness of our method, especially for capturing high-frequency
components. Video results are provided at https://gsgen3d.github.io. Our code
is available at https://github.com/gsgen3d/gsgen
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model
  Adaptation
Authors: name: Guy Yariv, name: Itai Gat, name: Sagie Benaim, name: Lior Wolf, name: Idan Schwartz, name: Yossi Adi
Published: 2023-09-28 13:26:26

We consider the task of generating diverse and realistic videos guided by
natural audio samples from a wide variety of semantic classes. For this task,
the videos are required to be aligned both globally and temporally with the
input audio: globally, the input audio is semantically associated with the
entire output video, and temporally, each segment of the input audio is
associated with a corresponding segment of that video. We utilize an existing
text-conditioned video generation model and a pre-trained audio encoder model.
The proposed method is based on a lightweight adaptor network, which learns to
map the audio-based representation to the input representation expected by the
text-to-video generation model. As such, it also enables video generation
conditioned on text, audio, and, for the first time as far as we can ascertain,
on both text and audio. We validate our method extensively on three datasets
demonstrating significant semantic diversity of audio-video samples and further
propose a novel evaluation metric (AV-Align) to assess the alignment of
generated videos with input audio samples. AV-Align is based on the detection
and comparison of energy peaks in both modalities. In comparison to recent
state-of-the-art approaches, our method generates videos that are better
aligned with the input sound, both with respect to content and temporal axis.
We also show that videos produced by our method present higher visual quality
and are more diverse.
CCEdit: Creative and Controllable Video Editing via Diffusion Models
Authors: name: Ruoyu Feng, name: Wenming Weng, name: Yanhui Wang, name: Yuhui Yuan, name: Jianmin Bao, name: Chong Luo, name: Zhibo Chen, name: Baining Guo
Published: 2023-09-28 15:03:44

In this work, we present CCEdit, a versatile framework designed to address
the challenges of creative and controllable video editing. CCEdit accommodates
a wide spectrum of user editing requirements and enables enhanced creative
control through an innovative approach that decouples video structure and
appearance. We leverage the foundational ControlNet architecture to preserve
structural integrity, while seamlessly integrating adaptable temporal modules
compatible with state-of-the-art personalization techniques for text-to-image
generation, such as DreamBooth and LoRA.Furthermore, we introduce
reference-conditioned video editing, empowering users to exercise precise
creative control over video editing through the more manageable process of
editing key frames. Our extensive experimental evaluations confirm the
exceptional functionality and editing capabilities of the proposed CCEdit
framework. Demo video is available at
https://www.youtube.com/watch?v=UQw4jq-igN4.
RealFill: Reference-Driven Generation for Authentic Image Completion
Authors: name: Luming Tang, name: Nataniel Ruiz, name: Qinghao Chu, name: Yuanzhen Li, name: Aleksander Holynski, name: David E. Jacobs, name: Bharath Hariharan, name: Yael Pritch, name: Neal Wadhwa, name: Kfir Aberman, name: Michael Rubinstein
Published: 2023-09-28 17:59:29

Recent advances in generative imagery have brought forth outpainting and
inpainting models that can produce high-quality, plausible image content in
unknown regions, but the content these models hallucinate is necessarily
inauthentic, since the models lack sufficient context about the true scene. In
this work, we propose RealFill, a novel generative approach for image
completion that fills in missing regions of an image with the content that
should have been there. RealFill is a generative inpainting model that is
personalized using only a few reference images of a scene. These reference
images do not have to be aligned with the target image, and can be taken with
drastically varying viewpoints, lighting conditions, camera apertures, or image
styles. Once personalized, RealFill is able to complete a target image with
visually compelling contents that are faithful to the original scene. We
evaluate RealFill on a new image completion benchmark that covers a set of
diverse and challenging scenarios, and find that it outperforms existing
approaches by a large margin. See more results on our project page:
https://realfill.github.io
Language models in molecular discovery
Authors: name: Nikita Janakarajan, name: Tim Erdmann, name: Sarath Swaminathan, name: Teodoro Laino, name: Jannis Born
Published: 2023-09-28 08:19:54

The success of language models, especially transformer-based architectures,
has trickled into other domains giving rise to "scientific language models"
that operate on small molecules, proteins or polymers. In chemistry, language
models contribute to accelerating the molecule discovery cycle as evidenced by
promising recent findings in early-stage drug discovery. Here, we review the
role of language models in molecular discovery, underlining their strength in
de novo drug design, property prediction and reaction chemistry. We highlight
valuable open-source software assets thus lowering the entry barrier to the
field of scientific language modeling. Last, we sketch a vision for future
molecular design that combines a chatbot interface with access to computational
chemistry tools. Our contribution serves as a valuable resource for
researchers, chemists, and AI enthusiasts interested in understanding how
language models can and will be used to accelerate chemical discovery.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and
  Planning
Authors: name: Qiao Gu, name: Alihusein Kuwajerwala, name: Sacha Morin, name: Krishna Murthy Jatavallabhula, name: Bipasha Sen, name: Aditya Agarwal, name: Corban Rivera, name: William Paul, name: Kirsty Ellis, name: Rama Chellappa, name: Chuang Gan, name: Celso Miguel de Melo, name: Joshua B. Tenenbaum, name: Antonio Torralba, name: Florian Shkurti, name: Liam Paull
Published: 2023-09-28 17:53:38

For robots to perform a wide variety of tasks, they require a 3D
representation of the world that is semantically rich, yet compact and
efficient for task-driven perception and planning. Recent approaches have
attempted to leverage features from large vision-language models to encode
semantics in 3D representations. However, these approaches tend to produce maps
with per-point feature vectors, which do not scale well in larger environments,
nor do they contain semantic spatial relationships between entities in the
environment, which are useful for downstream planning. In this work, we propose
ConceptGraphs, an open-vocabulary graph-structured representation for 3D
scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
their output to 3D by multi-view association. The resulting representations
generalize to novel semantic classes, without the need to collect large 3D
datasets or finetune models. We demonstrate the utility of this representation
through a number of downstream planning tasks that are specified through
abstract (language) prompts and require complex reasoning over spatial and
semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
video: https://youtu.be/mRhNkQwRYnc )