Orionnnn/hub-stats huggingface dataset:
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models
  without Specific Tuning
Authors: name: Yuwei Guo, name: Ceyuan Yang, name: Anyi Rao, name: Yaohui Wang, name: Yu Qiao, name: Dahua Lin, name: Bo Dai
Published: 2023-07-10 17:34:16

With the advance of text-to-image models (e.g., Stable Diffusion) and
corresponding personalization techniques such as DreamBooth and LoRA, everyone
can manifest their imagination into high-quality images at an affordable cost.
Subsequently, there is a great demand for image animation techniques to further
combine generated static images with motion dynamics. In this report, we
propose a practical framework to animate most of the existing personalized
text-to-image models once and for all, saving efforts in model-specific tuning.
At the core of the proposed framework is to insert a newly initialized motion
modeling module into the frozen text-to-image model and train it on video clips
to distill reasonable motion priors. Once trained, by simply injecting this
motion modeling module, all personalized versions derived from the same base
T2I readily become text-driven models that produce diverse and personalized
animated images. We conduct our evaluation on several public representative
personalized text-to-image models across anime pictures and realistic
photographs, and demonstrate that our proposed framework helps these models
generate temporally smooth animation clips while preserving the domain and
diversity of their outputs. Code and pre-trained weights will be publicly
available at https://animatediff.github.io/ .
Semantic-SAM: Segment and Recognize Anything at Any Granularity
Authors: name: Feng Li, name: Hao Zhang, name: Peize Sun, name: Xueyan Zou, name: Shilong Liu, name: Jianwei Yang, name: Chunyuan Li, name: Lei Zhang, name: Jianfeng Gao
Published: 2023-07-10 17:59:40

In this paper, we introduce Semantic-SAM, a universal image segmentation
model to enable segment and recognize anything at any desired granularity. Our
model offers two key advantages: semantic-awareness and granularity-abundance.
To achieve semantic-awareness, we consolidate multiple datasets across three
granularities and introduce decoupled classification for objects and parts.
This allows our model to capture rich semantic information. For the
multi-granularity capability, we propose a multi-choice learning scheme during
training, enabling each click to generate masks at multiple levels that
correspond to multiple ground-truth masks. Notably, this work represents the
first attempt to jointly train a model on SA-1B, generic, and part segmentation
datasets. Experimental results and visualizations demonstrate that our model
successfully achieves semantic-awareness and granularity-abundance.
Furthermore, combining SA-1B training with other segmentation tasks, such as
panoptic and part segmentation, leads to performance improvements. We will
provide code and a demo for further exploration and evaluation.
VampNet: Music Generation via Masked Acoustic Token Modeling
Authors: name: Hugo Flores Garcia, name: Prem Seetharaman, name: Rithesh Kumar, name: Bryan Pardo
Published: 2023-07-10 16:42:03

We introduce VampNet, a masked acoustic token modeling approach to music
synthesis, compression, inpainting, and variation. We use a variable masking
schedule during training which allows us to sample coherent music from the
model by applying a variety of masking approaches (called prompts) during
inference. VampNet is non-autoregressive, leveraging a bidirectional
transformer architecture that attends to all tokens in a forward pass. With
just 36 sampling passes, VampNet can generate coherent high-fidelity musical
waveforms. We show that by prompting VampNet in various ways, we can apply it
to tasks like music compression, inpainting, outpainting, continuation, and
looping with variation (vamping). Appropriately prompted, VampNet is capable of
maintaining style, genre, instrumentation, and other high-level aspects of the
music. This flexible prompting capability makes VampNet a powerful music
co-creation tool. Code and audio samples are available online.
Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation
Authors: name: Aditya Sanghi, name: Pradeep Kumar Jayaraman, name: Arianna Rampini, name: Joseph Lambourne, name: Hooman Shayani, name: Evan Atherton, name: Saeid Asgari Taghanaki
Published: 2023-07-08 00:45:01

Significant progress has recently been made in creative applications of large
pre-trained models for downstream tasks in 3D vision, such as text-to-shape
generation. This motivates our investigation of how these pre-trained models
can be used effectively to generate 3D shapes from sketches, which has largely
remained an open challenge due to the limited sketch-shape paired datasets and
the varying level of abstraction in the sketches. We discover that conditioning
a 3D generative model on the features (obtained from a frozen large pre-trained
vision model) of synthetic renderings during training enables us to effectively
generate 3D shapes from sketches at inference time. This suggests that the
large pre-trained vision model features carry semantic signals that are
resilient to domain shifts, i.e., allowing us to use only RGB renderings, but
generalizing to sketches at inference time. We conduct a comprehensive set of
experiments investigating different design factors and demonstrate the
effectiveness of our straightforward approach for generation of multiple 3D
shapes per each input sketch regardless of their level of abstraction without
requiring any paired datasets during training.
On decoder-only architecture for speech-to-text and large language model
  integration
Authors: name: Jian Wu, name: Yashesh Gaur, name: Zhuo Chen, name: Long Zhou, name: Yimeng Zhu, name: Tianrui Wang, name: Jinyu Li, name: Shujie Liu, name: Bo Ren, name: Linquan Liu, name: Yu Wu
Published: 2023-07-08 06:47:58

Large language models (LLMs) have achieved remarkable success in the field of
natural language processing, enabling better human-computer interaction using
natural language. However, the seamless integration of speech signals into LLMs
has not been explored well. The "decoder-only" architecture has also not been
well studied for speech processing tasks. In this research, we introduce
Speech-LLaMA, a novel approach that effectively incorporates acoustic
information into text-based large language models. Our method leverages
Connectionist Temporal Classification and a simple audio encoder to map the
compressed acoustic features to the continuous semantic space of the LLM. In
addition, we further probe the decoder-only architecture for speech-to-text
tasks by training a smaller scale randomly initialized speech-LLaMA model from
speech-text paired data alone. We conduct experiments on multilingual
speech-to-text translation tasks and demonstrate a significant improvement over
strong baselines, highlighting the potential advantages of decoder-only models
for speech-to-text conversion.
Large Language Models as General Pattern Machines
Authors: name: Suvir Mirchandani, name: Fei Xia, name: Pete Florence, name: Brian Ichter, name: Danny Driess, name: Montserrat Gonzalez Arenas, name: Kanishka Rao, name: Dorsa Sadigh, name: Andy Zeng
Published: 2023-07-10 17:32:13

We observe that pre-trained large language models (LLMs) are capable of
autoregressively completing complex token sequences -- from arbitrary ones
procedurally generated by probabilistic context-free grammars (PCFG), to more
rich spatial patterns found in the Abstract Reasoning Corpus (ARC), a general
AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern
completion proficiency can be partially retained even when the sequences are
expressed using tokens randomly sampled from the vocabulary. These results
suggest that without any additional training, LLMs can serve as general
sequence modelers, driven by in-context learning. In this work, we investigate
how these zero-shot capabilities may be applied to problems in robotics -- from
extrapolating sequences of numbers that represent states over time to complete
simple motions, to least-to-most prompting of reward-conditioned trajectories
that can discover and represent closed-loop policies (e.g., a stabilizing
controller for CartPole). While difficult to deploy today for real systems due
to latency, context size limitations, and compute costs, the approach of using
LLMs to drive low-level control may provide an exciting glimpse into how the
patterns among words could be transferred to actions.
RLTF: Reinforcement Learning from Unit Test Feedback
Authors: name: Jiate Liu, name: Yiqin Zhu, name: Kaiwen Xiao, name: Qiang Fu, name: Xiao Han, name: Wei Yang, name: Deheng Ye
Published: 2023-07-10 05:18:18

The goal of program synthesis, or code generation, is to generate executable
code based on given descriptions. Recently, there has been an increasing number
of studies employing reinforcement learning (RL) to improve the performance of
large language models (LLMs) for code. However, these RL methods have only used
offline frameworks, limiting their exploration of new sample spaces.
Additionally, current approaches that utilize unit test signals are rather
simple, not accounting for specific error locations within the code. To address
these issues, we proposed RLTF, i.e., Reinforcement Learning from Unit Test
Feedback, a novel online RL framework with unit test feedback of
multi-granularity for refining code LLMs. Our approach generates data in
real-time during training and simultaneously utilizes fine-grained feedback
signals to guide the model towards producing higher-quality code. Extensive
experiments show that RLTF achieves state-of-the-art performance on the APPS
and the MBPP benchmarks. Our code can be found at:
https://github.com/Zyq-scut/RLTF.
Toward Interactive Dictation
Authors: name: Belinda Z. Li, name: Jason Eisner, name: Adam Pauls, name: Sam Thomson
Published: 2023-07-08 16:30:13

Voice dictation is an increasingly important text input modality. Existing
systems that allow both dictation and editing-by-voice restrict their command
language to flat templates invoked by trigger words. In this work, we study the
feasibility of allowing users to interrupt their dictation with spoken editing
commands in open-ended natural language. We introduce a new task and dataset,
TERTiUS, to experiment with such systems. To support this flexibility in
real-time, a system must incrementally segment and classify spans of speech as
either dictation or command, and interpret the spans that are commands. We
experiment with using large pre-trained language models to predict the edited
text, or alternatively, to predict a small text-editing program. Experiments
show a natural trade-off between model accuracy and latency: a smaller model
achieves 30% end-state accuracy with 1.3 seconds of latency, while a larger
model achieves 55% end-state accuracy with 7 seconds of latency.
Large Language Models for Supply Chain Optimization
Authors: name: Beibin Li, name: Konstantina Mellou, name: Bo Zhang, name: Jeevan Pathuri, name: Ishai Menache
Published: 2023-07-08 01:42:22

Supply chain operations traditionally involve a variety of complex decision
making problems. Over the last few decades, supply chains greatly benefited
from advances in computation, which allowed the transition from manual
processing to automation and cost-effective optimization. Nonetheless, business
operators still need to spend substantial efforts in explaining and
interpreting the optimization outcomes to stakeholders. Motivated by the recent
advances in Large Language Models (LLMs), we study how this disruptive
technology can help bridge the gap between supply chain automation and human
comprehension and trust thereof. We design  -- a framework that accepts
as input queries in plain text, and outputs insights about the underlying
optimization outcomes. Our framework does not forgo the state-of-the-art
combinatorial optimization technology, but rather leverages it to
quantitatively answer what-if scenarios (e.g., how would the cost change if we
used supplier B instead of supplier A for a given demand?). Importantly, our
design does not require sending proprietary data over to LLMs, which can be a
privacy concern in some circumstances. We demonstrate the effectiveness of our
framework on a real server placement scenario within Microsoft's cloud supply
chain. Along the way, we develop a general evaluation benchmark, which can be
used to evaluate the accuracy of the LLM output in other scenarios.
SVIT: Scaling up Visual Instruction Tuning
Authors: name: Bo Zhao, name: Boya Wu, name: Tiejun Huang
Published: 2023-07-09 03:25:14

Thanks to the emerging of foundation models, the large language and vision
models are integrated to acquire the multimodal ability of visual captioning,
dialogue, question answering, etc. Although existing multimodal models present
impressive performance of visual understanding and reasoning, their limits are
still largely under-explored due to the scarcity of high-quality instruction
tuning data. To push the limits of multimodal capability, we Sale up Visual
Instruction Tuning (SVIT) by constructing a dataset of 3.2 million visual
instruction tuning data including 1.6M conversation question-answer (QA) pairs
and 1.6M complex reasoning QA pairs and 106K detailed image descriptions.
Besides the volume, the proposed dataset is also featured by the high quality
and rich diversity, which is generated by prompting GPT-4 with the abundant
manual annotations of images. We empirically verify that training multimodal
models on SVIT can significantly improve the multimodal performance in terms of
visual perception, reasoning and planing.
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Authors: name: Markus Anderljung, name: Joslyn Barnhart, name: Jade Leung, name: Anton Korinek, name: Cullen O'Keefe, name: Jess Whittlestone, name: Shahar Avin, name: Miles Brundage, name: Justin Bullock, name: Duncan Cass-Beggs, name: Ben Chang, name: Tantum Collins, name: Tim Fist, name: Gillian Hadfield, name: Alan Hayes, name: Lewis Ho, name: Sara Hooker, name: Eric Horvitz, name: Noam Kolt, name: Jonas Schuett, name: Yonadav Shavit, name: Divya Siddarth, name: Robert Trager, name: Kevin Wolf
Published: 2023-07-06 17:03:25

Advanced AI models hold the promise of tremendous benefits for humanity, but
society needs to proactively manage the accompanying risks. In this paper, we
focus on what we term "frontier AI" models: highly capable foundation models
that could possess dangerous capabilities sufficient to pose severe risks to
public safety. Frontier AI models pose a distinct regulatory challenge:
dangerous capabilities can arise unexpectedly; it is difficult to robustly
prevent a deployed model from being misused; and, it is difficult to stop a
model's capabilities from proliferating broadly. To address these challenges,
at least three building blocks for the regulation of frontier models are
needed: (1) standard-setting processes to identify appropriate requirements for
frontier AI developers, (2) registration and reporting requirements to provide
regulators with visibility into frontier AI development processes, and (3)
mechanisms to ensure compliance with safety standards for the development and
deployment of frontier AI models. Industry self-regulation is an important
first step. However, wider societal discussions and government intervention
will be needed to create standards and to ensure compliance with them. We
consider several options to this end, including granting enforcement powers to
supervisory authorities and licensure regimes for frontier AI models. Finally,
we propose an initial set of safety standards. These include conducting
pre-deployment risk assessments; external scrutiny of model behavior; using
risk assessments to inform deployment decisions; and monitoring and responding
to new information about model capabilities and uses post-deployment. We hope
this discussion contributes to the broader conversation on how to balance
public safety risks and innovation benefits from advances at the frontier of AI
development.
Decomposing the Generalization Gap in Imitation Learning for Visual
  Robotic Manipulation
Authors: name: Annie Xie, name: Lisa Lee, name: Ted Xiao, name: Chelsea Finn
Published: 2023-07-07 15:26:03

What makes generalization hard for imitation learning in visual robotic
manipulation? This question is difficult to approach at face value, but the
environment from the perspective of a robot can often be decomposed into
enumerable factors of variation, such as the lighting conditions or the
placement of the camera. Empirically, generalization to some of these factors
have presented a greater obstacle than others, but existing work sheds little
light on precisely how much each factor contributes to the generalization gap.
Towards an answer to this question, we study imitation learning policies in
simulation and on a real robot language-conditioned manipulation task to
quantify the difficulty of generalization to different (sets of) factors. We
also design a new simulated benchmark of 19 tasks with 11 factors of variation
to facilitate more controlled evaluations of generalization. From our study, we
determine an ordering of factors based on generalization difficulty, that is
consistent across simulation and our real robot setup.
One Step of Gradient Descent is Provably the Optimal In-Context Learner
  with One Layer of Linear Self-Attention
Authors: name: Arvind Mahankali, name: Tatsunori B. Hashimoto, name: Tengyu Ma
Published: 2023-07-07 13:09:18

Recent works have empirically analyzed in-context learning and shown that
transformers trained on synthetic linear regression tasks can learn to
implement ridge regression, which is the Bayes-optimal predictor, given
sufficient capacity [Aky\"urek et al., 2023], while one-layer transformers with
linear self-attention and no MLP layer will learn to implement one step of
gradient descent (GD) on a least-squares linear regression objective [von
Oswald et al., 2022]. However, the theory behind these observations remains
poorly understood. We theoretically study transformers with a single layer of
linear self-attention, trained on synthetic noisy linear regression data.
First, we mathematically show that when the covariates are drawn from a
standard Gaussian distribution, the one-layer transformer which minimizes the
pre-training loss will implement a single step of GD on the least-squares
linear regression objective. Then, we find that changing the distribution of
the covariates and weight vector to a non-isotropic Gaussian distribution has a
strong impact on the learned algorithm: the global minimizer of the
pre-training loss now implements a single step of pre-conditioned
GD. However, if only the distribution of the responses is changed, then this
does not have a large effect on the learned algorithm: even when the response
comes from a more general family of nonlinear functions, the global
minimizer of the pre-training loss still implements a single step of GD on a
least-squares linear regression objective.
Becoming self-instruct: introducing early stopping criteria for minimal
  instruct tuning
Authors: name: Waseem AlShikh, name: Manhal Daaboul, name: Kirk Goddard, name: Brock Imel, name: Kiran Kamble, name: Parikshith Kulkarni, name: Melisa Russak
Published: 2023-07-05 09:42:25

In this paper, we introduce the Instruction Following Score (IFS), a metric
that detects language models' ability to follow instructions. The metric has a
dual purpose. First, IFS can be used to distinguish between base and instruct
models. We benchmark publicly available base and instruct models, and show that
the ratio of well formatted responses to partial and full sentences can be an
effective measure between those two model classes. Secondly, the metric can be
used as an early stopping criteria for instruct tuning. We compute IFS for
Supervised Fine-Tuning (SFT) of 7B and 13B LLaMA models, showing that models
learn to follow instructions relatively early in the training process, and the
further finetuning can result in changes in the underlying base model
semantics. As an example of semantics change we show the objectivity of model
predictions, as defined by an auxiliary metric ObjecQA. We show that in this
particular case, semantic changes are the steepest when the IFS tends to
plateau. We hope that decomposing instruct tuning into IFS and semantic factors
starts a new trend in better controllable instruct tuning and opens
possibilities for designing minimal instruct interfaces querying foundation
models.
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Authors: name: Shilong Zhang, name: Peize Sun, name: Shoufa Chen, name: Min Xiao, name: Wenqi Shao, name: Wenwei Zhang, name: Kai Chen, name: Ping Luo
Published: 2023-07-07 13:43:44

Instruction tuning large language model (LLM) on image-text pairs has
achieved unprecedented vision-language multimodal abilities. However, their
vision-language alignments are only built on image-level, the lack of
region-level alignment limits their advancements to fine-grained multimodal
understanding. In this paper, we propose instruction tuning on
region-of-interest. The key design is to reformulate the bounding box as the
format of spatial instruction. The interleaved sequences of visual features
extracted by the spatial instruction and the language embedding are input to
LLM, and trained on the transformed region-text data in instruction tuning
format. Our region-level vision-language model, termed as GPT4RoI, brings brand
new conversational and interactive experience beyond image-level understanding.
(1) Controllability: Users can interact with our model by both language and
spatial instructions to flexibly adjust the detail level of the question. (2)
Capacities: Our model supports not only single-region spatial instruction but
also multi-region. This unlocks more region-level multimodal capacities such as
detailed region caption and complex region reasoning. (3) Composition: Any
off-the-shelf object detector can be a spatial instruction provider so as to
mine informative object attributes from our model, like color, shape, material,
action, relation to other objects, etc. The code, data, and demo can be found
at https://github.com/jshilong/GPT4RoI.
BiPhone: Modeling Inter Language Phonetic Influences in Text
Authors: name: Abhirut Gupta, name: Ananya B. Sai, name: Richard Sproat, name: Yuri Vasilevski, name: James S. Ren, name: Ambarish Jash, name: Sukhdeep S. Sodhi, name: Aravindan Raghuveer
Published: 2023-07-06 22:31:55

A large number of people are forced to use the Web in a language they have
low literacy in due to technology asymmetries. Written text in the second
language (L2) from such users often contains a large number of errors that are
influenced by their native language (L1). We propose a method to mine phoneme
confusions (sounds in L2 that an L1 speaker is likely to conflate) for pairs of
L1 and L2. These confusions are then plugged into a generative model (Bi-Phone)
for synthetically producing corrupted L2 text. Through human evaluations, we
show that Bi-Phone generates plausible corruptions that differ across L1s and
also have widespread coverage on the Web. We also corrupt the popular language
understanding benchmark SuperGLUE with our technique (FunGLUE for Phonetically
Noised GLUE) and show that SoTA language understating models perform poorly. We
also introduce a new phoneme prediction pre-training task which helps byte
models to recover performance close to SuperGLUE. Finally, we also release the
FunGLUE benchmark to promote further research in phonetically robust language
models. To the best of our knowledge, FunGLUE is the first benchmark to
introduce L1-L2 interactions in text.