Orionnnn/hub-stats huggingface dataset:
Meta-Personalizing Vision-Language Models to Find Named Instances in
  Video
Authors: name: Chun-Hsiao Yeh, name: Bryan Russell, name: Josef Sivic, name: Fabian Caba Heilbron, name: Simon Jenni
Published: 2023-06-16 20:12:11

Large-scale vision-language models (VLM) have shown impressive results for
language-guided search applications. While these models allow category-level
queries, they currently struggle with personalized searches for moments in a
video where a specific object instance such as ``My dog Biscuit'' appears. We
present the following three contributions to address this problem. First, we
describe a method to meta-personalize a pre-trained VLM, i.e., learning how to
learn to personalize a VLM at test time to search in video. Our method extends
the VLM's token vocabulary by learning novel word embeddings specific to each
instance. To capture only instance-specific features, we represent each
instance embedding as a combination of shared and learned global category
features. Second, we propose to learn such personalization without explicit
human supervision. Our approach automatically identifies moments of named
visual instances in video using transcripts and vision-language similarity in
the VLM's embedding space. Finally, we introduce This-Is-My, a personal video
instance retrieval benchmark. We evaluate our approach on This-Is-My and
DeepFashion2 and show that we obtain a 15% relative improvement over the state
of the art on the latter dataset.
BayLing: Bridging Cross-lingual Alignment and Instruction Following
  through Interactive Translation for Large Language Models
Authors: name: Shaolei Zhang, name: Qingkai Fang, name: Zhuocheng Zhang, name: Zhengrui Ma, name: Yan Zhou, name: Langlin Huang, name: Mengyu Bu, name: Shangtong Gui, name: Yunji Chen, name: Xilin Chen, name: Yang Feng
Published: 2023-06-19 14:30:52

Large language models (LLMs) have demonstrated remarkable prowess in language
understanding and generation. Advancing from foundation LLMs to
instructionfollowing LLMs, instruction tuning plays a vital role in aligning
LLMs to human preferences. However, the existing LLMs are usually focused on
English, leading to inferior performance in non-English languages. In order to
improve the performance for non-English languages, it is necessary to collect
language-specific training data for foundation LLMs and construct
language-specific instructions for instruction tuning, both of which are heavy
loads. To minimize human workload, we propose to transfer the capabilities of
language generation and instruction following from English to other languages
through an interactive translation task. We have developed BayLing, an
instruction-following LLM by utilizing LLaMA as the foundation LLM and
automatically constructing interactive translation instructions for instructing
tuning. Extensive assessments demonstrate that BayLing achieves comparable
performance to GPT-3.5-turbo, despite utilizing a considerably smaller
parameter size of only 13 billion. Experimental results on translation tasks
show that BayLing achieves 95% of single-turn translation capability compared
to GPT-4 with automatic evaluation and 96% of interactive translation
capability compared to GPT-3.5-turbo with human evaluation. To estimate the
performance on general tasks, we created a multi-turn instruction test set
called BayLing-80. The experimental results on BayLing-80 indicate that BayLing
achieves 89% of performance compared to GPT-3.5-turbo. BayLing also
demonstrates outstanding performance on knowledge assessment of Chinese GaoKao
and English SAT, second only to GPT-3.5-turbo among a multitude of
instruction-following LLMs. Demo, homepage, code and models of BayLing are
available.
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT
  Models
Authors: name: Boxin Wang, name: Weixin Chen, name: Hengzhi Pei, name: Chulin Xie, name: Mintong Kang, name: Chenhui Zhang, name: Chejian Xu, name: Zidi Xiong, name: Ritik Dutta, name: Rylan Schaeffer, name: Sang T. Truong, name: Simran Arora, name: Mantas Mazeika, name: Dan Hendrycks, name: Zinan Lin, name: Yu Cheng, name: Sanmi Koyejo, name: Dawn Song, name: Bo Li
Published: 2023-06-20 17:24:23

Generative Pre-trained Transformer (GPT) models have exhibited exciting
progress in capabilities, capturing the interest of practitioners and the
public alike. Yet, while the literature on the trustworthiness of GPT models
remains limited, practitioners have proposed employing capable GPT models for
sensitive applications to healthcare and finance - where mistakes can be
costly. To this end, this work proposes a comprehensive trustworthiness
evaluation for large language models with a focus on GPT-4 and GPT-3.5,
considering diverse perspectives - including toxicity, stereotype bias,
adversarial robustness, out-of-distribution robustness, robustness on
adversarial demonstrations, privacy, machine ethics, and fairness. Based on our
evaluations, we discover previously unpublished vulnerabilities to
trustworthiness threats. For instance, we find that GPT models can be easily
misled to generate toxic and biased outputs and leak private information in
both training data and conversation history. We also find that although GPT-4
is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more
vulnerable given jailbreaking system or user prompts, potentially due to the
reason that GPT-4 follows the (misleading) instructions more precisely. Our
work illustrates a comprehensive trustworthiness evaluation of GPT models and
sheds light on the trustworthiness gaps. Our benchmark is publicly available at
https://decodingtrust.github.io/.
Full Parameter Fine-tuning for Large Language Models with Limited
  Resources
Authors: name: Kai Lv, name: Yuqing Yang, name: Tengxiao Liu, name: Qinghui Gao, name: Qipeng Guo, name: Xipeng Qiu
Published: 2023-06-16 11:37:15

Large Language Models (LLMs) have revolutionized Natural Language Processing
(NLP) but demand massive GPU resources for training. Lowering the threshold for
LLMs training would encourage greater participation from researchers,
benefiting both academia and society. While existing approaches have focused on
parameter-efficient fine-tuning, which tunes or adds a small number of
parameters, few have addressed the challenge of tuning the full parameters of
LLMs with limited resources. In this work, we propose a new optimizer,
LOw-Memory Optimization (LOMO), which fuses the gradient computation and the
parameter update in one step to reduce memory usage. By integrating LOMO with
existing memory saving techniques, we reduce memory usage to 10.8% compared to
the standard approach (DeepSpeed solution). Consequently, our approach enables
the full parameter fine-tuning of a 65B model on a single machine with 8 RTX
3090, each with 24GB memory.
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image
  Editing
Authors: name: Kai Zhang, name: Lingbo Mo, name: Wenhu Chen, name: Huan Sun, name: Yu Su
Published: 2023-06-16 17:58:58

Text-guided image editing is widely needed in daily life, ranging from
personal use to professional applications such as Photoshop. However, existing
methods are either zero-shot or trained on an automatically synthesized
dataset, which contains a high volume of noise. Thus, they still require lots
of manual tuning to produce desirable outcomes in practice. To address this
issue, we introduce MagicBrush (https://osu-nlp-group.github.io/MagicBrush/),
the first large-scale, manually annotated dataset for instruction-guided real
image editing that covers diverse scenarios: single-turn, multi-turn,
mask-provided, and mask-free editing. MagicBrush comprises over 10K manually
annotated triples (source image, instruction, target image), which supports
trainining large-scale text-guided image editing models. We fine-tune
InstructPix2Pix on MagicBrush and show that the new model can produce much
better images according to human evaluation. We further conduct extensive
experiments to evaluate current image editing baselines from multiple
dimensions including quantitative, qualitative, and human evaluations. The
results reveal the challenging nature of our dataset and the gap between
current baselines and real-world editing needs.
OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for
  Object-Centric Learning
Authors: name: Yinxuan Huang, name: Tonglin Chen, name: Zhimeng Shen, name: Jinghao Huang, name: Bin Li, name: Xiangyang Xue
Published: 2023-06-16 08:26:57

Humans possess the cognitive ability to comprehend scenes in a compositional
manner. To empower AI systems with similar abilities, object-centric
representation learning aims to acquire representations of individual objects
from visual scenes without any supervision. Although recent advancements in
object-centric representation learning have achieved remarkable progress on
complex synthesis datasets, there is a huge challenge for application in
complex real-world scenes. One of the essential reasons is the scarcity of
real-world datasets specifically tailored to object-centric representation
learning methods. To solve this problem, we propose a versatile real-world
dataset of tabletop scenes for object-centric learning called OCTScenes, which
is meticulously designed to serve as a benchmark for comparing, evaluating and
analyzing object-centric representation learning methods. OCTScenes contains
5000 tabletop scenes with a total of 15 everyday objects. Each scene is
captured in 60 frames covering a 360-degree perspective. Consequently,
OCTScenes is a versatile benchmark dataset that can simultaneously satisfy the
evaluation of object-centric representation learning methods across static
scenes, dynamic scenes, and multi-view scenes tasks. Extensive experiments of
object-centric representation learning methods for static, dynamic and
multi-view scenes are conducted on OCTScenes. The results demonstrate the
shortcomings of state-of-the-art methods for learning meaningful
representations from real-world data, despite their impressive performance on
complex synthesis datasets. Furthermore, OCTScenes can serves as a catalyst for
advancing existing state-of-the-art methods, inspiring them to adapt to
real-world scenes. Dataset and code are available at
https://huggingface.co/datasets/Yinxuan/OCTScenes.
CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller
Authors: name: Yuxiang Yang, name: Guanya Shi, name: Xiangyun Meng, name: Wenhao Yu, name: Tingnan Zhang, name: Jie Tan, name: Byron Boots
Published: 2023-06-16 00:19:14

We present CAJun, a novel hierarchical learning and control framework that
enables legged robots to jump continuously with adaptive jumping distances.
CAJun consists of a high-level centroidal policy and a low-level leg
controller. In particular, we use reinforcement learning (RL) to train the
centroidal policy, which specifies the gait timing, base velocity, and swing
foot position for the leg controller. The leg controller optimizes motor
commands for the swing and stance legs according to the gait timing to track
the swing foot target and base velocity commands using optimal control.
Additionally, we reformulate the stance leg optimizer in the leg controller to
speed up policy training by an order of magnitude. Our system combines the
versatility of learning with the robustness of optimal control. By combining RL
with optimal control methods, our system achieves the versatility of learning
while enjoys the robustness from control methods, making it easily transferable
to real robots. We show that after 20 minutes of training on a single GPU,
CAJun can achieve continuous, long jumps with adaptive distances on a Go1 robot
with small sim-to-real gaps. Moreover, the robot can jump across gaps with a
maximum width of 70cm, which is over 40% wider than existing methods.
Robot Learning with Sensorimotor Pre-training
Authors: name: Ilija Radosavovic, name: Baifeng Shi, name: Letian Fu, name: Ken Goldberg, name: Trevor Darrell, name: Jitendra Malik
Published: 2023-06-16 17:58:10

We present a self-supervised sensorimotor pre-training approach for robotics.
Our model, called RPT, is a Transformer that operates on sequences of
sensorimotor tokens. Given a sequence of camera images, proprioceptive robot
states, and past actions, we encode the interleaved sequence into tokens, mask
out a random subset, and train a model to predict the masked-out content. We
hypothesize that if the robot can predict the missing content it has acquired a
good model of the physical world that can enable it to act. RPT is designed to
operate on latent visual representations which makes prediction tractable,
enables scaling to 10x larger models, and 10 Hz inference on a real robot. To
evaluate our approach, we collect a dataset of 20,000 real-world trajectories
over 9 months using a combination of motion planning and model-based grasping
algorithms. We find that pre-training on this data consistently outperforms
training from scratch, leads to 2x improvements in the block stacking task, and
has favorable scaling properties.
AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation
Authors: name: Yifei Zeng, name: Yuanxun Lu, name: Xinya Ji, name: Yao Yao, name: Hao Zhu, name: Xun Cao
Published: 2023-06-16 14:18:51

We introduce AvatarBooth, a novel method for generating high-quality 3D
avatars using text prompts or specific images. Unlike previous approaches that
can only synthesize avatars based on simple text descriptions, our method
enables the creation of personalized avatars from casually captured face or
body images, while still supporting text-based model generation and editing.
Our key contribution is the precise avatar generation control by using dual
fine-tuned diffusion models separately for the human face and body. This
enables us to capture intricate details of facial appearance, clothing, and
accessories, resulting in highly realistic avatar generations. Furthermore, we
introduce pose-consistent constraint to the optimization process to enhance the
multi-view consistency of synthesized head images from the diffusion model and
thus eliminate interference from uncontrolled human poses. In addition, we
present a multi-resolution rendering strategy that facilitates coarse-to-fine
supervision of 3D avatar generation, thereby enhancing the performance of the
proposed system. The resulting avatar model can be further edited using
additional text descriptions and driven by motion sequences. Experiments show
that AvatarBooth outperforms previous text-to-3D methods in terms of rendering
and geometric quality from either text prompts or specific images. Please check
our project website at https://zeng-yifei.github.io/avatarbooth_page/.
CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained
  Language-Vision Models
Authors: name: Hao-Wen Dong, name: Xiaoyu Liu, name: Jordi Pons, name: Gautam Bhattacharya, name: Santiago Pascual, name: Joan Serrà, name: Taylor Berg-Kirkpatrick, name: Julian McAuley
Published: 2023-06-16 05:42:01

Recent work has studied text-to-audio synthesis using large amounts of paired
text-audio data. However, audio recordings with high-quality text annotations
can be difficult to acquire. In this work, we approach text-to-audio synthesis
using unlabeled videos and pretrained language-vision models. We propose to
learn the desired text-audio correspondence by leveraging the visual modality
as a bridge. We train a conditional diffusion model to generate the audio track
of a video, given a video frame encoded by a pretrained contrastive
language-image pretraining (CLIP) model. At test time, we first explore
performing a zero-shot modality transfer and condition the diffusion model with
a CLIP-encoded text query. However, we observe a noticeable performance drop
with respect to image queries. To close this gap, we further adopt a pretrained
diffusion prior model to generate a CLIP image embedding given a CLIP text
embedding. Our results show the effectiveness of the proposed method, and that
the pretrained diffusion prior can reduce the modality transfer gap. While we
focus on text-to-audio synthesis, the proposed model can also generate audio
from image queries, and it shows competitive performance against a
state-of-the-art image-to-audio synthesis model in a subjective listening test.
This study offers a new direction of approaching text-to-audio synthesis that
leverages the naturally-occurring audio-visual correspondence in videos and the
power of pretrained language-vision models.
Scaling Open-Vocabulary Object Detection
Authors: name: Matthias Minderer, name: Alexey Gritsenko, name: Neil Houlsby
Published: 2023-06-16 08:27:46

Open-vocabulary object detection has benefited greatly from pretrained
vision-language models, but is still limited by the amount of available
detection training data. While detection training data can be expanded by using
Web image-text pairs as weak supervision, this has not been done at scales
comparable to image-level pretraining. Here, we scale up detection data with
self-training, which uses an existing detector to generate pseudo-box
annotations on image-text pairs. Major challenges in scaling self-training are
the choice of label space, pseudo-annotation filtering, and training
efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which
address these challenges. OWLv2 surpasses the performance of previous
state-of-the-art open-vocabulary detectors already at comparable training
scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples,
yielding further large improvement: With an L/14 architecture, OWL-ST improves
AP on LVIS rare classes, for which the model has seen no human box annotations,
from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale
training for open-world localization, similar to what has been seen for image
classification and language modelling.
Block-State Transformer
Authors: name: Mahan Fathi, name: Jonathan Pilault, name: Pierre-Luc Bacon, name: Christopher Pal, name: Orhan Firat, name: Ross Goroshin
Published: 2023-06-15 22:48:08

State space models (SSMs) have shown impressive results on tasks that require
modeling long-range dependencies and efficiently scale to long sequences owing
to their subquadratic runtime complexity. Originally designed for continuous
signals, SSMs have shown superior performance on a plethora of tasks, in vision
and audio; however, SSMs still lag Transformer performance in Language Modeling
tasks. In this work, we propose a hybrid layer named Block-State Transformer
(BST), that internally combines an SSM sublayer for long-range
contextualization, and a Block Transformer sublayer for short-term
representation of sequences. We study three different, and completely
parallelizable, variants that integrate SSMs and block-wise attention. We show
that our model outperforms similar Transformer-based architectures on language
modeling perplexity and generalizes to longer sequences. In addition, the
Block-State Transformer demonstrates more than tenfold increase in speed at the
layer level compared to the Block-Recurrent Transformer when model
parallelization is employed.
Inverse Scaling: When Bigger Isn't Better
Authors: name: Ian R. McKenzie, name: Alexander Lyzhov, name: Michael Pieler, name: Alicia Parrish, name: Aaron Mueller, name: Ameya Prabhu, name: Euan McLean, name: Aaron Kirtland, name: Alexis Ross, name: Alisa Liu, name: Andrew Gritsevskiy, name: Daniel Wurgaft, name: Derik Kauffman, name: Gabriel Recchia, name: Jiacheng Liu, name: Joe Cavanagh, name: Max Weiss, name: Sicong Huang, name: The Floating Droid, name: Tom Tseng, name: Tomasz Korbak, name: Xudong Shen, name: Yuhui Zhang, name: Zhengping Zhou, name: Najoung Kim, name: Samuel R. Bowman, name: Ethan Perez
Published: 2023-06-15 20:11:23

Work on scaling laws has found that large language models (LMs) show
predictable improvements to overall loss with increased scale (model size,
training data, and compute). Here, we present evidence for the claim that LMs
may show inverse scaling, or worse task performance with increased scale, e.g.,
due to flaws in the training objective and data. We present empirical evidence
of inverse scaling on 11 datasets collected by running a public contest, the
Inverse Scaling Prize, with a substantial prize pool. Through analysis of the
datasets, along with other examples found in the literature, we identify four
potential causes of inverse scaling: (i) preference to repeat memorized
sequences over following in-context instructions, (ii) imitation of undesirable
patterns in the training data, (iii) tasks containing an easy distractor task
which LMs could focus on, rather than the harder real task, and (iv) correct
but misleading few-shot demonstrations of the task. We release the winning
datasets at https://inversescaling.com/data to allow for further investigation
of inverse scaling. Our tasks have helped drive the discovery of U-shaped and
inverted-U scaling trends, where an initial trend reverses, suggesting that
scaling trends are less reliable at predicting the behavior of larger-scale
models than previously understood. Overall, our results suggest that there are
tasks for which increased model scale alone may not lead to progress, and that
more careful thought needs to go into the data and objectives for training
language models.
Explore, Establish, Exploit: Red Teaming Language Models from Scratch
Authors: name: Stephen Casper, name: Jason Lin, name: Joe Kwon, name: Gatlen Culp, name: Dylan Hadfield-Menell
Published: 2023-06-15 18:49:50

Deploying Large language models (LLMs) can pose hazards from harmful outputs
such as toxic or dishonest speech. Prior work has introduced tools that elicit
harmful outputs in order to identify and mitigate these risks. While this is a
valuable step toward securing language models, these approaches typically rely
on a pre-existing classifier for undesired outputs. This limits their
application to situations where the type of harmful behavior is known with
precision beforehand. However, this skips a central challenge of red teaming:
developing a contextual understanding of the behaviors that a model can
exhibit. Furthermore, when such a classifier already exists, red teaming has
limited marginal value because the classifier could simply be used to filter
training data or model outputs. In this work, we consider red teaming under the
assumption that the adversary is working from a high-level, abstract
specification of undesired behavior. The red team is expected to refine/extend
this specification and identify methods to elicit this behavior from the model.
Our red teaming framework consists of three steps: 1) Exploring the model's
behavior in the desired context; 2) Establishing a measurement of undesired
behavior (e.g., a classifier trained to reflect human evaluations); and 3)
Exploiting the model's flaws using this measure and an established red teaming
methodology. We apply this approach to red team GPT-2 and GPT-3 models to
systematically discover classes of prompts that elicit toxic and dishonest
statements. In doing so, we also construct and release the CommonClaim dataset
of 20,000 statements that have been labeled by human subjects as
common-knowledge-true, common-knowledge-false, or neither. Code is available at
https://github.com/thestephencasper/explore_establish_exploit_llms. CommonClaim
is available at https://github.com/thestephencasper/common_claim.