Orionnnn/hub-stats huggingface dataset:
Ambient Adventures: Teaching ChatGPT on Developing Complex Stories
Authors: name: Zexin Chen, name: Eric Zhou, name: Kenneth Eaton, name: Xiangyu Peng, name: Mark Riedl
Published: 2023-08-03 12:52:49

Imaginative play is an area of creativity that could allow robots to engage
with the world around them in a much more personified way. Imaginary play can
be seen as taking real objects and locations and using them as imaginary
objects and locations in virtual scenarios. We adopted the story generation
capability of large language models (LLMs) to obtain the stories used for
imaginary play with human-written prompts. Those generated stories will be
simplified and mapped into action sequences that can guide the agent in
imaginary play. To evaluate whether the agent can successfully finish the
imaginary play, we also designed a text adventure game to simulate a house as
the playground for the agent to interact.
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive
  Vision-Language Models
Authors: name: Anas Awadalla, name: Irena Gao, name: Josh Gardner, name: Jack Hessel, name: Yusuf Hanafy, name: Wanrong Zhu, name: Kalyani Marathe, name: Yonatan Bitton, name: Samir Gadre, name: Shiori Sagawa, name: Jenia Jitsev, name: Simon Kornblith, name: Pang Wei Koh, name: Gabriel Ilharco, name: Mitchell Wortsman, name: Ludwig Schmidt
Published: 2023-08-02 19:10:23

We introduce OpenFlamingo, a family of autoregressive vision-language models
ranging from 3B to 9B parameters. OpenFlamingo is an ongoing effort to produce
an open-source replication of DeepMind's Flamingo models. On seven
vision-language datasets, OpenFlamingo models average between 80 - 89% of
corresponding Flamingo performance. This technical report describes our models,
training data, hyperparameters, and evaluation suite. We share our models and
code at https://github.com/mlfoundations/open_flamingo.
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like
  Models at All Scales
Authors: name: Zhewei Yao, name: Reza Yazdani Aminabadi, name: Olatunji Ruwase, name: Samyam Rajbhandari, name: Xiaoxia Wu, name: Ammar Ahmad Awan, name: Jeff Rasley, name: Minjia Zhang, name: Conglong Li, name: Connor Holmes, name: Zhongzhu Zhou, name: Michael Wyatt, name: Molly Smith, name: Lev Kurilenko, name: Heyang Qin, name: Masahiro Tanaka, name: Shuai Che, name: Shuaiwen Leon Song, name: Yuxiong He
Published: 2023-08-02 18:49:57

ChatGPT-like models have revolutionized various applications in artificial
intelligence, from summarization and coding to translation, matching or even
surpassing human performance. However, the current landscape lacks an
accessible, efficient, and cost-effective end-to-end RLHF (Reinforcement
Learning with Human Feedback) training pipeline for these powerful models,
particularly when training at the scale of billions of parameters. This paper
introduces DeepSpeed-Chat, a novel system that democratizes RLHF training,
making it accessible to the AI community. DeepSpeed-Chat offers three key
capabilities: an easy-to-use training and inference experience for ChatGPT-like
models, a DeepSpeed-RLHF pipeline that replicates the training pipeline from
InstructGPT, and a robust DeepSpeed-RLHF system that combines various
optimizations for training and inference in a unified way. The system delivers
unparalleled efficiency and scalability, enabling training of models with
hundreds of billions of parameters in record time and at a fraction of the
cost. With this development, DeepSpeed-Chat paves the way for broader access to
advanced RLHF training, even for data scientists with limited resources,
thereby fostering innovation and further development in the field of AI.
Revisiting DETR Pre-training for Object Detection
Authors: name: Yan Ma, name: Weicong Liang, name: Yiduo Hao, name: Bohan Chen, name: Xiangyu Yue, name: Chao Zhang, name: Yuhui Yuan
Published: 2023-08-02 17:39:30

Motivated by that DETR-based approaches have established new records on COCO
detection and segmentation benchmarks, many recent endeavors show increasing
interest in how to further improve DETR-based approaches by pre-training the
Transformer in a self-supervised manner while keeping the backbone frozen. Some
studies already claimed significant improvements in accuracy. In this paper, we
take a closer look at their experimental methodology and check if their
approaches are still effective on the very recent state-of-the-art such as
H-Deformable-DETR. We conduct thorough experiments on COCO object
detection tasks to study the influence of the choice of pre-training datasets,
localization, and classification target generation schemes. Unfortunately, we
find the previous representative self-supervised approach such as DETReg, fails
to boost the performance of the strong DETR-based approaches on full data
regimes. We further analyze the reasons and find that simply combining a more
accurate box predictor and Objects365 benchmark can significantly improve the
results in follow-up experiments. We demonstrate the effectiveness of our
approach by achieving strong object detection results of AP=59.3% on COCO
val set, which surpasses H-Deformable-DETR + Swin-L by +1.4%.
Last, we generate a series of synthetic pre-training datasets by combining the
very recent image-to-text captioning models (LLaVA) and text-to-image
generative models (SDXL). Notably, pre-training on these synthetic datasets
leads to notable improvements in object detection performance. Looking ahead,
we anticipate substantial advantages through the future expansion of the
synthetic pre-training dataset.
From Sparse to Soft Mixtures of Experts
Authors: name: Joan Puigcerver, name: Carlos Riquelme, name: Basil Mustafa, name: Neil Houlsby
Published: 2023-08-02 05:20:55

Sparse mixture of expert architectures (MoEs) scale model capacity without
large increases in training or inference costs. Despite their success, MoEs
suffer from a number of issues: training instability, token dropping, inability
to scale the number of experts, or ineffective finetuning. In this work, we
proposeSoft MoE, a fully-differentiable sparse Transformer that addresses these
challenges, while maintaining the benefits of MoEs. Soft MoE performs an
implicit soft assignment by passing different weighted combinations of all
input tokens to each expert. As in other MoE works, experts in Soft MoE only
process a subset of the (combined) tokens, enabling larger model capacity at
lower inference cost. In the context of visual recognition, Soft MoE greatly
outperforms standard Transformers (ViTs) and popular MoE variants (Tokens
Choice and Experts Choice). For example, Soft MoE-Base/16 requires 10.5x lower
inference cost (5.7x lower wall-clock time) than ViT-Huge/14 while matching its
performance after similar training. Soft MoE also scales well: Soft MoE Huge/14
with 128 experts in 16 MoE layers has over 40x more parameters than ViT
Huge/14, while inference time cost grows by only 2%, and it performs
substantially better.
ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based
  Image Manipulation
Authors: name: Yasheng Sun, name: Yifan Yang, name: Houwen Peng, name: Yifei Shen, name: Yuqing Yang, name: Han Hu, name: Lili Qiu, name: Hideki Koike
Published: 2023-08-02 01:57:11

While language-guided image manipulation has made remarkable progress, the
challenge of how to instruct the manipulation process faithfully reflecting
human intentions persists. An accurate and comprehensive description of a
manipulation task using natural language is laborious and sometimes even
impossible, primarily due to the inherent uncertainty and ambiguity present in
linguistic expressions. Is it feasible to accomplish image manipulation without
resorting to external cross-modal language information? If this possibility
exists, the inherent modality gap would be effortlessly eliminated. In this
paper, we propose a novel manipulation methodology, dubbed ImageBrush, that
learns visual instructions for more accurate image editing. Our key idea is to
employ a pair of transformation images as visual instructions, which not only
precisely captures human intention but also facilitates accessibility in
real-world scenarios. Capturing visual instructions is particularly challenging
because it involves extracting the underlying intentions solely from visual
demonstrations and then applying this operation to a new image. To address this
challenge, we formulate visual instruction learning as a diffusion-based
inpainting problem, where the contextual information is fully exploited through
an iterative process of generation. A visual prompting encoder is carefully
devised to enhance the model's capacity in uncovering human intent behind the
visual instructions. Extensive experiments show that our method generates
engaging manipulation results conforming to the transformations entailed in
demonstrations. Moreover, our model exhibits robust generalization capabilities
on various downstream tasks such as pose transfer, image translation and video
inpainting.
ELIXR: Towards a general purpose X-ray artificial intelligence system
  through alignment of large language models and radiology vision encoders
Authors: name: Shawn Xu, name: Lin Yang, name: Christopher Kelly, name: Marcin Sieniek, name: Timo Kohlberger, name: Martin Ma, name: Wei-Hung Weng, name: Attila Kiraly, name: Sahar Kazemzadeh, name: Zakkai Melamed, name: Jungyeon Park, name: Patricia Strachan, name: Yun Liu, name: Chuck Lau, name: Preeti Singh, name: Christina Chen, name: Mozziyar Etemadi, name: Sreenivasa Raju Kalidindi, name: Yossi Matias, name: Katherine Chou, name: Greg S. Corrado, name: Shravya Shetty, name: Daniel Tse, name: Shruthi Prabhakara, name: Daniel Golden, name: Rory Pilgrim, name: Krish Eswaran, name: Andrew Sellergren
Published: 2023-08-02 17:59:45

Our approach, which we call Embeddings for Language/Image-aligned X-Rays, or
ELIXR, leverages a language-aligned image encoder combined or grafted onto a
fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight
adapter architecture using images paired with corresponding free-text radiology
reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance
on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13
findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898
across five findings (atelectasis, cardiomegaly, consolidation, pleural
effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images)
training data), and semantic search (0.76 normalized discounted cumulative gain
(NDCG) across nineteen queries, including perfect retrieval on twelve of them).
Compared to existing data-efficient methods including supervised contrastive
learning (SupCon), ELIXR required two orders of magnitude less data to reach
similar performance. ELIXR also showed promise on CXR vision-language tasks,
demonstrating overall accuracies of 58.7% and 62.5% on visual question
answering and report quality assurance tasks, respectively. These results
suggest that ELIXR is a robust and versatile approach to CXR AI.
More Context, Less Distraction: Visual Classification by Inferring and
  Conditioning on Contextual Attributes
Authors: name: Bang An, name: Sicheng Zhu, name: Michael-Andrei Panaitescu-Liess, name: Chaithanya Kumar Mummadi, name: Furong Huang
Published: 2023-08-02 17:57:25

CLIP, as a foundational vision language model, is widely used in zero-shot
image classification due to its ability to understand various visual concepts
and natural language descriptions. However, how to fully leverage CLIP's
unprecedented human-like understanding capabilities to achieve better zero-shot
classification is still an open question. This paper draws inspiration from the
human visual perception process: a modern neuroscience view suggests that in
classifying an object, humans first infer its class-independent attributes
(e.g., background and orientation) which help separate the foreground object
from the background, and then make decisions based on this information.
Inspired by this, we observe that providing CLIP with contextual attributes
improves zero-shot classification and mitigates reliance on spurious features.
We also observe that CLIP itself can reasonably infer the attributes from an
image. With these observations, we propose a training-free, two-step zero-shot
classification method named PerceptionCLIP. Given an image, it first infers
contextual attributes (e.g., background) and then performs object
classification conditioning on them. Our experiments show that PerceptionCLIP
achieves better generalization, group robustness, and better interpretability.
For example, PerceptionCLIP with ViT-L/14 improves the worst group accuracy by
16.5% on the Waterbirds dataset and by 3.5% on CelebA.
Three Bricks to Consolidate Watermarks for Large Language Models
Authors: name: Pierre Fernandez, name: Antoine Chaffin, name: Karim Tit, name: Vivien Chappelier, name: Teddy Furon
Published: 2023-07-26 17:56:36

The task of discerning between generated and natural texts is increasingly
challenging. In this context, watermarking emerges as a promising technique for
ascribing generated text to a specific model. It alters the sampling generation
process so as to leave an invisible trace in the generated output, facilitating
later detection. This research consolidates watermarks for large language
models based on three theoretical and empirical considerations. First, we
introduce new statistical tests that offer robust theoretical guarantees which
remain valid even at low false-positive rates (less than 10^{-6}).
Second, we compare the effectiveness of watermarks using classical benchmarks
in the field of natural language processing, gaining insights into their
real-world applicability. Third, we develop advanced detection schemes for
scenarios where access to the LLM is available, as well as multi-bit
watermarking.
Tool Documentation Enables Zero-Shot Tool-Usage with Large Language
  Models
Authors: name: Cheng-Yu Hsieh, name: Si-An Chen, name: Chun-Liang Li, name: Yasuhisa Fujii, name: Alexander Ratner, name: Chen-Yu Lee, name: Ranjay Krishna, name: Tomas Pfister
Published: 2023-08-01 17:21:38

Today, large language models (LLMs) are taught to use new tools by providing
a few demonstrations of the tool's usage. Unfortunately, demonstrations are
hard to acquire, and can result in undesirable biased usage if the wrong
demonstration is chosen. Even in the rare scenario that demonstrations are
readily available, there is no principled selection protocol to determine how
many and which ones to provide. As tasks grow more complex, the selection
search grows combinatorially and invariably becomes intractable. Our work
provides an alternative to demonstrations: tool documentation. We advocate the
use of tool documentation, descriptions for the individual tool usage, over
demonstrations. We substantiate our claim through three main empirical findings
on 6 tasks across both vision and language modalities. First, on existing
benchmarks, zero-shot prompts with only tool documentation are sufficient for
eliciting proper tool usage, achieving performance on par with few-shot
prompts. Second, on a newly collected realistic tool-use dataset with hundreds
of available tool APIs, we show that tool documentation is significantly more
valuable than demonstrations, with zero-shot documentation significantly
outperforming few-shot without documentation. Third, we highlight the benefits
of tool documentations by tackling image generation and video tracking using
just-released unseen state-of-the-art models as tools. Finally, we highlight
the possibility of using tool documentation to automatically enable new
applications: by using nothing more than the documentation of GroundingDino,
Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the
just-released Grounded-SAM and Track Anything models.
Predicting masked tokens in stochastic locations improves masked image
  modeling
Authors: name: Amir Bar, name: Florian Bordes, name: Assaf Shocher, name: Mahmoud Assran, name: Pascal Vincent, name: Nicolas Ballas, name: Trevor Darrell, name: Amir Globerson, name: Yann LeCun
Published: 2023-07-31 17:59:08

Self-supervised learning is a promising paradigm in deep learning that
enables learning from unlabeled data by constructing pretext tasks that require
learning useful representations. In natural language processing, the dominant
pretext task has been masked language modeling (MLM), while in computer vision
there exists an equivalent called Masked Image Modeling (MIM). However, MIM is
challenging because it requires predicting semantic content in accurate
locations. E.g, given an incomplete picture of a dog, we can guess that there
is a tail, but we cannot determine its exact location. In this work, we propose
FlexPredict, a stochastic model that addresses this challenge by incorporating
location uncertainty into the model. Specifically, we condition the model on
stochastic masked token positions to guide the model toward learning features
that are more robust to location uncertainties. Our approach improves
downstream performance on a range of tasks, e.g, compared to MIM baselines,
FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for
semi-supervised video segmentation using ViT-L.
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning
Authors: name: Ning Miao, name: Yee Whye Teh, name: Tom Rainforth
Published: 2023-08-01 10:31:36

The recent progress in large language models (LLMs), especially the invention
of chain-of-thoughts (CoT) prompting, makes it possible to solve reasoning
problems. However, even the strongest LLMs are still struggling with more
complicated problems that require non-linear thinking and multi-step reasoning.
In this work, we explore whether LLMs have the ability to recognize their own
errors, without resorting to external resources. In particular, we investigate
whether they can be used to identify individual errors within a step-by-step
reasoning. To this end, we propose a zero-shot verification scheme to recognize
such errors. We then use this verification scheme to improve question-answering
performance, by using it to perform weighted voting on different generated
answers. We test the method on three math datasets-GSM8K, MathQA, and MATH-and
find that it successfully recognizes errors and, in turn, increases final
predictive performance.
Skills-in-Context Prompting: Unlocking Compositionality in Large
  Language Models
Authors: name: Jiaao Chen, name: Xiaoman Pan, name: Dian Yu, name: Kaiqiang Song, name: Xiaoyang Wang, name: Dong Yu, name: Jianshu Chen
Published: 2023-08-01 05:54:12

We consider the problem of eliciting compositional generalization
capabilities in large language models (LLMs) with a novel type of prompting
strategy. Compositional generalization empowers the LLMs to solve problems that
are harder than the ones they have seen (i.e., easy-to-hard generalization),
which is a critical reasoning capability of human-like intelligence. However,
even the current state-of-the-art LLMs still struggle with this form of
reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting,
which instructs LLMs how to compose basic skills to resolve more complex
problems. We find that it is crucial to demonstrate both the skills and the
compositional examples within the same prompting context. With as few as two
examplars, our SKiC prompting initiates strong synergies between skills and
their composition capabilities. Notably, it empowers LLMs to solve unseen
problems that require innovative skill compositions, achieving near-perfect
generalization on a broad range of challenging compositionality tasks.
Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling
them to leverage pre-existing internal skills acquired during earlier
pre-training stages, even when these skills are not explicitly presented in the
prompting context. This results in the capability of LLMs to solve unseen
complex problems by activating and composing internal competencies. With such
prominent features, SKiC prompting is able to achieve state-of-the-art
performance on challenging mathematical reasoning benchmarks (e.g., MATH).
Discovering Adaptable Symbolic Algorithms from Scratch
Authors: name: Stephen Kelly, name: Daniel S. Park, name: Xingyou Song, name: Mitchell McIntire, name: Pranav Nashikkar, name: Ritam Guha, name: Wolfgang Banzhaf, name: Kalyanmoy Deb, name: Vishnu Naresh Boddeti, name: Jie Tan, name: Esteban Real
Published: 2023-07-31 17:57:48

Autonomous robots deployed in the real world will need control policies that
rapidly adapt to environmental changes. To this end, we propose
AutoRobotics-Zero (ARZ), a method based on AutoML-Zero that discovers zero-shot
adaptable policies from scratch. In contrast to neural network adaption
policies, where only model parameters are optimized, ARZ can build control
algorithms with the full expressive power of a linear register machine. We
evolve modular policies that tune their model parameters and alter their
inference algorithm on-the-fly to adapt to sudden environmental changes. We
demonstrate our method on a realistic simulated quadruped robot, for which we
evolve safe control policies that avoid falling when individual limbs suddenly
break. This is a challenging task in which two popular neural network baselines
fail. Finally, we conduct a detailed analysis of our method on a novel and
challenging non-stationary control task dubbed Cataclysmic Cartpole. Results
confirm our findings that ARZ is significantly more robust to sudden
environmental changes and can build simple, interpretable control policies.
UniVTG: Towards Unified Video-Language Temporal Grounding
Authors: name: Kevin Qinghong Lin, name: Pengchuan Zhang, name: Joya Chen, name: Shraman Pramanick, name: Difei Gao, name: Alex Jinpeng Wang, name: Rui Yan, name: Mike Zheng Shou
Published: 2023-07-31 14:34:49

Video Temporal Grounding (VTG), which aims to ground target clips from videos
(such as consecutive intervals or disjoint shots) according to custom language
queries (e.g., sentences or words), is key for video browsing on social media.
Most methods in this direction develop taskspecific models that are trained
with type-specific labels, such as moment retrieval (time interval) and
highlight detection (worthiness curve), which limits their abilities to
generalize to various VTG tasks and labels. In this paper, we propose to Unify
the diverse VTG labels and tasks, dubbed UniVTG, along three directions:
Firstly, we revisit a wide range of VTG labels and tasks and define a unified
formulation. Based on this, we develop data annotation schemes to create
scalable pseudo supervision. Secondly, we develop an effective and flexible
grounding model capable of addressing each task and making full use of each
label. Lastly, thanks to the unified framework, we are able to unlock temporal
grounding pretraining from large-scale diverse labels and develop stronger
grounding abilities e.g., zero-shot grounding. Extensive experiments on three
tasks (moment retrieval, highlight detection and video summarization) across
seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights,
TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed
framework. The codes are available at https://github.com/showlab/UniVTG.