Orionnnn/hub-stats huggingface dataset:
Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
Authors: name: Chaoning Zhang, name: Dongshen Han, name: Yu Qiao, name: Jung Uk Kim, name: Sung-Ho Bae, name: Seungkyu Lee, name: Choong Seon Hong
Published: 2023-06-25 16:37:25

Segment anything model (SAM) is a prompt-guided vision foundation model for
cutting out the object of interest from its background. Since Meta research
team released the SA project, SAM has attracted significant attention due to
its impressive zero-shot transfer performance and high versatility of being
compatible with other models for advanced vision applications like image
editing with fine-grained control. Many of such use cases need to be run on
resource-constraint edge devices, like mobile Apps. In this work, we aim to
make SAM mobile-friendly by replacing the heavyweight image encoder with a
lightweight one. A naive way to train such a new SAM as in the original SAM
paper leads to unsatisfactory performance, especially when limited training
sources are available. We find that this is mainly caused by the coupled
optimization of the image encoder and mask decoder, motivated by which we
propose decoupled distillation. Concretely, we distill the knowledge from the
image encoder ViT-H in the original SAM to a lightweight image encoder, which
can be automatically compatible with the mask decoder in the original SAM. The
training can be completed on a single GPU within less than one day, and the
resulting lightweight SAM is termed MobileSAM which is more than 60 times
smaller yet performs on par with the original SAM. For inference speed,
MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the
mask decoder. With superior performance and a higher versatility, our MobileSAM
is 7 times smaller and 4 times faster than the concurrent FastSAM, making it
more suitable for mobile applications. The code for MobileSAM project is
provided at https://github.com/ChaoningZhang/MobileSAM
RVT: Robotic View Transformer for 3D Object Manipulation
Authors: name: Ankit Goyal, name: Jie Xu, name: Yijie Guo, name: Valts Blukis, name: Yu-Wei Chao, name: Dieter Fox
Published: 2023-06-26 17:59:31

For 3D object manipulation, methods that build an explicit 3D representation
perform better than those relying only on camera images. But using explicit 3D
representations like voxels comes at large computing cost, adversely affecting
scalability. In this work, we propose RVT, a multi-view transformer for 3D
manipulation that is both scalable and accurate. Some key features of RVT are
an attention mechanism to aggregate information across views and re-rendering
of the camera input from virtual views around the robot workspace. In
simulations, we find that a single RVT model works well across 18 RLBench tasks
with 249 task variations, achieving 26% higher relative success than the
existing state-of-the-art method (PerAct). It also trains 36X faster than
PerAct for achieving the same performance and achieves 2.3X the inference speed
of PerAct. Further, RVT can perform a variety of manipulation tasks in the real
world with just a few (sim10) demonstrations per task. Visual results, code,
and trained model are provided at https://robotic-view-transformer.github.io/.
Supervised Pretraining Can Learn In-Context Reinforcement Learning
Authors: name: Jonathan N. Lee, name: Annie Xie, name: Aldo Pacchiano, name: Yash Chandak, name: Chelsea Finn, name: Ofir Nachum, name: Emma Brunskill
Published: 2023-06-26 17:58:50

Large transformer models trained on diverse datasets have shown a remarkable
ability to learn in-context, achieving high few-shot performance on tasks they
were not explicitly trained to solve. In this paper, we study the in-context
learning capabilities of transformers in decision-making problems, i.e.,
reinforcement learning (RL) for bandits and Markov decision processes. To do
so, we introduce and study Decision-Pretrained Transformer (DPT), a supervised
pretraining method where the transformer predicts an optimal action given a
query state and an in-context dataset of interactions, across a diverse set of
tasks. This procedure, while simple, produces a model with several surprising
capabilities. We find that the pretrained transformer can be used to solve a
range of RL problems in-context, exhibiting both exploration online and
conservatism offline, despite not being explicitly trained to do so. The model
also generalizes beyond the pretraining distribution to new tasks and
automatically adapts its decision-making strategies to unknown structure.
Theoretically, we show DPT can be viewed as an efficient implementation of
Bayesian posterior sampling, a provably sample-efficient RL algorithm. We
further leverage this connection to provide guarantees on the regret of the
in-context algorithm yielded by DPT, and prove that it can learn faster than
algorithms used to generate the pretraining data. These results suggest a
promising yet simple path towards instilling strong in-context decision-making
abilities in transformers.
Swin-Free: Achieving Better Cross-Window Attention and Efficiency with
  Size-varying Window
Authors: name: Jinkyu Koo, name: John Yang, name: Le An, name: Gwenaelle Cunha Sergio, name: Su Inn Park
Published: 2023-06-23 20:19:58

Transformer models have shown great potential in computer vision, following
their success in language tasks. Swin Transformer is one of them that
outperforms convolution-based architectures in terms of accuracy, while
improving efficiency when compared to Vision Transformer (ViT) and its
variants, which have quadratic complexity with respect to the input size. Swin
Transformer features shifting windows that allows cross-window connection while
limiting self-attention computation to non-overlapping local windows. However,
shifting windows introduces memory copy operations, which account for a
significant portion of its runtime. To mitigate this issue, we propose
Swin-Free in which we apply size-varying windows across stages, instead of
shifting windows, to achieve cross-connection among local windows. With this
simple design change, Swin-Free runs faster than the Swin Transformer at
inference with better accuracy. Furthermore, we also propose a few of Swin-Free
variants that are faster than their Swin Transformer counterparts.
Restart Sampling for Improving Generative Processes
Authors: name: Yilun Xu, name: Mingyang Deng, name: Xiang Cheng, name: Yonglong Tian, name: Ziming Liu, name: Tommi Jaakkola
Published: 2023-06-26 17:48:25

Generative processes that involve solving differential equations, such as
diffusion models, frequently necessitate balancing speed and quality. ODE-based
samplers are fast but plateau in performance while SDE-based samplers deliver
higher sample quality at the cost of increased sampling time. We attribute this
difference to sampling errors: ODE-samplers involve smaller discretization
errors while stochasticity in SDE contracts accumulated errors. Based on these
findings, we propose a novel sampling algorithm called Restart in order to
better balance discretization errors and contraction. The sampling method
alternates between adding substantial noise in additional forward steps and
strictly following a backward ODE. Empirically, Restart sampler surpasses
previous SDE and ODE samplers in both speed and accuracy. Restart not only
outperforms the previous best SDE results, but also accelerates the sampling
speed by 10-fold / 2-fold on CIFAR-10 / ImageNet 64 times 64. In addition,
it attains significantly better sample quality than ODE samplers within
comparable sampling times. Moreover, Restart better balances text-image
alignment/visual quality versus diversity than previous samplers in the
large-scale text-to-image Stable Diffusion model pre-trained on LAION 512
times 512. Code is available at
https://github.com/Newbeeer/diffusion_restart_sampling
ViNT: A Foundation Model for Visual Navigation
Authors: name: Dhruv Shah, name: Ajay Sridhar, name: Nitish Dashora, name: Kyle Stachowicz, name: Kevin Black, name: Noriaki Hirose, name: Sergey Levine
Published: 2023-06-26 16:57:03

General-purpose pre-trained models ("foundation models") have enabled
practitioners to produce generalizable solutions for individual machine
learning problems with datasets that are significantly smaller than those
required for learning from scratch. Such models are typically trained on large
and diverse datasets with weak supervision, consuming much more training data
than is available for any individual downstream application. In this paper, we
describe the Visual Navigation Transformer (ViNT), a foundation model that aims
to bring the success of general-purpose pre-trained models to vision-based
robotic navigation. ViNT is trained with a general goal-reaching objective that
can be used with any navigation dataset, and employs a flexible
Transformer-based architecture to learn navigational affordances and enable
efficient adaptation to a variety of downstream navigational tasks. ViNT is
trained on a number of existing navigation datasets, comprising hundreds of
hours of robotic navigation from a variety of different robotic platforms, and
exhibits positive transfer, outperforming specialist models trained on singular
datasets. ViNT can be augmented with diffusion-based subgoal proposals to
explore novel environments, and can solve kilometer-scale navigation problems
when equipped with long-range heuristics. ViNT can also be adapted to novel
task specifications with a technique inspired by prompt-tuning, where the goal
encoder is replaced by an encoding of another task modality (e.g., GPS
waypoints or routing commands) embedded into the same space of goal tokens.
This flexibility and ability to accommodate a variety of downstream problem
domains establishes ViNT as an effective foundation model for mobile robotics.
For videos, code, and model checkpoints, see our project page at
https://visualnav-transformer.github.io.
DragDiffusion: Harnessing Diffusion Models for Interactive Point-based
  Image Editing
Authors: name: Yujun Shi, name: Chuhui Xue, name: Jiachun Pan, name: Wenqing Zhang, name: Vincent Y. F. Tan, name: Song Bai
Published: 2023-06-26 06:04:09

Precise and controllable image editing is a challenging task that has
attracted significant attention. Recently, DragGAN enables an interactive
point-based image editing framework and achieves impressive editing results
with pixel-level precision. However, since this method is based on generative
adversarial networks (GAN), its generality is upper-bounded by the capacity of
the pre-trained GAN models. In this work, we extend such an editing framework
to diffusion models and propose DragDiffusion. By leveraging large-scale
pretrained diffusion models, we greatly improve the applicability of
interactive point-based editing in real world scenarios. While most existing
diffusion-based image editing methods work on text embeddings, DragDiffusion
optimizes the diffusion latent to achieve precise spatial control. Although
diffusion models generate images in an iterative manner, we empirically show
that optimizing diffusion latent at one single step suffices to generate
coherent results, enabling DragDiffusion to complete high-quality editing
efficiently. Extensive experiments across a wide range of challenging cases
(e.g., multi-objects, diverse object categories, various styles, etc.)
demonstrate the versatility and generality of DragDiffusion.
SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models
Authors: name: Lizao Li, name: Rob Carver, name: Ignacio Lopez-Gomez, name: Fei Sha, name: John Anderson
Published: 2023-06-24 22:00:06

Probabilistic forecasting is crucial to decision-making under uncertainty
about future weather. The dominant approach is to use an ensemble of forecasts
to represent and quantify uncertainty in operational numerical weather
prediction. However, generating ensembles is computationally costly. In this
paper, we propose to generate ensemble forecasts at scale by leveraging recent
advances in generative artificial intelligence. Our approach learns a
data-driven probabilistic diffusion model from the 5-member ensemble GEFS
reforecast dataset. The model can then be sampled efficiently to produce
realistic weather forecasts, conditioned on a few members of the operational
GEFS forecasting system. The generated ensembles have similar predictive skill
as the full GEFS 31-member ensemble, evaluated against ERA5 reanalysis, and
emulate well the statistics of large physics-based ensembles. We also apply the
same methodology to developing a diffusion model for generative
post-processing: the model directly learns to correct biases present in the
emulated forecasting system by leveraging reanalysis data as labels during
training. Ensembles from this generative post-processing model show greater
reliability and accuracy, particularly in extreme event classification. In
general, they are more reliable and forecast the probability of extreme weather
more accurately than the GEFS operational ensemble. Our models achieve these
results at less than 1/10th of the computational cost incurred by the
operational GEFS system.
DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image
  Generation using Limited Data
Authors: name: Jingyuan Zhu, name: Huimin Ma, name: Jiansheng Chen, name: Jian Yuan
Published: 2023-06-25 07:40:39

Denoising diffusion probabilistic models (DDPMs) have been proven capable of
synthesizing high-quality images with remarkable diversity when trained on
large amounts of data. Typical diffusion models and modern large-scale
conditional generative models like text-to-image generative models are
vulnerable to overfitting when fine-tuned on extremely limited data. Existing
works have explored subject-driven generation using a reference set containing
a few images. However, few prior works explore DDPM-based domain-driven
generation, which aims to learn the common features of target domains while
maintaining diversity. This paper proposes a novel DomainStudio approach to
adapt DDPMs pre-trained on large-scale source datasets to target domains using
limited data. It is designed to keep the diversity of subjects provided by
source domains and get high-quality and diverse adapted samples in target
domains. We propose to keep the relative distances between adapted samples to
achieve considerable generation diversity. In addition, we further enhance the
learning of high-frequency details for better generation quality. Our approach
is compatible with both unconditional and conditional diffusion models. This
work makes the first attempt to realize unconditional few-shot image generation
with diffusion models, achieving better quality and greater diversity than
current state-of-the-art GAN-based approaches. Moreover, this work also
significantly relieves overfitting for conditional generation and realizes
high-quality domain-driven generation, further expanding the applicable
scenarios of modern large-scale text-to-image models.
Zero-shot spatial layout conditioning for text-to-image diffusion models
Authors: name: Guillaume Couairon, name: Marlène Careil, name: Matthieu Cord, name: Stéphane Lathuilière, name: Jakob Verbeek
Published: 2023-06-23 19:24:48

Large-scale text-to-image diffusion models have significantly improved the
state of the art in generative image modelling and allow for an intuitive and
powerful user interface to drive the image generation process. Expressing
spatial constraints, e.g. to position specific objects in particular locations,
is cumbersome using text; and current text-based image generation models are
not able to accurately follow such instructions. In this paper we consider
image generation from text associated with segments on the image canvas, which
combines an intuitive natural language interface with precise spatial control
over the generated content. We propose ZestGuide, a zero-shot segmentation
guidance approach that can be plugged into pre-trained text-to-image diffusion
models, and does not require any additional training. It leverages implicit
segmentation maps that can be extracted from cross-attention layers, and uses
them to align the generation with input masks. Our experimental results combine
high image quality with accurate alignment of generated content with input
segmentations, and improve over prior work both quantitatively and
qualitatively, including methods that require training on images with
corresponding segmentations. Compared to Paint with Words, the previous
state-of-the art in image generation with zero-shot segmentation conditioning,
we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.
Kosmos-2: Grounding Multimodal Large Language Models to the World
Authors: name: Zhiliang Peng, name: Wenhui Wang, name: Li Dong, name: Yaru Hao, name: Shaohan Huang, name: Shuming Ma, name: Furu Wei
Published: 2023-06-26 16:32:47

We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
capabilities of perceiving object descriptions (e.g., bounding boxes) and
grounding text to the visual world. Specifically, we represent refer
expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
object descriptions are sequences of location tokens. Together with multimodal
corpora, we construct large-scale data of grounded image-text pairs (called
GrIT) to train the model. In addition to the existing capabilities of MLLMs
(e.g., perceiving general modalities, following instructions, and performing
in-context learning), Kosmos-2 integrates the grounding capability into
downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
including (i) multimodal grounding, such as referring expression comprehension,
and phrase grounding, (ii) multimodal referring, such as referring expression
generation, (iii) perception-language tasks, and (iv) language understanding
and generation. This work lays out the foundation for the development of
Embodiment AI and sheds light on the big convergence of language, multimodal
perception, action, and world modeling, which is a key step toward artificial
general intelligence. Data, demo, and pretrained models are available at
https://aka.ms/kosmos-2.
MotionGPT: Human Motion as a Foreign Language
Authors: name: Biao Jiang, name: Xin Chen, name: Wen Liu, name: Jingyi Yu, name: Gang Yu, name: Tao Chen
Published: 2023-06-26 15:53:02

Though the advancement of pre-trained large language models unfolds, the
exploration of building a unified model for language and other multi-modal
data, such as motion, remains challenging and untouched so far. Fortunately,
human motion displays a semantic coupling akin to human language, often
perceived as a form of body language. By fusing language data with large-scale
motion models, motion-language pre-training that can enhance the performance of
motion-related tasks becomes feasible. Driven by this insight, we propose
MotionGPT, a unified, versatile, and user-friendly motion-language model to
handle multiple motion-relevant tasks. Specifically, we employ the discrete
vector quantization for human motion and transfer 3D motion into motion tokens,
similar to the generation process of word tokens. Building upon this "motion
vocabulary", we perform language modeling on both motion and text in a unified
manner, treating human motion as a specific language. Moreover, inspired by
prompt learning, we pre-train MotionGPT with a mixture of motion-language data
and fine-tune it on prompt-based question-and-answer tasks. Extensive
experiments demonstrate that MotionGPT achieves state-of-the-art performances
on multiple motion tasks including text-driven motion generation, motion
captioning, motion prediction, and motion in-between.
Language models are weak learners
Authors: name: Hariharan Manikandan, name: Yiding Jiang, name: J Zico Kolter
Published: 2023-06-25 02:39:19

A central notion in practical and theoretical machine learning is that of a
weak learner, classifiers that achieve better-than-random
performance (on any given distribution over data), even by a small margin. Such
weak learners form the practical basis for canonical machine learning methods
such as boosting. In this work, we illustrate that prompt-based large language
models can operate effectively as said weak learners. Specifically, we
illustrate the use of a large language model (LLM) as a weak learner in a
boosting algorithm applied to tabular data. We show that by providing (properly
sampled according to the distribution of interest) text descriptions of tabular
data samples, LLMs can produce a summary of the samples that serves as a
template for classification and achieves the aim of acting as a weak learner on
this task. We incorporate these models into a boosting approach, which in some
settings can leverage the knowledge within the LLM to outperform traditional
tree-based boosting. The model outperforms both few-shot learning and
occasionally even more involved fine-tuning procedures, particularly for tasks
involving small numbers of data points. The results illustrate the potential
for prompt-based LLMs to function not just as few-shot learners themselves, but
as components of larger machine learning pipelines.
H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large
  Language Models
Authors: name: Zhenyu Zhang, name: Ying Sheng, name: Tianyi Zhou, name: Tianlong Chen, name: Lianmin Zheng, name: Ruisi Cai, name: Zhao Song, name: Yuandong Tian, name: Christopher Ré, name: Clark Barrett, name: Zhangyang Wang, name: Beidi Chen
Published: 2023-06-24 20:11:14

Large Language Models (LLMs), despite their recent impressive
accomplishments, are notably cost-prohibitive to deploy, particularly for
applications involving long-content generation, such as dialogue systems and
story writing. Often, a large amount of transient state information, referred
to as the KV cache, is stored in GPU memory in addition to model parameters,
scaling linearly with the sequence length and batch size. In this paper, we
introduce a novel approach for implementing the KV cache which significantly
reduces its memory footprint. Our approach is based on the noteworthy
observation that a small portion of tokens contributes most of the value when
computing attention scores. We call these tokens Heavy Hitters (H_2). Through
a comprehensive investigation, we find that (i) the emergence of H_2 is
natural and strongly correlates with the frequent co-occurrence of tokens in
the text, and (ii) removing them results in significant performance
degradation. Based on these insights, we propose Heavy Hitter Oracle (H_2O),
a KV cache eviction policy that dynamically retains a balance of recent and
H_2 tokens. We formulate the KV cache eviction as a dynamic submodular
problem and prove (under mild assumptions) a theoretical guarantee for our
novel eviction algorithm which could help guide future work. We validate the
accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of
tasks. Our implementation of H_2O with 20% heavy hitters improves the
throughput over three leading inference systems DeepSpeed Zero-Inference,
Hugging Face Accelerate, and FlexGen by up to 29times, 29times, and
3times on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the
latency by up to 1.9times. The code is available at
https://github.com/FMInference/H2O.