Orionnnn/hub-stats huggingface dataset:
ArrayBot: Reinforcement Learning for Generalizable Distributed
  Manipulation through Touch
Authors: name: Zhengrong Xue, name: Han Zhang, name: Jingwen Cheng, name: Zhengmao He, name: Yuanchen Ju, name: Changyi Lin, name: Gu Zhang, name: Huazhe Xu
Published: 2023-06-29 11:07:14

We present ArrayBot, a distributed manipulation system consisting of a 16
times 16 array of vertically sliding pillars integrated with tactile sensors,
which can simultaneously support, perceive, and manipulate the tabletop
objects. Towards generalizable distributed manipulation, we leverage
reinforcement learning (RL) algorithms for the automatic discovery of control
policies. In the face of the massively redundant actions, we propose to reshape
the action space by considering the spatially local action patch and the
low-frequency actions in the frequency domain. With this reshaped action space,
we train RL agents that can relocate diverse objects through tactile
observations only. Surprisingly, we find that the discovered policy can not
only generalize to unseen object shapes in the simulator but also transfer to
the physical robot without any domain randomization. Leveraging the deployed
policy, we present abundant real-world manipulation tasks, illustrating the
vast potential of RL on ArrayBot for distributed manipulation.
BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike
  Animated Motion
Authors: name: Michael J. Black, name: Priyanka Patel, name: Joachim Tesch, name: Jinlong Yang
Published: 2023-06-29 13:35:16

We show, for the first time, that neural networks trained only on synthetic
data achieve state-of-the-art accuracy on the problem of 3D human pose and
shape (HPS) estimation from real images. Previous synthetic datasets have been
small, unrealistic, or lacked realistic clothing. Achieving sufficient realism
is non-trivial and we show how to do this for full bodies in motion.
Specifically, our BEDLAM dataset contains monocular RGB videos with
ground-truth 3D bodies in SMPL-X format. It includes a diversity of body
shapes, motions, skin tones, hair, and clothing. The clothing is realistically
simulated on the moving bodies using commercial clothing physics simulation. We
render varying numbers of people in realistic scenes with varied lighting and
camera motions. We then train various HPS regressors using BEDLAM and achieve
state-of-the-art accuracy on real-image benchmarks despite training with
synthetic data. We use BEDLAM to gain insights into what model design choices
are important for accuracy. With good synthetic training data, we find that a
basic method like HMR approaches the accuracy of the current SOTA method
(CLIFF). BEDLAM is useful for a variety of tasks and all images, ground truth
bodies, 3D clothing, support code, and more are available for research
purposes. Additionally, we provide detailed information about our synthetic
data generation pipeline, enabling others to generate their own datasets. See
the project page: https://bedlam.is.tue.mpg.de/.
NeuralFuse: Learning to Improve the Accuracy of Access-Limited Neural
  Network Inference in Low-Voltage Regimes
Authors: name: Hao-Lun Sun, name: Lei Hsiung, name: Nandhini Chandramoorthy, name: Pin-Yu Chen, name: Tsung-Yi Ho
Published: 2023-06-29 11:38:22

Deep neural networks (DNNs) have become ubiquitous in machine learning, but
their energy consumption remains a notable issue. Lowering the supply voltage
is an effective strategy for reducing energy consumption. However, aggressively
scaling down the supply voltage can lead to accuracy degradation due to random
bit flips in static random access memory (SRAM) where model parameters are
stored. To address this challenge, we introduce NeuralFuse, a novel add-on
module that addresses the accuracy-energy tradeoff in low-voltage regimes by
learning input transformations to generate error-resistant data
representations. NeuralFuse protects DNN accuracy in both nominal and
low-voltage scenarios. Moreover, NeuralFuse is easy to implement and can be
readily applied to DNNs with limited access, such as non-configurable hardware
or remote access to cloud-based APIs. Experimental results demonstrate that, at
a 1% bit error rate, NeuralFuse can reduce SRAM memory access energy by up to
24% while improving accuracy by up to 57%. To the best of our knowledge, this
is the first model-agnostic approach (i.e., no model retraining) to address
low-voltage-induced bit errors. The source code is available at
https://github.com/IBM/NeuralFuse.
Dynamic-Resolution Model Learning for Object Pile Manipulation
Authors: name: Yixuan Wang, name: Yunzhu Li, name: Katherine Driggs-Campbell, name: Li Fei-Fei, name: Jiajun Wu
Published: 2023-06-29 05:51:44

Dynamics models learned from visual observations have shown to be effective
in various robotic manipulation tasks. One of the key questions for learning
such dynamics models is what scene representation to use. Prior works typically
assume representation at a fixed dimension or resolution, which may be
inefficient for simple tasks and ineffective for more complicated tasks. In
this work, we investigate how to learn dynamic and adaptive representations at
different levels of abstraction to achieve the optimal trade-off between
efficiency and effectiveness. Specifically, we construct dynamic-resolution
particle representations of the environment and learn a unified dynamics model
using graph neural networks (GNNs) that allows continuous selection of the
abstraction level. During test time, the agent can adaptively determine the
optimal resolution at each model-predictive control (MPC) step. We evaluate our
method in object pile manipulation, a task we commonly encounter in cooking,
agriculture, manufacturing, and pharmaceutical applications. Through
comprehensive evaluations both in the simulation and the real world, we show
that our method achieves significantly better performance than state-of-the-art
fixed-resolution baselines at the gathering, sorting, and redistribution of
granular object piles made with various instances like coffee beans, almonds,
corn, etc.
Generate Anything Anywhere in Any Scene
Authors: name: Yuheng Li, name: Haotian Liu, name: Yangming Wen, name: Yong Jae Lee
Published: 2023-06-29 17:55:14

Text-to-image diffusion models have attracted considerable interest due to
their wide applicability across diverse fields. However, challenges persist in
creating controllable models for personalized object generation. In this paper,
we first identify the entanglement issues in existing personalized generative
models, and then propose a straightforward and efficient data augmentation
training strategy that guides the diffusion model to focus solely on object
identity. By inserting the plug-and-play adapter layers from a pre-trained
controllable diffusion model, our model obtains the ability to control the
location and size of each generated personalized object. During inference, we
propose a regionally-guided sampling technique to maintain the quality and
fidelity of the generated images. Our method achieves comparable or superior
fidelity for personalized objects, yielding a robust, versatile, and
controllable text-to-image diffusion model that is capable of generating
realistic and personalized images. Our approach demonstrates significant
potential for various applications, such as those in art, entertainment, and
advertising design.
Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text
  Aligned Latent Representation
Authors: name: Zibo Zhao, name: Wen Liu, name: Xin Chen, name: Xianfang Zeng, name: Rui Wang, name: Pei Cheng, name: Bin Fu, name: Tao Chen, name: Gang Yu, name: Shenghua Gao
Published: 2023-06-29 17:17:57

We present a novel alignment-before-generation approach to tackle the
challenging task of generating general 3D shapes based on 2D images or texts.
Directly learning a conditional generative model from images or texts to 3D
shapes is prone to producing inconsistent results with the conditions because
3D shapes have an additional dimension whose distribution significantly differs
from that of 2D images and texts. To bridge the domain gap among the three
modalities and facilitate multi-modal-conditioned 3D shape generation, we
explore representing 3D shapes in a shape-image-text-aligned space. Our
framework comprises two models: a Shape-Image-Text-Aligned Variational
Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model
(ASLDM). The former model encodes the 3D shapes into the shape latent space
aligned to the image and text and reconstructs the fine-grained 3D neural
fields corresponding to given shape embeddings via the transformer-based
decoder. The latter model learns a probabilistic mapping function from the
image or text space to the latent shape space. Our extensive experiments
demonstrate that our proposed approach can generate higher-quality and more
diverse 3D shapes that better semantically conform to the visual or textural
conditional inputs, validating the effectiveness of the
shape-image-text-aligned space for cross-modality 3D shape generation.
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape
  Optimization
Authors: name: Minghua Liu, name: Chao Xu, name: Haian Jin, name: Linghao Chen, name: Mukund Varma T, name: Zexiang Xu, name: Hao Su
Published: 2023-06-29 13:28:16

Single image 3D reconstruction is an important but challenging task that
requires extensive knowledge of our natural world. Many existing methods solve
this problem by optimizing a neural radiance field under the guidance of 2D
diffusion models but suffer from lengthy optimization time, 3D inconsistency
results, and poor geometry. In this work, we propose a novel method that takes
a single image of any object as input and generates a full 360-degree 3D
textured mesh in a single feed-forward pass. Given a single image, we first use
a view-conditioned 2D diffusion model, Zero123, to generate multi-view images
for the input view, and then aim to lift them up to 3D space. Since traditional
reconstruction methods struggle with inconsistent multi-view predictions, we
build our 3D reconstruction module upon an SDF-based generalizable neural
surface reconstruction method and propose several critical training strategies
to enable the reconstruction of 360-degree meshes. Without costly
optimizations, our method reconstructs 3D shapes in significantly less time
than existing methods. Moreover, our method favors better geometry, generates
more 3D consistent results, and adheres more closely to the input image. We
evaluate our approach on both synthetic data and in-the-wild images and
demonstrate its superiority in terms of both mesh quality and runtime. In
addition, our approach can seamlessly support the text-to-3D task by
integrating with off-the-shelf text-to-image diffusion models.
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
Authors: name: Yunpeng Bai, name: Xintao Wang, name: Yanpei Cao, name: Yixiao Ge, name: Chun Yuan, name: Ying Shan
Published: 2023-06-29 13:33:02

This paper introduces DreamDiffusion, a novel method for generating
high-quality images directly from brain electroencephalogram (EEG) signals,
without the need to translate thoughts into text. DreamDiffusion leverages
pre-trained text-to-image models and employs temporal masked signal modeling to
pre-train the EEG encoder for effective and robust EEG representations.
Additionally, the method further leverages the CLIP image encoder to provide
extra supervision to better align EEG, text, and image embeddings with limited
EEG-image pairs. Overall, the proposed method overcomes the challenges of using
EEG signals for image generation, such as noise, limited information, and
individual differences, and achieves promising results. Quantitative and
qualitative results demonstrate the effectiveness of the proposed method as a
significant step towards portable and low-cost ``thoughts-to-image'', with
potential applications in neuroscience and computer vision.
Benchmarking Large Language Model Capabilities for Conditional
  Generation
Authors: name: Joshua Maynez, name: Priyanka Agrawal, name: Sebastian Gehrmann
Published: 2023-06-29 08:59:40

Pre-trained large language models (PLMs) underlie most new developments in
natural language processing. They have shifted the field from
application-specific model pipelines to a single model that is adapted to a
wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside
techniques like few-shot learning, have additionally shifted the output
modality to generation instead of classification or regression. Despite their
ubiquitous use, the generation quality of language models is rarely evaluated
when these models are introduced. Additionally, it is unclear how existing
generation tasks--while they can be used to compare systems at a high
level--relate to the real world use cases for which people have been adopting
them. In this work, we discuss how to adapt existing application-specific
generation benchmarks to PLMs and provide an in-depth, empirical study of the
limitations and capabilities of PLMs in natural language generation tasks along
dimensions such as scale, architecture, input and output language. Our results
show that PLMs differ in their applicability to different data regimes and
their generalization to multiple languages and inform which PLMs to use for a
given generation task setup. We share best practices to be taken into
consideration when benchmarking generation capabilities during the development
of upcoming PLMs.
An Efficient Sparse Inference Software Accelerator for Transformer-based
  Language Models on CPUs
Authors: name: Haihao Shen, name: Hengyu Meng, name: Bo Dong, name: Zhe Wang, name: Ofir Zafrir, name: Yi Ding, name: Yu Luo, name: Hanwen Chang, name: Qun Gao, name: Ziheng Wang, name: Guy Boudoukh, name: Moshe Wasserblat
Published: 2023-06-28 23:55:51

In recent years, Transformer-based language models have become the standard
approach for natural language processing tasks. However, stringent throughput
and latency requirements in industrial applications are limiting their
adoption. To mitigate the gap, model compression techniques such as structured
pruning are being used to improve inference efficiency. However, most existing
neural network inference runtimes lack adequate support for structured
sparsity. In this paper, we propose an efficient sparse deep learning inference
software stack for Transformer-based language models where the weights are
pruned with constant block size. Our sparse software accelerator leverages
Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
order of magnitude on a wide range of GEMM shapes under 5 representative
sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
widely used in industry. We apply our sparse accelerator on widely-used
Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
Services under proxy production latency constraints. We also compare our
solution with two framework-based inference solutions, ONNX Runtime and
PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
PyTorch on Xeon under the latency constraints. All the source code is publicly
available on Github: https://github.com/intel/intel-extension-for-transformers.
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text
  Documents
Authors: name: Hugo Laurençon, name: Lucile Saulnier, name: Léo Tronchon, name: Stas Bekman, name: Amanpreet Singh, name: Anton Lozhkov, name: Thomas Wang, name: Siddharth Karamcheti, name: Alexander M. Rush, name: Douwe Kiela, name: Matthieu Cord, name: Victor Sanh
Published: 2023-06-21 14:01:01

Large multimodal models trained on natural documents, which interleave images
and text, outperform models trained on image-text pairs on various multimodal
benchmarks. However, the datasets used to train these models have not been
released, and the collection process has not been fully specified. We introduce
the OBELICS dataset, an open web-scale filtered dataset of interleaved
image-text documents comprising 141 million web pages extracted from Common
Crawl, 353 million associated images, and 115 billion text tokens. We describe
the dataset creation process, present comprehensive filtering rules, and
provide an analysis of the dataset's content. To show the viability of OBELICS,
we train vision and language models of 9 and 80 billion parameters named
IDEFICS, and obtain competitive performance on different multimodal benchmarks.
We release our dataset, models and code.
Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4,
  and Human Tutors
Authors: name: Tung Phung, name: Victor-Alexandru Pădurean, name: José Cambronero, name: Sumit Gulwani, name: Tobias Kohn, name: Rupak Majumdar, name: Adish Singla, name: Gustavo Soares
Published: 2023-06-29 17:57:40

Generative AI and large language models hold great promise in enhancing
computing education by powering next-generation educational technologies for
introductory programming. Recent works have studied these models for different
scenarios relevant to programming education; however, these works are limited
for several reasons, as they typically consider already outdated models or only
specific scenario(s). Consequently, there is a lack of a systematic study that
benchmarks state-of-the-art models for a comprehensive set of programming
education scenarios. In our work, we systematically evaluate two models,
ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human
tutors for a variety of scenarios. We evaluate using five introductory Python
programming problems and real-world buggy programs from an online platform, and
assess performance using expert-based annotations. Our results show that GPT-4
drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human
tutors' performance for several scenarios. These results also highlight
settings where GPT-4 still struggles, providing exciting future directions on
developing techniques to improve the performance of these models.
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image
  Understanding
Authors: name: Yanzhe Zhang, name: Ruiyi Zhang, name: Jiuxiang Gu, name: Yufan Zhou, name: Nedim Lipka, name: Diyi Yang, name: Tong Sun
Published: 2023-06-29 17:08:16

Instruction tuning unlocks the superior capability of Large Language Models
(LLM) to interact with humans. Furthermore, recent instruction-following
datasets include images as visual inputs, collecting responses for image-based
instructions. However, visual instruction-tuned models cannot comprehend
textual details within images well. This work enhances the current visual
instruction tuning pipeline with text-rich images (e.g., movie posters, book
covers, etc.). Specifically, we first use publicly available OCR tools to
collect results on 422K text-rich images from the LAION dataset. Moreover, we
prompt text-only GPT-4 with recognized texts and image captions to generate 16K
conversations, each containing question-answer pairs for text-rich images. By
combining our collected data with previous multi-modal instruction-following
data, our model, LLaVAR, substantially improves the LLaVA model's capability on
text-based VQA datasets (up to 20% accuracy improvement) while achieving an
accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
evaluation also demonstrates the improvement of our model on both natural
images and text-rich images. Through qualitative analysis, LLaVAR shows
promising interaction (e.g., reasoning, writing, and elaboration) skills with
humans based on the latest real-world online content that combines text and
images. We make our code/data/models publicly available at
https://llavar.github.io/.
Automatic Calibration and Error Correction for Large Language Models via
  Pareto Optimal Self-Supervision
Authors: name: Theodore Zhao, name: Mu Wei, name: J. Samuel Preston, name: Hoifung Poon
Published: 2023-06-28 21:11:15

Large language models (LLMs) have demonstrated remarkable capabilities out of
box for a wide range of applications, yet accuracy still remains a major growth
area, especially in mission-critical domains such as biomedicine. An effective
method to calibrate the confidence level on LLM responses is essential to
automatically detect errors and facilitate human-in-the-loop verification. An
important source of calibration signals stems from expert-stipulated
programmatic supervision, which is often available at low cost but has its own
limitations such as noise and coverage. In this paper, we introduce a Pareto
optimal self-supervision framework that can leverage available programmatic
supervision to systematically calibrate LLM responses by producing a risk score
for every response, without any additional manual efforts. This is accomplished
by learning a harmonizer model to align LLM output with other available
supervision sources, which would assign higher risk scores to more uncertain
LLM responses and facilitate error correction. Experiments on standard relation
extraction tasks in biomedical and general domains demonstrate the promise of
this approach, with our proposed risk scores highly correlated with the real
error rate of LLMs. For the most uncertain test instances, dynamic prompting
based on our proposed risk scores results in significant accuracy improvement
for off-the-shelf LLMs, boosting GPT-3 results past state-of-the-art (SOTA)
weak supervision and GPT-4 results past SOTA supervised results on challenging
evaluation datasets.
REFLECT: Summarizing Robot Experiences for Failure Explanation and
  Correction
Authors: name: Zeyi Liu, name: Arpit Bahety, name: Shuran Song
Published: 2023-06-27 18:03:15

The ability to detect and analyze failed executions automatically is crucial
for an explainable and robust robotic system. Recently, Large Language Models
(LLMs) have demonstrated strong common sense reasoning skills on textual
inputs. To leverage the power of LLM for robot failure explanation, we propose
a framework REFLECT, which converts multi-sensory data into a hierarchical
summary of robot past experiences and queries LLM with a progressive failure
explanation algorithm. Conditioned on the explanation, a failure correction
planner generates an executable plan for the robot to correct the failure and
complete the task. To systematically evaluate the framework, we create the
RoboFail dataset and show that our LLM-based framework is able to generate
informative failure explanations that assist successful correction planning.
Project website: https://roboreflect.github.io/
Accelerating Transducers through Adjacent Token Merging
Authors: name: Yuang Li, name: Yu Wu, name: Jinyu Li, name: Shujie Liu
Published: 2023-06-28 08:33:13

Recent end-to-end automatic speech recognition (ASR) systems often utilize a
Transformer-based acoustic encoder that generates embedding at a high frame
rate. However, this design is inefficient, particularly for long speech signals
due to the quadratic computation of self-attention. To address this, we propose
a new method, Adjacent Token Merging (A-ToMe), which gradually combines
adjacent tokens with high similarity scores between their key values. In this
way, the total time step could be reduced, and the inference of both the
encoder and joint network is accelerated. Experiments on LibriSpeech show that
our method can reduce 57% of tokens and improve the inference speed on GPU by
70% without any notable loss of accuracy. Additionally, we demonstrate that
A-ToMe is also an effective solution to reduce tokens in long-form ASR, where
the input speech consists of multiple utterances.