Orionnnn/hub-stats huggingface dataset:
GPT Models Meet Robotic Applications: Co-Speech Gesturing Chat System
Authors: name: Naoki Wake, name: Atsushi Kanehira, name: Kazuhiro Sasabuchi, name: Jun Takamatsu, name: Katsushi Ikeuchi
Published: 2023-05-10 10:14:16

This technical paper introduces a chatting robot system that utilizes recent
advancements in large-scale language models (LLMs) such as GPT-3 and ChatGPT.
The system is integrated with a co-speech gesture generation system, which
selects appropriate gestures based on the conceptual meaning of speech. Our
motivation is to explore ways of utilizing the recent progress in LLMs for
practical robotic applications, which benefits the development of both chatbots
and LLMs. Specifically, it enables the development of highly responsive chatbot
systems by leveraging LLMs and adds visual effects to the user interface of
LLMs as an additional value. The source code for the system is available on
GitHub for our in-house robot
(https://github.com/microsoft/LabanotationSuite/tree/master/MSRAbotChatSimulation)
and GitHub for Toyota HSR
(https://github.com/microsoft/GPT-Enabled-HSR-CoSpeechGestures).
Binary and Ternary Natural Language Generation
Authors: name: Zechun Liu, name: Barlas Oguz, name: Aasish Pappu, name: Yangyang Shi, name: Raghuraman Krishnamoorthi
Published: 2023-06-02 18:01:02

Ternary and binary neural networks enable multiplication-free computation and
promise multiple orders of magnitude efficiency gains over full-precision
networks if implemented on specialized hardware. However, since both the
parameter and the output space are highly discretized, such networks have
proven very difficult to optimize. The difficulties are compounded for the
class of transformer text generation models due to the sensitivity of the
attention operation to quantization and the noise-compounding effects of
autoregressive decoding in the high-cardinality output space. We approach the
problem with a mix of statistics-based quantization for the weights and elastic
quantization of the activations and demonstrate the first ternary and binary
transformer models on the downstream tasks of summarization and machine
translation. Our ternary BART base achieves an R1 score of 41 on the
CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while
being 16x more efficient. Our binary model, while less accurate, achieves a
highly non-trivial score of 35.6. For machine translation, we achieved BLEU
scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full
precision mBART model score of 26.8. We also compare our approach in the 8-bit
activation setting, where our ternary and even binary weight models can match
or outperform the best existing 8-bit weight models in the literature. Our code
and models are available at:
https://github.com/facebookresearch/Ternary_Binary_Transformer
MotionDiffuser: Controllable Multi-Agent Motion Prediction using
  Diffusion
Authors: name: Chiyu Max Jiang, name: Andre Cornman, name: Cheolho Park, name: Ben Sapp, name: Yin Zhou, name: Dragomir Anguelov
Published: 2023-06-05 17:55:52

We present MotionDiffuser, a diffusion based representation for the joint
distribution of future trajectories over multiple agents. Such representation
has several key advantages: first, our model learns a highly multimodal
distribution that captures diverse future outcomes. Second, the simple
predictor design requires only a single L2 loss training objective, and does
not depend on trajectory anchors. Third, our model is capable of learning the
joint distribution for the motion of multiple agents in a permutation-invariant
manner. Furthermore, we utilize a compressed trajectory representation via PCA,
which improves model performance and allows for efficient computation of the
exact sample log probability. Subsequently, we propose a general constrained
sampling framework that enables controlled trajectory sampling based on
differentiable cost functions. This strategy enables a host of applications
such as enforcing rules and physical priors, or creating tailored simulation
scenarios. MotionDiffuser can be combined with existing backbone architectures
to achieve top motion forecasting results. We obtain state-of-the-art results
for multi-agent motion prediction on the Waymo Open Motion Dataset.
PLANNER: Generating Diversified Paragraph via Latent Language Diffusion
  Model
Authors: name: Yizhe Zhang, name: Jiatao Gu, name: Zhuofeng Wu, name: Shuangfei Zhai, name: Josh Susskind, name: Navdeep Jaitly
Published: 2023-06-05 01:36:39

Autoregressive models for text sometimes generate repetitive and low-quality
output because errors accumulate during the steps of generation. This issue is
often attributed to exposure bias - the difference between how a model is
trained, and how it is used during inference. Denoising diffusion models
provide an alternative approach in which a model can revisit and revise its
output. However, they can be computationally expensive and prior efforts on
text have led to models that produce less fluent output compared to
autoregressive models, especially for longer text and paragraphs. In this
paper, we propose PLANNER, a model that combines latent semantic diffusion with
autoregressive generation, to generate fluent text while exercising global
control over paragraphs. The model achieves this by combining an autoregressive
"decoding" module with a "planning" module that uses latent diffusion to
generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed
method is evaluated on various conditional generation tasks, and results on
semantic generation, text completion and summarization show its effectiveness
in generating high-quality long-form text in an efficient manner.
HeadSculpt: Crafting 3D Head Avatars with Text
Authors: name: Xiao Han, name: Yukang Cao, name: Kai Han, name: Xiatian Zhu, name: Jiankang Deng, name: Yi-Zhe Song, name: Tao Xiang, name: Kwan-Yee K. Wong
Published: 2023-06-05 16:53:58

Recently, text-guided 3D generative methods have made remarkable advancements
in producing high-quality textures and geometry, capitalizing on the
proliferation of large vision-language and image diffusion models. However,
existing methods still struggle to create high-fidelity 3D head avatars in two
aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model
whilst missing the necessary 3D awareness and head priors. This makes them
prone to inconsistency and geometric distortions in the generated avatars. (2)
They fall short in fine-grained editing. This is primarily due to the inherited
limitations from the pre-trained 2D image diffusion models, which become more
pronounced when it comes to 3D head avatars. In this work, we address these
challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt
for crafting (i.e., generating and editing) 3D head avatars from textual
prompts. Specifically, we first equip the diffusion model with 3D awareness by
leveraging landmark-based control and a learned textual embedding representing
the back view appearance of heads, enabling 3D-consistent head avatar
generations. We further propose a novel identity-aware editing score
distillation strategy to optimize a textured mesh with a high-resolution
differentiable rendering technique. This enables identity preservation while
following the editing instruction. We showcase HeadSculpt's superior fidelity
and editing capabilities through comprehensive experiments and comparisons with
existing methods.
The Surprising Effectiveness of Diffusion Models for Optical Flow and
  Monocular Depth Estimation
Authors: name: Saurabh Saxena, name: Charles Herrmann, name: Junhwa Hur, name: Abhishek Kar, name: Mohammad Norouzi, name: Deqing Sun, name: David J. Fleet
Published: 2023-06-02 21:26:20

Denoising diffusion probabilistic models have transformed image generation
with their impressive fidelity and diversity. We show that they also excel in
estimating optical flow and monocular depth, surprisingly, without
task-specific architectures and loss functions that are predominant for these
tasks. Compared to the point estimates of conventional regression-based
methods, diffusion models also enable Monte Carlo inference, e.g., capturing
uncertainty and ambiguity in flow and depth. With self-supervised pre-training,
the combined use of synthetic and real data for supervised training, and
technical innovations (infilling and step-unrolled denoising diffusion
training) to handle noisy-incomplete training data, and a simple form of
coarse-to-fine refinement, one can train state-of-the-art diffusion models for
depth and optical flow estimation. Extensive experiments focus on quantitative
performance against benchmarks, ablations, and the model's ability to capture
uncertainty and multimodality, and impute missing values. Our model, DDVM
(Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth
error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\%
on the KITTI optical flow benchmark, about 25\% better than the best published
method. For an overview see https://diffusion-vision.github.io.
PolyVoice: Language Models for Speech to Speech Translation
Authors: name: Qianqian Dong, name: Zhiying Huang, name: Chen Xu, name: Yunlong Zhao, name: Kexin Wang, name: Xuxin Cheng, name: Tom Ko, name: Qiao Tian, name: Tang Li, name: Fengpeng Yue, name: Ye Bai, name: Xi Chen, name: Lu Lu, name: Zejun Ma, name: Yuping Wang, name: Mingxuan Wang, name: Yuxuan Wang
Published: 2023-06-05 15:53:15

We propose PolyVoice, a language model-based framework for speech-to-speech
translation (S2ST) system. Our framework consists of two language models: a
translation language model and a speech synthesis language model. We use
discretized speech units, which are generated in a fully unsupervised way, and
thus our framework can be used for unwritten languages. For the speech
synthesis part, we adopt the existing VALL-E X approach and build a unit-based
audio language model. This grants our framework the ability to preserve the
voice characteristics and the speaking style of the original speech. We examine
our system on Chinese rightarrow English and English rightarrow Spanish
pairs. Experimental results show that our system can generate speech with high
translation quality and audio quality. Speech samples are available at
https://speechtranslation.github.io/polyvoice.
A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean
  Language Models
Authors: name: Hyunwoong Ko, name: Kichang Yang, name: Minho Ryu, name: Taekyoon Choi, name: Seungmu Yang, name: jiwung Hyun, name: Sungho Park
Published: 2023-06-04 04:04:04

Polyglot is a pioneering project aimed at enhancing the non-English language
performance of multilingual language models. Despite the availability of
various multilingual models such as mBERT (Devlin et al., 2019), XGLM (Lin et
al., 2022), and BLOOM (Scao et al., 2022), researchers and developers often
resort to building monolingual models in their respective languages due to the
dissatisfaction with the current multilingual models non-English language
capabilities. Addressing this gap, we seek to develop advanced multilingual
language models that offer improved performance in non-English languages. In
this paper, we introduce the Polyglot Korean models, which represent a specific
focus rather than being multilingual in nature. In collaboration with TUNiB,
our team collected 1.2TB of Korean data meticulously curated for our research
journey. We made a deliberate decision to prioritize the development of Korean
models before venturing into multilingual models. This choice was motivated by
multiple factors: firstly, the Korean models facilitated performance
comparisons with existing multilingual models; and finally, they catered to the
specific needs of Korean companies and researchers. This paper presents our
work in developing the Polyglot Korean models, which propose some steps towards
addressing the non-English language performance gap in multilingual language
models.
SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model
Authors: name: Dingyuan Zhang, name: Dingkang Liang, name: Hongcheng Yang, name: Zhikang Zou, name: Xiaoqing Ye, name: Zhe Liu, name: Xiang Bai
Published: 2023-06-04 03:09:21

With the development of large language models, many remarkable linguistic
systems like ChatGPT have thrived and achieved astonishing success on many
tasks, showing the incredible power of foundation models. In the spirit of
unleashing the capability of foundation models on vision tasks, the Segment
Anything Model (SAM), a vision foundation model for image segmentation, has
been proposed recently and presents strong zero-shot ability on many downstream
2D tasks. However, whether SAM can be adapted to 3D vision tasks has yet to be
explored, especially 3D object detection. With this inspiration, we explore
adapting the zero-shot ability of SAM to 3D object detection in this paper. We
propose a SAM-powered BEV processing pipeline to detect objects and get
promising results on the large-scale Waymo open dataset. As an early attempt,
our method takes a step toward 3D object detection with vision foundation
models and presents the opportunity to unleash their power on 3D vision tasks.
The code is released at https://github.com/DYZhang09/SAM3D.
VisualGPTScore: Visio-Linguistic Reasoning with Multimodal Generative
  Pre-Training Scores
Authors: name: Zhiqiu Lin, name: Xinyue Chen, name: Deepak Pathak, name: Pengchuan Zhang, name: Deva Ramanan
Published: 2023-06-02 19:19:43

Vision-language models (VLMs) discriminatively pre-trained with contrastive
image-text matching losses such as P(match|text, image)
have been criticized for lacking compositional understanding. This means they
might output similar scores even if the original caption is rearranged into a
different semantic statement. To address this, we propose to use the {bf
V}isual {bf G}enerative {bf P}re-{bf T}raining Score ({bf
VisualGPTScore}) of P(text|image), a multimodal
generative score that captures the likelihood of a text caption conditioned
on an image using an image-conditioned language model. Contrary to the belief
that VLMs are mere bag-of-words models, our off-the-shelf VisualGPTScore
demonstrates top-tier performance on recently proposed image-text retrieval
benchmarks like ARO and Crepe that assess compositional reasoning. Furthermore,
we factorize VisualGPTScore into a product of the marginal P(text)
and the Pointwise Mutual Information (PMI). This helps to (a)
diagnose datasets with strong language bias, and (b) debias results on other
benchmarks like Winoground using an information-theoretic framework.
VisualGPTScore provides valuable insights and serves as a strong baseline for
future evaluation of visio-linguistic compositionality.
Probabilistic Adaptation of Text-to-Video Models
Authors: name: Mengjiao Yang, name: Yilun Du, name: Bo Dai, name: Dale Schuurmans, name: Joshua B. Tenenbaum, name: Pieter Abbeel
Published: 2023-06-02 19:00:17

Large text-to-video models trained on internet-scale data have demonstrated
exceptional capabilities in generating high-fidelity videos from arbitrary
textual descriptions. However, adapting these models to tasks with limited
domain-specific data, such as animation or robotics videos, poses a significant
computational challenge, since finetuning a pretrained large model can be
prohibitively expensive. Inspired by how a small modifiable component (e.g.,
prompts, prefix-tuning) can adapt a large language model to perform new tasks
without requiring access to the model weights, we investigate how to adapt a
large pretrained text-to-video model to a variety of downstream domains and
tasks without finetuning. In answering this question, we propose Video Adapter,
which leverages the score function of a large pretrained video diffusion model
as a probabilistic prior to guide the generation of a task-specific small video
model. Our experiments show that Video Adapter is capable of incorporating the
broad knowledge and preserving the high fidelity of a large pretrained video
model in a task-specific small video model that is able to generate
high-quality yet specialized videos on a variety of tasks such as animation,
egocentric modeling, and modeling of simulated and real-world robotics data.
More videos can be found on the website https://video-adapter.github.io/.
Transformer-based Vulnerability Detection in Code at EditTime:
  Zero-shot, Few-shot, or Fine-tuning?
Authors: name: Aaron Chan, name: Anant Kharkar, name: Roshanak Zilouchian Moghaddam, name: Yevhen Mohylevskyy, name: Alec Helyar, name: Eslam Kamal, name: Mohamed Elkamhawy, name: Neel Sundaresan
Published: 2023-05-23 01:21:55

Software vulnerabilities bear enterprises significant costs. Despite
extensive efforts in research and development of software vulnerability
detection methods, uncaught vulnerabilities continue to put software owners and
users at risk. Many current vulnerability detection methods require that code
snippets can compile and build before attempting detection. This,
unfortunately, introduces a long latency between the time a vulnerability is
injected to the time it is removed, which can substantially increases the cost
of fixing a vulnerability. We recognize that the current advances in machine
learning can be used to detect vulnerable code patterns on syntactically
incomplete code snippets as the developer is writing the code at EditTime. In
this paper we present a practical system that leverages deep learning on a
large-scale data set of vulnerable code patterns to learn complex
manifestations of more than 250 vulnerability types and detect vulnerable code
patterns at EditTime. We discuss zero-shot, few-shot, and fine-tuning
approaches on state of the art pre-trained Large Language Models (LLMs). We
show that in comparison with state of the art vulnerability detection models
our approach improves the state of the art by 10%. We also evaluate our
approach to detect vulnerability in auto-generated code by code LLMs.
Evaluation on a benchmark of high-risk code scenarios shows a reduction of up
to 90% vulnerability reduction.
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video
  Understanding
Authors: name: Hang Zhang, name: Xin Li, name: Lidong Bing
Published: 2023-06-05 13:17:27

We present Video-LLaMA, a multi-modal framework that empowers Large Language
Models (LLMs) with the capability of understanding both visual and auditory
content in the video. Video-LLaMA bootstraps cross-modal training from the
frozen pre-trained visual \& audio encoders and the frozen LLMs. Unlike
previous vision- LLMs that focus on static image comprehensions such as
MiniGPT-4~zhu2023minigpt and LLaVA~liu2023visualit, Video-LLaMA
tackles two challenges in video understanding: (1) capturing the temporal
changes in visual scenes, (2) integrating audio-visual signals. For the first
challenge, we propose Video Q-former to extend the pre-trained image encoder to
a video encoder and introduce a video-to-text generation task to learn
video-language correspondence. For the second challenge, we leverage
ImageBind~girdhar2023imagebind as the pre-trained audio encoder which
performs exceptionally well in aligning different modalities to a common
embedding space. And then introduce an Audio Q-former to learn auditory query
tokens. To align the output of both visual \& audio encoder with LLM's
embedding space, we train Video-LLaMA on a large-scale vision caption dataset
and a hign-quantity vision-instruction-tuning dataset. We found Video-LLaMA
showcases the ability to perceive and comprehend video content, generating
meaningful responses that are grounded in the visual and auditory information
present in the videos. This highlights the potential of Video-LLaMA as a
promising prototype for audio-visual AI assistants. Our code, pre-trained
model, and demo are available at
https://github.com/DAMO-NLP-SG/Video-LLaMA.
InstructZero: Efficient Instruction Optimization for Black-Box Large
  Language Models
Authors: name: Lichang Chen, name: Jiuhai Chen, name: Tom Goldstein, name: Heng Huang, name: Tianyi Zhou
Published: 2023-06-05 17:55:22

Large language models~(LLMs) are instruction followers, but it can be
challenging to find the best instruction for different situations, especially
for black-box LLMs on which backpropagation is forbidden. Instead of directly
optimizing the discrete instruction, we optimize a low-dimensional soft prompt
applied to an open-source LLM to generate the instruction for the black-box
LLM. On each iteration of the proposed method, which we call InstructZero, a
soft prompt is converted into an instruction using the open-source LLM, which
is then submitted to the black-box LLM for zero-shot evaluation, and the
performance is sent to Bayesian optimization to produce new soft prompts
improving the zero-shot performance. We evaluate InstructZero on different
combinations of open-source LLMs and APIs including Vicuna and ChatGPT. Our
results show that InstructZero outperforms SOTA auto-instruction methods across
a variety of downstream tasks. Our code and data are publicly available at
https://github.com/Lichang-Chen/InstructZero.
PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge
Authors: name: Laura Cabello, name: Jiaang Li, name: Ilias Chalkidis
Published: 2023-06-05 16:44:27

The recently released ChatGPT model demonstrates unprecedented capabilities
in zero-shot question-answering. In this work, we probe ChatGPT for its
conversational understanding and introduce a conversational framework
(protocol) that can be adopted in future studies. The Pok\'emon universe serves
as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to
its closed world assumption. After bringing ChatGPT's background knowledge (on
the Pok\'emon universe) to light, we test its reasoning process when using
these concepts in battle scenarios. We then evaluate its ability to acquire new
knowledge and include it in its reasoning process. Our ultimate goal is to
assess ChatGPT's ability to generalize, combine features, and to acquire and
reason over newly introduced knowledge from human feedback. We find that
ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in
battle scenarios to a great extent, even when new information is introduced.
The model performs better with collaborative feedback and if there is an
initial phase of information retrieval, but also hallucinates occasionally and
is susceptible to adversarial attacks.
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and
  Generative Fusion
Authors: name: Dongfu Jiang, name: Xiang Ren, name: Bill Yuchen Lin
Published: 2023-06-05 03:32:26

We present LLM-Blender, an ensembling framework designed to attain
consistently superior performance by leveraging the diverse strengths of
multiple open-source large language models (LLMs). Our framework consists of
two modules: PairRanker and GenFuser, addressing the observation that optimal
LLMs for different examples can significantly vary. PairRanker employs a
specialized pairwise comparison method to distinguish subtle differences
between candidate outputs. It jointly encodes the input text and a pair of
candidates, using cross-attention encoders to determine the superior one. Our
results demonstrate that PairRanker exhibits the highest correlation with
ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates,
generating an improved output by capitalizing on their strengths and mitigating
their weaknesses. To facilitate large-scale evaluation, we introduce a
benchmark dataset, MixInstruct, which is a mixture of multiple instruction
datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly
outperform individual LLMs and baseline methods across various metrics,
establishing a substantial performance gap.