Orionnnn/hub-stats huggingface dataset:
Chain-of-Dictionary Prompting Elicits Translation in Large Language
  Models
Authors: name: Hongyuan Lu, name: Haoyang Huang, name: Dongdong Zhang, name: Haoran Yang, name: Wai Lam, name: Furu Wei
Published: 2023-05-11 05:19:47

Large language models (LLMs) have shown surprisingly good performance in
multilingual neural machine translation (MNMT) even when trained without
parallel data. Yet, despite the fact that the amount of training data is
gigantic, they still struggle with translating rare words, particularly for
low-resource languages. Even worse, it is usually unrealistic to retrieve
relevant demonstrations for in-context learning with low-resource languages on
LLMs, which restricts the practical use of LLMs for translation -- how should
we mitigate this problem? To this end, we present a novel method, CoD, which
augments LLMs with prior knowledge with the chains of multilingual dictionaries
for a subset of input words to elicit translation abilities for LLMs. Extensive
experiments indicate that augmenting ChatGPT with CoD elicits large gains by up
to 13x ChrF++ points for MNMT (3.08 to 42.63 for English to Serbian written in
Cyrillic script) on FLORES-200 full devtest set. We further demonstrate the
importance of chaining the multilingual dictionaries, as well as the
superiority of CoD to few-shot demonstration for low-resource languages.
Bot or Human? Detecting ChatGPT Imposters with A Single Question
Authors: name: Hong Wang, name: Xuan Luo, name: Weizhi Wang, name: Xifeng Yan
Published: 2023-05-10 19:09:24

Large language models like ChatGPT have recently demonstrated impressive
capabilities in natural language understanding and generation, enabling various
applications including translation, essay writing, and chit-chatting. However,
there is a concern that they can be misused for malicious purposes, such as
fraud or denial-of-service attacks. Therefore, it is crucial to develop methods
for detecting whether the party involved in a conversation is a bot or a human.
In this paper, we propose a framework named FLAIR, Finding Large language model
Authenticity via a single Inquiry and Response, to detect conversational bots
in an online manner. Specifically, we target a single question scenario that
can effectively differentiate human users from bots. The questions are divided
into two categories: those that are easy for humans but difficult for bots
(e.g., counting, substitution, positioning, noise filtering, and ASCII art),
and those that are easy for bots but difficult for humans (e.g., memorization
and computation). Our approach shows different strengths of these questions in
their effectiveness, providing a new way for online service providers to
protect themselves against nefarious activities and ensure that they are
serving real users. We open-sourced our dataset on
https://github.com/hongwang600/FLAIR and welcome contributions from the
community to enrich such detection datasets.
V2Meow: Meowing to the Visual Beat via Music Generation
Authors: name: Kun Su, name: Judith Yue Li, name: Qingqing Huang, name: Dima Kuzmin, name: Joonseok Lee, name: Chris Donahue, name: Fei Sha, name: Aren Jansen, name: Yu Wang, name: Mauro Verzetti, name: Timo I. Denk
Published: 2023-05-11 06:26:41

Generating high quality music that complements the visual content of a video
is a challenging task. Most existing visual conditioned music generation
systems generate symbolic music data, such as MIDI files, instead of raw audio
waveform. Given the limited availability of symbolic music data, such methods
can only generate music for a few instruments or for specific types of visual
input. In this paper, we propose a novel approach called V2Meow that can
generate high-quality music audio that aligns well with the visual semantics of
a diverse range of video input types. Specifically, the proposed music
generation system is a multi-stage autoregressive model which is trained with a
number of O(100K) music audio clips paired with video frames, which are mined
from in-the-wild music videos, and no parallel symbolic music data is involved.
V2Meow is able to synthesize high-fidelity music audio waveform solely
conditioned on pre-trained visual features extracted from an arbitrary silent
video clip, and it also allows high-level control over the music style of
generation examples via supporting text prompts in addition to the video frames
conditioning. Through both qualitative and quantitative evaluations, we
demonstrate that our model outperforms several existing music generation
systems in terms of both visual-audio correspondence and audio quality.
Exploiting Diffusion Prior for Real-World Image Super-Resolution
Authors: name: Jianyi Wang, name: Zongsheng Yue, name: Shangchen Zhou, name: Kelvin C. K. Chan, name: Chen Change Loy
Published: 2023-05-11 17:55:25

We present a novel approach to leverage prior knowledge encapsulated in
pre-trained text-to-image diffusion models for blind super-resolution (SR).
Specifically, by employing our time-aware encoder, we can achieve promising
restoration results without altering the pre-trained synthesis model, thereby
preserving the generative prior and minimizing training cost. To remedy the
loss of fidelity caused by the inherent stochasticity of diffusion models, we
introduce a controllable feature wrapping module that allows users to balance
quality and fidelity by simply adjusting a scalar value during the inference
process. Moreover, we develop a progressive aggregation sampling strategy to
overcome the fixed-size constraints of pre-trained diffusion models, enabling
adaptation to resolutions of any size. A comprehensive evaluation of our method
using both synthetic and real-world benchmarks demonstrates its superiority
over current state-of-the-art approaches.
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency
  Model
Authors: name: Zhen Ye, name: Wei Xue, name: Xu Tan, name: Jie Chen, name: Qifeng Liu, name: Yike Guo
Published: 2023-05-11 15:51:46

Denoising diffusion probabilistic models (DDPMs) have shown promising
performance for speech synthesis. However, a large number of iterative steps
are required to achieve high sample quality, which restricts the inference
speed. Maintaining sample quality while increasing sampling speed has become a
challenging task. In this paper, we propose a "Co"nsistency "Mo"del-based
"Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a
single diffusion sampling step while achieving high audio quality. The
consistency constraint is applied to distill a consistency model from a
well-designed diffusion-based teacher model, which ultimately yields superior
performances in the distilled CoMoSpeech. Our experiments show that by
generating audio recordings by a single sampling step, the CoMoSpeech achieves
an inference speed more than 150 times faster than real-time on a single NVIDIA
A100 GPU, which is comparable to FastSpeech2, making diffusion-sampling based
speech synthesis truly practical. Meanwhile, objective and subjective
evaluations on text-to-speech and singing voice synthesis show that the
proposed teacher models yield the best audio quality, and the one-step sampling
based CoMoSpeech achieves the best inference speed with better or comparable
audio quality to other conventional multi-step diffusion model baselines. Audio
samples are available at https://comospeech.github.io/.
HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
Authors: name: Mustafa Işık, name: Martin Rünz, name: Markos Georgopoulos, name: Taras Khakhulin, name: Jonathan Starck, name: Lourdes Agapito, name: Matthias Nießner
Published: 2023-05-10 17:59:55

Representing human performance at high-fidelity is an essential building
block in diverse applications, such as film production, computer games or
videoconferencing. To close the gap to production-level quality, we introduce
HumanRF, a 4D dynamic neural scene representation that captures full-body
appearance in motion from multi-view video input, and enables playback from
novel, unseen viewpoints. Our novel representation acts as a dynamic video
encoding that captures fine details at high compression rates by factorizing
space-time into a temporal matrix-vector decomposition. This allows us to
obtain temporally coherent reconstructions of human actors for long sequences,
while representing high-resolution details even in the context of challenging
motion. While most research focuses on synthesizing at resolutions of 4MP or
lower, we address the challenge of operating at 12MP. To this end, we introduce
ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160
cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We
demonstrate challenges that emerge from using such high-resolution data and
show that our newly introduced HumanRF effectively leverages this data, making
a significant step towards production-level quality novel view synthesis.
Reconstructing Animatable Categories from Videos
Authors: name: Gengshan Yang, name: Chaoyang Wang, name: N Dinesh Reddy, name: Deva Ramanan
Published: 2023-05-10 17:56:21

Building animatable 3D models is challenging due to the need for 3D scans,
laborious registration, and manual rigging, which are difficult to scale to
arbitrary categories. Recently, differentiable rendering provides a pathway to
obtain high-quality 3D models from monocular videos, but these are limited to
rigid categories or single instances. We present RAC that builds category 3D
models from monocular videos while disentangling variations over instances and
motion over time. Three key ideas are introduced to solve this problem: (1)
specializing a skeleton to instances via optimization, (2) a method for latent
space regularization that encourages shared structure across a category while
maintaining instance details, and (3) using 3D background models to disentangle
objects from the background. We show that 3D models of humans, cats, and dogs
can be learned from 50-100 internet videos.
DexArt: Benchmarking Generalizable Dexterous Manipulation with
  Articulated Objects
Authors: name: Chen Bao, name: Helin Xu, name: Yuzhe Qin, name: Xiaolong Wang
Published: 2023-05-09 18:30:58

To enable general-purpose robots, we will require the robot to operate daily
articulated objects as humans do. Current robot manipulation has heavily relied
on using a parallel gripper, which restricts the robot to a limited set of
objects. On the other hand, operating with a multi-finger robot hand will allow
better approximation to human behavior and enable the robot to operate on
diverse articulated objects. To this end, we propose a new benchmark called
DexArt, which involves Dexterous manipulation with Articulated objects in a
physical simulator. In our benchmark, we define multiple complex manipulation
tasks, and the robot hand will need to manipulate diverse articulated objects
within each task. Our main focus is to evaluate the generalizability of the
learned policy on unseen articulated objects. This is very challenging given
the high degrees of freedom of both hands and objects. We use Reinforcement
Learning with 3D representation learning to achieve generalization. Through
extensive studies, we provide new insights into how 3D representation learning
affects decision making in RL with 3D point cloud inputs. More details can be
found at https://www.chenbao.tech/dexart/.
Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era
Authors: name: Chenghao Li, name: Chaoning Zhang, name: Atish Waghwase, name: Lik-Hang Lee, name: Francois Rameau, name: Yang Yang, name: Sung-Ho Bae, name: Choong Seon Hong
Published: 2023-05-10 13:26:08

Generative AI (AIGC, a.k.a. AI generated content) has made remarkable
progress in the past few years, among which text-guided content generation is
the most practical one since it enables the interaction between human
instruction and AIGC. Due to the development in text-to-image as well 3D
modeling technologies (like NeRF), text-to-3D has become a newly emerging yet
highly active research field. Our work conducts the first yet comprehensive
survey on text-to-3D to help readers interested in this direction quickly catch
up with its fast development. First, we introduce 3D data representations,
including both Euclidean data and non-Euclidean data. On top of that, we
introduce various foundation technologies as well as summarize how recent works
combine those foundation technologies to realize satisfactory text-to-3D.
Moreover, we summarize how text-to-3D technology is used in various
applications, including avatar generation, texture generation, shape
transformation, and scene generation.
Alternating Gradient Descent and Mixture-of-Experts for Integrated
  Multimodal Perception
Authors: name: Hassan Akbari, name: Dan Kondratyuk, name: Yin Cui, name: Rachel Hornung, name: Huisheng Wang, name: Hartwig Adam
Published: 2023-05-10 17:22:06

We present Integrated Multimodal Perception (IMP), a simple and scalable
multimodal multi-task training and modeling approach. IMP integrates multimodal
inputs including image, video, text, and audio into a single Transformer
encoder with minimal modality-specific components. IMP makes use of a novel
design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts
(MoE) for efficient model \& task scaling. We conduct extensive empirical
studies about IMP and reveal the following key insights: 1) performing gradient
descent updates by alternating on diverse heterogeneous modalities, loss
functions, and tasks, while also varying input resolutions, efficiently
improves multimodal understanding. 2) model sparsification with MoE on a single
modality-agnostic encoder substantially improves the performance, outperforming
dense models that use modality-specific encoders or additional fusion layers
and greatly mitigating the conflicts between modalities. IMP achieves
competitive performance on a wide range of downstream tasks including image
classification, video classification, image-text, and video-text retrieval.
Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves
new state-of-the-art in zero-shot video classification. Our model achieves
77.0% on Kinetics-400, 76.8% on Kinetics-600, and 76.8% on Kinetics-700
zero-shot classification accuracy, improving the previous state-of-the-art by
+5%, +6.7%, and +5.8%, respectively, while using only 15% of their total
training computational cost.
Multi-Task End-to-End Training Improves Conversational Recommendation
Authors: name: Naveen Ram, name: Dima Kuzmin, name: Ellie Ka In Chio, name: Moustafa Farid Alzantot, name: Santiago Ontanon, name: Ambarish Jash, name: Judith Yue Li
Published: 2023-05-08 22:42:48

In this paper, we analyze the performance of a multitask end-to-end
transformer model on the task of conversational recommendations, which aim to
provide recommendations based on a user's explicit preferences expressed in
dialogue. While previous works in this area adopt complex multi-component
approaches where the dialogue management and entity recommendation tasks are
handled by separate components, we show that a unified transformer model, based
on the T5 text-to-text transformer model, can perform competitively in both
recommending relevant items and generating conversation dialogue. We fine-tune
our model on the ReDIAL conversational movie recommendation dataset, and create
additional training tasks derived from MovieLens (such as the prediction of
movie attributes and related movies based on an input movie), in a multitask
learning setting. Using a series of probe studies, we demonstrate that the
learned knowledge in the additional tasks is transferred to the conversational
setting, where each task leads to a 9%-52% increase in its related probe score.
VideoChat: Chat-Centric Video Understanding
Authors: name: KunChang Li, name: Yinan He, name: Yi Wang, name: Yizhuo Li, name: Wenhai Wang, name: Ping Luo, name: Yali Wang, name: Limin Wang, name: Yu Qiao
Published: 2023-05-10 17:59:04

In this study, we initiate an exploration into video understanding by
introducing VideoChat, an end-to-end chat-centric video understanding system.
It integrates video foundation models and large language models via a learnable
neural interface, excelling in spatiotemporal reasoning, event localization,
and causal relationship inference. To instructively tune this system, we
propose a video-centric instruction dataset, composed of thousands of videos
matched with detailed descriptions and conversations. This dataset emphasizes
spatiotemporal reasoning and causal relationships, providing a valuable asset
for training chat-centric video understanding systems. Preliminary qualitative
experiments reveal our system's potential across a broad spectrum of video
applications and set the standard for future research. Access our code and data
at https://github.com/OpenGVLab/Ask-Anything
StarCoder: may the source be with you!
Authors: name: Raymond Li, name: Loubna Ben Allal, name: Yangtian Zi, name: Niklas Muennighoff, name: Denis Kocetkov, name: Chenghao Mou, name: Marc Marone, name: Christopher Akiki, name: Jia Li, name: Jenny Chim, name: Qian Liu, name: Evgenii Zheltonozhskii, name: Terry Yue Zhuo, name: Thomas Wang, name: Olivier Dehaene, name: Mishig Davaadorj, name: Joel Lamy-Poirier, name: João Monteiro, name: Oleh Shliazhko, name: Nicolas Gontier, name: Nicholas Meade, name: Armel Zebaze, name: Ming-Ho Yee, name: Logesh Kumar Umapathi, name: Jian Zhu, name: Benjamin Lipkin, name: Muhtasham Oblokulov, name: Zhiruo Wang, name: Rudra Murthy, name: Jason Stillerman, name: Siva Sankalp Patel, name: Dmitry Abulkhanov, name: Marco Zocca, name: Manan Dey, name: Zhihan Zhang, name: Nour Fahmy, name: Urvashi Bhattacharyya, name: Wenhao Yu, name: Swayam Singh, name: Sasha Luccioni, name: Paulo Villegas, name: Maxim Kunakov, name: Fedor Zhdanov, name: Manuel Romero, name: Tony Lee, name: Nadav Timor, name: Jennifer Ding, name: Claire Schlesinger, name: Hailey Schoelkopf, name: Jan Ebert, name: Tri Dao, name: Mayank Mishra, name: Alex Gu, name: Jennifer Robinson, name: Carolyn Jane Anderson, name: Brendan Dolan-Gavitt, name: Danish Contractor, name: Siva Reddy, name: Daniel Fried, name: Dzmitry Bahdanau, name: Yacine Jernite, name: Carlos Muñoz Ferrandis, name: Sean Hughes, name: Thomas Wolf, name: Arjun Guha, name: Leandro von Werra, name: Harm de Vries
Published: 2023-05-09 08:16:42

The BigCode community, an open-scientific collaboration working on the
responsible development of Large Language Models for Code (Code LLMs),
introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context
length, infilling capabilities and fast large-batch inference enabled by
multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced
from The Stack, a large collection of permissively licensed GitHub repositories
with inspection tools and an opt-out process. We fine-tuned StarCoderBase on
35B Python tokens, resulting in the creation of StarCoder. We perform the most
comprehensive evaluation of Code LLMs to date and show that StarCoderBase
outperforms every open Code LLM that supports multiple programming languages
and matches or outperforms the OpenAI code-cushman-001 model. Furthermore,
StarCoder outperforms every model that is fine-tuned on Python, can be prompted
to achieve 40\% pass@1 on HumanEval, and still retains its performance on other
programming languages. We take several important steps towards a safe
open-access model release, including an improved PII redaction pipeline and a
novel attribution tracing tool, and make the StarCoder models publicly
available under a more commercially viable version of the Open Responsible AI
Model license.
Privacy-Preserving Recommender Systems with Synthetic Query Generation
  using Differentially Private Large Language Models
Authors: name: Aldo Gael Carranza, name: Rezsa Farahani, name: Natalia Ponomareva, name: Alex Kurakin, name: Matthew Jagielski, name: Milad Nasr
Published: 2023-05-10 08:30:31

We propose a novel approach for developing privacy-preserving large-scale
recommender systems using differentially private (DP) large language models
(LLMs) which overcomes certain challenges and limitations in DP training these
complex systems. Our method is particularly well suited for the emerging area
of LLM-based recommender systems, but can be readily employed for any
recommender systems that process representations of natural language inputs.
Our approach involves using DP training methods to fine-tune a publicly
pre-trained LLM on a query generation task. The resulting model can generate
private synthetic queries representative of the original queries which can be
freely shared for any downstream non-private recommendation training procedures
without incurring any additional privacy cost. We evaluate our method on its
ability to securely train effective deep retrieval models, and we observe
significant improvements in their retrieval quality without compromising
query-level privacy guarantees compared to methods where the retrieval models
are directly DP trained.
Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text
  Analytics? An Examination on Several Typical Tasks
Authors: name: Xianzhi Li, name: Xiaodan Zhu, name: Zhiqiang Ma, name: Xiaomo Liu, name: Sameena Shah
Published: 2023-05-10 03:13:54

The most recent large language models such as ChatGPT and GPT-4 have garnered
significant attention, as they are capable of generating high-quality responses
to human input. Despite the extensive testing of ChatGPT and GPT-4 on generic
text corpora, showcasing their impressive capabilities, a study focusing on
financial corpora has not been conducted. In this study, we aim to bridge this
gap by examining the potential of ChatGPT and GPT-4 as a solver for typical
financial text analytic problems in the zero-shot or few-shot setting.
Specifically, we assess their capabilities on four representative tasks over
five distinct financial textual datasets. The preliminary study shows that
ChatGPT and GPT-4 struggle on tasks such as financial named entity recognition
(NER) and sentiment analysis, where domain-specific knowledge is required,
while they excel in numerical reasoning tasks. We report both the strengths and
limitations of the current versions of ChatGPT and GPT-4, comparing them to the
state-of-the-art finetuned models as well as pretrained domain-specific
generative models. Our experiments provide qualitative studies, through which
we hope to help understand the capability of the existing models and facilitate
further improvements.
Sketching the Future (STF): Applying Conditional Control Techniques to
  Text-to-Video Models
Authors: name: Rohan Dhesikan, name: Vignesh Rajmohan
Published: 2023-05-10 02:33:25

The proliferation of video content demands efficient and flexible neural
network based approaches for generating new video content. In this paper, we
propose a novel approach that combines zero-shot text-to-video generation with
ControlNet to improve the output of these models. Our method takes multiple
sketched frames as input and generates video output that matches the flow of
these frames, building upon the Text-to-Video Zero architecture and
incorporating ControlNet to enable additional input conditions. By first
interpolating frames between the inputted sketches and then running
Text-to-Video Zero using the new interpolated frames video as the control
technique, we leverage the benefits of both zero-shot text-to-video generation
and the robust control provided by ControlNet. Experiments demonstrate that our
method excels at producing high-quality and remarkably consistent video content
that more accurately aligns with the user's intended motion for the subject
within the video. We provide a comprehensive resource package, including a demo
video, project website, open-source GitHub repository, and a Colab playground
to foster further research and application of our proposed method.