Orionnnn/hub-stats huggingface dataset:
Smart Word Suggestions for Writing Assistance
Authors: name: Chenshuo Wang, name: Shaoguang Mao, name: Tao Ge, name: Wenshan Wu, name: Xun Wang, name: Yan Xia, name: Jonathan Tien, name: Dongyan Zhao
Published: 2023-05-17 06:15:41

Enhancing word usage is a desired feature for writing assistance. To further
advance research in this area, this paper introduces "Smart Word Suggestions"
(SWS) task and benchmark. Unlike other works, SWS emphasizes end-to-end
evaluation and presents a more realistic writing assistance scenario. This task
involves identifying words or phrases that require improvement and providing
substitution suggestions. The benchmark includes human-labeled data for
testing, a large distantly supervised dataset for training, and the framework
for evaluation. The test data includes 1,000 sentences written by English
learners, accompanied by over 16,000 substitution suggestions annotated by 10
native speakers. The training dataset comprises over 3.7 million sentences and
12.7 million suggestions generated through rules. Our experiments with seven
baselines demonstrate that SWS is a challenging task. Based on experimental
analysis, we suggest potential directions for future research on SWS. The
dataset and related codes is available at
https://github.com/microsoft/SmartWordSuggestions.
NerfBridge: Bringing Real-time, Online Neural Radiance Field Training to
  Robotics
Authors: name: Javier Yu, name: Jun En Low, name: Keiko Nagami, name: Mac Schwager
Published: 2023-05-16 19:27:17

This work was presented at the IEEE International Conference on Robotics and
Automation 2023 Workshop on Unconventional Spatial Representations.
  Neural radiance fields (NeRFs) are a class of implicit scene representations
that model 3D environments from color images. NeRFs are expressive, and can
model the complex and multi-scale geometry of real world environments, which
potentially makes them a powerful tool for robotics applications. Modern NeRF
training libraries can generate a photo-realistic NeRF from a static data set
in just a few seconds, but are designed for offline use and require a slow pose
optimization pre-computation step.
  In this work we propose NerfBridge, an open-source bridge between the Robot
Operating System (ROS) and the popular Nerfstudio library for real-time, online
training of NeRFs from a stream of images. NerfBridge enables rapid development
of research on applications of NeRFs in robotics by providing an extensible
interface to the efficient training pipelines and model libraries provided by
Nerfstudio. As an example use case we outline a hardware setup that can be used
NerfBridge to train a NeRF from images captured by a camera mounted to a
quadrotor in both indoor and outdoor environments.
  For accompanying video https://youtu.be/EH0SLn-RcDg and code
https://github.com/javieryu/nerf_bridge.
CostFormer:Cost Transformer for Cost Aggregation in Multi-view Stereo
Authors: name: Weitao Chen, name: Hongbin Xu, name: Zhipeng Zhou, name: Yang Liu, name: Baigui Sun, name: Wenxiong Kang, name: Xuansong Xie
Published: 2023-05-17 16:01:27

The core of Multi-view Stereo(MVS) is the matching process among reference
and source pixels. Cost aggregation plays a significant role in this process,
while previous methods focus on handling it via CNNs. This may inherit the
natural limitation of CNNs that fail to discriminate repetitive or incorrect
matches due to limited local receptive fields. To handle the issue, we aim to
involve Transformer into cost aggregation. However, another problem may occur
due to the quadratically growing computational complexity caused by
Transformer, resulting in memory overflow and inference latency. In this paper,
we overcome these limits with an efficient Transformer-based cost aggregation
network, namely CostFormer. The Residual Depth-Aware Cost Transformer(RDACT) is
proposed to aggregate long-range features on cost volume via self-attention
mechanisms along the depth and spatial dimensions. Furthermore, Residual
Regression Transformer(RRT) is proposed to enhance spatial attention. The
proposed method is a universal plug-in to improve learning-based MVS methods.
Transfer Learning for Fine-grained Classification Using Semi-supervised
  Learning and Visual Transformers
Authors: name: Manuel Lagunas, name: Brayan Impata, name: Victor Martinez, name: Virginia Fernandez, name: Christos Georgakis, name: Sofia Braun, name: Felipe Bertrand
Published: 2023-05-17 07:51:35

Fine-grained classification is a challenging task that involves identifying
subtle differences between objects within the same category. This task is
particularly challenging in scenarios where data is scarce. Visual transformers
(ViT) have recently emerged as a powerful tool for image classification, due to
their ability to learn highly expressive representations of visual data using
self-attention mechanisms. In this work, we explore Semi-ViT, a ViT model fine
tuned using semi-supervised learning techniques, suitable for situations where
we have lack of annotated data. This is particularly common in e-commerce,
where images are readily available but labels are noisy, nonexistent, or
expensive to obtain. Our results demonstrate that Semi-ViT outperforms
traditional convolutional neural networks (CNN) and ViTs, even when fine-tuned
with limited annotated data. These findings indicate that Semi-ViTs hold
significant promise for applications that require precise and fine-grained
classification of visual data.
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Authors: name: Sang Michael Xie, name: Hieu Pham, name: Xuanyi Dong, name: Nan Du, name: Hanxiao Liu, name: Yifeng Lu, name: Percy Liang, name: Quoc V. Le, name: Tengyu Ma, name: Adams Wei Yu
Published: 2023-05-17 17:58:13

The mixture proportions of pretraining data domains (e.g., Wikipedia, books,
web text) greatly affect language model (LM) performance. In this paper, we
propose Domain Reweighting with Minimax Optimization (DoReMi), which first
trains a small proxy model using group distributionally robust optimization
(Group DRO) over domains to produce domain weights (mixture proportions)
without knowledge of downstream tasks. We then resample a dataset with these
domain weights and train a larger, full-sized model. In our experiments, we use
DoReMi on a 280M-parameter proxy model to find domain weights for training an
8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves
perplexity across all domains, even when it downweights a domain. DoReMi
improves average few-shot downstream accuracy by 6.5% over a baseline model
trained using The Pile's default domain weights and reaches the baseline
accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has
no knowledge of downstream tasks, even matches the performance of using domain
weights tuned on downstream tasks.
SLiC-HF: Sequence Likelihood Calibration with Human Feedback
Authors: name: Yao Zhao, name: Rishabh Joshi, name: Tianqi Liu, name: Misha Khalman, name: Mohammad Saleh, name: Peter J. Liu
Published: 2023-05-17 17:57:10

Learning from human feedback has been shown to be effective at aligning
language models with human preferences. Past work has often relied on
Reinforcement Learning from Human Feedback (RLHF), which optimizes the language
model using reward scores assigned from a reward model trained on human
preference data. In this work we show how the recently introduced Sequence
Likelihood Calibration (SLiC), can also be used to effectively learn from human
preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human
feedback data collected for a different model, similar to off-policy, offline
RL data. Automatic and human evaluation experiments on the TL;DR summarization
task show that SLiC-HF significantly improves supervised fine-tuning baselines.
Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF
implementation used in past work while being much simpler to implement, easier
to tune and more computationally efficient in practice.
Searching for Needles in a Haystack: On the Role of Incidental
  Bilingualism in PaLM's Translation Capability
Authors: name: Eleftheria Briakou, name: Colin Cherry, name: George Foster
Published: 2023-05-17 14:58:06

Large, multilingual language models exhibit surprisingly good zero- or
few-shot machine translation capabilities, despite having never seen the
intentionally-included translation examples provided to typical neural
translation systems. We investigate the role of incidental bilingualism -- the
unintentional consumption of bilingual signals, including translation examples
-- in explaining the translation capabilities of large language models, taking
the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method
approach to measure and understand incidental bilingualism at scale. We show
that PaLM is exposed to over 30 million translation pairs across at least 44
languages. Furthermore, the amount of incidental bilingual content is highly
correlated with the amount of monolingual in-language content for non-English
languages. We relate incidental bilingual content to zero-shot prompts and show
that it can be used to mine new prompts to improve PaLM's out-of-English
zero-shot translation quality. Finally, in a series of small-scale ablations,
we show that its presence has a substantial impact on translation capabilities,
although this impact diminishes with model scale.
Improving Language Model Negotiation with Self-Play and In-Context
  Learning from AI Feedback
Authors: name: Yao Fu, name: Hao Peng, name: Tushar Khot, name: Mirella Lapata
Published: 2023-05-17 11:55:32

We study whether multiple large language models (LLMs) can autonomously
improve each other in a negotiation game by playing, reflecting, and
criticizing. We are interested in this question because if LLMs were able to
improve each other, it would imply the possibility of creating strong AI agents
with minimal human intervention. We ask two LLMs to negotiate with each other,
playing the roles of a buyer and a seller, respectively. They aim to reach a
deal with the buyer targeting a lower price and the seller a higher one. A
third language model, playing the critic, provides feedback to a player to
improve the player's negotiation strategies. We let the two agents play
multiple rounds, using previous negotiation history and AI feedback as
in-context demonstrations to improve the model's negotiation strategy
iteratively. We use different LLMs (GPT and Claude) for different roles and use
the deal price as the evaluation metric. Our experiments reveal multiple
intriguing findings: (1) Only a subset of the language models we consider can
self-play and improve the deal price from AI feedback, weaker models either do
not understand the game's rules or cannot incorporate AI feedback for further
improvement. (2) Models' abilities to learn from the feedback differ when
playing different roles. For example, it is harder for Claude-instant to
improve as the buyer than as the seller. (3) When unrolling the game to
multiple rounds, stronger agents can consistently improve their performance by
meaningfully using previous experiences and iterative AI feedback, yet have a
higher risk of breaking the deal. We hope our work provides insightful initial
explorations of having models autonomously improve each other with game playing
and AI feedback.
DinoSR: Self-Distillation and Online Clustering for Self-supervised
  Speech Representation Learning
Authors: name: Alexander H. Liu, name: Heng-Jui Chang, name: Michael Auli, name: Wei-Ning Hsu, name: James R. Glass
Published: 2023-05-17 07:23:46

In this paper, we introduce self-distillation and online clustering for
self-supervised speech representation learning (DinoSR) which combines masked
language modeling, self-distillation, and online clustering. We show that these
concepts complement each other and result in a strong representation learning
model for speech. DinoSR first extracts contextualized embeddings from the
input audio with a teacher network, then runs an online clustering system on
the embeddings to yield a machine-discovered phone inventory, and finally uses
the discretized tokens to guide a student network. We show that DinoSR
surpasses previous state-of-the-art performance in several downstream tasks,
and provide a detailed analysis of the model and the learned discrete units.
The source code will be made available after the anonymity period.
Explaining black box text modules in natural language with language
  models
Authors: name: Chandan Singh, name: Aliyah R. Hsu, name: Richard Antonello, name: Shailee Jain, name: Alexander G. Huth, name: Bin Yu, name: Jianfeng Gao
Published: 2023-05-17 00:29:18

Large language models (LLMs) have demonstrated remarkable prediction
performance for a growing array of tasks. However, their rapid proliferation
and increasing opaqueness have created a growing need for interpretability.
Here, we ask whether we can automatically obtain natural language explanations
for black box text modules. A "text module" is any function that maps text to a
scalar continuous value, such as a submodule within an LLM or a fitted model of
a brain region. "Black box" indicates that we only have access to the module's
inputs/outputs.
  We introduce Summarize and Score (SASC), a method that takes in a text module
and returns a natural language explanation of the module's selectivity along
with a score for how reliable the explanation is. We study SASC in 3 contexts.
First, we evaluate SASC on synthetic modules and find that it often recovers
ground truth explanations. Second, we use SASC to explain modules found within
a pre-trained BERT model, enabling inspection of the model's internals.
Finally, we show that SASC can generate explanations for the response of
individual fMRI voxels to language stimuli, with potential applications to
fine-grained brain mapping. All code for using SASC and reproducing results is
made available on Github.
CoEdIT: Text Editing by Task-Specific Instruction Tuning
Authors: name: Vipul Raheja, name: Dhruv Kumar, name: Ryan Koo, name: Dongyeop Kang
Published: 2023-05-17 00:05:24

Text editing or revision is an essential function of the human writing
process. Understanding the capabilities of LLMs for making high-quality
revisions and collaborating with human writers is a critical step toward
building effective writing assistants. With the prior success of LLMs and
instruction tuning, we leverage instruction-tuned LLMs for text revision to
improve the quality of user-generated text and improve the efficiency of the
process. We introduce CoEdIT, a state-of-the-art text editing model for writing
assistance. CoEdIT takes instructions from the user specifying the attributes
of the desired text, such as "Make the sentence simpler" or "Write it in a more
neutral style," and outputs the edited text. We present a large language model
fine-tuned on a diverse collection of task-specific instructions for text
editing (a total of 82K instructions). Our model (1) achieves state-of-the-art
performance on various text editing benchmarks, (2) is competitive with
publicly available largest-sized LLMs trained on instructions while being
sim60x smaller, (3) is capable of generalizing to unseen edit instructions,
and (4) exhibits compositional comprehension abilities to generalize to
instructions containing different combinations of edit actions. Through
extensive qualitative and quantitative analysis, we show that writers prefer
the edits suggested by CoEdIT, relative to other state-of-the-art text editing
models. Our code and dataset are publicly available.
Application-Agnostic Language Modeling for On-Device ASR
Authors: name: Markus Nußbaum-Thom, name: Lyan Verwimp, name: Youssef Oualil
Published: 2023-05-16 19:31:18

On-device automatic speech recognition systems face several challenges
compared to server-based systems. They have to meet stricter constraints in
terms of speed, disk size and memory while maintaining the same accuracy. Often
they have to serve several applications with different distributions at once,
such as communicating with a virtual assistant and speech-to-text. The simplest
solution to serve multiple applications is to build application-specific
(language) models, but this leads to an increase in memory. Therefore, we
explore different data- and architecture-driven language modeling approaches to
build a single application-agnostic model. We propose two novel feed-forward
architectures that find an optimal trade off between different on-device
constraints. In comparison to the application-specific solution, one of our
novel approaches reduces the disk size by half, while maintaining speed and
accuracy of the original model.
A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them
  In Zero Shot
Authors: name: Aanisha Bhattacharya, name: Yaman K Singla, name: Balaji Krishnamurthy, name: Rajiv Ratn Shah, name: Changyou Chen
Published: 2023-05-16 19:13:11

Multimedia content, such as advertisements and story videos, exhibit a rich
blend of creativity and multiple modalities. They incorporate elements like
text, visuals, audio, and storytelling techniques, employing devices like
emotions, symbolism, and slogans to convey meaning. While previous research in
multimedia understanding has focused mainly on videos with specific actions
like cooking, there is a dearth of large annotated training datasets, hindering
the development of supervised learning models with satisfactory performance for
real-world applications. However, the rise of large language models (LLMs) has
witnessed remarkable zero-shot performance in various natural language
processing (NLP) tasks, such as emotion classification, question-answering, and
topic classification. To bridge this performance gap in multimedia
understanding, we propose verbalizing story videos to generate their
descriptions in natural language and then performing video-understanding tasks
on the generated story as opposed to the original video. Through extensive
experiments on five video-understanding tasks, we demonstrate that our method,
despite being zero-shot, achieves significantly better results than supervised
baselines for video understanding. Further, alleviating a lack of story
understanding benchmarks, we publicly release the first dataset on a crucial
task in computational social science, persuasion strategy identification.
What You See is What You Read? Improving Text-Image Alignment Evaluation
Authors: name: Michal Yarom, name: Yonatan Bitton, name: Soravit Changpinyo, name: Roee Aharoni, name: Jonathan Herzig, name: Oran Lang, name: Eran Ofek, name: Idan Szpektor
Published: 2023-05-17 17:43:38

Automatically determining whether a text and a corresponding image are
semantically aligned is a significant challenge for vision-language models,
with applications in generative text-to-image and image-to-text tasks. In this
work, we study methods for automatic text-image alignment evaluation. We first
introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets
from both text-to-image and image-to-text generation tasks, with human
judgements for whether a given text-image pair is semantically aligned. We then
describe two automatic methods to determine alignment: the first involving a
pipeline based on question generation and visual question answering models, and
the second employing an end-to-end classification approach by finetuning
multimodal pretrained models. Both methods surpass prior approaches in various
text-image alignment tasks, with significant improvements in challenging cases
that involve complex composition or unnatural images. Finally, we demonstrate
how our approaches can localize specific misalignments between an image and a
given text, and how they can be used to automatically re-rank candidates in
text-to-image generation.
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized
  Attention
Authors: name: Guangxuan Xiao, name: Tianwei Yin, name: William T. Freeman, name: Frédo Durand, name: Song Han
Published: 2023-05-17 17:59:55

Diffusion models excel at text-to-image generation, especially in
subject-driven generation for personalized images. However, existing methods
are inefficient due to the subject-specific fine-tuning, which is
computationally intensive and hampers efficient deployment. Moreover, existing
methods struggle with multi-subject generation as they often blend features
among subjects. We present FastComposer which enables efficient, personalized,
multi-subject text-to-image generation without fine-tuning. FastComposer uses
subject embeddings extracted by an image encoder to augment the generic text
conditioning in diffusion models, enabling personalized image generation based
on subject images and textual instructions with only forward passes. To address
the identity blending problem in the multi-subject generation, FastComposer
proposes cross-attention localization supervision during training, enforcing
the attention of reference subjects localized to the correct regions in the
target images. Naively conditioning on subject embeddings results in subject
overfitting. FastComposer proposes delayed subject conditioning in the
denoising step to maintain both identity and editability in subject-driven
image generation. FastComposer generates images of multiple unseen individuals
with different styles, actions, and contexts. It achieves
300times-2500times speedup compared to fine-tuning-based methods and
requires zero extra storage for new subjects. FastComposer paves the way for
efficient, personalized, and high-quality multi-subject image creation. Code,
model, and dataset are available at
https://github.com/mit-han-lab/fastcomposer.
Common Diffusion Noise Schedules and Sample Steps are Flawed
Authors: name: Shanchuan Lin, name: Bingchen Liu, name: Jiashi Li, name: Xiao Yang
Published: 2023-05-15 12:21:08

We discover that common diffusion noise schedules do not enforce the last
timestep to have zero signal-to-noise ratio (SNR), and some implementations of
diffusion samplers do not start from the last timestep. Such designs are flawed
and do not reflect the fact that the model is given pure Gaussian noise at
inference, creating a discrepancy between training and inference. We show that
the flawed design causes real problems in existing implementations. In Stable
Diffusion, it severely limits the model to only generate images with medium
brightness and prevents it from generating very bright and dark samples. We
propose a few simple fixes: (1) rescale the noise schedule to enforce zero
terminal SNR; (2) train the model with v prediction; (3) change the sampler to
always start from the last timestep; (4) rescale classifier-free guidance to
prevent over-exposure. These simple changes ensure the diffusion process is
congruent between training and inference and allow the model to generate
samples more faithful to the original data distribution.