Orionnnn/hub-stats huggingface dataset:
Increasing Diversity While Maintaining Accuracy: Text Data Generation
  with Large Language Models and Human Interventions
Authors: name: John Joon Young Chung, name: Ece Kamar, name: Saleema Amershi
Published: 2023-06-07 04:27:09

Large language models (LLMs) can be used to generate text data for training
and evaluating other models. However, creating high-quality datasets with LLMs
can be challenging. In this work, we explore human-AI partnerships to
facilitate high diversity and accuracy in LLM-based text data generation. We
first examine two approaches to diversify text generation: 1) logit
suppression, which minimizes the generation of languages that have already been
frequently generated, and 2) temperature sampling, which flattens the token
sampling probability. We found that diversification approaches can increase
data diversity but often at the cost of data accuracy (i.e., text and labels
being appropriate for the target domain). To address this issue, we examined
two human interventions, 1) label replacement (LR), correcting misaligned
labels, and 2) out-of-scope filtering (OOSF), removing instances that are out
of the user's domain of interest or to which no considered label applies. With
oracle studies, we found that LR increases the absolute accuracy of models
trained with diversified datasets by 14.4%. Moreover, we found that some models
trained with data generated with LR interventions outperformed LLM-based
few-shot classification. In contrast, OOSF was not effective in increasing
model accuracy, implying the need for future work in human-in-the-loop text
data generation.
Text-only Domain Adaptation using Unified Speech-Text Representation in
  Transducer
Authors: name: Lu Huang, name: Boyu Li, name: Jun Zhang, name: Lu Lu, name: Zejun Ma
Published: 2023-06-07 00:33:02

Domain adaptation using text-only corpus is challenging in end-to-end(E2E)
speech recognition. Adaptation by synthesizing audio from text through TTS is
resource-consuming. We present a method to learn Unified Speech-Text
Representation in Conformer Transducer(USTR-CT) to enable fast domain
adaptation using the text-only corpus. Different from the previous textogram
method, an extra text encoder is introduced in our work to learn text
representation and is removed during inference, so there is no modification for
online deployment. To improve the efficiency of adaptation, single-step and
multi-step adaptations are also explored. The experiments on adapting
LibriSpeech to SPGISpeech show the proposed method reduces the word error
rate(WER) by relatively 44% on the target domain, which is better than those of
TTS method and textogram method. Also, it is shown the proposed method can be
combined with internal language model estimation(ILME) to further improve the
performance.
Certified Reasoning with Language Models
Authors: name: Gabriel Poesia, name: Kanishk Gandhi, name: Eric Zelikman, name: Noah D. Goodman
Published: 2023-06-06 21:49:00

Language models often achieve higher accuracy when reasoning step-by-step in
complex tasks. However, their reasoning can be unsound, inconsistent, or rely
on undesirable prior assumptions. To tackle these issues, we introduce a class
of tools for language models called guides that use state and incremental
constraints to guide generation. A guide can be invoked by the model to
constrain its own generation to a set of valid statements given by the tool. In
turn, the model's choices can change the guide's state. We show how a general
system for logical reasoning can be used as a guide, which we call LogicGuide.
Given a reasoning problem in natural language, a model can formalize its
assumptions for LogicGuide and then guarantee that its reasoning steps are
sound. In experiments with the PrOntoQA and ProofWriter reasoning datasets,
LogicGuide significantly improves the performance of GPT-3, GPT-3.5 Turbo and
LLaMA (accuracy gains up to 35%). LogicGuide also drastically reduces content
effects: the interference of prior and current assumptions that both humans and
language models have been shown to suffer from. Finally, we explore
bootstrapping LLaMA 13B from its own reasoning and find that LogicGuide is
critical: by training only on certified self-generated reasoning, LLaMA can
self-improve, avoiding learning from its own hallucinations.
Triggering Multi-Hop Reasoning for Question Answering in Language Models
  using Soft Prompts and Random Walks
Authors: name: Kanishka Misra, name: Cicero Nogueira dos Santos, name: Siamak Shakeri
Published: 2023-06-06 20:45:18

Despite readily memorizing world knowledge about entities, pre-trained
language models (LMs) struggle to compose together two or more facts to perform
multi-hop reasoning in question-answering tasks. In this work, we propose
techniques that improve upon this limitation by relying on random walks over
structured knowledge graphs. Specifically, we use soft prompts to guide LMs to
chain together their encoded knowledge by learning to map multi-hop questions
to random walk paths that lead to the answer. Applying our methods on two T5
LMs shows substantial improvements over standard tuning approaches in answering
questions that require 2-hop reasoning.
M^3IT: A Large-Scale Dataset towards Multi-Modal Multilingual
  Instruction Tuning
Authors: name: Lei Li, name: Yuwei Yin, name: Shicheng Li, name: Liang Chen, name: Peiyi Wang, name: Shuhuai Ren, name: Mukai Li, name: Yazheng Yang, name: Jingjing Xu, name: Xu Sun, name: Lingpeng Kong, name: Qi Liu
Published: 2023-06-07 12:35:37

Instruction tuning has significantly advanced large language models (LLMs)
such as ChatGPT, enabling them to align with human instructions across diverse
tasks. However, progress in open vision-language models (VLMs) has been limited
due to the scarcity of high-quality instruction datasets. To tackle this
challenge and promote research in the vision-language field, we introduce the
Multi-Modal, Multilingual Instruction Tuning (M^3IT) dataset, designed to
optimize VLM alignment with human instructions. Our M^3IT dataset comprises
40 carefully curated datasets, including 2.4 million instances and 400 manually
written task instructions, reformatted into a vision-to-text structure. Key
tasks are translated into 80 languages with an advanced translation system,
ensuring broader accessibility. M^3IT surpasses previous datasets regarding
task coverage, instruction number and instance scale. Moreover, we develop
Ying-VLM, a VLM model trained on our M^3IT dataset, showcasing its potential
to answer complex questions requiring world knowledge, generalize to unseen
video tasks, and comprehend unseen instructions in Chinese. To encourage
further research, we have open-sourced both the dataset and trained models.
Learning to Ground Instructional Articles in Videos through Narrations
Authors: name: Effrosyni Mavroudi, name: Triantafyllos Afouras, name: Lorenzo Torresani
Published: 2023-06-06 15:45:53

In this paper we present an approach for localizing steps of procedural
activities in narrated how-to videos. To deal with the scarcity of labeled data
at scale, we source the step descriptions from a language knowledge base
(wikiHow) containing instructional articles for a large variety of procedural
tasks. Without any form of manual supervision, our model learns to temporally
ground the steps of procedural articles in how-to videos by matching three
modalities: frames, narrations, and step descriptions. Specifically, our method
aligns steps to video by fusing information from two distinct pathways: i) {\em
direct} alignment of step descriptions to frames, ii) {\em indirect} alignment
obtained by composing steps-to-narrations with narrations-to-video
correspondences. Notably, our approach performs global temporal grounding of
all steps in an article at once by exploiting order information, and is trained
with step pseudo-labels which are iteratively refined and aggressively
filtered. In order to validate our model we introduce a new evaluation
benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of
HowTo100MA test server is accessible at
\url{https://eval.ai/web/challenges/challenge-page/2082.} with steps sourced
from wikiHow articles. Experiments on this benchmark as well as zero-shot
evaluations on CrossTask demonstrate that our multi-modality alignment yields
dramatic gains over several baselines and prior works. Finally, we show that
our inner module for matching narration-to-video outperforms by a large margin
the state of the art on the HTM-Align narration-video alignment benchmark.
Recognize Anything: A Strong Image Tagging Model
Authors: name: Youcai Zhang, name: Xinyu Huang, name: Jinyu Ma, name: Zhaoyang Li, name: Zhaochuan Luo, name: Yanchun Xie, name: Yuzhuo Qin, name: Tong Luo, name: Yaqian Li, name: Shilong Liu, name: Yandong Guo, name: Lei Zhang
Published: 2023-06-06 09:00:10

We present the Recognize Anything Model (RAM): a strong foundation model for
image tagging. RAM can recognize any common category with high accuracy. RAM
introduces a new paradigm for image tagging, leveraging large-scale image-text
pairs for training instead of manual annotations. The development of RAM
comprises four key steps. Firstly, annotation-free image tags are obtained at
scale through automatic text semantic parsing. Subsequently, a preliminary
model is trained for automatic annotation by unifying the caption and tagging
tasks, supervised by the original texts and parsed tags, respectively. Thirdly,
a data engine is employed to generate additional annotations and clean
incorrect ones. Lastly, the model is retrained with the processed data and
fine-tuned using a smaller but higher-quality dataset. We evaluate the tagging
capabilities of RAM on numerous benchmarks and observe impressive zero-shot
performance, significantly outperforming CLIP and BLIP. Remarkably, RAM even
surpasses the fully supervised manners and exhibits competitive performance
with the Google API. We are releasing the RAM at
https://recognize-anything.github.io/ to foster the advancements of large
models in computer vision.
Emergent Correspondence from Image Diffusion
Authors: name: Luming Tang, name: Menglin Jia, name: Qianqian Wang, name: Cheng Perng Phoo, name: Bharath Hariharan
Published: 2023-06-06 17:33:19

Finding correspondences between images is a fundamental problem in computer
vision. In this paper, we show that correspondence emerges in image diffusion
models without any explicit supervision. We propose a simple strategy to
extract this implicit knowledge out of diffusion networks as image features,
namely DIffusion FeaTures (DIFT), and use them to establish correspondences
between real images. Without any additional fine-tuning or supervision on the
task-specific data or annotations, DIFT is able to outperform both
weakly-supervised methods and competitive off-the-shelf features in identifying
semantic, geometric, and temporal correspondences. Particularly for semantic
correspondence, DIFT from Stable Diffusion is able to outperform DINO and
OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k
benchmark. It even outperforms the state-of-the-art supervised methods on 9 out
of 18 categories while remaining on par for the overall performance. Project
page: https://diffusionfeatures.github.io
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
Authors: name: Zhenhui Ye, name: Ziyue Jiang, name: Yi Ren, name: Jinglin Liu, name: Chen Zhang, name: Xiang Yin, name: Zejun Ma, name: Zhou Zhao
Published: 2023-06-06 08:50:13

We are interested in a novel task, namely low-resource text-to-talking
avatar. Given only a few-minute-long talking person video with the audio track
as the training data and arbitrary texts as the driving input, we aim to
synthesize high-quality talking portrait videos corresponding to the input
text. This task has broad application prospects in the digital human industry
but has not been technically achieved yet due to two challenges: (1) It is
challenging to mimic the timbre from out-of-domain audio for a traditional
multi-speaker Text-to-Speech system. (2) It is hard to render high-fidelity and
lip-synchronized talking avatars with limited training data. In this paper, we
introduce Adaptive Text-to-Talking Avatar (Ada-TTA), which (1) designs a
generic zero-shot multi-speaker TTS model that well disentangles the text
content, timbre, and prosody; and (2) embraces recent advances in neural
rendering to achieve realistic audio-driven talking face video generation. With
these designs, our method overcomes the aforementioned two challenges and
achieves to generate identity-preserving speech and realistic talking person
video. Experiments demonstrate that our method could synthesize realistic,
identity-preserving, and audio-visual synchronized talking avatar videos.
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive
  Bias
Authors: name: Ziyue Jiang, name: Yi Ren, name: Zhenhui Ye, name: Jinglin Liu, name: Chen Zhang, name: Qian Yang, name: Shengpeng Ji, name: Rongjie Huang, name: Chunfeng Wang, name: Xiang Yin, name: Zejun Ma, name: Zhou Zhao
Published: 2023-06-06 08:54:49

Scaling text-to-speech to a large and wild dataset has been proven to be
highly effective in achieving timbre and speech style generalization,
particularly in zero-shot TTS. However, previous works usually encode speech
into latent using audio codec and use autoregressive language models or
diffusion models to generate it, which ignores the intrinsic nature of speech
and may lead to inferior or uncontrollable results. We argue that speech can be
decomposed into several attributes (e.g., content, timbre, prosody, and phase)
and each of them should be modeled using a module with appropriate inductive
biases. From this perspective, we carefully design a novel and large zero-shot
TTS system called Mega-TTS, which is trained with large-scale wild data and
models different attributes in different ways: 1) Instead of using latent
encoded by audio codec as the intermediate feature, we still choose spectrogram
as it separates the phase and other attributes very well. Phase can be
appropriately constructed by the GAN-based vocoder and does not need to be
modeled by the language model. 2) We model the timbre using global vectors
since timbre is a global attribute that changes slowly over time. 3) We further
use a VQGAN-based acoustic model to generate the spectrogram and a latent code
language model to fit the distribution of prosody, since prosody changes
quickly over time in a sentence, and language models can capture both local and
long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K
hours of speech and evaluate its performance on unseen speakers. Experimental
results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on
zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior
naturalness, robustness, and speaker similarity due to the proper inductive
bias of each module. Audio samples are available at
https://mega-tts.github.io/demo-page.
LEACE: Perfect linear concept erasure in closed form
Authors: name: Nora Belrose, name: David Schneider-Joseph, name: Shauli Ravfogel, name: Ryan Cotterell, name: Edward Raff, name: Stella Biderman
Published: 2023-06-06 16:07:24

Concept erasure aims to remove specified features from a representation. It
can be used to improve fairness (e.g. preventing a classifier from using gender
or race) and interpretability (e.g. removing a concept to observe changes in
model behavior). In this paper, we introduce LEAst-squares Concept Erasure
(LEACE), a closed-form method which provably prevents all linear classifiers
from detecting a concept while inflicting the least possible damage to the
representation. We apply LEACE to large language models with a novel procedure
called "concept scrubbing," which erases target concept information from every
layer in the network. We demonstrate the usefulness of our method on two tasks:
measuring the reliance of language models on part-of-speech information, and
reducing gender bias in BERT embeddings. Code is available at
https://github.com/EleutherAI/concept-erasure.
Natural Language Commanding via Program Synthesis
Authors: name: Apurva Gandhi, name: Thong Q. Nguyen, name: Huitian Jiao, name: Robert Steen, name: Ameya Bhatawdekar
Published: 2023-06-06 07:28:49

We present Semantic Interpreter, a natural language-friendly AI system for
productivity software such as Microsoft Office that leverages large language
models (LLMs) to execute user intent across application features. While LLMs
are excellent at understanding user intent expressed as natural language, they
are not sufficient for fulfilling application-specific user intent that
requires more than text-to-text transformations. We therefore introduce the
Office Domain Specific Language (ODSL), a concise, high-level language
specialized for performing actions in and interacting with entities in Office
applications. Semantic Interpreter leverages an Analysis-Retrieval prompt
construction method with LLMs for program synthesis, translating natural
language user utterances to ODSL programs that can be transpiled to application
APIs and then executed. We focus our discussion primarily on a research
exploration for Microsoft PowerPoint.
Large Language Models of Code Fail at Completing Code with Potential
  Bugs
Authors: name: Tuan Dinh, name: Jinman Zhao, name: Samson Tan, name: Renato Negrinho, name: Leonard Lausen, name: Sheng Zha, name: George Karypis
Published: 2023-06-06 06:35:27

Large language models of code (Code-LLMs) have recently brought tremendous
advances to code completion, a fundamental feature of programming assistance
and code intelligence. However, most existing works ignore the possible
presence of bugs in the code context for generation, which are inevitable in
software development. Therefore, we introduce and study the buggy-code
completion problem, inspired by the realistic scenario of real-time code
suggestion where the code context contains potential bugs -- anti-patterns that
can become bugs in the completed program. To systematically study the task, we
introduce two datasets: one with synthetic bugs derived from semantics-altering
operator changes (buggy-HumanEval) and one with realistic bugs derived from
user submissions to coding problems (buggy-FixEval). We find that the presence
of potential bugs significantly degrades the generation performance of the
high-performing Code-LLMs. For instance, the passing rates of CodeGen-2B-mono
on test cases of buggy-HumanEval drop more than 50% given a single potential
bug in the context. Finally, we investigate several post-hoc methods for
mitigating the adverse effect of potential bugs and find that there remains a
large gap in post-mitigation performance.
A Static Evaluation of Code Completion by Large Language Models
Authors: name: Hantian Ding, name: Varun Kumar, name: Yuchen Tian, name: Zijian Wang, name: Rob Kwiatkowski, name: Xiaopeng Li, name: Murali Krishna Ramanathan, name: Baishakhi Ray, name: Parminder Bhatia, name: Sudipta Sengupta, name: Dan Roth, name: Bing Xiang
Published: 2023-06-05 19:23:34

Large language models trained on code have shown great potential to increase
productivity of software developers. Several execution-based benchmarks have
been proposed to evaluate functional correctness of model-generated code on
simple programming problems. Nevertheless, it is expensive to perform the same
evaluation on complex real-world projects considering the execution cost. On
the contrary, static analysis tools such as linters, which can detect errors
without running the program, haven't been well explored for evaluating code
generation models. In this work, we propose a static evaluation framework to
quantify static errors in Python code completions, by leveraging Abstract
Syntax Trees. Compared with execution-based evaluation, our method is not only
more efficient, but also applicable to code in the wild. For experiments, we
collect code context from open source repos to generate one million function
bodies using public models. Our static analysis reveals that Undefined Name and
Unused Variable are the most common errors among others made by language
models. Through extensive studies, we also show the impact of sampling
temperature, model size, and context on static errors in code completions.
Deductive Verification of Chain-of-Thought Reasoning
Authors: name: Zhan Ling, name: Yunhao Fang, name: Xuanlin Li, name: Zhiao Huang, name: Mingu Lee, name: Roland Memisevic, name: Hao Su
Published: 2023-06-06 17:18:56

Large Language Models (LLMs) significantly benefit from Chain-of-Thought
(CoT) prompting in performing various reasoning tasks. While CoT allows models
to produce more comprehensive reasoning processes, its emphasis on intermediate
reasoning steps can inadvertently introduce hallucinations and accumulated
errors, thereby limiting models' ability to solve complex reasoning tasks.
Inspired by how humans engage in careful and meticulous deductive logical
reasoning processes to solve tasks, we seek to enable language models to
perform explicit and rigorous deductive reasoning, and also ensure the
trustworthiness of their reasoning process through self-verification. However,
directly verifying the validity of an entire deductive reasoning process is
challenging, even with advanced models like ChatGPT. In light of this, we
propose to decompose a reasoning verification process into a series of
step-by-step subprocesses, each only receiving their necessary context and
premises. To facilitate this procedure, we propose Natural Program, a natural
language-based deductive reasoning format. Our approach enables models to
generate precise reasoning steps where subsequent steps are more rigorously
grounded on prior steps. It also empowers language models to carry out
reasoning self-verification in a step-by-step manner. By integrating this
verification process into each deductive reasoning stage, we significantly
enhance the rigor and trustfulness of generated reasoning steps. Along this
process, we also improve the answer correctness on complex reasoning tasks.
Code will be released at https://github.com/lz1oceani/verify_cot.
Neuralangelo: High-Fidelity Neural Surface Reconstruction
Authors: name: Zhaoshuo Li, name: Thomas Müller, name: Alex Evans, name: Russell H. Taylor, name: Mathias Unberath, name: Ming-Yu Liu, name: Chen-Hsuan Lin
Published: 2023-06-05 17:59:57

Neural surface reconstruction has been shown to be powerful for recovering
dense 3D surfaces via image-based neural rendering. However, current methods
struggle to recover detailed structures of real-world scenes. To address the
issue, we present Neuralangelo, which combines the representation power of
multi-resolution 3D hash grids with neural surface rendering. Two key
ingredients enable our approach: (1) numerical gradients for computing
higher-order derivatives as a smoothing operation and (2) coarse-to-fine
optimization on the hash grids controlling different levels of details. Even
without auxiliary inputs such as depth, Neuralangelo can effectively recover
dense 3D surface structures from multi-view images with fidelity significantly
surpassing previous methods, enabling detailed large-scale scene reconstruction
from RGB video captures.