Orionnnn/hub-stats huggingface dataset:
GeneCIS: A Benchmark for General Conditional Image Similarity
Authors: name: Sagar Vaze, name: Nicolas Carion, name: Ishan Misra
Published: 2023-06-13 17:59:58

We argue that there are many notions of 'similarity' and that models, like
humans, should be able to adapt to these dynamically. This contrasts with most
representation learning methods, supervised or self-supervised, which learn a
fixed embedding function and hence implicitly assume a single notion of
similarity. For instance, models trained on ImageNet are biased towards object
categories, while a user might prefer the model to focus on colors, textures or
specific elements in the scene. In this paper, we propose the GeneCIS
('genesis') benchmark, which measures models' ability to adapt to a range of
similarity conditions. Extending prior work, our benchmark is designed for
zero-shot evaluation only, and hence considers an open-set of similarity
conditions. We find that baselines from powerful CLIP models struggle on
GeneCIS and that performance on the benchmark is only weakly correlated with
ImageNet accuracy, suggesting that simply scaling existing methods is not
fruitful. We further propose a simple, scalable solution based on automatically
mining information from existing image-caption datasets. We find our method
offers a substantial boost over the baselines on GeneCIS, and further improves
zero-shot performance on related image retrieval benchmarks. In fact, though
evaluated zero-shot, our model surpasses state-of-the-art supervised models on
MIT-States. Project page at https://sgvaze.github.io/genecis/.
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
Authors: name: Arnav Chavan, name: Zhuang Liu, name: Deepak Gupta, name: Eric Xing, name: Zhiqiang Shen
Published: 2023-06-13 17:59:32

We present Generalized LoRA (GLoRA), an advanced approach for universal
parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
GLoRA employs a generalized prompt module to optimize pre-trained model weights
and adjust intermediate activations, providing more flexibility and capability
across diverse tasks and datasets. Moreover, GLoRA facilitates efficient
parameter adaptation by employing a scalable, modular, layer-wise structure
search that learns individual adapter of each layer. Originating from a unified
mathematical formulation, GLoRA exhibits strong transfer learning, few-shot
learning and domain generalization abilities, as it adjusts to new tasks
through additional dimensions on weights and activations. Comprehensive
experiments demonstrate that GLoRA outperforms all previous methods in natural,
specialized, and structured benchmarks, achieving superior accuracy with fewer
parameters and computations on various datasets. Furthermore, our structural
re-parameterization design ensures that GLoRA incurs no extra inference cost,
rendering it a practical solution for resource-limited applications. Code is
available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.
WebGLM: Towards An Efficient Web-Enhanced Question Answering System with
  Human Preferences
Authors: name: Xiao Liu, name: Hanyu Lai, name: Hao Yu, name: Yifan Xu, name: Aohan Zeng, name: Zhengxiao Du, name: Peng Zhang, name: Yuxiao Dong, name: Jie Tang
Published: 2023-06-13 16:57:53

We present WebGLM, a web-enhanced question-answering system based on the
General Language Model (GLM). Its goal is to augment a pre-trained large
language model (LLM) with web search and retrieval capabilities while being
efficient for real-world deployments. To achieve this, we develop WebGLM with
strategies for the LLM-augmented retriever, bootstrapped generator, and human
preference-aware scorer. Specifically, we identify and address the limitations
of WebGPT (OpenAI), through which WebGLM is enabled with accuracy, efficiency,
and cost-effectiveness advantages. In addition, we propose systematic criteria
for evaluating web-enhanced QA systems. We conduct multi-dimensional human
evaluation and quantitative ablation studies, which suggest the outperformance
of the proposed WebGLM designs over existing systems. WebGLM with the
10-billion-parameter GLM (10B) is shown to perform better than the
similar-sized WebGPT (13B) and even comparably to WebGPT (175B) in human
evaluation. The code, demo, and data are at
https://github.com/THUDM/WebGLM.
STUDY: Socially Aware Temporally Casual Decoder Recommender Systems
Authors: name: Eltayeb Ahmed, name: Diana Mincu, name: Lauren Harrell, name: Katherine Heller, name: Subhrajit Roy
Published: 2023-06-02 14:47:56

With the overwhelming amount of data available both on and offline today,
recommender systems have become much needed to help users find items tailored
to their interests. When social network information exists there are methods
that utilize this information to make better recommendations, however the
methods are often clunky with complex architectures and training procedures.
Furthermore many of the existing methods utilize graph neural networks which
are notoriously difficult to train. To address this, we propose Socially-aware
Temporally caUsal Decoder recommender sYstems (STUDY). STUDY does joint
inference over groups of users who are adjacent in the social network graph
using a single forward pass of a modified transformer decoder network. We test
our method in a school-based educational content setting, using classroom
structure to define social networks. Our method outperforms both social and
sequential methods while maintaining the design simplicity of a single
homogeneous network that models all interactions in the data. We also carry out
ablation studies to understand the drivers of our performance gains and find
that our model depends on leveraging a social network structure that
effectively models the similarities in user behavior.
Image Captioners Are Scalable Vision Learners Too
Authors: name: Michael Tschannen, name: Manoj Kumar, name: Andreas Steiner, name: Xiaohua Zhai, name: Neil Houlsby, name: Lucas Beyer
Published: 2023-06-13 17:18:01

Contrastive pretraining on image-text pairs from the web is one of the most
popular large-scale pretraining strategies for vision backbones, especially in
the context of large multimodal models. At the same time, image captioning on
this type of data is commonly considered an inferior pretraining strategy. In
this paper, we perform a fair comparison of these two pretraining strategies,
carefully matching training data, compute, and model capacity. Using a standard
encoder-decoder transformer, we find that captioning alone is surprisingly
effective: on classification tasks, captioning produces vision encoders
competitive with contrastively pretrained encoders, while surpassing them on
vision & language tasks. We further analyze the effect of the model
architecture and scale, as well as the pretraining data on the representation
quality, and find that captioning exhibits the same or better scaling behavior
along these axes. Overall our results show that plain image captioning is a
more powerful pretraining strategy than was previously believed.
Instant Multi-View Head Capture through Learnable Registration
Authors: name: Timo Bolkart, name: Tianye Li, name: Michael J. Black
Published: 2023-06-12 21:45:18

Existing methods for capturing datasets of 3D heads in dense semantic
correspondence are slow, and commonly address the problem in two separate
steps; multi-view stereo (MVS) reconstruction followed by non-rigid
registration. To simplify this process, we introduce TEMPEH (Towards Estimation
of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads
in dense correspondence from calibrated multi-view images. Registering datasets
of 3D scans typically requires manual parameter tuning to find the right
balance between accurately fitting the scans surfaces and being robust to
scanning noise and outliers. Instead, we propose to jointly register a 3D head
dataset while training TEMPEH. Specifically, during training we minimize a
geometric loss commonly used for surface registration, effectively leveraging
TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric
feature representation that samples and fuses features from each view using
camera calibration information. To account for partial occlusions and a large
capture volume that enables head movements, we use view- and surface-aware
feature fusion, and a spatial transformer-based head localization module,
respectively. We use raw MVS scans as supervision during training, but, once
trained, TEMPEH directly predicts 3D heads in dense correspondence without
requiring scans. Predicting one head takes about 0.3 seconds with a median
reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art.
This enables the efficient capture of large datasets containing multiple people
and diverse facial motions. Code, model, and data are publicly available at
https://tempeh.is.tue.mpg.de.
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
Authors: name: Shuai Yang, name: Yifan Zhou, name: Ziwei Liu, name: Chen Change Loy
Published: 2023-06-13 17:52:23

Large text-to-image diffusion models have exhibited impressive proficiency in
generating high-quality images. However, when applying these models to video
domain, ensuring temporal consistency across video frames remains a formidable
challenge. This paper proposes a novel zero-shot text-guided video-to-video
translation framework to adapt image models to videos. The framework includes
two parts: key frame translation and full video translation. The first part
uses an adapted diffusion model to generate key frames, with hierarchical
cross-frame constraints applied to enforce coherence in shapes, textures and
colors. The second part propagates the key frames to other frames with
temporal-aware patch matching and frame blending. Our framework achieves global
style and local texture temporal consistency at a low cost (without re-training
or optimization). The adaptation is compatible with existing image diffusion
techniques, allowing our framework to take advantage of them, such as
customizing a specific subject with LoRA, and introducing extra spatial
guidance with ControlNet. Extensive experimental results demonstrate the
effectiveness of our proposed framework over existing methods in rendering
high-quality and temporally-coherent videos.
3D molecule generation by denoising voxel grids
Authors: name: Pedro O. Pinheiro, name: Joshua Rackers, name: Joseph Kleinhenz, name: Michael Maser, name: Omar Mahmood, name: Andrew Martin Watkins, name: Stephen Ra, name: Vishnu Sresht, name: Saeed Saremi
Published: 2023-06-13 00:38:51

We propose a new score-based approach to generate 3D molecules represented as
atomic densities on regular grids. First, we train a denoising neural network
that learns to map from a smooth distribution of noisy molecules to the
distribution of real molecules. Then, we follow the neural empirical Bayes
framework [Saremi and Hyvarinen, 2019] and generate molecules in two steps: (i)
sample noisy density grids from a smooth distribution via underdamped Langevin
Markov chain Monte Carlo, and (ii) recover the ``clean'' molecule by denoising
the noisy grid with a single step. Our method, VoxMol, generates molecules in a
fundamentally different way than the current state of the art (i.e., diffusion
models applied to atom point clouds). It differs in terms of the data
representation, the noise model, the network architecture and the generative
modeling algorithm. VoxMol achieves comparable results to state of the art on
unconditional 3D molecule generation while being simpler to train and faster to
generate molecules.
AniFaceDrawing: Anime Portrait Exploration during Your Sketching
Authors: name: Zhengyu Huang, name: Haoran Xie, name: Tsukasa Fukusato, name: Kazunori Miyata
Published: 2023-06-13 00:43:47

In this paper, we focus on how artificial intelligence (AI) can be used to
assist users in the creation of anime portraits, that is, converting rough
sketches into anime portraits during their sketching process. The input is a
sequence of incomplete freehand sketches that are gradually refined stroke by
stroke, while the output is a sequence of high-quality anime portraits that
correspond to the input sketches as guidance. Although recent GANs can generate
high quality images, it is a challenging problem to maintain the high quality
of generated images from sketches with a low degree of completion due to
ill-posed problems in conditional image generation. Even with the latest
sketch-to-image (S2I) technology, it is still difficult to create high-quality
images from incomplete rough sketches for anime portraits since anime style
tend to be more abstract than in realistic style. To address this issue, we
adopt a latent space exploration of StyleGAN with a two-stage training
strategy. We consider the input strokes of a freehand sketch to correspond to
edge information-related attributes in the latent structural code of StyleGAN,
and term the matching between strokes and these attributes stroke-level
disentanglement. In the first stage, we trained an image encoder with the
pre-trained StyleGAN model as a teacher encoder. In the second stage, we
simulated the drawing process of the generated images without any additional
data (labels) and trained the sketch encoder for incomplete progressive
sketches to generate high-quality portrait images with feature alignment to the
disentangled representations in the teacher encoder. We verified the proposed
progressive S2I system with both qualitative and quantitative evaluations and
achieved high-quality anime portraits from incomplete progressive sketches. Our
user study proved its effectiveness in art creation assistance for the anime
style.
Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for
  Speech Understanding
Authors: name: Mingqiu Wang, name: Izhak Shafran, name: Hagen Soltau, name: Wei Han, name: Yuan Cao, name: Dian Yu, name: Laurent El Shafey
Published: 2023-06-08 22:33:22

Large Language Models (LLMs) have been applied in the speech domain, often
incurring a performance drop due to misaligned between speech and language
representations. To bridge this gap, we propose a joint speech and language
model (SLM) using a Speech2Text adapter, which maps speech into text token
embedding space without speech information loss. Additionally, using a
CTC-based blank-filtering, we can reduce the speech sequence length to that of
text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the
dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to
address errors on rare entities, we augment SLM with a Speech2Entity retriever,
which uses speech to retrieve relevant entities, and then adds them to the
original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the
DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with
the dialog understanding task improves the ASR performance from 9.4% to 8.5%
WER.
GPT-Calls: Enhancing Call Segmentation and Tagging by Generating
  Synthetic Conversations via Large Language Models
Authors: name: Itzik Malkiel, name: Uri Alon, name: Yakir Yehuda, name: Shahar Keren, name: Oren Barkan, name: Royi Ronen, name: Noam Koenigstein
Published: 2023-06-09 15:47:22

Transcriptions of phone calls are of significant value across diverse fields,
such as sales, customer service, healthcare, and law enforcement. Nevertheless,
the analysis of these recorded conversations can be an arduous and
time-intensive process, especially when dealing with extended or multifaceted
dialogues. In this work, we propose a novel method, GPT-distilled Calls
Segmentation and Tagging (GPT-Calls), for efficient and accurate call
segmentation and topic extraction. GPT-Calls is composed of offline and online
phases. The offline phase is applied once to a given list of topics and
involves generating a distribution of synthetic sentences for each topic using
a GPT model and extracting anchor vectors. The online phase is applied to every
call separately and scores the similarity between the transcripted conversation
and the topic anchors found in the offline phase. Then, time domain analysis is
applied to the similarity scores to group utterances into segments and tag them
with topics. The proposed paradigm provides an accurate and efficient method
for call segmentation and topic extraction that does not require labeled data,
thus making it a versatile approach applicable to various domains. Our
algorithm operates in production under Dynamics 365 Sales Conversation
Intelligence, and our research is based on real sales conversations gathered
from various Dynamics 365 Sales tenants.
SayTap: Language to Quadrupedal Locomotion
Authors: name: Yujin Tang, name: Wenhao Yu, name: Jie Tan, name: Heiga Zen, name: Aleksandra Faust, name: Tatsuya Harada
Published: 2023-06-13 07:09:11

Large language models (LLMs) have demonstrated the potential to perform
high-level planning. Yet, it remains a challenge for LLMs to comprehend
low-level commands, such as joint angle targets or motor torques. This paper
proposes an approach to use foot contact patterns as an interface that bridges
human commands in natural language and a locomotion controller that outputs
these low-level commands. This results in an interactive system for quadrupedal
robots that allows the users to craft diverse locomotion behaviors flexibly. We
contribute an LLM prompt design, a reward function, and a method to expose the
controller to the feasible distribution of contact patterns. The results are a
controller capable of achieving diverse locomotion patterns that can be
transferred to real robot hardware. Compared with other design choices, the
proposed approach enjoys more than 50% success rate in predicting the correct
contact patterns and can solve 10 more tasks out of a total of 30 tasks. Our
project site is: https://saytap.github.io.
TART: A plug-and-play Transformer module for task-agnostic reasoning
Authors: name: Kush Bhatia, name: Avanika Narayan, name: Christopher De Sa, name: Christopher Ré
Published: 2023-06-13 04:37:00

Large language models (LLMs) exhibit in-context learning abilities which
enable the same model to perform several tasks without any task-specific
training. In contrast, traditional adaptation approaches, such as fine-tuning,
modify the underlying models for each specific task. In-context learning,
however, consistently underperforms task-specific tuning approaches even when
presented with the same examples. While most existing approaches (e.g., prompt
engineering) focus on the LLM's learned representations to patch this
performance gap, our analysis actually reveal that LLM representations contain
sufficient information to make good predictions. As such, we focus on the LLM's
reasoning abilities and demonstrate that this performance gap exists due to
their inability to perform simple probabilistic reasoning tasks. This raises an
intriguing question: Are LLMs actually capable of learning how to reason in a
task-agnostic manner? We answer this in the affirmative and propose TART which
generically improves an LLM's reasoning abilities using a synthetically trained
Transformer-based reasoning module. TART trains this reasoning module in a
task-agnostic manner using only synthetic logistic regression tasks and
composes it with an arbitrary real-world pre-trained model without any
additional training. With a single inference module, TART improves performance
across different model families (GPT-Neo, Pythia, BLOOM), model sizes (100M -
6B), tasks (14 NLP binary classification tasks), and even across different
modalities (audio and vision). Additionally, on the RAFT Benchmark, TART
improves GPT-Neo (125M)'s performance such that it outperforms BLOOM (176B),
and is within 4% of GPT-3 (175B). Our code and models are available at
https://github.com/HazyResearch/TART .
arXiVeri: Automatic table verification with GPT
Authors: name: Gyungin Shin, name: Weidi Xie, name: Samuel Albanie
Published: 2023-06-13 17:59:57

Without accurate transcription of numerical data in scientific documents, a
scientist cannot draw accurate conclusions. Unfortunately, the process of
copying numerical data from one paper to another is prone to human error. In
this paper, we propose to meet this challenge through the novel task of
automatic table verification (AutoTV), in which the objective is to verify the
accuracy of numerical data in tables by cross-referencing cited sources. To
support this task, we propose a new benchmark, arXiVeri, which comprises
tabular data drawn from open-access academic papers on arXiv. We introduce
metrics to evaluate the performance of a table verifier in two key areas: (i)
table matching, which aims to identify the source table in a cited document
that corresponds to a target table, and (ii) cell matching, which aims to
locate shared cells between a target and source table and identify their row
and column indices accurately. By leveraging the flexible capabilities of
modern large language models (LLMs), we propose simple baselines for table
verification. Our findings highlight the complexity of this task, even for
state-of-the-art LLMs like OpenAI's GPT-4. The code and benchmark will be made
publicly available.
Transformers learn through gradual rank increase
Authors: name: Enric Boix-Adsera, name: Etai Littwin, name: Emmanuel Abbe, name: Samy Bengio, name: Joshua Susskind
Published: 2023-06-12 11:41:42

We identify incremental learning dynamics in transformers, where the
difference between trained and initial weights progressively increases in rank.
We rigorously prove this occurs under the simplifying assumptions of diagonal
weight matrices and small initialization. Our experiments support the theory
and also show that phenomenon can occur in practice without the simplifying
assumptions.
Scalable 3D Captioning with Pretrained Models
Authors: name: Tiange Luo, name: Chris Rockwell, name: Honglak Lee, name: Justin Johnson
Published: 2023-06-12 17:59:03

We introduce Cap3D, an automatic approach for generating descriptive text for
3D objects. This approach utilizes pretrained models from image captioning,
image-text alignment, and LLM to consolidate captions from multiple views of a
3D asset, completely side-stepping the time-consuming and costly process of
manual annotation. We apply Cap3D to the recently introduced large-scale 3D
dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted
using 41k human annotations from the same dataset, demonstrates that Cap3D
surpasses human-authored descriptions in terms of quality, cost, and speed.
Through effective prompt engineering, Cap3D rivals human performance in
generating geometric descriptions on 17k collected annotations from the ABO
dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions,
and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E,
and DreamFusion.