Orionnnn/hub-stats huggingface dataset:
FitMe: Deep Photorealistic 3D Morphable Model Avatars
Authors: name: Alexandros Lattas, name: Stylianos Moschoglou, name: Stylianos Ploumpis, name: Baris Gecer, name: Jiankang Deng, name: Stefanos Zafeiriou
Published: 2023-05-16 17:42:45

In this paper, we introduce FitMe, a facial reflectance model and a
differentiable rendering optimization pipeline, that can be used to acquire
high-fidelity renderable human avatars from single or multiple images. The
model consists of a multi-modal style-based generator, that captures facial
appearance in terms of diffuse and specular reflectance, and a PCA-based shape
model. We employ a fast differentiable rendering process that can be used in an
optimization pipeline, while also achieving photorealistic facial shading. Our
optimization process accurately captures both the facial reflectance and shape
in high-detail, by exploiting the expressivity of the style-based latent
representation and of our shape model. FitMe achieves state-of-the-art
reflectance acquisition and identity preservation on single "in-the-wild"
facial images, while it produces impressive scan-like results, when given
multiple unconstrained facial images pertaining to the same identity. In
contrast with recent implicit avatar reconstructions, FitMe requires only one
minute and produces relightable mesh and texture-based avatars, that can be
used by end-user applications.
Online Continual Learning Without the Storage Constraint
Authors: name: Ameya Prabhu, name: Zhipeng Cai, name: Puneet Dokania, name: Philip Torr, name: Vladlen Koltun, name: Ozan Sener
Published: 2023-05-16 08:03:07

Online continual learning (OCL) research has primarily focused on mitigating
catastrophic forgetting with fixed and limited storage allocation throughout
the agent's lifetime. However, the growing affordability of data storage
highlights a broad range of applications that do not adhere to these
assumptions. In these cases, the primary concern lies in managing computational
expenditures rather than storage. In this paper, we target such settings,
investigating the online continual learning problem by relaxing storage
constraints and emphasizing fixed, limited economical budget. We provide a
simple algorithm that can compactly store and utilize the entirety of the
incoming data stream under tiny computational budgets using a kNN classifier
and universal pre-trained feature extractors. Our algorithm provides a
consistency property attractive to continual learning: It will never forget
past seen data. We set a new state of the art on two large-scale OCL datasets:
Continual LOCalization (CLOC), which has 39M images over 712 classes, and
Continual Google Landmarks V2 (CGLM), which has 580K images over 10,788 classes
-- beating methods under far higher computational budgets than ours in terms of
both reducing catastrophic forgetting of past data and quickly adapting to
rapidly changing data streams. We provide code to reproduce our results at
https://github.com/drimpossible/ACM.
Understanding 3D Object Interaction from a Single Image
Authors: name: Shengyi Qian, name: David F. Fouhey
Published: 2023-05-16 17:59:26

Humans can easily understand a single image as depicting multiple potential
objects permitting interaction. We use this skill to plan our interactions with
the world and accelerate understanding new objects without engaging in
interaction. In this paper, we would like to endow machines with the similar
ability, so that intelligent agents can better explore the 3D scene or
manipulate objects. Our approach is a transformer-based model that predicts the
3D location, physical properties and affordance of objects. To power this
model, we collect a dataset with Internet videos, egocentric videos and indoor
images to train and validate our approach. Our model yields strong performance
on our data, and generalizes well to robotics data.
Towards Expert-Level Medical Question Answering with Large Language
  Models
Authors: name: Karan Singhal, name: Tao Tu, name: Juraj Gottweis, name: Rory Sayres, name: Ellery Wulczyn, name: Le Hou, name: Kevin Clark, name: Stephen Pfohl, name: Heather Cole-Lewis, name: Darlene Neal, name: Mike Schaekermann, name: Amy Wang, name: Mohamed Amin, name: Sami Lachgar, name: Philip Mansfield, name: Sushant Prakash, name: Bradley Green, name: Ewa Dominowska, name: Blaise Aguera y Arcas, name: Nenad Tomasev, name: Yun Liu, name: Renee Wong, name: Christopher Semturs, name: S. Sara Mahdavi, name: Joelle Barral, name: Dale Webster, name: Greg S. Corrado, name: Yossi Matias, name: Shekoofeh Azizi, name: Alan Karthikesalingam, name: Vivek Natarajan
Published: 2023-05-16 17:11:29

Recent artificial intelligence (AI) systems have reached milestones in "grand
challenges" ranging from Go to protein-folding. The capability to retrieve
medical knowledge, reason over it, and answer medical questions comparably to
physicians has long been viewed as one such grand challenge.
  Large language models (LLMs) have catalyzed significant progress in medical
question answering; Med-PaLM was the first model to exceed a "passing" score in
US Medical Licensing Examination (USMLE) style questions with a score of 67.2%
on the MedQA dataset. However, this and other prior work suggested significant
room for improvement, especially when models' answers were compared to
clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by
leveraging a combination of base LLM improvements (PaLM 2), medical domain
finetuning, and prompting strategies including a novel ensemble refinement
approach.
  Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM
by over 19% and setting a new state-of-the-art. We also observed performance
approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
clinical topics datasets.
  We performed detailed human evaluations on long-form questions along multiple
axes relevant to clinical applications. In pairwise comparative ranking of 1066
consumer medical questions, physicians preferred Med-PaLM 2 answers to those
produced by physicians on eight of nine axes pertaining to clinical utility (p
< 0.001). We also observed significant improvements compared to Med-PaLM on
every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form
"adversarial" questions to probe LLM limitations.
  While further studies are necessary to validate the efficacy of these models
in real-world settings, these results highlight rapid progress towards
physician-level performance in medical question answering.
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Authors: name: Ziheng Li, name: Shaohan Huang, name: Zihan Zhang, name: Zhi-Hong Deng, name: Qiang Lou, name: Haizhen Huang, name: Jian Jiao, name: Furu Wei, name: Weiwei Deng, name: Qi Zhang
Published: 2023-05-16 03:53:30

Recent studies have shown that dual encoder models trained with the
sentence-level translation ranking task are effective methods for cross-lingual
sentence embedding. However, our research indicates that token-level alignment
is also crucial in multilingual scenarios, which has not been fully explored
previously. Based on our findings, we propose a dual-alignment pre-training
(DAP) framework for cross-lingual sentence embedding that incorporates both
sentence-level and token-level alignment. To achieve this, we introduce a novel
representation translation learning (RTL) task, where the model learns to use
one-side contextualized token representation to reconstruct its translation
counterpart. This reconstruction objective encourages the model to embed
translation information into the token representation. Compared to other
token-level alignment methods such as translation language modeling, RTL is
more suitable for dual encoder architectures and is computationally efficient.
Extensive experiments on three sentence-level cross-lingual benchmarks
demonstrate that our approach can significantly improve sentence embedding. Our
code is available at https://github.com/ChillingDream/DAP.
Pre-Training to Learn in Context
Authors: name: Yuxian Gu, name: Li Dong, name: Furu Wei, name: Minlie Huang
Published: 2023-05-16 03:38:06

In-context learning, where pre-trained language models learn to perform tasks
from task examples and instructions in their contexts, has attracted much
attention in the NLP community. However, the ability of in-context learning is
not fully exploited because language models are not explicitly trained to learn
in context. To this end, we propose PICL (Pre-training for In-Context
Learning), a framework to enhance the language models' in-context learning
ability by pre-training the model on a large collection of "intrinsic tasks" in
the general plain-text corpus using the simple language modeling objective.
PICL encourages the model to infer and perform tasks by conditioning on the
contexts while maintaining task generalization of pre-trained models. We
evaluate the in-context learning performance of the model trained with PICL on
seven widely-used text classification datasets and the Super-NaturalInstrctions
benchmark, which contains 100+ NLP tasks formulated to text generation. Our
experiments show that PICL is more effective and task-generalizable than a
range of baselines, outperforming larger language models with nearly 4x
parameters. The code is publicly available at https://github.com/thu-coai/PICL.
SoundStorm: Efficient Parallel Audio Generation
Authors: name: Zalán Borsos, name: Matt Sharifi, name: Damien Vincent, name: Eugene Kharitonov, name: Neil Zeghidour, name: Marco Tagliasacchi
Published: 2023-05-16 17:41:25

We present SoundStorm, a model for efficient, non-autoregressive audio
generation. SoundStorm receives as input the semantic tokens of AudioLM, and
relies on bidirectional attention and confidence-based parallel decoding to
generate the tokens of a neural audio codec. Compared to the autoregressive
generation approach of AudioLM, our model produces audio of the same quality
and with higher consistency in voice and acoustic conditions, while being two
orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5
seconds on a TPU-v4. We demonstrate the ability of our model to scale audio
generation to longer sequences by synthesizing high-quality, natural dialogue
segments, given a transcript annotated with speaker turns and a short prompt
with the speakers' voices.
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Authors: name: Tong Wu, name: Zhihao Fan, name: Xiao Liu, name: Yeyun Gong, name: Yelong Shen, name: Jian Jiao, name: Hai-Tao Zheng, name: Juntao Li, name: Zhongyu Wei, name: Jian Guo, name: Nan Duan, name: Weizhu Chen
Published: 2023-05-16 15:10:22

Diffusion models have gained significant attention in the realm of image
generation due to their exceptional performance. Their success has been
recently expanded to text generation via generating all tokens within a
sequence concurrently. However, natural language exhibits a far more pronounced
sequential dependency in comparison to images, and the majority of existing
language models are trained utilizing a left-to-right auto-regressive approach.
To account for the inherent sequential characteristic of natural language, we
introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
the generation of tokens on the right depends on the generated ones on the
left, a mechanism achieved through employing a dynamic number of denoising
steps that vary based on token position. This results in tokens on the left
undergoing fewer denoising steps than those on the right, thereby enabling them
to generate earlier and subsequently influence the generation of tokens on the
right. In a series of experiments on various text generation tasks including
text summarization, machine translation, and common sense generation,
AR-Diffusion clearly demonstrated the superiority over existing diffusion
language models and that it can be 100timessim600times faster when
achieving comparable results. Our code will be publicly released.
Make-An-Animation: Large-Scale Text-conditional 3D Human Motion
  Generation
Authors: name: Samaneh Azadi, name: Akbar Shah, name: Thomas Hayes, name: Devi Parikh, name: Sonal Gupta
Published: 2023-05-16 17:58:43

Text-guided human motion generation has drawn significant interest because of
its impactful applications spanning animation and robotics. Recently,
application of diffusion models for motion generation has enabled improvements
in the quality of generated motions. However, existing approaches are limited
by their reliance on relatively small-scale motion capture data, leading to
poor performance on more diverse, in-the-wild prompts. In this paper, we
introduce Make-An-Animation, a text-conditioned human motion generation model
which learns more diverse poses and prompts from large-scale image-text
datasets, enabling significant improvement in performance over prior works.
Make-An-Animation is trained in two stages. First, we train on a curated
large-scale dataset of (text, static pseudo-pose) pairs extracted from
image-text datasets. Second, we fine-tune on motion capture data, adding
additional layers to model the temporal dimension. Unlike prior diffusion
models for motion generation, Make-An-Animation uses a U-Net architecture
similar to recent text-to-video generation models. Human evaluation of motion
realism and alignment with input text shows that our model reaches
state-of-the-art performance on text-to-motion generation.
Improved baselines for vision-language pre-training
Authors: name: Enrico Fini, name: Pietro Astolfi, name: Adriana Romero-Soriano, name: Jakob Verbeek, name: Michal Drozdzal
Published: 2023-05-15 14:31:49

Contrastive learning has emerged as an efficient framework to learn
multimodal representations. CLIP, a seminal work in this area, achieved
impressive results by training on paired image-text data using the contrastive
loss. Recent work claims improvements over CLIP using additional
non-contrastive losses inspired from self-supervised learning. However, it is
sometimes hard to disentangle the contribution of these additional losses from
other implementation details, e.g., data augmentation or regularization
techniques, used to train the model. To shed light on this matter, in this
paper, we first propose, implement and evaluate several baselines obtained by
combining contrastive learning with recent advances in self-supervised
learning. In particular, we use the loss functions that were proven successful
for visual self-supervised learning to align image and text modalities. We find
that these baselines outperform a basic implementation of CLIP. However, when a
stronger training recipe is employed, the advantage disappears. Indeed, we find
that a simple CLIP baseline can also be improved substantially, up to a 25%
relative improvement on downstream zero-shot tasks, by using well-known
training techniques that are popular in other subfields. Moreover, we discover
that it is enough to apply image and text augmentations to make up for most of
the improvement attained by prior works. With our improved training recipe for
CLIP, we obtain state-of-the-art performance on four standard datasets, and
consistently outperform prior work (up to +4% on the largest dataset), while
being substantially simpler.
AutoRecon: Automated 3D Object Discovery and Reconstruction
Authors: name: Yuang Wang, name: Xingyi He, name: Sida Peng, name: Haotong Lin, name: Hujun Bao, name: Xiaowei Zhou
Published: 2023-05-15 17:16:46

A fully automated object reconstruction pipeline is crucial for digital
content creation. While the area of 3D reconstruction has witnessed profound
developments, the removal of background to obtain a clean object model still
relies on different forms of manual labor, such as bounding box labeling, mask
annotations, and mesh manipulations. In this paper, we propose a novel
framework named AutoRecon for the automated discovery and reconstruction of an
object from multi-view images. We demonstrate that foreground objects can be
robustly located and segmented from SfM point clouds by leveraging
self-supervised 2D vision transformer features. Then, we reconstruct decomposed
neural scene representations with dense supervision provided by the decomposed
point clouds, resulting in accurate object reconstruction and segmentation.
Experiments on the DTU, BlendedMVS and CO3D-V2 datasets demonstrate the
effectiveness and robustness of AutoRecon.
Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
Authors: name: Zhengxuan Wu, name: Atticus Geiger, name: Christopher Potts, name: Noah D. Goodman
Published: 2023-05-15 17:15:40

Obtaining human-interpretable explanations of large, general-purpose language
models is an urgent goal for AI safety. However, it is just as important that
our interpretability methods are faithful to the causal dynamics underlying
model behavior and able to robustly generalize to unseen inputs. Distributed
Alignment Search (DAS) is a powerful gradient descent method grounded in a
theory of causal abstraction that uncovered perfect alignments between
interpretable symbolic algorithms and small deep learning models fine-tuned for
specific tasks. In the present paper, we scale DAS significantly by replacing
the remaining brute-force search steps with learned parameters -- an approach
we call DAS. This enables us to efficiently search for interpretable causal
structure in large language models while they follow instructions. We apply DAS
to the Alpaca model (7B parameters), which, off the shelf, solves a simple
numerical reasoning problem. With DAS, we discover that Alpaca does this by
implementing a causal model with two interpretable boolean variables.
Furthermore, we find that the alignment of neural representations with these
variables is robust to changes in inputs and instructions. These findings mark
a first step toward deeply understanding the inner-workings of our largest and
most widely deployed language models.
Natural Language Decomposition and Interpretation of Complex Utterances
Authors: name: Harsh Jhamtani, name: Hao Fang, name: Patrick Xia, name: Eran Levy, name: Jacob Andreas, name: Ben Van Durme
Published: 2023-05-15 14:35:00

Natural language interfaces often require supervised data to translate user
requests into programs, database queries, or other structured intent
representations. During data collection, it can be difficult to anticipate and
formalize the full range of user needs -- for example, in a system designed to
handle simple requests (like find my meetings tomorrow or
move my meeting with my manager to noon), users may also express
more elaborate requests (like swap all my calls on Monday and
Tuesday). We introduce an approach for equipping a simple language-to-code
model to handle complex utterances via a process of hierarchical natural
language decomposition. Our approach uses a pre-trained language model to
decompose a complex utterance into a sequence of smaller natural language
steps, then interprets each step using the language-to-code model. To test our
approach, we collect and release DeCU -- a new NL-to-program benchmark to
evaluate Decomposition of Complex Utterances. Experiments show that the
proposed approach enables the interpretation of complex utterances with almost
no complex training data, while outperforming standard few-shot prompting
approaches.
DarkBERT: A Language Model for the Dark Side of the Internet
Authors: name: Youngjin Jin, name: Eugene Jang, name: Jian Cui, name: Jin-Woo Chung, name: Yongjae Lee, name: Seungwon Shin
Published: 2023-05-15 12:23:10

Recent research has suggested that there are clear differences in the
language used in the Dark Web compared to that of the Surface Web. As studies
on the Dark Web commonly require textual analysis of the domain, language
models specific to the Dark Web may provide valuable insights to researchers.
In this work, we introduce DarkBERT, a language model pretrained on Dark Web
data. We describe the steps taken to filter and compile the text data used to
train DarkBERT to combat the extreme lexical and structural diversity of the
Dark Web that may be detrimental to building a proper representation of the
domain. We evaluate DarkBERT and its vanilla counterpart along with other
widely used language models to validate the benefits that a Dark Web domain
specific model offers in various use cases. Our evaluations show that DarkBERT
outperforms current language models and may serve as a valuable resource for
future research on the Dark Web.
Symbol tuning improves in-context learning in language models
Authors: name: Jerry Wei, name: Le Hou, name: Andrew Lampinen, name: Xiangning Chen, name: Da Huang, name: Yi Tay, name: Xinyun Chen, name: Yifeng Lu, name: Denny Zhou, name: Tengyu Ma, name: Quoc V. Le
Published: 2023-05-15 01:59:58

We present symbol tuning - finetuning language models on in-context
input-label pairs where natural language labels (e.g., "positive/negative
sentiment") are replaced with arbitrary symbols (e.g., "foo/bar"). Symbol
tuning leverages the intuition that when a model cannot use instructions or
natural language labels to figure out a task, it must instead do so by learning
the input-label mappings.
  We experiment with symbol tuning across Flan-PaLM models up to 540B
parameters and observe benefits across various settings. First, symbol tuning
boosts performance on unseen in-context learning tasks and is much more robust
to underspecified prompts, such as those without instructions or without
natural language labels. Second, symbol-tuned models are much stronger at
algorithmic reasoning tasks, with up to 18.2% better performance on the List
Functions benchmark and up to 15.3% better performance on the Simple Turing
Concepts benchmark. Finally, symbol-tuned models show large improvements in
following flipped-labels presented in-context, meaning that they are more
capable of using in-context information to override prior semantic knowledge.
ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding
Authors: name: Le Xue, name: Ning Yu, name: Shu Zhang, name: Junnan Li, name: Roberto Martín-Martín, name: Jiajun Wu, name: Caiming Xiong, name: Ran Xu, name: Juan Carlos Niebles, name: Silvio Savarese
Published: 2023-05-14 23:14:09

Recent advancements in multimodal pre-training methods have shown promising
efficacy in 3D representation learning by aligning features across 3D modality,
their 2D counterpart modality, and corresponding language modality. However,
the methods used by existing multimodal pre-training frameworks to gather
multimodal data for 3D applications lack scalability and comprehensiveness,
potentially constraining the full potential of multimodal learning. The main
bottleneck lies in the language modality's scalability and comprehensiveness.
To address this bottleneck, we introduce ULIP-2, a multimodal pre-training
framework that leverages state-of-the-art multimodal large language models
(LLMs) pre-trained on extensive knowledge to automatically generate holistic
language counterparts for 3D objects. We conduct experiments on two large-scale
datasets, Objaverse and ShapeNet55, and release our generated three-modality
triplet datasets (3D Point Cloud - Image - Language), named "ULIP-Objaverse
Triplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself and
eliminates the need for any manual annotation effort, demonstrating its
scalability; and ULIP-2 achieves remarkable improvements on downstream
zero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2
sets a new record on the real-world ScanObjectNN benchmark (91.5% Overall
Accuracy) while utilizing only 1.4 million parameters(~10x fewer than current
SOTA), signifying a breakthrough in scalable multimodal 3D representation
learning without human annotations. The code and datasets are available at
https://github.com/salesforce/ULIP.