Orionnnn/hub-stats huggingface dataset:
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities
  of Large Language Models
Authors: name: Xiaoxuan Wang, name: Ziniu Hu, name: Pan Lu, name: Yanqiao Zhu, name: Jieyu Zhang, name: Satyen Subramaniam, name: Arjun R. Loomba, name: Shichang Zhang, name: Yizhou Sun, name: Wei Wang
Published: 2023-07-20 07:01:57

Recent advances in large language models (LLMs) have demonstrated notable
progress on many mathematical benchmarks. However, most of these benchmarks
only feature problems grounded in junior and senior high school subjects,
contain only multiple-choice questions, and are confined to a limited scope of
elementary arithmetic operations. To address these issues, this paper
introduces an expansive benchmark suite SciBench that aims to systematically
examine the reasoning capabilities required for complex scientific problem
solving. SciBench contains two carefully curated datasets: an open set
featuring a range of collegiate-level scientific problems drawn from
mathematics, chemistry, and physics textbooks, and a closed set comprising
problems from undergraduate-level exams in computer science and mathematics.
Based on the two datasets, we conduct an in-depth benchmark study of two
representative LLMs with various prompting strategies. The results reveal that
current LLMs fall short of delivering satisfactory performance, with an overall
score of merely 35.80%. Furthermore, through a detailed user study, we
categorize the errors made by LLMs into ten problem-solving abilities. Our
analysis indicates that no single prompting strategy significantly outperforms
others and some strategies that demonstrate improvements in certain
problem-solving skills result in declines in other skills. We envision that
SciBench will catalyze further developments in the reasoning abilities of LLMs,
thereby ultimately contributing to scientific research and discovery.
The Role of Entropy and Reconstruction in Multi-View Self-Supervised
  Learning
Authors: name: Borja Rodríguez-Gálvez, name: Arno Blaas, name: Pau Rodríguez, name: Adam Goliński, name: Xavier Suau, name: Jason Ramapuram, name: Dan Busbridge, name: Luca Zappella
Published: 2023-07-20 14:29:51

The mechanisms behind the success of multi-view self-supervised learning
(MVSSL) are not yet fully understood. Contrastive MVSSL methods have been
studied through the lens of InfoNCE, a lower bound of the Mutual Information
(MI). However, the relation between other MVSSL methods and MI remains unclear.
We consider a different lower bound on the MI consisting of an entropy and a
reconstruction term (ER), and analyze the main MVSSL families through its lens.
Through this ER bound, we show that clustering-based methods such as
DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of
distillation-based approaches such as BYOL and DINO, showing that they
explicitly maximize the reconstruction term and implicitly encourage a stable
entropy, and we confirm this empirically. We show that replacing the objectives
of common MVSSL methods with this ER bound achieves competitive performance,
while making them stable when training with smaller batch sizes or smaller
exponential moving average (EMA) coefficients.
  Github repo: https://github.com/apple/ml-entropy-reconstruction.
PASTA: Pretrained Action-State Transformer Agents
Authors: name: Raphael Boige, name: Yannis Flet-Berliac, name: Arthur Flajolet, name: Guillaume Richard, name: Thomas Pierrot
Published: 2023-07-20 15:09:06

Self-supervised learning has brought about a revolutionary paradigm shift in
various computing domains, including NLP, vision, and biology. Recent
approaches involve pre-training transformer models on vast amounts of unlabeled
data, serving as a starting point for efficiently solving downstream tasks. In
the realm of reinforcement learning, researchers have recently adapted these
approaches by developing models pre-trained on expert trajectories, enabling
them to address a wide range of tasks, from robotics to recommendation systems.
However, existing methods mostly rely on intricate pre-training objectives
tailored to specific downstream applications. This paper presents a
comprehensive investigation of models we refer to as Pretrained Action-State
Transformer Agents (PASTA). Our study uses a unified methodology and covers an
extensive set of general downstream tasks including behavioral cloning, offline
RL, sensor failure robustness, and dynamics change adaptation. Our goal is to
systematically compare various design choices and provide valuable insights to
practitioners for building robust models. Key highlights of our study include
tokenization at the action and state component level, using fundamental
pre-training objectives like next token prediction, training models across
diverse domains simultaneously, and using parameter efficient fine-tuning
(PEFT). The developed models in our study contain fewer than 10 million
parameters and the application of PEFT enables fine-tuning of fewer than 10,000
parameters during downstream adaptation, allowing a broad community to use
these models and reproduce our experiments. We hope that this study will
encourage further research into the use of transformers with first-principles
design choices to represent RL trajectories and contribute to robust policy
learning.
Meta-Transformer: A Unified Framework for Multimodal Learning
Authors: name: Yiyuan Zhang, name: Kaixiong Gong, name: Kaipeng Zhang, name: Hongsheng Li, name: Yu Qiao, name: Wanli Ouyang, name: Xiangyu Yue
Published: 2023-07-20 12:10:29

Multimodal learning aims to build models that can process and relate
information from multiple modalities. Despite years of development in this
field, it still remains challenging to design a unified network for processing
various modalities (e.g. natural language, 2D images, 3D point
clouds, audio, video, time series, tabular data) due to the inherent gaps among
them. In this work, we propose a framework, named Meta-Transformer, that
leverages a frozen encoder to perform multimodal perception without
any paired multimodal training data. In Meta-Transformer, the raw input data
from various modalities are mapped into a shared token space, allowing a
subsequent encoder with frozen parameters to extract high-level semantic
features of the input data. Composed of three main components: a unified data
tokenizer, a modality-shared encoder, and task-specific heads for downstream
tasks, Meta-Transformer is the first framework to perform unified learning
across 12 modalities with unpaired data. Experiments on different benchmarks
reveal that Meta-Transformer can handle a wide range of tasks including
fundamental perception (text, image, point cloud, audio, video), practical
application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph,
tabular, and time-series). Meta-Transformer indicates a promising future for
developing unified multimodal intelligence with transformers. Code will be
available at https://github.com/invictus717/MetaTransformer
Brain2Music: Reconstructing Music from Human Brain Activity
Authors: name: Timo I. Denk, name: Yu Takagi, name: Takuya Matsuyama, name: Andrea Agostinelli, name: Tomoya Nakai, name: Christian Frank, name: Shinji Nishimoto
Published: 2023-07-20 17:55:17

The process of reconstructing experiences from human brain activity offers a
unique lens into how the brain interprets and represents the world. In this
paper, we introduce a method for reconstructing music from brain activity,
captured using functional magnetic resonance imaging (fMRI). Our approach uses
either music retrieval or the MusicLM music generation model conditioned on
embeddings derived from fMRI data. The generated music resembles the musical
stimuli that human subjects experienced, with respect to semantic properties
like genre, instrumentation, and mood. We investigate the relationship between
different components of MusicLM and brain activity through a voxel-wise
encoding modeling analysis. Furthermore, we discuss which brain regions
represent information derived from purely textual descriptions of music
stimuli. We provide supplementary material including examples of the
reconstructed music at https://google-research.github.io/seanet/brain2music
Improving Multimodal Datasets with Image Captioning
Authors: name: Thao Nguyen, name: Samir Yitzhak Gadre, name: Gabriel Ilharco, name: Sewoong Oh, name: Ludwig Schmidt
Published: 2023-07-19 17:47:12

Massive web datasets play a key role in the success of large vision-language
models like CLIP and Flamingo. However, the raw web data is noisy, and existing
filtering methods to reduce noise often come at the expense of data diversity.
Our work focuses on caption quality as one major source of noise, and studies
how generated captions can increase the utility of web-scraped datapoints with
nondescript text. Through exploring different mixing strategies for raw and
generated captions, we outperform the best filtering method proposed by the
DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
candidate pool of 128M image-text pairs. Our best approach is also 2x better at
Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
effective source of text supervision. In experimenting with different image
captioning models, we also demonstrate that the performance of a model on
standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
indicator of the utility of the captions it generates for multimodal training.
Finally, our experiments with using generated captions at DataComp's large
scale (1.28B image-text pairs) offer insights into the limitations of synthetic
text, as well as the importance of image curation with increasing training data
quantity.
TokenFlow: Consistent Diffusion Features for Consistent Video Editing
Authors: name: Michal Geyer, name: Omer Bar-Tal, name: Shai Bagon, name: Tali Dekel
Published: 2023-07-19 18:00:03

The generative AI revolution has recently expanded to videos. Nevertheless,
current state-of-the-art video models are still lagging behind image models in
terms of visual quality and user control over the generated content. In this
work, we present a framework that harnesses the power of a text-to-image
diffusion model for the task of text-driven video editing. Specifically, given
a source video and a target text-prompt, our method generates a high-quality
video that adheres to the target text, while preserving the spatial layout and
motion of the input video. Our method is based on a key observation that
consistency in the edited video can be obtained by enforcing consistency in the
diffusion feature space. We achieve this by explicitly propagating diffusion
features based on inter-frame correspondences, readily available in the model.
Thus, our framework does not require any training or fine-tuning, and can work
in conjunction with any off-the-shelf text-to-image editing method. We
demonstrate state-of-the-art editing results on a variety of real-world videos.
Webpage: https://diffusion-tokenflow.github.io/
Instruction-following Evaluation through Verbalizer Manipulation
Authors: name: Shiyang Li, name: Jun Yan, name: Hai Wang, name: Zheng Tang, name: Xiang Ren, name: Vijay Srinivasan, name: Hongxia Jin
Published: 2023-07-20 03:54:24

While instruction-tuned models have shown remarkable success in various
natural language processing tasks, accurately evaluating their ability to
follow instructions remains challenging. Existing benchmarks primarily focus on
common instructions that align well with what the model learned during
training. However, proficiency in responding to these instructions does not
necessarily imply strong ability in instruction following. In this paper, we
propose a novel instruction-following evaluation protocol called verbalizer
manipulation. It instructs the model to verbalize the task label with words
aligning with model priors to different extents, adopting verbalizers from
highly aligned (e.g., outputting ``postive'' for positive sentiment), to
minimally aligned (e.g., outputting ``negative'' for positive sentiment).
Verbalizer manipulation can be seamlessly integrated with any classification
benchmark to examine the model's reliance on priors and its ability to override
them to accurately follow the instructions. We conduct a comprehensive
evaluation of four major model families across nine datasets, employing twelve
sets of verbalizers for each of them. We observe that the instruction-following
abilities of models, across different families and scales, are significantly
distinguished by their performance on less natural verbalizers. Even the
strongest GPT-4 model struggles to perform better than random guessing on the
most challenging verbalizer, emphasizing the need for continued advancements to
improve their instruction-following abilities.
LLMs as Workers in Human-Computational Algorithms? Replicating
  Crowdsourcing Pipelines with LLMs
Authors: name: Tongshuang Wu, name: Haiyi Zhu, name: Maya Albayrak, name: Alexis Axon, name: Amanda Bertsch, name: Wenxing Deng, name: Ziqi Ding, name: Bill Guo, name: Sireesh Gururaja, name: Tzu-Sheng Kuo, name: Jenny T. Liang, name: Ryan Liu, name: Ihita Mandal, name: Jeremiah Milbauer, name: Xiaolin Ni, name: Namrata Padmanabhan, name: Subhashini Ramkumar, name: Alexis Sudjianto, name: Jordan Taylor, name: Ying-Jui Tseng, name: Patricia Vaidos, name: Zhijin Wu, name: Wei Wu, name: Chenyang Yang
Published: 2023-07-19 17:54:43

LLMs have shown promise in replicating human-like behavior in crowdsourcing
tasks that were previously thought to be exclusive to human abilities. However,
current efforts focus mainly on simple atomic tasks. We explore whether LLMs
can replicate more complex crowdsourcing pipelines. We find that modern LLMs
can simulate some of crowdworkers' abilities in these "human computation
algorithms," but the level of success is variable and influenced by requesters'
understanding of LLM capabilities, the specific skills required for sub-tasks,
and the optimal interaction modality for performing these sub-tasks. We reflect
on human and LLMs' different sensitivities to instructions, stress the
importance of enabling human-facing safeguards for LLMs, and discuss the
potential of training humans and LLMs with complementary skill sets. Crucially,
we show that replicating crowdsourcing pipelines offers a valuable platform to
investigate (1) the relative strengths of LLMs on different tasks (by
cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
complex tasks, where they can complete part of the tasks while leaving others
to humans.
DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity
  Human-centric Rendering
Authors: name: Wei Cheng, name: Ruixiang Chen, name: Wanqi Yin, name: Siming Fan, name: Keyu Chen, name: Honglin He, name: Huiwen Luo, name: Zhongang Cai, name: Jingbo Wang, name: Yang Gao, name: Zhengming Yu, name: Zhengyu Lin, name: Daxuan Ren, name: Lei Yang, name: Ziwei Liu, name: Chen Change Loy, name: Chen Qian, name: Wayne Wu, name: Dahua Lin, name: Bo Dai, name: Kwan-Yee Lin
Published: 2023-07-19 17:58:03

Realistic human-centric rendering plays a key role in both computer vision
and computer graphics. Rapid progress has been made in the algorithm aspect
over the years, yet existing human-centric rendering datasets and benchmarks
are rather impoverished in terms of diversity, which are crucial for rendering
effect. Researchers are usually constrained to explore and evaluate a small set
of rendering problems on current datasets, while real-world applications
require methods to be robust across different scenarios. In this work, we
present DNA-Rendering, a large-scale, high-fidelity repository of human
performance data for neural actor rendering. DNA-Rendering presents several
alluring attributes. First, our dataset contains over 1500 human subjects, 5000
motion sequences, and 67.5M frames' data volume. Second, we provide rich assets
for each subject -- 2D/3D human body keypoints, foreground masks, SMPLX models,
cloth/accessory materials, multi-view images, and videos. These assets boost
the current method's accuracy on downstream rendering tasks. Third, we
construct a professional multi-view system to capture data, which contains 60
synchronous cameras with max 4096 x 3000 resolution, 15 fps speed, and stern
camera calibration steps, ensuring high-quality resources for task training and
evaluation. Along with the dataset, we provide a large-scale and quantitative
benchmark in full-scale, with multiple tasks to evaluate the existing progress
of novel view synthesis, novel pose animation synthesis, and novel identity
rendering methods. In this manuscript, we describe our DNA-Rendering effort as
a revealing of new observations, challenges, and future directions to
human-centric rendering. The dataset, code, and benchmarks will be publicly
available at https://dna-rendering.github.io/
Android in the Wild: A Large-Scale Dataset for Android Device Control
Authors: name: Christopher Rawles, name: Alice Li, name: Daniel Rodriguez, name: Oriana Riva, name: Timothy Lillicrap
Published: 2023-07-19 15:57:24

There is a growing interest in device-control systems that can interpret
human natural language instructions and execute them on a digital device by
directly controlling its user interface. We present a dataset for
device-control research, Android in the Wild (AITW), which is orders of
magnitude larger than current datasets. The dataset contains human
demonstrations of device interactions, including the screens and actions, and
corresponding natural language instructions. It consists of 715k episodes
spanning 30k unique instructions, four versions of Android (v10-13),and eight
device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It
contains multi-step tasks that require semantic understanding of language and
visual context. This dataset poses a new challenge: actions available through
the user interface must be inferred from their visual appearance. And, instead
of simple UI element-based actions, the action space consists of precise
gestures (e.g., horizontal scrolls to operate carousel widgets). We organize
our dataset to encourage robustness analysis of device-control systems, i.e.,
how well a system performs in the presence of new task descriptions, new
applications, or new platform versions. We develop two agents and report
performance across the dataset. The dataset is available at
https://github.com/google-research/google-research/tree/master/android_in_the_wild.
On the Origin of LLMs: An Evolutionary Tree and Graph for 15,821 Large
  Language Models
Authors: name: Sarah Gao, name: Andrew Kean Gao
Published: 2023-07-19 07:17:43

Since late 2022, Large Language Models (LLMs) have become very prominent with
LLMs like ChatGPT and Bard receiving millions of users. Hundreds of new LLMs
are announced each week, many of which are deposited to Hugging Face, a
repository of machine learning models and datasets. To date, nearly 16,000 Text
Generation models have been uploaded to the site. Given the huge influx of
LLMs, it is of interest to know which LLM backbones, settings, training
methods, and families are popular or trending. However, there is no
comprehensive index of LLMs available. We take advantage of the relatively
systematic nomenclature of Hugging Face LLMs to perform hierarchical clustering
and identify communities amongst LLMs using n-grams and term frequency-inverse
document frequency. Our methods successfully identify families of LLMs and
accurately cluster LLMs into meaningful subgroups. We present a public web
application to navigate and explore Constellation, our atlas of 15,821 LLMs.
Constellation rapidly generates a variety of visualizations, namely
dendrograms, graphs, word clouds, and scatter plots. Constellation is available
at the following link: https://constellation.sites.stanford.edu/.
FABRIC: Personalizing Diffusion Models with Iterative Feedback
Authors: name: Dimitri von Rütte, name: Elisabetta Fedele, name: Jonathan Thomm, name: Lukas Wolf
Published: 2023-07-19 17:39:39

In an era where visual content generation is increasingly driven by machine
learning, the integration of human feedback into generative models presents
significant opportunities for enhancing user experience and output quality.
This study explores strategies for incorporating iterative human feedback into
the generative process of diffusion-based text-to-image models. We propose
FABRIC, a training-free approach applicable to a wide range of popular
diffusion models, which exploits the self-attention layer present in the most
widely used architectures to condition the diffusion process on a set of
feedback images. To ensure a rigorous assessment of our approach, we introduce
a comprehensive evaluation methodology, offering a robust mechanism to quantify
the performance of generative visual models that integrate human feedback. We
show that generation results improve over multiple rounds of iterative feedback
through exhaustive analysis, implicitly optimizing arbitrary user preferences.
The potential applications of these findings extend to fields such as
personalized content creation and customization.
Text2Layer: Layered Image Generation using Latent Diffusion Model
Authors: name: Xinyang Zhang, name: Wentian Zhao, name: Xin Lu, name: Jeff Chien
Published: 2023-07-19 06:56:07

Layer compositing is one of the most popular image editing workflows among
both amateurs and professionals. Motivated by the success of diffusion models,
we explore layer compositing from a layered image generation perspective.
Instead of generating an image, we propose to generate background, foreground,
layer mask, and the composed image simultaneously. To achieve layered image
generation, we train an autoencoder that is able to reconstruct layered images
and train diffusion models on the latent representation. One benefit of the
proposed problem is to enable better compositing workflows in addition to the
high-quality image output. Another benefit is producing higher-quality layer
masks compared to masks produced by a separate step of image segmentation.
Experimental results show that the proposed method is able to generate
high-quality layered images and initiates a benchmark for future work.
DialogStudio: Towards Richest and Most Diverse Unified Dataset
  Collection for Conversational AI
Authors: name: Jianguo Zhang, name: Kun Qian, name: Zhiwei Liu, name: Shelby Heinecke, name: Rui Meng, name: Ye Liu, name: Zhou Yu, name: Huan Wang, name: Silvio Savarese, name: Caiming Xiong
Published: 2023-07-19 17:57:53

Despite advancements in conversational AI, language models encounter
challenges to handle diverse conversational tasks, and existing dialogue
dataset collections often lack diversity and comprehensiveness. To tackle these
issues, we introduce DialogStudio: the largest and most diverse collection of
dialogue datasets, unified under a consistent format while preserving their
original information. Our collection encompasses data from open-domain
dialogues, task-oriented dialogues, natural language understanding,
conversational recommendation, dialogue summarization, and knowledge-grounded
dialogues, making it an incredibly rich and diverse resource for dialogue
research and model training. To further enhance the utility of DialogStudio, we
identify the licenses for each dataset and design domain-aware prompts for
selected dialogues to facilitate instruction-aware fine-tuning. Furthermore, we
develop conversational AI models using the dataset collection, and our
experiments in both zero-shot and few-shot learning scenarios demonstrate the
superiority of DialogStudio. To improve transparency and support dataset and
task-based research, as well as language model pre-training, all datasets,
licenses, codes, and models associated with DialogStudio are made publicly
accessible at https://github.com/salesforce/DialogStudio