Orionnnn/hub-stats huggingface dataset:
Mind2Web: Towards a Generalist Agent for the Web
Authors: name: Xiang Deng, name: Yu Gu, name: Boyuan Zheng, name: Shijie Chen, name: Samuel Stevens, name: Boshi Wang, name: Huan Sun, name: Yu Su
Published: 2023-06-09 17:44:31

We introduce Mind2Web, the first dataset for developing and evaluating
generalist agents for the web that can follow language instructions to complete
complex tasks on any website. Existing datasets for web agents either use
simulated websites or only cover a limited set of websites and tasks, thus not
suitable for generalist web agents. With over 2,000 open-ended tasks collected
from 137 websites spanning 31 domains and crowdsourced action sequences for the
tasks, Mind2Web provides three necessary ingredients for building generalist
web agents: 1) diverse domains, websites, and tasks, 2) use of real-world
websites instead of simulated and simplified ones, and 3) a broad spectrum of
user interaction patterns. Based on Mind2Web, we conduct an initial exploration
of using large language models (LLMs) for building generalist web agents. While
the raw HTML of real-world websites are often too large to be fed to LLMs, we
show that first filtering it with a small LM significantly improves the
effectiveness and efficiency of LLMs. Our solution demonstrates a decent level
of performance, even on websites or entire domains the model has never seen
before, but there is still a substantial room to improve towards truly
generalizable agents. We open-source our dataset, model implementation, and
trained models (https://osu-nlp-group.github.io/Mind2Web) to facilitate further
research on building a generalist agent for the web.
Can Large Language Models Infer Causation from Correlation?
Authors: name: Zhijing Jin, name: Jiarui Liu, name: Zhiheng Lyu, name: Spencer Poff, name: Mrinmaya Sachan, name: Rada Mihalcea, name: Mona Diab, name: Bernhard Schölkopf
Published: 2023-06-09 12:09:15

Causal inference is one of the hallmarks of human intelligence. While the
field of CausalNLP has attracted much interest in the recent years, existing
causal inference datasets in NLP primarily rely on discovering causality from
empirical knowledge (e.g., commonsense knowledge). In this work, we propose the
first benchmark dataset to test the pure causal inference skills of large
language models (LLMs). Specifically, we formulate a novel task Corr2Cause,
which takes a set of correlational statements and determines the causal
relationship between the variables. We curate a large-scale dataset of more
than 400K samples, on which we evaluate seventeen existing LLMs. Through our
experiments, we identify a key shortcoming of LLMs in terms of their causal
inference skills, and show that these models achieve almost close to random
performance on the task. This shortcoming is somewhat mitigated when we try to
re-purpose LLMs for this skill via finetuning, but we find that these models
still fail to generalize -- they can only perform causal inference in
in-distribution settings when variable names and textual expressions used in
the queries are similar to those in the training set, but fail in
out-of-distribution settings generated by perturbing these queries. Corr2Cause
is a challenging task for LLMs, and would be helpful in guiding future research
on improving LLMs' pure reasoning skills and generalizability. Our data is at
https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at
https://github.com/causalNLP/corr2cause.
Embodied Executable Policy Learning with Language-based Scene
  Summarization
Authors: name: Jielin Qiu, name: Mengdi Xu, name: William Han, name: Seungwhan Moon, name: Ding Zhao
Published: 2023-06-09 06:34:09

Large Language models (LLMs) have shown remarkable success in assisting robot
learning tasks, i.e., complex household planning. However, the performance of
pretrained LLMs heavily relies on domain-specific templated text data, which
may be infeasible in real-world robot learning tasks with image-based
observations. Moreover, existing LLMs with text inputs lack the capability to
evolve with non-expert interactions with environments. In this work, we
introduce a novel learning paradigm that generates robots' executable actions
in the form of text, derived solely from visual observations, using
language-based summarization of these observations as the connecting bridge
between both domains. Our proposed paradigm stands apart from previous works,
which utilized either language instructions or a combination of language and
visual data as inputs. Moreover, our method does not require oracle text
summarization of the scene, eliminating the need for human involvement in the
learning loop, which makes it more practical for real-world robot learning
tasks. Our proposed paradigm consists of two modules: the SUM module, which
interprets the environment using visual observations and produces a text
summary of the scene, and the APM module, which generates executable action
policies based on the natural language descriptions provided by the SUM module.
We demonstrate that our proposed method can employ two fine-tuning strategies,
including imitation learning and reinforcement learning approaches, to adapt to
the target test tasks effectively. We conduct extensive experiments involving
various SUM/APM model selections, environments, and tasks across 7 house
layouts in the VirtualHome environment. Our experimental results demonstrate
that our method surpasses existing baselines, confirming the effectiveness of
this novel learning paradigm.
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Authors: name: Lianmin Zheng, name: Wei-Lin Chiang, name: Ying Sheng, name: Siyuan Zhuang, name: Zhanghao Wu, name: Yonghao Zhuang, name: Zi Lin, name: Zhuohan Li, name: Dacheng Li, name: Eric. P Xing, name: Hao Zhang, name: Joseph E. Gonzalez, name: Ion Stoica
Published: 2023-06-09 05:55:52

Evaluating large language model (LLM) based chat assistants is challenging
due to their broad capabilities and the inadequacy of existing benchmarks in
measuring human preferences. To address this, we explore using strong LLMs as
judges to evaluate these models on more open-ended questions. We examine the
usage and limitations of LLM-as-a-judge, such as position and verbosity biases
and limited reasoning ability, and propose solutions to migrate some of them.
We then verify the agreement between LLM judges and human preferences by
introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot
Arena, a crowdsourced battle platform. Our results reveal that strong LLM
judges like GPT-4 can match both controlled and crowdsourced human preferences
well, achieving over 80\% agreement, the same level of agreement between
humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate
human preferences, which are otherwise very expensive to obtain. Additionally,
we show our benchmark and traditional benchmarks complement each other by
evaluating several variants of LLaMA/Vicuna. We will publicly release 80
MT-bench questions, 3K expert votes, and 30K conversations with human
preferences from Chatbot Arena.
Background Prompting for Improved Object Depth
Authors: name: Manel Baradad, name: Yuanzhen Li, name: Forrester Cole, name: Michael Rubinstein, name: Antonio Torralba, name: William T. Freeman, name: Varun Jampani
Published: 2023-06-08 17:59:59

Estimating the depth of objects from a single image is a valuable task for
many vision, robotics, and graphics applications. However, current methods
often fail to produce accurate depth for objects in diverse scenes. In this
work, we propose a simple yet effective Background Prompting strategy that
adapts the input object image with a learned background. We learn the
background prompts only using small-scale synthetic object datasets. To infer
object depth on a real image, we place the segmented object into the learned
background prompt and run off-the-shelf depth networks. Background Prompting
helps the depth networks focus on the foreground object, as they are made
invariant to background variations. Moreover, Background Prompting minimizes
the domain gap between synthetic and real object images, leading to better
sim2real generalization than simple finetuning. Results on multiple synthetic
and real datasets demonstrate consistent improvements in real object depths for
a variety of existing depth networks. Code and optimized background prompts can
be found at: https://mbaradad.github.io/depth_prompt.
Tracking Everything Everywhere All at Once
Authors: name: Qianqian Wang, name: Yen-Yu Chang, name: Ruojin Cai, name: Zhengqi Li, name: Bharath Hariharan, name: Aleksander Holynski, name: Noah Snavely
Published: 2023-06-08 17:59:29

We present a new test-time optimization method for estimating dense and
long-range motion from a video sequence. Prior optical flow or particle video
tracking algorithms typically operate within limited temporal windows,
struggling to track through occlusions and maintain global consistency of
estimated motion trajectories. We propose a complete and globally consistent
motion representation, dubbed OmniMotion, that allows for accurate, full-length
motion estimation of every pixel in a video. OmniMotion represents a video
using a quasi-3D canonical volume and performs pixel-wise tracking via
bijections between local and canonical space. This representation allows us to
ensure global consistency, track through occlusions, and model any combination
of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and
real-world footage show that our approach outperforms prior state-of-the-art
methods by a large margin both quantitatively and qualitatively. See our
project page for more results: http://omnimotion.github.io/
Scaling Spherical CNNs
Authors: name: Carlos Esteves, name: Jean-Jacques Slotine, name: Ameesh Makadia
Published: 2023-06-08 17:59:08

Spherical CNNs generalize CNNs to functions on the sphere, by using spherical
convolutions as the main linear operation. The most accurate and efficient way
to compute spherical convolutions is in the spectral domain (via the
convolution theorem), which is still costlier than the usual planar
convolutions. For this reason, applications of spherical CNNs have so far been
limited to small problems that can be approached with low model capacity. In
this work, we show how spherical CNNs can be scaled for much larger problems.
To achieve this, we make critical improvements including novel variants of
common model components, an implementation of core operations to exploit
hardware accelerator characteristics, and application-specific input
representations that exploit the properties of our model. Experiments show our
larger spherical CNNs reach state-of-the-art on several targets of the QM9
molecular benchmark, which was previously dominated by equivariant graph neural
networks, and achieve competitive performance on multiple weather forecasting
tasks. Our code is available at
https://github.com/google-research/spherical-cnn.
R-MAE: Regions Meet Masked Autoencoders
Authors: name: Duy-Kien Nguyen, name: Vaibhav Aggarwal, name: Yanghao Li, name: Martin R. Oswald, name: Alexander Kirillov, name: Cees G. M. Snoek, name: Xinlei Chen
Published: 2023-06-08 17:56:46

Vision-specific concepts such as "region" have played a key role in extending
general machine learning frameworks to tasks like object detection. Given the
success of region-based detectors for supervised learning and the progress of
intra-image methods for contrastive learning, we explore the use of regions for
reconstructive pre-training. Starting from Masked Autoencoding (MAE) both as a
baseline and an inspiration, we propose a parallel pre-text task tailored to
address the one-to-many mapping between images and regions. Since such regions
can be generated in an unsupervised way, our approach (R-MAE) inherits the wide
applicability from MAE, while being more "region-aware". We conduct thorough
analyses during the development of R-MAE, and converge on a variant that is
both effective and efficient (1.3% overhead over MAE). Moreover, it shows
consistent quantitative improvements when generalized to various pre-training
data and downstream detection and segmentation benchmarks. Finally, we provide
extensive qualitative visualizations to enhance the understanding of R-MAE's
behaviour and potential. Code will be made available at
https://github.com/facebookresearch/r-mae.
LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
Authors: name: Zezhou Cheng, name: Carlos Esteves, name: Varun Jampani, name: Abhishek Kar, name: Subhransu Maji, name: Ameesh Makadia
Published: 2023-06-08 17:56:22

A critical obstacle preventing NeRF models from being deployed broadly in the
wild is their reliance on accurate camera poses. Consequently, there is growing
interest in extending NeRF models to jointly optimize camera poses and scene
representation, which offers an alternative to off-the-shelf SfM pipelines
which have well-understood failure modes. Existing approaches for unposed NeRF
operate under limited assumptions, such as a prior pose distribution or coarse
pose initialization, making them less effective in a general setting. In this
work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses
and neural radiance fields with relaxed assumptions on pose configuration. Our
approach operates in a local-to-global manner, where we first optimize over
local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and
geometry for this challenging few-shot task. The mini-scene poses are brought
into a global reference frame through a robust pose synchronization step, where
a final global optimization of pose and scene can be performed. We show our
LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making
restrictive assumptions on the pose prior. This allows us to operate in the
general SE(3) pose setting, unlike the baselines. Our results also indicate our
model can be complementary to feature-based SfM pipelines as it compares
favorably to COLMAP on low-texture and low-resolution images.
Unsupervised Compositional Concepts Discovery with Text-to-Image
  Generative Models
Authors: name: Nan Liu, name: Yilun Du, name: Shuang Li, name: Joshua B. Tenenbaum, name: Antonio Torralba
Published: 2023-06-08 17:02:15

Text-to-image generative models have enabled high-resolution image synthesis
across different domains, but require users to specify the content they wish to
generate. In this paper, we consider the inverse problem -- given a collection
of different images, can we discover the generative concepts that represent
each image? We present an unsupervised approach to discover generative concepts
from a collection of images, disentangling different art styles in paintings,
objects, and lighting from kitchen scenes, and discovering image classes given
ImageNet images. We show how such generative concepts can accurately represent
the content of images, be recombined and composed to generate new artistic and
hybrid images, and be further used as a representation for downstream
classification tasks.
Optimizing ViViT Training: Time and Memory Reduction for Action
  Recognition
Authors: name: Shreyank N Gowda, name: Anurag Arnab, name: Jonathan Huang
Published: 2023-06-07 23:06:53

In this paper, we address the challenges posed by the substantial training
time and memory consumption associated with video transformers, focusing on the
ViViT (Video Vision Transformer) model, in particular the Factorised Encoder
version, as our baseline for action recognition tasks. The factorised encoder
variant follows the late-fusion approach that is adopted by many state of the
art approaches. Despite standing out for its favorable speed/accuracy tradeoffs
among the different variants of ViViT, its considerable training time and
memory requirements still pose a significant barrier to entry. Our method is
designed to lower this barrier and is based on the idea of freezing the spatial
transformer during training. This leads to a low accuracy model if naively
done. But we show that by (1) appropriately initializing the temporal
transformer (a module responsible for processing temporal information) (2)
introducing a compact adapter model connecting frozen spatial representations
((a module that selectively focuses on regions of the input image) to the
temporal transformer, we can enjoy the benefits of freezing the spatial
transformer without sacrificing accuracy. Through extensive experimentation
over 6 benchmarks, we demonstrate that our proposed training strategy
significantly reduces training costs (by sim 50%) and memory consumption
while maintaining or slightly improving performance by up to 1.79\% compared to
the baseline model. Our approach additionally unlocks the capability to utilize
larger image transformer models as our spatial transformer and access more
frames with the same memory consumption.
SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions
Authors: name: Yuseung Lee, name: Kunho Kim, name: Hyunjin Kim, name: Minhyuk Sung
Published: 2023-06-08 13:18:23

The remarkable capabilities of pretrained image diffusion models have been
utilized not only for generating fixed-size images but also for creating
panoramas. However, naive stitching of multiple images often results in visible
seams. Recent techniques have attempted to address this issue by performing
joint diffusions in multiple windows and averaging latent features in
overlapping regions. However, these approaches, which focus on seamless montage
generation, often yield incoherent outputs by blending different scenes within
a single image. To overcome this limitation, we propose SyncDiffusion, a
plug-and-play module that synchronizes multiple diffusions through gradient
descent from a perceptual similarity loss. Specifically, we compute the
gradient of the perceptual loss using the predicted denoised images at each
denoising step, providing meaningful guidance for achieving coherent montages.
Our experimental results demonstrate that our method produces significantly
more coherent outputs compared to previous methods (66.35% vs. 33.65% in our
user study) while still maintaining fidelity (as assessed by GIQA) and
compatibility with the input prompt (as measured by CLIP score).
Matting Anything
Authors: name: Jiachen Li, name: Jitesh Jain, name: Humphrey Shi
Published: 2023-06-08 17:51:58

In this paper, we propose the Matting Anything Model (MAM), an efficient and
versatile framework for estimating the alpha matte of any instance in an image
with flexible and interactive visual or linguistic user prompt guidance. MAM
offers several significant advantages over previous specialized image matting
networks: (i) MAM is capable of dealing with various types of image matting,
including semantic, instance, and referring image matting with only a single
model; (ii) MAM leverages the feature maps from the Segment Anything Model
(SAM) and adopts a lightweight Mask-to-Matte (M2M) module to predict the alpha
matte through iterative refinement, which has only 2.7 million trainable
parameters. (iii) By incorporating SAM, MAM simplifies the user intervention
required for the interactive use of image matting from the trimap to the box,
point, or text prompt. We evaluate the performance of MAM on various image
matting benchmarks, and the experimental results demonstrate that MAM achieves
comparable performance to the state-of-the-art specialized image matting models
under different metrics on each benchmark. Overall, MAM shows superior
generalization ability and can effectively handle various image matting tasks
with fewer parameters, making it a practical solution for unified image
matting. Our code and models are open-sourced at
https://github.com/SHI-Labs/Matting-Anything.
Grounded Text-to-Image Synthesis with Attention Refocusing
Authors: name: Quynh Phung, name: Songwei Ge, name: Jia-Bin Huang
Published: 2023-06-08 17:59:59

Driven by scalable diffusion models trained on large-scale paired text-image
datasets, text-to-image synthesis methods have shown compelling results.
However, these models still fail to precisely follow the text prompt when
multiple objects, attributes, and spatial compositions are involved in the
prompt. In this paper, we identify the potential reasons in both the
cross-attention and self-attention layers of the diffusion model. We propose
two novel losses to refocus the attention maps according to a given layout
during the sampling process. We perform comprehensive experiments on the
DrawBench and HRS benchmarks using layouts synthesized by Large Language
Models, showing that our proposed losses can be integrated easily and
effectively into existing text-to-image methods and consistently improve their
alignment between the generated images and the text prompts.
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
Authors: name: Bo Li, name: Yuanhan Zhang, name: Liangyu Chen, name: Jinghao Wang, name: Fanyi Pu, name: Jingkang Yang, name: Chunyuan Li, name: Ziwei Liu
Published: 2023-06-08 17:59:56

High-quality instructions and responses are essential for the zero-shot
performance of large language models on interactive natural language tasks. For
interactive vision-language tasks involving intricate visual scenes, a large
quantity of diverse and creative instruction-response pairs should be
imperative to tune vision-language models (VLMs). Nevertheless, the current
availability of vision-language instruction-response pairs in terms of
quantity, diversity, and creativity remains limited, posing challenges to the
generalization of interactive VLMs. Here we present MultI-Modal In-Context
Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal
instruction-response pairs, with 2.2 million unique instructions derived from
images and videos. Each pair is accompanied by multi-modal in-context
information, forming conversational contexts aimed at empowering VLMs in
perception, reasoning, and planning. The instruction-response collection
process, dubbed as Syphus, is scaled using an automatic annotation pipeline
that combines human expertise with GPT's capabilities. Using the MIMIC-IT
dataset, we train a large VLM named Otter. Based on extensive evaluations
conducted on vision-language benchmarks, it has been observed that Otter
demonstrates remarkable proficiency in multi-modal perception, reasoning, and
in-context learning. Human evaluation reveals it effectively aligns with the
user's intentions. We release the MIMIC-IT dataset, instruction-response
collection pipeline, benchmarks, and the Otter model.
Modular Visual Question Answering via Code Generation
Authors: name: Sanjay Subramanian, name: Medhini Narasimhan, name: Kushal Khangaonkar, name: Kevin Yang, name: Arsha Nagrani, name: Cordelia Schmid, name: Andy Zeng, name: Trevor Darrell, name: Dan Klein
Published: 2023-06-08 17:45:14

We present a framework that formulates visual question answering as modular
code generation. In contrast to prior work on modular approaches to VQA, our
approach requires no additional training and relies on pre-trained language
models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA
examples used for in-context learning. The generated Python programs invoke and
compose the outputs of the visual models using arithmetic and conditional
logic. Our approach improves accuracy on the COVR dataset by at least 3% and on
the GQA dataset by roughly 2% compared to the few-shot baseline that does not
employ code generation.