Orionnnn/hub-stats huggingface dataset:
trajdata: A Unified Interface to Multiple Human Trajectory Datasets
Authors: name: Boris Ivanovic, name: Guanyu Song, name: Igor Gilitschenski, name: Marco Pavone
Published: 2023-07-26 02:45:59

The field of trajectory forecasting has grown significantly in recent years,
partially owing to the release of numerous large-scale, real-world human
trajectory datasets for autonomous vehicles (AVs) and pedestrian motion
tracking. While such datasets have been a boon for the community, they each use
custom and unique data formats and APIs, making it cumbersome for researchers
to train and evaluate methods across multiple datasets. To remedy this, we
present trajdata: a unified interface to multiple human trajectory datasets. At
its core, trajdata provides a simple, uniform, and efficient representation and
API for trajectory and map data. As a demonstration of its capabilities, in
this work we conduct a comprehensive empirical evaluation of existing
trajectory datasets, providing users with a rich understanding of the data
underpinning much of current pedestrian and AV motion forecasting research, and
proposing suggestions for future datasets from these insights. trajdata is
permissively licensed (Apache 2.0) and can be accessed online at
https://github.com/NVlabs/trajdata
Adaptive Frequency Filters As Efficient Global Token Mixers
Authors: name: Zhipeng Huang, name: Zhizheng Zhang, name: Cuiling Lan, name: Zheng-Jun Zha, name: Yan Lu, name: Baining Guo
Published: 2023-07-26 07:42:28

Recent vision transformers, large-kernel CNNs and MLPs have attained
remarkable successes in broad vision tasks thanks to their effective
information fusion in the global scope. However, their efficient deployments,
especially on mobile devices, still suffer from noteworthy challenges due to
the heavy computational costs of self-attention mechanisms, large kernels, or
fully connected layers. In this work, we apply conventional convolution theorem
to deep learning for addressing this and reveal that adaptive frequency filters
can serve as efficient global token mixers. With this insight, we propose
Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers
a latent representation to the frequency domain via a Fourier transform and
performs semantic-adaptive frequency filtering via an elementwise
multiplication, which mathematically equals to a token mixing operation in the
original latent space with a dynamic convolution kernel as large as the spatial
resolution of this latent representation. We take AFF token mixers as primary
neural operators to build a lightweight neural network, dubbed AFFNet.
Extensive experiments demonstrate the effectiveness of our proposed AFF token
mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs
compared to other lightweight network designs on broad visual tasks, including
visual recognition and dense prediction tasks.
Points-to-3D: Bridging the Gap between Sparse Points and
  Shape-Controllable Text-to-3D Generation
Authors: name: Chaohui Yu, name: Qiang Zhou, name: Jingliang Li, name: Zhe Zhang, name: Zhibin Wang, name: Fan Wang
Published: 2023-07-26 02:16:55

Text-to-3D generation has recently garnered significant attention, fueled by
2D diffusion models trained on billions of image-text pairs. Existing methods
primarily rely on score distillation to leverage the 2D diffusion priors to
supervise the generation of 3D models, e.g., NeRF. However, score distillation
is prone to suffer the view inconsistency problem, and implicit NeRF modeling
can also lead to an arbitrary shape, thus leading to less realistic and
uncontrollable 3D generation. In this work, we propose a flexible framework of
Points-to-3D to bridge the gap between sparse yet freely available 3D points
and realistic shape-controllable 3D generation by distilling the knowledge from
both 2D and 3D diffusion models. The core idea of Points-to-3D is to introduce
controllable sparse 3D points to guide the text-to-3D generation. Specifically,
we use the sparse point cloud generated from the 3D diffusion model, Point-E,
as the geometric prior, conditioned on a single reference image. To better
utilize the sparse 3D points, we propose an efficient point cloud guidance loss
to adaptively drive the NeRF's geometry to align with the shape of the sparse
3D points. In addition to controlling the geometry, we propose to optimize the
NeRF for a more view-consistent appearance. To be specific, we perform score
distillation to the publicly available 2D image diffusion model ControlNet,
conditioned on text as well as depth map of the learned compact geometry.
Qualitative and quantitative comparisons demonstrate that Points-to-3D improves
view consistency and achieves good shape controllability for text-to-3D
generation. Points-to-3D provides users with a new way to improve and control
text-to-3D generation.
Composite Diffusion | whole >= Σparts
Authors: name: Vikram Jamwal, name: Ramaneswaran S
Published: 2023-07-25 17:58:43

For an artist or a graphic designer, the spatial layout of a scene is a
critical design choice. However, existing text-to-image diffusion models
provide limited support for incorporating spatial information. This paper
introduces Composite Diffusion as a means for artists to generate high-quality
images by composing from the sub-scenes. The artists can specify the
arrangement of these sub-scenes through a flexible free-form segment layout.
They can describe the content of each sub-scene primarily using natural text
and additionally by utilizing reference images or control inputs such as line
art, scribbles, human pose, canny edges, and more.
  We provide a comprehensive and modular method for Composite Diffusion that
enables alternative ways of generating, composing, and harmonizing sub-scenes.
Further, we wish to evaluate the composite image for effectiveness in both
image quality and achieving the artist's intent. We argue that existing image
quality metrics lack a holistic evaluation of image composites. To address
this, we propose novel quality criteria especially relevant to composite
generation.
  We believe that our approach provides an intuitive method of art creation.
Through extensive user surveys, quantitative and qualitative analysis, we show
how it achieves greater spatial, semantic, and creative control over image
generation. In addition, our methods do not need to retrain or modify the
architecture of the base diffusion models and can work in a plug-and-play
manner with the fine-tuned models.
WavJourney: Compositional Audio Creation with Large Language Models
Authors: name: Xubo Liu, name: Zhongkai Zhu, name: Haohe Liu, name: Yi Yuan, name: Meng Cui, name: Qiushi Huang, name: Jinhua Liang, name: Yin Cao, name: Qiuqiang Kong, name: Mark D. Plumbley, name: Wenwu Wang
Published: 2023-07-26 17:54:04

Large Language Models (LLMs) have shown great promise in integrating diverse
expert models to tackle intricate language and vision tasks. Despite their
significance in advancing the field of Artificial Intelligence Generated
Content (AIGC), their potential in intelligent audio content creation remains
unexplored. In this work, we tackle the problem of creating audio content with
storylines encompassing speech, music, and sound effects, guided by text
instructions. We present WavJourney, a system that leverages LLMs to connect
various audio models for audio content generation. Given a text description of
an auditory scene, WavJourney first prompts LLMs to generate a structured
script dedicated to audio storytelling. The audio script incorporates diverse
audio elements, organized based on their spatio-temporal relationships. As a
conceptual representation of audio, the audio script provides an interactive
and interpretable rationale for human engagement. Afterward, the audio script
is fed into a script compiler, converting it into a computer program. Each line
of the program calls a task-specific audio generation model or computational
operation function (e.g., concatenate, mix). The computer program is then
executed to obtain an explainable solution for audio generation. We demonstrate
the practicality of WavJourney across diverse real-world scenarios, including
science fiction, education, and radio play. The explainable and interactive
design of WavJourney fosters human-machine co-creation in multi-round
dialogues, enhancing creative control and adaptability in audio production.
WavJourney audiolizes the human imagination, opening up new avenues for
creativity in multimedia content creation.
Large Language Models are Competitive Near Cold-start Recommenders for
  Language- and Item-based Preferences
Authors: name: Scott Sanner, name: Krisztian Balog, name: Filip Radlinski, name: Ben Wedin, name: Lucas Dixon
Published: 2023-07-26 14:47:15

Traditional recommender systems leverage users' item preference history to
recommend novel content that users may like. However, modern dialog interfaces
that allow users to express language-based preferences offer a fundamentally
different modality for preference input. Inspired by recent successes of
prompting paradigms for large language models (LLMs), we study their use for
making recommendations from both item-based and language-based preferences in
comparison to state-of-the-art item-based collaborative filtering (CF) methods.
To support this investigation, we collect a new dataset consisting of both
item-based and language-based preferences elicited from users along with their
ratings on a variety of (biased) recommended items and (unbiased) random items.
Among numerous experimental results, we find that LLMs provide competitive
recommendation performance for pure language-based preferences (no item
preferences) in the near cold-start case in comparison to item-based CF
methods, despite having no supervised training for this specific task
(zero-shot) or only a few labels (few-shot). This is particularly promising as
language-based preference representations are more explainable and scrutable
than item-based or vector-based representations.
Measuring Faithfulness in Chain-of-Thought Reasoning
Authors: name: Tamera Lanham, name: Anna Chen, name: Ansh Radhakrishnan, name: Benoit Steiner, name: Carson Denison, name: Danny Hernandez, name: Dustin Li, name: Esin Durmus, name: Evan Hubinger, name: Jackson Kernion, name: Kamilė Lukošiūtė, name: Karina Nguyen, name: Newton Cheng, name: Nicholas Joseph, name: Nicholas Schiefer, name: Oliver Rausch, name: Robin Larson, name: Sam McCandlish, name: Sandipan Kundu, name: Saurav Kadavath, name: Shannon Yang, name: Thomas Henighan, name: Timothy Maxwell, name: Timothy Telleen-Lawton, name: Tristan Hume, name: Zac Hatfield-Dodds, name: Jared Kaplan, name: Jan Brauner, name: Samuel R. Bowman, name: Ethan Perez
Published: 2023-07-17 01:08:39

Large language models (LLMs) perform better when they produce step-by-step,
"Chain-of-Thought" (CoT) reasoning before answering a question, but it is
unclear if the stated reasoning is a faithful explanation of the model's actual
reasoning (i.e., its process for answering the question). We investigate
hypotheses for how CoT reasoning may be unfaithful, by examining how the model
predictions change when we intervene on the CoT (e.g., by adding mistakes or
paraphrasing it). Models show large variation across tasks in how strongly they
condition on the CoT when predicting their answer, sometimes relying heavily on
the CoT and other times primarily ignoring it. CoT's performance boost does not
seem to come from CoT's added test-time compute alone or from information
encoded via the particular phrasing of the CoT. As models become larger and
more capable, they produce less faithful reasoning on most tasks we study.
Overall, our results suggest that CoT can be faithful if the circumstances such
as the model size and task are carefully chosen.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Authors: name: Shuyan Zhou, name: Frank F. Xu, name: Hao Zhu, name: Xuhui Zhou, name: Robert Lo, name: Abishek Sridhar, name: Xianyi Cheng, name: Yonatan Bisk, name: Daniel Fried, name: Uri Alon, name: Graham Neubig
Published: 2023-07-25 22:59:32

With generative AI advances, the exciting potential for autonomous agents to
manage daily tasks via natural language commands has emerged. However, cur rent
agents are primarily created and tested in simplified synthetic environments,
substantially limiting real-world scenario representation. In this paper, we
build an environment for agent command and control that is highly realistic and
reproducible. Specifically, we focus on agents that perform tasks on websites,
and we create an environment with fully functional websites from four common
domains: e-commerce, social forum discussions, collaborative software
development, and content management. Our environment is enriched with tools
(e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
human-like task-solving. Building upon our environment, we release a set of
benchmark tasks focusing on evaluating the functional correctness of task
completions. The tasks in our benchmark are diverse, long-horizon, and are
designed to emulate tasks that humans routinely perform on the internet. We
design and implement several autonomous agents, integrating recent techniques
such as reasoning before acting. The results demonstrate that solving complex
tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
task success rate of 10.59%. These results highlight the need for further
development of robust agents, that current state-of-the-art LMs are far from
perfect performance in these real-life tasks, and that WebArena can be used to
measure such progress. Our code, data, environment reproduction resources, and
video demonstrations are publicly available at https://webarena.dev/.
Strivec: Sparse Tri-Vector Radiance Fields
Authors: name: Quankai Gao, name: Qiangeng Xu, name: Hao Su, name: Ulrich Neumann, name: Zexiang Xu
Published: 2023-07-25 03:30:09

We propose Strivec, a novel neural representation that models a 3D scene as a
radiance field with sparsely distributed and compactly factorized local tensor
feature grids. Our approach leverages tensor decomposition, following the
recent work TensoRF, to model the tensor grids. In contrast to TensoRF which
uses a global tensor and focuses on their vector-matrix decomposition, we
propose to utilize a cloud of local tensors and apply the classic
CANDECOMP/PARAFAC (CP) decomposition to factorize each tensor into triple
vectors that express local feature distributions along spatial axes and
compactly encode a local neural field. We also apply multi-scale tensor grids
to discover the geometry and appearance commonalities and exploit spatial
coherence with the tri-vector factorization at multiple local scales. The final
radiance field properties are regressed by aggregating neural features from
multiple local tensors across all scales. Our tri-vector tensors are sparsely
distributed around the actual scene surface, discovered by a fast coarse
reconstruction, leveraging the sparsity of a 3D scene. We demonstrate that our
model can achieve better rendering quality while using significantly fewer
parameters than previous methods, including TensoRF and Instant-NGP.
Contrastive Example-Based Control
Authors: name: Kyle Hatch, name: Benjamin Eysenbach, name: Rafael Rafailov, name: Tianhe Yu, name: Ruslan Salakhutdinov, name: Sergey Levine, name: Chelsea Finn
Published: 2023-07-24 19:43:22

While many real-world problems that might benefit from reinforcement
learning, these problems rarely fit into the MDP mold: interacting with the
environment is often expensive and specifying reward functions is challenging.
Motivated by these challenges, prior work has developed data-driven approaches
that learn entirely from samples from the transition dynamics and examples of
high-return states. These methods typically learn a reward function from
high-return states, use that reward function to label the transitions, and then
apply an offline RL algorithm to these transitions. While these methods can
achieve good results on many tasks, they can be complex, often requiring
regularization and temporal difference updates. In this paper, we propose a
method for offline, example-based control that learns an implicit model of
multi-step transitions, rather than a reward function. We show that this
implicit model can represent the Q-values for the example-based control
problem. Across a range of state-based and image-based offline control tasks,
our method outperforms baselines that use learned reward functions; additional
experiments demonstrate improved robustness and scaling with dataset size.
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA
  Composition
Authors: name: Chengsong Huang, name: Qian Liu, name: Bill Yuchen Lin, name: Tianyu Pang, name: Chao Du, name: Min Lin
Published: 2023-07-25 05:39:21

Low-rank adaptations (LoRA) are often employed to fine-tune large language
models (LLMs) for new tasks. This paper investigates LoRA composability for
cross-task generalization and introduces LoraHub, a strategic framework devised
for the purposive assembly of LoRA modules trained on diverse given tasks, with
the objective of achieving adaptable performance on unseen tasks. With just a
few examples from a novel task, LoraHub enables the fluid combination of
multiple LoRA modules, eradicating the need for human expertise. Notably, the
composition requires neither additional model parameters nor gradients. Our
empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest
that LoraHub can effectively mimic the performance of in-context learning in
few-shot scenarios, excluding the necessity of in-context examples alongside
each inference input. A significant contribution of our research is the
fostering of a community for LoRA, where users can share their trained LoRA
modules, thereby facilitating their application to new tasks. We anticipate
this resource will widen access to and spur advancements in general
intelligence as well as LLMs in production. Code will be available at
https://github.com/sail-sg/lorahub.
ARB: Advanced Reasoning Benchmark for Large Language Models
Authors: name: Tomohiro Sawada, name: Daniel Paleka, name: Alexander Havrilla, name: Pranav Tadepalli, name: Paula Vidas, name: Alexander Kranias, name: John J. Nay, name: Kshitij Gupta, name: Aran Komatsuzaki
Published: 2023-07-25 17:55:19

Large Language Models (LLMs) have demonstrated remarkable performance on
various quantitative reasoning and knowledge benchmarks. However, many of these
benchmarks are losing utility as LLMs get increasingly high scores, despite not
yet reaching expert performance in these domains. We introduce ARB, a novel
benchmark composed of advanced reasoning problems in multiple fields. ARB
presents a more challenging test than prior benchmarks, featuring problems in
mathematics, physics, biology, chemistry, and law. As a subset of ARB, we
introduce a challenging set of math and physics problems which require advanced
symbolic reasoning and domain knowledge. We evaluate recent models such as
GPT-4 and Claude on ARB and demonstrate that current models score well below
50% on more demanding tasks. In order to improve both automatic and assisted
evaluation capabilities, we introduce a rubric-based evaluation approach,
allowing GPT-4 to score its own intermediate reasoning steps. Further, we
conduct a human evaluation of the symbolic subset of ARB, finding promising
agreement between annotators and GPT-4 rubric evaluation scores.
Predicting Code Coverage without Execution
Authors: name: Michele Tufano, name: Shubham Chandel, name: Anisha Agarwal, name: Neel Sundaresan, name: Colin Clement
Published: 2023-07-25 10:07:02

Code coverage is a widely used metric for quantifying the extent to which
program elements, such as statements or branches, are executed during testing.
Calculating code coverage is resource-intensive, requiring code building and
execution with additional overhead for the instrumentation. Furthermore,
computing coverage of any snippet of code requires the whole program context.
Using Machine Learning to amortize this expensive process could lower the cost
of code coverage by requiring only the source code context, and the task of
code coverage prediction can be a novel benchmark for judging the ability of
models to understand code. We propose a novel benchmark task called Code
Coverage Prediction for Large Language Models (LLMs). We formalize this task to
evaluate the capability of LLMs in understanding code execution by determining
which lines of a method are executed by a given test case and inputs. We curate
and release a dataset we call COVERAGEEVAL by executing tests and code from the
HumanEval dataset and collecting code coverage information. We report the
performance of four state-of-the-art LLMs used for code-related tasks,
including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's
Claude, on the Code Coverage Prediction task. Finally, we argue that code
coverage as a metric and pre-training data source are valuable for overall LLM
performance on software engineering tasks.
Multiscale Video Pretraining for Long-Term Activity Forecasting
Authors: name: Reuben Tan, name: Matthias De Lange, name: Michael Iuzzolino, name: Bryan A. Plummer, name: Kate Saenko, name: Karl Ridgeway, name: Lorenzo Torresani
Published: 2023-07-24 14:55:15

Long-term activity forecasting is an especially challenging research problem
because it requires understanding the temporal relationships between observed
actions, as well as the variability and complexity of human activities. Despite
relying on strong supervision via expensive human annotations, state-of-the-art
forecasting approaches often generalize poorly to unseen data. To alleviate
this issue, we propose Multiscale Video Pretraining (MVP), a novel
self-supervised pretraining approach that learns robust representations for
forecasting by learning to predict contextualized representations of future
video clips over multiple timescales. MVP is based on our observation that
actions in videos have a multiscale nature, where atomic actions typically
occur at a short timescale and more complex actions may span longer timescales.
We compare MVP to state-of-the-art self-supervised video learning approaches on
downstream long-term forecasting tasks including long-term action anticipation
and video summary prediction. Our comprehensive experiments across the Ego4D
and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs
state-of-the-art methods by significant margins. Notably, MVP obtains a
relative performance gain of over 20% accuracy in video summary forecasting
over existing methods.
MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised
  Learning of Motion and Content Features
Authors: name: Adrien Bardes, name: Jean Ponce, name: Yann LeCun
Published: 2023-07-24 11:27:14

Self-supervised learning of visual representations has been focusing on
learning content features, which do not capture object motion or location, and
focus on identifying and differentiating objects in images and videos. On the
other hand, optical flow estimation is a task that does not involve
understanding the content of the images on which it is estimated. We unify the
two approaches and introduce MC-JEPA, a joint-embedding predictive architecture
and self-supervised learning approach to jointly learn optical flow and content
features within a shared encoder, demonstrating that the two associated
objectives; the optical flow estimation objective and the self-supervised
learning objective; benefit from each other and thus learn content features
that incorporate motion information. The proposed approach achieves performance
on-par with existing unsupervised optical flow benchmarks, as well as with
common self-supervised learning approaches on downstream tasks such as semantic
segmentation of images and videos.