Orionnnn/hub-stats huggingface dataset:
GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS
Authors: name: Saman Kazemkhani, name: Aarav Pandya, name: Daphne Cornelisse, name: Brennan Shacklett, name: Eugene Vinitsky
Published: 2024-08-02 21:37:46

Multi-agent learning algorithms have been successful at generating superhuman
planning in a wide variety of games but have had little impact on the design of
deployed multi-agent planners. A key bottleneck in applying these techniques to
multi-agent planning is that they require billions of steps of experience. To
enable the study of multi-agent planning at this scale, we present GPUDrive, a
GPU-accelerated, multi-agent simulator built on top of the Madrona Game Engine
that can generate over a million steps of experience per second. Observation,
reward, and dynamics functions are written directly in C++, allowing users to
define complex, heterogeneous agent behaviors that are lowered to
high-performance CUDA. We show that using GPUDrive we are able to effectively
train reinforcement learning agents over many scenes in the Waymo Motion
dataset, yielding highly effective goal-reaching agents in minutes for
individual scenes and generally capable agents in a few hours. We ship these
trained agents as part of the code base at
https://github.com/Emerge-Lab/gpudrive.
MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh
  Tokenization
Authors: name: Yiwen Chen, name: Yikai Wang, name: Yihao Luo, name: Zhengyi Wang, name: Zilong Chen, name: Jun Zhu, name: Chi Zhang, name: Guosheng Lin
Published: 2024-08-05 15:33:45

We introduce MeshAnything V2, an autoregressive transformer that generates
Artist-Created Meshes (AM) aligned to given shapes. It can be integrated with
various 3D asset production pipelines to achieve high-quality, highly
controllable AM generation. MeshAnything V2 surpasses previous methods in both
efficiency and performance using models of the same size. These improvements
are due to our newly proposed mesh tokenization method: Adjacent Mesh
Tokenization (AMT). Different from previous methods that represent each face
with three vertices, AMT uses a single vertex whenever possible. Compared to
previous methods, AMT requires about half the token sequence length to
represent the same mesh in average. Furthermore, the token sequences from AMT
are more compact and well-structured, fundamentally benefiting AM generation.
Our extensive experiments show that AMT significantly improves the efficiency
and performance of AM generation. Project Page:
https://buaacyw.github.io/meshanything-v2/
Self-Taught Evaluators
Authors: name: Tianlu Wang, name: Ilia Kulikov, name: Olga Golovneva, name: Ping Yu, name: Weizhe Yuan, name: Jane Dwivedi-Yu, name: Richard Yuanzhe Pang, name: Maryam Fazel-Zarandi, name: Jason Weston, name: Xian Li
Published: 2024-08-05 17:57:02

Model-based evaluation is at the heart of successful model development -- as
a reward model for training, and as a replacement for human evaluation. To
train such evaluators, the standard approach is to collect a large amount of
human preference judgments over model responses, which is costly and the data
becomes stale as models improve. In this work, we present an approach that aims
to im-prove evaluators without human annotations, using synthetic training data
only. Starting from unlabeled instructions, our iterative self-improvement
scheme generates contrasting model outputs and trains an LLM-as-a-Judge to
produce reasoning traces and final judgments, repeating this training at each
new iteration using the improved predictions. Without any labeled preference
data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct)
from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms
commonly used LLM judges such as GPT-4 and matches the performance of the
top-performing reward models trained with labeled examples.
ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation
  Extraction on an Academic Budget
Authors: name: Riccardo Orlando, name: Pere-Lluis Huguet-Cabot, name: Edoardo Barba, name: Roberto Navigli
Published: 2024-07-31 18:25:49

Entity Linking (EL) and Relation Extraction (RE) are fundamental tasks in
Natural Language Processing, serving as critical components in a wide range of
applications. In this paper, we propose ReLiK, a Retriever-Reader architecture
for both EL and RE, where, given an input text, the Retriever module undertakes
the identification of candidate entities or relations that could potentially
appear within the text. Subsequently, the Reader module is tasked to discern
the pertinent retrieved entities or relations and establish their alignment
with the corresponding textual spans. Notably, we put forward an innovative
input representation that incorporates the candidate entities or relations
alongside the text, making it possible to link entities or extract relations in
a single forward pass and to fully leverage pre-trained language models
contextualization capabilities, in contrast with previous
Retriever-Reader-based methods, which require a forward pass for each
candidate. Our formulation of EL and RE achieves state-of-the-art performance
in both in-domain and out-of-domain benchmarks while using academic budget
training and with up to 40x inference speed compared to competitors. Finally,
we show how our architecture can be used seamlessly for Information Extraction
(cIE), i.e. EL + RE, and setting a new state of the art by employing a shared
Reader that simultaneously extracts entities and relations.
In-Context Example Selection via Similarity Search Improves Low-Resource
  Machine Translation
Authors: name: Armel Zebaze, name: Benoît Sagot, name: Rachel Bawden
Published: 2024-08-01 09:07:32

The ability of generative large language models (LLMs) to perform in-context
learning has given rise to a large body of research into how best to prompt
models for various natural language processing tasks. In this paper, we focus
on machine translation (MT), a task that has been shown to benefit from
in-context translation examples. However no systematic studies have been
published on how best to select examples, and mixed results have been reported
on the usefulness of similarity-based selection over random selection. We
provide a study covering multiple LLMs and multiple in-context example
retrieval strategies, comparing multilingual sentence embeddings. We cover
several language directions, representing different levels of language
resourcedness (English into French, German, Swahili and Wolof). Contrarily to
previously published results, we find that sentence embedding similarity can
improve MT, especially for low-resource language directions, and discuss the
balance between selection pool diversity and quality. We also highlight
potential problems with the evaluation of LLM-based MT and suggest a more
appropriate evaluation protocol, adapting the COMET metric to the evaluation of
LLMs. Code and outputs are freely available at
https://github.com/ArmelRandy/ICL-MT.
RelBench: A Benchmark for Deep Learning on Relational Databases
Authors: name: Joshua Robinson, name: Rishabh Ranjan, name: Weihua Hu, name: Kexin Huang, name: Jiaqi Han, name: Alejandro Dobles, name: Matthias Fey, name: Jan E. Lenssen, name: Yiwen Yuan, name: Zecheng Zhang, name: Xinwei He, name: Jure Leskovec
Published: 2024-07-29 14:46:13

We present RelBench, a public benchmark for solving predictive tasks over
relational databases with graph neural networks. RelBench provides databases
and tasks spanning diverse domains and scales, and is intended to be a
foundational infrastructure for future research. We use RelBench to conduct the
first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024),
which combines graph neural network predictive models with (deep) tabular
models that extract initial entity-level representations from raw tables.
End-to-end learned RDL models fully exploit the predictive signal encoded in
primary-foreign key links, marking a significant shift away from the dominant
paradigm of manual feature engineering combined with tabular models. To
thoroughly evaluate RDL against this prior gold-standard, we conduct an
in-depth user study where an experienced data scientist manually engineers
features for each task. In this study, RDL learns better models whilst reducing
human work needed by more than an order of magnitude. This demonstrates the
power of deep learning for solving predictive tasks over relational databases,
opening up many new research opportunities enabled by RelBench.
Measuring Progress in Dictionary Learning for Language Model
  Interpretability with Board Game Models
Authors: name: Adam Karvonen, name: Benjamin Wright, name: Can Rager, name: Rico Angell, name: Jannik Brinkmann, name: Logan Smith, name: Claudio Mayrink Verdun, name: David Bau, name: Samuel Marks
Published: 2024-07-31 18:45:13

What latent features are encoded in language model (LM) representations?
Recent work on training sparse autoencoders (SAEs) to disentangle interpretable
features in LM representations has shown significant promise. However,
evaluating the quality of these SAEs is difficult because we lack a
ground-truth collection of interpretable features that we expect good SAEs to
recover. We thus propose to measure progress in interpretable dictionary
learning by working in the setting of LMs trained on chess and Othello
transcripts. These settings carry natural collections of interpretable features
-- for example, "there is a knight on F3" -- which we leverage into
supervised metrics for SAE quality. To guide progress in
interpretable dictionary learning, we introduce a new SAE training technique,
p-annealing, which improves performance on prior unsupervised
metrics as well as our new metrics.
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language
  Models
Authors: name: Benno Weck, name: Ilaria Manco, name: Emmanouil Benetos, name: Elio Quinton, name: George Fazekas, name: Dmitry Bogdanov
Published: 2024-08-02 15:34:05

Multimodal models that jointly process audio and language hold great promise
in audio understanding and are increasingly being adopted in the music domain.
By allowing users to query via text and obtain information about a given audio
input, these models have the potential to enable a variety of music
understanding tasks via language-based interfaces. However, their evaluation
poses considerable challenges, and it remains unclear how to effectively assess
their ability to correctly interpret music-related inputs with current methods.
Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music
understanding in multimodal language models focused on audio. MuChoMusic
comprises 1,187 multiple-choice questions, all validated by human annotators,
on 644 music tracks sourced from two publicly available music datasets, and
covering a wide variety of genres. Questions in the benchmark are crafted to
assess knowledge and reasoning abilities across several dimensions that cover
fundamental musical concepts and their relation to cultural and functional
contexts. Through the holistic analysis afforded by the benchmark, we evaluate
five open-source models and identify several pitfalls, including an
over-reliance on the language modality, pointing to a need for better
multimodal integration. Data and code are open-sourced.
Medical SAM 2: Segment medical images as video via Segment Anything
  Model 2
Authors: name: Jiayuan Zhu, name: Yunli Qi, name: Junde Wu
Published: 2024-08-01 18:49:45

In this paper, we introduce Medical SAM 2 (MedSAM-2), an advanced
segmentation model that utilizes the SAM 2 framework to address both 2D and 3D
medical image segmentation tasks. By adopting the philosophy of taking medical
images as videos, MedSAM-2 not only applies to 3D medical images but also
unlocks new One-prompt Segmentation capability. That allows users to provide a
prompt for just one or a specific image targeting an object, after which the
model can autonomously segment the same type of object in all subsequent
images, regardless of temporal relationships between the images. We evaluated
MedSAM-2 across a variety of medical imaging modalities, including abdominal
organs, optic discs, brain tumors, thyroid nodules, and skin lesions, comparing
it against state-of-the-art models in both traditional and interactive
segmentation settings. Our findings show that MedSAM-2 not only surpasses
existing models in performance but also exhibits superior generalization across
a range of medical image segmentation tasks. Our code will be released at:
https://github.com/MedicineToken/Medical-SAM2
TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and
  Resampling
Authors: name: Dong Huo, name: Zixin Guo, name: Xinxin Zuo, name: Zhihao Shi, name: Juwei Lu, name: Peng Dai, name: Songcen Xu, name: Li Cheng, name: Yee-Hong Yang
Published: 2024-08-02 14:24:40

Given a 3D mesh, we aim to synthesize 3D textures that correspond to
arbitrary textual descriptions. Current methods for generating and assembling
textures from sampled views often result in prominent seams or excessive
smoothing. To tackle these issues, we present TexGen, a novel multi-view
sampling and resampling framework for texture generation leveraging a
pre-trained text-to-image diffusion model. For view consistent sampling, first
of all we maintain a texture map in RGB space that is parameterized by the
denoising step and updated after each sampling step of the diffusion model to
progressively reduce the view discrepancy. An attention-guided multi-view
sampling strategy is exploited to broadcast the appearance information across
views. To preserve texture details, we develop a noise resampling technique
that aids in the estimation of noise, generating inputs for subsequent
denoising steps, as directed by the text prompt and current texture map.
Through an extensive amount of qualitative and quantitative evaluations, we
demonstrate that our proposed method produces significantly better texture
quality for diverse 3D objects with a high degree of view consistency and rich
appearance details, outperforming current state-of-the-art methods.
Furthermore, our proposed texture generation technique can also be applied to
texture editing while preserving the original identity. More experimental
results are available at https://dong-huo.github.io/TexGen/
POA: Pre-training Once for Models of All Sizes
Authors: name: Yingying Zhang, name: Xin Guo, name: Jiangwei Lao, name: Lei Yu, name: Lixiang Ru, name: Jian Wang, name: Guo Ye, name: Huimei He, name: Jingdong Chen, name: Ming Yang
Published: 2024-08-02 06:13:29

Large-scale self-supervised pre-training has paved the way for one foundation
model to handle many different vision tasks. Most pre-training methodologies
train a single model of a certain size at one time. Nevertheless, various
computation or storage constraints in real-world scenarios require substantial
efforts to develop a series of models with different sizes to deploy. Thus, in
this study, we propose a novel tri-branch self-supervised training framework,
termed as POA (Pre-training Once for All), to tackle this aforementioned issue.
Our approach introduces an innovative elastic student branch into a modern
self-distillation paradigm. At each pre-training step, we randomly sample a
sub-network from the original student to form the elastic student and train all
branches in a self-distilling fashion. Once pre-trained, POA allows the
extraction of pre-trained models of diverse sizes for downstream tasks.
Remarkably, the elastic student facilitates the simultaneous pre-training of
multiple models with different sizes, which also acts as an additional ensemble
of models of various sizes to enhance representation learning. Extensive
experiments, including k-nearest neighbors, linear probing evaluation and
assessments on multiple downstream tasks demonstrate the effectiveness and
advantages of our POA. It achieves state-of-the-art performance using ViT, Swin
Transformer and ResNet backbones, producing around a hundred models with
different sizes through a single pre-training session. The code is available
at: https://github.com/Qichuzyy/POA.
Enhancing Semantic Similarity Understanding in Arabic NLP with Nested
  Embedding Learning
Authors: name: Omer Nacar, name: Anis Koubaa
Published: 2024-07-30 19:03:03

This work presents a novel framework for training Arabic nested embedding
models through Matryoshka Embedding Learning, leveraging multilingual,
Arabic-specific, and English-based models, to highlight the power of nested
embeddings models in various Arabic NLP downstream tasks. Our innovative
contribution includes the translation of various sentence similarity datasets
into Arabic, enabling a comprehensive evaluation framework to compare these
models across different dimensions. We trained several nested embedding models
on the Arabic Natural Language Inference triplet dataset and assessed their
performance using multiple evaluation metrics, including Pearson and Spearman
correlations for cosine similarity, Manhattan distance, Euclidean distance, and
dot product similarity. The results demonstrate the superior performance of the
Matryoshka embedding models, particularly in capturing semantic nuances unique
to the Arabic language. Results demonstrated that Arabic Matryoshka embedding
models have superior performance in capturing semantic nuances unique to the
Arabic language, significantly outperforming traditional models by up to
20-25\% across various similarity metrics. These results underscore the
effectiveness of language-specific training and highlight the potential of
Matryoshka models in enhancing semantic textual similarity tasks for Arabic
NLP.
Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy
  Curvature of Attention
Authors: name: Susung Hong
Published: 2024-08-01 17:59:09

Conditional diffusion models have shown remarkable success in visual content
generation, producing high-quality samples across various domains, largely due
to classifier-free guidance (CFG). Recent attempts to extend guidance to
unconditional models have relied on heuristic techniques, resulting in
suboptimal generation quality and unintended effects. In this work, we propose
Smoothed Energy Guidance (SEG), a novel training- and condition-free approach
that leverages the energy-based perspective of the self-attention mechanism to
enhance image generation. By defining the energy of self-attention, we
introduce a method to reduce the curvature of the energy landscape of attention
and use the output as the unconditional prediction. Practically, we control the
curvature of the energy landscape by adjusting the Gaussian kernel parameter
while keeping the guidance scale parameter fixed. Additionally, we present a
query blurring method that is equivalent to blurring the entire attention
weights without incurring quadratic complexity in the number of tokens. In our
experiments, SEG achieves a Pareto improvement in both quality and the
reduction of side effects. The code is available at
https://github.com/SusungHong/SEG-SDXL.
Generalized Out-of-Distribution Detection and Beyond in Vision Language
  Model Era: A Survey
Authors: name: Atsuyuki Miyai, name: Jingkang Yang, name: Jingyang Zhang, name: Yifei Ming, name: Yueqian Lin, name: Qing Yu, name: Go Irie, name: Shafiq Joty, name: Yixuan Li, name: Hai Li, name: Ziwei Liu, name: Toshihiko Yamasaki, name: Kiyoharu Aizawa
Published: 2024-07-31 17:59:58

Detecting out-of-distribution (OOD) samples is crucial for ensuring the
safety of machine learning systems and has shaped the field of OOD detection.
Meanwhile, several other problems are closely related to OOD detection,
including anomaly detection (AD), novelty detection (ND), open set recognition
(OSR), and outlier detection (OD). To unify these problems, a generalized OOD
detection framework was proposed, taxonomically categorizing these five
problems. However, Vision Language Models (VLMs) such as CLIP have
significantly changed the paradigm and blurred the boundaries between these
fields, again confusing researchers. In this survey, we first present a
generalized OOD detection v2, encapsulating the evolution of AD, ND, OSR, OOD
detection, and OD in the VLM era. Our framework reveals that, with some field
inactivity and integration, the demanding challenges have become OOD detection
and AD. In addition, we also highlight the significant shift in the definition,
problem settings, and benchmarks; we thus feature a comprehensive review of the
methodology for OOD detection, including the discussion over other related
tasks to clarify their relationship to OOD detection. Finally, we explore the
advancements in the emerging Large Vision Language Model (LVLM) era, such as
GPT-4V. We conclude this survey with open challenges and future directions.
Finch: Prompt-guided Key-Value Cache Compression
Authors: name: Giulio Corallo, name: Paolo Papotti
Published: 2024-07-31 21:33:56

Recent large language model applications, such as Retrieval-Augmented
Generation and chatbots, have led to an increased need to process longer input
contexts. However, this requirement is hampered by inherent limitations.
Architecturally, models are constrained by a context window defined during
training. Additionally, processing extensive texts requires substantial GPU
memory. We propose a novel approach, Finch, to compress the input context by
leveraging the pre-trained model weights of the self-attention. Given a prompt
and a long text, Finch iteratively identifies the most relevant Key (K) and
Value (V) pairs over chunks of the text conditioned on the prompt. Only such
pairs are stored in the KV cache, which, within the space constrained by the
context window, ultimately contains a compressed version of the long text. Our
proposal enables models to consume large inputs even with high compression (up
to 93x) while preserving semantic integrity without the need for fine-tuning.
Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual
  Inversion
Authors: name: Manuel Kansy, name: Jacek Naruniec, name: Christopher Schroers, name: Markus Gross, name: Romann M. Weber
Published: 2024-08-01 10:55:20

Recent years have seen a tremendous improvement in the quality of video
generation and editing approaches. While several techniques focus on editing
appearance, few address motion. Current approaches using text, trajectories, or
bounding boxes are limited to simple motions, so we specify motions with a
single motion reference video instead. We further propose to use a pre-trained
image-to-video model rather than a text-to-video model. This approach allows us
to preserve the exact appearance and position of a target object or scene and
helps disentangle appearance from motion. Our method, called motion-textual
inversion, leverages our observation that image-to-video models extract
appearance mainly from the (latent) image input, while the text/image embedding
injected via cross-attention predominantly controls motion. We thus represent
motion using text/image embedding tokens. By operating on an inflated
motion-text embedding containing multiple text/image embedding tokens per
frame, we achieve a high temporal motion granularity. Once optimized on the
motion reference video, this embedding can be applied to various target images
to generate videos with semantically similar motions. Our approach does not
require spatial alignment between the motion reference video and target image,
generalizes across various domains, and can be applied to various tasks such as
full-body and face reenactment, as well as controlling the motion of inanimate
objects and the camera. We empirically demonstrate the effectiveness of our
method in the semantic video motion transfer task, significantly outperforming
existing methods in this context.