Orionnnn/hub-stats huggingface dataset:
RE-AdaptIR: Improving Information Retrieval through Reverse Engineered
  Adaptation
Authors: name: William Fleshman, name: Benjamin Van Durme
Published: 2024-06-20 22:28:11

Large language models (LLMs) fine-tuned for text-retrieval have demonstrated
state-of-the-art results across several information retrieval (IR) benchmarks.
However, supervised training for improving these models requires numerous
labeled examples, which are generally unavailable or expensive to acquire. In
this work, we explore the effectiveness of extending reverse engineered
adaptation to the context of information retrieval (RE-AdaptIR). We use
RE-AdaptIR to improve LLM-based IR models using only unlabeled data. We
demonstrate improved performance both in training domains as well as zero-shot
in domains where the models have seen no queries. We analyze performance
changes in various fine-tuning scenarios and offer findings of immediate use to
practitioners.
Complexity of Symbolic Representation in Working Memory of Transformer
  Correlates with the Complexity of a Task
Authors: name: Alsu Sagirova, name: Mikhail Burtsev
Published: 2024-06-20 11:27:29

Even though Transformers are extensively used for Natural Language Processing
tasks, especially for machine translation, they lack an explicit memory to
store key concepts of processed texts. This paper explores the properties of
the content of symbolic working memory added to the Transformer model decoder.
Such working memory enhances the quality of model predictions in machine
translation task and works as a neural-symbolic representation of information
that is important for the model to make correct translations. The study of
memory content revealed that translated text keywords are stored in the working
memory, pointing to the relevance of memory content to the processed text.
Also, the diversity of tokens and parts of speech stored in memory correlates
with the complexity of the corpora for machine translation task.
Low-Resource Machine Translation through the Lens of Personalized
  Federated Learning
Authors: name: Viktor Moskvoretskii, name: Nazarii Tupitsa, name: Chris Biemann, name: Samuel Horváth, name: Eduard Gorbunov, name: Irina Nikishina
Published: 2024-06-18 12:50:00

We present a new approach based on the Personalized Federated Learning
algorithm MeritFed that can be applied to Natural Language Tasks with
heterogeneous data. We evaluate it on the Low-Resource Machine Translation
task, using the dataset from the Large-Scale Multilingual Machine Translation
Shared Task (Small Track #2) and the subset of Sami languages from the
multilingual benchmark for Finno-Ugric languages. In addition to its
effectiveness, MeritFed is also highly interpretable, as it can be applied to
track the impact of each language used for training. Our analysis reveals that
target dataset size affects weight distribution across auxiliary languages,
that unrelated languages do not interfere with the training, and auxiliary
optimizer parameters have minimal impact. Our approach is easy to apply with a
few lines of code, and we provide scripts for reproducing the experiments at
https://github.com/VityaVitalich/MeritFed
EvTexture: Event-driven Texture Enhancement for Video Super-Resolution
Authors: name: Dachun Kai, name: Jiayao Lu, name: Yueyi Zhang, name: Xiaoyan Sun
Published: 2024-06-19 11:27:44

Event-based vision has drawn increasing attention due to its unique
characteristics, such as high temporal resolution and high dynamic range. It
has been used in video super-resolution (VSR) recently to enhance the flow
estimation and temporal alignment. Rather than for motion learning, we propose
in this paper the first VSR method that utilizes event signals for texture
enhancement. Our method, called EvTexture, leverages high-frequency details of
events to better recover texture regions in VSR. In our EvTexture, a new
texture enhancement branch is presented. We further introduce an iterative
texture enhancement module to progressively explore the
high-temporal-resolution event information for texture restoration. This allows
for gradual refinement of texture regions across multiple iterations, leading
to more accurate and rich high-resolution details. Experimental results show
that our EvTexture achieves state-of-the-art performance on four datasets. For
the Vid4 dataset with rich textures, our method can get up to 4.67dB gain
compared with recent event-based methods. Code:
https://github.com/DachunKai/EvTexture.
A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
Authors: name: Florin Cuconasu, name: Giovanni Trappolini, name: Nicola Tonellotto, name: Fabrizio Silvestri
Published: 2024-06-21 08:31:02

Retrieval Augmented Generation (RAG) represents a significant advancement in
artificial intelligence combining a retrieval phase with a generative phase,
with the latter typically being powered by large language models (LLMs). The
current common practices in RAG involve using "instructed" LLMs, which are
fine-tuned with supervised training to enhance their ability to follow
instructions and are aligned with human preferences using state-of-the-art
techniques. Contrary to popular belief, our study demonstrates that base models
outperform their instructed counterparts in RAG tasks by 20% on average under
our experimental settings. This finding challenges the prevailing assumptions
about the superiority of instructed LLMs in RAG applications. Further
investigations reveal a more nuanced situation, questioning fundamental aspects
of RAG and suggesting the need for broader discussions on the topic; or, as
Fromm would have it, "Seldom is a glance at the statistics enough to understand
the meaning of the figures".
Towards Retrieval Augmented Generation over Large Video Libraries
Authors: name: Yannis Tevissen, name: Khalil Guetari, name: Frédéric Petitpont
Published: 2024-06-21 07:52:01

Video content creators need efficient tools to repurpose content, a task that
often requires complex manual or automated searches. Crafting a new video from
large video libraries remains a challenge. In this paper we introduce the task
of Video Library Question Answering (VLQA) through an interoperable
architecture that applies Retrieval Augmented Generation (RAG) to video
libraries. We propose a system that uses large language models (LLMs) to
generate search queries, retrieving relevant video moments indexed by speech
and visual metadata. An answer generation module then integrates user queries
with this metadata to produce responses with specific video timestamps. This
approach shows promise in multimedia content retrieval, and AI-assisted video
content creation.
Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images
Authors: name: Haruo Fujiwara, name: Yusuke Mukuta, name: Tatsuya Harada
Published: 2024-06-19 09:36:18

We propose a simple yet effective pipeline for stylizing a 3D scene,
harnessing the power of 2D image diffusion models. Given a NeRF model
reconstructed from a set of multi-view images, we perform 3D style transfer by
refining the source NeRF model using stylized images generated by a
style-aligned image-to-image diffusion model. Given a target style prompt, we
first generate perceptually similar multi-view images by leveraging a
depth-conditioned diffusion model with an attention-sharing mechanism. Next,
based on the stylized multi-view images, we propose to guide the style transfer
process with the sliced Wasserstein loss based on the feature maps extracted
from a pre-trained CNN model. Our pipeline consists of decoupled steps,
allowing users to test various prompt ideas and preview the stylized 3D result
before proceeding to the NeRF fine-tuning stage. We demonstrate that our method
can transfer diverse artistic styles to real-world 3D scenes with competitive
quality.
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and
  Benchmarking
Authors: name: Daniel Dauner, name: Marcel Hallgarten, name: Tianyu Li, name: Xinshuo Weng, name: Zhiyu Huang, name: Zetong Yang, name: Hongyang Li, name: Igor Gilitschenski, name: Boris Ivanovic, name: Marco Pavone, name: Andreas Geiger, name: Kashyap Chitta
Published: 2024-06-21 17:59:02

Benchmarking vision-based driving policies is challenging. On one hand,
open-loop evaluation with real data is easy, but these results do not reflect
closed-loop performance. On the other, closed-loop evaluation is possible in
simulation, but is hard to scale due to its significant computational demands.
Further, the simulators available today exhibit a large domain gap to real
data. This has resulted in an inability to draw clear conclusions from the
rapidly growing body of research on end-to-end autonomous driving. In this
paper, we present NAVSIM, a middle ground between these evaluation paradigms,
where we use large datasets in combination with a non-reactive simulator to
enable large-scale real-world benchmarking. Specifically, we gather
simulation-based metrics, such as progress and time to collision, by unrolling
bird's eye view abstractions of the test scenes for a short simulation horizon.
Our simulation is non-reactive, i.e., the evaluated policy and environment do
not influence each other. As we demonstrate empirically, this decoupling allows
open-loop metric computation while being better aligned with closed-loop
evaluations than traditional displacement errors. NAVSIM enabled a new
competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting
in several new insights. On a large set of challenging scenarios, we observe
that simple methods with moderate compute requirements such as TransFuser can
match recent large-scale end-to-end driving architectures such as UniAD. Our
modular framework can potentially be extended with new datasets, data curation
strategies, and metrics, and will be continually maintained to host future
challenges. Our code is available at
https://github.com/autonomousvision/navsim.
Cognitive Map for Language Models: Optimal Planning via Verbally
  Representing the World Model
Authors: name: Doyoung Kim, name: Jongwon Lee, name: Jinho Park, name: Minjoon Seo
Published: 2024-06-21 16:10:05

Language models have demonstrated impressive capabilities across various
natural language processing tasks, yet they struggle with planning tasks
requiring multi-step simulations. Inspired by human cognitive processes, this
paper investigates the optimal planning power of language models that can
construct a cognitive map of a given environment. Our experiments demonstrate
that cognitive map significantly enhances the performance of both optimal and
reachable planning generation ability in the Gridworld path planning task. We
observe that our method showcases two key characteristics similar to human
cognition: generalization of its planning ability to extrapolated
environments and rapid adaptation with limited training data. We hope our
findings in the Gridworld task provide insights into modeling human cognitive
processes in language models, potentially leading to the development of more
advanced and robust systems that better resemble human cognition.
Two Giraffes in a Dirt Field: Using Game Play to Investigate Situation
  Modelling in Large Multimodal Models
Authors: name: Sherzod Hakimov, name: Yerkezhan Abdullayeva, name: Kushal Koshti, name: Antonia Schmidt, name: Yan Weiser, name: Anne Beyer, name: David Schlangen
Published: 2024-06-20 06:56:19

While the situation has improved for text-only models, it again seems to be
the case currently that multimodal (text and image) models develop faster than
ways to evaluate them. In this paper, we bring a recently developed evaluation
paradigm from text models to multimodal models, namely evaluation through the
goal-oriented game (self) play, complementing reference-based and
preference-based evaluation. Specifically, we define games that challenge a
model's capability to represent a situation from visual information and align
such representations through dialogue. We find that the largest closed models
perform rather well on the games that we define, while even the best
open-weight models struggle with them. On further analysis, we find that the
exceptional deep captioning capabilities of the largest models drive some of
the performance. There is still room to grow for both kinds of models, ensuring
the continued relevance of the benchmark.
Judging the Judges: Evaluating Alignment and Vulnerabilities in
  LLMs-as-Judges
Authors: name: Aman Singh Thakur, name: Kartik Choudhary, name: Venkat Srinik Ramayapally, name: Sankaran Vaidyanathan, name: Dieuwke Hupkes
Published: 2024-06-18 13:49:54

Offering a promising solution to the scalability challenges associated with
human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an
approach to evaluating large language models (LLMs). However, there are still
many open questions about the strengths and weaknesses of this paradigm, and
what potential biases it may hold. In this paper, we present a comprehensive
study of the performance of various LLMs acting as judges. We leverage TriviaQA
as a benchmark for assessing objective knowledge reasoning of LLMs and evaluate
them alongside human annotations which we found to have a high inter-annotator
agreement. Our study includes 9 judge models and 9 exam taker models -- both
base and instruction-tuned. We assess the judge model's alignment across
different model sizes, families, and judge prompts. Among other results, our
research rediscovers the importance of using Cohen's kappa as a metric of
alignment as opposed to simple percent agreement, showing that judges with high
percent agreement can still assign vastly different scores. We find that both
Llama-3 70B and GPT-4 Turbo have an excellent alignment with humans, but in
terms of ranking exam taker models, they are outperformed by both JudgeLM-7B
and the lexical judge Contains, which have up to 34 points lower human
alignment. Through error analysis and various other studies, including the
effects of instruction length and leniency bias, we hope to provide valuable
lessons for using LLMs as judges in the future.
Learning Molecular Representation in a Cell
Authors: name: Gang Liu, name: Srijit Seal, name: John Arevalo, name: Zhenwen Liang, name: Anne E. Carpenter, name: Meng Jiang, name: Shantanu Singh
Published: 2024-06-17 19:48:42

Predicting drug efficacy and safety in vivo requires information on
biological responses (e.g., cell morphology and gene expression) to small
molecule perturbations. However, current molecular representation learning
methods do not provide a comprehensive view of cell states under these
perturbations and struggle to remove noise, hindering model generalization. We
introduce the Information Alignment (InfoAlign) approach to learn molecular
representations through the information bottleneck method in cells. We
integrate molecules and cellular response data as nodes into a context graph,
connecting them with weighted edges based on chemical, biological, and
computational criteria. For each molecule in a training batch, InfoAlign
optimizes the encoder's latent representation with a minimality objective to
discard redundant structural information. A sufficiency objective decodes the
representation to align with different feature spaces from the molecule's
neighborhood in the context graph. We demonstrate that the proposed sufficiency
objective for alignment is tighter than existing encoder-based contrastive
methods. Empirically, we validate representations from InfoAlign in two
downstream tasks: molecular property prediction against up to 19 baseline
methods across four datasets, plus zero-shot molecule-morphology matching.
Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical
  Report
Authors: name: Franz Louis Cesista
Published: 2024-06-17 10:45:47

Multimodal Foundation Models (MMFMs) have shown remarkable performance on
various computer vision and natural language processing tasks. However, their
performance on particular tasks such as document understanding is still
limited. They also require more compute, time, and engineering resources to
finetune and deploy compared to traditional, unimodal models. In this report,
we present Multimodal Structured Generation, a general framework which
constrains the output logits of frozen MMFMs to force them to reason before
responding with structured outputs that downstream APIs can parse and use. We
provide a detailed account of our approach, including the technical details,
theoretical discussions, and final evaluation results in the 2nd Multimodal
Foundation Models Challenge hosted by the Computer Vision and Pattern
Recognition (CVPR) conference. Our approach achieved the second highest score
in the hidden test set for Phase 2 and third highest overall. This shows the
method's ability to generalize to unseen tasks. And that simple engineering can
beat expensive & complicated modelling steps as we first discussed in our
paper, Retrieval Augmented Structured Generation: Business Document Information
Extraction as Tool Use. All of our scripts, deployment steps, and evaluation
results can be accessed in https://github.com/leloykun/MMFM-Challenge
Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
Authors: name: Chia-Yu Hung, name: Navonil Majumder, name: Ambuj Mehrish, name: Soujanya Poria
Published: 2024-06-21 14:35:16

The widespread applicability and increasing omnipresence of LLMs have
instigated a need to align LLM responses to user and stakeholder preferences.
Many preference optimization approaches have been proposed that fine-tune LLM
parameters to achieve good alignment. However, such parameter tuning is known
to interfere with model performance on many tasks. Moreover, keeping up with
shifting user preferences is tricky in such a situation. Decoding-time
alignment with reward model guidance solves these issues at the cost of
increased inference time. However, most of such methods fail to strike the
right balance between exploration and exploitation of reward -- often due to
the conflated formulation of these two aspects - to give well-aligned
responses. To remedy this we decouple these two aspects and implement them in
an evolutionary fashion: exploration is enforced by decoding from mutated
instructions and exploitation is represented as the periodic replacement of
poorly-rewarded generations with well-rewarded ones. Empirical evidences
indicate that this strategy outperforms many preference optimization and
decode-time alignment approaches on two widely accepted alignment benchmarks
AlpacaEval 2 and MT-Bench. Our implementation will be available at:
https://darwin-alignment.github.io.
Stylebreeder: Exploring and Democratizing Artistic Styles through
  Text-to-Image Models
Authors: name: Matthew Zheng, name: Enis Simsar, name: Hidir Yesiltepe, name: Federico Tombari, name: Joel Simon, name: Pinar Yanardag
Published: 2024-06-20 17:59:56

Text-to-image models are becoming increasingly popular, revolutionizing the
landscape of digital art creation by enabling highly detailed and creative
visual content generation. These models have been widely employed across
various domains, particularly in art generation, where they facilitate a broad
spectrum of creative expression and democratize access to artistic creation. In
this paper, we introduce STYLEBREEDER, a comprehensive dataset of 6.8M
images and 1.8M prompts generated by 95K users on Artbreeder, a platform that
has emerged as a significant hub for creative exploration with over 13M users.
We introduce a series of tasks with this dataset aimed at identifying diverse
artistic styles, generating personalized content, and recommending styles based
on user interests. By documenting unique, user-generated styles that transcend
conventional categories like 'cyberpunk' or 'Picasso,' we explore the potential
for unique, crowd-sourced styles that could provide deep insights into the
collective creative psyche of users worldwide. We also evaluate different
personalization methods to enhance artistic expression and introduce a style
atlas, making these models available in LoRA format for public use. Our
research demonstrates the potential of text-to-image diffusion models to
uncover and promote unique artistic expressions, further democratizing AI in
art and fostering a more diverse and inclusive artistic community. The dataset,
code and models are available at https://stylebreeder.github.io under a Public
Domain (CC0) license.
MantisScore: Building Automatic Metrics to Simulate Fine-grained Human
  Feedback for Video Generation
Authors: name: Xuan He, name: Dongfu Jiang, name: Ge Zhang, name: Max Ku, name: Achint Soni, name: Sherman Siu, name: Haonan Chen, name: Abhranil Chandra, name: Ziyan Jiang, name: Aaran Arulraj, name: Kai Wang, name: Quy Duc Do, name: Yuansheng Ni, name: Bohan Lyu, name: Yaswanth Narsupalli, name: Rongqi Fan, name: Zhiheng Lyu, name: Yuchen Lin, name: Wenhu Chen
Published: 2024-06-21 15:43:46

The recent years have witnessed great advances in video generation. However,
the development of automatic video metrics is lagging significantly behind.
None of the existing metric is able to provide reliable scores over generated
videos. The main barrier is the lack of large-scale human-annotated dataset. In
this paper, we release VideoFeedback, the first large-scale dataset containing
human-provided multi-aspect score over 37.6K synthesized videos from 11
existing video generative models. We train MantisScore (initialized from
Mantis) based on VideoFeedback to enable automatic video quality assessment.
Experiments show that the Spearman correlation between MantisScore and humans
can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about
50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and
VBench show that MantisScore has consistently much higher correlation with
human judges than other metrics. Due to these results, we believe MantisScore
can serve as a great proxy for human raters to (1) rate different video models
to track progress (2) simulate fine-grained human feedback in Reinforcement
Learning with Human Feedback (RLHF) to improve current video generation models.
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs
Authors: name: Ziyan Jiang, name: Xueguang Ma, name: Wenhu Chen
Published: 2024-06-21 17:23:21

In traditional RAG framework, the basic retrieval units are normally short.
The common retrievers like DPR normally work with 100-word Wikipedia
paragraphs. Such a design forces the retriever to search over a large corpus to
find the `needle' unit. In contrast, the readers only need to extract answers
from the short retrieved units. Such an imbalanced `heavy' retriever and
`light' reader design can lead to sub-optimal performance. In order to
alleviate the imbalance, we propose a new framework LongRAG, consisting of a
`long retriever' and a `long reader'. LongRAG processes the entire Wikipedia
into 4K-token units, which is 30x longer than before. By increasing the unit
size, we significantly reduce the total units from 22M to 700K. This
significantly lowers the burden of retriever, which leads to a remarkable
retrieval score: answer recall@1=71% on NQ (previously 52%) and answer
recall@2=72% (previously 47%) on HotpotQA (full-wiki). Then we feed the top-k
retrieved units (approx 30K tokens) to an existing long-context LLM to
perform zero-shot answer extraction. Without requiring any training, LongRAG
achieves an EM of 62.7% on NQ, which is the best known result. LongRAG also
achieves 64.3% on HotpotQA (full-wiki), which is on par of the SoTA model. Our
study offers insights into the future roadmap for combining RAG with
long-context LLMs.