Orionnnn/hub-stats huggingface dataset:
Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation
  Model Internet Agents
Authors: name: Yifei Zhou, name: Qianlan Yang, name: Kaixiang Lin, name: Min Bai, name: Xiong Zhou, name: Yu-Xiong Wang, name: Sergey Levine, name: Erran Li
Published: 2024-12-17 18:59:50

The vision of a broadly capable and goal-directed agent, such as an
Internet-browsing agent in the digital world and a household humanoid in the
physical world, has rapidly advanced, thanks to the generalization capability
of foundation models. Such a generalist agent needs to have a large and diverse
skill repertoire, such as finding directions between two travel locations and
buying specific items from the Internet. If each skill needs to be specified
manually through a fixed set of human-annotated instructions, the agent's skill
repertoire will necessarily be limited due to the quantity and diversity of
human-annotated instructions. In this work, we address this challenge by
proposing Proposer-Agent-Evaluator, an effective learning system that enables
foundation model agents to autonomously discover and practice skills in the
wild. At the heart of PAE is a context-aware task proposer that autonomously
proposes tasks for the agent to practice with context information of the
environment such as user demos or even just the name of the website itself for
Internet-browsing agents. Then, the agent policy attempts those tasks with
thoughts and actual grounded operations in the real world with resulting
trajectories evaluated by an autonomous VLM-based success evaluator. The
success evaluation serves as the reward signal for the agent to refine its
policies through RL. We validate PAE on challenging vision-based web
navigation, using both real-world and self-hosted websites from WebVoyager and
WebArena.To the best of our knowledge, this work represents the first effective
learning system to apply autonomous task proposal with RL for agents that
generalizes real-world human-annotated benchmarks with SOTA performances. Our
open-source checkpoints and code can be found in https://yanqval.github.io/PAE/
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal
  Retrieval-Augmented Generation
Authors: name: Manan Suri, name: Puneet Mathur, name: Franck Dernoncourt, name: Kanika Goswami, name: Ryan A. Rossi, name: Dinesh Manocha
Published: 2024-12-14 06:24:55

Understanding information from a collection of multiple documents,
particularly those with visually rich elements, is important for
document-grounded question answering. This paper introduces VisDoMBench, the
first comprehensive benchmark designed to evaluate QA systems in multi-document
settings with rich multimodal content, including tables, charts, and
presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval
Augmented Generation (RAG) approach that simultaneously utilizes visual and
textual RAG, combining robust visual retrieval capabilities with sophisticated
linguistic reasoning. VisDoMRAG employs a multi-step reasoning process
encompassing evidence curation and chain-of-thought reasoning for concurrent
textual and visual RAG pipelines. A key novelty of VisDoMRAG is its
consistency-constrained modality fusion mechanism, which aligns the reasoning
processes across modalities at inference time to produce a coherent final
answer. This leads to enhanced accuracy in scenarios where critical information
is distributed across modalities and improved answer verifiability through
implicit context attribution. Through extensive experiments involving
open-source and proprietary large language models, we benchmark
state-of-the-art document QA methods on VisDoMBench. Extensive results show
that VisDoMRAG outperforms unimodal and long-context LLM baselines for
end-to-end multimodal document QA by 12-20%.
When to Speak, When to Abstain: Contrastive Decoding with Abstention
Authors: name: Hyuhng Joon Kim, name: Youna Kim, name: Sang-goo Lee, name: Taeuk Kim
Published: 2024-12-17 04:38:08

Large Language Models (LLMs) demonstrate exceptional performance across
diverse tasks by leveraging both pre-trained knowledge (i.e., parametric
knowledge) and external knowledge (i.e., contextual knowledge). While
substantial efforts have been made to leverage both forms of knowledge,
scenarios in which the model lacks any relevant knowledge remain underexplored.
Such limitations can result in issues like hallucination, causing reduced
reliability and potential risks in high-stakes applications. To address such
limitations, this paper extends the task scope to encompass cases where the
user's request cannot be fulfilled due to the lack of relevant knowledge. To
this end, we introduce Contrastive Decoding with Abstention (CDA), a
training-free decoding method that empowers LLMs to generate responses when
relevant knowledge is available and to abstain otherwise. CDA evaluates the
relevance of each knowledge for a given query, adaptively determining which
knowledge to prioritize or which to completely ignore. Extensive experiments
with four LLMs on three question-answering datasets demonstrate that CDA can
effectively perform accurate generation and abstention simultaneously. These
findings highlight CDA's potential to broaden the applicability of LLMs,
enhancing reliability and preserving user trust.
MIVE: New Design and Benchmark for Multi-Instance Video Editing
Authors: name: Samuel Teodoro, name: Agus Gunawan, name: Soo Ye Kim, name: Jihyong Oh, name: Munchurl Kim
Published: 2024-12-17 13:00:04

Recent AI-based video editing has enabled users to edit videos through simple
text prompts, significantly simplifying the editing process. However, recent
zero-shot video editing techniques primarily focus on global or single-object
edits, which can lead to unintended changes in other parts of the video. When
multiple objects require localized edits, existing methods face challenges,
such as unfaithful editing, editing leakage, and lack of suitable evaluation
datasets and metrics. To overcome these limitations, we propose a zero-shot
Multi-Instance Video Editing
framework, called MIVE. MIVE is a general-purpose mask-based framework, not
dedicated to specific objects (e.g., people). MIVE introduces two key modules:
(i) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage and
(ii) Instance-centric Probability Redistribution (IPR) to ensure precise
localization and faithful editing. Additionally, we present our new MIVE
Dataset featuring diverse video scenarios and introduce the Cross-Instance
Accuracy (CIA) Score to evaluate editing leakage in multi-instance video
editing tasks. Our extensive qualitative, quantitative, and user study
evaluations demonstrate that MIVE significantly outperforms recent
state-of-the-art methods in terms of editing faithfulness, accuracy, and
leakage prevention, setting a new benchmark for multi-instance video editing.
The project page is available at https://kaist-viclab.github.io/mive-site/
Emergence of Abstractions: Concept Encoding and Decoding Mechanism for
  In-Context Learning in Transformers
Authors: name: Seungwook Han, name: Jinyeop Song, name: Jeff Gore, name: Pulkit Agrawal
Published: 2024-12-16 19:00:18

Humans distill complex experiences into fundamental abstractions that enable
rapid learning and adaptation. Similarly, autoregressive transformers exhibit
adaptive learning through in-context learning (ICL), which begs the question of
how. In this paper, we propose concept encoding-decoding mechanism to
explain ICL by studying how transformers form and use internal abstractions in
their representations. On synthetic ICL tasks, we analyze the training dynamics
of a small transformer and report the coupled emergence of concept encoding and
decoding. As the model learns to encode different latent concepts (e.g.,
``Finding the first noun in a sentence.") into distinct, separable
representations, it concureently builds conditional decoding algorithms and
improve its ICL performance. We validate the existence of this mechanism across
pretrained models of varying scales (Gemma-2 2B/9B/27B, Llama-3.1 8B/70B).
Further, through mechanistic interventions and controlled finetuning, we
demonstrate that the quality of concept encoding is causally related and
predictive of ICL performance. Our empirical insights shed light into better
understanding the success and failure modes of large language models via their
representations.
OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in
  Financial Domain
Authors: name: Shuting Wang, name: Jiejun Tan, name: Zhicheng Dou, name: Ji-Rong Wen
Published: 2024-12-17 15:38:42

As a typical and practical application of Large Language Models (LLMs),
Retrieval-Augmented Generation (RAG) techniques have gained extensive
attention, particularly in vertical domains where LLMs may lack domain-specific
knowledge. In this paper, we introduce an omnidirectional and automatic RAG
benchmark, OmniEval, in the financial domain. Our benchmark is characterized by
its multi-dimensional evaluation framework, including (1) a matrix-based RAG
scenario evaluation system that categorizes queries into five task classes and
16 financial topics, leading to a structured assessment of diverse query
scenarios; (2) a multi-dimensional evaluation data generation approach, which
combines GPT-4-based automatic generation and human annotation, achieving an
87.47\% acceptance ratio in human evaluations on generated instances; (3) a
multi-stage evaluation system that evaluates both retrieval and generation
performance, result in a comprehensive evaluation on the RAG pipeline; and (4)
robust evaluation metrics derived from rule-based and LLM-based ones, enhancing
the reliability of assessments through manual annotations and supervised
fine-tuning of an LLM evaluator. Our experiments demonstrate the
comprehensiveness of OmniEval, which includes extensive test datasets and
highlights the performance variations of RAG systems across diverse topics and
tasks, revealing significant opportunities for RAG models to improve their
capabilities in vertical domains. We open source the code of our benchmark in
https://github.com/RUC-NLPIR/OmniEval{https://github.com/RUC-NLPIR/OmniEval}.
Multi-Dimensional Insights: Benchmarking Real-World Personalization in
  Large Multimodal Models
Authors: name: YiFan Zhang, name: Shanglin Lei, name: Runqi Qiao, name: Zhuoma GongQue, name: Xiaoshuai Song, name: Guanting Dong, name: Qiuna Tan, name: Zhe Wei, name: Peiqing Yang, name: Ye Tian, name: Yadong Xue, name: Xiaofei Wang, name: Honggang Zhang
Published: 2024-12-17 07:06:10

The rapidly developing field of large multimodal models (LMMs) has led to the
emergence of diverse models with remarkable capabilities. However, existing
benchmarks fail to comprehensively, objectively and accurately evaluate whether
LMMs align with the diverse needs of humans in real-world scenarios. To bridge
this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which
includes over 500 images covering six common scenarios of human life. Notably,
the MDI-Benchmark offers two significant advantages over existing evaluations:
(1) Each image is accompanied by two types of questions: simple questions to
assess the model's understanding of the image, and complex questions to
evaluate the model's ability to analyze and reason beyond basic content. (2)
Recognizing that people of different age groups have varying needs and
perspectives when faced with the same scenario, our benchmark stratifies
questions into three age categories: young people, middle-aged people, and
older people. This design allows for a detailed assessment of LMMs'
capabilities in meeting the preferences and needs of different age groups. With
MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related
tasks, indicating that existing LMMs still have considerable room for
improvement in addressing real-world applications. Looking ahead, we anticipate
that the MDI-Benchmark will open new pathways for aligning real-world
personalization in LMMs. The MDI-Benchmark data and evaluation code are
available at https://mdi-benchmark.github.io/
Are Your LLMs Capable of Stable Reasoning?
Authors: name: Junnan Liu, name: Hongwei Liu, name: Linchen Xiao, name: Ziyi Wang, name: Kuikun Liu, name: Songyang Gao, name: Wenwei Zhang, name: Songyang Zhang, name: Kai Chen
Published: 2024-12-17 18:12:47

The rapid advancement of Large Language Models (LLMs) has demonstrated
remarkable progress in complex reasoning tasks. However, a significant
discrepancy persists between benchmark performances and real-world
applications. We identify this gap as primarily stemming from current
evaluation protocols and metrics, which inadequately capture the full spectrum
of LLM capabilities, particularly in complex reasoning tasks where both
accuracy and consistency are crucial. This work makes two key contributions.
First, we introduce G-Pass@k, a novel evaluation metric that provides a
continuous assessment of model performance across multiple sampling attempts,
quantifying both the model's peak performance potential and its stability.
Second, we present LiveMathBench, a dynamic benchmark comprising challenging,
contemporary mathematical problems designed to minimize data leakage risks
during evaluation. Through extensive experiments using G-Pass@k on
state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights
into both their maximum capabilities and operational consistency. Our findings
reveal substantial room for improvement in LLMs' "realistic" reasoning
capabilities, highlighting the need for more robust evaluation methods. The
benchmark and detailed results are available at:
https://github.com/open-compass/GPassK.
GeoX: Geometric Problem Solving Through Unified Formalized
  Vision-Language Pre-training
Authors: name: Renqiu Xia, name: Mingsheng Li, name: Hancheng Ye, name: Wenjie Wu, name: Hongbin Zhou, name: Jiakang Yuan, name: Tianshuo Peng, name: Xinyu Cai, name: Xiangchao Yan, name: Bin Wang, name: Conghui He, name: Botian Shi, name: Tao Chen, name: Junchi Yan, name: Bo Zhang
Published: 2024-12-16 15:20:03

Despite their proficiency in general tasks, Multi-modal Large Language Models
(MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands
understanding diagrams, interpreting symbols, and performing complex reasoning.
This limitation arises from their pre-training on natural images and texts,
along with the lack of automated verification in the problem-solving process.
Besides, current geometric specialists are limited by their task-specific
designs, making them less effective for broader geometric problems. To this
end, we present GeoX, a multi-modal large model focusing on geometric
understanding and reasoning tasks. Given the significant differences between
geometric diagram-symbol and natural image-text, we introduce unimodal
pre-training to develop a diagram encoder and symbol decoder, enhancing the
understanding of geometric images and corpora. Furthermore, we introduce
geometry-language alignment, an effective pre-training paradigm that bridges
the modality gap between unimodal geometric experts. We propose a
Generator-And-Sampler Transformer (GS-Former) to generate discriminative
queries and eliminate uninformative representations from unevenly distributed
geometric signals. Finally, GeoX benefits from visual instruction tuning,
empowering it to take geometric images and questions as input and generate
verifiable solutions. Experiments show that GeoX outperforms both generalists
and geometric specialists on publicly recognized benchmarks, such as GeoQA,
UniGeo, Geometry3K, and PGPS9k.
SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D
  Gaussians from Monocular Video
Authors: name: Jongmin Park, name: Minh-Quan Viet Bui, name: Juan Luis Gonzalez Bello, name: Jaeho Moon, name: Jihyong Oh, name: Munchurl Kim
Published: 2024-12-13 09:09:14

Synthesizing novel views from in-the-wild monocular videos is challenging due
to scene dynamics and the lack of multi-view cues. To address this, we propose
SplineGS, a COLMAP-free dynamic 3D Gaussian Splatting (3DGS) framework for
high-quality reconstruction and fast rendering from monocular videos. At its
core is a novel Motion-Adaptive Spline (MAS) method, which represents
continuous dynamic 3D Gaussian trajectories using cubic Hermite splines with a
small number of control points. For MAS, we introduce a Motion-Adaptive Control
points Pruning (MACP) method to model the deformation of each dynamic 3D
Gaussian across varying motions, progressively pruning control points while
maintaining dynamic modeling integrity. Additionally, we present a joint
optimization strategy for camera parameter estimation and 3D Gaussian
attributes, leveraging photometric and geometric consistency. This eliminates
the need for Structure-from-Motion preprocessing and enhances SplineGS's
robustness in real-world conditions. Experiments show that SplineGS
significantly outperforms state-of-the-art methods in novel view synthesis
quality for dynamic scenes from monocular videos, achieving thousands times
faster rendering speed.
Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion
  Models
Authors: name: Namhyuk Ahn, name: KiYoon Yoo, name: Wonhyuk Ahn, name: Daesik Kim, name: Seung-Hun Nam
Published: 2024-12-16 03:46:45

Recent advancements in diffusion models revolutionize image generation but
pose risks of misuse, such as replicating artworks or generating deepfakes.
Existing image protection methods, though effective, struggle to balance
protection efficacy, invisibility, and latency, thus limiting practical use. We
introduce perturbation pre-training to reduce latency and propose a
mixture-of-perturbations approach that dynamically adapts to input images to
minimize performance degradation. Our novel training strategy computes
protection loss across multiple VAE feature spaces, while adaptive targeted
protection at inference enhances robustness and invisibility. Experiments show
comparable protection performance with improved invisibility and drastically
reduced inference time. The code and demo are available at
https://webtoon.github.io/impasto
RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
Authors: name: Charles Xu, name: Qiyang Li, name: Jianlan Luo, name: Sergey Levine
Published: 2024-12-13 04:57:55

Recent advances in robotic foundation models have enabled the development of
generalist policies that can adapt to diverse tasks. While these models show
impressive flexibility, their performance heavily depends on the quality of
their training data. In this work, we propose Reinforcement Learning Distilled
Generalists (RLDG), a method that leverages reinforcement learning to generate
high-quality training data for finetuning generalist policies. Through
extensive real-world experiments on precise manipulation tasks like connector
insertion and assembly, we demonstrate that generalist policies trained with
RL-generated data consistently outperform those trained with human
demonstrations, achieving up to 40% higher success rates while generalizing
better to new tasks. We also provide a detailed analysis that reveals this
performance gain stems from both optimized action distributions and improved
state coverage. Our results suggest that combining task-specific RL with
generalist policy distillation offers a promising approach for developing more
capable and efficient robotic manipulation systems that maintain the
flexibility of foundation models while achieving the performance of specialized
controllers. Videos and code can be found on our project website
https://generalist-distillation.github.io
MaxInfoRL: Boosting exploration in reinforcement learning through
  information gain maximization
Authors: name: Bhavya Sukhija, name: Stelian Coros, name: Andreas Krause, name: Pieter Abbeel, name: Carmelo Sferrazza
Published: 2024-12-16 18:59:53

Reinforcement learning (RL) algorithms aim to balance exploiting the current
best strategy with exploring new options that could lead to higher rewards.
Most common RL algorithms use undirected exploration, i.e., select random
sequences of actions. Exploration can also be directed using intrinsic rewards,
such as curiosity or model epistemic uncertainty. However, effectively
balancing task and intrinsic rewards is challenging and often task-dependent.
In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and
extrinsic exploration. MaxInfoRL steers exploration towards informative
transitions, by maximizing intrinsic rewards such as the information gain about
the underlying task. When combined with Boltzmann exploration, this approach
naturally trades off maximization of the value function with that of the
entropy over states, rewards, and actions. We show that our approach achieves
sublinear regret in the simplified setting of multi-armed bandits. We then
apply this general formulation to a variety of off-policy model-free RL methods
for continuous state-action spaces, yielding novel algorithms that achieve
superior performance across hard exploration problems and complex scenarios
such as visual control tasks.
VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video
  Face Swapping
Authors: name: Hao Shao, name: Shulun Wang, name: Yang Zhou, name: Guanglu Song, name: Dailan He, name: Shuo Qin, name: Zhuofan Zong, name: Bingqi Ma, name: Yu Liu, name: Hongsheng Li
Published: 2024-12-15 18:58:32

Video face swapping is becoming increasingly popular across various
applications, yet existing methods primarily focus on static images and
struggle with video face swapping because of temporal consistency and complex
scenarios. In this paper, we present the first diffusion-based framework
specifically designed for video face swapping. Our approach introduces a novel
image-video hybrid training framework that leverages both abundant static image
data and temporal video sequences, addressing the inherent limitations of
video-only training. The framework incorporates a specially designed diffusion
model coupled with a VidFaceVAE that effectively processes both types of data
to better maintain temporal coherence of the generated videos. To further
disentangle identity and pose features, we construct the Attribute-Identity
Disentanglement Triplet (AIDT) Dataset, where each triplet has three face
images, with two images sharing the same pose and two sharing the same
identity. Enhanced with a comprehensive occlusion augmentation, this dataset
also improves robustness against occlusions. Additionally, we integrate 3D
reconstruction techniques as input conditioning to our network for handling
large pose variations. Extensive experiments demonstrate that our framework
achieves superior performance in identity preservation, temporal consistency,
and visual quality compared to existing methods, while requiring fewer
inference steps. Our approach effectively mitigates key challenges in video
face swapping, including temporal flickering, identity preservation, and
robustness to occlusions and pose variations.