Orionnnn/hub-stats huggingface dataset:
Unraveling the Complexity of Memory in RL Agents: an Approach for
  Classification and Evaluation
Authors: name: Egor Cherepanov, name: Nikita Kachaev, name: Artem Zholus, name: Alexey K. Kovalev, name: Aleksandr I. Panov
Published: 2024-12-09 14:34:31

The incorporation of memory into agents is essential for numerous tasks
within the domain of Reinforcement Learning (RL). In particular, memory is
paramount for tasks that require the utilization of past information,
adaptation to novel environments, and improved sample efficiency. However, the
term ``memory'' encompasses a wide range of concepts, which, coupled with the
lack of a unified methodology for validating an agent's memory, leads to
erroneous judgments about agents' memory capabilities and prevents objective
comparison with other memory-enhanced agents. This paper aims to streamline the
concept of memory in RL by providing practical precise definitions of agent
memory types, such as long-term versus short-term memory and declarative versus
procedural memory, inspired by cognitive science. Using these definitions, we
categorize different classes of agent memory, propose a robust experimental
methodology for evaluating the memory capabilities of RL agents, and
standardize evaluations. Furthermore, we empirically demonstrate the importance
of adhering to the proposed methodology when evaluating different types of
agent memory by conducting experiments with different RL agents and what its
violation leads to.
Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
Authors: name: Yuying Ge, name: Yizhuo Li, name: Yixiao Ge, name: Ying Shan
Published: 2024-12-05 18:53:04

In recent years, there has been a significant surge of interest in unifying
image comprehension and generation within Large Language Models (LLMs). This
growing interest has prompted us to explore extending this unification to
videos. The core challenge lies in developing a versatile video tokenizer that
captures both the spatial characteristics and temporal dynamics of videos to
obtain representations for LLMs, and the representations can be further decoded
into realistic video clips to enable video generation. In this work, we
introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the
diffusion process for self-supervised video representation learning. We posit
that if a video diffusion model can effectively de-noise video clips by taking
the features of a video tokenizer as the condition, then the tokenizer has
successfully captured robust spatial and temporal information. Additionally,
the video diffusion model inherently functions as a de-tokenizer, decoding
videos from their representations. Building upon the Divot tokenizer, we
present Divot-Vicuna through video-to-text autoregression and text-to-video
generation by modeling the distributions of continuous-valued Divot features
with a Gaussian Mixture Model. Experimental results demonstrate that our
diffusion-based video tokenizer, when integrated with a pre-trained LLM,
achieves competitive performance across various video comprehension and
generation benchmarks. The instruction tuned Divot-Vicuna also excels in video
storytelling, generating interleaved narratives and corresponding videos.
Exploring Multi-Grained Concept Annotations for Multimodal Large
  Language Models
Authors: name: Xiao Xu, name: Tianhao Niu, name: Yuxi Xie, name: Libo Qin, name: Wanxiang Che, name: Min-Yen Kan
Published: 2024-12-08 13:45:44

Multimodal Large Language Models (MLLMs) excel in vision--language tasks by
pre-training solely on coarse-grained concept annotations (e.g., image
captions). We hypothesize that integrating fine-grained concept annotations
(e.g., object labels and object regions) will further improve performance, as
both data granularities complement each other in terms of breadth and depth in
concept representation. We introduce a new dataset featuring Multimodal
Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we
explore the impact of different data recipes on multimodal comprehension and
generation. Our analyses reveal that multi-grained concept annotations
integrate and complement each other, under our structured template and a
general MLLM framework. We clearly explore and demonstrate the potential of
MMGiC to help MLLMs better locate and learn concepts, aligning vision and
language at multiple granularities. We further validate our hypothesis by
investigating the fair comparison and effective collaboration between MMGiC and
image--caption data on 12 multimodal comprehension and generation benchmarks,
e.g., their appropriate combination achieve 3.95% and 2.34% absolute
improvements over image--caption data alone on POPE and SEED-Bench. Code, data
and models will be available at https://github.com/LooperXX/MMGiC.
You See it, You Got it: Learning 3D Creation on Pose-Free Videos at
  Scale
Authors: name: Baorui Ma, name: Huachen Gao, name: Haoge Deng, name: Zhengxiong Luo, name: Tiejun Huang, name: Lulu Tang, name: Xinlong Wang
Published: 2024-12-09 17:44:56

Recent 3D generation models typically rely on limited-scale 3D `gold-labels'
or 2D diffusion priors for 3D content creation. However, their performance is
upper-bounded by constrained 3D priors due to the lack of scalable learning
paradigms. In this work, we present See3D, a visual-conditional multi-view
diffusion model trained on large-scale Internet videos for open-world 3D
creation. The model aims to Get 3D knowledge by solely Seeing the visual
contents from the vast and rapidly growing video data -- You See it, You Got
it. To achieve this, we first scale up the training data using a proposed data
curation pipeline that automatically filters out multi-view inconsistencies and
insufficient observations from source videos. This results in a high-quality,
richly diverse, large-scale dataset of multi-view images, termed WebVi3D,
containing 320M frames from 16M video clips. Nevertheless, learning generic 3D
priors from videos without explicit 3D geometry or camera pose annotations is
nontrivial, and annotating poses for web-scale videos is prohibitively
expensive. To eliminate the need for pose conditions, we introduce an
innovative visual-condition - a purely 2D-inductive visual signal generated by
adding time-dependent noise to the masked video data. Finally, we introduce a
novel visual-conditional 3D generation framework by integrating See3D into a
warping-based pipeline for high-fidelity 3D generation. Our numerical and
visual comparisons on single and sparse reconstruction benchmarks show that
See3D, trained on cost-effective and scalable video data, achieves notable
zero-shot and open-world generation capabilities, markedly outperforming models
trained on costly and constrained 3D datasets. Please refer to our project page
at: https://vision.baai.ac.cn/see3d
Robust Multi-bit Text Watermark with LLM-based Paraphrasers
Authors: name: Xiaojun Xu, name: Jinghan Jia, name: Yuanshun Yao, name: Yang Liu, name: Hang Li
Published: 2024-12-04 08:43:12

We propose an imperceptible multi-bit text watermark embedded by paraphrasing
with LLMs. We fine-tune a pair of LLM paraphrasers that are designed to behave
differently so that their paraphrasing difference reflected in the text
semantics can be identified by a trained decoder. To embed our multi-bit
watermark, we use two paraphrasers alternatively to encode the pre-defined
binary code at the sentence level. Then we use a text classifier as the decoder
to decode each bit of the watermark. Through extensive experiments, we show
that our watermarks can achieve over 99.99\% detection AUC with small (1.1B)
text paraphrasers while keeping the semantic information of the original
sentence. More importantly, our pipeline is robust under word substitution and
sentence paraphrasing perturbations and generalizes well to
out-of-distributional data. We also show the stealthiness of our watermark with
LLM-based evaluation. We open-source the code:
https://github.com/xiaojunxu/multi-bit-text-watermark.
ProcessBench: Identifying Process Errors in Mathematical Reasoning
Authors: name: Chujie Zheng, name: Zhenru Zhang, name: Beichen Zhang, name: Runji Lin, name: Keming Lu, name: Bowen Yu, name: Dayiheng Liu, name: Jingren Zhou, name: Junyang Lin
Published: 2024-12-09 15:11:40

As language models regularly make mistakes when solving math problems,
automated identification of errors in the reasoning process becomes
increasingly significant for their scalable oversight. In this paper, we
introduce ProcessBench for measuring the ability to identify erroneous steps in
mathematical reasoning. It consists of 3,400 test cases, primarily focused on
competition- and Olympiad-level math problems. Each test case contains a
step-by-step solution with error location annotated by human experts. Models
are required to identify the earliest step that contains an error, or conclude
that all steps are correct. We conduct extensive evaluation on ProcessBench,
involving two types of models: process reward models (PRMs) and critic models,
where for the latter we prompt general language models to critique each
solution step by step. We draw two main observations: (1) Existing PRMs
typically fail to generalize to more challenging math problems beyond GSM8K and
MATH. They underperform both critic models (i.e., prompted general language
models) and our own trained PRM that is straightforwardly fine-tuned on the
PRM800K dataset. (2) The best open-source model, QwQ-32B-Preview, has
demonstrated the critique capability competitive with the proprietary model
GPT-4o, despite that it still lags behind the reasoning-specialized o1-mini. We
hope ProcessBench can foster future research in reasoning process assessment,
paving the way toward scalable oversight of language models.
Training Large Language Models to Reason in a Continuous Latent Space
Authors: name: Shibo Hao, name: Sainbayar Sukhbaatar, name: DiJia Su, name: Xian Li, name: Zhiting Hu, name: Jason Weston, name: Yuandong Tian
Published: 2024-12-09 18:55:56

Large language models (LLMs) are restricted to reason in the "language
space", where they typically express the reasoning process with a
chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue
that language space may not always be optimal for reasoning. For example, most
word tokens are primarily for textual coherence and not essential for
reasoning, while some critical tokens require complex planning and pose huge
challenges to LLMs. To explore the potential of LLM reasoning in an
unrestricted latent space instead of using natural language, we introduce a new
paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden
state of the LLM as a representation of the reasoning state (termed "continuous
thought"). Rather than decoding this into a word token, we feed it back to the
LLM as the subsequent input embedding directly in the continuous space.
Experiments show that Coconut can effectively augment the LLM on several
reasoning tasks. This novel latent reasoning paradigm leads to emergent
advanced reasoning patterns: the continuous thought can encode multiple
alternative next reasoning steps, allowing the model to perform a breadth-first
search (BFS) to solve the problem, rather than prematurely committing to a
single deterministic path like CoT. Coconut outperforms CoT in certain logical
reasoning tasks that require substantial backtracking during planning, with
fewer thinking tokens during inference. These findings demonstrate the promise
of latent reasoning and offer valuable insights for future research.
CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive
  Prediction
Authors: name: Zhefei Gong, name: Pengxiang Ding, name: Shangke Lyu, name: Siteng Huang, name: Mingyang Sun, name: Wei Zhao, name: Zhaoxin Fan, name: Donglin Wang
Published: 2024-12-09 18:59:18

In robotic visuomotor policy learning, diffusion-based models have achieved
significant success in improving the accuracy of action trajectory generation
compared to traditional autoregressive models. However, they suffer from
inefficiency due to multiple denoising steps and limited flexibility from
complex constraints. In this paper, we introduce Coarse-to-Fine AutoRegressive
Policy (CARP), a novel paradigm for visuomotor policy learning that redefines
the autoregressive action generation process as a coarse-to-fine, next-scale
approach. CARP decouples action generation into two stages: first, an action
autoencoder learns multi-scale representations of the entire action sequence;
then, a GPT-style transformer refines the sequence prediction through a
coarse-to-fine autoregressive process. This straightforward and intuitive
approach produces highly accurate and smooth actions, matching or even
surpassing the performance of diffusion-based policies while maintaining
efficiency on par with autoregressive policies. We conduct extensive
evaluations across diverse settings, including single-task and multi-task
scenarios on state-based and image-based simulation benchmarks, as well as
real-world tasks. CARP achieves competitive success rates, with up to a 10%
improvement, and delivers 10x faster inference compared to state-of-the-art
policies, establishing a high-performance, efficient, and flexible paradigm for
action generation in robotic tasks.
RL Zero: Zero-Shot Language to Behaviors without any Supervision
Authors: name: Harshit Sikchi, name: Siddhant Agarwal, name: Pranaya Jajoo, name: Samyak Parajuli, name: Caleb Chuck, name: Max Rudolph, name: Peter Stone, name: Amy Zhang, name: Scott Niekum
Published: 2024-12-07 18:31:16

Rewards remain an uninterpretable way to specify tasks for Reinforcement
Learning, as humans are often unable to predict the optimal behavior of any
given reward function, leading to poor reward design and reward hacking.
Language presents an appealing way to communicate intent to agents and bypass
reward design, but prior efforts to do so have been limited by costly and
unscalable labeling efforts. In this work, we propose a method for a completely
unsupervised alternative to grounding language instructions in a zero-shot
manner to obtain policies. We present a solution that takes the form of
imagine, project, and imitate: The agent imagines the observation sequence
corresponding to the language description of a task, projects the imagined
sequence to our target domain, and grounds it to a policy. Video-language
models allow us to imagine task descriptions that leverage knowledge of tasks
learned from internet-scale video-text mappings. The challenge remains to
ground these generations to a policy. In this work, we show that we can achieve
a zero-shot language-to-behavior policy by first grounding the imagined
sequences in real observations of an unsupervised RL agent and using a
closed-form solution to imitation learning that allows the RL agent to mimic
the grounded observations. Our method, RLZero, is the first to our knowledge to
show zero-shot language to behavior generation abilities without any
supervision on a variety of tasks on simulated domains. We further show that
RLZero can also generate policies zero-shot from cross-embodied videos such as
those scraped from YouTube.
BigDocs: An Open and Permissively-Licensed Dataset for Training
  Multimodal Models on Document and Code Tasks
Authors: name: Juan Rodriguez, name: Xiangru Jian, name: Siba Smarak Panigrahi, name: Tianyu Zhang, name: Aarash Feizi, name: Abhay Puri, name: Akshay Kalkunte, name: François Savard, name: Ahmed Masry, name: Shravan Nayak, name: Rabiul Awal, name: Mahsa Massoud, name: Amirhossein Abaskohi, name: Zichao Li, name: Suyuchen Wang, name: Pierre-André Noël, name: Mats Leon Richter, name: Saverio Vadacchino, name: Shubbam Agarwal, name: Sanket Biswas, name: Sara Shanian, name: Ying Zhang, name: Noah Bolger, name: Kurt MacDonald, name: Simon Fauvel, name: Sathwik Tejaswi, name: Srinivas Sunkara, name: Joao Monteiro, name: Krishnamurthy DJ Dvijotham, name: Torsten Scholak, name: Nicolas Chapados, name: Sepideh Kharagani, name: Sean Hughes, name: M. Özsu, name: Siva Reddy, name: Marco Pedersoli, name: Yoshua Bengio, name: Christopher Pal, name: Issam Laradji, name: Spandanna Gella, name: Perouz Taslakian, name: David Vazquez, name: Sai Rajeswar
Published: 2024-12-05 21:41:20

Multimodal AI has the potential to significantly enhance
document-understanding tasks, such as processing receipts, understanding
workflows, extracting data from documents, and summarizing reports. Code
generation tasks that require long-structured outputs can also be enhanced by
multimodality. Despite this, their use in commercial applications is often
limited due to limited access to training data and restrictive licensing, which
hinders open access. To address these limitations, we introduce BigDocs-7.5M, a
high-quality, open-access dataset comprising 7.5 million multimodal documents
across 30 tasks. We use an efficient data curation process to ensure our data
is high-quality and license-permissive. Our process emphasizes accountability,
responsibility, and transparency through filtering rules, traceable metadata,
and careful content analysis. Additionally, we introduce BigDocs-Bench, a
benchmark suite with 10 novel tasks where we create datasets that reflect
real-world use cases involving reasoning over Graphical User Interfaces (GUI)
and code generation from images. Our experiments show that training with
BigDocs-Bench improves average performance up to 25.8% over closed-source
GPT-4o in document reasoning and structured output tasks such as
Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a
preference for outputs from models trained on BigDocs over GPT-4o. This
suggests that BigDocs can help both academics and the open-source community
utilize and improve AI tools to enhance multimodal capabilities and document
reasoning. The project is hosted at https://bigdocs.github.io .
CompCap: Improving Multimodal Large Language Models with Composite
  Captions
Authors: name: Xiaohui Chen, name: Satya Narayan Shukla, name: Mahmoud Azab, name: Aashu Singh, name: Qifan Wang, name: David Yang, name: ShengYun Peng, name: Hanchao Yu, name: Shen Yan, name: Xuewen Zhang, name: Baosheng He
Published: 2024-12-06 18:22:47

How well can Multimodal Large Language Models (MLLMs) understand composite
images? Composite images (CIs) are synthetic visuals created by merging
multiple visual elements, such as charts, posters, or screenshots, rather than
being captured directly by a camera. While CIs are prevalent in real-world
applications, recent MLLM developments have primarily focused on interpreting
natural images (NIs). Our research reveals that current MLLMs face significant
challenges in accurately understanding CIs, often struggling to extract
information or perform complex reasoning based on these images. We find that
existing training data for CIs are mostly formatted for question-answer tasks
(e.g., in datasets like ChartQA and ScienceQA), while high-quality
image-caption datasets, critical for robust vision-language alignment, are only
available for NIs. To bridge this gap, we introduce Composite Captions
(CompCap), a flexible framework that leverages Large Language Models (LLMs) and
automation tools to synthesize CIs with accurate and detailed captions. Using
CompCap, we curate CompCap-118K, a dataset containing 118K image-caption pairs
across six CI types. We validate the effectiveness of CompCap-118K by
supervised fine-tuning MLLMs of three sizes: xGen-MM-inst.-4B and
LLaVA-NeXT-Vicuna-7B/13B. Empirical results show that CompCap-118K
significantly enhances MLLMs' understanding of CIs, yielding average gains of
1.7%, 2.0%, and 2.9% across eleven benchmarks, respectively.
PanoDreamer: 3D Panorama Synthesis from a Single Image
Authors: name: Avinash Paliwal, name: Xilong Zhou, name: Andrii Tsarov, name: Nima Khademi Kalantari
Published: 2024-12-06 07:42:48

In this paper, we present PanoDreamer, a novel method for producing a
coherent 360^circ 3D scene from a single input image. Unlike existing
methods that generate the scene sequentially, we frame the problem as
single-image panorama and depth estimation. Once the coherent panoramic image
and its corresponding depth are obtained, the scene can be reconstructed by
inpainting the small occluded regions and projecting them into 3D space. Our
key contribution is formulating single-image panorama and depth estimation as
two optimization tasks and introducing alternating minimization strategies to
effectively solve their objectives. We demonstrate that our approach
outperforms existing techniques in single-image 360^circ scene
reconstruction in terms of consistency and overall quality.
Momentum-GS: Momentum Gaussian Self-Distillation for High-Quality Large
  Scene Reconstruction
Authors: name: Jixuan Fan, name: Wanhua Li, name: Yifei Han, name: Yansong Tang
Published: 2024-12-06 09:31:12

3D Gaussian Splatting has demonstrated notable success in large-scale scene
reconstruction, but challenges persist due to high training memory consumption
and storage overhead. Hybrid representations that integrate implicit and
explicit features offer a way to mitigate these limitations. However, when
applied in parallelized block-wise training, two critical issues arise since
reconstruction accuracy deteriorates due to reduced data diversity when
training each block independently, and parallel training restricts the number
of divided blocks to the available number of GPUs. To address these issues, we
propose Momentum-GS, a novel approach that leverages momentum-based
self-distillation to promote consistency and accuracy across the blocks while
decoupling the number of blocks from the physical GPU count. Our method
maintains a teacher Gaussian decoder updated with momentum, ensuring a stable
reference during training. This teacher provides each block with global
guidance in a self-distillation manner, promoting spatial consistency in
reconstruction. To further ensure consistency across the blocks, we incorporate
block weighting, dynamically adjusting each block's weight according to its
reconstruction accuracy. Extensive experiments on large-scale scenes show that
our method consistently outperforms existing techniques, achieving a 12.8%
improvement in LPIPS over CityGaussian with much fewer divided blocks and
establishing a new state of the art. Project page:
https://jixuan-fan.github.io/Momentum-GS_Page/
DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling
Authors: name: Minzheng Wang, name: Xinghua Zhang, name: Kun Chen, name: Nan Xu, name: Haiyang Yu, name: Fei Huang, name: Wenji Mao, name: Yongbin Li
Published: 2024-12-06 10:01:38

Large language models (LLMs) have made dialogue one of the central modes of
human-machine interaction, leading to the accumulation of vast amounts of
conversation logs and increasing demand for dialogue generation. A
conversational life-cycle spans from the Prelude through the Interlocution to
the Epilogue, encompassing various elements. Despite the existence of numerous
dialogue-related studies, there is a lack of benchmarks that encompass
comprehensive dialogue elements, hindering precise modeling and systematic
evaluation. To bridge this gap, we introduce an innovative research task
Dialogue Element MOdeling, including
Element Awareness and Dialogue Agent Interaction, and
propose a novel benchmark, DEMO, designed for a comprehensive
dialogue modeling and assessment. Inspired by imitation learning, we further
build the agent which possesses the adept ability to model dialogue elements
based on the DEMO benchmark. Extensive experiments indicate that existing LLMs
still exhibit considerable potential for enhancement, and our DEMO agent has
superior performance in both in-domain and out-of-domain tasks.