Orionnnn/hub-stats huggingface dataset:
Affordance-based Robot Manipulation with Flow Matching
Authors: name: Fan Zhang, name: Michael Gienger
Published: 2024-09-02 09:11:28

We present a framework for assistive robot manipulation, which focuses on two
fundamental challenges: first, efficiently adapting large-scale models to
downstream scene affordance understanding tasks, especially in daily living
scenarios where gathering multi-task data involving humans requires strenuous
effort; second, effectively learning robot trajectories by grounding the visual
affordance model. We tackle the first challenge by employing a
parameter-efficient prompt tuning method that prepends learnable text prompts
to the frozen vision model to predict manipulation affordances in multi-task
scenarios. Then we propose to learn robot trajectories guided by affordances in
a supervised Flow Matching method. Flow matching represents a robot visuomotor
policy as a conditional process of flowing random waypoints to desired robot
trajectories. Finally, we introduce a real-world dataset with 10 tasks across
Activities of Daily Living to test our framework. Our extensive evaluation
highlights that the proposed prompt tuning method for learning manipulation
affordance with language prompter achieves competitive performance and even
outperforms other finetuning protocols across data scales, while satisfying
parameter efficiency. Learning multi-task robot trajectories with a single flow
matching policy also leads to consistently better performance than alternative
behavior cloning methods, especially given multimodal robot action
distributions. Our framework seamlessly unifies affordance model learning and
trajectory generation with flow matching for robot manipulation.
Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
Authors: name: Yuxiang Wei, name: Hojae Han, name: Rajhans Samdani
Published: 2024-09-03 22:36:42

Recent studies have been increasingly demonstrating that high-quality data is
crucial for effective pretraining of language models. However, the precise
definition of "high-quality" remains underexplored. Focusing on the code
domain, we introduce Arctic-SnowCoder-1.3B, a data-efficient base code model
pretrained on 555B tokens through three phases of progressively refined data:
(1) general pretraining with 500B standard-quality code tokens, preprocessed
through basic filtering, deduplication, and decontamination, (2) continued
pretraining with 50B high-quality tokens, selected from phase one by a
BERT-style quality annotator trained to distinguish good code from random data,
using positive examples drawn from high-quality code files, along with
instruction data from Magicoder and StarCoder2-Instruct, and (3) enhanced
pretraining with 5B synthetic data created by Llama-3.1-70B using phase two
data as seeds, adapting the Magicoder approach for pretraining. Despite being
trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art
performance on BigCodeBench, a coding benchmark focusing on practical and
challenging programming tasks, compared to similarly sized models trained on no
more than 1T tokens, outperforming Phi-1.5-1.3B by 36%. Across all evaluated
benchmarks, Arctic-SnowCoder-1.3B beats StarCoderBase-3B pretrained on 1T
tokens. Additionally, it matches the performance of leading small base code
models trained on trillions of tokens. For example, Arctic-SnowCoder-1.3B
surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a
benchmark that evaluates function-level code generation, and remains
competitive on BigCodeBench. Our evaluation presents a comprehensive analysis
justifying various design choices for Arctic-SnowCoder. Most importantly, we
find that the key to high-quality data is its alignment with the distribution
of downstream applications.
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding
  Benchmark
Authors: name: Xiang Yue, name: Tianyu Zheng, name: Yuansheng Ni, name: Yubo Wang, name: Kai Zhang, name: Shengbang Tong, name: Yuxuan Sun, name: Ming Yin, name: Botao Yu, name: Ge Zhang, name: Huan Sun, name: Yu Su, name: Wenhu Chen, name: Graham Neubig
Published: 2024-09-04 15:31:26

This paper introduces MMMU-Pro, a robust version of the Massive
Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
MMMU-Pro rigorously assesses multimodal models' true understanding and
reasoning capabilities through a three-step process based on MMMU: (1)
filtering out questions answerable by text-only models, (2) augmenting
candidate options, and (3) introducing a vision-only input setting where
questions are embedded within images. This setting challenges AI to truly "see"
and "read" simultaneously, testing a fundamental human cognitive skill of
seamlessly integrating visual and textual information. Results show that model
performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8%
to 26.9% across models. We explore the impact of OCR prompts and Chain of
Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT
generally improves performance. MMMU-Pro provides a more rigorous evaluation
tool, closely mimicking real-world scenarios and offering valuable directions
for future research in multimodal AI.
Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion
  Dependency
Authors: name: Jianwen Jiang, name: Chao Liang, name: Jiaqi Yang, name: Gaojie Lin, name: Tianyun Zhong, name: Yanbo Zheng
Published: 2024-09-04 11:55:14

With the introduction of diffusion-based video generation techniques,
audio-conditioned human video generation has recently achieved significant
breakthroughs in both the naturalness of motion and the synthesis of portrait
details. Due to the limited control of audio signals in driving human motion,
existing methods often add auxiliary spatial signals to stabilize movements,
which may compromise the naturalness and freedom of motion. In this paper, we
propose an end-to-end audio-only conditioned video diffusion model named Loopy.
Specifically, we designed an inter- and intra-clip temporal module and an
audio-to-latents module, enabling the model to leverage long-term motion
information from the data to learn natural motion patterns and improving
audio-portrait movement correlation. This method removes the need for manually
specified spatial motion templates used in existing methods to constrain motion
during inference. Extensive experiments show that Loopy outperforms recent
audio-driven portrait diffusion models, delivering more lifelike and
high-quality results across various scenarios.
LongCite: Enabling LLMs to Generate Fine-grained Citations in
  Long-context QA
Authors: name: jiajie Zhang, name: Yushi Bai, name: Xin Lv, name: Wanjun Gu, name: Danqing Liu, name: Minhao Zou, name: Shulin Cao, name: Lei Hou, name: Yuxiao Dong, name: Ling Feng, name: Juanzi Li
Published: 2024-09-04 17:41:19

Though current long-context large language models (LLMs) have demonstrated
impressive capacities in answering user questions based on extensive text, the
lack of citations in their responses makes user verification difficult, leading
to concerns about their trustworthiness due to their potential hallucinations.
In this work, we aim to enable long-context LLMs to generate responses with
fine-grained sentence-level citations, improving their faithfulness and
verifiability. We first introduce LongBench-Cite, an automated benchmark for
assessing current LLMs' performance in Long-Context Question Answering with
Citations (LQAC), revealing considerable room for improvement. To this end, we
propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs
to automatically generate long-context QA instances with precise sentence-level
citations, and leverage this pipeline to construct LongCite-45k, a large-scale
SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the
LongCite-45k dataset, successfully enabling their generation of accurate
responses and fine-grained sentence-level citations in a single output. The
evaluation results on LongBench-Cite show that our trained models achieve
state-of-the-art citation quality, surpassing advanced proprietary models
including GPT-4o.
General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model
Authors: name: Haoran Wei, name: Chenglong Liu, name: Jinyue Chen, name: Jia Wang, name: Lingyu Kong, name: Yanming Xu, name: Zheng Ge, name: Liang Zhao, name: Jianjian Sun, name: Yuang Peng, name: Chunrui Han, name: Xiangyu Zhang
Published: 2024-09-03 08:41:31

Traditional OCR systems (OCR-1.0) are increasingly unable to meet people's
usage due to the growing demand for intelligent processing of man-made optical
characters. In this paper, we collectively refer to all artificial optical
signals (e.g., plain texts, math/molecular formulas, tables, charts, sheet
music, and even geometric shapes) as "characters" and propose the General OCR
Theory along with an excellent model, namely GOT, to promote the arrival of
OCR-2.0. The GOT, with 580M parameters, is a unified, elegant, and end-to-end
model, consisting of a high-compression encoder and a long-contexts decoder. As
an OCR-2.0 model, GOT can handle all the above "characters" under various OCR
tasks. On the input side, the model supports commonly used scene- and
document-style images in slice and whole-page styles. On the output side, GOT
can generate plain or formatted results (markdown/tikz/smiles/kern) via an easy
prompt. Besides, the model enjoys interactive OCR features, i.e., region-level
recognition guided by coordinates or colors. Furthermore, we also adapt dynamic
resolution and multi-page OCR technologies to GOT for better practicality. In
experiments, we provide sufficient results to prove the superiority of our
model.
PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in
  Action
Authors: name: Yijia Shao, name: Tianshi Li, name: Weiyan Shi, name: Yanchen Liu, name: Diyi Yang
Published: 2024-08-29 17:58:38

As language models (LMs) are widely utilized in personalized communication
scenarios (e.g., sending emails, writing social media posts) and endowed with a
certain level of agency, ensuring they act in accordance with the contextual
privacy norms becomes increasingly critical. However, quantifying the privacy
norm awareness of LMs and the emerging privacy risk in LM-mediated
communication is challenging due to (1) the contextual and long-tailed nature
of privacy-sensitive cases, and (2) the lack of evaluation approaches that
capture realistic application scenarios. To address these challenges, we
propose PrivacyLens, a novel framework designed to extend privacy-sensitive
seeds into expressive vignettes and further into agent trajectories, enabling
multi-level evaluation of privacy leakage in LM agents' actions. We instantiate
PrivacyLens with a collection of privacy norms grounded in privacy literature
and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM
performance in answering probing questions and their actual behavior when
executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4
and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even
when prompted with privacy-enhancing instructions. We also demonstrate the
dynamic nature of PrivacyLens by extending each seed into multiple trajectories
to red-team LM privacy leakage risk. Dataset and code are available at
https://github.com/SALT-NLP/PrivacyLens.
GenAgent: Build Collaborative AI Systems with Automated Workflow
  Generation -- Case Studies on ComfyUI
Authors: name: Xiangyuan Xue, name: Zeyu Lu, name: Di Huang, name: Wanli Ouyang, name: Lei Bai
Published: 2024-09-02 17:44:10

Much previous AI research has focused on developing monolithic models to
maximize their intelligence and capability, with the primary goal of enhancing
performance on specific tasks. In contrast, this paper explores an alternative
approach: collaborative AI systems that use workflows to integrate models, data
sources, and pipelines to solve complex and diverse tasks. We introduce
GenAgent, an LLM-based framework that automatically generates complex
workflows, offering greater flexibility and scalability compared to monolithic
models. The core innovation of GenAgent lies in representing workflows with
code, alongside constructing workflows with collaborative agents in a
step-by-step manner. We implement GenAgent on the ComfyUI platform and propose
a new benchmark, OpenComfy. The results demonstrate that GenAgent outperforms
baseline approaches in both run-level and task-level evaluations, showing its
capability to generate complex workflows with superior effectiveness and
stability.
Know When to Fuse: Investigating Non-English Hybrid Retrieval in the
  Legal Domain
Authors: name: Antoine Louis, name: Gijs van Dijck, name: Gerasimos Spanakis
Published: 2024-09-02 16:19:13

Hybrid search has emerged as an effective strategy to offset the limitations
of different matching paradigms, especially in out-of-domain contexts where
notable improvements in retrieval quality have been observed. However, existing
research predominantly focuses on a limited set of retrieval methods, evaluated
in pairs on domain-general datasets exclusively in English. In this work, we
study the efficacy of hybrid search across a variety of prominent retrieval
models within the unexplored field of law in the French language, assessing
both zero-shot and in-domain scenarios. Our findings reveal that in a zero-shot
context, fusing different domain-general models consistently enhances
performance compared to using a standalone model, regardless of the fusion
method. Surprisingly, when models are trained in-domain, we find that fusion
generally diminishes performance relative to using the best single system,
unless fusing scores with carefully tuned weights. These novel insights, among
others, expand the applicability of prior findings across a new field and
language, and contribute to a deeper understanding of hybrid search in
non-English specialized domains.
The MERIT Dataset: Modelling and Efficiently Rendering Interpretable
  Transcripts
Authors: name: I. de Rodrigo, name: A. Sanchez-Cuadrado, name: J. Boal, name: A. J. Lopez-Lopez
Published: 2024-08-31 12:56:38

This paper introduces the MERIT Dataset, a multimodal (text + image + layout)
fully labeled dataset within the context of school reports. Comprising over 400
labels and 33k samples, the MERIT Dataset is a valuable resource for training
models in demanding Visually-rich Document Understanding (VrDU) tasks. By its
nature (student grade reports), the MERIT Dataset can potentially include
biases in a controlled way, making it a valuable tool to benchmark biases
induced in Language Models (LLMs). The paper outlines the dataset's generation
pipeline and highlights its main features in the textual, visual, layout, and
bias domains. To demonstrate the dataset's utility, we present a benchmark with
token classification models, showing that the dataset poses a significant
challenge even for SOTA models and that these would greatly benefit from
including samples from the MERIT Dataset in their pretraining phase.
Kvasir-VQA: A Text-Image Pair GI Tract Dataset
Authors: name: Sushant Gautam, name: Andrea Storås, name: Cise Midoglu, name: Steven A. Hicks, name: Vajira Thambawita, name: Pål Halvorsen, name: Michael A. Riegler
Published: 2024-09-02 19:41:59

We introduce Kvasir-VQA, an extended dataset derived from the HyperKvasir and
Kvasir-Instrument datasets, augmented with question-and-answer annotations to
facilitate advanced machine learning tasks in Gastrointestinal (GI)
diagnostics. This dataset comprises 6,500 annotated images spanning various GI
tract conditions and surgical instruments, and it supports multiple question
types including yes/no, choice, location, and numerical count. The dataset is
intended for applications such as image captioning, Visual Question Answering
(VQA), text-based generation of synthetic medical images, object detection, and
classification. Our experiments demonstrate the dataset's effectiveness in
training models for three selected tasks, showcasing significant applications
in medical image analysis and diagnostics. We also present evaluation metrics
for each task, highlighting the usability and versatility of our dataset. The
dataset and supporting artifacts are available at
https://datasets.simula.no/kvasir-vqa.
Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive
  Content Generation
Authors: name: Qihua Chen, name: Yue Ma, name: Hongfa Wang, name: Junkun Yuan, name: Wenzhe Zhao, name: Qi Tian, name: Hongmei Wang, name: Shaobo Min, name: Qifeng Chen, name: Wei Liu
Published: 2024-09-02 08:28:57

This paper explores higher-resolution video outpainting with extensive
content generation. We point out common issues faced by existing methods when
attempting to largely outpaint videos: the generation of low-quality content
and limitations imposed by GPU memory. To address these challenges, we propose
a diffusion-based method called Follow-Your-Canvas. It builds upon two
core designs. First, instead of employing the common practice of "single-shot"
outpainting, we distribute the task across spatial windows and seamlessly merge
them. It allows us to outpaint videos of any size and resolution without being
constrained by GPU memory. Second, the source video and its relative positional
relation are injected into the generation process of each window. It makes the
generated spatial layout within each window harmonize with the source video.
Coupling with these two designs enables us to generate higher-resolution
outpainting videos with rich content while keeping spatial and temporal
consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g.,
from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically
pleasing results. It achieves the best quantitative results across various
resolution and scale setups. The code is released on
https://github.com/mayuelala/FollowYourCanvas
ContextCite: Attributing Model Generation to Context
Authors: name: Benjamin Cohen-Wang, name: Harshay Shah, name: Kristian Georgiev, name: Aleksander Madry
Published: 2024-09-01 14:36:36

How do language models use information provided as context when generating a
response? Can we infer whether a particular generated statement is actually
grounded in the context, a misinterpretation, or fabricated? To help answer
these questions, we introduce the problem of context attribution: pinpointing
the parts of the context (if any) that led a model to generate a particular
statement. We then present ContextCite, a simple and scalable method for
context attribution that can be applied on top of any existing language model.
Finally, we showcase the utility of ContextCite through three applications: (1)
helping verify generated statements (2) improving response quality by pruning
the context and (3) detecting poisoning attacks. We provide code for
ContextCite at https://github.com/MadryLab/context-cite.
Diffusion Policy Policy Optimization
Authors: name: Allen Z. Ren, name: Justin Lidard, name: Lars L. Ankile, name: Anthony Simeonov, name: Pulkit Agrawal, name: Anirudha Majumdar, name: Benjamin Burchfiel, name: Hongkai Dai, name: Max Simchowitz
Published: 2024-09-01 02:47:50

We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic
framework including best practices for fine-tuning diffusion-based policies
(e.g. Diffusion Policy) in continuous control and robot learning tasks using
the policy gradient (PG) method from reinforcement learning (RL). PG methods
are ubiquitous in training RL policies with other policy parameterizations;
nevertheless, they had been conjectured to be less efficient for
diffusion-based policies. Surprisingly, we show that DPPO achieves the
strongest overall performance and efficiency for fine-tuning in common
benchmarks compared to other RL methods for diffusion-based policies and also
compared to PG fine-tuning of other policy parameterizations. Through
experimental investigation, we find that DPPO takes advantage of unique
synergies between RL fine-tuning and the diffusion parameterization, leading to
structured and on-manifold exploration, stable training, and strong policy
robustness. We further demonstrate the strengths of DPPO in a range of
realistic settings, including simulated robotic tasks with pixel observations,
and via zero-shot deployment of simulation-trained policies on robot hardware
in a long-horizon, multi-stage manipulation task. Website with code:
diffusion-ppo.github.io
FLUX that Plays Music
Authors: name: Zhengcong Fei, name: Mingyuan Fan, name: Changqian Yu, name: Junshi Huang
Published: 2024-09-01 02:43:33

This paper explores a simple extension of diffusion-based rectified flow
Transformers for text-to-music generation, termed as FluxMusic. Generally,
along with design in advanced
Fluxhttps://github.com/black-forest-labs/flux model, we transfers it
into a latent VAE space of mel-spectrum. It involves first applying a sequence
of independent attention to the double text-music stream, followed by a stacked
single music stream for denoised patch prediction. We employ multiple
pre-trained text encoders to sufficiently capture caption semantic information
as well as inference flexibility. In between, coarse textual information, in
conjunction with time step embeddings, is utilized in a modulation mechanism,
while fine-grained textual details are concatenated with the music patch
sequence as inputs. Through an in-depth study, we demonstrate that rectified
flow training with an optimized architecture significantly outperforms
established diffusion methods for the text-to-music task, as evidenced by
various automatic metrics and human preference evaluations. Our experimental
data, code, and model weights are made publicly available at:
https://github.com/feizc/FluxMusic.
Accurate Compression of Text-to-Image Diffusion Models via Vector
  Quantization
Authors: name: Vage Egiazarian, name: Denis Kuznedelev, name: Anton Voronov, name: Ruslan Svirschevski, name: Michael Goin, name: Daniil Pavlov, name: Dan Alistarh, name: Dmitry Baranchuk
Published: 2024-08-31 16:09:20

Text-to-image diffusion models have emerged as a powerful framework for
high-quality image generation given textual prompts. Their success has driven
the rapid development of production-grade diffusion models that consistently
increase in size and already contain billions of parameters. As a result,
state-of-the-art text-to-image models are becoming less accessible in practice,
especially in resource-limited environments. Post-training quantization (PTQ)
tackles this issue by compressing the pretrained model weights into lower-bit
representations. Recent diffusion quantization techniques primarily rely on
uniform scalar quantization, providing decent performance for the models
compressed to 4 bits. This work demonstrates that more versatile vector
quantization (VQ) may achieve higher compression rates for large-scale
text-to-image diffusion models. Specifically, we tailor vector-based PTQ
methods to recent billion-scale text-to-image models (SDXL and SDXL-Turbo), and
show that the diffusion models of 2B+ parameters compressed to around 3 bits
using VQ exhibit the similar image quality and textual alignment as previous
4-bit compression techniques.