ADOPD-Instruct: A Large-Scale Multimodal Dataset for Document Editing

Authors: Wanrong Zhu, Xiangxi Shi, Yufan Zhou, Ruiyi Zhang, Tong Sun, Jiuxiang Gu
Primary area: datasets and benchmarks
Keywords: document editing, multimodal dataset, empirical study

Abstract:
Visually-rich document editing is a complex multimodal task with a wide range of real-world applications. Despite increasing interest, there is a significant lack of publicly available datasets offering detailed entity-level annotations and step-by-step instructions for the editing process. To address this, we introduce ADOPD-Instruct, a multimodal dataset designed specifically for document editing tasks. ADOPD-Instruct includes visually-rich documents, precise entity-level masks highlighting elements to be edited, and step-by-step edit instructions, targeting both the masking and inpainting processes for text and non-text design elements. ADOPD-Instruct instructions have been carefully curated by human annotators to ensure high quality across the dataset.
We conduct extensive evaluations of current Multimodal Large Language Models (MLLMs) and image editing models using various image backbones to assess their performance on document editing. The results reveal substantial challenges: current MLLMs struggle to generate accurate and detailed instructions, while image editing models often fail to follow instructions precisely, particularly with text edits. These findings underscore the limitations of existing models and highlight the importance of annotated datasets like ADOPD-Instruct for advancing this domain. Dataset is available at: https://huggingface.co/datasets/adopd-instruct/ADOPD-Instruct.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730375661108

thread_type:
official_review

<thread_object>
creator: reviewer_ogda

<content>
summary:
The paper builds the large-scale multimodal dataset with entity-level annotations and step-by-step instructions for visually-rich document editing. They evaluate the performance of MLLMs in visually-rich document understanding, as well as the efficacy of leading image editing models in document editing tasks.

strengths:
1. They introduce the publicly available multimodal dataset with detailed annotations and step-by-step instructions for entity-level editing in visually-rich documents. Builting upon ADOPD documents, they first use GPT-4o to generate initial editing instructions, which are then refined by human annotators to ensure accuracy and validity.
2. Experimental results show that the instructions generated by current open-source MLLMs did not fully achieve the level of detail and precision found in human-written instructions when describing intricate edits between visually-rich document.
3. They found the limitations of current MLLMs and image editing models in performing visuallyrich document editing tasks, emphasizing the necessity for more sophisticated methodologies to enhance model performance in this domain.

weaknesses:
1. The evaluation metrics are not comprehensive enough, for example, they could be based on GPT-4. 
2. There is a lack of more insightful analysis, such as whether newer models perform worse than older ones, and the reasons behind the differences among various models.
3. Lacking some error analysis and insights to optimize the model's performance.

questions:
see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730446277983

thread_type:
official_review

<thread_object>
creator: reviewer_8sc2

<content>
summary:
The paper introduces a multimodal dataset called ADOPD-Instruct, designed to benchmark model capabilities for document editing. This dataset is derived from the existing ADOPD dataset (ICLR 2024) but has been re-annotated. The annotation process involves both an automated procedure using GPT-4o and human verification. Consequently, the dataset assesses the capabilities of MLLMs in instruction generation (referred to as "masking" in the paper) and the capabilities of editing models in instruction following (referred to as "inpainting" in the paper). Both text and non-text elements are considered. The study examines 8 MLLMs and 4 image editing models.

strengths:
+ The task of document editing holds significant practical value in real-world applications, yet it remains under-explored. 
+ Observing the limitations of current models in executing document editing tasks provides valuable insights for further research.
+ The paper extensively evaluates 8 MLLMs and 4 image editing models.

weaknesses:
- Major: The reviewer recognizes the empirical contributions of this work. However, the paper falls short in terms of technical contributions and providing insightful knowledge to the community. Specifically, the dataset introduced in the paper is derived from an existing public dataset. While new annotations are provided, the dataset is relatively small in scale and offers limited insights for future research. Additionally, the evaluation of existing models highlights the challenges they currently face, but it does not offer any guidance on how the community could enhance model performance for these tasks.

- Although the work might be new in the field of visually-rich document editing, it is related to tasks such as text-rich and layout multimodal reasoning and editing. It is recommended to carefully discuss these previous works.

> TRINS: Towards Multimodal Language Models that Can Read. CVPR 2024.

> LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding. CVPR 2024.

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619947995

thread_type:
official_review

<thread_object>
creator: reviewer_tknm

<content>
summary:
This paper presents ADOPD-Instruct, a comprehensive large-scale multimodal dataset aimed at visually-rich document editing tasks based on the instruction. The dataset stands out for its detailed annotations, step-by-step edit instructions, and its ability for this rich-text image editing tasks. 
The authors also evaluate the performance of existing MLLMs on the editing instruction generated tasks and image editing models on document editing tasks, highlighting key limitations in current models and underscoring the dataset’s importance for advancing this field.

strengths:
ADOPD-Instruct provides a novel contribution that fills a critical gap in multimodal document editing by addressing the scarcity of datasets that support entity-level text edits in document images.

The dataset presented in the paper shows to be highly applicable for real-world scenarios, making it particularly relevant to document understanding and editing fields.

The experimental section offers valuable insights on current model limitations, revealing significant challenges in current methods, particularly around instruction-following and complex text rendering.

weaknesses:
- Lack of Clarity in Data Generation:  The dataset generation process is unclear, particularly regarding instruction and image generation. A thorough description of the data curation process would improve the clarity of the paper. Given the paper's length (8 pages), additional information on dataset curation could be great. A more detailed explanation in the revised version would positively influence my review score.

- Image Sources and Annotation Details: More details of the image sources and human annotators to imporve the soundness of the paper.

- The paper could benefit from a more detailed classification of edit tasks beyond just Masking and Inpainting for single text and non-text elements.

- More related works on image editing model and datasets[1-4], as well as the text rendering ability of generative models[5-8] will benfit to the paper.

[1]MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

[2]Emu Edit: Precise Image Editing via Recognition and Generation Tasks

[3]HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing

[4]UltraEdit: Instruction-based Fine-Grained Image Editing at Scale

[5]TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering

[6]GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models

[7]A spark of vision-language intelligence: 2-dimensional autoregressive transformer for efficient finegrained image generation

[8]GPT4O

questions:
1. How were the source and target images generated, especially given the high quality of examples in the paper? The data generation process remains ambiguous after reading.

2. Use of GPT-4o Instructions: Given the failure rate of GPT-4o instructions (nearly 60%, per Table 2), what is the justification for relying on these instructions, especially when human annotators are required to review and refine them? I also wonder the performance of gpt4o on ADOPD-Instruct for image editing.

3. Do the Masking and Inpainting tasks share the same image sources, with the tasks simply reversed?

4. Since ADOPD-Instruct focuses on Masking and Inpainting tasks with changes only occurring in the masked areas, are the metrics in Table 5 appropriate for evaluation? Local metrics such as SSIM or CLIP score in the masked area might provide a more accurate assessment. The best performance of LaMa may be due to simply copying the input image, resulting in relatively low scores.

5. Can more details be provided about the specific split of ADOPD-Instruct used in the evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730969049627

thread_type:
official_review

<thread_object>
creator: reviewer_ywzh

<content>
summary:
The paper introduces ADOPD-Instruct, a novel large-scale multimodal dataset designed to address the complexities of visually-rich document editing. The authors have curated a collection of visually-rich documents, precise masks highlighting elements to be edited, and human-curated instructions that target both text and non-text design elements. The paper also presents an extensive evaluation of current Multimodal Large Language Models and image editing models, revealing the challenges these models face in accurately generating and following detailed instructions for document editing.

strengths:
1. ADOPD-Instruct is quite useful in downstream tasks requiring document-level image editing. It stands out for its scale and the level of detail in its annotations. The inclusion of both text and non-text elements, along with segmentation masks and human-curated instructions, makes it a rich resource for researchers in document editing.
2. The annotation process is clear and easy to understand.
3. The paper benchmarks a wide range of available models, showing that the proposed task is challanging according to the results.

weaknesses:
- There is a lack of clarity regarding the involvement of human annotators in refining the instructions generated by GPT-4-O, as indicated in Table 2, which shows that a significant portion (>60%) of the instructions produced by the GPT model are incorrect.
- The need for an instruction generation task is also difficult to comprehend, particularly in the context of document editing. Typically, initial instructions are crafted by humans, so if the edited image and mask are already prepared, it raises the question of why further editing is necessary. This task setup appears to be rather disconnected from real-world applications.

questions:
1. Why separate the image editing task into detailed instruction generation and instruction-following document editing?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a dataset for document editing and establishes evaluations of current Multimodal Large Language Models (MLLMs) and image editing models to assess their performance on this task. While the reviewers acknowledge the value of the dataset, they highlight several weaknesses: (1) a lack of significant technical contribution, (2) insufficient insights on improving task performance, and (3) limited empirical verification. Two reviewers find that the paper is below ICLR standards.