Agent-DocEdit: Language-Instructed LLM Agent for Content-Rich Document Editing

Authors: te-lin  wu
rajiv  jain
yufan  zhou
puneet  mathur
vlad i morariu
multimodal  grounder
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 00:38:34

Abstract: Editing content-rich and multimodal documents, such as posters, flyers, and slides, can be tedious if the edits are complex, repetitive, or require subtle skills and deep knowledge of the editing software.
Motivated by recent advancements in both Large Language Model (LLM) agents and multimodal modeling, we propose a framework that automates document editing which takes as input a linguistic edit request from the user and then performs sequential editing actions to the document the satisfy the request.
Our proposed method, Agent-DocEdit, first grounds the edit request directly in the underlying document structure to identify the elements that need to be manipulated. Then, we rely on the agent capabilities of LLMs to generate an edit program which calls a set of pre-defined APIs to modify the underlying structure of the document.
To improve the generated edit program, we leverage a feedback mechanism incorporating a deterministic code executor and a multimodal LLM.
We demonstrate the effectiveness of our proposed modularized LLM editing agent on the DocEdit dataset, where Agent-DocEdit outperforms existing state-of-the-art methods by 70+% in document element grounding and 16+% on final rendition generation.

Main paper: 
 introduction: 
 Editing rich multi-modal content as found in flyers, graphic designs, posters, etc., requires knowledge of specialized tools and patience to apply multiple edits to image and text layers. Recent AI trends have increased interest in automating the process of editing such documents to reduce the skills and time required-powerful multimodal models are not only able to understand a user's language description of their edit intent but also generate new text (Achiam et al., 2023) , visual content (Geng et al., 2023) , or even code for carrying out the edits (Mathur et al., 2023) .There are two major approaches for applying recent techniques to document editing. The first is to model editing commands as structured and composable set of document elements, actions, and states and then analyze the document and user's description of their editing intent to produce an editing command. DocEdit (Mathur et al., 2023) is a recent example of this approach, and it uses a multimodal language model to understand the author edit request, identify the bounding box of the document element that is the target of the desired edit, and generate a short structured command. However, DocEdit does not model the underlying structure of the document (i.e., the individual text, image, and shape elements along with the required information for composing them into a document) operating instead on the raster image of the document as input, nor does it carry out the edits to produce the edited document (i.e., the command and bounding box are the final outputs). The second type of approach, such as InstructDiffusion (Geng et al., 2023) , focuses on directly producing the resulting raster image of the edited document, resulting in an architecture that can be FANCY DOG FANCY DOGCurve the text "National Moon Day" with the date and drag it to the top beside the moon image.Figure 1 : Instruction based document editing: Given a language user edit request as input (in the grey box), our framework automatically edits the before document to produce the after document.The illustrative examples above are similar to the training and evaluation data found in the DocEdit dataset (Mathur et al., 2023) .optimized end-to-end but which can generate artifacts or distortions due to the constraint that it can only manipulate the final pixels and must infer all the semantic groupings, constraints, and interactions among document elements as it performs modifications.Building on recent LLMs based approaches such as Toolformer (Schick et al., 2024) , we propose a method, Agent-DocEdit, that (1) takes as input a document with its full structure rather than just a raster image, (2) identifies within the document structure the elements involved in the edits (through a process called grounding), (3) produces an edit program to carry out the edits, and (4) executes it to produce a modified document structure that can be rendered by the document viewer if a raster of the document is desired, thus addressing the main weaknesses of the aforementioned approaches. One advantage of such an approach is that the edit program can leverage already existing and well-tested pre-defined APIs for editing documents rather than having to learn these transformations from scratch.We formulate our problem as follows. Given a user editing request (in the form of natural language) and a document with multimodal content (a layout of text, shape, and image elements), the goal of our system is to automate the requested edits and produce the desired edited document by identifying in the underlying document structure the document elements involved in edits, producing an edit program to modify these document elements, and then executing the modifications through an API to produce a modified document structure that can be rendered to produce an image of the edited document. Figure 1 provides a few example editing commands as well as examples of before and after documents that our system is designed to work with. The first (leftmost) example will be used to illustrate different components of our proposed framework in the remaining sections. The proposed Agent-DocEdit framework is extensively evaluated on the design subset of the DocEdit dataset. Our main contributions are:• We propose an LLM-based framework for document editing that, unlike previous work, operates directly on the structured representation of a document and produces structured content as output. This is done by grounding edit requests in the documents structure, generating an edit program, and executing the program through an external API to effectuate the desired edits. Compared to instruction-tuned diffusion baselines, we improve the image similarity-based metric by >16%.• We propose two new techniques for document element grounding given a language instruction, both of which are consistent with our approach of using tools (where the visual phrase grounder is the tool). Both approaches significantly improve the ability of the system to localize the document elements involved in edits by >70% against the prior DocEdit baseline. This is particularly impactful, since an error in grounding results in the wrong document element to be modified.• We propose a feedback mechanism that can improve the generated code by providing language feedback to guide the LLM to iteratively improve the generated plan, improving results by >2% in image similarity-based metric; and >35% in human judgements, against that without the feedback. 2 Method (The Agent-DocEdit)Figure 2 provides an overview of the proposed Agent-DocEdit framework, which is composed of four main components: (1) the Edit Program Generator takes input human user editing request (in natural language) as input and generates the edit program (i.e., a series of pre-defined API function calls as a high-level planning of the editing actions) by instructing an LLM to perform the code-generation task.(2) the Multimodal Grounder identifies the document elements referred to by the editing request, using an architecture derived from phrase grounding models for computer vision tasks. The ultimate output of the grounder will be the ID of the document element(s) within the document's content structure.(3) the Executor performs a deterministic conversion transforming the edit program to actual executable programs that modify with the content structure according to the editing request.(4) the Feedback Provider is a separately learned module that compares the edited outcome against the original edit request and produces language feedback that is then utilized to (iteratively) instruct the LLM to refine the generated edit program. We will now discuss each component in details below. 
generating an edit program via llm-code-gen.: 
 We instruct the LLM to produce a Python edit program which carries out the user edit request by calling a set of pre-defined APIs, acting as an agent that manipulates a set of provided tools. The API is provided to the LLM through the docstring of each function following similar designs in Surís et al. (2023) and Gupta & Kembhavi (2023) . The docstring includes a description of the goal of the API function, input and output arguments, and exemplar usage. In our early studies, we empirically found that usage examples play a crucial role as in-context examples in instructing the LLM to produce higher quality (more accurate) programs. The detailed implementation of each API is not provided to the LLM, as the LLM can generate programs simply by following the API guidelines without needing to refer to how each function is implemented, i.e., for the LLM to generate the edit program, it only needs to refer to the API function header as well as its docstring context.In this work, we generally follow the actions defined by DocEdit (Mathur et al., 2023) (i.e., MOVE, DELETE, ADD, COPY, MODIFY, REPLACE), with additional localization actions. We categorize these pre-defined API functions into three major categories: • Localization Operations: The FIND function performs entity extraction on the language request to identify which portions of the request refer to target document elements, and then it performs visual phrase grounding to localize the elements in the document. The underlying grounding model predicts bounding boxes, which are used to search over the document structure. The canonical ID of the document element with the highest intersection over union (IoU) will be returned.1 In addition to the FIND function, we also implement an INFER function that predicts the destination location of a document element after editing for actions such as MOVE.• Spatial Operations: Functions such as MOVE, DELETE, ADD, and COPY generally deal with spatial arrangements of the target components. Each function has its specific set of arguments, for instance, MOVE can take specified position delta or the actual target position to re-position a component. ADD takes object type (or description) to insert contents (either retrieved from an asset library or generated by a visual generative model) to a specified position, while DELETE simply takes the canonical object ID for deletion action.• Modification Operations: Functions REPLACE and MODIFY perform content modifications (or swapping). The MODIFY function takes as arguments the type of modification (e.g., text color or font-size) and the actual modification context (e.g.red for changing color) that will be acted on the target document element. REPLACE function will change a specified content (e.g., a piece of text, an imagery) to a targeted type of content following the specification.2 By including these pre-defined APIs in the system prompt of an LLM, we instruct the LLM to perform natural language to code generation to derive a Python-based edit program where each action to be performed comes from one of the pre-defined APIs.We leverage the instructed code-generation capability of a strong LLM to generate a Pythonbased edit program. Given the user request in natural language, we instruct the LLM to follow a specifically crafted guideline for it to strictly refer to the pre-defined APIs described in Section 2.1.1 when devising the edit program. As illustrated, the left hand side of Figure 3 shows a sample API function (MOVE), where its right hand side exemplifies a sample generated edit program.Using Layout Canvas. In this work, we find that defining the boundaries of the document contents (i.e., the canvas where components are within) and supplementing the coarse level of content spatial locations, is helpful for the LLM to devise more accurate edit program. We thus include the canvas dimension of the document, locations (as bounding boxes) of major contents in the document (via PaddleOCR (Du et al., 2020) ) in the LLM-instruction.sJoin us in commemorating a great achievement!Component Grounder (Adapted-GLIP)Inherited from the editing APIs but only returns the component-level information As described in Section 2.1.1, the FIND (and the INFER) function can be implemented via a visual phrase grounding model. In this work, we utilize GLIP (Li et al., 2022) , an open vocabulary object detection model, to serve as the basis fulfilling such grounding requirement. However, most phrase grounding models require the "phrase to ground" to be specified beforehand, usually as a set of starting and ending spans of the phrases within the input sentence. We explore the two following methods for extracting the phrase to ground from the given user editing request:NLP Parsing. The entity extraction task can be generally approached as a parsing task followed by simple heuristics. We use the semantic role labeling (SRL) (Shi & Lin, 2019) technique to parse the user request into a few arguments (ARGs) and then take the ARG1 as the main document element that needs to be edited. For instance, in the following SRL-parsed request: "[V: Change] [ARG1: the date in second line] [ARG2: to 02.22.2024] .", we regard the ARG1 span (the date in second line) as the "phrase to ground".Leveraging Pre-Defined APIs. While parsing techniques such as SRL are specifically designed for textual entity extraction, we find that our code-generation framework also naturally supports such extraction capability. As illustrated by Figure 4 , we devise a set of APIs that, when they are executed, the LLM is instructed to fill the function arguments with the extracted corresponding entities.With the two aforementioned automatic entity extraction pipelines, we construct a finetuning dataset from DocEdit to adapt the GLIP model to our document editing domain, and utilize its predicted bounding boxes for the target component localization. Note that the entity extraction pipeline will be absorbed into part of the FIND and INFER functions.Once the edit program is generated, each major spatial and modification API operations will call its corresponding executable implementations to perform the actual editing process. These operations are deterministic and are editing software-dependent, while the major API operations defined in Section 2.1.1 are generally applicable and hence software-agnostic. 3The execution of FIND and INFER are similar, the predicted bounding boxes of the target component will be used to search over the document structure and return the canonical component ID with the highest IoU score. Such IDs will be the major medium used for other APIs to refer to a specific component. For ADD and REPLACE, there are some occasions that new content needs to be generated (e.g., changing the background image to something else). We execute such request by feeding the extracted component (i.e., the extracted entity should be the content to be added or replaced with) to a text-to-image generative model as its prompt.The LLM-code-generation pipeline in Section 2.1.2 often generates high-quality edit programs to fulfill user requests, but these programs can occasionally produce errors (bugs) that either prevent the program form executing (syntax errors) or which produces unsatisfying results (semantic errors).• Syntax Error: Although as mentioned in Section 2.1.2, we instruct the LLM to perform code-generation strictly following the defined APIs and their argument structures, we find that the LLM can occasionally hallucinate how these functions are used. Concretely, we find that the LLM can hallucinate unspecified usages of the functions (e.g., using arguments that are not specified) or even unspecified API calls (e.g., the LLM can have a tendency to invent nonexistent APIs).• Semantic Error: This type of error occurs when the edit program is syntax bug-free, but exhibit behaviours that are not perfectly satisfying the user editing request. For instance, a request of moving a target component upwards but below another reference component, might result in a generated program that accidentally moves the component too far to the top. Another representative (and major) error is when the component grounder fails to localize the target component perfectly. In such cases, even though the generated edit program is semantically correct, the outcome can still be suboptimal because the program is applied to the wrong document element.We address these issues with a feedback mechanism, as illustrated in Figure 5 . For a syntax error, we devise a dummy executor that executes the generated edit program without calling the executable API, i.e.it calls a dummy API for the purpose of checking the syntax correctness of the generated edit program. Whenever an error occurs, we extract the error message (STDERR) and feed it back to the LLM to obtain the corrected edit program via a re-prompting (see Figure 5 ). We bound the number of re-prompting trials maximally at 5.Since the second type of error, the semantic error, will only be recognizable after the editing outcome is produced, and furthermore, there is no real deterministic "compiler" to identify whether the outcome is satisfactory or not, we propose to learn a multimodal model for providing semantic error feedback. Specifically, we train a multimodal LLM (Figure 6 ) to predict the type of semantic error (i.e., whether the error comes from semantic mistake of the edit program or errors from the grounder) by analyzing the edit request along with the before-and after-edit document renditions. The edit action is: Change the "Happy Birthday" color to red. Is the editing correct?No. The edited component is not the correct one to edit on.Yes the editing is correct.The edit action is: Move the "Retirement Party" to a bit left of "3821 Beach Drive, Ocean City". Is the editing correct?No. The "Retirement Party" is moved to right. Yes the editing is correct.Figure 6 : Multimodal feedback provider is trained with synthetically constructed datasets as shown on the right hand side. We utilize our programmable pipeline to construct positive and negative pairs of edited outcomes associated with the original request, in order to train a multimodal LLM to provide the feedback (in language).To learn the multimodal feedback module, we use a popular open-sourcel LLaVa model (Liu et al., 2024) and extend it to take paired images (i.e., before-and after-edit document content) to provide natural language feedback of the editing outcome. To equip the model with capability of inferring whether an edited outcome is satisfactory, we train it with correct (positive) and incorrect (negative) results of the editing actions. Since each API defined in Section 2.1.1 has a corresponding executable implementation, we can construct a deterministic mapping between a series of these API calls with the actual outcome after the execution of such API series. Dataset Construction. We craft a set of templated editing requests such as "[MOVE] {component a } to [direction] of {component b }", or "[REPLACE] {component a }'s {color} to [red]". Given document content, items within the braces ({}) will be replaced randomly with the document element to be manipulated and its corresponding attribute(s) to edit. Items in the brackets ([]) are sampled from a pre-defined set of allowed vocabulary, e.g., REPLACE can have synonyms such as CHANGE, and colors can be sampled from red, blue, green, and more. It is noticeable that a deterministic mapping can be constructed between the editing APIs and these templated requests. As illustrated in the right hand side of Figure 6 , we construct a dataset based on the templated editing requests, where a negative outcome is generated by by deliberately causing the API to incorrectly apply the desired edit request (e.g., sampling the wrong [direction], [color] or the incorrect {component}). Therefore, the negative samples can be constructed by manipulating the contents with exactly opposite or wrong actions (i.e., the right/wrong actions are known beforehand from the ground truths) against a sampled request -guaranteeing the resulting samples contradict it.The fine-tuned LLaVa model is trained on producing a simple yes/no-based feedback, which can also be deterministically written based on constructed positive or negative pairs.Feedback-based refinement. Once the natural language feedback is obtained following the procedure in Section 2.4.1, we can provide such feedback to the LLM to refine the generated edit program. We only care about negative feedback, which is of two main types:(1) For semantically wrong editing we perform a "re-prompting" of the LLM to generate the refined program with the instruction depicted in Figure 5 . (2) For wrongly grounded component, we sample the next highest ranked proposed bounding box from the grounder (Section 2.2) without modifying the LLM generated program. The underlying assumption is: at least one of the proposed boxes would correctly localize the target component, which is not necessarily true in all the cases. We leave the general cases as future work. 
experiments: 
 Our experiments seek to answer following research questions: (1) How does the proposed modularized editing-agent framework compare to the baseline end-to-end model reported in Mathur et al. (2023) on its various evaluation metrics? (2) How does the proposed framework compare to other instruction-tuned generative editing models? (3) How effective is the proposed feedback mechanism? Implementation Details. For the document element grounder, the GLIP model (Li et al., 2022 ) is adopted and extended from the original author's repository and pretrained model weights. For the code-generator, as well as the entity extractor, we use OpenAI's GPT-4 model (Achiam et al., 2023) . For the image generative model, we use the stable diffusion version 1.5 (SD-v1.5) (Rombach et al., 2022) and cap the denoising steps at 50. We adopt the LLaVa version 1.5 (Liu et al., 2024) with the language model part being LLaMa-2-13b (Touvron et al., 2023) for the feedback mechanism in Section 2.4.1.DocEdit Metrics. For various component-wise and user request intent understanding metrics, we use the ones reported in the original DocEdit work (Mathur et al., 2023) . Specifically, the two main evaluations are conducted on the accuracy of both action parameters (Action Para.) and component parameters (Component Para.), where the parameters here refer to the type(s) of actions/components the user editing request concerns. As mentioned in Section 2.1.1, the design of our pre-defined APIs follow the major actions defined in the DocEdit dataset, and hence we regard our generated edit program as correct on predicting the action parameters if a line of the generated program contains the API names same as the ground truth action. Similarly for the components, we evaluate the extracted component types from the grounder (Section 2.2) against the ground truth.Image Similarity Metrics. In addition to the original DocEdit defined metrics, we further evaluate how similar the edited outcome from our framework is, compared to the ground truth post-edit documents. Motivated by (Geng et al., 2023) , we extend the CLIP-Score (Hessel et al., 2021) to consider the relationship between the edit request and the before/after-edit changes to the document (as an image). Specifically, we use the following formula to quantify the similarity measures (ϕ indicates cosine similarity from either CLIP visual embeddings or text-to-image CLIPScore):ϕ CLIP (e img gt , e img edit ) -∥ϕ CLIPScore (e img gt , e text )ϕ CLIPScore (e img edit , e text )∥ 1 In addition to using the entire document image for comparisons, we also report the similarity measures focused only on the contents within the ground truth bounding box(es) (Boxed-).Instruction-Tuned Editing Baselines. Our two instruction-based editing baselines are: InstructPix2Pix (Brooks et al., 2023) , and InstructDiffusion (Geng et al., 2023) . They are similar in that they both accept a human language instruction on how the images are to be edited, while a text-to-image diffusion model (Rombach et al., 2022) is trained to follow such an instruction to generate an edited version of the input image. While InstructPix2Pix mainly leverages the Prompt2Prompt technique (Hertz et al., 2022) for the editing data curation, InstructDiffusion adopts a wider range of more sophisticated data construction methods for editing tasks such as object removal/replacement, image enhancement, and style transfer, making it an improved version of InstructPix2Pix.Quantitative Results. Table 1 summarizes the essential DocEdit performance metrics. It can be observed that for the Action Para. metric, our framework (AD as Agent-DocEdit) performs somewhat worse (66.7) than the original DocEdit model. However, an empirical inspection suggests that the generated edit program can actually be correct even if the main APIs used do not exactly match the ground truth action parameter (e.g., REPLACE can be a combination of DELETE and ADD, and MODIFY and REPLACE can be equivalent for textual editing). We thus establish a mapping (Append. Sec. A.1) between equivalent DocEdit action parameters and the results in Table 1 (after the slash) indicates that our LLM pipeline is actually significantly better than the baseline end-to-end trained model.Our framework outperforms the baseline by a large margin in the component parameter prediction, where the feedback mechanism improves further.The edit program generator averagely generates 10.75 lines of code with 1.4 major API calls. The semantic error rate (number of errors/edits) within a held-out human judged subset is 34.6% without feedback and 12.3% with feedback, where the feedback provider reaches 71% accuracy on a held-out test set. In Table 1 we also report the passing rate (bug-free during execution) of the generated edit program, the syntax error re-prompting within 5 trials can improve the passing rate to almost perfect. These results suggest that our feedback mechanism significantly reduces both syntax and semantic errors in the edit programs.In addition to the results reported in Table 1 , we also examine the individual semantic error rates per each major API action. The error rates (without feedback mechanism) of the top three frequent actions in a held-out subset are: REPLACE: 22%, MODIFY: 30%, and MOVE: 37%. It is noticeable that our system does particularly well at the action REPLACE, and we hypothesize MOVE can benefit from improved target position predictors. We also observe that our component grounder performs quite evenly well on all kinds of components -no particular bias against or towards certain kinds of components is observed.The comparisons between the two grounding methods in Section 2.2 is reported in Table 2 , where a significant improvement can be observed using the code-generation method with our defined APIs. Both methods perform well above the end-to-end baseline.Table 3 reports the performance for the similarity-based editing scores. For fair comparison, if the generated edit program fails during the execution (with syntax errors), we do not proceed with any editing to the document, i.e., a failed program makes the content unchanged.It is notable that our framework, with or without the feedback mechanism, surpasses the two instruction-tuned baselines. The results reveal our general observations, which are: The two (instruction-tuned) generative baselines often exhibit severe degeneration of the parts that are supposed to be intact, or fail to capture the intentions of the user editing request. Our modularized agent framework more successfully interprets the user requests and manipulates the components much more accurately. And moreover, the feedback mechanism alleviates initial errors from both the edit program generation as well as inaccurate grounding.To verify these numerical results align with human judgements, we further conduct a human study on a randomly sampled 100 subset, each ranked by three annotators of which model outcome is the more preferred, and averaged as in Table 4 . The human preference trends verify the aforementioned results. Note that human judgement prefers the outcome with our feedback mechanism significantly more than that without (the gap between them is also larger than that of the similarity-based metric). This is reasonable since similarity-based metric may not be very detail-oriented in judging our task. 
related work: 
 LLM Agent. Equipping an LLM with the ability to use external tools makes it a powerful model that enjoys both strong generalization of language modeling capability as well as the faithful information manipulation. Prior work such as ToolFormer (Schick et al., 2024) , TALM (Parisi et al., 2022) , and program-aided language models (Madaan et al., 2022; Chen et al., 2022; Gao et al., 2023; Wang et al., 2022c) , successfully learn an LM that is able to manipulate tools such as calculators, calendars, and programs, for achieving desired tasks.Program-based LLMs are also applied to control tasks, such as ProgPrompt (Singh et al., 2023), and Eureka (Ma et al., 2023) . One type of tool-equipped LLM is able to perform visual programming, i.e., generate programs that call APIs, such as object detectors, to deal with visually-related queries, such as VQA (Gupta & Kembhavi, 2023) . Our work is most closely related to ViperGPT (Surís et al., 2023) , where we extend the visual programming ability to content editing with additionally created feedback mechanism.Document/Content Understanding. Our work is also related to document content understanding, as well as layout comprehension. Prior works have investigated applying modern transformer architecture for an end-to-end trainable document processing models, such as DocFormer (Appalaraju et al., 2021 ), text-image-layout (Powalski et al., 2021) , Lay-outLMv2/3 (Xu et al., 2021; Huang et al., 2022) , Dessurt (Davis et al., 2022) , and UI/layout parsing or understanding (Lee et al., 2023; Li & Li, 2023; Wang et al., 2022b) Along this line of work as well as the grounding capability (Deng et al., 2021; 2023) , the DocEdit dataset also features an end-to-end learnable model (Mathur et al., 2023) (in this work we use GLIP (Li et al., 2022) ). We show that modularized framework leveraging LLM as an agent to program the edit actions with the help of external tools (e.g., component grounders) is a strong competitor for understanding and editing content-rich documents.Content Editing. Our work is closely related to learnable content editing models. Several prior works have also studied language-instructed (Chen et al., 2018; El-Nouby et al., 2018; Li et al., 2020; Lin et al., 2020b; Jiang et al., 2021a) or conversational (Lin et al., 2020a; Jiang et al., 2021b) image-editing, where a language processing module is learned to estimate the requested intents to guide the image-editing tasks. Recent works have adopted modern transformer architecture for better facilitate the instruction comprehension and planning (Shi et al., 2021) , as well as diffusion models for the edited outcome generation (Wang et al., 2022a; Hertz et al., 2022) . Notable works, including InstructPix2Pix (Brooks et al., 2023) , InstructDiffusion (Geng et al., 2023) , and Emu-Edit (Sheynin et al., 2023) push further the generative editing performance with large-scale and multi-tasking data. Inspired by the prior literature, as well as the motivation of the tool (grounding and editing software) usage and keeping unedited parts intact, our framework is able to more seamlessly (and faithfuly) process wider-range of editing actions. 
conclusion: 
 In The description of the object that is to be generated or retrieved . position : tuple If specified , the position in XY -coordinates ( px space ) to add the object .If not specified , add the object to a random position .Examples ------->>> # Add a bullet point object . >>> ADD ( object_type =" BULLET ") """ pass def COPY ( object_id : str , position : tuple = None ): """ Duplicates an object by its given object ID . Parameters -------object_id : str The ID of the object to be deleted . position : tuple If specified , the position in XY -coordinates ( px space ) to add the newly duplicated object .If not specified , duplicate the object to a random position .Examples ------->>> # Copy a text box object . >>> COPY ( object_id = object_id ) """ pass def REPLACE ( object_id_to_replace : str , type_of_object_replaced : str , new_object_type : str , new_object_content : str ) :""" Replaces an object by its object ID with a newly added object by its type and content description . Parameters -------object_id_to_replace : str The ID of the object to be replaced . type_of_object_replaced : strThe type of the object to be replaced . new_object_type : strThe type of the new object to be added to replace the original object . new_object_content : strThe content or description of the new object to be added to replace the original object .Examples ------->>> # Replace a text box object with an image of a cat . >>> REPLACE ( object_id_to_replace = object_id , new_object_type =" image ", new_object_content =" a cat ") """ pass def MODIFY ( object_id_to_modify : str , modification : str , type_of_modification : str ): """ Modifies the content or attributes of an object . For text objects the modification is the textual content to modify or replace the original one .For non -textual objects the modification is the edited attribute to the object being modified . Parameters -------object_id_to_modify : str The ID of the object to be modified .return postedit_position , postedit_bbox def GET_METADATA ( component_id : str , attr : str , document = None ): """ Returns the metadata information of the given component . Parameters -------component_id : str The canonical id of the component to get the metadata from . attr : strThe type of the metadata attribute to return . document :The object for the slide document . ------->>> # Returns the metadata information such as `fontsize `, or `bbox `. >>> bbox = GET_METADATA ( >>> component_id =" some id ", >>> attr =" bbox ", >>> document = document , >>> ) """ return information And additionally, the actual instructions to the OpenAI GPT-4 code-generation pipeline is as follows:  
 
 Appendix: Table 5 summarizes the main hyperparameters in all of our learning modules.The grounding model will propose several possible bounding boxes ranked with their logit scores, we simply take the top-1 scoring bounding box as the prediction.The REPLACE function can be viewed as a combination of DELETE and ADD, except that the added position is the same as the document element being modified.The main editing software used in this work implements a TypeScript-based low-level API library that can fulfill all the functionalities our pre-defined APIs require, where these low-level APIs will eventually be able to interact with the underlying document structure and perform the actual component manipulations/editing. 


Decision: This paper presents a method to edit visually rich documents by generating code to call APIs. After author rebuttal, it received scores of 5679. On one hand, reviewers agree that (1) the proposed method to consider this as a tool-use agent problem is interesting, convincing and novel, compared with end-to-end diffusion models; and (2) experiments show the effectiveness of the method, both in terms of automatic metrics and human evaluation. 

On the other hand, the drawbacks of the paper include that (1) many implementation details are missing, and the proposed method can be hard to reproduce; (2) more ablations are needed. There are many moving parts to the system, and breaking it down would give valuable insights regarding all the design decisions that eventually led to the proposed architecture.

The authors have done a relatively good job of rebuttal. Given the overall positive score with one strong support from one reviewer, the AC thinks that the merits of the paper outweigh the flaws, and decided to recommend acceptance by the end.

Reviews: review: paper_summary: paper_summary: This paper presents a method to edit visually rich documents by generating code to call APIs. It achieves this by combining multiple components, including SRL, visual grounding, code generation, and execution-based feedback. It improves end-to-end performance of visually rich document editing.
strength_weakness: strength_weakness: - The proposed method works by combining multiple existing components in each stage of processing. More discussions / comparison on the choice of components will be useful.
- Analysis of success and failure examples will be useful too
paper_summary: paper_summary: In this paper, a multimodal LLM-agent-based framework, Agent-DocEdit is proposed for the task of automatic (multimodal) document content editing. It consists of four major components: 
(1) Edit Program Generator -- takes the user's editing request as input and generates a sequence of editing function calls via LLM code generation. 
(2) Multimodal Grounder -- identifies the element IDs in the document that need to be actioned upon, using an architecture derived from phrase grounding models for computer vision tasks. 
(3) Executor -- rewrites the edit program to executable programs that modify the document content.
(4) Feedback Provider -- An LLM-based evaluator that compares the executor output against the original request, and generates feedback which is used to instruct the EPG (stage 1) to refine the generated edit program.
strength_weakness: strength_weakness: There are a lot of implementation details that are missing, which would make it hard to reproduce the system or reason about the robustness of the results. Here are a few: 
* 2.4.1: Dataset construction. How are negative images constructed and verified for quality (i.e. how do you reject bad training examples)? What is the proportion of positive and negative samples? If only minor templates are used, is the model robust to newer kinds of instructions, such as the ones that heavily paraphrase the templates or follow completely different grammatical structures (e.g. "hbd chage color to red", instead of the well-formatted "Change the color of the happy birthday to red")?
* 2.4.1: Feedback-based refinement. How is the quality of the Multimodal Feedback module evaluated? 
* 3.1: Error analysis of both action parameters and component parameters is missing. What types of actions does the system do well at, what does it not do well at? What kinds of components are easy for the system to handle? 
* 3.3: A lot of the details of human evaluation are missing. Who are the human evaluator? What is the input shown to a human evaluator? What are the guidelines given to the human evaluators? How many annotations are done per test sample? How are conflicts adjudicated?
* Discussion on the limitations of the feedback mechanism seems to be missing. Are there any cases that pass the semantic and syntactic tests yet do not fulfill the request accurately? For example, what if the system tampers with an original text that needs to be modified or changes its font or size in an undesirable way? What about changes to elements not specified in the original request (i.e. maybe some text was asked to be changed but led to a change in other components, etc.)

* It will be useful to see an end-to-end analysis of system robustness and error analysis. Is the system able to recover from noise in the inputs? Which stage of the pipeline is the most robust? Which one is the least robust? If a given input fails at a certain stage, can the later stages ameliorate any issues? A Sankey diagram visualization would be useful. 

Overall, while this is an interesting work, but seems to be lacking in terms of novelty. It reuses various pre-existing components from prior works. While that itself is not a problem, the lack of rigor in describing the details or conducting the evaluations limits its potential impact.
questions: questions: - How well does the system scale to new tools?
- How well does it scale to new actions (e.g. ROTATE, FLIP, CROP, SCALE, etc.)
- How sensitive is the system to different kinds of prompts? 
- There are a lot of incoherent sentences in the paper, e.g. "the Edit Program Generator takes input human user editing request (in
natural language) as input." I'll urge the authors to please review the manuscript to make it more readable.
paper_summary: paper_summary: This paper reads like an industry-track system description paper about a system architecture for document editing. There are 4 system components working together to allow users to edit visual documents (in this work: an image rendered from an underlying document structure). An interesting feature of the system is a learned feedback loop that yields pretty impressive gains over the baseline.
strength_weakness: strength_weakness: Even though the overall evaluation metrics look good, I would have wished for more ablations. There are many moving parts to the system, and breaking it down would give valuable insights regarding all the design decisions that eventually led to the proposed architecture. Aspects like the complexity (or simply lines of code) of the generated edit programs, the number of syntax and semantic errors without feedback mechanism, the accuracy of the semantic error classifier, etc. would give the reader a better intuition.

My two main concerns, however, are about clarity and limitations. Sec. 2.3 states that the API operations in the edit program "are deterministic and are editing software-dependent, while the major API operations defined in Section 2.1.1 are generally applicable and hence software-agnostic.", which is confusing to me. The python codes shown in the figures actually use the API operations defined in Sec. 2.1.1. IIUC the executor eventually limits what kind of edit operations are supported by the system, but details are not mentioned. For example, MODIFY() in Fig. 3b seems to support "modification=curve" - what other modifications are supported? In general, I feel that authors could be more upfront about the limitations of the system. 
- What are the programmatic limitations of the API? Can it handle multiple bounding boxes (e.g. paint all zebras in the image blue) and/or multiple document elements (e.g. change the national moon day document to a daytime scene)?
- What are the runtime implications of such a complex system design with a feedback mechanism?
- Another limitation compared to approaches operating on the raster image is that the underlying document structure has to be available, so it cannot be applied to any image.
questions: questions: - Not sure about the term "grounding" as this usually has a different meaning in the context of NLP/LLMs.
paper_summary: paper_summary: The authors report on a very interesting and very relevant problem related to document engineering: enabling users to perform complex document editing tasks which can be formulated as short natural language requests or edits. The use of LLMs to transform these requests into sequences of operations calling various already existing APIs is very convincing and the evaluation shows encouraging results. The paper is in general well written and easy to follow, the running example is well chosen. 

What is not clear from the evaluation, however: is there any comparison on human execution of the requested edits? Would humans reacting to feedback to first drafts usually given in short notes or sentences just like used here, execute similar or other actions? Would the result be similar or identical? I think the emphasis would not be in efficiency as in "getting more edits done in less time so paying layouters less" but to be able to present more variants easily to a client to show different solutions to a more general feedback like "can you do something about X here". 

I have two additional issues the authors should resolve for submitting the final paper, see below.
strength_weakness: strength_weakness: There are no reasons to reject the paper
questions: questions: Please position the section on related work just after the introduction. This way readers can grasp why you are using which existing components. The introduction with the very short reference to the two existing but unsatisfactory approaches is fine as it is now.


The formulation "linguistic request" is rather odd: One would assume edit requests related to linguistic features of text like shortening sentence or paragraphs, reversing arguments, changing from passive to active voice, etc. On some instances you use the phrasing "natural language" or "language"—which is more appropriate. In fact, the main advantage is that users can formulate ediing requests by using unrestricted natural language. Please change the respective instances in the text accordingly.

Comments: comment: Thanks for the response!
Dear Reviewer 5Bmb,

We appreciate the constructive feedback you provided, and have made our best efforts to address all your concerns and suggestions in detail.

As the end of the discussion period is approaching, we hope you could kindly let us know if they successfully addressed your concerns and helped you re-evaluate our paper.

Please do not hesitate to let us know if you have any further questions! We appreciate your efforts in reviewing our work!

Thank you very much!

Authors
Thanks for the response. I'm keeping my positive rating.
Thanks for the informative feedback! I keep the positive review of the paper based on the feedback.