Visual Prompting with Iterative Refinement for Design Critique Generation

Authors: Peitong Duan, Chin-Yi Cheng, Bjorn Hartmann, Yang Li
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: User Interface Design Critique, Multimodal LLM, Visual Grounding, Prompting Techniques

Abstract:
Feedback is crucial for every design process, such as user interface (UI) design, and automating design critiques can significantly improve the efficiency of the design workflow. Although existing multimodal large language models (LLMs) excel in many tasks, they often struggle with generating high-quality design critiques---a complex task that requires producing detailed design comments that are visually grounded in a given design's image. Building on recent advancements in iterative refinement of text output and visual prompting methods, we propose an iterative visual prompting approach for UI critique that takes an input UI screenshot and design guidelines and generates a list of design comments, along with corresponding bounding boxes that map each comment to a specific region in the screenshot. The entire process is driven completely by LLMs, which iteratively refine both the text output and bounding boxes using few-shot samples tailored for each step. We evaluated our approach using Gemini-1.5-pro and GPT-4o, and found that human experts generally preferred the design critiques generated by our pipeline over those by the baseline, with the pipeline reducing the gap to human performance by 50\% for one rating metric. To assess the generalizability of our approach to other multimodal tasks, we applied our pipeline to open-vocabulary object and attribute detection, and experiments showed that our method also outperformed the baseline.

Decision: reject
Reviewer scores: 5, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1729858134395

thread_type:
official_review

<thread_object>
creator: reviewer_s1k2

<content>
summary:
This work focuses on generating high-quality design critiques, where the inputs are an UI screen and design guidelines and outputs are design comments along with corresponding boxes that map each comment to specific region in the screenshot. It proposes an iterative visual prompting approach for UI critique, where LLMs are leveraged to iteratively refine both the text output and bounding boxes using few-shot samples tailored for each step.

strengths:
1.	It focuses on a novel, practical and difficult task - generating design critiques, which will have great impact on UI design by saving much design effort and accelerating design process.

2.	The insight into designing each component of the prompting pipeline is clearly explained.

3.	The experiments demonstrate the proposed method effectively.

weaknesses:
1.	There is no qualitative results. Without good examples, I cannot concretely know what design critiques can be generated. For example, are they diverse enough? Can the difficult design critiques be generated? Without bad examples, I cannot clearly know the shortcomings of the proposed method.

2.	There is no discussion about the cost in terms of time or number of calls. The prompting pipelines involves many rounds of refinement and relies on the visual capabilities of LLMs. All of these will make it a slow one, which will influence the user experience.

3.	There is no comparison with methods based on fine-tuning, e.g., fine-tuning open-source multimodality LLMs like LLAVA or continually fine-tuning the one from Bravo et al. (2023). If fine-tuning based methods perform comparable or even better than the proposed prompting pipeline, we should use fine-tuning since the prompting pipeline with iterative refinement is slow and expensive.

======================

After discussion with authors, most of my concerns have been addressed. While the techniques may not be highly novel and could have a limited impact on the broader AI/ML community, I appreciate its contribution to the specific domain of AI-assisted design. I would like to raise my score from 5 to 6.

questions:
How many rounds of iterative refinement are used in the experiment (averagely)? What will happen if we use less rounds of refinement or allow for more rounds of refinement?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730529829198

thread_type:
official_review

<thread_object>
creator: reviewer_wlxh

<content>
summary:
The paper presents an LLM based method to provide a text based critique (design feedback) on a UI design, given an image of that design.  In addition, the method can ground its feedback using bounding boxes overlaid on that image to indicate the basis for the feedback.

The technical contribution is the engineering of an LLM based pipeline, which consists of LLMs coupled together in multiple stages: 1) TextGen – generates design comments as text given a task/system prompt without any grounding; 2) TextFilter – a further LLM is used to prune spurious comments; 3) BoxGen – creates the grounding bounding boxes for each comment; 4) BoxRefine – Improves accuracy of the bounding boxes.

The paper explores the efficacy of various LLMs (GPT4 vs. Gemini1.5) at the task and ablates these stages and parameter choices within them to justify the pipeline design.  The experiments are done on Duan et al’s UICrit dataset and baselined against the recent Duan et al. CHI 2024 paper which addresses the same task.

strengths:
Automated UI design critique seems a well motivated task; there is a potential for UX design efficiencies in using an LLM for this purpose although without actually sequencing / simulating the interaction with an interface the feedback is limited only to superficial judgements around appearances of UI elements.  There may be value therefore in this idea as a kind of visual ‘linter’ for UI design that could be an assistant or validation check for less experienced designers.  That said, the paper is not the first to propose the task – see the recent (2024) dataset and baseline of Duan et al. cited.

The paper is clearly written – the stages of the LLM pipeline are straightforward and reproducabile, as relevant system prompt examples are given in the appendix.

Given that the paper essentially proposes a 4 stage pipeline, the necessary ablation is in place to show the value of each of the 4 stages.  A brief baseline comparison is made to the prior work of Duan et al (CHI 2024) on the same dataset (UICrit).

weaknesses:
There is limited technical innovation or scientific insight in the paper.  Essentially, the authors report a way to couple together several LLM based processes to create a grounded description of the image in the context of UX design guidelines (derived from the classic Jakub HCI paper on the same).  Whilst the design justifications for each stage are given, there is no real insight into the LLM’s capabilities or limitations.  Why is an LLM able to do this?  Which guidelines can it advise best on, and why?  Rather the innovation is presented at face value, as an engineering result.

The paper introduces a formalised notation for the UI design critique tasks which seems unused later in the paper.  Whilst it is welcome to have a clear task definition, I do not see the value of the math notation defined in Section 3 when it is not actually used later i.e. in the Method section of the paper.  The use of this formalism appears to distract from what is otherwise largely an engineering based contribution sequencing multiple LLMs together.

An attempt is made to assess the alignment of the UI critiques with human critiquers.  However the experiment is performed on only 33 UI examples, using 18 humans.  Whilst it is laudable to try to quantify human alignment, picking just 33 UIs out of a the huge design space of UIs can give no meaningful information on the performance of the LLMs at the general UI design critique task.  Also the task differs from the method proposed in the  baseline Duan et al. baseline, in that users are asked only to validate the comments for each UI region (i.e. check correctness) they are not asked to comment on the entire UI (i.e. check completeness).

I’m confused as to the purpose of Section 6 which seems to apply the UI critiquing 4 stage pipeline to the general open-world object detection/description task similar to Grounding DINO.  Why would this task be relevant to the proposed pipeline?  This entire section should be removed in my view.

Minor: The paper describes the pipeline as having 3 stages (comprising 6 LLMs) but then describes 4 stages (TextGen, TextFilter, BoxGen, BoxRefine).  pp.9 ‘Section’ typo in cross-reference:

questions:
Overall this paper addresses the relatively novel task of UI design critique using LLMs.  However the contribution is largely engineering based, creating a particular sequence of LLM promptings to create UI design critiques.  There is limited depth of insight as to the capabilities and limitations of this pipeline, beyond a baseline comparison to one prior work  and some ablation of the design itself.  The user study is so sparse as to not really provide any insight, and deviates without good justification from the approach of the baseline method.  An extra study (Section 6) that seems out of context, as well as some math formulation (Section 3) serve to bulk up the paper but don’t provide useful additionality to the exposition or problem defined.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565074056

thread_type:
official_review

<thread_object>
creator: reviewer_pcxv

<content>
summary:
The paper presents a method that coordinates multiple LMMs for design critique generation. The method takes as input a UI design image along with a set of design principles, and produces text comments on the design issues as well as a set of bounding boxes for visual grounding of the comments (i.e., localizing the problematic regions).

The main contribution is to improve (Duan et al., 2024b) through iterative refinement of outputs and a set of specially designed prompting techniques.

strengths:
1. Building computational methods for automatic generation of design critiques is an important problem to solve.

 2. The system is well designed and illustrated, and evaluated extensively.

weaknesses:
1. The scale of technical novelty is limited. What the paper actually does is to bring an existing idea, i.e., iterative output refinement for LMMs (Madaan et al., 2023; Xu et al., 2024a) into an existing problem domain, i.e., multimodal design critique generation (Duan et al., 2024b). The paper introduces several prompting techniques, such as including zoom-in image regions for the predicted bounding boxes into the visual prompts for bounding box refinement. However, the amount of novelty involved in these simple prompting methods is not significant enough. To increase the level of technical novelty, one possibility is to come up with more sophisticated prompting techniques, e.g., to generate better few-shot examples than those created via simple random perturbation for BoxRefine, Validation and TextRefine, or to iteratively modify the UI image beyond adding coordinate markers for more efficient bounding box refinement.

2. The quantitative scores of (Duan et al., 2024b) in terms of Comment Similarity and Estimated IoU (used in Table 2) are missing, and should be added. 

3. The improvement upon (Duan et al., 2024b) in terms of comment quality (from 0.45 to 0.47 in Table 3) is small. For an incremental work, a more noticeable performance boost is expected, e.g., from 0.45 to 0.5 that almost lies midway between (Duan et al., 2024b) and Human.

questions:
In the human evaluation, why were the participants not asked to rate bounding box accuracy as in (Duan et al., 2024b)? Is “BBox IoU” in Table 3 computed by comparing with the ground truth bounding boxes? If so, is it possible for “BBox IoU” to penalize a predicted bounding box that is valid but different from any ground truth ones?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649431208

thread_type:
official_review

<thread_object>
creator: reviewer_jbj1

<content>
summary:
This paper proposes a pipeline for generating design critique through interactions among large language models (LLMs), which provide design suggestions for UI interfaces and generate corresponding bounding boxes. By incorporating feedback techniques, the system refines bounding boxes or text based on the output of “Validation” step, resulting in more accurate text or bounding boxes. The authors also explore the model’s performance in tasks such as open vocabulary attribute detection and object detection.

strengths:
1. The paper proposes Validation and Iterative Refinement modules that, depending on the specific context, selectively optimize either the box or the text, thereby further enhancing the model's accuracy.

2. The model outperforms previous works on the UICrit dataset.

weaknesses:
1. **Minor Contributions**: Compared to previous work, the authors primarily added the Validation and Iterative Refinement modules; however, these modules lack novel and crucial design. Additionally, the IOU performance of the box generation module only reaches around 35%, limiting its practical applicability.

2. **Poor Result Presentation**: For both the design critique task and OVAD task, the paper provides minimal visual examples of the model's actual outputs. Instead, it predominantly uses ground truth images as illustrations, lacking clear demonstrations of the model's generated results.

questions:
1. **Visualization of Results** 

- **a)** Showcase as many successful and failed cases as possible. Provide a clear analysis of improvements compared to previous models and discuss reasons for unsuccessful cases.

- **b)** Visualize the refinement process with intuitive displays of box and text feedback, as well as refinement results. This will help validate the roles of the respective modules.

2. **Trainable VLM Choice** : Would directly fine-tuning advanced open-source VLM models, such as LLava or InternVL, on the UICrit dataset yield more accurate results?

3. **Simultaneous Generation of Box and Comment** : Since comments correspond to specific boxes, humans typically focus on the area (box) first and then generate corresponding text. Generating the text before locating the box seems counterintuitive. How would generating the comment and box simultaneously impact the effectiveness of the model?

4. **Ambiguous Metrics** : In both attribute detection and object detection tasks, mAP is a metric based on the percentage of successful samples, so performance differences are typically reported using absolute differences. However, in the paper, performance improvements are expressed as percentage increases in several sections, including the abstract, line 075, and line 483. This unconventional representation may introduce ambiguity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Although one of the reviewers suggested acceptance, others raised concerns regarding the limited novelty and insufficient analysis. Specifically, the reviewers agreed that the technical contributions were marginal. The AC also reviewed the paper, the reviews, and the rebuttal. The paper is well written (acknowledged by other reviewers), but iterative refinement has been widely explored in the literature, and applying this idea to the given task is considered incremental. Therefore, the AC agrees that the current version is not ready for acceptance. The AC strongly encourages the authors to address all the reviewers' concerns and resubmit to a future venue.