# FusAIn: Composing Generative AI Visual Prompts Using Pen-based Interaction

Xiaohan Peng, Janin Koch, Wendy E. Mackay

<figure><img src="image_1.png" /></figure>

## To cite this version:

Xiaohan Peng, Janin Koch, Wendy E. Mackay. FusAIn: Composing Generative AI Visual Prompts Using Pen-based Interaction. CHI 2025 - CHI Conference on Human Factors in Computing Systems, Apr 2025, Yokohama, Japan. pp.1 - 20, 10.1145/3706598.3714027. hal-05065936

HAL Id: hal-05065936

https://hal.science/hal-05065936v1

Submitted on 13 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# FusAIn: Composing Generative AI Visual Prompts Using
Pen-based Interaction

Xiaohan Peng

ExSitu, LISN

Université Paris-Saclay, CNRS, Inria

Paris, France

xiaohan.peng@lisn.fr

Janin Koch

UMR 9189 CRISTAL

Univ. Lille, Inria, CNRS, Centrale Lille

Lille, France

janin.koch@inria.fr

Wendy E. Mackay

ExSitu, LISN

Université Paris-Saclay, CNRS, Inria

Paris, France

wendy.mackay@inria.fr

<figure><img src="image_3.png" /><figcaption>**Figure 1: FusAIn:** A pen-based GenAI visual prompt composition tool where designers use (1) Source panel for image search, upload, and display. They extract visual properties with “smart” pens from (2) Pen sets (object, color, texture, basic, eraser) and compose visual prompts on (3) Canvas with editing tools. They can add additional text prompts at (4) Text prompt area, and use (5) Brush scope visualizing pen status. (6) Generation modes include *Guided Generate* and *Merge*. The result appears in (7) Generation panel with style lock, and can be added to (8) Image gallery.</figcaption></figure>

## Abstract

Although current generative AI (GenAI) enables designers to create novel images, its focus on text-based and whole-image interaction limits expressive engagement with visual materials. Based on the design concept of deconstruction and reconstruction of digital visual attributes for visual prompts, we present *FusAIn*, a GenAI prompt composition tool that lets designers create personalized pens by loading them with objects or attributes such as color or texture. GenAI then fuses the pen's contents to create new images. Extracting and reusing inspirational material matches designers' existing work practices, making GenAI more contextualized for professional design. A study with 12 designers shows how *FusAIn* improves their ability to define visual details at different levels that

are difficult to express with current GenAI prompts. Pen-based interaction lets them maintain fine-grained control over generated results, increasing GenAI image's editability and reusability. We discuss the benefits of “composition as prompts” and directions for future research.

## CCS Concepts

* Human-centered computing → Interactive systems and tools; * Applied computing → Arts and humanities*.

## Keywords

Creativity Support Tools, Human-AI Interaction, Design Practice, Machine Learning, Generative AI

## ACM Reference Format:

Xiaohan Peng, Janin Koch, and Wendy E. Mackay. 2025. FusAIn: Composing Generative AI Visual Prompts Using Pen-based Interaction. In *CHI Conference on Human Factors in Computing Systems (CHI '25), April 26-May 01, 2025, Yokohama, Japan*. ACM, New York, NY, USA, 20 pages. https://doi.org/10.1145/3706598.3714027

<figure><img src="image_4.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.
*CHI '25, Yokohama, Japan*

© 2025 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-1394-1/25/04
https://doi.org/10.1145/3706598.3714027
# 1 Introduction

As GenAI image generation offers creative professionals a wealth of new creative possibilities, designers and artists have begun to adopt new workflows that incorporate GenAI. For example, some artists have learned a GenAI-specific vocabulary for creating unique art pieces [12]. Some designers combine their own sketches with text prompts, and continuously tune and iterate them to reach satisfactory results [75]. Researchers have also begun to identify best practices for text-to-image prompt engineering [36], as well as defining prompting strategies that produce satisfactory images or conform to designers' aesthetics [13].

Nevertheless, text-to-image generation still requires users to formulate precise, descriptive text prompts to achieve their expected results [36]. This is not only difficult for visually oriented designers [32] but also forces them to shift from thinking visually to formulating verbal instructions for the system [17, 74]. Articulating and capturing the nuances of the designer's tacit design knowledge poses an even greater challenge for GenAI [44]. Designers also struggle to create professionally usable design assets, largely because current GenAI design workflows prioritize optimizing prompts over supporting creative expression [71]. GenAI design applications with more advanced features include whole-image input with masks for more local visual detail control [49]. However, since the feature directly implements image-to-image pipelines, it assumes the presence of existing visual material with some degree of completion. This workflow changes a designer's role from crafting images to "fixing" images and AI's suggestions, thus limiting engagement with visual materials.

Besides, current GenAI design tools are fragmented — designers must transition among different applications to take advantage of each tool's specific strengths and features [34]. Professional design tools have begun to include GenAI features without fully considering how they will affect real-world design practices [14]. Designers want to avoid GenAI "randomness" and seek greater precision and control [70], even when engaging in open-ended design exploration [37]. Although current GenAI tools can simplify repetitive operations, they do not replace tasks that require comprehension, decisions, or accuracy [34]. This raises a key challenge: How can we support both the designer's creative expression and sense of control over GenAI and reclaim originality in the creative process?

We present FusAIn<fnref n="1" />, a pen-based GenAI visual prompt composition tool that offers designers fine-grained control over visual details, including objects, colors, and texture. FusAIn lets designers (1) extract visual attributes from inspiration image collection; (2) use pens loaded with visual details to compose and control GenAI visual prompts precisely; (3) switch between local and global generation to take active control of image details; and (4) easily re-edit visual compositions by reusing generated design material. The key contributions of this paper include: (1) the concept of "smart" pen that supports GenAI visual prompt composition; (2) FusAIn, a GenAI design tool that illustrates this style of interacting with GenAI features within a holistic process; (3) insights from an empirical comparative study with 12 professional designers; and (4) a discussion of how to apply this concept to enhance interaction

with design-oriented GenAI tools and the generalizability of the interaction paradigm.

This paper first examines related work on expressive interaction through sketching, GenAI for controllable image creation, and GenAI use within design practice. We then introduce the design concept of "smart" pen and different pen functions. Next, we describe the implementation of FusAIn, including its user interface and GenAI functions, accompanied by a use scenario. We then present a structured observation study that compares the interaction approaches in FusAIn to another GenAI design tool. We discuss the benefits of using dedicated GenAI pens to interactively fuse objects and visual attributes as visual prompts, and our concerns regarding the editability and reusability of the generated images. We discuss "composition as prompts" as a GenAI interactive prompt paradigm and implications for future research.

# 2 Related Work

In the following section, we briefly introduce insights from previous work on Expressive Interaction through Sketching with a focus on pen use, before outlining existing approaches for Controlling GenAI Image Creation and the potential of GenAI in Design Practice.

## 2.1 Expressive Interaction through Sketching

Expressiveness in interaction is "the way and degrees to which people can convey thoughts or feelings" [9, 16]. Sketch-based interaction techniques were first introduced from Sutherland's Sketchpad [60], which inspired sketch-based graphical computer interfaces and interaction with "light pen" and mouse. Sketch-based tools have shown the effectiveness of combining sketch and computer interfaces, enabling users to quickly draw and transform digital (SILK [33]) and physical (Musink [67]) elements into high-fidelity interactive objects. Augmented pens are often used as input devices for enabling sketch-like interaction. Pen-based interaction offers a well-defined contact with the surface allowing "precision, writing and sketching tasks" [24]. Pen strokes can be also used as ink or gesture commands on digital surfaces, resulting in a plethora of potential interaction and application cases. The level of precision can be adapted to produce detailed interaction techniques, such as Pen + Touch [25] and Thumb + Pen [47], which allow users to have fine-grained control over the input. More expressive pens, such as I/O Brush [54], in contrast, can be used to extract real-world color, texture, and movements and apply them as expressive paint.

Research has also explored augmented pens and sketching interaction beyond conventional use cases, particularly in creative settings. Pen augmentation such as Conté utilizes different pen contact properties for various digital commands and drawing functionalities [73], while InfinitePaint augments pens in VR drawing environments with realistic haptics [19]. Sketching interaction in filtered.ink allows using customizable SVG filters as pen inks [78], while Vignette allows smooth interaction transitions between pen inks and commands to create and apply textures [30]. Sketching beyond creative use, as in RealitySketch, enables dynamic motion and trace definition with responsive sketches [61].

The pen's expressive yet controllable characteristics and creative uses open a design space for human-GenAI interaction in creative contexts. Professional design practices often require precision

<footnote n="1">¹The name FusAIn is inspired by "Fusain", the French word for the charcoal pencil used in creative practice. We highlight "AI" to indicate its use as part of this process.</footnote>
and control, which can be challenging for current GenAI design tools to achieve [75]. Using pens to enhance the controllability of professional design tools and extend it to GenAI usages offers an opportunity to reach a balance between expressiveness and accuracy in human-GenAI interaction.

## 2.2 Controlling Generative AI Image Creation

While text prompts let non-AI professionals customize input with few constraints to generate AI images, the output quality varies according to the input techniques and model behaviors. Text entry remains the most common interaction in current image generation GenAI applications such as *Adobe Firefly* [1], *Dall•E* [43], *Midjourney* [42] and *Stable Diffusion* [58]. However, as the initial excitement surrounding AI-generated images gradually fades, controllability is increasingly acknowledged as a key factor in ensuring these images are truly practical and usable [70]. The literature emphasizes solutions including crafting and refinement of text prompts to better control image generation [6, 13, 28, 55].

Advanced image creation AI applications now allow more defined image input and techniques, enabling users to control generation over parts of an image. For instance, some research has delved into locally applying visual features, such as visual styling [35], texture [26] and color schemes [69] to another image area. Some other works use sketching as input to indicate local area modification in shadow [76], lighting [41] and facial features [48].

Input workflows with model control techniques are also emerging for controlling image generation, alongside the use of more defined prompting modalities and input techniques. Node-based workflows in *ComfyUI*<fnref n="2" /> and *InvokeAI*<fnref n="3" /> enable users to tune generation related parameters to control diffusion-based image generation. These workflows allow using customized model input and control units. For example, adding *ControlNet* [77] to these workflows enables different types of control during diffusion image generations, such as persevering edges, depth, and poses. Using *InstructPix2Pix* [8] and *DreamBooth* [53] allows users to further edit and specify more image details with additional text prompts. Even though these improvements in models and input techniques can make image generation more accessible and controllable, they are still siloed in standalone AI applications primarily driven by AI features, rather than supporting a holistic and continuous process.

Recent research in human-AI interaction has begun to explore more controllable yet expressive interaction with alternative modalities, mediums, and design spaces. Using images, colors, and semantics in *DesignPrompt* lets designers decompose prompts visually and semantically [45]. Direct manipulation with text, code, and images in *DirectGPT* [40] enables users to construct their prompts more expressively to control the generated output. *PromptPaint* [14] lets users steer the text-to-image generation using painting-inspired interaction for more expressive and controllable results. Expressive interaction modalities such as the combination of sketching and speaking in *DrawTalking* demonstrate building stories and scenes with AI interactively [52]. Multi-dimensions of writing generations in *Luminate* let users further explore an expressive design space with structured contexts [59]. While these studies provide direct,

contextualized, and controllable interaction, their outputs remain exploratory to varying degrees, which highlights a research gap in developing precise, intuitive input techniques with controllable technology. The combination can offer greater control and flexibility, which are crucial, particularly in professional settings.

## 2.3 Generative AI within Design Practice

Design is a visual and reflective practice [57]. Professional designers understand and contextualize their work through ongoing reflection within the design process [23, 56]. A major aspect within this process is the ability to explore and understand the design material at hand to appropriate it to the task [23]. Research showed that designers started using GenAI in their professional and personal practice in recent years [62, 70]. The majority highlighted GenAI's inspirational potentials in design activities [37] such as visual concept decomposition [72], aesthetics and UI inspirations [34], design co-ideation [64], fashion ideation [18], moodboarding [45] and semi-functional prototyping [46].

Nonetheless, the limited engagement with design materials may indicate the limits of GenAI's value beyond inspiration in professional design practice. While non-explainability and "ungroundness" [22] of GenAI can be perceived as positively accelerating inspirations in divergent creative phases [70], it often fails in design activities that demand greater precision, human control, and critical thinking [11, 65].

Control in design context aims to guide GenAI outputs toward either expected or unexpected directions, ensuring the results are both applicable and contextually appropriate [65]. Balancing user control and system generation remains a critical challenge [63] and will determine the overall usefulness of GenAI to create meaningful output for more professional use [62]. An interaction paradigm that enables designers to engage with digital material and maintain control while using GenAI is yet to be explored.

## 3 “Smart” Pen Design Concept

Designers collect inspiration from a wide range of sources [31] using various mediums. A common way designers express visual concepts is through visual collages composed of sketches and images of different fidelity [21]. Designers often use visual material to capture various aspects of inspirational sources and “merge” them into new visual concepts [45]. The deconstruction and reconstruction of inspiration (an example of a fashion design process in Fig.2) occurs through reflective iterations at different phases of the design process, including conceptualization, design development, and prototyping [29]. The continuous engagement with design materials helps designers better understand and contextualize their work [66]. Conventional digital design tools let designers visualize concepts with fine details, however, they are usually time-consuming to learn and use. Manual composition such as collage or sketch is often faster, but may not produce the required level of visual fidelity. GenAI offers a possible solution, and more specifically research has shown that designers want to utilize richer GenAI input beyond text or single image [45]. Our goal is to develop a system that enables designers to use GenAI for high-fidelity compositions while maintaining precise control over the results. This makes GenAI

<footnote n="2">²https://github.com/comfyanonymous/ComfyUI</footnote>

<footnote n="3">³https://github.com/invoke-ai/InvokeAI</footnote>
more usable in the design process without requiring modifications to models or algorithms.

<figure><img src="image_5.png" /><figcaption>**Figure 2:** A fashion designer's design process: initial visual inspirational materials from the designer's moodboard [2, 39, 50]; deconstructed design elements; and a reconstructed design. From collection HÊTÉROGENÈSE ©Yuexuan Wang</figcaption></figure>

Sketching is an archetypal activity in design [10]. Designers prefer using pens to sketch rather than using text-to-image AI generation when developing new concepts. One important reason for this preference is the direct nature of using pens when noting concepts [79]. Hence we use a pen-based design inspired by the *reification* principle of Instrumental Interaction theory [3, 4], which transforms the user's commands into interactive instruments. The “smart” pen design concept reifies text- and image-based prompts into personalized pens: users “pick up” objects or visual properties such as color and texture from inspirational source images and load them into different pens. Users can then “draw” to fuse those objects or properties into a destination image. Such “smart” pens are not only expressive and versatile but also precise: users can fine-tune both the details of what they load into the pens and where and how to apply the results to the destination image. These personalized “smart” pen offers a novel approach for interacting and exploring the creative capabilities of GenAI.

The design concept of “smart” pen integrates tangible interaction for enhanced user control while deconstructing and constructing visual attributes to improve the contextualization and precision of GenAI prompts. Designers can interact with a set of digital “smart” pens like drawing and sketching on a digital canvas. These pens let them work with and manipulate digital visual material to their needs to harvest GenAI’s ability to create new images inspired by user input. Designers often consider fundamental properties of inspirational material such as color, texture, and shapes in a typical professional design process, e.g. see Fig.2. We considered multiple visual levels at which pens can interact and demonstrate their capabilities, identifying potential functionalities to ensure our design concept supports diverse design disciplines. Thus we outline five examples of pens reflecting high and low-level visual details: object pen, color pen, texture pen, shape pen, and depth pen (concept sketches shown in Fig.3).

**Object Pen** can extract parts and objects from visual material. The extracted objects can be applied to a canvas as well as combined with others in the form of a collage. These extracted objects will be merged with other visuals in the generated image. Example Use

*Case: Extract a couch from an inspirational image and brush it onto a collage composed of living room furniture.*

**Color Pen** interacts with the color property of existing visual material. The pen can select one or many colors of an existing image and the designer can draw with it onto another image. The main purpose of this pen is to do context-aware coloring efficiently while preserving its other visual properties such as layout, texture, and shapes. GenAI will apply the color onto the objects and locations the designer marked. Example Use Case: Select a dark green color from an image of a forest and brush it over the above couch to color it dark green.

**Texture Pen** allows designers to extract interesting textures from existing images and apply them to other objects effectively while preserving other visual attributes such as colors and shapes. A generative model will re-texture the applied area based on how the texture is brushed. Example Use Case: Select a velvet texture from an image of an evening dress and brush it over the above couch to make it look like velvet.

**Shape Pen** interacts with the shape property of objects. Designers can extract shapes from other objects and apply them to image objects or simply create different outlines of image objects themselves. The GenAI model will reshape the object based on the shape provided, while preserving other properties such as colors and textures as much as possible but adjust accordingly based on the new shape. Example Use Case: Draw outlines on the couch to show how much to increase the size of the couch's arms and reduce its back.

**Depth Pen** interacts with the depth property of objects. Designers can extract depth information from single or multiple objects and apply it to other image objects. They can also adjust the relative depth of image objects. The GenAI model will relocate the object based on the depth provided while preserving the overall visual properties and the object's surroundings as much as possible. Example Use Case: Pick and rearrange the furniture in the room, automatically adjusting the depth as needed.

The design rationale emphasizes preserving original pen interaction while embedding AI capabilities through intuitive interaction that is easily understandable by non-AI-expert users. Additionally, while each pen functions differently, the design aims to maintain consistency in the interaction across all pens.

Grounded in design practice, HCI theories, and presented literature, we explore the design concept of “smart” pen to meaningfully integrate GenAI into design practice. In particular, we aim to (1) support existing practical design processes, including deconstructing and reconstructing design materials; (2) enable controllable and editable visual prompts that align with visual and reflective processes; and (3) encourage designers to create and engage with visual design material, not just to fix existing images.
<table><thead><tr><th></th><th>Object Pen</th><th>Color Pen</th><th>Texture Pen</th><th>Shape Pen</th><th>Depth Pen</th></tr></thead><tbody><tr><th>Apply</th><td></td><td></td><td></td><td></td><td></td></tr><tr><th>Generate</th><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>Figure 3: Design concept sketches for object pen, color pen, texture pen, shape pen, and depth pen.</figcaption></figure>

# 4 FusAln Implementation

We developed FusAln, a digital composition tool that uses pen interaction to compose visual prompts for GenAI. Our goal is to evaluate the design concept of “smart” pen and demonstrate their core benefits. We integrated pens that represent the fundamental visual prompt functionalities necessary for the deconstruction-reconstruction within a visual design process. This includes an object pen, color pen, and texture pen, besides more traditional sketching pen (hereinafter basic pen) and eraser. Other “smart” pens are possible, see for example section 3, but are above the scope of this demonstration on core interaction approaches and design process. The basic functionalities of individual pens are showcased in Fig.5. These pens can extract and load multiple visual details from inspirational images and can be used individually or together.

FusAln is a web application built with Vue3 (front-end) and Express.js (back-end). The preferred setup includes an interactive display (Wacom Cintiq 16) and a digital pen (Wacom Pro Pen 2), but other input devices are possible.

## 4.1 Pen Functionality

When a pen is chosen, the canvas will be active for brushing only with the chosen pen properties. The brush scope panel (Fig.6.5) updates to show the current selection as well as further functionalities available for the pen. Users can apply multiple pens to the canvas. After applying different pens, users can select “Guided Generate” or “Merge” (Fig.6.7) to fuse all the applied visual details together.

**Object Pen.** Users can circle the area of interest on an inspirational image and confirm their selection by clicking the upper button on the pen. This transmits the circled object snipped to the FusAln canvas. Alternatively, clicking the lower button of the pen transfers the whole image to fill it as the canvas background. The transmitted object is freely movable and resizable on canvas (Fig.4.a). Once objects are on the canvas, users can select the upper button again in mid-air to trigger an extraction editor pop-up which allows auto background or manual brush removal to have a cleaner

extraction result (Fig.4.b). In the example given in Fig.4.c, users resize the object to fit it in the background.

**Color Pen.** Color pens extract six main colors of the user-selected area from an inspirational image to a color palette shown under the canvas. The color extraction uses color thief <fnref n="4" />, an open-source color palette extraction package for javascript. Users can select a color from the color palette to load it, or choose a customized color from a standard color wheel, and then brush over an area to recolor.

**Texture Pen.** The texture is extracted from the user's selected area on the inspirational image as a textured pattern, then users can use the pen to brush textured strokes. The default size of the

<figure><img src="image_16.png" /><figcaption>Figure 4: Main operations with Object Pen. (a) Transmitting the extracted object from the source panel to canvas. (b) Using extraction editor to remove background. (c) Moving and resizing objects on Canvas.</figcaption></figure>

<footnote n="4">⁴https://github.com/lokesh/color-thief</footnote>
<figure><img src="image_17.png" /><figcaption>Figure 5: FusAIn Pen implementation (object pen, color pen, texture pen, basic pen) overview and basic use (extract, load, apply, generate) of an individual pen. These pens can also be combined together to achieve a more complex composition.</figcaption></figure>

texture depends on the size of the selected area but can be changed in size and repetition gap between textures using the brush scope slider below the canvas.

**Basic Pen.** A basic pen allows users to brush with strokes of various thicknesses and colors, mimicking traditional pens. The ink of the basic pen is directly painted onto the canvas.

**Supporting Pen functionalities.** Brush scope is a panel visualizing the loaded information of the currently selected pen. It also contains additional functionalities for adjusting pen properties. When the color pen and basic pen are active, the brush scope allows brush size adjustment and updating in real time. The size and gap of textures are also adjustable so that users can choose their preferred way of applying texture. Since there are multiple pen designs in FusAIn,

brush scope can help remind users about current pen and actions in a visual way.

## 4.2 User Interface Implementation

**Source Panel** is used for searching, uploading, and displaying inspirational images as shown in Fig.6.A. Users can either choose to upload images, e.g. from their moodboard, or search images using Unsplash [68], an open-source search engine to provide high-quality images. Users can freely scroll over inspirational images by sliding the Wacom pen up and down on the source panel. They can use the different pens to circle on images to extract interested areas and then click on the upper pen button to transmit them to canvas.
<figure><img src="image_18.png" /><figcaption>Figure 6: FusAln User interface composed of (A) Source Panel (B) Composition Panel (C) Generation Panel.</figcaption></figure>

**Composition Panel** consists of a canvas (Fig.6.B.4) where users compose their visual prompt as shown in Fig.6.B.

* **Pen Sets** Users can choose different pens or erasers from pen sets (Fig.6.B.3) and draw with different properties.

* **Text Prompt Area** enables users to use text as additional input to composed images (Fig.6.B.1).

* **Canvas Operation Tools** enable users to move objects forward or backward, clear the whole canvas, and select local AI generation area (Fig.6.B.2).

* **Brush Scope** visualizes the current pen 'load' and lets users adjust its properties (Fig.6.B.5).

**Generation Panel** displays generated image (Fig.6.C.8) and allows saving it to the gallery (Fig.6.C.9).

* **Style Lock** allows users to preserve a certain style and overall appearance of a generated image in the following generations (Fig.6.C.6). Activating style lock is equivalent to keeping the same “seed” in the AI generation. On default it is de-activated, resulting in users’ generations being independent of the previous generation.

* **Generation Mode Panel** shows different generation modes such as “Guided Generate” (image-first generation) or “Merge” (text-first generation). This triggers the GenAI to generate a new image based on the current composition panel (Fig.6.C.7). Users can choose to continue working based on the newly generated image by selecting “Use Image” to send the image back to the canvas.

* **Image Gallery** allows the user to store generated images for later use by clicking on the “+” sign.

## 4.3 GenAI Implementation

The GenAI image generation functionalities are implemented using a stable diffusion model [51] v1-5⁵. We have added a neural network structure —ControlNet<fnref n="6" />, to control different visual attributes during the generation. Both, the stable diffusion model and ControlNet implementations are deployed locally and run on local NVIDIA RTX 4000 ADA GPU. For ease of access, we use the local API provided

by Stable Diffusion web UI <fnref n="7" />. FusAln's detailed code and parameters for generation can be accessed from Github<fnref n="8" />.

**Image-First Generation** Users can click on “Guided Generate” (Fig.6.C.7) after finishing visual prompts to generate images with Image-First Generation (example input and output shown at Fig.7.1.a and Fig.7.1.b). This Image-First Generation lets users generate an image that is based on the visually composed prompt on the canvas and preserve different smart pen strokes they have applied. By default, Image-First Generation in FusAln will generate based on the whole canvas as an image input while text input is optional. This is made possible by using stable diffusion image-to-image pipelines [27] combined with three ControlNet units, which preserve the color and texture visual feature separately during the diffusion process by applying slightly different image input for each unit. The detailed stable diffusion, ControlNet units, and their corresponding input can be found in the appendix in Fig.13.

**Local Image-First Generation** We have implemented Local Image-First Generation as a compliment to Image-First Generation to provide users the flexibility of switching between partial and whole image generation (example input and output shown at Fig.7.2.a and Fig.7.2.b). Local Image-First Generation generates contents only in selected areas while not affecting other parts of the image. It is applied by activating the “select GenAI area”, which enables users to prompt a new generation in one or several circled areas on the canvas. The generation mechanism adds user-selected areas as a mask and only generates new content in these areas based on user inputs. If any of the color pen and texture pen is applied, the generation will run through ControlNet units. If there is no pen applied in the selected area but there is a text prompt, the masked area will be generated based on text prompts.

**Text-First Generation** Users can click on “Merge” (Fig.6.C.7) to generate images with Text-First Generation (example input and output shown at Fig.7.3.a and Fig.7.3.b). This Text-First Generation allows users to generate an image that is primarily based on the text prompt but uses the visually composed prompt from the canvas as a visual reference. The technical implementation uses stable diffusion's text-to-image pipelines and two ControlNet units with different image inputs. The implementation details can be found also in the appendix in Fig.14.

## 5 Scenario

The following scenario illustrates how Bob, an interior designer, would use FusAln to make a visual composition for a living room design. Bob launches FusAln at his Wacom display. He picks up his pen and starts by uploading a set of inspirational images from his previously saved moodboard. He finds a living room picture he likes as a background, then selects the *object pen* and draws on the image to select the area he wants to put on canvas. Then he taps on the upper button of the pen to transfer the extracted part to the canvas. He resizes the part to fit it on the canvas at an appropriate size. Next, he decides to place some drinks on the living room table.

<footnote n="5">⁵https://github.com/Stability-AI/stablediffusion</footnote>

<footnote n="6">⁶https://github.com/lillyasviel/ControlNet</footnote>

<footnote n="7">⁷https://github.com/AUTOMATIC1111/stable-diffusion-webui</footnote>

<footnote n="8">⁸https://github.com/exsitu-projects/FusAln</footnote>
<figure><img src="image_19.png" /><figcaption>(1.a) Image Input
+Text Input (optional)</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>(2.a) Image Input
+Text Input (optional)
+ Local Mask</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>(3.a) Image Input
+ Text Input "Cyberpunk architecture, high quality"</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>(1.b) Image Output</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>(2.b) Image Output</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>(3.b) Image Output</figcaption></figure>

<caption>(1) Global Image-First Generation</caption>

<caption>(2) Local Image-First Generation</caption>

<caption>(3) Text-First Generation</caption>

<caption>**Figure 7:** Three types of Generation in *FusAIn*. (1) Global Image-First Generation. (2) Local Image-First Generation. (3) Text-First Generation. (1.a) User applied 1. Basic Pen 2. Object Pen and Color Pen 3. Texture Pen 4. Object Pen in different parts of the image. (1.b) The generated result with *Global Image-First Generation*. (2.a) User applied 1. Object Pen 2. Color Pen in different parts of the image. (2.b) The generated result with *Local Image-First Generation*. (3.a) User applied 1. Basic Pen 2. Object Pen and Color Pen 3. Texture Pen 4. Object Pen in different parts of the image. (3.b) The generated result with *Text-First Generation*.</caption>

He roughly selects two drinks from another image using the same pen and transfers them again to the canvas (Fig.8.2) again. Then he opens the *extraction editor* by clicking the upper button again to auto-remove their background. He adjusts the drinks to the right size, places them on the table, and then uses *color pen* to pick up a few colors he likes from an advertising image (Fig.8.1). He applies the selected orange color to some couch cushions by brushing over them (Fig.8.3). Then Bob sees the painting on the living room wall and wonders if he could try a different texture. With *texture pen*, he selects an interesting texture from one of his inspirational materials and adjusts its size and gap with the pen scope before applying it to the painting (Fig.8.4).

Bob now wants to see how these combinations would look in a complete image so he clicks *Guided Generate*. After a few seconds,

he sees the generated image (Fig.8.5), which he finds intriguing so he saves it to gallery by clicking the '+' button (Fig.8.6). But Bob notices the lamp that he wants to keep changed in the generated image, so he decides to reuse the composition but with *Local Image-First Generation* mode. He selects *Select Gen Area* and then draws the areas he wants the GenAI to alter. After clicking *Guided Generate* again, he sees the last image but this time with the lamp unaltered.

Finally, Bob wants to experiment with different living room design styles. He types in the text prompt area “blue-tone living room”, and selects *Merge*. The generated image features a futuristic blue tone, and Bob is happy about the styling. He activates the *style lock* to ensure future generated images maintain a similar style.
<figure><img src="image_25.png" /><figcaption>Figure 8: Interface showing (1) source colors selection, (2) applied “drink”, (3) orange color, and (4) icy texture. (5) Generated images can be saved with the (6) '+' button.</figcaption></figure>

# 6 Study Methodology

We are interested in how users interact with FusAln, in particular:

* RQ1 How does pen-based visual composition affect users' ability to express ideas to GenAI?

* RQ2 How does visual prompt composition change the designers' perceptions of control? and,

* RQ3 How does the prompt composing process affect participants' design workflow?

Thus we conducted a comparative structured observation study [38], similar to [20, 32]. Participants engage in two tasks with each design variant, followed by answering questions to assess and compare the advantages and limitations of each variant. An experimenter observes their interaction with each variant.

## 6.1 Identifying Baseline Condition

We aimed to identify a condition that closely reflects the functionalities of FusAln (hereinafter Pen condition) to enable a fair comparison. As no existing applications offer both similar editing features and GenAI utilities, we tried to identify a comparison condition that approximates these functionalities.

We first explored a workflow most closely aligning with FusAln using a common stable diffusion web UI with exactly the same model and ControlNet pre-set as FusAln, plus an open-source design software Krita<fnref n="9" /> for basic image editing features (hereinafter Stable condition). Stable condition provides the same GenAI model features, more technical possibilities, and visual editing features. The design process with Stable condition, however, requires users to switch between two applications and to adapt the GenAI features in the UI. We conducted a pilot study with a UX/UI designer familiar with mainstream design tools such as Figma and Photoshop, and highly experienced with image creation GenAI such as ChatGPT, Midjourney, and Firefly. Each design task with a design variant, Stable condition and Pen condition, was designed to take 15 minutes led by a 10-minute video tutorial for practice.

Our pilot study showed that the designer was able to finish the design task in Pen condition. However, the participant encountered difficulties completing the task in Stable condition within the allocated time, eventually producing two images using only text-based

input. We received feedback that a 10-minute video training of Stable condition was insufficient for understanding and remembering all the basic features.

To provide a more comparable experience, we, therefore, chose a state-of-art design tool Photoshop (including Firefly features) [1] as the main software for our baseline condition (hereinafter Firefly condition). It offers powerful image editing features, and built-in high-quality GenAI features such as Generative Fill and Generative Expand. Besides, designers with over 3 years of experience are generally familiar with standard design tools such as Photoshop, which helps form a high but realistic baseline. Although the features offered as well as the underlying GenAI models are different from Pen condition, we believe Photoshop (including Firefly features) is the closest comparable design software with GenAI features (a feature comparison is shown in Table 1).

## 6.2 Participants

We recruited 12 participants (9 women, 3 men; aged 24-30) who are experienced in diverse creative domains and have at least three years of design training or practice (Table.2). The participants rated their experience with GenAI from “Not Familiar at all” to “Very Familiar”. We followed our institution’s IRB policy both in our explanations of the study to participants and study design. All participants signed informed consent forms and agreed to voice and screen recordings. We anonymized the results and followed European privacy law (GDPR).

## 6.3 Setup

A Microsoft Windows laptop running FusAln and Photoshop (with Firefly) sits on a table, with a large external monitor, a Wacom Cintiq 16 tablet, a Wacom Pro 2 pen, a mouse, and a keyboard. The experimenter launches the corresponding software of Pen condition or Firefly condition according to the experimental protocol.

## 6.4 Procedure

The study lasted approximately 1 hour to 1 hour 30 minutes (the varied time was due to participants' varied time taken for training). We use a [2x2] within-participant design with two factors: The primary factor related to the INTERACTION STRATEGIES: Firefly condition (baseline) and Pen condition (variant). The secondary factor is the task, for which we created two equivalent DESIGN BRIEFS. The tasks are designed to require general design skills without highly specialized expertise. Both, the design task and interaction strategies, were counterbalanced using a Latin-Square to account for order effects. The study protocol consists of five steps:

(1) **Introduction**: The experimenter first describes the study and obtains informed consent.

(2) **Training**: Each task begins with a 10-15 minute training session. The experimenter walks users through an interactive tutorial describing all AI and basic image editing features. The training includes hands-on exercises, which cover all the fundamental features participants might use in the design tasks and ensure participants understand the conditions. In Pen condition condition particularly, the experimenter teaches participants how to operate the pen.

<footnote n="9">⁹https://krita.org/en/</footnote>
<table><caption>Table 1: A comparison of features that might be used in design tasks in Pen condition and Firefly condition.</caption><thead><tr><td>Feature</td><td>Firefly Condition</td><td>Pen Condition</td></tr></thead><tbody><tr><td>Text to Image Inpainting</td><td>Yes</td><td>Yes</td></tr><tr><td>Image Inpainting (No text input)</td><td>Yes</td><td>Yes</td></tr><tr><td>Text to Image Outpainting</td><td>Yes</td><td>No</td></tr><tr><td>Image Outpainting (No text input)</td><td>Yes</td><td>No</td></tr><tr><td>Image to Image</td><td>No</td><td>Yes</td></tr><tr><td>Background Remove</td><td>Yes</td><td>Yes</td></tr><tr><td>Image Lasso Selection and Positioning</td><td>Yes</td><td>Yes</td></tr><tr><td>Advanced Image Editing Features</td><td>Yes</td><td>No</td></tr></tbody></table>

(3) **Design Tasks:** Participants perform a 15-minute design task with the assigned condition: Pen condition or Firefly condition using a think-aloud protocol. After each condition, participants fill out a short questionnaire with three Likert-scale questions regarding the ease of visual composition, intention alignment, and versatility. Then participants repeat the process with the second design task and the other condition.

(4) **Comparative Questionnaire:** The participant fills out a Likert-scale questionnaire with ten questions to compare the two conditions.

(5) **Semi-Structured Interview:** The experimenter conducts a semi-structured interview that asks participants to reflect upon different workflows, interaction techniques, expectation alignment, and their perceived role in the design process while interacting with the GenAI.

The two design briefs are:

**Design Brief A:** Your client recently rented a new space and she wants to open a small bar that serves primarily cocktails, mocktails, and wine. To set her bar apart from others in the neighborhood, she intends to provide a modern living room-style experience in which smaller groups can gather for lengthy conversations over good drinks. She is about to decorate the interior of her new location and would like you to provide an interior design concept image or several images that suit these needs.

**Design Brief B:** Your client is planning a new fashion exhibition later this year. As part of the preparation for this exhibition, he is still looking for inspiration in new dress forms that reflect the vibrant autumn styles. He requests that you submit a fashion concept image or several images of a dress depicting new forms, textures, and colors to brighten this colorful autumn.

## 6.5 Data Collection and Analysis

We gathered screen and audio recordings of the think-aloud study and collected responses from questionnaires and images created by each participant. Additionally, we collected handwritten notes from the experimenter, taken while observing the participants.

We performed a reflexive thematic analysis [7] on the recorded audio data, utilizing a combination of deductive (top-down) and inductive (bottom-up) methods. In the deductive phase, we analyzed and coded the data guided by our research questions. The preliminary themes included visual vs. textual expression, visual attribute separation, perceptions of control, user intention alignment, user agency, precision and directness of pen, feeling of sketching, perceived output predictability, reflected aesthetics and styles. Subsequently,

we examined patterns within and across conditions through an inductive approach to develop a more detailed understanding of the data. This process resulted in four key themes: diverse but defined visual details communication, design decision ownership, fine-grained control with pen, perceptions of GenAI predictability. In each step, one author took primary responsibility for coding, while co-authors sampled, coded, and engaged in discussions throughout the process. To highlight common insights among participants, representative quotes and counts are provided. Additionally, we reported participants' responses from the comparative questionnaire.

## 7 Results

This section provides an overview of the questionnaire's quantitative results and qualitative findings from our structured observations. We also present some images generated by participants at the end of design tasks (Fig.10 and Fig.11).

### 7.1 Questionnaire Results

The short questionnaire results showed that the ease of visual composition, intention alignment, and versatility were preferred in the Pen condition, which was confirmed in the comparative questionnaire as well.

The results of the comparative questionnaire (Fig.9) highlight that participants strongly preferred Pen condition over Firefly condition in terms of collaging and composing ideas, and learning about how to compose with AI through interaction. More than 75% participants preferred Pen condition over Firefly condition in ease of defining input and expressing ideas, encouraging creativity and exploration, and understanding and controlling visual details. Over 50% participants preferred Pen condition over Firefly condition in terms of feeling controllable, intention alignment, and reflection on aesthetics and preferences, while less than 24% preferred Firefly condition over Pen condition in these attributes. In terms of ease of predicting generation results, responses from participants' responses show slight preferences over Pen condition but seem more mixed, with 41% strongly preferred or preferred Pen condition, 41% remaining neutral, and the rest preferred Firefly condition.

### 7.2 Visual prompt composition communicates diverse, fine-grained detail

All designers used visual prompts without texts in the Pen condition in design tasks, while 4 of them tried also to include text prompts. In contrast, all participants used mainly text prompts, especially Generative Fill, in the Firefly condition (12/12). Quantitative results
<table><caption>Table 2: Study participants' backgrounds and experience. (12 participants (9 women, 3 men; aged 24-30) with diverse creative backgrounds and design experience (training and practice), and various familiarities with GenAI).</caption><thead><tr><th>ID</th><th>Gender</th><th>Design Background</th><th>Design Experience</th><th>GenAI Experience</th></tr></thead><tbody><tr><td>1</td><td>M</td><td>Industrial Design</td><td>10 yrs</td><td>Familiar</td></tr><tr><td>2</td><td>F</td><td>Graphic Design</td><td>6 yrs</td><td>Somewhat Familiar</td></tr><tr><td>3</td><td>F</td><td>Digital Media Technology</td><td>8 yrs</td><td>Very Familiar</td></tr><tr><td>4</td><td>F</td><td>Human-Computer Interaction</td><td>3 yrs</td><td>Familiar</td></tr><tr><td>5</td><td>F</td><td>Graphical Design, Fine Art</td><td>4 yrs</td><td>Not Familiar</td></tr><tr><td>6</td><td>M</td><td>Human-Computer Interaction</td><td>3 yrs</td><td>Familiar</td></tr><tr><td>7</td><td>F</td><td>Architectural Design</td><td>10 yrs</td><td>Somewhat Familiar</td></tr><tr><td>8</td><td>M</td><td>Human-Computer Interaction</td><td>3 yrs</td><td>Familiar</td></tr><tr><td>9</td><td>F</td><td>Industrial & Product Design</td><td>7 yrs</td><td>Familiar</td></tr><tr><td>10</td><td>M</td><td>Architectural Design</td><td>8 yrs</td><td>Somewhat Familiar</td></tr><tr><td>11</td><td>F</td><td>HCI, Interaction Design</td><td>5 yrs</td><td>Somewhat Familiar</td></tr><tr><td>12</td><td>F</td><td>Graphical & Packaging Design</td><td>9 yrs</td><td>Not Familiar</td></tr></tbody></table>

showed that most designers preferred *Pen condition* to understand and control different visual details when composing images (10/12) and being creative and exploratory in the design process (9/12).

### 7.2.1 Communicating Diverse but Defined Visual Details.
Designers generally prefer expressing their ideas using visuals over text (P1, P2, P3, P5, P8, P9, P12). Two designers described their problems with connecting visual ideas to text in the *Firefly condition* (P3, P6). One mentioned the “autumn element” described in the design brief made him easily connect with the color and texture of the autumn landscape, but there was no way to express it precisely in the *Firefly condition* (P6). One designer found the main problem of text prompts is the misaligned wording interpretations (P1). He put the word “aqua” as part of the prompt, however, “I thought about the color (of aqua), it (AI) thought about the object”. He did not “believe (textual) expression could be sufficient for the computer to understand” (him). Some designers thought the *Pen condition* allowed them to have more diverse choices over defining visual details (P1, P4, P7, P9). One mentioned composing visual prompts in the *Pen condition* allowed him to express via “multiple dimensions” (P1), one is through choosing preferred inspirational images, and the second is “which type of inspiration” (which visual attribute to load and draw). Another designer said such “flexibility” over choosing color or texture to express allowed her to “break down the steps” in design (P9). Others enjoyed the specificity of visual composition (P3, P4). One participant described that using “more specific parts (of inspirational images)” is like accessing “layers of the part like if it’s the texture the color” (P4). The other mentioned she preferred to express her ideas in a “more defined” way because she found challenging to describe a plaid pattern she imagined using text alone (P3).

### 7.2.2 Styling with Textual Input.
The text prompt in the *Pen condition* remained available on the interface, and it was utilized by four designers (P1, P4, P7, P9) toward the end of the study. On top of detailed visual input, we noticed designers used text prompts to add abstract styling details uniformly to selected parts of their composition. One designer put “icy zen bar couch” as an addition because he did not want to “destroy the whole color balance but

just add more details” (P1). Another designer mentioned at the end of her design process she just wanted to change the style as a whole so she put “vibrant style” as shown in Fig.10 (P7). One designer wanted to have a more “aligned tone”, so she put “make the room a bit darker like it’s at night time” (P9).

## 7.3 Controllable composition supports designers to take ownership of decisions

More than half of designers felt more in control over the generated results in the *Pen condition* (7/12). Most designers also think the *Pen condition* enables their intentions to be well aligned with generated results (8/12). One designer described the whole design process as “drawing what I imagined and then seeing if it generates content that matches what I imagine” (P6). Half of the designers (P4, P6, P8, P9, P11, P12) expressed that in the *Pen condition* they felt more “like the person deciding how it’s gonna look like” (P11).

### 7.3.1 Intention Alignment in Output.
For some designers, the ownership of decisions comes from the alignment between their intentions and their generated results (P4, P8, P11, P12). One designer explicitly mentioned that it was not really about “a better result”, but “about realizing what I had in my head” (P8). Another one said she had “some intention with what I’m doing and it does something that aligns with it”, while in the *Firefly condition*, she felt “intention that I have is a bit more lost” (P11). One designer felt she only “write that sentence” and “passively accept what it gives you” (P3).

### 7.3.2 Clear Intention vs. Exploration.
Taking ownership also requires clear thinking for a few designers (P3, P9). Specifically, one designer mentioned that to “pilot the procedure” well in *Pen condition*, she had to “think very clearly” (P9). Whereas some designers in *Firefly condition* thought the “quick” text prompts helps especially in the more exploratory “inspiration phase” (P3, P8, P9).

## 7.4 Pen interaction is precise, direct, and closer to designers' habits

Most participants used the same hardware setup with monitor and keyboard for the *Firefly condition* and Wacom display plus
<figure><img src="image_26.png" /><figcaption>Figure 9: Comparative Questionnaire Results</figcaption></figure>

keyboard (for those who have typed) for the Pen condition (11/12). One designer chose a particular setup with a monitor, keyboard (at non-dominant hand), and Wacom display (at dominant hand) with both the Pen condition and Firefly condition (P1). Quantitative results showed that almost all designers thought in the Pen condition it was easier to define their input (10/12). Most designers preferred the Pen condition in terms of better and easier expressing ideas through interaction (9/12) and from the interaction they learned to compose images with GenAI (9/12). Most designers (P1 P3, P4, P6, P7, P8, P9, P10, P11, P12) enjoyed using pen, but for various reasons. Half of them mentioned pen interaction is closer to sketching or drawing (P1, P6, P7, P9, P10, P11). Some said it was “closer to (their design) habit” and “design workflow” (P7, P9, P10). For one product designer (P9) the interaction was similar to “drawings and mock-ups” while P7 and P10 who have an architecture background connected the interaction with “architecture sketching” specifically. P1 mentioned he liked to “draw not talk”, and pens make him take advantage of “imagery thinking rather than talking”.

Some designers preferred pen for its precision (P3, P4, P8). One designer mentioned that for some detailed tasks such as removing the background, using a pen rather than a mouse to “move from here to there” would feel more direct (P3). Another designer enjoyed the “tangibility” of the pen since she preferred to use tangible devices such as a touch-screen to do precise operations (P8). Two designers mentioned pen interaction helped them express (P7, P12). One designer perceived pen interaction as “a direct expression of what you’re thinking” (P12), While for another designer (P7) it was about thinking while interacting, “I can sketch while I’m thinking”. Another two designers found Pen condition was particularly suitable for “creation” (P3, P11). One designer felt she was “more into a creator’s role” while holding the pen (P3), and another designer thought she would want to use FusAIn “for more creation” type of work since it enables making “something more from scratch” (P11).

## 7.5 Designers perceived predictability based on input controllability and output quality

Quantitative results show that an equal number of designers preferred the Pen condition and hold neutral opinions (5/12) for ease of

predicting GenAI results, and the other few prefer the Firefly condition (2/12). More than half of designers preferred the Pen condition (7/12) while some designers (3/12) preferred the Firefly condition in reflections on aesthetics and preferences in output. When asked about predictability in both conditions in the interview, designers had mixed opinions.

Almost all designers mentioned in the Pen condition the generated results were more predictable (P1, P2, P3, P4, P6, P7, P8, P9, P10, P12). One designer said it was because in the Pen condition “it was easier to guess what it is going to be” (P2). Another designer attributed the predictability to “more information in communication” and the ability to explain “I want this color...texture...only this region to be generated in this way” (P1). Some designers found using text input in the Firefly condition sometimes produced fairly “random” results (P2, P4, P10, P12), which made them think it was unpredictable. Some doubted it was because they were not “good prompters” and their prompts were “too abstract” (P4, P10, P11).

For some designers, the predictability seems also relevant to the image quality generated. Half of designers (P2, P5, P8, P9, P10, P11) mentioned the image output quality produced by the Firefly condition was high and seemed more “mature” (P8). One designer liked the Firefly condition for its great “merging” effects, which she already used in her post-processing visual work (P11). One designer complimented on the output predictability in the Pen condition but she also mentioned some visual details were not fine enough so it affected her perceptions slightly, “it is like a quick preview that helps to anticipate the result, although sometimes the image details are not fine enough” (P2). One designer seemed to think the unpredictability stemmed from the technological perspective of GenAI, regardless of the condition (P5). She was not familiar with GenAI (self-reported in demographics) and considered outputs from both conditions unpredictable, “(in the Pen condition) maybe I have control over the elements that I want, but the results that come out of it are still not exactly what I expected” (P5).
<figure><img src="image_27.png" /><figcaption>**Figure 10:** The top two images show P4 used an object pen to put different objects on the table and then used a texture and color pen to decorate her design; The bottom two images show P7 used texture pen stacked with color pen on the dress object, then used short text prompts for styling.</figcaption></figure>

## 7.6 Interaction with objects instead of layers reduces designers' confusion

When one designer carefully reflected on his design process with the Pen condition and the Firefly condition, he brought up that “layers seem to be an obstacle, especially when you look at different masks and stuff” in the context of AI image generation (P6).

From our observation, similar mistakes with operating layers in the Firefly condition were also observed among some other participants (P5, P7, P8, P9), especially when they tried to do post-processing on the generated image as shown in Fig.12. After participants realized their “mistakes”, they usually stopped further editing on the generated image and reverted to the previous actions. We noticed in the Pen condition the objects participants picked were more frequently reused. As a designer interpreted, “Because I prefer

<figure><img src="image_28.png" /><figcaption>**Figure 11:** P11 used multiple generative fills to compose a cozy bar image in Photoshop Firefly.</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>**Figure 12:** Example of “mistakes” in Firefly condition due to layer confusion: (a) Moving a plant and also shifting environmental pixels on the same layer (P7). (b) applying generative expand at wrong layer, causing non-contextual expansion (P8). (c) rotating images despite on separate layers (P9).</figcaption></figure>

to understand it as an object. And then I'm going to put the object over here and over there instead of a layer" (P6).

## 8 Discussion

In the following section, we discuss interactively defined visual GenAI prompts, considerations of editability and reusability in GenAI image creation, insights into using tangible tools connected with GenAI functions, followed by visual “composition as prompts” as our identified interaction paradigm. We end with a discussion on the system and study limitations and future research directions.

### 8.1 Interactively Defined Visual GenAI Prompt

Designers faced several challenges with text prompts in our study: difficulty articulating prompts, unexpected interpretations by GenAI, and self-perceived inadequacy as “good promoters”. While extensive research aims to simplify prompt crafting for optimal results, we question whether such motifs still stand given that words are inherently open to varying interpretations, even among humans [15].

FusAln demonstrates an interactive visual prompting paradigm that bridges human-controllable and machine-interpretable inputs. Designers explained that FusAln provided more defined visual detail expression at different levels. The “put that there” [5] style of interaction with high-level visual details such as objects, aligns more with the designer’s understanding of visual composition (P6). Drawing-like interaction with low-level details such as color and texture acts on high-level details. Compared to prior augmented “pens” such as [19, 30, 73, 78], a single pen in FusAln supports “extract, load and apply” actions with a single property, but combining different pens offers greater flexibility and controllability when working with different levels of details (example shown in appendix Fig.15). FusAln has improved designers perceptions of control (RQ2) mainly from two aspects: Precise and contextual visual details support controllable visual prompt formation, while controlled “communication” with GenAI models facilitates alignment between designers’ intents and generated results. Deconstructing and reconstructing high- to low-level visual details can be applied to a variety of visual attributes, thus they enable designers to create more customized “smart” pens to expand the space for customizable visual prompt design.

On the other hand, designers used text prompts to describe certain visual details, especially when language could better define
them, such as for styling or comparative changes. Text input allows uniform feature application across the image, whereas achieving similar results manually could be time-consuming. Designers also mentioned text prompting as particularly useful and efficient when broadly exploring inspiration with less defined expectations.

## 8.2 Reusability and Editability in GenAI Image Creation

Layers in digital design tools traditionally separate distinct objects for visual and operational clarity<fnref n="10" />. However, generated layers in *Firefly condition* merge objects with surrounding regions, potentially complicating further user interaction (see Fig.12.a). Separate original and generated layers can also mislead users to do unintended operations (Fig.12.b and Fig.12.c). While users can manually change the mask or combine and rework the generated object, this method is time-consuming and discourages iterative refinement. In contrast, *FusAIn*'s object-level operations facilitated more frequent image reuse through simplified editing and composition, which may encourage the reusability of GenAI images overall.

While certain techniques from computer vision and graphics facilitate feature transfer and contextual editing, their direct integration into design tools necessitates careful consideration. The direct “implant” approach allows designers to use advanced editing techniques but raises questions about their compatibility with iterative creative workflows. This approach, with more focus on features over process, risks overlooking practical but crucial workflow considerations, including material prerequisites, design requirements for specific techniques to function effectively, and asset reusability. Our findings demonstrate that different interaction approaches—generative in/outpainting across layers or iterative composition—impact users’ editing behavior and asset reuse patterns. The “smart” pen concept, enabling fine-grained visual feature manipulation and editable visual prompts, could provide more meaningful control in iterative design processes.

We conclude that visual prompt composition enables designers to define but also re-edit visual details efficiently (RQ2). The improved editability compared to whole-image or layer-based image generation, could potentially improve GenAI adoption among creative professionals and encourage the reuse of generated assets.

## 8.3 Expressive Pen and Added Intelligence

Visually oriented designers enjoy drawing or sketching-like interaction with pens. Pens bring tangibility, directness, precision as well as expressivity that can be unleashed through interacting with the visual material. Some designers hence reported feeling more in a creator mindset when using *FusAIn*.

“Smart” pen introduces new possibilities for digital pen interaction beyond inking or gestural commands by utilizing ink as visual guidance for GenAI generation. The combination of traditional pen use in design with the notion of design materials, such as objects or texture, resulted in the impression the pen is “a direct expression of what you’re thinking” (P7).

Combining GenAI capabilities with pen interaction expands digital pen’s versatility with added intelligence, providing users

tangible control while actively engaging with digital creative GenAI material. It adds expressiveness to current human-AI interaction and interfaces, while also reflecting the possibility of preserving instrumentality in the GenAI context (RQ1).

## 8.4 Visual “Composition as Prompts” in Design

Designers mentioned *FusAIn* encouraged them to be creative and take ownership of design decisions. They enjoyed the precise control in *FusAIn* to deciding what they take from inspirational images, rather than passively accepting generated images and reiterating the prompting process. Using any inspirational material, the “smart” pen facilitates the deconstruction and reconstruction of design materials, offering precision, expressive freedom, and an alignment with existing design habits, while GenAI functions as a powerful tool for visual transformation, producing more contextually relevant outcomes.

The integration of manual composition and controlled GenAI generation engages designers more actively than traditional prompt-iteration workflows, fitting into a holistic design process without adding GenAI as a separate feature (RQ3). Composing ahead of GenAI generation preserves the aesthetics of the design, and avoids disruptions caused by fragmented AI tools [34, 44]. While the modality of pen fits designers’ habits and comes with pen-specific properties, we still see visual “composition as prompts” as a universally applicable interaction paradigm for general purported image generation that offers ease of defining and controlling visual details.

## 8.5 Limitations and Future Work

Participants raised concerns about *FusAIn*'s image quality, such as rough object edges and less refined textures compared to state-of-art [26]. Due to the local computer's performance limitations, we did not incorporate an “upscaler” or more sophisticated model to improve image quality. Despite this, the study was sufficient to demonstrate “smart” pen for professional design and the style of interacting with GenAI features in *FusAIn*.

Study time constraints led some participants to use only a few features in *FusAIn*, which may have affected perceptions of “aesthetic predictability”. A few participants noted that hardware setups, such as Wacom tablets and pens, introduced unfamiliar interaction styles that may lead them to lean toward *Firefly condition*. However, these factors did not impact the overall preference for *Pen condition*.

The study involved creative professionals with 3-10 years of experience (avg. 6.3 years), without distinguishing between novices and experts. Future research could focus on narrower experience levels to assess the professional generalizability of “smart” pen in the design tool context.

While both software in the study are GenAI-based design tools with similar functionalities, they differ in models and image editing features. Since our focus is on comparing how users construct or input prompts rather than the tools themselves, the two variants chosen enable a fair comparison, particularly as *Firefly condition* represents the state-of-art.

Both *Pen condition* tasks used the same GenAI parameters, which may not have produced optimal results in all cases. Nonetheless, the findings demonstrate the interaction style's broad applicability.

<footnote n="10">¹⁰https://helpx.adobe.com/photoshop/using/layer-basics.html</footnote>
Further research could examine varying parameter configurations for a more thorough evaluation.

Finally, FusAln exemplifies how a set of high- and low-level “smart” pens can be integrated into a holistic design workflow. Future applications could include more instances of high-level pens such as environment pens that interact with the high-level details of ambiance, or low-level pens that interact with more artistic forms. Further research could explore extending FusAln’s use to fields such as architecture, where complex aesthetic decisions—such as spatial arrangements, lighting, and material combinations are critical.

## 9 Conclusion

This paper demonstrates how pen-based visual GenAI prompt composition enables more expressive and controllable designer-GenAI interaction. We propose the design concept of “smart” pen to compose more defined visual prompts. A single “smart” pen integrates tangible pen interaction to augment user control, with decomposed visual attributes, i.e. texture, to improve the precision of GenAI visual prompts. “Smart” pens interacting with different visual attributes can be combined together to create a defined visual composition. To evaluate our design concept we developed FusAln, a GenAI visual prompt composition tool using pen-based interaction. It enables designers to create visual prompts using various visual elements, such as objects, color, and texture, to enable finer control of GenAI generation.

Our evaluation of FusAln involving 12 professional designers shows that interactively defined visual GenAI composition in FusAln provides designers with a more detailed and direct means to visually express their ideas to GenAI. FusAln demonstrates a compositional means of forming a GenAI visual prompt, from extracting design material from inspirational images, loading and applying visual details at different levels, guiding GenAI generation, to re-editing the composition. The process of deconstructing, reconstructing, and re-editing design materials results in more editable GenAI visual input and contextually relevant output, and shifts designers’ attention toward crafting visual composition rather than constantly refining input for GenAI.

From a long-term perspective, the ability to manipulate fine-grained visual elements and generate customized images may enhance the reusability of GenAI content. The research demonstrates how “smart” pen concept can enable “composition as prompts” as a GenAI interaction paradigm, and presents FusAln as an example of how GenAI can be integrated into a professional design tool context. Future research could explore novel “smart” pen as well as their combinations to encompass a customized range of visual attributes, thereby expanding the design space for customizable visual prompt design.

## Acknowledgments

We sincerely thank Yuexuan Wang for sharing assets from her fashion design project. We would also like to thank Mathieu Nancel for contributing the name “FusAln” to this project. This research was supported by the European Commission through the Human- AI-net project (grant agreement no. 952026) and by Université Paris- Saclay’s doctoral program (D23-ET046).

## References

[1] Adobe. 2024. Adobe Firefly. https://firefly.adobe.com/. [Accessed 08-02-2024].

[2] Joanna Allen. 2019. Stranglehold. Consequence Project, [Accessed 5-12-2024].

[3] Michel Beaudouin-Lafon. 2000. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (The Hague, The Netherlands) (CHI '00). Association for Computing Machinery, New York, NY, USA, 446–453. doi:10.1145/332040.332473

[4] Michel Beaudouin-Lafon and Wendy E. Mackay. 2000. Reification, Polymorphism and Reuse: Three Principles for Designing Visual Interfaces. In *Proceedings of the Working Conference on Advanced Visual Interfaces* (Palermo, Italy) (AVI '00). ACM, New York, NY, USA, 102–109. doi:10.1145/345513.345267

[5] Richard A. Bolt. 1980. "Put-that-there": Voice and gesture at the graphics interface. In *Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques* (Seattle, Washington, USA) (SIGGRAPH '80). Association for Computing Machinery, New York, NY, USA, 262–270. doi:10.1145/800250.807503

[6] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. In *Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23)*. Association for Computing Machinery, New York, NY, USA, 1–14. doi:10.1145/3586183.3606725

[7] Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. *Qualitative research in sport, exercise and health* 11, 4 (2019), 589–597.

[8] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023. InstructPix2Pix: Learning to Follow Image Editing Instructions. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE, Vancouver, 18392–18402. doi:10.1109/CVPR52729.2023.01764

[9] Miguel Bruns, Stijn Ossevoort, and Marianne Graves Petersen. 2021. Expressivity in Interaction: a Framework for Design. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems* (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 163, 13 pages. doi:10.1145/3411764.3445231

[10] Bill Buxton. 2007. *Sketching user experiences: Getting the design right and the right design*. Morgan Kaufmann, Oxford, England.

[11] Tara Capel and Margot Brereton. 2023. What is Human-Centered about Human-Centered AI? A Map of the Research Landscape. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems* (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 359, 23 pages. doi:10.1145/3544548.3580959

[12] Minsuk Chang, Stefania Druga, Alexander J. Fiannaca, Pedro Vergani, Chimmay Kulkarni, Carrie J Cai, and Michael Terry. 2023. The Prompt Artists. In *Proceedings of the 15th Conference on Creativity and Cognition (C&amp;C '23)*. Association for Computing Machinery, New York, NY, USA, 75–87. doi:10.1145/3591196.3593515

[13] Li-Yuan Chiou, Peng-Kai Hung, Rung-Huei Liang, and Chun-Teng Wang. 2023. Designing with AI: An Exploration of Co-Ideation with Image Generators. In *Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS '23)*. Association for Computing Machinery, New York, NY, USA, 1941–1954. doi:10.1145/3563657.3596001

[14] John Joon Young Chung and Eytan Adar. 2023. PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions. In *Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23)*. Association for Computing Machinery, New York, NY, USA, 1–17. doi:10.1145/3586183.3606777

[15] Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In *Perspectives on socially shared cognition*. American Psychological Association, Washington, DC, US, 127–149. doi:10.1037/10096-006

[16] Peter Dalsgaard, Kim Halskov, and Ole Sejer Iversen. 2016. Participation Gestalt: Analysing Participatory Qualities of Interaction in Public Space. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 4435–4446. doi:10.1145/2858036.2858147

[17] Hai Dang, Sven Goller, Florian Lehmann, and Daniel Buschek. 2023. Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems* (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 408, 17 pages. doi:10.1145/3544548.3580969

[18] Richard Lee Davis, Thiemo Wambsganss, Wei Jiang, Kevin Gonyop Kim, Tanja Käser, and Pierre Dillenbourg. 2024. Fashioning Creative Expertise with Generative AI: Graphical Interfaces for Design Space Exploration Better Support Ideation Than Text Prompts. In *Proceedings of the CHI Conference on Human Factors in Computing Systems* (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 167, 26 pages. doi:10.1145/3613904.3642908

[19] Andreas Rene Fender, Thomas Roberts, Tiffany Luong, and Christian Holz. 2023. InfinitePaint: Painting in Virtual Reality with Passive Haptics Using Wet Brushes and a Physical Proxy Canvas. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems* (Hamburg, Germany) (CHI '23). Association for
Computing Machinery, New York, NY, USA, Article 21, 13 pages. doi:10.1145/3544548.3580927

[20] Jérémy Garcia, Theophanis Tsandilas, Carlos Agon, and Wendy E. Mackay. 2014. Structured observation with polyphonic: a multifaceted tool for studying music composition. In *Proceedings of the 2014 Conference on Designing Interactive Systems* (Vancouver, BC, Canada) (DIS '14). Association for Computing Machinery, New York, NY, USA, 199–208. doi:10.1145/2598510.2598512

[21] William Gaver. 2011. Making spaces: how design workbooks work. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11)*. Association for Computing Machinery, New York, NY, USA, 1551–1560. doi:10.1145/1978942.1979169

[22] Brett A. Halperin and Stephanie M. Lukin. 2024. Artificial Dreams: Surreal Visual Storytelling as Inquiry Into AI 'Hallucination'. In *Proceedings of the 2024 ACM Designing Interactive Systems Conference (Copenhagen, Denmark) (DIS '24)*. Association for Computing Machinery, New York, NY, USA, 619–637. doi:10.1145/3643834.3660685

[23] Björn Hartmann, Scott R. Klemmer, Michael Bernstein, Leith Abdulla, Brandon Burr, Avi Robinson-Mosher, and Jennifer Gee. 2006. Reflective physical prototyping through integrated design, test, and analysis. In *Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology* (Montreux, Switzerland) (UIST '06). Association for Computing Machinery, New York, NY, USA, 299–308. doi:10.1145/1166253.1166300

[24] Ken Hinckley and Daniel Wigdor. 2012. *Input Technologies and Techniques (2012).* CRC Press, Boca Raton, Florida, United States, Chapter 6, 95–132. https://www.microsoft.com/en-us/research/publication/input-technologies-techniques/

[25] Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen + touch = new tools. In *Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology* (New York, New York, USA) (UIST '10). Association for Computing Machinery, New York, NY, USA, 27–36. doi:10.1145/1866029.1866036

[26] Anita Hu, Nishkrit Desai, Hassan Abu Alhaja, Seung Wook Kim, and Maria Shugrina. 2024. Diffusion Texture Painting. In *ACM SIGGRAPH 2024 Conference Papers* (Denver, CO, USA) (SIGGRAPH '24). Association for Computing Machinery, New York, NY, USA, Article 5, 12 pages. doi:10.1145/3641519.3657458

[27] huggingface. 2024. Stable Diffusion pipelines. https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/overview. [Accessed 10-09-2024]

[28] Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. PromptMaker: Prompt-based Prototyping with Large Language Models. In *Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems* (New Orleans, LA, USA) (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 35, 8 pages. doi:10.1145/3491101.3503564

[29] Yu Jin, Juhyeok Yoon, James Andrew Self, and Kyungho Lee. 2024. AI as a Catalyst for Creativity: Exploring the Use of Generative Approach in Fashion Design for Improving Their Inspiration. In *DRS2024: Boston (DRS2024).* Design Research Society, Boston, USA. doi:10.21606/drs.2024.680

[30] Rubaiai Habib Kazi, Takeo Igarashi, Shengdong Zhao, and Richard Davis. 2012. Vignette: interactive texture design and manipulation with freeform gestures for pen-and-ink illustration. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 1727–1736. doi:10.1145/2207676.2208302

[31] Janin KOCH, Magda LÁSZLÓ, Andrés LUCERO, and Antti OULAVSINTA. 2018. Surfing for Inspiration: digital inspirational material in design practice. In *DRS2018: Catalyst (DRS2018, Vol. 3).* Design Research Society, Limerick, Ireland. doi:10.21606/drs.2018.352

[32] Janin Koch, Nicolas Taffin, Andrés Lucero, and Wendy E. Mackay. 2020. SemanticCollage: Enriching Digital Mood Board Design with Semantic Labels. In *Proceedings of the 2020 ACM Designing Interactive Systems Conference (DIS '20).* Association for Computing Machinery, New York, NY, USA, 407–418. doi:10.1145/3357236.3395494

[33] J.A. Landay and B.A. Myers. 2001. Sketching interfaces: toward more human interface design. *Computer* 34, 3 (2001), 56–64. doi:10.1109/2.910894

[34] Jie Li, Hancheng Cao, Laura Lin, Youyang Hou, Ruihao Zhu, and Abdallah El Ali. 2024. User Experience Design Professionals' Perceptions of Generative Artificial Intelligence. In *Proceedings of the CHI Conference on Human Factors in Computing Systems* (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 381, 18 pages. doi:10.1145/3613904.3642114

[35] Zheng Lin, Zhao Zhang, Kang-Rui Zhang, Bo Ren, and Ming-Ming Cheng. 2022. Interactive Style Transfer: All is Your Palette. *arXiv:2203.13470* [cs.CV] https://arxiv.org/abs/2203.13470

[36] Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22).* Association for Computing Machinery, New York, NY, USA, 1–23. doi:10.1145/3491102.3501825

[37] Yuwen Lu, Chengzhi Zhang, Iris Zhang, and Toby Jia-Jun Li. 2022. Bridging the Gap Between UX Practitioners&x#2019; Work Practices and AI-Enabled Design Support Tools. In *Extended Abstracts of the 2022 CHI Conference on Human Factors*

in *Computing Systems (CHI EA '22).* Association for Computing Machinery, New York, NY, USA, 1–7. doi:10.1145/3491101.3519809

[38] Wendy Mackay and Joanna McGrenere. 2025. Comparative Structured Observation. *ACM Trans. Comput.-Hum. Interact.* (Jan. 2025). doi:10.1145/3711838 Just Accepted.

[39] Martin Martinček. 1964–1967. Zatva I. https://www.webumenia.sk/en/dielo/SVK: SNGUP-DK_2881-a. storočie, 2. polovica, Slovenská národná galéria, SNG, UP-DK 2881/a, [Accessed 5-12-2024].

[40] Damien Masson, Sylvain Malacria, Géry Casiez, and Daniel Vogel. 2024. Direct-GPT: A Direct Manipulation Interface to Interact with Large Language Models. In *Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24).* Association for Computing Machinery, New York, NY, USA, Article 975, 16 pages. doi:10.1145/3613904.3642462

[41] Yiqun Mei, He Zhang, Xuaner Zhang, Jianming Zhang, Zhixin Shu, Yilin Wang, Zijun Wei, Shi Yan, HyunJoon Jung, and Vishal M. Patel. 2023. LightPainter: Interactive Portrait Relighting with Freehand Scribble. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).* IEEE, Vancouver, 195–205. doi:10.1109/CVPR5279.2023.00027

[42] Midjourney. 2024. Midjourney. https://www.midjourney.com/. [Accessed 26-01-2024].

[43] Openai. 2024. DALL-E 3 – openai.com. https://openai.com/dall-e-3. [Accessed 26-01-2024].

[44] Srishti Palani and Gonzalo Ramos. 2024. Evolving Roles and Workflows of Creative Practitioners in the Age of Generative AI. In *Proceedings of the 16th Conference on Creativity & Cognition* (Chicago, IL, USA) (C&C '24). Association for Computing Machinery, New York, NY, USA, 170–184. doi:10.1145/3635636.3656190

[45] Xiaohan Peng, Janin Koch, and Wendy E. Mackay. 2024. DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI. In *Proceedings of the 2024 ACM Designing Interactive Systems Conference* (IT University of Copenhagen, Denmark) (DIS '24). Association for Computing Machinery, New York, NY, USA, 804–818. doi:10.1145/3643834.3661588

[46] Savvas Petridis, Michael Terry, and Carrie J Cai. 2024. PromptInformer: How Tightly Coupling AI and UI Design Impacts Designers' Workflows. In *Proceedings of the 2024 ACM Designing Interactive Systems Conference (Copenhagen, Denmark) (DIS '24).* Association for Computing Machinery, New York, NY, USA, 743–756. doi:10.1145/3643834.3661613

[47] Ken Pfeuffer, Ken Hinckley, Michel Pahud, and Bill Buxton. 2017. Thumb + Pen Interaction on Tablets. In *Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems* (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 3254–3266. doi:10.1145/3025453.3025567

[48] Tiziano Portenier, Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker. 2018. Faceshop: deep sketch-based face image editing. *ACM Trans. Graph.* 37, 4, Article 99 (July 2018), 13 pages. doi:10.1145/3197517.3201393

[49] Recraft. 2024. Infinite AI Artboard - Recraft — recraft.ai. https://www.recraft.ai/. [Accessed 25-11-2024].

[50] Alexander Rodchenko. 1928. Parade of the Dynamo Sports Club. https://www.wikiart.org/en/alexander-rodchenko/parade-of-the-dynamo-sports-club-1928-1928. [Accessed 5-12-2024].

[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).* IEEE, New Orleans, 10684–10695.

[52] Karl Toby Rosenberg, Rubaiai Habib Kazi, Li-Yi Wei, Haijun Xia, and Ken Perlin. 2024. DrawTalking: Towards Building Interactive Worlds by Sketching and Speaking. In *Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA '24).* Association for Computing Machinery, New York, NY, USA, Article 113, 8 pages. doi:10.1145/3613905.3651089

[53] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).* IEEE, Vancouver, 22500–22510. doi:10.1109/CVPR52729.2023.02155

[54] Kimiko Ryokai, Stefan Marti, and Hiroshi Ishii. 2004. I/O brush: drawing with everyday objects as ink. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI '04).* Association for Computing Machinery, New York, NY, USA, 303–310. doi:10.1145/985692.985731

[55] Téo Sanchez. 2023. Examining the Text-to-Image Community of Practice: Why and How Do People Prompt Generative AIs?. In *Proceedings of the 15th Conference on Creativity and Cognition (C&amp;C '23).* Association for Computing Machinery, New York, NY, USA, 43–61. doi:10.1145/3591196.3593051

[56] Donald Schön. 2017. *The Reflective Practitioner: How Professionals Think in Action.* Taylor & Francis, London, United Kingdom. https://www.taylorfrancis.com/books/mono/10.4324/9781315237473/reflective-practitioner-donald-sch%C3%B6n
[57] Donald Schön and John Bennett. 1996. *Reflective conversation with materials*. Association for Computing Machinery, New York, NY, USA, 171–189. https://doi.org/10.1145/229868.230044

[58] Stable Diffusion. 2022. Stability AI. https://stability.ai/stable-diffusion. [Online; accessed 19-April-2023].

[59] Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In *Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24)*. Association for Computing Machinery, New York, NY, USA, Article 644, 26 pages. doi:10.1145/3613904.3642400

[60] Ivan E. Sutherland. 1963. Sketchpad: a man-machine graphical communication system. In *Proceedings of the May 21-23, 1963, Spring Joint Computer Conference (Detroit, Michigan) (AFIPS '63 (Spring))*. Association for Computing Machinery, New York, NY, USA, 329–346. doi:10.1145/1461551.1461591

[61] Ryo Suzuki, Rubaiat Habib Kazi, Li-yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In *Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20)*. Association for Computing Machinery, New York, NY, USA, 166–181. doi:10.1145/3379337.3415892

[62] Macy Takaffoli, Sijia Li, and Ville Mäkelä. 2024. Generative AI in User Experience Design and Research: How Do UX Practitioners, Teams, and Companies Use GenAI in Industry?. In *Proceedings of the 2024 ACM Designing Interactive Systems Conference (Copenhagen, Denmark) (DIS '24)*. Association for Computing Machinery, New York, NY, USA, 1579–1593. doi:10.1145/3643834.3660720

[63] Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The Metacognitive Demands and Opportunities of Generative AI. In *Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24)*. Association for Computing Machinery, New York, NY, USA, Article 680, 24 pages. doi:10.1145/3613904.3642902

[64] Jakob Tholander and Martin Jonsson. 2023. Design Ideation with AI - Sketching, Thinking and Talking with Generative Machine Learning Models. In *Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS '23)*. Association for Computing Machinery, New York, NY, USA, 1930–1940. doi:10.1145/3563657.3596014

[65] Katja Thoring, Sebastian Huettemann, and Holand M. Mueller. 2023. THE AUGMENTED DESIGNER: A RESEARCH AGENDA FOR GENERATIVE AI-ENABLED DESIGN. *Proceedings of the Design Society* 3 (2023), 3345–3354. doi:10.1017/pds.2023.335

[66] Martin Tricaud and Michel Beaudouin-Lafon. 2024. Revisiting creative behaviour as an epistemic process: lessons from 12 computational artists and designers. In *Proceedings of the 35th Australian Computer-Human Interaction Conference (Wellington, New Zealand) (OzCHI '23)*. Association for Computing Machinery, New York, NY, USA, 175–190. doi:10.1145/3638380.3638395

[67] Theophanis Tsandilas, Catherine Letondal, and Wendy E. Mackay. 2009. Musink: composing music through augmented drawing. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI '09)*. Association for Computing Machinery, New York, NY, USA, 819–828. doi:10.1145/1518701.1518827

[68] Unsplash. [n. d.]. Unsplash Image API | Free HD Photo API — api.unsplash.com. https://api.unsplash.com/. [Accessed 29-01-2024].

[69] Chaitat Utintu, Pinaki Nath Chowdhury, Aneeshan Sain, Subhadeep Koley, Ayan Kumar Bhunia, and Yi-Zhe Song. 2024. SketchDeco: Decorating B&W Sketches with Colour. arXiv:2405.18716 [cs.CV] https://arxiv.org/abs/2405.18716

[70] Severi Uusitalo, Antti Salovaara, Tero Jokela, and Marja Salmimaa. 2024. "Clay to Play With": Generative AI Tools in UX and Industrial Design Practice. In *Proceedings of the 2024 ACM Designing Interactive Systems Conference (Copenhagen, Denmark) (DIS '24)*. Association for Computing Machinery, New York, NY, USA, 1566–1578. doi:10.1145/3643834.3661624

[71] Veera Vimpari, Annakaisa Kultima, Perttu Hämäläinen, and Christian Guckelsberger. 2023. "An Adapt-or-Die Type of Situation": Perception, Adoption, and Use of Text-to-Image-Generation AI by Game Industry Professionals. *Proc. ACM Hum.-Comput. Interact.* 7, CHI PLAY, Article 379 (oct 2023), 34 pages. doi:10.1145/3611025

[72] Yael Vinker, Andrey Voynov, Daniel Cohen-Or, and Ariel Shamir. 2023. Concept Decomposition for Visual Exploration and Inspiration. *ACM Trans. Graph.* 42, 6, Article 241 (dec 2023), 13 pages. doi:10.1145/3618315

[73] Daniel Vogel and Géry Casiez. 2011. Conté: multimodal input inspired by an artist's crayon. In *Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST '11)*. Association for Computing Machinery, New York, NY, USA, 357–366. doi:10.1145/2047196.2047242

[74] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23)*. Association for Computing Machinery, New York, NY, USA, 1–21. doi:10.1145/3544548.3581388

[75] Chengzhi Zhang, Weijie Wang, Paul Pangaro, Nikolas Martelaro, and Daragh Byrne. 2023. Generative Image AI Using Design Sketches as input: Opportunities and Challenges. In *Proceedings of the 15th Conference on Creativity and Cognition (Virtual Event, USA) (C&C '23)*. Association for Computing Machinery, New York, NY, USA, 254–261. doi:10.1145/3591196.3596820

[76] Lvmin Zhang, Jinyue Jiang, Yi Ji, and Chunping Liu. 2021. SmartShadow: Artistic Shadow Drawing Tool for Line Drawings. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*. IEEE, virtual, 5391–5400.

[77] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*. IEEE, Paris, 3813–3824. doi:10.1109/ICCV51070.2023.00355

[78] Tongyu Zhou, Connie Liu, Joshua Kong Yang, and Jeff Huang. 2023. filtered.ink: Creating Dynamic Illustrations with SVG Filters. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23)*. Association for Computing Machinery, New York, NY, USA, Article 129, 15 pages. doi:10.1145/3544548.3581051

[79] Alexander "Freddie" Holliman and Ross Brisco. 2024. Is pen-to-paper the buggy whip of design? Assessing the use of ai tools for design sketching. In *DRS2024: Boston (DRS2024)*. Design Research Society, Boston. doi:10.21606/drs.2024.289
# A APPENDIX

## A.1 Stable Diffusion and ControlNet Implementation

<figure><img src="image_30.png" /><figcaption>Figure 13: Image-First Generation Technical Implementation. This process happens after users click on “Guided Generate” on the interface. The implementation uses user composition which is deconstructed as different input images for image-to-image pipeline input and different ControlNet units (while text prompt is optional). Specifically tile_resample module is used to preserve the color, canny and ‘softedge anyline’ are used to preserve texture. The overall purpose is to prioritize image input and preserve the visual features of applied strokes in the diffusion process.</figcaption></figure>
<figure><img src="image_31.png" /><figcaption>Figure 14: *Text-First Generation* Technical Implementation. This process happens after users click on “Merge” on the interface. The implementation uses text prompt as the text-to-image pipeline input, and deconstructed user composition as different input images for different ControlNet units. The overall purpose is to prioritize text input but preserve overall visual features in the diffusion process.</figcaption></figure>
## A.2 Alternative Pen Combination Use Case

<figure><img src="image_32.png" /><figcaption>Figure 15: The figure illustrates an alternative “smart” pen use case, showcasing how iterative and mixed-fidelity visual prompts help create content with a consistent style. The visual prompts (1.a) and (2.a) are composed with a combination of object pens and basic pens. The generation mode is Text-First Generation with text prompt “a flat design of planets, blue tone”. With style lock enabled and a hand-drawn music note added in (2.a), the generated result (2.b) differs from (1.b) primarily on the stylized music note. The visual prompt can be easily re-arranged due to the compositional nature of such interaction. Different “smart” pen combinations can enable diverse iterative workflows with flexible technical configurations.</figcaption></figure>