Fantastic Copyrighted Beasts and How (Not) to Generate Them

## Abstract

Abstract Recent studies show that image and video generation models can be prompted to reproduce copyrighted content from their training data, raising serious legal concerns around copyright infringement. Copyrighted characters, in particular, pose a difficult challenge for image generation services, with at least one lawsuit already awarding damages based on the generation of these characters. Yet, little research has empirically examined this issue.
We conduct a systematic evaluation to fill this gap. First, we build CopyCat , an evaluation suite consisting of diverse copyrighted characters and a novel evaluation pipeline. Our evaluation considers both the detection of similarity to copyrighted characters and generated image’s consistency with user input. Our evaluation systematically shows that both image and video generation models can still generate characters even if characters’ names are not explicitly mentioned in the prompt, sometimes with only two generic keywords (e.g., prompting with “videogame, plumber” consistently generates Nintendo’s Mario character). We then introduce techniques to semi-automatically identify such keywords or descriptions that trigger character generation. Using our evaluation suite, we study runtime mitigation strategies, including both existing methods and new strategies we propose. Our findings reveal that commonly employed strategies, such as prompt rewriting in the DALL·E system, are not sufficient as standalone guardrails. These strategies must be coupled with other approaches, like negative prompting, to effectively reduce the unintended generation of copyrighted characters. Our work provides empirical grounding to the discussion of copyright mitigation strategies and offers actionable insights for model deployers actively implementing them.

## 1 Introduction

State-of-the-art image and video generation models demonstrate a remarkable ability for generating high-quality visual content based on free-form user inputs (Rombach et al., ; Betker et al., ; Chen et al., ; Li et al., ; Blattmann et al., ; Esser et al., ).
However, recent research has shown that generative models, including image and video generation models, are susceptible to memorizing and generating entire datapoints or concepts from their training data (Somepalli et al., ; Carlini et al., , ).
Since some training data originates from copyrighted materials (Carlini et al., ; Kumari et al., ), regurgitation of such content may lead to legal intellectual property liability for users and model deployers who further make use of the generated content.
In particular, this liability may stem not only from verbatim generation of training data points, but generation of some copyrightable repeating motifs highly similar to those from the training data. A notable example of such copyrightable data is copyrighted characters. Several legal scholars have highlighted that these characters will pose a particularly difficult, and under-explored, challenge for image or video generation services (Sag, ; Henderson et al., ; Lee et al., ). At least one lawsuit in China has already resulted in liability for an image generation service that generated the copyrighted character, Ultraman (Shimbun, ).

In this work, we provide a study focused on a subset of copyrighted content: *copyrighted characters*, such as popular Intellectual Property (IP) from Disney, Nintendo, and Dreamworks.(^1^11 We study the phenomenon of copyrighted characters generation from an image-based reasoning and natural language processing perspective. Artist compensation and other downstream policy issues are also important, but the formal discussion on these topics are beyond the scope of this work. For additional legal background on the distinction between copyrighted characters and other types of memorization studies, see Appendix A.) Given the legal risks involved and the need to respect individuals’ IP rights to copyrighted characters, commercial services like DALL·E have begun deploying interventions that prevent generating copyrighted characters.
For example, DALL·E rejects user requests that directly name copyrighted characters (e.g., “Mario”). It also rewrites user’s prompts that target specific characters into more generic descriptions (OpenAI, ).
However, such interventions have never been systematically assessed.
Our work fills this gap by making the following key contributions.

First, we build CopyCat , an evaluation suite with a diverse set of popular copyrighted characters and a novel evaluation pipeline (§2), to better ground our investigation and evaluation. Our evaluation captures the inherent trade-off between elimination of similar output and consistency with user intent. We then evaluate five image generation models: Playground v2.5 (Li et al., ), Stable Diffusion XL (Podell et al., ), PixArt-$\alpha$ (Chen et al., ), DeepFloyd IF (StabilityAI, ), and DALL·E 3 (Betker et al., ), as well as one video generation model, VideoFusion (Luo et al., ).

Figure: (a) Target copyrighted character: Mario
Refer to caption: x1.png

Second, we demonstrate that as little as two generic keywords associated with a character are enough to generate their image, without mentioning their name (Figure 1). We categorize text that triggers copyrighted character generation into different modes, depending on whether the text explicitly contains character name (Character Name Anchoring), or generic keywords and descriptions only (Indirect Anchoring). We use the term ‘anchor’ to refer to these different sources of generation. Similar to our indirect anchoring concept, some previous work have observed that to obtain certain objects or concepts in generated image, the prompt does not necessarily need to include its name (Zhang et al., ; Chin et al., ; Lee et al., ). Indirect anchoring is particularly crucial for both model deployers and model users: even a non-malicious user could inadvertently generate copyrighted characters when using seemingly innocuous prompts, leading to potential legal liability for the model deployer as well as any unsuspecting user attempting to monetize the generated image.

We systematically evaluate these two modes of generation using CopyCat. We propose a two-stage method to identify indirect anchors: We first use a language model to generate candidate keywords and descriptions that might evoke the likeliness of a copyrighted character. Then we rerank based on embedding space distance to, and co-occurrence with, character names in popular training corpora (§3). Our findings suggest a mismatch between the level of generality in the prompt versus the specificity of the output: For example, $40\%$ of our target characters can be recreated with only 5 automatically selected keywords using our method (§5.1), and some with as few as 2 keywords. These indirect anchors also extend to video generation models and product systems with built-in safeguards, like DALL·E.

Third, we find that existing mitigations are not fully effective and suggest new strategies (§5.2). In this paper we focus on runtime approaches only, assuming that models cannot be modified to remove copyrighted characters. We explore practical solutions that model deployers can incorporate into a production system. We find that prompt rewriting—an important technical piece for copyright protection adopted by model deployers—is far from perfect (§5.2). Applying this intervention alone can only reduce the number of generated characters by half compared to no intervention—and many of the failed rewritten prompts contain indirect anchoring keywords we identified. Instead, combining this strategy with negative prompting (i.e., steering models away from concepts like “red hat”, a defining feature of Mario, during inference) significantly boost mitigation effectiveness. Applying this combination reduces the generation of over 80% of copyrighted characters compared to no mitigation. We also demonstrate that a combination of mitigation strategies can strike a balance between effectively eliminating similar outputs and adhering to user intent.

We summarize the key takeaways for users and model deployers as follows:

- •
We call for more awareness of indirect anchoring, where models can generate copyrighted characters without explicitly mentioning the character’s name. For deployers, this may bypass safeguards that rely on direct name detection. For users, such prompts can result in the generation of characters substantially similar to copyrighted ones, leading to potential liability even if they did not intend to generate them.
- •
For model deployers who adopt mitigation strategies and intend to prevent the generation of copyrighted characters, we recommend investing in techniques beyond prompt rewriting. Our work suggests a relatively simple set of strategies, such as combining prompt rewriting and negative prompts, to make progress on this issue.

## 2 CopyCat : An Evaluation Suite for Copyrighted Characters Detection

To systematically study the anchors leading to the generation of copyrighted characters and how to prevent it, we curate a list of copyrighted characters
(§2.1) and develop a novel evaluation pipeline (§2.2) that we call CopyCat (Copyrighted CharAcTers).

### 2.1 A Curation of Copyrighted Characters

We first curate a character list comprising a diverse selection of copyrighted characters to concretely study the effects of character name and indirect anchoring. We source copyrighted characters from popular studios and franchises, as they are more likely to have been present in the training process of image and video generation models. These characters represent a diverse distribution from superhero movies (e.g., Batman, Iron Man, Hulk), animations (e.g., Lightning McQueen, Monkey D. Luffy, Elsa), and video games (e.g., Mario, Pikachu, Link). In addition to U.S. studios like Disney and DreamWorks, we also include international ones like Nintendo and Shogakukan. In total, our collection includes 50 diverse popular copyrighted characters from 18 different studios and subsidiaries. The full list of characters in CopyCat can be found in Appendix B. Throughout this paper, we refer to this curated list of characters as $\mathcal{D}$.

### 2.2 Evaluation Metrics

In the context of copyright and generative models, especially when any intervention strategy is applied, there is an inherent trade-off between two key factors: 1) dissimilarity from copyrighted entities (e.g., avoiding the generation of specific copyrighted characters like Mario) and 2) consistency with user intent (e.g., if the user requests a plumber, still generating a plumber). Both factors warrant quantification, and an effective intervention method should strike a balance between these two aspects.
Specifically, for a copyrighted character $C$ and a corresponding generated image $\mathcal{I}=f_{p,m}(C)$, where $f(\cdot)$ is the generation model, $p$ is the given prompt, and $m$ is mitigation (if any), we calculate the following two metrics:

Detected similarity to copyrighted characters.
In general, the more similar a generated image is to existing copyrighted characters, the higher the likelihood of potential legal issues like copyright infringement. In our evaluation, we use GPT-4V (Achiam et al., ) to detect whether $C$ is present in $\mathcal{I}$ (see Appendix C).(^2^22We use GPT-4V detection for research evaluation purposes, but we are not suggesting that this is a reliable detector for legal judgement of copyright infringement, as such judgments typically require case-by-case analysis.) The detector outputs $d(C,\mathcal{I})\in\{0,1\}$, indicating the presence (1) or absence (0) of character $C$ in image $\mathcal{I}$. We then define the metric $\mathsf{DETECT}$ for a model $f$, prompting method $p$, mitigation $m$ as:

$$ $\mathsf{DETECT}(f,p,m)=\sum_{C\in\mathcal{D}}d(C,f_{p,m}(C)),$ $$

which sums the binary detection scores across the character list $\mathcal{D}$. A lower $\mathsf{DETECT}$ score indicates that fewer copyrighted characters were generated. We include human evaluation of copyrighted character detection in Appendix D.

Consistency with user intent. On the other hand, if a model always outputs a random image or rejects the user’s request, it can achieve near-perfect elimination of similar output to copyrighted characters but would fail to fulfill the user’s intent. Therefore, we quantify the consistency between the generation and the user’s intent as a potential metric for user satisfaction: we test whether the key characteristics in user prompt can be found in the generated image. Since we assume that users are prompting the models to obtain certain characters, we can generate the ground-truth key characteristics using the list of target characters. For each copyrighted character $C$ in CopyCat’s curated list of characters, we ask GPT-4 to automatically identify its main general characteristics $s(C)$, which we manually verify and adjust if necessary (e.g., “cartoon mouse” for Mickey Mouse). We then use VQAScore (Lin et al., ) to measure the consistency between image $\mathcal{I}$ and characteristics $s(C)$, defined as $c(s(C),\mathcal{I})=\mathbb{P}(\texttt{``Yes''}|\mathcal{I},``\texttt{Does
this figure show}~{}s(C)\texttt{? Please answer yes or no.}")$. For example, we calculate $\mathbb{P}(\texttt{``Yes''}|\mathcal{I},``\texttt{Does this figure show a
cartoon mouse}\texttt{? Please answer yes or no.}")$ when the character is Mickey Mouse. The consistency metric $\mathsf{CONS}$ for a model $f$, input prompt $p$, and intervention $m$ is the average consistency score across the curated list of characters in CopyCat’s character list:

$$ $\mathsf{CONS}(f,p,m)=\frac{1}{|\mathcal{D}|}\sum_{C\in\mathcal{D}}c(s(C),f_{p, m}(C)).$ $$

A higher $\mathsf{CONS}$ indicates better consistency with user intent (see Section C.4 for examples). The underlying assumption is that as long as the main general characteristic (e.g., a “cartoon mouse”) is present in the generation, the user may still be more satisfied with the result despite some alterations. We note that this only captures some aspects of consistency with user requests and that future work may improve this metric.

We omit $(f,p,m)$ for $\mathsf{DETECT}$ and $\mathsf{CONS}$ if they are clear from the context. When studying the effect of Character Name and Indirect Anchoring, our evaluation only relies on $\mathsf{DETECT}$, since this task’s focus is whether specific characters can be generated. The trade-off becomes more relevant when involving intervention, hence both $\mathsf{DETECT}$ and $\mathsf{CONS}$ are used for evaluating these interventions: An effective intervention strategy $m$ should aim to minimize $\mathsf{DETECT}$ while maximizing $\mathsf{CONS}$.
CopyCat provides a useful framework for understanding different modes of copyrighted character generation and the effectiveness of mitigation strategies, which we will discuss in the following sections.

## 3 Identifying Indirect Anchors

Not surprisingly, prompting with “Mario” would likely generate this Nintendo character. We refer to this type of generation as *Character Name Anchoring*. However, if users ask for a generic “video game plumber” they will also receive the iconic character’s likeness from most models (Figure 1(a)). We refer to this mode of generation, using keywords or descriptions without the character’s name, as *Indirect Anchoring*. To reduce copyright violation risk coming from the Indirect Anchoring mode in particular, we would like to first generate a set of indirect anchors for a certain character. To systematically identify such anchoring keywords and understand how they become effective triggers, we further use a two-stage approach involving generation and reranking.
We generate a set of candidate descriptions and keywords related to the given character and then rerank the candidates to select the most likely anchors that can effectively trigger the desired character generation.

Generation. First, we use GPT-4 to generate a set of candidate descriptions and keywords pertaining to the visual appearance of the characters in CopyCat, using the prompting template in C.(^3^33The textual descriptions are around 60 words in length. This length limit provides maximal descriptive information while keeping under the 77 token limit for stable diffusion models (Urbanek et al., ).)

Ranking. Given the generated candidates, we use different ranking methods to investigate what prompts most likely trigger a character generation, even when the character’s name is not present.

Figure: Algorithm 1 EmbeddingSim Ranking

- •
EmbeddingSim: We leverage embedding space similarity to rerank and obtain the top $k$ indirect anchor candidates, which can be descriptions or keywords. The algorithm is illustrated in Algorithm 1, and is applicable for both descriptions and keywords. Specifically, for each character name and candidate word, we use the text encoder of the image generation model to calculate their textual embeddings. We then rank candidate keywords by their embedding’s cosine similarity with the character name embedding, computed as the averaged token embedding at the last hidden layer. We hypothesize that keywords with embeddings more similar to the character’s name may incline the model to generate that character.
- •
Co-Occurrence: For keywords, we can also rank by their *co-occurrence with the character’s name* in popular training corpora (see Algorithm 2). We hypothesize that models learn to associate characters with words commonly found in their descriptions or references, turning these seemingly generic adjectives into anchoring words for specific characters. We examine common training corpora, including captions from image-captioning datasets: LAION-2B (Schuhmann et al., )), as well as text-only datasets (C4 (Raffel et al., ), OpenWebText (Radford et al., ), and The Pile (Gao et al., ). We follow the indexing and search procedure discussed in Elazar et al. () to rank and select keywords.
- •
LM-Ranked: For keywords, we also obtain an inherently LM-ranked list as a baseline for comparison. This is achieved by obtaining the top $k$ keywords associated with certain characters using greedy decoding, based on the prompt template provided in Appendix C. Note that the LM may generate words *not* present in the candidate list, but we maintain $k$ as the same for a fair comparison between LM-Ranked, EmbeddingSim, and Co-Occurrence.

While we focus on keywords re-ranking in later parts of this paper as they provide valuable information for the design of mitigation strategy, we also include relevant analysis on descriptions in §F.3. As a concrete example can be seen in Figure 6 in Section C.5.

## 4 Mitigation Strategies

We first discuss known mitigation strategies adopted by current producion-level image generation services. We then propose new mitigation strategies, especially leveraging negative prompts, that can improve upon current implementation.

Prompt rewriting is an existing mitigation used in production-level systems such as DALL·E. Specifically, the DALL·E interface contains a prompt-rewriting step that first processes the user’s text input into a format that DALL·E can use to generate images and comply with OpenAI’s policies, such as avoiding copyrighted content. In order to simulate the prompt-rewriting pipeline, we query GPT-4 with the DALL·E’s full system prompt (see full template in §C.6) and the keywords or descriptions to be rewritten.

Prompt-rewriting changes short prompts (e.g., one-word character name) most significantly, transforming them into a longer descriptive prompt that adds modification in order to create a more generic output. At a high level, such intervention is compromising faithfulness of certain visual aspects for copyright protection. The exact features to be prioritized or de-prioritized can be customized in the rewriting instructions.

Negative prompts are often used in deployed diffusion model deployments (Playground AI, ) to allow users to exclude undesired concepts or elements from the generated output. Negative prompts are incorporated through classifier-free guidance during the decoding process (Ho & Salimans, ). For example, the official prompt guide from Playground suggests using phrases like “ugly, deformed hands” to discourage unwanted aesthetics.(^4^44[https://playground.com/prompt-guide/negative-prompts](https://playground.com/prompt-guide/negative-prompts)) Despite their utility, negative prompts are currently under-studied as a means to exclude specific copyrighted elements from generated outputs.

We test negative prompts as a mitigation strategy based on the important anchoring keywords selected via our methods in §3. Specifically, negative prompts are “Copyrighted character” or specific target’s name paired with one of the following options: EmbeddingSim Co-Occurrence LM-Ranked options: 1) $k$ LM-Ranked keywords; 2) $k$ EmbeddingSim keywords; 3) $k$ Co-Occurrence keywords; 4) $k$ EmbeddingSim + $k$ Co-Occurrence keywords.

In addition, we propose to further combine prompt rewriting and negative prompts to strengthen the copyright protection.

## 5 Experiments and Discussion

This section presents our empirical results, where we seek to answer the following two key questions:

- •
Which method introduced in §3 most effectively identifies indirect anchors (§5.1)?
- •
How effective are the mitigation strategies discussed in §4, namely prompt rewriting and negative prompting, in reducing the generation of copyrighted characters (§5.2)?

#### Experimental setup.

To ensure a clear understanding and better control over model behaviors, our evaluation primarily focuses on four state-of-the-art open-source image generation models: Playground v2.5 (Li et al., ), Stable Diffusion XL (SDXL) (Podell et al., ), PixArt-$\alpha$ (Chen et al., ), and DeepFloyd IF (StabilityAI, ), DALL·E 3 Betker et al. (), as well as one video generation model, VideoFusion (Luo et al., ).(^5^55Video generation pipelines can be broadly categorized into the two types: 1) image generation model followed by an image-to-video model, and 2) a direct text-to-video pipeline. For models in the first category (eg. Stable Video Diffusion (Blattmann et al., )), our findings on image generation models are also applicable. Therefore, we focus our video experiments on models of the second type, e.g. VideoFusion (Luo et al., ).) The configuration details for each model used in our experiments can be found in Section C.2.

Our main analysis focuses on the Playground v2.5 due to its superior generation quality. We also report results for other models in Section F.5.

### 5.1 Identifying Prompts That Generate Copyrighted Characters

First, not too surprisingly, we have verified that when using character names, ~60% of tested characters can be generated.
(^6^66However, we find that models are not robust to misspellings of character names and generally do not result in generation of characters even with minor misspellings, see Section F.4.)
For the remainder of this section, we focus on indirect anchoring, where the prompt does not explicitly contain the character’s name. We examine the effect of two types of indirect anchors: textual descriptions and keywords, as well as how to automatically discover them (§3), by checking $\mathsf{DETECT}$, the number of detected copyrighted characters in the generation.

Figure: Figure 2: Number of characters detected using different top keywords ranked by various methods on (a) image generation and (b) video generation models. Ranking keywords based on their co-occurrence with the character’s name in the LAION corpus is the most effective and could generate more characters than using a 60-word description when only 20 keywords are used.
Refer to caption: x3.png

Figure: (a) Prompt: Character’s name
Refer to caption: x4.png

60-word descriptions lead to the generation of ~48% characters. As described in §3, the first type of indirect prompt uses around 60 words to describe a character’s visual appearance. Despite omitting character names, these descriptions often lead to successful character generation, as shown in Figure 2.
Furthermore, prompts with higher embedding similarity to a character’s name tend to generate that character more reliably. Among 100 randomly generated 60-word descriptions per character, the top-ranked description by embedding similarity generates 24 characters successfully, versus only 16 for the bottom-ranked (see Section F.3).

We note that a concurrent study by Kim et al. () also examine keywords potentially important for image generation, but only include the character name along with the associated movie or TV program as keywords. They also show that LLM-optimized descriptions can generate images similar to copyrighted characters on proprietary models such as ChatGPT, Copilot, and Gemini. However, their optimized prompts do not explicitly exclude the characters’ names.

A few keywords, especially those with most frequent co-occurrence with character names in LAION, also easily generate copyrighted characters. We examine the effectiveness of keywords with top co-occurrence frequency with the copyrighted characters’ names (§3) and visualize results in Figure 2. For most selections, we find that keywords chosen from LAION are more effective than using other methods. This is likely because this multimodal dataset is more common in training of image generation models compared to the other text-only ones. Notably, using 5 LAION keywords can almost match performance of using 60-word descriptions. Top 20 LAION and embedding-ranked keywords can both generate more copyrighted characters than using the more detailed paragraph descriptions. Figure 3 shows some examples of these generated images with the descriptions and keywords discussed above.

Descriptions and identified keywords also transfer to generating characters from DALL·E 3 and video models. We further test indirect anchors on production-level models, such as DALL·E 3.

Figure: (a) DALL·E 3, 60-word description
Refer to caption: x7.png

Surprisingly, indirect anchors like descriptions can still bypass system safeguards and result in the generation of copyrighted characters (Figure 4). This further suggests that current safeguards are not fully effective. More results can be found in Appendix E.In addition, we also test indirect anchors on the video generation model VideoFusion (Luo et al., )(see Figure 4 for examples). We compare selection methods for indirect anchors in Figure 2. LAION is the most useful corpus for identifying such keywords, and has a smaller gap to 60-word description on video generation compared to image generation.

### 5.2 Mitigation Effectiveness

The next question is: can we effectively prevent the models from recreating these copyrighted characters? We mainly evaluate the intervention strategies discussed in §4, specifically: 1) using prompt rewriting only, 2) using negative prompts only,(^7^77To effectively apply the proposed negative prompts, model deployers need a mechanism to detect the identity of the intended copyrighted character (if any) from the user’s prompt. As the primary focus of this work is not end-to-end system building but the evaluation of specific mitigation methods, we assume the existence of such a method. However, we provide more discussion on this in Section F.2 and demonstrate two possible implementations for detecting whether a prompt may reference (directly or indirectly) a popular character.) and 3) combining negative prompts and prompt rewriting.

We evaluate these strategies on CopyCat using $\mathsf{DETECT}$ and $\mathsf{CONS}$ as described in §2.2: $\mathsf{DETECT}$ counts the number of detected copyrighted characters, and $\mathsf{CONS}$ measures the image’s consistency with user input. A good mitigation strategy achieves low $\mathsf{DETECT}$ and high $\mathsf{CONS}$. We run each strategy three times and report the mean and standard deviation of $\mathsf{DETECT}$ and $\mathsf{CONS}$ in Table 1.

Prompt rewriting alone is not entirely effective at eliminating outputs similar to copyrighted characters. Our evaluation starts with prompt rewriting (§4), which has been adopted as an intervention strategy for production-level models like DALL·E. However, as demonstrated in Table 1, solely adopting prompt rewriting can only reduce $\mathsf{DETECT}$ from 30 to 14. Nonetheless, an advantage of prompt rewriting is that the $\mathsf{CONS}$ scores modestly improve, likely due to the rewritten prompts containing more detailed information.

**Table 1: Performance of all intervention strategies on the Playground v2.5 model. We run each strategy three times, and report the mean and standard deviation of the number of detected copyrighted characters ($\mathsf{DETECT}$, lower is better) and the consistency with user intent ($\mathsf{CONS}$, higher is better). Including the character’s name in the negative prompts is highly important for reducing $\mathsf{DETECT}$. Combining prompt rewriting and negative prompts can effectively reduce $\mathsf{DETECT}$ from 30 to 5, without significantly degrading $\mathsf{CONS}$.**
| Negative Prompt | Prompt: Target’s name | Prompt: Rewritten prompt |  |  |
| --- | --- | --- | --- | --- |
| $\mathsf{DETECT}$ ($\downarrow$) | $\mathsf{CONS}$ ($\uparrow$) | $\mathsf{DETECT}$ ($\downarrow$) | $\mathsf{CONS}$ ($\uparrow$) |  |
| None | $30.33_{\pm 1.89}$ | $0.75_{\pm 0.01}$ | $14.33_{\pm 2.62}$ | $0.80_{\pm 0.01}$ |
| "Copyrighted character" | $30.33_{\pm 1.25}$ | $0.74_{\pm 0.01}$ | $17.33_{\pm 1.70}$ | $0.80_{\pm 0.01}$ |
| + 5 LM-Ranked keywords | $30.33_{\pm 1.89}$ | $0.71_{\pm 0.01}$ | $14.33_{\pm 1.70}$ | $0.80_{\pm 0.00}$ |
| + 5 EmbeddingSim keywords | $28.00_{\pm 1.41}$ | $0.72_{\pm 0.03}$ | $15.67_{\pm 1.25}$ | $0.80_{\pm 0.00}$ |
| + 5 Co-Occurrence-LAION keywords | $27.33_{\pm 0.00}$ | $0.73_{\pm 0.01}$ | $14.33_{\pm 0.94}$ | $0.80_{\pm 0.00}$ |
| + 5 EmbeddingSim & 5 Co-Occurrence-LAION keywords | $23.33_{\pm 3.30}$ | $0.72_{\pm 0.03}$ | $7.00_{\pm 1.63}$ | $0.81_{\pm 0.00}$ |
| Target’s name | $23.67_{\pm 2.62}$ | $0.76_{\pm 0.01}$ | $7.67_{\pm 0.47}$ | $0.81_{\pm 0.01}$ |
| + 5 LM-Ranked keywords | $25.00_{\pm 1.63}$ | $0.74_{\pm 0.01}$ | $7.00_{\pm 1.63}$ | $0.81_{\pm 0.02}$ |
| + 5 EmbeddingSim keywords | $22.67_{\pm 2.36}$ | $0.73_{\pm 0.02}$ | $5.67_{\pm 0.47}$ | $0.80_{\pm 0.00}$ |
| + 5 Co-Occurrence-LAION keywords | $20.67_{\pm 2.05}$ | $0.75_{\pm 0.01}$ | $5.00_{\pm 0.82}$ | $0.81_{\pm 0.01}$ |
| + 5 EmbeddingSim & 5 Co-Occurrence-LAION keywords | $20.67_{\pm 0.47}$ | $0.72_{\pm 0.03}$ | $4.33_{\pm 0.47}$ | $0.81_{\pm 0.00}$ |

**Table 2: The combination of prompt-rewriting and negative prompts (target’s name & 5 EmbeddingSim & 5 Co-Occurrence-LAION keywords) can significantly reduce $\mathsf{DETECT}$ while mostly preserving $\mathsf{CONS}$ across all 5 open-source models tested, making it a promising candidate for copyright-protection intervention.**
| Model | w/o Intervention | w/ Prompt Rewriting & Negative Prompt |  |  |
| --- | --- | --- | --- | --- |
| $\mathsf{DETECT}$ ($\downarrow$) | $\mathsf{CONS}$ ($\uparrow$) | $\mathsf{DETECT}$ ($\downarrow$) | $\mathsf{CONS}$ ($\uparrow$) |  |
| Playground v2.5 (Li et al., ) | $30.33_{\pm 1.89}$ | $0.75_{\pm 0.01}$ | $4.33_{\pm 0.47}$ | $0.81_{\pm 0.00}$ |
| Stable Diffusion XL (Podell et al., ) | $33.00_{\pm 1.00}$ | $0.73_{\pm 0.01}$ | $1.67_{\pm 0.94}$ | $0.77_{\pm 0.03}$ |
| PixArt-$\alpha$ (Chen et al., ) | $24.67_{\pm 0.58}$ | $0.79_{\pm 0.01}$ | $4.67_{\pm 0.47}$ | $0.79_{\pm 0.01}$ |
| DeepFloyd IF (StabilityAI, ) | $33.67_{\pm 1.53}$ | $0.71_{\pm 0.01}$ | $2.00_{\pm 1.00}$ | $0.72_{\pm 0.01}$ |
| VideoFusion (Luo et al., ) | $28.33_{\pm 1.89}$ | $0.68_{\pm 0.01}$ | $11.33_{\pm 1.53}$ | $0.76_{\pm 0.01}$ |

Figure: (a) Prompt: Character’s name, Negative Prompt: None
Refer to caption: x9.png

We then investigate potential reasons for prompt rewriting sometimes failing. Specifically, we calculate the average number of Top-5 LAION keywords present in rewritten prompts that result in $\mathsf{DETECT}=0$ (success) and $\mathsf{DETECT}=1$ (failure). We find that the failed rewritten prompts contain on average more LAION keywords—$0.667$ for failure cases compared to $0.387$ for success cases. Similarly, we also observe that failing rewritten prompts tend to share higher embedding similarity with the character’s name (see Section F.3). This again suggests the existence of indirect anchors, and potentially their inclusion in rewritten prompts could impair this strategy.

Using negative prompts improves elimination of similar output with modest impact on consistency. In addition to existing countermeasures like prompt-rewriting, we also explore negative prompts (§4). Specifically, we use keywords identified with different methods (§4, with $k=5$) as negative prompts. We generally observe that including Co-Occurrence-LAION results in higher reduction in $\mathsf{DETECT}$ compared to including LM-Ranked and EmbeddingSim (Table 1). This shows that LAION keywords are more effective than those ranked by LM or embedding space distance.

Including character names in the negative prompt is also helpful. As shown in Table 1, compared to the upper half, the lower half (target name included in negative prompt) consistently has lower $\mathsf{DETECT}$ scores.(^8^88We also examine the effectiveness of adding the character name to the negative prompt when user input does not contain the character name and also find a consistent effect. For example, in the case of paragraph-length descriptions, the number of detected characters is reduced by over $50\%$ while maintaining consistency (see Section F.1).)
Incorporating LAION keywords into the negative prompts in addition to character name further reduces $\mathsf{DETECT}$. The combination of these words in the negative prompt significantly reduces the original $\mathsf{DETECT}$ score from 30 to 4. Notably, the addition of negative prompts does not significantly impair generated image’s consistency with user’s intended prompt, as the $\mathsf{CONS}$ scores typically remain similar or only slightly lower compared to the no intervention setting, but still substantially above $0.33$, the value which indicates very high consistency (see Section C.4). Figure 5 and Figure 12 (in Section F.4) visualize some qualitative examples.

Combining prompt rewriting and negative prompts shows promise for elimination of similar output. Finally, we combine prompt rewriting and negative prompts. Specifically, we send the rewritten prompts as inputs to the image generation models. Then we apply negative prompts during generation. Surprisingly, as demonstrated in Table 2, this simple technique is already quite promising in alleviating copyright concerns and is effective across all open-source models evaluated.(^9^99DALL·E does not allow customizing negative prompts.) The number of detected copyrighted characters is significantly reduced for all models. Notably, the number of detection decreases to only 5% of the original in the case of DeepFloyd. At the same time, the $\mathsf{CONS}$ scores remain mostly stable. This suggests that despite the pressing concern of image generation models generating copyrighted characters, we can use this simple yet effective method for meaningful mitigation. Figure 5 and Figure 13 (in Section F.4) present some examples. As shown, most generated images still align with the user’s intent in the sense that the generated figure is of a similar entity as the requested copyrighted character, but the generation result is already drastically different from the requested copyrighted characters.

Nonetheless, even this combination of strategies is not perfect at stopping the generation of copyrighted characters, which calls for more future research efforts.

## 6 Related Work

#### Diffusion Models

Diffusion models is a type of generative models that synthesize images through two intertwined processes: the forward diffusion and the reverse diffusion paths (Rombach et al., ; Podell et al., ). In the forward diffusion process, an image gradually transitions from its original state to a fully noised version by incrementally adding noise. The reverse process aims to reconstruct the original image from this noisy state. These models can approach the reverse process in two ways: by either predicting the clean image directly at each step or by estimating the noise to be subtracted from the noisy image. Training diffusion models requires extensive datasets, such as LAION-5B, which consists of a vast collection of publicly accessible copyrighted materials (Schuhmann et al., ). As these models evolve, diffusion models can generate copies of samples from their training data (Carlini et al., ; Vyas et al., ), which raises potential concerns regarding privacy and copyright. Recent works have explored some potential pathways to suppress certain concepts from being generated in the diffusion process (Kumari et al., ; Li et al., ). While these methods further fine-tune models and address memorized styles and images individually, we aim to examine operationalizable ways to add copyright protection without updating the parameters.

#### Copyright and Generative Models

Recent studies have delved into the copyright implications of generative models such as diffusion models and language models (Sag, ; Henderson et al., ; Lee et al., ; Sag, ).
Lee et al. (), Sag (), and Henderson et al. () in particular point to copyrighted characters as a challenging legal area.
Each of them note that it may be possible for characters to be generated even when users don’t explicitly input the character name, though they do not systematically evaluate this phenomenon.

Others have demonstrated that these models can potentially reconstruct or replicate copyrighted content from their training data (Carlini et al., , ). Efforts to mitigate these risks include provable copyright protection strategies inspired by differential privacy (Vyas et al., ), decoding-time prevention (Golatkar et al., ) that guide the generation process away from copyright concepts and model editing and unlearning that aim to remove copyrighted content from model weights (Chefer et al., ; Zhang et al., ).
However, the legal framework remains underdeveloped, posing a significant challenge as the capabilities of these generative models continue to advance. A concurrent study by Kim et al. () leverages a large language model optimizer to generate prompts that potentially maximize the likelihood of generating copyrighted content in proprietary image-generation models. Similarly, Zhang et al. () focus on building attacks that can generated particular concepts—where concepts are broadly defined and include some copyrighted characters.
These works focus on attacks and do not explore effective mitigation methods. Our work, however, focuses on building an evaluation framework for understanding the effectiveness of defenses.
This necessarily includes similarly identifying indirect anchors that might generate copyrighted characters.
These studies also focus on longer prompts, while our work demonstrates the possibility of such violations with just a few keywords (around five).

## 7 Limitations and Future Work

Our work provides an initial step forward for systematically evaluating the likelihood of generating copyrighted characters and the effectiveness of inference-time mitigation strategies. Future works can improve these evaluation protocols and mitigations in several ways. First, they can leverage optimization-based approaches to identify more complicated indirect anchors.
Second, they can explore improved mechanisms to identify user intent to generate copyright characters from prompts alone.
For example, if a user writes a complicated prompt “A video game plumber with a red hat and an M on the hat, in blue overalls….”, model improvements could better map the description to a potential character so that their name could be included in the negative prompt.
Third, future work can address additional broader types of similarly challenging visual content, like trademarks, as well as broader sets of less-popular characters—our assessment is limited to a relatively small set of popular characters.
Fourth, metrics like consistency scoring and detection could be improved to better capture legally-relevant and human-centered notions of consistency and character similarity.
While our work will likely be re-usable for these broader categories of copyrighted and trademarked content, we did not explicitly evaluate them here.

## 8 Conclusion

In this paper, we investigate two main research questions: 1) Which textual prompts can trigger generation of copyrighted characters; and 2) How effective are current runtime mitigation strategies and how we can improve them? To systematically study these questions, we curate a diverse set of copyrighted characters and develop a novel evaluation suite CopyCat that considers both elimination of similar output to copyrighted characters and generated image’s consistency with user input. We show how to leverage embedding space distance and common training corpora to extract useful indirect anchors—descriptions and keywords not explicitly mentioning the characters’ names.
We find that these indirect anchors can be effective in triggering copyrighted character generation.
Existing mitigations, namely prompt rewriting, are not fully effective and we suggest new runtime methods to improve them.
Our work calls for more attention to the indirect anchoring challenge and the effectiveness of deployed mitigation strategies for copyrighted character protection. The insights we provide here can be operationalized by model deployers for copyright-aware image and video generation systems in the future.

## Acknowledgement

We thank Yanai Elazar for his insights on the WIMBD indices. We thank Colin Wang, Mengzhou Xia, Dan Friedman, Howard Yen, Jiayi Geng, Xindi Wu, Boyi Wei, Samyak Gupta, and Eric Wallace for providing helpful feedback. Luxi He is supported by the Gordon Y. S. Wu Fellowship. Yangsibo Huang is supported by the Wallace Memorial Fellowship. This research is partially supported by a Princeton SEAS Innovation Grant.
Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.

## Appendix A Legal Background and Broader Societal Impacts

While past work has studied the setting of verbatim regurgitation of images (Carlini et al., ), and some lawsuits focus on this particular legal issue (Vincent, ; *Andersen et al. v. Stability AI et al.*, ), copyrighted characters pose a unique legal challenge (Sag, ; Henderson et al., ; Lee et al., ). Unlike in the verbatim memorization setting, copyrighted characters are computationally more like general concepts that can appear in many poses, sizes, and variations in the training data. So typical deduplication, or even near access free learning approaches (Vyas et al., ), will not work—something discussed by others (Henderson et al., ).

Copyrighted characters are a somewhat distinct area of copyright law with distinct rules to determine infringement (Schreyer, ; Hennessey, ).
To simplify the legal rules, characters are defined by key distinctive features that as a whole comprise the character. This can lead to interesting situations.
For example, in 2023 the copyright for the original version of Mickey Mouse character (Steamboat Willie) entered the public domain. But this version of the character did not wear white gloves. However, the gloved version of Mickey Mouse that is now well known has not yet entered the public domain. A number of legal scholars and commentators have pointed out that this means that using a visual depiction of the modern Mickey Mouse would likely lead to an infringement claim, but using the old style of Mickey Mouse (Steamboat Willie) would not (Liu, ; Lee, ).

In some cases, characters can also be trademarked, leading to other distinct legal challenges not available for memorization of datapoints as a general problem (Hennessey, ).

The paper studies how companies can respect the intellectual property rights of creators and their visual copyrighted characters from an inference-time technical perspective. Leveraging methods here will both improve likelihood that rights are respected and reduce litigation risk for companies. However, we do not address broader societal discussions on how artists should be compensated for training on images that may contain their intellectual property (such as their characters). This is a larger, worthy, discussion broader than the scope of our work. However, we note that current fair use doctrine in the United States may allow this training provided that mitigation strategies are used to prevent substantially similar outputs (Lee et al., ; Lemley & Casey, ; Henderson et al., ; Pasquale & Sun, ). Similar fair use standards exist in other countries as well, but globally there are also countries where even training may not be allowed and different approaches may be needed. There are also general labor displacement concerns that are important that go beyond the scope of this work.

## Appendix B Full List of Characters and Studios in CopyCat

Ariel, Astro Boy, Batman, Black Panther, Bulbasaur, Buzz Lightyear, Captain America, Chun-Li, Cinderella, Cuphead, Donald Duck, Doraemon, Elsa, Goofy, Groot, Hulk, Iron Man, Judy Hopps, Kirby, Kung Fu Panda, Lightning McQueen, Link, Maleficent, Mario, Mickey Mouse, Mike Wazowski, Monkey D. Luffy, Mr. Incredible, Naruto, Nemo, Olaf, Pac-Man, Peter Pan, Piglet, Pikachu, Princess Jasmine, Puss in boots, Rapunzel, Snow White, Sonic The Hedgehog, Spider-Man, SpongeBob SquarePants, Squirtle, Thanos, Thor, Tinker Bell, Wall-E, Winnie-the-Pooh, Woody, Yoda.

Walt Disney Animation Studios, Disney subsidiaries (Marvel Studios, Pixar Animation Studios, Lucasfilm), Tezuka Productions, DC Comics (Warner Bros.), Nintendo, Capcom, Shin-Ei Animation, Studio MDHR, HAL Laboratory, DreamWorks Animation (Universal Pictures), Toei Animation, Pierrot, Bandai Namco Entertainment, Sega, Nickelodeon Animation Studio, Sony Pictures.

## Appendix C Experimental details

### C.1 Compute resource

All experiments are conducted on 2 NVIDIA A100 GPU cards, each with 80GB of memory. Table 3 provides statistics on the time cost for each image generation across all the evaluated models, using the character’s name as the input prompt.

**Table 3: Averaged time cost per generation for evaluated models using 2 NVIDIA A100 GPU cards.**
| Model | Time cost (seconds) per generation |
| --- | --- |
| Playground v2.5 (Li et al., ) | $5.1$ |
| Stable Diffusion XL (Podell et al., ) | $36.4$ |
| PixArt-$\alpha$ (Chen et al., ) | $8.3$ |
| DeepFloyd IF (StabilityAI, ) | $16.4$ |
| VideoFusion (Luo et al., ) | $6.7$ |

We also report the time cost per evaluation for a single image in Table 4, including the cost of running the GPT-4V detector on the image and calculating the consistency score between the image and its key characteristics using VQAScore (Lin et al., ). Note that the time cost of the GPT-4V detector is obtained via querying the API, so it may also depend on the real-time network traffic.

**Table 4: Averaged time cost per evaluation on 2 NVIDIA A100 GPU cards. Note that the GPT-4v detector does not require local computational resources, as we query the API provided by OpenAI.**
| Evaluation | Time cost (seconds) per generation |
| --- | --- |
| GPT-4V detector | $3.8$ |
| VQAScore | $<0.1$ |

### C.2 Model Generation Configurations

For Playground v2.5, Stable Diffusion XL (SDXL), and PixArt-$\alpha$, we use 50 iterative steps to progressively refine the image from noise to a coherent output. We set guidance_scale to 3 for the strength of the conditioning signal.

For DeepFloyd IF, we use the standard 3-stage set-up. Models for the 3 stages are DeepFloyd’s IF-I-XL-v1.0, IF-II-L-v1.0, and Stability AI’s stable-diffusion-x4-upscaler respectively. All generation configurations are the model’s default.

For video generation on VideoFusion, we use the model’s default parameters to generate a 16-frame video, and take the first, middle, and last frames for detailed study.

### C.3 GPT-4V evaluator

We use GPT-4V to check whether there exists a character in the image that can be recognized as an existing copyrighted character. For each image, we prompt the multimodal model with the alongside the following prompt:

### C.4 Consistency Score

As described in §2, we use the VQAScore (Lin et al., ) to measure consistency ($\mathsf{CONS}$) by checking if the main subject of the target copyrighted character exists in the generated image. The backbone model for computing VQAScore is CLIP-FlanT5.(^10^1010[https://huggingface.co/zhiqiulin/clip-flant5-xxl](https://huggingface.co/zhiqiulin/clip-flant5-xxl))

To establish reference points, we consider two settings that yield high and low $\mathsf{CONS}$ scores:

- •
Prompting Playground v2.5 with the character’s name yields $\mathsf{DETECT}=33$ and $\mathsf{CONS}=0.741$. Hence, $\mathsf{CONS}\approx 0.75$ indicates high consistency.
- •
Prompting Playground v2.5 with the character’s name but randomly replacing 3 letters yields $\mathsf{DETECT}=1$ and $\mathsf{CONS}=0.329$. Hence, $\mathsf{CONS}\approx 0.33$ indicates low consistency.

### C.5 Generation of Indirect Anchors

50 candidate keywords for indirect anchors ranking. We use GPT-4 to generate a pool of $n$ candidate keywords related to each character, where $n=50$ in our main experiments. Below is an example prompt for collecting $50$ keywords for Mario:

60-word descriptions. We use GPT-4 to generate 60-word descriptions for studying description-based indirect anchors. Below is an example prompt for collecting such a description for Mario:

### C.6 Prompt-Rewriting Template

For prompt-rewriting intervention, we adapt the DALL·E system prompt to simulate mitigation strategies adopted by model deployers. The prompt is modified to highlight copyright concerns as well as restrict prompt length to be around 60 words.
(^11^1111The original full DALL·E system prompt is discussed at [https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/_system-prompts/gpts/dalle.md](https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/_system-prompts/gpts/dalle.md))

The full prompt used in our experiment is shown below:

Figure 6 shows an example of the generated keywords and descriptions for Mario.

Figure: Figure 6: Indirect anchors (keywords and descriptions) that trigger models to generate Mario. Both keywords and descriptions in the figure are LM-generated indirect anchors.
Refer to caption: x13.png

## Appendix D Human evaluation (authors)

To verify the reliability of judgments provided by the GPT-4V evaluator, we conduct an internal human evaluation process among the authors.

Specifically, we first sample 200 generated images (20 characters $\times$ 10 images per character) from various prompting configurations, including direct prompting with character names and indirect prompting using keywords or descriptions, with or without the application of mitigation strategies. We then ask 6 authors to independently annotate these images, following guidelines similar to those used for GPT-4V (described in Section C.3).

For these 200 records, we examine the accuracy of GPT-4V, with the majority-human scores as ground truths. We find that the scores assigned by GPT-4V obtain a fairly high accuracy of $82.5\%$.
To further analyze the consistency and agreement, we compute the Cohen Kappa value (Cohen, ) between GPT-4V scores with the majority-human scores.
As evaluated, we observe a Cohen Kappa value of 0.648, representing a substantial agreement between human annotators and GPT-4V. We also accompany the pair-wise agreement measurements among human annotators and GPT-4V in Figure 7.

Figure: Figure 7: Cohen Kappa score matrix between human annotators and GPT-4V. As shown, at most time, human annotators themselves reach substantial agreements (Cohen Kappa score in between $0.6\sim 0.8$). Meanwhile, GPT-4V achieves a slightly lower scores when compared with human (yet still substantial agreement on average).
Refer to caption: extracted/5681539/figures/exp/matrix_agreement.png

## Appendix E More results on DALL·E

Character name anchoring does not work on DALL·E system due to its built-in filter that detects and blocks requests that explicitly mention copyrighted characters. However, indirect anchoring is still able to bypass the system guardrails and generate high-quality images that highly resemble the target copyrighted characters, as illustrated in Figure 8.

Figure: Figure 8: Using 60-word descriptions to circumvent built-in safeguards like character name detection and prompt rewriting, we are able to push DALL·E 3 to generate copyrighted characters.
Refer to caption: x14.png

## Appendix F More results on open-source models

### F.1 Effect of Target’s Name as Negative Prompt for Indirect Anchoring

We also examine intervention strategies in cases where users provide keywords or descriptions to generate images. As shown in Table 5, consistent with our previous observations in Table 1 when the character name is part of user input, adding character name as negative prompt is still a very effective method to reduce recreating copyrighted characters. In particular, when the original prompt consists of 10 keywords or descriptions, incorporating target’s name as negative reduce $\mathsf{DETECT}$ by 50% or more, while $\mathsf{CONS}$ values remain almost constant. For all experiment setup, the $\mathsf{CONS}$ values either remain stable or show a slight decrease with the addition of negative prompts. From a practical perspective, adding copyright character detection and target name as negative prompt is a simple yet effective way of reducing the recreation of copyrighted characters, at the cost slight compromise in adhering to user request.

**Table 5: Effect of adding character names as negative prompts on different indirect anchors set-up.**
|  | Negative Prompt: None | Negative Prompt: Target’s name |  |  |
| --- | --- | --- | --- | --- |
| Original Prompt | $\mathsf{DETECT}$ ($\downarrow$) | $\mathsf{CONS}$ ($\uparrow$) | $\mathsf{DETECT}$ ($\downarrow$) | $\mathsf{CONS}$ ($\uparrow$) |
| 10 curated keywords | $14.00_{\pm 3.0}$ | $0.76_{\pm 0.01}$ | $7.00_{\pm 2.00}$ | $0.76_{\pm 0.00}$ |
| 20 curated keywords | $28.00_{\pm 2.65}$ | $0.78_{\pm 0.00}$ | $20.67_{\pm 3.21}$ | $0.76_{\pm 0.00}$ |
| 50 curated keywords | $29.67_{\pm 2.08}$ | $0.78_{\pm 0.01}$ | $16.00_{\pm 1.00}$ | $0.76_{\pm 0.00}$ |
| 5 keywords from LAION | $19.67_{\pm 2.89}$ | $0.74_{\pm 0.00}$ | $12.33_{\pm 2.31}$ | $0.72_{\pm 0.01}$ |
| Description | $21.00_{\pm 2.65}$ | $0.78_{\pm 0.01}$ | $10.33_{\pm 0.58}$ | $0.78_{\pm 0.01}$ |

### F.2 Intent detection

In practice, user inputs can include both standard requests for generating non-copyrighted images and requests for generating copyrighted characters. In our evaluation, we assume the presence of an oracle capable of detecting whether a user input is likely to lead a text-to-image model to generate a copyrighted character. To validate this assumption, we explore two methods:

- 1.
LLM-based detector that uses an LLM to determine if the user input is associated with a copyrighted character. It directly queries the LLM with the prompt, "Does the following description resemble any copyrighted character?" We then compare the model’s prediction to the correct answer.
- 2.
Retriever-based detector that uses a retriever to compare the user input against a database of copyrighted character descriptions. For a given user query, the retriever searches for similar descriptions based on OpenAI embeddings(^12^1212text-embedding-3-small). If no description with a cosine similarity greater than 0.7 is found, we conclude that the user query does not intend to generate characters substantially similar to copyrighted ones.

#### Experimental Setup

To evaluate our detection methods, we curated a dataset comprising 200 descriptions of copyrighted characters and 200 standard prompts unlikely to cause copyright issues selected from MJHQ benchmarks(^13^1313[huggingface.co/datasets/playgroundai/MJHQ-30K](huggingface.co/datasets/playgroundai/MJHQ-30K)). We report accuracy, true positive rates (TPR) and false positive rates (FPR) as our evalaution metrics.

#### Results

As shown in Table 6, both methods achieve over 90% accuracy. The LM-based detector achieved an accuracy of 95%, slightly outperforming the retriever-based detector. This high performance indicates that both methods are effective in identifying potential copyright issues in user inputs. It is therefore reasonable to assume that building such a detection oracle is feasible and can be done relatively easily.

**Table 6: Accuracy, true positive rate (TPR) and false positive rate (FPR) of LM-based and retriever-based detectors.**
| Detection Method | Accuracy (%) | TPR (%) | FPR (%) |
| --- | --- | --- | --- |
| LM-based detector | $95.14$ | $93.68$ | $3.32$ |
| Retriever-based detector | $93.28$ | $91.26$ | $4.36$ |

### F.3 Embedding similarity analysis

#### 60-word description as indirect anchors.

We randomly generate 100 60-word prompts per character using the template described in Appendix C, and rank them by embedding similarity to the corresponding character name. As shown in Figure 9, the top-ranked prompt by embedding similarity generates 26 characters successfully, versus only 16 for the bottom-ranked prompt.

Figure: Figure 9: Character generation success ($\mathsf{DETECT}$ scores) for 60-word descriptions with varying embedding similarity to the target character’s name. Prompts with higher name similarity tend to generate the desired character more often.
Refer to caption: x15.png

#### Rewritten prompts.

We also study how the success and failure of rewritten prompts correlate with their embedding similarity to the corresponding character name. Specifically, for each character, we generate 100 rewritten prompts and rank them by their embedding similarity to the character’s name. As shown in Figure 10, the top-ranked rewritten prompt by embedding similarity generates 20 characters successfully, versus only 12 for the bottom-ranked rewritten prompt. This suggests that potentially, rewritten prompts that fail to avoid character generation could be due to their high similarity to the character’s name.

Figure: Figure 10: Character generation success ($\mathsf{DETECT}$ scores) for rewritten prompts with varying embedding similarity to the target character’s name. Rewritten prompts with higher name similarity tend to generate the desired character more often (i.e., tend to fail in mitigating).
Refer to caption: x16.png

### F.4 More results for Playground v2.5

#### Robustness analysis of character name anchoring.

Interestingly, the model exhibits high sensitivity to even minor perturbations in the character’s name. For instance, if we randomly replace a single letter in the character’s name with a different letter, the model can only generate 8 out of the 50 characters successfully. The situation is even more extreme when we randomly replace 3 letters – in this case, the model could only generate 1 out of the 50 characters accurately (see Figure 11(b)).

On the other hand, if the character’s name is present in the prompt, and irrelevant keywords such as "dancing" or "swimming" are added, this generally does not affect the number of characters generated (see Figure 11(c) and Figure 11(d)). These findings suggest that the character name anchoring mode heavily relies on the exact spelling of the target character’s name to generate copyrighted characters.

Figure: (a) Prompt: Character’s name
Refer to caption: x17.png

#### More visualization.

Figure 12 visualizes results using the character’s name as the prompt and various keywords as negative prompts. Including the character’s name in the prompt, even with detailed negative prompts, still leads to the generation of copyrighted characters. This suggests that T2I models are deeply anchored to these character names.

However, once we apply prompt rewriting and combine it with various negative prompts, the model is no longer inclined to generate these characters, as shown in Figure 13.

Figure: (a) Negative Prompt: None
Refer to caption: x21.png

Figure: (a) Negative Prompt: None
Refer to caption: x25.png

### F.5 Results for PixArt- α 𝛼 \alpha italic_α , Stable Diffusion XL, and DeepFloyd IF

Figure 14 visualizes results from the PixArt-$\alpha$ model (Chen et al., ). With higher generation quality, the findings are also consistent with those observed using the Playground v2.5 model—adding more fine-grained negative prompts and applying prompt rewriting significantly reduces the similarity of the generated images to the original copyrighted character.

Figure 15 visualizes results from the Stable Diffusion XL (SDXL) model (Podell et al., ). Although the generation quality of SDXL is generally lower compared to the Playground model (see Figure 5), adding more fine-grained negative prompts and applying prompt rewriting significantly reduces the similarity of the generated images to the original copyrighted character.

Figure: (a) Prompt: Character’s name, Negative Prompt: None
Refer to caption: x29.png

Figure: (a) Prompt: Character’s name, Negative Prompt: None
Refer to caption: x33.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Why are detailed results only available for Playground v2.5?
To which extent negative Prompts in diffusion models would be more reliable than in LLM?
Could this approach be similarly "jailbreaked" using standard additions such as <I NEED to test how the tool works with extremely simple prompts. DO NOT add any detail, just use it AS-IS:>
soundness: 3
strengths: The paper addresses a very relevant topic in a way that goes way beyond standard approaches provided with image generation systems, such as default Prompt rewriting. The research questions (Q1/Q2) are clearly formulated, and the empirical metrics developed are well-aligned to these questions. Consistency via the VQAS score is an important addition to the method. 
Despite its empirical stance, the paper is rather rigorous on the technical aspects, which includes an extensive choice of image generation systems, an examination of training corpora, including captions from image-captioning datasets. 
The presentation of results in Table I and Table II is detailed and convincing, with potentially a 7-fold decrease in copyrighted character detection. 
There is a comprehensive section on supplementary material giving additional technical details (although sometimes difficult to distinguish in nature from the paper itself). Moreover, including a self-contained, yet technically relevant, legal perspective will no doubt be a great use to the readers.
summary: This paper addresses the issue of copyright violation in image or video generation from text. More specifically, it explores empirically which Prompting techniques can be used to avoid regenerating visual content that was part of these systems' training base (e.g. for Diffusion Systems). It investigates the risk of accidental copyrighted content generation from user Prompts in order to develop mitigating strategies.
Experimenting with four image generation (Playground v2.5, Stable Diffusion XL, PixArt-α, DeepFloyd IF) and one video generation (VideoFusion) systems, the authors evaluate the following mitigation strategies: 1) using prompt rewriting only, 2) using negative prompts only, and 3) combining negative prompts and prompt rewriting. The authors develop two specific metrics (DETECT and CONS) measuring copyrighted character generation and consistency with user intent, to assess intervention performance and lack of side-effects. Based on this metrics they report positive results for their approach, in some cases reducing copyright infringement from 30% to 5%.
weaknesses: Poor document structure, with some important elements appearing in the additional material section, and no clear, systematic progression, sometimes leaving an impression of separate, quasi-anecdotal testing. While this may not affect the quality or significance of individual results, it makes the paper tedious to read at times and weakens the overall use case.
Aside from the quantitative reporting (DETECT, CONS) there are a number of evaluation issues:
- lack of a discussion on whether the test sample is sufficient for results' significance
- differences expressed in absolute values with no statistical testing
- qualitative results difficult to assess, and uneven across characters
- (supplementary material) testing by the authors themselves, which is not good practice.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can you please explain the disconnect from jailbreaking and red teaming? Specifically, what aspects of the copyright character phenomenon studied in this work differentiate it so substantially to motivate such a disconnected framing?
2. Can you explain and/or justify the focus on prompt rewriting?
3. Can you explain and/or justify the disconnect from related work on "debiasing" generative models?
4. Can you provide an explicit statement on the contributions that this manuscript makes in light of #1 and #3?
soundness: 3
strengths: The paper has several key strengths:

## 1. Writing Quality and Clarity
The paper is remarkably concise, clear, and articulate in its writing. Despite aligning to terminologies consistent with our area of research, I imagine that the manuscript's writing style and depth would allow many types of stakeholders beyond ICLR's more technical community (e.g., software engineers with limited machine learning experience) to engage with it. 

## 2. Evaluation Depth and Methodological Rigor
The paper's presented relies on methodological rigor to conduct a through examination of the phenomenon of copyright character generation. I found the authors' choice of methods, metrics, and datasets to be appropriate. One might argue that the methods described and employed in portions of their Appendix are prominent enough to be their own contributions.

## 3. Practical Motivation and Commit to Reproducibility
The paper's motivation is thematically oriented toward model use in practice and consumer reproducibiltiy, which makes aspects of the paper (e.g the Discussion) have a rather unique feel.
summary: This paper introduces an evaluation framework for assessing a generated image's similarity to copyrighted characters (i.e., avatars) and its consistency with user intent. The paper uses the framework to illustrate how state-of-the-art image and video generation models continue to generate copyrighted characters, despite not being prompted to do so explicitly by name. The authors study the effectiveness of semi-automatic techniques to identify keywords or descriptions that trigger this behavior alongside strategies for mitigating them as well. The authors' work is framed as a form of "empirical grounding" to facilitate discussion on copyright mitigation strategies that can be leveraged by stakeholders interested in safeguarding generated content.
weaknesses: My concerns with the paper are primarily related to its positioning and contribution with respect to prior work in related areas. Below, I highlight three key concerns:

## 1. Weak Connection to Jailbreaking and Red Teaming
The paper is motivated by two key questions that related to the generation of copyrighted characters and mitigation strategies against attempts at such generation. I find this conceptually identically to the challenge of jailbreaking, yet the authors appear to have explicitly chosen to differentiate the two (e.g., See: L871 for the sole reference to jailbreaking). There is a wide array of preexisting work related to jailbreaking LLMs and a growing body of literature on jailbreaking MLLMs (e.g., Ma et al.). I'm perplexed with the connection with this area of research is weak.

Ma et al. “Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Characte.” ArXiv abs/2405.20773 (2024): n. pag.

## 2. Narrow Scope of Semi-Automatic Techniques: Prompt Rewriting
I found the scope of semi-automatic techniques (i.e. on prompt rewriting) to be quite thin in its novelty. As the authors note, prompt rewriting is a well-established technique both within text-to-image models and beyond it. There are a large number of techniques (i.e., from studies related to jailbreaking and red teaming) that go beyond prompt rewriting (e.g. Yang et al's token perturbation approach). 

Yang et al. “SneakyPrompt: Jailbreaking Text-to-image Generative Models.” 2024 IEEE Symposium on Security and Privacy (SP) (2023): 897-912.

## 3. Relevance to "Debiasing" Generative Models
The phenomenon of copyrighted characters being generated on the basis of indirect anchoring can be viewed and understood as a form of generative bias. There are prior works (e.g. He et al.) that have studied techniques for debiasing text-to-image models in similar ways. 

He et al. In Proceedings of the 1st ACM Multimedia Workshop on Multi-modal Misinformation Governance in the Era of Foundation Models (MIS '24). Association for Computing Machinery, New York, NY, USA, 29–36. https://doi.org/10.1145/3689090.3689387

## 4. Significance and Contribution
Amid the other weaknesses, I find myself asking if the authors' motivating statement remains true. There exists a need to revisit the significance and contribution of the work and provide an explicit statement regarding the novelty and significance of the conducted work.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Do you consider a metric with 82.5% of accuracy appropriate? Why?
2. Why did you not incorporate the metric's accuracy in the analysis of the results?
soundness: 2
strengths: Although the ability of image-generation engines to create, by instruction or indirectly, is known, it is important to measure how common the problem is and how susceptible they are to produce them even without user intent. It is also important to measure how effective mitigation strategies are. In many ways, the authors aim to have a complete discussion about the subject, which is highly positive.

I also liked that the authors included imagery in the paper, it is very important in this context.
summary: The paper presents a study of how easy is to make popular image-generation engines to create images with copyrighted characters. It explores indirect methods to make the engines produce those characters, including description of the characters and references. It also explores the effectiveness of different mitigation strategies. It suggests that it is fairly easy to make those engines to produce copyrighted characters, but the actual study results are not provided in detail.
weaknesses: I see two key problems with this paper which, for me, warrant it not to be accepted.

First, the authors use GPT-4V as the evaluator of their images. According to the appendix, the accuracy is only 82.5%, and the Kappa agreement with humans is merely 0.65. There are also problems with the methodology used in the human evaluation, but even dis-considering this problem, this characterizes as a quite flawed metric: it incorrectly evaluates 1 in 5 of images. This is never mentioned in the main paper, I had to dig through the appendixes to find it, and makes all the results questionable. Many of the differences depicted in figure 3 are indistinguishable if we consider an interval of plus/minus 20% of confidence in the metric. Similarly, all results in tables 1 and 2 are virtually identical if we consider that the metric has, at minimum, 20% of error. In other words, almost all of the results presented in the paper are not valid when we consider the problems with the metric.

Second, the presentation of the results of the study is improperly done. The baseline result of the paper, when character names are used, is described as "not too surprisingly, we have verified that when using character names, ~60% of tested characters can be generated." How does this break down among engines? A table describing the results would be nice (I tried to find in the appendix, but could not, if it is there please point specifically). Similarly, what the results are for 60-word and other indirect methods.

The latter problem can be fixed by better writing, but the former cannot without redesigning and redoing the study. That is the reason I recommend rejection of this paper.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.) In Figure 1, 'Gotham' is provided as an example of indirect anchoring. However, this feels somewhat like 'cheating' as Gotham is very well known as the fictional city that Batman watches over. For example, 'weak to kryptonite' does not mention Superman directly but kyptonite is a fictional material that only exists within the superman comic. Could this be considered a form of data leakage, and if so, what was the extent?

2.) How were the 50 diverse copyrighted characters chosen?

3.) Is there an available source for the claim that DALL.E leverages prompt rewriting? (Completely understand if this is not published but a citation here would be ideal)

4.) I'm a bit confused about negative prompts. My understanding is that this is just an instruction to avoid generating images that fit the description of the negative prompt, is this correct? 

5.) In footnote 5, the authors note that a slight misspelling of the character name often is enough to have the model avoid generating the character. Why was this not used as a baseline?
soundness: 2
strengths: The authors tackle a very important and rising issue in today's world with generative AI: Issues surrounding Intellectual Property(IP) and copyright in generated materials. Their approach for identifying 'indirect anchors' is well founded with both the 'embeddingsim' and 'co-occurrence' being interesting ways of investigating how the model 'thinks' about the copyrighted images and their attributes. While not having access to the method DALL-E uses to rewrite the prompt specifically, the authors do offer a reasonable alternative to allow them to perform a similar mitigation tactic for experiments with other models.
summary: The authors investigate how often image-generation models will generate copyrighted characters, directly or indirectly and evaluate the effectiveness of common mitigation approaches and preventing such copyrighted material from being generated. They present results claiming that it is fairly easy to both purposely and accidentally generate copyrighted characters and that current mitigation strategies are not sufficient to prevent this from happening.
weaknesses: A major weakness in the paper is how the image generation is evaluated whether or not to be copyrighted material with a lesser weakness being the diversity of the character pool.

The authors themselves discuss how 'legal judgements of copyright infringement are usually multifaceted and made on a case-by-case basis'. The alternative question "Is the generated characters so similar as to be recognized as an existing copyrighted character?" seems sufficient in a vacuum, however, does not account for peoples' satellite knowledge and potential misconceptions regarding the characters. Beyond just a binary rating on whether a character resembles another in the human evaluation, the confidence of the human evaluator in that evaluation also seems important.

The character pool also seems somewhat small and arbitrarily chosen. While they are definitely popular and trademarked characters (I myself recognize most of them), it would be nice if the authors could provide some concrete metric to ensure these are not just characters well known to those in the CS-field.

Meta-review: Interesting paper, interesting topic, well-written. The most severe criticism is that GPT-4V as the evaluator model, and that it does not have a very high accuracy. However, this does not change the relative ordering of the results, so the results are still broadly valid. Also, there does not seem to be any other way the evaluations could have been done without huge human effort.

Decision: Accept (Poster)