\copyright Plug-in Authorization for Human Copyright Protection in Text-to-Image Model
Authors: Chao Zhou
Huishuai Zhang
Jiang Bian
Weiming Zhang
Nenghai Yu
Venue: Accepted by TMLR ()

Abstract: This paper addresses the contentious issue of copyright infringement in images generated by text-to-image models, sparking debates among AI developers, content creators, and legal entities. State-of-the-art models create high-quality content without crediting original creators, causing concern in the artistic community and model providers. To mitigate this, we propose the ©Plug-in Authorization framework, introducing three operations: addition, extraction, and combination. Addition involves training a ©plug-in for specific copyright, facilitating proper credit attribution. The extraction allows creators to reclaim copyright from infringing models, and the combination enables users to merge different ©plug-ins. These operations act as permits, incentivizing fair use and providing flexibility in authorization. We present innovative approaches, ``Reverse LoRA'' for extraction and ``EasyMerge'' for seamless combination. Experiments in artist-style replication and cartoon IP recreation demonstrate ©plug-ins' effectiveness, offering a valuable solution for human copyright protection in the age of generative AIs. The code is available at \url{https://github.com/zc1023/-Plug-in-Authorization.git}

Published in Transactions on Machine Learning Research (02/2025)
Pl Authorization for Human Copyright Protection in Text-to-lmage Model
Chao Zhou"
se1 2e1666:1.0..00mmmauusst..et...
University of Science and Technology of China
Huishuai Zhang
shanghusshuni@hhhuccccc
Wangruan Institute of Computer Technology, Peking University National Key Laboratory of General Artificial Intelligence
Jiang Bian
jiabia@microsoft..com
Microsoft Research
Weiming Zhang
shangumm@ustc..cuu.cm
University of Science and Technology of China
Nenghai Yu
ynh@ustt.edu.co
University of Science and Technology of China
Reviewed on OpenReview: https: // openreview. net/ met/ forum? 40=408t1Fkpp
Abstract
This paper addresses the contentious issue of copyright infringement in images generated by text-to-image models, sparking debates among Al developers, content creators, and legal en- titics. Stato-of-tho-art models create high-quality content without crediting original creators, causing concern in the artistic communily and model providers. To mitigate this, we propose the @Plug-in Authorization framework, introducing three operations: addition, extraction, and combination. Addition involves training @plug-in for specific copyright, facilitating proper credit attribution. The extraction allows creators to reclaim copyright from infringing models, and the combination enables users to merge different Oplug-ins. These operations act as permits, incentivizing fair use and providing flexibility in authorization. We present innova- live approaches, "Reverse LoRA" for extraction and "Easy Merge" for seamless combination. Experiments in artist-style replication and cartoon IP recreation demonstrate Oplug-ins effec- tiveness, offering valuable solution for human copyright protection in the age of generative com otr.
Als. The code is available at https:
1 Introduction
Large foundation models Brown el al. Touvron (2023): Radford el al. (2021 b
OpenAl (023); Rombach et al. (022b) are trained with extensive, high-quality datasets like The Pile Gao et al. (0220). C4 Raffel
ct al. contain copyrighted human contents. At the same lime, these models not only excel at generating content
2020). LAION Schuhmann et al. (2022 and other enormous undisclosed data sources, which definitely
based on user prompts cha OpenAl (023); Rombach et al. (2022b): Ramesh et al. 2021 2022), but also have the potential of memorizing the exact training data thanks to the huge capacity in their gigantic numbers of
parameters Carlini et al. 20021 2023a
Such training procedure and utilization of AT models have sparked copyright infringement concerns among content providers, artists, and users. A notable instance is the lawsuit filed by The New York Times against
* "indicttes equal contribution. Corresponding Huishuai Zhang.

---

Published in Transactions on Machine Learning Research (02/2025)
OpenAI and Microsoft NYT alleging the unauthorized use of a vast number of articles for the purpose of training automated chatbots. The lawsuit seeks the destruction of the allegedly infringing chatbots and their associated training data. Similar concerns and legal actions are also emerging in the field of text-to-image generation law
Indeed, these concerns are well justified as these powerful models could disrupt the existing reward system in creative arts, adding anxiety to the content providers and artist community. The proficiency of Al in generating artworks that rival human creations is noteworthy, particularly in its ability to replicate characters from major intellectual properties (IPs). For instance, the use of stable diffusion models Rombach et al. (2022a), combined with controlled generation techniques like ControlNet Zhang et al. (2023b), enables users to effortlessly create well-known characters, such as those from Disney. This ease of replication significantly lowers the barriers to potential copyright infringement, raising concerns about increased piracy risks.
One debating point is whether using copyrighted matcrial to train machine learning models is prohibited by copyright laws. It is known that copyright does not ban all forms of copying or replication due to the fair use doctrine, which allows certain copying and distribution if it can be justified as fair use. It is not clear whether Al companies can successfully argue that their training procedures fall under this 'fair use' exception in copyright laws NYT law Additionally, academic research is underway to develop methods ensuring AI models do not generate copyrighted concepts, as seen in works like Vyas et al. 20233
In this paper, we step back and advocate to rethink the motivation for enforcing copyright laws. Copyright is a type of intellectual property that intends to protect the original expression of ideas in creative works, which can include literary, artistic, Article I, Scction 8, Clause 8 of the U.S. Constitution USC| is "To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries". The primary objective of copyright is to incentivize authors to create new works and to facilitate the dissemination of these works to the public by granting them property rights. However, existing generative Al models present challenges in appropriately attributing proper rewards to the copyright holders, which can significantly impact society. Artists, who depend on attribution for recognition and income, may be affected. Additionally, domain experts contributing to knowledge exchange websites like StackOverflow and Quora might hesitate to provide answers if they do not reccive reasonable rewards. This situation could backfire machine learning, as generative models might soon face shortage of fresh data due to reduced contributions from these sources.
musical forms copp The foundational goal of copyright laws, as stated in
User
Model Provider
rtist/PP Owner
Buee Model
Quacy beerzcd
Copesigian Data
Mode
Regiskkr
+ 7lug--a
Quey bose model Ergh dylegrn
plag la posl
phapetningg at sallives Cogli
Ciedit plag 1 usgg
Pay forplagen rgary
Figure 1: OPlug-in Authorization Process. The authorization process consists of three lypes of entities: user, model provider, and IP owners (artists). Users can generate copyrighted images only by accessing the relevant plug-in. The model provider offers services to users, tracks usage of plug-ins, and attributes rewards to the IP owner. The IP owners can achieve authorization by registering their Oplug-ins through addition or extraction. These Øplug-ins form a pool where users can get @plug-ins to produce content with the TP owner's authorization.
To address the attribution challenge in generative AI models, the concept of Stable Attribution Troynikov (2023) has been proposed, aiming lo credit ast and share revenue with creators based on their cibuutittonn. Specifically, Stable Attribution attempts to trace back an Al-generated image to the most similar examples in the training datasct. However, achieving this with reasonable cost and ensuring fairness is challenging, given the vast size and heterogeneous nature of the training set. Content providers and model owners may have completely different views on the evaluation of the content NYT While copyright data is unique, its impact.
2

---

Published in Transactions on Machine Learning Research (02/2025)
A ddition
Extraction
Combination
Coprightt
Midda
Bus Nodi
Boe
Torn Modil
Nuw
Nenwinelingrng Madal
aacdian in Vin Oogh
Fradioonss 0xgh
pl
rettaging ny
Plug-
Figure 2: Three foundational operations achieving OPlug-in Authorization: addition, extraction, and combination. The plug-in can be created by addition if the copyrighled work is new to the base model. Meanwhile, the plug-in can be created by extracting from the base model if the copyrighted work is already infringed by the base model. Once a pool of @plug-ins is constructed, the combination operation can merge multiple Oplug-ins ins featuring the generation of multiple concepts and leave a non-infringing model exchuding all multiple concepts.
on model performance may not be able to be fairly measured, especially when compared to the benefits derived from the vast size of open-domain public data. The value of artistic works might be underestimated by model owners. Addressing these issues is essential to create a balanced ecosystem in generative AI.
In addressing the challenge of practical altribution in generalive models, we introduce the "Copyright Plug-in Authorization" framework (see Figure [ designed to align with existing Intellectual Property (IP) management practices. This framework involves basc model providers, like Stability AI, functioning as repositories for copyright plug-ins. Copyright holders, such as artists, can register their works as plug-ins, receiving rewards for their use. End users, in turn, pay for the generation of images involving copyrighted concepts using these plug-ins. This system offers positive incentives for all involved: copyright holders are compensated for their creative contributions, end users can use copyrighted plug-ins ins without risking infringement, and base model providers profit from plug-in registration and model usage. The framework also facilitates explicit tracking of copyrighted work usage, ensuring a fair and straightforward reward system. By successfully implementing this authorization process, we can enable a more equitable distribution of copyright benefits across the generative model landscape.
Technically to enable effective and efficient copyright authorization, the plug-ins, as permits, should be easily created by addition if copyrighted works are new to the base models, by extraction if the copyrighted works are already infringed by the base model. Moreover, the plug-ins should be easily combined, which allows copyright holders to merge multiple plug-ins into a new one or enables end users to generate images with multiple copyrighted works. Mcanwhile, for efficient execution, these operations should be implemented as light, adaptations to the base model, e.g., parameter-efficient luning methods or prompt designs.
In this paper, we introduce three foundational operations- addition, extraction, and combination implemented using the Low-Rank Adaptor (LoRA) method Hu et al. (2022). These operations are essential for realizing the Copyright Plug-in Authorization (Figure for an overview).
It is noteworthy that Civitai civv represents a commendable attempt to instantiate the addition operation, as users can train and share LoRA components to generate corresponding figures. However, the operations extraction and combination are currently not publicly available and pose greater challenges.
The extraction operation involves separating the generative model into a non-infringing base model and some copyrighted plug-ins. A conventional approach might involve retraining the model from scratch using only non-infringing data, and then applying LoRA with copyrighted data. However, this method is impractical, if not impossible, due to high training costs and complex data-cleaning processes. Alternatively, this paper roduces a "Reverse LoRA" approach to extract a plug-in from an infringing base model. This process begins by capturing the target concept: we LoRA-tune the model on the target concept and then take the negative of the LoRA weights to achieve concept destruction. Then we finc-tune the LoRA on surrounding contexts to repair the non-infringing model's contextual generation ability. Finally, we reverse the LoRA to be the Oplug-in.
3

---

Published in Transactions on Machine Learning Research (02/2025)
The combination operation entails merging multiple copyrighted plug-ins into unified one. Simply adding these plug-ins together could lead to unpredictable results due to the correlation among copyrighted plug- ins. In this paper, we have successfully developed a method to fuse multiple components. We introduce "EasyMerge", a method termed "data-free ayer-wise distillation" for the combination process. Inspired by conditional generation in generative models, we utilize a LoRA component designed to learn the layer- wise outputs of @plug-ins under corresponding conditions. Consequently, the LoRA component can mimic the behavior of these @plug-ins when subjected to the corresponding conditions, effectively achieving the combination of Oplug-ins.
Our contributions are summarized as follows:
Conceptual contribution: A OPlug-in Authorization framework. We advocate to solve the problem of copyright infringement in foundation models with @Plug-in Authorization framework. It can offer fair and practical solution for the attribution challenge in text-to-image generative models. We further introduce three operations addition, extraction and combination to instantiate the framework with efficient human content copyright authorizations.
Technical contribution: A novel "Reverse LoRA" algorithm for extraction. It effec- lively extract copyrighted concepts from the base model, achieving competitive performance for concept extraction with flexible plug-ins.
Technical contribution: A novel Merge" approach for combination. It is a data- free laser-wise distillation approach, which effectively and efficiently address the challenge of combining multiple LoRA components.
The structure of the paper as follows. Section introduces the "OPlug-in Authorization" framework and delves into the three operations addition, extraction and combination. Section 3 presents experiments to validate the effectiveness of the proposed operations. Section 5 concludes the paper, offering discussion on the limitations of our work.
2 OPlug-in in Authorization with Addition, Extraction and Combination
As detailed in the Introduction, we implement the "OPlug-in Authorization" " by utilizing the publicly available pretrained diffusion generative model, Stable Diffusion (Rombach et al., 2022a), along with LoRA components (Hu et al. 2022). is important to note that our framework not confined to specific model structures, thereby facilitating compatibility with other foundation models such as the GPT series (Brown et al.). Additionally, it is capable of synergizing with various light fine-tuning or prompt tuning techniques (Li
& Liang 2021: Lester al. 2021 Edalati et al. 2022 Hyeon-Wor et al., 2021 Tn the subsequent, sections, revisit the fundamentals of diffusion generative models and introduce the three basic operations of the framework, along with our innovalive algorithms.
2.1 Preliminary Diffusion Generative Model
Diffusion models (Schl-Dickstein al. 2015 Song et al. 2020 Ho et al. 2020) are probabilistic models designed to learn a data distribution. Tn the forward pass, Gaussian noises are successively added T times to an image Xo, thereby creating a sequence Xo.....y that simulates Markov process. Conversely, the reverse process trains the model to denoise, effectively emulating the reversal of the Markoy Chain. New images are generated by initially sampling random Gaussian noises and then denoising them using the model. Importantly, this process can be conditioned on inputs, such as a prompt text c. The denoising process, denoted as P(w) (Xe, e, (), is trained to predict the noise under the textual prompt c any timestep / E [0, T], outlined in
(X,,
arg min Ee. (Xe, (We,, el
(1)
Recent advancements have introduced latent diffussio models as a solution to mitigate the drawbacks associated with evaluating and optimizing models in pixel space, such low inference speed and high training
4

---

Published in Transactions on Machine Learning Research (02/2025)
costs. These latent diffusion models operate within compressed latent space, exemplified by publicly available models such as the Stable Diffusion Model (SDM), as detailed in Rombach et al. (2022a). The SDM architecture consists of a variational autoencoder (VAE) that maps images to latent space, a U-Net that learns the diffusion process, and a CLIP encoder for text embedding. Our work primarily focuses on the attention structure within the U-Net, which has been identified as the most influential component in diffusion models.
To implement the "OPlug-in Authorization", incorporate three foundational operations into the Stable Diffusion Model (SDM): addition, which allows copyright owners to add plug-in for their works; catraction, which enables owners to extract plug-in from an infringing base model; and combination, which permits users to merge plug-ins for multiple copyrighted concepts. The addition operation employs LoRA components that are added to SDM's attention matrices, and these are then trained with copyrighted data. While the specifics of the addition operation are covered on existing model-sharing platforms like civ our discussion will primarily focus on the extraction and combination operations in the subsoquent sections.
2.2 Extraction: Reverse LoRA
DE-CONCEPT
RE-CONTEXT
he painting of building
painting
building
model
model
X
LoRR
Loes
-
Jax
a
(O
X
((0,))
Image with prompt "I painting building by Picasso"
Image with prompt pointing of building"
Figure 3: The method extraction consists two steps:de-concep and re-contert. The de-concept step tries to capture the larget concept "Picasso" by luning the LoRA component to match copyrighted images with the contextual prompt "The painting of building". In the re-context step, we reverse the LoRA (so that successfully forget "Picasso") and then further tune the LoRA with surrounding contextual prompt and non-copyrighted image pairs, to ensure the capabilities of contextual generation.
In our endeavor achieve extraction, we introduce a method known as "Reverse LoRA". This approach encompasses two crucial steps to effectively extract the target copyrighted concept while maintaining the ability to generate contextual concepts. The initial step, termed the de-comcepl step, involves removing the target concept from the base model. The subsequent step is a counterbalance to the de-concept step, involving relearning the surrounding semantic context, referred to as the re-conlezt step. Figure 3 illustrates these two steps in the context of extracting the target concept "Picasso".
2.2.1
Stepl: De-concept
Our goal to extract Picasso-related information from the base model to a copyright plug-in using LoRA (Wz). This involves identifying the information that represents "Picasso" via an alignment process and documenting the changes in model parameters that occur during the alignment process.
Specifically, align copyrighted image generation, e.g,, images of "the painting of building by Picasso", with the prompt without copyrighted text, e.g., "the painting of a building" in the base infringing model. The alignment objective is mathematically expressed as follows:
(w) c², ,()] =E (ww+z) (e, (c,c,
(2)
5

---

Published in Transactions on Machine Learning Research (02/2025)
where @ is the denoising function, denotes the original network parameter, wz is the LoRA component, is the prompt "the painting of a building", ee" is the prompt "the painting of a building by Picasso", is the initial noise, and f is the sampling timestep.
To achieve the alignment as defined in (2), we optimize the following objective function with respect to the LoRA parameters Wz, while keeping all other model parameters frozen. The objective function is defined as follows:
arg min Ex,x .c, Wr.
Il (w (XT, - ell
(3)
where X* the copyrighted image (or generated by the infringing model with the prompt "the painting of Vl-ate the noisy version of X*, c is the prompt of "the painting of building"; is the original network and wr, is the LoRA weight. This optimization aims to adjust wn, that it effectively captures the desired information related to the target concept.
building by Picasso"), X; vaiX* + v11
By incorporating such a LoRA component, the base model can generate Picasso-style images even when the prompts do not explicitly mention "Picasso". Hence, the LoRA represents the copyrighted Picasso style, and w - wp would give us a non non-infringing model, which can thought of as an analogy of a 'negative LoRA', However, directly using w wp as the non-infringing model compromiscs its ability to generate images with surrounding context, e.g., "the painting of a building", as shown in Figure in Appendix A.1 This observation leads us to further tune the LoRA with pairs of images and texts of surrounding semantic context.
2.2.2 Step2: Re-context
To mitigate the performance degradation of the non-infringing model when generating images with contextually related prompts, we introduce re-context step following the de-concept step. This step involves fine-tuning the LoRA component with images and textual prompts of surrounding contexts, e-g., "the painting of a
building". To curate the dataset, we randomly generate images with the base model using the contextual prompt "the painting of a building", while leveraging the negative prompt (Ho & Salimans 2022) "Picasso" to steer the generation as far away from the larget concept "Picasso" as possible.
Specifically, we further optimize wt with the objective,
arg min E,,.,c,ll (w m (Xe,, ccl,,
(4)
where X, c represent the constructed pairs (image, prompt) to recover the generation capability of surrounding contexts.
Overall, after the de-concept step, the model wr is unable to generate images in the Picasso style, yet it performs well with surrounding prompts thanks to the re-context step. Therefore, through the extraction operation, we obtain a non-infringing model w w - wy and a Oplug-in wz. By incorporating the Oplug-in in, the model is restored to the original base model we, regaining the capability of successfully generating the artworks in the "Picasso" style. The intermediate results of the extraction process are visually illustrated in Figure 7 in Appendix A. showcasing the successful extraction of the targeted copyright, while preserving the model's ability generate images with surrounding contexts.
2.3 Combination: EasyMerge
In this section, we consider the operation combination. The combination of existing Oplug-ins becomes essential when aiming to generate an image featuring both "Snoopy" and "Mickey" concepts.
It is worth noting that simply adding these plug-ins together could yicld unpredictable outcomes due to inherent correlations among these plug-ins. To facilitate the combination of multiple copyrighted concepts, we propose a novel approach named EasyMerge. This method employs a data-free, layer-wise distillation technique that only requires plug-ins and corresponding text prompts. Furthermore, with layer-wise distillation, EasyMerge achieves efficient combination in just a few iterations. The versatility of EasyMerge extends beyond the current, context, potentially also applicable in other scenarios like continual learning.
6

---

Published in Transactions on Machine Learning Research (02/2025)
Specifically, we use new LoRA component wz to mimic the functionalities of each plug-in that needs to be combined. The objective is defined as follows:
arg min
Σ RES.JESt
Be.llldow, (s,ce, ou- (e, e)1?"
(5)
WL
where S is the set of text prompts to be combined, St is the set of layers that are added with LoRA components, and O¹ is the output of layer j's LoRA component. Similarly to the previous section, w denotes the base model parameter, wt, denotes the combined plug-in, d denotes the prompt h; WL, is the plug-in of context C₁2 E is initial noise and the sampling timestep of the diffusion process. The non-infringing model Wr, is that simultancously excludes multiple styles related ck: which is called the combination of estraction. Algorithm 1 describes concrete steps of optimizing the objective 5
Algorithm 1: Combination: Easy Merge method
Input: set S of indices of plug-ins be combined, base model w, diffusion stop Output: Combined LoRA wz repeat
for WL,&G G S do Uniform([1..T7 6 <~ NN(00, AddHooo((L.))
// Capture input le, and output O for each layer j // Denoise obtain features // Get layer-output through new LORA
Ror." O'n t Pwtwe, (E,
Oh, + Pbg (T
C Σjest 10 mz
- 0 Oie,
WL. Wt. VuzCe L;;
end
until converyence;
3 Experiments to Verify Basic Operations
As position paper, regard primary contribution the proposal of the copyright authorization framework. Nonetheless, also aim to validate the practical effectiveness and efficiency of the proposed basic operations. Given that the addition operation has already been well demonstrated by existing practices, we focus on evaluating the extraction and combination operations, We choose two typical scenarios of copyright. infringement: artist-style replication and cartoon intellectual property (IP) recreation.
3.1 Experiment Setup, Metrics and Baselines
Experiment Setup. In all experiments, we fine-tune the attention component in the U-Net architecture of Stable Diffusion Model v1.5, as described in Rombach et al. (202aa).
For the estraction operation, we need to generate data with pre-trained models. For the case of extracting a given artist style, we leverage ChatGPT (cha) to generate 10 common content paintings. In the de-concept. step, for cach itcration, we select one of these content to generate 8 images with prompts The painting of (content] by [artist]". Similarly, during re-context step, we select. imagery generate images with prompts The painting of [content]" while using negative prompts "by [artist]". For the case of extracting particular IP character, it follows the same procedure as above except that the prompts become "The cartoon of the IPP character]" for the de-concept process and "The cartoon of the [character]" for the re-context process, respectively. For both the de-concept. process and the re-contex: ext process, the training consists of 10 iterations, with each iteration 30 epochs. We use a learning rate of 1.5e-4, T 50 steps for the diffusion process, and a rank of 40 for LoRA.
For the combination operation, learning rate of 1e-3 and rank value of 140 for LoRA.
7

---

Published in Transactions on Machine Learning Research (02/2025)
Metric. To evaluate the effectiveness of the extraction operation, measure the discrepancy between the set of images generated by the base model and that generated by the non-infringing model after extraction with the same set of prompts. We want to observe large discrepancy when the prompts are with target concepts while having a small discrepancy with surrounding concepts. This means that the extraction operation achieves its goal: the non-infringing model cannot generale images with larget concepts but can generate high-quality images with surrounding prompts.
We acknowledge that for image generation tasks, the ultimate evaluation criterion is human judgment. Therefore, we provide the generated images from various scenarios for readers' assessment. Nevertheless, to reduce costs and facilitate comparisons with existing approaches, we also employ an objective metric known as the Kernel Inceplion Distance (KID) (Biíkowski et al. 2018)) to quantify the aforementioned discrepancy. KID is akin to the Fréchet Inception Distance (FID) (Heusel et al. 2017) but is considered to exhibit less bias and possess asymptotical normality. Moreover, we also employ the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.) 2018) to quantify the discrepancy of artistic style artworks. LPIPS is robust measurement tool that effectively caplures differences in human perception between two images, offering comprehensive evaluation of stylistic variations in generated artworks.
Baseline. We compare our extraction operation with the concept ablation approach (Kumari et al. 2023 and Frased Stable Diffusion (ESD) (Rohit. Gandikota and Joanna Materzyíska and Jaden Fiotto- Kaufman and David Bau 2023), which achieve concept removal by aligning latent representations of target concepts with those of anchor concepts.
In general, we find it hard to compare the results with existing methods because of the complex setups in image generation, e.g., the tuning steps and the tradc-off between removing the target concept and keeping the surrounding concept. Therefore we take conservative approach and only consider the generation with similar scenarios and the same metric as in the original paper.
3.2
Extraction and Combination of Artists' Styles
Non-infringing Model
Non-infringing Model bastAl
Plug-in Integration
Base Model
Ddoet "Ventagg
Dilact Those"
Indect "Maeet
bageo Coigh"
xteegat "Phasor
Haugele "Maee"
Sathwm
Pronig Conditiooal Sample
Prongeh Craclteznd Sample
110 pus
(a) Results of extraction in style replication
(b) Results of combination in style replication
Figure 4: Results of style replication. In Figure (a), we show samples from different non-infringing models in each column. Each non-infringing model exhibits a deficiency in one style generation ability, with all other style generation capabilities remaining unaffected. In Figure (b), we present samples generated after integrating certain @Plug-ins in each column. Each of these @Plug-ins serves to exclusively restore the generation of one particular style, while the generation of other styles continues to exhibit diminished performance.
Extraction. We extract artist styles from the Stable Diffusson V1.5, referred to as the "base model". We consider three renowned artists: (1) Vincent van Gogh, (2) Pablo Ruiz Picasso and (3) Cicar-Claude Monet. The results of individual style extractions are visually presented in Figure 4a). These images showcase outputs generated by both the base model and the non-infringing model, encompassing both the target styles and surrounding styles.
8

---

Published in Transactions on Machine Learning Research (02/2025)
In Figure 4 images within red boxes represent the target styles, while the rest images embody surrounding styles. A notable contrast is observable between the images within the red boxes and those generated by the base model. However, the images representing surrounding styles exhibit a substantial similarity to those generated by the base model. This demonstrates the success of the extraction operation in isolating the target style from the base model while preserving the quality of images with surrounding styles.
Table 1: Quantitative comparison with basclines in artist-style extraction. Compared Concepts- Ablation, ours extracts the target style more thoroughly, and compared to ESSD, ours enjoys less damage to surrounding styles.
Metrics
Methods
Target style
Surrounding style
KIDx10"
EXTRACTION (Ours)
187
32
CONCErTS-ABLATION
42
12
EXTRACTION (OuRs)
0.31
0.14
LPIPS
ESD
0.38
0.21
we employ quantitative metrics evaluate the effectiveness of the extraction operation comparison to bascline methods. Our method demonstrates notable improvements, as indicated by the KID metric increasing from 42 to 187 for the larget style when compared to Concepts-Ablation (Kumari et al. 2023). This increase indicates an enhanced removal of the target style. Additionally, in comparative assossment with the Erasing method (Rohit Gandikota and Joanna Materzyíska and Jaden Pootto-Kaffman
In Table T
and David Bau 2023), our method achieves a reduction in LPIPS from 0.21 to 0.14 for surrounding styles. This reduction implies less degradation of the surrounding artistic styles, affirming our method's ability to preserve the quality of generated images when using surrounding style prompts.
Combination. In this part, we show the effectiveness of the combination operation for extracting multiple artist styles and then adding them back with corresponding plug-ins.
Given three artistic styles of Van Gogh, Picasso, and Monet and their Oplug-ins, first extract these three styles from the base model, producing a non-infringing model, which is illustrated in the leftmost column of Figure bb). Notably, all the images generated by the non-infringing model significantly differ from those generated by the base model in the leftmost column of Figure aa). This underscores the efficacy of the combination of multiple extraction operations.
We then individually integrate each style copyright plug-in into the non-infringing model. The images highlighted within red boxes represent the target style achieved after integrating the respective copyright plug-in. Notably, the target style images after integration are distinetly different from those produced by the non-infringing model, showing a closer resemblance to the images generated by the base model. This obscrvation indicates that the copyright plug-in can reinstate the modol's ability to create artworks in the target styles, without infringing upon the copyright restrictions associated with other artistic styles.
3.3 Extraction and Combination of Cartoon IPs
In the context of intellectual property (IP) recreation, we demonstrate the capabilities of our framework through both extraction and combination operations. Specifically, Figure 3 displays the outcomes of extracting three iconic TP characters: Mickey, R2D2, and Snoopy. The images framed red boxes were generated by the non-infringing model using prompts specific to the target IP, after extraction. These images significantly deviate from those produced by the base model. In contrast, images outside the red boxes, which represent. other IPs, show resemblance to those generated by the base model, indicating the targeted nature of the extraction process.
Our approach to IP extraction effectively isolates the specified IP, ensuring that the generation capabilities for other IPs remain intact. The efficacy of our extraction method in the realm of IP recreation is quantified in Table 2 where we document notable improvement, i.c., approximately a 2.6-fold incrcase in the Kernel Inception Distance (KID) metric for the targeted IP, while the KID metrics for other IPs remain relatively stable.
9

---

Published in Transactions on Machine Learning Research (02/2025)
Non-infringing Model
Base Model
Extract "Mickey"
Extract "R2D2"
Extract "Snoopy"
Prompt Conditional Sample
Mickey
R2D2
Snoopy
Figure 5: Results of IP Recreation. Each column on the right represents the output of a distinct non-infringing model. We successfully extract the unique IPs of Mickey, R2D2, and Vader independently, preserving the generation of other IPs.
Additionally, comprchensive large-scale experiments demonstrate that the extraction process does not impact the model's ability lo generate non-IP-related content. The performance of the model after extraction remains consistent with routine or everyday use cases, as further detailed in the appendix A.2 This ensures that while the model respects copyright constraints by effectivoly removing or isolating specific IPs, it does not compromise on its general utility or the breadth of its creative outputs.
Table 2: Quantitative comparison in IP recreation. We increase the KID of the target IP about 2.6 times compared with Concepts-Ablation while keeping the KID of the surrounding IP on par.
Metrics
Methods
Target IP
Surrounding
3IDx10 EXTRACTION (OURS)
131
17
CONLTIOS-ABLLTTIO
Furthermore, we demonstrate the combination of multiple IP @plug-ins, as illustrated in Figure 6 The initial image is produced by the non-infringing model after extracting Mickey Mouse and Darth Vader, where the IPs are hard recognize. Subsequent images, the second and third, are created after individually adding the Mickey and Vader @plug-ins, which distinctly feature the respective IP. The final image is generated upon
10

---

Published in Transactions on Machine Learning Research (02/2025)
adding the combined Oplug-in, successfully displaying both IPs. This procedure underscores the plug-in's efficacy in selectively and collectively restoring the model's ability to generate IP-specific content.
Combined extraction
Mickey Integration
Vader Integration
C Combinated Integration
Prompt
The cartoon of Mickey and Vader
Figure 6: IP Recrcation in a single image. We can integrate Oplug-in into the non-infringing model to generale either Mickey or Vader in a single image or integrate the combined Øplug-in to generate both of them.
These results indicate the efficacy of combining multiple extractions, where the non-infringing model's ability to produce images themed with either Mickey Mouse or Darth Vader disabled. With the integration of the respective copyright plug-in, the model regains the capability to create content related to the spocific IP.
4 Related Work
To position our work in the vast literature, we review related work through two perspectives: scope and technique. It is worth noting that some of the literature touches both sides and organize them in way most related to ours.
4.1 Scope Related: Copyright, Data Contribution and Credit Attribution
Recent text-to-image generative models trained with large-scale datasets (Schuhmann et al. 2022 Liu et al. 2022), which cannot be guaranteed free of copyrighted data. At the same time, the state-of-the-art models are capable of generating high-quality and valuable creative images comparable to human creators or even memorizing the data points in the training set (Carlini et al. 2023b), which arouses copyright concerns about the training data and brings anxicty to the artist community.
Numerous efforts have been made for copyright protection of training data (Zhong et al. 2023). direct approach removing the copyrighted images from the training set, which may involve cumbersome costs due to the size of the training sets and may significantly degrade the model performance (Feldman 2020 Another direct approach is post filtering, refusing to generate images with copyrighted concepts, e.g., Schramowski et al. (023) proposes Safe Latent Diffusion to guide latent representation away from target concepts in the inference process, which nonetheless can be bypassed by a user with access to the model (Rando et al.) 2022). As an example, OpenAl Dall-E3 (OpenAl 2023) declines requests for generating an image in the style of a living artist and promises that creators can also opt their images out from training of future image generation models. Many papers discuss the idea of concept removal, which will be reviewed in later section.
Shan al. (2033 propose Image Cloaking that suggests adding adversarial perturbations before posting artistic works on the internet so as to make them unlearnable for machine learning model, which has been pointed out to be hard to defend against future learning algorithms (Radiya- et al. 2021
Theoretically, Bousquet et al. 2020): Elkin-Koren et al. (023) connect the copyright protection of training data with the concept of differential privacy and discuss their subtle differences. Vyas et al. (2023) further formulate the copyright problem with a near free access (NAF) notion bound the distance the generative distributions of the models trained with and without the copyrighted data.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Our paper distinguishes largely from all previous works as do not try to prohibit generating copyrighted concepts but instead we introduce copyright authorization for the generative model to reward the copyright owners with fairness and transparency, From this aspect, our paper is also related with literature of monetizing
the training data (Vincent & Hecht 2021 Vincent et al. 2021 Li et al. 2022b a or attributing credits for the generative contents (Troynikov]
2023
but we establish a very distinct way to reward the authorship.
Heated discussion is also around the copyright for Al generated artwork Franceschelli & Musolesi (2022); Abbott & Rothman (022). The Review Board of the United States Copyright Office has recently refused the copyright registration of two-dimensional Al generated artwork entitled "A "A Recent Entrance to Paradise", However, Abbott & Rothman (2022) argues for giving the copyright to AI generated works, which will encourage people to develop and use creative AI, promote transparency and eventually benefit the public interest.
4.2 Technique Related: Concept Removal, and Negative Sampling
Our extruction operation is closely related with the concept removal for generative models. Rohit Gandikola and Joanna Materzyúska and Jaden Fiotto-Kaufman and David Bau (2023); Kumari et al. (2023) remove target concepts by matching the generation distribution of contexts with target concepts and that of contexts without target concepts. Zhang al. (2023a) forget target concepts by minimizing the cross attention of larget concepts with that of larget images. Heng & Soh (2023) leverage the reverse process of continual learning to promote the controllable forgetting of target contents in deep generative models.
2022) can also prevent generating certain concepts. Specifically, end users can set conditional context and negative context to guide the diffusion process generate images conforming the conditional context while being far away from the negative context. Only negative sampling cannot stop copyright infringing generation because the contexts are set freely and adversarially by end users.
We note that negative sampling (Ho & Salimans
In contrast, for a specific copyrighted concept, our extraction operation takes an "Reverse LoRA" approach to disentangle the base model into two part: non-infringing base model and the plug-in LoRA component for copyrighted concept. Specifically, we use negative sampling to generate non-infringing images, which serves as training data for copyright plug-in. From the aspect of parameter efficient. fine-tuning, our paper relatod
with literature (Alaluf al. 2022 Ruiz al.
2023 Gal ct al. 2022 Hu et al.] 2022 Huang al. 2023).
Our combination operation is related with the widely studied "knowledge distillation" (Liang et al. 2023 Lopes et al., 2017 Sun et al. 2019 Hinton et al. 2015 Fang et al. 2019), but entails large difference from previous work. We combine multiple copyright plug-ins that are LoRA components for different targets, and we take data free approach due to practical constraint.
5 Discussion, Open Questions and Limitations
The growing concerns regarding generative AT models stem from their capacity to produce copyrightt-iffrnging content. This issue becomes more pronounced as state-of-the-ae the-art models continue to improve the quality of generated images, often without adequately acknowledging the contributions of human content creators. In response, we propose the "Copyright Plug-in Authorization" framework to address these societal worries, drawing inspiration from the purpose of copyright law. Our approach demonstrates that copyrighted data can be incorporated into LoRA plug-ins, enabling straightforward tracking of usage and equitable distribution of rewards.
A key challenge for this framework is the efficient management of large number of plug-ins, which is essential to ensure user-frieudly access to specific generations. Moreover, updating the base model poses another challenge, as retraining the entire suite of plug-ins can be costly, raising the issue of ensuring backward compatibility. One limitation of our current rescarch is the potential degradation the performance of the non-infringing model due a large number of extraction operations, factor that has yet to be thoroughly investigated.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Acknowledgement
We would like to express our sincere gratitude to the anonymous reviewers for their constructive suggestions, which have significantly improved the quality of this manuscript. This work of Huishuai Zhang was supported (in part) by the State Key Taboratory of General Artificial Intelligence.
References
NYT Complaint Dec2023. http:////wyyoooooc..........eeep Comppp/iii/////TTTmmmaaaiaiintt Complaint, Deeettt_eee223eeeee pdf
Constitution of the United States.
ChalGPT. https::/chat. com/
Civitai. https://civitai.com
Copyright. http:///ww...iiippeaaa..or//////
AI Art Generators Spark Multiple Copyright Lawsuits. htttg::/www.bollyyoorrrpprppp com
Ryan Abbott and Elizabeth Rothman. Disrupting creativity: Copyright law the age of generative artificial intelligence. Florida Law Review, Forthcoming, 2022.
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with rrpernelworks for real image editing. In Proceedings of the IEEE/CVE conference on compuler Vision and pattern recognition, pp. 18511-18521, 2022.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Olivier Bousquet, Roi Livni, and Shay Moran. Synthetic data generators sequuettai and private. Advances in Neural Information Processing Systems, 33:7114 7124, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901. Curran Associates, Inc.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633 2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models. International Conference on Learning Represen- lalions, 2023a.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium Caniiiummmuse (USENIX Security 23), pp. 5253-5270, 2023b.
Ali Edalati, Marzieh Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXi::2212.106: 106500 2022.
Niva Elkin-Koren, Uri Hacohen, Roi Livni, and Shay Moran. Can copyright be reduced to privacy? arXic preprint arXiv:2305.14 14822, 2023.
Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation. arXiv preprint arXiw:1912.11006 2019.
Vitaly Feldman. Does learning require memorization? short tale about a long tail. In Annual ACM SIGACT Symposium Theory of Computing, 2020.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Giorgio Franceschelli and Mirco Musolesi. Copyright in generative deep learning. Data 8 Policy, 4:c17, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashmik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing sotatto-mmage generation using textual inversion. arXiv preprint arXiv:2208.0100 016618 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arxiv preprint ar 00..000110007, 2020.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. arXiv preprint rrXii:2355100100 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503. 02031, 2015.
Jonathan Ho and Tim Salimans. Classifier-fres diffusion guidance. arXin:2207.15500 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840 6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and
Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference on Leurning Representations, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient. cross-task generalization via dynamic lora composition. arXiv preprint 13X2092307.13269, 2023.
Nam Hyeon- Moon Ye-Bin, and Tac-Hyun Oh. Fedpara: Low-rank hadamard product for communication- efficient federated learning. In International Conference on Learning Representations, 2021.
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts text-to-image diffusion models. In ICCV, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for eaffmetterefffiiett prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), EMNLP '21. Association for Computational Linguistics, 2021.
Hanlin Li, Brent Hecht, and Stevie Chancellor. All that's happening behind the scenes: Putting the spotlight on volunteer moderator labor in reddit. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pp. 584-595, 2022a.
Hanlin Li, Brent Hecht, and Stevie Chancellor. Measuring the monetary value of online volunteer work. Proceedings of the International AAAI Conference on Web and Social Medio, volume 16, pp. 596-600 606, 2022b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), ACI-LCCNL 221, pp. Association for Computational Linguistics, 2021.
Chen Liang, Simiao Zuo, Qingru Zhang, Pengeheng He, Weizhu Chen, and Tuo Zhao. Less more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pp. 20852 -00677. PMLRR 2023.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiang Wu, and Jinqiao Wang. Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training. Advances in Newral Information Processing Systems, 35:16705 16717, 2022.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arviv preprint orXii:1710000555 2017.
OpenAL. Dall-E 3. pttps:///ppenat.cmmmaall//
2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xix preprint arXiv:2304.07199 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Evani Radiya- Dixit, Sanghyun Hong, Nicholas Carlini, and Florian Tramer. Data poisoning won't save you from facial recognition. In International Conference on Learning Representations, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Reseorch, 2115555485-5551, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 061125 1(2):3 2022.
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer. Red-teaming the stable diffusion safcty filter. In NewrIPS ML Safety Workshop, 2022.
Rohit Gandikota and Joanna Materzyúska and Jaden Kaaito-Kaufman and David Bau. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image syuthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion (CVPR), pp. 10684-10695, June 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference. on computer vision and pattern recognition, pp. 10684 10695, 2022b.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting, Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open largo-scale dataset for training next generation imago-text models. Advances in Neural Information Processing Systems, 85:25278-25200 2022.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y. Zhao. Glaze: protecting artists from style mimicry by text-to-image models. In Proceedings of the 32nd USENIX Conference on Security Symposium, SEC '23, USA, 2023. USENIX Association. ISBN 978- 178-1-939133-37-3
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference machine learning, pp. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2020.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4323 4332, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. aixis preprint orX0::2307.00258, 2023.
Anton Troynikov. Stable Attribution. http:://ww..ttabbeetrrriiittttot com 2023.
Nicholas Vincent and Brent Hecht. A deeper investigation of the importance of wikipedia links search engine results. Proceedings of the ACM Fluman-Compule Interaction, 5(CSCW1):1-15, 2021.
Nicholas Vincent, Hanlin Li, Nicole Tilly, Stevie Chancellor, and Brent Hecht. Data leverage: A framework for empowering the public in its relationship with technology companies. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 215-222222 125-227,- 2021.
Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models. arXiv preprint Xiv:22302. 10870, 2023.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint Xiv:22303 1 17591, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In IEFF International Conference on Computer Vision (ICCV) 2023b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018.
Haonan Zhong, Jiamin Chang, Ziyue Yang, Tingmin Wu, Pathum Chamikara Mahawaga Arachchige, Chchara Pathmabandu, and Minhui Xue. Copyright protection and accountability of generative ai: Attack, watermarking and altribution. In Companion Proceedings of the ACM Web Conference 2023, pp. 94 98, 2023.
16

---

Published in Transactions on Machine Learning Research (02/2025)
A Appendix
A.1 Intermediate results of extraction
We present an intricate analysis of the extraction process, clucidating distinct phases that delincate the evolutionary Irajectory of the non-infringing model's performance. Following the de-concept slep, the model experiences a transient phase marked by a temporary impairment in its ability generate semantically meaningful images. Subsequently, the re-context step engenders a noticeable restoration, enhancing the model's proficiency in generating semantically rich images. Importantly, despite this recuperation, the model retains its inherent limitation - the incapacity to generate artwork in the distinctive style synonymous with Picasso. This observation underscores the success of the de-concept step, wherein the LoRA component effectively captures and removes the target concept, leading to the temporary impairment in the non- infringing model. The subsequent re-context step rectifies this performance decrease without reintroducing any information about the target concept. In essence, the extraction process successfully achieves the targeted concept extraction.
Prompt
Base Model
After De- Doncept
After Re contextt
The painting of building by Picasso
The painting of building by Tyler Edlin
Figure 7: Intermediate results of estraction. After the de-concept step, the non-infringing model's gencrative abilitics become significantly limited, predominantly manifesting as the production noise. After the re-context slep, the generation's prowess is rejuvenated, but due to the absence of learning Picasso-style images, the model remains unable to generate artwork in the style of Picasso.
A.2 Experiment on ordinary objects generation
We evaluated the influence of extraction on the generation of ordinary objocts. We utilize 5000 textual captions selected from the validation in MS-COCO (Lin al.) 2015)) as prompts, generating 5000 images using SD1.5 and the non-infringing model that extracts R2D2 and Picasso, respectively. Several randomly selected images are displayed in Figure S For illustrative purposes, we also generated results for concept-ablation and ESD on MS-COCO, respectively. Images within the same column exhibit substantial similarity, indicating that extraction does not exert. an impact on the generation ordinary items.
As depicted Tablc we calculate some quantitative metrics like FID and KID. noteworthy that Concept-Ablation Kumari et al. 2023) only releases the checkpoint of ablating "R2D2" and ESD (Rohit)
Gandikota and Joanna Materzyáska and Jaden Fiotto-Kaufman and David Bau 2023 only releases the
17

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Quantitative results on MS-COCO. FID and KID metrics for removing the Picasso style are presented the upper two rows, while those for removing R2D2 are displayed in the lower rows.
Domain
Method
FID
KIDx 10
Extract Picasso
24.04
2.83
Style replication
Erase Picasso
25.20
3.39
Extract R2D2
20.55
2.36
IP recreation
Ablate R2D2
18.97
1.34
'A yellow bus with tinted windows driving uphill down street.'
black Honda motorcycle parked in front of garage."
Two stuffed animals table with honey."
'an imaage slicing small pizza.
'a small kitten inside a laptop on the floor."
SD1.5
Extract
R2D2
Ablate
R2D2
Extract
Picasso
Erase Picasso
Figure 8: Ordinary objects generation after extraction. Row displays images generated by Stable Diffusion V-1.5. Rows and illustrate images generated after the removal of the IP character R2D2, while Rows 4 and 5 showcase images generated after the elimination of Picasso's style. Rows 3 and serve as the bascline, representing concopt-ablation and ESD, respectively. Notably, after the extraction of R2D2 and Picasso, the non-infringing model retains the capability to generate commonplace objects sourced from the MS-COCO datasct [Lin et al. 2015
18

---

Published in Transactions on Machine Learning Research (02/2025)
checkpoint of erasing "Picasso". Thus, we compare them separately by using their respective checkpoints. Evaluation metries consistently maintain low values, further affirming that extraction does not compromise the generation of ordinary items.
A.3 More Quantitative Results
We compare the results of three methods following the removal of Van Gogh's influence. Like LPIPS, DINO-v2 Oquab et al. (2023) is self-supervised model used for extracting visual features from images, allowing to compute the similarity between generated images generated by the non-infringing model and the base model. The higher the value of DINO-v2, the closer the image is. Also, we calculate the CLIP-t distance to measure the semantic similarity between images generated by the non-infringing model and their corresponding textual prompts. The higher the CLIP-t value, the closer the sematic similarity is.
To ensure fairness in our evaluation, we implemented various settings to control the extraction effect and align it closely as possible with the surrounding style. The quantitative results, as detailed in Table ! demonstrate that method achieves superior removal of the target style while preserving the integrity of the surrounding style.
Table 4: Quantitative comparison of different methods. The symbol 1 indicates that the higher value is better on the metric, whereas + symbol signifies the lower value more preferable. The colors of symbols f and & should be read according to the colors of the most left column (Style).
Style
Methods
KID x10
LPIPSN
DINO-v2%
CLIP-t%|1
ESD
138
0.385
46.06
26.9
Target Style
EXTRACTION (OURS)
187
0.387
38.8
23.9
ESD
27
0.212
71.42
32.15
Surrounding Style
EXTRACTION (OURS)
32
0.157
79.02
31.9
CONCEPT-ABLATION
42
0.255
52.1
28.1
Target Style
EXTRACTION (OURS)
58
0.293
48.6
28.2
CONCEPT-ALLATION EXTRACTION (OURS)
12
0.123
81.2
29.6
Surrounding Style
7.4
0.115
86.2
32.8
A.4 Experiment on seen and unseen contents generation
In extraction, we sample 10 common contents (training set) leveraging ChatGPT fine-tune the base model. These contents have been previously processed by the non-infringing model. Additionally, generate 10 supplementary contents for evaluation. Figure g shows the images generated with these 20 contents. The images on the left represent the seen contents (training set), while those on the right are the unseen contents (evaluation set). Within each image block, we extract the corresponding artistic style. The top row is generated by the base model, wheras the bottom row is generated by the non-infringing model. All images are generated with the prompt "[content] by [artist]". The specifics of the artistic and contents are detailed Table 5
To evaluate the performance of the base model and the non-infringing model, computed various image distance metrics, including KID, LPIPS, and DINO-v2, between images generated by these two models. Additionally, we assessed the CLIP-t distance to measure the semantic alignment between textual prompts and images produced by the non-infringing model.
The quantitative results, averaged on the seen and unseen contents separately, are summarized Table 0 The lack of significant differences across all metrics for the seen versus unscen content suggests that the model did not exhibit signs of over-fitting
19

---

Published in Transactions on Machine Learning Research (02/2025)
condents
Creem contentss
Vann
Figure 9: Scen and Unscen contents generation after extraction. On the left are 10 contents already seen during the extraction, while on the right aren't 't seen. Within each image block, the top row generated by the base model, and the non-infringing model generates the bottom one. Zoom in for better visualization.
A.5 Extracting specific prompt v.s. extracting style
Figure [0 compares the outputs the base model and the non con-infringing model under two different extraction sconarios, extracting specific prmpt "sunflowers by van gogh" and entire style 'van gogh'. For each scenario presented, the left column exhibits images generated by the base model, whereas the right column features images produced by the non-infringing model. The results indicate that the extraction method is effective
20

---

Published in Transactions on Machine Learning Research (02/2025)
Table 5: Details of artistic style and contents
Artistic Style
Seen Contents (training set)
Unseen Contents (evaluation set)
Claude Monet
vase flowers
barn in a rural setting bustling street market boat on a calm river group of animals in field crowded café scene horse grazing in a pasture vintage clock on a mantelpiece window with a view of the countryside
Vincent van Gogh
bowl of fruit
Pablo Picasso
still life with candles
Johannes Vermeer
landscape with rolling hills
Leonardo da Vinci
cityscape with buildings
Michelangelo Frida Kahlo
forest with sunlight filtering through trees
portrail of a person quiet beach at sunset mountain range with snow tranquil lake with reflections
Rembrandt van Rijn
Salvador Dalí
room with antique furniture close-up of tree's bark and leaves
Henri Matisse
Table 6: Quantitative results on seen and unseen contents. The gap in metrics between seen content and unseen content is not significant. The symbol indicates that the higher value is better on the metric, whereas & symbol signifies the lower value is more preferable.
Style
Contents
KID 1
LPIPSN
DINO-v2%L
CLIP-t%41
Seen Contents
137
0.315
46.8
28.5
Target Style
Unseen Contents
168
0.303
43.3
28.0
Seen Contents
17.92 16.66
0.139
81.0
32.3
Surrounding Style
Unseen Contents
0.135
81.6
32.1
not only in isolating and modifying individual content-style associations but also in comprehensively altering an artist's entire style.
Table shows that both extracting specific prompts and styles have little impact on the model's generative ability.
Table 7: Quantitative results on COCO caption set.
Method
FID,
KID x10 3
4
Extract "van gogh'
22
3.4
Extract "sunflower by van gogh"
20.8
1.4
A.6 Style-IP Gplug-in in combination
Figure I1 shows the combination of style and IP Oplug-ins. The initial image is generated by the non-infringing model after extracting R2D2 and Gogh, where the TP and style ate hard to recognize. Subsequent images, the second and third, are produced by the model after separately adding the R2D2 and Gogh Oplug-in, thus distinctly featuring the respective IP and style. The final image is created after adding the combined Oplug-in, successfully achieving IP recreation and style replication.
A.7 The limitation of potential degradation in the performance of the non-infringing model
We observe a decline in the performance of the non-infringing model as the number of extracted styles increascs. This degradation can be effectively mitigated by increasing the rate of re-context iterations and de-concept iterations, ie., we can sample more contexts in the re-contexts in the re-context sub-process maintain the generative ability of the non-infringing model.
21

---

Published in Transactions on Machine Learning Research (02/2025)
Extract sunflowers by van gogh
Extract van gogh
Sunflowers
Stary Night
Portrait
Figure 10: The results of the non-infringing model after extracting specific prompt and style. In the left block, only the specific prompt 'sunflowers by van gogh' is extracted, while in the right block the entire style "van gogh" is extracted. For each block, the left column displays images generated by the base model, whereas the right column shows images generated by the corresponding non-infringing model.
Combined extraction
R2D2 Integration
Van gogh Integration
Combined Integration
Prompt
R2D2 on the stany night by van gogh
Figure 11: IP Recrcation and style replication in a single image. We can integrate Oplug-in into the non-infringing model to recreate R2D2 or replicate van Goghis style in single image. Also, we can integrate the combined Oplug-in achieve them both in an image.
22

---

Published in Transactions on Machine Learning Research (02/2025)
During the ertract process, set the number of de-concept iterations to be 10 and set the number of re-context iterations to 10 The default value for the rate is 1. For extraction, we leverage ChatGPT to generate 10 r common contents. For each iteration, we select one of these contents to generate 8 images. A higher r allows the re-context phase to learn more contextual information, thereby maintaining the model's ability to generale high-quality images even as the number of extracted styles grows.
Figure 12 illustrates this phenomenon. The first row displays the image generated by the base model. Subsequent rows, from top bottom, images generated by non-infringing models that extract 1 to 10 styles, respectively. As the number of extracted styles increases, there is noticeable decline in image quality, However, by increasing the rate, this degradation is significantly alleviated, as evidenced by improved image fidelity. The styles and contexts are detailed in Table When the rate is greater than 1, we use the context. in the evaluation set for the re-context phase.
0.
Rate=10
Relo-1.5
Rato=20
The degradation and alleviation the performance of the non-infringing model. We randomly sample 8 images generated by different models. The first line is the image generated by the base model. The rest, from top to bottom, are images generated by non-infringing models that extract 1, 2, 3..., 10 styles, respectively. As the extracted concept mumber increases, the quality of images continuously declines. With the increase of the rate of re-context iteration number and de-concept iteration number, this decline has been effectively alleviated. Zoom in for better visualization.
Figure 12:
23


Reviews:
[{"review_id": "9aX0E6YVug", "reviewer": "Reviewer_t6UR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes an innovative, scalable solution to copyright attribution in generative AI, directly aligning with existing intellectual property management practices.\n\n2. From the technique perspective, Reverse LoRA is a method to disentangle copyrighted concepts from base models effectively. EasyMerge is a data-free, layer-wise distillation approach to combine multiple plug-ins seamlessly.\n\n\nWeaknesses:\n\n1. The evaluation metrics adopted in the work include KID and LPIPS metrics. More diverse performance metrics of T2I models could strengthen the argument.\n\n2. The paper acknowledges potential performance degradation in non-infringing models after extensive extractions. It will be better to provide a quantitative or qualitative analysis of this impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel framework, Plug-in Authorization, to address copyright infringement concerns in text-to-image generative models. It introduces three operations—Addition, Extraction, and Combination—to handle copyright attribution and protection for artists and intellectual property (IP) owners. By leveraging methods like \"Reverse LoRA\" for extraction and \"EasyMerge\" for combination, the framework allows the separation of copyrighted styles or IPs from base models and the merging of multiple plug-ins into a cohesive model while maintaining non-infringing capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes are the same as the weaknesses.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PLSZDgQstp", "reviewer": "Reviewer_LCJv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The problem addressed is meaningful and worth studying.\n- The proposed method is relatively complete, covering both extraction and combination.\n- The results show the effectiveness of the proposed method.\n\n**Weaknesses**\n\n- Main concern: The authors use ChatGPT to extract 10 context prompts for a given artistic style or IP character during training and employ the same set of prompts for evaluation. I am concerned that the model may become overfitted to this specific prompt set and may not generalize to a broader variety of prompts. If this is the case, the copyright protection may ultimately be ineffective.\n- While addition is mentioned in the paper, it is actually an original contribution of Civitai. The authors should highlight this earlier in the introduction to avoid misleading the audience.\n- Why do the authors compare their method with concept ablation only on KID and with ESD only on LPIPS? It is recommended to compare both methods using both metrics.\n- The results in the paper only include combinations of multiple artistic styles or multiple IP characters. Can the authors provide results that demonstrate the combination of artistic styles with IP characters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a LoRA-based method to extract and combine copyright plug-ins for T2I diffusion models. This addresses the copyright infringement issues commonly encountered in the generative AI industry. The reported results demonstrate the effectiveness of the proposed method in extracting specific IPs and disentangling protected concepts from general concepts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gKfBm9ZGlO", "reviewer": "Reviewer_L9GJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The overall concept of a plug-in component in the image generation model that authorizes a certain IP is interesting and could be very useful for real-world applications of large-scale pretrained image generation models.\n2. Experimental results demonstrate the effectiveness of the method for removing and restoring simple concepts.\n\nWeaknesses:\n1. The proposed plug-in lora component seems to be related to a specific prompt, instead of an IP or a whole concept. For example, I would like to know after extracting \"Sunflower by Van Gogh\" from a pretrained SD model, can the model still generate valid images based other Van Gogh-related prompts (e.g. Starry night by Van Gogh, etc)? If the model can still generate such images, then it's probably very hard for the method to scale up as essentially every prompt needs a different lora for plug-in authorization.\n2. What is the $w-w_L$ operation mentioned in Section 2.2.1? Is it subtracting the Lora output from the pretrained SD output? Can the author provide more intuition on why this operation leads to the model forgetting the generation related to the IP specific prompt, but in the meantime doesn't affect the generation of other objects too much?\n3. The experiments on the model's generation performance after extracting a certain prompt are very important and shouldn't be placed in the appendix. For the experiment in Appendix A.2 and Table 3, I'd also like to see the comparison of FID between pretrained Stable Diffusion model and the model after extracting a certain prompt. The method will not make sense if extracting a prompt from the model will dramatically hurt the image generation performance of the base model.\n\nMinor concerns:\n1. In Section 1, first paragraph: \"these models not only excel at generating content based on user prompts cha...\" Is this \"cha\" a typo or a broken reference to the citation?\n2. In Section 2 first paragraph: \"\"As detailed in the Introduction...\" There is a \" symbol at the beginning of the paragraph.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a plug-in authorization framework that learns to disentangle a specific prompt from a pretrained image generation model into a lora module. This plug-in lora module can act as an authorization key to avoid copyright infringement. Only with this plug-in module can the base image generation model generate images related to the concept encoded in the lora module. The author also provided some discussions on methods to merge multiple concepts into a single lora module and included some examples to demonstrate the effect of the plug-in modules.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting and important problem, copyright protection, for generative models. It is of value to researchers in the area. However, the novelty is limited and hence, it doesn't meet the standards for certification or presentation at ICLR.
Decision: Accept (poster)