SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To alter the style, we set  $x_{\mathrm{new}} =$  "a portrait of a mathematician, in Fauvism style". We define the concept subspace using prompts of the form "a portrait of a mathematician in [xstyle] style", where  $x_{\mathrm{style}}$  takes value from a list of styles. During sampling, the original prompt is utilized in the first 20% of timesteps to better retain the content.
The list of styles is generated by ChatGPT. They are: Art Deco, Minimalist, Baroque, Abstract Expressionist, Cubist, Fauvism, Impressionist, Steampunk, Neoclassical, Japanese Ukiyo-e, Surrealism, Memphis Design, Scandinavian, Bauhaus, Pop Art, Art Nouveau, Street Art, American West, Victorian Gothic, Futurism, Photorealistic, Mannerist, Flemish, Byzantine, Medieval, Romanesque, Trompe-l'oeil, and Dutch Golden Age.
C.3 Stress-test experiments (Figure 3)
In this experiment, we deliberately chose a diverse array of artistic styles and contrasting content to challenge our model. The styles used are Baroque painting, traditional Chinese painting, Pop art, Gothic stained glass, the Pre-Raphaelite painting style, Victorian botanical illustration, and Japanese Edo period art. In terms of content, we use: a bustling train station, a nuclear power plant, an artificial intelligence lab, a jazz music concert, a modern dance festival, a modern minimalist house, and a contemporary office setting. The rationale behind this selection was to pair styles and content that would rarely co-occur in training data. This rarity poses a significant challenge for the model in generating realistic outputs, testing its capabilities and adaptability to unconventional combinations.
We implement concept algebra the same way as in Figure 1 example. For concept composition, we use the software in [Liu+22].
In fig. 6 we show some examples where concept algebra is not preferred.
Figure 6: Some examples where concept algebra is not preferred. Each row has style-reference (leftmost), a content-reference (2nd from left), and left-to-right (here, but randomly permuted in the survey) images generated from concept-algebra, composition and direct prompting.
C.4 The nurse example (Figure 4)
We initiate the process with an original prompt,  $x_{\mathrm{orig}} =$  "a portrait of a nurse". Our goal is to perform concept projection to manipulate the sex attribute. Similar to the mathematician example, we define the sex direction using a pair of prompts:  $(x_1,x_2) =$  ("a man", "a woman"). However, instead of setting the distribution of sex as one of the delta functions or a fair one corresponding to the neutral prompt "a person", we wish to see what the concept's arithmetic average will define. Specifically, we take the sex direction of the average of a female nurse and a male nurse, calculated as  $\frac{1}{2} s(a \text{female nurse}) + \frac{1}{2} s(a \text{male nurse})$ . It turns out the arithmetic mean realize the interpolation between two extremal sex points in the sex subspace, and the score function after concept projection returns images of androgynous nurses.
Concept Algebra beats negative prompting in simple tasks Negative prompting, aims to eliminate target concept expressions by subtracting relevant scores. Unlike concept algebra and similar to concept composition, this method does not confine manipulations to specific subspaces. As predicted, we see negative prompting inadvertently modify off-target concepts tied to the primary concept, whereas concept algebra succeeds, as in .
Concept algebra fails when causal separability (Theorem 3.2) is violated We show one concrete example of failures.
Figure 8 shows that we are unable to transfer the gender of the nurse when we calculate the score function of a male nurse by  $(\mathbb{I} - \mathrm{proj}_Z)s[$  "a portrait of a nurse]  $^+$  projz  $s$  [a man] where  $\mathrm{proj}_Z$  is computed by  $s$  [a buck on the grass]-  $s$  [a doe on the grass]. The target concept  $Z$  and  $W$  are sex  $\in$  {male, female} and species  $\in$  {human, deer}. It's obvious that the sex and species have an interaction effect on the image  $Y$  - different species induce different sexual characteristics.
Figure 7: Concept-algebra can succeed when negative prompting fails. Images (matched on random seed) from top to bottom: direct prompting "a portrait of a king", negative prompting w/ "male", negative prompting w/ "male" but using concept projection, concept algebra with prompt "female". Note negative prompting does not remove the maleness of the output.
(b)  $(\mathbb{I} - \mathrm{proj}_Z)s[\text{"a portrait of a nurse"}] + \mathrm{proj}_Zs[\text{"a man"}]$  where  $\mathrm{proj}_Z$  is computed by  $s[\text{"a buck on the grass"}] - s[\text{"a doe on the grass"}]$
(c)  $(\mathbb{I} - \mathrm{proj}_Z)s[a$  portrait of a nurse]  $^+$  projz  $s$  [a man] where projz is computed by s["amn]-s[a woman]
Figure 8: Necessity of Assumptions: the validity of concept algebra depends on causal separability