SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additional zero-shot classification We show in both Table 2 and Table 3 additional results for the baseline zero-shot classification protocol (following the exact same setup in the main paper) with the similarity metric  $S(\Pi_N(\mathbf{z}_I), \mathbf{z}_T)$  after the noun subspace projection. As can be seen, the proposed subspace leads to improved zero-shot classification on a wide range of datasets, for multiple CLIP architectures.
Table 2: Top-1 ZS classification accuracy with CLIP ViT-B-16.
ImageNET	MIT-states	UT Zap-	DomainNET	StanfordCars	Caltech101	Food101	CIFAR10	CIFAR100	OxfordPeds	Flowers102	Caltech256	STL10	MNIST	FER2013
CLIP	61.60	47.80	89.10	54.40	62.50	82.00	81.50	88.80	62.40	76.10	65.20	81.70	96.90	48.00	41.90
PoS PCA	61.60	47.90	89.00	54.40	62.50	82.10	81.50	88.80	62.30	76.00	65.30	81.80	97.00	47.70	41.70
PoS PGA	61.80	47.70	89.60	54.30	60.00	82.20	81.70	87.90	62.70	78.60	64.60	82.30	96.60	50.00	42.70
Noun Submanifold	62.80	48.30	88.10	54.70	62.70	82.90	82.10	89.10	63.90	77.00	65.40	83.40	97.00	55.00	48.60
Table 3: Top-1 ZS classification accuracy with CLIP ViT-L-14.
ImageNET	MIT-states	UT Zap-	DomainNET	StanfordCars	Caltech101	Food101	CIFAR10	CIFAR100	OxfordPeds	Flowers102	Caltech256	STL10	MNIST	FER2013
CLIP	69.00	51.80	90.40	59.60	74.70	80.90	87.40	91.70	72.70	82.50	74.60	86.80	97.70	59.70	35.70
PoS PCA	69.00	51.80	90.40	59.60	74.70	80.90	87.40	91.70	72.70	82.50	74.60	86.80	97.70	59.70	35.70
PoS PGA	69.10	51.90	90.50	59.60	74.60	80.80	87.50	91.60	72.80	82.60	74.80	86.80	97.70	59.70	35.80
Noun Submanifold	70.10	52.50	90.10	60.00	76.00	82.80	87.70	90.60	73.90	83.10	74.70	88.10	96.90	58.90	46.40
C.1 Relationship to alternative component analyses
We first compare 1D subspaces learnt with our method, FDA [48], and FKT [46] shown in Figure 14 on toy data chosen to illustrate the qualitative differences in the properties of the subspaces. Given the very different goals of FDA in minimising intra-class variation, the resulting FDA subspace is near-orthogonal to that of FKT and the proposed method. Whilst the FKT-given subspace is very close to ours, for this particular illustrative toy data our subspace better kills the variance in the red data (as illustrated in Figure 14 b.iii). This figure also provides a visualisation of the learnt subspace's proximity to the principal component of the target class and the bottom principal component of the red class' datapoints—the proximity to the two extremes being controlled by  $\lambda$ .
Figure 14: (a) A visual comparison of the leading eigenvector of  $\mathbf{C}_i$  to the first FDA component (centred for comparison), to the first FKT component. Shown in (b) are the points' coordinates in the three subspaces. As can be seen, the learnt  $\mathbf{w}_{B1}$  captures large variance in the blue target class and is close to orthogonal to data points of the red class.
C.2 Role of  $k$
We show the impact of various choices of  $k$  (dimensionality of the subspaces) as visualised with text-to-image models in Figure 15 with the projections onto the orthogonal complements to kill the undesired variation. As can be seen, increasing  $k$  removes more and more visual information relevant to the particular visual mode. For example, we see the image feature increasingly less snow on the right, whilst increasing less of London on the left.
Figure 15: Ablation study on the value of  $k$  on the  $d = 1024$  OpenCLIP VL space when projecting onto the orthogonal complements of  $k$ -dimensional adjective and noun subspaces.
Figure 16: Without using PoS as 'negative guidance' (i.e. when  $\lambda \coloneqq 0$ ), related concepts (e.g. 'cranberry juice') can be visually removed to a much greater extent than when using the PoS guidance ( $\lambda \coloneqq 0.5$ ).
C.3 Role of  $\lambda$
We next provide an ablation study on the value of  $\lambda$ . We provide both a visual qualitative ablation and a quantitative one.
Qualitative Concretely, in each subfigure (row) of Figure 17, we take the first 5000 WordNET text strings from each part of speech and compute their CLIP text embeddings  $\mathbf{z}_T \in \mathbb{R}^d$ . We then calculate  $\hat{\mathbf{W}}_i^\top \mathrm{Log}_{\mu}(\mathbf{z}_T)$ , plotting the first coordinate along the  $x$ -axis and the second along the  $y$ -axis of each subfigure in Figure 17. Ideally, the data points in the target class should be the only ones with a large norm if this hyperplane captures visual variation that is unique to a particular word class. We see that  $\lambda := 0$  preserves the most variance in the target class' embeddings but the different categories' projections are clearly entangled—the other classes' datapoints also have large norm. Conversely,  $\lambda := 1.0$  maps all points effectively to the zero vector—killing the variance in all categories. As can be seen in Figure 17c,  $\lambda := 0.5$  offers a reasonable balance of both properties in this 4-class setting. The exact same experiments are run on the larger version of CLIP, shown in Figure 18, where similar conclusions can be drawn about the practical impact of  $\lambda$ .
Quantitative For quantifying this in more dimensions, we compute the class invariance metric (used in the main paper) in Figure 19 and Figure 20 for various values of  $\lambda$ , where we observe that  $\lambda := 0.5$  is a sensible choice for multiple CLIP architectures.
Visual subspaces Finally, we demonstrate the importance of using PoS as 'negative examples' in the summation in the main objective when learning visual theme-specific subspaces. Intuitively, whilst we want to maximise the variation for phrases of a particular theme (such as 'gory'), we also want to preserve the ability to generate other concepts with the TIIM, which is what the objective provides through the hyperparameter  $\lambda$ .
In particular, we show in the second row of Figure 16 the visual results when projecting onto the orthogonal complement of a 'gory' subspace learnt when we do not use the PoS as 'negative guidance' (i.e. when  $\lambda := 0$ ). As can be seen in comparison to the third row of Figure 16, using the PoS is critical in this instance for retaining the ability to synthesise existing related concepts (here ensuring e.g. 'cranberry juice' can still be synthesised even though visually 'gory' appearances are removed).
(a)  $\lambda := 0$
(b)  $\lambda \coloneqq 0.25$
(c)  $\lambda := 0.5$
(d)  $\lambda \coloneqq 0.75$
Figure 17: Embeddings' first two coordinates in the tangent space(es), with various values of  $\lambda$  in the main objective (axis limits are fixed to compare length of vectors across values of  $\lambda$ ). The base CLIP model clip-vit-base-patch32 is used here.
(e)  $\lambda \coloneqq 1.0$
Figure 18: Embeddings' first two coordinates in the tangent space(es), with various values of  $\lambda$  in the main objective (axis limits are fixed to compare length of vectors across values of  $\lambda$ ). The larger CLIP model clip-vit-large-patch14 is used here.
(b)  $\lambda \coloneqq 0.1$
(a)  $\lambda := 0$
(d)  $\lambda := 0.5$
(e)  $\lambda \coloneqq 0.75$
(c)  $\lambda \coloneqq 0.25$
(f)  $\lambda \coloneqq 1.0$
Figure 19: Ablation on  $\lambda$  with the quantity  $\frac{1}{n_j} ||(\hat{\mathbf{W}}_i^\top \mathbf{Y}_j)||_F^2$  introduced in the main paper. Row-normalisation is performed to highlight the relative representation of each class' embeddings within each subspace. The base CLIP model clip-vit-base-patch32 is used here.
(a)  $\lambda := 0$
(d)  $\lambda \coloneqq 0.5$
Figure 20: Ablation on  $\lambda$  with the quantity  $\frac{1}{n_j} ||(\hat{\mathbf{W}}_i^\top \mathbf{Y}_j)||_F^2$  introduced in the main paper. Row-normalisation is performed to highlight the relative representation of each class' embeddings within each subspace. The larger CLIP model clip-vit-large-patch14 is used here.
(b)  $\lambda \coloneqq 0.1$
(e)  $\lambda \coloneqq 0.75$
D Experimental details
For reference in this section, we first include the dimensionalities of the shared VL spaces and public links to implementations used of the three CLIP models in this paper in Table 4.
Table 4: Dimensionality of the VL space representations in the four CLIP models.
Model name	clip-vit-base-patch32	clip-vit-base-patch16	clip-vit-large-patch14	‘OpenCLIP’
Dimensionality of zI, zT	512	512	768	1024
Public link	HuggingFace	HuggingFace	HuggingFace	Github
Visual disentanglement The Paella TTIM [15] used in the main paper adopts the 'OpenCLIP' [50] model with a larger  $d = 1024$ -dimensional VL representation. For all 'visual disentanglement' results throughout both the paper and supplementary material, we use  $k = 768$  dimensional 'adjective' and 'noun' subspaces for all text prompts (apart from when removing the 'content' representations in visually polysemous phrases, where we find only  $k = 32$  components are necessary).
Visual theme subspaces For the custom visual subspaces, we produce a list of phrases related to the visual theme of interest by asking ChatGPT [51] questions of the format: Please give me a list of 250 words and phrases related to the concept of  $\{\mathbf{x}\}$ , where  $\mathbf{x}$  is the visual concept of interest (such as 'gore'). For the case of the artist subspace, we ask: Please give me a list of 250 of the most famous painters and visual artists of all time. In each scenario, we follow up twice more asking for additional responses (given the limited response length), specifying that it tries not to repeat any of the previous answers in the list. For the experiments in the 'gory' and 'artist' custom subspaces, ChatGPT gave us 371 and 830 unique phrases respectively (taking just the provided artists' surnames as additional examples for the latter), and use a  $k = 128$ - and  $k = 512$ -dimensional subspace respectively (given the limited number of phrases for the gore subspace provided by ChatGPT).
Concurrent work Recent preprints [52, 53] explicitly address the task of ablating particular concepts in diffusion models specifically. However, in contrast to the proposed method, these concurrent works fine-tune Stable Diffusion-specific [4] submodules, and do not focus on the final CLIP vector representations. Thus, there is no straightforward way to compare the proposed method working in CLIP's shared vision-language space directly nor the alternative Paella [15] TIIM. One methodological benefit to [52] over the proposed method however (purely in the context of text-to-image synthesis) is in the requirement of only a single text prompt describing a concept, relative to our necessary collection<sup>4</sup>. On the other hand, our subspaces are learnt in closed form—for example, the 'gory' subspace takes only 0.28 seconds to compute on a V100 GPU, given the CLIP embeddings. This is in contrast to [52]'s models which are stated to require 1000 gradient descent steps to compute, and [53] taking 5 minutes per concept.
Compute time and hardware To run the Paella model, we use a 32GB NVIDIA Tesla V100 GPU. Learning the subspaces is particularly fast given the closed-form solution, taking just 1.1 seconds to compute all 4 (noun, adjective, verb, and adverb) PoS subspaces. Encoding all WordNet PoS examples with CLIP takes 28.91 minutes, however, this is a fixed cost and only needs to be done once at the beginning (after which any number of additional subspaces can be computed very quickly).
In particular,  $n$  phrases' embeddings can span a subspace with a maximum of  $n$  dimensions