SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dólar, and Ross B. Girshick. Segment anything. ArXiv, abs/2304.02643, 2023.
[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[3] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[6] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2018.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] OpenAI. Chatgpt (mar 14, 2023 version). Large language model, 2023. URL https://chat.openai.com/chat.
[65] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
[66] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
[67] Matthew D Zeiler. Adadelta: an adaptive learning rate method. ArXiv, abs/1212.5701, 2012.
[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
A Compositional Sculpting of Iterative Generative Processes (Section 5, continued)
A.1 Classifier Training (GFlowNets): Algorithm
Algorithm A.1 Compositional Sculpting: classifier training
1: Initialize  $\phi$  and set  $\overline{\phi} = \phi$
2: for step  $= 1$  ,...,num_steps do
3: for  $i = 1,\dots ,m$  do
4: Sample  $\widehat{\tau}_i\sim p_i(\tau)$
5: end for
6:  $\mathcal{L}_T(\phi) = -\sum_{i=1}^m \log \widetilde{Q}_{\phi}(y_1 = i | x = \widehat{x}_i)$  {Terminal state loss (16)}
7:  $w_{i}(\widehat{x}_{j};\overline{\phi}) = \widetilde{Q}_{\overline{\phi}}^{-}(y_{k} = i|x = \widehat{x}_{j}), i,j\in \{1,\ldots m\}$  {Terminal probability estimates}
{Non-terminal state loss (17)-(18)}
$$
8: \quad \mathcal {L} _ {N} (\phi , \bar {\phi}) = \sum_ {\hat {y} _ {1} = 1} ^ {m} \dots \sum_ {\hat {y} _ {n} = 1} ^ {m} \left(\prod_ {k = 2} ^ {n} w _ {\hat {y} _ {k}} (\hat {x} _ {\hat {y} _ {1}}; \bar {\phi})\right) \ell (\hat {\tau} _ {\hat {y} _ {1}}, \hat {y} _ {1}, \dots \hat {y} _ {n}; \phi)
$$
9:  $\mathcal{L}(\phi, \overline{\phi}) = \mathcal{L}_T(\phi) + \gamma (\mathrm{step}) \cdot \mathcal{L}_N(\phi, \overline{\phi})$
10: Update  $\phi$  using  $\nabla_{\phi}\mathcal{L}(\phi ,\overline{\phi})$  ; update  $\overline{\phi} = \beta \overline{\phi} +(1 - \beta)\phi$
11: end for
A.2 Composition of Diffusion Models
In this section, we show how the method introduced above can be applied to diffusion models. First, we adapt the model we introduced in (9)-(11) to diffusion models. A diffusion model trained to sample from  $p_i(x)$  generates a trajectory  $\tau = \{x_t\}_{t=0}^T$  over a range of time steps which starts with a randomly sampled state  $x_T$  and ends in  $x_0$ , where  $x_0$  has distribution  $p_{i,t=0}(x) = p_i(x)$ . Thus, we must adapt our model to reflect this. We introduce a set of mutually dependent variables  $x_t$  for  $t \in (0,T]$  with as conditional distribution the transition kernel of the diffusion model  $p_i(x_t|x_0)$ .
Given  $m$  base diffusion models that sample from  $p_1, \ldots, p_m$  respectively, we define the prior  $\widetilde{p}(x)$  as a mixture of these diffusion models. Proposition A.1 shows that this mixture is a diffusion model that can be constructed directly from the base diffusion models. We then apply classifier guidance to this mixture to sample from the composition. We present an informal version of the proposition below. The required assumptions and the proof are provided in Appendix D.5.
Proposition A.1 (Diffusion mixture SDE). Suppose distributions  $p_1(x), \ldots, p_m(x)$  are realized by diffusion models with forward SDEs  $dx_{i,t} = f_{i,t}(x_{i,t})dt + g_{i,t}dw_{i,t}$  and score functions  $s_{i,t}(\cdot)$ , respectively. Then, the mixture distribution  $p_M(x) = \sum_{i=1}^{m}\omega_i p_i(x)$  with  $\omega_1 \ldots \omega_m \geq 0$  and  $\sum_{i=1}^{m}\omega_i = 1$  is realized by a diffusion model with forward SDE
$$
d x _ {t} = \underbrace {\left[ \sum_ {i = 1} ^ {m} p (y = i | x _ {t}) f _ {i , t} \left(x _ {t}\right) \right]} _ {f _ {M, t} \left(x _ {t}\right)} d t + \underbrace {\sqrt {\sum_ {i = 1} ^ {m} p (y = i | x _ {t}) g _ {i , t} ^ {2}}} _ {g _ {M, t} \left(x _ {t}\right)} d w _ {t}, \tag {19}
$$
and backward SDE
$$
d x _ {t} = \left[ \sum_ {i = 1} ^ {m} p (y = i | x _ {t}) \left(f _ {i, t} \left(x _ {t}\right) - g _ {i, t} ^ {2} s _ {i, t} \left(x _ {t}\right)\right) \right] d t + \sqrt {\sum_ {i = 1} ^ {m} p (y = i | x _ {t}) g _ {i , t} ^ {2}} d \bar {w} _ {t}, \tag {20}
$$
$$
p (y = i \mid x _ {t}) = \frac {\omega_ {i} p _ {i , t} \left(x _ {t}\right)}{\sum_ {j = 1} ^ {m} \omega_ {j} p _ {j , t} \left(x _ {t}\right)}. \tag {21}
$$
If the base diffusion models have a common forward SDE  $dx_{i,t} = f_t(x_{i,t})dt + g_tdw_{i,t}$ , equations (19)-(20) simplify to
$$
d x _ {t} = f _ {t} (x _ {t}) d t + g _ {t} d w _ {t}, \quad d x _ {t} = \left[ f _ {t} (x _ {t}) - g _ {t} ^ {2} \left(\sum_ {i = 1} ^ {m} p (y = i | x _ {t}) s _ {i, t} (x _ {t})\right) \right] d t + g _ {t} d \overline {{w}} _ {t}. \qquad (2 2)
$$
Theorem A.2 summarizes the overall approach.
Theorem A.2. Suppose distributions  $p_1(x), \ldots, p_m(x)$  are realized by diffusion models with forward SDEs  $dx_{i,t} = f_{i,t}(x_{i,t})dt + g_{i,t}dw_{i,t}$  and score functions  $s_{i,t}(\cdot)$ , respectively. Let  $y_1, \ldots, y_n$  be random variables defined by (9). Then, the conditional  $\widetilde{p}(x|y_1, \ldots, y_n)$  is realized by a classifier-guided diffusion with backward SDE
$$
d x _ {t} = v _ {C, t} \left(x _ {t}, y _ {1}, \dots , y _ {n}\right) d t + g _ {C, t} \left(x _ {t}\right) d \bar {w} _ {t}, \tag {23}
$$
$$
v _ {C, t} \left(x _ {t}, y _ {1}, \dots , y _ {n}\right) = \sum_ {i = 1} ^ {m} \widetilde {p} (y = i \mid x _ {t}) \left(f _ {i, t} \left(x _ {t}\right) - g _ {i, t} ^ {2} \left(s _ {i, t} \left(x _ {t}\right) + \nabla_ {x _ {t}} \log \widetilde {p} \left(y _ {1}, \dots , y _ {n} \mid x _ {t}\right)\right)\right), \tag {24}
$$
$$
g _ {C, t} \left(x _ {t}\right) = \sqrt {\sum_ {i = 1} ^ {m} \widetilde {p} (y = i \mid x _ {t}) g _ {i , t} ^ {2}}. \tag {25}
$$
The proof of Theorem A.2 is provided in Appendix D.6.
A.3 Classifier Training (Diffusion Models)
We approximate the inferential distributions in equations (22) and (23) with a time-conditioned classifier  $\widetilde{Q}_{\phi}(y_1,\ldots ,y_n|x_t)$  with parameters  $\phi$ . Contrary to GFlowNets, which employed a terminal and non-terminal state classifier, here we only need a single time-dependent classifier. The classifier is trained with different objectives on terminal and non-terminal states. The variables  $y_{1},\dots,y_{n}$  are dependent given a state  $x_{t}$  for  $t\in [0,T)$ , but are independent given the terminal state  $x_{T}$ . Thus, when training on terminal states we can exploit this independence. Furthermore, we generally found it beneficial to initially train only on terminal states. The loss for the non-terminal states depends on classifications of the terminal state of the associated trajectories, thus by minimizing the classification error of terminal states first, we reduce noise in the loss calculated for the non-terminal states later.
For a terminal state  $x_0$ , the classifier  $\widetilde{Q}_{\phi}(y_1, \ldots, y_n | x_t)$  can be factorized as  $\prod_{k=1}^{n} \widetilde{Q}_{\phi}(y_k | x_0)$ . Hence we can train  $\widetilde{Q}$  by minimizing the cross-entropy loss
$$
\mathcal {L} _ {\mathrm {T}} (\phi) = \underset {(\hat {x} _ {0}, \hat {y} _ {1}) \sim \widetilde {p} (x, y _ {1})} {\mathbb {E}} \left[ - \log \widetilde {Q} _ {\phi} \left(y _ {1} = \hat {y} _ {1} \mid x _ {0} = \hat {x} _ {0}\right) \right]. \tag {26}
$$
Samples  $\widetilde{p}(x_0, y_1)$  can be generated according to the factorization  $\widetilde{p}(y_1) \widetilde{p}(x_0 | y_1)$ . First,  $\widehat{y}_1$  is sampled from  $\widetilde{p}(y_1)$ , which is uniform under our choice of  $\widetilde{p}(x_0)$ . Then,  $\widehat{x}_0 | (y_1 = \widehat{y}_1)$  is generated from the reverse SDE of base diffusion model  $p_{\widehat{y}_1}(x)$ . Note that equation (9) implies that all observations have the same conditional distribution given  $x$ . Thus,  $\widetilde{Q}_{\phi}(y_1 | x_0)$  is also a classifier for observations  $y_2, \ldots, y_n$ .
For a non-terminal state  $x_{t}$  with  $t \in (0, T]$ , we must train  $\widetilde{Q}$  to predict  $y_{1}, \ldots, y_{n}$  jointly. For a non-terminal state  $\hat{x}_{t}$  and observations  $\hat{y}_{1}, \ldots, \hat{y}_{n}$ , the cross-entropy loss is
$$
\ell \left(\widehat {x} _ {t}, \widehat {y} _ {1}, \dots , \widehat {y} _ {n}; \phi\right) = - \log \widetilde {Q} _ {\phi} \left(y _ {1} = \widehat {y} _ {1}, \dots , y _ {n} = \widehat {y} _ {n} \mid x _ {t} = \widehat {x} _ {t}\right). \tag {27}
$$
Tuples  $(\widehat{x}_t,\widehat{y}_1,\dots ,\widehat{y}_n)$  are obtained as follows: 1)  $\widehat{y}_1\sim \widetilde{p} (y_1)$  ; 2) A trajectory  $\tau = \{x_{t}\}_{t = 0}^{T}$  is sampled from the reverse SDE of diffusion model  $y_{1}$  . At this point, we would ideally sample  $\widehat{y}_2,\ldots ,\widehat{y}_n$  given  $\widehat{x}_0$  but this requires access to  $\widetilde{p} (y_k = \widehat{y}_k|\widehat{x}_0)$  . Instead, we approximate this with  $w_{i}(\widehat{x};\phi) = \widetilde{Q}_{\phi}(y_{1} = i|x_{0} = \widehat{x}_{0})$  and marginalize over  $\widehat{y}_2,\ldots ,\widehat{y}_n$  to obtain the cross-entropy loss
$$
\mathcal {L} _ {N} (\phi , \bar {\phi}) = \underset {(\hat {\tau}, \hat {y} _ {1}) \sim \widetilde {p} (\tau , y _ {1})} {\mathbb {E}} \left[ \sum_ {\hat {x} _ {t} \in \hat {\tau} \backslash \{\hat {x} _ {0} \}} \sum_ {\hat {y} _ {2} = 1} ^ {m} \dots \sum_ {\hat {y} _ {n} = 1} ^ {m} \left(\prod_ {k = 2} ^ {n} w _ {\hat {y} _ {k}} (\hat {x} _ {0}; \bar {\phi})\right) \ell (\hat {x} _ {t}, \hat {y} _ {1}, \dots , \hat {y} _ {n}; \phi) \right]. \tag {28}
$$
B Classifier Guidance for Parameterized Operations
This section covers the details of classifier guidance and classifier training for the parameterized operations (Section 4.1).
The complete probabilistic model for the parameterized operations on two distributions is given by
$$
\widetilde {p} (x, y _ {1}, y _ {2}; \alpha) = \widetilde {p} (x) \widetilde {p} (y _ {1} | x) \widetilde {p} (y _ {2} | x; \alpha), \quad \widetilde {p} (x) = \frac {1}{2} p _ {1} (x) + \frac {1}{2} p _ {2} (x), \tag {29a}
$$
$$
\widetilde {p} (y _ {1} = i | x) = \frac {p _ {i} (x)}{p _ {1} (x) + p _ {2} (x)}, \quad \widetilde {p} (y _ {2} = i | x; \alpha) = \frac {\left(\alpha p _ {1} (x)\right) ^ {[ i = 1 ]} \cdot \left((1 - \alpha) p _ {2} (x)\right) ^ {[ i = 2 ]}}{\alpha p _ {1} (x) + (1 - \alpha) p _ {2} (x)}. \tag {29b}
$$
While in the probabilistic model (3) all observations  $y_{i}$  are exchangeable, in the parameterized model (29)  $y_{1}$  and  $y_{2}$  are not symmetric. This difference requires changes in the classifier training algorithm for the parameterized operations.
We develop the method for the parameterized operations based on two observations:
-  $y_{1}$  appears in (29) in the same way as in (3);
- the likelihood  $\widetilde{p}(y_2|x;\alpha)$  of  $y_2$  given  $x$  can be expressed as the function of  $\widetilde{p}(y_1|x)$  and  $\alpha$ :
$$
\begin{array}{l} \widetilde {p} \left(y _ {2} = 1 \mid x; \alpha\right) = \frac {\alpha p _ {1} (x)}{\alpha p _ {1} (x) + (1 - \alpha) p _ {2} (x)} = \frac {\alpha \frac {p _ {1} (x)}{p _ {1} (x) + p _ {2} (x)}}{\alpha \frac {p _ {1} (x)}{p _ {1} (x) + p _ {2} (x)} + (1 - \alpha) \frac {p _ {2} (x)}{p _ {1} (x) + p _ {2} (x)}} \tag {30a} \\ = \frac {\alpha \widetilde {p} (y _ {1} = 1 | x)}{\alpha \widetilde {p} (y _ {1} = 1 | x) + (1 - \alpha) \widetilde {p} (y _ {1} = 2 | x)}, \\ \end{array}
$$
$$
\begin{array}{l} \widetilde {p} \left(y _ {2} = 2 \mid x; \alpha\right) = \frac {(1 - \alpha) p _ {2} (x)}{\alpha p _ {1} (x) + (1 - \alpha) p _ {2} (x)} = \frac {(1 - \alpha) \frac {p _ {2} (x)}{p _ {1} (x) + p _ {2} (x)}}{\alpha \frac {p _ {1} (x)}{p _ {1} (x) + p _ {2} (x)} + (1 - \alpha) \frac {p _ {2} (x)}{p _ {1} (x) + p _ {2} (x)}} \tag {30b} \\ = \frac {(1 - \alpha) \widetilde {p} (y _ {1} = 2 | x)}{\alpha \widetilde {p} (y _ {1} = 1 | x) + (1 - \alpha) \widetilde {p} (y _ {1} = 2 | x)}. \\ \end{array}
$$
These two observations combined suggest the training procedure where 1) the terminal state classifier is trained to approximate  $\widetilde{p}(y_1 = i|x)$  in the same way as in Section 5.2; 2) the probability estimates  $w_i(\hat{x},\hat{\alpha};\phi) \approx \widetilde{p}(y_2 = i;\hat{\alpha})$  are expressed through the learned terminal state classifier  $\widetilde{p}(y_1 = i|x)$  via (30). Below we provide details of this procedure for the case of GFlowNet composition.
Learning the terminal state classifier. The marginal  $y_{1}$  classifier  $\widetilde{Q}_{\phi}(y_1|x)$  is learned by minimizing the cross-entropy loss
$$
\mathcal {L} _ {T} (\phi) = \underset {(\hat {x}, \hat {y} _ {1}) \sim \widetilde {\rho} (x, y _ {1})} {\mathbb {E}} \left[ - \log \widetilde {Q} _ {\phi} \left(y _ {1} = \hat {y} _ {1} \mid x = \hat {x}\right) \right]. \tag {31}
$$
Then, the joint classifier  $\widetilde{Q}_{\phi}(y_1,y_2|x;\alpha)$  is constructed as
$$
\widetilde {Q} _ {\phi} \left(y _ {1}, y _ {2} \mid x; \alpha\right) = \widetilde {Q} _ {\phi} \left(y _ {1} \mid x\right) \widetilde {Q} _ {\phi} \left(y _ {2} \mid x; \alpha\right), \tag {32}
$$
where  $\widetilde{Q}_{\phi}(y_2|x;\alpha)$  can be expressed through the marginal  $\widetilde{Q}_{\phi}(y_1|x)$  via (30).
Learning the non-terminal state classifier. The non-terminal state classifier  $\widetilde{Q}(y_1, y_2 | s; \alpha)$  models  $y_1$  and  $y_2$  jointly. Note that  $\alpha$  is one of the inputs to the classifier model. Given a sampled trajectory  $\hat{\tau}$ , labels  $\hat{y}_1, \hat{y}_2$ , and  $\hat{\alpha}$ , the total cross-entropy loss of all non-terminal states in  $\hat{\tau}$  is
$$
\ell (\hat {\tau}, \hat {y} _ {1}, \hat {y} _ {2}, \hat {\alpha}; \phi) = \sum_ {t = 0} ^ {| \tau | - 1} \left[ - \log \widetilde {Q} _ {\phi} \left(y _ {1} = \hat {y} _ {1}, y _ {2} = \hat {y} _ {2} \mid s = \hat {s} _ {t}; \hat {\alpha}\right) \right]. \tag {33}
$$
(a) Base distributions  $p_1, p_2$
(b)  $\mathrm{HM}_{\mathrm{(ours)}}$  ,Product
(c) Contrasts (ours)  $p_1\mathbf{Q}_{(1 - \alpha)}p_2$
Figure C.1: Compositional sculpting and energy operations applied to 1D Gaussian distributions. (a) Densities of base 1D Gaussian distributions  $p_1(x) = \mathcal{N}(x; -5/4, 1)$  and  $p_2(x) = \mathcal{N}(x; 5/4, 1/2)$ . (b) harmonic mean  $p_1 \otimes p_2$  and product  $p_1$  prod  $p_2$ . (c) parameterized contrasts  $p_1 \odot_{(1-\alpha)} p_2$  at different values of  $\alpha$ . (d) negations  $p_1 \neg \gamma$ ,  $p_2$  at different values of  $\gamma$ . Curves show the PDFs of distributions.
(d) Negations  $p_1 \operatorname{neg}_\gamma p_2$
The pairs  $(\widehat{\tau}, \widehat{y}_1)$  can be generated via a sampling scheme similar to the one used for the terminal state classifier loss above: 1)  $\widehat{y}_1 \sim \widetilde{p}(y_1)$  and 2)  $\widehat{\tau} \sim p_{\widehat{y}_1}(\tau)$ . An approximation of the distribution of  $\widehat{y}_2$  given  $\widehat{\tau}$  is constructed using (30):
$$
w _ {1} (\widehat {x}, \widehat {\alpha}; \phi) = \frac {\widehat {\alpha} \widetilde {Q} _ {\phi} (y _ {1} = 1 | x = \widehat {x})}{\widehat {\alpha} \widetilde {Q} _ {\phi} (y _ {1} = 1 | x = \widehat {x}) + (1 - \widehat {\alpha}) \widetilde {Q} _ {\phi} (y _ {1} = 2 | x = \widehat {x})} \approx \widetilde {p} (y _ {2} = 1 | x = \widehat {x}; \widehat {\alpha}), \qquad (3 4 a)
$$
$$
w _ {2} (\hat {x}, \hat {\alpha}; \phi) = \frac {(1 - \hat {\alpha}) \widetilde {Q} _ {\phi} (y _ {1} = 2 | x = \hat {x})}{\hat {\alpha} \widetilde {Q} _ {\phi} (y _ {1} = 1 | x = \hat {x}) + (1 - \hat {\alpha}) \widetilde {Q} _ {\phi} (y _ {1} = 2 | x = \hat {x})} \approx \widetilde {p} (y _ {2} = 2 | x = \hat {x}; \hat {\alpha}). \qquad (3 4 b)
$$
Since these expressions involve outputs of the terminal state classifier which is being trained simultaneously, we again (see Section 5.2) introduce the target network parameters  $\overline{\phi}$  that are used to compute the probability estimates (34).
The training loss for the non-terminal state classifier is
$$
\mathcal {L} _ {N} (\phi , \bar {\phi}) = \underset {\hat {\alpha} \sim p (\alpha) (\hat {\tau}, \hat {y} _ {1}) \sim \tilde {p} (\tau , y _ {1})} {\mathbb {E}} \underset {\hat {y} _ {2} = 1} {\mathbb {E}} \left[ \sum_ {\hat {y} _ {2} = 1} ^ {2} w _ {\hat {y} _ {2}} (\hat {x}, \hat {\alpha}; \bar {\phi}) \ell (\hat {\tau}, \hat {y} _ {1}, \hat {y} _ {2}, \hat {\alpha}; \phi) \right], \tag {35}
$$
where  $p(\alpha)$  is sampling distribution over  $\alpha \in (0,1)$ . In our experiments, we used the following sampling scheme for  $\alpha$ :
$$
\widehat {z} \sim U [ - B, B ], \quad \widehat {\alpha} = \frac {1}{1 + \exp (- \widehat {z})}. \tag {36}
$$
C Analysis of Compositional Sculpting and Energy Operations
The harmonic mean and contrast operations we have introduced are analogous to the product and negation operations for EBMs respectively. Although the harmonic mean and product operations are quite similar in practice, unlike the negation operation our proposed contrast operation always results in a valid probability distribution. Figure C.1 shows the results of these operations applied to two Gaussian distributions. The harmonic mean and product, shown in panel (b), are both concentrated on points that have high probability under both Gaussians. Figure C.1(c) shows parameterized contrasts  $p_1 \mathbf{0}_{(1 - \alpha)} p_2$  at different values of  $\alpha$ , and panel (d) shows negations  $p_1 \neg \mathrm{g}_{\gamma} p_2$  at different values of  $\gamma$ . The effect of negation at  $\gamma = 0.1$  resembles the effect of the contrast operation: the density retreats from the high likelihood region of  $p_2$ . However, as  $\gamma$  increases to 0.5 the distribution starts to concentrate excessively on the values  $x < -3$ . This is due to the instability of division  $p_1(x) / (p_2(x))^{\gamma}$  in regions where  $p_2(x) \to 0$ . Proposition C.1 shows that negation  $p_1 \neg \mathrm{g}_{\gamma} p_2$  in many cases results in an improper (non-normalizable) distribution.
Mathematical analysis of operations. Harmonic mean and product are not defined for pairs of distributions  $p_1, p_2$  which have disjoint supports. In such cases, attempts at evaluation of the expressions for  $p_1 \otimes p_2$  and  $p_1$  prod  $p_2$  will lead to impossible probability distributions that have zero
probability mass (density) everywhere<sup>3</sup>. The result of both harmonic mean and product are correctly defined for any pair of distributions  $p_1, p_2$  that have non-empty support intersection.
Notably, contrast is well-defined for any input distributions while negation is ill-defined for some input distributions  $p_1, p_2$  as formally stated below (see Figure C.1 (d) for a concrete example).
Proposition C.1.
1. For any  $\alpha \in (0,1)$  the parameterized contrast operation  $p_1\mathbf{O}_{(1 - \alpha)}$ $p_2$  (7) is well-defined: gives a proper distribution for any pair of distributions  $p_1$ ,  $p_2$ .
2. For any  $\gamma \in (0,1)$  there are infinitely many pairs of distributions  $p_1$ ,  $p_2$  such that the negation
$$
\left(p _ {1} \operatorname {n e g} _ {\gamma} p _ {2}\right) (x) \propto \exp \left\{- \left(E _ {1} (x) - \gamma E _ {2} (x)\right) \right\} \propto \frac {p _ {1} (x)}{\left(p _ {2} (x)\right) ^ {\gamma}}. \tag {37}
$$
results in an improper (non-normalizable) distribution.
Proof. Without loss of generality, we prove the claims of the proposition assuming absolutely continuous distributions  $p_1, p_2$  with probability density functions  $p_1(\cdot), p_2(\cdot)$ .
Claim 1. For any two distributions  $p_1, p_2$  we have  $p_1(x) \geq 0$ ,  $p_2(x) \geq 0$ ,  $\int_{\mathcal{X}} p_1(x) dx = \int_{\mathcal{X}} p_2(x) dx = 1 < \infty$ . Then, the RHS of the expression for the parameterized contrast operation  $p_1 \mathbf{O}_{(1 - \alpha)} p_2$  (7) satisfies
$$
\frac {p _ {1} (x) ^ {2}}{\alpha p _ {1} (x) + (1 - \alpha) p _ {2} (x)} = \frac {p _ {1} (x)}{\alpha} \cdot \underbrace {\frac {p _ {1} (x)}{p _ {1} (x) + \frac {(1 - \alpha)}{\alpha} p _ {2} (x)}} _ {\leq 1} \leq \frac {p _ {1} (x)}{\alpha}, \quad \forall x \in \operatorname {s u p p} \left(p _ {1}\right) \cup \operatorname {s u p p} \left(p _ {2}\right). \tag {38}
$$
For points  $x \notin \operatorname{supp}(p_1) \cup \operatorname{supp}(p_2)$ , we set  $\frac{p_1(x)^2}{\alpha p_1(x) + (1 - \alpha)p_2(x)} = 0$  since by construction the composite distributions do not have probability mass outside of the union of the supports of the original distributions. The above implies that
$$
\int_ {\mathcal {X}} \frac {p _ {1} ^ {2} (x)}{\alpha p _ {1} (x) + (1 - \alpha) p _ {2} (x)} d x \leq \frac {1}{\alpha} \int_ {\mathcal {X}} p _ {1} (x) d x = \frac {1}{\alpha} <   \infty . \tag {39}
$$
Therefore, the RHS of the expression for the parameterized contrast operation  $p_1 \mathbf{O}_{(1 - \alpha)} p_2$  (7) can be normalized, and the distribution  $p_1 \mathbf{O}_{(1 - \alpha)} p_2$  is well-defined.
Claim 2. For any  $\gamma \in (0,1)$  we provide an infinite collection of distribution pairs  $p_1$  and  $p_2$  such that negation  $p_1\mathrm{neg}_\gamma p_2$  results in a non-normalizable distribution.
For the given  $\gamma \in (0,1)$  we select four numbers  $\mu_1\in \mathbb{R}$ ,  $\mu_{2}\in \mathbb{R}$ ,  $\sigma_{1} > 0$ ,  $\sigma_{2} > 0$  such that
$$
\sigma_ {1} ^ {2} \geq \frac {1}{\gamma} \sigma_ {2} ^ {2}, \tag {40}
$$
Consider univariate normal distributions  $p_1(x) = \mathcal{N}(x; \mu_1, \sigma_1^2)$ ,  $p_2(x) = \mathcal{N}(x; \mu_2, \sigma_2^2)$  with density functions
$$
p _ {i} (x) = \mathcal {N} \left(x; \mu_ {i}, \sigma_ {i} ^ {2}\right) = \frac {1}{\sqrt {2 \pi \sigma_ {i} ^ {2}}} \exp \left\{- \frac {\left(x - \mu_ {i}\right) ^ {2}}{2 \sigma_ {i} ^ {2}} \right\}, \quad i \in \{1, 2 \}. \tag {41}
$$
For such  $p_1$  and  $p_2$ , the RHS of (37) is