SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a)  $p_0$ : Even Digits
Figure G.3: Composing even digits and multiples of three. (a,b) samples from base diffusion models. (c-e) samples from the resulting harmonic mean and contrast compositions.
(b)  $p_2\colon \{0,3,6,9\}$
(c)  $p_0\otimes p_2$
(d)  $p_0\bullet p_2$
(e)  $p_0\mathbf{O}p_2$
(a)  $p_1$ : OddDigits
Figure G.4: Composing odd digits and multiples of three. (a,b) samples from base diffusion models. (c-e) samples from the resulting harmonic mean and contrast compositions.
G.3 Chaining: Sequential Composition of Multiple Distributions
We present results on chaining binary composition operations sequentially on a custom colored MNIST dataset.
Setup. We start with three base generative models that are trained to generate  $p_1$ ,  $p_2$  and  $p_3$  in Figure G.5. Specifically,  $p_1$  is a uniform distribution over digits  $\{0, 1, 2, 3, 4, 5\}$ ,  $p_2$  is a uniform distribution over even digits  $\{0, 2, 4, 6, 8\}$ , and  $p_3$  is a uniform distribution over digits divisible by 3:  $\{0, 3, 6, 9\}$ . Note that we use a different color for each digit consistent across  $p_1$ ,  $p_2$ ,  $p_3$ . Our goal is to produce combinations of chained binary operations involving all three distributions, where two of them were combined first, then in a second step, combined with the third distribution through either harmonic mean  $\otimes$  or contrast  $\mathbf{1}$ .
Binary Classifier Training. Consider, for example, the operation  $(p_1 \otimes p_2) \oplus p_3$ . We use the same classifier training procedure for  $p_1$  versus  $p_2$ , as well as the composite model  $(p_1 \otimes p_2)$  versus  $p_3$ , except that in the later case we sample from composite model as a whole. Our classifier training simultaneously optimizes the terminal classifier and the intermediate state classifier.
Implementation Detains. We adapted diffusion model training code for the base distributions from [33]. Our diffusion model used a UNet backbone with four latent layers on both the contracting and the expansive path. The contracting channel dimensions were [64, 128, 256, 256] with the kernel size 3, and strides [1, 2, 2, 2]. The time embedding used a mixture of 128 sine and cosine feature pairs, with a total of 256 dimensions. These features were passed through a sigmoid activation and then expanded using a different linear head for each layer. The activations were then added to the 2D feature maps at each layer channel-wise. We used a fixed learning rate of 0.01 with Adam optimizer [59].
We adapted the classifier training code from the MNIST example in Pytorch [68]. Our binary classifier has two latent layers with channel dimensions [32, 64], stride 1 and kernel width of 3. We use dropout on both layers:  $p_1 = 25\%$ ,  $p_2 = 50\%$ . We train the network for 200 epochs on data sampled online in batches of 256 from each source model, and treat the composite model in the second step the same way. We use the Adadelta [67] optimizer with the default setting of learning rate 1.0.
Sampling. We generate samples according to Appendix A.2 and multiply the gradient by  $\alpha = 20$ .
Results. Row 3 from Figure G.5 contains mostly zeros with a few exceptions. This is in agreement with harmonic mean being symmetric. In  $p_1 \otimes (p_2 \otimes p_3)$ , digits that are outside of the intersection still appear with thin strokes.
Figure G.5: Chaining Binary Operations. (a-c) Samples from 3 pre-trained diffusion models. (d-1) Samples from binary compositions. (row 3) The harmonic mean between all three. (row 4 and beyond) various ways to chain the operations. Parentheses indicate the order of composition.
Figure G.6: Training curves of the classifier  $\widetilde{Q}_{\phi}(y_1, y_2|\cdot)$  in GFlowNet 2D grid domain. The experimental setup corresponds to Section 6 ("2D distributions via GFlowNet") and Figure 2 (top row). Left: Terminal state loss and non-terminal state loss (as defined in Algorithm A.1) as functions of the number of training steps. Right:  $L_{1}$  distance between learned distributions (compositions obtained through classifier-based mixture and guidance) and ground-truth composition distributions as the function of the number of training steps.  $L_{1}(p,q) = \sum_{x \in \mathcal{X}} |p(x) - q(x)|$ .
Figure G.7: Training curves of the classifier  $\widetilde{Q}_{\phi}(y_1,y_2|\cdot)$  in GFlowNet molecule generation domain. The experimental setup corresponds to Section 6 ("Molecule generation via GFlowNet") and Figure 3 (a-c). The curves show terminal state loss and non-terminal state loss (as defined in Algorithm A.1) as functions of the number of training steps.
Figure G.8: Training curves of the classifier  $\widetilde{Q}_{\phi}(y_1,y_2|\cdot)$  in diffusion MNIST image generation domain. The experimental setup corresponds to Section G.1 and Figure G.1. The curves show terminal state loss and non-terminal state loss (as defined in equations (26), (28)) as functions of the number of training steps. The nonterminal loss optimization begins after the first 100 training steps (shown by the black dashed line).
G.4 Classifier Learning Curves and Training Time
We empirically evaluated classifier training time and learning curves. The results are shown in Figures G.6, G.7, G.8 and Tables G.1, G.2.
Figures G.6, G.7, G.8 show the cross-entropy loss of the classifier for terminal (16) and non-terminal states (18) as a function of the number of training steps for the GFlowNet 2D grid domain, the molecular generation domain, and the Colored MNIST digits domain respectively. They show that the loss drops quickly but remains above 0. Figure G.6 further shows the distances between the learned compositions and the ground truth distributions as a function of the number of training steps of the classifier. For all compositions, as the classifier training progresses, the distance to the ground truth distribution decreases. Compared to the distance at initialization we observe almost an order of magnitude distance reduction by the end of the training.
The runtime of classifier training is shown in Tables G.1 and G.2. We report the total runtime, as well as separate measurements for the time spent sampling trajectories and training the classifier. The classifier training time is comparable to the base generative model training time. However, most of the classifier training time (more than  $70\%$ , or even  $90\%$ ) was spent on sampling trajectories from base generative models. Our implementation of the training could be improved in this regard, e.g. by sampling a smaller set of trajectories once and re-using trajectories for training and by reducing the number of training steps (the loss curves in Figures G.6, G.7, G.8 show that classification losses plateau quickly).
Table G.1: Summary of base GFlowNet and classifier training time in molecule generation domain. The experimental setup corresponds to Section 6 ("Molecule generation via GFlowNet") and Figure 3 (a-c). All models were trained with a single GeForce RTX 2080 Ti GPU.
Base GFlowNet training steps	20000
Base GFlowNet batch size	64
Base GFlowNet training elapsed real time	6h 47m 11s
Classifier training steps	15000
Classifier batch size	8 trajectories per base model (all states used)
Classifier training total elapsed real time	9h 2m 19s
Classifier training data generation time	6h 35m 58s (73%)
Table G.2: Summary of base diffusion and classifier training time in MNIST image generation domain. The experimental setup corresponds to Section G.1 and Figure G.1. All models were trained with a single Tesla V100 GPU.
Base diffusion training steps	200
Base diffusion batch size	32
Base diffusion training elapsed real time	10m 6s
Classifier training steps	200
Classifier batch size	128 trajectories per base model (35 time-steps per trajectory)
Classifier training total elapsed real time	30m 12s
Classifier training data generation time	29m 22s (97%)
Table G.3: Average pairwise similarity [36] of molecules generated by GFlowNets trained on 'SEH', 'SA', 'QED' rewards at different values of  $\beta$ . For each combination (reward,  $\beta$ ) a GFlowNet was trained with the corresponding reward  $R(x)^{\beta}$ . Then, 5000 molecules were generated. The numbers in the table reflect the average pairwise Tanimoto similarity of top 1000 molecules (selected according to the target reward function).
SEH	SA	QED
β = 1	0.527	0.539	0.480
β = 4	0.529	0.527	0.464
β = 10	0.535	0.500	0.438
β = 16	0.548	0.465	0.422
β = 32	0.585	0.411	0.398
β = 96	0.618	0.358	0.404
Table G.4: Number of Tanimoto-separated modes found above reward threshold. For each combination (reward,  $\beta$ ) a GFlowNet was trained with the corresponding reward  $R(x)^{\beta}$ , and then 5000 molecules were generated. Cell format is " $A/B"$ , where  $A$  is the number of Tanimoto-separated modes found above the reward threshold, and  $B$  is the total number of generated molecules above the threshold. Analogously to Figure 14 in [36], we consider having found a new mode representative when a new molecule has Tanimoto similarity smaller than 0.7 to every previously found mode's representative molecule. Reward thresholds (in [0, 1], normalized values) are 'SEH': 0.875, 'SA': 0.75, 'QED': 0.75. Note that the normalized threshold of 0.875 for 'SEH' corresponds to the unnormalized threshold of 7 used in [36].
SEH	SA	QED
β = 1	15 /	17	37 /	37	0 /	0
β = 4	12 /	17	82 /	82	0 /	0
β = 10	85 /	109	332 /	337	18 /	18
β = 16	190 /	280	886 /	910	253 /	253
β = 32	992 /	1821	2859 /	3080	3067 /	3124
β = 96	1619 /	4609	4268 /	4983	4470 /	4980
G.5 Analysis of Sample Diversity of Base GFlowNets in Molecule Generation Domain
In order to assess the effect of the reward exponent  $\beta$  on mode coverage and sample diversity, we evaluated samples generated from GFlowNets pre-trained with different values of  $\beta$ . The results are in Tables G.3 and G.4. The details of the evaluation and the reported metrics are described in the table captions. As expected, larger reward exponents shift the learned distributions towards high-scoring molecules (the total number of molecules with scores above the threshold increases). For 'SA' and 'QED' models we don't observe negative effects of large  $\beta$  on sample diversity and mode coverage: the average pairwise similarity of top 1000 molecules doesn't grow as  $\beta$  increases and the ratio of Tanimoto-separated modes remains high. For 'SEH' models we observe a gradual increase in the average pairwise similarity of top 1000 molecules and a gradual decrease in the ratio of Tanimoto-separated modes. However, the total number of separated modes grows as  $\beta$  increases, which indicates that larger reward exponents do not lead to mode dropping.
G.6 Summary of Pairwise Distribution Distances in Molecule Generation Domain
Table G.5: Estimated pairwise earth mover's distances between distributions shown in Table 1.
y=SEH	y=SA	y=QED	y=SEH,SA	y=SEH,QED	y=SA,QED	y=SEH,SA,QED	y=SEH×3	y=SA×3	y=QED×3
y=SEH	0	4.42	5.77	3.39	4.20	4.88	4.10	2.46	4.44	5.73
y=SA	4.42	0	5.88	3.26	5.15	4.59	4.20	4.39	2.55	5.89
y=QED	5.77	5.88	0	5.40	4.02	3.85	4.20	5.80	5.90	3.20
y=SEH,SA	3.39	3.26	5.40	0	4.25	4.19	3.68	3.39	3.30	5.39
y=SEH,QED	4.20	5.15	4.02	4.25	0	3.80	3.67	4.22	5.19	4.00
y=SA,QED	4.88	4.59	3.85	4.19	3.80	0	3.65	4.91	4.59	3.87
y=SEH,SA,QED	4.10	4.20	4.20	3.68	3.67	3.65	0	4.12	4.23	4.20
y=SEH×3	2.46	4.39	5.80	3.39	4.22	4.91	4.12	0	4.43	5.73
y=SA×3	4.44	2.55	5.90	3.30	5.19	4.59	4.23	4.43	0	5.90
y=QED×3	5.73	5.89	3.20	5.39	4.00	3.87	4.20	5.73	5.90	0