SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B.4 Particles versus MC simulations
There is an important distinction between the number of particles and the number of MC simulations. In this work, we assume that the participant uses a single particle for inferring the cluster assignments, but this is not observed from the experimenter's viewpoint. The reported internal construct may be associated with one out of many possible cluster
$^{2}$ https://github.com/stevengj/nlopt
Table 2: Table of all model parameters.
Parameter name	Symbol	DEF component	Equation	Log-space?	In default?	Notes
Expansion rate	α0	rational	(1)	Y	Y
Decay rate	r	rational	(1)	N	N	optimized in economical, fixed to 0 in exchangeable
Cluster count distortion	β	rational	(1)	Y	N	optimized in economical, fixed to 1 in exchangeable
Prior mean	μ0	rational	(2)	-	-	fixed to zero, not fitted
Pseudocount of prior mean	λ0	rational	(2)	Y	Y
Prior variance	σ0	rational	(2)	Y	Y
Pseudocount of prior variance	a0	aleatoric	(2)	Y	Y
Slack on cluster count	ε	aleatoric	(5)	Y	Y
Concentration scaling in reporting cluster weight	c	aleatoric	(6), (12)	Y	Y
Gaussian noise variance in reporting cluster mean	σv	aleatoric	(6), (13)	Y	Y
Gaussian noise variance in reporting cluster log variance	σm	aleatoric	(6), (14)	Y	Y
Visual noise variance	σn	-	(7)	Y	Y
Poisson prior mean	K	batch rational	(9)	Y	N	only in the baseline batch ICP
Weight on model size penalty	γ	batch rational	(10)	Y	N	only in the baseline batch ICP
Cluster weight prior	-	batch rational	-	Y	N	only in the baseline batch ICP
assignments. As such, we run many simulations of the single-particle particle filter. The number of MC simulations is  $M$ , and each simulation is indexed by  $i$ . Since we stick to the single-particle assumption throughout the paper, we do not need an additional index for the number of particles within each MC simulation.
Future work may consider multiple particles. In this case, each MC simulation will contain multiple sequences of cluster assignments. Maintaining multiple cluster assignment sequences allows re-assignment or re-weighting of the observations at each time step, achieving some retrospective correction. However, it is unknown how the participant makes a decision on reporting the cluster properties from multiple samples, especially when the number of clusters do not agree across different particles. We thus leave multiple-particle models for future work.
B.5 Additional modeling results
After training the DEFs, we evaluate the log-likelihood using a large number of simulations:  $10^{6}$  for the sequential DEFs in our main results, and  $10^{5}$  for the baseline batch DEF.
B.5.1 Ablation studies on DEF
Our initial experiment design used the exchangeable DEF (with a CRP-GMM rational component) as a reference model. We explored the model space by making small modifications to this DEF motivated by resource constraints and other heuristics. Here, we show that the two modifications in the economical DEF, namely the Decay in expansion rate  $\alpha_{t}$  and Distortion in cluster count  $n_{t,k}$  produce reliable improvements over the reference exchangeable DEF. The DEFs resulting from other modifications either did not produce reliable improvements or were insignificant compared to the reference model. All model parameters are listed in Table 2. We introduce the modifications below.
Decay: adding a model-size dependent decay rate  $r$ , as in (1).
Distort: adding an exponential transformation to the size of the clusters, as in (1).
Fixed Splitting: when  $\hat{K}^i < K^i$  and splitting a cluster, the parameters of the splitting are fixed with  $b_{\sigma} = \sqrt{3}/2$  and  $s_{\sigma} = 1/4$
Merge Cluster: when  $\hat{K}^i < K^i$ , instead of removing a cluster, the function  $f(\varphi, \hat{K})$  merges the smallest cluster into the cluster with closest mean. The cluster properties of the new component are computed by moment matching, as detailed in Appendix B.2.
Local MAP: instead of sampling the cluster assignment according to (4), take the cluster with largest posterior probability. This is the local MAP procedure described in [15].
Constant Variance: instead of updating the variance of each cluster according to (3), the cluster mean is fixed to the trainable parameter  $\sigma_0$
Fixed Mean: instead of updating the mean of each cluster according to (3), the mean is fixed at the first observation assigned to the cluster. This checks if participants are able to update the mean or simply remember the first observations assigned to the clusters.
Fixed Mean Confidence: instead of fitting  $\lambda_0$  as a parameter, we fix it at  $\lambda_0 = 0.01$
No Visual Noise: do not add Gaussian noise to the observations.
No Distribution Prior: when computing the per-step assignment posterior in (4), the likelihood is a Student's  $t$ -distribution obtained from having a conjugate prior over the Gaussian distribution parameters. Instead of computing this likelihood using the  $t$ -distribution, this modification computes this likelihood by a Gaussian with mean  $\mu_{t,k}$  and variance  $\sigma_{t,k}$ .
No Counting Prior: in (1), instead of using the  $n_{t,k}$  maintained for each cluster, use the average cluster size.
The results of these DEFs, averaged over participants, are shown in ??. Clearly, only Decay and Distort produced reliable improvement on AICs compared to the CRP-GMM DEF across all Experiments. All other modifications that deviate from a rational approximation
of the Bayes rule (e.g. Fixed Mean, No Counting Prior, etc.) resulted in significantly worse fit to the reported internal constructs from our participants. We also see that removing visual noise is detrimental to the quality of the fit.
The stability of the likelihood approximated from (7) is shown in Fig. 11. All DEFs fitting converged well as the number of restarts increases, except for a few worse DEFs in Experiment 3.
(a) Experiment 1
(b)Experiment 2
Figure 12: Model ablation comparisons for all Experiments. Lower values are better. Error bars are 1 sems. Blue bar indicates the reference DEF with CRP-GMM as the rational component. Green bars indicate significant differences to the reference model (Wilcoxon signed-rank test,  $p < 0.05$ ), and grey bars indicate insignificant comparison. The Proposed (Distort + Decay) is the economical ICP.
(c)Experiment 3
Figure 13: Distribution of fitted ICP parameters using data in Experiment 2.
B.5.2 Strong prior on cluster width
We found in most fitted ICPs that the prior cluster standard deviation is small (see Fig. 5 and Fig. 13), and the confidence indicated by its pseudocount  $a_0$  is much greater than zero. This high confidence contrasts with the low pseudocount associated with the prior cluster mean  $\lambda_0$ . This means that participants could learn the cluster mean mostly driven by the observations while failing to adapt to the cluster uncertainty. This contributes to the large number of clusters seen when there is a single Gaussian in the true data distribution. We noted that Gershman and Niv [35] also used a relatively high  $a_0 = 10$  parameter for their task.
[36] Sylvia Richardson and Peter J. Green. On Bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Statistical Society: Series B (Statistical Methodology), 1997.
[37] Christopher M. Bishop. Pattern Recognition and Machine Learning, Springer, 2006.