SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[50] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[51] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In International Conference on Learning Representations, 2018.
[52] Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object discovery. arXiv preprint arXiv:2306.00600, 2023.
[53] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[54] Christoph von der Malsburg. Binding in models of perception and brain function. Current opinion in neurobiology, 5(4):520-526, 1995.
[55] Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual review of neuroscience, 18(1):555-586, 1995.
[56] John G.R Jefferys, Roger D Traub, and Miles A Whittington. Neuronal networks for induced '40 hz' rhythms. Trends in Neurosciences, 19(5):202-208, 1996.
[57] Pascal Fries, Danko Nikolic, and Wolf Singer. The gamma cycle. Trends in neurosciences, 30 (7):309-316, 2007.
[58] Christoph von der Malsburg and Werner Schneider. A neural cocktail-party processor. Biological cybernetics, 54(1):29-40, 1986.
[59] Christoph von der Malsburg and Joachim Buhmann. Sensory segmentation with coupled neural oscillators. Biological cybernetics, 67(3):233-242, 1992.
[60] Andreas K Engel, Peter König, Andreas K Kreiter, Thomas B Schillen, and Wolf Singer. Temporal coding in the visual cortex: new vistas on integration in the nervous system. Trends in neurosciences, 15(6):218-226, 1992.
[61] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011.10287, 2020.
[62] Federico Baldassarre and Hossein Azizpour. Towards self-supervised learning of global and object-centric representations. arXiv preprint arXiv:2203.05997, 2022.
[63] Olivier J Henaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In European Conference on Computer Vision, pages 123–143, 2022.
Datasets. We evaluate all models (i.e., CAE, CAE++ and CtCAE) on a subset of Multi-Object dataset [41] suite. We use three datasets: Tetrominoes which consists of colored tetris blocks on a black background, dSprites with colored sprites of various shapes like heart, square, oval, etc., on a grayscale background, and lastly, CLEVR, a dataset from a synthetic 3D environment. For CLEVR, we use the filtered version [45] which consists of images containing less than seven objects sometimes referred to as CLEVR6 as in Locatello et al. [15]. We normalize all input RGB images to have pixel values in the range [0, 1] consistent with prior work [36].
Models. Table 6 shows the architecture specifications such as number of layers, kernel sizes, stride lengths, number of filter channels, normalization layers, activations, etc., for the convolutional neural networks used by the Encoder and Decoder modules in CAE,  $\mathrm{CAE} + +$  and CtCAE.
Table 6: Encoder and Decoder architecture specifications for CAE, CAE++ and CtCAE models.
Encoder
3 × 3 conv, 128 channels, stride 2, ReLU, BatchNorm
3 × 3 conv, 128 channels, stride 1, ReLU, BatchNorm
3 × 3 conv, 256 channels, stride 2, ReLU, BatchNorm
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 × 3 conv, 256 channels, stride 2, ReLU, BatchNorm
For 64×64 and 96×96 inputs, 2 additional encoder layers:
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 × 3 conv, 256 channels, stride 2, ReLU, BatchNorm
Linear Layer
Flatten
Linear, 256 ReLU units, LayerNorm
Decoder
For 64×64 and 96×96 inputs, 2 additional decoder layers:
Bilinear upsample x2
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
Bilinear upsample x2
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
Bilinear upsample x2
3 × 3 conv, 256 channels, stride 1, ReLU, BatchNorm
3 × 3 conv,128 channels, stride 1, ReLU, BatchNorm
Bilinear upsample x2
3 × 3 conv, 128 channels, stride 1, ReLU, BatchNorm
Output Layer (CAE only)
1 × 1 conv, 3 channels, stride 1, sigmoid
Training Details. Table 7 and Table 8 show the hyperparameter configurations used to train CAE/CAE++ and to contrastively train the CtCAE model(s) respectively.
Computational Efficiency. We report the training and inference time (wall-clock) for our models across the various image resolutions of the 3 multi-object datasets. First, we find that inference time(s) is similar for all models, and it only depends on the image resolution and number of ground truth clusters in the input image. Inference for the test set containing 320 images of  $32 \times 32$  resolution takes 25 seconds, for  $64 \times 64$  images it takes 65 seconds and for the  $96 \times 96$  images it takes 105 seconds. Training time(s) on the other hand differs both across models and image resolutions. We report all
Table 7: General training hyperparameters.
Hyperparameter	Tetrominoes	dSprites	CLEVR
Training Steps	50’000	100’000	100’000
Batch size	64	64	64
Learning rate	4e-4	4e-4	4e-4
Table 8: Contrastive learning hyperparameters for the CtCAE model.
Common Hyperparameters
Loss coefficient (in total loss sum)	1e-4
Temperature	0.05
Contrastive Learning Addresses	Magnitude
Contrastive Learning Features	Phase
Encoder Hyperparameters	32x32	64x64	96x96
Number of anchors	4	4	6
Number of positive pairs	1	1	1
Top-K to select positive pairs from	1	1	5
Number of negative pairs	2	2	6
Bottom-M to select negative pairs from	2	2	18
Decoder Hyperparameters	32x32	64x64	96x96
Patch Size	1	2	3
Number of anchors	100	100	100
Number of positive pairs	1	1	1
Top-K to select positive pairs from	5	5	5
Number of negative pairs	100	100	100
Bottom-M to select negative pairs from	500	500	500
training time(s) using a single Nvidia V100 GPU. CAE and  $\mathrm{CAE}++$  have similar training times: for  $32 \times 32$  images training for  $100 \mathrm{k}$  steps takes 3.2 hours, for  $64 \times 64$  images it takes 4.8 hours, and for  $96 \times 96$  images it takes 7 hours. CtCAE on the other hand takes 5 hours, 9.2 hours and 10.2 hours to train on dataset of  $32 \times 32$ ,  $64 \times 64$  and  $96 \times 96$  images respectively. To reproduce all the results/tables (mean and std-dev across 5 seeds) reported in this work we estimate the compute requirement to be  $840 \mathrm{GPU}$  hours in total. Further, we estimate that the total compute used in this project is approximately  $5 - 10 \mathrm{x}$  more, including experiments with preliminary prototypes.
Object Assignments from Phase Maps. We describe the process of extracting discrete-valued object assignments from continuous-valued phase components of decoder outputs  $\mathbf{y} \in \mathbb{C}^{h \times w \times 3}$ . First, the phase components of decoder outputs are mapped onto a unit circle and those phase values are masked out whose magnitudes are below the threshold value of 0.1. After this normalization and filtering step, the resultant phase values are converted from polar to Cartesian form on a per-channel basis. Finally, we apply K-means clustering where the number of clusters  $K$  parameter is retrieved from the ground-truth segmentation mask for each image. This extraction methodology of object assignments from output phase maps is consistent with Löwe et al. [36].
We show some additional experimental results: generalization capabilities of CtCAE w.r.t. the number of objects, and ablation studies on various design choices made in CAE++ and CtCAE models.
Table 9: Grouping results for CAE++ and CtCAE models with different threshold values applied to post-process the continuous output phase maps.
Dataset	Model	Threshold=0.1	Threshold=0.0	No threshold
ARI-FG ↑	ARI-FULL ↑	ARI-FG ↑	ARI-FULL ↑	ARI-FG ↑	ARI-FULL ↑
Tetrominoes	CAE++	0.78 ± 0.07	0.84 ± 0.01	0.77 ± 0.07	0.79 ± 0.02	0.54 ± 0.09	0.21 ± 0.05
CtCAE	0.84 ± 0.09	0.85 ± 0.01	0.86 ± 0.05	0.82 ± 0.01	0.67 ± 0.11	0.26 ± 0.09
dSprites	CAE++	0.38 ± 0.05	0.49 ± 0.15	0.38 ± 0.05	0.49 ± 0.12	0.37 ± 0.05	0.49 ± 0.12
CtCAE	0.48 ± 0.03	0.68 ± 0.13	0.46 ± 0.07	0.69 ± 0.10	0.47 ± 0.06	0.69 ± 0.10
CLEVR	CAE++	0.22 ± 0.10	0.30 ± 0.18	0.33 ± 0.04	0.32 ± 0.25	0.34 ± 0.04	0.32 ± 0.25
CtCAE	0.50 ± 0.05	0.69 ± 0.25	0.52 ± 0.05	0.69 ± 0.20	0.52 ± 0.06	0.72 ± 0.21
Table 10: Generalization evaluation on the CLEVR dataset. Training only on a subset, but evaluating on all possible subsets containing images with 3, 4, 5 or 6 objects.
Evaluation	ARI-FG↑	ARI-FULL↑
Training Subset: 4 or less objects
3 objects	0.39 ± 0.20	0.53 ± 0.30
4 objects	0.41 ± 0.18	0.55 ± 0.29
5 objects	0.38 ± 0.15	0.54 ± 0.28
6 objects	0.38 ± 0.13	0.55 ± 0.24
All images	0.39 ± 0.04	0.54 ± 0.28
Training Subset: 5 or less objects
3 objects	0.49 ± 0.04	0.69 ± 0.25
4 objects	0.50 ± 0.03	0.66 ± 0.24
5 objects	0.46 ± 0.02	0.65 ± 0.24
6 objects	0.45 ± 0.03	0.64 ± 0.23
All images	0.48 ± 0.02	0.66 ± 0.24
B.1 Generalization to Higher Number of Objects
Here we evaluate generalization capabilities of CtCAE by training only on a subset of images containing less than a certain number of objects, on CLEVR. We have two cases—training on subsets with either up to 4 or up to 5 objects while the original trainig split contain between 3 and 6 objects. The results in Table 10 show that CtCAE generalizes well. The performance drops only marginally when trained on up to 4 objects and tested on 5 and 6 objects, or when trained on up to 5 objects and tested on 6 objects. We also observe that training on 5 or less objects also consistently improves ARI scores for every subset of less than 5 objects. We suspect that the reason for this, apart from having more training data, is that the network has more pressure to separate objects in the phase space when observing more objects, which in turn also helps for images with a smaller number of objects.
B.2 Architecture Modifications
Table 11 shows the results from the ablation study that measures the effect of applying each of our proposed architectural modifications cumulatively to finally result in the CAE++ model. Across all datasets, we consistently observe that, starting from the vanilla CAE baseline which completely fails, grouping performance gradually improves as more components of the CAE++ model is added. Lastly, our proposed contrastive objective used to train CtCAE further improves CAE++ across all 3 datasets.
B.3 Contrastive Learning Ablations
Table 12 shows an ablation study for the choice of layer(s) to which we apply our contrastive learning method. Please note that all variants here always use magnitude components of complex-valued activations as addresses and phase components as features to contrast. We observe that the variant
Table 11: Grouping metrics achieved by various model variants from our proposed architectural modifications and resulting finally in the CAE++ model. Extends the results of Table 4 to all 3 multi-object datasets.
Dataset	Model	ARI-FG ↑	ARI-FULL ↑
Tetrominoes	CAE	0.00 ± 0.00	0.12 ± 0.02
CAE-(fout 1x1 conv)	0.00 ± 0.00	0.00 ± 0.00
CAE-(fout sigmoid)	0.12 ± 0.12	0.35 ± 0.36
CAE-transp.+upsamp.	0.10 ± 0.21	0.10 ± 0.22
CAE++ (above combined)	0.78 ± 0.07	0.84 ± 0.01
CtCAE	0.84 ± 0.09	0.85 ± 0.01
dSprites	CAE	0.02 ± 0.00	0.07 ± 0.01
CAE-(fout 1x1 conv)	0.06 ± 0.01	0.07 ± 0.07
CAE-(fout sigmoid)	0.02 ± 0.01	0.07 ± 0.01
CAE-transp.+upsamp.	0.19 ± 0.02	0.10 ± 0.04
CAE++ (above combined)	0.38 ± 0.05	0.49 ± 0.15
CtCAE	0.48 ± 0.03	0.68 ± 0.13
CLEVR	CAE	0.09 ± 0.05	0.08 ± 0.06
CAE-(fout 1x1 conv)	0.05 ± 0.01	0.06 ± 0.01
CAE-(fout sigmoid)	0.06 ± 0.02	0.01 ± 0.02
CAE-transp.+upsamp.	0.19 ± 0.07	0.10 ± 0.11
CAE++ (above combined)	0.22 ± 0.10	0.30 ± 0.18
CtCAE	0.50 ± 0.05	0.69 ± 0.25
which applies the contrastive objective to both the outputs of the encoder and decoder ('enc+dec') outperforms the others which apply it only to either one ('enc-only' or 'dec-only'). This behavior is consistent across Tetrominoes, dSprites and CLEVR. We hypothesize that this is because these two contrastive strategies are complementary: one is on low-level cues (decoder) and the other one based on high-level abstract features (encoder).
Table 13 shows an ablation study for the choice of using magnitude or phase as addresses (and the other one as features) in our contrastive learning method. As we apply contrastive learning to both the decoder and encoder outputs, we can make this decision independently for each output (resulting in four possible combinations). We observe that the variant which uses magnitude for both the encoder and decoder ( $\text{mg+mg}$ ) outperforms other variants across all 3 multi-object datasets. These ablations support our initial intuitions when designing our contrastive objective.
Table 12: Grouping metrics achieved by CtCAE model variants that apply the contrastive loss on output features from only the encoder (enc-only) or only the decoder (dec-only) or both (enc+dec) for all 3 multi-object datasets. Extends the results of Table 5 to all 3 multi-object datasets.
Dataset	Model	ARI-FG ↑	ARI-FULL ↑
Tetrominoes	CtCAE (enc-only)	0.81 ± 0.06	0.85 ± 0.01
CtCAE (dec-only)	0.74 ± 0.04	0.86 ± 0.00
CtCAE (enc+dec)	0.84 ± 0.09	0.85 ± 0.01
dSprites	CtCAE (enc-only)	0.40 ± 0.06	0.58 ± 0.05
CtCAE (dec-only)	0.48 ± 0.05	0.72 ± 0.07
CtCAE (enc+dec)	0.48 ± 0.03	0.68 ± 0.13
CLEVR	CtCAE (enc-only)	0.21 ± 0.11	0.29 ± 0.15
CtCAE (dec-only)	0.38 ± 0.17	0.69 ± 0.18
CtCAE (enc+dec)	0.50 ± 0.05	0.69 ± 0.25
Table 13: Grouping metrics on all 3 multi-object datasets achieved by CtCAE model variants that use either magnitude or phase components of the encoder/decoder outputs as the addresses for contrastive learning. For example, 'mg+ph' means that magnitude components used as addresses of the encoder outputs and phase components used as addresses of the decoder outputs (conversely, the phase components of the encoder outputs are used as features and the magnitude components of the decoder outputs are used as features).
Dataset	Model	ARI-FG ↑	ARI-FULL ↑
Tetrominoes	CtCAE w/ mg+mg	0.84 ± 0.09	0.85 ± 0.01
CtCAE w/ ph+ph	0.85 ± 0.05	0.84 ± 0.01
CtCAE w/ mg+ph	0.86 ± 0.03	0.85 ± 0.02
CtCAE w/ ph+mg	0.77 ± 0.04	0.85 ± 0.01
dSprites	CtCAE w/ mg+mg	0.48 ± 0.03	0.68 ± 0.13
CtCAE w/ ph+ph	0.38 ± 0.08	0.42 ± 0.16
CtCAE w/ mg+ph	0.36 ± 0.07	0.57 ± 0.13
CtCAE w/ ph+mg	0.45 ± 0.05	0.67 ± 0.18
CLEVR	CtCAE w/ mg+mg	0.50 ± 0.05	0.69 ± 0.25
CtCAE w/ ph+ph	0.22 ± 0.06	0.22 ± 0.09
CtCAE w/ mg+ph	0.18 ± 0.08	0.28 ± 0.15
CtCAE w/ ph+mg	0.40 ± 0.15	0.65 ± 0.19
C Additional Visualizations
We show additional visualization samples of groupings from CAE,  $\mathrm{CAE} + +$  and CtCAE on Tetrominoes, dSprites and CLEVR, and qualitatively highlight both grouping failures and successes of CAE,  $\mathrm{CAE} + +$  and CtCAE.
Figure 5: Visualizations for CAE (row 1), CAE++ (row 2) and CtCAE (row 3) on the Tetrominoes dataset example from Figure 3. We see that CtCAE achieves near perfect grouping (column 4) and better separability in phase space (column 5) compared to our improved CAE++ variant which shows significant grouping interference (parts of all three objects modelled by same group, e.g., orange cluster). We also observe that the baseline CAE completely fails to reconstruct or group the image.
Figure 6: Visualizations for CAE (row 1), CAE++ (row 2) and CtCAE (row 3) on the dSprites dataset example from Figure 3. We see that CtCAE successfully separates the 4 scene objects into 4 clusters while CAE++ mistakenly groups 2 scene objects into 1 cluster, i.e., the blue cluster (column 4). Again, the baseline CAE fails to reconstruct (no color) or group the image.
Figure 7: Visualizations for CAE (row 1), CAE++ (row 2) and CtCAE (row 3) on the CLEVR dataset example from Figure 3. We see that the CtCAE in this example is able to separate the 4 scene objects into 4 distinct clusters while our improved CAE++ fails (column 4). This good separation in phase space is reflected in the continuous phase maps (column 6) where the pixel colors in the phase maps are representative of their values/differences when comparing CtCAE and CAE++. Finally, we see that although the CAE is able to reasonably reconstruct the image it shows very poor grouping of the scene objects.
Figure 8: Visualization on grouping performance for CAE,  $\mathrm{CAE} + +$  and CtCAE on images containing same colored objects on Tetrominoes. CAE struggles to even reconstruct the images, whereas  $\mathrm{CAE} + +$  often groups objects with the same colour together. CtCAE on the other hand has no problem separating objects of the same colour on Tetrominoes.
Figure 9: Visualization on grouping performance for CAE,  $\mathrm{CAE}++$  and CtCAE on Tetrominoes. CtCAE shows a larger spread in phase values for different clusters (column 5) compared to CAE. We hypothesize that this facilitates CtCAE to perform better grouping in the scenario of multiple object instances with the same color.
Figure 10: Visualization on grouping performance for CAE,  $\mathrm{CAE}++$  and CtCAE on images containing similar colored objects on dSprites. CAE has very poor grouping capability, whereas  $\mathrm{CAE}++$  often groups objects with the same colour together. For example, purple ellipse and square in the first example (top panel) or red and brown hearts grouped together in the last example (bottom panel) are grouped together. CtCAE on the other hand has significantly better performance, grouping correctly objects with the same (or similar) colors, except in the first example (top panel) where it groups the pink square and ellipse together.
Figure 11: Performance for CAE, CAE++ and CtCAE on images containing many objects on dSprites. CAE shows poor grouping performance, as noted previously. CtCAE has no issues grouping up to 5 objects in the same image, whereas CAE++ struggles, often grouping multiple objects (typically of same or similar colors) together into the same cluster.
Figure 12: Visualization on grouping performance for CAE, CAE++ and CtCAE on CLEVR dataset. We are particularly interested in the phase space spread (column 5) here. CAE has a very large spread in phase values (column 5), but almost random object discovery performance, often grouping all objects together. But if we see the actual phase values assigned to objects (column 6), we observe that CAE tends to assign very similar phase values (e.g. all object phases' are in shades of magenta seen in column 6, rows 1, 4, and 7) to regions belonging to different objects whereas the corresponding maps show better separation in CAE++ and much superior in CtCAE. This explains better grouping performance (column 4) shown by CAE++ and CtCAE over the vanilla CAE.
Figure 13: Visualization on grouping performance for CAE, CAE++ and CtCAE on samples with 4, 5 and 6 objects from the CLEVR dataset. CAE consistently shows poor object discovery, often grouping all objects together. CAE++ improves grouping substantially over the CAE baseline, but also struggles in certain scenarios. For example, in the scene with 4 objects it groups two different objects together, whereas for scenes with 5 and 6 objects on two samples here we see that it wrongly groups two objects together. CtCAE improves on these grouping failures of CAE++. CtCAE correctly discovers all objects in the scene with 4 and 5 objects, even though it makes an error grouping two objects together in the hardest example scene containing 6 objects.
Figure 14: One of the limitations (failure modes) we observe is that in some images containing many objects, with some of them having the same (or similar) colors, our CtCAE model groups two such objects into the same cluster. Even in this example, CtCAE still clearly outperforms  $\mathrm{CAE}++$ , which in turn outperforms the CAE baseline.