SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) Real-world datasets
Method	SVHN	IDRiD
ARI	mDice	mIoU	ARI	mDice	mIoU
SLIC	5.3±0.0	50.1±0.0	34.5±0.0	32.2±0.0	12.7±0.0	8.8±0.0
Felzenszwalb	31.7±0.0	51.6±0.0	39.8±0.0	14.7±0.0	19.0±0.0	15.4±0.0
Slot Attention	38.9±1.5	51.7±1.8	36.7±1.7	28.7±1.1	8.6±1.7	5.0±1.6
EfficientMORL	32.2±1.7	49.2±2.0	34.0±1.8	16.8±1.5	11.1±2.7	7.0±1.8
GENESIS-V2	28.6±1.4	60.8±1.5	45.9±1.4	18.3±1.6	8.8±1.9	5.4±1.6
SLATE	21.2±1.2	57.0±1.3	41.7±1.5	35.6±2.1	8.1±1.2	4.7±1.8
SysBinder	15.8±1.6	49.5±1.9	34.1±1.8	25.2±1.3	16.6±1.7	11.1±1.8
BO-QSA	24.3±1.2	62.0±1.6	48.3±1.3	27.7±2.0	7.0±1.9	4.5±1.7
OC-Net	39.7±0.1	64.6±0.1	49.9±0.1	39.0±0.4	38.1±0.2	31.2±0.2
(c) Complex textures dataset
Method	CLEVRTEX	CLEVRTEX-OOD
ARI	mDice	mIoU	ARI	mDice	mIoU
SLIC	27.4±0.0	20.0±0.0	13.0±0.0	25.8±0.0	21.7±0.0	14.0±0.0
Felzenszwalb	57.3±0.0	33.6±0.0	26.8±0.0	44.6±0.0	29.6±0.0	23.4±0.0
Slot Attention	58.6±1.6	35.0±1.6	26.7±1.5	51.3±1.9	34.1±1.4	25.1±1.3
EfficientMORL	59.5±1.7	37.7±1.5	31.1±1.4	53.9±2.5	32.2±2.4	25.3±2.8
GENESIS-V2	65.6±1.8	36.9±1.4	30.4±1.4	67.6±1.6	34.2±1.5	27.6±1.9
SLATE	57.5±1.8	33.3±1.6	24.4±1.5	56.6±1.3	34.7±2.1	25.3±1.8
SysBinder	61.4±1.7	31.3±1.5	23.1±1.4	61.0±2.4	32.3±2.0	23.8±1.8
BO-QSA	70.9±1.9	42.9±1.8	34.7±1.7	66.1±1.3	42.8±1.4	33.9±1.3
OC-Net	70.7±0.9	45.1±0.9	37.5±0.7	69.8±0.8	43.5±0.7	35.0±0.6
(d) Common objects datasets
Method	Flowers	Birds	COCO
Dice	IoU	Dice	IoU	mDice	mIoU
SLIC	30.5±0.0	18.4±0.0	33.1±0.0	20.3±0.0	36.2±0.0	24.4±0.0
Felzenszwalb	43.7±0.0	30.4±0.0	34.3±0.0	23.0±0.0	36.6±0.0	27.1±0.0
Slot Attention	43.0±1.5	28.6±1.2	42.9±2.0	27.9±1.8	24.8±2.0	15.0±1.7
EfficientMORL	59.5±2.1	45.2±2.2	44.0±1.9	30.8±1.8	28.4±2.3	18.9±2.1
GENESIS-V2	63.7±2.2	50.2±2.2	41.4±1.7	27.7±1.5	25.1±2.1	16.1±1.7
SLATE	55.6±1.2	40.8±1.8	39.5±1.5	25.9±1.8	37.0±1.9	24.4±1.8
SysBinder	45.0±1.8	30.8±1.6	33.7±1.3	21.1±2.0	18.4±1.6	10.7±1.4
BO-QSA	65.8±1.9	51.7±1.9	44.6±1.7	30.3±1.5	34.9±1.1	23.6±0.9
OC-Net	67.2±0.2	54.4±0.2	47.8±0.2	33.5±0.2	48.2±0.2	35.6±0.2
(a) Multi-dSprites
(b) Tetrominoes-NM
(c) SVHN
(d) IDRiD
(e) CLEVRTEX
(f) CLEVRTEX-OOD
(g) Flowers
(h) Birds
(i) COCO
Figure 2: Visualization of discovered objects.
4.3 Experiments on Model Generalizability
Ideally, an unsupervised object discovery model should be trained to understand common visual appearances such as the difference between foreground vs background, so as to discover objects from
(b) SVHN
(c) CLEVRTEX
Figure 3: mIoU scores vs decreasing number of training samples.
Table 3: mIoU scores for model generalizability after training on Multi-dSprites.
Method	Tetrominoes-NM	SVHN	IDRiD	CLEVRTEX	CLEVRTEX-OOD
Slot Attention	21.8±3.5	19.5±3.8	7.5±2.5	12.2±2.2	12.3±2.2
EfficientMORL	21.2±3.8	23.4±2.8	6.5±2.6	12.7±3.2	15.2±2.0
GENESIS-V2	42.9±4.9	31.1±2.8	8.5±2.4	21.9±1.6	21.3±2.5
SLATE	51.4±1.6	21.1±2.0	10.0±1.7	12.7±2.2	12.9±1.8
SysBinder	28.5±1.8	23.8±1.1	13.9±1.8	10.6±1.5	11.6±1.9
BO-QSA	41.8±1.8	24.3±2.0	4.0±1.5	24.4±1.4	22.8±2.5
OC-Net	100.0±0.0	47.5±0.5	29.1±0.5	31.7±0.6	31.3±0.6
out-of-domain images. In this set of experiments, we compare the generalization ability of OC-Net with the baselines by training the models on Multi-dSprites and testing them on the other datasets.
Table 3 shows the results. For Tetrominoes-NM, there is a decrease in performance for all methods while OC-Net still obtains perfect score. For SVHN, CLEVRTEX and CLEVRTEX-OOD, the performance of all models decreases due to the shift in data distribution. However, OC-Net experiences the smallest drop in performance and still significantly surpasses the best performing method. For IDRiD, the methods show improvement in performance. One possible reason is that the larger training set in Multi-dSprites enables the circular shape of the optic disc to be better segmented. Despite this, OC-Net remains the top performer.
Next, we examine the effect of feature connectivity and regularization terms  $\mathcal{L}_{sep}$  and  $\mathcal{L}_{ent}$  on the performance of OC-Net. We implemented three variants of OC-Net: (a) w/o connectivity. Here, we do not require that a path should exist between  $i$  and  $k$  when computing  $\mathrm{dist}[i,k]$ ; (b) w/o  $\mathcal{L}_{ent}$ . This network is trained without the entanglement regularization term; (c) w/o  $\mathcal{L}_{sep}$ . The separation regularization term is not used in the training of OC-Net.
Table 4 shows the mIoU scores for all the datasets. Without feature connectivity, we observe a drop in performance across all datasets since it is common for images to have multiple identical objects, that is, same color and shape. As such, OC-Net w/o connectivity tend to cluster these blocks as a single object.
Removing the entanglement term (OC-Net w/o  $\mathcal{L}_{ent}$ ) leads to a slight decrease in the performance as the object representations may still be separated even when the dimensions are entangled. The largest performance drop in all datasets is seen when the object representation separation term is removed (OC-Net w/o  $\mathcal{L}_{sep}$ ), indicating the importance of having a well-separated object representation space for effective object discovery.
Table 4: mIoU scores for variants of OC-Net.
Method	Multi-dSprites	Tetro-NM	SVHN	IDRiD	CTEX	CTEX-OOD
OC-Net	99.1±0.0	100.0±0.0	49.9±0.1	31.2±0.2	37.5±0.7	35.0±0.6
w/o connectivity	98.8±0.1	89.2±0.1	36.6±0.1	17.3±0.1	20.3±0.9	17.7±0.9
w/o Lent	98.7±0.3	99.6±0.1	48.8±0.1	25.8±0.2	26.4±0.8	18.9±0.4
w/o Lsep	49.3±0.3	51.2±0.2	35.0±0.2	4.0±0.1	18.5±0.6	11.0±0.4
5 Prediction based on Learned Object Representation
One characteristic of an effective object-centric representation is its ability to encode object properties such as color, position and shape [42]. In this section, we show that the learned object representations are disentangled and can be used to predict the values of these properties.
Given the representations and their corresponding ground truth values of a target object property, we employ them as features to train a gradient boosted tree (GBT) [16]. To evaluate how well the properties of unseen objects are predicted by the GBT, we use the coefficient of determination  $R^2$  [49]. We perform an experiment using the learned object representations from the simulated datasets to predict the properties of object such as color, position and shape. We use the mIoU score to match the discovered object to the ground truth object. We split the 320 test images equally into two sets, one for training the GBT model for each object property, and the other for evaluation.
Table 5 shows the average  $R^2$  scores of the GBT models on the evaluation set. The GBT models trained with OC-Net representations achieved the highest  $R^2$  scores compared to the models trained using the representations from other methods. This suggests that the object representations learned by OC-Net are effective in encoding the object properties.
Table 5:  $R^2$  scores for object property prediction on simulated datasets
Method	Multi-dSprites	Tetrominoes-NM
Color	Position	Shape	Color	Position	Shape
Slot Attention	72.2±12	96.8±0.1	38.2±0.0	86.5±6.5	98.7±0.6	36.3±0.0
EfficientMORL	86.5±6.2	95.8±0.1	61.7±0.0	94.9±3.2	97.9±0.7	68.5±0.0
GENESIS-V2	78.1±7.5	97.1±0.7	75.8±0.0	88.1±5.8	94.6±2.6	37.9±0.0
SLATE	87.5±0.7	90.6±4.4	31.7±0.0	85.5±3.9	89.6±0.7	10.5±0.0
SysBinder	73.6±1.0	69.3±3.4	33.3±0.0	97.9±0.6	77.8±2.7	19.9±0.0
BO-QSA	96.3±1.6	97.4±0.1	75.2±0.0	98.1±0.7	98.9±0.2	52.5±0.0
OC-Net	98.0±0.6	98.3±0.1	78.1±0.0	100.0±0.0	99.4±0.1	98.7±0.0
In this work, we have described a framework called OC-Net that learns object-centric representations in a fine-grained manner without supervision. OC-Net leverages on feature connectivity and two new regularization terms to learn disentangled representations and to ensure the representations of different objects are well-separated. From the results of experiments conducted on simulated, real-world, complex texture and common object images, we have demonstrated the superior quality of the object representations over current state-of-the-art. Moreover, we have highlighted the sample efficiency and generalizability of OC-Net. Finally, we have shown how the discovered object representations can be used to predict object properties in a downstream task, indicating its potential use for other computer vision applications where samples and ground truth labels are limited.
There are still obstacles that have to be overcome for successful application of our framework to the full visual complexity of the real world. A natural next step would be to extend OC-Net to handle real-world scenes containing objects with more complex part-whole hierarchies. It is also promising to explore explicit representation of the discovered objects into a dictionary of prototypes to better handle occlusion between objects. Lastly, real-world scenes with multiple objects is still of higher visual complexity than the datasets considered here and reliably bridging this gap is an open problem.
This research is supported by the National Research Foundation Singapore under its AI Singapore Programme (Award Number: AISG-GC-2019-001-2A).
[1] Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., and Süsskrunk, S. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274-2282, 2012. doi: 10.1109/TPAMI.2012.120.
[2] Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. Advances in NIPS 2016 Deep Learning Symposium, 2016.
[3] Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In The Tenth International Conference on Learning Representations, ICLR, 2022.
[4] Bear, D., Fan, C., Mrowca, D., Li, Y., Alter, S., Nayebi, A., Schwartz, J., Fei-Fei, L. F., Wu, J., Tenenbaum, J., et al. Learning physical graph representations from visual scenes. Advances in Neural Information Processing Systems, 33:6027–6039, 2020.
[5] Burgess, C. P., Matthew, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M. M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. CoRR, 1901.11390, 2019.
[6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
[7] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
[8] Crawford, E. and Pineau, J. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3412-3420, 2019.
[9] Dittadi, A., Papa, S. S., Vita, M. D., Schölkopf, B., Winther, O., and Locatello, F. Generalization and robustness implications in object-centric learning. In International Conference on Machine Learning, ICML, 2022.
[10] Eastwood, C. and Williams, C. K. I. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[11] Emami, P., He, P., Ranka, S., and Rangarajan, A. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In International Conference on Machine Learning, pp. 2970–2981, 2021.
[12] Engelcke, M., Kosiorek, A. R., Jones, O. P., and Posner, I. GENESIS: generative scene inference and sampling with object-centric latent representations. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[13] Engelcke, M., Parker Jones, O., and Posner, I. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems, 34: 8085-8094, 2021.
[14] Eslami, S., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Hinton, G. E., et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29, 2016.
[15] Felzenszwalb, P. F. and Huttenlocher, D. P. Efficient graph-based image segmentation. International journal of computer vision, 59(2):167-181, 2004.
[35] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525-11538, 2020.
[36] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[37] Nilsback, M.-E. and Zisserman, A. Delving deeper into the whorl of flower segmentation. Image and Vision Computing, 28(6):1049-1062, 2010.
[38] Palmer, S. and Rock, I. Rethinking perceptual organization: The role of uniform connectedness. Psychonomic bulletin & review, 1(1):29-55, 1994.
[39] Paszke, A. e. a. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 2019.
[40] Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., and Meriaudeau, F. Indian diabetic retinopathy image dataset (idrid), 2018.
[41] Rezende, D. J. and Viola, F. Taming vaes. ArXiv, abs/1810.00597, 2018.
[42] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634, 2021. doi: 10.1109/JPROC.2021.3058954.
[43] Shi, J. and Malik, J. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-905, 2000.
[44] Shwartz-Ziv, R., Balestriero, R., Kawaguchi, K., Rudner, T. G., and LeCun, Y. An information-theoretic perspective on variance-invariance-covariance regularization. arXiv preprint arXiv:2303.00633, 2023.
[45] Singh, G., Deng, F., and Ahn, S. Illiterate dall-e learns to compose. In International Conference on Learning Representations, 2021.
[46] Singh, G., Kim, Y., and Ahn, S. Neural systematic binder. In The Eleventh International Conference on Learning Representations, 2023.
[47] Wang, X., Girdhar, R., Yu, S. X., and Misra, I. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3124-3134, 2023.
[48] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., and Perona, P. Caltechucsd birds 200. 2010.
[49] Wright, S. Correlation and causation. Journal of agricultural research, 1921.
[50] Yang, Y. and Yang, B. Promising or elusive? unsupervised object segmentation from real-world single images. Advances in Neural Information Processing Systems, 35:4722-4735, 2022.
[51] Yuan, J., Chen, T., Li, B., and Xue, X. Compositional scene representation learning via reconstruction: A survey. arXiv preprint arXiv:2202.07135, 2022.