Unsupervised learning of features and object boundaries from local prediction
Authors: Heiko H. Schütt
Wei Ji Ma
Venue: Submitted to ICLR 2023 (2023)
Keywords: unsupervised learning
segmentation
prediction
Markov random field
objects

Abstract: A visual system has to learn both which features to extract from images and how to group locations into (proto-)objects. Those two aspects are usually dealt with separately, although predictability is discussed as a cue for both. To incorporate features and boundaries into the same model, we model a layer of feature maps with a pairwise Markov random field model in which each factor is paired with an additional binary variable, which switches the factor on or off. Using one of two contrastive learning objectives, we can learn both the features and the parameters of the Markov random field factors from images without further supervision signals. The features learned by shallow neural networks based on this loss are local averages, opponent colors, and Gabor-like stripe patterns. Furthermore, we can infer connectivity between locations by inferring the switch variables. Contours inferred from this connectivity perform quite well on the Berkeley segmentation database (BSDS500) without any training on contours. Thus, computing predictions across space aids both segmentation and feature learning, and models trained to optimize these predictions show similarities to the human visual system. We speculate that retinotopic visual cortex might implement such predictions over space through lateral connections.

Under review as a conference paper at ICLR 2023
UNSUPERVISED LEARNING OF FEATURES AND OBJECT BOUNDARIES FROM LOCAL PREDICTION
Anonymous authors Paper under double-bling review
ABSTRACT
The human visual system has to learn both which features to extract from images and how to group locations into (proto-Jobjects. Those two aspects are usually dealt with separately, although predictability is discussed as cue for both. To incorporate features and boundaries into the same model, we model a retinotopic visual cortex with pairwise Markov random field model in which each factor off.
is paired with additional binary variable, which switches the factor Using one of two contrastive learning objectives, we can learn both the features and the parameters of the Markov random field factors from images without further supervision signals. The features learned by shallow neural networks based on this loss are local averages, opponent colors, and Gabor-like stripe patterns observed in early human visual cortices. Furthermore, we can infer connectivity between locations by inferring the switch variables. Contours inferred from this connectivity perform quite well on the Berkeley segmentation database (BSDS500) without any training on contours. Thus, optimizing predictions across space aids both segmentation and feature learning, and models trained this way show similarities the human visual system. We speculate that retinotopic visual cortex might implement such predictions over space through lateral connections.
INTRODUCTION
long-standing question about human vision is how representations initially be based on parallel processing of retinotopic feature maps can represent objects in useful way. Most research on this topic has focused on computing later object-centered representations from the feature map representations. Psychology and neuroscience identified features that lead to objects being grouped together (Kofikal [935] Köhler 1967). established feature integration into coherent objects as sequential process (Treisman & Gelade 1980), and developed solutions to the binding problem, i.e. ways how neurons could signal whether they represent parts of the same object (Finger König 2014; Peter et al. 2019 Singer & Gray. 1995 Treisman 1996). In computer vision, researchers also focused on how feature map representations could be turned into segmentations and object masks. Classically, segmentation algorithm were clustering algorithms operating extracted feature spaces (Arbeláez et al., 2011 Comannici & Meer, [2022] Cour et al.) 20055 Felzenszwalb & Huttenlocheri 2004] Shi & Malik 2000), and this approach is still explored with more complex mixture models today (Vacher et al. 2022). Since the advent of deep neural network models, the focus has shifted towards models that directly map to contour maps or semantic segmentation maps (Girshick et al.
2014; He et al. 2019| Kokkinos, 2016 Liu et al. 2017 Shen et al. 2015 Xie & Tu 2015). as
reviewed by Minaee al. (2021).
Diverse findings suggest that processing within the feature maps take object boundaries into account.
For example, neurons appear to encode border ownership (Jeurissen et al.) 20013] Peter al. 2019) Self et al., 2019) and to fill in information across surfaces (Komatsu) (2066) and along il illusory contours (Grosof et al.) 1993; von der Heydt et al. 1984). Also, attention spreading through the feature maps seems to respect object boundaries (Baldauf & Desimone 2014 Roelfsema et al. 1998).
And selecting neurons that correspond to an object takes time, which scales with the distance between the points to be compared (Jeurissen al. 2016 Korjoukov al.) (002).. Finally, long history of psychophysical studies showed that changes in spatial frequency and orientation content can define
(texture) boundaries (e.g. Beck et al. 1987 Landy & Bergen 1991 Wolfson & Landy 1995). In

---

Under review as a conference paper at ICLR 2023
both human vision and computer vision, relatively little attention has been given to these effects of grouping or segmentation on the feature maps themselves.
Additionally, most theories for grouping and segmentation take the features in the original feature
maps as given. In human vision, these features are traditionally chosen by the experimenter (Kotika
1935 Treisman & Gelade 1980, Treisman (996) are inferred based other research (Peter
et al. 2019; Seelf et al. 2019). Similarly, computer vision algorithms used off- off-the-shelf feature
banks originally (Arbeláez et al.] 2011 Comaniciu & Meer 2002) Cour et al. 2005 Felzenszwalb
& Huttenlocher 2004]|Shi Shi & Malik] 2000), and have recently moved towards deep neural network
representations trained for other tasks as source for feature maps (Girshick et al. 2014 He et al.
2019 Kokkinos 2016; Liu et al.) 2017: Shen et al.) 2015) Xie & Tu 2015).
(iu
Interestingly, predictability of visual inputs over space and time has been discussed as a solution for both these limitations of earlier theories. Predictability has been used as cue for segmentation since the law of common fate of Gestalt psychology (Kofika 1935), and both lateral interactions in
visual cortices and contour integration respect the statistics of natural scenes (Geisler & Perry] 2009)]
Geisler et al. (200T). Among other signals like sparsity (Olshausen & Field 1996) or reconstruction
(Kingma & Welling 2014), predictability is also well known signal for self-supervised learning of features (Wiskott Sejnowski 2002), which has been exploited by many recent contrastive learning (e.g. Feichtenhofer et al.) 2021 Gutmann & Hyvarinen 2010 [Hénfff et al. (020)] van den
Oord et al. (019)) and predictive coding schemes (e.g. (Ltterr et al. [017]] [018]] van den Oord et al.,
2019) for self-supervised learning. However, these uses of predictability for feature learning and for segmentation are usually studied separately.
Here, we propose a model that learns both features and segmentation without supervision. Predictions between locations provide a self-supervised loss to learn the features, how to perform the prediction and how infer which locations should be grouped. Also, this view combines contrastive learning (Gutmann & Hyvarinen 20100; van den Oord et al. 2019), a Markov random field model for the feature maps [Li 2012) and segmentation into a coherent framework. We implement our model using some shallow architectures. The learned features resemble early cortical responses and the object boundaries we infer from predictability align well with human object contour reports from the Berkeley segmentation database (BSDS500 (Arbeláez et al. 2011)). Thus, retinotopic visual cortex might implement similar computational principles as we propose here.
2
MODEL
To explain our combined model of feature maps and their local segmentation information, we start with Gaussian Markov random field model (Li) 2012) with pairwise factors. We then add a variable E 40. 1} to each factor that governs whether the factor enters the product or not. This yields a joint distribution for the whole feature map and all w's. Marginalizing out the w's yields a Markov random field with "robust" factors for the feature map, which can vectors at neighboring positions. We find two contrastive losses based on these predictions that can be used to optimize the feature extraction and the factors in the Markov random field model.
predict feature vectors from the
We model the distribution of k-dimensional feature maps f T Rk.m' ,n' that are computed from input images I E Rsm. with e = 3 color channels (see Fig. A & B). We use a Markov random field model with pairwise factors, i.e. we define the probability of encountering a feature map f with entries f; at locations E [1... [1...m) m²] x [1... [1...m) .2'] as follows:
pff) II (AA II
(ty(fi,)))
(1)
(J)EN
where dh is the local factor, N is the set of all neighboring pairs, and dhey is the pairwise factor between positions and I We will additionally assume shift invariance, i.c. each point has the same set of nearby relative positions in the map as neighbors, v is the same factor for each position, and each factor dhy depends only on the relative position of i and j.
i and j thus have two entries each
2

---

Under review as a conference paper at ICLR 2023
A
B
c
U/1./1 =1)
v/ 12/:22111111 =0)
V f2)
h 12
f1 f0
fi
fr
f1
f2
f2
f2
D
f1
fi
fi
8
12
20
f2
f2
fa
Figure 1: Illustration our Markov random field model for the feature maps. A: An example input image. B: Feature map with 4 neighborhood connectivity and pixel color as the extracted feature. In the actual models, these feature maps are higher dimensional maps extracted by a convolutional neural network. C: Illustration of the factor that links the feature vectors two neighboring locations for a ID feature. Top row: projection of the factor Viy onto the difference between the features value f: fj+ showing the combination of Gaussian around and constant function for the connection variable waj being 1 or 0 respectively. Middle row: 2D representation of the factor and its parts plotted against both feature values. Bottom row: Multiplication standard normal factor for each position yielding the joint distribution of two isolated positions. D: Neighborhoods of different sizes used the models, scaling from 4 to neighbors for each location.
the middle row with the
We now add a binary variable w E {0, 1} to each pairwise factor that encodes whether the factor
is "active" (w 1) for that particular image (Fig. C). To scale the probability of 1 and w = 0 relative to each other, we add a factor that scales them with constants Pij E [0, 1] and Psj respectively:
p(f. w)
II (f)) II
Pij Wi (1 Pij) 1-ws wis (fs
(1J)N
Finally, we assume that the factors are Gaussian and the feature vectors are originally normalized to have mean 0 and variance 1:
II p(f, w) Zo (1.)eN
1 ().
Púù (1 p(j) 1- Z(wiy, Cts)
cxp (- Waj (fr Co (3)
where Zo the overall normalization constant, (f,, O, I) is the density of a standard normal distribution with k m' x n' dimensions, Cj governs the strength of the coupling in the form of precision matrix, which we will assume to be diagonal, and Z(wij Ciy) scales the distributions with Wij 0 and Wiy 1 relative to each other.
We set (wij. C.y) to the normalization constant the Gaussian with standard Gaussian factors for fü and fy respectively. For w = this is just -² the normalization constant of a standard Gaussian in 2k dimensions. For we get:

---

Under review as a conference paper at ICLR 2023
[(wy = 1, C,)) = l
(-
1 1
JT ft
2 2 1
1
- - cill - 1)) dfidts (4)
1
1 Ct
(2r) * det
(5)
Cis
I 1 Cis
= (2T) v VI+ 2011
(6)
which we get by computing the normalization constant of a Gaussian with the given precision and then using the assumption that Ci is diagonal matrix with diagonal entries en-
This normalization depends only valid probability distribution on the feature maps. Two points are notable about this normalization though: First, once other factors also constrain fi and/or fj. this normalization will not guarantee p(W;; 1) Pij. Second, the Wi are not independent in the resulting distribution. For example, if pairwise factors connect a to b, to c and a to c the corresponding w are dependent, because wab = 1 and Whe 1 already imply smaller difference between fa and fa than if these factor were inactive, which increases the probability for Wac =1. 1.
and the coupling matrix C of the factor bhy and thus induces
2.1 LEARNING
To learn model from data, contrastive learning objective on the marginal likelihood p(f). To do so, we first need to marginalize out the w's, which is fortunately simple, because each wo affects only single factor:
1 O(,,,,,D) II Zo
p(f) p(t,, w)
(f P()) (J)NN
Using this marginal likelihood directly for fitting is infeasible though, because computing Zo, i.c. normalizing this distribution is not computationally tractable.
We resort contrastive learning fit the unnormalized probability distribution (Gutmann & Hyvari nen (0100). i.c. we optimize discrimination from a noise distribution with the same support as the target distribution. Following /van den Oord et al. 2019) we do not optimize the Markov random field directly, but optimize predictions based on the model using features from other locations as the noise distribution. For this noise distribution, the factors that depend only on single location (the first product in (1) will cancel. We thus ignore the N (f, O, I) in our optimization and instead normalize the feature maps to mean and unit variance across each image. We define two alternative losses that make predictions for positions based on all their neighbors or for single factor respectively.
2.1.1 POSITION LOSS
The position loss optimizes the probability of the feature vector at each location relative to the probability of randomly chosen other feature vectors from different locations and images:
4pos (f) Elog log P(ALVV E N())
(8)
= =Σ Σ log (f fz)
Σ log
Σ exp
Σ JEN()
log
(9)
JEN(O
where N(1) is the set of neighbors of i.
Instead, p(u;, = 1) will be higher, because other factors increase the precision for the feature vectors, which makes the normalization constants more similar.
4

---

Under review as a conference paper at ICLR 2023
2.1.2 FACTOR LOSS
The factor loss instead maximizes each individual factor for the correct feature vectors relative to random pairs of feature vectors sampled from different locations and images:
Viey (fi Ene V4 ff)
lact
Σ
log
(10)
6,
Σ (.
log Uis (fi, fz) Σ log Σ volle.fy).
(11)
where i index the correct locations and i', J index randomly drawn locations, in our implementation generated by shuffling the feature maps and taking all pairs that occur in these shuffled maps.
2.1.3 OPTIMIZATION
We optimize all weights of the neural network used for feature extraction and the parameters of the random field, i.c. the C' and Pi, for the different relative spatial locations simultancously. As an optimization algorithm, we use stochastic gradient descent with momentum. Both losses succeed to learn the model, but the factor loss is substantially more efficient. We discuss the distinction between the losses and further details of the optimization in the supplementary materials.
2.2 SEGMENTATION INFERENCE
Computing the probability for any individual pair of locations (i,j) to be connected, i.c. computing p(wi; = 1)), depends only on the two connected feature vectors fi and fz:
p(wi, 1[f) Pij
Z(wiy 0, Ciy) exp C.) f)) p(w;r = 0ff) (1 - Pb)) Z(wiy = 1, Cty)
(12)
This inference effectively yields a connectivity measure for each pair of neighboring locations, i.c. sparse connectivity matrix. Given that we did not apply any prior information enforcing continuous objects or contours, the inferred Wij do not necessarily correspond to a valid segmentation or set of contours, Finding the best fitting contours or segmentation for given probabilities for the ws is an additional process, which in humans appears to be an attention-dependent serial process (Jeurissen jet al. 2016, Self et al. 2019).
To evaluate the detected boundaries in computer vision benchmarks, we nonetheless need to convert the connectivity matrix we extracted into a contour image. To do so, we use the spectral-clustering based globalization method developed by Arbeláez et al. (201T). This method requires that all connection weights between nodes are positive. To achieve this, we transform the log-probability find the 30% quantile of the values, subtract
ratios for the Why follows: For each image, from all log-probability ratios, and set all values below 0.01 to 0.01. We then compute the smallest eigenvectors of the graph Laplacian in graph spectral clustering. These eigenvectors are then transformed back into image space and are filtered with simple edge detectors to find the final contours.
EVALUATION
We implement 3 model types implementing feature extractions of increasing complexity in Py Porch (Paszke et al. 2019):
Pixel value model. For illustrative purposes, first apply ideas to the rgb pixel values of an image as features. This provides us with an example, where we can easily show the feature values and connections. Additionally, this model provides easy benchmark for all evaluations.
Linear model. As the simplest kind of model that allows learning features, we use a single convolu- tional deep neural network layer feature model. Here, 50 11 11 linear features.
5

---

Under review as a conference paper at ICLR 2023
Linear, Position loss
Linear, Factor loss
Linear, best segmentation
Predseg layer
E Predseg 1, Position Loss
Predseg 1, Factor Loss
Figure 2: Example linear filter weights learned by our models. Each individual filter is normalized to minimum 0 and maximum 1. As weights can be negative even a zero weight can lead to pixel having some brightness. For example, number of channels load similarly on red and green across positions. Where these weights are positive the filter appears yellow and where the weights are negative filter appears blue, even if the blue channel has zero weight. A-C: Feature weights learned by the linear model. A: Using the position loss. B: Using the factor loss. C: The weights of the model that leads to the best segmentation performance, i.c. the one shown in Figure 3 D: Weights of the first convolution in predsegl. Next to the filter shapes, which are nearly constant, average weight of each channel onto the three color channels of the image. Predsegl filters in the second convolution for network trained with the position based loss. F: Predseg filters in the second convolution for network trained with the factor based loss.
plot the
Predseg1: To show that our methods work for more complex architecture with non non- linearities, we use relatively small deep neural network with 4 layers (2 convolutional layers and residual blocks with subsampling layers between them, see supplement for details).
For each of these architectures, we train different networks with all combinations of the following settings: 4 different sizes of neighborhoods (4, 8, 12, or 20 neighbors, see Fig. [P:: different noise levels (0, 0.1, 0.2) and the two learning objectives. As a training set, we used unlabeled image set from MS COCO (Lin et al. 2015), which contains 123,404 color images with varying resolution. To enable batch processing, we randomly crop these images to 256 256 pixel resolution, but use no other data augmentation (See supplementary information for further training details).
We want to evaluate whether our models learn human-like features and segmentations. To do so, we first analyze the features in the first layers of our networks where can judge whether features are representative of biological visual systems. In particular, we extract segmentations from our activations and evaluate those the Berkeley Segmentation Dataset (Arbeláez et al. 2011 BSDS500)
3.1 LEARNED FEATURES
Linear Model We first analyze the weights in our linear models (Fig 2 A-C). All instances learn local averages and Gabor-like striped features, i.e. spatial frequency and orientation tuned features with limited spatial extend. These features clearly resemble receptive fields neurons in primary visual cortex. Additionally, there appears be some preference for features that weight the red and

---

Under review as a conference paper at ICLR 2023
A
Pixel
Linear
PS1-LO
PS1-L1
B
Mumbo condstency
0.8
Preciaion
Koraar
gfb--owt-aaom (2011)
Pridaweg LG
0.2
Canny (1985)
Pradacgl:L
Mean Stiih (202P)
Lir oar features
Feteersawall (2004)
Pixcl values
0
Narmal and Cate (2005)
olgu curves
0.2
0.4
0.8
Recall
Figure 3: Contour detection results. A: Example segmentations from our models. B: Precision-recall curves for our models on the Berkeley segmentation dataset, with some other models for comparison as evaluated by Arbeláez. et al. (201T): gPb-uwt-ucm, the final algorithm combining all improvements (Arbeláez et al., (011). Canny's 's classical edge detector (Canny) 1986). the mean shift algorithm (Comaniciu & Meer 2002). Felzenschwalbs algorithm (Felzenszwalb & Huttenlocher (0044 and segmentation based on normalized cuts (Cour et al. 2005). For all comparison algorithms evaluations on BSDS were extracted from the figure by Arbellezz al. (2011)
green color channels much stronger than the blue channel, similar to the human luminance channel, which leads the yellow-blue contrasts in the plots. There is some difference between the two learning objectives though. The position based loss generally leads to lower frequency and somewhat noisier features. This could either be due the higher learning efficiency of the factor based loss, i.e. the factor based loss is closer to convergence, or due to a genuinely different optimization goal.
Predseg1 In Predsegl, we first analyze the layer 0 convolution (Fig. (D). which has only 3 channels with 3 3 receptive fields, which we originally introduced as learnable downsampling. This layer consistently converges to applying near constant weights over space. Additionally, exactly one of the channels has differences between two of the channels (red vs green and green vs. blue in the examples). This parallels the luminance and opponent color channels of human visual perception.
(the 3rd, 1st and 3rd in Fig. ( and the other take balanced
In the second convolution, we observe a similar pattern of oriented filters and local averages as in the linear model albeit in false color the input channels are rotated by the weighting of the layer convolution (Fig. E & F).
3.2 CONTOUR EXTRACTION
To evaluate whether the connectivity information extracted by our model corresponds to human perceived segmentation, extract contours from models and compare them contours reported by humans for the Berkeley Segmentation Database (Arbeláez et al., 2011 Martin et al. (001I. This database contains human drawn object boundaries for 500 natural images and is accompanied by methods for evaluating segmentation models. Using the methods provided with the database, we compute precision-recit curves for each model and use the best F-value (geometric mean of precision and recall) as the final evaluation metric.
As we had multiple models to choose from, we choose the models from each class that perform best on the training data for our reports. For all models this was one of the models with the largest neighborhood, i.e. using 20 neighbors, and the factor loss. seems the factor loss performed better simply due to its technical efficiency advantage discussed above. Performance increases monotonically with neighborhood size and Markov random field based approaches to semantic segmentation also increased their performance with larger neighborhoods up fully connected Markov random fields (Krähenbühl & Koltun 2012 Chen et al., 2014 2017). We thus expect that larger neighborhoods could work even better.
Qualitatively, we observe that all our models yield sensible contour maps (see Fig. A). Additionally, note that the linear model and Layer 1 of the predseg model tend to produce double contours,
7

---

Under review as a conference paper at ICLR 2023
Table 1: Numerical evaluation for various algorithms on the BSDS500 dataset. Precision and recall are only given for ODS, i.. with a the threshold fixed across the whole dataset. model
Recall
Precision F(ODS)
F(OIS)
Area PR
Deep Contour* *
Shen et al.
2015
0.76
0.78
0.80
HED** **
X1e & Tu
2015
0.79
0,81
0.84
RC
Liu et al.
2017
0.81
0.83
Deep Boundary
Kokkinos
2016
0.813
0.831
0.866
BDCN*
(He ct al.
2019
0.83
0.84
0.89
Canny (Canny
1986
0.60
0.63
0.58
Mean Shift (Comaniciu & Meer
2002
0,64
0.68
0.56
Felzenszwalb*
Felzenszwwlb Huttenlocher
2004
0.61
0.64
0.56
Normalized Cuts (Cour al.
2005
0.64
0.68
0,45
gPb-owt-ucm (Arbelaez et al.
2011
0.73
0.73
0.73
0.76
0.73
Pixel
0.73
0.66
0.69
0.69
0.73
linear
0.78
0.66
0.72
0.73
0.75
0
0.79
0.69
0.74
0.73
0.80
Predseg1-Laye 1
0.74
0.47
0.57
0.59
0.45
*, Evaluation these algorithms taken from Arbeláez et al. (2011).
#8: Supervised DNNs, evaluation taken from He et al. (2019).
i.e. they tend produce contours either side of the contour reported by human subjects with some area between them connected to neither side of the contour.
Quantitatively, our models also perform well except for the deeper layers of Predseg (Fig. 3 and Table []. The other models beat most hand-crafted contour detection algorithms that were tested
on this benchmark (Canny) 1986 Comaniciu & Meer 2002 Cour al.) 2005 Felzenszwalb Huttenlocher 2004) and perform close to the glb contour detection and segmentation algorithm (Arbelãez al. 201T) that was the state of the
the time. Layer-0 of Predseg 1 performs best followed by the linear feature model and finally the pixel value model. Interestingly, the best performing models seem be mostly the local averaging models (ef. Fig. C In particular, the high performance of the first layer of Predseg 1 is surprising, because it uses only 3 x 3 pixel local color averages as features.
Since the advent of deep neural network models, networks trained to optimize performance image segmentation have reached much higher performance on the BSDS500 benchmark, essentially
reaching perfect performance up human inconsistency (e.g. He et al. 2019 Kokkinos 2016
Linsley et al. 2020. Liu et al., 2017] Shen et al.) 2015 Su et al. 2021 Xie & Tu 2015] see Table However, these models all require direct training
human reported contours and often features
learned for other tasks. There are also a few deep neural network models that attempt unsupervised
segmentation (e.g. Chen et al. 2019: Lin et al.) 2021 Xia & Kulis 2017). but we were unable find any that were evaluated on the contour task of BSDS00. The closest perhaps the W-net (Xia & Kulis 2017). which used an autoencoder structure with additional constraints and was evaluated on the segmentation task on BSDS500 performing slighly better than gPb-owt-ucm.
4 DISCUSSION
We present a model that can learn features and local segmentation information from images without further supervision signals. This model integrates the prediction task used for feature learning and the segmentation task into the same coherent probabilistic framework. This framework and the dual use for the connectivity information make it seem sensible to represent this information. Furthermore, the features Iearned by our models resemble receptive fields in the retina and primary visual cortex and the contours we extract from connectivity information match contours drawn by human subject fairly well, both without any training towards making them more human-like.
To improve biological plausibility, all computations our model are local and all units are connected to the same small, local set of other units throughout learning and inference, which matches early visual cortex, in which the lateral connections that follow natural image statistics are implemented
anatomically (Buzás al. 2006 Hunt et al. 2011
Roelfsema et al. 1998. Stettler al. 2002)).
8

---

Under review as a conference paper at ICLR 2023
This in contrast to other ideas that require flexible pointers to arbitrary locations and features (as discussed by Shadlen & Movshon 1999) or capsules that flexibly encode different parts of the imput (Doerig et al. 2020; Kosiorek et al. 2019 Saaoou et al., 2017: 202I). Nonetheless, we employ contrastive learning objectives and backpropagation here, for which we do not provide a biologically plausible implementations. However, there is currently active research towards biologically plausible alternatives to these algorithms (c.g. [Iling et al. 2021 Xiong et al. 2020).
Selecting the neurons that react a specific object appears to rely on some central resource (Treisman 1996 Treisman& & Gelade, 1980) and to spread gradually through the feature maps (Jeurissen et al.
2013 2016; Self et al.j 2019). We used a computer vision algorithm for this step, which centrally computes the eigenvectors of the connectivity graph Laplacian (Arbeláez et al. (011). which does not immediately look biologically plausible. However, a recent theory for hippocampal place and grid cells suggests that these cells compute the same eigenvectors of graph Laplacian albeit of successor representation (Stachenfeld et al. 2014 2017). Thus, this might be an abstract description of an operation brains are capable of. In particular, earlier accounts that model the selection
marker that spreads to related locations (e.g. Finger & König 2014 Roelisema 2006 Siinger & Gray
1995) have some similarities with iterative algorithms to compute eigenvectors. Originally, phase coherence was proposed as a marker (Finger & König 2014 Peter et al. 2019 [Singer & Gray| [195). but a simple gain increase within attended objects (Roelfsema 2006) and random gain modulation were also proposed (Haimerl et al.) 2021 2019). Regardless of the mechanistic implementation of the marker, connectivity information of the type our model extracts would be extremely helpful to explain the gradual spread of object selection.
Our implementation of the model is not fully optimized, as it is meant as proof of concept. In particular, we did not optimize the architectures or training parameters of our networks for the task, like initialization, optimization algorithm, learning rate, or regularization. Presumably, better performance in all benchmarks could be reached by adjusting any or all of these parameters.
One possible next step for our model would be to train deeper architectures. such that the features could be used for complex tasks like object detection and classification. Contrastive losses like the one we use here are successfully applied for pretraining for large scale tasks such as ImageNet (Russakovsky et al. 2015) or MS Coco (Lin et al. 2015). These large scale applications often require modifications for better learning (Chen et al. 2020) Feichtenhofer et al., 2021 Grill et al. 2020: He et al. 2020 Hénaff et al. 2020: van den Oord et al. (019)). For example: Image augmentations to explicitly train networks to be invariant to some image changes, prediction heads that allow more complex distributions for the predictions, and memory banks other methods decrease the reliance on many negative samples. For understanding human vision, this line of reasoning opens the exciting possibility that higher visual cortex could be explained based on similar principles, as representations from contrastive learning also yield high predictive power for these cortices (Zhuang et al.) 2021).
The model we propose here is probabilistic model of the feature maps. Based on this model, we could also infer the feature values. Thus, our model implies a pattern how neurons should combine their bottom-up inputs with predictions from nearby other neurons, once we include some uncertainty for the bottom-up up inputs. In particular, the combination ought to take into account which nearby neurons react to the same object and which ones do not. Investigating this pooling could provide insights and predictions for phenomena that are related to local averaging. Crowding for example (Balas et al.) 2009 Freeman& & Simoncelli [011 Herzog et al. 2015 Wallis et al. 2016 201 7, 2019) is currently captured best by summary statistic models (Balas et al. 2009 Freeman & Simoncelli 2011 Wallis et al. 2017). but deviations from these predictions suggest that object boundaries change
processing (Herzog al. 2015 Wallis al. 2016 2019).
Another promising extension of our model would be processing over time, because predictions over time were found to be potent signal for contrastive learning (Feichtenhoter al.] 2021 and because coherent object motion is among the strongest grouping signals for human observers (Köhler) 1967) and computer vision systems (Yang et al. 2021). Beside the substantial increases in processing capacity necessary to move to video processing instead of image processing, this step would require some extension of our framework to include object motion into the prediction. Nonetheless, including processing over time seems to be an interesting avenue for future research, especially because segmentation annotations for video are extremely expensive to collect such that unsupervised learning is particularly advantageous and popular in recent approaches (Araslanov et al. 2021 Jabri et al. 2020 Lai et al. 2020).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nikita Araslanov, Simone Schaub-Meyer, and Stefan Roth. Dense Unsupervised Learning for Video Segmentation. Advances Neural Information Processing Systems, 35:12, 2021.
Arbeláez, M Maire, C. Fowlkes, and Malik. Contour Detection and Hierarchical Image Segmen- tation. IEEE Transactions Pattern Analysis and Machine Intelligence, 33(5):888 2011. ISSN 0162-8828, 2160-9292. doi: 10. 109/TPAMI..000.111.
B. Balas, I.. Nakano, and R. Rosenholtz. A summary-statistic representation in peripheral vision explains visual crowding. Journal of Vision, 912):13-13, 2009. ISSN 1534-7362. doi: 10.1167/9. 12.13.
Daniel Baldauf and Robert Desimone. Neural mechanisms of object-based attention. Science, 344 (6182):424 427, 2014.
Jacob Beck, Anne Sutter, and Richard Ivry. Spatial frequency channels and perceptual grouping in texture segregation. Computer Vision, Graphics, and Image Processing, 37(2):299-325, 1987.
Péter Buzás, Krisztina Kovács, Alex S. Ferecskó, Julian M.L. Budd, Ulf T. Eysel, and Zoltán F. Kisvárday. Model-based analysis excitatory lateral connections in the visual cortex. The Journal of Comparative Neurology, 499(6))861-881, 2006. ISSN 00219967, 10969861. doi: 10.1002//ne:21134.
John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, 00::679-99, 1986.
Liang-Chiel Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected erfs. arXiv preprint arXiv:1412.7062, 2014.
Liang- Main--Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised Object Segmentation by Redrawing. Advances in Neural Information Processing Systems, 33:12, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Puttern Analysis and Machine Intelligence, 24(5):603-619, 2002. ISSN 01628828. doi: 10.1109/34. 001100944.10002333 1000336.
Cour, Benezit, and Jianbo Shi. Spectral Segmentation with Multiscale Graph Decomposition. In
2005 IEEE Computer Sociely Conference Computer Vision and Puttern Recognition (CVPR ()), volume 2. pp. 1124-1131, San Diego, CA, USA, 2005. IEEE. ISBN 978-0-7695-2372- doi: 10. 10.1109//VPR.2200.3322
Adrien Doerig, Lynn Schmittwilken, Bilge Sayim, Mauro Manassi, and Michael Herzog, Capsule networks as recurrent models of grouping and segmentation. PLOS Computational Biology, 16(7):
e1008017, 2020. ISSN 1553-7358. doi: 10. 1/.37//jouraal.cch. 100.0..00.........
1008017.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3299-3309, 2021.
Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient Graph-Bast Based Image Segmentation. International Journal of Computer Vision, 1772), 2004. ISSN 0920-5691. doi: 10.30B 2ww.0000022288.197776..
10

---

Under review as a conference paper at ICLR 2023
Holger Finger and Peter König. Phase synchrony facilitates binding and segmentation of natural images in coupled neural oscillator network. Frontiers in Computational Neuroscience, 2014. ISSN 1662-5188. doi: (0.3389/fncom.2113.001999
Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. Nature Neuroscience, 14 (9):1195-1201, 2011. ISSN 1097-6256, 1546-1726. doi: 10.1038/mn.2889
Wilson S. Geisler and Jeffrey S. Perry. Contour statistics in natural images: Grouping across
ocelusions. Visual Neuroscience, 26(1):109-12) 101, 2009. ISSN 0952-5238, 1 1469-8714 -774. doi: 10.1017/000525288800000033
W.S. Geisler, J.S. Perry, B.J. Super, and D.P. Gallogly. Edge co-oceurrence in natural images predicts contour grouping performance. Vision Research, 41(6):711-724, 2001. ISSN 00426989, doi:
10.016//0042-669900000777-
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Mann-Batieen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21221 2020.
David H. Grosof, Robert M. Shapley, and Michael J. Hawken. Macaque VI neurons can signal 'illusory' contours. Nature, 365(6446):550-522 1993. ISSN 1476-4687. doi: 10. 10.1003/365550a0.
Michael Gutmann and Aapo Hyvarinen. Noise-comtrastive estimation: A new estimation principle for unnormalized statistical models. International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 297-304, 2010.
Caroline Haimerl, Cristina Savin, and Ecro Simoncelli. Flexible information routing in neural populations through stochastic comodulation. Advances in Neural Information Processing Systems, 32, 2019.
Caroline Haimerl, Douglas A Ruff, Marlene R Cohen, Cristina Savin, and Eero P Simoncelli. Targeted comodulation supports flexible and accurate decoding in V1. bioRxiv the preprint server for biology, 2021.
Jianzhong He, Shiliang Zhang, Ming Yang. Yanhu Shan, and Tiejun Huang. Bi-Directional Cascade Network for Perceptual Edge Detection. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3823-3832, Long Beach, CA, USA, 2019. IEEE. ISBN 978-1-72813-29333 doi: 10. 21110///V..229.0000
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, Seattle, WA, USA, 2020. IEEE. ISBN 978-1-72817-168-5 doi: 10. 11111///VV222200200000000.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, and S M Ali Eslami. Data-Efficient Image Recognition with Contrastive Predictive Coding. Proceedings of the 37th International Conference Machine Learning (PMLR), pp. 119, 2020.
Michael H. Herzog, Bilge Sayim, Vitaly Chicherov, and Mauro Manassi. Crowding, grouping, and object recognition: matter of appearance. Journal of Vision 15(6):5, 2015. ISSN 1534-7362. doi: 10.1167/15.6.5.
Jonathan J Hunt, William H Bosking, and Geoffrey Goodhill. Statistical structure of lateral connections in the primary visual cortex. Neural Systems & Circuits, 1(1):3, 2011. ISSN 2042- 1001. doi: 10. 186/2042-10011
Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using supfrvipe contrastive predictions. Advances in Neural Information Processing Systems, 35:15, 2021.
11

---

Under review as a conference paper at ICLR 2023
Allan A Jabri, Andrew Owens, and Alexei A Efros. Space-Time Time Correspondence as Contrastive Random Walk. Advances Neural Information Processing Systems, 34:16, 2020.
Danique Jeurissen, Matthew W. Self, and Pieter R. Roelfsema. Surface reconstruction, figure-ground modulation, and border-ownership. Cognitive neuroscience, 4(1):00-52, 2013.
Danique Jeurissen, Matthew W. Self, and Pieter R. Roelfsema. Serial grouping of 2D-image regions with object-based attention humans. Elife, Ste14320, 2016.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. arXiv: [cs, stat], 2014.
K Koffka. Principles of gestalt psychology. 1935.
Wolfgang Köhler. Gestalt psychology. Psychologische Forschung, 31(1):XVII-XXX, 1967.
lasonas Kokkinos. Pushing the Boundaries of Boundary Detection using Deep Learning.
arXiv:1511.07386 [cs], 2016.
Hidehiko Komatsu. The neural mechanisms of perceptual filling-in, Nature Reviews Neuroscience, (3):220-231, 2006. ISSN 1471-003X, 1471-0048, doi: 10.1038/mm (0138/mm1869. 1869.
Ilia Korjoukov, Danique Jeurissen, Niels A. Kloosterman, Josine E. Verhoeven, Steven Scholte, and Pieter R. Roelfsema. The Time Course of Perceptual Grouping in Natural Scenes. Psychological Science, 23(12):1482-148,
2012. ISSN 0956-7976. doi: 10. 1117/00967796614438822
Adam R. Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E. Hinton. Stacked Capsule Autoen- coders. arXiv:1906.000 06818 [cs, stat], 2019.
Philipp Krähenbühl and Vladlen Koltun. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. 120055240 [es], 2012.
Zihang Lai, Erika Lu, and Weidi Xie. MAST: A Memory-Augmented Self-Supervised Tracker. CVPR, pp. 6479-6488, 2020.
Michael Landy and James R Bergen. Texture segregation and orientation gradient. Vision research, 31(4):679-69) 6917,9--9,111 1991.
Li. Markov Random Field Modeling in Computer Vision. Springer Science & Business Media, 2012.
Qinghong Lin, Weichan Zhong, and Jianglin Lu. Deep Superpixel Cut for Unsupervised Image Segmentation. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 8870- 8876, Milan, Italy, 2021. IEEE. ISBN 778-1-72818-808.2 doi: 10. 10.1199//C44888062000 9411968.
Tsung- Yi Lin, Michael Maire, Serge Belongie. Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects Context. arXiv: 145.:330
[cs), 2015.
Drew Linsley, Junkyung Kim, Alekh Ashok, and Thomas Serre. Recurrent neural circuits for contour detection. In International Conference on Learning Representations, 2020.
Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang, and Xiang Bai. Richer Convolutional Features for Edge Detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3000-3009, 2017.
William Lotter, Gabriel Kreiman, and David Cox. Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning. arXiv: 08404//0//0 [cs, q-bio], 2017.
William Lotter, Gabriel Kreiman, and David Cox. A neural network trained to predict fu- ture video frames mimics critical properties of biological neuronal responses and perception. arXiv: 1805. 10734 (es, q-bio), 2018
12

---

Under review as a conference paper at ICLR 2023
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pp. 416-423, July 2001.
Shervin Minace, Yuri Y. Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image Segmentation Using Deep Learning: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. ISSN 0162-8828, 2160-9292, 1939-3539. doi: 10.1 1011099//MM.2000 2021 20113399......3359966
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (6583):607--66) 1996.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, higl-performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pp. 8024-8035. 2019. URL http::/ nwwrips.
ibbrar.
pdf
Alina Peter, Cem Uran, Johanna Klon-Lipok, Rasmus Roese, Sylvia van Stijn, William Barnes, Jarrod Dowdall, Wolf Singer, Pascal Fries, and Martin Vinck. Surface color and predictability determine contextual modulation of V1 firing and gamma oscillations. eLife, 8:e42101, 2019. ISSN 2050-084X. doi: 10. 10.7554.el.ffe.22100..
Pieter R Roelfsema. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci., 29:203-222 227, 2006.
Pieter R Roelfsema, Victor AF Lamme, and Henk Spekreijse. Object-based attention in the primary visual cortex of the macaque monkey. Nature, 395(6700))3766381) 1998.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0577 [cs], 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic Routing Between Capsules. arXiv:1710.09829 2017.
Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E. Hinton, and David J. Fleet. Unsu- pervised part representation by Flow Capsules. urXiv:2011.13920 [cs], 2021.
Matthew W. Self, Danique Jeurissen, Anne F. Ham, Bram van Vugt, Jasper Poort, and Pieter R. Roelfsema. The Segmentation of Proto-Objects in the Monkey Primary Visual Cortex. Current Biology, 29(6):1019-108 2019. ISSN 09609822, doi: w...166f..bb22119022.0..
Michael N Shadlen and J Anthony Movshon. Synchrony unbound: A critical evaluation of the temporal binding hypothesis. Neuron, 24(1):67-77, 1999.
Wei Shen, Xinggang Wang, Yan Wang. Xiang Bai, and Zhijiang Zhang. DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3982-3991, Boston, MA, USA, 2015, IEEE, ISBN 978- 178-1-4473-6964--0 doi: 10.119//VP222015.299922
Jianbo Shi and Jitendra Malik. Normalized Cuts and Image Segmentation. IEEE Transactions On Pattern Analysis and Machine Intelligence, 22(8):18, 2000.
Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual review of neuroscience, (8(1)5555-586, 1995.
Kimberly L Stachenfeld, Matthew Botvinick, and Samuel Gershman. Design Principles of the Hippocampal Cognitive Map. Advances in Neural Information Processing Systems, pp. 9, 2014.
13

---

Under review as a conference paper at ICLR 2023
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. The hippocampus predictive map. Nature Neuroscience, 20(11):1643 2017. ISSN 1097-6256, 1546-1726. doi: 10.1038/nn.4600.
Dan D Stettler, Aniruddha Das, Jean Bennett, and Charles D Gilbert. Lateral Connectivity and Contextual Interactions in Macaque Primary Visual Cortex. Newron, 36(4):739-73) 2002. ISSN 08966273. doi: (021006//009666633300)000
Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pictikäinen, and Li Liu. Pixel Difference Networks for Efficient Edge Detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 5117-5127, 2021.
Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
Anne M. Treisman and Garry Gelade. A Feature- Inteuratinnnriiio Theory of Attention, Cognitive Psychology, 12:97-136, 1980.
Jonathan Vacher, Claire Launay, and Ruben Coen-Cagli. Flexibly regularized mixture models and application to image segmentation. Neural Networks, 149:107-12 123, 2022. ISSN 0893-6080. doi: htw.:///f..eumet222222220
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. 1rXiv:1807.05: [cs, star), 2019.
R. von der Heydt, E. Peterhans, and G. Baumgartner. Illusory Contours and Cortical Neuron Responses. Science, 22A(4654) 1984. ISSN 0036-8075, 1095-9203. doi: 10.1126/ science.653955
Thomas S. A. Wallis, Matthias Bethge, and Felix A. Wichmann. Testing models of peripheral encoding using metamerism in oddity paradigm. Journal of Vision, 16(2):4, 2016. ISSN 1534-7362. doi: 10.1167/16.2.4.
Thomas S. A. Wallis, Christina M. Funke, Alexander S. Ecker, Leon A. Gatys, Felix A. Wichmann, and Matthias Bethge. A parametric texture model based on deep convolutional features closely matches texture appearance for humans. Journal of Vision, 17(12):5-5, 2017. ISSN 1534-7362. doi: 10.1167/17.12.
Thomas SA Wallis, Christina M Funke, Alexander S Ecker, Leon Gatys, Felix A Wichmann, and Matthias Bethge. Image content is more important than Bouma's Law for scene metamers. eLife, doi: 1255
8:e42512, 2019, ISSN 2050-084X.
Laurenz Wiskott and Terrence Sejnowski. Slow Feature Analysis: Unsupervised Learning of Invariances. Neural Computation, 14(4):115-770 2002. ISSN 0899-7667, 1530-888X. doi: 10.1 (0.11662088997660233333999999
Sabina Wolfson and Michael Landy. Discrimination of orientation-deffmel texture edges. Vision research, 35(20):2863-2877 1995.
Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprint arXiv: 1.08506, 2017.
Saining Xie and Zhuowen Tu. Holistically-Nested Edge Detection. In Proceedings of the IEEE International Conference Computer Vision, pp. 1395-1403, 2015.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. LoCo: Local Contrastive Representation Learning. Advances in Neural Information Processing Systems, 34:1 12, 2020.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xic. Self-supervised Video Object Segmentation by Motion Grouping. arXiv:2104.07588 [cs], 2021.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf. Michael C. Frank, James J. DiCarlo, and Daniel L. K. Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy Sciences, (00)))222141966 18, 2021. ISSN 0027-8424, 1091-666 -6490. doi: 10. 20003//paas.004119666
14

---

Under review as a conference paper at ICLR 2023
A SUPPLEMENTARY MATERIAL: TRAINING DETAILS
We trained 24 networks of each of the three types. The versions differed in the size of the neighborhood
(4, 8, 12, or 20 neighbors), the amount of noise added (a E 0, 0.1, 0.2), and the used loss (position or factor loss).
The parameters we trained were:
all weights of the underlying network
the logit transform of p for each relative position of two neighbors
the logarithms of the diagonal entries of C' for each relative position of neighbors
We trained models using the standard stochastic gradient descent implemented in pytorch (Paszke al. 2019) with learning of 0.001, momentum of 0.9 and slight weight decay of 0.0001. To speed convergence we increased the learning rate by factor of 10 for the parameters of the prediction, i.c. C' and p. For the gradient accumulation for the position based loss, we accumulate repetitions for the pixel model and 10 for the linear model and for predsegl. Each repetition contained 10 random negative locations. Batch size was set to fit onto the smaller GPU type used in our local cluster. The resulting sizes are listed in Table 2
A.1 ARCHITECTURE DETAILS
The pixel model was implemented as a single Identity layer.
The linear model was implemented single 50 x 11 x 11 convolutional layer.
The Predsegl model was implemented as a sequential model with 4 processing steps separated by subsampling layers (1 x convolutional layers with stride >> 1). The first processing step 3 x 3 convolutional layer with 3 channels followed by subsampling by factor of 3. The second step was a 11 11 convolutional layer with 64 features followed by subsampling by factor of 2. The third and fourth steps were residual processing blocks, i.e. two convolutional layers with rectified linear unit non-linearity between them whose results were added to the inputs. They had 128 and 256 features respectively and were separated by another subsampling by factor of 2.
A.2 ADDED NOISE
To prevent individual features dimensions from becoming perfectly predictive, we added small amount of Gaussian noise the feature maps before applying the loss. To yield variables with mean 0 and variance after adding the noise we implemented this step as:
fnoise /1
a² + ar
(13)
where a E [0, 1] controls the noise variance and is standard normal random variable.
Adding this noise did change any of our results substantially and the three versions with different amounts of noise (a 0. 0.1 or 0.2) performed within 1 - 2% in all performance metrics.
A.3 TRAINING DURATION
Networks were trained in training jobs that were limited to either 48 hours of computation time or 10 epochs of training. As listed in table we used single such job for the pixel models, for the linear models and 9 for the predsegl models. Most larger networks were limited by the 48 hour limit, not by the epoch limit.
A.4 USED COMPUTATIONAL RESOURCES
The vast majority of the computation time used for training the network parameters. Computing segmentations for the BSDS500 images and evaluating them took only a few hours of pure CPU processing.
15

---

To enable a sufficiently large set of negative points with the available memory, we compute this loss multiple times with few negative samples and sum the gradients. This trick saves memory, because we can free the memory for the loss computation after each repetition. the initial computation of the feature maps is the same for all negative samples, we save some computation for this procedure by computing the feature maps only once. To propagate the gradients through this single computation, we add up the gradients of the loss repetitions with regard to the feature maps and then propagate this summed gradient through the feature map computation. This procedure does not save computation still need to calculate the evaluation for Under review as a conference paper at ICLR 2023
Table 2: Training parameters and training time for the different networks. Networks were trained on single V100 (32GB) or RTX8000 (48GB) GPUs depending on availability. Training times were approximately read out from the computation logs. # of training jobs indicates how many 48 hour jobs we started for each model. batch size
Model
training time (hh:mm per epoch)
# training jobs
4
8
12
20
pixel (position loss) pixel (factor loss)
32
0:30
1:00
1:30
2:30
32
0:45
1:20
2:00
3:15
linear (position loss)
6
10:20
20:20
30:15
50:00
linear (factor loss)
6
4:00
7:50
11:30
19:00
predsegl I (position loss)
16
4:05
7:45
11:25
18:40
predsegl (factor loss)
24
2:20
4:40
6:45
11:10
Networks were trained on an internal cluster using one GPU at time and 6 CPUs for data loading. We list the training time per epoch table If every job had for the full 48 hours we would have used (1 + 7 + 9) x 24 x 2 = 816 days of GPU processing time, which is relatively close upper bound on the time we actually used.
A.S COMPARISON OF THE TWO LOSSES
The position loss is consistent with the prediction made by the whole Markov random field, but is relatively inefficient, because the predicted distribution pi P(filJyy E N(i) and the normalization constants for these conditional distributions are different for every location i. Thus, the second term in equation (9) cannot be reused across the locations i. Instead, we need to compute the second term for each location separately, which requires similar amount of memory the whole feature representation for each negative sample i' and each neighbor.
time compared to the loss with many negative samples, each position and each sample in the normalization set.
The factor loss does not lead to a consistent estimation of the MRF model, because the prediction p(filfs) should not be based only on the factor thij but should include indirect effects as fj also constrains the other neighbors of i. Optimizing each factor separately will thus overaccount for information that could be implemented in two factors. However, the factor loss has the distinct advantage that the same noise evaluations can be used for all positions and images in minibatch, which enables a much larger number of noise samples and thus much faster learning.
16


Reviews:
[{"review_id": "sIRon7-NzG", "reviewer": "Reviewer_khhk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Markov random field model that learns both features and connectivity (whether two locations belong to the same segment), in an unsupervised manner using contrastive learning.  The trained model is able to extract contours, and gives good performance on the Berkeley segmentation database.  ", "summary_of_the_review": "While the proposed method has some novelty and interesting results on unsupervised contour extraction, given the concerns above (would like more empirical validation, some strangeness in the ordering of the model performance, in particular the good performance of the simpler feature map models, and the poor performance of Predseg1-Layer 1), I would lean toward reject.", "strength_and_weaknesses": "Strengths:\n\n+ Empirical performance - unsupervised method for contour extraction gives good performance on the Berkeley database, comparable to or better than hand-crafted methods.\n\nWeaknesses:\n\n- Extent of empirical validation - only studied on one benchmark, did not compare against existing deep neural networks for unsupervised segmentation (as opposed to contour extraction), would have been nice to get some sense of a comparison even if some simplifying assumptions had to be made.\n\n- It's strange to me that the pixel-based model performs well, relative to the other proposed models, and also that the best performing model seems to be Predseg1-Layer 0, which is, as noted, only using 3x3 local color averages as features.  This seems to suggest that the model is good at connectivity/contour extraction, but not necessarily in learning good features.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z7F34ES8pl", "reviewer": "Reviewer_FyoF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a pairwised Markov random field model to learn both features and segmentation from images without further supervision signals. They showed that the features learned by the shallow neural networks based on the contrastive learning loss are local averages, opponent colors and Gabor-like stripe patterns as observed in early visual cortex. They can also inder connectivity between locations by inferring the switch variables, and contours inferred from such connectivty perform well on a benchmark.", "summary_of_the_review": "The novelty and contribution are not enough. It is hard to know what the authors want to say in some parts of the paper.", "strength_and_weaknesses": "Strength:\nThe idea behind is interesting and the authors did a lot of careful empirical study to verify it. \n\nWeaknesses:\n1, The introduction is not convincing enough. It is good to do a broad review of visual processing, but it is hard to get the point of what the authors want to claim in the introduction (except the last paragraph). A lot of stuff in the introduction seems like unnecessary. It is better to say more words about the model in the introduction. \n2, It is unclear how the proposed model contribute to the computer vision community. Can this method outperform current deep-learning based methods for image feature extraction and segmentation?\n3, Many of the implementation details can be put into Appendix.\n4, The authors mentioned \"human-like\" features at several place in the paper. However, what is human-like features? Gabor-like patterns are first found on cats' visual cotex. It is not enough to say it is human like from Fig2, Fig3A  and Table1\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JROTb8fBzJv", "reviewer": "Reviewer_Pt4H", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a local model for boundary detection. At the core of the method, it assumes a local Gaussian Markov model in the feature space, where the features can be jointly learnt or derived from a neural network. The model uses a contrastive learning scheme to optimize features and pixel location connectivity, which gives the final contour prediction results.", "summary_of_the_review": "Given the concerns above, I vote to reject this paper in its current state. The author needs to improve clarity and make sure the reader can understand what's really going on with minimum guessing.", "strength_and_weaknesses": "---\n### strength\nIf everything holds up, the strength of this paper is its novelty. It shows that even with local gaussian Markov models, under the proposed contrastive learning goal, one can recover meaningful boundary maps by looking at the pixel connectivity matrix. (Spectral clustering is needed to generate segments first and use edge detectors for contours.)\n\n---\n### Weaknesses\nThis paper is poorly written with many missing pieces and handwavy arguments.\n\n1. The learning objective.\nIn the loss section, only $\\psi_{ij}$ appears in both terms, so why $p_{ij}$ is being optimized as well? $\\psi_{ij}$ is just a gaussian error term between $f_i$ and $f_j$, with optimizable diagonal variance $C_{ij}$. How is the related to the connectivity $w_{ij}$? Also please avoid using $p_{ij}$ because you are also using $p(\\cdot)$ as probability.\n\nI would highly recommend the author rewrite the method section in a way that the learning objective is obvious. According to the formulation in the text, the final prediction result is $p(w | f)$, then how is this target linked with the proposed objective? This needs a more detailed derivation, and since this is the core of the proposed method, it should be written as a formal theorem where you start from the contrastive objectives and ends with a MLE or approximation of $p(w | f)$. \n\nRight now, from the current version of the manuscript, I can only make an educated guess that after optimizing with the contrastive feature learning goal, a MLE estimation of $p(w|f)$ is performed, given the learnt feature and covariance parameters. I don't think this is what happens because the authors mentioned in the text that $p_ij$ is also being optimized, which never appears. I suspect the learning goal is wrong, where you need the $p_ij$ terms as well, otherwise you are approximating some distribution (not stated clearly) with a different Markov field than the one stated in Eq2.\n\n2.  Assuming it should not be just $\\psi_{ij}$ in the loss term, but with $p_ij$ as well, the authors should make it explicit about what exactly are they approximating using the similar formulation of NCE, with equations as proof. I find handwavy arguments like: \n\n> we do not optimize the Markov random field\ndirectly, but optimize predictions based on the model using features from other locations as the noise\ndistribution. For this noise distribution, the factors that depend only on a single location (the first\nproduct in (1)) will cancel. \n\nvery vague and hard to follow. What do you mean by optimizing a markov field? Optimizing for an MLE estimate? What are the parameters? MLE of what distribution? What noise distribution? Why did it cancel out? they really should be equations in addition to words.\n\n\n3. There's weird organization issues and formulation issues. Starting from Eq.4 to Sec 2.1, the derivation of normalization factor is for what? According to text in the later section, it seems you need this for some logit calculation. But it's funny to say that something is a proper distribution after deriving normalization factor: normalization factor is exactly what makes it a proper distribution, by definition. A better way to formulate this is to just let the pair-wise potential (factor) to be $f(w_ij, f_i, f_j)$, and define this function to be the terms inside the prod operator. for each individual factor, p(w_ij=1) is p_ij by definition, but not the marginal distribution on the markov field, a simple fact . The authors make statements about this in a weird way, but it's just a simple distinction between marginal distribution vs local definition of the distribution type. I'm still confused about why stressing this point though; are you approximating the marginal distribution P(w_{ij}) somewhere?\n\nBecause of the conflicting arguments about optimizing $p_ij$ and not having it in the objective, I can't make sense of what the optimization is about. Even if $p_ij$ is included, it's still not obvious to see what distribution approximation the authors are making, and how to get a point estimate of the connectivity in the end (is it MLE over some distribution? ).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J0xGh7niFj", "reviewer": "Reviewer_BpGS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a model that learns both features and segmentation without supervision. The method combines contrastive learning, a Markov random field model for the feature maps and segmentation. Learning utilizes two losses; position loss which optimizes the probability of the feature vector at each location relative to the probability of randomly chosen other feature vectors from different locations and images and factor loss, which maximizes factors for the correct feature vectors relative to random pairs of feature vectors sampled from different locations and images. The goal of the research is to evaluate, if it is possible to learn human-like features and segmentations.", "summary_of_the_review": "Very interesting and well written paper with novel method, however, the relevance of the developed method was left a bit unclear.", "strength_and_weaknesses": "The paper is very well written and scientifically and technically sound. The method includes novelty, it has been carefully discussed and evaluated. However, the relevance of the paper was left a bit unclear. Unsupervised segmentation methods exist, why is it relevant to learn the features simultaneously, or wasn't this the main contribution here?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject