Unsupervised learning of features and object boundaries from local prediction
Authors: Heiko H. Schütt
Wei Ji Ma
Venue: NeurIPS 2022 Submitted (2022)
Keywords: unsupervised
self-supervised
segmentation
grouping
prediction
local
probabilistic
contours
object boundaries

Abstract: A visual system has to learn both which features to extract from images and how to group locations into (proto-)objects. Those two aspects are usually dealt with separately, although predictability is discussed as a cue for both. To incorporate features and boundaries into the same model, we model a layer of feature maps with a pairwise Markov random field model in which each factor is paired with an additional binary variable, which switches the factor on or off. Using one of two contrastive learning objectives, we can learn both the features and the parameters of the Markov random field factors from images without further supervision signals. The features learned by shallow neural networks based on this loss are local averages, opponent colors, and Gabor-like stripe patterns. Furthermore, we can infer connectivity between locations by inferring the switch variables. Contours inferred from this connectivity perform quite well on the Berkeley segmentation database (BSDS500) without any training on contours. Thus, computing predictions across space aids both segmentation and feature learning and models trained to optimize these predictions show similarities to the human visual system. We speculate that retinotopic visual cortex might implement such predictions over space through lateral connections.

Unsupervised learning of features and object boundaries from local prediction
Anonymous Author(s)
Affiliation Address email
Abstract
A visual system has to learn both which features to extract from images and how to group locations into (proto-)object Those two aspects are usually dealt with separately, although predictability is discussed features and boundaries into the same model, we model a layer of feature maps with a pairwise Markov random field model in which each factor is paired with additional binary variable, which switches the factor on or off. Using one of two contrastive learning objectives, the Markov random field factors from images without further supervision signals. The features learned by shallow neural networks based on this loss are local aver- ages, opponent colors, and Gabor-like stripe patterns. Furthermore, we can infer connectivity between locations by inferring the switch variables. Contours inferred from this connectivity perform quite well on the Berkeley segmentation database (BSDS500) without any training on contours. Thus, computing predictions across space aids both segmentation and feature learning, and models trained to optimize these predictions show similarities the human visual system. We speculate that retinotopic visual cortex might implement such predictions over space through lateral connections.
2
cue for both. To incorporate
4
learn both the features and the parameters of
10
11
13
14
15
17
Introduction
18
19
long-standing question about human vision is how representations initially be based on parallel processing of retinotopic feature maps can represent objects in useful way. Most research on this topic has focused on computing later object-centered representations from the feature map representations. Psychology and neuroscience identified features that lead to objects being grouped established feature integration into coherent objects as a sequential process 73] and developed solutions to the binding problem, i.e. ways how neurons could signal whether they represent
20
21
23
together 37 38
24
25
parts of the same object 17 67 12] In computer vision, researchers also focused on how feature map representations could be turned into segmentations and object masks. Classically, segmentation algorithm were clustering algorithms operating on extracted feature spaces 22 12 [3 [6 66), and
27
this approach is still explored with more complex mixture models today 77. Since the advent of deep neural network models, the focus has shifted towards models that directly map to contour maps
29
semantic segmentation maps 21 277
69
50
65]
83 as reviewed in [54].
31 Diverse findings suggest that processing within the feature maps take object boundaries into account. For example, neurons appear to encode border ownership 34
57 [X] and to fill in information across [6 Also, attention spreading through the feature 59 And selecting neurons that correspond to object
33 surfaces e0 and along illusory contours
maps seems to respect object boundaries 4
35 takes time, which scales with the distance between the points to be compared [35] M. Finally, long history of psychophysical studies showed that changes in spatial frequency and orientation
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37
content can define (texture) boundaries [e.g. relatively little attention has been given to these effects of grouping or segmentation on the feature maps themselves.
45 81
In both human vision and computer vision,
38
Additionally; most theories for grouping and segmentation take the features in the original feature maps as given. In human vision, these features are traditionally chosen by the experimenter [37 are inferred based on other research [57 Similarly, computer vision algorithms used and have recently moved towards deep neural network representations trained for other tasks as souree for feature maps
41
42
73
12
43
off-the-shelf feature banks originally
44
39 50
45 Interestingly, predictability of visual inputs over space and time has been discussed as a solution for both these limitations of earlier theories. Predictability has been used cue for segmentation since the law of common fate of Gestalt psychology (77, and both lateral interactions in visual cortices and contour integration respect the statistics of natural scenes 19
47
Among other signals like sparsity 551 or reconstruction [36]. predictability is also a well known signal for self-supervised learning 49
20
50
of features [S,, which has been exploited by many recent contrastive learning [e.g. [5
2 2y 75
51
and predictive coding schemes [e.g. [5], 52 [75] for self-supervised learning. However, these uses of predictability for feature learning and for segmentation are usually studied separately.
52
53
Here, we propose a model that learns both features and segmentation without supervision. Predictions between locations provide self supervieed loss to learn the features, how perform the prediction and how to infer which locations should be grouped. Also, this view combines contrastive learning Markov random field model for the feature maps 46 and segmentation into coherent framework. We implement our model using some shallow architectures. The learned features resemble early cortical responses and the object boundaries infer from predictability align well with human object contour reports from the Berkeley segmentation database (BSDS500 (2). Thus, retinotopic visual cortex might implement similar computational principles as we propose here.
55
24
57
59
80
2
Model
61
62
To explain our combined model of feature maps and their local segmentation information, we start with a Gaussian Markov random field model 46 with pairwise factors. We then add a variable 63
64 E {0, }} to each factor that governs whether the factor enters the product or not. This yields a joint 65 distribution for the whole feature map and all w's. Marginalizing out the w's yields a Markov random field with "robust" factors for the feature map, which
to predict feature vectors from the 87 vectors at neighboring positions. We find two contrastive losses based on these predictions that can be used to optimize the feature extraction and the factors in the Markov random field model.
89 We model the distribution of k-dimensional feature maps f E R4.m' .n'm that are computed from input
70 images Rem.. with e = color channels (see Fig. D & B). We use a Markov random field model with pairwise factors, i.e. we define the probability of encountering feature map f with
72 entries fi at locations E [1... m²) x [1 .I...1 n²' as follows:
pff x II (.. II St)(
(GO)E
73 where u); is the local factor, N is the set of all neighboring pairs, and thi is the pairwise factor
74. between positions and I We will additionally assume shift invariance, i.c. each point has the same
75 set of nearby relative positions in the map as neighbors, Yb is the same factor for each position, and
76 each factor vha depends only on the relative position of i and j.
(0, 1} each pairwise factor that encodes whether the factor 78 is "active" (w = 1) for that particular image (Fig. C). To scale the probability of w = and
We now add binary variable
79 0 relative to each other, we add factor that scales them with constants Pij [0, 1] and Pij
80 respectively:
pff, w) x II ((A
(1-p)))= Pij (1 - pty)= 1- (fr( ( (KJ)N
II
(2)
L and j thus have two entries each

---

A
falwar
0)
(2).
f2
f1 - f2
f1
fi
f1
+
f2
f2
f2
D
f1
fi
fi
4
12
20
f2
f2
f2
Figure I: Illustration our Markov random field model for the feature maps. A: An example input image. B: Feature map with 4 neighborhood connectivity and pixel color as the extracted feature. In the actual models, these feature maps are higher dimensional maps extracted by a convolutional neural network model. C: Illustration of the factor that links the feature vectors at neighboring locations for a 1D feature. Top row: projection features value the connection variable Wij being 1 or O respectively. Middle row: 2D representation of the factor and its parts plotted against both feature values. Bottom row: Multiplication of the middle row with the standard normal factor for each position yielding the joint distribution of two isolated positions. D: Neighborhoods of different sizes used in the models, scaling from 4 to 20 neighbors for each location.
the factor Viy onto the difference between the f.. showing the combination of Gaussian around 0 and constant function for
81 Finally, we assume that the factors are Gaussian and the feature vectors are originally normalized to 82 have mean 0 and variance 1:
Pij 1h
(1 (1-1P) Pij)
pf. w) = (f. II (i.))N
exp (- Wii (fr Cigf
(3)
Zo
Z(wij, Cis)
2
where Zo is the overall normalization constant, (f,, 0, I) is the density of standard normal 84 distribution with x m' x n' dimensions, Ci governs the strength the coupling in the form of precision matrix, which will assume to be diagonal, and Z(wij Cit) scales the distributions with 86 Way = 0 and Wij = relative to each other.
We set (Wwi, Cij) to the normalization constant of the Gaussian with standard Gaussian factors for Si and Sj respectively. For 0 this is just (2m)-*, the normalization constant of standard 89 Gaussian in 2k dimensions. For we get:
ʃ/exp
2 fTTf ft 15 (1. -f 81) dfidf, (4)
1 1
Z(wiy = 1,C4)
Cts
Cū
(2T) del.
Cjj
C+
(5)
(2m)
1 2chl
so which we get by computing the normalization constant of a Gaussian with the given precision and then using the assumption that Ot is diagonal matrix with diagonal entries cu-

---

92 This normalization depends only on w and the coupling matrix C' of the factor Viy and thus induces a valid probability distribution on the feature maps. Two points are notable about this normalization
93
94 though: First, once other factors also constrain Ji and/or fj. this normalization will not guarantee P(W3; = 1) = Pij. Second, the Wij are not independent in the resulting distribution. For example, if pairwise factors connect a to and a to e the corresponding w are dependent, because wab =
95
96
97
and Whe = 1 already imply smaller difference between fa and fc than if these factor were inactive, which increases the probability for Wac
96
2.1
Learning
100
To learn our model from data, we use a contrastive learning objective on the marginal likelihood p(f). To do so, first need to marginalize out the w's ss which is fortunately simple, because each w affects only a single factor:
102
1 OF,,O,I p(f) = Σ plf, w) Zo
(n + pc)) (CJ)E
II
(7)
w
103
Using this marginal likelihood directly for fitting is infeasible though, because computing Zo. i.c. normalizing this distribution is not computationally tractable.
104
105
We resort to contrastive learning to fit the unnormalized probability distribution 24 i.e. we optimize discrimination from noise distribution with the same support the target distribution. Following we do not optimize the Markov random field directly, but optimize predictions based on the model using features from other locations as the noise distribution. For this noise distribution, the
107
75]
108
109
factors that depend only on a single location (the first product in (I) will cancel. We thus ignore the (f, O, I) in our optimization and instead normalize the feature maps mean 0 and unit variance
110
across each image. We define two alternative losses that make predictions for positions based on all their neighbors or for single factor respectively.
112
113
2.1.1
Position loss
114
The position loss optimizes the probability of the feature vector at each location relative to the probability of randomly chosen other feature vectors from different locations and images:
P(fifijj E N()) Ee
Apos (f)
Elog log
(8)
Ee PifelfNii E N(1))
Σ Σ log (hr Ji) Elog
Σ exp
log (oolee
jEN(
(EN((
116
where N(i) is the set of neighbors of i.
This loss is consistent with the prediction made by the whole Markov random field, but is relatively inefficient, because the predicted distribution pffiffvvj E N()) and the normalization constants for these conditional distributions
118
different for every location i. Thus, the second term in equation 9 cannot be reused across the locations i. Instead, we need to compute the second term for each location separately, which requires similar amount of memory as the whole feature representation for each negative sample '' and each neighbor.
120
121
122
123 To enable a sufficiently large set of negative points i' with the available memory, we compute this loss multiple times with few negative samples and sum the gradicnts. This trick saves memory, because can free the memory for the loss computation after each repetition. As the initial computation of the feature maps is the same for all negative samples, we can save some computation for this procedure by computing the feature maps only once. To propagate the gradients through this single computation, we add up the gradients of the loss repetitions with regard to the feature maps and then propagate this summed gradient through the feature map computation. This procedure does not computation time compared to the loss with many negative samples, as we still need to calculate the evaluation for each position and each sample in the normalization set.
124
126
128
130
131
2 Instead, p(w,, = 1) will be higher, because other factors increase the precision for the feature vectors, which makes the normalization constants more similar.

---

132
2.1.2
Factor loss
133
The factor loss instead maximizes each individual factor for the correct feature vectors relative to random pairs of feature vectors sampled from different locations and images:
134
(f) Eng (b(
Mact
Σ log
(10)
ij
Σ
log Vi (fr,
Σ log E folleeffy)
(11)
where J index the correct locations and i', index randomly drawn locations, in our implementation generated by shuffling the feature maps and taking all pairs that occur in these shuffled maps.
136
137
This loss does not lead to a consistent estimation of the MRF model, because the prediction p( p(/:),)) should not be based only on the factor diiy. but should include indirect effects as fy also constrains the other neighbors of i. Optimizing each factor separately will thus overaccount for information that could be implemented in two factors. However. this loss has the distinct advantage that the same noise evaluations can be used for all positions and images in minibatch, which enables much larger number of noise samples and thus much faster convergence.
138
139
140
142
143
2.1.3
Optimization
144
We optimize all weights of the neural network used for feature extraction and the parameters of the random field, i.e. the connectivity matrices C and the Ps for the different relative spatial locations simultancously. As an optimization algorithm we use stochastic gradient descent with momentum. Further details of the optimization can be found in the supplementary materials.
145
148
147
2.2
Segmentation inference
149
Computing the probability for any individual pair of locations (i,j) to be connected, i.c. computing p(wy = f). depends only the two connected feature vectors fi and fy:
Z(w, = 0, C,),) p(w; 0ff) (1 (1-Pa) Pb) Z(wij 1, Ci,)
p(w, = 1|f) Pij
exp (-
(12)
151
This inference effectively yields a connectivity measure for each pair of neighboring locations, i.e. a sparse connectivity matrix. Given that we did not apply any prior information enforcing continuous objects or contours, the inferred Wij do not necessarily correspond to a valid segmentation or set of contours. Finding the best fitting contours or segmentation for given probabilities for the is
152
153
154
155
additional process, which in humans appears to be an attention-dependen serial process B5 63
To evaluate the detected boundaries in computer vision benchmarks, nonetheless need to convert the connectivity matrix we extracted into a contour image. To do so, we use the based globalization method developed by [2] This method requires that all connection weights between nodes are positive. To achieve this, we transform the log- tegg-rbbabblity ratios for the Way as follows: For each image, find the 30% quantile of the values, subtract it from all probabiiiyyy ratios, and set all values below 0.01 to 0.01. We then compute the smallest eigenvectors of the graph Laplacian in graph spectral clustering. These eigenvectors are then transformed back into image space and are filtered with simple edge detectors to find the final contours.
157
159
161
162
163
Evaluation
184
We implement model types implementing feature extractions of increasing complexity in PyTorch
186
Pixel value model. For illustrative purposes, we first apply our ideas to the rgb pixel values of an image as features. This provides us with an example, where we can easily show the feature values and connections. Additionally, this model provides easy benchmark for all evaluations.
188
5

---

Linear, Position loss
Linear, Factor loss
Linear, best segmentation
D Predseg layer
B
E Predseg 1, Position Loss
F Predseg Factor Loss
Figure 2: Example linear filter weights learned by models. Each individual filter is normalized to minimum and maximum 1. As weights can be negative even a zero weight can lead to pixel having some brightness. For example, number of channels load similarly red and green across positions. Where these weights are positive the filter appears yellow and where the weights are negative filter appears blue, even if the blue channel has zero weight. A-C: Feature weights learned by the linear model. A: Using the position loss. B: Using the factor loss. C: The weights of the model that leads the best segmentation performance, i.e. the one shown in Figure E the first convolution in predsegl. Next to the filter shapes, which are nearly constant, we plot the average weight of each channel onto the three color channels of the image. Predsegl filters in the second convolution for network trained with the position based loss. F: Predseg filters in the second convolution for network trained with the factor based loss.
D: Weights of
170 Linear model. As the simplest kind of model that allows learning features, we use a single convolu- tional deep neural network layer feature model. Here, use 50 11 11 linear features.
172
Predsegl: show that our methods work for more complex architecture with non-lineasities, we relatively small deep neural network with 4 layers (2 convolutional layers and 2 residual blocks with subsampling layers between them, see supplement details).
174
175
For each these architectures, we train 24 different networks with all combinations of the following settings: 4 different sizes neighborhoods (4, 12, or 20 neighbors, see Fig. ID): different noise levels (0, 0. 0.2) and the two learning objectives. As training set, we used the unlabeled image from MS COCO 48) which contains 123,404 color images with varying resolution. To enable batch processing, we randomly crop these images to 256 x 256 pixel resolution, but use no other data augmentation (See supplementary information for further training details).
178
177
178
179
180
181
We want to evaluate whether our models learn meaningful features and segmentations. To do so, we first analyze the features in the first layers of our networks where we can judge whether features are representative of biological visual systems. In particular, we extract segmentations from our activations and evaluate those on the Berkeley Segmentation Dataset BSDS500]
182
183
184
185
3.1
Learned features
186
Linear Model
We first analyze the weights in our linear models (Fig 2 A-C). All instances learn local averages and Gibor-like striped features, i.c. spatial frequency and orientation tuned features

---

A
Pixel
Linear
PS1-LO
PS1-L1
B
Humia consisieney
0.8
Precision
0.4
9PD ao0---w---ommmmmmm (201
Freddngg 1-0
0.2
Cahryy (1088)
Fradeeg1-4
Mean Steft (P009)
Lit car features
Velberaswwat: (2000)
Pixcl values
Cuts (2025)
curvee
0.2
0.4
0.6
0.8
Recall
Figure 3: Contour detection results. A: Example segmentations from our models. B: Precision-
recall curves for our models on the Berkeley segmentation dataset, with some other models for
comparison as evaluated by 22 gPb-uwt-ucm, the final algorithm combining all improvements [2].
Canny's classical edge detector 77. the mean shift algorithm [ Felzenschwalbs algorithm []6]
and segmentation based on
normalized cuts [3. For all comparison algorithms evaluations on BSDS
were extracted from the figure by
188 with limited spatial extend. These features clearly resemble receptive fields neurons in primary visual cortex. Additionally, there appears to be some preference for features that weight the red and green color channels much stronger than the blue channel, similar to the human luminance channel, which leads to the yellow-blue contrasts in the plots. There is some difference between the two learning objectives though. The position based loss generally leads to lower frequency and somewhat noisier features. This could either be due the higher learning efficiency of the factor based loss, i.e. the factor based loss closer to convergence, or due to a genuinely different optimization goal.
190
192
194
Predseg1 In Predsegl, first analyze the layer O convolution (Fig. 2D). which has only 3 channels with 3 3 receptive fields, which we originally introduced as a learnable downsampling. This layer consistently converges to applying near constant weights over space. Additionally, exactly the channels has a non-zero mean (the 3rd, 1st and 3rd in Fig. 2) and the other two take balanced differences between two of the channels (red green and green blue in the examples). This parallels the luminance and opponent color channels of human visual perception.
196
of
198
200
In the second convolution, we observe similar pattern of oriented filters and local averages in the linear model albeit in false color as the input channels are rotated by the weighting of the layer convolutionn (Fig. E & F).
202
204
3.2
Contour detection
205
To evaluate whether the connectivity information extracted by our model corresponds to human perceived segmentation, extract contours from models and compare them to contours reported by humans for the Berkeley Segmentation database [2] [53]. This database contains human drawn object boundaries for 500 natural images and is accompanied by methods for evaluating segmentation models. Using the methods provided with the database, we compute precision-recall curves for each model and the best F-value (geometric mean of precision and recall) the final evaluation metric.
207
209
210
211 As we had multiple models to choose from, we choose the models from each class that perform best on the training data for our reports. For all models this was one of the models with the largest neighborhood, i.c. using 20 neighbors, and the factor loss. seems the factor loss performed better simply due to its technical efficiency advantage discussed above. Performance increases monotonically with neighborhood size and Markov random field based approaches to semantic segmentation also increased their performance with larger neighborhoods up fully connected We thus expect that larger neighborhoods could work even better.
213
215
217
Markov random fields 43 g.
Qualitatively, we observe that all our models yield sensible contour maps (see Fig. 3 A). Even the contours extracted from the pixel model yield sensible contours. Additionally, we note that the linear model and Layer 1 of the predseg model tend to produce double contours, i.c. they tend to produce
219
7

---

Table 1: Numerical evaluation for various algorithms on the BSDS500 dataset. Precision and recall are only given for ODS, i.e. with the threshold fixed across the whole dataset.
model
Recall
Precision
F(ODS)
F(OIS)
Area PR
Deep Contour*> #1
0.76
0.78
0.80
HED*
0.79
0.81
0.84
RCF*
50
0.81
0.83
Deep Boundary
39
0.813
0.831
0.866
*DCN**
27
0.83
0.84
0.89
Canny*
0.60
0.63
0.58
Mean Shift
0.64
0.68
0.56
welzenszwalb*
16
0.61
0.64
0.56
Normalized Cuts
13
0.64
0.68
0,45
gPb-owt-ucm
0.73
0.73
0.73
0.76
0.73
Pixel
0.73
0.66
0.69
0.69
0.73
linear
0.78
0.66
0.72
0.73
0.75
Predseg 1 1-Layer 0
0.79
0.69
0.74
0.73
0.80
Predseg L-LLaerr
0.74
0.47
0.57
0.59
0.45
*; Evaluation of these algorithms taken from [2]
**; Supervised DNNs, evaluation taken from 27
221
two contours on either side of the contour reported by human subjects with some area between them connected to neither side of the contour.
223
Quantitatively, our models also perform well except for the deeper layers of Predseg (Fig. and Table D]. The other models beat most hand-crafiee contour detection algorithms that were tested on this benchmark 7
224
225
[6] and perform close to the gPb-owt-uem contour detection and segmentation algorithm [2] that was the state of the art at the time. Layer-0 of Predseg 1 performs best followed by the linear feature model and finally the pixel value model. Interestingly, the best performing models seem to be mostly the local averaging models (ef. Fig, 2
227
C). In particular, the high performance the first layer of Predseg is surprising, because it uses only 3 pixel local color averages as features.
229
Since the advent of deep neural network models, networks trained optimize performance on image segmentation have reached much higher performance on the BSDS500 benchmark, essentially reaching perfect performance up human inconsistency [e.g. 27 39 50
232
233
see Table D However, these models all require direct training on human reported contours and often use features learned for other tasks. There are also a few deep neural network models that attempt unsupervised segmentation [e.g. [C
/1 X
235
but we were unable to find any that were evaluated on the contour task of BSD500. The closest is perhaps the W-net [82), which used an autoencoder structure with additional constraints and was evaluated on the segmentation task on BSDS500 performing slighly better than gPb-owtuucm.
237
239
4
Discussion
We present model that can learn features and local segmentation information from images without further supervision signals. This model integrates prediction task used for feature learning and the segmentation task into the same coherent probabilistic framework. This framework and the dual use for the connectivity information make it seem sensible to represent this information. Furthermore. the features Iearned by our models resemble receptive fields in the retina and primary visual cortex and the contours we extract from connectivity information match contours drawn by human subject fairly well, both without any training towards making them more human-like.
242
244
246
248
To improve biological plausibility, all computations our model are local and all units are connected to the same small, local set of other units throughout learning and inference, which matches early visual cortex, in which the lateral connections that follow natural image statistics are implemented anatomically G
250
59 70 This in contrast to other ideas that require flexible pointers to arbitrary locations and features Jas discussed by 34 or capsules that flexibly encode different parts of the input Nonetheless, employ contrastive learning objectives and backpropagation here,
252
14
2
51 62

---

254
for which we do not provide a biologically plausible implementations. However, there is currently active research towards biologically plausible alternatives to these algorithms [e.g.
255
32
M4
256
Selecting the neurons that react to a specific object appears to rely on some central resource
73
257
and to spread gradually through the feature maps 62
55 [%]
We used a computer vision algorithm
258
for this step, which centrally computes the eigenvectors of the connectivity graph Laplacian [2]. which does not immediately look biologically plausible. However, a recent theory for hippocampal place and grid cells suggests that these cells compute the same eigenvectors of graph Laplacian of a prediction network, albeit of a successor representation, i.e. of predictions of the animals state transitions [68, 691. Thus, this might be an abstract description of an operation brains are capable of. In particular, earlier accounts that model the selection as a marker that spreads to related locations le.g. 17 558 677 have some similarities with iterative algorithms to compute eigenvectors. Originally,
259
280
261
262
263
284
265
phase coherence between the neurons encoding the same object was proposed 11 57 67 but gain increase with object based attention [58] or known random modulation is also sufficient to select a task relevant set of neurons 25
266
267
26 Regardless of the mechanistic implementation of the marker,
connectivity information of the type our model extracts would be extremely helpful to explain the gradual spread of object selection.
269
270
Our implementation of the model is not fully optimized, as it is meant as a proof of concept. In particular, we did not optimize the architectures or training parameters of our networks for the task, like initialization, optimization algorithm, learning rate, or regularization. Presumably, better performance in all benchmarks could be reached by adjusting any or all of these parameters.
271
272
273
274
One possible next step for our model would be to train deeper architectures, such that the features could be used for complex tasks like object detection and classification. Contrastive losses like the one we use here are successfully applied for such pretraining purposes even for large scale tasks such as ImageNet (60) MS Coco 48 These large scale applications often require modifications for better learning though I
275
278
277
278
75 For example: Image augmentations to explicitly train networks be invariant to some image changes, prediction heads that allow more complex shapes for the predictions, and memory banks or other methods to decrease the reliance on many negative samples. Similar modifications might be necessary to apply our formulation deeper architectures for pretraining purposes. For understanding human vision. this line of reasoning opens the exciting possibility that higher visual cortex could be explained based on similar principles, as representations from contrastive learning also yield high predictive power for these cortices [86]
22
28
279
280
281
282
283
284
The model propose here is a probabilistic model of the feature maps. One implication of this is that we could also infer the feature values if they were not fixed based on the input. Thus, our model implies pattern how neurons should combine their bottom-up inputs with predictions from nearby other neurons, once we include some uncertainty for the bottom-up inputs. In particular, the combination ought to take into account which nearby neurons react to the same object and which ones do not. Investigating this pooling could provide insights and predictions for phenomena that are related to local averaging like crowding for example [ 18 30 77-779. where summary statistic models currently capture perceptual limitations best B.
286
287
288
209
290
291
292
18
78
but deviations from these predictions
293
suggest that object boundaries change processing 30
294
Another promising extension of our model would be processing over time, because predictions over time were found to be a potent signal for contrastive learning 15 and because coherent object motion is among the strongest grouping signals for human observers 38 and computer vision systems (85) Beside the substantial increases in processing capacity necessary to move video processing instead of image processing, this step would require some extension of our framework to include object motion into the prediction. Nonetheless, including processing over time seems to be an interesting avenue for future research, especially because segmentation annotations for video are extremely expensive to collect such that unsupervised learning is particularly advantageous and popular in
295
296
287
298
299
300
301
302
recent approaches 11 33
303
This work aims to move us closer to understanding how human visual perception can take object structure into account in retinotopic feature map processing and may help us to build systems with similar capabilities in the future. We acknowledge that such technological progress can have unknown
304
305
306
societal consequences, but we do not foresee specific negative consequences of this work.

---

References
307
308
[1] Nikita Araslanov, Simone Sehaur,-eyer, and Stefan Roth. Dense Unsupervised Learning for
Video Segmentation. Advances in Neural Information Processing Systems, 35:12, 2021.
310
[2] P Arbelácz, M Maire, C Fowkkes, and J Malik. Contour Detection and Hierarchical Image Segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 33(5):898- -96691 2011.
312
313
[3] B. Balas, L.. Nakano, and R. Rosenholtz. A summary-statistic representation in peripheral vision explains visual crowding. Journal of Vision, 9(12):13-13, 2009.
314
315
[4] Daniel Baldauf and Robert Desimone. Neural mechanisms of object- attention. Science, 344(6182):444-277, 2014.
316
317
[5] Jacob Beck, Anne Sutter, and Richard Ivry. Spatial frequency channels and perceptual grouping in texture segregation. Computer Vision, Graphics, and Image Processing, 37(2):999-325, 1987.
319
[6] Péter Buzás, Krisztina Kovács, Alex S. Ferecskó, Julian M.L. Budd, Ulf T. Eysel, and Zoltán F. Kisvárday, Model-based analysis of excitatory lateral connections in the visual cortex. The Journal of Comparative Neurology, 499(6):861-88) 2006.
321
[7] John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (00:679-698, 1986.
323
325
[8] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv: 14 2.7622 2014.
326
327
328
[9] Liang- Chichh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution. and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
329
330
331
332
110] Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised Object Segmentation by Redrawing. Advances in Neural Information Processing Systems, 33:12, 2019.
333
334
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
336
[12] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603-619, 2002.
338
339
[13] T. Cour, F. Benezit, and Jianbo Shi. Spectral Segmentation with Multiscale Graph Decomposi- tion. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR105), volume 2, pages 1 1124-1131 San Diego, CA, USA, 2005. IEFF.
340
342
[14] Adrien Doerig, Lynn Schmittwilken, Bilge Sayim, Mauro Manassi, and Michael H. Herzog. Capsule networks as recurrent models of grouping and segmentation. PLOS Computational Biology, 16(7)xe1008017 2020.
343
344
345
[15] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study unsupervised spatiotemporal representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3299-3309, 2021.
346
347
[16] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient Graph Graph-Based Based Image Segmentation. International Journal of Computer Vision, 188) 2004.
349
[17] Holger Finger and Peter König. Phase synchrony facilitates binding and segmentation of natural images in a coupled neural oscillator network. Frontiers in Computational Neuroscience, 7, 2014.
351
10

---

353
[18] Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. Nature Neuroscience, 1499):1195-120) 2011.
354
355
[19] Wilson S. Geisler and Jeffrey S. Perry. Contour statistics in natural images: Grouping across occlusions. Visual Neuroscience, 26(1):109-121) 2009.
356
357
[0
W.S. Geisler, J.S. Perry, B.J. Super, and D.P. Gallogly. Edge co-occurrence in natural images predicts contour grouping performance. Vision Research, 41(6):711-722 2001.
359
[21] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
360
361
382
[22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural information processing systems, 12211222222244 2020.
383
384
385
366
[23] David H. Grosof, Robert M. Shapley, and Michael J. Hawken. Macaque VI neurons can signal 'illusory' contours. Nature, 365(6446):500-552, 1993.
367
368
[24] Michael Gutmann and Aapo Hyvarinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. International ConerertoonnCCoffeeennee on Artificial Intelligence and Statistics (AISTATS), pages 297-304, 2010.
370
371
[25] Caroline Haimerl, Douglas A Ruff, Marlene R Cohen, Cristina Savin, and Eero P Simoncelli. Targeted comodulation supports flexible and accurate decoding in V1. bioRxiv the preprint server for biology, 2021.
372
374
[26] Caroline Haimerl, Cristina Savin, and Eero Simoncelli. Flexible information routing in neural populations through stochastic comodulation. Advances in Neural Information Processing Systems, 32, 2019.
375
378
377
[27] Jianzhong He, Shiliang Zhang, Ming Yang. Yanhu Shan, and Tiejun Huang. Bi-Directional Cascade Network for Perceptual Edge Detection. In 2019 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
379
Long Beach, CA, USA, 2019. IEEE
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 9726-9735, Seattle, WA, USA, 2020. IEEE.
381
383
[29] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, and S M Ali Eslami. Data-Efficient Image Recognition with Contrastive Predictive Coding. Proceedings of the 37th International Conference on Machine Learning (PMLR), page 119, 2020.
384
305
386
[30] Michael H. Herzog, Bilge Sayim, Vitaly Chicherov, and Mauro Manassi. Crowding, grouping, and object recognition: A matter of appearance. Journal of Vision 15(6):5, 2015.
387
388
[31] Jonathan Hunt, William H Bosking, and Geoffrey Goodhill. Statistical structure of lateral connections in the primary visual cortex. Neural Systems & Circuits, 1(1):3, 2011.
390
[32] Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using seff-supervieed contrastive predictions. Advances in Neural Information Processing Systems, 35; 2021.
381
392
393
[33] Allan A Jabri, Andrew Owens, and Alexci A Efros. Space-Time Correspondence as Contrastive Random Walk. Advances in Neural Information Processing Systems. 34:16, 2020.
394
385
(34] Danique Jeurissen, Matthew W. Self, and Pieter R. Roelfsema. Surface reconstruction, figure- ground modulation, and border-ownership. Cognitive neuroscience, 4(1):50-52, 2013.
396
397
[35] Danique Jeurissen, Matthew W. Self, and Pieter R. Roelfsema. Serial grouping of 2D-image regions with object-based attention in humans. Elife, 5:e 14300 2016.
11

---

389
[36] Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. [cs. stat], 2014.
400
401
[37] K Koffka. Principles of gestalt psychology. 1935.
402
[38] Wolfgang Köhler. Gestalt psychology. Psychologische Forschung, 31()):XVIIIXXXX, 1967.
[39] Lasonas Kokkinos. Pushing the Boundaries of Boundary Detection using Deep Learning.
404
arXiv:1511.07386 [cs], 2016.
405
[40] Hidehiko Komatsu. The neural mechanisms of perceptual filling-in. Nature Reviews Neuro- science, 7(3):220-231,
408
407
[41] Ilia Korjoukov, Danique Jeurissen, Niels A. Kloosterman, Josine E. Verhoeven, H. Steven Scholte, and Pieter R. Roelfsema. The Time Course of Perceptual Grouping in Natural Scenes. Psychological Science, 23(12);1482-1453 2012.
409
410
[42] Adam R. Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E. Hinton. Stacked Capsule Autoencoders. arXiv: 06818 [cs, stat), 2019.
412
[43] Philipp Krähenbühl and Vladlen Koltun. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. 1rXiv:1210.564 [cs], 2012.
414
[44] Zihang Lai, Erika Lu, and Weidi Xie. MAST: A Memory-Augmented Self-Supervised Tracker. CVPR, pages 6479-6488, 2020.
415
416
[45] Michael S Landy and James R Bergen. Texture segregation and orientation gradient. Vision research, 31(4):679-691, 1991.
417
418
[46] SZ. Li. Markov Random Field Modeling in Computer Vision. Springer Science & Business Media, 2012.
420
[47] Qinghong Lin, Weichan Zhong, and Jianglin Lu. Deep Superpixel Cut for Unsupervised Image Segmentation. In 2020 25th International Conference Putlern Recognition (ICPR), pages 8876,8876, Milan, Italy, 2021. IEEE.
422
423
[48] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. arXiv: [cs], 2015.
425
[49] Drew Linsley, Junkyung Kim, Alekh Ashok, and Thomas Serre. Recurrent neural circuits for contour detection. In International Conference on Learning Representations, 2020.
427
428
[50] Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang. and Xiang Bai. Richer Convolutional Features for Edge Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3000-3009, 2017.
429
430
[51] William Lotter, Gabriel Kreiman, and David Cox. Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning. arXiv:1605. 00140 [cs, q-bio], 2017.
432
439
[52] William Lotter, Gabriel Kreiman, and David Cox. A neural network trained to predict fu- ture video frames mimics critical properties of biological neuronal responses and perception. 1805.
435
[cs, q-bio], 2018
436
[53] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pages 416-423, July 2001.
437
[54] Shervin Minaee, Yuri Y. Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image Segmentation Using Deep Learning: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
440
441
442
[55] Bruno Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381(6583):007-609 1996.
12

---

444
[56] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. 2019.
445
447
448
449
450
[57] Alina Peter, Cem Uran, Johanna Klon-Lipok, Rasmus Roese, Sylvia van Stijn, William Barnes, Jarrod R Dowdall, Wolf Singer, Pascal Fries, and Martin Vinck. Surface color and predictability determine contextual modulation of V1 firing and gamma oscillations. eLife, 8:e42101, 2019.
451
452
(58)
Pieter R Roelfsema. Cortical algorithms for perceptual grouping. Annu. Rev. Neurocii, 29:203- 227, 2006.
454
[59] Picter R Roelfsema, Victor AF Lamme, and Henk Spekreijse. Object-based attention in the primary visual cortex of the macaque monkey. Nature, 395(6700))376-381, 1998.
456
457
[60] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv: 1409.055 [cs], 2015.
458
459
460
[61] Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic Routing Between Capsules. arXiv:1710.09829 [cs]. 2017.
481
622]
Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E. Hinton, and David J. Fleet. Unsupervised part representation by Flow Capsules, arXiv:2011.13900 [cs], 2021.
483
[63] Matthew W. Self, Danique Jeurissen, Anne F. van Ham, Bram van Vugt, Jasper Poort, and Pieter R. Roelfsema. The Segmentation of Proto-Objects in the Monkey Primary Visual Cortex. Current Biology,
465
2019..
467
[64] Michael N Shadlen and J Anthony Movshon. Synchrony unbound: A critical evaluation of the temporal binding hypothesis. Neuron, 24(1):67-77, 1999.
468
[65]
Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang Zhang. DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3982-3991, Boston, MA, USA, 2015. IEEE.
470
471
472
[66] Jianbo Shi and Jitendra Malik. Normalized Cuts and Image Segmentation. IEEE Transactions On Pattern Analysis and Machine Intelligence, 22(8):18, 2000.
474
475
[67] Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual review of neuroscience, 18(1):555-588 586, 1995.
478
477
[68]
Kimberly L Stachenfeld, Matthew Botvinick, and Samuel J Gershman. Design Principles of the Hippocampal Cognitive Map. Advances in Neural Information Processing Systems, page 2014.
478
479
[69]
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. The hippocampus as a predictive map. Nature Neuroscience, 166(1))::63316666 2017.
481
482
[70] Dan D Stettler, Aniruddha Das, Jean Bennett, and Charles D Gilbert. Lateral Connectivity and
483
Contextual Interactions in Macaque Primary Visual Cortex. Neuron, 36(4):739-750, 2002.
484
[71] Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu. Pixel Difference Networks for Efficient Edge Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5117-5127, 2021.
485
486
[72]
Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
488
[73] Anne M. Treisman and Garry Gelade, Feature-Integratioon Theory of Attention. Cognitive Psychology, 12:97-136. 1980.
13

---

[74] Jonathan Vacher, Claire Launay, and Ruben Coen-Cagli. Flexibly regularized mixture models and application to image segmentation. Neural Networks, 149:107-123, 2022.
491
492
[75] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv: 1807.03748 [cs, stat], 2019.
[76]
R. von der Heydt, E. Peterhans, and G. Baumgartner. Illusory Contours and Cortical Neuron Responses. Science, 224(4654):126--1224 1984.
495
486
[77] Thomas S. A. Wallis, Matthias Bethge, and Felix A. Wichmann. Testing models of peripheral encoding using metamerism in an oddity paradigm. Journal of Vision, 16(2):4, 2016.
497
498
[78] Thomas A. Wallis, Christina M. Funke, Alexander S. Ecker, Leon A. Gatys, Felix A. Wichmann, and Matthias Bethge. A parametric texture model based on deep convolutional features closely matches texture appearance for humans. Journal of Vision, 17(12):5-5, 2017.
500
501
[79] Thomas SA Wallis, Christina M Funke, Alexander Ecker, Leon A Gatys, Felix A Wichmann, and Matthias Bethge. Image content is more important than Bouma's Law for scene metamers. eLife, 8:e42512, 2019.
503
504
[80] Laurenz Wiskott and Terrence Sejnowski. Slow Feature Analysis: Unsupervised Learning of Invariances. Neural Computation, 14(4):715-770, 2002.
505
508
[81] Sabina Wolfson and Michael Landy. Discrimination of orientation-defined texture edges. Vision research, 35(20):2833-287, 1995.
507
508
[22
Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprint 1rXiv:
509
2017.
510
[83]
Saining Xie and Zhuowen Tu. Holistically-Neste Edge Detection. In Proceedings of the IEEE International Conference Computer Vision, pages 1395-1403, 2015.
[84]
Local Contrastive Representation Learning. Advances in Neural Information Processing Systems, 34:12, 2020.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. LoCo:
513
514
[85] Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised Video Object Segmentation by Motion Grouping. S.Ki::2104.06655 [cs], 2021.
515
516
[86] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C. Frank, James DiCarlo, and Daniel I.. K. Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3)=e20141)) 41961 18, 2021.
517
518
14

---

Checklist
519
520
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
522
contributions and scope? [Yes]
523
(b) Did you descrihe the limitations of your work? [Yes]
524
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
526
527
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
529
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] In the supplementary material.
532
534
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the supplementary material.
535
536
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] This would require undue amounts of computation for results, which we interpret only qualitatively anyway.
537
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In the supplementary material.
540
541
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
542
(a) your work uses existing assets, did you eite the creators? [Yes]
543
(b) Did you mention the license of the assets? [N/A]
544
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
546
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
547
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
549
550
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
552
553
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
554
555
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "x2E-Q2XCQlc", "reviewer": "Reviewer_TYhZ", "summary": "* In this work the authors propose a model that can learn visual input features and segmentation without supervision which combines contrastive learning + Markov Random Fields (MRFs) to implement spatial feature prediction for feature maps + image segmentation.\n* Self supervised learning approach: prediction between locations yield self supervised loss to learn feature maps, how to infer which locations should be grouped.\n* Gaussian Markov Random Fields are used to model pairwise connections among features given feature maps and binary variables that indicate activity between connections. \n* When considering pairwise connections the four adjacent pixels from the neighbours of position i. The factors of features (point or pairwise) model the strength and influence of a point on its neighbours wrt to a feature. \n* There are two contrastive Learning Objectives defined on the marginal likelihood p(f)  1. Position Loss: optimizes probability of randomly chosen positions from other features + images 2. Factor Loss: maximizes factor for correct feature vectors relative to random feature vectors sampled from different locations and images.  Optimization is done on all parameters of the MRF in parallel via SGD with momentum \n* Segmentation Inference: p(w_i,j = 1 | f) is the probability that two points are connected.  This depends on the two feature vectors f_i ^& f_j.  This is computed by forming a sparse connectivity matrix from: p(w_i,j = 1 | f) / p(w_i,j = 0 | f) (claimed to be an attention dependent serial process)\n* Features are trained over three Models: Identity, Linear, and ResNet. Models are trained on MS COCO image dataset. The authors provide an analysis of the learned features for the linear and ResNet model.  Contours are extracted from the model and compared to the Berkeley (human) Segmentation Database (BSDS500).  The authors present recall and precision scores along with other segmentation metrics against a number of other baselines that may have been trained with additional supervision signals ( human contour / segmentation data + pretraining on other tasks). \n\nMain claims of the work:\n\n1. Predictions across space aid in segmentation and feature learning without further supervision signals (human contour data or pre-training on other tasks/data).\n2. Models optimized under this approach have similarities to the human visual system (retinotopic visual cortex)\n", "questions": "\nIt was stated that models using the factor loss + neighbourhood of size 20 performed best.  Was there a sense of whether further increases in the neighbourhood would yield further improvement in the scores?  Were any other losses considered?\n\nIn section 3.2 it's stated: \"... the best performing models seem to be mostly the local averaging models ...\" Do you have any intuition or explanation as to why the best performing models were those that do local averaging?\n\nCould this approach induce good representations for image or video data for downstream tasks or other applications?  Was this explored at all?\n\nWould prediction across features in the manner presented be applicable to other input modalities (e.g. language/text) or sensory data (e.g. speech or audio)?\n", "limitations": "The authors discuss the limits of their model in the context of low scale and minimal tuning - in effect this is a proof of concept.  They also note that the contrastive losses + learning are not optimal and could be modified to achieve better optimization.  Further the model does include any temporal signal which may also aid in helping to better model probabilistic features. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* Overall I believe this is a pretty interesting paper although I'm not deeply familiar with all of the ties to cognitive science and the human visual system.  However, I think that the authors do a good job of demonstrating links to the human visual system via the contour & segmentation analysis and the feature visualization without any specific supervision toward human visual cognition.\n* The results are clearly demonstrated and show that even with simple feature modeling that spatial feature prediction modeled by the MRF + contrastive losses the model achieves reasonable segmentation and contour prediction accuracy approaching the level of deep neural networks making use of pretraining on other tasks and training on human reported contours.\n* In the discussion there are some highlights regarding how this method performs object selection and how temporal data might be used to further improve model capacity to detect object structure (the binding problem).  Also the discussion of biological plausibility helps to underpin the relevance of this work. \n* The scores achieved look fairly good despite relatively little tuning and relatively small models even though optimizing over these metrics wasn't the aim of this research. \n\n**Weaknesses**\n\n* May need more analysis/explanation on the reasons behind why the factor loss performed well vs. the position loss.\n* Sub-section 2.3 may be more appropriately placed under section 3.  Are these meant to be baselines? Finally, Fig. 2 appears fairly early in the text relative to this subsection.  \n* The authors stated a number of future directions for this work in section 5 including processing video input.  It would be helpful to know if we could reasonably expect these models to advance state of the art and how this approach fits into the broader family of models tackling image/video segmentation and representation.  \n* Figure 2 D & E were a bit tricky to understand and might benefit from some clarification in the description or main text.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tb63_P9OTlL", "reviewer": "Reviewer_NeiE", "summary": "The paper proposes an unsupervised model for segmentation and contour detection. The unsupervised learning is based on training Markov Random Field model find new feature maps. The MRF is constructed by connectivity of pixels in a local image patch, and Gaussian responses (called factors). The feature maps are inserted into shallow neural networks, and ideal features are learned via contrastive loss based on noisy and correct locations of the pixels and gaussian factors. as well as target which can then be inserted into forward DNNs.", "questions": "See above. \n\nMinor: \nLine 3: .”..dealt with separately,” -> “ dealt separately” ?\n", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n•\tI agree with the authors that more research on grouping mechanisms for neural network models is valuable and can contribute to improved CNN for downstream visual recognition tasks. \n\nWeaknesses: \n•\tThe paper is hard to follow, and more intuitive explanations on the mathematical derivations are needed. Figure captions are lacking, and require additional explanations and legends (e.g., explain the colors in Fig. 2). Fig. 1 and 2 did not contribute much to my understanding, and I had to read the text few times instead. \n•\tAt the end of the day the model proposes a method to learn features for detecting boundaries, which is an old computer vision task. Indeed, it uses a new MRF framework, and contrastive loss, but it is not clear why not using DNNs with contrastive loss for doing that, besides that maybe the learned features are more like human vision features. \n•\tThe results of the models are not compared against unsupervised DNN models. I think it is interesting to see such a comparison, e.g., to unsupervised segmentation models that can be adjusted to the BSDS500 contour detection task.  \n•\tThere is not review of previous related work, and these is not section for “related work”. Can unsupervised segmentation model be relevant to your work?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dE2aQadmvJ", "reviewer": "Reviewer_6B8P", "summary": "The authors propose an unsupervised learning method which learns local features from image pixels both with weights that allow to group those features together.\n\nThe contributions are the following :\n- a description of their original model based on Markov Random Fields,\n- explanations about how to trained the proposed models following principle from contrastive learning,\n- a comparison of the learned features under variation of their algorithm (loss, linear/seg),\n- a comparison of the performance of their algorithm on a contour detection task (BSD500).", "questions": "1. The issue with the position loss and the trick to circumvent it are obscure to me. Can you elaborate more on the issue and its solution ? I do not really understand the issue and how the proposed trick solves it.\n\n2. I disagree with the fact that the linear models learn Gabor-like features. In particular for the position loss (Figure 3.A), the features are more like *non-local* wave-like features (Fourier).\n\n3. I would like to see some maps of p_ij or w_ij associated to some features to understand how the algorithm has grouped those.\n\n4. A related question is that the post-processing step to turn p_ij into contours makes hard to understand what was learned ? Is the use of spectral clustering transparent ie has the algorithm learned to group contours over any other type of grouping (eg textures) ?\n\nFew additional references about texture-based segmentation :\n- Beck, J., Sutter, A., & Ivry, R. (1987). Spatial frequency channels and perceptual grouping in texture segregation. Computer Vision, Graphics, and Image Processing, 37(2), 299-325.\n- Landy, M. S., & Bergen, J. R. (1991). Texture segregation and orientation gradient. Vision research, 31(4), 679-691.\n- Wolfson, S. S., & Landy, M. S. (1995). Discrimination of orientation-defined texture edges. Vision research, 35(20), 2863-2877.\n- Vacher, J., Launay, C., & Coen-Cagli, R. (2022). Flexibly regularized mixture models and application to image segmentation. Neural Networks, 149, 107-123.\n\nTypos :\nl 63 -> which we can us to\n", "limitations": "Limitations are sufficiently assessed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n- modelisation of grouping using a binary weight which is on or off whether the feature are grouped or not,\n- a method to train a seemingly untrainable model (without supervision),\n- multiple variations of their model are compared,\n- model comparison with SOTA algorithms,\n- a real concern about visual perception\n\nWeaknesses :\n- results are not SOTA,\n- non-negligible post-processing to extract image contours\n- learned features are shown but not the local weights given a feature and a location\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject