Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes an unsupervised model for segmentation and contour detection. The unsupervised learning is based on training Markov Random Field model find new feature maps. The MRF is constructed by connectivity of pixels in a local image patch, and Gaussian responses (called factors). The feature maps are inserted into shallow neural networks, and ideal features are learned via contrastive loss based on noisy and correct locations of the pixels and gaussian factors. as well as target which can then be inserted into forward DNNs.
strengths_and_weaknesses: Strengths: 
•	I agree with the authors that more research on grouping mechanisms for neural network models is valuable and can contribute to improved CNN for downstream visual recognition tasks. 

Weaknesses: 
•	The paper is hard to follow, and more intuitive explanations on the mathematical derivations are needed. Figure captions are lacking, and require additional explanations and legends (e.g., explain the colors in Fig. 2). Fig. 1 and 2 did not contribute much to my understanding, and I had to read the text few times instead. 
•	At the end of the day the model proposes a method to learn features for detecting boundaries, which is an old computer vision task. Indeed, it uses a new MRF framework, and contrastive loss, but it is not clear why not using DNNs with contrastive loss for doing that, besides that maybe the learned features are more like human vision features. 
•	The results of the models are not compared against unsupervised DNN models. I think it is interesting to see such a comparison, e.g., to unsupervised segmentation models that can be adjusted to the BSDS500 contour detection task.  
•	There is not review of previous related work, and these is not section for “related work”. Can unsupervised segmentation model be relevant to your work?

questions: See above. 

Minor: 
Line 3: .”..dealt with separately,” -> “ dealt separately” ?

limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: * In this work the authors propose a model that can learn visual input features and segmentation without supervision which combines contrastive learning + Markov Random Fields (MRFs) to implement spatial feature prediction for feature maps + image segmentation.
* Self supervised learning approach: prediction between locations yield self supervised loss to learn feature maps, how to infer which locations should be grouped.
* Gaussian Markov Random Fields are used to model pairwise connections among features given feature maps and binary variables that indicate activity between connections. 
* When considering pairwise connections the four adjacent pixels from the neighbours of position i. The factors of features (point or pairwise) model the strength and influence of a point on its neighbours wrt to a feature. 
* There are two contrastive Learning Objectives defined on the marginal likelihood p(f)  1. Position Loss: optimizes probability of randomly chosen positions from other features + images 2. Factor Loss: maximizes factor for correct feature vectors relative to random feature vectors sampled from different locations and images.  Optimization is done on all parameters of the MRF in parallel via SGD with momentum 
* Segmentation Inference: p(w_i,j = 1 | f) is the probability that two points are connected.  This depends on the two feature vectors f_i ^& f_j.  This is computed by forming a sparse connectivity matrix from: p(w_i,j = 1 | f) / p(w_i,j = 0 | f) (claimed to be an attention dependent serial process)
* Features are trained over three Models: Identity, Linear, and ResNet. Models are trained on MS COCO image dataset. The authors provide an analysis of the learned features for the linear and ResNet model.  Contours are extracted from the model and compared to the Berkeley (human) Segmentation Database (BSDS500).  The authors present recall and precision scores along with other segmentation metrics against a number of other baselines that may have been trained with additional supervision signals ( human contour / segmentation data + pretraining on other tasks). 

Main claims of the work:

1. Predictions across space aid in segmentation and feature learning without further supervision signals (human contour data or pre-training on other tasks/data).
2. Models optimized under this approach have similarities to the human visual system (retinotopic visual cortex)

strengths_and_weaknesses: **Strengths**

* Overall I believe this is a pretty interesting paper although I'm not deeply familiar with all of the ties to cognitive science and the human visual system.  However, I think that the authors do a good job of demonstrating links to the human visual system via the contour & segmentation analysis and the feature visualization without any specific supervision toward human visual cognition.
* The results are clearly demonstrated and show that even with simple feature modeling that spatial feature prediction modeled by the MRF + contrastive losses the model achieves reasonable segmentation and contour prediction accuracy approaching the level of deep neural networks making use of pretraining on other tasks and training on human reported contours.
* In the discussion there are some highlights regarding how this method performs object selection and how temporal data might be used to further improve model capacity to detect object structure (the binding problem).  Also the discussion of biological plausibility helps to underpin the relevance of this work. 
* The scores achieved look fairly good despite relatively little tuning and relatively small models even though optimizing over these metrics wasn't the aim of this research. 

**Weaknesses**

* May need more analysis/explanation on the reasons behind why the factor loss performed well vs. the position loss.
* Sub-section 2.3 may be more appropriately placed under section 3.  Are these meant to be baselines? Finally, Fig. 2 appears fairly early in the text relative to this subsection.  
* The authors stated a number of future directions for this work in section 5 including processing video input.  It would be helpful to know if we could reasonably expect these models to advance state of the art and how this approach fits into the broader family of models tackling image/video segmentation and representation.  
* Figure 2 D & E were a bit tricky to understand and might benefit from some clarification in the description or main text.

questions: 
It was stated that models using the factor loss + neighbourhood of size 20 performed best.  Was there a sense of whether further increases in the neighbourhood would yield further improvement in the scores?  Were any other losses considered?

In section 3.2 it's stated: "... the best performing models seem to be mostly the local averaging models ..." Do you have any intuition or explanation as to why the best performing models were those that do local averaging?

Could this approach induce good representations for image or video data for downstream tasks or other applications?  Was this explored at all?

Would prediction across features in the manner presented be applicable to other input modalities (e.g. language/text) or sensory data (e.g. speech or audio)?

limitations: The authors discuss the limits of their model in the context of low scale and minimal tuning - in effect this is a proof of concept.  They also note that the contrastive losses + learning are not optimal and could be modified to achieve better optimization.  Further the model does include any temporal signal which may also aid in helping to better model probabilistic features.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an unsupervised learning method which learns local features from image pixels both with weights that allow to group those features together.

The contributions are the following :
- a description of their original model based on Markov Random Fields,
- explanations about how to trained the proposed models following principle from contrastive learning,
- a comparison of the learned features under variation of their algorithm (loss, linear/seg),
- a comparison of the performance of their algorithm on a contour detection task (BSD500).
strengths_and_weaknesses: Strengths :
- modelisation of grouping using a binary weight which is on or off whether the feature are grouped or not,
- a method to train a seemingly untrainable model (without supervision),
- multiple variations of their model are compared,
- model comparison with SOTA algorithms,
- a real concern about visual perception

Weaknesses :
- results are not SOTA,
- non-negligible post-processing to extract image contours
- learned features are shown but not the local weights given a feature and a location


questions: 1. The issue with the position loss and the trick to circumvent it are obscure to me. Can you elaborate more on the issue and its solution ? I do not really understand the issue and how the proposed trick solves it.

2. I disagree with the fact that the linear models learn Gabor-like features. In particular for the position loss (Figure 3.A), the features are more like *non-local* wave-like features (Fourier).

3. I would like to see some maps of p_ij or w_ij associated to some features to understand how the algorithm has grouped those.

4. A related question is that the post-processing step to turn p_ij into contours makes hard to understand what was learned ? Is the use of spectral clustering transparent ie has the algorithm learned to group contours over any other type of grouping (eg textures) ?

Few additional references about texture-based segmentation :
- Beck, J., Sutter, A., & Ivry, R. (1987). Spatial frequency channels and perceptual grouping in texture segregation. Computer Vision, Graphics, and Image Processing, 37(2), 299-325.
- Landy, M. S., & Bergen, J. R. (1991). Texture segregation and orientation gradient. Vision research, 31(4), 679-691.
- Wolfson, S. S., & Landy, M. S. (1995). Discrimination of orientation-defined texture edges. Vision research, 35(20), 2863-2877.
- Vacher, J., Launay, C., & Coen-Cagli, R. (2022). Flexibly regularized mixture models and application to image segmentation. Neural Networks, 149, 107-123.

Typos :
l 63 -> which we can us to

limitations: Limitations are sufficiently assessed.
review_title: null

Meta-review: metareview: There was some disagreement on the value of this work. The paper received 1 strong accept, 1 accept and 1 reject. The positive reviewers recommended the paper to be accepted because it proposes a novel unsupervised approach to semantic segmentation and contours detection and because of connections to neuroscience. Some of the main criticisms from the more negative reviewer included a lack of a discussion to related work and a lack of sufficient experimental evaluation (in particular comparisons to related work). 

The AC found the response of the authors in the rebuttal unconvincing. There is quite a bit of prior work using MRF for segmentation (yes prior supervised segmentation work needs to be properly cited and not just in passing in the results section (line 216-217). Wrt a lack lack of baseline comparisons, I also agree with the reviewer and in that respect the statement in the paper (line 236-237) is not convincing ("There are also a few deep neural network models that attempt unsupervised segmentation [e.g. 10, 47, 82], but we were unable to find any that were evaluated on the contour task of BSD500"). It seems that the authors should at least consider running these models on BSD500 or run their models on other datasets. And as acknowledged by one of the more positive reviewers the results are not SOTA. As stated in the discussion, the authors plan to continue tweaking the architecture to improve results. The AC thinks this is needed for this work to make a sufficient contribution to the conference since neither the use of MRFs or contrastive loss are novel on their own -- the burden is on the authors to demonstrate that they can engineer a system from these two ideas with at least competitive results. 

As for the neuroscience contribution, I am quoting the discussion ("the features learned by our models resemble receptive fields in the retina and primary visual cortex and the contours we extract from connectivity information match contours drawn by human subject fairly well, both without any training towards making them more human-like.").  This seems like an unsubstantiated statement since no quantitative analysis is provided. Almost any CNN trained for any reasonable task will return some sort of center-surround, orientation-selective and color-opponency filters (take AlexNet trained on ImageNet for instance) -- so it is unclear what is new here. For this statement to be meaningful that authors should formulate null models and demonstrate empirically that their proposed model are more cortex-like than other reasonable alternative model. 

Overall because of a lack of sufficient technical or neuroscience contribution, the AC recommends this paper to be rejected.
confidence: Certain
Decision: Reject