Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a model that learns both features and segmentation without supervision. The method combines contrastive learning, a Markov random field model for the feature maps and segmentation. Learning utilizes two losses; position loss which optimizes the probability of the feature vector at each location relative to the probability of randomly chosen other feature vectors from different locations and images and factor loss, which maximizes factors for the correct feature vectors relative to random pairs of feature vectors sampled from different locations and images. The goal of the research is to evaluate, if it is possible to learn human-like features and segmentations.
strength_and_weaknesses: The paper is very well written and scientifically and technically sound. The method includes novelty, it has been carefully discussed and evaluated. However, the relevance of the paper was left a bit unclear. Unsupervised segmentation methods exist, why is it relevant to learn the features simultaneously, or wasn't this the main contribution here?
clarity,_quality,_novelty_and_reproducibility: The paper is very clear and easy to follow, the method has been reported in details and conclusions are drawn from various viewpoints. However, the relevance of the contribution is left quite open, was the main goal to be able to analyze the similarities of computer vision perception with human's, or was the main point to achieve simultanous segmentation and feature extraction?
summary_of_the_review: Very interesting and well written paper with novel method, however, the relevance of the developed method was left a bit unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Markov random field model that learns both features and connectivity (whether two locations belong to the same segment), in an unsupervised manner using contrastive learning.  The trained model is able to extract contours, and gives good performance on the Berkeley segmentation database.  
strength_and_weaknesses: Strengths:

+ Empirical performance - unsupervised method for contour extraction gives good performance on the Berkeley database, comparable to or better than hand-crafted methods.

Weaknesses:

- Extent of empirical validation - only studied on one benchmark, did not compare against existing deep neural networks for unsupervised segmentation (as opposed to contour extraction), would have been nice to get some sense of a comparison even if some simplifying assumptions had to be made.

- It's strange to me that the pixel-based model performs well, relative to the other proposed models, and also that the best performing model seems to be Predseg1-Layer 0, which is, as noted, only using 3x3 local color averages as features.  This seems to suggest that the model is good at connectivity/contour extraction, but not necessarily in learning good features.
clarity,_quality,_novelty_and_reproducibility: The main novelty of the proposed method seems to be in the joint unsupervised learning of feature maps and local segmentation information.  However, a counter-point to the usefulness of this is the fact that the best empirical results on the contour extraction task, as noted above, either are just using the pixel rgb values as features, or doing simple 3x3 averaging.  This seems to suggest that, within this model, learning non-trivial/complex feature maps is not needed or in some way detrimental to learning the segmentation.  
summary_of_the_review: While the proposed method has some novelty and interesting results on unsupervised contour extraction, given the concerns above (would like more empirical validation, some strangeness in the ordering of the model performance, in particular the good performance of the simpler feature map models, and the poor performance of Predseg1-Layer 1), I would lean toward reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a pairwised Markov random field model to learn both features and segmentation from images without further supervision signals. They showed that the features learned by the shallow neural networks based on the contrastive learning loss are local averages, opponent colors and Gabor-like stripe patterns as observed in early visual cortex. They can also inder connectivity between locations by inferring the switch variables, and contours inferred from such connectivty perform well on a benchmark.
strength_and_weaknesses: Strength:
The idea behind is interesting and the authors did a lot of careful empirical study to verify it. 

Weaknesses:
1, The introduction is not convincing enough. It is good to do a broad review of visual processing, but it is hard to get the point of what the authors want to claim in the introduction (except the last paragraph). A lot of stuff in the introduction seems like unnecessary. It is better to say more words about the model in the introduction. 
2, It is unclear how the proposed model contribute to the computer vision community. Can this method outperform current deep-learning based methods for image feature extraction and segmentation?
3, Many of the implementation details can be put into Appendix.
4, The authors mentioned "human-like" features at several place in the paper. However, what is human-like features? Gabor-like patterns are first found on cats' visual cotex. It is not enough to say it is human like from Fig2, Fig3A  and Table1


clarity,_quality,_novelty_and_reproducibility: The writting is not clear, and some of the modeling parts seem like have been studied previously.
summary_of_the_review: The novelty and contribution are not enough. It is hard to know what the authors want to say in some parts of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a local model for boundary detection. At the core of the method, it assumes a local Gaussian Markov model in the feature space, where the features can be jointly learnt or derived from a neural network. The model uses a contrastive learning scheme to optimize features and pixel location connectivity, which gives the final contour prediction results.
strength_and_weaknesses: ---
### strength
If everything holds up, the strength of this paper is its novelty. It shows that even with local gaussian Markov models, under the proposed contrastive learning goal, one can recover meaningful boundary maps by looking at the pixel connectivity matrix. (Spectral clustering is needed to generate segments first and use edge detectors for contours.)

---
### Weaknesses
This paper is poorly written with many missing pieces and handwavy arguments.

1. The learning objective.
In the loss section, only $\psi_{ij}$ appears in both terms, so why $p_{ij}$ is being optimized as well? $\psi_{ij}$ is just a gaussian error term between $f_i$ and $f_j$, with optimizable diagonal variance $C_{ij}$. How is the related to the connectivity $w_{ij}$? Also please avoid using $p_{ij}$ because you are also using $p(\cdot)$ as probability.

I would highly recommend the author rewrite the method section in a way that the learning objective is obvious. According to the formulation in the text, the final prediction result is $p(w | f)$, then how is this target linked with the proposed objective? This needs a more detailed derivation, and since this is the core of the proposed method, it should be written as a formal theorem where you start from the contrastive objectives and ends with a MLE or approximation of $p(w | f)$. 

Right now, from the current version of the manuscript, I can only make an educated guess that after optimizing with the contrastive feature learning goal, a MLE estimation of $p(w|f)$ is performed, given the learnt feature and covariance parameters. I don't think this is what happens because the authors mentioned in the text that $p_ij$ is also being optimized, which never appears. I suspect the learning goal is wrong, where you need the $p_ij$ terms as well, otherwise you are approximating some distribution (not stated clearly) with a different Markov field than the one stated in Eq2.

2.  Assuming it should not be just $\psi_{ij}$ in the loss term, but with $p_ij$ as well, the authors should make it explicit about what exactly are they approximating using the similar formulation of NCE, with equations as proof. I find handwavy arguments like: 

> we do not optimize the Markov random field
directly, but optimize predictions based on the model using features from other locations as the noise
distribution. For this noise distribution, the factors that depend only on a single location (the first
product in (1)) will cancel. 

very vague and hard to follow. What do you mean by optimizing a markov field? Optimizing for an MLE estimate? What are the parameters? MLE of what distribution? What noise distribution? Why did it cancel out? they really should be equations in addition to words.


3. There's weird organization issues and formulation issues. Starting from Eq.4 to Sec 2.1, the derivation of normalization factor is for what? According to text in the later section, it seems you need this for some logit calculation. But it's funny to say that something is a proper distribution after deriving normalization factor: normalization factor is exactly what makes it a proper distribution, by definition. A better way to formulate this is to just let the pair-wise potential (factor) to be $f(w_ij, f_i, f_j)$, and define this function to be the terms inside the prod operator. for each individual factor, p(w_ij=1) is p_ij by definition, but not the marginal distribution on the markov field, a simple fact . The authors make statements about this in a weird way, but it's just a simple distinction between marginal distribution vs local definition of the distribution type. I'm still confused about why stressing this point though; are you approximating the marginal distribution P(w_{ij}) somewhere?

Because of the conflicting arguments about optimizing $p_ij$ and not having it in the objective, I can't make sense of what the optimization is about. Even if $p_ij$ is included, it's still not obvious to see what distribution approximation the authors are making, and how to get a point estimate of the connectivity in the end (is it MLE over some distribution? ).

clarity,_quality,_novelty_and_reproducibility: ---
#### Clarity
This paper lacks clarity. It is hard to follow and requires a reader to guess that the author is trying to say. 

---
#### The quality of the results is acceptable, but the paper writing is preventing the readers from understanding what's actually going on.

--- 
#### The authors did provide full training and evaluation code in the submission. I believe the method/results are reproducible.
summary_of_the_review: Given the concerns above, I vote to reject this paper in its current state. The author needs to improve clarity and make sure the reader can understand what's really going on with minimum guessing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
The paper proposes a model that learns both features and segmentation without supervision. Markov random field is proposed to learn both features and connectivity (whether two locations belong to the same segment), in an unsupervised manner using contrastive learning. The trained model is able to extract contours, and gives good performance on the Berkeley segmentation database.
# Strengths:
- The strength of this paper is its novelty. It shows that even with local gaussian Markov models, under the proposed contrastive learning goal, one can recover meaningful boundary maps by looking at the pixel connectivity matrix. 
- Empirical performance - unsupervised method for contour extraction gives good performance on the Berkeley database, comparable to or better than hand-crafted methods.
# Weaknesses:
- This paper is poorly written with many missing pieces and handwavy arguments.
- Extent of empirical validation - only studied on one benchmark, did not compare against existing deep neural networks for unsupervised segmentation (as opposed to contour extraction), would have been nice to get some sense of a comparison even if some simplifying assumptions had to be made.
-  The authors mentioned "human-like" features at several place in the paper. However, what is human-like features? Gabor-like patterns are first found on cats' visual cotex. It is not enough to say it is human like from Fig2, Fig3A and Table1

justification_for_why_not_higher_score: Based on the recommendation of the reviewers and the absence of author feedback, all reviewers agreed that the paper tackles an important problem but there are many weaknesses in order to accept the paper.
Decision: Reject