Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents multiple techniques for training models with different feature priors (i.e. inclinations to focus on different aspects of the training data) and combining them, either post hoc via ensembles or by allowing the models to provide augmented pseudo-labelled training data to each other via co-training. When using simple ensembling techniques, ensembles with a diversity of feature priors are show to perform better than ensembles where the individual models have similar feature priors. Co-training is shown to boost performance substantially when models with diverse feature priors supply pseudo-labels to each other. The problem domain is image classification. The feature priors concern shape and texture. Different preprocessing and or architecture constraint techniques are used for different models so as to predispose them to focus on shape but not texture or vice versa.
main_review: The implementation of shape feature priors (via edge detection preprocessing) and texture feature priors (via limited receptive field via bagnet) makes a lot of sense and does a good job of illustrating a concrete example of a collection of different feature priors.  Some of the co-training experimental results are strong. 

One concern I have is that the ensemble results presented in section 3.2 are generated using very primitive ensembling techniques. Appendix A.4 says the combination techniques were simply max, average and lowest rank. It is more common to treat this kind of ensembling as a 2nd-level machine learning problem with the outputs of the models forming the inputs to a 2nd-level model. I would not have expected a fancy 2nd-level model but I was hoping at a minimum that the first-level models would be combined via. e.g. linear regression on first-level outputs from a held-out validation set. Fancier combinations (e.g. neural nets) are also possible, of course. I would encourage the authors to read a few writeups by winners of Kaggle competitions and/or read about the ensembling done in the Netflix Prize to get a better sense of what constitutes state-of-the-art ensemble combination techniques.

I was also concerned about the assumption made on page 7 that spurious correlations are likely not to exist in unlabelled data, because unlabelled data supposedly comes from a more diverse collection process. While this may be true in some cases, there will also be many real-world situations where all the input data, whether labelled or unlabelled, comes from the same distribution and it may all have the same spurious correlation. It is often the case that a small portion of the data is labelled simply because it is very time intensive for humans to do the labeling but nonetheless, the remaining unlabelled data comes from the same distribution. I hope that in future versions of this work, the authors make clear that practitioners should think hard about whether their unlabelled data will have the same spurious correlations as their labelled data, rather than assuming that this is likely the case.

For these reasons, I can only give the paper a 5. I would prefer that the ensemble section be redone with more sophisticated ensembling and/or removed and I would prefer that the absence-of-spurious-correlation-in-unlabelled data assumption be presented more cautiously. 
 
A minor complaint: from a presentation point of view, it is non-standard and a bit strange to add additional related work in section 6. I don't normally expect to read about related work after the results and towards the end of the paper. Related work is normally presented earlier in a paper. The authors might consider moving this section to an earlier point in the paper. 

On page 8, I found the bolding of +BagNet cotraining results to be a bit confusing. Normally the 'winning' algorithm results are bolded, which in this case is Canny. I realize that the message of the huge boost of cotraining for +BagNet is what is intended but it still confused me that the bolded numbers were not the best numbers.

It also would be nice to show the method on another domain aside from image classification, although I realize space constraints might make this difficult. The authors might consider removing the ensembling section in future versions of the work and instead using that space for cotraining results on another type of problem.

*** Update after author rebuttal *** In light of the addition of stacking experiments for the ensembling, I have raised my score to a 6.
summary_of_the_review: The authors achieve some positive results from cotraining of groups image classification models designed to focus on shape but not texture or vice versa. However, their ensembling results are acheived using very primitive ensembling which is not state of the art. They also overstate the odds that spurious correlations are unlikely to exist in unlabelled data.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is an empirical study of combining multiple feature priors along with some pre-processing to solve a variety of computer vision tasks.
main_review: Positives

+ The study seems to be interesting and maybe useful for practitioners.

Concerns

- Very meagre contribution in terms of technical novelty and framework. 

- Looks like an empirical study without much conviction and direction.

- Experimental evaluation and comparisons seem dated, not state of the art.

- The work is very much below the expected standards of ICLR.
summary_of_the_review: The paper is clearly below par, can be rejected.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a formalized framework for imposing priors on the feature extraction in deep visual processing models. There has been earlier work on encouraging certain feature representations (e.g. suppressing the focus on texture in feature extraction) and also making feature representations robust to domain shift. The core contribution of this paper is the systematic formulation and investigation of how different, distinct feature priors leads to complementary feature representations that can be combined to provide more robust data representations - in other words, creating synthesized multi-view data representations.
The paper ties back to early (1998) work on co-training (which essentially is multi-modal bootstrapping) and ties this to the more recent body of work on self-supervision and self-training.
Experiments are performed with classical shape- and texture-biased models, and show that the hypothesis - that diverse feature priors are able to robustly create a set of complementary data views - holds.
main_review: This paper has a number of strengths, that combined makes me recommend the paper for acceptance:
+ The topic of this paper, creating and combining robust, generalizable and diverse feature representations, is of high relevance to a large portion of the ICLR audience.
+ It provides an interesting and valuable formal framework for steering feature representations in different directions, creating multi-view representations of the data.
+ It is well written, well organized, technically correct, and easy to read.
+ The experimental design is sound and well done.

One weakness can be pointed out, not however any cause for not accepting this paper in my opinion:
- The experiments are performed on old datasets, CIFAR-10 and STL-10, both with quite clear class structure and simplistic image setting (e.g. the object centered in the image). It would be interesting to see experiments on more difficult data with fine-grained and hierarchical class structure for example. 

summary_of_the_review: This paper has a number of strengths, that combined makes me recommend the paper for acceptance: Of high relevance, well written and correct, proposes a valuable framework, and contains sound and well designed experiments.
One weakness can be pointed out, not however any cause for not accepting this paper in my opinion: The experiments are performed on old datasets.

In summary, I propose acceptance for this paper and believe it will be of interest to a large portion of the ICLR audience.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The goal of the paper is to improve model generalisation. The authors consider feature priors as distinct perspectives on the data. The results show that models trained with diverse sets of various feature priors have less overlapping modes and are more efficiently combined.  
main_review: Strengths. The experimental part is relatively clear. 

Weaknesses. The paper is not very clearly written. First, I would appreciate some (even informal) definition of a feature prior. Later in the text, the co-training is mentioned, and it seems that using different priors = co-training using different views. Is it the idea of the paper? I did not really understand the first contribution: "We demonstrate that training models with diverse feature priors results in them making mistakes on different parts of the data distribution, even if their overall accuracy is similar." What is meant?

As far as I understand, there are two "priors" only explored in the paper: shape and texture. 

summary_of_the_review: The current contribution is an exploratory work, combining several state-of-the-art methods (for instance, self-training and co-training are used in the experiments). 
There is a lack of technical novelty.
review_title: null

Meta-review: The manuscript proposes a framework for imposing priors on the feature extraction in deep visual processing models. The core contribution of this manuscript is the systematic formulation and investigation of how different, distinct feature priors leads to complementary feature representations that can be combined to provide more robust data representations. The manuscript uses early work on co-training and also the more recent work on self-supervision and self-training. Experiments are performed with classical shape- and texture-biased models, and show that diverse feature priors are able to robustly create a set of complementary data views.

Positive aspects of the manuscript includes:
1. The topic of this paper, creating and combining robust, generalizable and diverse feature representations, is of high relevance;
2. Positive results from co-training of groups image classification models designed to focus on shape but not texture or vice versa.

There are also several major concerns, including:
1. The ensemble results presented in section 3.2 are generated using very primitive ensembling techniques;
2. The absence-of-spurious-correlation-in-unlabelled data assumption be presented more cautiously;
3. Definition of feature prior;
4. Analysis on another domain aside from image classification.

During the rebuttal period, the Authors provided additional experiments using a more sophisticated method (“stacking”), and additional discussion of where spurious correlations are likely to occur. The manuscript has high rating variance. Some reviewers think that the manuscript lacks the technical novelty, and the results presented are the results of an empirical study. The focus of this manuscript is on two natural feature priors (i.e., shape and texture). It would strengthen the manuscript if the Authors can provide further analysis to emphasise the generality of the proposed framework that it could accommodate any two feature priors as long as they are sufficiently diverse.
Decision: Reject