Title: Training Group Orthogonal Neural Networks with Privileged Information
Authors: Yunpeng Chen
Xiaojie Jin
Jiashi Feng
Shuicheng Yan
Venue: ICLR 2017
Keywords: Deep learning
Computer vision
Supervised Learning

Abstract: Learning rich and diverse feature representation are always desired for deep convolutional neural networks (CNNs). Besides, when auxiliary annotations are available for specific data, simply ignoring them would be a great waste. In this paper, we incorporate these auxiliary annotations as privileged information and propose a novel CNN  model that is able to maximize inherent diversity of a CNN model such that the model can learn better feature representation with a stronger generalization ability. More specifically, we propose a group orthogonal convolutional neural network (GoCNN) to learn features from foreground and background in an orthogonal way by exploiting privileged information for optimization, which automatically emphasizes feature diversity within a single model. Experiments on two benchmark datasets, ImageNet and PASCAL VOC, well demonstrate the effectiveness and high generalization ability of our proposed GoCNN models.

TL;DR: A convolutional neural network for image classification which encourages learning more diverse feature representations by using image segmentations as privileged information.

Review discussion:
Could you please discuss the relationship between your method and the "DeCov" method presented in https://arxiv.org/abs/1511.06068? How would your approach compare to this baseline? Thanks.
"DeCov" and our "GoCNN" share similar motivation: more diverse learned features have stronger generalization ability;
thus both of them emphasize and aim to explicitly pursuit such feature diversity.
However, our "GoCNN" proposes a different approach to optimize diversity compared with "DeCov". Concretely, "DeCov" penalizes the covariance in an unsupervised fashion and cannot utilize extra available annotations, leading to insignificant performance improvement over vanilla models.
For instance, the performance enhancement brought by "DeCov"(NiN) is quite marginal on ImageNet dataset compared with "Dropout"(NiN), see Table 5 (val top1/top5) in DeCov paper.
When they intentionally increase the model complexity (i.e., NiN->AlexNet) with greater overfitting risk, the "DeCov" method performs even worse than a simple baseline method ("Dropout"), see Fig. 5 (left; val accuracy) in the DeCov paper.
In contrast, "GoCNN" regularizes the covariance (such that increases feature independency and diversity) in a supervised way.
Thus "GoCNN" can fully utilize rich extra information (segmentation masks here). Benefitting from this, "GoCNN" provides significant performance boost on ImageNet dataset even with a more complex model (i.e. ResNet-50), compared with AlexNet.
Thanks for pointing out this related work. We would add it into the related works.
The use of the background "suppression term" (eq 3) in the objective is not intuitive to me.
While I understand that the foreground object should be more informative than the background for the object classification task, why penalize the model for using the background when it is informative?  Do you have a baseline result that ablates this term and/or the group-wise correlation term?
Thanks for your comments.
Sec. 4.1 explains the way GoCNN learns group orthogonal features.
- Both Eq.2 and Eq.3 are designed ONLY for the FOREGROUND group (not the whole model). They only penalize the FOREGROUND group from using background information.
- GoCNN does utilize background information when it is informative, and the background features are extracted by the BACKGROUND group which also corresponds to two similar terms as Eq.2 and Eq.3, but with an inverted Mask (check the last paragraph in Sec.4.1 for more details).
In the experiment section, we visualized the learnt feature vectors on Val dataset and performed controlled experiments. As can be seen in Fig.2, the features within the foreground group are well orthogonal with features in the background group, Fig2(b) v.s. Fig2(c);
Also, richer features are extracted in GoCNN compared with the baseline, Fig2(d) v.s. Fig2(e). The terms introduced in Sec.4.1 play a crucial role in achieving this. (Please check Sec 5.2 for more detailed analysis and deeper discussions.)
The starting point of this work is the understanding that by having decorrelated neurons (e.g. neurons that only fire on background, or only on foreground regions) one provides independent pieces of information to the subsequent decisions.
As such one gives "complementary viewpoints" of the input to the subsequent layers, which can be thought of as performing ensembling/expert combination within the model, rather than using an ensemble of networks.
For this, the authors propose a sensible method to decorrelate the activations of intermediate neurons, with the aim of delivering complementary inputs to the final classification layers: they split intermediate neurons to a "foreground" and a "background" subset, and append side-losses that force them to be zero on background and foreground pixels respectively.
They demonstrate that this can improve classification on a mid-scale classification example (a fraction of imagenet, and a ResNet with 18, rather than 150 layers), when compared to a "vanilla" baseline that does not use these losses.
I enjoyed reading the paper because the idea is simple, smart, and seems to be effective.
But there are a few concerns;
-firstly, the way of doing this seems very particular to vision. In vision one knows that masking the features (during both training and testing) helps, e.g. https://arxiv.org/abs/1412.1283
To be fair, this is not truly the same thing as what the authors are doing, because in the reference above the masking is computed  during both training and testing, while here it is used as a method of decorrelating neurons at training time.
But I understand that to the broader iclr community this may seem as "yet another vision-specific trick", while to the vision community one would ask why not just use the mask during both training and testing, since one can compute it in the first place.
More importantly, the evaluation is quite limited; the authors use only one network (18 rather than 150 layers) and only part of imagenet for testing. They do get a substantial boost, but it is not clear if this will transfer to more data/layers.
The authors could at least have also tried CIFAR-10/100. I would expect to see some more results during the rebuttal period.
Thanks for your comments.
Q: The evaluation is quite limited; the authors use only one network (18 rather than 150 layers) and only part of ImageNet for testing. They do get a substantial boost, but it is not clear if this will transfer to more data/layers.
A: We have added new experiments on the full ImageNet-1k dataset where only 10% privileged information is provided and we used 152-layer ResNet as the basic model. In the experiments, our proposed GoCNN model achieves 21.8% top-1 error while the top-1 error of the vanilla ResNet-152 is 23.0%. Such performance boost is consistent with the results shown in Table 4., which again confirms the effectiveness of the GoCNN.
Q: The authors could at least have also tried CIFAR-10/100.
A: Because CIFAR dataset does not provide segmentation masks for the images and any other type of privileged information, we are not able to conduct such experiment.
However, ImageNet-1k is much larger and more diverse than CIFAR-10/100. We believe the results on ImageNet-1k are convincing.
This paper proposes a modification to ConvNet training so that the feature activations before the linear classifier are divided into groups such that all pairs of features across all pairs of groups are encouraged to have low statistical correlation.
Instead of discovering the groups automatically, the work proposes to use supervision, which they call privileged information, to assign features to groups in a hand-coded fashion. The developed method is applied to image classification.
Pros:
- The paper is clear and easy to follow
- The experimental results seem to show some benefit from the proposed approach
Cons:
(1) The paper proposes one core idea (group orthogonality w/ privileged information), but then introduces background feature suppression without much motivation and without careful experimentation
(2) No comparison with an ensemble
(3) Full experiments on ImageNet under the "partial privileged information" setting would be more impactful
This paper is promising and I would be willing to accept an improved version.
However, the current version lacks focus and clean experiments.
First, the abstract and intro focus on the need to replace ensembles with a single model that has diverse (ensemble like) features. The hope is that such a model will have the same boost in accuracy, while requiring fewer FLOPs and less memory. Based on this introduction, I expect the rest of the paper to focus on this point.
But it does not; there are no experimental results on ensembles and no experimental evidence that the proposed approach in able to avoid the speed and memory cost of ensembles while also retaining the accuracy benefit.
Second, the technical contribution of the paper is presented as group orthogonality (GO).
However, in Sec 4.1 the idea of background feature suppression is introduced.
While some motivation for it is given, the motivation does not tie into GO. GO does not require bg suppression and the introduction of it seems ad hoc.
Moreover, the experiments never decouple GO and bg suppression, so we are unable to understand how GO works on its own. This is a critical experimental flaw in my reading.
Minor suggestions / comments:
- The equation in definition 2 has an incorrect normalizing factor (1/c^(k)^2)
- Figure 1 seems to have incorrect mask placements. The top mask is one that will mask out the background and only allow the fg to pass
Thanks for your comments.
Q:
(1) The paper proposes one core idea (group orthogonality w/ privileged information), but then introduces background feature suppression without much motivation and without careful experimentation
(2) No comparison with an ensemble
A:
We feel the introduction section and the abstract are not well organized. We have rewritten these two sections and added more explanation (on the motivation) and comparison. We also highlight the difference from ensemble models in the related works. Please check our revised version.
Q:
(3) Full experiments on ImageNet under the "partial privileged information" setting would be more impactful.
A:
Thanks for the suggestion. We added a complementary experiment on the FULL ImageNet-1k dataset where only 10% privileged information is provided and we used 152-layer ResNet as the basic model. In the experiments, our proposed GoCNN model achieves 21.8% top-1 error while the top-1 error of the vanilla ResNet-152 is 23.0%. Such performance boost is consistent with the results shown in Table 4., which again confirms the effectiveness of the GoCNN.
Q:
Figure 1 seems to have incorrect mask placements. The top mask is one that will mask out the background and only allow the fg to pass
A:
We found that the Fig 1. is a bit confusing and leads to misunderstanding for the reviewer. We have already updated it in the revised revision. In GoCNN, the feature extractor only serves for the suppression and the extracted features are not used by the image classifier. More specifically, since there is no such mask to separate foreground from background feature during the testing phase, the CNN has to learn to separate them by itself through pursuing group orthogonality in the training.
Therefore, the classifier cannot directly use the masked features for training.
To guide the CNNs to learn group orthogonal features, we used the suppression term to penalize the contaminated parts if there is any. In this way, the suppression term guides the CNN to separate the foreground and background feature space.
Therefore, it is not redundant. In contrary, it is a very important part.
Q:
The equation in definition 2 has an incorrect normalizing factor (1/c^(k)^2)
A:
Yes, the normalization factor should be equal to the combinatorial number of the different groups multiplied by the number of paired features within two groups. The expression would be a bit complicated. Here we use the upper bound value c^{(k)}^2 that is sufficient to make sure the summation term to be smaller than 1.
This paper proposes to learn groups of orthogonal features in a convnet by penalizing correlation among features in each group.  The technique is applied in the setting of image classification with “privileged information” in the form of foreground segmentation masks, where the model is trained to learn orthogonal groups of foreground and background features using the correlation penalty and an additional “background suppression” term.
Pros:
Proposes a “group-wise model diversity” loss term which is novel, to my knowledge.
The use of foreground segmentation masks to improve image classification is also novel.
The method is evaluated on two standard and relatively large-scale vision datasets: ImageNet and PASCAL VOC 2012.
Cons:
The evaluation is lacking.  There should be a baseline that leaves out the background suppression term, so readers know how much that term is contributing to the performance vs. the group orthogonal term.  The use of the background suppression term is also confusing to me -- it seems redundant, as the group orthogonality term should already serve to suppress the use of background features by the foreground feature extractor.
It would be nice to see the results with “Incomplete Privileged Information” on the full ImageNet dataset (rather than just 10% of it) with the privileged information included for the 10% of images where it’s available.  This would verify that the method and use of segmentation masks remains useful even in the regime of more labeled classification data.
The presentation overall is a bit confusing and difficult to follow, for me.
For example, Section 4.2 is titled “A Unified Architecture: GoCNN”, yet it is not an overview of the method as a whole, but a list of specific implementation details (even the very first sentence).
Minor: calling eq 3 a “regression loss” and writing “||0 - x||” rather than just “||x||” is not necessary and makes understanding more difficult -- I’ve never seen a norm regularization term written this way or described as a “regression to 0”.
Minor: in fig. 1 I think the FG and BG suppression labels are swapped: e.g., the “suppress foreground” mask has 1s in the FG and 0s in the BG (which would suppress the BG, not the FG).
An additional question: why are the results in Table 4 with 100% privileged information different from those in Table 1-2?  Are these not the same setting?
The ideas presented in this paper are novel and show some promise, but are currently not sufficiently ablated for readers to understand what aspects of the method are important.
Besides additional experiments, the paper could also use some reorganization and revision for clarity.
Edit (1/29/17): after considering the latest revisions -- particularly the full ImageNet evaluation results reported in Table 5 demonstrating that the background segmentation 'privileged information' is beneficial even with the full labeled ImageNet dataset -- I've upgraded my rating from 4 to 6.
(I'll reiterate a very minor point about Figure 1 though: I still think the "0" and "1" labels in the top part of the figures should be swapped to match the other labels.  e.g., the topmost path in figure 1a, with the text "suppress foreground", currently has 0 in the background and 1 in the foreground, when one would want the reverse of this to suppress the foreground.)
Thanks for your comments.
Q:
- The evaluation is lacking. There should be a baseline that leaves out the background suppression term, so readers know how much that term is contributing to the performance vs. the group orthogonal term.
- The use of the background suppression term is also confusing to me -- it seems redundant, as the group orthogonality term should already serve to suppress the use of background features by the foreground feature extractor.
- Minor: in fig. 1 I think the FG and BG suppression labels are swapped: e.g., the “suppress foreground” mask has 1s in the FG and 0s in the BG (which would suppress the BG, not the FG).
A:
We found that the Fig 1. is a bit confusing and leads to misunderstanding for the reviewer. We have already updated it in the revised revision. In GoCNN, the feature extractor only serves for the suppression and the extracted features are not used by the image classifier. More specifically, since there is no such mask to separate foreground from background feature during the testing phase, the CNN has to learn to separate them by itself through pursuing group orthogonality in the training.
Therefore, the classifier cannot directly use the masked features for training.
To guide the CNNs to learn group orthogonal features, we used the suppression term to penalize the contaminated parts if there is any. In this way, the suppression term guides the CNN to separate the foreground and background feature space.
Therefore, it is not redundant. In contrary, it is a very important part.
Q:
It would be nice to see the results with “Incomplete Privileged Information” on the full ImageNet dataset (rather than just 10% of it) with the privileged information included for the 10% of images where it’s available. This would verify that the method and use of segmentation masks remains useful even in the regime of more labeled classification data.
A:
Thanks for the suggestion. We added a complementary experiment on the FULL ImageNet-1k dataset where only 10% privileged information is provided and we used 152-layer ResNet as the basic model. In the experiments, our proposed GoCNN model achieves 21.8% top-1 error while the top-1 error of the vanilla ResNet-152 is 23.0%. Such performance boost is consistent with the results shown in Table 4., which again confirms the effectiveness of the GoCNN.
Q:
The presentation overall is a bit confusing and difficult to follow, for me.
For example, Section 4.2 is titled “A Unified Architecture: GoCNN”, yet it is not an overview of the method as a whole, but a list of specific implementation details (even the very first sentence).
A:
Thanks for the suggestion. We change the title to “Architecture and Implementation Details of The GoCNN”.
Q:
Minor: calling eq 3 a “regression loss” and writing “||0 - x||” rather than just “||x||” is not necessary and makes understanding more difficult -- I’ve never seen a norm regularization term written this way or described as a “regression to 0”.
A:
Eqn 3. has been corrected. Thank you for the suggestion.
Q:
An additional question: why are the results in Table 4 with 100% privileged information different from those in Table 1-2? Are these not the same setting? The ideas presented in this paper are novel and show some promise, but are currently not sufficiently ablated for readers to understand what aspects of the method are important.
A:
Table 1-2 shows validation result by using 10 crops in the testing, while Table 4 with 100% privileged information only shows the result using 1 centre crop in the testing. Since 10-crop testing is time-consuming, and the main target of the experiments is to verify the effectiveness of GoCNN compared with baseline method, results under the 1-crop setting are convincing enough to demonstrate it.
See edit at the bottom of review: I've upgraded my rating in light of the latest revisions.
We've already updated the paper.
- The abstract and introduction have been rewritten with more explanation (on the motivation) and comparison.
- The difference from ensemble models was highlighted in the related works.
- We found that the Fig 1. is a bit confusing and have already updated it in the revised revision.
- Eqn 3. has been corrected.
- New results on ImageNet dataset.

Meta-review:
role: PC
data: comment: This paper was reviewed by three experts. While they find interesting ideas in the manuscript, all three point to deficiencies (lack of clean experiments, clarity in the manuscript, etc) and recommend rejection. I believe there are promising ideas here, and this manuscript will be stronger for a future deadline.

Decision: Reject