On the Capability of CNNs to Generalize to Unseen Category-Viewpoint Combinations

Abstract: Object recognition and viewpoint estimation lie at the heart of visual understanding. Recent works suggest that convolutional neural networks (CNNs) fail to generalize to category-viewpoint combinations not seen during training. However, it is unclear when and how such generalization may be possible. Does the number of combinations seen during training impact generalization? What architectures better enable generalization in the multi-task setting of simultaneous category and viewpoint classification? Furthermore, what are the underlying mechanisms that drive the network’s generalization? In this paper, we answer these questions by analyzing state-of-the-art CNNs trained to classify both object category and 3D viewpoint, with quantitative control over the number of category-viewpoint combinations seen during training. We also investigate the emergence of two types of specialized neurons that can explain generalization to unseen combinations—neurons selective to category and invariant to viewpoint, and vice versa. We perform experiments on MNIST extended with position or scale, the iLab dataset with vehicles at different viewpoints, and a challenging new dataset for car model recognition and viewpoint estimation that we introduce in this paper - the Biased-Cars dataset. Our results demonstrate that as the number of combinations seen during training increase, networks generalize better to unseen category-viewpoint combinations, facilitated by an increase in the selectivity and invariance of individual neurons. We find that learning category and viewpoint in separate networks compared to a shared one leads to an increase in selectivity and invariance, as separate networks are not forced to preserve information about both category and viewpoint. This enables separate networks to significantly outperform shared ones at classifying unseen category-viewpoint combinations.

Reviews and author responses:

Strength: This submission provides a novel perspective in understanding the relationship between jointly learned tasks. Recent works such as Standley et al. [1] empirically showed that shared-weight training behavior in a pairwise setting could vary substantially depending on the task. This submission tells us that it also depends on the familiarity of testing examples. Figure 3b shows that, in the usual supervised training setting in which training and testing examples both come from "seen" combinations, shared branch architecture is better. But it is the opposite when evaluated on unfamiliar examples.

Strength: Novel experiments. The finding that _"CNNs generalize better to unseen category-viewpoint combinations as the training data diversity grows"_ by itself is not a surprising one in my opinion; I would expect a nearest-neighbor baseline to behave similarly. But I think the paper's value mainly has to do with the experimental design. The symmetry between the two tasks and quantitative analysis of invariance and selectivity are useful in validating findings from prior work (covered in part by Yang 2019, cited). It would not have been immediately apparent to me how to set up fair experiments because it involves two seemingly unrelated tasks. The 2D grid classification (which uniformly discretizes the task space) and joint geometric mean evaluation made sense.

Weakness: In Figure 3b, it appears that getting better at predicting _unseen_ combinations degrades performance on _seen_ combinations. This is not a weakness on its own, but it's easy to miss, and I think it should have been discussed more. The resnet architecture seems big enough to max out performance on MNIST, so I would expect the biased-cars dataset to present the limitations better as the authors suggested.  As the number of _seen_ combinations increases, there's less of _unseen_ examples to extrapolate. But it also increases the scope of predictions you have to make on seen examples. I'm personally inclined to think there are almost always (hidden or obvious) trade-offs, in one form or another, and some more useful than others. In [1], getting better at one task often comes at the cost of sacrificing performance on another task. [2] also evaluated on unfamiliar categories and unfamiliar viewpoints and found that those two have somewhat mutually exclusive properties.

Another observation is that a _shared branch_ architecture can be computationally the same as a _separate branch_ architecture if you insert zero weights at the correct channels (and had more channels to make up for the sparsity). Although specialized neurons do naturally emerge when learning many tasks (Yang et al.), current supervision methods do not seem to make it easy to learn this. It is still unclear to me why exactly the separate branch architecture works better.

My current rating of the paper is "above threshold". I'm willing to change the rating based on author feedback and discussion.

- [1] ICML 2020, "Which Tasks Should Be Learned Together in Multi-task Learning?"
- [2] CVPR 2018, "Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction"

Post-discussion comment: I'd like to keep my rating (6: Marginally above acceptance threshold). I think R1's concerns regarding similarity between seen and unseen set are valid. As the number of seen combinations increases, the extrapolation problem becomes an interpolation problem. But in my view, that's not a weakness as long as the symmetric grid setting appropriately captures the relationship between the two tasks. I am leaning towards thinking that it does. It eliminates other confounding factors such as the relative weights of the loss terms too, and I think it is fair overall. Hopefully future work will provide more insight. Ultimately I think this submission is above threshold.

This is an analytical study which investigates how good are CNNs in generalizing to new viewpoints. They also investigate whether mixing with an extra task of classifying the viewpoint would help to generalize to new viewpoints or not. 

This study is focused on 3 datasets: MNIST (scale/position), ilab-2m (vehicle toys) and their own novel dataset of photo realistic 5 category car dataset. 

One interesting aspect of their experimental setup is to withhold a set of category/viewpoint pairs for the test set. Essentially for each category the test viewpoint is not seen in the training set. So they are evaluating to what extent can a model generalize from one category to another category. As opposed to withholding a viewpoint from all categories (for example no frontal view for any of the cars during training).
There has been several previous papers that referred to viewpoint generalization as generalization to unseen viewpoints rather than generalization to viewpoints unseen for this category. (mnist-affnist or mnist-mnistROT or smallNorb experiments in viewpoint equivarience papers). 

They also investigate whether classifying the viewpoint jointly can improve viewpoint generalization. Their analysis shows that it hurts the accuracy to add the viewpoint classification task using architectures like ResNet and sharing the backbone. 

Furthermore, this paper studies specialization, selectivity and invariance of neurons to category/viewpoint. 

This paper has an extensive number of experiments on various setups measuring different criterias. For the sheer amount of experimental data present in this study, it is valuable to community and answers some questions. Most significantly it shows that adding a viewpoint predication head hurts viewpoint generalization. This finding is counter intuitive, therefore intriguing and merits further studies in this regard. 
Although the accuracy reported in the experiments is a geometric mean of viewpoint and category. I would be very interested in just comparing the category accuracy. It is valuable to at least have one version that only category accuracy is reported for shared/separate. (train with viewpoint and category classification, for reporting ignore the viewpoint accuracy). To verify whether adding viewpoint classification improves category classification generalization to unseen viewpoints.

Question: 
Can you please clarify whether the images in the test set are unseen instances of the categories as well as unseen viewpoint? i.e. the mnist-scale test split: is it mnist test set images scaled to the unseen scale? As well as for cars, etc.

Can you clarify the choice of unseen viewpoint for a category rather than viewpoint generalization over all categories?

There has been several works addressing the viewpoint generalizability of cnns, including group equivarient convolutional networks and capsule networks. Adding these architectures to this study would verify to what extent the findings here are limited to resnet style architectures. 

The findings regarding separate/shared architecture are interesting, but the significance of the findings and framework of study at its current state is short of a conference standards.


-------------------------------------Post Author Response
Thank you for providing the results and clarifying the setup. I am increasing the score to 6. 

This paper poses the following question: whether or not CNNs that learn to recognize a car from only its frontal view can correctly recognize the same car from a side view that they have never seen before. Previous studies report negative result. I interpret the question as "How would CNNs' learning to recognize various different cars in various different poses generalize to the above case of recognition?" I think it's an intriguing question, and it would be beneficial to know how categories and viewpoints are intertwined in their recognition in CNNs.

The main conclusion is summarized as follows: i) learning with a greater number of category-viewpoint combinations results in higher prediction accuracy for unseen combinations, and ii) separately recognizing category and viewpoint yields more accurate prediction for each. I don't feel these results necessarily deepen our understanding of CNNs regarding the above question.

The first result seems obvious. In the experiments, the authors divide category-viewpoint combinations into a training and a test split without overlap. They then observe how prediction accuracy for the test split (i.e., unseen combinations) changes as the number of combinations in the training split (i.e., seen combinations) increases while its cardinality is kept constant. They conclude that increasing the number of seen combinations improves prediction for the test split. However, I suspect this result can be fully explained because increasing the number of seen combinations results in more samples having closer viewpoints to the test split samples. An identical (category) object tends to have similar appearances when seen from closer viewpoints. It will be easier to recognize the object from viewpoints closer, if not equal, to seen viewpoints.

As for the second result, although the authors state that it is surprising, I don't feel so. The result implies the employed CNN finds little in common between recognition of category and viewpoint, and thus we cannot expect generalization across the two. It is reasonable that there is little or no synergistic effect in multi-task learning of two unrelated tasks. From the standpoint of the above question, it would have been more interesting if recognition of category and viewpoint were not separated but coupled with each other.

UPDATE: 
I read the authors' response. They say "Thus, it is unclear that generalization to unseen viewpoints shown in our results is completely explained by the similarity between the seen and unseen set,..."
I agree with this comment. I don't say we can prove that the similarity fully explains generalization to unseen viewpoints. It's my conjecture. However, the authors themselves seem to recognize my conjecture could be right; at least, it cannot be eliminated. In other words, a major issue with the current manuscript is that we cannot derive a firm conclusion from the experimental results. If my conjecture is true, then the results are almost obvious and not interesting. 
I want to lower the score from 5 to 4, as the authors' response makes me believe my concern is valid.
I think the authors tackle quite a hard problem and admire their efforts, though. 

OVERVIEW:
The authors present an analysis of the capability of CNNs to generalize to unseen category-viewpoint combinations. They study various network architectures (shared, separate and splits in their terminology) for the task of joint category and viewpoint classification. This is done by setting up training-testing data in a specific pattern where some category + viewpoint combinations are held-out and never seen in training data. The question the authors try to answer is what kind of training data diversity is needed to be able to do well on this held-out test set. This done using empirical evidence on the following datasets: (i) MNIST-Position, MNIST-Scale, (ii) iLab-2M and (iii) Biased-Cars dataset (which is a new dataset that they introduce). They define and measure selectivity, invariance and specialization scores for different models with different training data to comment on **when** and **how** CNNs generalize to these unseen combinations. 

PROS:
- The authors tackle an interesting and relevant problem of generalization of CNNs for joint viewpoint and category estimation. In a realistic setting, we will not have enough images of all viewpoints and all categories of interest and a systematic study to roughly quantify how much is needed to be able to successfully predict any viewpoint and category of interest will be helpful.
- I liked the selectivity, invariance and specialization score definitions as quantitative ways of measuring generalizability. 
- The paper is well-written and the ideas/experiments are presented in an easy-to-read manner.
- The experiments are well-designed to validate the authors' hypotheses. Sections 5 and 6, answer the key questions and tie it nicely with experiments and results presented. 

CONS:
- The authors correctly identify a few variations of the network architecture that could be used for the joint viewpoint and category estimation task and study how this affects generalization. However, they are missing a key component of how these networks are trained. Let us consider the overall network to contain three components: (i) stem, (ii) category head and (iii) viewpoint head. Say we train the category estimation task first (stem + category head). The features learned by the stem will be viewpoint invariant because the category estimation task requires viewpoint invariance. If we not train the viewpoint head on top of such features, it is going to suffer. The other two possibilities are (a) train the viewpoint estimation task first and then category estimation and (b) train the viewpoint and category estimation task jointly (which is what the authors have done). The viewpoint and category tasks compete compared to other join tasks like detection + categorization and so training protocol might make a significant difference to network performance. 
- The authors are missing some references of works that do joint category and viewpoint estimation:
  - Elhoseny et al, "A Comparative Analysis and Study of Multiview CNN Models for Joint Object Categorization and Pose Estimation", ICML 2016. [link](http://proceedings.mlr.press/v48/elhoseiny16.pdf)
  - Afifi et al, "Simultaneous Object Classification and Viewpoint Estimation using Deep Multi-task Convolutional Neural Network", VISIGRAPP 2018. [link](https://researchoutput.csu.edu.au/ws/portalfiles/portal/23232236/23231703_Published_paper.pdf)
  - Mahendran et al, "Convolutional Networks for Object Category and 3D Pose Estimation from 2D Images", ECCVW 2018. [link](https://openaccess.thecvf.com/content_ECCVW_2018/papers/11129/Mahendran_Convolutional_Networks_for_Object_Category_and_3D_Pose_Estimation_from_ECCVW_2018_paper.pdf)
- In Mahendran et al, some more network architectures are considered specifically ones that combine the category and viewpoint estimation tasks at the prediction stage along with the feature stage. Including it in your analysis as well will be helpful.
- The new proposed dataset of Biased-Cars is synthetic even if useful. Why not use a real dataset like UIUC Dataset of 3D object categories? [paper](http://vision.stanford.edu/documents/SavareseFei-Fei_ICCV2007.pdf), [link](http://www.eecs.umich.edu/vision/data/3Ddataset.zip). Demonstration that the conclusions you make on Biased-Cars carry to actual cars will provide stronger evidence for your claims/findings. 
- For the MNIST experiments, a rotated MNIST [link](https://sites.google.com/a/lisa.iro.umontreal.ca/public_static_twiki/variations-on-the-mnist-digits) would be a better choice compared to MNIST-position and MNIST-scale. 

REASON FOR RATING:
I like the idea and analysis present in the paper for generalization to unseen category-viewpoint combinations. At the same time, I think training protocol plays a key role in such multi-task networks and is missing from current analysis. Adding that (or at the very least commenting about it) and including results on the UIUC 3D Object dataset will help me go from borderline to yes. 

UPDATE:
I have read the author feedback and other reviews & discussions. I have updated my rating from 6 to 7. The authors responded to my comments with updated analysis to further prove their claims.

We thank all the reviewers for their constructive feedback. We have addressed all the concerns raised in the reviews, and have run new experiments that validate our results with additional datasets, architectures and training protocols as requested by the reviewers. We are looking forward to hearing back if there are any concerns that have not been addressed, so that we can provide more evidence before the deadline.

Thank you for the detailed review and your comments.

***Comment 1: Performance improves because samples become more similar to test set***

Note that the similarity between the seen and unseen set mentioned in the review is with respect to human visual perception; such similarity does not necessarily translate to similar activations or predictions by CNNs. This is because CNNs do not guarantee generalization to unseen viewpoints (as discussed in Sec. 5). In fact, a tiny change in the viewpoint can lead to a big difference in object classification accuracy as reported extensively in previous work (cited in our paper: Engstrom et al., 2019; Azulay & Weiss, 2019; Srivastava et al., 2019). Moreover, adversarial examples show that similar images (for humans) can lead to very different object classifications. Thus, it is unclear that generalization to unseen viewpoints shown in our results is completely explained by the similarity between the seen and unseen set, as the network could have easily overfit to the seen viewpoints.

***Comment 2: Shared does not do well due to lack of synergy between category and pose***

If tasks lacked synergy, shared architectures would suffer for both seen and unseen combinations, but our results show this is not true. See figure 3 in the main paper and figures D7, D8, and D9 in Supplement: Shared is worse than separate only for unseen combinations but not for the seen combinations. Thus, these results can not be explained by synergy between tasks. Our proposed specialization score offers an explanation for both seen and unseen cases.

Thank you for the detailed comments and suggestions.

***Comment 1: Trade-off between seen and unseen***

We agree our results indicate a trade-off between performance on seen combinations and extrapolating to unseen combinations. We have added a discussion on this to Section 5 (copied below):

Also, note that as seen combinations increase, the task becomes more challenging as images with each category and pose become more diverse, leading to some drop in accuracy on seen combinations. This suggests an inherent trade-off between getting better on seen combinations and extrapolating to unseen combinations, which is impacted by training data diversity. This result adds to a growing body of works investigating the trade-offs inherent to multi-task learning (Standley et al., 2019; Shin et al., 2018). 

***Comment 2: Shared architecture is computationally the same as separate***

This is correct as long as the number of neurons is the same. By controlling the number of neurons per layer in “Shared-Wide”, we ensure that this architecture has the capability to instantiate the same solutions as “Separate”. While these two architectures can instantiate the same solutions, training with bias leads to different solutions with different degrees of specialization and generalization. Regarding improving current methods of supervision, we are currently exploring how to enforce invariance in order to improve specialization and generalization. 

***Thank you for bringing these references to our attention, we have added them to the paper.***

Thank you for your detailed comments and suggested experiments.

***Question 1: Accuracy of only category prediction***
These results were provided in Supplement section D.3, as mentioned in Section 5 in the main paper.

***Question 2: Unseen images being unseen instances of categories as well as unseen viewpoints***
Yes, the MNIST-Scale test data was created by taking images from the standard MNIST test set and scaling them to unseen scales. Similarly, for MNIST-Position, we placed MNIST test images in unseen positions. For all datasets (including iLab and Biased-Cars), the unseen test set consists of unseen instances of categories at unseen viewpoints. 

***Question 3: Choice of unseen viewpoint for one category rather than all categories***
We provided these results in Supplement section D.4. In Fig D6 (b), the unseen viewpoint is held out from all categories. We also offer similar results for unseen cars held out from all viewpoints (Fig. D6 (a)).

***Experiment 1: Results on group equivariant CNNs (G-CNNs).***
We have added analysis on two group equivariant CNNs, namely Spherical CNNs and Icosahedral CNNs in Sec. D7. As shown in Fig. D9 (a) and (b), our findings hold for G-CNN architectures as well. Anonymous link: https://imgur.com/a/hoYGZra 

***Experiment 2: Results on MNIST Rotation.***
We have added these experiments to Supplement Sec. D6. As shown in Fig D8 (b), our findings hold for this dataset as well. Anonymous link: https://imgur.com/a/4PewhX2.


Thank you for the detailed review and suggested experiments.

***Comment 1: Training protocol variations - first training category and then rotation + vice-versa***

We have added these two new training protocols for the ResNet backbone to Supplement Sec. D5. As shown in Fig. D7, our findings are consistent for these training protocols as well. Anonymous link: https://imgur.com/a/sotp8dh

It is important to note that the symmetric treatment of the two tasks is an important control in our analysis. As the category dependent viewpoint architectures described in Mahendran et al. do not treat the two tasks symmetrically, we leave these architectures for future work.

***Comment 2: Missing References***
We have added these references to the paper. Thank you for bringing them to our attention.

***Comment 3: UIUC Dataset***
We have added these experiments to Supplement Sec. D6. As shown in Fig D8 (a), our findings hold for this dataset as well. Anonymous link: https://imgur.com/a/AacW0oT. Note that the generalization numbers are lower than other datasets due to this small size dataset, but the findings still hold.

***Comment 4 Rotated MNIST***
We have added these experiments to Supplement Sec. D6. As shown in Fig D8 (b), our findings hold for this dataset as well. Anonymous link: https://imgur.com/a/4PewhX2.

Meta-review: This paper received 2 borderline accepts, 1 accept, and 1 reject.

In general, there is broad agreement that this is solid experimental work and that the differences found between recognition and viewpoint estimation were interesting.

The main issue brought up by the more negative reviewer is that some of the experiments are subject to interpretation (specifically the extrapolation problem could become more of an interpolation problem as the number of training examples increases). This issue is acknowledged by the other reviewers who nonetheless see some value in the paper being published and potentially paving the way for additional studies. My suggestion for the authors is to prominently discuss this issue in a revision of this paper. Unfortunately, because of space constraints,  I have to recommend this paper be rejected.

Decision: withdrawn-rejected-submissions