Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The last layer of a neural network trained to classify instances usually linearly projects the feature vector from the previous layer to compute log odds of a class. This paper proposes replacement of that linear projection with a norm of the projection of the feature vector into a subspace. The paper shows how to optimize all weights of such a network and studies the advantages of the representation. The authors argue that the learned features transfer better in downstream tasks.
strength_and_weaknesses: The strengths: 
The paper is clearly written and well-motivated. The method is sufficiently described for a reader's implementation. The experiments are diverse: Beyond testing accuracy on ImageNet, the paper also studies feature transfer to new datasets as well as learning from small number of examples.

The weaknesses: Some of the improvements demonstrated in experiments are fairly small (e.g. Table 1). In fact, given the variability in numbers one might get in all the experiments through small tinkering with known tricks, it is difficult to know if the demonstrated advantages would hold under further optimization for any given task (and if they are statistically significant across initializations in learning), although experiments and illustrations as a whole do paint a convincing picture. Some of the choices of the implementation are not fully explained.

Overall, my first impression is positive.

Questions:
1. Given the need for step 5 in the algorithm (orthogonalization) for numerical stability, why bother with the geodesic at all and simply do gradient descent on S followed by step 5?  Are there some illustrations of why it fails?
2. Just looking at Eq. 6, one might ask if the orthogonal subspace basis and Grassman manifold view is really necessary, or if the benefit simply comes from the quadratic form of the logit computation (instead of linear). I.e., going beyond the previous question: Can the optimization be simply done on unconstrained S_i? Or for that matter can the logit be l_i=x' W x, with unconstrained W (gradient descent optimization of W tends to regularize it to be low rank anyhow).
3. Given the quadratic nature of the computation, is there a relationship to tensor product representations (Smolenski et al), where such computations are done in all layers of a network? (and do you plan to move your subspace projection into earlier layers, too?)
4. Norm regularization (12), as well as in equation 4 (to renormalize x to constant \gamma norm) may play big roles in learning reducing the real effect of the subspace modeling (and also do you do both of these things or just (12)?)
5. In the first ImageNet experiment, how would you account for the change in the modeling power by simply having more parameters in the last layer? 
6. In the transfer experiments, I am assuming that the issue above no longer exists, because you treat the features from the previous layer the same way (i.e. not through fine-tuned subspace projections, but using a linear classifier).  Is that right?
7. If the above is right, then Table 2 may be slightly confusing, as results for ImageNet seem to be copied from Table 1, where logits a computed using norm of the subspace projections, but for the rest of the datasets, they are computed using linear projections. 
8. Finally, the premise of the experiments is that the joint training of the backbone and the (subspace-based) classifier results in features that are better in the ways described in the paper.  If you initialize the network trained with regular softmax or cosine softmax classifier layer, and then switch to the subspace-based layer, what happens? Can keeping the features fixed and finding good subspaces increase accuracy? Does further training of the network change the features and how? (or is this not a meaningful experiment because of the lack of the bias term in your model?)
clarity,_quality,_novelty_and_reproducibility: As I mentioned above, I think the paper is clear and reproducible. 
summary_of_the_review: While I do have questions, I also like the discussion of the ideas in the paper. I don't regret reading it :), so I imagine others would not, either. So, unless something essentially equivalent has already been done (and I am not aware of it), then publication is justified. I am looking forward to the authors' response so I can understand the ideas and the details even better.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The overall goal of the submission is a learning formulation that simultaneously models inter-class discrimination while promoting intra class variation in classification. To this end, it considers a linear subspace approach that is scaleless and thus in theory more suitable for long tail classification than the vector counterpart. Since set of subspaces form a Grassmann manifold, the submission replaces the fully connected layer in deep networks with a geometric one and optimizes it with Riemannian SGD. The method is validated on various benchmarks where it is shown to improve the vanilla baseline on transfer learning as well as long tail classification tasks.
strength_and_weaknesses: Strength:

-Presentation: The submission is easy to read and follow. It is well written with intuitions provided where necessary. The problem is well motivated and contextualized in the broader scientific context.

-Technically solid and grounded work.


-Interesting empirical results wrt baselines considered in the submission and thus a promising direction.


Weakness:

- Technical Novelty, from Deep (Riemannian) Manifold Learning perspective, is somewhat marginal.

- Experimental Validation is heavily centered around Grassmanian baselines. While this submission cites several works that explicitly encourage/promote intra-class variablity, comparison with such baselines is completely missing.

- While the problem of promoting intra class variability is of great interest in deep learning, the proposed method does not explicitly model it as such. I concede it is not a strong weakness and based on the empirical findings in this work, future work can address this explicitly.
clarity,_quality,_novelty_and_reproducibility: -The paper is well written and easy to read. Related work is fairly covered.

- The idea of replacing fully connected layer with a geometric layer and the resulting impact on transfer learning and long tail classification is an interesting technical contribution.

- The authors have promised to released the code for reproducibility and also provided enough technical details in the submission.
summary_of_the_review: The approach is based on a principled framework even though the technical novelty is not very strong from a broader manifold learning perspective. The resulting gain wrt Grassmanian and vanilla baselines are interesting even though the method does not explicitly model the intra class variability in the formulation unlike some existing work in this direction which are not compared with.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper points out that using softmax does not take the intra-class and inter-class feature variation into account, and this paper aims to interpret the high dimensional feature output that lies in a union of linear subspaces. In classification, each feature representation falls into one of the subspaces (K classes), where each subspace is a Grassmannian manifold. To achieve this, this paper incorporates the Riemannian SGD into the ResNet50-D backbones to optimize the network and the k subspaces. The authors have validated that such an assumption is powerful and outperforms the softmax and cross-entropy combination in ImageNet-1K classification, feature transfer, and Long-tail classification.
strength_and_weaknesses: 
Pros,

1. Replacing the de-facto combination of softmax and cross-entropy is interesting. They provide intensive experiments to validate their claim.

2. The experiments on the long-tail tasks provide a new sight to handle the data imbalance issue. It correlated to an extra memory bank or dictionary somehow.

Cons,
1. The authors miss an important reference [1]. The proposed method has been employed and used in subspace clustering instead of classification. The authors should clarify clearly your contribution and difference.


[1] Scalable Deep k-Subspace Clustering, Tong Zhang, Pan Ji, Mehrtash Harandi, Richard Hartley, Ian Reid, ACCV 2018.

Unclear parts:
1.  When there are more than two identical eigenvalues, there will be a sign flip issue in the corresponding eigenvectors. It may lead to the subspace update in a different way. Thus, I would like the authors to provide an analysis of such randomness due to subspace updating.

2. In table 2, as the Grassmannian with 16 dimensions is better than 8, why there is no FN experiment on 16 dimensions? I am also wondering how the sparsity is related to the FN. Comparing dimension 8, with FN and without FN the accuracy is quite similar but sparsity changes a lot. 

3. The batch normalization will project the feature space to a unit sphere, which will be against the linear subspace assumption. Could the authors explain more on this direction and how you solve this issue? 

4. Besides, I am also wondering how data augmentation affects the accuracy since the authors did not use augmentation in their implementation.

clarity,_quality,_novelty_and_reproducibility: Clarity. This paper put their goal straightforward and clear.  The paper is very easy to follow.

Novelty, considering the reference [1], the novelty of the algorithm is limited. But I do appreciate that the paper finds good applications instead.

Reproducibility, it should be easy to reproduce the paper given the details.
summary_of_the_review: Since the algorithm novelty of the paper is limited, I will need more details from the authors to justify whether they indeed bring new insights into this area.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, traditional class representative vectors deep neural networks are replaced by linear subspaces on Grassmann manifolds. It is supposed to be more informative for intra-class feature variations. The proposed method optimizes the subspaces using geometric optimization, with an efficient Riemannian SGD implementation tailored for Grassmannians. Experiments on image classification, feature transfer, and long-tail classification tasks show that the new method improves the vanilla softmax and cosine softmax.
strength_and_weaknesses: Strength:

(1) Introducing subspace learning in deep neural networks is interesting, and introducing geometric optimization with Riemannian SGD is useful to solve this problem.

(2) Experimental results on three tasks show improvements over traditional softmax methods.

(3) The paper is well written and organized.

Weaknesses:

(1) With larger k, the proposed method introduces much more parameters, not to say the SVD operation in the Riemannian SGD solver. As in Table 2, without FN only k=16 shows improvements. This would make a very difficult scalability for large-scale learning, for example, with millions of classes in training face recognition models.

(2) If computation is not an issue, traditionally there are also methods in expanding the class representative vectors, e.g. multiple experts and fusion. It is not clear if the improvement is due to enlarged classification parameters or due to the new learning framework. Therefore, it would be better to show a comparison against multiple experts.

(3) The FN loss contributes a lot for the improvements. However, it should be a general trick that can also be applied on the traditional softmax baselines, which should also be reported for a fair comparison. Without the FN loss it appears that the improvement of the proposed method is still limited.

(4) With k=1 the linear subspaces degrade to class vectors. In such case what is the difference between the proposed method with k=1 and cosine softmax? They perform quite similar with each other across all the three tables.

(5) How will the proposed method incorporate margin parameters and what would be its effect? 
clarity,_quality,_novelty_and_reproducibility: Clarity: good.

Quality: fair.

Novelty: good.

Reproducibility: good.
summary_of_the_review: The proposed new formulation for learning linear subspaces in deep neural networks is very interesting. However, from my point of view this work is still not yet solid. As I can understand the proposed method is novel and I expect a lot from it. However, after reading the experimental results I'm not that excited, and I think the proposed method is not yet fully validated in some aspects, as listed in weaknesses. To me the only reason to accept the current paper is that I would like to encourage this novel study.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to represent a class as a subspace in the deep learning regime. The contributions of this paper are the formulation of classes as subspaces, the Grassmannian layer to update subspaces, learning the Grassmannian layer using constraint optimization. The core contribution is to represent a class as a subspace where we can easily replace the softmax layer with fully-connected networks with the projection between subsapaces and the input feature. The experiments show strong performance improvement on some cases on large datasets (e.g., ImageNet). As an example of its effectiveness compared to the softmax one, the top-1 accuracy of ImageNet1K is improved by 1.3%.

strength_and_weaknesses: Strengths:
- The feature norm regularization is somewhat novel for training the neural network under geometry constraints.
- This work has a strong performance in term of empirical results compared to prior methods in image classification and transfer learning.
- The empirical results also show that the representations using the Grassmannian layer is more sparse.

Weaknesses :
- This work has unclear motivation why second order representations in the form of linear subspaces yields better performance compared to the first order representations.  There is no motivating examples nor theories when subspaces are suitable representing classes. Citing the work Watanabe and Pakvasa does not directly describe why the linear subspace approach is a better model to represent classes in the era of deep learning.  
- The novelty of this work is marginal with many overlapping points and contributions compared to the work of Simon et al. The problems of image classification and transfer learning are covered by the work of Simon et al. that enjoys the superiority of linear subspaces over prototypes (a single vector) to represent classes in few-shot learning. In experiments,  this work does not discuss or even compare with the proposed method. 
- The proposed method updaates linear subspaces in classifiers with some constraints, and also this is not novel as some other works by Harandi and Fernando “Generalized BackPropagation Etude De Cas: Orthogonality” and Roy et al., “Siamese Networks: The Tale of Two Manifolds” have discussed similar concepts (i.e. the geometry aware layers) and implemented the proposed method for image classification but this works has no comparison to these prior works.
- Moreover, the types of data feasibly represented using the linear subspace method are also not discussed in the paper. Is the proposed method only applicable for visual data?
- This statement “The whole matrix S needs not be orthonormal”. For discriminative purposes, even though it needs more investigation, the straightforward idea is to force subspaces as different as possible to avoid collapses (see Simon et al., Arjovsky et al., Ozay and Okatani, and Harandi et al.). However, this work does not have any discussion about this idea nor include the idea to discriminate between subspaces.
- There are no properties of the proposed Grassmannian class representation layers.  For instance, what are the properties and benefits preserving the orthogonality for each subspace? what are properties of not preserving subspaces coming from different to be orthogonal?
- The design of this approach is somehow limited for a neural network module. How is the design of the proposed method with the multi-layer version of fully-connected layers (if possible)?
- The experiments require some prior methods for comparison with some variants in class representations, e.g., prototypes (the average of all representations within a class), non-learnable subspaces (a similar concept as in Simon et al.), .
- The performance of long-tail classification is marginally improved compared to cosine softmax.  That shows that the proposed method might not be quite effective in addressing such issue compared to transfer learning and common image classification.
- Is there any comparison in terms of speed between softmax strategies and the Grassmanian one? The discussion of trade-off between the performance gain and the processing time is crucial for this type of method because it usually requires additional processing time especially with contstraint optimization.
- The experiments are also lacking of comparison to some other models as a backbone. For instance, the proposed method can compare the methods using transformer models, another ResNet type (e.g., ResNet101), VGG, Inception.
- The feature sparsity is not very clear, is that 78% zero activations on the feature before the Grasmannian layer? or the elements of the Grassmannian layer (i.e., each subspace)? or the output after the Grassmannian layer?



References:

Simon et al, “Adaptive Subspaces for Few-Shot Learning,“ CVPR, 2020.

Harandi et al., “Extrinsic Methods for Coding and Dictionary Learning on Grassmann Manifolds,” IJCV, 2015.

Arjovsky et al. "Unitary evolution recurrent neural networks," ICML, 2016.

Ozay and Okatani, "Training CNNs with normalized kernels," AAAI, 2018.
clarity,_quality,_novelty_and_reproducibility: The novelty of this work is considered marginal with some reasons explained in the weaknesses, the clarity of the statement needs to be improved, and currently there is no code for reproducibility.


summary_of_the_review: This paper has some insights about the use of linear subspaces for image classification especially comparing with the vanilla fully-connected layer in neural networks. However, there are some issues regarding novelty, comparison, and experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was evaluated by five reviewers and received 2x5 and 3x6 borderline scores. While the reviewers liked the paper they had also a number of important concerns that remained including lack of strong comparisons with existing bilinear methods, matrix square root methods, log-Euclidean and other non-Euclidean classifiers [A,B,C] (there are many more works than A B C). Reviewers also noted improvements were marginal and evaluations could have been conducted on a larger number of datasets often used with the above methods. Also, vector-to-subspaces has been used before as a classifier in several works, for example in FSL [D,E,F]. Overall, this is a well established concept and as such the paper requires non-incremental contributions, evaluations and findings to be accepted. On this occasion, the paper is below the acceptance bar and requires a new round of reviews after fixing the above issues.

A. Improved Bilinear Pooling with CNNs, Lin et al.
B. Fast Differentiable Matrix Square Root, Song et al.
C. Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling? Song t al.
D. Regression Networks for Meta-Learning Few-Shot Classification, Devos  et al.
E. Adaptive Subspaces for Few-Shot Learning, Simon et al.
F. TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning, Yoon et al.
justification_for_why_not_higher_score: After rebuttal and AC-rev. discussions, several issues as described above lingered (issues with the limited novelty, limited evaluations, and marginal improvements).
Decision: Reject