Grassmannian Class Representation in Deep Learning
Authors: Haoqi Wang
Zhizhong Li
Wayne Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Grassmannian
geometric optimization
classification
feature transfer
long-tail

Abstract: We generalize the class representative vector found in deep classification networks to linear subspaces and show that the new formulation enables the simultaneous enhancement of the inter-class discrimination and intra-class feature variation. Traditionally, the logit is computed by the inner product between a feature and the class vector. In our modeling, classes are subspaces and the logit is defined as the norm of the projection from a feature onto the subspace. Since the set of subspaces forms Grassmann manifolds, finding the optimal subspace representation for classes is to optimize the loss on a Grassmannian. We integrate the Riemannian SGD into existing deep learning frameworks such that the class subspaces in a Grassmannian are jointly optimized with other model parameters in Euclidean. Compared to the vector form, subspaces have two appealing properties: they can be multi-dimensional and they are scaleless. Empirically, we reveal that these distinct characteristics improve various tasks. (1) Image classification. The new formulation brings the top-1 accuracy of ResNet50-D on ImageNet-1K from 78.04% to 79.37% using the standard augmentation in 100 training epochs. This confirms that the representative capability of subspaces is more powerful than vectors. (2) Feature transfer. Subspaces provide freedom for features to vary and we observed that the intra-class variability of features increases when the subspace dimensions are larger. Consequently, the quality of features is better for downstream tasks. The average transfer accuracy across 6 datasets improves from 77.98% to 80.12% compared to the strong baseline of vanilla softmax. (3) Long-tail classification. The scaleless property of subspaces benefits classification in the long-tail scenario and improves the accuracy of ImageNet-LT from 46.83% to 48.94% compared to the standard formulation. With these encouraging results, we believe that more applications could benefit from the Grassmannian class representation. Codes will be released.

Under review as a conference paper at ICLR 2023
GRASSMANNIAN CLASS REPRESENTATION IN DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We generalize the class representative vector found in deep classification networks to linear subspaces and show that the new formulation enables the simultaneous enhancement of the inter-class discrimination and intra-class feature variation. Traditionally, the logit is computed by the inner product between a feature and the class vector. In modeling, classes subspaces and the logit is defined as the norm of the projection from a feature onto the subspace. Since the set subspaces forms Grassmann manifolds, finding the optimal subspace representation for classes is to optimize the loss on a Grassmannian. We integrate the Riemannian SGD into existing deep learning frameworks such that the class subspaces in a Grassmannian are jointly optimized with other model parameters in Euclidean. Compared to the vector form, subspaces have two appealing properties: they can be multi-dimensional and they are scaleless. Empirically, we reveal that these distinct characteristics improve various tasks. (1) Image classification. The new formulation brings the top-1 accuracy of ResNet50-D on ImageNet- IK from 78.04% to 79.37% using the standard augmentation in 100 training epochs. This confirms that the representative capability of subspaces is more powerful than vectors. (2) Feature transfer. Subspaces provide freedom for features to vary and we observed that the intra-class variability of features increases when the subspace dimensions larger. Consequently, the quality of features is better for downstream tasks. The average transfer accuracy across 6 datasets improves from 77.98% 3 80. compared the strong bascline of vanilla softmax. (3) Long- tail classification. The scaleless property of subspaces benefits classification in the long-tail scenario and improves the accuracy of ImageNet-LT LT from 46.83 to 48. 94% compared to the standard formulation. With these encouraging results, we believe that more applications could benefit from the Grassmannian class representation. Codes will be released.
INTRODUCTION
The idea of representing classes linear subspaces in machine learning be dated back, at least, to 1973 (Watanahe Pakvasa (1973)), yet is mostly ignored in the current deep learning literature. In this paper, we revisit the scheme of representing classes linear subspaces in the deep learning context. To be specific, each class is associated with linear subspace S,, and for any feature vector x, the i-th class logit is defined as the norm of projection
4 |projss #||
Since a subspace is point in the Grassmann manifold (Absil et al. (2009)), we call this formulation the Grassmannian class representation. In the following, we answer the two critical questions,
1. Is Grassmannian class representation useful in real applications?
2. How optimize the subspaces in training?
The procedure fully-connected layer + softmax - cross-entropy loss is the standard practice in deep classification networks. Each column of the weight matrix of the fully- connected layer is called the class representative vector and serves as a prototype for one class. This representation of class has achieved huge success, yet it is not without imperfections.

---

Under review as a conference paper at ICLR 2023
In the study of transferable features, researchers noticed dilemma that representations with higher classification accuracy on the original task lead to less transferable features for downstream tasks (Kornblith et al. (2021); Müller et al. (2019)). This is connected to the fact that they tend to collapse intra-class variability of representations, resulting in loss of information in the logits about the resemblances between instances of different classes. Furthermore, the neural collapse phenomenon (Papyan et al. (2020)) indicates that as training progresses, the intra-class variation becomes negligible, and features collapse to their class-means. So this dilemma inherently originates from the practice of representing classes by single vector. The Grassmannian class representation shed light on this issue as features of each class are allowed to vary in a high-dimensional subspace without incurring losses in classification.
In the study of the long-tail classification, researchers found that the norm of class representative vectors is highly related to the number of training instances in the corresponding class (Kang et al. (2019)) and the recognition accuracy is affected. To counter this effect, the class representative vector is typically been rescaled to unit length during training (Liu et al. (2019)) or re-calibrated in an extra post-procsssing step (Kang et al. (2019)). In addition to these techniques, the Grassmannian class representation provides a natural and elegant solution for this subspace is scaleless.
is well known that the set of k-dimensional linear subspaces form a Grassmann manifold, so finding the optimal subspace representation for classes is to optimize the Grassmann manifold. Thus for the second question, the natural solution is to use the geometric optimization (Edelman et al. (1998)), which optimizes an objective function under the constraint of a given manifold. Points being optimized are moving along geodesics instead of following the direction of Euclidean gradients. The preliminary concepts of geometric optimization are reviewed in Section 3, and the technical details of subspace learning are presented in Section 4. We implemented an efficient Riemannian SGD for optimization in Grassmann manifold as shown in Algorithm 1, which integrates the geometric optimization algorithms to deep learning frameworks so that both the linear subspaces in Grassmannian and model weights in Fuclidean are jointly optimized.
Going back to the first question, we experiment on three concrete tasks in Section 5 to demonstrate the practicality and effectiveness of Grassmannian class representation. We find that (1) Grassmannian class representation improves large-scale image classification accuracy. (2) Grassmannian class representation produces high-quality features that can better transfer to downstream tasks. (3) Grass- mannian class representation improves the long-tail tail classification accuracy. With these encouraging results, we believe that Grassmannian class representation is a promising formulation and more applications may benefit from its attractive features.
RELATED WORK
Geometric Optimization Edelman et al. (1998) developed the geometric Newton and conjugate gradient algorithms on the Grassmann and Stiefel manifolds in their seminal paper. Riemannian SGD was introduced in Bonnabel (2013) with analysis convergence and there are variants such as Riemannian SGD with momentum (Roy et al. (2018)) or adaptive (Kasai et al. (2019)). Other popular Euclidean optimization methods such as Adam also studied in the Riemannian manifold context (Becigneul & Ganea (2019)). Lezcano-Casado & Martinez- Rubio (2019) study the special case of SO(n) and ('n) and uses the exponential map enable Euclidean optimization methods for Lie groups. The idea was generalized into trivialization in Lezcano Casado 2201).. Our Riemannian SGD Algorithm is tailored for Grassmannian, so we have a closed-form equation for geodesics. Other applications of geometric optimization include matrix completion (Mishra & Sepulchre (2014); Li al. (2015b;a); Nimishakavi et (2018)), hyperbolic taxonomy embedding (Nickel & Kiela (2018)), etc. Hamm & Lee (2008) propose the Grassmann discriminant analysis, in which features are modeled as linear subspaces. These applications are mostly using shallow models. Zhang et al. (2018) use subspaces to model clusters in unsupervised learning, which share similar spirit with our work. Simon et al. (2020) model classes as subspaces in few-shot learning, however, their subspaces computed from data matrix rather than explicitly parametrized and learned. Roy et al. (2019) use Stiefel manifold to construct Mahalanobis distance matrix in Siamese networks in order to improve feature embeddings of deep metric learning.
2

---

Under review as a conference paper at ICLR 2023
Orthogonal Constraints in Deep Learning There works that enforce orthogonality on weights, which study the regularization effect of orthogonal constraints, Contrastingly, we used orthogonal matrices as the numerical representation of the geometry object of subspaces and focus the representation of classes. The approaches of enforcing orthogonality include regularizations (Arjovsky et al. (2016); Xie et al. (2017a) 7a); Bansal al. (2018); Qi et al. (2020); Wang et al. (2020), etc.), geometric constraints (Ozay & Okatani (2018); Harandi & Fernando (2016)) and paraunitary systems (Su et al. (2022)). Orthogonally constrained data is also explored by Huang al. (2018).
Improving Diversity in Feature Learning Grassmannian class representation encourages the intra-class variation implicitly by providing a subspace to vary. In metric learning. there are efforts to explicitly encourage feature diversity. For example, SoftTriplet Loss (Qian al. (2019)) models each class as local clusters with several centers. Zhang et al. (2017) use a global orthogonal regularization to encourage local descriptors to spread out in the features space. Yu et al. (2020) propose to learn low-dimensional structures from the maximal coding rate reduction principle. The subspaces are estimated using PCA on feature vectors after the training. In our formulation, subspaces are directly optimized in the Grassmann manifold during training.
Normalized Classification Weights Normalizing class representative vectors has been found use- ful in representation learning (Wang et al. (2017; 2018); Deng et al. (2019)) and long-tail classification (Liu et al. (2019); Wang al. (2021)). However, works such as ArcFace (Deng et al. (2019)) focus on adding an extra margin to suppress intra-class variance. In contrast, our subspace formulation encourages intra-class variation.
3 PRELIMINARIES
In this section, briefly review the essential concepts in geometric optimization. Detailed exposition can be found in Edelman et al. (1998) and Absil et al. (2009). Given an n-dimensional Euclidean space "R,", the set of k-dimensional linear subspaces forms the Grassmann manifold (kk: n)) A compurationallfriendtf representation for subspace $ E G(k, n)) is an orthonormal matrix S E Ra where 5 = I₁ and I& the k x k identity matrix. Columns of matrix S can be interpreted as an orthonormal basis for the subspace S. The matrix representation is not unique, as right multiplying by an orthonormal matrix will get a new matrix representing the same subspace. Formally, Grassmannian is a quotient space of the Stiefel manifold and the orthogonal group ((k, n) = St(k, (k)),,) where St(k, n) {X E R=* X X } and O(k) = {X E R XX X Ix When the context is clear, we use the notation of space S and one of its matrix representations S interchangeably. The tangent space of the Grassmann manifold at S consists of all n x k matrices such that S T =
Given function f: G(k,n) + R defined on the Grassmann manifold, the Riemannian gradient of at point S E (kk, n)) is given by (Edelman et al., 1998, Equ. (2.70))
Vf(S) fs - SST fs.
(2)
Us When performing gradient descend on the Grassmann manifold, and suppose the current point is S and the current Riemannian gradient is G. then the next point is the endpoint of S moving along the geodesic toward the tangent G with some step size. The formula of the geodesic is given by (Edelman et al., 1998, Equ. (2.65)),
where fs is the Euclidean gradient with elements (fs) = 21
S(t) = (SV cos cts + U sin(t VT.
(3)
where UEVT = - is the thin singular value decomposition of G.
LEARNING THE GRASSMANNIAN CLASS REPRESENTATION
Denote the weight of the last fully- layer in a classification network by W E R and the bias by b E C where n is the dimension of features and C' is the number of classes. The i-th column vector w; of W is called the i-th class representative vector. The i-th logit is computed as the inner product between feature x and the class vector (and optionally offset by bias by), namely w x + b.. We extend this well-established formula to a multi--iimensional subspace form
4 [[pojss. 20
(4)
3

---

Under review as a conference paper at ICLR 2023
9(1.2)
wf
X0
Figure 1:
Geometric optimization Grassmann manifold ( 2). Each point (e.g. w²) in the black circle represent the I-dimensional linear subspace passing through it, g is the Riemannian gradient obtained by the projection of Euclidean gradient d. w² moves along the geodesic towards the direction to new point w ++1
where St E G(k, n) is a k-dimensional subspace in the n-dimensional feature space. We call S; the i-th class representative space, or class space in short. Comparing the new logit to the standard one, the inner product of feature a with class vector is replaced by the norm of the subspace projection projs, z and the bias term is omitted. We found that re- neenooallizig features
constant length ? -/2 To simplify notation, we |[=]l
improves training. Incorporating this, Equation (4) becomes projs,
assume feature has been properly re-normalized throughout this paper unless otherwise specified.
The application of the subspace class representation requires two modifications to an existing network. Firstly, the last fully-connected layer replaced by its geometric counterpart, which is detailed in Section 4.1. The new geometric layer will transform features to logits using Equation (4). Secondly, the optimizer should be extended to process the new geometric layer simultaneously, which is explained in Section 4.2. Parameters of the geometric layer are optimized using Geometric SGD, while all other parameters are optimized usual using the standard SGD algorithm.
4.1 GRASSMANNIAN CLASS REPRESENTATION
Suppose for class i, 1, 2, ... C, its subspace representation is S; G(k,, nt). where the dimension k; is hyperparameter and is fixed during training. Then the tuple of subspaces (S1, S Sc) will be optimized in the product space G(kı, n) x G(k,, n) x... x G(ke, n). Denote matrix instantiation of S; as S; E IR" Rx where the column vectors form an orthonormal basis S,, then we concatenate the matrices into big matrix
(5) The matrix S contains the parameters that optimized numerically. For feature æ, the product ST x gives the coordinate of projs, x under the orthonormal basis formed by the columns S,. By definition in Equation (4), the logit for class and feature æ: is computed by
S SS S2 Scl R
4 projs, x = ||ST ae|
(6)
Grassmannian Fully-Connected Layer We can implement a geometric fully-connected layer using the plain old fully-connected layer. The shape of the weight is n x (k1 + k2 - + kc), as shown in Equation (5). In the forward pass, the input feature is multiplied with the weight matrix to get a temporary vector S æ. then the first element the output is the norm of the sub-vector (f1...., tR,)., ), and the second element of the output is the norm of (tx, +k, +2....... tx,
etc.
Parameter Initialization Each matrix instantiation of the subspace should be initialized as an orthonormal matrix. The geometric optimization algorithm described in Section 4.2 ensures their orthonormality during training. Specifically, for Grassmannian fully-connected layer, each block S; of the weight in Equation (5) is orthonormal. The whole matrix needs be orthonormal.
4.2 OPTIMIZE THE SUBSPACESS
Geometric optimization is to optimize functions defined manifolds. The key step is to find the Riemannian gradient of the loss function and then descend along the geodesic. Here the manifold in concern is the Grassmannian (k,, nt). As intuitive example, ((1 consists of all lines through the origin a two-dimensional plane. We can visualize it as a unit circle where each point on the unit circle represents the line passing through it. Antipodal points represent the same line. To illustrate
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 An Iteration of the Riemannian SGD with Momentum for Grassmannian at Step
Input: Learning rate > 0, momentum I E [0, 1), Grassmannian weight matrix S E B
momentum buffer M (t-1) |R=* Euclidean gradient D IR**k.
Compute Riemannian gradient G (In S
Equation (8)
2: Approximately parallel transport M to the tangent space current point S by projection
M (In SST) ((
-
(11)
3: New momentum M + μM + G.
PyTorch version
4: Move along geodesic using equation (3), FUEVI = ( is the thin singular value decompo- sition, then
s(++1) (s(t) cos(>>) U sin(12)) VT.
5: (Optional) Re-orthogonalization S(++1) by QR decomposition.
For numerical stability
how geometric optimization works, define toy problem G 2) that maximizes the norm of the projection of a fixed vector æp onto line through the origin, namely
max ||projs SE9(1.2)
As shown in Figure 1, we represent 5 with unit vector w E S. Suppose at step 1, the current point is w(t)) then it is easy to compute that the Euclidean gradient at w((1) is d = æo, and the Riemannian gradient is the Euclidean gradient d projected to the tangent space of G 2) at point w((") The next iterative point (t+1)) is to move (0(() along the geodesic toward the direction g. Without geometric optimization, the next iterative point would have lied at (() + ,d, jumping outside of the manifold.
The following proposition computes the Riemannian gradient needed.
Proposition 1. Let S Rank be matrix instantiation of subspace S E Gk., n). and æ E R is a vector in Euclidean space, then the Riemannian gradient G of x) p w.t.. S is
G 1 [In -SS (s"')xx''s S. i
(8)
Proof. Rewrite ||projææ||| = VxT SS x, and compute the Euclidean derivatives as
0l
1 xxx s,
Øl dx 7 S a.
as
Then Equation (8) follows from Equation (2).
We give a geometrie interpretation of Proposition 1. Let wi be the unit vector along direction projg then expand it to an orthonormal basis of S, say {ur,, W2,..., wp.- Since Riemannian gradient invariant to the matrix instantiation, we can set S = [w) we wx). Then Equation (8) becomes
G = [ SS æ
0].
(10)
since u; 1. i. =, and w/ x: I. Equation (10) shows that in the single-sample case, only one basis vector uy needs to be rotated towards vector æ. where wi is the unit vector S that closest to æ.
Riemannian SGD During training, parameters of non-geometric layers are optimized as usual using the vanilla SGD algorithm. For geometric layers such as the Grassmannian fully-connected layer, their parameters are optimized using the Riemannian SGD algorithm. The pseudo-code of the Riemannian SGD with momentum, which we implemented in our experiments, described in Algorithm 1. We only show the code for the single-sample, single Grassmannian case. It is trivial to extend them to the batch version and the product Grassmannians. Note that in step 2, we use projection to approximate the parallel translation of momentum for efficiency, and in step 5 an optional extra orthogonalization can improve numerical stability. The momentum update formula is adapted from the Py Torch implementation of the vanilla SGD. Weight decay does not apply here since spaces are scaleless. Algorithm works together with the vanilla SGD and modifies the gradient from Fuclidean Grassmannian on-the-fly for geometric parameters.
5

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENT
In this section, we study the influence of Grassmannian class representation through experiments. Firstly, in Section 5.1, we show that the expressive power of Grassmannian class representation improves accuracy in large-scale image classification. Secondly, in Section 5.2, we show that the Grassmannian class representation improves the feature transferability by allowing larger intra-class variation. Thirdly, in Section 5.3, we demonstrated that the scaleless property of the Grassman- nian class representation improves the classification accuracy in the long-tail scenario. Additional experiments hyper-parametee choices and design decisions are presented in Appendix B.
We choose the vanilla softmax loss and the cosine softmax loss (without margin) as baselines since they reflect the current typical class representations. The former uses plain vector and the latter uses a normalized vector. Other innovations on losses, such as adding margins (Deng et al. (2019)), re- re-balancing class-wiss gradients (Wang al. (2021)), orthogonal to contribution.
5.1 GRASSMANNIAN CLASS REPRESENTATION IMPROVES CLASSIFICATION ACCURACY
We apply the Grassmannian class representation to large-scale classification, where consistent improvement over baselines shown. We then analyze the characteristics both the learned features and the learned class subspaces. On the feature representation side, compare the feature sparsity and intra-class variability. On the class representation side, we visualize the principal angles between any pair of classes, concept that only appears when classes are Grassmannian.
Experimental Setting We use the ResNet50-D - (He et al. (2019)) architecture as the base model, and benchmark on ImageNet-IK (Deng et al. (2009)). ResNer50-D is a slight modification of the original ResNet-50 (He et al. (2016)) with about 1 1 improvement in accuracy. ImageNet- I is large-scale image classification dataset containing 1.28M training images and 50K validation images in 1000 categories. We set y 25 for both cosine softmax and the Grassmannian class representation. Our method replaces the last fully-connected layer of ResNet50-D by a Grassmannian fully fully-connected layer. To reduce the number of apper-parameters we simply set the subspace dimension k be the same for all classes. We vary the
& in the range [1, 2, 4, 8, 16]. Since the dimension of feature is 2048, the Grassmannian fully-connected layer has the geometry of 11 (k; 2048).
Training Strategy All settings share the same training strategy. Each training includes 100 epochs with total batch size 256 on NVIDIA Tesla V100 GPUs SGD is used for baselines and Riemannian SGD described in Algorithm 1 is used for Grassmannian class representations. The momentum is 0.9 and the weight decay is 0.0001. The initial learning rate is 0.1 and then follows the cosine learning rate decay. The checkpoint with best validation score is used. The input size is 224 x 224 and we use the standard augmentation for ImageNet, namely, random resized crop followed by random horizontal flip. The code is implemented using the mmclassification (MMClassificttion Contributors (2020)) package, and uses PyTorch as the training backend. Note that to make the number of experiments tractable due to our limited computation resources, we omitted many tricks that has shown to improve representation learning, such as stronger augmentation (Cubuk et al. (2020), longer training (Wightman et (2021)), adding margins (Deng et al. (2019)) etc., and focus on the improvements solely contributed by the Grassmannian formulation.
Feature Norm Regularization We noticed that the norm of the feature (before re-normalization) decreases as training progresses (details see Appendix A). For example, the case of k: 16, the average norm of feature decreases from 1.051 at epoch 10 to 0.332 at epoch 100. Although the norm of the feature does not affect inference result due to the feature re-normalization when computing logits, we empirically find that encouraging the norm to be larger than a constant L improves the training. Specifically, we propose a feature norm regularization loss PN
LFN
Σ (relu (L - 11xx K
(12)
where æ; is the feature of the i-th sample before normalization and K is the number of features with norm larger than I.. In experiments, I. 1 and the loss is directly added to the softmax loss

---

Under review as a conference paper at ICLR 2023
Table ResNet50-D on ImageNet- IK classification dataset, under different logit formulations. sparsity
validation features and the variability of training features are listed. FN denotes the feature norm regularization.
Variability
Accuracy
Setting
Dim
Sparsity
Intra-Class
Inter-Class
Top1
Top5
Softmax
0.55
60.12
90.01
78.04
93.89
Cosine Softmax
77.70
56.87
89.98
78.30
91.07
Grassmannian
77.92
56.52
89.98
78.48
94.24
Grassmannian
78.96
59.20
89.98
78.92
94.32
Grassmannian
78.49
61.25
89.99
78.86
94.34
Grassmannian
78.03
63.57
89.99
79.12
94.41
Grassmannian
16
78.12
65.81
90.00
79.21
94.29
Grassmannian + FN Grassmannian FN
33.51
56.32
90.02
78.65
94.24
45.55
63.01
90.02
79.37
94.53
with equal weight. We also tried larger values of L or regularize the norm of feature on both sides, however, they degrade the performance.
The validation accuracies of different models on ImageNet-1 is listed in Table 1. All models with the Grassmannian class representation achieve higher top-1 and top-5 accuracies than the vanilla softmax and the cosine softmax. A general trend that, with larger subspace dimension the accuracy improvement is greater. When subspace dimension is 16, the top-1 accuracy is 79.21 %, which is 1.17% points higher than the vanilla softmax loss. With feature norm regularization, the top-1 accuracy further improves from 79.12% to 79.37% for dimension 8.
Results
Intra-Class Class Variability Increases with Dimension The intra-class variability measured by the mean pair-wise angles (in degrees) between features within the same class, and then average over all classes. The inter-class variability is the average of mean pair-wise angles between features from different classes. Following the convention in the study of neural collapse (Papyan et al. (2020)), we use the global centered training feature compute variabilities. Kornblith et al. (2021) showed that alternative objectives that improve accuracy, including label smoothing, dropout, sigmoid, cosine softmax, logit normalization, etc., collapse the intra-class variability in representation, which in consequence degrades the quality of feature downstream tasks. However, this conclusion does not apply when the classes are modeled by subspaces. The intra-class variability does reduces from baseline's 's 60.12 to Grassmannian formulation's 56.52 when the subspace dimension k = however, as increases, both the top-1 accuracy and the intra-class variability grow. This indicates that representing classes subspaces enables the simultaneous improvement of class discriminative power and expansion of intra-class variability.
Feature Sparsity The feature sparsity is measured by the average percentage of zero activations on the validation set. As shown Table 1, the feature from vanilla softmax networks are very dense, with only 0.55% zero activations. Cosine softmax and Grassmannian class representations all result in more sparse representations, with around 78% zero activations. The feature norm regularization decreases the sparsity about half.
Principal Angles Between Class Representative Spaces When classes are subspaces, relation- ships between two classes can be measured by k angles called principal angles, which contain richer information than a single angle between two class vectors. The principal angles between two k-dimensional subspaces S and R are recursively defined (Absil et al. (2006)
cos(B: (6:) max max s" s² Tú, |s|| = ||||| = 1, s. s, = - = = i (13) #ES PER
for I,... A and 0, E x/ In Figure 2, we illustrate the smallest and largest principal angles between any pair of classes for a model with 1 = 8. From the figure, we can see that the smallest principal angle reflects class similarity, and the largest principal angle is around 7/2. A smaller angle means the two classes are correlated in some directions, and π 7/2 angle means that some directions in one class subspace is completely irrelevant (orthogonal) to the other class.
7

---

Under review as a conference paper at ICLR 2023
110
105
95
90
85
80
75
(b)
(c)
Figure 2: Each sub-figure is a heatmap of 1000 x 1000 grids. The color the i-th row and the j-th column represent an angle between class and class j in ImageNet- IK. (a) Pair-wise angles between class vectors of the ResNet50-D trained by vanilla softmax. Grids with red hue is large 90% and blue hue means smaller than 90". (b) Pair-wise smallest principal angles between 8-dimensional class subspaces of a ResNeL50-D model. Deeper blue colors indicates smaller angles. (c) Pair-wise largest principal angles of the same model as in (b). Grayish color means they close to 90"
Table 2: Linear transfer using SVM. Models are the same as in Table 1. R is the class separation.
ImageNet Top-1
Transfer
Setting
Dim
CIFARIO
CIFARIO0
Food
Pets
Cars
Flowers
Avg.
Softmax
78.04
0.495
90.79
67.76
72.13
92.49
51.55
93.17
77.90
Cosine Softmax
78.30
0.528
89.34
65.32
64.79
91.68
43.92
87.28
73.72
Grassmannian
78.48
0.534
89.62
64.40
64.94
91.91
43.49
84.67
73.17
Grassmamnian
78.92
0.493
89.71
66.33
66.08
91.44
45.11
87.81
74.41
Grassmannian
78.86
0.463
90.01
67.72
67.68
91.68
50.49
88.64
76.04
Grassmannian
79.12
0.429
90.63
68.25
69.85
91.83
51.83
94.06
77.74
Grassmannian
16
79.21
0.395
90.74
68.98
71.17
92.18
56.15
93.36
78.76
Grassmannian FN
78.65
0.541
90.20
67.58
69.07
92.02
53.45
94.49
77.80
Grassmannian FN
79.37
0.439
91.31
69.67
72.46
92.59
58.91
95.77
80.12
5.2 GRASSMANNIAN CLASS REPRESENTATION IMPROVES FEATURE TRANSFERABILLTT
In this section compare the linear transferability of the features learned by different models trained on the ImageNet-1K K dataset. The feature transfer benchmark dataset includes CIFAR-10 10 (Krizhevsky et (2009)), CIFAR- (Krizhevsky et al. (2009)), Food-101 (Bossard al. (2014)), Oxford-IIIT Pets (Parkhi al. (2012)), Stanford Cars (Krause et al. (2013)), and Oxford 102 Flowers (Nilsback Zisserman (2008)). For each the transfer dataset, we use the same trained models as in Table 1 to extract their features. Then all features are normalized to unit length. We fit linear SVM with one-vs-rest multi-clase policy on the training set, and report the accuracies on their test set. The regularization hyper-paramette for SVM is grid searched with candidates 1,.1,0.2, 0.5, 1,2,5 5, 10, 15, 20) and determined by five-fold cross-validation on the training set.
Results As shown in Table the cosine softmax and the Grassmannian with subspace dimension has comparable transfer performance, but both are lower than the vanilla softmax. However, when the subspace dimension increases, the transfer performance gradually improves, and when k 16, the transfer performance is on par with vanilla softmax. The feature norm regularization improves the transfer quality, as shown the 1, cases. We hypothesize that this might relate to the fact that features with norm regularization are less sparse, more information are encoded.
Class Separation The class separation is measured by the index R², which is defined as one minus the ratio of the average intra-class cosine distance to the overall average cosine distance (Kornblith et al., 2021, Eq. (1)).. Komblith et al. (2021) found that greater class separation R² is associated with less transferable features. This may explain the feature transfer performance of Grassmannian class
8

---

Under review as a conference paper at ICLR 2023
Table 3: Long-tail classification on ImageNett-LT. Test accuracies are shown.
Base Model
Dim
Many(> 100)
Mcdium(20-100)
Few(<: 20)
Overall
Softmax
65.42
40.64
15.97
46.83
Cosine Softmax
67.16
42.37
16.87
48.45
Grassmannian
67.70
42.87
17.22
48.94
Grassmannian
2
67.75
42.75
15.44
48.66
Grassmannian
4
68.02
40.82
12.88
17.49
Grassmannian
67.78
37.60
10.88
45.59
representations. The vanilla softmax has lower separation (0.495) compared to the cosine softmax (0.528) and the Grassmannian class representation with subspace dimension k = (0.534). From subspace dimension high value (0.534) to a low value (0.395). The change in class separation is roughly in line with the change of transfer performances.
16, the separation from Grassmannian models decreases from
5.3 SCALELESS OF SUBSPACE IMPROVES LONG-TAIL RECOGNITION
We benchmark its effectiveness long- tail classification using the ImageNet-LT dataset (Liu et al. (2019)). ImageNet-LT is subset of ImageNet-IK, where the number of images class ranges from to 1280. There are totally 115.8K images, roughly 1/10 size of ImageNet-IK. We use same ResNet50-D networks in Section 5.1. All training settings including optimizer, augmentation, initial learning rate are also kept the same except we modify the total epochs to 200 and the learning rate is decayed by 1/10 epoch 150, 180, and 195. The last checkpoint is used for evaluation. We use the sampling, and square-root sampling both degrade the performance.
sampling, as it was reported by Kang et al. (2019) that class-balanced
We report the top-1 accuracies on the test set in Table 3. We find that both the cosine softmax and the Grassmannian class representation with small subspace dimension improve the long-tai classification accuracy. Specifically, the cosine softmax is 1.62% higher in score compared to the vanilla softmax, and the Grassmannian class representation with subspace dimension & is 2.11% higher score compared to the vanilla softmax. However, when the subspace dimension increases, the accuracy drops. We notice that for few-shot classes, there are not enough sample to learn good higher dimensional subspace for its representation, as the accuracy of few-shot classes degrade significantly when dimension are large. Too few training data for a class an example scenario when larger dimension does not offer much help.
LIMITATION AND FUTURE DIRECTION
One problem that remains open is how to choose the optimal dimension. Currently, treat it as a hyper-parameter and decide it through experiments. Computational side, geometric optimization incurs some computational overhead since it contains SVD decomposition. This might hinder the training speed when is very large. The Grassmannian class representation allows for greater intra-class variability, but we did not explicitly promote the intra-class variability in any form. will be very interesting to explore ways to explicitly encourage intra-class variability. For example, a potential way is combine with supervised learning. We hope our work would stimulate progress in these directions.
7 CONCLUSION
In this work, we proposed to use linear subspaces the class prototype in deep neural networks. The geometric structure of the related Grassmannian fully-connected layer and the Grassmannian convolutional layer are products of Grassmannian. We optimize the subspaces using geometric optimization and provide an efficient Riemannian SGD implementation tailored for Grassmannians. We apply the new formulation large-scale image classification, feature transfer, and long-tul tail classification tasks. Experiments demonstrate that the new Grassmannian class representation is able improve performances in these settings.

---

Under review as a conference paper at ICLR 2023
REFERENCES
P-A Absil, Alan Edelman, and Plamen Koev. On the largest principal angle between random subspaces. Linear Algebra and its applications, (:128))288---- 2944, 2006.
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms matrix manifolds. In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks, In International Conference on Machine Learning, pp. 1120-1128. PMLR, 2016.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can gain more from orthogonality regu- larizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
Gary Becigneul and Octavian-Eugee Ganea. Riemannian adaptive optimization methods. In Inter- national Conference on Learning Representations, 2019. URL https: http::///peeeeeeeeee/moo net/ forum? cerum?ideee wwwuummdddrriiiii 99K7.
Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 580))2217-22229, 2013.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 1o0d-i00g discriminative compo- nents with random forests. In European conference computer vision, pp. 446-461. Springer, 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248-255, 2009. doi: 10. 10..C....2.2 520444
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1690-4699 2019.
Alan Edelman, Tomás A Arias, and Steven T Smith. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20(2):303-353, 1998.
Jihun Hamm and Daniel D Lee. Grassmann discriminant analysis: unifying view subspace-based learning. In Proceedings of the 25th international conference on Machine learning, pp. 376-383, 2008.
Mehrtash Harandi and Basura Fermando. Generalized backpropagation, \ {B} tude de cas: Orthogo- nality, arXiv preprint arXiv:1611.05927 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Zhiwu Huang, Jiqing Wu, and Lue Van Gool. Building deep networks grassmann manifolds. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailes recognition. In International Conference Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2023
Hiroyuki Kasai, Pratik Jawanpuria, and Bamdev Mishra. Riemannian adaptive stochastic gradient algorithms on matrix manifolds. In Interaational Conference on Machine Learning, pp. 2262-3271. PMLR. 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// openreview. net/ htte//ormmm?// (ooouummid-8:wpp555555555
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Mario Lezcano Casado, Trivializations for gradient-based optimization on manifolds. Advances in Neural Information Processing Systems, 32, 2019.
Mario Lezcano-Casade and David Martunez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pp. 3794 3803. PMLR, 2019.
Zhizhong Li, Deli Zhao, Zhouchen Lin, and Edward Y. Chang. Determining step sizes geometric optimization algorithms. In 2015 IEEE International Symposium on Information Theory (ISIT), pp. 1217-1221, 2015a. doi: 10.1
Zhizhong Li, Deli Zhao, Zhouchen Lin, and Edward Y. Chang. A new retraction for accelerating the riemannian three-factor low-rank matrix completion algorithm. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015b.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Bamdev Mishra and Rodolphe Sepulchre. R3me: A riemannian three-factor algorithm for low-rank matrix completion. In 53rd IEEE Conference on Decision and Control, pp- 1137-1142, 2014. doi: 10.1 10.1109/CDDC.20147009993
MMClassification Contributors. OpenMMLab's Image Classification Toolbox and Benchmark, 7 2020. URL https:///gggtt http///gtttttttt com/ open mml ab/ h00/mmclc /mmclmmassssiom..mmmmmmmmmmmmmam fication.
Rafael Müller, Simon Komblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Maximillian Nickel and Douwe Kiela. Learning continuous hicrarchies in the lorentz model of hyperbolic geometry. International Conference on Machine Learning, pp. 3779-3788. PMLR, 2018.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Madhav Nimishakavi. Pratik Kumar Jawanpuria, and Bamdev Mishra. dual framework for low-rank tensor completion. Advances in Neural Information Processing Systems, 31, 2018.
Mete Ozay and Takayuki Okatani. Training cnns with normalized kernels. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 117(40): 24652-24663, 2020.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012
IEFE conference computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
11

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buse E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019, URL http: cttp:::/pppprrs.errra....c. ccppppppppcppppppprpppppeppppp htt.://wwcoo..aaaeeeeeteeeeeeeee pdf.
Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep isometric learning for visual recognition. In International Conference on Machine Learning, pp. 7824-7835. PMLR, 2020.
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6450-6458, 2019.
Soumava Kumar Roy, Zakaria Mhammedi, and Mehrtash Harandi. Geometry aware constrained optimization techniques for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460-4460 2018.
Soumava Kumar Roy, Mehrtash Harandi, Richard Nock, and Richard Hartley. Siamese networks: The tale of two manifolds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3046-3055, 2019.
Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4136 4145, 2020.
Jiahao Su, Wonmin Byeon, and Furong Huang. Scaling-up diverse orthogonal convolutional networks by paraunitary framework. In International Conference on Machine Learming, pp. 20546-20579. PMLR, 2022.
Feng Wang, Xiang Xiang. Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1041-1049, 2017.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1265-5274, 2018.
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang. Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp.
0695-9704 2021.
Jiyyu Wang. Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. Proceedings the the IEEE/CVE conference on computer vision and paltern recognition,
pp. 11505-11515, 2020.
Satosi Watanabe and Nikhil Pakvasa. Subspace method of pattern recognition. In Proc. 1st. IJCPR, pp. 25-32, 1973.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NewrlPS 202/ Workshop on ImageNet: Past, Present, and Future, 2021.
URL www.ttttp//.... / http///pppeeeiiiieeeee net. forum? id=NG6MJnV1 6M5.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176-6185, 2017a.
12

---

Under review as a conference paper at ICLR 2023
Table Validation accuracy of Grassmannian ResNet50-D on ImageNet with different retractions. The first two rows are two implementations of Riemannian SGD, while the third row is the vanilla SGD without any manifold constraints.
Setting
Dim
Optimization
Retraction
Topl
Top5
Grassmannian
RSGD Algorithm
Move along geodesic
79.12
94.41
Grassmannian
8
RSGD Algorithm variant
Retraction via QR
79.13
94.45
Unconstrained
8
SGD
None
78.55
94.18
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017b.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 13:9422-9434 2020.
Tong Zhang, Pan Ji, Mehrtash Harandi, Richard Hartley, and Ian Reid. Scalable deep k-subspace clustering. In Asian Conference on Computer Vision, pp. 466 461-481. Springer, 2018.
Xu Zhang, Felix Yu, Sanjiv Kumar, and Shih-Fu Chang. Learning spread-out local feature descriptors. In Proceedings of the IEEE international conference on computer vision, pp. 4595- 4603, 2017.
A TECHNICAL DETAILS
Alternative Implementation of Riemannian SGD The step 4 of Algorithm called retraction in geometric optimization. There are alternative implementations of retraction other than moving parameters along the geodesic. For example, replace step 4 with the Euclidean gradient update and followed by the re-orthogonalizatoon via QR decomposition in Step 5. The subspace parameter may move away from the Grassmannian after the Euclidean gradient update, but will be pulled back to the manifold after the QR re-orthogonalizztion (details see Absil et al. (2009, Equ. (4.11))). For ease of reference, we call this version of Riemannian SGD as "Algorithm variant". We compare the two implementations in the first two rows of Table The results show that the Grassmannian class representation is effective on both versions of Riemannian SGD.
Necessity Grassmannian Formulation and Geometric Optimization To show that the neces- sity of constraining the subspace parameters to lie in the Grassmannian, replace the Riemannian SGD with the vanilla SGD and compare with Riemannian SGD, Note that with SGD, the logit formula ||S?" z=| no longer means the projection norm because S, is not orthogonal anymore. The result is shown at the third row of Table 4, from which we observe a significant performance drop for the unconstrained setting.
Numerical Stability of Algorithm The numerical stability issue is caused by the accumulation of tiny computational errors of Equation (3). After many iterations. the resultant matrix S might not be perfectly orthogonal. For example, after 100, 1000, and 5000 iterations of the Grassmannian ResNet50-D with subspace dimension k 8, we observed that the error max | m S: I| is 1.9e-5, 9.6e-5 and 3.7e-4, respectively. After 50 epochs, the error accumulates to 0.0075. One can run step 5 every 100 iterations to keep the error at low level and the computational cost is neglectable. For this reason, we marked this step as "optional".
Decreasing Feature Norm During Training We show the changes of average norm on the valida- tion set of ImageNet from epoch 10 epoch 100 in Figure 3. The subspace dimension k 16.
13

---

Under review as a conference paper at ICLR 2023
1.6
14
122
Avenge
10 20
30
40
30
60
70
90
100
Epoch
Figure 3: The average norm of features on the validation set. The error bar shows the std of norms.
Table 5:
Validation accuracy of Grassmannian ResNet50-D on ImageNet with varying 7. Different from other experiments, this set comparison was conducted using the step learning schedule.
Setting
Dim
Lr Schedule
Top1
Top5
Grassmannian
8
Step
16
78.27
93.87
Grassmannian
8
Step
20
78.55
94.05
Grassmannian
8
Step
25
78.71
94.11
Grassmannian
8
Step
32
78.47
94.07
B SYPER-PARAMENE AND DESIGN DECISIONS
Choice Gamma We use 7 = 25 throughout the main text. Here we give more results with different choice of y when subspace dimension & = 8 in Table 5. Due to that conducted this set of experiments early exploration stage, the learning rate decay policy is to divide by 10 epochs 30, 60 and 90, which is different from main results using the cosine learning rate schedule. The top-1 accuracy is slightly lower than the cosine learning rate counter part. Other training settings such augmentation the same in Table 1.
Importance of Re-Normalizing Features Re-normalizing the feature is critical to effectively learn the class representative subspaces. Below we provide training results without feature re-normalization in Table 6. There are significant performance drop without re-normalization, For reference, the cosine softmax also requires feature re-normalization for effective learning.
Importance of Joint Training Joint training the subspaces and the features essential. To support this claim, we add an experiment that only fine-tunes the class subspaces from weights pre-trained using the regular softmax (third row of Table 7). For comparison, we also add another experiment that fine-tunes all parameters (fourth row of Table 7). We find that if the feature is fixed, changing the regular to the geometric version does not increase performance noticeably (top-1 from 78. 18.04% to 78.1 14%). But when all parameters are free to learn, the pre-trained weights is better initialization than the random initialization (top-1 from 79. 12% % to 79.44%).
More Results of FN We present more results using the feature norm regularization trick in Table 8. From the results, we observe that FN also works for the baseline Cosine Softmax. For Grassmannian FN, the performance reaches peak dimension i and then decreases when k 16.
Stronger Augmentation Improves Accuracy Generally speaking, stronger augmentation miti- gates the overfitting problem and benefits models with larger capacity. To demonstrate the effect of experiments using RandAug (Cubuk et al. (2020)) in Table 9. We can see that stronger augmentation indeed further increases the accuracy. Together with longer training and SyncBN, the top-1 accuracy for ResNet50-D reaches 80. 17%.
stronger augmentations,
14

---

Under review as a conference paper at ICLR 2023
Table Compare the validation accuracy of Grassmannian ResNet50-D on ImageNet with/without
re-normalization.
Setting
Dim
Re-Normalization
Top1
Top5
Grassmannian
78.48
94.24
Grassmannian
77.91
93.78
Grassmannian
79.12
94.41
Grassmannian
8
78.12
93.90
Table 7: Validation accuracy of ResNet50-D on ImageNet trained with fine-tuning, Models of the last two rows are initialized using the weights from the first row.
Setting
Dim
Initialization
Fine- Tune
Top1
Top5
Softmax
Random
78.04
93.89
Grassmann
8
Random
79.12
94.41
Grassmann
Weights trained using softmax Weights trained using softmax
Only last layer
78.14
93.97
Grassmann
All parameters
79.44
94.58
C MORE BASELINES
We have compared the proposed method with vanilla softmax and the cosine softmax the main text. In this section we compare with baselines that use the same amount of parameters, and run experiments on different network structures.
Multi-FC
We add multiple classification fc layer to the network. During training, these independent fes are trained side by side, and their losses are averaged. During testing, the logits are first averaged, and then followed by softmax to output the prediction probability.
SoftTriple In the SoftTriple loss (Qian et al. (2019)), each class is modeled by multiple centers. The logit is a weighted average of logits computed from individual class centers. We adapted the official code into our codebase to train on the ImageNet dataset. The recommended parameters are used. Specifically, À 20, 0.1, 7 0.2 and 0.01.
For the above two settings, we use the same training protocols as Table 1. Results are shown in Table 10, from which we find that the Grassmannian class representation is the most effective one.
More Architectures We show experiments on ResNet101-D and ResNeXt (Xic et al. (2017b)) in Table 11. The training settings are the cosine learning rate schedule, and train for 100 epochs. The results show that our formulation is effective across different model architectures.
in Table 1, namely, the standard augmentation,
D TRAINING SPEED AND SVD SPEED
During inference, the computational cost is K times the vanilla softmax. Since is mostly matrix multiplication, the GPU acceleration can speed up even further. For example, on a V100 GPU, the average time of multiplying 1000 x 2048 matrix with a 2048 dimensional vector is 20 +: 2.9us, while multiplying an 8000 2048 matrix with a 2048 dimensional vector takes about 105 + 7.6 The cost is neglectable compared to the network forward time.
During training, the most costly operation in Algorithm is SVD. We measure the actual iteration time during training in Table 12. We observe that when K is small, it is as fast the vanilla softmax When & 8, the full training needs roughly 1.7x time compared to vanilla softmax (this can be reduced greatly with the new version of Py Torch, as we will discuss below).
Since the release of PyTorch 1.13, they supported the fast approximate SVD algorithm GESVDA. We saw great speed improvement in the case of and 16. The benchmark time is shown
15

---

Under review as a conference paper at ICLR 2023
Table 8: Validation accuracy of ResNet50-D on ImageNet with FN regularization.
Setting
Dim
Top1
TopS
Cosine Softmax FN
78.64
94.24
Grassmannian FN
78.65
94.24
Grassmannian FN
2
78.87
94.43
Grassmannian + FN
4
79.10
94.58
Grassmannian FN
8
79.37
94.53
Grassmannian + FN
16
79.09
94.37
Table 9: Validation accuracy of ResNet50-D on ImageNet with stronger augmentation and longer training. SyneBN also improves accuracy.
Setting
Dim
Augmentation
SyncBN
Epochs
Top1
Top5
Softmax
Standard
100
78.04
93.89
Softmax
RandAug
100
78.04
94.05
Cosine Softmax
Standard
100
78.30
94.07
Cosine Softmax
RandAug
100
78.95
94.55
Grassmannian
8
Standard
100
79.12
94.41
Grassmannian
8
RandAug
100
79.32
94.46
Grassmannian
8
RandAug
100
79.49
94.64
Grassmannian
8
RandAug
300
79.91
94.87
Grassmannian
8
RandAug
300
80.03
94.77
Grassmannian
16
Standard
100
79.21
94.29
Grassmannian
RandAug
/
100
79.53
94.58
Grassmannian
16
RandAug
300
80.17
94.93
in Table 13. With computational optimizations as such, we expect the computational cost of SVD would be minimal for k < 32.
E PYTORCH CODE FOR RIEMANNIAN SGD
We provide sample implementation of Algorithm in Figure 4 using PyTorch (Paszke et al. (2019)). The sample code checks if parameter geometric by checking whether it has an 'geometry attribute. If not, then it runs the original SGD on that parameter. If the 'geometry' property is not None, then it is list of numbers indicating the dimension class representative subspaces for all classes. If all the dimensions are the same, then goes to the batch version (line 23 of the code in Figure 4). Otherwise, it goes to the for loop version (line 46 of the code in Figure 4).
16

---

Under review as a conference paper at ICLR 2023
Table 10: Validation aceuracy of ResNet50-D ImageNet trained with Grassmannian, Multi-FC and SoftTriple. They use the same amount of parameters to represent a class.
Setting
Class Parameters
Top1
Top5
Grassmannian
k 8 dimensional subspaces
79.12
94.41
Multi-FC
8 fe classification layers
77.34
93.65
SoftTriple
8 centers for each class
75.55
92.62
Table 11: Validation accuracy of ResNet101-D and ResNeXt55 on ImageNet.
Architecture
Setting
Dim
Topl
Top5
ResNeXt50
Softmax
78.02
93.98
ResNeX150
Grassmannian
8
79.00
94.28
ResNer101-D
Softmax
79.32
94.62
ResNet101-D
Grassmannian
8
80.03
94.81
Table 12: Average iteration time (forward + backward) during training 8xV100 server. The PyTorch version used in this table is 1.11.0. The SVD step is computed on CPU due to PyTorch performance regression prior version 1.13.0.
Setting
Dim
Avg. Iter Time (ms)
Softmax
147
Cosine Softmax
150
Grassmann
1
149
Grassmann
2
145
Grassmann
4
177
Grassmann
Grassmann
16
449
Table 13: SVD time with Approximation on Nvidia GeForce GTX 1080 Ti using PyTorch 1.13.0. The tested matrix has shape (num classes, feature dimension, subspace dimension). The numbers should be smaller when running on more powerful devices such as V100.
Matrix Size
CPU (ms)
GPU (ms)
GPU+GESVDA (ms)
1000 x 2048
3.1
117.7
41.3
1000 x 2048 2
8.2
196.8
41.1
1000 x 2048 4
18.9
366.3
41.8
1000 x 2048
80.5
495.1
45.2
1000 x 2048 16
211.9
627.1
55.4
1000 x 2048 32
640.3
834.9
92.5
17

---

Under review as a conference paper at ICLR 2023
def -single_sensorr_____ ppprassoooorrpprran List[Tensor,, d_p_list: List[tensor), romentun list: [istiooto [I weight_dcccup Flost, Float, In: dosponing: float, nexieize: bool):
For: panes snuserate<(paraan)::
q_p (_S_11
if weight_decay 1a e and not hasattn(paras, "gronetry'):
d_p d_prad, alphemeeight_
if sosentun 8:
but norentur_ crrttuu__ufffe_reee [ist[i]
18
buf None: buf torch.cloneed (d_) ()
11
pueentur_bufferritttt.
11
else:
14
(step
15
buffrruu_cooeentuuuodd.d.. sipha-1 corpening)
dp but
17
18
3ipha Ir roximize cise Ir
nomentur Sartii__fffeer_ssti
28
hasattr(parar,, 'gconetry'):
22
peonetric 560 for paraseters with "geonetry" attribute
23
if (tgp.u/i(uc(orre..easst))) -- batch
25
subviin = paran-pereirr(()]
foatdir
27
# reshope
batch_p paran.reshapee-11, subdin, featdie).
20
satch satch_d_n d_pp.esshape(-1, sundta, (sstcttn).peeurraae 2, 1])
38
batch_r but rershrestpe(-1, subdte, featain), (eatutin).peruut(((e 2, 1])
gred
grad (eautioo
(13))
32
batch, g = batch_d_p. batch_F (batch_p. p.peruut(e,,, 1]) batch_d_p)
momentun
34
2, 1l) reshopppp) festein) buf)
35
# svd
(batch U, batchue, batch_it) Borch.1nnilg.ovd(aatta (ull_natrico==llee))
37
new paraseter by soving along geodesic
batch_p
39
(batch_p # batch peteehLt.. 2, 11)) torsh.cos(alpha batchhs). reshapett, 1, subdie)
betch_U + torch. sintalai reshape,,,,,,,,pe(-,,,, subdin)) betch_Vt decomposition
41
42
nou batch. warch.iinalg.ce or (nex. batch.p).0
update
44
add bnthhbattchhhhaaaaaeeaeaaaaaaeht batch (er 2, reshapephp, foatdin) paras)
clse:
general case
17
dims [8, "tp.cussuu(panee.eeeeeee
zsp(dins[:-1],, dirs[i:]):
49
# reshape
paran[e:se).T
51
d. 3.p[c:oc].T
buf[cnec].1
suclidean grad rierannian grad (equation (10) and (13))
54
d (p.T
update a_add_(8 svd
96
58
(U, 5, Vt) perch-linolg..ee svd(g, full ratricesssss))))) # parameter by moving along geodesic
59
nox torch.caz((p Vt.T, U), torch.cat(
61
(torch. wet..ddiigllttttrhh..cmmm
s)), www.dd/flaatttor.sssiiia..a >33), 6)
83
nov_p torch. Mimalg.gg(row,.....mmmm
update
p_add_(neu, P)
66
s1se:
original
68
aid, (d_p, alphamalpha))
Figure 4: Pytorch implementation of the Riemannian SGD of Algorithm 1. The function shown in this figure is the core part. To make runnable, one can copy an implementation of SGD (https: github. com/ httpp/ytoccchccccccto..tooocthht wttppppppytttooo////pppppppppctt http////ooo/nnttt//rrrrtttto opt im/ sgd. py), and replace the single tensor sgd () function with wsingnnnett. tensor regd () shown above.
18


Reviews:
[{"review_id": "vThYg5t9eS", "reviewer": "Reviewer_Ut4X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out that using softmax does not take the intra-class and inter-class feature variation into account, and this paper aims to interpret the high dimensional feature output that lies in a union of linear subspaces. In classification, each feature representation falls into one of the subspaces (K classes), where each subspace is a Grassmannian manifold. To achieve this, this paper incorporates the Riemannian SGD into the ResNet50-D backbones to optimize the network and the k subspaces. The authors have validated that such an assumption is powerful and outperforms the softmax and cross-entropy combination in ImageNet-1K classification, feature transfer, and Long-tail classification.", "summary_of_the_review": "Since the algorithm novelty of the paper is limited, I will need more details from the authors to justify whether they indeed bring new insights into this area.", "strength_and_weaknesses": "\nPros,\n\n1. Replacing the de-facto combination of softmax and cross-entropy is interesting. They provide intensive experiments to validate their claim.\n\n2. The experiments on the long-tail tasks provide a new sight to handle the data imbalance issue. It correlated to an extra memory bank or dictionary somehow.\n\nCons,\n1. The authors miss an important reference [1]. The proposed method has been employed and used in subspace clustering instead of classification. The authors should clarify clearly your contribution and difference.\n\n\n[1] Scalable Deep k-Subspace Clustering, Tong Zhang, Pan Ji, Mehrtash Harandi, Richard Hartley, Ian Reid, ACCV 2018.\n\nUnclear parts:\n1.  When there are more than two identical eigenvalues, there will be a sign flip issue in the corresponding eigenvectors. It may lead to the subspace update in a different way. Thus, I would like the authors to provide an analysis of such randomness due to subspace updating.\n\n2. In table 2, as the Grassmannian with 16 dimensions is better than 8, why there is no FN experiment on 16 dimensions? I am also wondering how the sparsity is related to the FN. Comparing dimension 8, with FN and without FN the accuracy is quite similar but sparsity changes a lot. \n\n3. The batch normalization will project the feature space to a unit sphere, which will be against the linear subspace assumption. Could the authors explain more on this direction and how you solve this issue? \n\n4. Besides, I am also wondering how data augmentation affects the accuracy since the authors did not use augmentation in their implementation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uVEm5Cwh2qy", "reviewer": "Reviewer_iWx6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The overall goal of the submission is a learning formulation that simultaneously models inter-class discrimination while promoting intra class variation in classification. To this end, it considers a linear subspace approach that is scaleless and thus in theory more suitable for long tail classification than the vector counterpart. Since set of subspaces form a Grassmann manifold, the submission replaces the fully connected layer in deep networks with a geometric one and optimizes it with Riemannian SGD. The method is validated on various benchmarks where it is shown to improve the vanilla baseline on transfer learning as well as long tail classification tasks.", "summary_of_the_review": "The approach is based on a principled framework even though the technical novelty is not very strong from a broader manifold learning perspective. The resulting gain wrt Grassmanian and vanilla baselines are interesting even though the method does not explicitly model the intra class variability in the formulation unlike some existing work in this direction which are not compared with.\n", "strength_and_weaknesses": "Strength:\n\n-Presentation: The submission is easy to read and follow. It is well written with intuitions provided where necessary. The problem is well motivated and contextualized in the broader scientific context.\n\n-Technically solid and grounded work.\n\n\n-Interesting empirical results wrt baselines considered in the submission and thus a promising direction.\n\n\nWeakness:\n\n- Technical Novelty, from Deep (Riemannian) Manifold Learning perspective, is somewhat marginal.\n\n- Experimental Validation is heavily centered around Grassmanian baselines. While this submission cites several works that explicitly encourage/promote intra-class variablity, comparison with such baselines is completely missing.\n\n- While the problem of promoting intra class variability is of great interest in deep learning, the proposed method does not explicitly model it as such. I concede it is not a strong weakness and based on the empirical findings in this work, future work can address this explicitly.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q7r8vYSZ56", "reviewer": "Reviewer_ShD7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, traditional class representative vectors deep neural networks are replaced by linear subspaces on Grassmann manifolds. It is supposed to be more informative for intra-class feature variations. The proposed method optimizes the subspaces using geometric optimization, with an efficient Riemannian SGD implementation tailored for Grassmannians. Experiments on image classification, feature transfer, and long-tail classification tasks show that the new method improves the vanilla softmax and cosine softmax.", "summary_of_the_review": "The proposed new formulation for learning linear subspaces in deep neural networks is very interesting. However, from my point of view this work is still not yet solid. As I can understand the proposed method is novel and I expect a lot from it. However, after reading the experimental results I'm not that excited, and I think the proposed method is not yet fully validated in some aspects, as listed in weaknesses. To me the only reason to accept the current paper is that I would like to encourage this novel study.", "strength_and_weaknesses": "Strength:\n\n(1) Introducing subspace learning in deep neural networks is interesting, and introducing geometric optimization with Riemannian SGD is useful to solve this problem.\n\n(2) Experimental results on three tasks show improvements over traditional softmax methods.\n\n(3) The paper is well written and organized.\n\nWeaknesses:\n\n(1) With larger k, the proposed method introduces much more parameters, not to say the SVD operation in the Riemannian SGD solver. As in Table 2, without FN only k=16 shows improvements. This would make a very difficult scalability for large-scale learning, for example, with millions of classes in training face recognition models.\n\n(2) If computation is not an issue, traditionally there are also methods in expanding the class representative vectors, e.g. multiple experts and fusion. It is not clear if the improvement is due to enlarged classification parameters or due to the new learning framework. Therefore, it would be better to show a comparison against multiple experts.\n\n(3) The FN loss contributes a lot for the improvements. However, it should be a general trick that can also be applied on the traditional softmax baselines, which should also be reported for a fair comparison. Without the FN loss it appears that the improvement of the proposed method is still limited.\n\n(4) With k=1 the linear subspaces degrade to class vectors. In such case what is the difference between the proposed method with k=1 and cosine softmax? They perform quite similar with each other across all the three tables.\n\n(5) How will the proposed method incorporate margin parameters and what would be its effect? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "jB0wt9t33r", "reviewer": "Reviewer_Acxf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to represent a class as a subspace in the deep learning regime. The contributions of this paper are the formulation of classes as subspaces, the Grassmannian layer to update subspaces, learning the Grassmannian layer using constraint optimization. The core contribution is to represent a class as a subspace where we can easily replace the softmax layer with fully-connected networks with the projection between subsapaces and the input feature. The experiments show strong performance improvement on some cases on large datasets (e.g., ImageNet). As an example of its effectiveness compared to the softmax one, the top-1 accuracy of ImageNet1K is improved by 1.3%.\n", "summary_of_the_review": "This paper has some insights about the use of linear subspaces for image classification especially comparing with the vanilla fully-connected layer in neural networks. However, there are some issues regarding novelty, comparison, and experiments. ", "strength_and_weaknesses": "Strengths:\n- The feature norm regularization is somewhat novel for training the neural network under geometry constraints.\n- This work has a strong performance in term of empirical results compared to prior methods in image classification and transfer learning.\n- The empirical results also show that the representations using the Grassmannian layer is more sparse.\n\nWeaknesses :\n- This work has unclear motivation why second order representations in the form of linear subspaces yields better performance compared to the first order representations.  There is no motivating examples nor theories when subspaces are suitable representing classes. Citing the work Watanabe and Pakvasa does not directly describe why the linear subspace approach is a better model to represent classes in the era of deep learning.  \n- The novelty of this work is marginal with many overlapping points and contributions compared to the work of Simon et al. The problems of image classification and transfer learning are covered by the work of Simon et al. that enjoys the superiority of linear subspaces over prototypes (a single vector) to represent classes in few-shot learning. In experiments,  this work does not discuss or even compare with the proposed method. \n- The proposed method updaates linear subspaces in classifiers with some constraints, and also this is not novel as some other works by Harandi and Fernando “Generalized BackPropagation Etude De Cas: Orthogonality” and Roy et al., “Siamese Networks: The Tale of Two Manifolds” have discussed similar concepts (i.e. the geometry aware layers) and implemented the proposed method for image classification but this works has no comparison to these prior works.\n- Moreover, the types of data feasibly represented using the linear subspace method are also not discussed in the paper. Is the proposed method only applicable for visual data?\n- This statement “The whole matrix S needs not be orthonormal”. For discriminative purposes, even though it needs more investigation, the straightforward idea is to force subspaces as different as possible to avoid collapses (see Simon et al., Arjovsky et al., Ozay and Okatani, and Harandi et al.). However, this work does not have any discussion about this idea nor include the idea to discriminate between subspaces.\n- There are no properties of the proposed Grassmannian class representation layers.  For instance, what are the properties and benefits preserving the orthogonality for each subspace? what are properties of not preserving subspaces coming from different to be orthogonal?\n- The design of this approach is somehow limited for a neural network module. How is the design of the proposed method with the multi-layer version of fully-connected layers (if possible)?\n- The experiments require some prior methods for comparison with some variants in class representations, e.g., prototypes (the average of all representations within a class), non-learnable subspaces (a similar concept as in Simon et al.), .\n- The performance of long-tail classification is marginally improved compared to cosine softmax.  That shows that the proposed method might not be quite effective in addressing such issue compared to transfer learning and common image classification.\n- Is there any comparison in terms of speed between softmax strategies and the Grassmanian one? The discussion of trade-off between the performance gain and the processing time is crucial for this type of method because it usually requires additional processing time especially with contstraint optimization.\n- The experiments are also lacking of comparison to some other models as a backbone. For instance, the proposed method can compare the methods using transformer models, another ResNet type (e.g., ResNet101), VGG, Inception.\n- The feature sparsity is not very clear, is that 78% zero activations on the feature before the Grasmannian layer? or the elements of the Grassmannian layer (i.e., each subspace)? or the output after the Grassmannian layer?\n\n\n\nReferences:\n\nSimon et al, “Adaptive Subspaces for Few-Shot Learning,“ CVPR, 2020.\n\nHarandi et al., “Extrinsic Methods for Coding and Dictionary Learning on Grassmann Manifolds,” IJCV, 2015.\n\nArjovsky et al. \"Unitary evolution recurrent neural networks,\" ICML, 2016.\n\nOzay and Okatani, \"Training CNNs with normalized kernels,\" AAAI, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TcJS5YkfM2", "reviewer": "Reviewer_K7X9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The last layer of a neural network trained to classify instances usually linearly projects the feature vector from the previous layer to compute log odds of a class. This paper proposes replacement of that linear projection with a norm of the projection of the feature vector into a subspace. The paper shows how to optimize all weights of such a network and studies the advantages of the representation. The authors argue that the learned features transfer better in downstream tasks.", "summary_of_the_review": "While I do have questions, I also like the discussion of the ideas in the paper. I don't regret reading it :), so I imagine others would not, either. So, unless something essentially equivalent has already been done (and I am not aware of it), then publication is justified. I am looking forward to the authors' response so I can understand the ideas and the details even better.", "strength_and_weaknesses": "The strengths: \nThe paper is clearly written and well-motivated. The method is sufficiently described for a reader's implementation. The experiments are diverse: Beyond testing accuracy on ImageNet, the paper also studies feature transfer to new datasets as well as learning from small number of examples.\n\nThe weaknesses: Some of the improvements demonstrated in experiments are fairly small (e.g. Table 1). In fact, given the variability in numbers one might get in all the experiments through small tinkering with known tricks, it is difficult to know if the demonstrated advantages would hold under further optimization for any given task (and if they are statistically significant across initializations in learning), although experiments and illustrations as a whole do paint a convincing picture. Some of the choices of the implementation are not fully explained.\n\nOverall, my first impression is positive.\n\nQuestions:\n1. Given the need for step 5 in the algorithm (orthogonalization) for numerical stability, why bother with the geodesic at all and simply do gradient descent on S followed by step 5?  Are there some illustrations of why it fails?\n2. Just looking at Eq. 6, one might ask if the orthogonal subspace basis and Grassman manifold view is really necessary, or if the benefit simply comes from the quadratic form of the logit computation (instead of linear). I.e., going beyond the previous question: Can the optimization be simply done on unconstrained S_i? Or for that matter can the logit be l_i=x' W x, with unconstrained W (gradient descent optimization of W tends to regularize it to be low rank anyhow).\n3. Given the quadratic nature of the computation, is there a relationship to tensor product representations (Smolenski et al), where such computations are done in all layers of a network? (and do you plan to move your subspace projection into earlier layers, too?)\n4. Norm regularization (12), as well as in equation 4 (to renormalize x to constant \\gamma norm) may play big roles in learning reducing the real effect of the subspace modeling (and also do you do both of these things or just (12)?)\n5. In the first ImageNet experiment, how would you account for the change in the modeling power by simply having more parameters in the last layer? \n6. In the transfer experiments, I am assuming that the issue above no longer exists, because you treat the features from the previous layer the same way (i.e. not through fine-tuned subspace projections, but using a linear classifier).  Is that right?\n7. If the above is right, then Table 2 may be slightly confusing, as results for ImageNet seem to be copied from Table 1, where logits a computed using norm of the subspace projections, but for the rest of the datasets, they are computed using linear projections. \n8. Finally, the premise of the experiments is that the joint training of the backbone and the (subspace-based) classifier results in features that are better in the ways described in the paper.  If you initialize the network trained with regular softmax or cosine softmax classifier layer, and then switch to the subspace-based layer, what happens? Can keeping the features fixed and finding good subspaces increase accuracy? Does further training of the network change the features and how? (or is this not a meaningful experiment because of the lack of the bias term in your model?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject