Maximum Class Separation as Inductive Bias in One Matrix
Authors: Tejaswi Kasarla
Gertjan J. Burghouts
Max van Spengler
Elise van der Pol
Rita Cucchiara
Pascal Mettes
Venue: NeurIPS 2022 Accept (2022)
Keywords: maximum separation
inductive bias

Abstract: Maximizing the separation between classes constitutes a well-known inductive bias in machine learning and a pillar of many traditional algorithms. By default, deep networks are not equipped with this inductive bias and therefore many alternative solutions have been proposed through differential optimization. Current approaches tend to optimize classification and separation jointly: aligning inputs with class vectors and separating class vectors angularly. This paper proposes a simple alternative: encoding maximum separation as an inductive bias in the network by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind our approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. We outline a recursive approach to obtain the matrix consisting of maximally separable vectors for any number of classes, which can be added with negligible engineering effort and computational overhead. Despite its simple nature, this one matrix multiplication provides real impact. We show that our proposal directly boosts classification, long-tailed recognition, out-of-distribution detection, and open-set recognition, from CIFAR to ImageNet. We find empirically that maximum separation works best as a fixed bias; making the matrix learnable adds nothing to the performance. The closed-form implementation and code to reproduce the experiments are available on github.

Maximum Class Separation as Inductive Bias in One Matrix
Tejaswi Kasarla University of Amsterdam
Gertjan Burghouts TNO, Intelligent Imaging
Max van Spengler University of Amsterdam
Elise van der Pol Microsoft Research Al4Science
Rita Cucchiara
Pascal Mettes University of Amsterdam
University of Modena and Reggio Emilia
Abstract
Maximizing the separation between classes constitutes well-known inductive bias in machine learning and a pillar of many traditional algorithms. By default, deep networks are equipped with this inductive bias and therefore many alternative solutions have been proposed through differential optimization. Current approaches tend to optimize classification and separation jointly: aligning inputs with class vectors and separating class vectors angularly. This paper proposes a simple alternative: encoding maximum separation an inductive bias the network by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind our approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. We outline a recursive approach obtain the matrix consisting of maximally separable vectors for any number of classes, which can be added with negligible engineering effort and computational overhead. Despite its simple nature, this one matrix multiplication provides real impact. We show that our proposal directly boosts classification, long-tailes recognition, and open-set recognition, from CIFAR to ImageNet. We find empirically that maximum separation works best as fixed bias; making the matrix learnable adds nothing to the performance. The closed-form matrices and code to reproduce the experiments are available on github
detection,
Introduction
An inductive bias of learning algorithm describes set of assumptions about the target function independent of training data (37]. Inductive biases play a vital role in the design machine learning algorithms, consider for example inductive biases for image structures (e.g., the convolution operator), symmetries (e.g., rotational equivariance), or relational structures (e.g., graph layers). Specifically for categorization, an important and long-standing inductive bias is optimal class separation. Perhaps the most well-known example of the use of this bias is Support Vector Machines, which maximize the margin of the hyperplane between samples of two classes [6]. Given many possible hyperplanes, the rationale is to select the one that represents the largest margin, i.e., separation, known as a maximum-maree margin classifier. Similarly, boosting algorithms give weights to samples depending their margin, such as AdaBoost which focuses on low margin samples [14]. In such algorithms and in many more, separation has been incorporated in their design to pull apart the classes, thereby improving generalization in classification settings to unseen data.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In deep learning- based classification, neural networks are by default not equipped with a maximum separation inductive bias. The cross-entropy loss with a softmax function a common choice train a classifier, aiming for discriminative power. Yet this framework does not explicitly drive maximum separation between the classes. Liu et al. [31 observed that networks by themselves try to decouple inter-class differences based on angles and intra-class variations based on norms. This decoupling is however limited out-of-the-box and a wide range of works have investigated optimization strategies to explicitly force classes away from each other. A common approach is to introduce additional losses which incorporate a notion of class margins, see e.g., [25] 33 M or by including orthogonality constraints [25]. Specifically, hyperspherical uniformity has shown to be an effective way to separate classes, whether it be as additional objectives in a softmax cross-entrope optimization [26 29 30 70
or through hyperspherical prototypes (3)].
This paper provides an alternative view to embed separation in deep networks with a straightforward solution: encoding maximum separation as an inductive bias in the network architecture itself with one fixed matrix. Rather than treating separation as an optimization problem, we find that the problem can be optimally solved in closed-form. The closed-form solution is given by a recursive algorithm which embeds all class vectors uniformly, such that all are equiangular and with zero mean. The solution holds for any finite number of classes and results in single matrix that can be added priori as a matrix multiplication on top of any network. The outputs of the proposed embedding are the inner product scores that feed into the softmax function, allowing to still leverage powerful objectives such as cross-entrops on softmax activations.
We demonstrate that remarkably, adding only one fixed matrix in the neural network provides compelling improvements across various classification tasks. We show that enriching ResNet with maximum separation inductive bias improves standard classification and especially long-tailed recognition. We also show that our proposal improves methods specifically designed for long- tailed recognition, highlighting its general utility. The results generalize to ImageNet and we show improvement in both shallow and deep networks. The added inductive bias helps to balance classes by design, avoiding the under-representtation of rare classes. Beyond classification, we illustrate that out-of of-distribution and open-set recognition benefit from an embedded maximum separation. Our general- pureoos solution comes with no measurable difference in runtime and can be added in one line of code, enabling it to be adopted generally. To highlight the broad applicability of our proposal, we also show how task-specific sfate-of-the-art approaches for open-set recognition benefit directly when adding the proposed maximum separation.
2 Closed-Form Maximum Separation between Classes
Classification constitutes supervised setting, where we are given training set of N examples {(x;,;)) b where x; denotes input data at index i and yı E (1...., C} denotes the corresponding label for one of C classes. Conventionally, this problem can be addressed by feeding inputs through
network, i.e., o; = Φ(xx (x)) E C The outputs denote class logits, which serve as input for a softmax activation followed by a cross-entropy loss.
Our goal is to embed maximum separation as an inductive bias into any network, so that the network no longer needs to learn this behaviour. We do so by adding a single matrix multiplication on
top of the output of network (((.). The matrix itself should contain all necessary information about maximum separation between classes. While the general problem of optimally separating arbitrary numbers of classes on arbitrary numbers of dimensions is an open problem [29] 36), we note that we only need maximum separation of C classes in particular output space. To that end, let x; =
+x(x.) ) E R C-11 now denote the output of a network to a (C - -)-dimensional output space. We
add the matrix multiplication to obtain C-dimensional class logits: o, = pT &; with P E RCC-1)xC The matrix P denotes set of C vectors, one for each class, each of (C - 1) dimensions. This matrix is fixed such that the class vectors are separated both uniformly and maximally with respect to their pair-wise angles, which we in turn embed on top of a network. Below, we outline the definition and implementation of class vectors on a hypersphere with maximum separation. In what follows, the number of classes denoted by le + 1= C. for notational convenience. Moreover, this definition is restricted matrices of the form (x++1))
2

---

10.00
V8
Hing u/
v8 2
va
(1.0)
(a) Recursive update from to 3 classes.
(b) Recursive update from to 4 classes.
Figure 1: Closed-form separation through recursion. In (a), we visualize how to obtain maximally separated vectors for three classes $1 given class vectors
the line. In (b), do the same for the step from three to four classes on $2 For any new (k: + 1) 1-tth class (red), we introduce a new k-th dimension and position the prototype along its axis (1, (1,0,.. ... 0). All other prototypes
are shrunken by a factor / 11- _ (purple) and repositioned such that their center is placed at (- O,...,0 in the hyperplane perpendicular the new class (green). Now, the matrices P2 and Ps are the matrices containing the green vectors shown in (a) and (b), respectively, as their columns. This recursive approach ensures that all classes remain of unit norm, with equal pair- wise similarity, and with zero mean.
Definition (Maximally separated matrix) For classes, Ps. = [po,.... ..o.....90 PR. denote matrix of column vectors { gk-1, such that Vigkk41137k (p.. (pis Pk) and Σ = 0.
Here, ( denotes the cosine similarity. Definition 1 ensures that all class vectors are as far away from each other as possible based on two requirements. First, the angle between any two class vectors should be the same. Equal pair-wise similarity entails uniformity as any deviation from this setup leads to classes that move closer to each other [50]. Second, the mean over all class vectors should be the origin. This constraint prevents trivial solutions such as identical embedding of all classes as well as suboptimal embeddings in terms of separation such as one-hot encodings. In short, the definition prescribes that k + classes can be optimally separated specifically when using output dimensions. To be able to construct such embedding, we first need to know what the pair-wise angular similarity is for any fixed number of classes:
Lemma 1. For maximally separated matrix Ph. Vi.jai (pt.
Proof.
With Ps maximally separated, we have Ei Pi = and hence:
0 (po. Σ p.)-=11
11
(po. (p) Σ
k<po.P1).
(I)
i=0
d=1
since all class vectors at equal angle to each other. We have (po.P1) 1 and again by extension of equidistance: Vig.ist (Pr.;)
The implication of the lemma above is that an optimal separation of l: + 1 classes the mani- fold g results in classes being separated beyond orthogonality, a direct result of using the full space and different from classical one-hot encodings, which only span the positive sub-space and

---

therefore provide only orthogonal separation. The outcome of the lemma allows to construct Pk recursively T 38).
For any finite number of classes k: + 1, the closed-form solution for Ps is given as:
P1 (1
-1) R1x2
(2)
- 11 Pk V 1 VI-
E R+x((+1)
2o Px-1
The closed-form solution for classes (P1) is trivially given +1 and I on the line. For more than 2 classes, k 2 2, we start from the solution given for k - 1 on S6-2, We add the vector for the new class Pı (1, . 0). We place the solution Pa-1 at (- K 0,. ....... : 0), the hyperplane
perpendicular to ps. where P is shrunk so that its radius is Pk are of radius and uniformly distributed, as proven below. Figure shows the construction of our embeddings for 3 and 4 classes.
1Z In this manner, all points in
Theorem 1. For any k 2 Ps is maximally separated matrix.
Proof. We proceed by induction. Note that, for k = 1, the angular similarity of the two vectors is clearly given by = 1. Now, assume the theorem holds for k i-th column vector of Pn for arbitrary n E N. Then, for & and + j. we obtain the case where either i or is and the case where neither For the first case, without loss of generality assume that the i-th element is the first, so that:
N and let (P²x )-s denote the
1 ((Pa)ss; (Pk) k
1 12
(0. (Pa-1))))))
+ V
For the second case, we derive:
1
((Pr) (Px)as)
+
(PP= (PR-1)+s)
k
k
1
1 -11
k.2
22
k'
since ((PR-1)+i, (Pa-1)as) = - by assumption. Therefore, the vectors of Ps have a pair-wise
angular similarity of -.- To prove that Ps has zero mean, again proceed by induction. Note that for Pi, the statement obviously holds. Now, assume the statement holds for k- - 1 E N, then
(
1 - k
(Pe).s=
E.= (Ph-1)) = 0,
since Ek-1 (Pk-1) by assumption.
For C -= t + classes, the solution Pa consists of C' vectors of unit length. The main idea of this work is to utilize Ps obtain class logits under fixed maximum separation, see Figure 2 For sample x,, we want its network output x; the vector of its corresponding class. Given Ps. the alignment is maximized if the network output points in the same direction as the class vector. This can be obtained by computing the dot product between both. Since we need to compute the logits for all class vectors in the final softmax activation, we simply augment the network output with single matrix multiplication: o, pT*; By enforcing maximum separation from the start, a network no longer needs to learn this behaviour, essentially simplifying the learning objective to maximizing the alignment between samples and their fixed class vectors.
S
(x:) to align with
Input
Backbone
Output
Figure
Maximum separation pipeline.

---

C-F
CIFAR-10
0.2
0.1
0.02
0.01
0.2
0.1
0.02
0.01
ConvNet
56.70
45.97
40.34
27.35
16.59
86.68
79.47
73.90
51.40
43.67
+ +This paper
57.05
46.59
40.44
28.27
18.40
86.76
79.63
75.88
55.25
48.05
+0.35
+0.62
+0.10
+0.92
+1.81
+0.08
+0.16
+1.98
+3.85
+4.38
ResNet-32
75.77
65.74
58.98
42.71
35.02
94.63
88.17
83.10
68.64
56.98
+ +This paper
76.54
66.01
60.54
45.12
38.85
95.09
91.42
88.16
77.02
69.70
+0.77
+0.27
+1.56
+2.41
+3.83
+0.46
+3.25
+5.06
+8.38
+12.72
Table Adding maximum separation as inductive bias on CIFAR-10 and CIFAR-I0U for stan- dard and imbalanced settings. Both the AlexNet and ResNet architectures benefit from an embedded maximum separation, especially when imbalance increases and networks are more expressive.
We note that the norm of the class vectors is not taken into account during their construction as we operate on the hypersphere. We can optionally include a radius hyperparameter which controls the norm of the class vectors, i.e., the network output is given as: o; = pP &; with hyperparameter. In practice however, the radius of the class vectors of little influence to the task performance and we set it to I unless specified otherwise.
3 Experiments
Implementation details. Across our experiments, we train a range of network architectures including AlexNet (22), multiple ResNet architectures [17 677, and VGG32 477, as provided by PyTorch [40]. All networks are optimized with SGD with a cosine annealing learning rate scheduler, initial learning rate of 0.1, momentum 0.9, and weight decay 5c-4. For AlexNet use a step learning rate decay scheduler. All hyperparameters and experimental details are also available in the provided code.
3.1 Classification and long-tailed recognition
Classification with maximum separation. In the first set of experiments, we evaluate the potential of maximum separation as inductive bias in classification and long-tailed recognition settings using the CIFAR-100 and CIFAR-10 datasets along with their long-tailed variants [.]. Our maximum separation is expected to improve learning especially when dealing with under-represented classes which will be separated by design with our proposal. We evaluate on a standard ConvNet and a ResNet-32 architecture with four imbalance factors: 0.2, 0.1. 0.02, and 0.01. We set p 0.1 for ResNet-32 as this provides a minor improvement over p = 1. The results are shown in Table [ We report the confidence intervals for these experiments in the supplementary materials. For an Nee style ConvNet, classification accuracy improves in settings, especially when imbalances is highest. With more expressive network such ResNet, find that embedding maximum separation provides a bigger boost in performance. The higher the imbalance, the bigger the improvements; on CIFAR- 100 the accuracy improves from 35.02 to 38.85, improvement of 12.72 percent point. For further analysis, we have investigated the relation between accuracy improvement and train sample frequency. In Figure 3 we show the results over the imbalance factors, highlighting that our proposal improves all classes as imbalance increases, especially classes with lower sample frequencies.
CIFAR- 10 from 56.98 to 69.70, an
In the supplementary materials, we furthermore report the Angular Fisher Score as given by Liu el al. [32] This score evaluates the angular feature discrininativeness of the trained models. We also compare our closed-form fixed matrix to the optimization based fixed matrix of Mettes er al. (36). These results show that the closed-form maximum separation is more discriminative than optimization- based separation.
Should maximum separation be fixed inductive bias? Separation in this work takes the form of a fixed matrix multiplication that can be plugged on top of any network. But is the information in this matrix as inductive bias optimal for classification? To test this hypothesis, we compare the previous results on CIFAR- 100 and CIFAR-10 and their long-tailed variants [TT] to two baselines. The first baseline makes the matrix P learnable and our proposal acts as an initialization of the learnable matrix. Intuitively, if the results improve by making the matrix learnable instead of fixed, there is more information that is not captured in the closed-form solution. The second baseline
5

---

adds a learnable fully-connected layer with standard random initialization instead of a fixed matrix multiplication, to investigate whether our obtained improvements are not simply a result of extra computational efforts.
The results are shown in Figure On both datasets across all imbalance factors, embedding maximum separation as fixed constraint in a network works best. Making the matrix learnable actually decreases the performance, indicating that further optimization discards important information and highlighting that separation is an inductive bias we should not steer away from. Simply adding an extra learnable layer with standard initialization is not effective, which indicates that improvements are not guaranteed when adding more fully- colle-coneettd layers. We conclude that maximum separation best added in a rigid manner as strict constraint for networks adapt to.
Enriching long-tailed recognition approaches. So far, we have shown that conventional network architectures are better off with maximum separation inductive bias for classification and long- tailed recognition. In the third experiment, we investigate the potential of adding our proposal on top of methods that are specifically designed for long-tailed recognition. We perform experiments on two methods, LDAM [8] and MiSLAS [69]. LDAM addresses learning on imbalanced data through aabelldistribution-awwee
margin loss which replaces the conventional cross-entropy loss. We can easily augment LDAM by placing the matrix on top of the used backbone. Their hyperparameter 1 LDAM set 0.5 when using their method as is and set to 0.4 when augmented with our the-artt art approach for long-tailed recognition that improves two-stage methods by means of label-aware smoothing and shifted batch normalization. We again plug in our approach in the provided code simply as fixed matrix multiplication on top of the used backbone.
proposal as this maximizes the scores for both settings. MiSLAS is a recent state-of oate--of-hhe-ar
In Table we report the results on CIFAR-100 using a ResNet-32 backbone across three imbalance factors. For LDAM, we find that both for the SGD and the DRW variants, adding maximum separation on top improves the accuracy. For imbalance factor 0.01 for LDAM, we improve the accuracy from 39.87 to 42.02 with the SGD variant and from 42.37 43.19 for the DRW variant. The recent MiSLAS approach to long-tain ang-tailed recogmition also benefits from maximum separation after both stages. The most competitive results after their stage are improved by 1.59, 0.83, and 0.42 percent point for respectively imbalance factors 0.1, 0.02, and 0.01. The long-tailed basclines are specifically designed to address class imbalance but additionally benefit from maximum separation across the board. We conclude that maximum separation as a fixed inductive bias complements task-specific approaches, highlighting its general applicability and usefulness.
ImageNet experiments. To highlight that our proposal is also beneficial with many classes and deeper networks, we provide results on ImageNet for two ResNet backbones in Table 3 With a ResNet-50 backbone, we obtain improvements of 1.6 percent point on ImageNet and 3.5 percent point on its long-tailed variant in top accuracy. With a deeper 1 the improvements are 0.6 and 1.4 percent points for the standard and loigltailed benchmarks. We also find that the top 5 accuracy improves from 92.4 to 94.9 with ResNet-50 and from 94.3 to 95.1 with ResNet-152. We conclude that the proposed inductive bias in matrix form is also useful for larger-scale classification.
imiselacee 0.2
imfudlnce 0.1
infaslance 0.02
imbalance 0.01
15
500 1211 2. 201 240 2011 1126
200 250 11N Images per class
200
Figure 3: Effect of maximum separation on class accuracy as a function of sample frequency. We show the per class accuracy improvements when adding separation on CIFAR-10 with a ResNet-32 for imbalance factors (from left to right) 0.2, 0. 1, 0.02, 0.01. The accuracy increases over all classes as imbalance increases, especially classes with lower sample frequencies.

---

100
100
80
Accuracy
60
Accuracy
60
40
Fixed (This paper)
20
Learnable (This paper)
Learnable (Random)
None
0.2
0.1
0.02
0.01
None
0.2
0.1
0.02
0.01
Imbalance factior
Imbalance factor
(a) CIFAR-IO
(b) CIFAR- 10.
Figure 4: Should maximum separation be embedded as fixed inductive bias? On both CIFAR- 100 and CIFAR-10, 10, we find that embedding a fixed separation (orange) works best, outperforming baselines that seek to further optimize matrix initialized with maximum separation and that simply add another learnable matrix initialized like any other layer. Maximum separation should be computed a priori and kept fixed in the network architecture.
Imbalance factor
Imbalance factor
0.1
0.02
0.01
0.1
0.02
0.01
LDAM-SGD
55.05
43.85
39.87
MiSLAS (stage 1)
58.36
44.69
40.29
This paper
57.72
45.14
42.02
Thiis paper
59.63
45.65
40.56
+2.67
+1.29
+2.20
+1.27
+0.96
+0.27
LDAM-DRW 57.45
47.56
42.37
MiSLAS (stage 2)
61.93
52.53
48.00
+ This paper
58.37
48.02
43.19
+ This paper
63.52
53.36
48.42
+0.92
+0.46
+0.82
+1.59
+0.83
+0.42
Table 2: Enriching imbalanced learning algorithms with our maximum separation CIFAR-
100 for imbalance factors 0.1, 0.02, and 0.01. We perform experiments with two variants of LDAM of ( and with the recent MiSLAS of (69] after both stages of their approach. All results are obtained by running the author-provided code. Across all methods and imbalances factors, we find that maximum separation inductive bias provides consistent boost in accuracy.
On feature dimensionality and scaling. We set the last layer feature dimension 1024 depending on the dataset. The size of the learnable last layer is D x k. On the top of this we add our fixed matrix of size parameters, there is no noticeable difference between the standard setup and our maximum separation formulation. With many classes however, the fixed final matrix will be large, which can lead to extra computational effort. At the ImageNet scale (1,000 classes) training and inference times are similar, but we have yet to investigate extreme classification cases.
512 or
+ to get class logits. This means that in terms of learnable
3.2 Out-of-distribution detection and open-set recognition
We also investigate the potential of our proposal outside the closed set of known classes. To that end, we perform additional experiments on out-of-distribution detection and open-set recognition.
distif-distribution with maximum separation. Intuitively, maximum separation and uniformity between classes allows for more opportunities for samples outside the distribution of known classes to fall in between the spaces of class vectors, especially since all classes are positioned beyond orthogonal to each other as per Lemma [] We evaluate this intuition first on out-of-distribution detection. Following Liu et.al. (28] we use CIFAR- 100 in-distribution set and experiment with SVHN and Placed 365 as out-of-distribution sets. Using the same experiment settings reported their paper, we train on WideResNet and compare the out-of-distribution performance the common
7

---

Resnet-50
Resnet-152
top 1
top 5
top
top 5
Basc
+ Ours
Basc
+ Ours
Basc
Ours
Basc
Ours
Imagenet 73.2
74,8
92.4
94.9
77.9
78.5
94.3
95.1
Imagenet-L.
43.8
47.3
70.4
73.6
48.3
49.7
73.9
74.8
Table 3: Evaluations ImageNet and ImageNet-LT for two ResNet architectures. Both the top
1 and top 5 accuracies improve on ImageNet when embedding maximum separation on top of the architectures, with further boost for the long-tailed ImageNet variant[
Metric
w/o maximum separation
w/ maximum separation
FPR95
AUROCT
AUPR
FPR95 L
AUROC f
AUPR 1
Softmax Score
84.00
71.40
92.87
83.04
75.58
94,59
SVHN
Energy Score
85.76
73.94
93.91
78.86
85.42
96.92
Mahalanobis
44.02
90.48
97.83
35.88
91.45
97.93
Softmax Score
82.86
73.46
93.14
83.08
73.53
93.54
Places 365
Energy Score
80.87
75.17
93.40
81.36
76.01
93.64
Mahalanobis 88.83
67.87
90.71
89.16
69.33
91.49
Table 4: Out-of-distribution detection on CIFAR-100 with and without maximum separation embedded in Wide ResNet, using SVHN and Placed 365 as out-of-distribution datasets. On SVHN we observe improvements across all matrics and measures. On Places 365, cuttof-distribution detection is better for the AUROC and AUPR metrics, but not the FPR95.
metrics FPR95, AUROC, and AUPR. We determine whether sample in- or out-of-distribution using three scoring functions. The first directly uses the maximum softmax probability to distinguish in- and out-of-distribution samples [19]. The second is the energy score as introduced in Liu er al. (28), which classifies an input as outttf-disttibution if the negative energy score is smaller than threshold value, The third is the confidence score based on Mahalanobis distance from Lee et al. [23
The results are shown in Table 4 With SVHN as out-of distribiibttinn set, embedding maximum separation inductive bias improves all metrics across all out-of-distribution detection methods. Espe- cially the energy score benefits from maximum separation with an improvement of 6.90 in FPR95, 11.48 in AUROC, and 3.01 AUPR. Places 365, the results are overall closer. Maximum separation improves on the AUROC and AUPR metrics but not the FPR95 metric. We observe that adding maximum separation performs best for near out-of-distributie data. In Figure we provide further analyses the effect of maximum separation for scores, highlighting that separation increases the gap in energy scores between both sets, making casier to discriminate both. We conclude that a maximum separation inductive bias is valuable for out-of-distribute detection, especially for near out-of-distribution data.
detection with energy
Open-set recognition with maximum separation. We also investigate the potential of maximum separation for open-set recognition, which can be viewed as a generalization of -ut-of-distribution detection 335 To take our proposal to the ultimate test, start from the recent state-of-the-art work of Vaze el al. (55). We experiment with adding our proposal to two approaches outlined in their work. The first applies various training tricks and hyperparameter tuning to improve the closed-set classifier, followed by open-set recognition using the maximum softmax probability (MSP+) score. The second performs the same training with the Maximum Logits Score (MLS) before the softmax activations to differentiate closed- from open-set examples. All experiments are run on author protore-roviied code on four benchmarks: SVHN, CIFARIO, CIFAR + 10, and CIFAR 50, where we use the 5 splits and report the average scores as outlined by Vaze et al. [55]
The results of the open-set experiments are shown in Table with MSP+ and the state-of-th -ttt-arttt MLS scores, maximum separation is positive addition. This result follows the intuition of the original approach that improving the closed-set classifier here by means of the extra inductive bias is also beneficial for open-set recognition. We additionally visualize the 2d features of the last layer for our approach in the supplementary materials. The visualization shows
with the AUROC metric. For both the

---

in ditributtion (w/o seporation) teparation)
0.15
in dici busion (w/o separation) separation)
0.15
out of dieribution(w// oearattin)) out-of- diteddittbtuiin separation)
out of ditrizationn/// sepanttio)) out-of ofdiiitiiuution(/ seporation)
Froquency
Frequency
0.10
0.10
0.05
0.05
0.00
0.00
10
20
30
40
10
20
30
40
Energy Score
Energy Score
(a) SVHN OOD.
(b) Places365 OOD.
Figure 5: Energy scores for citoo--distribution detection. We plot the inverse of the energy scores for ease of visualization. With SVHN out-of-distribution dataset for CIFAR- 1-AAR-100 as in-distribution dataset, we find that embedding maximum separation increases the gap in energy scores between both datasets, making it casier to differentiate between both. With Places 365 as out-of-distribution dataset, the energy scores are affected less, resulting in smaller improvements with maximum separation.
SVHN
CIFARIO
CIFAR + 10
CIFAR + 50
MSP+
95.94
90.10
94.48
93.58
+ This paper
96.22
91.05
95.57
94.31
+0.95
+1.09
+0.73
MI.S
97.10
93.82
97.94
96.48
+ This paper
97.58
95.30
98.33
96.74
+0.48
+1.48
+0.39
+0.26
Table 5: Open-set recognition on four benchmarks with and without maximum separation building on the experimental settings of Vaze et.al. (55). Whether using the maximum probability or the recently proposed maximum logit to perform open-set recognition, fixed maximum class separation helps differentiate closed-set from open-set samples.
that, as training progresses, the norm of the known classes gets bigger than the norm of the open-set classes, making differentiation between both feasible.
Related work
In deep learning literature, the principle of separation has been investigated from several perspectives. Separation can be achieved by enforcing or regularizing orthogonality. For example, Li et al. [25) propose to regularise class vectors orthogonal each other. Orthogonality also often used as
regularisation for intermediate network layers, see e.g., B 42 59
64 65]. Orthogonal constraints
are fixed and easy to obtain by design. For classification specifically however, orthogonality does not result in maximal separation between classes, as only the positive sub-space is utilized. We strive for a solution that makes use of the entire output space.
Beyond orthogonality, multiple works improve class separation by promoting angular diversity between classes. common approach do so is by incorporating class margin losses 12 32
58
68). For example, the generalized large-margin softmax loss enforces both intra-class compactness and inter-class separation [33]. These works are in line with the observations of Liu al. [3] that deep networks have a natural tendency to break classification down into maximizing class separation and minimizing intra-class variance. In particular, separation has in recent years been successfully tackled by adopting the perspective of hyperspherical uniformity. Such uniformity can for example be optimized in deep networks by minimizing the hyperspherical energy between class vectors [26]
29 30 16] Recently, Zhou ef al. (70) extend this line of work by learning towards the largest margin with zero-centroid regularization. Hyperspherical uniformity can also approximated by minimizing the maximum cosine similarity between pairs of class vectors [1 36 61]. In this work.

---

we also start from hyperspherical uniformity and deviate from current literature on one important axis: hyperspherical uniformity does not require optimization. We show and prove that maximum separation has closed- solution and can be added as fixed matrix to any network. Mroueh et al. (38] also show that such closed-form solution is beneficial when added SVM loss functions and mention it could be generalized to other loss functions. Having maximum separation fixed inductive bias does not require additional hyperparameters and comes with minimal engineering effort, lowering the barrier towards broad integration in the field.
Our work fits into a broader tendency of incorporating inductive biases into deep learning, for example 51
by incorporating rotational symmetries [O 24
46
49
54 [57 63], scale equivariance 4%]
62
gauge equivariance [9] [8], graph structure
20 56 physical inductive biases [2] 41
symmetries between reinforcement learning agents S 27 53 60), and visual inductive biases
45
52 66
Embedding maximum separation as inductive bias is complementary to the listed works.
Relation to Neural Collapse Neural Collapse is an empirical phenomenon that arises in the features of the last layer and the classifiers of neural networks during terminal phase training [39] In neural collapse, the class means and the classifiers themselves collapse to the vertices of simplex equiangular tight frame. From these insights, recent work by Zhu el al. [71] fix the last layer classifier with a fixed simplex of size (k + 1) x d with d >= (k + 1) for k: + 1 classes. Our work reaches a similar conclusion from the perspective of maximum separation and we find its potential also reaches long- maillt and out-of-distribution detection. Our recursive algorithm is more compact size (k + 1) x & and we plug our approach on top of any network architecture, rather than replace the final layer as done in (71]. Due to the recursive nature of our approach, there is also potential for incremental learning with maximum separation.
Conclusions
This paper strives to embed a well-known inductive bias into deep networks: separate classes maximally. Separation is not optimization problem and can be solved optimally in closed-form. We outline a recursive algorithm to compute maximum class separation and add as a fixed matrix multiplication on top of any network. In this manner, maximum separation can be embedded with negligible effort and computational complexity. To showcase that our proposal is an effective building block for deep networks, we perform various experiments classification, long-tailed recognition, out-of- of-distribution detection, and open-set recognition. We find that our solution to maximum separation as inductive bias (i) improves classification, especially when classes are imbalanced, (ii) is best treated as fixed matrix, (iii) improves standard networks and task-specifie state-of of-the-art algorithms, and (iv) helps to detect samples from outside the training distribution. With separation decoupled and solved prior to training, a network only needs to optimize the alignment between samples and their fixed class vectors. We conclude that our one fixed matrix is an effective, broadly applicable, and easy-to-use addition to classification in networks. Our approach is currently focused on multi-class settings only, where exactly one label needs to be assigned to each example. Maximum separation does also not naturally generalize to zero-shot settings, as such settings require that classes are represented by semantic vectors that point in similar directions based on semantic similarities. While focus the classification supervised setting, maximum separation has potential for unsupervised learning as well as self-supervised learning paradigm. As the matrix imposes geometric structure in the output space it could potentially help in improving the uniformity of classes.
Acknowledgements
This work is financially supported by the ELLIS Amsterdam Unit.
References
[1] http://ww...elllooooooggg/n/////
n-sphere/7147811 147 4
[2] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. NeurlPS, 2019. 10
10

---

[3]
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations training deep networks? NeurIPS, 2018. y
[4] Peter Battaglia, Jessica B. Hamrick, Vietor Bapst, Alvaro Gonzaez, Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv, 2018. TO
[5]
Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In ICLR, 2020. I00
[6] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, 1992.
[7] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J. Bekkers, and Max Welling. Geometric and physical quantities improve E(3) equivariant message passing. ICLR, 2022. TO
[8] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distrbbution-awar margin loss. In NeurIPS, 2019.
6 7
[9] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In ICML, 2019. I0
[10] Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In ICML, 2016.
10
[11] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
S g
[12] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[13] Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting eyclic symmetry in convolutional neural networks. In International conference on machine learning, pages 1 1889-1898. PMLR, 2016. TO
[14] Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. JCSS, 1997. [
[15] Mina Ghadimi Atigh, Martin Keller-Ressel, and Pascal Mettes. Hyperbolic busemann learning with ideal prototypes. NeurlPS, 2021. 9
[16] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In ICML, 2021.
9
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
5
[18] Lingshen Hc, Yiming Dong, Yisen Wang, Dacheng Tao, and Zhouchen Lin. Gauge equivariant
transformer. NewrlPS, 2021. 10
[19] Dan Hendryeks and Kevin Gimpel. baseline for detecting misclassified and -i-tt--dittiitttion examples in neural networks. ICLR, 2017. g
[20] Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. In ICLR, 2020. 10
[21] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for
molecular graphs. ICLR, 2020. 10
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 2012. 5
[23] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-tf-distributiin samples and adversarial attacks. NewrIPS, 2018. g
[24] Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. NewrIPS, 2018
I0

---

[25]
Jing-Hao Xiaoxu Li, Dongliang Chang, Zhanyu Ma, Zheng- theng-Hua Tan, Jing- Hao Xue, Jie Cao, Jingyi Yu, and Jun Guo. Oslnet: Deep small- -ample classification with an orthogonal softmax layer. TIP.
2020.
9
[26]
Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James M Rebg, Li Xiong, and
Le Song. Regularizing neural networks via minimizing hyperspherical energy. In CVPR, 2020.
2
[27] Iou-Jen Liu, Raymond A. Ych, and Alexander G. Schwing. PIC: Permutation invariant critic for multi-agent deep reinforcement learning. CoRL. 2019. TT
[28] Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-of-distribttion
detection. NeurlPS, 2020. 7
[29] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. NeurIPS, 2018. 2 9
[30] Weiyang Liu, Rongmei Lin, Zhen Liu, Li Xiong, Bernhard Schölkopf, and Adrian Weller, Learning with hyperspherical uniformity. In AISTATS, 2021.
2
[31] Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang, James M Rehg, and Le Song. Decoupled networks. In CVPR, 2018. 2
[32] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017. 5
[33] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In ICML, 2016.
2
[34] Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. NeurlPS, 2017. 2
[35] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
[36] Pascal Mettes, Elise van der Pol, and Cees G M Snoek. Hyperspherical prototype networks.
NeurlPS, 2019. 2 9
[37] Tom M Mitchell. The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research .... 1980. !
[38] Youssef Mroueh, Tomaso Poggio, Lorenzo Rosasco, and Jean-Jeacques Slotine. Multiclass
learning with simplex coding. Advances in Neural Information Processing Systems, 25, 2012.
10
[39]
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40):4452-2466)
2020. 10
[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. NeurlPS, 2019.
[41] David Pfau, James S Spencer, Alexander GDG Matthews, and W Matthew C Foulkes. Ab initio solution of the many-electron schrödinger equation with deep neural networks. Physical Review Research, 2020. 10
[42] Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In ICCV, 2021. 9
[43] Pau Rodríguez, Jordi Gonzalez, Guillem Cucurull, Josep M Gonfaus, and Xavier Roca. Regu- larizing cnns with locally constrained decorrelations. ICLR, 2017. 99]
[44] Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 2018. I0
[45] Zenglin Shi, Pascal Mettes, Subhransu Maji, and Cees G M Snoek. On measuring and controlling the spectral bias of the deep image prior. IJCV, 2022. I0
[46] Gregor NC Simm, Robert Pinsler, Gábor Csányi, and José Miguel Hermandez-Lobato. Symmetry-aware actor-critic for 3d molecular design. arXiv, 2020. I0
12

---

[47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale
image recognition. ICLR, 2015. 5
[48] Ivan Sosnovik, Michal Szmaja, and Amold Smeulders. Scale-equivariant steerable networks.
arXiv preprint arXiv: 1109,, 2019. TO
[49] Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. In International Conference on Machine Learning, pages 6086-6095, PMLR, 2019. [T]
[50] Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit on the surface of collen-grains Recueil des travaux botaniques néerlandais, 1930. [
[51] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and cunslation-eqiivriinn neural networks for 3d point clouds. arXiv, 2018. I0
[52] Nergis Tomen, Silvia-Laura Pintea, and Jan Van Gemert. Deep continuous networks. In ICML, 2021. 10
[53] Elise van der Pol, Herke van Hoof, Frans A Olichoek, and Max Welling, Multi-agent mdp homomorphic networks. ICLR, 2022. [T]
[54] Elise van der Pol, Daniel E. Worrall, Herke van Hoof, Frans Olichoek, and Max Welling. MDP homomorphic networks: Group symmetries in reinforcement learning. In NeurIPS, 2020. 10
[55] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need? In ICLR, 2022. g g
[56] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
TO
[57] Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant g Icarning in spatial action spaces. In CoRL, 2022. I0
[58] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and
Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In CVPR, 2018. /
[59] Jayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional
neural networks. In CVPR, 2020. /
[60] Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In ICLR, 2018. I0
[61]
Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles. NewrlPS, 2020. 9
[62] Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. NeurIPS, 2019.
10
[63] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. In CVPR, 2017. TO
[64] D. Xie, Xiong, and S. Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In
CVPR, 2017. g
[65] Pengtao Xie, Yuntian Deng, Yi Zhou, Abhimanu Kumar, Yaoliang Yu, James Zou, and Eric P.
Xing. Learning latent space models with angular constraints. In ICML, 2017. 9
[66] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced
by exploring intrinsic inductive bias. NeurIPS, 2021. TO
[67] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016. 5
[68] Yutong Zheng. Dipan K Pal, and Marios Savvides. Ring loss: Convex feature normalization for face recognition. In CVPR, 2018. 9
[69] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In CVPR, 2021. 7
[70] Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, Xin Gao, and Xiangyang Ji. Learning towards the largest margins. In ICLR, 2022. 2 9
[71] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. NeurlPS, 2021. 10
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yPm3kip21hv", "reviewer": "Reviewer_xgNV", "summary": "The paper proposes to integrate a closed-form structural prior in deep learning architectures, which has a clear geometric interpretation. For any neural network designed for classification, hence having a final softmax layer, the contribution takes the form of a fixed (non-learnable) matrix multiplication before the computation of the softmax. For a classification problem with K classes, the construction of the matrix consists of deriving K vectors on a (K-1)-dimensional hypersphere that are maximally separated. The authors provide a simple recursive procedure to construct such a matrix that only needs to be called once before training. The proposed contribution is evaluated on different learning tasks, showcasing its relevance in various common benchmarks.", "questions": "Since the authors experimented with various learning tasks to showcase the relevance of their contribution, it could be of interest to additional information on the limitation of the proposed contribution.\nCan the authors provide examples showcasing such limits ?", "limitations": "The authors stated that their work is limited to supervised settings, and cannot be applied to settings that require relational information between classes. However, in the case of unsupervised learning with variational-auto-encoders, could such a matrix be integrated at the top of the encoder for better latent space covering ?\nAnswering such a question could additionally broaden the significance of the contribution, since performing clustering in the latent space of an auto-encoder can provide additional information that can be very valuable in a few-shot learning setting.\nGenerally speaking, even if the construction of the matrix is somehow tied to a classification problem with $K$ classes, the fact that the matrix impose a geometric structure in the resulting output space should also be beneficial for unsupervised learning.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is to provide a simple geometric interpretation of maximum class separation within the context of neural architecture design. Contrary to many engineering tricks used in deep learning to stabilize training or improve performances, the presented contribution does not contain any hyperparameter that needs to be fine-tuned. The originality and quality of the proposed contribution lies in the fact that the authors addressed the question of maximum separation from a theoretical point of view, and proves that their matrix design is relevant when followed by a softmax activation function. The recursive procedure used to construct the matrix closely resembles an algorithm that could be used to build an orthonormal basis for a vector space, except that in the presented case, the class vectors are required to have a fixed pairwise scalar product (equal to $-\\frac{1}{K}$) instead of being orthogonal to each others. The experiments are well-chosen to prove the significance of the contribution across different challenging learning tasks, and authors focus on proving that their contribution improves on many baselines, rather than seeking for a single state-of-the-art metric in a very specific setting. Additionally, the authors shows that making their matrix learnable, either with their initialization or with random initialization, can even degrade performances, thus showing the optimality of their contribution to a certain extent. Such paper shows that it is possible to make consistent improvements in neural architectures when the chosen operations are motivated by strong theoretical arguments, somehow deviating from the traditional viewpoint that neural networks are differentiable hence optimizable black box functions. One criticism that could be made is that settings where the proposed contribution falls short are only mentioned in one sentence at the conclusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cZ1jwnDsIJK", "reviewer": "Reviewer_BhPp", "summary": "In this paper, the author explores how to achieve maximizing class separation by directly adding a pre-calculated layer into the deep neural networks. The weights of the layer are solved in closed-form by a recurve algorithm. The effectiveness of the maximally separable vectors is verified on various datasets from long-tailed recognition, out-of-distribution and open-set recognition tasks.", "questions": "- As shown in Table.1, the performance gap of adding maximum separation layer over baseline enlarges on more imbalanced data. However, the phenomenon does not hold in Table. 2. More explanations are needed.\n- Why the proposed methods can benefit OOD detection and OSR recognition? More explanations and evidences are needed.\n", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The topic of maximum class separation is interesting.\n- The experimental evaluation is adequate.\n- The paper is well written and easy to follow.\n\nWeaknesses\n- Limited novelty and contribution. To my knowledge, the maximum class separation has been discussed by a few works [1-2]. Moreover, the proposed method is similar with [1], and the matrix estimation in [1] does not need recursive calculation.\n- Lack of insights from the experiments. The experiments mainly focus on the performance evaluation. \n\n[1] Do We Really Need a Learnable Classifier at the End of Deep Neural Network.\n\n[2] Dissecting Supervised Contrastive Learning.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Og87ZAxhqc", "reviewer": "Reviewer_bBZR", "summary": "This paper proposes a closed-form design of the classifier layer, which encourages the maximal separation between any two classifiers on a hypersphere. The design is simple and straighforward, and it introduces a recursive algorithm to place $k+1$ classifiers in $k$ dimensional space. The experimental results demonstrate effectiveness in standard visual recognitio, long-tailed recognitio, OOD detection and open-set recognition.", "questions": "- My major questions are given in the \"Strengths And Weaknesses\" section.\n\n- Some visualization on the learned features could really improve the paper. I am also curious about how the learned features will be given a fixed set of classifiers.\n\n- While the classifiers are no longer learned, I am wondering whether the convergence performance will also be improved (since less number of parameters are needed to learn)? Some experiments on convergence speed (eg, iteration vs. testing accuracy) would be nice.", "limitations": "For technical limitations, I have discussed them in the \"Strengths And Weaknesses\" section. For potential negative societal impact, I am unaware of any.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper is well written and structured. I generally enjoy reading this paper and find this idea quite interesting. Placing static classifiers and encourage their maximal separability is very natural and well motivated. The inductive bias from hyperspherical uniformity is encoded in the classifier layer which is used to generate corresponding gradients to update the whole network. Due to such a design, I think the learned features should also be quite discriminative (maybe with some sort of angular margin). I would suggest the authors to conduct some visualizations or Fisher discrimination analysis (say angular Fisher score in Appendix E of [1]) to demonstrate such advantages in a more intuitive way.\n\n- The resursive algorithm is neat and efficient. I like its simplicity and the theoretical analysis that motivates it. The analysis is intuitive and should be correct as far as I'm concerned. I am wondering how it compares to gradient-based optimization. For example, you can directly minimize a matrix's hyperspherical energy to obtain its maximally separable column vectors, and then keep them fixed during the network training. Although it may not be the optimal design, I am curious how it compares empirically to the proposed method.\n\n- I believe the static design of the classifier layer is of sufficient interest to the community and is of sufficient novelty, because it may be a potential solution to both encode better inductive bias and address the computational difficulty of training million-level number of classes. Besides the proposed static design, I would point the authors to a probabilistic design (in Section 6.4 of [2]) that also pre-specifies a set of fixed classifiers (which are randomly initialized) and then learns an orthogonal matrix that applies to them. I find the same idea can also be used in this paper, meaning that you learn an orthogonal matrix (using the same way as [2]) just to rotate/reflect these maximally separable classifiers. This does not change their pairwise distance / similarity, which means their hyperspherical energy stays the same. I think it will further improve the proposed method by granting it more flexibilty (while their maximal separability does not change).\n\n- The paper demonstrates the effectiveness of the hyperspherical uniformity inductive bias in a number of recognition experiments. Its generalization ability is well verified. To me, the experiments should also be easily reproducible.\n\n\nWeaknesses:\n\n- While I like the general idea of static classifier design, the proposed method has an obvious weakness: the dimensionality of classifier scales linearly with the number of classifiers. This may be okay for benchmarks like CIFAR and ImageNet, but it still constrains its practical usage in large categorical training (say extreme classification). I can understand this particular algorithm design (ie, $k+1$ classifiers in $k$-dimensional space) is due to the fact that static assignment of maximally separable classifiers is very challenging when the dimensioanltiy is way smaller than the number of classifiers. I think this limitation, as a future direction to study, should be explicitly discussed in the paper.\n\n- I appreciate the simplicity of the static classifier design, but the lack of flexibility (learnability) may limit its performance. Learning an orthogonal transformation (or even simpler, learning a rotation matrix) can be beneficial, as I mention in the third point in Strengths.\n\n- As Lemma 1 shows, the inner product approaches zero as $k$ becomes larger. Does it mean that we can use orthogonal matrix (which is $k\\times k$ instead of $k+1 \\times k$) to replace the maximally separated matrix? It seems to be an interesting conclusion that one can take advantage of. This is not exactly a weakness, but it could be an interesring connection to orthogonality. Maybe the authors can comment / elaborate on this.\n\n- The definition of maximally seprated matrix is a bit inappropriate in the sense that the equal inner product between any two classifiers does not always hold for matrices of any size. It only holds when the matrix is of the size $(k+1)\\times k$. I suggest the authors to clarify this in order to avoid confusion.\n\n\nSummary:\n- I am in favor of the core idea (as well as the overall direction in static classifier design) and find this paper a solid work in general. I would vote for clear acceptance given the authors properly address my concerns.\n\n\n[1] SphereFace: Deep Hypersphere Embedding for Face Recognition, CVPR 2017\n\n[2] Orthogonal Over-Parameterized Training, CVPR 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0D1_NLqYv2Z", "reviewer": "Reviewer_9rpc", "summary": "The paper proposes to bake maximum separation between classes as an inductive prior to a deep learning model. The proposed approach maximally separates class vectors, which is cleverly proven to have a closed form (assuming hyperspherical uniformity), and is implementable as a simple multiplication with a fixed matrix. Results are good on conventional tasks (Cifar, ImageNet), alongside long-tailed and open set problems. ", "questions": "Can this applied to other tasks, like segmentation?\nCan an approach like this be modified to work well in an unsupervised setting? ", "limitations": "I'd be interested in knowing if this work can be applied to other different tasks, like segmentation, or be applied in other settings, like unsupervised. \nThere are obviously improvements with this method, but when I quickly ran the code a couple of times, there seemed to be a large amount of variance of the test performance, exp in the imbalanced setting. Calculating some error bars/confidence intervals would be helpful.  ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe method improves classification in the imbalanced, long tailed setting, alongside OOD and open set settings. \nThe approach easily slots in with what appears to be any/most existing approach, as seen by the comprehensive experiments with various different methods on various problems. \nGood theoretical analysis. It appears correct to the best of my understanding.  \nUtilizing a fixed matrix that you do not learn and improving results in a deep learning framework is an interesting approach with some novelty/originality to it. \nThe result of the ablation with learnable prototypes is interesting, highlighting the importance of class separation as a solid inductive prior. \nHow to run the code is very clearly laid out in README.md (albeit with minor mistakes). \nI was able to reproduce similar numbers to a subset of the results (table 1, alexnet) using the code with minor fixes. Did not have time to try more.\n\nWeakness:\nDiagrams are confusing and could be better labeled. For example, in Figure 1, it would be most useful to label what is $P_k$ and what is $p_k$.  Figure 3 could be much more clear (sub-title each subplot with amount of imbalance).\nSome figures appear to contain the wrong description i.e. figure 4 \"green\" (there is no green). \nThe code was troublesome to get running; many simple errors to fix. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept