Spherization Layer: Representation Using Only Angles
Authors: Hoyong Kim
Kangil Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: representation learning
hyperspherical learning
angular similarity
spherization

Abstract: In neural network literature, angular similarity between feature vectors is frequently used for interpreting or re-using learned representations. 
However, the inner product in neural networks partially disperses information over the scales and angles of the involved input vectors and weight vectors. Therefore, when using only angular similarity on representations trained with the inner product, information loss occurs in downstream methods, which limits their performance. In this paper, we proposed the $\textit{spherization layer}$ to represent all information on angular similarity. The layer 1) maps the pre-activations of input vectors into the specific range of angles, 2) converts the angular coordinates of the vectors to Cartesian coordinates with an additional dimension, and 3) trains decision boundaries from hyperplanes, without bias parameters, passing through the origin. This approach guarantees that representation learning always occurs on the hyperspherical surface without the loss of any information unlike other projection-based methods. Furthermore, this method can be applied to any network by replacing an existing layer. We validate the functional correctness of the proposed method in a toy task, retention ability in well-known image classification tasks, and effectiveness in word analogy test and few-shot learning. Code is publicly available at https://github.com/GIST-IRR/spherization_layer

Spherization Layer: Representation Using Only Angles
Hoyong Kim, Kangil Kim'
Artificial Intelligence Graduate School Gwangju Institute of Science and Technology, Gwangju 61005, South Korea
hoyong- kim. 21@gm. gist. ac ac.kr, kangil. kim. 01 01@gmail..... com
Abstract
In neural network literature, angular similarity between feature vectors is frequently used for interpreting re-using learned representations. However, the inner prod- uct in neural networks partially disperses information over the scales and angles of the involved input vectors and weight vectors. Therefore, when using only angular similarity on representations trained with the inner product, information loss occurs in downstream methods, which limits their performance. In this pa- per, we proposed the spherization layer to represent all information on angular similarity. The layer maps the pre-activations of input vectors into the specific range of angles, 2) converts the angular coordinates of the vectors to Cartesian coordinates with an additional dimension, and 3) trains decision boundaries from hyperplanes, without bias parameters, passing through the origin. This approach guarantees that representation learning always occurs on the hyperspherical sur- face without the loss of any information unlike other mojection-bbsse methods. Furthermore, this method can be applied to any network by replacing an exist- ing layer. We validate the functional correctness of the proposed method in a toy task, retention ability well-known image classification tasks, and effective- ness in word analogy test and few-shot learning. Code is publicly available at htttps://githei cww.coo/cc/ooooo Swo//ISTT-a Bttp/IIISTTTRRRSETTEE htt.ss--nR//pperiiiiioo.....
1
Introduction
The inner product is a key element constituting layers in deep neural networks with a nonlinear acti- vation function. The inner product with the Euclidean norms and the angle, that is, ||w:|l||x; cos Orj- has been analyzed in terms of the norms ||x||| and the angle cos O,y. independently [1] 12 31). All factors of the inner product learn distinct information. Therefore, using only one factor results in information loss when re-using or understanding the information in downstream tasks. This problem termed dispersion problem. The angular similarity between features frequently used. How- ever, this technique causes the dispersion problem in advanced methods in neural network literature such as decoupled network [12, representation learning [5] 6. 20 21 228. regularization [9] 30),
zero-shot learning (22), and generative model 23 To mitigate the dispersion problem, numerous angle-based learning approaches I
32 have been proposed. However, these studies are based on projection onto the hyperspherical surface. In this projection method, distinction by the scale of features ignored. Therefore, information loss occurs.
g
10
13
15
We proposed the spherization layer as an explicit solution for the dispersion to completely eliminate the interference of the norms in training without drawbacks. This layer is used locate all represen-
corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

cutput
pulput
Spherization Layer
Jepn: R 30 (e.g.. nt
3D visunlization on MNIST
al
(
E
Activation
Activation
will gonsis
z10 - h(s:Wo))
x/!
Target
h: h:201~ -
Fiidden Layer
(3) Layer
w/"
(2)
[
cnsup, s
(2)
[-3]
Conversion to Cartesian
((p)
EEB
8: gg+e1
(I)
Activation function
Angularization fanction
(1)
(
|g!l-1]
x/2
f(x)
Hidden Layer
Hidden Layer
+ (Pun
f:8"
input
input
Figure I: Overview of spherization layer. The red box indicates the existing fully connected layer to be replaced with the spherization layer in the blue box. After replacement, the pre-activations from (1- 1)th layer are passed to the angularization function, not activation function. These pre-activations converted to angular coordinates in (1). Through (2), spherized representations are located on the (n + 1)-spherical surface. Finally, a hidden layer without bias parameters is trained on these spherized representations for using only the angles in (3). These relations of the input and output of the stage (1), (2), and (3) are illustrated as graphs, which are clearly defined in Eq. 3 Eq. 6 and Eq. [9] respectively. The right side displays how to generate angular coordinates and convert them to Cartesian coordinates in a a 3-dimensional space on MNIST
tations onto a constrained region on the hyperspherical surface and train hyperplanes passing through the origin to learn representations with only the angles.
The spherization layer consists of three main components: Angularization funiiiiaaaiiooniiioiii that converts the pre-activations from the previous hidden layer to angles; Conversion from spherical coordinates to Cartesian coordinates located on the hyperspherical surface; No-bias layer, hidden layer without bias parameters, that determines decision boundaries by using only the angles. Figure illlstrates the design of the spherization layer. Through spherization. representations are located on the hyperspherical surface and the effect of the norms in representation learning is completely eliminated. Thus, neural networks are enforced to express all representations differently by using only the angles.
We experimentally verified the functional correctness of the spherization layer in a toy task and its applicability to feedforward and convolutional neural networks by evaluating performance image classification tasks. The results reveal that the training ability of original networks is preserved after applying the spherization layer. Furthermore, the effect of projection in the spherization layer, and the influence of the spherization layer on the gradient flows in training. Finally, we investigated the effect of the proposed method in downstream tasks through visualization, word analogy test, and few-shot learning.
analyzed the sensitivity to width and depth,
In summary, contributions are three-fold:
To address the dispersion problem, we propose the spherization layer to represent all feature vectors the hyperspherical surface and learn the representations with only the angles.
We validate the wide applccppliy and scalability of the spherization layer without any loss of performance through experiments on various vell-known networks.
We empirically show that the spherization layer can be used in many applications in which angular similarity is a critical metric.

---

2 Background
Conversion Spherical the Cartesian Coordinate System most neural networks, all input samples on the n-dimennionnl space are represented as Cartesian coordinates, in which i-th column value denotes the distance from the origin along the i-th axis, and neural networks train them by using neurons in which the inner product between the input and weight vector occurred. The are-activations from neurons are used determine whether neurons should be activated by using the following activation function. In this process, the pre-activations are calculated by using weight and bias parameters, represented as Cartesian coordinates. These Cartesian coordinates can be converted from spherical coordinates. Given vector represented as Cartesian coordinates on the n n-dimensional space, 8 can be defined as spherical coordinates [r. q]. composed of a radial coordinate and n-l angular coordinates with Eq.
[Q), 02..... Pn-1]. In this case, the k-th axis of can be computed from
S cos Q1 coS Q& | sin Pi.
[= sin qi]
where Siccen r cos Q& 11 sin Qi-
Generally, the spherical coordinate system is a 3-dimensional version of the polar coordinate system. However, the spherical coordinate system the followings indicates all n-dimensional versions of the polar coordinate system, where и is greater than or equal to 2.
3 Spherization Layer
We proposed the spherization layer as shown in Figure[]to locate feature vectors on the hyperspherical surface without any loss of information and learn hyperplanes by using only the angles. In this method, a layer of an original network selected learn representations by angular similarity. Next, the layer is replaced to spherization layer through three sequential stages: Angularization, Conversion to Cartesian, and No-bias training notated as f. g. and h functions, respectively. The final form of spherization layer feph is expressed as Eq. 2
fsph (hogof), fsph R" RM
(2)
After training with the layer, can obtain the representations the (n + 1) -)spheriica surface, namely spherized representations, as the outputs of (g f) operations.
The common goal of all stages is to preserve the training ability of the original network while the spherization layer trains all information on the 1)-spherical surface. The angularization locates all pre-activations the safe spherical surface. The conversion to Cartesian results in the generation of compatible representations to the ordinary layer. No-bias training enforces training by only the angles. In the followings, we elaborate each stages more detail and only annotate the (1 2)th and th layer as [ and [/]. respectively, for the simplicity.
3.1 Angularization
f(2), f :: - R"
Angularization the stage to map a pre-activation vector z, passed from (1 1)th layer, to angular coordinates qp The indicates the dimension of the pre-activation vector. The role of this stage is to configure the shape the mapped region on the (n + 1)-spherical surface for resolving training and computational difficulty.
Angularization f is implemented by applying the following element-wiss function all coordinates of an activation function, and f is illustrated as Eq.
f(x) (4 41.) (a z) + 41.
where the terms and form are used by three following motivations.
Converting Pre-Activation to the Angular Coordinate The first step is to convert the input vector into angular coordinates. To ensure the conversion as bijective mapping, we restrict the range of the function as [0, J. To allow unrestricted input representations on the real-valued domain, the sigmoid function (-). is used with weight I for the range setting. As the sigmoid function used,
3

---

the input vector should be pre-activations, not activations because when activations from ReLU or another sigmoid are passed the angularization, inefficient use of the spherical surface or gradient amplification, respectively, may occur. After converting pre-activations to angular coordinates, the representations on the hyperspherical surface the same range were located by setting a consistent radius over all inputs. This radius scale is controlled in the Caneersiiont--Crtessiiee stage.
Tailoring Angular Representation Space In the conversion from angular to Cartesian coordinates, the last coordinate can be an extremely small value because trigonometric values in [0, 1] are multiplied many times. This scale descent can map all values in the axis to only a single value by the limit of the floating point data type. To reduce this effect in the angularization, we introduce a lower bound PL of angles to guarantee distinguishable values in its corresponding converted Cartesian coordinates as the following equation (Eq. 5
(5) where ō is a minimal trigonometric value to guarantee the distinguishable representations. The details are presented in Appendix A. We set 5 to the empirically obtained proper value 10 for all experiments.
fr. sin (61/m)
Scaling Pre-Activations because of the lower bound. This concentration renders training difficult with the decrease in the variance. To reduce the effect, we set a learnable parameter a as a weight of z. which controls the variance Using this scale factor, the generated angular representations become abundant.
In angularization, the activations are concentrated onto the small region
3.2 Conversion-to--Crtesiin
g()).
g R: R*+1
(6)
In the Snneersionnt--Cartsiii stage, the angular coordinates of the previous stage converted to Cartesian coordinates on the (n + 1)-spherical surface. This conversion ensures the consistency between the output of angularization (polar coordinate system) and the input of the following no-bias layer (Cartesian coordinate system), and enables the layer to be trained in the same way as general neural networks. Furthermore, an additional dimension makes the spherization layer have enough capacity to be compatible with the ordinary layer. This implementation is based on Eq. [ with the modified range of angles as the following equation (Eq. 7
g (<p)
[x P1. , cos Q&
k-1 sin Pi.-. I7= sin Φı),
Φi [9.. 3]
Calculation Trick Implementation of Eq. as tensor operation requires the trick defined in the following equation (Eq.
8
WT 6 4p
(8)
exp WT In (sin (b) In (cos (Φ bal)
where @ is dimension-expendee vector in R=+l and Φn+1 Φn. and is a constant to control radius. Here, W p. W and ba are constant matrices and vector in [=x(+++)) R (=+1)x((++1 and R (x+1)) respectively. We = =[,,; v]. where y [0; I] E R", used for matching the dimension between f and s. The other constants are used for calculating logarithm trigonometric values from expanded angular coordinates in the way of matrix multiplication, where W is an upper triangular matrix in which all diagonals are zero and (W (W&).at1 is also zero, and b [0; [0:-2]T, See Appendix B for detail hyperparameters and process.
3.3 No-bias Training
! = h(s), h: :: + RR
(9)
Through angularization and conversion-to-CCrtesii spherical surface. In this case, any update to the representations is determined by the change of angular similarity. However, hyperplanes on the (n + -)-dimensional space from the next layer may use only the angular similarity, which may assign semantic information to the Euclidean norm of the spherized representation. To synchronize these parameters, we used no-bias training using only the weight parameters Will as illustrated in Eq. 10
all representations are located on the (n + 1)-
1
Wlll
(10)

---

Effect of No-Bias on Training In the ordinary layer, the problem of no-bias is that hyperplanes passing through the origin cannot be shifted another parallel hyperplanes. However, the problem disappears when all feature vectors are located the (n1 + 1)-spherical surface because the decision boundary can be shifted by only the angle changes of (n + 1)-dimensional hyperplanes even though they pass through the origin.
3.4 Optimization with Overall Process
training loss L is calculated in the same manner the original network. The gradient of L for the spherization layer is calculated by multiplying the following partial derivative
øl to the original SW backpropagation step from the (I 2) to (/)th layer, as illustrated in Eq. 11 The first term Tg calculated by the subsequent layers in the same way of the original network.
aL aL
0z [1]
Ōs
Qqp
Oz.
aw dz
as
δq
Oz
aw
θz [1-2]
ow
dę π
Oz (
PL a o' z)
(11)
Øs
|- sin P1.
sin
sin Pr. , cos QD
sin p:l
0Φ
i=1
0zl'1
wil
Ōs
4 Experiments
First, we define two terms for simplicity on indicating two networks: one is the network before applying the spherization layer, called original network, and the other is the network after substituting an ordinary layer with the spherization layer, called spherized network. In the followings, we used these two terms consistently. All experiments were performed five times with random seeds and their training and test accuracy were evaluated except word analogy test and few-shot learning. The mean and standard deviation of accuracy are represented μ in each table.
4.1 Functional Correctness Test on a Toy Task
Implementation Details We verified the spherization layer for learning decision boundaries a simple binary classification task. We set up the simple binary classification task: given (N,, pairs, where x; is the i-th input sample in 2 and E {0, }} is the label of x;, we randomly generated 100 input samples located around (0, 0) for the label 0, and the other 100 samples for the label around (1, 1), as shown in Figure We set 2-layer neural network as the original network, and trained it with the softmax function, cross-entropy, and SGD at a learning rate of 0.01. We applied the proposed method to the original network by replacing the last fully connected layer with the spherization layer. For comparison of representations in the same dimensional space, we set the dimension of the spherization layer to 1, where it is in the original network. The other settings are identical the original network. We trained both networks the input samples for 100 epochs with 16 mini-batches, where the networks converged and achieved 100% training accuracy.
0.
046
0. 1.0
Figure 2: Input samples for the toy task
Distribution of Representations Initially, all representations are randomly distributed into two groups a 2-dimensional space. After convergence, the feature vectors divided into two dis- jointed groups, as illustrated in Figure This means the spherized network locates all representations on the 2-spherical surface in both the initial and final epoch, whereas the original network spreads them out.
5

---

boundery
643
2112
Banadirn- Spach
Eparch 100
Spherication Spach
Figure 3: Visualization of Hyperplanes, Decision Boundary, and Representations in the Toy Task. (w: hyperplanes, boundary: decision boundary, red or green points: representations for label or 1)
Table I: Retention of the Training Ability on Image Classification with Various Datasets and Models (Accuarcy($)); (%): j = o))
Reference
Reproduced
Spherized
Network
Dalasel
train
test
train
test
train
test
SimpleFNN LeNet S
98.47
99. 99.99+0.01
98-58+0.03
99.99-0.01
98.65+0.0M4
MNIST
99.05
09.55
99. 99.10=0.05
99.79=0.09
99.14=0.0M
F-MNIST
94.70
99.24.0.16 18
94.3620.06
98.92a0.36
94.34.0.17
VGG-11 11 2F
CIFARIO
90.90
100.00+0.00
92.38:+0.06
100.00=00.00
92.49:0.11
CIFAR 100
66.80
99.71+0.03
68.42+0.12 12
99.82-0.02
69.03-0.24
Table 2: Retention of Training Ability on Image Classification with CIFARI00 in Various Network Width and Depth Settings (Accuarcy($h)) (%): μ o))
Referenc
Reproduccd
Spherized
Depth
Width
train
Imain
test
train
16/32/4/1228
79.23
60.40::0.35
98.34=0.37
64.89+0.338
96.98+3.67
65.38+00288
VGG-11
64/128/256/512
99.7120.03
68.4240.12
99.220.02
69.03.0.24
128/256/512/1022
99.90=0.00
70.53+0.35
99.93+0.00
70.89=0.19
99.90=0.01
41.43-2222
99.93+0.01
71.94-0.19
VGG-11
68 64
99.71=0.03
68.42+0.11 12
99 00.8200002
69.03::0224
VGG-16
64/128/256/112
72.93
99.3920.07
72.51100.26
99.5420.05
72.53a0.17
VGG-19
72.23
97.95=0.81
71.53+0.32
99.30=0.05
72.17=0.33
Hyperplanes Hyperplanes and decision boundary in the feature space are defined as follows with the parameters of the output layer:
Wi Wi x bj =
W/2 W2 x b2
D12 : (W] - w2) x (b; -b) =0
where W, is the hyperplane determined by weight parameter and bias b, and Dij is linear decision boundary whose points satisfy 0.5 confidence for both labels. In the spherized network, all bias terms are eliminated. In Figure 3 the hyperplanes are illustrated yellow lines and the decision boundary as a gray line. In the spherized network, hyperplanes and decision boundary pass through the origin from the initial to the last epoch, unlike the original network. The results imply that the spherization layer can learn the correct decision boundary by changing only the angles of hyperplanes passing through the origin.
4.2 Retention of Training Ability on Image Classification Benchmarks
Implementation Details this task, we empirically verified that replacing an existing layer to spherization layer still maintains the training ability of the original networks on well-known image classification tasks. We reproduced all networks and their performance on the image classification tasks with each dataset. Then, we validated our proposed method on the same settings with them, where only the last fully connected layer replaced by the spherization layer. See Appendix C for the detail about networks and datasets.

---

Table 3: Analysis on the Effect of Projection on Image Classification with CIFARIO (acc. accuracy, # err. the number of emrors in overlapping samples, # ovlp.: the number of overlapping samples, ratio(%) ratio of # err. to # ovlp.) (the number of test data 10000)
Train
Test
Role
Operator
No-bias
ncc.
ncc.
een:
# #ovlp.
(ratio)
base
Original Come.
99.47 + 0.42
92.46 0.10
0+
0+ 0
( 0.00 0.00 %)
Sigmoid
81.74 = 4.48
79.03 2.71
1097 266
5222 : 391
(20.75
3,70 %)
direct
Lincur
74.71 = 1.06
72.15 z 0.32
1631 a 39
6849 1: 286
(23.84 0.63 %)
Cosine
77.41 0.89
76.15 0.86
776 132
3170 330
(24.39 2.66 %)
SW-Softmax
98.32 = 0.07
91,31 = 0.19
167
7925 + 58
( ( 0.11 %)
indirect
LW-Softmax
86.74 = 4.06
82,12 3.82
946 374
7669 + 66
(12,30 2 4,84 %)
CW-Softmax
99.66 = 0.04
92.29 0.18
80
7051 # 134
( [1114 a: 0.05 %)
proposed
Spherization
99.66 = 0.05
92.38 0.14
0
499 4 106
0.00 0.00 (8)
Results and Analysis The accuracy results of original and spherized networks are compared in Table The reference and reproduced results of each setting are similar. The accuracy results of the spherized networks implemented on the reproduced code are similar or slightly higher on both training and test data than those of the original networks. These results imply that the spherization layer maintains the training ability of the original network. In Table 2 the accuracy results of original and spherized networks are shown in various width and depth settings. The spherized network consistently exhibits similar or higher test accuracy over all width and depth settings. The results reveal that the spherization layer again preserves the training ability of the original network.
4.3 Analysis: Effect of Projection
Implementation Details We compared the method with another angle-based approach to analyze the effect of reducing information loss by projection. For comparison, we used 9-layer CNN, namely CNN-9, with the same experimental setup of [6] but reproduced in PyTorch. To apply a spherization layer, changed the last fully connected layer in each model. We considered samples overlapped when their cosine similarity is greater than or equal to (1 - 10-6).
Results and Analysis After training, extracted representations from the target layer and analyzed the effect of projection. First, we count the number of representations overlapping at least one another representation. Then, we get the total errors caused by the incorrectly classified and overlapping representations. Finally, we calculate the ratio of the incorrectly classified representations to the overlapping representations. The results of them are shown in Table 3 # ovlp., # err. and (ratio), respectively. As the result, large proportion of representations suffer the overlap problem from projection approach, and most significant errors are caused by the overlapping representations. Furthermore, representation learning relying on bias parameters (direct projection case) causes more significant errors than no-bias layer (indirection projection case) because the representations are more difficult to be distinguished than those of no-bias layer. In comparison, the spherization layer removes the errors and decreases the upperbound of errors measured by the number of overlapping representations.
4.4 Analysis: Gradient Flows
The replacement of hidden layer and the operations in the spherization layer, such as angularization function and conversion-tor Co-aatesii,,,,,,in,,, might make the gradient flow unstable in the original networks. To empirically verify how the spherization layer affects to the gradients during training, we qualitatively compare the gradient flows of original and spherized VGG-11. They were trained on the image classification with CIFAR100. We used the same setting with Section 4.2
Implementation Details
Results and Analysis As shown in Figure 4 the average of absolute gradients in the original and spherized network are not very different during the whole training. Furthermore, the results show similar flows not only at the last fully connected layer (fc3 or sph_fc3) but at the previous layers. This result implics the spherization layer does not destroy the gradient flows.
7

---

VGG11
SpherizecVGG11
#pech300 eporto
0.0
220
oV313ge gratiemt
0.25
0.5
0.10
3.10
816
8.00
8.00
vomus
cancs
con
convea
comu
cone
conca
cond
layors
(a) Avg. of Absolute Gradients at Each Layer
(b) Histograms
Figure 4: Analysis of Gradient Flow from Image Classification Model trained on CIFAR100. (a)
The y-axis means the average of absolute gradients which occurred at each layer. The left side shows the gradient flow in VGG-11 (VGGII), and the right side shows the spherized VGG-11 (SpherizedVGGI1) I), where the last fully connected layer is substituted with the spherization layer. (b) The histograms show the frequency the average of absolute gradients in VGG-11 (red) and spherized VGG-11 (cyan), respectively.
(a) Original 2D
(b) Original 3D
(c) Spherized 3D
Figure 5: Visualization of Feature Representations MNIST. (a) and (b) are the visualization results of 2D and 3D feature vectors in the original networks, and (c) is the result the spherized network
4.5 Downstream Tasks: Visualization
Implementation Details To investigate the spherization layer locates feature vectors on the (n + D-ssheriiall surface, implemented simple CNN learn 3-dimensional feature vectors for visualization. This technique is called CNN-Vis3D, which generates 3-dimensional feature vectors and shows those representations as graph. Detailed configurations In the spherized network, the second fully connected layer was replaced by the spherization layer, which is located the previous of the last fully connected layer. After training, the aetivviiv before the no- bo-bias layer were used to visualization.
presented in Appendix C.
Results and Analysis The results of visualization illustrated in Figure In the 2D and 3D feature visualization results of the original network, the representations are distributed over wide ranges of both scale and angles, shown Figure 5a and [56] As illustrated in decoupled network [12], the angle accounts for semantic difference and the Euclidean norm accounts for intra- class variation. the spherized network, all representations are placed on the 3-spherical surface, as shown in Figure Based on these spherized representations and the following no-bias layer, all the roles are expressed by the angle change on the hyperspherical surface. Thus, in the spherization layer, all trained information be used in angular similarity-baac based interpretation.

---

Table Performance on the Word Analogy Test (S pmm 1)
Model
SAT
U2
U4
Google
BATS
Avg.
BERT
29.4/28.5/28.8
36.0/66.03668
7/34,
33.0033./33.
32.3/36.0033.2
33/3/33.6/33.2
BERT + sph
A/299
37.3/39.0/36.0
36.8/359//54
32.4/32.6/32.2
34,0/34.2/33.8
33.9/34.2/33.1
RoBERTa
29.4/31.2/29.7
35, 35.5/35.5/36.4
33.6/34.3/4455
32.8/33.
30.9/31 /6/30.9
32.4/33, 1/32.5
RoBERTa+ sph
29.1/29.4/3.00
36.4/35.5/34.2
34.0/34./3333
34.2/33.6/32.8
35.0/33.9/44.8
33.7/33.3/33.0
Table Performance of few-shot learning on Mini-ImageNet. Euclidean and Cosine mean euclidean distance and cosine similarity, respectively, which are the distance metrics that used in the experiments. (Accuarcy(%); μ + o))
Test Acc.
Test Acc.
Model
Model
Euclidean
Cosine
Buelidean
Cosine
ConvNet
50.29
52.8720.18
ResNet
37.6340.15
33.41:0.15
ConvNet
43.41=0.16
53.74:0.16
ResNet xph
31.7740.13
38.71=0.16
4.6 Downstream Tasks: Word Analogy Test
Implementation Details We used BERT 4 and RoBERTa 17 to conduct the word analogy test, in which angular similarity is used predict relation type between words. The settings were identical with [25]. We applied the spherization layer to the last fully connected layer of the encoder in each model. Next, trained them on WikiText [9] for 3 epochs with 8 mini-batches, the softmax function following cross-entropy, SGD at a learning rate 0.0001 on masked-language modeling.
Results and Analysis Table presents the performance evaluated with three metries [25] Applying the spherization layer to BERT and RoBERTa improves most average scores. The improvement implies that the spherized representations provide accurate information for angle-based distinction of word relations.
4.7 Downstream Tasks: Few-shot Learning
Implementation Details We used ProtoNet (24] with ConvNet and ResNet for few-shot learning on Mini-ImageNet (26), in which several feature vectors are compared with the feature vector of an input image by using distance metric. This task was performed according to the guidelines in (2]
Results and Analysis detaills the performance results in few-shot learning 2 Generally, the Fuclidean distance used discriminate feature vectors. However, the Fucliean distance also has the dispersion problem. To focus only the angles, we trained the models with cosine similarity and compared the performances. As shown in T5 all spherized models with cosine similarity outperform the other models. This improvement indicates the spherized representations are useful for the angle-based metric such as cosine similarity.
Related Works
Semantic Analysis on the Inner Product The inner product a crucial operator in current neural networks, in which the distance between input vector and weight vector w is encoded. At the decoupled networks [12], the inner product is reparametrized with the norms and the angle, and the intra-class variation and the semantic difference are modeled in neural networks by decoupling them. Furthermore, the substitutes of the inner product have been proposed, where the direction of gradient or the similarity between kernels are used as the key factor instead of the inner product [1]. The spherization layer can be considered be the substitute of the inner product, which normalizes the input vectors by locating them on the hyperspherical surface. However, the spherization layer is direct and specific method to convert feature vectors focused on the angles without information loss.
Angle-based Approach The semantic analysis on the inner product has revealed that the streams focuses on the information the angles. The angle a crucial factor, which the most abundant and discriminative information is preserved [ [0] [5]. In SphereFace variants [0] 13 14 32)

---

the angular softmax that enables CNNs to learn discriminative features angular separability was used. Furthermore, some of this angular information have been used for regularization [29 [0]. In most angle-based studies, the angular information was used indirectly by the objective function or regularization. In contrast, the spherization layer ensures the model directly learns the angular information on the hyperspherical surface.
Hyperspherical Representation Learning In some angle-based approaches, input vectors were directly projected onto the hyperphserical surface [12 16 18 These hyperspherical representation learning methods normalized the input vectors to ensure models are dependent on only the angles. However, this normalization is the projection onto the hyperspherical surface, and it can be less discriminative when some points overlap after the projection. The spherization layer locates the input vectors on the hyperspherical surface without the overlap problem through the spherization.
6 Conclusion
We introduced the dispersion problem of trained information to the Euclidean norm and angle on representations. To address the dispersion problem, proposed the spherization layer learn representations by using only the angles without information loss. We used the angularization for using pre-activations as angular coordinates, Cnvvession-toCCartssaa for locating them on the (n+ 1-spherical surface, and no-bias training to learn representations by using only the angles. In the experiments on toy, image classification benchmarks, few-shot learning, and word analogy test, the proposed method achieved accurate learning of the decision boundary and retention of the original training ability, and improved performance downstream tasks using angle-based information re- used or interpreted. The proposed method can be applied to numerous network layers and downstream applications. limit of this approach that the spherization layer should inserted to networks in training step to fully utilize its advantage, which restricts the use of pre-trained models trained without it. Recovering trained information on representations with sampling should be investigated in the future.
Acknowledgments and Disclosure of Funding
This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022RAA222010554) and by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01842 Artificial Intelligence Graduate School Program (GIST)).
References
[1] Beidi Chen, Weiyang Liu, Zhiding Yu, Jan Kautz, Anshumali Shrivastava, Animesh Garg, and Animashree Anandkumar. Angular visual hardness. In International Conference on Machine Learning, pages 1637- 1648. PMLR, 2020.
[2] Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue. Self-supervised learning for few-shot image classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1745 1749. IEEE, 2021.
[3] Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical variational auto-encoders. arXiv preprint
2018.
[4] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.
[5] Aiden Durrant and Georgios Leontidis. Hyperspherically regularized networks for byol improves feature uniformity and separability, arXiv preprint arXiv:2105 00925, 2021.
[6] Yi Hao, Nannan Wang. Jie Li, and Xinbo Gao. Hsme: Hypersphere manifold embedding for visible thermal person re- re-identification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1385-8392. 2019.
[7] Geoffrey E Hinton, Simon Osindero, and Yee- Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527-1554 2006.
10

---

[8] Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (0:211):27822333 1998.
[9] Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James M Rehg, Li Xiong, and Le Song. Regularizing neural networks via minimizing hyperspherical energy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 6917-6927, 2020.
[10] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018.
[11] Weiyang Liu, Zhen Liu, James M Rehg, and Le Song. Neural similarity learning. Advances in Neural Information Procexsing Systems. 32, 2019.
[12] Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang. James M Rehg, and Le Song. Decoupled networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2771-2779, 2018.
[13] Weiyang Liu, Yandong Wen, Bhiksha Raj, Rita Singh, and Adrian Weller. Sphereface revived: Unifying hyperspherical face recognition. arXiv preprint arXiv:2109.0585 2021.
[14] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and puttern recugnition, pages 212-220, 2017.
[15] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In ICML, volume 2, page 7, 2016.
[16] Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. Advances in neural information processing systemss 30, 2017.
[17] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 7ettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.1 2019.
[18] Chunjie Luo, Jianfeng Zhan, Xiaohe Xue, Lei Wang, Rui Ren, and Qiang Yang, Cosine normalization: Using cosine similarity instead of dot product neural networks. In International Conference on Artificial Neural Networks, pages 382-391. Springer, 2018.
[19] Stephen Merity. The wikitext long term dependency language modeling dataset. Salesforce Metamind, 2016.
[20] Pascal Mettes, Elise der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in neural information processing systems, 32, 2019.
[21] Rafacl S Pereira, Alexis Joly, Patrick Valduriez, and Fabio Porto. Hyperspherical embedding for novel class classification. arXiv preprint wwXiv:2102.03244 2021.
[22] Jiayi Shen, Zehao Xiao, Xiantong Zhen, and Lei Zhang. Spherical zero-shot learning. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
[23] Woohyeon Shim and Minsu Cho. Cirelegan: Generative adversarial learning across spherical circles. Advances in Neural Information Processing Systems, 33:21081-21091, 2020.
[24] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances neural information processing systems, 30, 2017.
[25] Asahi Ushio, Luis Espinosa-Amn Steven Schockaert, and Jose alexnet is to cv: can pre-trained language models identify analogies? arXiv preprint arXiv:2105.044: 2021.
Bert to nlp what
[26] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[27] Shengjie Wang. Tianyi Zhou, and Jeff Bilmes. Bias also matters: Bias attribution for deep neural network explanation. In International Conference on Machine Learning, pages 6659-6667. PMLR, 2019.
[28] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929 9939. PMLR, 2020.
11

---

[29] Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. Dma regularization: Enhancing discriminability of neural networks by decreasing the minimal angle. IEEE Signal Processing Letters, 27:2089-2093,, 2020.
[30] Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. Mma regularization: Decorrelatngg weights of neural networks by maximizing the minimal angles. Advances in Neural Information Processing Systems, 11119999-19100, 2020.
[31] Zhennan Wang, Wenbin Zou, and Chen Xu. Pr product: A substitute for inner product in neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6013-6022, 2019,
[32] Yandong Wen, Weiyang Liu, Adrian Weller, Bhiksha Raj, and Rita Singh. Sphereface2: Binary classifica- tion is all you need for deep face recognition, arXiv preprint arXiv22108.01513 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Abstract and Introduction (Section D
(b) Did you describe the limitations of your work? [Yes] See Conclusion (Section 6 and Appendix D
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of theoretical results? [Yes) See Background (Section 2 and Spherization Layer (Section (b) Did you include complete proofs of all theoretical results? [Yes] See Spherization (Section 3) Functional Correctness Test Toy Task in Experiments (Section 4), and Appendix A and
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] See the supplemental material (.zip file)
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Implementation Details in each experiments (Section 4 and Appendix C
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Tables in each experiments (Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix and the supplemental material (.zip file)
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We cited them by Reference. The detail about the citation is covered at the supplemental material (zzip file)
(b) Did you mention the license of the assets? Yes] See the supplemental material (.zip
file)
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See the supplemental material (.zip file)
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We used open datasets. When using curated open datasets, we cited by Reference
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We used open datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "_xuR4p36SDG", "reviewer": "Reviewer_B9zn", "summary": "This paper proposes spherization layer to represent all information on angular similarity, which avoids the information loss when using only angular similarity on representations trained with the inner product. It maps the pre-activations of input vectors into the specific range of angles, converts the angular coordinates of the vectors to Cartesian coordinates with an additional dimension, and trains decision boundaries from hyperplanes. This guarantees that representation learning always occurs on the hyperspherical surface without the loss of any information unlike other projection-based methods. Experiments have been evaluated on different tasks and show the effectiveness of the proposed methods.\n\n", "questions": "\nI have no problem", "limitations": "Weakness.\nSome related works should be listed and have a discussion and comparison, ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Novelty.\nThis paper proposes a novel method for representation learning only using the angle information. The subtle construction to covert the angular coordinates of the vectors to Cartesian coordinates, which avoids the information loss than the previous method.\n\nWriting.\nThis paper is of better organized and shows detailed description about the proposed method. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Wf0ZItZktXC", "reviewer": "Reviewer_ouKB", "summary": "The paper proposes a spherization layer to represent preactivations in a hypersphere. This layer can be plugged into any neural network. This representation is useful in cases where cosine similarity is used as the main metric to determine similarity in the feature space.", "questions": "1. It seems it is possible to theoretically show that the spherization layer retains all information as a Euclidean layer. Meaning, a spherization layer can learn any function that can be represented by a Euclidean layer. Can you comment on this?", "limitations": "Societal impact is not discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The spherization idea is interesting. It addresses a niche problem in a neat way.\n2. The formulation is clean and clear justifications are provided.\n3. Experiments are performed in various settings and significant improvements are observed in the few-shot learning setting.\n\n## Weaknesses\n1. The improvement is not clear compared to the projection approach (table 5). This raises doubts about the motivation for this approach. Is there really an information loss with the projection approach?\n2. More experiments where explicit similarity is used in the feature space could be performed. Examples include self-supervised learning and metric learning. This may show the benefits of the proposed approach in a better way.\n3. Training stability of the network could have been discussed. It is not clear, how the signal propagation (gradient flow) would be affected by the spherization layer.\n\n## Post rebuttal\nThanks for the clarifications. I'm in favour of accepting the paper. Please consider addressing the discussed comments in camera-ready or in an extension.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QCoBjFpsZM", "reviewer": "Reviewer_v1RE", "summary": "The authors propose a novel neural network layer that forces pre-activations into angles. Besides, the spherization Layer converts the angular coordinates to Cartesian coordinates and the training is done without bias parameters. Several different empirical experiments have shown the effectiveness of the proposed method. ", "questions": "It would be nice if the authors can add more explanation about the motivation and analysis of the conversion-to-Cartesian process.", "limitations": "The authors have adequately addressed the limitations of this paper. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. This paper is well written and clear. \n2. The proposed spherization layer can be easily applied to most existing neural network architectures and downstream tasks.\n\nWeakness:  \n\n1. There is not enough comparison between the proposed spherization layer and other recent angle-based learning methods such as [1] [2]. Since the authors claim that the proposed method solves the information loss problem that occurs in these projection-based learning methods, these comparisons are important experiments to support the authors claims.\n\n\n[1] Chen, Beidi, et al. \"Angular visual hardness.\" International Conference on Machine Learning. PMLR, 2020.\n[2] Lin, Rongmei, et al. \"Regularizing neural networks via minimizing hyperspherical energy.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KmC6tr50M7H", "reviewer": "Reviewer_Wemx", "summary": "This paper presents a new formulation for a layer in neural networks known as Spherization Layer. When inner product with Euclidean norm is calculated, often there exist a loss of scalar information due to normalization. Authors propose Spherization layer to form a bijective mapping from Cartesian coordinates to hyperspherical surface. Spherization layer can replace typical neural network layers. Empirical experiments, from simple synthetic data to real-world dataset such as MNIST, Mini-ImageNet and WikiText, show that performance of neural network with Spherization layer is scalable and can perform well.   ", "questions": "1. Although authors motivate the formulation in the beginning, stating that scalar information can be lost. What is that information exactly? Is it possible to show empirically, either via a small experiment or via real-world experiment, to show formulations that uses normalization loses information due to scaling?  \n\n2. Can authors further elaborate on what you mean by ”Robustness” in Figure 2? Robustness at first glance might remind some readers of adversarial robustness, or robustness to noise or some deformation on the input. Upon further reading, it seems to mean “maintaining trainability.” I recommend changing the wording here to avoid confusion, or easier readability.", "limitations": "Authors stated potential limitations of their work, in that spherization layer only takes on an effect if its used during training. It is mentioned that it will require retraining in order to use the proposed formulation in conventional pretrained models.  ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The formulation presented is clear and well-written. Background on spherical coordinates in $n+1$ dimensions, and step-by-step explanation of the spherization process is easy to follow with good choice of notation. Experiments are also well-documented. The toy example is a good starting point verifying the formulation. Experiments on both computer vision tasks such as image classification and language tasks such as word analogy are also pluses that show case formulation proposed is versatile for different use cases. Empirically, for some setting, proposed formulation can greatly outperform older related works. The formulation is original, and significant in that it is a flexible formulation and is applicable in many other related works.   \n\nI reserve the weakness and questions together for better coherence.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept