Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new formulation for a layer in neural networks known as Spherization Layer. When inner product with Euclidean norm is calculated, often there exist a loss of scalar information due to normalization. Authors propose Spherization layer to form a bijective mapping from Cartesian coordinates to hyperspherical surface. Spherization layer can replace typical neural network layers. Empirical experiments, from simple synthetic data to real-world dataset such as MNIST, Mini-ImageNet and WikiText, show that performance of neural network with Spherization layer is scalable and can perform well.   
strengths_and_weaknesses: The formulation presented is clear and well-written. Background on spherical coordinates in $n+1$ dimensions, and step-by-step explanation of the spherization process is easy to follow with good choice of notation. Experiments are also well-documented. The toy example is a good starting point verifying the formulation. Experiments on both computer vision tasks such as image classification and language tasks such as word analogy are also pluses that show case formulation proposed is versatile for different use cases. Empirically, for some setting, proposed formulation can greatly outperform older related works. The formulation is original, and significant in that it is a flexible formulation and is applicable in many other related works.   

I reserve the weakness and questions together for better coherence.  
questions: 1. Although authors motivate the formulation in the beginning, stating that scalar information can be lost. What is that information exactly? Is it possible to show empirically, either via a small experiment or via real-world experiment, to show formulations that uses normalization loses information due to scaling?  

2. Can authors further elaborate on what you mean by ”Robustness” in Figure 2? Robustness at first glance might remind some readers of adversarial robustness, or robustness to noise or some deformation on the input. Upon further reading, it seems to mean “maintaining trainability.” I recommend changing the wording here to avoid confusion, or easier readability.
limitations: Authors stated potential limitations of their work, in that spherization layer only takes on an effect if its used during training. It is mentioned that it will require retraining in order to use the proposed formulation in conventional pretrained models.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a spherization layer to represent preactivations in a hypersphere. This layer can be plugged into any neural network. This representation is useful in cases where cosine similarity is used as the main metric to determine similarity in the feature space.
strengths_and_weaknesses: ## Strengths
1. The spherization idea is interesting. It addresses a niche problem in a neat way.
2. The formulation is clean and clear justifications are provided.
3. Experiments are performed in various settings and significant improvements are observed in the few-shot learning setting.

## Weaknesses
1. The improvement is not clear compared to the projection approach (table 5). This raises doubts about the motivation for this approach. Is there really an information loss with the projection approach?
2. More experiments where explicit similarity is used in the feature space could be performed. Examples include self-supervised learning and metric learning. This may show the benefits of the proposed approach in a better way.
3. Training stability of the network could have been discussed. It is not clear, how the signal propagation (gradient flow) would be affected by the spherization layer.

## Post rebuttal
Thanks for the clarifications. I'm in favour of accepting the paper. Please consider addressing the discussed comments in camera-ready or in an extension.
questions: 1. It seems it is possible to theoretically show that the spherization layer retains all information as a Euclidean layer. Meaning, a spherization layer can learn any function that can be represented by a Euclidean layer. Can you comment on this?
limitations: Societal impact is not discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes spherization layer to represent all information on angular similarity, which avoids the information loss when using only angular similarity on representations trained with the inner product. It maps the pre-activations of input vectors into the specific range of angles, converts the angular coordinates of the vectors to Cartesian coordinates with an additional dimension, and trains decision boundaries from hyperplanes. This guarantees that representation learning always occurs on the hyperspherical surface without the loss of any information unlike other projection-based methods. Experiments have been evaluated on different tasks and show the effectiveness of the proposed methods.


strengths_and_weaknesses: Novelty.
This paper proposes a novel method for representation learning only using the angle information. The subtle construction to covert the angular coordinates of the vectors to Cartesian coordinates, which avoids the information loss than the previous method.

Writing.
This paper is of better organized and shows detailed description about the proposed method. 

questions: 
I have no problem
limitations: Weakness.
Some related works should be listed and have a discussion and comparison,
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a novel neural network layer that forces pre-activations into angles. Besides, the spherization Layer converts the angular coordinates to Cartesian coordinates and the training is done without bias parameters. Several different empirical experiments have shown the effectiveness of the proposed method. 
strengths_and_weaknesses: Strengths: 

1. This paper is well written and clear. 
2. The proposed spherization layer can be easily applied to most existing neural network architectures and downstream tasks.

Weakness:  

1. There is not enough comparison between the proposed spherization layer and other recent angle-based learning methods such as [1] [2]. Since the authors claim that the proposed method solves the information loss problem that occurs in these projection-based learning methods, these comparisons are important experiments to support the authors claims.


[1] Chen, Beidi, et al. "Angular visual hardness." International Conference on Machine Learning. PMLR, 2020.
[2] Lin, Rongmei, et al. "Regularizing neural networks via minimizing hyperspherical energy." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.





questions: It would be nice if the authors can add more explanation about the motivation and analysis of the conversion-to-Cartesian process.
limitations: The authors have adequately addressed the limitations of this paper.
review_title: null

Meta-review: metareview: This paper proposes spherization layer by first transforming the pre-activations into angles, then transforming the angles into cartesian coordinates on a sphere, and finally training weight parameters without bias. The proposed spherization layer is geometrically meaningful, and is generally applicable. It is demonstrated on a range of experiments. 

Reviewer v1RE, who gave a rating 5, pointed out two related references and asked for more explanation about motivation. The authors explained the difference between their paper and the two references, and explained the motivation. All other reviewers are positive about this paper. 
confidence: Certain
Decision: Accept