Conic Activation Functions

Authors: changqing  fu
laurent d cohen
marco  fumero
clementine  domine
zorah  lähner
donato  crisostomi
luca  moschella
kimberly  stachenfeld
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-03 14:14:56

Abstract: Most activation functions operate component-wise, which restricts the equivariance of neural networks to permutations. We introduce Conic Linear Units (CoLU) and generalize the symmetry of neural networks to continuous orthogonal groups. By interpreting ReLU as a projection onto its invariant set—the positive orthant—we propose a conic activation function that uses a Lorentz cone instead. Its performance can be further improved by considering multi-head structures, soft scaling, and axis sharing. CoLU associated with low-dimensional cones outperforms the component-wise ReLU in a wide range of models—including MLP, ResNet, and UNet, etc., achieving better loss values and faster convergence. It significantly improves diffusion models' training and performance. CoLU originates from a first-principles approach to various forms of neural networks and fundamentally changes their algebraic structure.

Main paper: 
 introduction: 
 Recurrent neural networks (RNNs), convolutional neural networks (CNNs) and Transformers [Vaswani et al., 2017] are examples of a symmetry principle in neural network architectures: they capture local patterns and uniformly apply them across the entire space. These architectures have laid a solid foundation for modern machine learning systems. RNNs repeatedly apply the same weights to the hidden states. This autoregressive form also inspires diffusion models [Sohl-Dickstein et al., 2015] -the patterns are uniform across intermediate states. Convolution layers share the same weights in a small local window to slide across a large domain-the patterns are uniform at arbitrary spatial positions. In Transformers, the self-attention function applies its weights homogeneously to the word or pixel embedding space-the patterns are uniform in arbitrary directions since a per-vector rotation or reflection on both the embedded query and key vectors does not change the attention mask. Different kinds of pattern uniformity are consequences of the associated space homogeneity. These homogeneities (symmetries) have been a principle that continually inspires new designs of model architectures. Recent works continue to push the limit of model performance in vision or language tasks with reduced complexity and different types of symmetry, such as state space models [Gu and Dao, 2023] and more efficient Transformers [Liu et al., 2023] .The convolution and self-attention functions' symmetries are characterized by the equivarance under spatial translation and vector rotation-a function λ is equivariant under a group G if and only if ∀P ∈ G, P λ = λP , where the operation between them is the composition of functions. The same principle applies to a basic multi-layer perceptron (MLP). First, the same activation function is used recurrently in the same space up to a linear embedding layer; second, it applies uniformly to each vector component (neuron) . The first property is the foundation of deep models using the same activation function. The second one results in permutation symmetry: ReLU is equivariant under G where G contains compositions of permutations and diagonal matrices with non-negative entries (positive scaling). The symmetry in models is induced by the symmetry of hidden states' space: by substituting the equality λ = P -1 λP, ∀P ∈ G into a two-layer neural network f (x) = wλ(w ′ x), the network stays the same except that the group acts on the weights (w, w ′ ) to obtain (wP -1 , P w ′ ), which means the order of rows and columns of the weight matrices are exchanged. While permutation symmetry has been a fundamental assumption in neural networks, we take another path to reflect on this axiomatic assumption and raise the question:Can forms of equivariance more general than permutation improve neural networks?The self-attention function in Transformers positively answers this question. We give a second answer and let activation functions be another solution. To further motivate the activation function, in Appendix G we start from symmetry principles to axiomatically infer the forms of different neural network structures from scratch, where we essentially modify the hypothesis that activation functions are component-wise. We further show in Appendix B that the proposed activation function and the self-attention function share the same type of symmetry, associated with Noether's Theorem. The symmetry group is related to linear mode connectivity explained in Appendix C, meaning that the loss landscape of neural networks is empirically convex modulo the group. Generalizing the group to infinite order fundamentally enlarges the algebraic structure of neural networks.We propose Conic Linear Units (CoLU), which introduces orthogonal group symmetry to neural networks. CoLU outperforms state-of-the-art component-wise activation functions such as ReLU in various models including ResNet and UNet for recognition and generation, and keeps the training and inference speed. It achieves remarkable gains in training diffusion models. 
background: 
 Component-Wise Activations Among the most commonly used activation functions are Rectified Linear Units (ReLU) and its variants, such as Leaky ReLU and Exponential Linear Units (ELU) [Clevert et al., 2015] . There are also bounded ones, such as the sigmoid function or the hyperbolic tangent function used in Hochreiter and Schmidhuber [1997] . In state-of-the-art vision and language models, soft approximations of ReLU are preferred for their better performance, such as Gaussian Error Linear Units (GELU) [Hendrycks and Gimpel, 2016] , Sigmoid-Weighted Linear Units (SiLU) [Elfwing et al., 2018] , etc. All these functions are component-wise.Non-Component-Wise Activations Previous works proposing non-component-wise activation functions are essentially different from CoLU, such as using layer normalizations [Ni et al., 2024] or multiplying the input by a radial function [Ganev et al., 2021] . In comparison, CoLU is a generalization of common activations, keeps the favorable conic-projective property unchanged, and improves the performance of neural networks. In the previous version of CoLU [Fu and Cohen, 2024] , it had not yet achieved universal improvement on all types of models, since its variants had not been developed.Equivariance in Linear Layers For symmetries in the linear part of the model, ensuring different equivariance improves the performance of recognition [Zhang, 2019] and generation [Karras et al., 2021] models, which repeatedly confirm the potential benefits of the symmetry principle. Group equivariant convolutional neural networks (GCNN) [Cohen and Welling, 2016] put symmetry constraints in the spatial domain so that the model admits spatial group actions such as 2D spatial rotations and reflections. Like in most convolutional neural networks, the channel dimensions of GCNNs are always fully connected. CoLU's symmetry assumption is on the channel axis of the states, which means that CoLU considers the tangent space of GCNN's symmetry space, and equally applies to fully connected layers without convolution structures.Invariant scattering convolutional networks [Bruna and Mallat, 2013] use wavelet bases as deterministic spatial correlations and only learn the pixel-wise linear layer or 1 × 1 convolution. It indicates that learning channel correlation plays a primary role in representing data patterns compared to spatial connections, and it motivates further investigations into general symmetries in the channel dimensions-the embedding space. Low-rank adaptation [Hu et al., 2022] and the Query-Key embeddings in the self-attention function are examples of putting low-rank assumptions in the embedding space to represent patterns efficiently. CoLU considers another assumption: it assumes potential subspace orthogonalities.Orthogonality in the Embedding Space Ensuring orthogonality of the embedding space in the linear layers is twofold. The hard constraint method uses a projection onto the Stiefel manifold during training to ensure the orthogonality of the weights [Jia et al., 2019] . The soft constraint method adds a regularization term to the loss function [Wang et al., 2020] and learns the orthogonality approximately. Orthogonal CNNs outperform conventional CNNs, suggesting that the orthogonality property helps neural networks gain robustness and generalization ability. The self-attention function in Transformers is also orthogonal equivariant. CoLU is compatible with these orthogonal layers to allow layerwise orthogonality in consecutive layers.Other Constructions in Nonlinearities Weiler and Cesa [2019] conduct a survey on some of the nonlinear functions for equivariant networks, which does not cover the form of CoLU. Liu et al. [2024] , Mantri et al. [2024] propose essentially component-wise nonlinearities by leveraging other properties, where the equivariance is still restricted to permutations.3 Conic Activation Functionsx ⊥ x 1 x λ(x) V H(x) (a) Conic projection x ⊥ x 1 wλ(R C ) V (b) Affine transform of a coneFigure 1 : Illustration of a CoLU function λ and an affine transform w of a cone V .x 1x 2 x 3 x ′ 1 x ′ 2 x ′ 3 x ′ 4 x ′ 5 x ′ 6 y ′ 1 y ′ 2 y ′ 3 y ′ 4 y ′ 5 y ′ 6 y 1 y 2 y 3 y 4 1 Pointwise λ : R → R w ′ λ λ λ λ λ λ w x 1 x 2 x 3 x ′ 1 x ′ 2 x ′ 3 x ′ 4 x ′ 5 x ′ 6 y ′ 1 y ′ 2 y ′ 3 y ′ 4 y ′ 5 y ′ 6 y 1 y 2 y 3 y 4 2 Conic (One Cone) λ : R C → R C w ′ w x 1 x 2 x 3 x ′ 1 x ′ 2 x ′ 3 x ′ 4 x ′ 5 x ′ 6 y ′ 1 y ′ 2 y ′ 3 y ′ 4 y ′ 5 y ′ 6 y 1 y 2 y 3 y 4 3 Grouped (Two Cones) λ : R C/G → R C/G w ′ w x 1 x 2 x 3 x ′ 1 x ′ 2 x ′ 3 x ′ 4 x ′ 5 x ′ 1 x ′ 2 x ′ 3 x ′ 4 x ′ 5 y 1 y 2 y 3 y 4 4 Grouped + Axis-Shared λ : R (C-1)/G+1 → R (C-1)/G+1 w ′ wFigure 2 : Connections between neurons in a two-layer neural network y = wλ(w ′ x) with component-wise / conic / group-conic / shared-axis group-conic activation functions. In this illustrative example, the network width is C = 6 except that in the last shared-axis case C = 5. The number of cones is G = 1 when there is one cone and G = 2 in the grouped case. The yellow arrows denote the maximum norm threshold on the output vector in each group, and the dashed frames denote the cones' axis dimensions.A basic conic activation function is defined as λ :EQUATIONwhere x = (x 1 , x 2 , . . . , x C ) is the input vector, | • | is the Euclidean norm, ε is a small constant taken as 10 -7 for numerical stability, and x ⊥ denotes the normal vector x ⊥ = (0, x 2 , x 3 , . . . , x C ), so that x = x 1 e 1 + x ⊥ holds. Here e 1 = (1, 0, . . . , 0) ∈ R C is a unit vector. Figure 1a visualizes a CoLU function with a red arrow and Figure 1b visualizes a transformed cone with a linear layer after CoLU. Figure 2 visualizes the connections between neurons of the basic CoLU and its variants to be defined in the sequel. The complexity of CoLU is O(C), which is the order of component-wise functions and is negligible compared to matrix multiplications. The design of CoLU is irrelevant to the choice of the first axis or another one since its adjacent linear layers are permutation equivariant. Figure 5 : The correlations between weights cov(w ′ ) and between states cov(x ′ ).The bright areas on the topleft corners are the correlated axes. 
soft scaling: 
 The sigmoid-weighted conic activation function is defined asEQUATIONwhere sigmoid(x) = 1/(1 + exp(-x)). Compared with Equation (1), the weighting function min{max{r, 0}, 1} is replaced by sigmoid(r -1/2), where r = x 1 /(|x ⊥ | + ε) is the cotangent value of the cone's opening angle α, r → 1/ tan(α) as ε → 0.The soft projection is inspired by the better performance of smooth functions such as SiLU λ(x) = sigmoid(x)x, compared to the piecewise-linear ReLU λ(x) = 1 R ≥0 (x)x. Figure 3 compares ReLU weighting with its sigmoid-weighted variant SiLU. Figure 4 compares the hard projection in Equation ( 1), firm projection weighted by sigmoid(4r -2) and sigmoid-weighted soft projection in Equation (2).Inspired by group normalization [Wu and He, 2018] , group convolution [Krizhevsky et al., 2012] , etc., the channel dimension can be divided into G heads of dimension S = C/G. The groupconic activation function is defined as a group-wise application of the conic activation function. Suppose λ : R S → R S is defined in Equation ( 1) or (2), and π G i : R C → R S , i = 1, 2, . . . , G are the G-partition subspace projections, then λ in higher dimension C is uniquely characterized byEQUATIONIn the trivial case G = 0, there is no axis to project towards, and we specify that the activation function coincides with the identity function. In the special case S = 2 or when the cones are in a 2D space, the 1D cone section degenerates to a line segment with no rotationality, so we specify that the CoLU coincides with the component-wise activation function.The shared-axis group CoLU is also uniquely defined by π G i λ = λπ G i , i = 1, 2, . . . , G but with the G-partition subspace projections defined differently:EQUATIONwhere π j , j = 1, 2, . . . , C are projections to each axis. π G i is a projection onto the first dimension (the cone axis) and S -1 other consecutive dimensions (the cone section). Therefore the relation among the dimension formula among (C, G, S) is C -1 = G(S -1) in the shared-axis case.Figure 5 illustrates the motivation of axis sharing: the colinear effect in the hidden states. In this example, w ′ is the first linear layer of a VAE's encoder x ∈ R 784 → wλ(w ′ x) ∈ R 20 pretrained on the MNIST dataset, and x ′ is the first hidden state x ′ = w ′ x ∈ R 500 where the 100 cone axes are permuted together for visualization. Therefore, the hidden dimension is C = 500, the number of groups is G = 100, the number of test examples is 10000, w ′ ∈ R 784×500 , x ′ ∈ R 10000×500 and cov(w ′ ), cov(x ′ ) ∈ R 500×500 . The upper-left parts of the matrices are very bright, meaning that the axis dimensions are highly colinear.An alternative form of CoLU ensures component homogeneity, by rotating the standard Lorentz Cone towards the all-one vector, and we call it a rotated conic activation function (RCoLU)EQUATIONwhere x e = x • e, x ⊥ = xx e and e = (1/ √ S, . . . , 1/ √ S). The axis-homogeneous cone avoids splitting operations in the calculation. It can also be combined with grouping using Equation (4), and with axis sharing by setting e = (1/ √ C, . . . , 1/ √ C) in Equation ( 5) instead of using Equation (4). RCoLU's performance boost over ReLU is similar to standard CoLU, so we omit it in the experiment section. 
why conic activation functions: 
 CoLU is motivated by the conic projection, which generalizes the equivariance in a neural network. The proofs are provided in Appendix E.To naturally characterize this projection, it is necessary to recall hyperbolic geometry detailed in Appendix A, where we define the Lorentz cone (the future Light Cone)V = {x ∈ R C : x 2 1 -x 2 2 - . . . -x 2 C ≥ 0, x 1 ≥ 0} and the hyperplane of simultaneity H(x) = {y ∈ R C : y 1 = x 1 }. We denote V = R ≤0 e 1 ∪ V , where R ≤0 e 1 = {(t, 0, . . . , 0) ∈ R C : t ≤ 0}. Definition 4.1 (Conic Projection). The conic projection is defined as x ∈ R C → π V ∩H(x) (x) where π is the nearest point projection, π A (x) = argmin y∈A |y -x|.The restriction of the projection on its image V is the identity function, so it satisfies the idempotent property λ 2 = λ. Constraining the projection in H(x) simplifies the computation while maintaining essential equivariance properties-it guarantees that the projection is always towards the cone axis. Since V ∩ H(x) = ∅ when x 1 < 0, the projection is not feasible in the negative half-space, so V is extended to V for the well-definedness-on the negative half-space, the projection is degenerate, π V ∩H(x) (x) = (x 1 , 0, . . . , 0). In other words, the past Light Cone has zero light speed and thus zero opening angle. Lemma 4.2 (CoLU is Conic Projection). Suppose λ is defined in Equation (1), then it coincides with a conic projection.EQUATIONwhereD = {x ∈ R C : x 1 = 1, C i=2 x i ≤ 1} is the (C -1)-dimensional disk.We note that V is the conic hull of D, and D is isometric to a hyperball in dimension C -1, and therefore it has the symmetry group O(C -1). In comparison, the invariant set of ReLU is the convex hull of the C -1 simplex ∆ C-1 , defined as the convex hull of the unit vectors {e i ∈ R C : i = 1, 2, . . . , C}. Next, we discuss the general link between algebraic and geometric symmetry.Inspired by the Erlangen program [Klein, 1893] bridging algebraic groups with geometric forms, the equivariant group is more intuitively motivated by the symmetry of the projections' invariant sets. Definition 4.3 (Invariant Set). The invariant set of a function λ : R C → R C is defined asI λ = {x ∈ R C : λ(x) = x}Moreover, the symmetry group G and the isometric symmetry group G * of a set A is the group of affine and rigid functions that preserves the set:G A = {P ∈ GA(C) : P (A) = A}, G * A = G A ∩ E(n)where GA(C) is the general affine group, andE(n) = {P ∈ Map(R C ) : |P (x) -P (y)| = |x -y|, ∀x, y ∈ R C } denotes the Euclidean group.Definition 4.4 (Symmetry Group). The equivariance group and the isometric equivariance group of a function λ : R C → R C is defined asG λ = {P ∈ GA(C) : P λ = λP }, G * λ = G λ ∩ E(n) Lemma 4.5 (Projective-Type Operators). If λ is either ReLU or CoLU, then G λ = G I λ , and G * λ = G * I λ .This algebra-geometry duality applies to more general neural network architectures, such as the self-attention function. The relation with Noether's theorem is discussed in Appendix B. Corollary 4.6 (Permutation Symmetry). Suppose λ is the component-wise ReLU, then I λ = R C + , G λ = G I λ = S(C) and G * λ = G * I λ = Perm(C), where R C + = {x ∈ R C : x i ≥ 0, i = 1, 2, . . . ,EQUATIONwhere I λ = V G . In the shared-axis case,I λ = V G / ∼ where the relation ∼ is defined as x ∼ y if and only if ∃i, j ∈ {1, 2, . . . , G} such that π G i (x) 1 = π G j (y) 1 and ∀k ∈ {2, 3, . . . , S}, π G i (x) k = π G j (y) k = 0.In Equation ( 7), S(G) represents the permutations among different cones and O(S -1) represents rotations or reflections within each cone. The motivation is that matrix conjugation modulo permutations reduce to block diagonal form, and we assume there are low-dimensional block sub-spaces that can hold orthogonal equivariance. The symmetry group is continuous and thus of order infinity, unprecedented in component-wise activations. We use the following construction to illustrate that it improves neural networks' generalization ability since component-wise activations fail to hold orthogonal equivariance whereas conic activations do. Lemma 4.8 (Layerwise Orthogornal Equivariance). Assume a two-layer neural network y = f θ (x) = wλ(w ′ x) with fixed width C and the training data D satisfies subspace orthogonal symmetry:∀(x, y) ∈ D, ∀P ∈ G, (P x, P y) ∈ G, where G = {P ∈ GL(C) : P [1, 2; 1, 2] ∈ O(2), P [3, . . . , C; 3, . . . , C] = I C-2 , P [1, 2; 3, . . . , C] = P [3, . . . , C; 1, 2] ⊤ = 0} ≃ O(2). Then, (1) (ReLU excludes orthogonal equivariance) If λ is component-wise activation function, then ∀θ ∈ (R C 2 \{0}) 2 , ∃x ∈ R C and P ∈ G such that P f θ (x) ̸ = f θ (P x).(EQUATIONAs a remark, we explain the sufficiency of rigid alignments with a compact group G * without scaling by adding a least-action regularization term, to justify the common practice in the literature, which answers the open issue of permutation-only alignments in Bökman and Kahl [2024] .Remark 4.9 (Soundness of Isometric Alignment). Suppose L is the alignment objective defined in the algorithms in Appendix F, then ∃η > 0 such that the regularized alignment coincides with isometric alignment: argmin P ∈G λ (L(P ) -η∥P ∥) = argmin P ∈G * λ L(P ), where ∥θ∥ = w ( i w p i ) 1/p is some norm of order p ≥ 1.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Yes, open-source code and data are presented. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 The experiments are conducted on an 8-core Google v4 TPU. For computational costs, CoLU introduces negligible computational overhead compared to ReLU in all experiments and all variants.5.1 Synthetic Data To demonstrate the advantage of the generalized symmetry of CoLU in the embedding space, we use a two-layer MLP to learn the rotation of a 2D hemisphere. The MLP is defined as1 0 1 1 0 1 1 0 1 x 0 1 2 0 1 2 0 1 2 (w x) 1 0 0 1 0 1 w (w x) 1 0 0 1 0 1 y (a) ReLU 1 0 1 1 0 1 1 0 1 x 0 1 2 0 1 2 0 1 2 (w x) 1 0 0 1 0 1 w (w x) 1 0 0 1 0 1 y (b) RCoLUx ∈ R 3 → wλ(w ′ x), where w, w ′ ∈ R 3×3 . The dataset D consists of polar grid points and their rotated counterparts (x, y = Rx), where R represents a rotation of 45 • around each of the three coordinate axes. As shown in Figure 6 in the third column, ReLU does not capture orthogonal equivariance (rotation around the hemisphere axis) near the equator, instead projecting the boundary onto a triangle. In contrast, RCoLU successfully preserves the rotational symmetry at every latitude, including at the boundary. This is due to the geometry of the projection boundary: ReLU cuts the hemisphere with the positive orthant and produces a boundary of the 2-simplex ∆ 2 , whereas CoLU projects onto a cone that naturally preserves the circular pattern.The toy generative model is a VAE with a two-layer encoder and a two-layer decoder, trained on the binarized MNIST dataset. The test loss is compared since CoLU is hypothesized to increase the model's generalization ability.Experimental Settings We use the Adam optimizer with a weight decay of 10 -2 and train 10 epochs for each run. The global batch size is set to 128 and the learning rate is set to 10 -3 . Each configuration is trained for 10 times with different random seeds. More detailed settings are provided in Appendix D. According to the hyperparameter search above, we set the cone dimensions to S = 4, which complies with the number of chips in hardware platforms. We compare test accuracies in the MNIST recognition tasks to test the hypothesis of CoLU's generalization ability.Experimental Settings We set the global batch size to 1024 and the learning rate to 10 -3 . Each configuration is trained 7 times with different random seeds. More detailed settings are provided in Appendix D. Results Table 3 compares ReLU with CoLU of low-dimensional orthogonal subspaces and shows the improvement from using axis sharing combined with soft projection.To test the performance of CoLU in deeper models, we scale up the network to ResNet-56 and train them on the CIFAR10 dataset. Axis sharing and soft projection are omitted for clean comparisons with ReLU in the sequel.The ResNet architecture and the training recipe follow He et al. [2016] .The runs are repeated for 10 times with different random seeds each lasting 180 epochs, and use the Adam optimizer with a batch size of 128, a learning rate of 10 -3 , and a weight decay coefficient of 10 -2 . Finer training settings will achieve better baselines, and CoLU remains superior to ReLU. Results Table 4 shows that CoLU outperforms ReLU and the training is stable across different initialization seeds.We compare CoLU and ReLU in unconditional generation with diffusion models [Sohl-Dickstein et al., 2015] trained on the CIFAR10 and Flowers datasets. Then we show the possibility of borrowing a pretrained text-to-image model [Rombach et al., 2022] We replace all activation functions in the UNet with soft-projected conic activation functions of G = 32 without axis sharing. Appendix D shows generated samples from the fine-tuned model and visually compares the original activation and CoLU models.CoLU is better than ReLU in the MLP part of a Generative Pretrained Transformer (GPT2) trained on Shakespeare's play corpus. Appendix D reports a comparison in the test loss. We also observe that CoLU achieves slower overfitting and lower test loss with the same training loss.CoLU enlarges the group of neural networks' linear mode connectivity, explained in Appendix C.Convolution Filter Symmetry Diffusion models with ReLU and CoLU have different symmetry patterns in the convolution filters. We show in Appendix D that between the last layer of two diffusion UNets trained with different initialization on CIFAR10, a ReLU model's convolution filters can be permuted to match each other, whereas a CoLU model cannot since the orthogonal symmetry relaxes to additional color rotations.Generative Model Alignment For completeness, we show alignment results on the ReLU and CoLU-based models in Appendix D. In the literature on linear mode connectivity, few works study generative models, and we show that the generative VAEs also reveal linear mode connectivity under the equivariance groups of activation functions.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: No dataset is released. The code is provided as supplemental.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 In this work, we introduced Conic Linear Units (CoLU) to let neural networks hold layerwise orthogonal equivariance. CoLU outperforms common component-wise activation functions and scales to a broad range of large models. The code will be publicly available at https://github.com/ EvergreenTree/di-f-fu-sion. x is defined in the C-dimensional Euclidean space (called space-time). A scalar product on R C is defined asEQUATIONThe hyperbolic geometry can be understood by the fact that along a rotation in the space, the quantity x 2 1x 2 2 -. . . x 2 C is unchanged. This scalar product induces a norm |x| M = ⟨x, x⟩ M , and the Lorentz cone is defined asV = {x ∈ R C : |x| M ≥ 0, x 1 ≥ 0}.It is usually called a light cone since if we regard x 1 := t as the time axis where the constant c is the speed of light and t is the time of the event x, then the cone is characterized by |x ⊥ | = ct, and c is the tangent value of the opening angle of the cone, and we set c = 1 without loss of generality. More precisely it is a future light cone since t ≥ 0, and the past light cone associates to the case when t ≤ 0. CoLU sets the past light cone with c = 0. The plane of simultaneity (under the rest frame of reference) is defined as 1a , CoLU is intuitively understood as the closest point to the input within the light cone and the plane of simultaneity. The meaning of the weight w after the activation function is visualized in Figure 1b , where the previous space-time is tilted by a linear transform (called Lorentz transform). In the grouped CoLU case, gluing the axes together is motivated by equalizing the time axes of each light cone (called an observer).H(x) = π -1 1 (x 1 e 1 ) = {y ∈ R C : y 1 = x 1 }. In Figure 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Yes, they are noted.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: It is an improvement regarding low-level architectures, so we do not find any negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Code, data, models used in the paper are credited. License and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we associate the CoLU equivariance with the conserved quantity in the tangent space of the spatial domain and show that CoLU and self-attention have the same type of symmetry. Definition B.1 (Lagrangian). A Lagrangian functional is defined as an integral L :T M -→ R such that L(x, ẋ, L) = L 0 L(x, ẋ, ℓ) dℓ (9)Theorem B.2 (Noether). Suppose ∀s ∈ R the Lagrangian L(x, ẋ, L) is invariant over a transformation h s parameterized by s, then the following quantity is constant over time.EQUATION)Corollary B.3 (Translation Momentum). Assume ω ∈ Ω = [-1, 1] 2 , e 1 = (1, 0) is a unit vector, and L(ω, ω, t) = ω2 /2. If h s (ω) = ω + se 1 then I = ω1 is conserved.The convolution function commutes with h s and associates with the translation momentum on Ω. s : x(σ) ∈ R Σ → R 2s/π x(σ), then I = ẋ × e 1 .Proposition B.5 (Attention Invariance). The self-attention function commutes with h s , so the Lagrangian of attention dynamics admits the orthogonal group. Therefore the attention dynamics in Equation ( 39) conserves the angular momentum for rotations in R C . Proposition B.6 (Conic-Activation Invariance). For the same reason as above, if the activation function is conic, The ResNet dynamics in Equation ( 40) conserves the angular momentum for rotations around the cone axis.The equivariance of activation functions is linked to the linear mode connectivity phenomenon: two neural networks trained with different initializations and (usually) on the same dataset can be aligned to be very close to each other [Izmailov et al., 2018 , Singh and Jaggi, 2020 , Entezari et al., 2022 , Ainsworth et al., 2023] . This phenomenon implies that neural network optimization is approximately convex modulo a group. The group characterizes the permutation symmetry of component-wise activation functions, and the proposed conic activation functions generalize the type of symmetry. This aligned representation phenomenon across different models at a larger scale is discussed in [Huh et al., 2024] . Note that there are other types of mode connectivity [Garipov et al., 2018] , which does not leverage permutation symmetry and requires more complicated paths such as piece-wise linear or Bézier spline, and we do not discuss here.Given the loss function L(θ) on two sets of model parameters θ 0 , θ 1 , the closeness of the two models is measured by the loss barrier. There are different definitions of loss barriers, and we define it asEQUATIONThe loss barrier signifies the relative loss increase of the linearly interpolated weights. With one model θ 0 fixed, an alignment on the other one θ 1 refers to finding the optimal permutation on each layer by matching either intermediate states or weights [Jordan et al., 2023 , Ainsworth et al., 2023] .The proposed activation function generalizes permutations to orthogonal matrices (where permutations are special cases). The orthogonal symmetry is continuous, meaning that there are infinitely many ways of alignment. This results in a loss landscape with infinite local minima forming connected components. The alignment matrices are associated with different manifold constraints.Experimental Settings The VAE's encoder and generator's parameters are θ E = (w E , w ′ E ) and θ G = (w G , w ′ G ). The inputs, latents and outputs are x, z, x, where z = w E λ(w ′ E x) and x = w G λ(w ′ G z). The dimension of input and output is 28 × 28 = 784 and the dimension of the hidden state z is fixed to d = 20. The loss function is defined asEQUATIONwhereH(x, x) = -n x n (log( x n ) + (1 -x n ) log(1 -x n ))is the binary cross-entropy, and D KL (p z |p 0 ) is the Kullback-Leibler Divergence from a standard Gaussian distribution p 0 ∼ N (0, 1) to the latent distribution p z ∼ N (µ z , σ z )D KL (p z |p 0 ) = - x p 0 (x) log(p z (x)/p 0 (x)) dx = 1 2 d j=1 1 + log(σ z 2 j ) -µ z 2 j -σ z 2 j (13)The last equality is obtained by setting µ z = (N n=1 z n )/N and σ z = (( N n=1 (z n -µ z ) 2 )/(N - 1)) 1/2with sample size N . The hyperparameter α is set to 1 so that the impact of the KL term is relatively small, given that the magnitude of the cross-entropy term is around 100 times larger.Figure 8 visualizes the test loss curves when the granularity of grouping varies. In summary, as the cone dimension S reduces, the performance of grouped conic activation functions improves until it outperforms component-wise activation functions. In the figures, only one case for high and low dimensional cones is shown for clarity.The cone dimension is among S ∈ {∞, 2400, 600, 4, 2}, or equivalently, the number of groups is among G ∈ {0, 1, 4, 800, 2400}. In the shared-axis case, the network width is fixed to C = 2401 so that the number of parameters is the same. In the no-sharing case, the network width is C = 2400 + G. Specifically, G = 0 reduces to the identity function, while S = 2 (or G = 2400) is specified as the component-wise activation of ReLU for hard projection and SiLU for soft projection since there is no orthogonality. The dashed line in the hard-projected shared-axis case means that the training is unstable over different random seeds: about 80% of the initializations do not converge, so only one converged training instance is visualized. Sharing the axes increases the performance of the activation function on toy examples when the projection is soft and the cones are low dimensional (S being small). In practice, this combination is the most meaningful one since it has the best performance and saves the most number of parameters. Intuitively, soft projection effectively stabilizes the training of CoLU models, which is the most obvious in the early training stage of the highest-dimensional conic functions (the single cone case). Especially, it makes the VAE with shared-axis activation functions easier to train.Experimental Settings The model is parameterized by θ = (w, w ′ ) and defined as x ∈ R 28×28 → y = softmax(wλ(w ′ x)) ∈ ∆ 9 , which is a two-layer MLP whose output is mapped to the probability simplex by a softmax function. The MNIST dataset is denoted as a collection of data pairs (x, y), where x is flattened as vectors and y is a unit vector among 10 classes. The network width is fixed to C = 512. The loss function is the cross entropy of the predicted probability relative to the labelH( y, y) = i y i log y i (14)Training Experimental Settings The UNet structure follows the Stable Diffusion model (LDM) [Rombach et al., 2022] without the VAE part. The network block widths are set to (128, 256, 256, 256) and the numbers of ResNet blocks are set to 1 for CIFAR10 (2 for Flowers).For unconditional generation, the cross-attention function is replaced with the self-attention function. All runs last 100K steps and use the Adam optimizer with a batch size of 128 for CIFAR10 (16 for Flowers), a learning rate of 10 -4 , and a weight decay coefficient of 10 -2 . Figure 9 shows comparisons on the Flowers dataset. Experimental Settings The Transformer follows Vaswani et al. [2017] with the block size of 64, an embedding size of 256, a number of heads 8, head size 32 and number of layers 6. Each run lasts 20K steps and uses the Adam optimizer with a batch size of 512, a learning rate of 10 -4 , and a weight decay coefficient of 10 -2 . Results Table 6 shows that CoLU is on par with ReLU in GPT2's MLP. We also observe a faster drop in the test loss and slower overfitting.The latent state's permutation symmetry is studied qualitatively on diffusion UNets and quantitatively on toy models. Generative Model Alignment We show linear mode connectivity results for the same toy model in Section 5.2, and we find out that linear mode connectivity also holds in generative models, which is rarely discussed in the literature.Weight matching and state matching algorithms in Appendix F are applied to align the VAE model, and the results are shown in Figure 12 . They have different advantages: weight matching produces a flatter barrier in our toy experiment, and state matching requires no data as the model input. Their convergence is analyzed in [Ainsworth et al., 2023 , Jordan et al., 2023] . The difference in the conic case is that the symmetry group is relaxed, so the Stiefel manifold optimization problem replaces the sum of bilinear assignment problem (SOBLAP). Figure 13 and 14 visualize the loss barrier and the KL Divergence barrier.Proof of Proposition 4.2. If |x ⊥ | ̸ = 0, Equation ( 6) holds component-wise, and the set {x ∈ R C :|x ⊥ | = 0} is negligible.Proof of Lemma 4.5. We assumeP ∈ GA(C) To prove G λ ⊂ G I λ , it suffices to show ∀P ∈ G λ , I λ = P P -1 I λ = P I λ ⊂ I λ .The last inclusion comes from ∀P ∈ G, there holds ∀x ∈ I λ , λ(P x) = P λ(x) = P x, so P x ∈ I λ . The first equality is from P ∈ G λ and the second one is from x ∈ I λ . Conversely, to prove G I λ ⊂ G λ , we need to strengthen the condition on λ to ∃A a convex set such that ∀x, λ(x) = P A (x). ∀z ∈ I λ , ⟨z -P λ(x), P x -P λ(x)⟩ ≥ 0, so λ(P x) = P λx Proof of Proposition 4.8. (1) is proven by taking x and P such thatw ′ x = (1, 0, 0, . . . , 0), P [1, 2; 1, 2] = √ 2 2 - √ 2 2 √ 2 2 √ 2 2(2) is proven by takingw † = 0 I 2 I C-2 0 , w ′ † = 0 I C-2 I 2 0Proof of Remark 4.9. It suffices to take η large enough so that D ∈ Diag(C) is determined by argmin P D∈G λ ∥P Dθ∥, since P ∈ Perm(C) does not change ∥P θ∥.Algorithm 1 and 2 from Jordan et al.[2023], Ainsworth et al. [2023] are applied to achieve linear mode connectivity of the toy VAE model.This section aims to establish a bottom-up framework from first principles to infer the form of neural network architectures, including the proposed activation function. For simplicity, we assume that each state is defined in a vector space with a fixed dimension M = R C . We separate the construction into several parts, including a general Neural Network, a Residual Network, a Convolutional Network, and an Attention Network.Proposition G.1 (Derivation of a Neural Network). The assumptions on the left of the following equations characterize the neural network in Equation ( 20).Require: θ 0 , θ 1 ▷ Pre-Trained Weights from different random initializations Require: x ∈ X ▷ Intermediate states ordered by forward pass Require: θ prev (x), θ next (x)▷ Linear weights prior to and after the state Require: G ▷ Symmetry group of the activation function Ensure:P = {P x : x ∈ X} ▷ Optimal alignment 1: Initialize P x = I dim(x)▷ Identity matrices with the same dimension of x 2: repeat 3: for x in RandPerm(X) do ▷ Shuffle the order of the states 4:L(P ) = 0 5: for w ′ in θ prev (x) do 6:for w in θ next (x) do 7:L(P ) ← L(P ) + tr(w ′ 0 ⊤ P w ′ 1 )/|θ prev (x)| + tr(w 0 P ⊤ w ⊤ 1 )/|θ next (x)| 8:end for 9:end for 10:Solve P x ← argmin P ∈G L(P )11:end for 12: until P ConvergesAlgorithm 2 State Matching Require: θ 0 , θ 1 , θ prev (x), θ next (x), G ▷ Same as above Require: x(0) ▷ Data as model input Require: x ∈ X(θ, x(0))▷ Following the order of the forward pass Ensure:P = {P x : x ∈ X(θ, x(0))} ▷ Optimal alignment 1: Initialize P x = I dim(x) 2: for (x 0 , x 1 ) in (X(θ 0 , x(0)), X(θ 1 , x(0))) do 3: Solve P ℓ ← argmin P ∈G L(P ) = x 0 ⊤ P x 1 4:for w ′ in θ prev (x) do 5:w 1 ← P ℓ w 1 6:end for 7:for w in θ next (x) do 8:w 1 ← P ℓ w 1 9:end for 10: end forEQUATIONEQUATIONIn the derivation above, equation ( 15) denotes an arbitrary function Λ with input x(0) and output x(1). Equation ( 16) holds by assuming the function decomposes into several ones, resulting in a process or a sequence of states x(0), x(1), . . . , x(L) ∈ M , where the terminal states x(0) and x(L) are the input and output. Equation ( 17) follows from assuming the sequence of functions to perform in a linear kernel space. Suppose the linear kernel function at layer ℓ parameterized by w ′ Φ :M -→ M λ x -→ w ′ (ℓ)xon the kernel space, there is a function Λ : M λ → W λ where W λ is the range of the activation function. Then the inverse kernel function is parameterized by wΦ : W λ -→ M x -→ w(ℓ)xAgain we assume for simplicity that the dimensionality of each kernel space is fixed: 17) is obtained by replacing x ∈ M with x ′ ∈ M λ in equation ( 16) and plugging in the change of variable x ′ = wxw ′ . Equation ( 18) is obtained by assuming time homogeneity modulo a linear group of the nonlinear functions: the function Λ is on the lifted space M λ in Equation ( 17) instead of M , where the lifting is determined by assuming that there exist proper w, w ′ in each space such that the functions are uniform over time, meaning Λ 1 = Λ 2 = . . . = Λ L = Λ. Equation ( 19) assumes that there exists a function λ : R → R so that the nonlinear function is represented asM λ = M = R C . Equation (Λ(x 1 e 1 + . . . + x n e n ) = λ(x 1 )e 1 + . . . + λ(x n )e n .In this paper, we replace this assumption with orthogonal symmetry instead. Note that the component-wise λ : M λ → M λ is equivariant under any permutation P . Equation ( 20) rewrites the process into steps between adjacent states. Proposition G.2 (Derivation of a Residual Network). Adding more assumptions, we continue to derive the form of a Residual Network in Equation ( 27).Linear SplittingEQUATIONEQUATIONEQUATIONEQUATIONIn the derivation above, Equation ( 21) splits the inverse kernel function's weight w into the identity (zeroth-order) part and the first-order part w -1. Equation ( 22) re-parameterize the weights by denoting 1-w as w without loss of generality. Equation ( 23) modifies the assumption in Equation ( 17) so that two copies of kernel functions are parameterized by w ′′ , w ′ , and the inverse kernel function remains the same. Equation ( 24) modifies the assumption in equation ( 18) different functions λ ′ , λ applies on each one. Equation ( 25) assumes that the first kernel function w ′′ is identity. Equation (26) further assumes w ′ is identity to simplify equations in the sequel. Equation ( 27) assumes that the function associating to the zeroth-order kernel space is identity. Proposition G.3 (Derivation of a Convolutional Network). Given a basic neural network, the form of a convolutional neural network in Equation ( 33) is determined by the following additional assumptions on the left.Space IndexationEQUATIONIn the derivation above, Equation ( 28 29) write the matrixvector product in the form of a summation. In Equation ( 30) we imposes core assumption of the Convolutional Neural Network, namely the spatial translation equivariance, so that (wx)(ωω ′′ ) = (wx(ω -ω ′′ )), ∀ω ′′ . This results in w(ω, ω ′ +ω ′′ ) = w(ω -ω ′′ , ω ′ ), ∀ω ′′ , so w(ω, ω ′ ) must take the form of w(±ω ∓ ω ′ ), and we set w(ω ′ω) without loss of generality. Equation ( 31) is a change of variable, replacing ω ′ω with ω ′ . Equation ( 32) imposes the condition that the spatial dependency on Ω is within a 3 × 3 neighbourhood. Note that the family of neighbourhoods defines the Topology of the space Ω. Finally, Equation ( 33) denotes the linear function with the ⋆ notation. Proposition G.4 (Derivation of an Attention Network). The construction of the cross-attention function is proceeded by imposing further assumptions.EQUATIONIn the above derivation, Equation (34) assumes the Topology to be discrete, or the neighbourhood of a spatial point is itself, which restricts the convolution to be on a 1 × 1 window. For the matrix w(ℓ, σ, σ ′ ) with σ, σ ′ ∈ [C], Equation ( 35) applies the linear transform k T k, where k(ℓ, σ ′′ , σ) can be regarded as a set of 36) assumes λ to be identity function denoted as 1. Equation ( 37) scales xw T with a softmax function softmax(x(σ, σ ′′ )) = exp(x(σ, σ ′′ ))/ σ ′′ ∈[C ′′ ] exp(x(σ, σ ′′ )). Finally, Equation ( 37) is obtained from setting the Query-Key-Value notations Q = x(ℓ -1, ω, σ ′ ), K = V = k(ℓ, σ ′ , σ ′′ ). Note that by cancelling the assumption in Equation ( 26), we may also take inC ′′ condition "pixels" of dimension C, or ω ∈ [C], ω ′′ = [C ′′ ]. Equation (Q = w Q (ℓ, σ, σ ′ )Q ′ , K = w K (ℓ, σ, σ ′ )K ′ , V = w V (ℓ, σ, σ ′ )V ′ .Proposition G.5 (Attention Network Dynamics). EQUATIONJustification: Yes, assumptions and proofs are provided. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Proceedings of the II edition of the Workshop on Unifying Representations in Neural Models (UniReps 2024). 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Big on theoretical motivation, short on emperical results
main_review: main_review: Strengths:
- The main theoretical motivation of the proposed activation function is strong: specifically that permutation symmetry and the work of Ainsworth et al. specifically, suggests NN optimization is approx. convex modulo a group. This motivation got me excited about the research direction.
- Well-written paper overall.
- Good background, covers everything needed to understand the work well.
- Excellently written and presented method section, specifically Figure 1 which I believe explains the method well.

Weaknesses:
- At the end of the day, no matter how beautiful the math, proposing a new activation function has a significant bar to pass empirically: it has to be demonstrated to work significantly better empirically than existing activation functions at least in some general setting across a range of datasets/models. To the author's credit, they have evaluated their proposed method over a variety of settings including image classification, diffusion models, and the MLP layer of NLP transformers. However, overall I did not find convincing evidence of a substantial improvement in any of the results. furthermore, the only compared "component-wise" activation function is ReLU, and there are obviously a lot more of them out there (as discussed in background)
I would encourage the authors to focus on a setting where their proposed method shows the biggest advantage and then show as many results across variants of models/datasets in that setting in comparison to a variety of existing activation functions.
   - Toy MLP/MNIST result. 95% on MNIST is an extremely poor baseline already even for an MLP, but to be fair the authors do call this a toy setting.
   - ResNet-56/CIFAR10: ResNet-56 (ReLU) test accuracy on CIFAR-10 in the original paper is 93.03%, so again I'm not sure why this baseline is so low. If it wasn't for that, I think this would be the strongest result in the paper (although not sufficient alone), and I would encourage the authors to improve upon the baseline and perhaps show results on ResNet with other datasets and component-wise activations for more convincing results.
  - UNet training loss differs by 0.0013, I don't believe this is significant although we don't have variance to judge with.
  - GPT2 MLP result shows a slightly higher training loss and virtually equivalent evaluation loss (differs by 0.001), I don't believe this is significant although we don't have variance to judge with.
- Figure 5 does not explain which "component-wise" activation function is being plotted.
- Give the empirical results, it makes myself at least wonder after the fact, indeed if NN optimization is approx. convex module a group, then doesn't training from any specific random init mean it's already approximately convex. Perhaps having the explicitly connected-local minima of the proposed conic activation function doesn't help if you're virtually guaranteed to be within a good one already? Perhaps outside of convincing empirical results, some linear-mode connectivity or other loss-based analysis would be more convincing that the resulting loss is in fact easier to optimize on?
- Notation $x_{-1}$ looks more like a typo at a glance than normal mathematical notation, would suggest changing this.
- Minor: Tables/figures should in general be at top/bottom of page rather than interspersed in text.
title: title: Official Review of Submission2 by Reviewer qTG8
main_review: main_review: **Summary:**

The paper introduces Conic Activation Functions (CoLU) to enhance symmetry in neural networks beyond standard permutations. Inspired by Lorentz cones, CoLU demonstrates superior performance compared to ReLU and SiLU across various architectures like ResNets, UNets, and Transformers, in both image and text tasks. The authors explore variations such as soft scaling, multi-head structures, and axis sharing, which further enhance CoLU’s efficacy in reducing loss and speeding up training.

**Strengths and Weaknesses:**


**Strengths:**  

- The paper proves that CoLU induces orthogonal symmetries in the latent space, potentially improving generalization. This aligns with findings from (Zhao et al., 2023b), where such symmetries are linked to flatter minima, which is often associated with better model robustness.

- CoLU can be applied selectively to subsets of neurons rather than globally, making it particularly useful in architectures like Regular Group Convolutions, where preserving equivariance by restricting cross-channel mixing is crucial.

- CoLU's superior performance compared to ReLU and SiLU across diverse tasks is well-demonstrated. Additionally, the connection between CoLU and physics-based concepts enhances the potential for interpretability, which could be an exciting avenue for future research.

**Weaknesses:**  

- While the concept of CoLU is relatively straightforward, its current presentation is overly complex, which could hinder accessibility for a broader audience. Simplifying the notation and adhering to established conventions from weight symmetry literature would enhance readability. Additionally, providing geometric visualizations (e.g., a 3D illustration of the conic projection) would make the concept more intuitive.

- Section B of the Appendix, though interesting, appears tangential to the core contribution of the paper and might dilute the focus. It may be worth revising or relocating this content to ensure relevance.

- Other activation functions, such as the radial scaling activation (Ganev et al., 2022), also provide orthogonal symmetries in latent spaces. Since the symmetry groups are similar when $G=1$, these functions should be acknowledged and compared in the experimental section for completeness.

- Corollary 4.6 addresses the monomial matrix group as the intertwiner group (notation from Godfrey, Charles, et al., 2022) for certain point-wise activation functions. However, if the activation is homogeneous, the diagonal matrix should be constrained to positive values. Similarly, for `tanh`, the matrix entries are restricted to $\pm 1$ (Tran, Hoang V., et al., 2024). A more precise statement in Corollary 4.6 is necessary to account for these nuances.

**Questions:**

- Could you elaborate on your claim that “CoLU considers finer symmetries than GCNN”? Specifically, what finer symmetries does CoLU capture, and how do they compare to those modeled by GCNN?

- Could you provide more detail regarding the experiment in Section 5.6, where you fine-tune a large diffusion model with component-wise activation functions towards its conic version? What insights or implications do you draw from this approach?

**Limitations:** Limitations are adequately addressed

**Ethical concerns:** No

**Soundness:** 2 fair

**Presentation:** 1 poor

**Contribution:** 3 good
title: title: Conic activation functions
main_review: main_review: The paper introduces Conic Activation Functions (CoLU), a new type of activation function for neural networks that generalizes the symmetry principles of component-wise activations like ReLU. The main motivation is to extend the symmetry in neural networks beyond permutations by leveraging continuous orthogonal groups, thus providing an alternative symmetry structure based on the Lorentz cone. CoLU maintains a similar computational cost as traditional activations but shows superior performance across some toy model architectures. Experimental results seem to show improved loss and training speed, particularly when low-dimensional cones are used, along with configurations like multi-head structures and axis sharing.

Strengths:
- The proposed methodology is quite interesting and new.
- The proposed approach can be effectively used on different architectures.
- The toy tests seem to show an improvement for models performances.
-  It's very interesting how the paper proposes geometric and algebraic symmetry for network design.

Weaknesses:
- While the paper claims that the computational complexity remains the same as ReLU, it doesn't provide an analysis of the overhead in large-scale networks, especially when using more complex configurations like multi-head structures.
- The paper seems quite strong on the theoretical side, but it might lack some practical consideration aside from the model performances.
title: title: Review
main_review: main_review: The paper presents a novel activation function called Conic Activation Functions (CoLU), which generalizes neural networks' symmetry from permutations to continuous orthogonal groups. This new activation is inspired by geometric principles, specifically the Lorentz cone, and is proposed as an alternative to traditional component-wise activation functions like ReLU. The core idea is that CoLU introduces new forms of symmetry into neural networks, which can improve performance in various tasks. 
The authors demonstrate that CoLU outperforms ReLU across multiple architectures such as MLPs, ResNet, Transformers, and UNet, particularly in image/text classification and generation. They further explore enhancements like multi-head structure, soft scaling, and axis sharing to improve performance. 

Strengths:
1. The method is novel. The introduction of a non-component-wise activation function (CoLU) that generalizes neural networks symmetry to continuous orthogonal groups is a fresh approach.

2. Good performance on various architectures is offered by CoLU shows significant improvements over ReLU in various deep learning models (MLP, ResNet, Diffusion Models).

3. The idea seems mathematically sound to me, and makes sense.

Weaknesses:
1. The paper can be better organised and written. Its overall flow can be improved. It is also not clear what is the purpose of the first paragraph in the Introduction section.

2. The authors should discuss the computational cost aspect of this method.

3. While the authors do consider many experiments, the method can be better benchmarked and compared with more architectures and recent activation functions, especially learned activation functions. (see question below)

Question to authors:
I think that the idea of multi-component activation function makes sense. Can you discuss or compare with the recent approach in [1] that offers a component-wise activation that is dependent on all components? would such an approach benefit Conic Activation Functions?

[1] DiGRAF: Diffeomorphic Graph-Adaptive Activation Function