Title: Rethinking 3D Convolution in $\ell_p$-norm Space
Venue: NeurIPS 2024
Accepted: yes
Keywords: $\ell_p$-norm; 3D Convolution
Abstract: Convolution is a fundamental operation in the 3D backbone. However, under certain conditions, the feature extraction ability of traditional convolution methods may be weakened. In this paper, we introduce a new convolution method based on  $\ell_p$-norm. 
For theoretical support, we prove the universal approximation theorem for $\ell_p$-norm based convolution, and analyze the robustness and feasibility of  $\ell_p$-norms in 3D point cloud tasks. Concretely, $\ell_{\infty}$-norm based convolution is prone to feature loss. $\ell_2$-norm based convolution is essentially a linear transformation of the traditional convolution.  $\ell_1$-norm based convolution is an economical and effective feature extractor. We propose customized optimization strategies to accelerate the training process of $\ell_1$-norm based Nets and enhance the performance. Besides, a theoretical guarantee is given for the convergence by \textit{regret} argument. We apply our methods to classic networks and conduct related experiments. Experimental results indicate that our approach exhibits competitive performance with traditional CNNs, with lower energy consumption and instruction latency.

Rethinking 3D Convolution in ℓp -norm Space

Abstract

Convolution is a fundamental operation in the 3D backbone. However, under certain conditions, the feature extraction ability of traditional convolution methods may be weakened. In this paper, we introduce a new convolution method based on ℓ p -norm. For theoretical support, we prove the universal approximation theorem for ℓ p -norm based convolution, and analyze the robustness and feasibility of ℓ p -norms in 3D point cloud tasks. Concretely, ℓ ∞ -norm based convolution is prone to feature loss. ℓ 2 -norm based convolution is essentially a linear transformation of the traditional convolution. ℓ 1 -norm based convolution is an economical and effective feature extractor. We propose customized optimization strategies to accelerate the training process of ℓ 1 -norm based Nets and enhance the performance. Besides, a theoretical guarantee is given for the convergence by regret argument. We apply our methods to classic networks and conduct related experiments. Experimental results indicate that our approach exhibits competitive performance with traditional CNNs, with lower energy consumption and instruction latency.

1 Introduction

The convolution-based 3D backbone networks have demonstrated substantial success in foundational tasks such as classification [1], object tracking [2], scene segmentation [3], etc. Some downstream tasks also heavily rely on these networks, such as interactive perception [4], object manipulation [5], imitation learning [6], and human-machine collaboration [7]. In the traditional 3D convolution, suppose K ∈ R m × n is the filter, and P t ∈ R m × n is the sampled matrix from the t -th sliding window on input data, 1 ≤ t ≤ T . T is the total sliding counts. For any t ≥ 1 , the t -th convolution is calculated as:

which is the same as inner product between vectors. To distinguish it from our new convolution framework, we refer to it as inner product based convolution in the following discussion. A geometric consideration arises when P t follows a certain symmetric distribution, such as a Gaussian or uniform distribution. By symmetry, there exist some of { P t } T t =1 situated close to the subspace perpendicular to K , which means K ⊙ P t ≈ 0 . This inevitably leads to explicit feature loss, diminishing the model's ability on information extraction.

In previous works, ℓ p -norms ( p = 1 , 2 , 3 , · · · , ∞ ) demonstrated strong performance across various domains [8, 9, 10]. These norms exhibit remarkable capabilities in expressing spatial structures and local relationships within sets of points. To address the limitations of inner product-based convolution in certain extreme cases and to explore the potential of ℓ p -norms in feature extraction, we propose ℓ p -norm-based convolution, i.e. , for any kernel K and sampled matrix P t , it can be formulated as Eq. 2:

More precisely, the goal of this paper is to leverage the power of ℓ p -norm measurement (Fig. 1 (a)) and devise efficient and robust optimization methods for it. Our solutions are as follows:

From the theoretical standpoint, we prove the universal approximation theorem of ℓ p -norm Nets (for p = 1 , 2 , 3 , · · · , ∞ ). Besides, we show that ℓ p -norm based convolutions are more robust than the traditional ones via variance analysis under random noise.

From the practical standpoint, we first discuss the performance of different ℓ p -norms in actual execution. 3D convolution in ℓ ∞ -norm space tends to lose multiple useful pieces of information since only the maximum absolute value is reserved. The ℓ 2 -norm measure is inherently a linear transformation of the traditional convolution (details can be found in Sec. A). In contrast, the ℓ 1 -norm has

unique potential for 3D point cloud tasks. However, directly replacing traditional convolution with an ℓ 1 -norm-based one is not feasible in practice due to the difficult convergence and local optima. To enhance network performance, we propose customized optimization strategies. The first strategy is a mixed gradient strategy (MGS), and the second is a dynamic learning rate controller (DLC). These strategies are applied in the training process (Algorithm 1) to accelerate network convergence and avoid local optima. We also provide a convergence guarantee for our optimization strategies from the perspective of regret .

We evaluate our method on several benchmarks, ranging from global, semi-dense, and dense prediction tasks. The experimental results show that ℓ 1 -norm Net has the same competitive performance as traditional convolution. Moreover, the proposed ℓ 1 -norm Net has three advantages: 1) ℓ 1 -norm (inherently addition operation) has lower computational complexity compared to multiplication; 2) addition significantly reduces energy consumption [11]; 3) ℓ 1 -norm operations (addition) has lower instruction latencies [12] than inner product process (multiplication). These properties facilitate the 3D point cloud tasks especially online tasks such as 3D real-time object detection, pose tracking, etc.

Contributions. 1) We prove the universal approximation for ℓ p -norm Nets. And we show that ℓ p -norm Nets are robust under random noise. 2) We compare different ℓ p -norm based convolutions, and further propose a reliable and efficient ℓ 1 -norm Net for 3D point cloud tasks with customized optimization strategies. We also give a theoretical guarantee for convergence by regret argument. 3) Experimental results demonstrate the effectiveness of our methods in 3D point cloud tasks, showing lower energy consumption and faster instruction execution.

2 Related Work

Different Convolution Methods. Convolutions have seen significant success, leading to various convolution methods aimed at improving performance and efficiency. Traditional convolutions, introduced by [13], use fixed-size kernels to extract features but are computationally intensive and may not capture diverse patterns effectively. To overcome these limitations, several alternatives have been proposed: 1) depthwise separable convolutions [14, 15]. Popularized by MobileNets, these decompose standard convolutions into depthwise and pointwise operations. 2) dilated convolutions [16, 17, 18]. These introduce spaces between kernel elements, expanding the receptive field without increasing parameters. 3) deformable convolutions [19, 20]. These adapt the sampling loca-

tions of the convolutional kernel, enhancing the network's ability to model geometric transformations. However, due to their unique strengths, they only excel at some specific tasks.

ℓ p -norm Measure in Different Tasks. Using the ℓ p -norm as a feature measurement function for convolutional kernels offers several advantages: 1) Flexibility: The ℓ p -norm allows adjusting the parameter p according to specific needs [21, 22, 23]. 2) Sparsity: It encourages most elements in the convolutional kernel to approach zero, reducing computational complexity and storage requirements [21, 24]. Overall, in diverse settings, employ distinctive approaches. The ℓ p -norm is widely used across various fields. For example, in image processing , the ℓ 1 -norm is used for sparse representation in image compression [25], enabling efficient storage and transmission. In machine learning and optimization , optimization problems also use ℓ p -norm constraints to impose sparsity or specific patterns in solutions [26, 27]. Despite progress, directly migrating these methods into 3D point cloud tasks causes a domain gap . In this work, we aim to explore ℓ p -norm measure for 3D point cloud tasks in depth.

3 Methodology

Notations. For the sake of simplicity, in what follows, we take the classic PointNet++ [28] as the basis model to estimate the efficiency of ℓ p -norm based Nets with the proposed optimization strategies. Note that, we directly replace the inner product based convolution by ℓ p -norms ( p = 1 , 2 , 3 , . . . , ∞ ) based one, and denote the corresponding network by ℓ p -PointNet++ or ℓ p -norm Net. Moreover, the proposed ℓ p -norm based convolution can also be called ℓ p -norm neuron.

3.1 Universal Approximation

The universal approximation ability of a neural network is crucial. Firstly, it establishes a solid theoretical foundation for the network's capabilities [29], which asserts that certain architectures and activation functions enable neural networks to approximate any continuous function. There is a series of works on the approximation capacity, such as theories for feedforward networks [30], RNNs [31], Transformer [32]. However, the universal approximation property of ℓ p -PointNet++ has not been studied thoroughly up to now.

Theorem 1. Assume S = { x 1 , · · · , x N } ⊂ R k is an arbitrary point cloud. J ⊂ R k is any compact set and S ⊂ J . For any continuous function f defined on 2 J with respect to Hausdroff distance d H ( · , · ) , there exists an ℓ p -PointNet++ P satisfying for any ϵ > 0 ,

Moreover, for any ℓ 1 -integrable function g defined on J , there exists an ℓ p -PointNet++ P ′ , for any ϵ ′ > 0 ,

Briefly speaking, f could be approximated by an MLP consisting of ℓ p -norm convolution layers and a max pooling layer. And g could be approximated by a network composed of an ℓ p -norm convolution layer and a fully connected layer. The detailed proof can be found in Sec. A from the appendix.

3.2 Robustness Analysis

In the following, we show that under Gaussian random noise on input data, ℓ p -norm based convolutions are more robust than that based on inner product. Suppose G ∈ R m × n is a Gaussian matrix. Each G ( i, j ) ∼ N (0 , σ 2 ) where σ > 0 is a constant. Let P t ∈ R m × n be the data at time t and K ∈ R m × n be the kernel function.

For inner product,

and

Suppose ∀ i ∈ [ m ] and ∀ j ∈ [ n ] , K ( i, j ) is a constant, we have V ar [ ( G + P t ) ⊙ K ] = Θ( mn ) .

For ℓ p -norm, first we could prove that when p = 2 , V ar [ ∥ G + X -K ∥ 2 ] = O (1) , which is significantly smaller than V ar [ ( G + P t ) ⊙

K ] . The details of calculation could be found in Sec. A from the appendix. For the more

general cases ( p = 1 , 2 , 3 , · · · , ∞ ), we show that ℓ p -norm has a small variance through numerical computation in the Tab. 1, where we take σ = 1 .

3.3 Implementation of ℓ p -norm Nets

Note that although Theorem 1 guarantees a universal approximation capability, it does not mean that all the ℓ p -norm Nets are efficient and feasible in practice. Therefore, we further discuss the characteristics of each specific ℓ p -norm Nets ( p = 1 , 2 , 3 , · · · , ∞ ) in detail.

Assume the input data follows Gaussian distribution, saying G is the standard Gaussian matrix. For ℓ p -norm based convolution, when p is greater than or equal to 3, the distribution of the output data is very close. We present the simulation results in Fig. 2. It's clear that when p is getting larger, the distribution of ∥ G ∥ p gradually overlaps with the distribution of ∥ G ∥ ∞ . There-

Distribution of Lp for different values of p

Distribution of Lp for different values of p

80%

p=1

80%

=d

70%

p=2

p=3

70%

p=5

p=4

Percentage

50%

60%

p=inf

ercentage

60%

50%

p=7

p=6

p=8

40%

40%

6=d

30%

P

30%

p=inf

20%

20%

10%

10%

0%

0%

0.0

2.5

5.0

7.5 10.012.515.017.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Lp value

Lp value

fore, we take p = ∞ as the representative case for p ≥ 3 .

Actually, l ∞ -norm exhibits weaknesses due to its overly simplistic emphasis on the largest element. Namely this approach tends to oversimplify the feature space by disproportionately emphasizing only one dimension, potentially discarding valuable information present in other dimensions. Also, this concept is supported by experimental results in Sec. 5. Besides, ℓ 2 -norm inherently is calculated by taking the square root of the sum of the squares of its elements. And ℓ 2 -norm based convolution C ℓ 2 can be regarded as an equivalence transformation of the traditional convolution C . Briefly speaking, we could show that C 2 ℓ 2 = α + β ×C , where α and β are constants.

ℓ 1 -norm can synthesize each element of the feature vector. And the ℓ 1 -norm Net is not equivalent to a translation transform, which we believe holds potential as a 3D convolutional similarity metric function according to the Theorem. 1. To this end, our method focuses on rationalizing the ℓ 1 -norm measure to maximize its potential in feature extraction. Mathematically, if the similarity measurement function between the input data and kernel function is replaced with the ℓ 1 -norm, the convolution can be re-formulated as:

The underlying operation of ℓ 1 -norm kernel function is addition, which has more development potential and application value in real scenarios. Specifically, 1) It contains almost no multiplication but addition, resulting in lower computational complexity of the model. 2) ℓ 1 -norm operation (addition) is proved to have lower energy consumption compared to the inner product (multiplication) calculation [33]. Take the operation of floating-point addition and multiplication as an example, which has energy costs of 0.9 pJ and 3.7 pJ , respectively. 3) Low latency is also a consideration in practical application scenarios. [12] tells us that multiplication (inner product process) has longer theoretical instruction wait times than addition operations. Table 1 of this study lists the instruction

latency, throughput, and micromanipulation faults for Intel, AMD, and VIA CPUs. For instance, the latency of float multiplication and addition is 4 and 2 in the VIA Nano 2000 series.

3.4 Regret

It's a good way [34, 35, 36] to demonstrate the convergence of an optimization process by analyzing the regret . Performance measurement [37], optimization guidance [38], and feedback mechanisms [39] can be summarized as its advantages. We employ it the construct the convergence theorem for our optimization strategies in Sec. 4.

Consider a general online optimization model between a player and an adversary. A subset F ∈ R m is non-empty, bounded and closed. For each iteration k ∈ [ T ∗ ] , the player choose a point x k ∈ F ( T ∗ is not known for player). After committing to this choice, a convex function h k will be revealed by the adversary. And we note the cost of this game by regret :

The player aims to carefully select x k to minimize regret as much as possible, while conversely the adversary aims to specifically choose h k to hinder the player. Intuitively, if an algorithm(the player) could bound regret by a sub-linear function of T ∗ , i.e. , R T ∗ = o ( T ∗ ) , we could conclude that 'on the average' the algorithm performs as well as the best fixed strategy in hindsight [40].

4 Optimization

By the argument above, we are motivated to devise a new convolution based on ℓ 1 -norm. However, direct training of ℓ 1 -norm Nets can easily lead to unsatisfactory results. Thus, two customized optimization strategies are proposed for training. Before introducing these optimization strategies, we clarify the notations in the following.

Notations Recall that K ∈ R m × n is the kernel and P t ∈ R m × n is the sliding window on the input data, 1 ≤ t ≤ T . Y ( P t , K ) is the convolution of K and P t . L denotes the loss function in training process. We use the m × n matrix ∂L ∂K to denote the gradient on of L on K , where ( ∂L ∂K ) i,j = ∂L ∂K ( i,j ) . Besides, define the vectors

and

4.1 MGS: Mixed Gradient Strategy

Now we focus on the gradient descent in training process, especially the partial derivative of loss function L on the kernel K . It should be pointed out that L is a function on ( Y ( P 1 , K ) , Y ( P 2 , K ) , . . . , Y ( P T , K )) . By chain rule of derivation we have for any given ( i, j ) ,

Notice that when loss function L is fixed, ∂L ∂Y is regardless of the choice of Y ( P t , K ) (inner product or ℓ p -norm). And we should only focus on the vector ∂Y ∂K ( i,j ) . In the context of ℓ 1 -PointNet++:

Here, sgn ( · ) represents the sign function.

There are two unavoidable problems: 1) the use of Eq. 10 results in a signSGD update. As discussed in [41], the direction of signSGD is not aligned with the steepest descent, and this misalignment exacerbates with increasing dimensionality. 2) The gradient of ℓ 1 -norm Net is significantly smaller than that of inner product convolution in the experiment. Namely, ∥ ∂L ∂K ∥ 2 is extremely small when we choose the convolution Y as ℓ 1 -norm. Taking PointNet++ on S3DIS as an example, we report the ℓ 2 norm of gradient of ℓ 1 -PointNet++ in Fig. 3. The gradient from ℓ 1 -PointNet++ is much smaller than that in PointNet++ ( e.g. , ℓ 1 -PointNet++: 0.0002, PointNet++: 0.3162 in layer I). Hence, this small gradient ∂L ∂K in ℓ 1 -norm Net would significantly slow down the training process.

Based on the above observations, we introduce a novel M ixed G radient S trategy ( MGS ) tailored for ℓ 1 -PointNet++ training. This approach strategically combines the gradients of the ℓ 1 -PointNet++ and that of ℓ 2 -PointNet++:

Actually, as we discussed above, ℓ 2 -norm based convolution is a linear transform of inner product convolution. So gradient of ℓ 2 -norm Net has a proper scale. The mixed strategy involves dynamically adjusting ∂Y ( P t ,K ) ∂K ( i,j ) during training, guided by a parameter 0 < λ < 1 and the training step k . The mixed gradient strategy is expressed as:

100

scale)

10-1

Gradient (log

10-2

10-3

f-PointNet++

PointNet++

IV

V

Layer

This dynamic adjustment introduces a controlled transition in the gradient computation as training progresses. Taking λ = 0 . 99 for example, when k is small, the term λ k dominates and ∂Y ( P t ,K ) ∂K ( i,j ) approximates to P t ( i, j ) -K ( i, j ) . This initial configuration aligns with the more efficient ℓ 2 -like update, providing stability and aiding in faster convergence. As training progresses ( k gets larger), the term λ k becomes more prominent, shifting the gradient computation towards sgn ( P t ( i, j ) -K ( i, j )) . This transition allows the model to leverage the advantages of the ℓ 1 -PointNet++ structure, facilitating sparsity in the learned features. By dynamically adapting the gradient computation based on the training step, the mixed strategy offers a flexible and adaptive approach to overcome the challenges associated with fixed gradient schemes. This dynamic adjustment provides a thoughtful compromise, combining the efficiency of ℓ 2 -like updates in the initial stages with the sparsity-inducing benefits of ℓ 1 -PointNet++ in later stages.

In fact, there is quite a bit of literature supporting the effectiveness of the signSGD update scheme, and in particular, it has been shown that it has some advantages in avoiding saddle points [42]. However, when certain random rotations of the objective appear, signSGD may become trapped in a periodic behavior that hinders convergence in such cases. To address this unexpected behavior, we additionally explored the introduction of momentum into the update rule. Our experimental results prove that this modification effectively breaks the symmetry induced by random rotations, preventing the model from getting stuck and fostering smoother convergence.

4.2 DLC: Dynamic Learning rate Controller

Considering the uniqueness of the mixed gradient strategy, we focus on achieving larger update magnitudes and faster convergence rates during the initial stages of training. However, in the later stages, we aim to revert to signSGD, implementing a more cautious update strategy to enhance the model's precision. Therefore, we propose a learning rate update strategy that adapts to this characteristic: D ynamic L earning rate C ontroller ( DLC ), maintaining a higher rate in the early training phase, and returning to a lower rate in the later phase.

To this end, we design two bound functions to control the learning rate: the lower bound

and the upper bound where p 1 , p 2 and p 3 are hyper-parameters to be determined and k denotes the training step. And we use simple comparison operations to make learning rate α ( k ) locate in [ α 1 ( k ) , α 2 ( k )] :

To enhance the universality of this dynamic control framework, A could be another learning rate optimization algorithm like the adaptive learning rate strategy of [43], which can be specifically switched according to the task at hand. However, regardless of A , we will later demonstrate that dynamic control alone is sufficient to provide theoretical convergence guarantees by the regret argument of Theorem 2, and it also performs well in experiments.

4.3 Training Framework

It has been noted from previous discussions that the momentum method can help signSGD avoid getting trapped in cycles, thereby improving training stability. Combining the methods above, we present the global optimization algorithm ( O ptimizer with M ixed gradient strategy and D ynamic learning rate controller, OMD ) for ℓ 1 -PointNet++ training. Details are shown in Algorithm. 1

Here we give a convergence guarantee for OMD under an online optimization framework, which is harder than offline optimization. We could show that regret R T ∗ of OMD is bounded by O ( √ T ∗ ) . Low regret means the algorithm

Algorithm 1 OMD

Input: Initial learning rate α , hyper-parameters p 1 , p 2 , p 3 , referred by Eq. 13 and Eq. 14. q 0 and q in (0 , 1) .

1: m 0 = 0 , α (0) = α , x 1 = ⃗ 0 .

2: Set the functions α 1 ( k ) and α 2 ( k ) by hyper-parameters p 1 , p 2 , p 3 .

3: for k = 1 to T ∗ do

4: g k ← ∂L ∂K ( x k ) # Consider the gradient ∂L ∂K as an vector here. ∂L ∂K ( i,j ) = ⟨ ∂L ∂Y , ∂Y ∂K ( i,j ) ⟩ . ∂L ∂Y only depends on the choice of loss function. See Eq. 12 for ∂Y ∂K ( i,j ) .

5: q k = q 0 · q k .

6: m k = q k · m k -1 +(1 -q k ) · g k

7: ˆ α ( k ) ← min { max { α 1 ( k ) , A [ α ( k )] } , α 2 ( k ) } √

8: α ( k ) ← ˆ α ( k ) / k

9: x k +1 = Π F ,α ( k ) -1 / 2 ( x k -α ( k ) · m k ) 10: end for

progressively gets closer to the optimal solution over time. This shows that OMD has reliable convergence properties, making it a dependable optimization method.

Theorem 2. Continue with the settings and notations of Algorithm 1. Suppose F ⊂ R n is bounded, saying max x , y ∈F ∥ x -y ∥ ∞ ≤ B ∞ Besides, suppose ∀ k ∈ [ T ∗ ] , ∥ g k ∥ 2 ≤ B 2 . we could show that for any convex functions { h k } T ∗ t =1 ,

where C 1 and C 2 are constants that rely on p 1 , p 2 , p 3 , B ∞ , B 2 , q 0 and q . And x ⋆ ≜ arg min x ∈F ∑ T ∗ k =1 h k ( x ) .

The proof could be found in the appendix, Sec. A.

5 Experiments

To validate the generalizability and robustness of the method and thus ensure its effectiveness and broad applicability, we verify the performance of our method in several tasks, ranging from Global Tasks ( i.e. , Parts Segmentation), Semi-dense Prediction ( i.e. , scenario semantic segmentation), and Dense Prediction ( i.e. , pose estimation) tasks. Shapenet, S3DIS, and GarmentNets Simulation are used as the datasets.

5.1 Dataset and Experimental Settings

Dataset. 1) ShapeNet. In ShapeNet, there are 16,881 shapes from 16 categories, which are annotated with 50 parts in total. Note that most object categories are labeled with two to five parts and Ground Truth annotations are labeled on sampled points on the shapes. This task can be regarded as a point-wise classification task. 2) S3DIS. The Stanford Large-Scale 3D Indoor Spaces Dataset, which encompasses 3D scans obtained from Matterport scanners across 6 distinct areas, comprising a total of 271 rooms. Within the S3DIS dataset, every point within the scans is labeled with a semantic category from a set of 13 distinct classes. These classes encompass various elements such as chairs, tables, floors, walls, among others, in addition to a category for clutter. 3) GarmentNets Simulation. GarmentNets Simulation is a large-scale dataset proposed by [44]. This dataset has six garment categories with a total data volume of 1.72TB. Dress, Jump, Skirt, Top, Pants and Shirt are included.

Experimental Settings. We train our frameworks using CrossEntropy loss and the AdamW optimizer [45], with an initial learning rate of 0.001, a weight decay of 10 -4 , Cosine Decay, and a batch size of 32. The total training consists of 200 epochs. All tasks use the same settings unless otherwise specified. All experiments are conducted on a computer workstation with three GeForce GTX 3090 GPUs using the PyTorch deep learning framework. The best model on the validation set is selected for testing.

5.2 Experiments on Global Task

Parts Segmentation. As a classic global task, 3D object parts segmentation is an important predecessor for articulated objects from the embodied intelligence community, such as pose estimation [46, 47], manipulation [48, 49], etc. In this section, we conduct experiments on ShapeNet part dataset [50].

Table 2: Quantitative segmentation results on ShapeNet part dataset . Note that a 3D fully convolutional network is proposed as the 3DCNN, mIoU(%) is reported as the metric on points.

From Tab. 2, see them all: we find that our method has almost equivalent performance to the conventional method when being equipped with PointNet, and achieves superior performance on 3DCNN and PointNet++. Treat them equally: we see that our method can often perform better in some categories ( e.g. , car, motor, rocket, etc.), these categories usually have a larger volume ( i.e. , a more sparsified point cloud) compared to other objects. We propose that the inner product within convolutional networks has a tendency to highlight local context among points, yet it is greatly affected by the overall translation and scaling of the dataset. Our method focuses on the points drawn from ℓ 1 -norm space and addresses this problem by integrating the inherent distance measure into our architecture. Specifically speaking, The Manhattan distance based ℓ 1 -norm Nets tend to avoid this problem, which notices point cloud features at a longer distance.

5.3 Experiments on Semi-dense Prediction Task

Scenario Semantic Segmentation. As a semi-dense prediction task, this task aims to segment distinct regions within a 3D scene based on their semantic meaning using point cloud data. Semantic scene segmentation is crucial for understanding and interpreting the spatial arrangement and relationships between objects in 3D scenes. For our study, we utilize the S3DIS dataset. The metrics and experimental settings follow those outlined in [28].

Following the training and test strategies used in [51], we first divide the point cloud using the room as the basic unit and then sample the room at a size of 1m * 1m (randomly sampling up to 4096

Model

Mean

IoU (%)

Overall

Accuracy (%)

Energy

(

µj

)

PointNet

47.7

78.6

7.981

ℓ

1

-PointNet

47.6

77.9

3.471

PointNet++

53.5

83.0

3.395

ℓ

1

-PointNet++

53.9

82.9

1.328

points during training, and all points are involved in the computation during testing), which in turn predicts the class of each point in each block. Note that we use a 9-dimensional vector to represent each point, representing XYZ, RGB, and normalized room location (ranging from 0 to 1). K-fold strategy is also used for training and testing.

The quantitative results are reported in Tab. 3. Experimental results show that although our approach achieves almost equivalent performance to inner product based networks, we maximize the potential of ℓ 1 -norm measure by relying on our proposed optimization strategy, which allows us to achieve similar performance but with less computational complexity and lower energy consumption (Almost 61% energy reductions). Also, we provide qualitative segmentation results for visualization in Fig. 4. Overall, our model generates consistent object predictions and is resilient to the presence of absent points and obstructions.

5.4 Experiments of Dense Prediction Task

Garment Pose Estimation Garments, vital in daily life, present unique challenges for machine perception and interaction due to properties like infinite degrees of freedom and thin structure. Garment pose estimation and tracking systems hold potential for applications in mixed reality [52, 53], augmented real-

ity [54, 52], and robotic manipulation [55, 49]. Addressing these challenges, mainstream methods typically employ Normalized Object Coordinate Space (NOCS) [56] for dense prediction tasks . In this section, we introduce GarmentNets [44], a baseline focusing on garment pose estimation using partial point clouds as input and generating complete point clouds as output. Our approach utilizes the GarmentNets Simulation Dataset to evaluate this task. The total epoch number is 200, and the batch size is 16.

Quantitative results are in Tab. 4. Note that we use Symmetric Chamfer Distance as the metric, This metric measures accuracy and completeness for surface reconstruction. The accuracy metric is defined as the mean L2 distance of points on the output mesh to their nearest neighbors on the GT mesh. From the table, it can be seen that our method performs comparably to the original method.

5.5 Ablation Experiments

Replacing Means. The most critical structure of PointNet++ is the 3 separate SetAbstractions modules (SA). Hence, to explore the effect of using the ℓ 1 -PointNet++ at different places and in different ratios, we remove the modules at different ratios and places on S3DIS. The experimental result is shown in Tab. 5. In many aspects, we can infer that the average mean IOU and accuracy are higher under the 66.7% ratio than those reported under the 33.3% ratio. This result tells the conclusion that our ℓ 1 -norm measure can exact more useful features from sparse point clouds. we hope these results can prompt further study on replacing means, such as different replacing ratios in

each inner module, creditable ways to combine hybrid convolutional blocks. etc. We leave this for more passionate researchers in the future.

Table 6: Ablation Results on S3DIS Dataset Using Different Variants of ℓ 1 -Nets. Mean IOU and overall Accuracy (%) are reported. Note that the results of ℓ 1 -PointNet are reported from I to IV, and ℓ 1 -PointNet++ are reported from V to VIII. Besides, vanilla Net represents the model without our customized optimization strategy while training.

Optimization Strategy. As demonstrated in Sec. 4, we propose mixed gradient strategy (MGS) to accelerate network convergence, while dynamic learning rate controller (DLC) helps our network move away from local optima. To evaluate the effectiveness of MGS and DLC, we remove them separately from ℓ 1 -PointNet++ and evaluate the scenario semantic segmentation performance on S3DIS. Tab. 6 presents the quantitative results. The baselines (I and V) indicate that we only use ℓ 1 -norm as the similarity measurement but without any optimization. It can be observed that they both resulted in huge performance degradation. Besides, we can see that both our MGS and DLC contribute to network convergence and optimization results.

6 Limitations and Broader Impact

Firstly, some of the other convolutions ( e.g. , sparse convolution, group convolution, dilated convolution) and additional computer vision tasks remain unexplored. Secondly, the inference speed of the ℓ 1 -norm Net is marginally slower than that of traditional one. This is attributed to the lack of CUDA and cuDNN optimized operations for Manhattan distance metrics. It's noteworthy that, beyond introducing a novel convolution based on the ℓ p -norm and proving the universal approximation theorem for theoretical support, this paper also presents customized optimization strategies.

7 Conclusion

In this paper, we are motivated to explore ℓ p -norm measure to replace the classic inner product convolution. we first prove the universal approximation of ℓ p -norm Nets. And then we compare different ℓ p -norm measures and propose the ℓ 1 -norm Net for 3D point cloud tasks. Furthermore, we design the customized optimization strategies ( i.e. , mixed gradient strategy and dynamic control on learning rate) for ℓ 1 -norm Net. When introducing our method to classical 3D networks, they achieve competitive performances at a lower energy cost. In summary, our ℓ 1 -norm Net can achieve similar performance to traditional convolution network, but with less computational cost and lower instruction latency.

References

Min Zhang, Yifan Wang, Pranav Kadam, Shan Liu, and C-C Jay Kuo. Pointhop++: A lightweight learning model on point sets for 3d classification. In 2020 IEEE International Conference on Image Processing (ICIP) , pages 3319-3323. IEEE, 2020.

Haozhe Qi, Chen Feng, Zhiguo Cao, Feng Zhao, and Yang Xiao. P2b: Point-to-box network for 3d object tracking in point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6329-6338, 2020.

Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 452-468, 2018.

Qiaojun Yu, Junbo Wang, Wenhai Liu, Ce Hao, Liu Liu, Lin Shao, Weiming Wang, and Cewu Lu. Gamma: Generalizable articulation modeling and manipulation for articulated objects. arXiv preprint arXiv:2309.16264 , 2023.

Haoyu Xiong, Haoyuan Fu, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, Animesh Garg, and Cewu Lu. Robotube: Learning household manipulation from human videos with simulated twin environments. In Conference on Robot Learning , pages 1-10. PMLR, 2023.

Fan Zhang and Yiannis Demiris. Learning garment manipulation policies toward robot-assisted dressing. Science robotics , 7(65):eabm6010, 2022.

Kailin Li, Lixin Yang, Haoyu Zhen, Zenan Lin, Xinyu Zhan, Licheng Zhong, Jian Xu, Kejian Wu, and Cewu Lu. Chord: Category-level hand-held object reconstruction via shape deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 94449454, 2023.

Marius Kloft, Ulf Brefeld, Sören Sonnenburg, and Alexander Zien. Lp-norm multiple kernel learning. The Journal of Machine Learning Research , 12:953-997, 2011.

Qingsong Gu and Po-Lam Yung. A new formula for the lp norm. Journal of Functional Analysis , 281(4):109075, 2021.

James A Cadzow. Minimum ℓ 1 , ℓ 2 , and ℓ ∞ norm approximate solutions to an overdetermined system of linear equations. Digital Signal Processing , 12(4):524-560, 2002.

William Dally. High-performance hardware for machine learning. Nips Tutorial , 2:3, 2015.

Instruction latencies of different operations. www.agner.org/optimize/ instruction_tables.pdf .

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1251-1258, 2017.

Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 , 2015.

Dren Gashi, Mike Pereira, and Valeriia Vterkovska. Multi-scale context aggregation by dilated convolutions machine learning-project. 2017.

Xin Wang, Rongrong Lv, Yang Zhao, Tangwen Yang, and Qiuqi Ruan. Multi-scale context aggregation network with attention-guided for crowd counting. In 2020 15th IEEE International Conference on Signal Processing (ICSP) , volume 1, pages 240-245. IEEE, 2020.

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 764-773, 2017.

Feng Chen, Fei Wu, Jing Xu, Guangwei Gao, Qi Ge, and Xiao-Yuan Jing. Adaptive deformable convolutional network. Neurocomputing , 453:853-864, 2021.

Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67(2):301-320, 2005.

Julianne Chung and Silvia Gazzola. Flexible krylov methods for \ ell _ p regularization. SIAM Journal on Scientific Computing , 41(5):S149-S171, 2019.

Martine Labbé, Justo Puerto, and Moisés Rodríguez-Madrena. Shortest paths and location problems in a continuous framework with different \ ell _ p -norms on different regions. arXiv preprint arXiv:2110.07866 , 2021.

Jinglai Shen and Seyedahmad Mousavi. Least sparsity of p-norm based optimization problems with p>1. SIAM Journal on Optimization , 28(3):2721-2751, 2018.

CS Sastry and Ashish Mishra. Application of l1-norm minimization technique to image retrieval. World Academy of Science, Engineering and Technology , 56(145):801-804, 2009.

Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350 , 2017.

Stephen Boyd and Venkataramanan Balakrishnan. A regularity result for the singular values of a transfer matrix and a quadratically convergent algorithm for computing its ℓ ∞ -norm. Systems &Control Letters , 15(1):1-7, 1990.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359-366, 1989.

Non-Polynomial Activation Functions Can. Multilayer feedforward networks with nonpolynomial activation functions can approximate any function-_-. 1991.

Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In Artificial Neural Networks-ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006. Proceedings, Part I 16 , pages 632-640. Springer, 2006.

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077 , 2019.

Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC) , pages 10-14. IEEE, 2014.

Noam Brown and Tuomas Sandholm. Regret transfer and parameter optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, 2014.

Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non-stationary environments. Advances in Neural Information Processing Systems , 33:6743-6754, 2020.

Zi Wang, Beomjoon Kim, and Leslie P Kaelbling. Regret bounds for meta bayesian optimization with an unknown gaussian process prior. Advances in Neural Information Processing Systems , 31, 2018.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Elad Hazan. 10 the convex optimization approach to regret minimization. Optimization for machine learning , page 287, 2012.

Jeremy Bernstein, Kamyar Azizzadenesheli, Yu-Xiang Wang, and Anima Anandkumar. Convergence rate of sign stochastic gradient descent for non-convex functions. 2018.

Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in Neural Information Processing Systems , 35:9955-9968, 2022.

Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1468-1477, 2020.

Cheng Chi and Shuran Song. Garmentnets: Category-level pose estimation for garments via canonical space shape completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3324-3333, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2019.

Liu Liu, Han Xue, Wenqiang Xu, Haoyuan Fu, and Cewu Lu. Toward real-world category-level articulation pose estimation. IEEE Transactions on Image Processing , 31:1072-1083, 2022.

Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song. Categorylevel articulated object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3706-3715, 2020.

Jianren Wang, Sudeep Dasari, Mohan Kumar Srirama, Shubham Tulsiani, and Abhinav Gupta. Manipulate by seeing: Creating manipulation controllers from pre-trained representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3859-3868, 2023.

Yezhou Yang, Yi Li, Cornelia Fermuller, and Yiannis Aloimonos. Robot learning manipulation action plans by" watching" unconstrained videos from the world wide web. In Proceedings of the AAAI conference on artificial intelligence , volume 29, 2015.

Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG) , 35(6):1-12, 2016.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Julie Carmigniani and Borko Furht. Augmented reality: an overview. Handbook of augmented reality , pages 3-46, 2011.

Dhiraj Amin and Sharvari Govilkar. Comparative study of augmented reality sdks. International Journal on Computational Science & Applications , 5(1):11-26, 2015.

Ronald T Azuma. A survey of augmented reality. Presence: teleoperators & virtual environments , 6(4):355-385, 1997.

Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Cloth funnels: Canonicalized-alignment for multi-purpose garment manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 5872-5879. IEEE, 2023.

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2642-2651, 2019.

Jooyoung Park and Irwin W Sandberg. Universal approximation using radial-basis-function networks. Neural computation , 3(2):246-257, 1991.

H Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908 , 2010.

and,

- Appendix -

In the Appendix, we present additional information on our methods. Concretely, we provides a detailed theoretical analysis of the theorems from the main paper, including the variance analysis, the proposed universal approximation, the regret argument, and the equivalence of ℓ 2 -norm Measure.

A Additional Theoretical Analysis

A.1 Omitted proof of variance analysis

Since adding a constant does not significantly affect variance, for ease of demonstration, we could assume V ar [ ∥ G + P t -K ∥ 2 ] ≈ V ar [ ∥ G ∥ 2 ] . Notice that

It's easy to verify that for any u ≥ 0 ,

Let u = ∥ G ∥ 2 2 m and calculate the expectation of G on both sides of the inequality, we have

̸

Because E [ ( ∥ G ∥ 2 2 m -1) 2 ] = 1 m 2 · E [ ∑ m i =1 ( G ( i ) 2 -1) 2 + ∑ i = j ( G ( i ) 2 -1)( G ( j ) 2 -1) ] and ∀ i , E [ G ( i ) 2 -1] = 0 , we can conclude that

where the first equation holds for all the G ( i ) s are i.i.d. The second equation holds for E [ G (1) 4 ] = 3 , E [ G (1) 2 ] = 1 . Combining inequality (17) and Equation (18b), E G ∼N (0 , I m ) [ ∥ G ∥ 2 ] ≥ √ m 2 · ( 2 -2 m ) . Therefore, by Equation (16) we have

Thus we have shown that V ar [ ∥ ( G + P t ) -K ∥ 2 ] = O (1) .

A.2 Proof of Theorem 1

Scaling S and J by 1 diam ( J ) where diam ( J ) = max x,y ∈ J {∥ x -y ∥ ∞ } , we could assume S = { x 1 , · · · , x N } and ∀ i ∈ [ N ] x i ∈ J ⊂ [0 , 1] k . For convenience, first we show the case k = 1 . Here we refer the construction of soft occupancy function in [51]. Because f is continuous function, for any ϵ > 0 , ∃ σ > 0 so that | f ( S 1 ) -f ( S 2 ) | < ϵ for any S 1 and S 2 with d H ( S 1 , S 2 ) < δ . Let M = ⌈ 1 δ ⌉ and h m ( x ) = exp ( -d H ( x, [ m -1 M , m M ])) be the soft occupancy function, for all m ∈ [ M ] . Next, for all m ∈ [ M ] define

v m ( S ) indicates the occupancy of the m -th interval by points in S . Define v : 2 J → { 0 , 1 } M and for any S ∈ 2 J , v ( S ) = ( v 1 ( S ) , v 2 ( S ) , · · · , v M ( S )) . And then define η : { 0 , 1 } M → 2 J , η ( v ( S )) = { m -1 M | v m ( S ) ≥ 1 } . Notice that by this construction, d H ( η ( v ( S )) , S ) < 1 M ≤ δ . So let ω : { 0 , 1 } M → R and ω ( v ) = f ( η ( v )) , we have

The last inequality holds for the definition of Hausdorff distance and continuity of f . Here ω and { h m } M m =1 could be made up of a multi-layer perceptron network [51]. { ˆ v m } M m =1 consist of a max pooling layer on { h m } M m =1 and { v m } M m =1 can be composed of a simple perceptron layer on { ˆ v m } M m =1 , which compares ˆ v m ( S ) and 1 . For the general cases k ≥ 1 , it suffices to get the same conclusion by simply extending the 1 dimensional functions h m , ˆ v m , v m to k dimension. So there is a ℓ p -PointNet++ P that can approximate any continuous function f on 2 J .

We employ the RBF theory of [57] to give the second conclusion. For completeness, we restate it here.

Theorem 3 ([57]) . The radio basis networks consist of a family of functions(RBF) noted by S K :

̸

where x ∈ R d , z i ∈ R d , σ ∈ R , H ∈ N . S K is dense in ℓ 1 ( R d ) , if K satisfies: 1.integrable bounded, 2. K is continuous almost everywhere, 3. ∫ K ( x ) dx = 0 .

It's clear that the ℓ p -norm ∥ · ∥ p : R d → R satisfies all the three conditions on K . Besides, a large enough ℓ p based convolution layer with a full connected layer could represent all the functions ∑ H i =1 a i · ∥ ( x -z i σ ) ∥ p . So for any ℓ 1 -integrable function g , there exists an ℓ p -PointNet++ P ′ such that for any ϵ > 0 , ∫ | g ( x ) -P ′ ( x ) | dx < ϵ .

A.3 Proof of Theorem 2

Before proving, we restate an important result in online learning and we will use it in the following.

Lemma 1 ([58]) . For any Q ∈ S d + and convex feasible set F ⊂ R d , suppose u 1 = min x ∈F ∥ Q 1 / 2 ( x -z 1 ) ∥ and u 2 = min x ∈F ∥ Q 1 / 2 ( x -z 2 ) ∥ then we have ∥ Q 1 / 2 ( u 1 -u 2 ) ∥ ≤ ∥ Q 1 / 2 ( z 1 -z 2 ) ∥ .

Our proof framework is similar to that of [58]. Here is a standard argument in momnet method.

Lemma 2. Suppose m t = γm t -1 +(1 -γ ) g t with m 0 = 0 and 0 < γ < 1 . We have

Proof. By Cauchy-Schwarz and Young's inequality, we have

Take the summation on t for both sides of the inequality, we have the conclusion.

Note that m 0 = 0 ,

So we have

So we could begin to prove Theorem 2. Suppose { x t } ⊂ R n . As the notation before, x ⋆ = arg min x ∈F ∑ T ∗ t =1 h t ( x ) and x t +1 = Π F ,α -1 / 2 t ( x t -α ( t ) · m t ) = min x ∈F ∥ α ( t ) -1 / 2 · ( x -( x t -α ( t ) · m t )) ∥ . By Lemma 1, ∥ α ( t ) -1 / 2 · ( x t +1 -x ⋆ ) ∥ 2 ≤ ∥ α ( t ) -1 / 2 · ( x t -α ( t ) · m t -x ⋆ ) ∥ 2 = ∥ α ( t ) -1 / 2 · ( x t -x ⋆ ) ∥ 2 + ∥ α ( t ) 1 / 2 · m t ∥ 2 -2 ⟨ q t m t -1 +(1 -q t ) g t , x t -x ⋆ ⟩ . Rearrange the inequality, we have

The second inequality holds for Cauchy-Schwarz inequality and for any a, b ∈ R , ab ≤ a 2 + b 2 2 . Because { h t } T ∗ t =1 are convex functions:

So we have

First we bound the part A:

Next we bound the part B. By definition of α ( t ) , α 1 (1) / √ t ≤ α ( t ) ≤ α 2 (1) / √ t . So we have

The second inequality holds for Jensen inequality and the third inequality follows from CauchySchwarz inequality. The forth inequalityholds for Lemma 2.

Combine the argument above and notice that α ( t ) -1 ≤ p -1 1 · √ T ∗ , we have

A.4 Equivalence of ℓ 2 -norm Measure and Classic Convolution in Convergence

We find that ℓ 2 -norm Net is the linear transformation to the inner product convolution network, here we give the detailed calculation.

The output of the ℓ 2 -norm Net in Eq. 25.

Therefore, we can express it as the following:

Notably, the term ∑ i,j K ( i, j ) 2 remains constant for each channel, and ∑ t ≥ 1 ∑ i,j P t ( i, j ) 2 represents the square of ℓ 2 -norm of each input patch. If this term is invariant across patches, the ℓ 2 -norm Net's output can be regarded as a linear transformation of the CNNs' output.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: see the abstract and introduction part.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: see Sec.A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Instructions about experimental settings are in Sec. 5, and the URL of the project will be released after the paper is accepted.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: see the zip files of codes in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: see the main manuscript.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: the effect of random seed could almost be negligible since we set the same initiation seed during experiments. Reproducibility can be guaranteed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: see the main manuscript.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: we have conformed with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: see Sec. 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: see the main manuscript.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using the \\( \\ell_p \\)-norm, specifically the \\( \\ell_1 \\)-norm, to replace the classic squared \\( \\ell_2 \\)-norm convolution in 3D tasks. The \\( \\ell_1 \\)-norm kernel function relies on addition, reducing computational cost. Initial gradient implementation revealed insufficient gradient values. To address this, the authors gradually transition the significant gradient from \\( \\ell_2 \\) to \\( \\ell_1 \\) as training progresses and employ momentum updates and learning rate scheduling. By replacing traditional 3D networks with their \\( \\ell_1 \\)-norm networks, the paper demonstrates competitive performance on ModelNet classification and S3DIS semantic segmentation tasks at significantly lower costs.\n\nSTRENGTHS:\ni)  The proposed 3D \\( \\ell_1 \\)-norm Net is novel and inspiring for 3D tasks,  which is broadly applicable somewhat.\n\nii)  The proposed tricks (the optimizer and the customized operator) make the \\ell_1-norm network get competitive performance at a lower cost.\n\niii)  The proof of the appendix is detailed and carefully.\n\nWEAKNESSES:\nMinor typos error:\n\ni) In L-651, it should be \"The *\\ell_p*-norm\" rather than \"\"The \\ell_p norm\"\"\n\nii) Very few words require consistent capitalization.\n\niii) Formulas in separate rows may have missing order numbers, it is recommended to unify this style.\n\niv) The reference format is not uniform enough.\n\nQUESTIONS:\ni)  The current experiments seem to be on small datasets. Does the current method perform well on larger datasets? \n\nii)  During training, The key idea of this paper is to use the $\\ell_2$ gradient. As far as I am concerned, another potential way is to use $\\ell-_\\infty$ gradients.  The dual space of $\\ell__\\infty$ is the $\\ell_1$ norm; Thus, the gradients of $\\ell_ _\\infty$ will be sparse. Why did the authors not try this idea and speed up the training time?\n\niii)  The \\ell-p-norm Space measurement can be also seen in other filed, such as  image analysis[1], Inpainting[2] , the author claim that \"these method can't be **directly** applied into 3D tasks\", why?\n\niv)  The proposed lower bound and upper bound are Eq(13) and Eq(14), why the authors design the bounds as these?\n\n\n\n[1] Generalized 2-D principal component analysis by Lp-norm for image analysis.\n\n[2] Rank-One Matrix Approximation With ℓ*p*-Norm for Image Inpainting.\n\nLIMITATIONS:\nMore generalization experiments on other datasets can be considered.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes for using the $\\ell_{p}$ norm in 3D convolution as a substitute for the inner product. Out of different choices, the authors pick the $\\ell_{1}$-norm because it's faster and uses less energy than the inner product. This is because the $\\ell_{1}$-norm relies on addition, which is simpler computationally. To optimize the $\\ell_{1}$-norm-based convolution models, they propose a tailored optimization approach leveraging Mixed Gradient Strategy and Dynamic Learning Rate Controller. Finally, the method is evaluated on ModelNet10 and ModelNet40 datasets, showcasing its usage in learning point cloud features for object classification.\n\nSTRENGTHS:\n1 ) The writing and organization of this paper is great.\n\n2 ) The experiments are sufficient in both main paper and supp, ranging from global Tasks, Semi-dense Prediction, Dense Prediction.\n\n3 ) The idea of replacing traditional convolution with a ℓ𝑝-based convolution is quite foundational and novel for 3D tasks.\n\n4 ) The proposed methods can be easily extended to other 3D backbones, which can be more environmentally friendly.\n\n5 ) The proposed methods demonstrate good performance on most tasks (such as ModelNet classification and S3DIS semantic segmentation) at a significantly lower cost.\n\nWEAKNESSES:\n1 )  The baseline models should be detailed introduced, including the model structure, integration (i.e., replacing) method of the proposed method, just as Table 8 from Ablation Experiments.\n\n2 ) The layout of Table 5 needs to be further optimized, such as splitting it into two independent tables for parallel layout.\n\nQUESTIONS:\n1 )  Is there any basis for the authors to assume that the mean of data **X** is 0 when discussing robustness?\n\n2 )  The authors discuss regret without detailed motivation, How does it works? I wonder if it is necessary to introduce the regret. \n\n3 ) There are many symbols in this article, it is recommended to add more explanations  in a separate section.\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of enhancing the representational capacity of traditional convolution methods. To tackle this issue, the authors introduce a novel convolution approach based on $\\ell_p$-norm and offer customized optimization strategies to expedite the training process. Extensive theoretical and empirical results have verified that the proposed algorithms demonstrate competitive performance compared to other baselines.\n\nSTRENGTHS:\nThe concept of using the $\\ell_p$ norm-based kernel to develop a convolution operator is intriguing. This proposed method has the potential to notably enhance the flexibility of the traditional convolution operator.\n\nWEAKNESSES:\n1. The writing of this paper requires improvement, particularly in providing additional details regarding the proposed method. The current version of this manuscript may lead to reader confusion.\n\n2. The role of the theoretical results presented in this paper is unclear. There appears to be a gap between the theoretical guarantees and the empirical evidence provided.\n\nQUESTIONS:\n1. The authors ought to include a formal definition of the \"$\\ell_p$ norm space\" concept in the title of this paper.\n\n2. The formulation of the proposed convolution operator in Eq. 2 on the 35th line of page 2 lacks an intuitive explanation, which may confuse the reader.\n\n3. What precisely is the definition of $\\mathcal{P}(S)$ in Eq. 3? Additionally, the primary method proposed in this paper is only briefly introduced in the introduction without further detailed description, making it less reader-friendly.\n\n4. It appears that the proposed training strategy \"Dynamic Learning rate Controller\" is seemingly unrelated to the $\\ell_p$ convolution method, if I haven't overlooked anything.\n\n5. There are several errors are evident in theorem 2. Initially, the mismatch between the subscripts $k$ and $t$ in the statement \"we could show that for any convex functions $\\lbrace h_k \\rbrace_{t=1}^{T^*}$ \" is notable. Furthermore, it would be beneficial if the authors could clarify the significance of $h$. Moreover, although this theorem addresses convex functions, it is crucial to acknowledge that numerous deep learning tasks entail non-convex functions. In conclusion, the reasoning behind the authors' choice to include a regret analysis in theorem 2 within this paper lacks clarity.\n\n6. Building on the previous point, it is advisable for the authors to undertake a conventional convergence analysis for the proposed optimization algorithms. This recommendation stems from the availability of numerous well-established mathematical tools designed for assessing convergence in offline optimization, particularly under non-convex or unbounded gradient conditions. These tools are often better suited for the intricacies of deep learning environments.\n\n7. In the experimental section, it is recommended that the authors provide a comprehensive outline of the setup for the online learning tasks they consider, particularly detailing how data arrives sequentially. Additionally, the experimental section overlooks sequential decision tasks or time series tasks. Therefore, I suggest that the authors include these related tasks to further strengthen the justification for providing a regret guarantee in theorem 2. Besides, as this paper is affiliated with the learning theory track, the empirical results presented seem somewhat disconnected from the theoretical findings. Specifically, while theorem 2 offers a regret guarantee for the proposed method, the metric used in the experimental section does not measure regret.\n\n8. In comparing the proposed method with baseline methods, the authors appear to have neglected to consider recent online learning methods, such as online non-convex learning using follow-the-perturbed-leader [1,2].\n\n[1] Suggala, Arun Sai, and Praneeth Netrapalli. \"Online non-convex learning: Following the perturbed leader is optimal.\" Algorithmic Learning Theory. PMLR, 2020.\n\n[2] Suggala, Arun, and Praneeth Netrapalli. \"Follow the perturbed leader: Optimism and fast parallel algorithms for smooth minimax games.\" Advances in Neural Information Processing Systems 33 (2020): 22316-22326.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new convolution method based on the \\( L_p \\)-norm. The authors provide a theoretical foundation by proving the universal approximation theorem for \\( L_p \\)-norm networks and analyzing the robustness and feasibility of \\( L_p \\)-norms in 3D tasks. Several key findings are highlighted in this work: \n\n1. \\( L_\\infty \\)-norm convolution is prone to feature loss. \n2. \\( L_2 \\)-norm convolution essentially performs a linear transformation in traditional CNNs. \n3. \\( L_1 \\)-norm convolution is an economical and effective method for feature extraction. \n\nTo further enhance the capabilities of \\( L_1 \\)-norm based networks, the paper proposes a series of customized training and optimization strategies.\n\nIn the experimental section, the authors apply their methods to classical 3D networks such as PointNet and PointNet++, achieving competitive performance at a lower cost. In summary, the \\( L_1 \\)-norm network can achieve similar performance to traditional convolutional networks but with reduced computational cost and lower instruction latency.\n\nSTRENGTHS:\n-- This paper is clear and easy to follow, which is well organized.\n\n-- The proof of the universal approximation theorem for Lp-norm Nets and the analysis of the robustness and feasibility of Lp-norms are interesting and beautiful.\n\n-- The comparative results between different $\\ell_{p}$-norm-based convolutions presented in this paper are valuable, offering a meaningful technical reference for further method design and subsequent research in 3D vision.\n\nWEAKNESSES:\nThere are some weakness/concerns need to be discussed:\n\n--Noise distribution is an interesting issue, but this paper only analyzes the impact of Gaussian noise when considering random noise. How do the authors solve other noise distribution conditions？\n\n--In this paper, how to come up with the idea of the \\( L_p \\)-norm convolution  is not discussed in detail, although its mechanism is well  illustrated.\n\nQUESTIONS:\n--This paper claim that \\( L_p \\)-norm are more robust than that based on inner product, however, why only \\( L_2 \\)-norm  is proved? how about the other cases when $p$ differs?\n\n--In Table 2, the results from \\( L_2 \\)-norm Net is also competitive, why not choose the \\( L_2 \\)-norm Net for the further study but  \\( L_1 \\)-norm?\n\n--In Sec 3.1, the proposed Universal Approximation, I wonder why  the second part is presented through integration? this is confusing compared with other works such as  \"Multilayer feedforward networks universal approximator\"？\n\nOverall, I have some questions about the theory and motivation above, which hopes to be answered.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}