Decodable and Sample Invariant Continuous Object Encoder
Authors: Dehao Yuan
Furong Huang
Cornelia Fermuller
Yiannis Aloimonos
Venue: ICLR 2024 poster (2024)
Keywords: Point Cloud
PointNet
Function Mapping
Encoder
Noise Robustness
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a function encoder that converts its sample into a fixed-length vector. The encoding is robust to distribution and density variation of the sampling.

Abstract: We propose Hyper-Dimensional Function Encoding (HDFE). Given samples of a continuous object (e.g. a function), HDFE produces an explicit vector representation of the given object, invariant to the sample distribution and density. Sample distribution and density invariance enables HDFE to consistently encode continuous objects regardless of their sampling, and therefore allows neural networks to receive continuous objects as inputs for machine learning tasks, such as classification and regression. Besides, HDFE does not require any training and is proved to map the object into an organized embedding space, which facilitates the training of the downstream tasks.  In addition, the encoding is decodable, which enables neural networks to regress continuous objects 
by regressing their encodings.  Therefore, HDFE serves as an interface for processing continuous objects. 

We apply HDFE to function-to-function mapping, where vanilla HDFE achieves competitive performance with the state-of-the-art algorithm. We apply HDFE to point cloud surface normal estimation, where a simple replacement from PointNet to HDFE leads to  12\% and 15\% error reductions in two benchmarks. 
In addition, by integrating HDFE into the PointNet-based SOTA network, we improve the SOTA baseline by 2.5\% and 1.7\% on the same benchmarks.

Published as a conference paper at ICLR 2024
DECODABLE AND SAMPLE INVARIANT CONTINUOUS OBJЕCT ENCODER
Dehao Yuan, Furong Huang, Cornelia Fermüller & Yiannis Aloimonos Department of Computer Science University of Maryland College Park, MD 20740, USA {chyuan, furongh, fermulcm, jyaloimoJ0ume edu
ABSTRACT
We propose Mlyeer-imenssin Function Encoding (HDFE). Given samples of continuous object (e.g. a function), HDFE produces an explicit vector representa- tion of the given object, invariant the sample distribution and density. Sample distribution and density invariance enables HDFE to consistently encode contin- uous objects regardless of their sampling, and therefore allows neural networks to receive continuous objects as inputs for machine learning tasks, such as clas- sification and regression. Besides, HDFE does not require any training and is proved to map the object into an organized embedding space, which facilitates the training the downstream tasks. addition, the encoding is decodable, which enables neural networks to regress continuous objects by regressing their encod- ings. Therefore, HDFE serves as an interface for processing continuous objects.
We apply HDFE to function-to ceann-toffufctee function mapping, where vanilla HDFE achieves competitive performance with the state-of-the-ar algorithm. We apply HDFE point cloud surface normal estimation, where simple replacement from PointNet to HDFE leads to 12% and 15% error reductions in two benchmarks. addition, by integrating HDFE into the PointNet-basse SOTA network, improve the SOTA baseline by 2.5% and 1.7% on the same benchmarks.
INTRODUCTION
Continuous objects objects that can be sampled with arbitrary distribution and density. Examples include point clouds (Guo et al., 2020), event-based vision data (Gallego et al. 2020). and sparse meteorological data (Lu al. 2021p. A crucial characteristic of continuous objects, which poses challenge for learning, is that their sample distribution and size varies between training and test sets. For example, point cloud data in the testing phase may be sparser or denser than that the training phase. A framework that handles this inconsistency is essential for continuous object learning.
When designing the framework, four properties are desirable: (1) Sample distribution invariance: the framework is not affected by the distribution from which the samples are collected. (2) Sample size invariance: the framework is not affected by the number of samples. (3) Explicit representation: the framework generates outputs with fixed dimensions, such as fixed-length vectors. (4) Decodabil- ity: the continuous object can be reconstructed arbitrary resolution from the representation.
Sample invariance (properties and 2) ensures that differently sampled instances of the same con- tinuous objects are treated consistently, thereby eliminating the ambiguity caused by variations in sampling. An explicit representation (property 3) enables a neural network to receive continuous objects as inputs, by consuming the encodings the objects. Decodability (property 4) enables neural network to predict continuous object, by first predicting the representation and then decod- ing it back to the continuous object. Fig. Iillustrates the properties and their motivations.
However, existing methodologies, which we divide into three categories, are limited when incorpo- rating the four properties.
(1) Discrete framework. The methods discretize continuous objects and process them with neural networks. For example, Liu et al. (2019) uses 3D-CNN to process elized point clouds, Kim et al.] (2017) uses an RNN to predict particle trajectories. These methods not sample invariant- the spatial and temporal resolution must be consistent across the training

---

Published as a conference paper at ICLR 2024
Netwadk
regresout
HDFE
dstance
Recoumad
Nound
Figure 1: Left: HDFE encodes continuous objects into fixed-length vectors without any training. The encoding is not affected by the distribution and size with which the object is sampled. The encoding can decoded to reconstruct the continuous object. Right: Applications of HDFE. HDFE can be used to perform machine learning tasks (e.g. classification, regression) continuous objects. HDFE also enables neural networks to regress continuous objects by predicting their encodings.
and testing phases. (2) Mesh- grid-based framework. They operate on continuous objects defined on mesh grids and achieve discretization invariance (the framework is not affected by the resolution of the grids). Examples include the Fourier transform (Salih) 2012) and the neural operator (Li et al.) 2020a). But they do not apply to sparse data like point clouds. (3) Sparse framework. They oper- ate on sparse samples drawn from the continuous object. Kernel methods (Hofmann al. 20088) work for non-linear regression, classification, etc. But they do not provide an explicit representation of the function. PointNet (Qi et al. 2017a) receives sparse point cloud input and produces plicit representation, but the representation is not decodable (see Appendix B). In addition, all the frameworks require extra training of the encoder, which is undesired in some scarce data scenarios.
Currently, only the vector function architecture (VFA) (Frady et al. 2021 can encode an explicit function into vector through sparse samples, while preserving all four properties. However, VFA is limited by its strong assumption of the functional form. VFA requires the input function to con- form to fzz) Σx ag K Kxx, Ty). where K X X - E kernel defined on X. If the input function does not conform to the form, VFA cannot apply induces large errors. In practice, such requirement rarely satisfied. For example, (ar cannot even approximate a constant function g(x:) = 1: to approximate the constant function, the kernel K must be constant. But with the con- stant kernel, fx) cannot approximate other non-constan functions. Such limitation greatly hinders the application of VFA. Kindly refer to Appendix C for failure cases and detailed discussions.
We propose hyper-dimensional function encoding (HDFE), which does not assume any explicit form of input functions but only requires Lipschitz continuity (Appendix input types). Consequently, HDFE can encode a much larger class of functions, while holding all four properties without any training. Thanks to the relaxation, HDFE can be applied multiple real-world applications that VFA fails, which will be elaborated on in the experiment section. HDFE maps the samples to high-dimensional space and computes weighted averages of the samples in that space to capture collective information of all the samples. A challenge in HDFE design is maintaining sample invariance, for which we propose novel iterative refinement process to decide the weight of each sample. The contributions of our paper can be summarized as follows:
some suitable
We present HDFE, an encoder for continuous objects without any training that exhibits sample invariance, decodability, and distance-pree preservation. To the best of our knowledge, HDFE is the only algorithm that can encode Lipschitz functions while upholding all the four properties.
We provide extensive theoretical foundation for HDFE. We prove that HDFE is equipped with all the desirable properties. We also verify them with empirical experiments.
We evaluate HDFE on mesh-grid data and sparse data. In the mesh-grid data domain, HDFE achieves competitive performance as the specialized state-of-the-art (SOTA) in function-to- function mapping tasks. In the sparse data domain, replacing PointNet with HDFE leads to average error decreases of 12% and 15% in two benchmarks, and incorporating HDFE into the PointNet-based SOTA architecture leads to average error decreases of 2.5% and 1.7%.
2 PROBLEM DEFINITION AND METHODOLOGY
Let F be the family of c-Lipschitz continuous functions defined on a compact domain with a compact range Y. In other words, vʃ F. F. X Y and dy (f()), f(x2) I dx (2),
2

---

Published as a conference paper at ICLR 2024
where dx and (Y, dy) are metric spaces, and is the Lipschitz constant. Our goal is to find a representation algorithm that can encode a function f E F into a vector representation E CN.. To construct it, we will feed samples of the function mapping { (z;, f(x;)) ] to the representation algorithm, which will generate the vector representation based on these samples.
We require the function representation satisfy the following: (1) Sample distribution invariance: the function representation is "not affected" by the distribution from which the samples are col- lected. (2) Sample size invariance: the function representation is "not affected" by the number of samples. (3) Fixed-length representation: all functions are represented by fixed-length vectors. (4) Decodability: as new inputs query the function representation, it can reconstruct the function values.
To better formalize the heuristic expression of "not affected" in Properties I and 2, we introduce the definition of asymptotic sample invariance to formulate an exact mathematical expression:
Definition 1 (Asymptotic Sample Invariance). Let ʃ X - Y be the function to be encoded, p : X (0, 1) be a probability density function (pdf) on X, { ~ p(X) be independent samples of X. Let Fn be the representation computed from the samples {ar, (x:)] asymptotic sample invariance implies F,, converges to limit F oo independent of the pdf p.
from
In this definition, sample size invariance is reflected because the distance between Fm and Fn can be arbitrarily small as m, n become large. Sample distribution invariance is reflected because the limit Fxx does not depend on the pdf p. as long as p is supported on the whole input space X.
With the problem definition above, present our hyper-dimensional function encoding (HDFE) approach. Sec. 2. introduces how HDFE encodes explicit functions. Sec. 2.2 generalizes HDFE to implicit function encoding. Sec. [.3 realizes HDFE for vector-valued function encoding. Finally, Sec. [.44estabbistt
the theorems that HDFE is asymptotic sample invariant and distance-preserving Throughout the section, we assume the functions are c-Lipschitz continuous. The assumption will also be explained in Section 2.4 Kindly refer to Appendix A for the table of notations.
2.1 EXPLICIT FUNCTION ENCODING
Encoding HDFE inspired by the methodology of hyper-dimensionss computing (HDC) (Kleyko et al. 2023), where one encodes an indefinite number of data points into fixed-length vector. The common practice is to first map the data points to a high-dimensional space and then average the data point representations in that space. The resulting superposed vector can represent the distribution of the data. Following the idea, we represent an explicit function as the superposition of its samples:
F =
Σ wi E(x;, Vs)
where maps function samples to a high-dimensional space C.V, The question remains (a) how to design the mapping to make the vector decodable; (b) how to determine the weight of each sample w; so that the representation is sample invariant. We will answer question (a) first and leave question (b) to the iterative refinement section.
Regarding the selection of E(x, y). a counter-example a linear mapping, where the average of the function samples in the high-dimensional space will degenerate to the average of the function samples, which does not represent the function. To avoid degeneration, the encodings of the samples should not interfere with each other if they are far from each other. Specifically, if dx (I1, x2) is larger than a threshold co. their function values f f(1 (x2) may be significantly different. case, want E(x1, y1) to be orthogonal to E(22, M22 to avoid interference. On the other hand, if dx (#1, is smaller than the threshold cu. by the Lipsehitz continuity, the distance between their function values of (z, fxz)) is bounded by cEg. In this case, want E(x,, 91) to be similar to F( 122. We call the tunable threshold én the receptive field of HDFE, which will be discussed be formulated as:
this
in Sec. 2.4 Denoting the similarity between vectors (1, :), the requirement
(E(x, v), E(x',''))
R
dx x') < cD
decays to 0 quickly dx(x, x') en
In addition to avoiding degeneration, we also require the encoding to be decodable. This can be achieved by factorizing E(x, y) into two components: We first map I, and ys to the high- dimensional space CN through two different mappings Ex and Ey. To ensure equation is satis- fied, we require (Fx(x), Ex (z')) ss 1 when dx (2r, x²) < en and that it decays 0 otherwise. The
3

---

Published as a conference paper at ICLR 2024
property of Ey will be mentioned later in the discussion of decoding. Finally, compute the joint embedding of z; and y through a binding operation @: E(,, U4) = Ex ( @ Ey (y.).
We will show that the representation is decodable if the binding operation satisfies these properties:
1. commutative: & y
2. distributive: 2 (
3. similarity preserving: (ar @ y,
(v.
4. invertible: there exists an associative, distributive, similarity preserving operator that undoes the binding, called unbinding @ satisfying (z @ v) Ø (ær @ z) @ and (x: 6 y) @ x y-
The binding and unbinding operations can be analogous to multiplication and division, where the difference is that binding and unbinding operate vectors and similarity preserving.
Decoding With the properties of the two operations, the decoding of the function representation be performed by similarity search. Given the function representation F E C.V. and query input z0 E X, the estimated function value yo is computed by:
1/0 argmax, (F 2 ; Ey
(3)
The distributive property allows the unbinding operation be performed sample-by-sample. The invertible property allows the unbinding operation to recover the encoding of the function values: Ex, @ By (f(z)) Ø Ex (#o) Ey (f(z)) Ey (f(o)) when dx (Ir, xς) small. The similarity preserving property ensures that [Ex (a:) @ Ex (zo) & (f produces vector orthogonal to Ey (zxo)) when the distance between two samples is large, resulting in summation of noise. The following formula illustrates the idea and Appendix E details the derivation.
F 2 Ex(xo)= = Σ (xx( @ Ey (f(x)) @ Ex(z))
Σ
w: Ey (f(x))
Σ
wi (Ex(a) 0 Ex & Ey (f(x))
d(
d(Fo.a))>es
(E(o)())
since orthogonal (f(zo)
After computing F @ (x())), we search for E Y such that the cosine similarity between Ey (v)
and F 2 (xo) ) is maximized. We desire that 8. By (By(()) (v), Ey > 0 for all and y' so that the optimization can be solved by gradient descent. See Appendix [F] for detailed formulation.
Since the decoding only involves measuring cosine similarity, in the last step, we normalize the function representation to achieve sample size invariance without inducing any loss:
F normalize Σ
w; [x (2x,) @ Ey (J(ra)))
(4)
Iterative refinement for sample distribution invari- ance In equation 4 are left to determine the weight of each sample so that the representation is sample invari- ant. To address this, propose an iterative refinement process to make the encoding invariant to the sample dis- tribution. We initialize w, and compute the initial function vector. Then we compute the similarity between the function vector and the encoding of each sample. We then add the sample encoding with the lowest similarity the function vector and repeat this process until the lowest similarity no longer increases. By doing so, the output will be the center of the smallest bull containing all the sample encodings. Such output is asymptotic sample invariant because the ball converges to the smallest ball contain- ing Uzex [x (x) @ Ey (f()))
Algorithm 1 Iterative Refinement
z - Ex (xx) @ Ey(((z;)) for all i. == Σx while min (F, (F,2) still increases do
j argminn (F, 2))
F- F+
end while
the sample size goes large, where the limit ball only depends on the function. We left the formal proof to the Appendix H..]] In Appendix [.3] we introduce a practical implementation of the iterative refinement for saving computational cost.
2.2 IMPLICIT FUNCTION ENCODING
Generalizing HDFE to implicit functions is fairly straightforward Without loss of generality, we assume implicit function is represented f(x O. Then it can be encoded using equation
4

---

Published as a conference paper at ICLR 2024
where the weights wa determined by the iterative refinement:
F y=u = normalize
Σ
Wx Ex (x))
(5)
(=)(=)=
The formula can be understood as encoding an explicit function g. where g(x) 1f f(x) 0 and
we can obtain equation 5 The formula can be interpreted in simple way: continuous object can be represented as the summation of its samples a nigh-dimenssonal space.
g(x) = 0 f(x) + O. Then by choosing Ey (1) and By (0) = 0 in equation 4
2.3 VECTOR- VALUED FUNCTION ENCODING
In the previous sections, we established theoretical framework for encoding e-Lipschitz continu- ous functions. In this section, we put this framework into practice by carefully choosing appropriate input and output mappings Ex. Ey, the binding operator 8, and the unbinding operator @ in equa- I We will first state our choice and then explain the motivation behind it.
Formulation Let (x, y) be one of the function samples, where Rd and y E R, the mapping Ex X + CNN, Fy IR + CN and the operations & and @ are chosen as:
Ex(x) exp (i (i.a a T
Ey) exp (18Vv)
m
Tx
Ex @ Ey == exp (i + 13
dx
(x 2 Ey == exp (i 1..0
1,30y)
where i is the imaginary unit, + E R Nxm and v E RNNN are random fixed matrices where all elements drawn from the standard normal distribution. and 3 are hyper-parameters controlling the properties of the mappings.
The above way of mapping real vectors high-dimensional spaces is modified from Komer & Eliasmith (0220) known as fractional power encoding (FPE). We introduce the motivation for adopting this technique heuristically. In Appendix G we elaborate on the relation between FPE and radial basis function (RBF) kernels, which gives rigorous reason for adopting this technique.
Motivation
First, the mappings are continuous, which can avoid losses when mapping samples the embedding space. Second, the receptive field of the input mapping Ex (the eo in equation 2) can be adjusted easily through manipulating a. Fig. [a demonstrates how manipulating a can alter the behavior of Ex. Typically, has a magnitude of TO for capturing the
component of the function. Thirdly, the decodability of the output mapping Ey can easily be achieved by selecting appropriate values. We select I9 such that EE (0), Ey (1)) equal to 0 to utilize the space CN maximally while keeping the gradient of (Ey (11); Ey (y2)) non-zero for all 3/1 and 1/2 Per the illustration in Fig. 2a the optimal choice for 3 is 2.5. Finally, the binding and unbinding operators defined the element-wiie multiplication and division of complex vectors, which satisfy the required properties.
2.4 PROPERTIES OF HDFE
HDFE produces an explicit decodable representation of functions. In this section, we state theorem on the asymptotic sample invariance, completing the claim that HDFE satisfies all four desirable properties. We study the effect of the receptive field on the behavior of HDFE. We also state that HDFE is pietanne- and discuss the potential of scaling HDFE nigh-dimensional data. We leave the proofs to Appendix HH and verify the claims with empirical experiments in Appendix [] We include several empirical experiments of HDFE Appendix [ including the cost of the iterative refinement and its practical implementation, the effectiveness of sample invariance in a synthetic regression problem, and the analysis of information loss when encoding continuous objects.
Theorem (Sample Invariance). HDFE is asymptotic sample invariant (defined Definition 7
HDFE being sample invariant ensures functions realized with different sampling schemes are treated invariantly. Kindly refer to Appendix H.I and I.I for the proof and empirical experiments.
5

---

Published as a conference paper at ICLR 2024
Theorem 2 (Distance Preserving). Let f.9 X - Y be both c-Lipschitz continuous, then their 12-distance is preserved in the encoding. In other words, HDFE is an isometry:
|lf g = -frex
=I JEE
ff(z) g(x)) " b- b. G)
HDFE being isometric indicates that HDFE encodes functions into a organized embedding space, which reduce the complexity of the machine learning architecture when training downstream tasks on the functions. Kindly refer to Appendix H.2 a.2 for the proof and empirical experiment.
Effect of receptive field Fig. shows the reconstruction results of 1d function / IR + I. which demonstrates that HDFE can reconstruct the original functions given a suitable receptive field and sufficiently large embedding space. When using large receptive field (Fig. 2bb the figh-freqeency components will be missed by HDFE. When using a small receptive field (Fig. 2c the high-frequency components
be captured, but it may cause incorrect reconstruction if the dimension of the embedding space is not large enough. Fortunately, reconstruction failures can be eliminated by increasing the dimension of the embedding space (Fig. 2d).
0.2.
GT Function Fixted Function
GT Function
GT Function
e-10
Fiored Furctiion
Fizedd Function
a=4n
04-
044
62-
20
(a) Similarity between Ex (x) and Ex (0).
(b) Large recept. field. Dimension = 1000.
(c) Small recept. field. Dimension = 1000.
(d) Small recept. field. Dimension = 2000.
Figure 2: (a): How a and 3 equation affects the receptive field of HDFE. (b)-(d): Functions can be reconstructed accurately given suitable receptive field and encoding dimension. To capture the high-frequene component of the function, a small receptive field and high dimension are required.
Scale to high-dimensional input HDFE produces the function encoding based on the sparsely collected samples. Unlike mesh-grid based methods, which require mesh-grid domain and suffer from an exponential increase in memory and computational cost as the data dimension increases, HDFE uses superposition encode all the samples defined in the support of the function. This means the required dimensionality only depends on the size of the support, not on the data dimen- sionality. Even if the data dimensionality is high, HDFE can mitigate this issue as long as the data reside in low-rank subspace. Appendix [L6] gives an empirical experiment of nigh-dimensionall input to show the potential of HDFE work in low-rank high-dimensional scenarios.
EXPERIMENT
In this section, present applications of HDFE. Sec. 3.1 showcases how HDFE can be lever- aged for solving partial differential equations (PDE). This exemplifies how HDFE can enhance neu- ral networks receive function inputs and produce function outputs. Sec. 3.2, we apply HDFE to predict the surface normal of point clouds. This demonstrates how HDFE can enhance neural networks to process implicit functions and extract relevant attributes.
3.1 PDE SOLVER
Several neural networks have been developed to solve partial differential equations (PDE), such as the Fourier neural operator (Li et al. (020aa). In this section, we compare our approach using HDFE against the current approaches and show that achieve on-par performance. VFA does not apply to the problem since the input and output functions do not conform to the form that VEA requires.
To solve PDEs using neural networks, first encode the PDE and its solution into their vector embeddings using HDFE. Then, we train a multi-layer perceptron to map the embedding of the PDE the embedding of its solution. The optimization target is the cosine similarity between
Architecture

---

Published as a conference paper at ICLR 2024
the predicted embedding and the true embedding. Since the embeddings complex vectors, adopt a Deep Complex Network (Trabelsi et al. 2017) as the architecture of the multi-layer percep- tron. The details are presented in Appendix J.I Once the model is trained, use it to predict the emhedding of the solution, which then decoded to obtain the actual solution.
Dataset We use 1d Burgers' Equation (Su & Gardner 1969) and 2d Darey Flow TTk 1957
for evaluating our method. The error measured by the absolute distance between the predicted solution and the ground-truth solution. The benchmark (Li al. 2020c) has been used evaluate neural operators widely. For the Id Burgers' Equation, it provides 2048 PDEs and their solutions, sampled a 1d mesh grid at a resolution of 8192. For the 2d Darcy Flow, it provides 2048 PDEs and their solutions, sampled at a 2d mesh grid at resolution of 241 x 241.
Baselines We evaluate our HDFE against other neural network PDE-solving methods. These in- clude: PCANN (Bhattacharya et al. 2021: ; MGKN: Multipole Graph Neural Operator (Li et al. 2020c): FNO: Fourier Neural Operator (Li al. 2020a
0.04
PCANN
HGKN
FNO
MIGN
0.025
Function Prodiction Error
acconstruction Error
0.06
HDFE trocon. em))
RNO
HDFE
HOTE pred,
Test Error
Test Error
Test Error
0.02
0.010
0.03
0.0
123
250 512 1024
2048 4030 8192
0.000
4000
8000
16000
24000
0.0
121
241
Resolution
Dimension
Resolution
Figure 3: HDFE solves PDE by predicting the encoding of its solution and then reconstructing at points, so the error consists of a function encoding prediction error and a reconstruction error. Left: Prediction error of different methods under different testing resolutions, evaluated the Id Burgers' equation. Mid: The reconstruction error (in HDFE) dominates the function encoding prediction error, while the reconstruction eπrоr can be reduced by increasing the dimensionality of the embedding. Right: Prediction error of different methods evaluated on 2d Darcy Flow.
When decoding required. our approach achieves 55% lower prediction error than MGKN and PCANN and competitive performance to FNO. The error of HDFE consists of two compo- nents. The first is the error arising from predicting the solution embedding, and the second is the reconstruction eтror arising when decoding the solution from the predicted embedding. In contrast, FNO directly predicts the solution, and hence, does not suffer from reconstruction error. If we con- sider both errors, HDFE achieves comparable performance FNO. Fig. shows the comparison.
On the other hand, when decoding not required, our approach achieves lower error than FNO. Such scenarios happen frequently when we functions only as input, for example, the local ge- ometry prediction problem in Experiment 3.2 Despite the presence of reconstruction error, the reconstruction error be reduced by increasing the embedding dimension, shown in Fig- ure 3 (Mid). Increasing the embedding dimension may slightly increase the function prediction error possibly because the network is not adequately trained due limited training data and some overfitting, We conjecture that this prediction error can be reduced with more training data.
In addition to comparable performance, HDFE overcomes two limitations of FNO. First, HDFE provides an explicit function representation, resolving the restriction of FNO, which only models the mappings between functions without extracting attributes from them. Second, HDFE not only works for grid-sampled functions but also for sparsely sampled functions.
3.2 UNORIENTED SURFACE NORMAL ESTIMATION
Next, apply HDFE to extract attributes of functions, setting where neither neural operators VFA applies, because neural operators do not consume sparse samples and VFA does not encode implicit functions. We predict the unoriented surface normal from 3d point cloud input.

---

Published as a conference paper at ICLR 2024
Baselines We compare HDFE with two baselines. In the first baseline, compare the vanilla HDFE with the PCPNet (Guerrero et al. 2018). which is a vanilla PointNet (Qi et al. 2017a) archi- tecture. We replace the PointNet with our HDFE appended with a deep complex network (Trabelsi eet al. 2017). In the second baseline, we incorporate HDFE into HSurf-Net (Li et al. 2022). which the state-of-the-ar PointNet-based normal estimator. In both settings, we compare the effect of data augmentation in the HDFE module, where we add noise to the weight of each sample when generating the patch encoding by HDFE. Kindly refer to Appendix J.2 for details.
Dataset and metries We use the root mean squared angle error (RMSE) as the metrics, evaluated on the PCPNet (Guerrero et al. 2018) and FamousShape (Li et al. 2023) datasets. We compare the robustness for two types of data corruption: (1) point density: sampling subsets of points with two regimes, where gradient simulates the effects of distance from the sensor, and strips simulates local occlusions. (2) point perturbations: adding Gaussian noise to the point coordinates. Table reports normal angle RMSE comparison with the baselines on PCPNet and FamousShape. Appendix [K] reports the ablation studies examining the effect of the receptive field size and the dimensionality.
Table I: Unoriented normal RMSE results on datasets PCPNet and FamousShape. Replacing from PointNet to HDFE improves performance. Integrating HDFE with the SOTA estimator improves its performance. Applying data augmentation to HDFE improves its performance.
PCPNe Dataset means improvementt)
Slape Dataset
mecrrs inspewvement)
None
Low
Nose
33ed
High
Sinpe
Deasiyy
Urodien
Ascrage
None
Low
Nose
Med
High
Strige
Deisity
Gradient
Aserage
PCPSC CEPECCR a. 2011
15.27
13:45
14.5%
21.07
32.00
24.95
PPPNENPSUENNET HDFE
9.48
11.05
17.16
11.61
10.19
13.67
15.66
17.92
31.24
17.26
22.59
PCPN PoimNer
17.89
12.88
17.99
31.23
14.01
21.16
Difference
0.361
0.08
1.70
5.43
1.37
1.36
4.13
7,37
HSarf-Nr Tretal (022)
4.90
8.78
16.15
21.64
5.18
508
10.18
2.54
15.56
79:47
38.61
7.33
7.44
17.24
HBarfNar- mmm
4.13
8.64
16.14
21.64
5.02
4.87
10.07
746
18,90
29.42
38.56
7.77
7.35
17.68
Dilleenne
0.77
0.14
0.01
0.00
0.16 3
0.16
0.13
0.08
0.06
0.04
0.05
0.05 1
0.09
0.06
TeSirf HDDEE Ang.
3.89
8:78
16.14
21265
4.60
4.31
9.90
7:01
15.57
29.84
38,57
6,99
6.98
17.44
Difference
0:41
0.00
0.01
0.01
0.58
0.50
0.25
0.43
0.01
0.00
0.04 1
0.85 1
0.46
0.30
HDFE significantly outperforms the PointNet baseline. When processing the local patches, we replace PointNet with HDFE followed by neural network. This replacement leads to an average reduction in error of 1.70 and 3.79 on each dataset. This is possibly because HDFE encodes the distribution of the local patch, which is guaranteed by the decodability property of HDFE. PointNet, on the other hand, does not have such guarantee. Specifically, PointNet aggregates point cloud features through a max-pooling operation, which may omit points within the point cloud and fail to adequately capture the patch's 's distribution. Consequently, in tasks where modeling the point cloud distribution is crucial, such as normal estimation, PointNet exhibits higher error compared to HDFE.
HDFE, as a plug-in module, improves the SOTA baseline significantly. HSurf-Net [Li et al, 2022). the SOTA method in surface normal estimation, introduces many features, such local aggregation layers, and global shift layers specifically for the task. Notably, HDFE does not compel such features. We incorporate HDFE into HSurf-Net (See Appendix(.33 for details), where it leads to average error reductions of 0.25/0.30 on each dataset. Notably, such incorporation can be performed on any PointNet-based architecture across various tasks. Incorporating HDFE to other PointNet- based architectures for performance and robustness gain can be future research direction.
HDFE promotes stronger robustness to point density variance. In both comparisons and both benchmarks, HDFE exhibits stronger robustness to point density variation than its PointNet counter- part, especially in the Density-Gradient setting (error reduction of 4.79/7.37/0.52//. the effectiveness of the HDFE's sample invariance property and the embedding augmentation. Sam- ple invariance ensures a stable encoding of local patches when the point density changes. The embedding augmentation is second assurance to make the system more robust to density variation.
This shows
4 RELATED WORK
4.1
MESII-GRI--BSED FRAMEWORK
These methods operate on continuous objects (functions) defined on mesh grids with arbitrary reso- lution. They enjoy discretization invariance, but they do not receive sparse samples input.
Fourier transform (FT) can map functions from their time domains to their frequency domains. By keeping finite number of frequencies, FT can provide a vector representation of functions. ID-
8

---

Published as a conference paper at ICLR 2024
FT has been a standard technique for signal processing (Salih 2012) and 2D-FT has been used for image processing (Jain 1989). FT is also incorporated into deep learning architectures (Fan et al.
2019 Sitzmann al. 2020). However, FT is not scalable since the n-dimensional Fourier transform returns an n-dimensional matrix, which is hard to process when n. gets large.
Neural operator is set of toolkits model the mapping between function spaces. The technique was pioneered in DeepONet (Lu et al. (2019) and a series of tools were developed (Li et al.) 2021
2020b c Guibas al. 2021: Kovachki et al. 2021) for studying the problem. The most well-known work is the Fourier Neural Operator (FNO) (Li et al. 2020a). The approach showed promising accuracy and computational efficiency. Though proposed in 2020, FNO is still the first choice when mapping between function spaces (Wen et al., 2023 Renn et al. 2023) Gopakumar et al., (2023). Despite their success, neural operators lack explicit function representations and their application limited to mappings between function spaces.
4.2 SPARSE FRAMEWORK
These methods, including HDFE, work with sparse samples from continuous objects.
PointNet (Qi al. 2 is neural network architecture for processing point cloud data. uses multi-layer perceptrons to capture local features of every point and then aggregates them into a global feature vector that's invariant to the order of the input points. PointNet and its variation (Zaheer et al.] 2017| Qi et al.] [2017;; Doseph-Rivlin et al.]
2019; Yang et al.] 2019 Zhao et al.) 2019 Duan et al. 2019 Yan et al. (020) have been widely applied to sparse data processing, for example, for object classification Yan et al., 2020 Lin et al. 2019), semantic segmentation (Ma) al. 2020 Li et al. 2019), and object detection (Qi et al. 2020 Yang et al. (020) with point cloud input. However, PointNet does not produce a decodable representation. Specifically, after encoding a point cloud with PointNet, it is difficult to decide whether a point is drawn from the point cloud distribution. Besides, PointNet is also sensitive to perturbations in the input point cloud.
Kernel methods (Hofmann et al., 2008 are a type of machine learning algorithm that transforms data into higher-dimensional feature space via a kernel function, such as the radial basis function (RBF) (Cortes& & Vapnik 1995), which can capture nonlinear relationships. Though kernel methods can predict function values any query input (i.e. decodable) and the prediction is invariant the size and distribution of the training data, kernel methods do not produce an explicit representation of functions, so they are only used for fitting functions but not processing or predicting functions.
Vector function architecture VFA Frady et al. 2021) encodes a function of the form f(x) Σn 0g. K(2 xx4) into a vector, where K : x + R is a kernel defined on the input space. VFA and HDFE share a similar high-level idea. They both map the samples to sigh-dimensional space and compute the weighted average in that space. However VFA determines the weight by relying on the assumption of the functional form. HDFE, on the other hand, uses iterative refinement to solve the weights. The iterative refinement coupled with the binding operation relaxes the assumption required by VFA and enables encoding functions across larger class of inputs. In addition. VFA is limited to empirical experiments such as non- non-linear regression and density estimation, without practical applications. In comparison, HDFE is demonstrated to be applicable to real-world problems.
CONCLUSION
We introduced Hyper-Diiennsionl Function Encoding (HDFE), which constructs vector represen- tations for continuous objects. The representation, without any training, is sample invariant, decod- abile, and isometric. These properties position HDFE as an interface for the processing of contin- uous objects by neural networks. Our study demonstrates that the significantly reduced errors compared to PointNet-based counterparts, especially in the presence of density perturbations. This reveals that HDFE presents a promising complement to PointNet and its variations for processing point cloud data. Adapting HDFE (e.g. imposing rotational invariance to HDFE) to tasks like point cloud classification and segmentation offers promising avenues for ex- ploration. Still, HDFE does possess limitations in encoding capacity. For functions defined over large domains or highly non-linear functions, HDFE can experience underfitting. The exploration of techniques to enhance HDFE's ' capacity remains promising research. Regardless, HDFE already shows strong applicability in dow-dimensional (1D, 2D, 3D) inputs.
architecture attains
9

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
The support of NSF under awards OISE 2020624 and BCS 2318255, and ARL under the Army Cooperative Agreement W91 INF2 is greatly acknowledged. The in-depth discussions with Denis Kleyko, Paxon Frady, Bruno Olshausen, Christopher Kymn, Friedrich Sommer, Pentti Kan- erva significantly improved the manuscript and we are grateful.
REFERENCES
Mahmoud Abbasi, Amin Shahraki, and Amir Taherkordi. Deep learning for network traffic moni- toring and analysis (ntma):: A survey. Computer Communications, 170:19 2021.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduc- tion and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121-157, 2021.
Yiang Chen, Dehao Yuan, Wanying Chen, Mingyun Hu, Jimmy CH Fung, Haochen Sun, and Xingcheng Lu. Estimation and variation analysis of secondary inorganic aerosols across the greater bay area in 2005 and 2015. Chemosphere, 292: 193:1333,3, 2022.
Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R Lyu. Experience report: Deep learning-bar system log analysis for anomaly detection. arXiv preprint arXiv:2107.05908, 2021.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning. 20:273-297, 1995.
Yueqi Duan, Yu Zheng, Jiwen Lu, Jie Zhou, and Qi Tian. Structural relational reasoning of point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 949-958, 2019.
Yuwei Fan, Cindy Orozco Bohorquez, and Lexing Ying. Ber-net: A neural network based on the nonstandard wavelet form. Journal of Computational Physics, 384:1-15, 2019.
Paxon Frady, Denis Kleyko, Christopher J Kymn, Bruno Olshausen, and Friedrich T Sommer. Computing on functions using randomized vector representations. arXiv preprint (.Xiv:2009.0349,, 2021.
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-180, 2020.
Vignesh Gopakumar, Stanislas Pamela, Lorenzo Zanisi, Zongyi Li, Anima Anandkumar, and MAST Team. Fourier neural operator for plasma modelling. arXiv preprint arXiv:2302.06444 2023.
Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J Mitra. Pepnet learning local shape properties from raw point clouds. In Computer graphics forum, volume 37, pp. 75-85, Wiley Online Library, 2018.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587 2021.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE transactions pattern analysis and machine intelligence, 43(12))4338-4333 2020.
Xian-Feng Han, Zhi-Ao Feng, Shi-Jie Sun, and Guo-Qian Guo-Qiang Xiao. 3d point cloud descriptors: state- of-the-art. Artificial Intelligence Review, pp. 1-51, 2023.
Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learn- ing. 2008
10

---

Published as a conference paper at ICLR 2024
Xiaoshui Huang, Guofeng Mei, Jian Zhang, and Rana Abbas. A comprehensive survey on point cloud registration. arXiv preprint arXiv:2103.0 0...000,,,,..... 2021.
Anil K Jain. Fundamentals of digital image processing. Prentice- Hall, Inc., 1989.
Mor Joseph-Rivlin, Alon Zvirin, and Ron Kimmel. Momen (e) t: Flavor the moments in learning to classify shapes. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pp. 0-0, 2019.
BoeoungDo Kim, Chang Mook Kang, Jaekyum Kim, Seung Hi Lee, Chung Choo Chung, and Jun Won Choi. Probabilistic vehicle trajectory prediction over occupancy grid map via recur- rent neural network. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 399 404. IEEE, 2017.
Denis Kleyko, Dmitri Rachkovskij. Evgeny Osipov, and Abbas Rahimi. survey on hyperdimen- sional computing aka vector symbolic architectures, part ii: Applications, cognitive models, and challenges. ACM Computing Surveys, 55(9): 2023.
Brent Komer and Chris Eliasmith. Efficient navigation using scalable, biologically inspired spatial representation. In CogSci, 2020.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint ar www.:::::80000. 080881 2021.
Qing Li, Yu-Shen Liu, Jin-San Cheng, Cheng Wang, Yi Fang, and Zhizhong Han. Hsurf-net: Nor- mal estimation for 3d point clouds by learning hyper surfaces. Advances in Neural Information Processing Systems, 35:4218-4230 2022.
Qing Li, Huifang Feng. Kanle Shi, Yue Gao, Yi Fang, Yu-Shen Liu, and Zhizhong Han. Shs-net: Learning signed hyper surfaces for oriented normal estimation of point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13591-13600 2023.
Zihao Li. Shu Zhang, Junge Zhang, Kaiqi Huang. Yizhou Wang, and Yizhou Yu. Mvp-net: multi- view fpn with position-aware attention for deep universal lesion detection. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2019: 22nd International Conference, Shenchen, China, October 13-17, 2019, Proceedings, Part VI 22, pp. 13-21. Springer, 2019.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial diffcrential equations. arXiv preprint arXiv:2010.00 2020a.
Zongyi LI, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprini 0rXiv:2003.03485 2020b.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhat- tacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differ- ential equations. Advances in Neural Information Processing Systems, 33:6755
66665556666,,, 6766, 2020c.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.037994 2021.
Hongxin Lin, Zelin Xiao, Yang Tan, Hongyang Chao, and Shengyong Ding. Justlookup: One millisecond deep feature extraction for point clouds by lookup tables. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 326-331. IEEE, 2019.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxed enn for efficient 3d deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for iden- tifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv: 190003900.03/9// 2019.
11

---

Published as a conference paper at ICLR 2024
Xingcheng Lu, Dehao Yuan, Yiang Chen, Jimmy CH Fung, Wenkai Li, and Alexis KH Lau. Estima- tions of long-term nss- and no3-wet depositions over east asia by use of ensemble machine- learning method. Environmental Science & Technology, 54(18):11118
126, 2020.
Xingcheng Lu, Dehao Yuan, Yiang Chen, and Jimmy CH Fung. Impacts of urbanization and long- term meteorological variations global pm2. 5 and its associated health burden. Environmental Pollution, 270: 16003, 2021.
Yanni Ma. Yulan Guo, Hao Liu, Yinjie Lei, and Gongjian Wen. Global context reasoning for se- mantic segmentation of 3d point clouds. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2931-2940, 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017a.
Charles Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Imvotenet: Boosting 3d object detection in point clouds with image votes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4404-4413, 2020.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Peter Renn, Cong Wang, Sahin Lale, Zongyi Li, Anima Anandkumar, and Morteza Gharib. Fore- casting subcritical cylinder wakes with fourier neural operators. arXiv preprint arX1v:2301.082900 2023.
Salih Salih. Fourier Transform: Signal Processing. BoD-Books on Demand, 2012.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7475 2020.
Chau Hsing Su and Clifford S Gardner. Korteweg-de vries equation and generalizations, iii deriva- tion of the korteweg-de vries equation and burgers equation. Journal of Mathematical Physics, 10(3):536-53, 1969.
MR Tek. Development of generalized darey equation. Journal of Petroleum Technology, 9006): 45-47, 1957.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Fe- lipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher Pal. Deep complex networks. arXiv preprint (tti:1170.09917 2017.
Gege Wen, Zongyi Li, Qirui Long, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. Real-time high-resolution co 2 geological storage prediction using nested fourier neural operators. Energy & Environmental Science, 173-)1173 2023.
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5589-5598, 2020.
Jiancheng Yang, Qiang Zhang, Binghing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3323-3332. 2019. 2019.
Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11040- 10048, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh. Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2024
Hengshuang Zhao, Li Jiang, Chi- Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighborhood features for point cloud processing. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5565-5573, 2019.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A NOTATION TABLE
Table Table of Notation
Notations of Continuous Functions
F
family of c-Lipschitz continuous functions
c
Lipschitz constant of the continuous function imput space, bounded domain of the function output space, bounded range of the function distance metrics in the input space X distance metrics in the output space Y collection of function samples.
X
Y
dx
dy
((x,, f(x))}
m
dimensionality of the input space X.
Notations HDFE
60
receptive field of HDFE.
F
N-ddimensional complex vector. Encoding of the explicit function f.
F /=0
N-dimensional complex vector. Encoding of the implicit function f(x) =
N
dimensionality of the function encoding. the input and output mapping from X, Y to the encoding space CN. binding operation.
Ex, Ey
0
unbinding operation.
(,)
cosine similarity between two complex vectors.
B POINTNET AS FUNCTION ENCODER
In this section, present empirical evidence highlighting PointNet's limitations in generating de- codable function encodings. Our designed pipeline, following the popular style of training recon- struction networks, demonstrates PointNet's inability produce even reasonable reconstructions. Consequently, we infer that without substantial modifications, PointNet unlikely to acquire the necessary capability for effective function encoding.
Specifically, we generate random functions by
1 2
f(x)
sin
k=1
where as Uniform(O.L 1) are the parameters controlling the generation and f(x) (0, 1). We 15000
randomly sample {x,, Yi 15500 where = f(x:) + with variance le-4. Then we stack x; and y/s into (5000, matrix and feed it a standard PointNet to generate a dimensiial vector. After encoding the function samples, we introduce a decoder that retrieves the function values from the encoding when receiving function inputs. We optimize the PointNet encoder and the decoder by minimizing the reconstruction loss. The procedure can be summarized as:
Uniform(O, 1) and e; is white noise
F PointNee(((aaa (((r,,,)))))))
V Decoder(F, x.), L= Σ
1 - -
The decoder is designed as followed: x; is lifted to a 1024-dimaniional vector X; through sequential layers of I.i near (1, 64), near (64, 128), near (128, 1024) with BatchNore rm and
14

---

Published as a conference paper at ICLR 2024
ReLU inserted between the linear layers. The retrieved function value is computed by the dot product between F and &, Figure (Left) plots the training curve and Figure 4 (Right) visualizes the reconstruction, which shows PointNet fails to learn decodable representation.
honthet tranning
fundton sumples
0.6
0.65
0.5 0.-
Meen Absolute
0.35
0.35
Epochs
0.0
0.2
0.4
0.6
0.8
1.0
Figure 4: PointNet fails to learn an effective function encoder. Left: Training curve of the PointNet function encoder. The enror is still significantly higher than HDFE though training for 700 epochs. Right: The PointNet function encoder does not produce a reasonable reconstruction.
C VECTOR FUNCTION ARCHITECTURE
In this section, we will give a brief introduction of VFA and its relation and comparison with HDFE. Vector Function Architecture (VFA) A function of the form
f(x) = Σ Og K(x,
represented by F En a2 E(xx), where the kernel K (:)-) and the encoder E(-) satisfy K (z, v) = (E(x), E(y)). Given the function encoding F, the function value at a query point To can be retrieved by f(zo) = (F. F(xo)).
VFA satisfies the four desired properties: VFA can encode an explicit function into a fixed-length vector, where the encoding sample invariant and decodable. Besides, the function encoding will preserve the overall similarity of the functions: (F, G) = Sz (x)) (f) dæ. Note that when encoding multiple functions, the kernel K (:.)) and the encoder F.)) must the same. Otherwise, the function encoding generated by VFA will be meaningless.
However, it strong assumption that function can be approximated by equation 7 In other words, under a fixed selection of K(,-) :) and E(.). equation 7 can only approximate small set of functions. There are large set of functions that cannot be approximated by equa- tion 7 In Fig. 5 we show failure case of VFA. We generate a function f(z) 0.1 El 04 K(,,
25
gucnd
_k=1
where Fa ~ N (0, 1), dg ~ fommor 1). K(I,y)
expl-20(x - = an RBF kernel. We generate 1000 function samples and add white noise to the output values form the training set. The experiment shows that VFA fails to reconstruct the function but HDFE succeeds.
In this paper, HDFE applicable for all e-Lipchitz func- tion, without compromising the four desired properties. Thanks to the improvement, we can apply function en- coding real-world applications. The first application.
Figure 5: VFA fails reconstruct the function but HDFE succeeds.
PDE solving, cannot be solved by VFA because the input and output functions cannot be approxi- mated by equation 7 The second application, local geometry prediction, has to deal with implicit function encoding, which cannot be solved by VEA either because VFA only encodes explicit func- tions.
15

---

Published as a conference paper at ICLR 2024
D SUITABLE INPUT TYPES FOR HDFE
We mentioned HDFE can encode Lipschitz functions. this section, we will discuss several data types that exhibits Lipschitz continuity, making them well-suited as inputs for HDFE. This enu- meration is not exhaustive of all potential suitable inputs for HDFE. Our objective is to furnish a conceptual understanding of the characteristics that define a suitable input, thereby guiding future considerations in this domain.
Point Cloud Local Geometry In many 3D vision problems, point cloud local features are important, such as point cloud registration (Huang et al. 202T). scene matching (Han et al. (203).. The local geometry around a point in a point cloud is usually a continuous surface, which inherently exhibits Lipschitz continuity. This characteristic makes HDFE suitable for encoding this type of data.
Meteorological Measurements the field of meteorology, machine learning plays a pivotal role in various applications, including pollutant estimation (Lu et al. (020) and weather forecasting (Chen et al. 2022). A standard practice in these applications involves inputting meteorological measurements from neighboring areas (for instance, windows of 100 km²) into machine learning models. HDFE a suitable tool for encoding these neighboring metcorological measurements, primarily due the inherent physical constraints that ensure these measurements do not rapidly change over time and space.
Time Series Data: Some time series data, especially in fields like event logging (Chen et al.) 2021 or network traffic (Abbasi et al., (021). are sparse in nature. Such sparse events can be treated as implicit functions and therefore HDFE is well-suited for it.
E DERIVATION OF DECODING
In this section, complete the details why F ( Ex can be decomposed into the summation
of Fyy (f(oo) and noises. This is an immediate outcome from the similarity preserving property. Since the unbinding
When dx (r0..2 x;)) is large. (Ex (Ex(xx) & Ey (f(o)))) (x,) & Ey (fxx))
operation is similarity preserving, we have (f((((z)) Ex & By (f(z:)) Ø Ex Ex((x)) & O. Therefore, O Ex (2x) can be decomposed into Ey (Fro) and the sum of vectors that are orthog- onal to it. Consequently. when we search for the y to maximize the (F Ø Ex (zu), Ey (v)), those orthogonal vectors will not bias the optimization.
F GRADIENT DESCENT FOR DECODING FUNCTION ENCODING
In equation 3 HDFE decodes the function encoding by similarity maximization. Given the func- tion encoding F E CN, and query point I'n E X, the function value do is reconstructed by:
yo = argmax,ex yEY ( 2 (xo), (y))
When Ex and Ey are chosen the fractional power encoding (equation 6) the optimization can be solved by gradient descent. In this section. we detail the gradient descent formulation. We assume the output space
By setting Ey as the fractional power encoding, the optimization can be rewritten as:
jo (0(0.) (F @ Ex exp(iVy) exppeii 0
where N is a random fixed vector where all elements are drawn from the normal distribution. Since F @ Ex Ex(x)) is constant vector, the optimization can be further simplified as:
Go argmaxyye(..1 (z, exp(iWg))
where F 2 (x( CN We write into its polar form:
z "0, a2e [x]]
In equation
6 (x) exp(i Since 33 and n two constant numbers, we rewrite B as V for
notation purpose.
16

---

Published as a conference paper at ICLR 2024
where ag [0, +00) and 0g (0,2x), We first simplify equation 8 with respect to y.
and then compute its gradient
1
(z, exp(iVy))
Z exp(iVyy)1
N2
age t0 2 ( 12x
N2
N
apage -00) -v)0
N2
N
apan cos [0, - -0g) - -V - va)yy
N2
p=
The gradient can be computed easily by taking the derivative:
d zz, exp(i = N2 Q - Vg)) sin [00p 0g) - -V - Wayyl dy
N
(9)
In practice, N can be large number like 8000. Computing the gradient by equation can be expensive. Fortunately, since is a random fixed vector, where all elements are independent of each other, unbiasedly estimate the gradient by sampling small number of entries in the vector. The decoding can be summarized by the pseudo-code below.
Algorithm 2 Gradient Descent for Decoding Function Encoding
Input: F, Tu. Ex, W
F is the function encoding. zo is the query point. Ex is the input mapping. v is the parameter in Ey-
z O Ex(o))
z = "0, a2e=">..... "0x
> Convert z into its polar form.
while (z, exp(iVy)) still increases do
Randomly select subset of [0, N) n Z. Denote S.
b We choose |S| 500.
= S 5|- Ep apa (V V Wy) sin [0pp 0g) (V, -u,-u)y1 2/ f y - a
D a is the learning rate.
end while
G RELATION BETWEEN FPE AND RBF KERNEL
In the main paper, we mention that we adopt fractional power encoding (FPE) as the mapping for the input and output space. In this section, explain the rational behind our choice by revealing its close relationship between the radial basis function (RBF) kernel. For explanation purpose, we discuss the single-variable case. is strwight-forward to generalize multi-variable scenarios.
RBF kernel is a similarity measure defined as K (z, y) exp ( (- 7 y)²). The feature space of
the kernel has infinite number of dimensions:
exp (->(x- - 2)
22 (x: y) (6(x),
(2k)
where (xx) = 7
1,
We will then show theorem that tells that FPE maps real numbers into finite complex vectors, where the similarity between the vectors can approximate a heavy- RBF kernel. However, the feature space of the heavy-tailed RBF kernel still has an infinite number of dimensions and therefore inherits all the blessings the RBF kernel. Notably, FPE provides a finite encoding, while approximating an infinite dimensional feature space.
17

---

Published as a conference paper at ICLR 2024
Theorem 3. Let 2, R. E(x) e where θ E RN and 4o E 0,0 NOO,, 2). then as
N + 0C,
(-1)
EEl)) (E e
28 y) (0(x),
(2k)!!
where (())= = -x
1, V T.
22 V
27
x
3
Proof.
(e e =
1
exp((>
exp(-ix00u)
N2
1
N²
N
1
N
p=1
Σ
COS
[T(0p - +g)((0 y)]
N
N
p#1
1) [r(ep 0g) y)} 2k
N
(2k)
pPg k=
(0p 22
0g)ah
72th
(a y)2k
N
N
(21:)
R=0
2k
(x y)2k
(21) 1)1!
(2k))
k=0
2x y)
(2k)
k=
Since 8p. 0g ~ ~N(0,, 2), 0p - 0g ~ (O, 1), and therefore, E[(e, - -0,)2a] = (2n 1)!!. So
Eppt , -0,))an converges to (2n - 1)!!. Nº
H PROOF OF THE HDFE'S PROPERTIES
H.1 ASYMPTOTIC SAMPLE INVARIANCE
In the main paper, claim that the iterative refinement (Algorithm[] will converge the center of the smallest ball containing all the sample encodings and therefore, HDFE leads to an asymptotic sample invariant representation. In this section, we detail the proof of the argument. To facilitate the understanding. Figure 6 sketches the proof from a high-level viewpoint. Recall the definition of asymptotic sample invariance (definition
Definition (Asymptotic Sample Invariance). Let f X - Y be the function to be encoded,
X (0, 1) be probability density function (pdf) on X, {x.)² p(X) be n independent samples of X. Let Fn, be the representation computed from the samples {xa, [(x)) (t asymptotic sample invariance implies F, n converges limit F o0 independent of the pdf p.
Proof. We begin by showing the iterative refinement converges to
F argmax|==||=1 min (2, E(z;, f(x:))
(10)
where fx)) is defined at equation in the original paper. maps a function sample to a high-dimensional space C..
18

---

Published as a conference paper at ICLR 2024
40th : : = U B ((a))
: = U.ex Biz,, f(x1))
Ball(S,))
Ball(S)
Figure 6: Proof of asymptotic sample invariance (overview). Ball(S) and Ball(Sn) the smallest ball containing S and Sn- As + oc, the Hausdorff distance between the two balls goes to zero with probability one. From elementary geometry, |ccette( (S,) center (Bll((S)) S d (S,ll(S,), Ball(S)). So the distance between the centers of the two balls goes to O.
To show the convergence the iterative refinement, follows from the gradient descent: since Vi- min,(2, E(x, f(x:))] -arggmnn
the gradient descent is formulated 2 + a argmin; (z, E (x,)))) which aligns with the iterative refinement in the paper.
Then will prove equation 10 produces sample invariant encoding by proving Fn converges to
F argmax |=||=|
min (2, Exx, (x)))
(11)
aex
Throughout the proof, the following definitions:
S
CN
A
Usex E(z, f(r) U B(xi, (as))
S, CCC
|1
1.2-norm of a complex vector.
dæ
maX minpe ||p all. Hausdorff distance between two compact
sets P C Q C..
Ball(P)
the smallest solid ball that contains the compact set P. center of the ball.
center(Bar center(Ball(.)
Recall from equation
6 E(r,y) v) F (c(((+40))))) so ||((r, y)1l = for all and y. Since
|| - =2ll2 = || 221|" 2 2211,, 22), we have (z, E(x, R)) = 1 - - Exx, w)1l if || 1.
Therefore, equation 10 equivalent to
F n argmin ||||||
mäx ||> E(x,, (x:))] i=1
(12)
Note that equation 12 implies F,, is the center of the smallest ball containing S,:
Fn center(Ball(S.,
(13)
because we were to construct balls containing S, with center F' + F,,, the radius of the ball must be larger than the radius of (S,)
When n + oo, the Hausdorff distance between Ball(S,) and Ball(S) goes to 0 with probabil- ity one. First, it is easy to see that dæ (Ball(S,), Ball(S)) is decreasing sequence and is positive, so the limit exists. Assume the limit is strictly positive, then there exists a point p' E such that minges. llp' qll for constant > O
co. This means sample is drawn from the ball (p'). This is contradictory to the definition of p X > (0, 1): p is positive over the input space X.
Finally, we conclude by |[E, Foell & du (Sa)l(Sa), Ball(S)). Since Ball(S,) C Ball( from elementary geometry, if A B are balls, then |ceeter(( B) center(A)) & radius B) radius(AA) < du( B, A). Therefore, we have ||center(Ball (Ball(S.)) senter(Ball(S du (S,),(S,), Ball()). Therefore, FF, Fooll S dnrBaal(Ss), Bal())),, which decays to 0 as n. + 00.
19

---

Published as a conference paper at ICLR 2024
H.2 ISOMETRY
In this section, we complete the proof that HDFE is an isometry.
Theorem. Let f.g: X - Y be both c- c-Lipschitz continuous, then their L2-distance is preserved in the encoding. In other words, HDFE is an isometry:
Ilf - gllza = -S.e [f(z) - x
d
Lemma 4. (x: 2 y, & 2) (y. 2).
Proof. Let xxx,, y = e eiz
(x 2 y, 2 = (e((x)))) (
= (xxxx)
(etY,e" e²2)
= 2)
Proof.
(F. G) = / (Ex(x) (f())) Fx(x)) (x(d) e Fy(g(d)))
(Ex(x) (x()) Er(g(<")))
(Ex(x) Er Ex (x') @ Ey Er (r(g(z))) da
(Ex(t) Ex @ Ey (o(z))) dr + noise
Bx
by Lemma
(-1) (2k)!!
32h (f(x) g(x))2 dr
by Theorem
3
2 a / f(x) - 22z by taking the first and second order terms
The second line holds because when dx (x x') is larger than the receptive field ep, Ex (x: and Ex(x') will be orthogonal, so the similarity between Ex (x) @ (y(((y) and Ex (x(xx))))))))))) Fy (v))) will be close to zero and they will be summed as noise.
I EMPIRICAL EXPERIMENT OF HDFE
In this section, we verify the properties claimed in Sec. 2.4 with empirical experiments.
I.1 SAMPLE INVARIANCE
In Fig. 7 we demonstrate that the function encoding produced by HDFE remains invariant of both the sample distribution and sample density. Specifically, we sample function values from three distinct input space distributions, namely left-skewed, right-skewed, and uniform distribution, each with sample sizes of either 5000 or 1000. We then calculate the similarity between the function vectors generated from these six sets of function samples. Before tuning the function vectors, the representation is influenced by the sample distributions (Fig. 7 Mid). However, after the tuning process, the function vector becomes immune the sample distribution (Fig. Right).
20

---

Published as a conference paper at ICLR 2024
Probability Density Function
rgnntskkwedd urifarm distrbbtioon
0.99
1.0
0.99
Uniorm
0.79
0.79
1.0
Untorm
St
0.99
0.99
1.0
0.98
0.99
0.89
0.89
1.0
Fight Talld
Right Tolle
0.50
0.51
0.86
0.85
0.90
1.0
0.98
0.98
0.98
0.98
0.99
1.0
1k
3k
1k
1K
0.
0.25
0.5
0.75
3.0
Let-Taled
Unioom
Fight-Talled
LehtTalled
Uniorm
Right Tailed
Figure 7: HDFE is invariant of sample distribution and sample size. Left: Three distributions where the function samples are drawn from. For each distribution, the sample size is either 5000 or 1000. Mid, Right: Similarity among the function vectors generated by the six sets of function samples, before and after the function vector tuning process, respectively.
1.2 ISOMETRY
In Fig. 8 generate pairs of random functions and compute their function encodings through
HDFE. We plot the L2-distance between the functions and the similarity between their encodings. We discover strong correlation between them. This coincides the isometric property claimed in Theorem 2
similarity between function vectors
0.9
0.6
distance between functions
Figure 8: HDFE is a distance preserving transformation. The L2-distance between functions is proportional the negative similarity between their encodings.
1.3 PRACTICAL CONSIDERATION OF ITERATIVE REFINEMENT
In Algorithm| we propose one implementation of iterative refinement, which iteratively adds the sample encoding that has the minimum similarity with the function encoding. This implementation conservative implementation that guarantees asymptotic sample invariance. However, in practi- cal applications, strict sample invariance may not be necessary. For example, achieving similarity threshold of 0.99 when constructing with different samples might not be required. This relaxation is viable because the downstream neural network possesses an inherent ability to handle some level of inconsistency. Therefore, we consider a practical adaptation of Algorithm by introducing slight modifications to accommodate these real-world considerations.
One-Shot Refinement Algorithm I the motivation of the iterative refinement is to balance the weights between dense and sparse samples. By iterative refinement, adjust the function encoding so that the sparse samples also contribute to the encoding. Such motivation can be achieved by another cheaper one-shot refinement. After obtaining the initial function encoding by averaging the sample encoding, we compute the similarity between this initial encoding and all the sample rough estimation of the sample density at partieular point.
encodings. The similarity
21

---

Published as a conference paper at ICLR 2024
Therefore, if we to balance the weights between dense and sparse samples, simply recompute the weights by the inverse estimated density. Algorithm illustrates the procedure.
Algorithm One-Shot Refinement
2, t (x;) & (f(()=)) for all i.
F Eizi
for do
w; (F, 2
wy is an estimation of the density (z,, f (ax)).
Wi = max E, wi)
Ensure numerical stability. b Compute inverse density.
W; = wi- 1 W
end for
F
Σ W1 24
Although one-shot refinement is not strictly sample distribution invariant, it is a quite good approximation of the sample invariant function encoding and is very cheap compute. We perform comparison among no refinement, one-shot refinement and iterative refinement with syn- thetic data, where we encode the same function sampled with two different distributions and com- pute the similarity between the two encodings. Specifically, we generate a random function by
1 (x)
(14)
k=1
where ag Uni Uniforml (0r 1) are the parameters controlling the generation and f(x) 1). We construct the encoding of the function by samples from two different sample distributions. The first distribution is computed by I; Uniform(O, 1) and computed by H; ~ Finffr left-tailed, and the second distribution is right-tailed. Then compare the similarity of function encodings generated by no iterative refinement, one-shot refinement, and iterative refinement. Figure shows the comparison, which demonstrates that one-shot refinement a quite good approximation of the sample invariant function encoding (the similarity increases from ~ 0.5 to 0.98 after one-shot refinement). In Appendix [.4 Table 3 we also compare the effectiveness of the three refinement schemes in a synthetic regression problem.
x². The second distribution and V; + 1 x²- Consequently, the first distribution is
Similarity between Function Encodings
Iteratvee refinemmnnt
refinement
ane-shot refinement
0.6-
0.-
0.5
100
175
Number of Iterations
Figure 9: When encoding the same function under two different sample distributions, one-shot refinement approximate the sample invariant function encoding well. It takes 75/90/1500 ms to encode 5000 samples on a CPU, and 7.5/8.0/25. ms on an NVIDIA Titan-X GPU when performing no refinement/one-shee refinement/200-steg iterative refinement.
1.4 EFFECTIVENESS OF SAMPLE INVARIANCE
In this section, examine the effectiveness of HDFE's sample invariance property through an synthetic function regression problem. We first generate random functions by equation [4] where ag Uni 1) are the parameters controlling the generation. The task is regress the
22

---

Published as a conference paper at ICLR 2024
Table 3:
Performance of function parameters regression. PointNet fails when sample distribution
varies between training and testing phases, while HDFE is robust to the sample distribution variation.
HDFE
PointNet
No Refinement
One-Shot Refinement 200-Step Iter. Ref.
MSE n
0.0037
0,0005
< 0.0000
< 0.0000
No Distr. Var.
0.978
> 0,9975
> 0.9975
> 0.9975
MSE
0.0717
0,003
< 0,00005
0.001
Distr. Var.
0.513
0.982
0.9975
0.992
coefficients [d1, 02, a3, 01] from the funetion samples { [(xa))) Regarding the sample size, the number of function samples is 5000 in the training phase and 2500 in the testing phase. Regarding sample distribution, we consider two different settings:
The sample distribution is consistent between train- ing phase and testing phase. We let I; ~ (ni 1) in both training phase and testing phase.
Setting 1 (No Sample Distribution Variation):
Setting 2 (Sample Distribution Variation): The sample distribution is different between the train- ing phase and testing phase. Specifically, in the training phase, we let I; Uni form(0, 1) and ar +- x². In the testing phase, let I; Uni /orm(0, 1) and æ,; + x Consequently, the sample distribution in the training phase is left-tailed, while in the testing phase is right-tailed.
We compare our HDFE with PointNet in terms of mean squared error (MSE) and the R-squared (Rt) metrics. For HDFE, we compare the performance among no refinement, one-shot refinement (introduced in Appendix L.3 and 200-step iterative refinement. Table 3 shows the comparison.
In Setting 1, when there is no distribution variation, HDFE achieves significantly lower error than PointNe. This is because HDFE capable of capturing the entire distribution of functions, while PointNet seems to struggle on that.
In Setting when there is distribution variation, PointNet fails miserably, while HDFE, even without iterative refinement, already achieves fairly good estimation, and even better than the PointNet in Setting In addition, the experiment also shows that both the one-shot refinement and the iterative refinement are effective techniques to improve the robustness to distribution variation.
1.5 INFORMATION LOss OF HDFE
In this section, we analyze the information loss when encoding continuous objects with HDFE. It intuitive that larger encoding dimensionality induces smaller information loss, and encoding function changing more rapidly induces larger information loss. We attempt to quantify the relation through empirical experiments. We generate random functions and we measure the "function com- plexity" by the integral the absolute gradient: complexity(f) = So (z) Consequently, functions changing more rapidly yield a higher complexity(f):
We study the relation between the reconstruction mean absolute error (MAE) R- R-squared (R²) and the function complexity under different encoding dimensions. Figure Figure[O revealss the MAE exhibits a linear relation with the input function complexity, while the R-squared seems not to be affected by the function complexity when the dimension is large enough.
dtme530
akm-1000
Mean Absolute Reconstruction
zimw2200
0.090
0.002
0.00
cinea20 48n-2500
0.005
Complexity of Input
Complexity Input Function
Figure 10: Empirical information loss when encoding functions of different complexities.
23

---

Published as a conference paper at ICLR 2024
1.6 LOW-RANK HIGH-DIMENSIONAL SCENARIOS
We generate random functions by first randomizing I& E R and ag R, the random function is constructed by:
f(x) = Σ ak K K( 24) k=1
where n can measure the complexity of the function, and d is the dimension of the function input. The testing samples are generated by z& + noise for all & E [n]]
In Fig. the reconstruction error increases n. increases, which indicates the encoding quality negatively correlated to the complexity of the function. However, the reconstruction error does not change as d increases, which indicates that the encoding quality does not depend on the explicit dimension of the function input.
This empirical experiment shows that HDFE has the potential to operate on high-dimensional data, because the encoding quality of HDFE does not depend on the dimension the function input, but only depends on the complexity of the function.
Broee
0.00
000
Figure 11: The reconstruction error of HDFE is negatively correlated the complexity of the func- tion, but does not depend on the dimension of the function input.
J EXPERIMENT DETAILS
J.1 PDE SOLVER
The PDE and the solution are encoded into two embedding vectors with length N. A deep complex network is trained to learn the mapping between two vectors. The ar- chitecture is a sequence of layers: [Complexiines]
(N, 256), ComplexReI. (), ComplexLinear (256, 256), ComplexReLU () ComplexLinear (256, 256), CompleXReLU (), ComplexLinear (256,M)) N) J. The network trained with Adam optimizer with learning rate of 0.001 for 20,000 iterations. The value in equation 6 is 15, 25, 42, 45 for N 4000, 8000, 16000, 24000 and the 3 value 2.5.
J.2 SURFACE NORMAL ESTIMATION
The architecture is sequence of layers: [ComplexLinear (N, 256),
ComplexRatchNoem () ComplexReLU (), Complextas near (256, 256),
ComplexBatcchNomm)) (), ComplexReLU (), ComplexLinears (256, 128),
ComplexBatcchNozm (), ComplexReLU 1. After the sequence layers, will pro- duce 128-dimensions complex vector Since desire 3-dimensional real vector output (normal vector in R²), we use two RealLinear (128,3) layers L.real and L.imag. The final output normal vector is L.real (z. real) L.imag (2 imag). . The network is trained with Adam optimizer with learning rate 0.001 for 270 epochs. The a chosen as 20 and the dimensionality is chosen 4096.
24

---

Published as a conference paper at ICLR 2024
J.3 ADDING HDFE MODULE TO HSURF-NET
Denote the input local patch as P with shape (B, N,3,, where B is the batch size, N is the number of points in local patch. HSurf-Net
their novel space transformation module to extract n keypoints and their K nearest neighbors. The resulting data is denoted as P_sub with shape (B, n, K, 3). HSurf-Net uses a PointNet to process P_sub, by first lifting the dimension to (B, n. K, C) and then doing a maxpooling to shape the data into (B. n. C.). We add our HDFE module here: we use HDFE to shape the data from (B, n, K,3) to (B, C), by first lifting the dimension from (B, n. 3,3) to (B, K, 512) using equation 6 and then average the embedding across the neighbors to shape it into (B, n, 512). Then we use a fully-connected layer to map the data into (B, n, C), which becomes the HDFE feature. Then sum the features generated by HSurf-Net and HDFE into a (B, n, C* matrix and pass to the output layer as HSurf-Net does.
K ABLATION STUDIES OF SURFACE NORMAL ESTIMATION
Table 4: Ablation Studies on the PCPNet dataset.
Dimension
2048
4096
Naise Level
Nome
Low
Med
High
Stripe
Gradient
Average
None
Low
Med
Hligh
Gradient 9.87
Average
10
8.98
10.80
17.40
22.18
10.62
9.92
13.32
9.00
10.60
17.40
22.48
10.50
13.31
15
8.98
11:02
17.73
22.72
10.85
9.46
13.46
8.29
10.77
17.46
22.63
10.17
9.10
13.07
9.93
10.72
25
8.86
11.43
17.94
22.85
10.42
9.33
13:47
8.40
11:23
17.66
22.
9.89
8.95
13.15
Table 5: Ablation Studies on the FamousShape dataset.
Dimension Naise Level
2048
4096
Nome
Med
High
Stripe
Gradient
Average
None
Low
Med
Hlgh
Gradient
10
13:12
18.43
30.86
38.06
13.52
13.82
22.07
13.05
18.15
30.61
38.50
16.31
13.71
22.14
15
14.42
17.97
30.45
38.92
15.47
13.81
21.84
13.68
17.77
31.17
38.79
14.83
13.06
21.55
31.40
17.99
38.57
0 25
14.70
19.71
31.52
38.95
15.04
13.56
22.25
14.03
19.16
31.39
38.65
14.38
13.22
21.81
In general, when the dimensionality is higher, the error is lower. When the receptive field is large (a is small), HDFE performs better when the noise level is high. When the receptive field is small (a large), HDFE performs better when the noise level is low. This coincides with the analysis in Fig. 2 A large receptive field tends to filter out the perturbations and therefore is more robust to noise. small receptive field can capture the figh-frequency details and therefore is more accurate.
L NOISE ROBUSTNESS
In this section, we further analyze why HDFE is robust to point perturbations. For visualization purposes, perform the analysis on 2d data, while the analysis generalizes well to higher dimen- sions. We randomly sample 1000 points from the unit circle x² + y² = 1 and add Gaussian noises the samples. Then we encode the points into vectors using HDFE with different receptive fields (a = 10, 15, 20, 25) and reconstruct the implicit function. The pseudo-color plot in Fig. 12 visual- izes the likelihood that a point lies on the unit circle.
When a is small, the visualization shows that the reconstructions under different noise levels are similar, which tells that the encodings under different noise levels are similar. So demonstrates a robustness to point perturbations. On the other hand, when is large, the reconstruction is more refined and captures the high-frequency details, but as price, it is more sensitive to the point perturbations.
25

---

Published as a conference paper at ICLR 2024
10
15
20
25
30
0.01
2² 0.05
0² 0.1
Figure 12: Reconstruction of implicit functions sampled with noisy inputs under different choices of receptive field.
26


Reviews:
[{"review_id": "fHogBLwig7", "reviewer": "Reviewer_oyrE", "summary": "This paper proposes Hyper-Dimensional Function Encoding (HDFE), which encodes a continuous object (eg, functions) into a fixed-size explicit vector representation without requiring training. While it maintains the benefits of vector function architecture (VFA), satisfying sample invariance and decodability, it relaxes the strict assumption on the function form in VFA into Lipschitz functions by introducing a novel iterative refinement process. While HDFE serves as a general interface for a continuous object encoder without training, substituting HDFE for domain-specific algorithms in experiments on mesh-grid data and sparse data shows comparable performance. \nThe main contributions of the papers are: \n\n(1) The authors propose a novel function encoding method that satisfies key properties of VFA while relaxing the strict assumption on function space to  Lipschitz continuity. \n\n(2) Theoretical foundations and empirical analysis support the validity of HDFE on key properties. \n\n(3) Experimental results confirm that replacing domain-specific algorithms with HDFE maintains competitive performance and robustness to the noise perturbation.", "questions": "- How long does the HDFE take compared to the baselines (eg, pointNet in Experiment 3.2)? Is the iterative refinement process applicable to a large number of samples? How long does it take for convergence in the process? \n\n- In the formulation on decoding, (i.e., equation between eq. (2) and eq.(3)), can you please clarify on why orthogonality property ensures that $E_X(x_i) ⊘ E_X(x_0) $ will produce a vector orthogonal to $E_X(x_0)$ when the distance between two samples is large? Also what does the noise mean? Does it mean that it’s near zero so that it is a negligible component?\n\n- In the formulation on decoding, (i.e., equation between eq. (2) and eq.(3)), it seems it misses $w_i$. \n\n- For an unbinding operation, element-wise division of complex vectors is used. But I don't think this operation is commutative, which violates the assumption. Can you please clarify on this? \n\n- In experiment 3.1, how does the function prediction error is measured? Is it measured in embedding space? And the paper states that “when decoding is not required, our approach achieves lower error than FNO”, but how can we compare to FNO, which directly predicts the solution? \n\n- While the authors claim that HDFE is robust to point perturbation, the experiments on  [PCPNet - PointNet + HDFE] in Table 1 shows that the performance boost becomes much less as the noise level increases. Can you please elaborate on this?\n\n- [Possible Typo] In the last sentence in section 2.1, “appendix F.1” should be “appendix E.1”.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow. \n\n- The formulation of the decodable encoder and iterative refinement process for sample invariance seems interesting and convincing. Also, theoretical analysis on each component is clear and supports the claims.\n\n- Despite the general and straightforward formulation, empirical results demonstrate the effectiveness of HDFE.", "weaknesses": "Method\n\n- One concern is the computational cost of HDFE induced by the iterative refinement process. In order to employ function representation for the downstream tasks, computational costs of HDFE is important. it may hinder application to large-scale tasks. \n\nExperiment\n\n- Overall, it’s convincing that HDFE is a reasonable and general interface for processing continuous objects, supported by the experiments. However, it’s less convincing why we should use HDFE instead of other domain-specific encoding methods. The authors claim that sample invariance is a crucial property for the machine learning tasks throughout the paper, but it lacks the supporting experiment revealing HDFE’s efficacy in those scenarios (i.e., sample distributions are different in training and test dataset). It would make the paper stronger if it presents the experiments with scenarios having disparate sample distributions between training and test datasets and compares the performance of HDFE compared to the baselines. \n\n- In the experiment section, it lacks the analysis why HDFE is more beneficial than the counterparts (e.g., PointNet) in terms of the performance. It would improve the understanding of HDFE if analysis on which component leads to the performance gap even when the noise is absent is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHNXVDcosf", "reviewer": "Reviewer_421T", "summary": "This submission propose a module, namely Hyper-Dimension Function Encoding (HDFE), to map a continuous object (data sample) into a fixed-dimension vector without any training.\nThe author asserts that the proposed approach possesses four key characteristics: (1) sample distribution invariance (2) sample size invariance (3) explicit representation (4) decodability.\nTo obtain the fixed-length vector representation, the input data for HDFE must adhere to Lipschitz continuity and will be transformed into a high-dimensional space, where a weighted average will be computed.", "questions": "1. In line 7 of page 2, why is the representation learned by PointNet (Qi et al., 2017a) not easily decodable? For instance, in their original paper (https://arxiv.org/pdf/1612.00593.pdf) in Figure 2, it seems possible to set m=3 and obtain normalized point clouds. Additionally, other works like [1] may also be able to 'decode' the input from the vector representation. Is there any difference between this manuscript and those works?\n\n2. When experimenting with batches, should the model visit all data samples to decide hyperparameters? According to Equation 3, the decoding step should visit all $\\bm Y$.\n\n3. By curious: why a high-dimensional input does not affect the size of the fixed-length vector representation $N$. Could the author provide further explanation, possibly an extension of the paragraph on 'Scale to high-dimensional input'?\n\n4. in Section 2.3, the manuscript only shows the picking of $E_X$ and $E_Y$ when the function output y is scalar. Are there more cases that can be considered?\n\n5. Although HDFE is a deterministic function, is there any empirical result available to estimate the information loss from the raw input to the fixed-length vector representation?\n\n\n\n[1] Learning Representations and Generative Models for 3D Point Clouds. PMLR", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The manuscript demonstrates excellent organization, a well-defined research problem, clear logic, and skillful writing.\n\n2. The topic holds significant importance: a method that can map data samples with varying distributions and sizes to fixed-length sequences may be highly appealing for pre-training models that utilize cross-domain data.\n\n3. The theory effectively connects with the experiment: the utilization of a weighted average operation has the potential to reduce noise effectively.", "weaknesses": "1. It is necessary to provide a clear definition of \"implicit representation\" and \"explicit representation\" in the manuscript. Some reviewers may intuitively refer to the fixed-length vector representation of a data sample as \"implicit representation\" since it may not be human-friendly. However, in this manuscript, the fixed-length vector representation is referred to as the \"explicit representation.\"\n2. The proposed method (HDFE) relies on the assumption that the input data follows Lipschitz continuity. While the reviewer agrees that point cloud data intuitively follows Lipschitz continuity, it would be beneficial for the manuscript to include an analysis of the types of input data that adhere to Lipschitz continuity.\n\n3. As a module that doesn't require any training, it is important to provide detailed guidance on selecting hyperparameters. This includes guidance on choosing the size of the fixed-length vector representation (denoted as $N$) and determining the hyperparameters $\\alpha$ and $\\beta$ in Equation 5, which are influenced by the receptive field $\\epsilon_0$ and the Lipschitz continuous constant $c$.\n\n4. 2The selection of weights ($w_i$ in Equation 1), hyperparameters ($\\alpha$ and $\\beta$), and the mapping functions $E_X$ and $E_Y$ are highly dependent on the dataset. This means that if the task or input data changes, all these variables need to be carefully decided and tested.\n\n5. There is a small concern regarding the experimental results on the PCPNet dataset. The proposed HDFE method is demonstrated to outperform the PCPNet model (the baseline in 2018) simply by replacing PointNet with HDFE. However, it is only comparable to the current state-of-the-art (SOTA) method, outperforming it in four out of twelve metrics, albeit with a slight drop in average performance. It would be valuable to provide insights or explanations for these observations and discuss any potential limitations or implications of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yMpq42Z4bw", "reviewer": "Reviewer_jCpW", "summary": "This paper proposes Hyper-Dimensional Function Encoding (HDFE), which does not require training and maps continuous objects for embedding space. The proposed method enables processing continuous objects. Experiments show that the proposed method can be plugged into and improve PointNet-based architectures.", "questions": "Can the proposed method be applied for any-resolution image encoder for complex natural images, e.g. ImageNet? What would the main challanges be for applying the method to the image domain?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Encoding continuous signals is an important research topic. The paper is clear and well-organized.\n2. The proposed method does not require any-training and can be plugged into existing structures, which makes it easy to apply in practice and could have wide applications.\n3. Evaluation is thorough and solid. The method shows advantages over various prior works, across different datasets and settings.", "weaknesses": "1. In Table 1, some metrics did not show improvement when comparing to the prior work HSurf-Net.\n2. The encoding capacity of the proposed method might be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)