Neural Fourier Transform: A General Approach to Equivariant Representation Learning
Authors: Masanori Koyama
Kenji Fukumizu
Kohei Hayashi
Takeru Miyato
Venue: ICLR 2024 poster (2024)
Keywords: Fourier transform
equivariance
harmonic analysis
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We present Neural Fourier Transform, a general framework of equivariance learning that does not assume any explicit knowledge of how the group acts on data.

Abstract: Symmetry learning has proven to be an effective approach for extracting the hidden structure of data, with the concept of equivariance relation playing the central role. 
However, most of the current studies are built on architectural theory and corresponding assumptions on the form of data. 
We propose Neural Fourier Transform (NFT), a general framework of learning the latent linear action of the group without assuming explicit knowledge of how the group acts on data.
We present the theoretical foundations of NFT and show that 
the existence of a linear equivariant feature, which has been assumed ubiquitously in equivariance learning, is equivalent to the existence of a group invariant kernel on the dataspace. 
We also provide experimental results to demonstrate the application of NFT in typical scenarios with varying levels of knowledge about the acting group.

Published as a conference paper at ICLR 2024
NEURAL FOURIER TRANSFORM: A GENERAL AP- PROACH TO EQUIVARIANT REPRESENTATION LEARN-
ING
Masanori Koyama' Kenji 2ukumizu Kohei Hayashi' Takeru Miyato Preferred Networks, Inc. 2he Institute of Statistical Mathematics UUniversity of Tübingen
ABSTRACT
Symmetry learning has proven to be an effective approach for extracting the hid- den structure of data, with the concept of equivariance relation playing the cen- tral role. However, most the current studies are built on architectural theory and corresponding assumptions Transform (NFT), a general framework of learning the latent linear action of the group without assuming explicit knowledge of how the group acts on data. We present the thcoretical foundations of NFT and show that the existence of a linear equivariant feature, which has been assumed ubiquitously in equivariance learn- ing, is equivalent to the existence a group invariant kernel on the dataspace. We also provide experimental results demonstrate the application of NFT in typical scenarios with varying levels of knowledge about the acting group.
the form of data. We propose Neural Fourier
INTRODUCTION
Various types of data admit symmetric structure explicitly or implicitly, and such symmetry is often formalized with action of group. As typical example, an RGB image can be regarded function defined on the set of 2D coordinates R² + and this image admits the standard shift and rotation the coordinates. Data of 3D object/scene accepts SO(3) action (Chen et al. 2021 Yu et al.) 2020), and molecular data accepts permutations (Raghunathan and Priyakumar 2021 as well. To leverage the symmetrical structure for various tasks, equivariant features are used in many applications in hope that such features extract essential information of data.
Fourier transform (FT) is one of the most classic tools in science that utilizes equivariance relation to investigate the symmetry in data. Originally, FT was developed to study the symmetry induced by the shift action a e f-) = f(.-fff... a) on a square-integraabl function to another function (f)) = E (RR) well known that FT satisfies (a (w) = e -iaw f(w) and hence the equivariant relation ( f) ei (f)). By this equivariant mapping, FT achieves the decomposition of L2 (R) into shift-equivariant subspaces (also called frequenciessirreddcibb representations). This idea has been extended the actions of general groups, and extensively studied as harmonic analysis on groups (Rudin) 1991). In recent studies of deep learning, group convolution (GC) is a popular approach to equivariance (Cohen and Welling 2017 Finzi et al., 2021 Cohen and Welling, 2014 2016)) /Willrr and Cesa (201)), and the theory of FT also provides its mathematical Foundation (Kondor and Trivedi 20188 Cohen al. 2018 2019).
(2R).. FT maps invertibbl
One practical limitation of FT and GC is that they can be applied only when we know how the group acts on the data. Moreover, FT and GC also assume that the group acts linearly on the constituent unit of imput (e.g. pixel). In many cases, however, the group action on the dataset may not be linear or explicit. For example, when the observation process involves an unknown nonlinear deformation such as fisheye transformation, the effect of the action on the observation is also intractable and nonlinear (Fig. [ left). Another such instance may occur for the 2D pictures of rotating 3D object rendered with some camera pose (Fig. I right). In both examples, any two consecutive frames are implicitly generated as (I. go.)) where is a group action of shiftirotation. They are clearly not linear transformations in the 2D pixel space. To learn the hidden equivariance relation describing the symmetry of data in wider situations, we must extend the equivariance learning and Fourier analysis to the cases in which the group action each data may be nonlinear implicit.

---

Published as a conference paper at ICLR 2024
Figure 1: Left: An image sequence produced by applying fisheye transformation after horizontal shifting. Right: 2D renderings of spinning chair.
Nonlinear
Encader (FonwrdNFT)
Decoder
(Invarse NFT)
Linear
Figure 2: NFT framework. Each block corresponds to irreducible represenntationiirqqqeneyy
To formally establish solution to this problem, propose Neural Fourier Transform (NFT), nonlinear extension of FT, as general framework for equivariant representation learning. We gen- eralize the approach explored in (Miyato al.] 2022) and provide novel theoretical foundation for the nonlinear learning equivariance. As an extension FT, the goal NFT is to express the data space the direct of linear equivariant spaces for nonlinear, analytically intractable actions. Given dataset consisting of short tuple/sequences (I1, ...)..) that are generated by an unknown group action, NFT conducts Fourier analysis that composed of (i) learning of an equiv- ariant latent feature on which the group action is linear, and (ii) the study of decomposed latent feature direct sum of action-equivarian subspaces, which correspond frequencies. Unlike the previous approaches to equivariance learning that rely on model architectures (Keller and Welling, 2021 Cohen and Welling 2017). the learning (i) NFT does not assume any analytically tractable knowledge of the group action in the observation space, and simply uses an autoencoder-type model infer the actions from the data. addition to the proposed framework of NFT, we detail our theoretical and empirical contributions follows.
1. We answer the essential theoretical questions of NET (Sec 4. In particular, Existence. We elucidate when
find linear equivariant latent features and hence when we can conduct spectral analysis on a generic dataset.
Uniqueness. We show that NFT associates nonlinear group action with of irreducible representations, assuring NFT's ability to find the unique set of the equivariant subspaces. finite-dimensionll approximation. We show that the autoencoder-type loss chooses set of irreducible representations in approximating the group action in the observation space.
2. We experimentally show that:
NFT conducts data-dependent, nonlinear spectral analysis. It compress the data under nonlinear deformation and favorably extract the dominant modes of symmetry (Sec 5.
By using knowledge about the group, NFT make inferences even when the action is not invertible in the observation space. For example, occlusion can be resolved (Sec 5.2).
By introducing prior knowledge of the irreducible representations, we can improve the out- of-domain (OOD) generalization ability of the features extracted by the encoder (Sec 5.2
PRELIMINARIES
In this paper, G is group, and its unit element. We say G acts X if there is map x X + X. (g. x) + g x, such that e x = x and (gh) or = go (h 2x) for any x X' and g, G. When G acts on X' and V. map & X - y is called equivariant if @(gex) go +(x))
2

---

Published as a conference paper at ICLR 2024
for any 9 E G. X. A group representation is defined linear action of group G on some vector space V. ie., a group homomorphism p G GL(). See also Appendix A for notations.
Fourier transform an equivariant map. We first explain the equivariance of the classic Discrete Fourier Transform (DFT) to motivate the model and learning of NFT. As is well known, DFT on the function space L2 := (f: : ZT - C). where Zy- {j/T E (0. J 0,..., T (mod 7) is the T grid points on the unit interval. DFT and its inverse (IDFT) are given by
1}
f== -
2m 2 ((/T),
SG/T) -े E 2 fu Vk,j 2T. (1)
== We can define the group of shifts == 2r (mod T)) acting on (f(i by m == (f(li-
m) j=0 T-1 With the notation (f) == fr. it is well known that 2 (m o f) = e 7 2 (f) for
all m, k E Zr, establishing DFT as an equivariant map; namely
((m f) ((f)
mo = Φ (D(m)
(2)
where D(m)= == Diag(e 2mi " ) is diagonal matrix. By definition, D(m) satisfies D(m++m') D(m) ((m'), meaning that G 3 m + D(m) E GL(C")) is group representation. Thus, the DFT map & is an equivariant linear encoding of L3 into the direct sum of eigen spaces (or the spaces that are invariant with respect to the shift actions), and & is the corresponding linear decoder.
In group representation theory, the diagonal element 2x of ((m) is known as an irreducible representation, which is a general notion of frequency. We shall therefore use the word frequency and the word irreducible representation interchangeably in this paper. A group representation of many groups can be decomposed into a direct sum of irredueible representations, or the finest unit of group invariant vector spaces. Generally, for a locally compact group G. the Fourier transform, the inversion formula, and the frequencies are all analogously defined (Rudin 1991). For the group action (g- (x) = fx) an equivariant formula analogous to eq. also holds with D(g) being a block diagonal matrix. Thus, the frequencies are not necessarily scalar-valued, but matrix-valuce
NEURAL FOURIER TRANSFORM
In NFT, we assume that group G acts on generic set X, and examples of (z, (o.g (r E EX can be observed as data. However, we do not know how the action o is given in X, and the action can only be inferred from the data tuples. As in FT, the basic framework of NFT involves an encoder
and a decoder, which are to be learned from a dataset to best satisfy the relations analogous to eq 2
( x) ((g (zx) and
T (f(x))
(Vz E X,, Vg E G),
(3)
where Mg) is some linear map dependent on g. which might be either known or unknown. It turns out that realizing 3 is enough to guarantee that M (g) is a group representation:
Lemma 3.1. If span 4 {X)} is equal to the entire latent space, then eq.(3) implies that M (g) is group representation, that is, (()) = Id and M (gg)- = M
The proof is given in Appendix B The encoder / and decoder V may be chosen without any restrictions on the architecture. In fact, we will experiment NFT with M.PPCCNNTTranfoomee whose architectures have no relation to the ground truth action.
Given pair ( V that satisfies eq. 3 also seek invertible linear map P to block-diagonalizz (g) unless we know M (g) a priori in a block-diagonal form. This corresponds to irreducible decomposition in representation theory. Assuming that the representation is completely reducible, we can seek a common change of basis matrix P for which B(g) = (M is block-diagonal for every g so that each block corresponds to an irreducible component of (g). See See A for the details on irreducible decomposition. Putting all together. NFT establishes the relation
gor To
(4)
We call the framework consisting of eqs. and 44) Neural Fourier Transform (NFT), where 2 Pœ(x) is the Fourier image of x, and (1P-1)) is the inverse Fourier image. See also Fig 2 for the visualization of the NFT framework. The classic DFT described in See 2 is instance of NET; X is RT. which is the function space on G = Zy with shift actions, and PC and v pPP11 1 are respectively DFT and IDFT (linear). (Miyato et al.) 2022) emerges as implementation of NET when (Ø, v is to be learned in a completely unsupervised manner with no prior knowledge of M (g) nor G itself. As we show next, however, NET can be conducted in other situations well.
3

---

Published as a conference paper at ICLR 2024
3.1 THREE TYPICAL SCENARIOS OF NFT
There can be various setting of data and prior knowledge of G, and accordingly various methods for obtaining pair (Φ, V) in eq. 3). However, we at least need dataset consisting of tuples of observations (e.g (z. g (x) from which we need to infer the effect of group action. A common strategy to optimize BIIll X (M (x))())))I where (g) is either estimated or provided, depending on the level of prior knowledge on or G.
Unsupervised NFT (U-NFT): Neither G nor g is known. In U-NFT, the dataset consists of tuples of observations (z == (x)) ..... .......7)) ) where each (0) is implicitly generated as xx(1) =
gi 020
(1) for some unobserved 9, sampled from G that is unknown. Such dataset may be obtained by collecting short consecutive frames of movies/time-serics for example. (Miyato et al. 2022) is
a method for U-NFT. MSP uses a dataset consisting of consecutive triplets (T 2), such as any
consecutive triplet of images in Fig | Given such a dataset, MSP trains (Φ, v) by minimizing
E| |22 - (*<x)))
where M = arg arg ([1) - (zo)))
(5)
is computed for each triplet æ (Fig 20, Appendix). By considering & with matrix output of dimen- sion _axmm _M* E _aax can be analytically solved as ((o))) where A is the Moore Penrose inverse of A (Inner optimization part). Thus (Φ. Y is trained in an end-to-end manner by minimiz- ing E|[ m2 V ( ( (2x)) (T (f(x1)) [1 For the U-NFT experiments, we used MSP as a method of choice. After training (Ø, V), we may obtain M for each xx() (say M,), and use (Maehara and Murota 2011 for example to search for P that simultaneously diagonalizes all M
G-supervised NFT (G-NFT): G' is known but not g. G-NFT has the same dataset assumption as U- NFT, but the user is allowed to know the group G from which each is sampled. In this case, we can assume that the matrix M (g) the latent space would be a direct sum of irreducible representations of G. For example, we may assume some parametric form of irreducible representations M (0) = BR (0) and estimate ((+) for every tuple of data (+()) However, estimating (()) for each i may not scale for a large dataset. Alternatively, we may just use the dimensional information of the matrix decomposition and estimate each block in the same manner as U-NFT. For instance. in the context of [ the user may know that the transformation between consecutive frames "some" periodic action (cyclic shift), for which it is guaranteed that the matrix representation is direct sum of 2 x 2 matrices. When T 2, we can minimize the same prediction loss as eq. (5. except that we put M = 0 Mt E Raxa where Mt arg m [ - 4x)()))) E 122x2 and 4 are the matrices constituting & by vertical concatenation +- [1; ....
g-supervised NET (g-NFT): g is known. In g-NFT, the user can observe set of (z. go x) for known g so that the data technically consists of ((x.gorr). setting not only allows the user to know that g is periodic, but also the velocity of the shift (e.g.. the size of the pixel-level shift before applying the fisheye transform). Thus, by deciding the irreducible representations to use in our approximation of the action, we can predetermine the explicit form of M (g). For g-NFT, we may train (4, )) by minimizing
In the context of Fig [ the g-NFT
Ellgg o -
E (g(=)) MM (z))(())11
The matrix (g) can be derived from the knowledge of representation theory. For example, if Σ is the group, M(g) corresponding to the frequencies (f1,--u) would be the direct sum of the 21) rotation matrices by angle 2T fegg/N (see also Fig. 21 in Appendix).
3.2 PROPERTIES OF NET
By realizing eq. approximately, NFT learns spectral information from the data and actions (Lemma 3.1 Here, we emphasize three practically important properties of NFT. First, by virtue of the nonlinear encoder and decoder, NFT achieves nonlinear spectral analysis for arbitrary data types. Second, NFT performs data-dependent spectral analysis; it provides decomposed represen- tations only through the frequencies necessary to describe the symmetry in the data. These two properties contrast with the standard FT, where the pre-fixed frequencies are used for expanding functions. Third, the NET framework has the flexibility to include spectral knowledge about the group in the latent space, as in G-NFT and NFT This further enables us to extract effective fea- tures for various tasks. These points will be verified through theory and experiments in the sequel.
4

---

Published as a conference paper at ICLR 2024
4 THEORY
Because the goal of NFT is to express the data space in the form of latent linear equivariant subspaces, is a fundamental question answer whether this goal is achievable theoretically. Additionally, it is important to ask if the linear action can be expressed by a unique set of irreducible representations. The following Thm 4.1 answers these questions in the affirmative using the notion of group invariant kernel. Let group G act on the space X. A positive definite kernel X X + IR called G-invariant kig o J. g gox') = k(r,z') for any a, x' E X and
Existence and uniqueness:
E G.
Theorem 4.1. (Existence, uniqueness, and invariant kernel) Let G be a compact group acting on
X. There exists a vector space V and non-trivial equivariant map $ - if and only if there exists a non-trivial G-invariant kernel on X. Moreover, the set of G-invariant kernels and the set of equivariant maps to a vector space are in one-to-one correspondence up to a G-isomorphism.
We provide the proof in Appendix C The implication of Thm is twofold. First, Thm 4.1 guar- antees the existence of an equivariant map & by the existence of an invariant kernel. In the proof, + emerges as an embedding into the associated reproducing kernel Hilbert space (RKHS). Thus, un- der mild conditions, there is a latent space that is capable of linearly representing sufficiently rich group action, since one can easily construct G-invariant kernel with dimenioadi RKHS which is dense in 1 space(TThm C.I.I)
Second, Thm implies the identifiability. When there are two invertible equivariant maps and c Thm 4.1 guarantees that there a pair of corresponding kernels, and we can induce from them a G-isomorphism (Sec(A) between the corresponding RKHSs. other words, & and & are connected via G-issomorphism, corresponding to the same set of irreducible representations. This way, may associate any linearlizable group action to a unique set of irreducible representations. Similarly, any two solutions for the g-NFT with the similar M (g)s would also differ only by G-isomorphism.
Finite dimensional Approximation: When the output dimension of & small, the space may not fit all members of irreducible representations corresponding to the target group action and it may not be possible to invertibly encode the action and observations as done in the standard FT. In such a case, the expression x1 (M (xx1)
(z)) )) in NFT would only provide approximation of the transition 21 + 22. However, what type of approximation would be in U-NFT and G-NFT? Below will present the claim guaranteeing that NFT conducts data-dependent filter that selec- tively extracts the dominant spectral information in the dataset describing the action. We present an informal statement here (see See C.3.2 for the formal version).
Theorem 4.2. (Nonlinear Fourier Filter) Suppose that there exists invariant kernel for the group action on X whose RKHS is large enough to invertibly encode X. Suppose further that (0", vr*) is the minimizer of
Egealllo -
among the set of all equivariant autoencoders of fixed latent dimension for which the actions are linear in the latent space. Then the frequencies to appear in the latent space of 4* are determined by the signal strength of each irreducible component in the RKHS of
This result claims that, in the application of NFT with small latent dimension, U-NFT and G-NFT automatically select the set irreducible representations that are dominant in describing the action in the observation space, functioning as a data-dependent nonlinear filter on the dataset. Please also see Sec C for the theory of NFT. As we will validate experimentally in Sec 5.1 NET does choose the major frequencies discretely even in the presence of noise frequencies.
EXPERIMENTS
5.1
NFT VS DFT
To verify that NFT performs nonlinear spectral analysis, conduct experiments ID time series with time-warped shifts. We first prepared a fixed or random set of frequencies F and constructed
ZN 1- E(I)
Ek cos (m -((p))))))(()))))) F= (fi, f2, -.. Sk),
5

---

Published as a conference paper at ICLR 2024
which is the time- wine-w version of the series r(.) ExSLK op cos(2 /N). The shift E NN acts on the latent space as e(-) by (m)))(() = r((1/N}3- m/N). We used N = 128. Note that the frequency spectrum obtained by the classical DFT does not reflect the true symmetry structure defined with F (Fig 3) Codes are in supplementary material, details are in Appendix
Spectral analysis of time-warped signals. In this experiment, verify that NFT recovers the fixed frequencies F from the time-warped signals. We generated 30000 sets of (c1....k) with
Average DFT after Deformation
15.0
12.5
K = 7, yielding 30000 instances of l as dataset X' IR 1212 The 0&S were sampled from the uniform distribution, and cs. Cs scaled by the factor of 0.1 to produce two noise frequencies. See Fig 4 for visualization. To train NFT, we prepared a set of sequences of length-3 (s = (,, 62) with l = r((t/N)3 - E 1125) shifting with random velocity Ug. We then conducted U-NFT as in Sec 3.c with latent dimension R R00x1e so that for each sequence s.
Cowff
10.0
7.5-
5.0
2.5
Frequencies
Figure 3: DFT result
the matrix M. E 1000 provides e, 2 (MM) With this setting, &b can express at most 10/2= = 5 frequencies.
To check whether the frequencies F obtained by NFT are correct, we use the result from the repre- sentation theory stating that, if Py : Z - Re xdy is the irreducible representation corresponding to the frequency f. the character inner product (Fulton and Harris 1991) satisfies
(0gler) = Sraee,,traae(pp trace(trrrrrt(t(t (g)) = 655
(ōfr is Kronecker's delta).
We can thus validate the frequencies captured in the simultaneously block-diagonalized M,s by taking the character inner product of each identified block B, with the known frequencies. The center plot in Fig i js the graph of (py B:)] plotted agasint f. We see that the spike only occurs at the major frequencies in (i.e. {8, 15, 22, 40, 45)), indicating that our NFT successfully captures the major symmetry modes hidden in the dataset without any supervision. When we repeated this experiment with 100 instances of randomly sampled F of the same size, NFT was able to recover the major members of F with (FN, FP) (0.035, 0.022) by thresholding E MM)]] score did not change much even when we reduced the number of samples to 5000 (0.04, 0.028). This experimental result also validates that the disentangled features reported in (Miyato al., 2022) in fact the set of major frequencies in the sense of classical DFT. We can also confirm that underlying symmetry can be recovered even when the deformation applied to 2D images (Fig 6).
)] at 0.5. This
42mioni D.DI
(2
#Z 3.0)
0.00
62meer 1.5)
My
100
Figure 4: Left:the sequence of length=128 128 signals constructed by applying the shift operation with constant speed. Right: the sequence of the same function with time deformation.
Data compression with NFT. In signal processing, FT is also used as a tool to compress data based on the shift-equivariance structure of signals. We generated 5000 instances of the fime-warped signals with F C {0 16), and applied U- G-, and g-NFT to compress the dataset using encoder output of dimension 32211 We also added an independent Gaussian noise at each time point in the training signals. As for G-NFT, we used the direct sum of 16 commutative 2 x 2 matrices of form ((a, -b), (b, a)) parameterize M For g-NFT, we used the 2 block diagonal matrix M(0) + 2 sin
co 1P - sin i0 with known 0.
cos
We evaluated the reconstruction losses of these NET variants on the test signals and compared them against DFT over frequencies of range (0 16) (Fig 12 Appendix). The mean squared errors together with standard deviations in parentheses are shown in Table []] which clearly demonstrates that NFT learns the nonlinearity of observation and compresses the data appropriately according to the hidden frequencies. It is also reasonable that g-NFT, which uses the identity of group element
When the scalar field is TR, 2 x 2 would be the smallest irreducible representation instead of 1 1 in complex numbers.

---

Published as a conference paper at ICLR 2024
#2miooup 0.001)
Frequencies of each block
digganal
bork?
C6-
brock
Brocks
50
100
02 9.87))
#2m0 19.741)
30
Figure 5: Left: Long horizon future prediction of the sequence of time-warped sigmals. Center: Ekpy [B) plotted against E [0,64] for each block B in the block diagonalized M,s learned from the dataset with 5 major frequencies ((8, 15, 22, 40, 45)) and 2 noise frequencies with small coefficients ((18, 43)) when M.s s can express at most frequencies. Note that E [M.] is linear with respect to M* (Appendix A.3). Right: Average absolute value of llok-diiggonllized M,s.
Figure 6: Horizontal shift of unseen objects in fisheye- view predicted from the left-most frame by U- NFT trained on CIFARIOS 100 (Krizhevsky et al. 2009) sequences with T = 4. (top:pred, bottom:gt). U-NFT learns the deformed shifts that are not expressible as linear functions on input coordinates.
acting each signal, achieves more accurate compression than G-NFT. DFT fails compress the data effectively, which is obvious from Fig B
5.2 APPLICATION TO IMAGES
We verify the representation ability of NET by applying challenging tasks which involve out- of-domain (OOD) generalization and recovery of occlusion. Codes are in supplementary material.
Rotated MNIST. We applied g-NFT MNIST (LeCun et al. 1998) dataset with SO(2) rotation action and used it as a tool in unsupervised representation learning for OOD generalization. To this end, we trained (Φ) v) on the in-domain dataset(rotated MNIST), and applied logistic regression on the output of Φ for the downstream task of classification as on rotated Fashion-MNIST (Xiao et al.) (017) and rotated Kuzushiji-MNin 2018) (two out-domains). Following the standard procedure as in (Chen et al. 2020), trained the regression classifier for the downstream task with fixed $. We used data consisting of triplets (go ox 9er orr ( f) with being a digit image and 0, 8 ~ Uniform(0, 2x) being random rotation angles. We used the same transition matrix (0) used in the data compression experiment (Sec 5.1 with the max frequency /max 2 plus one-dimensional trivial representation. Details are in Appendix D Because the representation learned with NFT is decomposed into frequencies, we can make a feature by taking the absolute value of each frequency component; that is, by interpreting the latent variable IR (2)mmx lation is I-dim), we may take the norm of each frequency component to produce (Lmax + 1) X dm- dimensional feature. This is analogous to taking the absolute value of each coefficient in DFT. We designate this approach as norm in Fig
the rotated MNIST (in-domain) as well
(Clanuwat et al.
as mmax + frequeecies of dm multiplicity each (trivial represen-
As we can see the right panel of Fig 7 both g-NFT and g-NFT-norm perform competitively compared to conventional methods. In particular, g-NFT norm consistently eclipses all competitors on OOD. In the left panel, although larger ''max generally benefits the OOD performance, too large a value of /max seems to result in overfit, just like in the analysis of FT. We also compared NFT against StecrableCNN (Cohen and Welling 2017). which assumes that acting G is a rotation group. We gave SteerableCNN competitive advantage by training the model with classification labels on rotMNIST, and fine-tuned the final logit layer on all three datasets, consisting of the in-domain dataset (rotMNIST) and two OOD datasets (roTFMNIST, rotkuzushij--MNISS supervision outperforms all our baselines in the in-domain dataset, but not on the OOD datasets. We believe that this is because, pointed out in (Cohen al. 2018), SteerableCNN functions
SteerableCNN with
7

---

Published as a conference paper at ICLR 2024
g-NFT
G-NFT
DFT (N, 16)
Noiseless
3.59 (#1.22) x10
1.98 (#1.89) x10-
8.10
0 0.01
2.62 (#0.26) x10-1
2.42 (#1.19) x10-
0 0.05
1.42 (400.44) x10-3
5.82 (+:1.15 x10-2
0.1
2.53 (1.0.09) x x10
1.16 (1.0222 x10
Table 1: Reconstruction error of data compression for time-warped 1d signals prediction (max f
Input
GT
16
32
(OOD)
KMNIST
100
accuracy
75
500
0.05
0.03
test 0.01
0.02
102
163
164
102
163
104
102
163
164
num_ num_ www.ffinttteeeeee data
2
B
32
max frequency
Supervised
SimCLA
Steerableic
9-NFT
-e- MNIST -e-
FMNIST (OOD) - KMNIST (OOD)
AE
IAE
Steerable1502)
9 NFT-nno
Figure/ 7: OOD generalization ability of NFT trained on rotMNIST. Top Left: Prediction by g- NFT with various maximum frequencies lmaax. Bottom Left: Prediction errors of g-NFT. Right: Classification accuracy of linear probe, compared against autoencoder (AE), SimCLR (Chen et al.) 2020), the invariant autoencoder (IAE) (Winter et al., (022) and supervised model including Cn steerable CNN (Cohen and Welling 2017) and SO(2) steerable CNN (Weiler et al. 2018). Each line is the mean of the accuracy over 10 seeds, with the shaded area being the standard deviation.
composition of filters that preferentially choose the frequencies that are relevant to the task used in the training, so that the model learns G-linear maps that overfitted to classify the rotMNIST dataset. Also see Appendix E for rotMNIST with more challenging condition involving occlusion.
Learning 3D Structure from 2D Rendering. We also applied g-NFT to the novel view synthesis from a single 2D rendering of a 3D object. This a challenging task because it cannot be formulated as pixel-based transformation in all renderings, the used three datasets: ModeINet10-SO3. (Liao et al. 2019) in 64 x 64 resolution, BRDFs (Greff et al., 2022) (224 x 224), and ABO-Material (Collins al. 2022 (224 224). Each dataset contains multiple 2D images rendered from different camera positions. We trained the autoencoder with the same procedure as for MNIST, except that we used Wigner D matrix representations of SO(3) with We used Vision Transformer Doooovitskiy et al. 2021 to model & and Y.
side of the object is always occluded. We
1max
The prediction results (Fig g) demonstrate that g-NFT accurately predicts the 2D rendering of 3D rotation. We also studied each frequency component by masking out all other frequencies before decoding the latent. Please also see the rendered movie in the Supplementary material. Note that 0- th frequency (FO) captures the features invariant to rotation, such as color. Fl (second row) captures the orientation information, and higher frequencies extract symmetries of the object shapes. For example, F3 depicts triangle-like shapes having rotational symmetry of degree 3, similar to the spherical harmonics decomposition done in 3D space (Fig 18). See Appendix for details.
6 RELATED WORKS AND DISCUSSIONS
As tool in equivariant deep learning, group convolution (GC) has been extensively studied (Cohen and Welling 2017 Cohen et al. 2019 Krizhevsky et al. 2012). NFT differs from GC as well as FT in that it does not assume to act linearly on the input. In the words of (Kondor and Trivedi 2018 FT and GC assume that each instance E X' is a function defined on homogeneous space, or the copy of the acting group modulo the stabilizers. These methods must install the explicit knowledge of the group structure into the model architecture (Finzi et al.! 2021) (Kondor 2008 Reisert and
We used SO(2) representation for BRDFs however, since its camera positions have a fixed elevation.
8

---

Published as a conference paper at ICLR 2024
prediction
input
input
input
Figure 8: Novel view synthesis from a single image of a test object that is unseen during training (from left to right: ModelNet, BRDFs, ABO). From bottom to top, we show ground truth, g-NFT prediction, and the decoder output with k-th frequency only. As for ABO, the same backgrounds were repeatedly used for both training and test datasets.
Burkhardt 2007) have used group invariant kernel, but limited their scope to situations similar to FT. When the action is linear, our philosophy of G-NFT also shares much in common with (Cohen and Welling 2014).
As other efforts to find the symmetry under nonlinear actions, (Dupont et al. 2020) took the ap- proach mapping the input to the latent space of volumetric form and applying the linear rotation operation the voxels, yielding instance of g-NFT. (Shakerinava al.) 2022 uses different types of invariants (polynomial) that are specific to group/family of groups, instead linearlized group actions in the form of representations. (Falorsi et al. 2018) maps the observations to the ma- trix group of (0(3) itself. (Park et al., 2022 first maps the input to an intermediate latent space with blackbox function and then applies the convolution of known symmetry for contrastive learning. Finally, Koopman operator (Brunton et al.) 2022) is closely related to the philosophy of NFT that it linearizes a single nonlinear dynamic, but NFT differs in that it seeks the latent linear transitions with structures (e.g frequencies) that are specific to group.
Most relevant this work, (Miyato al. 2022 presents an implementation of U-NFT and uses to disentangle the modes of actions. However, they do not present other versions of NFT (g-NFT, G-NFT) and, most importantly, neither provide the theoretical foundations that guarantee the identi- fiability nor establish the "learning of linearlized equivariance" as an extension of Fourier Transform. By formally connecting the Fourier analysis with their results, the current work has shown that the contextual disentanglement that was often analyzed in the framework of probabilistic generative model (Zhang et al., 2020) Kim and Mnih (018) or the direct product groups (Higgins et al.] 2018 Yang et al. 2222]) may be described in terms of Fourier frequency as well. To our knowledge, we are the first of a kind in providing a formal theory for seeking the linear representations on the dataset with group symmetries defined with nonlinear actions.
LIMITATIONS AND FUTURE WORK
As stated in Sec 3.1 NFT requires dataset consisting of (short) tuples because it needs to observe the transition in order to learn the equivariance hidden in nonlinear actions; this might be a practical limitation. Also, although NFT can identify the major frequencies of data-specifie symmetry, it does not identifiably preserve the norms in the latent space and hence the size of the coefficient in each frequency, because the scale of & is undecided. Finally, while NET is framework for which we have provided theoretical guarantees existence and identifiability in general cases, the work remains to establish algorithm that is guaranteed find the solution eq. (3). As stated in (Miyato et al. 2022), the proof has not been completed to assure that the AE loss of Sec can find the optimal solution, and resolving this problem is future work of great practical importance.

---

Published as a conference paper at ICLR 2024
REFERENCES
Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: next-generation hyperparameterr optimization framework. In SIGKDD, page 2623-2631, 2019.
L. Brunton, M. Budišić, E. Kaiser, and J. N. Kutz. Modern Koopman theory for dynamical systems. SIAM Review, 64(2):229 340, 2022. doi: 10. 00.117/21M1401243
21M14012433
T. Ceccherini- Press, 2007.
Ceacherini-Silberstiin,, Silberstein, F. Scarabotti, and F. Tolli. Harmonic Analysis finite groups. Cambridge University
G. Cesa, L. Lang, and M. Weiler. A program build B(N)-equivariant steerable CNNs. In Interna- tional Conference Learning Representations, 2022. URL. https://open forr wd- nQw
H. Chen, Liu, Chen, H. and R. Hill. Equivariant point network for 3D point cloud analysis. CVPR, pages 14514-14523, 2021.
Chen, S. Kornhlith, M. Norouzi, and Hinton. A simple framework for contrastive learning visual representations. In JCML, pages 1597-1607, 2020.
Clanuwat, M. Gober-Irizat A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.0173 2018.
M. Clausen and U. Baum. Faxt Fourier Transforms. Wissenschaftsverlag. 1993.
Cohen and M. Welling. Learning the irreducible representations commutative lie groups ICML, pages 1755-1763, 2014.
Cohen and M. Welling, Group equivariant convolutional networks. In ICML, pages 3990-2999, 2016.
Cohen, M. Geiger, and M. Weiler. general theory of equivariant CNNs homogeneous spaces. NeurIPS, 36, 2019.
Cohen and M. Welling, Steerable CNNs. ICLR, 2017.
Cohen, M. Geiger, and Weiler. Intertwiners between induced representations (with applications to theory of equivariant neural networks). arXiv preprint arXiv: 10743, 2018.
J. Collins, Goel, K. Deng, A. Luthra, L. Xu, E. Gundogdu, Zhang, T. F. Yago Vicente, T. Dideriksen, H. Arora, M. Guillaumin, and Malik, ABO: Dataset and benchmarks for real-world 3D object understand- ing. CVPR, 2022.
defisheye Contributors. defisheye, http::///ppp..oo/ooooeeeeee
2019. [Software].
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghami, M. Minderer, G. Hcigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
E. Dupont, M. A. Bautista, A. Colburn, A. Sankar, C. Guestrin, Susskind, and Q. Shan. Equivariant neural rendering. ICML, 2020.
L. Falorsi, P. de Haan, T. R. Davidson, N. De Cao, M. Weiler, P. Forré, and T. S. Cohen. Explorations in homeomorphic variational auto-encoding. arXiv preprint arXiv:1807.04689 2018.
M. Finzi, M. Welling, and A. Wilson. practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. ICML, 2021.
Folland. Course Abstract Harmonic Analysis. Taylor and Francis, ISBN 9780849384905.
W. Fulton and Harris. Representation Theory: First Course. Springer New York, 1991. ISBN 9780387974958. URL htt.hhts://bbokk wwwp:ookkk.ooolee.... www.le.oo..ooooooooooooo mooolleecoo co. jp/books 2iookssid--ao 6CU
K. Greff, F. Belletti, L. Beyer, C. Doersch,
D. Duckworth, D. Fleet, Gnanapragasam, Golemo, C. Hermmann, T. Kipf, A. Kundu, D. Lagun, Laradji. H.-T. D. Liu, H. Meyer, Y. Miao, D. Nowrrouzezahrai, C. Oztireli, E. Pot, N. Radwan, D. Rebain. S. Sabour, M. S. M. Sajjadi, M. Sela, V. Sitzmann. A. Stone, D. Sun, S. Vora, Z. Wang, T. Wu, K. M. Yi, F. Zhong, and A. Tagliasacchi. Kubrie: scalable dataset generator. In CVPR, 2022.
10

---

Published as a conference paper at ICLR 2024
L. Higgins, D. Amos. D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner. Towards definition of disentangled representations. arXiv preprint arXiv.8812.022300 2018.
Keller and M. Welling. Topographic VAEs learn equivariant capsules. In NeurIPS, pages 28585-28597, 2021.
H. Kim and A. Mnih. Disentangling by factorising. In ICML, pages 2649 2658, 2018.
R. Kondor. Group theoretical methods in machine learning. PhD thesis, Columbia University, May 2008.
R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. ICML, 2018.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009. URL https: //www cs. tcrento. wwwp/kkri//eeariingg weatariin--fetuueeeee tewaturree00001111
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 2012.
Y. LeCun, L.. Bottou, Y. Bengio, and Haffner. Gradient-based learning applied to document recognition. Proceedings of IEEE, 86(11):278-2233 1998. URL httt:://yaannn....m. Lecun. com/ exdb /mni//nnee st/ License
Liao, E. Gavves, and C. G. Snoek. Spherical regression: Learning viewpoints, surface normals and 3D rotations on n-spheres. In CVPR, pages 9759-97677, 2019.
1. Loshchilov and Hutter. Decoupled weight decay regularization. arXiv preprins arXiv:1711.05000 2017.
Maehara and Murota. Algorithm for error-controleed simultaneous block-diagonalizztion of matrices. SIAM Journal on Matrix Analysis and Applications, 60( 6200, 2011.
Miyato, M. Koyama, and K. Fukumizu. Unsupervised learning of equivariant structure from sequences. NeurlPS, 2022.
J. Y. Park, O. Biza. L. Zhao, J. W. van de Meent, and R. Walters. Learning symmetric embeddings for equiv- ariant world models. /CML, 2022.
S. Raghunathan and U. D. Priyakumar. Molecular representations for machine learning applications in chem- istry. Inernational Journal of Quantum Chemistry, 2021.
M. Reisert and H. Burkhardt. Learning equivariant functions with matrix valued kemels. Journal of Machine Learning Research (JMLR), 8(15)385-408, 2007.
W. Rudin. Fourier Analysis on Groups. Wiley, 1991. ISBN 9780471523642.
URL ht https://books
google..ccommooooooo co. stppbooksiddsee 2LLUWuU222UUUYY
M. Shakerinava, A. K. Mondal, and S. Ravanbakhsh. Structuring representations using group invari- ants. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Ad- vances in Neural Information Processing Systems, volume 35, pages 34162-34174, Curran Associates, Inc., 2022, URL httttss:/pprocood.nggg 21g233000000baa222233333333-----
H. Skibbe, Q Wang, O. Ronneberger, H. Burkhardt, and M. Reisert. Fast computation of 3D spherical Fourier harmonic descriptors complete orthonormal basis for a rotational invariant representation of three- dimensional objects. In /CCV Workshops, pages 1863-1869, 2009. doi: 10. 10.100//CCCW..00555555550
M. Weiler and G. Cesa. General E())-cquivariant steerable CNNs. In NewrlPS, pages 14334-14345, 2019.
M. Weiler, F. A. Hamprecht, and M. Storath. Learning stecrable filters for rotation cquivariant cnns. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 949-858, 2018.
R. Winter. M. Bertolini, T. Le, F. Noé, and D.-. Clevert. Unsupervised learning of group invariant and equivariant representations. arXiv preprint arXiv:2202.07559, 2022.
H. Xiao, K. Rasul, and R. Vollgraf. Mishion-MNISTT novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17Xvv:1708.077477 2017.
T. Yang, X. Ren, Y. Wang, W. 7eng, N. Zheng, and P. Ren. Groupify VAF: From group-hased definition to VAF-based unsupervised representation disentanglement. arXiv preprint a02iv:2 10303, 2021.
11

---

Published as a conference paper at ICLR 2024
A. Yu, R. Li. M. Tancik, H. Li, R. Ng. and A. Kanazawa, Plenoctrees for real-time rendering of neural radiance fields. ICCV, 2020.
R. Zhang, M. Koyama, and K. Ishiguro. Learning structured latent factors from dependent data: A generative model framework from information-theoreiic theoretic perspective. In ICML, pages 11141-11152, 2020.
12

---

Published as a conference paper at ICLR 2024
A ELEMENTS OF GROUP, ACTION, AND GROUP REPRESENTATION
In this first section of the Appendix, we provide a set of preliminary definitions and terminologies used throughout the main paper and the Appendix section (Sec C main results. For general reference on the topies discussed here, see (Fulton and Harris 1991). for example.
dedicated to the proofs of our
A.1 GROUP AND GROUP REPRESENTATION
A group is set G with a binary operation G x G 3 (g. h)) + gh E G such that (i) the operation is associative (gh)r g(hr), (ii) there is a unit element and (iii) for any g E G there is an inverse g so that 9g
G that eg = ge = g for any g E G,
Let G and H be groups. A map yp p:C G - H is called homomorphism if ((st) = (ps))((s()))) for any & G. If homomorphism
G - H is a bijection, it is called isomorphism.
Let V be vector space. The set of invertible linear transforms of V, denoted by GV( is group where the multiplication is defined by composition. When V = R", GL(R") is identified with the set of invertible n x n matrices as a multiplicative group.
For group G with unit element and set X, (left) action of G on X' is a map L: G x x denoted by I.g(2) == L(g. 2). such that (x) = x and Ligh = T, (x)). If there is no confusion, Lg (x) is often written by gz
For group G and vector space V, group representation (p, V) is a group homomorphism p G - GL(V), that is, p(e) Idy and p(gh) p()) A group representation is special type of action; consists of linear maps on vector space. Of particular importance, we say (p. V) is unitarizable if there exists change of basis on V that renders p(g) unitary for all g G. It is known that any representation of a compact group is unitarizable (Folland 1994). Through group representations, can analyze various properties of groups, and there is an extensive mathematical theory on group representations. See (Fulton and Harris 1991), for example.
Let (p, V) and (r, W) be group representations of G. A linear map f V W called G-linear map (or G-map) if (p(g)o) (v))[(v) for any E V and E G. that is, if the diagram in Fig commutes.
plg)
V
f
[(g)
W
W
Figure 9: G-linear map
A G-map is a homomorphism of the representations of G. If there is a bijective G-map between two representations of G. they are called G-isomorphic, or isomorphic for short.
Related to this paper, an important device in representation theory is irreducible decomposition. A group representation (p, V) is reducible if there is a non-trivial, proper subspace W of V such that W is invariant to G, that is, W C W holds for any g G. If is not reducible, it is called irreducible.
There is famous, important lemma about G-linear map between two irreducible representations.
Theorem A.1 (Schur's lemma), Let (V, p) and (W, r) be two irreducible representations of group G. If V W is G-linear, then f is either isomorphism or the zero map.
13

---

Published as a conference paper at ICLR 2024
Pp(a))
Po(s))em
relea)m
Pp(s.)P
P(R:)
p(D.) p(as)
p(g.)
[ P H{
PP-1
Figure 10: Simultraneous block diagonalization. Existence of such P is guaranteed if the represen- tation is completely reducible. Note that the matrix P for the change of basis is common for all
g E
A.2 IRREDUCIBLE DECOMPOSITION
A representation (p, V) is called decomposable if it isomorphic direct of two non-trivial representations (pu, U)) and (pw, W), that is
1 pw,
(6)
If a representation is not decomposable, we say that it is indecomposable.
is obvious that group representation decomposable, it is reducible. In other words, an irreducible representation is indecomposable. An indecomposable representation may not be ir- reducible in general. is known (Maschke's theorem) that for finite group, this true; for non-trivial subrepresentation (r, U) of (p, V) we can always find a complementary subspace W such that V U @ W and is G-invariant.
It is desirable if we express representation P direct sum of irreducible representations. We say that (p, V) is completely reducible there is finite number of irreducible representations {(e,, Vm of G such that
P1
pm.
completely reducible representation is also called semi-simple. the irreducible components are identified with subrepresentation of the original (p. V). the component can be uniquely obtained by restriction of p to the subspace V,. The irreducible decomposition is thus often expressed by decomposition of V, such as
== V1
Vm
(8)
In the irreducible decomposition, there may be some isomorphic components among V., By sum- marizing the isomorphic classes, we can have the isotypic decomposition
W
W
where W tiplicity the irreducible representation Wy. The isotypic decomposition unique if an irreducible decomposition exists.
We are mutually non-isomorphic irreducible representations of G, and n, is the mul-
A group representation may not completely reducible in general. However, some classes are known to be completely reducible. For example,
representation of a finite group a finit-dimenniioal representation a compact Lie group
fmite-dimensional representation of locally compact Abelian group
are completely reducible (Fulton and Harris 1991).
representation (p, V) is completely reducible, we can find a basis of V such that the represen- tation can be expressed with the basis in the form of block diagonal components, where each block corresponds to an irreducible component. Note that the basis does not depend on group element g, and thus we realize simultaneous block diagonalization (see Fig 10
When complex vector spaces are considered, any irreducible representation of a locally compact Abelian group is meediinnssiona.. This is the basis of Fourier analysis. For the additive group [0, 1). they are the Fourier basis functions 2 (x E 1), k = 0, 1,...... See (Rudin) 1991). for example, for Fourier analysis locally compact Abelian groups.
14

---

Published as a conference paper at ICLR 2024
A.3 CHARACTERS
In this section, we introduce the idea of characters which is used in Sec 5.1 For representation
(p, V), the character Xp : G - C of p defined as its trace
xp(g) = trace trace(p(g))
This function has extremely useful properties in harmonic analysis, and we list two of them that exploit in this paper.
Property (Invariance to change of basis) By its definition, character is invariant to change of basis. That is, if (p, V) and (p, W) are two representations that are isomorphic so that there exists a linear invertible map P: V - W, then
Xp(g) = trace
(10)
trace (r)))(
(11)
trace(p(g)) = Xp(g).
(12)
Indeed, if V = W, Pis is just the change of basis.
Property (Character orthogonality) Suppose that G is compact group. Then group invariant measure on G is a measure satisfying that
p(A) p(gA) p((g A)).
du is group invariant measure with for du(g) = 1, then for all irreducible representations p and
(p = [ (x( = {
if p and أ are isomorpohic otherwise.
(13)
Thus, if M is a representation that is isomorphic to the direct sum of {px}, then by the linear property of the trace, (p) is the number of components the direct sum that are isomorphic to and this is called multiplicity of in M. For more detailed theory of characters, see (Fulton and Harris 1991).
B PROOF OF LEMMA 3.1
The following a rephrase of Lemma 3.1
Lemma B.1. Suppose that
((g - x) = (g)
(14)
holds for any x X and g G. If span ( (xx) equals the entire latent space, then M M(g) is group representation, i.e., M(e) = Id and M ((gh) = (l).
Proof. Take g = e in eq. 14 Then, ((x) x() (() for any æ. It follows from the
assumption of (x(()) that M(e) Id holds. Next, by replacing a with how (h G) in eq, [1).
we have
( (hor).
This implies
M(gh)((z) ((g)M(())(ss,,
from which we have M (gh)= = (h).
C PROOFS OF THEOREMS IN SEC
This section gives the proofs the theorems in Sec Throughout this section, we use X' to denote data space, and assume that group G acts on X.
In this paper, we discuss an extension of Fourier transform to any data. Unlike the standard Fourier transform, it does not assume each data instance be function some homogeneous space. To
15

---

Published as a conference paper at ICLR 2024
this end, embed each point E X' to a function space on X, which linear action group representation is easily defined. Recall that we can define a regular representation on a function space H. on X by
bn : f() - S(0 E
easy to see that g - li is a group representation of G on the vector space H. order to discuss the existence of ④ into a latent space with linear action, we want to introduce an embedding X H and make use of this regular representation. The reproducing kernel Hilbert space natural device to discuss such an embedding, because, as is well known to the machine learning community, we can easily introduce the so-called feature map for such a L, which a mapping from space X' to the reproducing kernel Hilbert space.
C.I EXISTENCE
In this subsection, we will prove the existence part of Thm 4.1: that is, whenever there is a feature space with G-invariant inner product structure, we can find linear representation space for the data space, which means that we can find an encoding space where the group action is linear.
RKHS with G-invariant kernel:
Proposition C.1. Suppose that K is a positive definite kernel a topological space on X and G acts continuuusl on X. Define Kg(z.y) := K(g x, g y) for E G. Then H (Kg)) := {f(g-1) 1 f E HK)} is the RKHS corresponding to Kg for any g. and H(K) + H ((gg) defined by p(g) f(-) + f(g ((.) an isomorphism of Hilbert spaces.
}
Proof. It clear that Kg (z, y) is positive definite and that p(g) is an invertible linear map. Let Ho be the dense linear subspace spanned by (- }) in H H(g)). In the way of Moore's 's theorem, we equip Ho with the inner product via (kig -), (y, x),,, = (2 y)- First, note that Kg satisfies the reproducing property for any function f E Ho. In fact, for En Og K, Rn). we see
(f, (K((,)))K = E annKalan
f(x).
Also, H (Kg)) is the closure of the span of (kg .)} with respect to the norm defined by this inner product. is then sufficient to show that p(g) maps the linear span of {K( (K(xx)), to Ho isometrically. Consider (.)-) Em (n)(( E H and note that
p()) IN(g
Σ On (Fn,
(15)
N
an Kigi (gInn Ho.
(16)
a
From this relation, we see
(p())(s)))
Σ
amiiggg (9 gmm)
Σ an
ll
(17)
This completes the proof.
Now, this result yields the following important construction of representation space when K is G-invariant.
Corollary C.2. Suppose that K is positive definite kernel on X that is G-invariant in the sense that K(g ((gggg = K K(x, v) for any x,y E X and g E G. Then the reproducing kernel Hilbert space (KK) is closed under the linear representation of G defined by
p(g) : 5() + ((g
and p is unitary.
16

---

Published as a conference paper at ICLR 2024
Proof. If K = K, then (Kg)) (K) for any 9. and thus p(g) acts (K) linearly, Also, it follows from eq.(17) in the proof of Proposition C.1 that || [| ||,((--1)) k for h E H(K). Therefore p(g) unitary.
The following is just restatement of the corollary
Corollary C.3. Let X' be set for which G-invariant positive definite kernel K exists. Then there exists a latent vector space V that is representation space of G, i.e. there is an equivariant map 4: X + V such that the action of G on V is linear.
Proof. Simply define the map : x (-) = K(x,)) ). Note that p(g) defines an appropriate group
representation the span of (-) K })} because kyozi (-) [g og, K (xx
(-))ka(-) by the invariance property, and the result follows with V = H(K).
Thus, this result shows that an equivariant feature can be derived from any invariant kernel.
C.1.1 THE RICHNESS OF THE SPACE GENERATED BY GROUP INVARIANT KERNEL
Given positive definite kernel K(x,y), the integral with the Haar measure easily defines G- invariant kernel. An important requirement for the G-invariant kernel that it can introduce a sufficiently rich reproducing kernel Hilbert space (RKHS) as latent vector space that any irre- ducible representation is included. The following theorem shows that if the RKHS of the kernel K dense in (P)), then is the RKHS of the derived G-invariant kernel.
Theorem C.4. Let G be locally compact group acting continuously on a space X. and K be a continuous positive definite kernel on X such that the corresponding reproducing kernel Hilbert space Hg is dense in L2(P) with probability measure P on X. Then, assuming that, for any I, E X, the integral with right Haar measure bounded Sc [[g o x, g gg dpl())))) C for constant C, the positive definite kernel defined
Ko(ε, v) = =Jo (lgox.g=y)dul
(18)
is G-invariant such that the corresponding reproducing kernel Hilbert space is dense in L2(P).
Proof. The invariance is easily confirmed since, for any a E G.
Ka(a o y. = -s K(ga o ga g y)du(()) = [ K go)xxggo))d
holds by the right invariance property of μ.
For the denseness (2P),, suppose that L2(P) is orthogonal to Hkg in L2(P). suffices to show that h = 0.
follows from the orthogonality that
J/ [ go (fggss.gooyuuuccooeeeeeoeeoppppp =
(19)
Let (xx) = K(.#)) E Hx. From fx Jx Ja I Kegoa..gggglMlelhepepopooohoopp s
C']LI|2((P) Fubini's theorem tells that the left-hand side of eq equals to
- ( J (o0o=)))))) (P(E))d))dd
1 M dp(g),
(20)
where M2(g) is defined by
Mp(g) /  (()))))()))P(())
17

---

Published as a conference paper at ICLR 2024
Note that Mr. G + Hx is well-defined and continuous by the Bochner integral, since
/ | o + dP(x)
s (Le l dP(x)
1/2
(/s IM(o))PP(())
1/2
( K(g 2. (())xa)))))P(a))
1/2
[|
finite and continuous with respect to g-
follows from eq. 19) and eq. eq.20) 20 that
J M du(g) = 0
holds, which implies M, = 0. In particular, plugging g c, we have
J (z))(())dP(s)= O.
The denseness of HK in Lz(P) implies that the integral operator h + fx ((xx)))(()dd((())) is injective, and thus we have h = 0, which completes the proof.
C.2 UNIQUENESS
Now that we have shown that any given kernel can induce an equivariant map, we will show the other way around and establish Theorem 4.1
Theorem C.S. Up G-isomoprhism, the family of G-Invariant kernel K(x,)) y) X has one-to- one correspondence with the family of equivariant feature X - such that the action on V is unitarizable and V ( E
C.3 that an invariant kernel induces a linear representation space with unitary representation. Note that, in the construction of + in C.3 from the kernel, the + trivially satisfies the relation (2(x), (2)) = K (a, y). Next we show that this correspondence from ((K to & is in fact one to equivariant map & can correspond to a unique class of representations that are all isomorphic to ((K) (isomorphism class of (K)). In particular, + to construct K, and show that V is isomorphic to H(K) itself as a representation space.
Proof. We have shown in Proposition
reversing this construction and show that given unitarizable
start from unitarizable equivariant map
Φ X'
Let + X
V be equivariant map into linear representation space (M, V) with V x E X). We assume WLOG that M(g) is unitary for all g because the assumption guarantees that M can be unitarized with change of basis, which is G-isomorphic map. Let K be a kernel defined by K(x,1) y) = (o()), & (1)). This kernel is invariant by construction. We will show that V is G-isomorphic to (K)).
To show this, we consider the map from V to H(K) defined as
J.V H(K)
such that
J(u)
[r (0(x),)))
(21)
where Ju) is a member of H(K) by the definition of K. We claim that (0(g-to)) u) (0(z).geu) for all u. To see this, note that any E CI ()ap x X3 can be rewritten as = Σ, (2x) for some sequence of a, Thus
(f(x.ggom) (0(x), E (.M (M* ()) Ea (0(g oor) u)
(22)
18

---

Published as a conference paper at ICLR 2024
where (g) ( is the conjugate transpose of (g). Therefore it follows that J(M(g)u) = {i - (0()). (g) f + (0 - x))
(23)
(24)
(25)
where the action of g on (() is the very unitary representation with go f(x)
f(g that
we defined in Sec C.1 Because J is trivially linear, this shows that J is G-linear map with standard action on H K. Also, (9(x), = (o(x), e) for all x, then u = v necessarily V CI Span (xx) ) x &). Thus, the map J is injective. The map J is trivially surjective as
well, because (x) = le and H(K) = CiSpan lex x E 27. In fact, this is map induces an isometry well, because
(J - (Ka, konkky)) K = K(x v) (0(x),4())
(26)
and thus validating the reproducing property (J$(x), Ju) (.(x), Ju(x). Finally, this isomorphism holds for any 4 satisfying Kxx, y) = (x(x). ( In summary, Any group invariant K corresponds to unique linear representation space H(K) and any family of unitarizable equiv- ariant map 4s that are G-isomorphic to each other corresponds to a unique G-invariant kernel K
corresponding to (K) with K (x:, y) = (ar),, p(y)).
When the group of concern is compact, this result establishes the one-to-one correspondence with any equivariant map because all the representations of a compact group are unitarizable (Folland) 1994).
This result also derives the following collorary claimed Sec
Corollary C.6. Suppose that 4; X + V, is an invertible equivariant map into a linear represen- tation space (M, for 1,2. Then V1 and V22 differ by G-isomorphic map.
Proof. By the previous claim, if (x, y) (x)) pb())) then it suffices to show that H(K)) and H ((2)) are G-isomorphic. By the invertivility of 42 and the previous result, L (I, - K2(x, -) is an invertible map on K1 (Xx.). This trivially induces G-isomoprhism between H(Ki)
and H(22) because L(Ki(s < )) = K2(9 ox,,) = go K2(1,)) and Ki(x,-) generates H(K,).
C.3 FOURIER FILTER
In this paper, we are building theoretical framework that utilizes equivariant encoder & X V into linear representation space V satisfying
X) = (X) D: GL(V), D()) D(gh) h
(27)
together with a decoder Vr In this section, we make a claim regarding what we call "Filtering principle" that describes the information filtering that happens in the optimization of the following loss:
C DPP, Pc) Ex-P.g-e Illg X ( x)1l
(28)
where & is to be chosen from the pool of equivariant encoder and v is chosen from the set of all measurable maps mapping V to X. We consider this loss when both P and Pa are distributions over X' and G with full support.
C.3.1 LINEAR FOURIER FILTER
Let us begin from a restrictive case where X' is itself a representation space (inner product vector space) with unitary linear action of granting the isotypic decomposition
X
V.
(29)
where V3 is the direct sum of all irreducible representations of type b. V. is called isotypic space of type (Clausen and Baum 1993] Seecherini-Silbeesttee et al. (007).
In order to both reflect the actual implementation used in this paper as well as to ease the argument, consider the family of equivariant maps t X - W that satisfy the following properties:
19

---

Published as a conference paper at ICLR 2024
Definition C.7. (Multiplicity- Muulippiicty-umlimmiteed -nipinneeMMMapiiinng
Let A' be a representation space of G and Zx the set of all irreducible types present in X. Suppose have vector space W with linear action of G and equivariant map & X W. In this case, we use I to be the set of all irreducible types present in ((X). We say that & is a multiplicity- unlimited map if the multiplicity of any b Io is equal to that of b in X.
Thus, in terms of the characters, if G acts on ((2²) with the representation M4 and G acts on X with the representation M (that is, if (gg or) Mo (x)) and g g M(g)æ x fforalla E X),
then (p) (p) whenever (p| 0.
Given an equivariant map & X + W, where W a vector space with linear action, let Co (X) denote the set of all measurable map from W to X. We also use P. to be the projection of X' onto V.. and let V = P.X.. Because each V/ is a space that is invariant to the action of G, P. can be shown to be G-linear map (Clausen and Baum, 1993). With these definitions, then make the following claim:
Proposition C.8. Fix a representation space X, a distribution on X, and Pg on G. With the assumptions set forth above, let M be the set of all multiplicity unniiitte linear equivariant map from X to some vector space W with a linear action. For any given E M, define
L(Φ)=
min (2C)(x")
C(D, Pc).
where C(b, V/P. is given by eq. 28. Then
((
Σ
Ep ||
(30)
b
for some set-dependent functions Rs-
Proof. Let Ps be the projection of X onto Vo, that
C = Σ Blllg Vo Po (
Σ Rs V PP))
(31)
where Vú, = P&X and the integrating distributions (P, Pc) in the suffix of E are omitted for brevity.
Now, it follows from the definition of conditional expectation that
min ( I1P, PG) = Elg V/ Elg V. ( X)]]
= -Bg o Vi Eig Vs tgo Vizk I+
(32)
The second equality follows from Schur's lemma (Thm A. assuring that + restricted each V%, an isomorphism for each & Zz. We see in this expression that if E Iz. then ming Ro VV, 4) = V, whenever
because E[g - V,. {g- = Vx; k E I =
Zz. We therefore focus on E Io and
show that Rg with & In scales with [[]] |. First, note that because the norm on X' is based on G-invariant inner product, Billg V 12] = [[ Next note that, for any scalar >> O, we have
E'goaV, (go V4 Io aE Vo {go V4; k I+
(33)
Therefore
min Ex..xxxxPPlPllllle P x)
Ellgg V, Elg a ((g X]][
(34)
= Ellg - V, Ellg Vo ((g X)
(35)
a² min Rs ④, Pc)
(36)
Thus there is distribution Pt of X with Ppp [[[]]]]]]]] such that, by factoring out Ep [I] [b as
we did in the expression above, we can obtain
min Rs VP, Ep Ep [[V][1]]] min (o V/P!,
v/Pi.Pa) Pc)
(37)
20

---

Published as a conference paper at ICLR 2024
indicating that ming R&, P, PG) scales with [/] for every choice of Lp. Finally, see from cq. that ming ( w/ PP, Pa) depends only on I so define (T(Is) =
ming (2, 1 pa)) Because V can be chosen each Vú, separately for each b, see that
L
Ep |=
(38)
From this result we see that the optimal solution the optimal D², V/* in the linear case can be found by minimizing Eoez, [[ [ (I+)) about I& with $ E F that EseI, dim(V;) S dim and find the corresponding V by setting (V(V) = P. (V) E (V;; E I+ This is indeed an optimization problem about the discrete set Ip- We also claim the following about the depen- dency of the solution to this problem on the norms V..
Corollary C.9. For C In, define {(J) = Enes E V For any $1 and &2 we have L(#1) > L(D2) if(1) Iz, C CI,, or (2) E(Inq) is sufficiently larger than E(Tag).
Proof. If Ip, C Izz, then note that
L( min C( Pc) = Blllg X - ( X)1
|B[lg X Elg X ( [
(39)
Elg V E[g-V fg Va E I. Hll.
Because ( o Vz: E Io,]) 3 olgg Vi:; k E Io, }) where is the sigma field, it trivially follows that I((0)) I(( by the definition the conditional expectation.
Next, if To, & Io,, note that L( = minwecs (x) L IPP, Pc) satisfies
C. (Ix)
Σ E 2 L( Σ Ep 1 (Io))
Σ
Ep EPPIIV/l1 11
(40)
beLaa
beIz
for some Cu and q. Thus, if
CI(Ion)
Σ E
Σ
Ep EP[[11/13]
(41)
cI(,) beTTa
bgTT,
Then
L(Φ1) 2 (Io,)
Σ Ep 1I 2 cu (Io.) DEToz
Σ
1 L(
(42)
Then L(D1) > L( is guaranteed. In other words, We have L(Ø1) > L( when {(Ia;) is
sufficiently larger than ((Tog).
C.3.2 NONLINEAR FOURIER FILTER
In this section, we extend the argument in the previous section to a more general situation in which X' is not necessarily a representation space itself. Before we proceed, a word of caution is in order regarding the space on which the encoder & is to searched in the optimization process.
As is the case in many applications, we set our goal to find & from set of functions with certain smoothness, and we use this smoothness property to relate the harmonic information captured in the latent space. In analogy to this argument, we thus restrict the search space of our encoder & to a space of functions with a certain level of smoothness in order to make the claims meaningful. Therefore, in this section, assume that there exists G-invariant kernel K ((,,), and that searching the encoder from the space of equivariant functions from X' to some vector space over R satisfying the following properties:
21

---

Published as a conference paper at ICLR 2024
X
F H(K)
Figure 11: The commuting relation established by Lemma C.11
Definition C.10. Let K be G-invariant positive definite kernel on X. and H(K) be the RKHS corresponding to K. A map & X - V for some Hilbert space is called H H(K) ((K))continuous equivariant map if the following two conditions are satisfied;
The action g on ((x) is linear; that is, there exits a representation D+ : - GL&((V such that (
(x).))
2. For all V, the map + (v, <(x))v belongs H(K).
We first show that any H continuous equivariant map factors through the map from X to H(K) (I.
Lemma C.11. Suppose that is a G-invariant positive definite kernel on and + is a H-(K- continuous equivariant map from X to V. For the canonical embedding F X + H(K), X ka, there exists linear equivariant map μa H(K) - V such that 4 = F, where H(K) admits the linear regular action p(g):
f(g 1..).
Proof. Because (v. F()lv H(K)) for any by assumption, there is 1 E H(K) such that (v, ((x)) = (ft ka)H. where (, ) is the inner product of H(K). This being said, if {e;} is the orthonormal basis of V, we have
+ (xx) = Σ eeler F(z))v
= -eaff kkzH
Motivated by this expression, let define the linear map pa H(K) - V by
μbb h r Σ Eeelf h)..
(43)
We claim that the diagram Fig 11 commutes and up equivariant.
First, it follows from eq. (43) that
= u.p (ka) = Σ Cif Ko)H = edler =
(44)
which means μa o F(x) = (xx). Recall that the linear action p defined by p(g) : h(-)) + 1l.)) Then, we have p(g)kz K K(x, o.) K(gex,)) kg ors and thus, using eq. 44
ne (p(g)kk)) = po (kgoz) = (g(goz)
((g))(()).
Because the span of (kz x X3 is dense in H(K), the above equality means the equivariance of μg..
Now, with Lemma C.I1 we may extend the definition of multiplicity free map that is H(K)- continuous equivariant map.
Definition C.12. (Multiplicity
sinuiiliiiity-ulimiteedd unlimited mapping, General) We say that a H(K) -cotinnuuus equiv- ariant map 4 X + V is multiplicity unlimited if рa constructed in the way C is multiplicity unlimited map. Let us denote by I. the set of all irreducible types present in V. Also, let us use P, to denote the projection of F(X') onto the bth isotypic component V,, of F(2)) and let V, PxF(
22

---

Published as a conference paper at ICLR 2024
X x² H(K) = F& Féy x Fg V
With this definition in mind, we can extend the argument in Proposition C.11 to the nonlinear case if the F part of the decomposition + H. a F is a BiLLisssss injective map.
Theorem C.13. Suppose that there exists G-invariant kernel K on X for which the map F x (xx,-) )) is a -p-biiipschitz injective map, and define M(K) to be the set of all H((K)-contintou equivariant, multiplicity unlimited maps &b into a vector space V with a linear action. We use Fb to denote the map of X into b-th isotypic component of H((K). For any given क E .M(K), define I())= min L(D, 1p (P, where ((Y) is the set of all measurable maps from V X. with Vo being the bth isotypic component of (X)) and Also, define E(J) = EoeJ ExliEElllV
assume that there is some such that [ > 6. Then for any &1 and $ with Io, # Is, we have I((1)) > L( (f) Io, C Io, or (2) E(TT,) ) is sufficientll larger than {(T4.)
Proof. Using the similar logic in the linear case, establish the claim by showing that L(Φ) ==
ming C(2, V P PG) is bounded from above and below by
2 I( Σ (
B [V
(45)
for some Cj and Cu. Assume the same F used in Lemma C.II and let Fb be the 6th isotypic component of F. To show the lower bound, by BiLipschitz property of F we have
min L MPP) 2 cF min (g X) F (()+
(46)
CF min
(g - x)
(47)
2 CF
Σ
min
B||l Pig X) (g. x)11
(48)
M CF
Σ bgTo
Bllg- V3 - E.g VolVo: 6 [+]l
(49)
where in the last inequality, we used the optimality of conditional expectation and the fact that in the factorization & = F. μa (X) has the same sigma algebra as {Va: e Ip under the multiplicity unlimited setting. Because the formula above is of the same form as eq. 43.. we are done with the lower bound here with the same logic as in the linear case. As for the Upper bound, again we have from the BiLipschitz property that
L( = min L(D, (PP) S cF min
(50)
= Cp min Ell (g x))11
Σ
(51)
Now, noting that 2 = P F under multiplicity unlimited setting is injective on its image and writing Fg F to be the projection of F onto the representations corresponding to II. V can be chosen to an invertible map that maps the image of & back to its preimage so that, by using the fact that μs is invertible on Is component, we have (x:) F ( for E X.
To construct such v, write & F.FF and for some E X, consider n4 pa''( (+(x)), which is pos- sibly a set. Here, we may take a specific section Az(x) E F(X) of μa., and define v so that (((())) = F='Ag(a)) because F maps X' its image injectively. This way, FVD(()) = (x) and Fo (xx) us take the section such that the norm of its projection Pg (x) == Pg FV)(() is minimal amongst all members of pT p (#(z)) where PS is the projection to (Vi[Io). This way, we have [F)))()))] & [[)((=)]] whenever b & It.
Po Ap Because the choice of the section for each is arbitrary. let
As for when b E Ig. note by the definition (4 (z) = (xx) = [1) po and by Schur's 's lemma, (x) (z) u PoA(z)) Fo.).. Because μp is invertible on (x) by Schur's lemma and multiplicity- condition, this shows that Fo (x) = (x) because Po As) E (o)... Altogether we have Fo vD(x) Fo (x) as desired.
23

---

Published as a conference paper at ICLR 2024
With such V, have F, (T))) (x( whenever Io well. Thus, for such chosen V, L(Φ) may be bounded from above by
CF
Ellgo V3 x
(52)
Tg
Now, note that
Σ bgT
Bilgo Vo ( x])][ - Σ
Elllg Voll") (V
(53)
hgTT
2
E Elllg V
(54)
b
2
Σ E [3
(55)
bEIz
by the choice of the section chose to construct v. This would define the upper bound Eogi, V and the same logic as in the linear case follows from here.
This statement tells us that, if a certain set of isotypes captures significantly more dominant infor- mation than others, then NFT will prefer the very set of isotypes over others. Indeed, if ((Zg, is sufficiently larger than (ZZ.) ). will mean that the set of isotypes that O excluding from H (K) is so much larger in the dataset than those excluded by 4 that it is more preferable to choose $2 than 11 This way, the solution to the autoencoding problem determined by the contribution of each frequency space to the dataset.
24

---

Published as a conference paper at ICLR 2024
D DETAILS OF EXPERIMENTS
D.1 COMPRESSION OF DEFORMED DATA
In this section we provide the details of the signal compression experiment in Sec 5.1 along with the additional visualization of the reconstructed results.
Dataset: The dataset was constructed as described in Sec 5.1 and the velocity of the shift was chosen from the range [0, 64].
Models: For the encoder and decoder, MLP with two hidden layers used. The intermediate di- mensions and the activations of the network are as follows:
g-NFT (g known): Encoder 728-256-256-33 Decoder 32-256-256-11 128-128. activation=Rel.U. G-NFT (G known, but unknown): Encoder 128-512-512-32, Decoder 32-512-512-128. activa- tion=Tanh.
Visualization of the reconstructed signals:
(A) Blue: Reconstruction with NFT (g known). Red: noisy training data.
Reconstruction
valuss
trme
time
timee
time
B Blue: Reconstruction with DFT. Red: Noiseless test data.
DFT (Num freq 16 in complex)
MM
M
Figure 12: Reconsturction of NFT and DFT for nonlinearly transformed functions. Times these figures are scaled by 1/128.
As we can see in Fig 12 by the nonlinear transform t + t3 the left area around the origin is flattened, and the right area around = 1 is compressed. As result, the deformed signal has broader frequency distribution than the original latent signal (see Fig 3 also). The results in Fig [2] depict that the standard DFT, which is applied directly the deformed signal, fits the left area with undesirable higher frequencies, while fitting the right area with an over smoothed signal. The NFT accurately reconstructs the signal in all the areas.
D.2 1D SIGNAL HARMONIC ANALYSIS SEC 5.1
In this section we provide more details of the 1d signal experiment, along with the more detailed form of the objective we used to train the encoder and decoder. We also provide more visualizations of the spectral analysis (Fig 14).
The dataset was constructed as described in Sec 5.1 and the velocity of the shift was chosen from the range [0, 64), which suffices for the computation of the character inner product for real valued sequence of length 128.
Dataset:
Model: For both encoder and decoder, used the same architecture in D.I. However, the latent space dimension was set to be 10x x 16, which is capable expressing at most 10/2 frequencies.
25

---

Published as a conference paper at ICLR 2024
Frequencies learned from dataset
Frequencies of each block
1.0
est
biorkn
0.3
0.8
block1
block2
blockJ
0.5
0.6-
block4
0.4
0.2
0.0
0.0
0 10
20
30
40
50
60
10
20
30
40 50
60
Figure 13: The plot of ppr)] (Left) and the plot of [B; pyy)] for each block of the block- diagonalized M (tright) when the major frequencies are {8, 15, 22, 40, 45} and minor frequencies with weak coefficients are (18, 13). We can confirm in the plot that major spikes only occur at major frequencies, and that each block corresponds to single major frequency. Note that, when there are noise frequencies, they are slightly picked up in the overall spectrum of M (Left). However, as seen in See 5.1 each block in the block diagonalization of M contributes only slightly to the noise frequencies, distributing the noise over the entire M instead of corrupting specific dominant frequency. With this character analysis, we can identify the major frequencies almost perfectly by using an appropriate threshold (Figl) 14
Majon: 9 61 36 15 01, Weekn]69 59]
M5 16 27 17), Weaken|55 13]
1.0
3.5
0.
0.4
02-
M
14
49), Weaksu22.3
Major][18 Wesice|34
ROC
3.0.
TPR
04
0.5
3.5
0.0
0.2
0.6
0.6
FPR
Figure 14: (Left)Other visualizations of E(Mpll plotted against f on the deformed signal datasets generated with different sets of major/minor frequencies. On any one of these plots, we can iden- tify the major frequencies by simply looking for the set of fs for which ((MMp)) is above some threshold. (Right) ROC curve of the major- frequeefy identification with different thresholding val- ues, computed over 100 datasets with randomly selected 5 major frequencies and minor frequen- cies. Note that, on this setting of frequency detection, there are always 5/64 frequencies that are "positive". When normalized by (1-5/64), AUC was 0.97.
26

---

Published as a conference paper at ICLR 2024
Figure 15: Prediction of shift under fish-eye deformation. Each predicted sequence is placed above the ground truth sequence. The predictions computed V (M* (1/)(((x))) (ro)), where M is being regressed from the encoding of the first two frames (zn,
D.3 PREDICTION OF THE SHIFT UNDER FISHEYE DEFORMATION IN SEC 5.1
We provide the details of the experiment on fisheye-deformed shift in Sec 5.1 We also provide more visualizations of the predicted sequence (Fig
Dataset: For the fisheye prediction, the sequence of images was constructed from Cifar100 by first applying sequence of horizontal shifts with random velocity in the range of 16 pixels, and by applying the fisheye transformation (defisheye Contributors, 2019).
For the encoder and decoder, ViT architecture was used with mlp-ratio=4, depth=8, number of heads=6 and embed dimension = 256, and the model was trained over 200000 iterations with Adam (Ir=0.01). We decayed the learning rate linearly from 80000-th iteration. encoder and decoder in U-NFT, the transition matrix M from the first to the second frame was validated at the third and the fourth frame to compute the loss (Same as in eq. with T 2, but used Bill x2 (M, [21))
Model:
the training of
|a's (MM (21)) to optimize ( V))
D.4 DETAILS SO(2) EXPERIMENTS (SECTION 5.2
In this section we provide the details of the rotated MNIST experiment in Section 5.2 In this set of experiments, we used OOD task to compare the representation learned by the encoder & against other representation learning method as well as supervised methods. As representation learning methods, we trained an autoencoder (AE), SimCLR (Chen et al., (020). the invariant autoencoder (IAE) [Winter et al.] 2022) and MSP (Miyato et al. (022). We used two-layer CNN for both the encoder (and decoder) of these models. To evaluate these methods, we first trained the encoder & on the in-distribution training set, and then trained the linear classifier the out of distribution dataset by optimizing
Cross Entropyy So (mmaaW"" ((X)),))
with respect to W using the fixed 4. where Y is the label of X. In Figure reported the size of the out-of-distributiin dataset used to train W as num num-fmetuue-ddtte
For the supervised methods compare against NFT, used the two-layer CNN, C, (cyclic group) steerable CNN (Cohen and Welling 2017). and S0(2) steerable CNN Weiler et al. (2018). For the training of these models, simply trained the cross entropy loss the in-distribution training set,
27

---

Published as a conference paper at ICLR 2024
and fine-tuned the final softmax layer on the out-of-distribttion dataset to evaluate the OOD perfor- mance. In particular, we simultanesouly trained both W and the parameters of Net to minimize Cross Eroropyp So Lmaz (o X)), Y)
on the in-distribution set, and exclusively fine-tuned W on the
set.
For steerable networks, we implemented them based on esenn library (Cesa al. 2022) We used
the same architecture described in
ipynb
for CNN and hww.atttee/naadcoooooooeaaaattta
Eutorial http://awwaaaeeeeeeeeeeeeeeaeeaa
steerable cnns http::/paa.......aooeeoooneaaaan
for (0(2) CNN.
For g-NFT and MSP, we reshaped the dimensiinaa output of the encoder into a d, x dm matrix so that it is compatible with the group representation matrix M E Rd. xd, acting from left.
For g-NFT, because we know the irreducible representations of SO(2), we modeled the transition matrix in the latent space as representation (0) with frequency up to 4max 2. We thus used the same parametrization as in the compression experiment of See 1ec that we used x 1 identity matrix for the frequency 0. producing m2/ + 1) x (2lmax + 1) matrix. We then trained (L, V) with & having the latent dimension [g [1) We trained each model for 200 epochs, which took less than hour with a single V100 GPU. We used Adam optimizer (Loshchilov and
Hutter 2017) with 31 0.9 and B22 0.999.
The hyperparameter space and the selected hyperparameters for each method were as follows:
autoencoder leaming rate (LR): 1.7393071 130001138611--0 weight decay (WD): 2.3502942694559366-009
supervised LR: 0.0028334155536699777
WD: 4.88 4.881328998770217--77
SimCLR (Chen et al. 2020) projection head dim: 176, temperature: 0.32968516540864811
LR: 11000574900951993588
WD: WDD:277571155777055588
IAF WWnter et al. 2022) LR: 0.00137771092779155442
WD:: 1.2144236696665408--00
Steerable(C_N) angular resolution n: 28,
L.:: 0.0077362726795999055. WD:
6.5696449997770058---66
Steerable(SO02) maximum frequency: 4, LR: 0.0030013815048863377,,,, WD: 7.337868377360226--66
MSP (Miyato et al. 2022) d. 9, latent dimension dm 252, LR:
111 1.2124992115519022:.05
WD: 0.0163884899855
g-NFT maximum frequency /maxi 2, dm: 180, LR: 0.0005439955566956466 WD: 0.0006491622240310113
The hyperparameters of each baseline, such as the learning rate and maxx were selected by Optuna (Akiba et al. 2019) based the test prediction error MNIST.
Although IAE is designed to be able to estimate each group action g. we supervised g in this experiment to make fair comparison to other g-supervised baselines. As the loss function for g-NFT, we used the sum of the autoencoding loss defined in Theorem 4.2 together with the alignment loss: [[ X) ( (X)) This loss function promotes the equivariance of the encoder tD See Fig 16 for more visualizations of the reconstruction with different values of Imax-
D.5 DETAILS OF 3D EXPERIMENTS (SECTION 5.2
In this section we provide the details of the experiment the rendering of 3D objects (ModelNerIO-SO3. Complex BRDFs, ABO-Material). All the experiments herein were conducted with 4 V100 GPUs.
10SS The dataset in this experiment contains the object-centered images of 4,899 CAD model B
Each image was rendered with the uniformly random camera position g E SO(3).
https: / thub. com www/w/osh//eeecceemmoooee
Rotation
28

---

Published as a conference paper at ICLR 2024
The CAD models were separated into training set (100 images for each CAD model) and test set (20 images for each). We downscaled the image resolution to 64 x 64. We used the vision Transformer (ViT) for the encoder and the decoder. The encoder consisted of 9 blocks of 512-dimensional embedding and 12 heads. The decoder consisted of blocks of 256-dimensional emhedding and 6 heads. The patch size
4 and MIP-rrtio was 4 for both networks. We set dm = 64 and (max = 8. The encoder output was converted to the de x dm matrix by EncoderAdapter module defined by the following PyTorch pseudocode.
from torch import from einops layers. torch import Rearrange
class EncoderAdapter ( mn Module):
def -_init__ (self, num hum_patches,
mim, d_a, d_m):
self. net mn. Sequential
nn Linear (embed_diig dim, embed_din dim // 4),
Rearrange ('b -> b nn')
nn Linear (num_patcher nn .E.GEL))
num_patches // 4),
nn nn. LayerNoroom ([embed dim // 4, num num_patches // 4]), Rearrange ( bb
b (e n)").
nn Linear (embed _iii_ddim__ * num patchhht // 16, a_ a d_), Rearrange ( ('b (m a) -> b m m=d_m),
def forward (self, encoder encoder_output). return self net (encoderroutpuul
We also used similar network before the decoder adjust the latent variable. We used Adam optimizer with batch size 48, learning rate 10 10-4, and weight decay 0.05. We didn't use the alignment loss described in Sec D.4 We trained the model for 200 epochs, which took days.
Complex BRDFs The dataset this experiment contains the renderings of ShapeNet objects from evenly placed 24 views") The camera positions in this experiment are constrained on a circle with fixed radius the group action of moving the camera position can be interpreted as the action of SO(2) rather than SO(3). We used 80% of the objects for training and 20% for testing. Following the terminology of the ViT family] We used V for the encoder and ViT-S/16 for the decoder. The learning rate was 3e-4. We trained the model for 100 epochs, which took 1.5 days. Other settings were the same as the ones we used the ModelNet experiment.
The dataset in this experiment contains 2.1 million rendered images of 147,702 product items such as chairs and sofa The images were rendered from 91 fixed camera positions along the upper icosphere. We reduced the original 512 x 512 resolution to 224 x 224. The dataset was randomly partitioned into training (80%), validation (10%), and test (10%). The encoder and decoder architectures were the same as for the BRDFs experiment. We trained the model for 400 epochs with batch size 36, which took 12 days.
ABO-Material
E SUPPLEMENTARY EXPERIMENTS FOR ROTATED MNIST
To further investigate the capability of NFT, we also extended our experiments of D.4 to consider the cases of image occlusion as well. Specifically, we zeroed out three quadrants of the input images, so that only one quarter of the image is visible to the models at all time. This is more challenging task because the actions are not invertible at pixel level. We trained the encoder and decoder in the NET framework on this dataset, and conducted the same experiment in D.4 As
compl ex brdf
29

---

Published as a conference paper at ICLR 2024
Input
A
I
Prediction w/ freq 0 to
0
6 4
4
8
8 4
8
8 4
16
4
Target
PIPY 8 4
Figure 16: Reconstruction conducted with different values of Iman (1 16). Higher frequencies promote sharper reconstruction of the images.
illustrated in Fig 17 the NFT-trained feature consistently demonstrated competitive performance (g-NFT NFT, MSP).
MNIST
FMNIST (OOD)
KMNIST (OOD)
75
accuracy
50
25
1e2
1e3
1e4
1e2
1e3
1e4
1e2
1e3
1e4
num num_ www.ffnneeeaaate data
supervised
SimCLR
MSP
g-NFT
AE
IAE
MSP-norm
g-NFT-norm
Figure 17: Linear probe results on Rotated MNIST with image occlusions.
30

---

Published as a conference paper at ICLR 2024
Freq=O
5 IN
Figure 18: Spherical Decomposition of 3D point cloud representations of the same chair object in the lefumost panel of 8 conducted in the way of (Skibbe et al. 2009). Although our experiment on the ModelNet dataset does not access the 3D voxel information nor include such structure in the latent space, each frequency Fig 8 has much resemblance the result of the point-cloud derived harmonic decomposition. See the movie folder in the Supplementary material for the movie rendition of this visualization (modelnet_chanle Sahaarrsphericalggg.
MNIST
FMNIST (OOD)
KMNIST (OOD)
100
75
accuracy
50
25
1e2
1e3
1e4
1e2
1e3
1e4
1e2
1e3
1e4
num Sum_finetunn_dart data
Supervised
SimCLR
Steerable(C.N)
MSP
g-NFT
AE
IAE
Steerable(SO2)
MSP-norm
g-NFT-n
Figure 19: The result of the rotMNIST experiment (Fig including the MSP and MSP-norm.
31

---

Published as a conference paper at ICLR 2024
optimization
(4) Decode
2
11
(2)
space
((M))()))
(5) Compute the lose
Given observations
oig pin
32
zo
2
feature
|||Л||I| AF ((())
4H
((26)
((2)))
Validation
Figure 20: The meta sequential prediction (MSP) method used for U-NFT.
(1) Encode latent features 9
(Arig)
Given observations g or
Migi
(4) Compute the loss
Mi g
Miste:ai 0(g-a)
Mig)0())
Figure 21: The training algorithm for g-NFT.
32


Reviews:
[{"review_id": "iubJjgb2j4", "reviewer": "Reviewer_uMSh", "summary": "The main motivation behind the paper is the following: In pre-dominant approaches to equivariant learning, the underlying group and group action is assumed to be known. There are various approaches which then construct linear-equivariant layers followed by a suitable non-linearity and then stack them together. However, often the underlying group or action might not be known. The starting point of the paper is to consider the usual (and then the group) Fourier transform and its well-known equivariance relation. The idea is to construct a non-linear extension where the underlying action is not known, but has similar behaviour as the usual FT. This method dubbed as the non-linear Fourier transform affords finding a relation similar to the usual Fourier transform without having access to analytically tractable knowledge of the group action. It however, does require access to transformed versions of the data to set up a loss function that might be minimized. Various basic properties of this transform and shown. The work heavily relies on ideas from Miyakato et al. and Keller and Welling.", "questions": "Would appreciate if the authors could elaborate on the weaknesses mentioned.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea is quite natural and convincing. It does seem more suitable for time series and video data, due to the lack of availability of suitably transformed data in usual setting, but for a framework it seems reasonable. \n- It relies on the natural idea of using an invariant kernel mapping (while not cited, there is quite some work on non-neural variants of this, such as by Reisert and Bukhardt, JMLR 2007), and shows how the idea is sensible. \n- The idea of finding a data-dependent spectrum is appealing.", "weaknesses": "-- The primary weakness of this paper is that it relies on data triplets of tuples that gives a sense of the underlying group action. This is a weakness, since such data isn't easily available in most real-world settings. This is also the reason why the paper pivots to applications in time series in the experimental section. \n-- The applications presented don't seem compelling, and more or less follow a similar pattern as in the work of Miyato et al. 2022. \n-- Related to the above; The experimental section is relatively toy, which is a weakness, given the generality of the pitch of the paper.\n\nOverall I am just not clear about the applicability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KnTQXQUN7R", "reviewer": "Reviewer_XpFk", "summary": "This paper proposes Neural Fourier Transform for equivariant representation learning that deals with the cases in which the group action on the data may be nonlinear or implicit. The paper investigates three scenarios based on the level of prior knowledge on the group and group element that acts on each data. Experimental results are provided to demonstrate the effectiveness of the proposed method in nonlinear spectral analysis, data compression, image classification, and novel view synthesis from a single 2D rendering of a 3D object.", "questions": "1. With respect to the work of Miyato et al., (2022) that considers the properties of the learned model to be intra-orbital homogeneity or full equivariance, which of these properties that the proposed model owns in each setting (U-NFT, G-NFT, g-NFT) ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper has a clear motivation\n* The proposed method is accompanied with interesting theoretical results\n* Limitations are properly discussed\n* Codes are provied to support the main paper", "weaknesses": "* Overall, the paper is well written but it probably needs more checks for possible typo errors (there is a minor typo at the end of Section 2 \"scaler-value\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfMsj3rWJO", "reviewer": "Reviewer_g5qF", "summary": "This manuscript presents the Neural Fourier Transform (NFT), a novel framework for learning latent linear group actions without requiring explicit knowledge of how the group (or even what group) acts on the input data. The paper rigorously explores the theoretical underpinnings related to the existence of linear equivariant features. Moreover, empirical results are provided to substantiate the utility of NFT in a range of applications, each distinguished by differing levels of prior knowledge about the input group action.", "questions": "Please refer to the previous comment.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and well structured. Many remarks and explanation are provided, making it easily comprehensible\n2. The core concept—learning a latent group representation space by utilizing triples of group-acted inputs, without the need for prior information on the group—is both intriguing and innovative.\n3. The empirical findings presented are both compelling and substantiate the framework's utility.", "weaknesses": "1. I have only one question/comment: in Theorem 4.2, $(\\Phi^*, \\Psi^*)$ is the minimizer of\n$$E_{g\\in G}[\\|g\\circ X - \\Psi \\Phi (g\\circ X)\\|^2]$$\nDoes this merely imply that $\\Psi \\circ\\Phi $ is a \"good\" autoencoder on the group orbit, instead of transforming $X$ into a linear space on which there is a linear $G$ action? Should it be the following instead?\n$$E_{g\\in G}[\\|g\\circ X - \\Psi \\circ M(g) \\circ \\Phi ( X)\\|^2]$$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8NHwsOKGyf", "reviewer": "Reviewer_ef5j", "summary": "This paper proposes Neural Fourier Transform as a general approach for equivariant representation learning. The goal is to learn a latent linear action of the group, similar to how the DFT encodes linear representations of the shift group. Many symmetries (e.g. fisheye transformations) are obvious symmetries to the human eye, but there is no easy way to represent them using classic theory. The learning task is decomposed into three cases, where the network has to match using a linear action in latent space and a nonlinear action in data space. Theoretical results are verified on several tasks.", "questions": "- How did you combine the autoencoder framework with the downstream (e.g. classification) objective? Did you have to weight the two objectives? How did you find the hyperparameters?\n- Do you have an idea of how to get the tuples of nonlinear group actions in the wild?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles the interesting task of representing nonlinear image transformations in a neural network as linear group actions. This is an interesting and valuable direction.\n- The paper finds a simple yet effective strategy for solving the problem using tuples of $g$-transitions.\n- While restricted to synthetic settings, the experiments are promising.", "weaknesses": "- The experiments could have been detailed a bit more clearly. I.e., what are the objectives, the loss functions used, the in/outputs to the model, etc. \n- I find the title \"Neural Fourier Transform\" a bit misleading, as the Fourier transform is an extremely general signal processing tool, where in this work it is used as an example of finding linear representations of a group.\n- It would have been nice to have some results on a task that is a bit more \"in the wild\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)