Invariant Feature Coding using Tensor Product Representation
Authors: YUSUKE Mukuta
Tatsuya Harada
Venue: Accepted by TMLR ()

Abstract: In this study, a novel feature coding method that exploits invariance for transformations represented by a finite group of orthogonal matrices is proposed. We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier using convex loss minimization. Based on this result, a novel feature model that explicitly considers group action is proposed for principal component analysis and k-means clustering, which are commonly used in most feature coding methods, and global feature functions. Although the global feature functions are in general complex nonlinear functions, the group action on this space can be easily calculated by constructing these functions as tensor-product representations of basic representations, resulting in an explicit form of invariant feature functions. The effectiveness of our method is demonstrated on several image datasets. 

Published in Transactions on Machine Learning Research (06/2023)
Invariant Feature Coding using Tensor Product Representa- tion
Yusuke Mukuta
muksto@miit.uttoop.....
The University of Tokyo, RIKEN
Tatsuya Harada
The University of Tokyo, RIKEN
Reviewed on OpenReview: Attps: // openreview. net/ met/ forum? id= wo32J0dQuh
Abstract
In this study, a novel feature coding method that exploits invariance for transformations represented by finite group of orthogonal matrices proposed. We prove that the group- invariant feature vector contains sufficient discriminative information when learning línear classifier using convex loss minimization. Based on this result, novel feature model that explicitly considers group action is proposed for principal component analysis and k-means clustering, which are commonly used in most feature coding methods, and global feature functions. Although the global feature functions are in general complex nonlincar functions, the group action on this space can be easily calculated by constructing these functions as prosurtproduct representations of basic representations, resulting in an explicit form of invariant feature functions. The effectiveness of our method is demonstrated several image datasets.
1 Introduction
Feature coding is a method of calculating a single global feature by summarizing the statistics of the local features extracted from a single image. After obtaining the local features { E Rdhoos a nonlinear function F and F n E E Rdgiahal is used as the global feature. Currently, activations of convolutional layers of pre-trained convolutional neural networks (CNNs), such as VGG-Net (Simonyan & Zisserman 2014), are used as local features, to obtaín considcrable performance improvement (Sánchez et al. 2013 Wang et al. 2016). Furthermore, existing studies handle coding methods as differentiable layers and train them end-to-end obtain high accuracy (Arandjelovic al. 2016: Gao al. 2016 Lin al. 2015 Thus, feature coding is a general method for enhancing the performance of CNNs.
((x,)
The invariance of images under geometric transformations is essential for image recognition because com- pact and discriminative features can be obtained by focusing on the information that is invariant to the transformations that preserve image content. For example, some researchers constructed CNNs with more complex invariances, such as image rotation (Cohen Welling. 2016 2017 Worrall et al., 2017). and ob- tained a model with high accuracy and reduced model parameters. Therefore, we expect to construct a featuro-coding method that contains highly discriminative information per dimension and is robust to the considered transformations by exploiting the invariance information in the coding methods.
In this study, we propose novel feature-coding method that exploits invariance. Specifically, assume that transformations T that preserve image content act as finite group consisting of orthogonal matrices on each local feature 2n. For example, when concalenating pixel values in the image subregion as a local feature, image flipping acts as a change in pixel values. Hence, it can be represented by a permutation matrix that is orthogonal. Also, many cxisting CNNs, invariant complex transforms, implement invariance by restricting convolutional kernels such that transforms as permutations between filter responses, which be represented by the orthogonal matrices. Therefore, the orthogonal assumption is not strong one.

---

Published in Transactions on Machine Learning Research (06/2023)
Input Image
local feature
Local Feature Space
Encoded Feature Space
Invariant Feature Space
extraction
nonlinear mapping
F(r)
projection
invariant space Pss
transformation that perserves image contents
orthogonal
orthogonal
Pi(P(e)) A(P((()))))
transformation or (((
transformation st(a)
[9)
F(x(g)x) ((())
Figure 1: Overview of the proposed feature coding method. In the initially constructed global feature space, x' (x)) = (g)()))) holds for some orthogonal æ'. Subsequently, the projection is applied to the trivial representation of Pi to obtain the invariant global feature.
Ignoring the change in feature position, because global feature pooling is being applied, construct a nonlinear feature coding function F that exploits T. Our first result is that when learning the linear classifier using L2-regularized convex loss minimization on the vector space, where T acts as an orthogonal matrix, the learned weight cxists in the subspace invariant under the T action. From this result, we propose a guideline that first constructs a vector space which acts orthogonally on F(r,) to calculate the T-invariant subspace.
On applying theorem to feature coding, two problems occur when constructing the global feature. The first problem is that, in general, F exhibits complex noulinearity. The action of T the CNN features can be easily calculated because CNNs consist of linear transformations and point-wise activation functions. This is not the case for feature-coding methods. The second is that has to be learned from the training data. When we encode the feature, we first apply principal component analysis (PCA) on 2n to reduce the dimension. Clustering is often learned using the k-means or Gaussian mixture model (GMM) and F is calculated from the learned model. Therefore, we must consider the effect of T on the learnod model.
To solve these problems, exploit two concepts of group representation theory: reducible decomposition of the representation and tensor product of two representations. The former is the decomposition of the action of T on into a direct sum of irreducible representations. This dccomposition is important when constructing a dimensionality reduction method that is compatible with group actions. Subsequently, the tensor product of the representations calculated. The tensor product is a method by which we construct vector space where the group acts on the product of the input representations. Therefore, constructing nonlinear feature functions in which group action can be easily calculated is important.
Based on these concepts, propose novel feature coding method and model training method that exploit the group structure. We conducted experiments on image recognition datasets and observed an improvement in the performance and robustness to image transformations.
The contributions of this study are as follows:
We prove that when training the lincar classifier on the space where the group acts orthogonally, the learned weight lies in the invariant subspace.
We propose group-invariant extension to feature modeling and feature coding methods for groups acting orthogonally on local features.
We evaluated the accuracy and invariance of our methods on image recognition datasets.
2

---

Published in Transactions on Machine Learning Research (06/2023)
2
Related Work
2.1 Feature coding
The covariance-based approach models the distributions of local features based on Gaussian distributions and uses statistics as the global feature. For example, GLC (Nakayama et al. 2010a) uses the mean and covariance of the local descriptors as the features. The global Gaussian NNaaayama et al., 2010b) method applies an informatiin-ggomeettr metric to the statistical manifold of the Gaussian distribution as a similarity measure. Bilinear pooling (BP) (Lin et al., 2015) uses the mean of self-products instead of the mean and covariance, but its performance is similar. The BP is defined as F = vec NV Σm= =n where vec (A)
denotes the vector that stores the elements of A. Because of the simplicity of BP, there are various extensions, such as Lin & Maji 2017): Wang et al. (2017); Gou et al. (2018): Yu et al. (2020): Koniusz & Zhang (2021)
Song et al. 22021 2023) which demonstrate better performance than the original CNNs without a feature
coding module. For example, improved bilincar pooling (iBP) uses the matrix square root as a global
feature. These works mainly focus on the postprocessing of the bilinear matrix given covariance information; whereas, our motivation is to obtain the covariance information that is important to the classification. We use invariance as the criterion for feature selection.
The vector of locally aggregated descriptors (VLAD) (Jégou el. al. 2010: is a Cdioca-dimensiin vector that uses k-means clustering with C referring to the number of clustering components and consists of the sum of the differences between each local feature and the cluster centroid Me to which it assigned, expressed as Fe Exnes, (zn, Pe); Se is the set of local descriptors that are assigned to the c-th cluster. The vector of locally aggregated tensors (VLAT) (Picard & Gosselin, 2013) is an extension of VLAD that exploits second-order information. VLAT uses the sum of tensor products of the differences between each local descriptor and cluster centroid pa: Fe vec (E.,es, (In, [e)(I,, mo)T T - -Ta), where To is the mean of (z, of all the local descriptors assigned to the e-th cluster. VLAT contains information similar to the full covariance of the GMM. The Fisher vector (FV) (Sánchez et al. 2013 also exploits sccond-order information but uses only diagonal covariance. One work has also exploited local second-order information
with lower feature dimensions using local subspace clustering DDiit & Vasconcelos) 2016).
2.2 Feature extraction that considers invariance
One direction for exploiting the invariance in the model structure is to calculate all transformations and subsequently apply pooling with respect to the transformation to obtain the invariant feature. Tl-pooling Laptev et al.] 2016) first applies various transformations to input images, subsequently applies the same CNNs to the transformed images, and finally applies max-pooling to obtain the invariant feature. Anselmi itt al. (2016) also proposed a layer that averages the activations with respect to all considered transformations. Rot Eqte MMaroos et al. 2017) calculates the vector field by rotating the convolutional filters, lines them up with the activations, and subsequently applies pooling to the vector fields to obtain ootation-ivvarial features. Group cquivariant CNNs (Cohen & Welling. 2016) construct network based on the average of activations with respect to the transformed convolutional filters.
Another direction is to exploit the group structure of transformations and construct the group feature using group representations. Harmonic networks (Worrall et al.) 2017] consider continuous image rotation in constructing a layer with spherical harmonics, which is the basis for the representation of two-dimensional rotation groups. Steerable CNNs (Cohen & Welling 2016) construct filter using the direct sum of the irreducible representations of the D4 group, to reduce model parameters while preserving accuracy. |Weiler et al. 20188) implemented steerable CNNs the weighted sum of predefined equivariant filters and extended it to finer rotation equivariance. Variants of steerable CNNs are summarized in (Weiler & Cesa. 2019 Jenner & Weiler (2022) constructed the equivariant layer by combining the partial differential operators. These works mainly focus on equivariant convolutional layers, whereas we mainly focus on invariant feature coding given equivariant local features. Kondor et al.) (2018) also considered tensor product representation to construct DNNs that are cquivariant under 3D rotation for the recognition of molecules and 3D shapes. While Kondor et al. (2018) directly uses tensor product representation as the nonlinear activation function, we use tensor product representation as the tool for extending the existing coding methods for equivariance.
3

---

Published in Transactions on Machine Learning Research (06/2023)
Therefore, we introduce several equivariant modules, in addition to tensor product representation. The difficulty of invariant feature coding is in formulating complex nonlinear feature coding functions that are consistent with the considered transformations.
Furthermore, there exists the approach to learn invariance from training data instead of deciding the invari- ance beforehand. Rao & Ruderman (1988); Miao & Rao (2007) learned the group that preserves training data unsupervisedly by considering reconstruction loss between the original image and the transformed im- age assuming Gaussian noise. ? used the matrix exponential and applied the matrix backpropagation to optimize the reconstruction loss. Sohl-Dickstein et al. (2010) proposed an adaptive smoothing method to obtain good local minima. [Benton et al. (2020) tried to learn the neural network to be invariant under data augmentation by using consistency regularization. Lin et al. (2021) trained the auto-encoder parameterized by the Lie group. Chau et al. 2022) combined multiple 2D rotation groups with sparse coding to model the transformation group.
Another approach similar to ours is to calculate the nonlinear invariance with respect the transformations. Reisert & Burkhardt (2006) considered the 3D rotation invariance and used the coefficients with respect to spherical harmonics as the invariant feature. Kobayashi et al. (2011) calculated rotation-invariant feature using the norm of the Fourier coefficients. Kakarala (2012) used bilinear feature with respect to 3D rotation group. Morère et al. (2017) proposed group-invariant statistics by integrating all the considered transformations, such as Eq. (1) for image retrieval without assuming that the transformations act linearly. Kobayashi (2017) exploited the fact that the eigenvalues of the representation of flipping are #1 and used the absolute values of the coefficients of eigenvectors as the features. Ryu et al. (2018) used the magnitude of two-dimensional discrete Fourier transformations as the translation-invariant global feature. Compared to these approaches that manually caleulate the invariants, invariants given the transformations, in fact calculating all the invariants after constructing the vector space on which the group acts. Therefore, our method exhibits both versatility and high classification performance.
method can algorithmically calculate the
In summarr, our work is an extension of existing coding methods; where the effect of transformations and all the invariants are calculated algorithmically. Compared with existing invariant feature learning methods, we mainly focus on invariant feature coding from the viewpoint of group representation theory.
3 Overview of Group Representation Theory
In this section, we present an overview of the group representation theory that is necessary for constructing our method. The contents of this section are not original and can be found in books group representation theory (Fulton& & Harris 2014). We present the relationship between the concepts explained in this section and the proposed method as follows:
Group representation is the homomorphism from the group element to the matrix, meaning that the lincar action considered as the effect of transformations.
Group representation can be decomposed into the direct sum of irreducible representations. The linear operator that commutes with group action is restricted to the direct sum of linear operators between the same irreducible representations. These results are used to formulate PCA that preserves equivariance in Section 4.2. Further, this irreducible decomposition also used when calculating the tensor product.
We obtain a nonlinear fcature vector for the group that acts lincarly by considering tensor product representation. Furthermore, the invariant vectors can be determined by exploring the subspace with respect to the trivial representation 1, which can be obtained by Eq. (1. These results can be used to prove Theorem [ and construct the feature coding in Section 4.3
3.1 Theory
Group representation When set G and operation o: G G G satisfy the following properties:
4

---

Published in Transactions on Machine Learning Research (06/2023)
is associative: 91. 92, 9s G satisfies ((g1 92) 93) (g1 (92 gg))
The identity element e E exists and satisfies g g e - for all g E G
All g G contain the inverse g² that satisfies g gog - ogee
the pair = o) is called a group. The axioms above are abstractions of the properties of the trans- formation sets. For example, set consisting of 2D image rotations associated with the composition of transformations form a group. When the number of elements in G written as G is finite, 9 is called finite group. As mentioned in Section we consider finite group.
Now, we consider complex vector space Cd to simplify the theory; however, in our setting, the proposed global feature is real. The space of bijective operators on Cd can be identified with the space of x regular complex matrices written as GL C), which also a group with the matrix product as the operator. The homomorphism x from & to GL(d, C) is the mapping ग : G GL(d, C), which satisfies
For 91.92 E G, (g)) o x (g)) = (gi o 92)
(e)) laxd
and is called the representation of G on C², where Laxd denotes the d-dimensional identity matrix. The space in which the matrices act is denoted by (T) The representation that maps all g E G to laxd is called a trivial representation. The one-dimensional trivial representation is denoted as When all <(g) are unitary matrices, the representation called a unitary representation. Furthermore, when the r(g) are orthogonal matrices, this is called an orthogonal representation. The orthogonal representation is also a unitary representation. In this study, transformations assumed to be orthogonal.
Intertwining operator For two representations (x, Cd) and (I'' ((), a lincar operator A Cd + Cd' is called an intertwining operator satisfies 7'(g) o A = A Tg)). This implies that (T', AC4) is also representation. Thus, when applying linear dimension reduction, the projection matrix must be an in- tertwining operator. We denote the space of the intertwining operator Homg(T, x'). When a bijective A Homg (T, x') exists, we write x'. This implies that the two representations are virtually the same and that the only difference is the basis of the vector space.
Irreducible representation Given two representations æ and a, the mapping that associates g with the matrix which we concatenate (g) and (g) in block-diagonal form is called the direct sum representation of and a, written as ø. When the representation is equivalent to some direct sum representation, is called a completely reducible representation. The direct sum is the compasition of the space on which the group acts independently. Therefore, a completely reducible representation can be decomposed using independent and simpler representations. When the representation is unitary, the representation that cannot be decomposed is called an irreducible representation, and all representations are equivalent to the direct sum of the irreducible representations. The irreducible representation 7, is determined by the group structure, and स is decomposed into 1171 T2yT Br.,iTTr, where T' the number of different irreducible representations and net, is n, times direct sum of Tt. When denote the characteristic function of y X=(g) = TT((=g)), can calculate these coefficients n4 [ Egec Xx (g)x, (g). Furthermore, the projection operator P,
nyT; is calculated P.. dim(T:) ले Egea Xr. Xr, (g) Specifically, because X1 (g) 1, we calculate the projection to the trivial representation using
P
Σ (())-
G
gEG
This equation reflects the fact that the average of all T(g) is invariant to group action. Schur's lemma indicates that Homg T(,)= = (0) if 4 + la and Homa(T, 72) CA for some matrix A.
5

---

Published in Transactions on Machine Learning Research (06/2023)
Original Feature Space
Decomposed Feature Space
Decomposed Feafure Space
flipped image
y
Invariant Classifier
original
2
=
mip
train classifer
rotatitoo
flip
TL
x(6) (0
m) 9)
1
(o 1) mim) -)
0) 2)
vimn ( 0 -1)
Figure 2: Example of the irreducible decomposition and learned classifier.
Tensor product representation Finally, temsor product representation is important when constructing nonlinear feature functions. Given π and o, the mapping that associates g with the matrix tensor product of x (g)) (g) the representation of the space of the tensor product of the input spaces. We denote the tensor product representation л a The tensor product of the unitary representation is also unitary. The important properties of the lensor represenlation
as follows: (i) is distributive, where (I @ m2) @ (m3 @ 74) ( @ πT) + (I2 πa) @ (IT & ma) @ (mo @ TT), and (ii) Xago (g) Xx (g) (g). Thus, the irreducible decomposition of tensor representations can be calculated from the irreducible representations.
3.2
Illustrative example
The group representation theory described above is visualized in a simple setting in this section.
Group consisting identity mapping and image flipping. We consider group consists of identity mapping e and horizontal image flipping m. Because the original image was obtained by applying image e,
flipping twice, it follows that following three propertics of a group by setting m = m:
m,
m,
This definition satisfies the
is associative: 91, 92. Is G satisfy ((g) o 92) ga) (gi o (g2 ga)).
The identity element G exists and satisfics g g cog for all g G.
All G contain the inverse g- that satisfy gog-l g =c.
The irreducible representations are т. (e) 1, (m) and T-1 (e) (mmmmm) -11 This is
proven follows: the defined above satisfies the representation conditions
For 91.92 G, T(91) o ( ) = T(g1 g2).
r(e) ldxd-
trivially irreducible. When the dimension of the representation space is greater than 1, :T(e)= = laxd and ((m) = A E C). We apply cigendecomposition of A to obtain A = S-1AS, where A is diagonal, and we denote the i-th diagonal elements as A, Because π = S-1 S and (m)) = AAS, π is decomposed as a direct sum representation of n,(e) = 1 and π. (.m)) = X; for each i-th dimension, this representation not irreducible. Thus, the irreducible representation must be 1-dimensional. Moreover, T(m)² x(e) 1. Therefore, x(m) is or -1 and can be deomposed into the direct sum of and T-1-
where 1-dimensional representations

---

Published in Transactions on Machine Learning Research (06/2023)
Image feature and its irreducible decomposition. As example of the image feature, catenation of the luminosity of two horizontally adjacent pixels as a local feature and apply average pooling to get global feature. Thus, the feature dimension is 2. Since image flipping changes the order of the pixels,
con-
0
it permutates the first and second elements in feature space. Therefore, the group acts as (T)) 0 1)
((m) 0 .) as plotted in the left figure of Figure By applying orthogonal matrices calculated by eigendecomposition, the feature space written in the center of Figure 2 is obtained. In this space, the group acts as r(e)= = () 9) x(m) = ) Thus, this is the irreducible decomposition of x into Ti $ T-1 defined by the Irreducible representation paragraph in the previous subsection. Note that in the gen- eral case, each representation matrix is block-diagonalized instead of diagonalized, and each diagonal block becomes more complex, like those in Tables 1 and
Tensor product representation. Subsequently, consider the "product" of feature spaces to get non- linear features. Two feature spaces (h2 (i)) and (,, hit) with acting as T(e) () 0 ग ((m)
(i )) on both spaces are considered. Any imput spaces can be used whenever these spaces and satisfy the above obtain bilinear pooling, The tensor product of these spaces becomes 4-dimensional vector space consisting of hah h h and h. Since permutates hg and hg hy and hg simultaneously, m acts a permutator of high and
condition. For example, we use the same feature space for (h2, h2) and (hg,,,)
0
0
0
0
0
1
0
12 Wig and high simultaneously. Thus, x(m) is writlen as T(m)
Moreover, tensor
1
0
0
0
0
0
product space is also the group representation space. This space also be docomposed into irreducible
0
0
0
1
0
0
0
0
1
0
0
0
0
representations, such as x(e)=
and ((m) =
0
0
0
-1
0
0
0
0
0
0
-1
0
0
0
0
1
0
0
Intertwining operator Given (,, C4)
and (T', C²)
where T(e) =
and T(m) =
0
0
0
0
0
0
0
0
and re()) = 9). + 7'(m) = (0 2)) intertwining operator from a to T' maps
-1
0
-1
Ti elements to elements and T-1 elements to T-1 elements and therefore is represented by the form where 0gys are complex.
(11
012
0
0
423
221
4 Invariant Tensor Feature Coding
In this section, the proposed method is explained. Our goal was to construct an effective feature function F = E F(x,), where the transformations that preserve the image content act as finite group of orthogonal matrices on In Hence, we prove a theorem that reveals the condition of the invariant feat with sufficient discriminative information Section [.1 Subsequently, the feature modeling method necessary for constructing the coding in Section
4.2 explained. Our proposed invariant feature function is described we discuss the limitations of this theorem and extend under more general 4.5 the effectiveness of the proposed method is discussed in end-to-end setting.
in Section 4.3 In Section 4.4
assumptions. In Section

---

Published in Transactions on Machine Learning Research (06/2023)
4.1
Guideline for the invariant features
4.1.1 Theory
First, to determine what an effective feature 18, we prove the lollowing theorem:
Theorem 1. We denote by C the of image categories, W [Rdx[c] the linear classification weight whose e-th column vector corresponds to the weight for c-th category, I| llE the Frobenius norm of the matrix and I: IRC xC - R be the loss function conver with respect to the first argument. Let the finite group act orthogonal representation π on R and preserve the distribution of the training data {Vums 4m) }M m==1 E I xC,
which implies that {(Em Vm J E R x C exhibits the same distribution as {( Vm m== E R x C for any 9- The solution of the L-gregulorized convex loss minimization.
arg min WERAXICI 1I
M Σ
I(W
Um Fm.)
(2)
is G-invariant, implying that (g) = W for any 9 and therefore P.W W.
The G-invariance of the training data corresponds to the fact that does not change the image content. From another viewpoint, this corresponds to data augmentation that uses the transformed images as additional training data. For example, given the original dataset {(u., yo)] and a constructed augmented dataset [1ms )m))m JM as {(T(g)). 3o)1o=1,gc0 this augmented dataset invariant under the action of =(g) for any 9- We intend to this theorem by regarding the global feature F as Um, but it is applicable whenever the assumption holds. The proof is follows.
0. This is because if we assume that Egen T(g) A * 0, there exists e that satisfics Av & O, and CAAv is onc-dimensional G-invariant completely reducible. We denote the n,T, elements of W and Um as ((t) and (t) respectively, where the decomposition of W is applied column-wise. It follows that W Σ W(t) and V' Σ v! Furthermore, since W(')s and (tt) orthogonal among different ts. It follows that,
Proof. The non-trivial unitary representation satisfies Egea
subspace. violates the irreducibility of w. As (xx, Rd) is unitary representation,
s for different ts lie in a different subspace with respect to the irreducible decomposition, they are
M
Σ
1(W
Ums 3m)
M
m=1
M
T
T
Σ
Re
W(t))
M
m=1
M
T
G
Σ
Re
Σ
(w()))
-m
gEG
Σ
M
Σ
Re
(T(l))(l)))
T
3m
G
M
GEG
Σ Re
1
Σ (g)
(1)
G
3m
BEG
1 M
T (INi)))
(11) 3m
where Re the real part of complex number and W() and ii are the trivial representation elements under the irreducible decomposition of W and Um respectively, which be obtained by applying the

---

Published in Transactions on Machine Learning Research (06/2023)
projection matrix calculated by Eq. 1T). The first equation originates from the orthogonality of W and um's amoug different ts; the second equation comes from G-invariance of the training data such that ΣM I f( Vm) = Σm f(m(g))mm Vm.) holds for any f.g from the assumption; and the third comes from the unitarity of (g). The inequality comes from the convexity of I, additivity of Re, and inner products. The final equality comes from the fact that the average of Te(g) is equal to 0 for nontrivial 9;; W() and un are real. Therefore, this proof exploits the properties of convex functions and group representation theory.
Combined with the fact that | 2 | [2: the loss value of W is larger than that of W Therefore, the solution is G-invariant.
We can also prove the same results by focusing on the subgradient, which described in the Supplementary Material section. Facts from representation theory are necessary for both cases.
This theorem indicates that the complexity of the problem can be reduced by imposing invariance. Because the generalization error increases with complexity, this theorem explains one reason that invariance con- tributes to the good test accuracy. (Sokolic et al. (2017) analyzed the generalization error in similar setting, using a covering number. This work calculated the upper bound of complexity; whereas, we focus on the linear classifier, obtain the explicit complexity, and calculate the learned classifier. Furthermore, even when the obtained classifier is the same as that trained by the augmented samples, the invariant features are more suitable for training owing to the reduced number of leature dimension and training samples, which may affect the performance of the learned feature. Moreover, we can apply a larger size of code words for the clustering- based methods when we match the global feature dimension. Hence, our goal is to construct a feature that is invariant in the vector space where the group acts orthogonally.
4.1.2 Illustrative example
Theory is illustrated in the setting of Section 3.2 In this decomposed space, when the classifier is trained by considcring both original images and flipped images, the classification boundary learned with L2-regularized convex loss minimization acquires the form of = b, as plotted in the right figure of Figure 2 Thus, we can discard g-elements of features and obtain a compact feature vector. This result is validated as follows: when the fcature in the decomposed space that corresponds to the original image is denoted as (z., 2²), the 0
feature corresponding to the flipped image becomes -v²) because T(m) = -.) Expressing the
linear classifier as w-y² + w² + b, the loss for these two images can be written as 1(w-022 + w² + b) + (ww' - u2 + b). From Jensen's inequality, this is lower-bounded by 21(w" u + b). This means that the loss is minimized when w² = O, resulting in the classification boundary w + b 0. wi is invariant under the action of (e)) and (m))
4.2 Invariant feature modeling
First, featurc-modeling method is constructed for the calculation of invariant feature coding.
Invariant PCA PCA attempts to determine the subspace that maximizes the sum of the variances of the projected vectors. The original PCA is a solution to uax Tr (UT 1 N En (I - u)( 11 TT) U): where μ 1/4 Em= Zn. The solution is matrix consisting of the eigenvectors of N EN (on w)(xn PMT that correspond to the top eigenvectors.
Our method can only utilize the projected low-dimensional vector it also lies in the representation space of the considered group. Therefore, in addition to the original constraint U = we assume that U an intertwining operator in the projected space. From Schur's s lemma described in Section 3 the U that satisfies these conditions is the matrix obtained using the projection operator Pr, with the dimensionality reduced to
NTT.. Furthermore, when Tt = Dnr Tr,o and the -hh element of 2n as zn (t.o) the dimensionality reduction within n,T, is in the form of (!()) + Ene, ((t..)),(te) for (tt.o) E C because of Schur's lemma. Hence, our basic strategy to first calculate (1 that maximize the variance with the orthonormality preserved and subsequently choose & for larger variances per dimension. In fact, (1.00 can be caleulated using PCA with
9

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm 1 Calculation of Invariant PCA
Input: {
1N E "R, ",,,,dpro)
Output: E Rdxdpoi which is intertwining and orthonormal for t= to T do
for 01. 02 to n do
(1)
01,02 + 2/2 N
(1,01) Zn
-utttr)), xn (1.0z) (l(hoz))
end for
(A , up ) + cigendoccomposition of E(0
end for
U = empty matrix
while size of U is smaller than d dproj do
(t) U concat of U and (1p Tar,) Pr. for non-used p,t with maximum X /d end while
Algorithm Calculation of invariant k-means
Input: Ja E Rdd G,
Output: Pg.c for E G. fl....cc} C} randomly initialize Pe,c for c E (1,...C) C} for =t to maxiter do
Hg.c + v(g)ue.c for E G, {I,..., C} Sg.c + {n((g,c) arg min || - Hg,ell)
(g.c)
Ha,e E.eo 18x.el Eyec Enesp. 12,,, for fl..... C}.
end for
the sum of covariances between cach dimensional element of xn (t.0). This algorithm is presented in Algorithm Because the projected vector must be real, additional care is required when certain elements of t are complex. The modification for this case are described in the Supplementary Materials section. the experiment, we use groups in which all irreducible representations can be real; thus, Tr can be used directly.
As for computational complexity, standard PCA requires square order of the input feature dimen- sion times the number of input features plus cubic order of the input feature dimension and therefore o (N
(N (M (,dii(T))TT)) + (Σ m,dim (odim(t)))) On the other hand, the proposed invariant PCA requires
the computation of PCA for each irreducible representation and therefore O (I CEi=iNNn (Nnīdim(rr) + m3))
Note that PCA considers the 2D rotation using Fourier transformation and has been applied to cryo-EM images (Zhao & Singer 2013] Vonesch al.] 2015 2013 Zhao et al.) 2016). The proposed PCA can be regarded
extension of the general noncommutative group. Other studies (Zhao & Singer 2013
Vonesch et al. 2015 2013 Zhao et al. 2016 apply PCA for direct image modeling, whereas regard our proposed PCA as a preprocessing method that preserves the equivariance property for successive feature coding.
Invariant k-means The original k-mcans algorithm is calculated as min, ΣN min ||,, - Mell. To
ensure that & acts orthogonally on the learned model, we simply retain the cluster centroids by applying all transformations to the original centroid. This implies that we learn Mg.e for g E G.c e {l...., C) such that Pg.e satisfics (9)1) um computational complexity, the proposed k-means requires [G] times as large computation complexity as the original k-means since the size of the code words becomes G times large.
for all 91. C.. This algorithm listed in Algorithm 2 As for
10

---

Published in Transactions on Machine Learning Research (06/2023)
4.3 Invariant feature coding
Subsequenly, we construct feature coding function F as G-invariant vector in the space where g acts orthogonally, First, the spaces of the function In that G act orthogonally are calculated as the basis spaces for constructing more complex representation spaces using the tensor product. The first basis space is that of Zn- The second is the C[G]-dimmnsionnll 1-of-k vector that we assign In to the nearest Mg.c- When we apply (g) to In: the nearest μ is Pg.c which corresponds to the vector to which we apply =(g) on the nearest y In. Therefore, 9 acts as a permutation. We denote this representation by (g)
Invariant BP
Because BP is written as the tensor product of Zn. use
== vec P (In 8mm)
(4)
as the invariant global feature. Although P1 is a projection onto the trivial representation defined by Eqq we can calculate this feature from the irreducible decomposition of In and the irreducible decomposition of the lensor products. Normalization can also be applied to the invariant covariance with respect to each Pi (Tr,o @ Tr, element to obtain a variant of BP, such as iBP (Lin & Maji 2017). Because elements that are not invariant are discarded, the feature dimensions become smaller.
Invariant VLAD Subsequently, we propose an invariant version of VLAD. The first difficulty is that VLAD is not in the form of a tensor product, and second, the effect of transformation on the nearest code word is not consistent. We solved the first difficulty by considering VLAD as the lensor product of the local features and 1-of-k representation, and the second difficulty was solved by using the invariant k-means proposed in Section 4.2 to learn the code word.
The expression for the invariant VLAD is as follows:
N Σ ==1
F = vec
P1 (1a(zn) @ (En (aa))
(5)
N
where He denotes the nearest centroid to Tn- Because 1 (x())) is a vector in which only the element corre- sponding to the nearest μ is 1, the tensor product becomes the vector in which the elements corresponding to the nearest i are I, - Mes which is the same as the original VLAD. Bocause both (x,) and In 14 are orthogonal representation spaces, this space also an orthogonal representation space. Although the size of the codebook becomes times larger, the dimensions of the global feature are not as large because we used the invariant vector.
illustrative example of invariant k-means and VLAD As shown in Figure 3 when the learned codebook is for = 2, In and 22 are both assigned to μr. When these features are flipped, the flipped 21 is assigned to μ4, whereas the flipped 22 is assigned to un2 Thus, image flipping does not act consistently on the assignment vector (xx). As plotted in Figure 4 when the codebook is learned such that there exists μ with y-clement flipped for each code word, the group acts consistently on (x,)) This is bocause whenever on assigned to He,s, the flipped H, is assigned to pm.s. addition, the flipped on is assigned Meçi when F, is assigned to pm,i. Therefore, the group acts orthogonally (xx)).
Invariant VLAT Finally, propose the invariant VLAT that incorporates local second-order statistics. This feature can be calculated by combining the two aforementioned features.
(aa))Ta))) )
=
Σ P. (L,()))))) ((a, He) 8 (In -Mo) - Ta)))
(6)
N
The dimension of the invariant VLAT with C components is the same as that of VLAT with components.
Thus, we can model complex nonlinear statistics and calculate the invariants using the tensor product representation when using local polynomial statistics global feature.
11

---

Published in Transactions on Machine Learning Research (06/2023)
Decomposed Feature Space
Decomposed Feature Space
y
flipped.12
flipped.22
Me,2
flipped
f12
1mm11
f
flip
Pe,
flip
u3
Pe,1
Pm,3
Mm,2
22
22
Figure: 3: VLAD with code words
learned by k-means.
Figure 4: Invariant VLAD with code words learned by proposed k-means.
Table 1: Irreducible representations of the D4 group.
Rep.
m
mr
mr²
771
T1a
F1-1
1
1
1
-1
-1
-1
-1
T-1,1
1
-1
1
-1
1
-1
1
-1
T-1,-1
1
-1
1
-1
-1
1
-1
1
1
07
0
-1
--1
0
0
1
[-1
07
[
1
1
0
0
-17
1
1
0
0
-1
-1
0
0
-1
-
1
0
4.4 Limitation and extension
Our theorem and methods depend on two assumptions regarding the transformations: finiteness and or- thogonality. At first glance, these assumptions may seem strong. However, the outputs of a wide variety of equivariant CNNs satisfy these requirements. This is because the invariance with respect to infinite trans- formations can often be approximated by finite invariance. For example, the continuous rotation invariance is approximated with small discrete rotations (Marcos al. 2017 Weiler al. 2018). Moreover, invariance is often implemented by CNNs with special restrictions convolutional filters (e.g. filter "A" is filter "B" rotated by some 0). In this case, the transformations act as permutations between the filter responses, which are orthogonal. Therefore, these assumptions are not restrictive. Furthermore, we can extend Theorem | to a more general assumption by exploiting more advanced group representation theory. This case is discussed in the Supplementary Materials section.
4.5 Effectiveness of the end-to-end setting
Although Theorem assumees that the input features are fixed, the effectiveness of the proposed invariant feature can also be demonstrated when the local features are learned in an end-to-end manner. We write the entire classification model as W Fe (x), where Fg, and f,heta: denote the learnable feature coding function and feature extractor, respcctively. We assume that these functions preserve equivariance, moaning that the transformations act orthogonally on Fg.,o, (z). This assumption satisfied when equivariant CNNs used the local feature extractor along with the proposed invariant feature coding functions. Then, the
12

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: Irreducible representations tensor products of irreducible representations of the D4 group.
71,1
-
7-1,1
T-1,-1
72
1,,
71,
7,-1
T-1,1
T-1,-1
72
71-1
0,11
71
T-1,1
T-1,1
7-1,1
7-1-1
T-1,1
71,1
71.1
7-1,-1
T-1,-1
T-1,1
71,11
71,1
T2
72
T2
72
72
T2
TI.1
01,
① T-111 T-1,-1
Table 3: Comparison of accuracy using fixed features.
Methods
Dim.
FMD
DTD
CIUC
CUB
Cars
BP
525k
81.28 + 1.54
75.89 + 0.72
80.83 = 2.35
77.48
86.22
iBP
525k
81.38 + 1.38
75.88 + 0.62
81.94 = 3.19
75.90
86.74
VLAD
525k
80.38 + 1.61
75.12 + 0.77
80.37 -3.23
72.94
86.10
VLAT
525k
79.98 + 1.66
76.24 + 0.66
80.46 = 2.60
76.62
87.12
FV
525k
78.18 + 1.66
75.56 + 0.80
79.35 = 1.52
66.59
81.70
Inv BP (ours)
82k
83.34 + 1.48
77.19 = 0.60
81.48 = 1.75
81.79
87.45
Inv iBP (ours)
82k
83.46 + 1.27
77.96 + 0.46
83.33 + 1.80
82.12
88.80
Inv VLAD (ours)
525k
82.42 + 1.74
76.53 + 0.56
81.94 = 1.40
80.97
88.54
Inv VLAT (ours)
525k
81.88 + 1.34
77.45 + 0.55
82.13 = 1.64
83.25
88.59
learning problem can be formulated as
M
Σ I(WT (o Mm). m=1
arg min WER 1ER, 82.09
+
M
For any fixed 0, we can apply Theorem by considering Um Fe (xmm) Therefore, the invariant feature is effective, even in an end-to-end setting.
5
Experiment
In this section, the accuracy and invariance of the proposed method evaluated using the pretrained features in Section 5.1 In Section 5.2 we evaluate the method on end-to-end case.
5.1 Experiment with fixed local features
In this subsection, our methods are evaluated on image recognition datasets using pretrained CNN local features. Note that the local features were fixed to compare only the performance of coding methods; thus, the overall scores are lower than those of the existing end-to-end methods.
These methods were evaluated using the Flickr Material Dataset (FMD) Sharan et al.
2013
describable
texture datasets (DTD) Cimpoi et al. 2014b. UIUC material dataset (UIUC) [Liao et al.
2013). Caltech-
UCSD Birds (CUB) (Welinder et al.J 2010)) and Stanford Cars (Cars) (Krause et al. 2013
FMD contains
material categories with 1,000 images; DTD contains texture categories with 5,640 images; UIUC
contains 18 categories with 216 images; CUB contains 200 bird categories with 11,788 images; and Cars consists of 196 car categorics with 16,185 images. We used given training test splits for DTD, CUB, and Cars. We randomly split 10 times such that the sizes of the training and testing data would be the same for each category for FMD and UIUC.
5.1.1 Results on D4 Group
First, we applied our method to the D4 group used in Cohen & Welling (2017) which contains rich informa- tion and is easy to calculate. The D4 group consists of x/2 rotation and image flipping with |G] 8.
13

---

Published in Transactions on Machine Learning Research (06/2023)
Table 4: Comparison of accuracy using fixed features on the augmented test data.
Methods
Dim.
FMD
DTD
CIUC
CUB
Cars
BP
525k
78.17 + 1.32
70.90 + 0.64
69.12 + 1.53
37.35
27.45
iBP
525k
78.28 + 1.17
71.00 + 0.64
69.39 = 1.91
38.32
28.45
VLAD
525k
77.20 + 1.16
70.62 = 0.77
67.95 2.18
36.18
29.78
VLAT
525k
77.03 + 1.14
71.45 + 0.65
69.87 = 1.53
41.01
27.27
FV Inv BP (ours)
525k
75.28 + 1.41
70.78 + 0.76
65.89 = 1.06
29.36
27.37
82k
83.05 + 1.43
77.09 = 0.61
81.48 = 1.75
81.62
87.50
Inv iBP (ours)
82k
83.24 + 1.19
77.83 + 0.47
83.38 + 1.81
81.98
88.80
Inv VLAD (ours)
525k
82.14 + 1.67
76.38 + 0.50
81.68 = 1.32
80.75
88.54
Inv VLAT (ours)
525k
81.77 + 1.31 77.36 + 0.60
82.13 = 1.64
83.01
88.62
Table 5: Comparison of accuracy using invariant feature modeling and existing feature coding methods.
Methods
Dim.
FMD
DTD
UICC
CUB
Cars
BP
525k
81.20
1.60
75.09 + 0.56
80.19 3.00
78.24
85.67
iBP
525k
81.20 + 1.60
75.05 + 0.63
81.20
3.35
76.36
86.00
VLAD
525k
80.50 = 1.95
75.20 = 0.86
78.80 = 2.29
71.61
85.72
VLAT
525k
79.98 + 1.62
75.66 + 0.82
81.02 + 1.86
78.29
84.77
FV
525k
78.50 + 1.65
75.63 + 0.72
79.91 + 1.80
67.47
82.47
The irreducible reprosentations and dccomposition of the tensor products are summarized in Tables and respectively.
Because D4 is not orthogonal to the output of the standard CNNs, we pretrained the group equivariant CNNs and used the last convolutional activation as the local feature extractor. The group equivariant CNN is the model in which we preserve the G action using |G Limes the number of filters T(g) applied on the original filters and use the average with respect to for the convolution. When the feature is of dcNx x |G| dimension, can be regarded as dcNN times the direct sum of the eight-dimensional orthogonal representation space because D4 acts as a permutation. The representation was decomposed as follows: #CNN dCNNTI,1 @ deNNTI,-1 dCNN-T-1,1 dCNNT-1,-1 ( 2dcNN72-
The group equivariant CNNs with the VGG19 architecture were pretrained with convolutional filter sizes of 23, 45, 91, 181, 181 instead of 64, 128, 256, 512, 512 as the local feature extractor. Furthermore, we added batch normalization layers for each convolution layer to accelerate the training. The model was trained using the ILSVRC201 datasct (Deng et al. 2009 A standard data augmentation strategy was applicd, and the same learning settings as the original VGG-Net were used. The pretraining code was implemented using Pytorch (Paszke et al. 2019) with the group equivariant convolution layers implemented using Groupy [Cohen & Welling 2016).
We extracted the last. convolutional activation of this pretrained equivariant VGG19 group after rescaling the input images by 25, where = =3, -3...........
2.,, 1.5. For efficiency, the scales that increased the image size to greater than 1, 024 pixels, were discarded. Subsequently, the nonlinear embedding method proposed in Vedaldi & Zisserman 20122) was applied such that the feature dimensions were three times larger. As this embedding is a point-wise function, the output can be regarded as three times the direct sum of the original representations when considering the group action.
The local feature dimension was then reduced using PCA for the existing method and the proposed invariant PCA for the proposed method. We applied BP and iBP with dimensions of 1,024, VLAD with dimensions of 512 and 1,024 components, FV with 512 dimensions and 512 components, and VLAT with 256 dimensions and 8 components. We applied the proposed BP and iBP with the same settings, and VLAD and VLAT with eight times the number of components.
Training code given pretrained VGG19 was implemented using MATLAB. The linear SVM implemented in LIBLINEAR (Fan et. al.] 2008) was used to evaluate the average test accuracy. Furthermore, to validate
14

---

Published in Transactions on Machine Learning Research (06/2023)
Table 6: Comparison of accuracy using existing feature coding methods with augmented training data,
Scores in the parenthesos correspond to those of the proposed invariant features cited from Table Methods
Dim.
FMD
UICC
BP
525k
83.00 + 1.51 (83.3411.48)
81.11 + 1.46 (81.48:1..5))
iBP
525k
83.10 + 1.38 (83.46#1.27)
82.87 + 2.01 (83.33+1.80)
VLAD
525k
82.72 + 1.66 (82.42
80.56 + 1.75 (81.94+1.40)
VLAT
525k
82.66 + 1.27 (81.88+1.34)
81.67 + 1.89 (82.13+1.64)
FV
525k
81.44 = 1.60
80.74 + 2.46
Table 7: Irreducible representations of the D6 group.
Rep.
T1,1
71,-1
T-1,1
T-1,-1
-1
Coos (1/33) /3)
sin /r)
0
T2a
sin (TT /3)
cos (IT /3)
1
0
cos (2t//3 /3)
sin (2 /3]
0
T2b
sin (2T (2r33)
cos (2Tr/3
1 0
that the proposed feature is D4 invariant, we used the same training data and evaluated the accuracy by augmenting the test data eight-fold using the D4 group.
Tables 3 and 4 show the accuracy for the test as well the augmented test datasets. Our method demon- strated better accuracy than non-invariant methods with the dimensions of the invariant BP and iBP ap- proximately 1/7 of the original dimensions. Thus, we obtained features with significantly smaller dimensions and higher performances. Note that the higher accuracy is not a simple effect of dimensionality reduction because in general, the accuracy decreascs as the dimension is reduced. We observed that iBP with 400 local feature dimensions (80k global features) scored 81.76 on FMD and 81.30 on UIUC. Furthermore, compact bilincar pooling with 82k feature dimensions scored 81.54 FMD and 79.81 UTCC. Therefore, the result is significant as Inv iBP shows better accuracy despite the lower feature dinension.
This table also shows that the existing methods exhibit poor performance on the augmented test data, but the proposed mothods demonstrate performance with scores similar to the original. These results suggest that the existing methods use information that not related to image content, reflecting dataset bias instead. Our method can discard this bias and focus on the image content.
As an ablation study, we conducted experiments in which PCA and k-means were trained using the proposed invariant feature modeling and existing feature coding methods were then applied.
Table 3 demonstrates that the proposed invariant feature modeling, combined with the standard feature coding method, does not demonstrate better accuracy than the original methods. This result shows that only performance is enhanced when the proposed invariant feature modeling combined with invariant feature coding.
We also compared our results with those of data augmentation by training the existing non-invariant feature coding with eight-fold data augmentation. We conducted this by applying all transformations to the input images and made the number of training samples |G] times as large. Considering the much higher training cost, experiments were conducted on the FMD and UIUC datasets, which have a smaller number of training images.
Table shows that the accuracy comparable or little lower than the accuracy of the proposed invariant feature coding methods. This result is consistent with the argument of Theorem g that invariant features learned with the original training data are equivalent to non-invariant features learned with the augmented
15

---

Published in Transactions on Machine Learning Research (06/2023)
Table 8: Irreducible representations of the tensor products of irreducible representations of the D6 group.
71,
71-1
7-1,1
T-1,-1
72e
72b
71,
71,
1,-1
7-1,1
T-1,-1
T2a
72b
71,1
71-1
71,1
T-1,-1
7-1,1
72b
7-1,1
7-1,-1
T-1,1
71,-1
71,1
T2o
T-1,-1
T-1,-1
7-1,1
T1,-1
71.1
T2b
T2a
72a
T2a
72a
T2b
T2b
T1,1
13,111
4 T2b
T-1,1 ① T-1,-11
? T2a
T2b
726
T26
72a
T2a
T-1,1
T-1,-1 T2a
T1,1
1 71,-1
1 T26
Table 9: Comparison of lest accuracy using D6-equivarianl CNN.
Methods
Dim.
FMD
DTD
UIUC
CUB
Cars
BP
525k
77.26
73.46
75.93
72.66
80.15
iBP
525k
77.18+1142
73.51+0.65
75.56=1.70
71.19
81.13
VLAD
525k
75.80+1.55
71.65+0.99
70.19+3.223
64.69
78.44
VLAT
525k
75.70-11.40
72.60
72.3142.07
69.92
78.59
FV
525k
75.24+1.42
72.37
74.91
63.06
75.55
Inv BP (ours)
55k
80.26+1.70
75.41+1001
79.26+2.93
78.77
80.92
Inv iBP (ours)
55k
80.36-11.48
0.55
80.09
79.74
83.60
Inv VLAD (ours)
525k
78.00=1.17
72.76
73.98
76.12
82.47
Inv VLAT (ours)
525k
78.06=11.45
74.15+0.77
75.65+2.73
77.59
80.15
training data. Therefore, our method exploits the advantages of data augmentation with lower feature dimensions and lower training costs.
As for computation time, using an Intel Xeon E5-2698v4 x2 20 Core, 2.2 GHz CPU takes 13 seconds to extract the training features and 61 seconds to learn SVM train BP on UTUC, 9.4 seconds to extract the training features and 2.1 seconds to learn SVM to train Inv BP on UIUC in Table 3 and 81 seconds to extract the training urat and 130 seconds to learn SVM to train BP the augmented UTUC in Table The reduced number of feature dimensions by the proposed method contributes to the training efficiency.
5.1.2 Results on D6 Group
Furthermore, we applied our method to the D6 group, which was more complex than the D4 group. The D6 group consists of π 3/3 rotations r and an image flipping m with |G] 12. The irreducible representations and docomposition of the tensor products for this group are summarizod in Tables and respectively.
To obtain the local feature in which D6 acts orthogonally, CNN was pretrained with HexaConv (Hoogeboom) et al. 2018). HexaConv models the input images in the hexagonal axis and applies D6 group-equivariant convolutional layers to construct the CNN. Because D6 acts permutation of the positions along the hexagonal axis, the convolutional activations are D6-equivariant. Therefore, used this convolutional activation as the input for our coding methods.
The group equivariant CNNs with the VGG19 architecture were pretrained with convolutional filter sizes of 18,37, 74, 148, 148 instead of 64, 128, 556,,12,511 as the local feature extractor. Furthermore, batch normalization layers were added to cach convolution layer to accelerate the training speed. Therefore, the dimensionality of the local features is 148 12. The model was trained using the ILSVRC2012 dataset (Deng) et al. 2009). Because the input image size increases when the axes are changed from Euclidean to hexagonal, 160 160 image was randomly cropped from the original image rescaled to 192 192 for training. The remaining settings followed those of the original VGG.
We extracted the last convolutional activation of this pretrained model after rescaling the input images by 24, where -3 225, ..., 1.5. For efficiency, the scalings that increased the image size to greater than 5122 pixels were discarded. Subsequently, the nonlinear embedding method proposed in Vedaldi & Zisserman 2012) was applied. The dimensions and number of components used for the coding methods followed
16

---

Published in Transactions on Machine Learning Research (06/2023)
Table 10: Comparison of accuracy on Resnet50.
The score with c*: denotes the score reported in Li et al.
20018).
Method
iSQRT-COV* (Resnet50)
iSQRT-COV (equivariant Resnet55)
Inv iSQRT-COV
(equivariant Resnet55)) (ours)
Dim.
32k
265k
25k
Top-1 Err.
22.1 14
21.65
21.02
Top-5 Err.
6.22
5.92
5.47
Table 11: Comparison of accuracy on Resnet101. The score with :** denotess the score reported in Li et al.
2018
Method
iSQRT-COV* (Resnet101)
iSQRT-COV
Inv iSQRT-COV
(equivariant Resnet101)
(equivariani Resnet (ours)
Dim.
32k
265k
25k
Top-1 Err.
21.21
20.45
19.98
Top-5 Err.
5.68
5.35
4.96
those used for the D4 experiments. The pretraining was implemented by Pytorch and hexaconv library [Hoogeboom et al. 2018), and feature coding and classifier training was implemented with MATLAB.
In Table 9 the proposed coding methods consistently demonstrate better performance than non-invariant methods although the overall accuracy is lower than the D4 case, which may arise from the discriminative performance of the local feature extractor. Furthermore, the dimensionality of invariant BP and iBP is smaller than the dimensionality for D4. This is because the D6 group represents more complex transforma- tions than the D4 group; thus, the number of invariants with respect the D6 group smaller than the number with respect the D4 group.
Therefore, the proposed framework was also effective for the D6 group.
5.2 Experiment with end-to-end model
The proposed invariant BP was then applied an end-to-end learning framework.
5.2.1 Results on D4 Group with iSQRT-COV
We constructed a model based on iSQRT-COV [Li ct al. 2018 which is a variant of BP that demonstrates good performance and stable training. Group equivariant CNNs with Resnet50 and 101 (He et al. 2016) architecture were used for the local feature extractor, where the filter sizes were reduced, as in the case of VGG19. The iSQRT-COR and the proposed invariant iSQRT-COV were used instcad of global average pooling. We compared iSQRT-COV with Resnet50, iSQRT-COV with D4-equivariant Resnet50, and the proposed invariant iSQRT-COV with D4-equivariat Resnet50.
All the models were learned, including the feature extractor, using a momentum grad with an initial learning rate of 0.1, momentum of 0.9, and weight decay rate of le-4 for 65 epochs with a batch size of 160. The learning rate was multiplied by 0.1 at 30, 45, and 60 epochs. The top-1 and top-5 test errors were then evaluated, using the average score for the original and flipped images for the evaluation. The training code implemented with Pytorch, Groupy and fast-MPN-COV libraries (Li et al. 2018).
Tables 10 and 11 show that although iSQRT-COV achieves accuracy using the local features extracted from equivariant CNN, our Inv iSQRT-COV demonstrates better accuracy with smaller feature dimension.
The above pre-trained models were further fine-tuned and the accuracy was evaluated on fine- ine-grained recog- nition datascts: CUB, Cars, and Aircraft (Maji et al. 2013).. The Aircraft datasct consists of 100 categories with 6,667 training images and 3,333 test images. Following the settings of existing studies, each input image resized to 448 x 448 for fine-tuning and evaluation for CUB and Cars, and the other images resized
17

---

Published in Transactions on Machine Learning Research (06/2023)
Table 12:
Comparison of accuracy using fine-tuning on Resnet50.
The score with *** denotes the score
reported in Li ct al. 20188).
Method
ISQRT-COV*
iSQRT-COV
Inv iSQRT-COV
(Resnet50)
(equivariant Resnet50)
(equivariant Resnet50) (ours)
Dim.
32k
265k
25k
CUB
88.1
87.8:10.19
87.9
Cars
92.8
93.8+0.11
93.4+0.16
Aircraft
90.0
91.4
91.7
Table 13: Comparison of accuracy using fine-tuning on Resnet101. reported in Li et al. (0018).
The score with *** denotes the score
Method ISQRT-COV*
ISQRT-COV
Inv iSQRT-COV
(Resnet101)
(equivariant Resnet101)
(equivariant Resnet101) (ours)
Dim.
32k
265k
25k
CUB
88.7
44.7:+2.3
88.5++0.25
Cars
93.3
56.0=7.9
93.9
Aircraft
91.4
69.1+2.00
92.1+0.26
to 512 x 512, further cropping the 448 x 448 center image for the Aircraft dataset. The training setting followed the original setting of the iSQRT-COV (Li al., 2018 For each sctting, we evaluated the models len times and evaluated the average lest accuracy. Tables 12 and 13 show the results. As for ResNet55,, the proposed Inv iSQRT-COV displays comparable better accuracy than the reported iSQRT-COV and simi- lar accuracy to iSQRT-COV with equivariant ResNet50 with much smaller feature dimension. Furthermore, ESQRT-COY with equivariant ResNet101 overfits the training data and displays much lower accuracy than the proposed Inv iSQRT-COV ResNet101. This result suggests that our invariant feature coding contributes to the stability of tradient-basee training and results in the exploitation of complex local feature extractors with high recognition accuracy.
As for the computation time, when using 8 A100 GPUs it takes 0.17 seconds /band///bcchhhhc to train iSQRT-COV (Resnet50), 0.72 seconds/ batch to train iSQRT-COV (equivariant Resnet50) and 0.45 seconds brconds/batch batch to train the proposed Inv iSQRT-COV (equivariant Resnet50) on ImageNet. Though equivariant Resnet50 models take more time than the original Resnet50, which mainly results from feature extractor module instead of the feature coding module we target, we can reduce the computation time by the proposed invariant coding method compared to iSQRT-COV (equivariant Resnet50). The reduced complexity arises from (i) reduction of the feature dimension that we calculate the bilinear feature, and (ii) reduction of the size of the final fully-connected layer.
Subsequently, to visualize attention, GradCAM++ + (Chttopadhay al. 2018 was used after the dimension- reduction layer (Figure 5 In the existing methods (ISQRT-COV with Resnet50 and equivariant Resnet50), the shape of the heatmap changes when rotation and horizontal flip are applied to the input image. The shape of the heatmap is relatively preserved with the proposed invariant coding method. This result indicates that the baseline methods learned in a standard manner do not learn the invariance correctly, whereas our method was trained to both demonstrate high accuracy and preserve invariance, which is consistent with the result for the case of fixed local features. Furthermore, the heatmaps are quantatively compared using maximum mean discrepancy (MMD) by (i) inverting flipped and rotated heatmaps to match the original heatmaps, (ii) applying 11-normalization to the heatmaps, and (iii) calculating the MMD between the inverted heatmaps and heatmaps for the original image with the Gaussian kernel function exp( 0² dimensional positions of the image pixels normalized within [0. 1), and proposed method shows a lower MMD value, which means that the shape of the heatmaps are more similar to each other. Plots of the results the other images are available in the Supplementary Materials section.
l where 7,5 are two-
set 0.1. We can see that the
18

---

Published in Transactions on Machine Learning Research (06/2023)
Original image
(b) + the (c) CradCAM++ the (d) GradCAM++ the original image with iSQRT- Original image with iSQRT- original image with Inv COV (Resnet50)
COV (oquivariant Resnet50)
iSQRT-COV (equivariant Resnet50) (ours)
(e) Flipped image
(f) GradCAM++ the (g) CradCAM+ the (h) GradCAM++ the flipped image with iSQRI- flipeed image with iSQRT- flipped image with Inv COV (Resnet50) (MMD: COV (equivariant Resnet50) iSQRT-COV (equivariant 0.021)
(MMD: 0.068)
Resnet50) (ours) (MMD: 0.017)
Rotated image
CradCAM+ the (k) CradCAM+ the (I) GradCAM+ the
original image with iSQRT- original image with iSQRT- Original Image with COV (Resnet50) (MMD: COV (equivariant Resnet50) iSQRT-COV (cquivariant
0.031)
(MMD: 0.11)
Resnet50) (ours) (MMD: 0.022)
Figure 5: Comparison of GradCAM+ on the transformed input image. MMD indicates maximum mean discrepancy the heatmaps of the original image.
5.2.2 Results on the other groups
The proposed method was then applied to different groups.
First, we used D6 group with HexaConv (Hoogeboom et al.) 2018 as in Section 5.1.2 with D6-equivariant Resnct18 as the base feature extractor. Resnet18 was used reduce memory consumption. We then compared the standard Resnet18, D6-equivariant Resnet18, iSQRT-COV with D6-equivariant Resnet18, and the proposed invariant iSQRT-COV with D6-equivariant Resnet18.
19

---

Published in Transactions on Machine Learning Research (06/2023)
Table 14:
Comparison of accuracy for the end-to-end D6-equivariant model.
Method
Resnet18
equivariant Resnet18
iSQRT-COV
Inv iSQRT-COV
(equivariant Resnet18)
(cquivariant Resnet18) (ours)
Dim.
0.5k
1.8k
395k
22k
Top-1 Err.
30.24
28.93
27.27
26.42
Top-5 Err.
10.92
9.99
9.46
8.45
Table 15:
Comparison of accuracy for the end-to-end C8-equivariant model.
Method
Resnet18
equivariant Resnet18
iSQRI-COV
Inv iSQRT-COV
(equivariant Resnet18)
(cquivariant Resnet18) (ours)
Dim. 0.5k
1.5k
131k
10k
Top-1 Err.
30.24
27.32
25.33
25.25
Top-5 Err. 10.92
8.76
7.90
7.86
To learn the iSQRT-COV models, momentum grad with initial learning rate of 0.1, momentum of 0.9, and weight decay rale of le-4 were used for 65 epochs with a batch size of 160. The learning rale was further multiplied by 0.1 at 30, 45, and 60 epochs. Following the original setting, we learned the Resnet models using & mm grad with an initial learning rate of 0.1, momentum of 0.9, and weight decay rate of le-4 for 100 epochs with a batch size of 256. The learning rate was multiplied by 0,1 at 30, 70, and 90 epochs, and then the top-1 and top-5 test. errors were evaluated. The training code was implemented with Pytorch, hexacony and fast-MPN-COV libraries.
Table 14 demonstrates that the proposed invariant features show good accuracy with low feature dimension.
When considering the discrete rotation group, the proposed invariant bilinear pooling was reduced to the squared norm of the Fourier coefficients. The Fourier coefficients were then applied to the C8 group that consists of 7/8 rotations. The implementation provided by Weiler & Cesa (019) was used to implement the C8-oquivariant networks. The standard Resnet18, C8-cquivariant Resnet C8-cquivariant Resnet18 with ISQRT-COY pooling, and the proposed C8-cquivariatn Resnet18 with invariant iSQRT-COV pooling were then compared. The training setting followed that of the D6 group.
Table 15 demonstrates similar tendency for the D4 and D6 cases. Therefore, the proposed method is effective for several invariance settings.
5.2.3 Results on the other coding method
In this section, further apply method to another variant of bilinear pooling.
Scaling eigen branch (SEB) (Song et al. 2023) is method that multiplies improved bilinear pooling with 1+ EE Age-2X, 2X, enhance the importance of features with smaller variance, where A denotes the i-th eigenvalue of the bilinear matrix. Although the original method calculates the features using singular value decomposition, to reduce the computation cost and increase the stability, we calculated the features using [1+ Vtrace (Ee-22) iSQRT(E), where Σ is the input covariance feature. The computation does not require singular value docompasition. We refer this method as iSQRT-SEB and compare iSQRT-SEB Resnet50, D4-equivariant Resnet50 with iSQRT-SEB (denoted by "ISQRT-SEB") and the proposed D4- equivariant Resnet50 with invariant iSQRT-SEB (denoted by "Inv iSQRT-SEB (ours)"). For iSQRT-SEB with D4-oquivariant Resnet50, the global feature dimension was reduced to 32k by applying invariant PCA, considering memory limitations. The training strategy was the same as that of iSQRT-COV in the D4 group. The DifferentialSVD library (Song el al. 2023) was used for implementing SEB. Tables 16 and 17 display the results. Although the overall accuracy is lower than that of iSQRT, showing the same tendency as in the original paper, the tendency is the same as that of the proposed method which has better accuracy.
Therefore, invariant feature coding is effective for end-to-end training.
20

---

Published in Transactions on Machine Learning Research (06/2023)
Table 16: Comparison of accuracy for iSQRT-SEB on Resnet50.
Method
iSQRT-SEB
iSQRT-SEB
Inv iSQRT-SEB (oquivariant Resnet50) (ours)
(Resnet50)
(equivariant Resnet50)
Dim.
32k
32k
25k
Top-1 Err.
25.20
24.62
23.77
Top-5 Err.
7.92
7.71
7.14
Table 17: Comparison of accuracy for iSQRT-SEB on Resnet
Method
ISQRT-SEB (Resnet101)
iSQRT-SEB
Inv iSQRT-SEB (equivariant Resnet101) (ours)
(equivariant Resnet101)
Dim.
32k
32k
25k
Top-1 Err.
23.84
23.87
22.96
Top-5 Err.
7.7
7.11
6.62
6
Conclusion
In this study, we proposed a leature-coding method that considers image information preserving transforma- tions. Based group representation theory, propose guideline used to first construct feature space in which groups act orthogonally to calculate the invariant vector. Subsequently, novel model learning method and coding methods are introduced to explicitly consider group actions. We applied our method to image classification datasets and cemonstrated that the proposed model can yield high accuracy while preserving invariance, This study provides novel framework for constructing an invariant feature.
Acknowledgements
This work was partially supported by JSPS KAKENHI Grant Number JP19176033 and JP19K20290, JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR 4403 and JPMJCR2015 and Basic Rescarch Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo. We would like to thank Atsushi Kanehira, Takuhiro Kaneko, Toshihiko Maisuura, Takuya Nanri and Masatoshi Hidaka for the helpful discussion.
References
Fabio Anselmi, Joel 7. Teibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso Poggio. Unsu- pervised learning of invariant representations. Theoretical Computer Science, 633:112 121, 2016.
Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Con architecture for weakly supervised place recognition. In CVPR, 2016.
Gregory Benton, Marc Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data. In Neurips, 2020.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vinceth N Balasubramanian. Grad-cam++:: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, 2018.
Ho Yin Chau, Frank Yuchon Qiu, Yubei Chen, and Bruno Olshausen. Disentangling images with lic group transformations and sparse coding. In Neurips 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
Taco Cohen and Max Welling. Group equivariant convolutional networks, In ICML, 2016.
Taco Cohen and Max Welling, Steerable cmns. In ICLR, 2017.
21

---

Published in Transactions on Machine Learning Research (06/2023)
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
Mandar Dixit and Nuno Vasconcelos. Object based scene representations using fisher scores of local subspace projections. In Neurips, 2016.
Rong-En En Fan, Kai-Wei Chang, Cho-Jui Hsich, Xiang- Rui Wang, and Chih-Jen Lin. LIBLINEAR: A library for large linear classification. JMLR, 9:1871 1874, 2008.
William Fulton and Joe Harris. Represenlation Theory: A First Course. Springer, 2014.
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In CVPR, 2016.
Mengran Gou, Fei Xiong, Octavia Camps, and Mario Sznaier. Monet: Moments embedding network. In CVPR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Emiel Hoogeboom, Jorn W.T. Peters, Taco S. Cohen, and Max Welling. Hexaconv. In TCLR, 2018.
Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. In CVPR, 2010.
Erik Jenner and Maurice Weiler. Stecrable partial differential operators for equivariant neural networks. In ICLR, 2022.
Ramakrishna Kakarala. The bispectrum as source of phase-sensilive invariants for fourier descriptors: a group-theoretic approach. Journal of Mathematical Imaging and Vision, 44:341-352 2012.
Takumi Kobayashi. Flip-invariant motion representation. In ICCV, 2017.
Takumi Kobayashi, Koiti Hasida, and Nobuyuki Otsu. Rotation invariant feature extraction from 3-d accel- cration signals. In ICASSP, 2011.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch gordan nets: fully fourier space spherical convo- lutional neural network. In Neurips, 2018.
Piotr Koniusz and Hongguang Zhang. Power normalizations in ine-grained image, few-shot image and graph classification. TPAMI, 44(2):591 -09, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cate- gorization. In ICCV, 2013.
Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Mare Pollefeys. Ti-pooling: transformation-
invariant pooling for feature learning in convolutional neural networks. In CVPR, 2016.
Peihua Li, Jianglao Xie, Qilong Wang, and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In CVPR, 2018.
Zicheng Liao, Jason Rock, Yang Wang, and David Forsyth. Nar-meramttric filtering for geometric detail extraction and material representation. In CVPR, 2013.
Feng Lin, Haohang Xu, Houqiang Li, Hongkai Xiong, and Guo-Jun Qi. Auto-encoding transformations in reparameterized lie groups for unsupervised learning. In AAAI, 2021.
Tsung- Yu Lin and Subhransu Maji. Improved bilincar pooling with cmns. In BMVC, 2017.
Tsung- Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear enn models for ine-grained visual recognition. In ICCV, 2015.
22

---

Published in Transactions on Machine Learning Research (06/2023)
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In ICCV, 2017.
Xu Miao and Rajesh P. N. Rao. Learning the lie groups of visual invariance. Neural Computation, 19():): 2665-2693, 2007.
Olivier Morère, Antoine Veillard, Lin Jie, Julie Petta, Vijay Chandrasekhar, and Tomaso Poggio. Group invariant deep representations for image instance retrieval. In AAAI, 2017.
Hideki Nakayama, Tatsuya Harada, and Yasuo Kuniyoshi. Dense sampling low-level statistics of local features. IEICE TRANSACTIONS on Information and Systems, 93:1727-1736, 2010a.
Hideki Nakayama, Tatsuya Harada, and Yasuo Kuniyoshi. Global gaussian approach for scene categorization using information geometry. In CVPR, 2010b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Neurips, 2019.
David Picard and Philippe-Henri Gosselin. Efficient image signatures and similarities using tensor products of local descriptors. CVIU, 117:680 687, 2013.
Rajesh Rao and Daniel Ruderman. Learning lie groups for invariant visual perception. In Neurips, 1998.
Marco Reisert and Hans Burkhardt. Using irreducible group representations for invariant 3d shape descrip- tion. In Joint Pattern Recognition Symposium, 2006.
Jongbin Ryu, Ming- Hsuan Yang, and Jongwoo Lim. Dat-basod transformation invariant pooling layer for visual classification. In ECCV, 2018.
Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. Image classification with the fisher vector: Theory and practice. IJCV, 105:222 245, 2013.
Lavanya Sharan, Ce Liu, Ruth Rosenholtz, and Edward H Adelson. Recognizing materials using perceptually inspired features. 1JCV, 10333)3348 371, 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2014.
Jascha Sohl-Dicksteni Ching Ming Wang, and Bruno A Olshausen. An unsupervised algorithm for learning lie group transformations. arXie preprint orxvv10001.10e 2010.
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Generalization error of invariant classifiers. In AISTATS, 2017.
Yue Song, Nicu Sebe, and Wei Wang. Why approximate matrix square root outperforms accurate svd in global covariance pooling? In ICCV, 2021.
Yue Song, Nicu Sebe, and Wei Wang. On the cigenvalues of global covariance pooling for fine-grained visual recognition. TPAMI, 45(3):3554 3566, 2023.
Andrea Vedaldi and Andrew Zisserman. Efficient additive kernels via explicit feature maps. TPAMI, 34(3): 480-492, 2012.
Cédric Vonesch, Frédéric Stauber, and Michael Unser. Design of stoerable filters for the detection of micro- particles. In International Symposium on Biomedical Imaging, pp. 934-99 334-937. IEFE, 2013.
23

---

Published in Transactions on Machine Learning Research (06/2023)
Cédric Vonesch, Frédéric Stauber, and Michael Unser. Steerable pea for rotation-invarian image recognition. STAM Journal on Imaging Sciences, 8(3):1857-1877 2015.
Qilong Wang, Peihua Li, Wangmeng Zuo, and Lei Zhang, Raid-g: Robust estimation of approximate infinite dimensional gaussian with application to material recognition. In CVPR, 2016.
Qilong Wang, Peihua Li, and Lei Zhang. G2denet: Global gaussian distribution embedding network and its
application to visual recognition. In CVPR, 2017.
Maurice Weiler and Gabriele Cesa. General e2)-qqivari steerable cnns. In Neurips, 2019.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning stecrable filters for rotation equivariant cmns. In CVPR, 2018.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. Technical Report CNS-TR-201, Caltoch, 2010.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In CVPR, 2017.
Tan Yu, Yunfeng Cai, and Ping Li. Toward faster and simpler matrix normalization via rank-1 update. In ECCV, 2020.
Zhizhen Zhao and Amit Singer. Fourier bessel rotational invariant eigenimages. JOSA A, 30(5):871 877, 2013.
Zhizhen Zhao, Yoel Shkolnisky, and Amit Singer. Fast stecrable principal component analysis. IEEE trans- actions on compulational imaging, 2(1):1 12, 2016.
A Another Proof of Theorem i
In this section, the proof of Theorem provided.
Proof. Denoting the loss function L(W), the subgradient. ØL at Wo is defined as the set,
AL(W)) = (cl 2 f(W)) + (c, W Wo) for VW}.
(8)
From this definition, Wo minimizes L if and only 0 E OL(W)). From the assumption in Theorem I L(W)= ()) for any W.g. g. Therefore, (L)) = <gg) (T.(W)) and 0 E aL(W)) = 0 (L)))))) From the definition subgradient, when E @LW.)), it follows that
aT.
x Wo
n (L(T())W)) 90.
(9)
G
BEG
The inclusion comes from the fact that C E N2 (L())))Wo) then L(W) 2 क EvEc L(T(g) + (c. W
c Eneo W)) 2 Mid Eyea (g) (c, W क. Ener (g) Therefore, क Eges म (g) also
minimizes I,, which is invariant under (g))
B
Extension of Theorem 1
Non-orthogonal case
When omit the restriction the orthogonality of n
M
Re
T
(w())
T
T
Fym
M
m=
GEG
Σ
Σ
Re
Σ
(1)6))(i1))
m
Um
(10)
M
GEG
(11)
24

---

Published in Transactions on Machine Learning Research (06/2023)
does not follow. However, matrix T can be calculated such that TT(g)T becomes orthogonal for all g. Therefore, if we apply appropriate coordinate transformation beforehand, we can apply Theorem |
Infinite case The finite group can be extended to a compact group. The compact group is parameterized by closed bounded sct, and o and -1 are continuous with respect to these paramcters. Many results regarding the representation of finite group also hold for compact group when [ল] Egea is replaced with f dg, where dg is special measure called Haar mcasure. With this mcasure, the following holds.
Pr, = dim(Fi) /
dgxx))))(1))
(12)
and
P1 - dgT(g).
(13)
Theorem 1 can be rewritten as
Theorem 2. When assume that the compact group G acts an orthogonal representation on R' and preserves the distribution of the training data P, the solution to the L2-regularized convez loss minimization,
1W P(e.)) x,y)
arg min WeRexIa
(14)
is G-invariant, implying that T(g) W for any g and P.W W.
C Invariant PCA when the irreducible representations are not real.
When some has complex elements, Algorithm cannot be applied directly because the intertwining operator and covariance matrices bocome complex. In this case, we couple т with т. into 7' Tr @ Tz. Because Xr. = XF.: the multiplicities of 7, and 7. in the decompasition are equal because of Eq.(6), and the projected vectors are complex conjugates of each other because of (7). Thus, v2 times the concatenation of the real and imaginary parts constitute the r'-th clement. In Algorithm 1, is replaced with of to obtain an invariant PCA for the general case.
Proof that the dimension of Invariant VLAD and VLAT is the same as that of original feature
Using components, 1a can decomposed into CTT,1 @ CTI,11 @ CT-11 @ CT-1,-1 @ 2C' When the first- sccond-order statistics are docomposed as 11,7/,1 @ 01,-17,-1 @ 1-111-111 $
R-1,-17-1,11 n272, the multiplicity of T1,1 of (CT,, @ CT1-1 CT-11 0T-1111 @ 20Tg) (11,171,1 11,-171--1 n-1,17-1,1 which is the same the dimension of the original feature.
① n-1,-17-1,-1 @ n272) is C + m1-1 1-11 1-1,-1 + 2m2).
E GradCAM++ on the other images
GradCAM+ results on the other images are plotted in Figures and
F Results on SIFT features
In this section, we report the results using the SIFT feature which D4 acts orthogonally. The irroducible decomposition of the SIFT feature is described in Section F.1 The accuracy of the image recognition datasets evaluated in Section Section F2
25

---

Published in Transactions on Machine Learning Research (06/2023)
Original Image
(b) + the (c) CradCAM++ the (d) + the original image with iSQRT- original image with iSQRT- original image with Inv COV (Resnet50)
COV (oquivariant Resnet50)
iSQRT-COV (equivariant Resnet50) (ours)
(e) Flipped Image
(f) GradCAM++ the (g) CradCAM+ the (h) GradCAM++ the flipped image with iSQRI- flipeed image with iSQRT- flipped image with Inv COV (Resnet50) (MMD: COV (equivariant Resnet50) iSQRT-COV (equivariant 0.027)
(MMD: 0.050)
Resnet50) (ours) (MMD: 0.017)
Rotated Image
CradCAM+ the (k) CradCAM+ the (l) GradCAM++ the orig-
original image iSQRT- original image with iSQRT- inal image with Inv iQQRT- COV (Resnet50) (MMD: COV (oquivariant Resnet50) COV (equivariant Resnet50) 0.023)
(MMD: 0.048)
(ours) (MMD: 0.014)
Figure 6: Comparison of GradCAM+ on the transformed input image. MMD indicates maximum mean discrepancy to the heatmaps of the original image.
F.1 Irreducible decomposition of SIFT
An overview of the SIFT feature is plotted in Figure 8 where D4 group acts permutator both fa, d, e, f.g, hi and 1...66. We can further decompose these into permutations on {a.c,e,g): (b, db [1,4, 13, 16), {6, 7, 11), and (2, 8,9, 12, 13, 14). These permutations can be decomposed as: T1,1 $ T-1,1 + T2, T1,1 + T-1,-1 O 72, T1.1 @ T-1.-1 @ 72, T1,1 @ T-1.-1 @ 72, and T1.1 + T-1,1 T T1,-1 1 + T-1,-1 @ T2 respectively, which can be calculated using the characteristic functions. Thus, the permutations SIFT can be decomposed into (27,,1 @ T-1,1 @ T-1.-1 @ 27x) @ (37,,1 T1.11 T-1.1 37-1,-1 e 44g) 187,1
1471-1 @ 14 @ 187-1,-1 @ 3272-
26

---

Published in Transactions on Machine Learning Research (06/2023)
Original Image
(b) + original image with iSQRT- original image with iSQRT- original image with Inv COV (Resnet50)
the (c) CradCAM++ the (d) GradCAM++ the
COV (oquivariant Resnet50)
iSQRT-COV (equivariant
Resnet50) (ours)
(e) Flipped Image
(f) GradCAM++ the (g) CradCAM+ the (h) GradCAM++ the flipped image with iSQRI- flipped image with iSQRT- flipped image with Inv COV (Resnet50) (MMD: COV (equivariant Resnet50) iSQRT-COV (equivariant 0.0082)
(MMD: 0.012)
Resnet50) (ours) (MMD: 0.0082)
Rotated Image
CradCAM++ the (k) CradCAM+ the (1) GradCAM++ the orig-
original image with iSQRT- original image with iSQRT- inal image with Inv iSQRT- COV (Resnet50) (MMD: COV (oquivariant Resnet50) COV (equivariant Resnet50) 0.014)
(MMD: 0.013)
(ours) (MMD: 0.0092)
Figure 7: Comparison of GradCAM+ on the transformed input image. MMD indicates maximum mean
discrepancy to the heatmaps of the original image.
F.2
Experimental Results
The methods were evaluated on (FMD) (Sharan et al. 2013), (DTD) (Cimpoi et al. 2014), (UTUC) (Liao
et al.] 2013). and CUB (Welinder et al. 2010)). ). The evaluation protocol was the same as that for the uree
The dense SIFT feature was extracted from multi-scale images, as in the case VGG, and then nonlincar homogencous mapping (Vedaldi & Zisserman 2012) was applied to make the feature dimension three times larger.
27

---

Published in Transactions on Machine Learning Research (06/2023)
2
12
13
14
15
16
Figure 8: Overview of the SIFT feature.
Table 18: Comparison of accuracy using SIFT features.
Methods
Dim.
Test Accuracy
Augmented Test Accuracy
FMD
DTD
UIUC
FMD
DTD
UIUC
BP
33k
49.84-t 1.54441..57
51.54=0.93
56.02:12.440
43.36+11.19
41.55
42.50:12.23
VLAD
262k
60.11#1.41
59.37=1.23
58.70:3.50
54.68+1.11
51.54+0..0
46.11=2.01
VLAT
525k
58.09+11.40
58.74=0.84
59.72++2.87
52.01+1.25
50.40+0..1
47.85=1.66
FV
262k
61.84
6.920-0.96
64.26:1.3.30
56.78110.78
53.77
51.89-12.29
Inv BP (ours)
8k
55.19:11.62
54.78=1.37
60.7442.84
55.1811.75
54.79+11.3
60.7413.02
Inv VLAD (ours)
262k
67.53+1.03
64.50=1.01
68.98-2.55
67.55+1..11
64.53++.033
69.0442.58
Inv VLAT (ours)
525k
66.79111.47
64.60.11.12
67.50+14.15
66.7511.48
64.61+1.07
67.57
The dimension was reduced to 256 prior to applying BP, VLAD with 1,024 components, FV with 512 components, and VLAT with 16 components. We also applied the proposed invariant BP with the same setting, and VLAD and VLAT with eight times the number of components.
Table 18 shows a tendeney similar to the results of the VGG-feature. Our methods demonstrate better performance than existing methods for both the original test data and angmented test data.
28


Reviews:
[{"review_id": "qhaw42lq4Q", "reviewer": "Reviewer_ZzG6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe experiments using image recognition datasets and simple examples (for example Table 3, 4 and Figure 2) show that the proposed methods (based on invariant feature coding) are able to capture invariant image content while the baseline are not. \n\nWeaknesses\n1. The current version of the manuscript did not provide any complexity analyses of the proposed method. It even did not state about the computational environment where the experiments were done. It is not clear how much computational resources the proposed method would take compared to existing methods.\n2. The manuscript could be improved in terms of presentation. It also seems that some details about the experimental settings are missing. Please see the detailed comments in “Requested Changes”.\n3. The datasets used in the experiments are not large. Even UICU data includes around 12 images per each category in average. I am not sure if this size of the data sets is large enough to validate the performance of certain proposed models in this specific research area. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes variants of the feature coding method which calculates a single global feature by aggregating the local feature extracted from an image.  It proposes 1) group-invariant extensions of feature coding methods, including Invariant PCA and Invariant K-means, and provides 2) a proof that the optimal solution of L2-regularized convex loss minimization, i.e., a linear classifier, is group-invariant when trained on the space where the group acts orthogonally. From the experiments conducted on image recognition datasets, the proposed methods improved the recognition performance compared to the baselines which produce non-invariant features.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be better to add formal definitions or concrete examples for some notations at the place where the notations are first introduced. For example, what would the definitions of “group representations”, “group action” and “representations” be? In addition, about “g and G” in Section 3, I think the authors could introduce the D4 groups here as examples of “g and G”.   \n\nMinor: “\\pi(g)and\\sigama(g)” and “\\piand\\sigama”: space missing? >> “\\pi and \\sigama”?\n\nIt would be minor, but is not the loss in eq. (2) the form of multiclass classification losses? In linear multiclass classification, we need |C| number of weight parameters (w_{1, … ,|C|} unless the authors want to solve ordered multiclass classification problems. The descriptions are not clear enough. \nIn Section 4.5 and Section 5.2, it is not clear what kind of the loss function is used here. In addition, the parameter w should be a matrix whose column size is the same as the number of classes |C|. \n\nAbout Theorem 1. \n1. What is the formal definition of “T” in the eq. (3)? \n2. It is not clear how v_m is connected to the local feature x_n. v_m is the global feature F? \n3. Please include the explicit (formal) definition of the statement “The solution of the L2-regulazed convex loss minimization is G-invariant”? \n4. “Furthermore, w^{(t)}s and x_{i}^{(t)} are orthogonal”: x_{i}^{(t)} is a typo? It was not mentioned in Theorem 1 before. \n5. It is not clear the main point of Theorem 1. It is difficult to me to connect Theorem 1 with the generalization error. Could you elaborate more about the connection between Theorem 1 and generalization error? I am also wondering how practical this connection would be. What I mean, for example, this connection really explains why the proposed method worked better than the baselines in the experimental results?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E8QjGIniis", "reviewer": "Reviewer_y3WA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The ideas and results are presented well. The authors provided an extensive set of experimental results to support their position, including ablation studies. In general, I am convinced that the performance gains they see are real. They also provide theoretical results that support their claims in an idealistic setting.\n\nI don't see any weaknesses that should prevent the paper from being published.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method of augmenting feature maps from static or learned linear-nonlinear neural network layers to encode invariance to standard identity-preserving transformations. They provide theoretical as well as experimental support for their ideas, which together suggest that such a method can lead to improved performance over no augmentation at all. Additionally, they compare their method against augmenting training data with the same transformations, and provide results suggesting that their method achieves approximately equivalent performance with lower training time requirements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Easier:** These suggestions should be pretty easy to include in the final revision.\n\n* You might consider how your contributions fit in with prior work from Ramakrishna Kakarala [1], Rajesh Rao [2-3], Bruno Olshausen [4-6], and colleagues. To be clear, I think your background section is quite comprehensive, as this is a fairly large research area. But in my opinion these works have carved out unique perspectives on the problem.\n\n* You compare against augmenting the training data with the same transformations and note that your method has an improved training time. Isn't there alternatively an additional cost of combining the transformed feature representations at each activity map in your method? How do they actually compare? I think some numbers comparing the compute time or flops between the two methods would be interesting. Also, could you add a row repeating the scores of your methods without augmented training data to Table 5? This will make it easier to compare.\n\n* I'm curious if you have a suggestion for why the accuracy trends in Table 3 are not consistent between the Test Accuracy and Augmented Test Accuracy conditions. Specifically, FMD & UIUC go up; DTD & CUB go down; and Cars stay the same.\n\n**More involved:** These next suggestions might be a lot more work (re-running experiments; re-training networks with a new loss; implementing an alternative published analysis method). I think the paper is publication-worthy without answering them fully. However, I will leave it to the chair to require any of the changes in the context of other reviews.\n\n* I think the standard practice for reporting ML accuracies is to report mean & variance across random seeds (e.g. for weight initialization & data shuffles). It helps to understand how much your accuracy gains. For example, Table 3 Test Accuracy for iBP on FMD is 0.08% better. Is that significant?\n\n* I would be interested to see some more information on how you augmented the training data to obtain the results in Table 5. How does your method compare to [7]? Specifically, I'm interested to know if adding a regularization loss like they describe in eq 5 would improve the data augmentation results. [7] seems to suggest that this is necessary in addition to augmenting the data.\n\n* I don't really see how the GradCAM results support your claims. You note that \"The shape of the heatmap is relatively preserved with the proposed invariant coding method.\" I don't see this. The variance in the heatmap shape seems about the same between iSQRT-COV (Resnet50) (left column) and the proposed method. I didn’t notice much of a difference in the appendix figure either. Is there a way that you can quantify this? Perhaps some sort of intersection over union. You may also consider using GradCam++ [8]; the improved gradient masks might make your point more clear.\n\n**Minor suggestions / typo edits:**\n\n* Abstract: \"explicitly consider group action\" --> \"explicitly considers group action\"\n\n* Bottom of page 1: \"Therefore, this assumption is valid.\" --> I didn't follow this argument. I would appreciate it if you put a little more time into this part. It is a good section because it lays out a high-level description of your assumptions.\n\n* Bottom of page 2, first contribution: Can you reword this contribution? We know that orthogonal perturbations to a linear classifier's weights will result in no difference in activation levels, because of linearity. I don't think this is what you're saying, but it is not clear to me how what you are saying here is different.\n\n* Top of page 6, \"Section Section 4.3\" --> \"Section 4.3\"\n\n* Bottom of page 13, \"Table Table 8\" --> \"Table 8\"\n\n**Citations:**\n\n[1] R. Kakarala, “The bispectrum as a source of phase-sensitive invariants for fourier descriptors:\nA group-theoretic approach,” Journal of Mathematical Imaging and Vision, vol. 44, no. 3,\npp. 341–353, 2012.\n\n[2] R. P. Rao and D. L. Ruderman, “Learning lie groups for invariant visual perception,” Advances\nin neural information processing systems, pp. 810–816, 1999.\n\n[3] X. Miao and R. P. Rao, “Learning the lie groups of visual invariance,” Neural computation,\nvol. 19, no. 10, pp. 2665–2693, 2007.\n\n[4] B. J. Culpepper and B. A. Olshausen, “Learning transport operators for image manifolds.,” in\nNIPS, 2009, pp. 423–431.\n\n[5] J. Sohl-Dickstein, C. M. Wang, and B. A. Olshausen, “An unsupervised algorithm for learning\nlie group transformations,” arXiv preprint arXiv:1001.1027, 2010.\n\n[6] H. Y. Chau, F. Qiu, Y. Chen, and B. Olshausen, “Disentangling images with lie group transformations and sparse coding,” arXiv preprint arXiv:2012.12071, 2020.\n\n[7] G. Benton, M. Finzi, P. Izmailov, and A. G. Wilson, “Learning invariances in neural networks\nfrom training data,” Advances in neural information processing systems, vol. 33, pp. 17 605–\n17 616, 2020.\n\n[8] Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018, March). Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV) (pp. 839-847). IEEE.\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zyBqeteQ6q", "reviewer": "Reviewer_xXoe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths ##\nAs far as I can tell, the presented results appear novel (but I am not an expert in this particular niche), and empirical results seem to suggest modest improvements in hold-out predictive performance when enforcing invariant features. I also found the example in Fig. 2 to be quite illustrative: since the gradCam explanation algorithm is not equivariant to e.g. image flipping, then the associated explanations are not. The invariant features nicely improve this situation.\n\n## Weaknesses ##\nMy main concern with the paper is that I find it very difficult to read. I am not part of the intended audience, so I understand that the paper is not aimed directly at me. Yet, I had to read the paper several times to perform this review, and I still struggle with the elementary aspects of the work. Consequently, the remainder of my review will focus on highlighting places where I got stuck (see Requested Changes below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers learning invariant global features in convolutional neural networks. Here the considered invariance is the group of image content-preserving transformations (e.g. image flipping, 90-degree rotations, etc). Under the assumption that this group is restricted to consist of orthogonal matrices, the paper proves that a linear classifier trained on the resulting invariant features itself becomes invariant. The paper further shows how to perform PCA and k-means on these invariant features.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In line 3 of the introduction it states that $F = \\frac{1}{N}\\sum_{n=1}^N F(x_n) \\in \\mathbb{R}^{d_{global}}$. From this I deduce that $F$ is both a function $F: \\mathbb{R}^{d_{local}} \\rightarrow \\mathbb{R}^{d_{global}}$ *and* a vector $F \\in \\mathbb{R}^{d_{global}}$. Already, at this stage, I am stuck, because those two seem to contradict each other.\n\nThe main result of the paper is Theorem 1. I found this to be highly underspecified to the degree that I do not know what the theorem shows. Specifically:\n* The training data is defined as $\\\\{(v_m, y_m)\\\\}_{m=1}^M \\in \\mathbb{R}^d \\times \\mathcal{C}$. I assume that it's rather $(v_m, y_m) \\in \\mathbb{R}^d \\times \\mathcal{C}$, but even so: what is $\\mathcal{C}$? This space does not appear to be defined.\n* The loss function $l$, can that be any function? The text seems to indicate some notion of convexity of required, but I am unable to tell from the theorem precisely what is required (if anything).\n* I did not understand the meaning of the sentence:\n> ... which implies that $\\\\{(v_m, y_m)\\\\}_{m=1}^M \\in \\mathbb{R}^d \\times \\mathcal{C}$ exhibits the same distribution as $\\\\{(\\pi(g)v_m, y_m)\\\\}_{m=1}^M \\in \\mathbb{R}^d \\times \\mathcal{C}$ for any $g$.\n(apologies for formatting; I could not get Openreview to behave)\n* In the proof, I did not understand the step from the second to the third line of Eq 3.\n* In the last line of Eq 3, $w^{(1)}$ and $v_m^{(1)}$ does not appear to be defined.\n\nAll in all, I do not understand the statement in Theorem 1 nor do I understand its proof. This may be an issue on my side, but I see a significant lack of precision.\n\nAs a side-remark, I enjoyed the section \"Group consisting of identity mapping and image flipping\" in the appendix and I would recommend that it is moved to the main paper as it eased my second reading of the paper.\n\n*Remark:*\nBelow I answered No to Claims and Evidence; this answer is based on my lack of understanding of the theory. I'm happy to change this in case a more precise paper is submitted.\n*update:* After having read the revised version of the paper, I have changed this to Yes as the paper is now sufficiently clear.\n\n*Remark:*\nBelow I answer Yes to Audience, but this is highly speculative as I am not the intended reader group, but I naively assume one such exist.", "broader_impact_concerns": "I have no such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In general, the paper is well written and revised. One reviewer suggested to include computational complexities of considered methods using big O notation. Please take it into account for the final version.
Decision: Accept (poster)