Compact Bilinear Pooling via General Bilinear Projection
Authors: Kun Song
Junwei Han
Feiping Nie
Gong Cheng
Bin Gu
Fakhri Karray
Venue: Submitted to ICLR 2023 (2023)
Keywords: Bilinear Pooling
Bilinear Projection
fine-grained recognition

Abstract: Most factorized bilinear pooling (FBiP) employs Hadamard product-based bilinear projection to learn appropriate projecting directions to reduce the dimension of bilinear features. However, in this paper, we reveal that the Hadamard product-based bilinear projection makes FBiP miss a lot of possible projecting directions, which will significantly harm the performance of outputted compact bilinear features, including compactness and effectiveness. To address this issue, we propose a general matrix-based bilinear projection based on the rank-$k$ matrix base decomposition, where the Hadamard-based bilinear projection is a special case of our proposed one. Using the proposed bilinear projection, we design a novel low-rank factorized bilinear pooling (named RK-FBP), which does not miss any projecting directions. Thus, our RK-FBP can generate better compact bilinear features. To leverage high-order information in local features, we nest several RK-FBP modules together to formulate a multi-linear pooling that outputs compact multi-linear features. At last, we conduct experiments on several fine-grained image tasks to evaluate our models. The experiments show that our models achieve new state-of-the-art classification accuracy by the lowest dimension.

Under review as a conference paper at ICLR 2023
COMPACT BILINEAR POOLING VIA GENERAL BILIN- EAR PROJECTION
Anonymous authors Paper under double-bling review
ABSTRACT
Many factorized bilinear pooling (FBiP) algorithms employ Hadamard product- based bilinear projection to learn appropriate projecting directions to reduce the dimension of bilinear features. However, in this paper, reveal that the Hadamard product-based bilinear projection makes FBiP miss a lot of possible projecting directions, which will significantly harm the performance of outputted compact bilinear features, including compactness and effectiveness. To address this issue, we propose general matrix-based bilinear projection based on the rank-k matrix base decomposition, where the Hadamard-bass bilinear projection and Y = U XV are special cases of our proposed one. Thus, our proposed pro- jection can be used improve the algorithms based on the two types of bilinear pro- jections. Using the proposed bilinear projection, we design a novel low-rank fac- torized bilinear pooling (named RK-FBP), which considers the feasible projecting directions missed by the Hadamard product-basse bilinear projection. Thus, our RK-FBP can generate better compact bilinear features. To leverage high-order in- formation in local features, nest several RK-FBP modules together to formu- late a multi-linear pooling that outputs compact multi-linear features. At last, we conduct experiments
several -ee-iiai image tasks to evaluate models, which show that our models achieve new state-of-the-art classification accuracy by the lowest dimension.
INTRODUCTION
Bilinear pooling (BiP) Lin 2015) and its variants (Li al. 2017b Lin & Maji 2017 Wang et al. 2017) employ Kronecker product to yield expressive representations by mining the rich statistical information from a set of local features, and has attracted wide attentions in many applications, such as fine-gras tre-grained image classification, visual question answering, etc. Although achieving excellent performance, the bilinear features suffer from two shortcomings: (1) the ability of BiP to boost the discriminant information between different classes also magnifies the intra-class variances of representations, which makes BiP easily encounter the burstiness problem (Gao et al., 2020 Zheng ct al., 2019) and suffer from a performance deficit; (2) the Kronecker product exploited by BiP usually makes the bilinear features exceptionally high-dimensional, leading an overfitted training of the succeeding tasks and hefty computational load by increasing the memory storage. Thus, how to effectively solve the shortcomings of BiP is an important issue.
Several approaches have been proposed (Gao et al. 2016 Fukui et al. 2016 Yu et al. 2021 Li
let al. 2017b] Kim al.] 2016) solve the shortcomings of BiP. Among them, the factorized
bilinear pooling (FBiP) methods (Li et al. 20176; Kim et al. 2016 Amin et al. 2020 Yu et al. 2018 Gao al., (020) have been promising leads. The essence FBiP performs a dimension reduction operation on bilinear features. It finds linear projection to map bilinear features into a low-dimension space with their discriminant information among classes preserved using the least dimensions, and then employs Izznormmlizzation to project those low-dimension features on a hyper-sphere. (Bilinear pooling equals the non-linear projection determined by the polynomial kernel function y(x,y) (< y >>² Cao et al. (2016), which makes bilinear features probably linear discriminant.) Thus, the information reflecting the large intra-class variances is abandoned because they do not help distinguish different classes. In this way, the burstiness and high dimension problems are solved simultaneously (Gao et al.) 2020; Wei et al. 2018). This procedure depicted in Figure The sub-figure (a) shows set of samples with large variance. Sub-figure (c) is

---

Under review as a conference paper at ICLR 2023
1
(a)
[b]
[c)
(d)
(e)
(0)
Figure 1: (a): Original data {x, ): (b): The results projected by & [0.55, -0.45; 0.44, 0.55 x,; (c): La-
normalization on &;; 2, = /./|8.. |2 (d): Element-wise "signed--quarr-roo"" x, = sgn((x)) xx Samples in and [1, 1]. respectively. (e)(D) Lz-normalization on regions 1,2,3,4 shrink to points [1. 1,,,,,,,,,, 1..----11111
x. and x;, respectively. Compared (c) with (f), we know the intra-class variances in region 2 and 4 are reduced but those in and increased in (e). Thus, an accurate linear projection better than "signe--squareroon strategy for solving the burstiness problem.
the result after the linear projection and L2 -z-normalization, the intra-class variance is reduced significantly. Actually, the information along e, can be completely discarded, and the dimension of data becomes 1. Let compare sub-figure (c) with (e), we can find FBiP outperforms the combination of the signed-square-row good performance, the key step is to find the accurate projecting direction e2. the dimension and intra-variance reduction can not achieve successfully. Because of the extremely high dimension of bilinear features, it is not easy to find appropriate projecting directions by traditional linear projec- tion. Thus, how to find a set of parameter-effieien model to accurately depict those appropriate directions is curial for solving the shortcomings of bilinear features.
transformation and Lg-normalization To achieve such a
Most FBiP approaches formulate the projection for dimension reduction as Hadamard product-based bilinear projection (Kim et al. 2016 Gao et al. 2020 Li et al. 2017b): f p" (U' x,
y, where U and V are two learnable variables, respectively, and variable defined various from algorithms. Each di- mension of f can be seen as linear combination of dimensions of the Hadamard product z = (U" X, o yz). Consider each dimension of z shown in Figure 2 The Hadamard product only considers the values {uu x, (y, vz)li = j)} and ignores val- ues {uf x. (y7 Vý * j} which are considered by Kronecker product. In this paper, we prove that those ignored values are important for dimension reduction, be- cause they are coefficients of bilinear features on feasible matrix projecting directions (v:u li + j)- Thus, missing them will lead to inaccurate projecting directions of the dimension reduction, which inevitably affects the overcoming for shortcomings of BiP greatly. Consequently, the effectiveness and compactness of FBiP are seriously harmed.
Figure Two ways to connect di- mensions of U²x, and VTy.. (a) is Hadamard product; (b) Kro- necker product.
In this paper, from the perspective of finding accurate and parameter-efficient matrix projecting directions, we analyze the decomposition bases of matrix space, then propose a general bilinear projection based on decomposed rank-k matrix bases. Because of the solid mathematical foundation of the proposed bilinear projection, it can be seen as baseline analyze current FBiP. Employing our novel bilinear projection, we formulate a new FBiP model without missing possible projecting directions. The contributions are listed as follows:
(1) We make a detailed analysis to demonstrate why the traditional FBiP tends to miss a lot of feasible projecting directions. Based on our analysis, we propose a general bilinear projection that calculates the coefficients of matrix data a set of complete decomposed rank-k matrix bases.
(2) Based on the proposed general bilinear projection, we design a new FBiP method named rank-k factorized bilinear pooling (RK-FBP). Because of the capability to learn accurate projecting direc- tions, the calculated bilinear features are highly compact and effective. Utilizing this property, we nest several RK-FBP modules to calculate multi-linear features that are still compact and effective.
(3) We conduct experiments on several challenging image classification datasets to demonstrate the effectiveness of the proposed RK-FBP. Compared with output extremely compact bilinear features (the dimension is 512) and achieve comparable or better classification accuracy.
BiP methods, our model can
2

---

Under review as a conference paper at ICLR 2023
Notation. Throughout this paper, Tr(.) denotes the trace of matrix. tre(.) is the matrix vector- ization (i.e., reshaping a matrix to a vector by stacking its columns on top of one another). rank() represents the rank of matrix. The ij-th element of the matrix A [R=x is represented by the symbol (A jaj. and the i-th element x E Rd is represented by (x);-
PRELIMINARY
2.1
DIMENSION REDUCTION ON BILINEAR FEATURES
Given a set of training samples, among which each instance yields two groups of local features denoted by X2 {x, R" j==1 and y/8 {y, R" Ji=1 (in some cases, 3/8 Xy (Lin 2015
Kong & Fowlkes 2017)). Bilinear pooling (BiP) integrates those two groups of local features into an expressive representation by the following operation.
X
Σ x.y% E TR (a.))ES
(I)
where X is the bilinear feature, and S is the pair set of local features. For convenience, we write bilinear feature as = x.y² in the following content by ignoring the summation symbol. Because most deep neural networks satisfy [S] < min(m, n). the bilinear feature X is a low-rank matrix. Such property can help to design parameter-efficient dimension reduction algorithms.
Factorized bilinear pooling (FBiP) reduces X to h-dimensional vector by linear projection f (Tr(XWT), .... Tr(XW))) ) where W, E m is the r-th low-rank matrix projecting direction. For parameter-effiiinng each W, is decomposed into small matrices various ways, which leads to different FBiP algorithms.
For example, by decomposing W, as W, V, where U, E mx* and V, E Rank k is the rank of W,. The r-th dimension of f can be rewritten as follows.
(f) Tr (x.y7 (...V..U") 1 x V.y.)
(2)
where o is Hadamard product and 1 is a vector with all elements equaling 1. The Eq. is adopted by FBC (Gao et al. 2020) and LowFER (Amin et al. 2020).
Wei et al. 2018) let local feature sets X y and assume W, be symmetric low-rank matrix. The W, decomposed W, U (²( - -, (U,)²² where U. and U. are learned smaller matrices. Thus, let U, = [ U,7. r-th dimension also can be rewritten Hadamard product-based projection:
Besides, some FBiP algorithms (Kong& & Fowlkes 2017
(1) = TT (x (U+)T) - Tr (x.y?U (U,)²) = -1" U"
(3)
Some algorithms (Kim et al. 2016 Yu al.) 2018 Kim al. 2018 replace 1 in Eq. by learnalle vector p,. and transform Eq. to more general formulation presented as follows.
= p" U x v
where P = ....... .P.. E R E m and E gaxl are learnable matrices, respectively. Eq.44) is a general formulation of Eq and Eq. mathematically, analysis on (4..
only focus
2.2 SHORTCOMINGS OF TRADITIONAL FACTORIZED LOW-RANK BILINEAR POOLING
The linear dimension reduction finds the least projecting directions to mapping samples into the low-dimensional space with their discriminant information being preserved. Thus, the projecting directions should satisfy the following criteria: (1) they preserve discriminant information of sam- ples well, otherwise the classification performance will be poor; (2) those projecting directions are linear independent, or the calculated dineenmnnio embeddings can be reduced further.
Next, we demonstrate that the bilinear projection in Eq. can not find suitable projecting directions and harms the compactness and effectiveness of the compact bilinear features. Before doing this, first introduce theorem presented as follows.
3

---

Under review as a conference paper at ICLR 2023
Theorem 1. Suppose U f 1mxx) Jp=1 and V {rg E R 1q-1 are two groups of linear
independent vectors in spaces Rmx1 and [8xx], respectively. If the vector set W {vec( W.)
Rmax 1//2 is constructed by W; = upyg where i 11 (1 1) + p. then W is a set of linear
independent vectors.
The Proof is attached in the Appendix. 1 = m and 12 n, W a complete bases of Rmmxx The r-th element of the low-dimensional feature f presented in Eq. can be rewritten
(,) = Σ ((..),vec(u,r)) vec(x.y))
(5)
where (p,), is the j-th element in the r-th column of P. As seen from Eq. 5) (f), is the coefficient of the bilinear feature vec(x.y;) projected on the projecting direction Σ j==1 p
( vz) which
is a vector in the linear space spanned by vectors B = [vec(u,v] vT), vec(u2
v }}
Then, we prove that the linear combinations of vectors in B can not express all the possible projecting directions.
Let us start from the case I S max n). Because learning a low-rank matrix is challenging problem in the machine learning community (Candès et al. 2011; Liu et al.) [002] Wright et al.) 2009). U and V in Eq. 4 are much likely to be full rank matrices because of no additional constraints on them. Thus, columns in U (and V) are linear independent. Figure 3 gives experimental Proof to support this assumption. Because the columns of U trained by the FBiP model of Eq. are nearly vertical, U is full rank matrix.
According to Theorem I, columns of U and V can generate a set of linear indepen- dent vectors W. Obviously, B is small subset of W. According criterion (2), not only vectors in the space spanned by B but also vectors spanned by W - B can generate the possible projecting directions for reducing dimensions of samples in Rmax1 However, because Eq.(4) only calculates coefficients of the bi- linear feature projected on directions spanned by B, it misses the projecting directions generated by W - B. Considering the number of vectors in W B and B are 12 I and I, respec- tively, more than 99% feasible projecting directions are missed if max(m, n] > > 100.
Mistog am angic
20000
300000
300033
00000
Worse, such a serious issue can not be alleviated by increasing Without loss of generality, we let >> m = n and the first m columns in U and V be linear independent. Thus, the last i-m m columns in (or V) can be represented by the first m columns. To be specific, the r-th (r m) columns in U and are u, = E Up and v, = Ey= brrggg gg. respectively, where arp and brg are auxiliary parameters introduced for easy reading. Obviously, u, and v, can generate projecting direc- tion (ec(u,v,)) = 2p=1 Dapi(arpb Dec(u,, Let us list those generated projecting vectors of different r into a projecting matrix, i.e., í. [vec(um V vin+),,"... v.e_u,,e((uv((
AXC3
04
08
30
90 24
56
Figure 3: The angles between columns of U(2048 x 2048) trained by Eq.(4). Because the angle range is about [85", 955), the columns are nearly vertical to each other.
According to Theorem 1, (uec(u, xu)) (.,,""))))))
p=1.q=1 ))>""
complete base set in Rmax1 So vec(u,
can be any -in-dimensional vector by giving auxiliary parameters (arpbrg 1 mm, 1-11 suitable values. However, because those auxiliary parameters are implicit, we can not train them as other parameters of our model. Thus, the learnable projecting directions can be any possible ones in the solution space. The worst case is that L is rank-1 matrix, which means most of its columns do not preserve the discriminant information of samples. Thus, we can transform L to one column without perfor- mance reduction. implies that the missed projecting directions can not be found by increasing the value of l. Thus, FBiP can not find suitable projecting directions. Of course, auxiliary parameters can let L be full rank matrix. In this case, the projecting directions will be in the space spanned by the missed directions W B. The performance of deep learning crucially depends on tricks in stochastic gradient descent strategies, such as 'momentum' and 'decay'. Because those auxiliary parameters can not be improved by those tricks, their values may not as good as desired, which probably make the learned projecting directions unsuitable. Thus, the performance of FBiP will
4

---

Under review as a conference paper at ICLR 2023
be poor. To alleviate this issue, FBiP should adopt more projection directions to capture enough discriminant information. Nevertheless, due to keeping large intra-class variances, the performance of those unsuitable solutions has an upper bound less than that of the suitable projecting directions. Consequently, the effectiveness and compactness of outputted bilinear features are harmed.
For the case U V. the following Corollary can demonstrate Eq. also misses lot of feasible projecting matrices. The analysis can be made the same way mentioned above, needing to replace Theorem with the Corollary. Thus, do not present the analysis in our paper.
Corollary. Given a set of linear independent vectors {u: E [R=xI the symmetry matrices
S = [win] +m,m"" E 1Bxx Ti=lj=1 are also linear independent.
The Proof is attached in the Appendix.
GENERAL BILINEAR PROJECTION
3.1 DECOMPOSITION OF MATRIX BASES
Given a set of rank-k matrices W (wp p E [RMX)]m
1p-1, if those matrices are linear indepen- dent, then W is rank-k complete base set of the matrix space p=xx
Theorem 1 demonstrates that the rank-1 matrix base set can be decomposed into two vector base sets. According to Theorem I, we can derive the traditional bilinear projection Y U"'VV which is famous in the fields of linear algebra (Strang et al. (1993) and machine learning (Pirsiavash et al.) 2009 Nie et al. 2018). The conclusion is presented in the following theorem.
Theorem 2. If the coefficient of mR on the base " calculated as Upq TTr(X up then the coefficients of X on the whole base set W = {upp' [up E U. E p= are {Upq pm 1.--1 which can form coefficient matrix Y E R=** satisfying:
Y=U²X
where U [M um] mm and V [M Fnl E R"x=
The Proof is presented the Appendix.
Remark 1. Theorem 2 gives way to decompose matrix base set into two low-dimensional vector base sets. base set corresponds to a linear projection in the linear space. Thus, the solution to find an appropriate projection in matrix space (a digeeninensional space) can be transformed to find two vector projections in the low- cow-dimensional vector spaces, which will alleviate the overfitting problem and save the computational sources, including runming time and memory storage. Theorem 2 is a mathematical interpretation of why so many matrix-hased algorithms modeled by the projec- tion y= XV according Theorem 2, the feature reduction algorithms based E4.(14) only find rank-1 matrix projections. Rank-1 projections are a small port of feasible projections. If the applications prefer high-rank projections, Eq. (44 will miss them and lead to performance deficit.
XV (Pirsiavash et al. 2009 Nie et al. 2018 Fukui et al. 2016) work well. However,
In the next section, we will explore a more general bilinear projection which is based on matrix bases with high rank, i..., ks >
3.2 FORMULATION OF THE GENERAL BILINFAR PROJECTION
Consider the coefficient of an arbitrary matrix X E mm projected rank-k matrix base Wp W The coefficient can be caleulated as μp Tr(WX p = = TTU""' by decomposing W,
as W, = V. where Up E mx and V E m Since Tr = vec" vec" (XV,) = vecT (V,) & XU)) the calculation of yp is equivalent to:
yp = Tr (IIx X X)vec( VV,))ee" (U,))
where @ represents the Kronecker product, 1 is the k x A identity matrix.
(7 indicates that the coefficient 3/p equals the coefficient of I, X projected on the ma- trix vec(V (o,(())) Vp))ved? (Up) in mkxkk As the analysis presented in the previous section, the term
5

---

Under review as a conference paper at ICLR 2023
vec(V,) (p)uuc"( consists of a set of free implicit variables that make the appropriate projecting di- rections hard to learn. To overcome this shortcoming, we employ the property presented in Theorem 1 to separate the implicit variables from the linear independent projecting directions.
Since mn < mnk?, the matrices {rec(V (eeec( (Up) are more likely located in a subspace of Rmkxnn whose matrix base can be decomposed into two vector base sets (ūr) and (
Thus, for each vec(V vp)uec (Up), we can find matrix Lp R/ix to hold the following equation:
12 4
vec(V (Up) = Σ 3EE00 (L))
(8)
am1 1=1
where (Lp) is the ij-th element in the matrix Lp, 1 S mk and l2 S nk.
By substituting Eq. 8 in Eq. we obtain new equation to calculate yy by employing Lp. (6)) and (ū; وー Assigning pup) in vector denoted by y [y, [y,.....,, the general bilinear projection is presented as follows:
y= vec(U" (Is @ X)V)
(9)
where indicates the rank of the matrix base set, X Rmxn U j.,.... ز Rmxxi Vi2] -',,...
E 1ankxiz, P [vec(L)),... vee(La)] E [(lll)xx Worthy of note is that the sizes of U and V in ()) become mkk x 1 and nk x l2 respectively, which are different from the ones presented before this paper.
Worthy of note is that P stores the free implicit variables mentioned in the previous section. In our proposed projection, they become explicit variables whose values are easy to con- strain. Unlike Eq. (2) our bilinear projection considers all feasible projecting directions to reduce dimension and intra-class variance. Thus, our projection facilitates yielding more compact features than other FBiP approaches.
Remark 2.
Remark 3. BiP methods MPN (Li et al. 2017a). MPN-COVI et al. 2020 iSORT-COV (Li el al. 2018), SMSO (Yu & Salzmann 2018 and DBTNet-50 (Simonyan & Zisserman 2014) reduce the dimension of local feature I x, by linear projection &, U² x, and then calculate low- dimensenno bilinear features as U" xx,x"" U. According to our analysis, it equals to reduce the dimension of xx= by rank-1 matrix projecting directions, which misses much information. From Eq. D. the projection on local features should be x U (I& @ xx))
The term I @ X in (q. has a lot of zeros which cost lots of memory storage. By introducing the matrix partition as U [U" "LT and V = IVT,-.. VTJ" we can reformulate our general bilinear projection P R respectively.
pd veveel E i= XV.). where U, e Rmxl,, V, e R=xl and
4 RANK-k FACTORIZED BILINEAR POOLING FORMULATION
4.1
Bilinear pooling. Following traditional FBiP method (Kim et al.) 2016), we employ the Eq 9 to reduce the dimension of the bilinear feature x,y?" and formulate new compact bilinear pooling method named rank-k factorized bilinear pooling (RK-FBP) presented as follows.
f,
PT
vec Σ
(U.
bVVV + byy" b"
(10)
where by E R4x1, Dy E Blxx and b" E R are the bias terms of projections U, Rmmxi V, E Raxls and P E (hila)))), respectively.
Remark 4. As seen from Eq. 10), let set ( =1= h. we fix P E {0, 1) in which only the (i, (i 1)h + i)-th element equals "1". Then our model is equivalent to the bilinear pooling model used in (Gao et al. 2020 Amin et al. 2020). If we set the (i, (i 156-177 1)h i)-th element of P, i.e., Pik. as learnable parameters while other elements are set as "0", our model is equivalent to the bilinear pooling module used in (Kim et al. 2016 Lu al. 2016 Yu et al. 2017
Multi-linear pooling, We employ the proposed bilinear projection to generate high-order pooling features. However, the dimension of high-order pooling features is huge, so it is intractable to

---

Under review as a conference paper at ICLR 2023
(x
X
Backbone
RIGFFP
RK-FIP
RICFFIP
yt
Second order
Third owuthrrr
(y
Ostoomad
(b)
Figure 4: (a) The proposed multi-linear pooling structure. (b) RK-FBP module. In (a), we use three Rk-FBP modules to integrate high-order information into the multi-lincar representations. From left to right, the 1-th, 2-th and last modules integrates the second-order information, the third-order information, and the fourth-order information, respectively. Each RK-FBP has its own parameters. At last, choice several types of those features and concatenate them a more informative one.
perform the dimension reduction the high-order features directly. Thus, give recursive way to output the compact high-order pooling features by utilizing several RK-FBP modules. Without ambiguity, we denote f. as the f-th order pooling feature, then the (t 1)-th order pooling feature can be calculated as follows.
Pd
VEC Σ
U +.VC +b:) b"
(II)
For obtained features (-
concatenate them into feature vector, i.c., [(T))T (TTTTTT before feeding them into the classifier. Considering the information of dif- ferent orders may be conflicted with each other, some order pooling features can be ignored in the concatenation. The structure of our model depicted in Figure
Memory Analysis. Compared with the fully bilinear pooling algorithm (Lin 2015). our method results in a great saving of memory storage and computational load. For e-th classification tasks, U. V and P and the classifier hyperplane have km/1 + knl2 + 412h + ch elements. In image classification tasks adopting ResNet50 as backbone, there are m 2048, n 2048, BiP requires mnc = 200mn = 10 parameters to output the classification result. In our RK-FBP, we set l = 512 and
4 at most, there is about 107 parameters, which is smaller than that of BiP by two orders of magnitude, and same as that of traditional factorized bilinear pooling algorithm, e.g., FBC (Gao et al.) 2020 Fang et al., 2019), etc. Due to having a smaller amount of parameters, the computational load is also reduced. Because the high-order RK-FBP is recursively constructed by nesting several RK-FBP modules together, its storage memory is several times of the singular RK-FBP. Considering T-fold multi-lines feature has m. dimension for m-dimension first-order feature, the weights of our multi-linear version of RK-FBP are much smaller.
l2 300,
5
EXPERIMENTS AND ANALYSIS
5.1 EXPERIMENTAL SETTING
We conduct experiments the image classification tasks to evaluate the proposed RK-FBP model. The adopted image datasets are Describing Texture Dataset (DTD) (Cimpoi et al. 2014). MINC-
2500 (MINC) (Bell al.) 2015), MIT-Indoor (INDOOR) QQuattoni & Torral ba 2009) ) and Caltech- UCSD Bird (CUB200) (Xie et al. 2013). Cars196 (Krause et al.) 2013), Aircraft (Maji et al.] 2013) which are the texture image dataset in the wild, the material dataset, the indoor scene dataset, and two fine-grained image datasets, respectively. We incorporate our RK-FBP module in Deep Neural Networks as Figure 4 and implement by PyToretPaszkke et al. (2017). ImageNet-pretrainngg are taken from torchvision Marcel & Rodriguez (2010). The adopted backbones are VGG-16 and ResNet50. For those image data, each image resized into 418 418. VGG-16 generates x 28 x 512 feature map from con5 3 layer. The number of local features d 512 and number of local feature is 784. For ResNet50, generates a X 14 X 2048 feature map which result 196 2048-dimensional features. Following Gao et al. (2016), the compact bilinear feature f are sent to the softmax classifier. Each RK-FBP module has its own parameters U, V, and P. which are updated by the back- propagation algorithm. The Pytorch is on GPU Quadro RTX 6000. The optimization is done using SGD, weight decay is 0.01, learning rate is 0.01, momentum 0.9. The number of epoch is 60 and batch size is 32.

---

Under review as a conference paper at ICLR 2023
5.2 ABLATION EXPERIMENT
We adopt VGGI6 as the backbone for the ablation experiments.
Rank k. Different & means different matrix bases, the compact bilinear features also have different performance. To demonstrate how the rank affects Rk-FBP, we perform RK-FBP on Indoor data set with varied & E 11, ,1,,,.,1........ 15]. The number of column and row projections are set 4 122 I E {50, 100, 150, 200, 250} and the dimension of the bilinear feature is set to h = 512. The results are depicted in Figure. 6 We can find that While
250 and k = 2, RK-FBP achieves the best performance 83.5%. When / = 250 and & = L, the accuracy is about 81.9% with about 2.6% decrease. When 1 => 150, the accuracy is relatively stable after k > 2. The accuracy variance of k for each is within 2%. It means only h = 2 is enough. But when 1 < 150, the maximal accuracy corresponds large k 9. This means when the number of matrix bases less, large rank is preferred. Because large rank matrices have more parameters, so they can depict the projecting directions more accurate. Thus, the observation may indicates fewer dimensions need more accurate projecting directions to capture the discriminant information. Besides, we employ
70
325
256
302
1004 2045
3072 4004
6000
Figure 5: Accuracy RK-FBP with different Rank & on Indoor (dimension = 512). When is small, & should be large to achieve its largest accuracy.
Figure 6: Accuracy of RK-FBP with different dimensions Indoor. FBC and RK-HFBP need large dimensions achieve the best result.
8, = U [ 60 x.) to improve the dimension procedure in SMSO (Li et al. 70),a), MP & find that increase of the rank k, the accuracy of those methods can be improved. And the maximum gain is 6.6% achieved by MPN on the Indoor dataset. Besides, we can find that with the increase of k. the accuracy will increase first and then be stable. This is consistent with our RK-FBP when the number of matrix bases is s small. Both experiments illustrate the importance of rank k. This indicates that current dimension reduction methods in those bilinear pooling models limit their expressive ability. Dimensionality h. We compare our model with FBC, RK-FBP, and RK-HFBP (presented in (3.(38) on Indoor. We report the testing accuracy at h [128, 256, 512, 1024, 2048, 3072, 4096, 8192]. We set the rank k- = 2 for our RK-FBP and 2 = 12 = 250 for
Salzmann 2018 iSQRT-COV(Li et al., 2018). As seen from Table 1
Table 1: Accuracy (%) with different rank of the projection on different datasets.
7
10
13
RK-FBP. For RK-HFBP and FBC, the rank of them are empirically set as k = 8 and k = 14, respectively. As seen from Figure 6 RK-FBP outperforms FBC and RK-HFBP with different dimensionality. Specifically, RK-FBP is higher than FBC about 4.1%
SMSO
Indnor
EVTD
59.3
21.3
83.1
70.9
83.1
71.3
71.7
MPN
Indour
76.5
76.9
79.2
82.6
83.1
ISQRE-COV
Indoor
DTTD
69.7
79.5
80.5
70.5
81.9
71.5
82.5
71.5
83.4
70.9
when h 512. It validates that our model can find more discriminative projecting directions than RK-HFBP and FBC. After the dimension increase as h > 512, the performances of RK-FBP are relatively stable. Considering RK-FBP achieves the best result among the three approaches, this indicates that the performance of RK-FBP saturates. Nevertheless, for FBC and RK-HFBP, their performance continues to increase when > 512. It may be because they can not find accurate projecting directions, so more discriminant information is acquired when the feature dimension in- creases. However, they can not surpass the RK-FBP. This implies that the missing discriminant information can not be completely solved by increasing the dimension of the compact bilinear fea- tures.
5.3 COMPARISON WITH THE STATE- OF-TE-OF-THER ARE ALGORIHTM
VGG-16 backbone. We first compare with full bilinear pooling methods: BCNN, improved BCNN, DeepO2P, Ge et, RUN, DeepKSPD. Except BCNN, those methods adopt enhanced normal- ization strategies to overcome the bursitness problem. As seen from Table only DeDeNet slightly surpasses our RK-FBP on CUB200 dataset by 0.6%. On the rest five datasets, our RK-FBP achieves the best results, and the largest gain is 8.6% (more than DeepO2P) on the MINC dataset.
8

---

Under review as a conference paper at ICLR 2023
Table 2: Comparisons for BiP methods in terms of Average Precision (%)
Backtonee
Feature dim.
Param
MINC
CUB220
Cars196
VG
( moonyyn & Zisserman 2014
VGG226
4096 1314 KC
120.43
60.1
64.5
73.0
N0.4
70.7
74,1
BCNN
VGGI1
52.43
67.5
77.5
74.3
80.0
91.1
$7.1
Improwis
411M
85.8
88.5
VGGG
78.4
CBP
VGG16
SK
1.6:M
67.7
76.8
54.0
90.
$7.1
MPN
vul Pouin
VGG16
32K
13.1M
68.0
76.5
76.2
86.1
92.2
$9.9
Ga Waneest
2013
VGG316
133K
87.1
92.5
LXBP [Noir
2017
VGSG26
1250
65.8
N4.2
90.9
87.3
FBN 15 matan
VGG11
0.39
67.8
82.9
S7L7
5488
X01
2048
1.46M
69.3
79.5
78.0
8500
RUN
VaGi6
89
M118
VGG16
131K
$1.0
86.5
901
91.0
ReDro Rattonwsseee Piletl
VGGL6
33K
103M
M0.2
H6.7
92.2
91.0
ISORESOOO MoyaRRgee 20lls
VGG16
33K
308M
69.7
79.5
79.2
87.3
91.1
90.0
yl murr
87.2
McNaer Cim
VGGG6
10K
80.3
FCRN Miesein
VGGG1
4K
66.8
K3.6
90.5
TBC GoNEEEi PUKI
VGG16
$192
10M
71.5
79.9
S0.2
80.3
90.3
$7.1
TKB Careral 2022
VGG16
9X
96
68.2
80.5
78.2
86.0
$1.3
91.1
RK-FUP (21 ours]
VGGL6
512
45.2M
72.2
83.5
83.4
86.8
92.5
91.9
VGG16
TKBE
ReSNeeSo
DITN (A k010
DBTNNet
2K
87.5
94.5
91.2
ReDro Remioee AIM
ResNet50
33K
103M
80.2
86.7
91.2
91.0
ISORT
ResNet50
33K
312M
70.4
80.1
$1.3
87.9
91.1
90.9
SMSO
ResNet50
2K
5.75M
72.5
79.7
83.8
92.2
58.9
RK-FRP (2" 000s]
ResNer50
512
112M
73.2
84.5
84.9
86.7
92.9
92.8
RK-FDP (3 2
ResNet50
1024
112M
74,9
86.2
85,1
88.1
94.5
93.6
Then, we further compare with the medium-scale bilinear features, ReDro and iSQRT-COV. ReDro is grouped bilinear pooling method which reduces the dimension of bilinear features to 33K. As shown in Table 2 Our RK-FBP achieves comparable results with ReDR. Specifically, RK-FBP is more than ReDro by 0.1% and 0.3% CUB200 and Car196, and 3.3% and 0.9% Indoor and Aireraft datasets, respectively. iSQRT-COV reduces the dimension of the local features from 512 to 256 using 1 x convolution layer. So, the dimension of its bilinear features is 33K too. Our RK-FBP outperforms iSQRT-COV on the rest five datasets except CUB200. Especially, RK-FBP significantly surpasses iSQRT-COV by 4.0% Indoor. Because the ReDro and iSQRT-COV are designed to overcome the burstiness of bilinear features, their dimensions are relatively low.
We conclude two conclusions: (I) those positive results over the high dimensional BiP methods indicate that high dimension limits the efficiency of bilinear features in the succeeding classification tasks. (2) Considering that the above comparison methods employ matrix normalization strategies, the comparable results mean our RK-FBP can effectively solve the bursitness problem caused by the intra-class variances, that indicating that our method can find suitable projecting directions.
We also compare several state-of-the-art compact bilinear pooling methods: CBP, LRBP, FBC, and TKPF. As for those methods, we report the best results and their corresponding dimensions. As shown in Table 2 our RK-FBP achieves better classification accuracy using the lowest dimension. Compared to 4K, the smallest dimensions of those comparison methods, 512 and 1024 are ex- tremely compact. It is mainly because those compact methods can not find suitable projecting directions since they miss a lot of possible directions. As for the number of parameters, our RK- FBP uses more parameters than other compact methods. However, compared with the importance of accurate projecting directions, the cost of such an amount of storage is acceptable. Most of our model's parameters contributed by P, whose parameters can be reduced by constraining it using sparsity regularization. It will be done in our future work.
At last, we report the performance of our multi-linear models (3+ + + means capturing the 2- order and the 3-th order information). As seen from Table 2 our multi-linear model achieves the best accuracy while the dimension is 1024. Especially, our multi-linear features surpass HPB by 0.2%, where HBP is an enhanced bilinear pooling method fusing features across different layers by Hadamard product. Because the dimension of traditional compact bilinear pooling is high, is hard to calculate multi-linear features by nesting them together. This excellent result shows the benefit of extremely low dimension features in the community.
ResNet50 backbone. As shown in Table 2, our models (bilinear and multi-lineat surpass other comparison methods on most datasets for the comparison methods, which indicates our model is robust to the backbones. Besides, we also compare with DBTNet-50 is a deep structure constructed by bilinear transformation blocks. We find that DBTNet-50 surpasses RK-FBP on CUB200 and 196s196 being weaker than RK-FBP Aircraft. However, considering that DBTNet-50 applies the bilinear pooling in every layer, the results reported are not significant. Besides, our multi-lines features can outperform DBTNet-50 most datasets with smaller dimensions. It may be because

---

Under review as a conference paper at ICLR 2023
the transformation used DBTNet-50 is based on the rank-1 matrix bases whose learning ability is limited. So DBTNet-50 may be improved by our proposed rank-k bilinear projection.
6 CONCLUSION In this paper, we reveal that traditional factorized bilinear pooling tends miss feasible projecting direction. To overcome this disadvantage, we propose general bilinear projection to formulate a pooling module called rank-k factorized bilinear pooling (RK-FBP). Our RK-FBP has three ad- vances: (1) RK-FBP is derived from general bilinear projection based on complete matrix bases, so no feasible projecting directions will missed. (2) Because the projecting directions are ac- curate, the learned bilinear features not only compact but also discriminative. Those benefits give RK-FBP the power to produce more expressive compact bilinear features. Conducted exper- iments demonstrate that RK-FBP outperforms various Mate-of-the-art algorithms challenging image classification tasks.
REFERENCES
Saadullah Amin, Stalin Varanasi, Katherine Ann Dunfield, and Günter Neumann. Lowfer: Low- rank bilinear pooling for link prediction. In International Conference on Machine Learning, pp- 257-268. PMLR, 2020.
Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in context database. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3479-3487, 2015.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright, Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Anoop Cherian, Piotr Koniusz, and Stephen Gould. Higher-order pooling of cmn features via kernel linearization for action recognition. In 2017 IEEE Winter Conference on Applications of Com- puter Vision (WACV), pp. 130.138. IEEE, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie. Kernel pooling for convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921-2930, 2017.
Xiyang Dai, Joe Yue-Hei Ng, and Larry S Davis. Fason: First and second order information fusion network for texture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7352-7360, 2017.
Melih Engin, Lei Wang, I Luping Zhou, and Xinwang Liu. Deepkspd: Learning kernel-matrix- spd representation for fine-gru menegrained image recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 612-627, 2018.
Pengfei Fang, Jieming Zhou, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Bilinear attention networks for person retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8030-8039, 2019.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Mareus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint 1oX.::6606.018000 2016.
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling, In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 317-326, 2016.
Zhi Gao, Yuwei Wu, Xiaoxum Zhang, Jindou Dai. Yunde Jia, and Mehrtash Harandi. Revisiting bilinear pooling: A coding perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3964-3961, 2020.
10

---

Under review as a conference paper at ICLR 2023
Mengran Gou, Fei Xiong, Octavia Camps, and Mario Sznaier. Monet: work. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 3175-3183, 2018.
Moments embedding net-
2018.
Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix back propagation for deep net- works with structured layers. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2965-2973, 2015.
Hervé Jégou, Matthijs Douze, and Cordelia Schmid. On the burstiness of visual elements. In 2009 IEEE conference on computer vision and pattern reeognition, pp. 1169-1176. IEEE, 2009.
Jin-Hwa Kim, Kyoung- Woon On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Syoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv: arXiv:1610.0433 1610.04325, 2016.
Jin-Hwa Kim, Jachyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. NeurIPS, 2018.
Shu Kong and Charless Fowlkes. Low-rank bilinear pooling for fine-grained classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 365-374, 2017.
Piotr Koniusz, Fei Yan, Philippe-Henn Henri Gosselin, and Krystian Mikolajezyk. Higher-order occur- rence pooling for bags-of-words: Visual concept detection. IEEE transactions on pattern analysis and machine intelligence, 13-3),3 2016.
Piotr Koniusz, Hongguang Zhang, and Fatih Porikli. A deeper look at power normalizations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5774- 5783, 2018
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Peihua Li, Jiangtao Xie, Qilong Wang, and Wangmeng Zuo. second-order information helpful for large-scale visual recognition? In Proceedings of the IEEE international conference on computer vision, pp. 2070-2078. 2017a.
Peihua Li, Jiangtao Xie, Qilong Wang. and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 247-955,2018
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Factorized bilinear models for image recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2079- 2087, 2017b.
Tsung- Yu Lin. Bilinear enn models for fine-grained visual recognition. Proceedings of the IEEE international conference on computer vision, pp. 1449-144 1459, 2015.
Tsung-Yo Yu Lin and Subhransu Maji. Improved bilinear pooling with cnns. arXiv preprint
arXiv:1707.06772, 2017.
Guangean Liu, Zhouchen Lin, Shuicheng Yan. Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE transactions on pattern analysis and machine intelligence, 35(1):171-184 2012.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, 29:289-297, 2016.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine- vision package of torch. In Pro- ceedings of the 18th ACM international conference Multimedia, pp. 1485-1488, 2010.

---

Under review as a conference paper at ICLR 2023
Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recognition and quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14225-14234, 2021.
Feiping Nie, Han Zhang, Rui Zhang, and Xuelong Li. Robust multiple rank-A: bilinear projections for unsupervised learning. IEEE Trunsactions Image Processing, 28(5):2574-258 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hamed Pirsiavash, Deva Ramanan, and Charless C Fowlkes. Bilinear classifiers for visual recogni- tion. In NIPS, volume 1, pp. 3, 2009.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes, In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 413-420. IEEE, 2009.
Saimunur Rahman, Lei Wang, Changming Sun, and Luping Zhou. Redro: Efficiently learning large- sized spd visual representation. In European Conference on Computer Vision, pp. 1-17. Springer, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Gilbert Strang, Gilbert Strang, Gilbert Strang, and Gilbert Strang. Introduction to linear algebra, volume 3. Welleslyy-Cambridge Press Wellesley, MA, 1993.
Lei Wang, Jianjia Zhang, Luping Zhou, Chang Tang, and Wanqing Li. Beyond covariance: Feature representation with nonlinear kernel matrices. In Proceedings of the IEEE international confer- ence on computer vision, pp. 4570 4578, 2015.
Qilong Wang. Peihua Li, and Lei Zhang. G2denet: Glohal gaussian distribution embedding network and its application visual recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2730-2739, 2017.
Qilong Wang, Jiangtao Xie. Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep cnns meet global co- variance pooling: Better representation and generalization. JEEE transactions on pattern analysis and machine intelligence, 43(8):2582-2599 2597, 2020.
Xing Wei, Yue Zhang, Yihong Gong, Jiawei Zhang, and Nanning Zheng. Grassmann pooling as compact homogeneous bilinear pooling for fine-grained visual classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 355-370, 2018.
John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. Advances in neural information processing systems, 22, 2009.
Lingxi Xie, Qi Tian, Richang Hong, Shuicheng Yan, and Bo Zhang. Hierarchical part matching for fine-grained visual categorization. In Proceedings of the IEEE international conference on computer vision, pp. 1641-1648, 2013.
Jieping Yc, Ravi Janardan, and Qi Li. Two-dimensional linear discriminant analysis. Advances in neural information processing systems, 17:1569-1576, 2004.
Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. Hierarchical bilinear pooling for fine-grained visual recognition. In Proceedings of the European conference on computer vision (ECCV), pp. 574-589, 2018.
Kaicheng Yu and Mathieu Salzmann. Statistically second-order pooling. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 600-616, 2018.
Tan Yu. Yunfeng Cai, and Ping Li. Toward faster and simpler matrix normalization via rank- update. In European Conference Computer Vision, pp. 203-219. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2023
Tan Yu, Xiaoyun Li, and Ping Li. Fast and compact bilinear pooling by shifted random maclaurin. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3243-3251, 2021.
Tan Yu, Yunfeng Cai, and Ping Li. Efficient compact bilinear pooling via kronecker product. pp- 121-127, 2022.
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 1821-1830, 2017.
Jianjia Zhang, Lei Wang, Luping Zhou, and Wanqing Li. Beyond covariance: Sice and kernel based visual feature representation. International Journal of Computer Vision, 129(2):300-33)) 2021.
Xingfu Zhang and Xiangmin Ren. Two dimensional principal component analysis based indepen- dent component analysis for face recognition. In 201 International Conference on Multimedia Technology, pp. 934-936. IEEE, 2011.
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Learning deep bilinear transformation for sine-grained image representation. Advances in Neural Information Processing Systems, 32, 2019.
Appendix
7 DEFINITIONS AND PROOFS
Definition 1: In the linear space mmxx, the inner production between two matrices A mmx and m x. is defined as A, >= Tr(AB").
Definition 2: (W ynn a group of complete bases of RRmx", if and only if. for any matrix X pip=1 Rmxn, there is an unique non-zero vector = [yı, y2, -umn] to hold the following equation:
X
Up W
(12)
where is the projected coefficient o X on the p-th base Wp,
The vector y can be seen as the representation of the matrix X on the base set mm p=1
Definition Suppose W ( mmxn man with max frank( k. Wmnxn is sp=1 complete orthonormal base set of the matrix space Rmxx if the following equation holds
{
Tr WWW/W,)
Wg)
p=q p
(13)
Theorem Suppose U (up E Rmx linear independent vectors in two spaces R.mxl and Rax1, respectively. If the vector set W (vec(W;) E gmxa Ji=i constructed by W, unvy where = 4119 1) 12, then W is a set of linear independent vectors.
1 pp=1 and {r, E Bax1 1 are two groups of
Proof. Suppose U [u,.... [ and V [Vi, v.l. Because (..( vec (u, v
Up
Ng. there is U & V (vec(u,y]] vf', vec(u, v2 1,.... vec uu ( v..... vec(u,, vf) vec U, v )- @ is the Kronecker product. Thus, columns in Ug V are the vectors in W. According to the property
of Kronecker product, there is rank(U V) 1142 Thus, W is full rank, and the column in W is group of linearly independent vectors.
Theorem 2. If the coefficient of X TR on the base u is calculated as Upg = -Tr(X" up! 7) then the coefficients of X the whole base set W =upp" up E U. E V)m=i.q=1 1p which can form coefficient matrix Y E R satisfying:
Y U XV
(14)
where U '",,'.., um R mm and V [M vnl E Raxn
13

---

Under review as a conference paper at ICLR 2023
Proof.
Because Upg Tr(X" upxy))
Ny by arranging Ypg matrix, there is Y
U'Xv.
0
Corollary, Given a set of linear independent vectors {u; E RmxII
li=1 the synmetry matrices
[u:u] +u,u?) E mR 11 Jillj=1 are also linear independent.
Proof. According to Theorem 2, we know the matrices {u,uz matrices. Thus, the solution of following function is Ctj O.
M Jillj=1 is set of linear independent
Σ
(15)
Let us consider the following equations with variables C' = =(djlt i 1, .4j = 1, , j}
(u.u² + u
Σ Yu +
(16)
2
1=1 jCL
If 4, = O, we let Gj = 4 for i < and G; = de. The function in 16) has an non-zeros
solution. This violates the conclusion that Cij = 0 for Eq. Thus, all elements in C' are means the matrix set S is group of linear independent matrices.
DISCUSSION ON BURSTINESS AND NORMALIZATION STRATEGIES
8.1 WHY THE BILINEAR POOLING CAN ENHANCE THE DISCRIMINANT ABILITY OF LOCAL FEATURES
Given a local feature x; Rmx1, its corresponding bilinear feature is z, vec(x,]]). Let us calculate the inner product between bilinear features of x; and xj, there is
Z4 Zj >= vec(x;x (ee(xxxx)) (x?xx) x2
(17)
Comparing Eq. with the polynomial kernel function k(x,, x,) (a(x x,) d)p, the inner product between bilinear features cquals to polynomial kernel function with = 1, = 0 and
p 2. Therefore, we can claim that the bilinear feature the explicit result of the non-lincar projection determined by the polynomial kernel function.
How to let the features outputted by the backbone fit well to the hyper- paremeearssstrrs of polynomial kernel function is not the concern of our paper. Thus, we suppose the backbone can generate features good enough to satisfy the hyper--paaaae parameaaaameess....
X.X, m is just the sum of bilinear features of columms in X, So its enhanced discriminant ability can be also interpreted by the polynomial kernel function.
As for the local features X; [x:1 X.2.
xie] mxee, the bilinear pooling Z:
As well known, the polynomial kernel function can improve the classification performance of the support vector machines which is linear discriminant classifier. Thus, the discriminant information in bilinear features can be well depicted by the linear projection.
This the basis that we can employ linear projections to reduce the dimensionality of bilinear features, which is crucial step to solve the burstiness problem of bilinear features.
8.2 How THE BURSTINESS REDUCE THE PERFORMANCE OF MODELS
Burstiness phenomenon on the image features is first analysed the literature Jégou et al. (2009), which focus the feature vectors obtained by the bag-of-words frameworks.
In ofg-of-words frameworks, each dimension of feature vectors corresponds to an visual word collected from the whole image data. For the i-th image, the feature vector is x; [aa, F22, .. Fim]" T e mm where the value of æıy is the frequency of the j-th visual word ap- peared in the i-th image. Suppose the label of i-th image yi, the average value of the j-th
14

---

Under review as a conference paper at ICLR 2023
dimension of the whole e-th class of samples is E, of the e-th class.
N. N. En.ec Bay where Ne is the sample number
In some cases, the j-the visual word may appear a of times in the i-th image but does not appear so many times in other images. This will make the value of Zij much larger than the average value of the j-th dimension, i.e. Tz. This is the burstiness phenomenon of bag-of-words features. Geometrically, this burstiness phenomenon increases the variance of samples in each class in the feature space.
Therefore, literature Wei all 2018)) describes the burstiness phenomenon as "the problem that the feature descriptor is not invariant enough where the feature elements may have large variances within the same class."
Thus, the burstiness problem can be summarized as the problem of large intra-class variance.
The burstiness of bilinear features is caused by the outer product on the local features. Let us take the three-dimensional data as an example. Suppose there are three samples x1 [1, 1, 1.1
X2 [3, 1, 1), X3
1.2, 1.2, 1.2] The bilinear features of X1, X2, and X3 Zı ved xec[
1. 1,1,1,111111
Z2 pec(x2x3 = 3. 1, 1,3, 1, 111,1 1J", and Z3 vec(x333) = [1.44, 1.44, 1.44, 1.44, 1.44, 1.44, 1.44 44, 1.44"
1 T
We
can calculate the average of Z1 and Z3. i.c. Z
Z1 4-22
2
[1.22, 1.22, 1.22, 1.22, 1.22, 1. 1.22, 1.22, 1.22, 1.22 If most of local features are close to and X3, can be considered the average bilinear feature of the whole class. Thus, Z2 is far away from the average z. Because this phenomenon is similar to the burstiness of bag-of-words features, it is also called as the burstiness of bilinear features, which also expands the intra-class variance.
For some images, there are the illumination variations and appearance changes in them Gao et al. (2020;; Wei al. (2018), which make the features extracted by deep neural networks also have some variances within each class. The variance may reflect the singular values of the matrix storing local features, i.e., X Ei u. where a; is the i-the singular value and u; and V; are the corresponding singular vectors. Because the bilinear pooling on xx" = =i=1 variances will be expanded by the outer product and cause the burstiness of bilinear features in the deep frameworks.
=Ein u Those
Because the burstiness will affect the similarity between bilinear features Wei et 2018 will affect the performance of models based on similarity.
For classification tasks, the bilinear features likely linear discriminant. Due the high dimen- sionality of bilinear features, there are lot of feasible solutions of classifiers can fit the training data well. However, the large intra-class variance caused by the burstiness may let the classifier select a bad solution which has the bad generalization on the test dataset, and the performance of bilinear features harmed. Thus, how to alleviate the burstiness problem very important for learning the bilinear features.
8.3 SIGNED ELEMENTWISE SQUARE-ROOT OPERATION
Signed elementwise square-root operation transforms a vector x = [x1, T2, ", mm] to a new vector x 12, rmm 17 in which f;; is calculated
4 = sgn (x:
x
(18)
Consider the value of /2/ there is
| <= 1 x
(19)
[x:]]
In this way, we can find that / Varal let the value
|x:]|
close to 1. Let us consider the value sg)(
Eq. (18) lets the vector x E mmxx close to the centers [+1, +1,.... +1] E Rmxl where the symbol + is determined by s2.()...
Thus, if samples from different classes are well separated and are located in different quadrants in the feature space, the elementwise square-root operation can reduce the intra-class variance well.
15

---

Under review as a conference paper at ICLR 2023
And the generalization is good. Such a requirement be satisfied by the bag-of-wond words features. is because the bag-of-words frameworks generate feature vectors according the frequency of each visual elements appeared in each image, where the visual elements often generated by clustering algorithms and thus have explicit similarity meanings.
The literature Wei al. (018 reveals the features extracted by the deep neural networks do not meet the requirement signed Elementwise Square-root transformation.
8.4 Ly-NORMALIZATION
Lg dggnormalization is widely used in the deep neural networks to enhance the generalization ability of learned features. For a vector X; E gmx1 the Lg-normalization of x; is x, defined as
/
(20)
Therefore, the Euclidean distance between two & and 8, is - 8
Thus, after
L2 Ln-normalization the Euclidean distance between samples can be replaced by the cosine distance. For the features extracted by deep neural networks, L2 -y-normalization will reduce the variances in each class Meng et al. 2021), because the variance information is along the radius direction in the feature space.
8.5 HOW THE FACTORIZED BILINEAR POOLING ALLEVIATE THE BURSTINESS
Because the bilinear features are linear discriminant, so the dimension of bilinear features can be reduced by set of linear projections. In this way, the variance along those eliminated directions discarded. Then, the Lz-normalization strategy is adopted to reduce the variance along the radius direction in the low-dimensional feature space. In this way, the intra-class variances are reduced and the burstiness problem is alleviated.
RELATED WORK
We review works improving bilinear pooling in two aspects: 1) Enhancing the bilinear feature's 's effectiveness; 2) Reducing the dimension of the bilinear feature for greater efficiency.
Effectiveness improvement. There are several different routines to improve the bilinear feature's effectiveness. For example, G2DeNet (Wang et al. 2017), FASON (Dai et al. 2017) and MoNet (Gou et al. (018) take the first-order statistics into consideration beside the bilinear pooling, KP (Cui al. 70.7). HOK (Koniusz et al., 2016) and HOP (Cherian al. 2017) extend the second-order pooling higher-order pooling. By considering bilinear pooling depicting correlations across different features a specific kernel space, Ker-RP (Wang et al. 2015 Zhang al. 202T) employs kernel functions instead of inner-produce to strengthen the representative capability of the bilinear feature. Bilinear pooling can be enhanced by capturing correlations between features yielded by different layers Yuu al., 2018). Besides, the strategies of normalization are also proved to be effective in improving the discriminant ability bilinear features. For instance, improved bilinear pooling (IBP) (Lin & Maji 2017). MPN-COV (Li et al. (018)) and their variants (Koniusz et al.) 2018) explore the matrix normalization to moderate the singular values of the bilinear matrix. By using those strategies, the discriminant ability of bilinear features increased by a large margin compared with the original bilinear features.
Dimension reduction. The bilinear feature is high dimensional matrix, which limits its efficiency and makes it prone to over-fitting. To speed up the classification and suppress over-fitting, FBN (Li et al., 2017b) reduces the dimension of the weight matrices in the classifier by replacing each weight matrix with product of two low-rank matrices. In paraliel, some methods focus on the dimension reduction for the original finst-order features before bilinear pooling. BCNN (Lin) 2015) uses principal component analysis (PCA) to learn the projection matrix for reducing the dimension of first-order features. Specifically, uses the projection matrix learned from PCA as initialization of 1 1 convolution layer and then trains the network in end-to-end manner. The PCA is also utilized in LRBP (Kong & Fowlkes] (017) and iSQRT-COV (Li et al. 2018) for dimension reduction on the first-order features. Except those linear projection with learnable parameters, CBP
16

---

Under review as a conference paper at ICLR 2023
(Gao et al. 2016 Yu al. 2021) approximates the operation of bilinear pooling by the polynomial kernel function. Then CBP employs two kernel approximation methods to reduce the dimension of bilinear feature, tensor sketch and random Maclaurin (RM), which achieve better performance than the PCA. Inspired by CBP, the strategy of CBP is adopted in MLB (Fukui et al. 2016) for cross-modal understanding. In CBP, the projections is formulated first-order features by two random projections which is then fused by Hadamard product. Hadamard product-based low-rank factorized bilinear pooling (HFBP) (Kim et al. 2016) replaces the random matrices with two learnable matrices and obtains a new compact bilinear pooling algorithm. Such technique is then adopted by HBP (Yu et al., 2018) for fusing features in different layers. Although so many dimension reduction algorithms have been proposed in past years, there lacks a general perspective to understand them. In this paper, we review the dimension reduction algorithms from the perspective of finding appropriate projection directions. Moreover, we reveal that the projections used in those algorithms tend to miss a lot of possible projecting directions, which reduces the effectiveness of the compact bilinear feature.
reduce the dimension of the
9.1 INTERPRETATION OF BILINEAR FEATURES FROM THE PERSPECTIVE OF SUPERVISED SPECTRAL GRAPH PARTITIONING
9.1.1 SPECTRAL GRAPH PARTITIONING
Let us consider the similarity between two samples x and with polynomial kernel function K yx, = (7xx* y) + d)² where 7 and d are two hyper-parameters adjusted for different data distri- butions. If we calculate the inner product between two bilinear features vec(xx") and vec(y" y"): < [xxxxyy² >= (xy=)²,
find that it is the special case of the similarity calculated by the polynomial kernel function with y = and O. The similarity matrix S on a data set can be used to describe the discriminant information between
As well known, the polynomial kernel function can be adopted to solve the inseparable problem by support vector machines (SVMs) and k-means algorithms. The ability to solve the inseparable problem also enhance the discriminant ability of vectors x.
Let us adopt the spectral graph partitioning as tool to analyze the performance of bilinear features. Before doing this, introduce the procedure of spectral graph partitioning.
Definition 4. Given set of samples {x, N the adjacent matrix S is defined by
Syy K (x xj), itj
(21)
0,
i=j
where K (X;,x)) is a kernel function whose value monotonically decreases with respective to the distance between x and y. e.g., K (x, y) = exp( - y13) (7 > 0).
The normalized Laplace matrix is defined as I, = I D 1/2 1/2 where D is diagonal matrix with D., Ez=1 S.y. The following equation is held:
SJ و Vdj
L'L-=
(22)
where f = f2, ... fNJ E R For traditional bi-class spectral graph partitioning, the objective function to minimize Normalized cuts whose the objective function is presented as follows.
J
f
min
[E EE[-1118
Vd
Vd
Because the distance between x; and x, is smaller, the value k(x,, x;) is larger, the solution the above optimization problem is
Vdl, E Ci Vdh,
f. =
(23)
I E Cz
17

---

Under review as a conference paper at ICLR 2023
C1
C
C2
C3
C
C3
C3
C
C
(a)
(b)
Figure 7: (a) Traditional spectral graph partitioning. Our designed spectral graph partitioning. There four classes {Ci) When we use bi-class based model to deal with the fourth classes, the partitioning of two models are different. In (a), the first class consists of C1, and the second class consists of {C2, Ca, C } in (b). the first class consists of {Ci, C2). and the second class consists of (C3, C.}. When extend the bi-class model to multi-class version, the strategy different.
where C; is the collection of samples in the i-th class.
The above is the traditional graph partitioning model. In our graph partitioning model, we let the ker- nel function k(x,, x,) whose value monotonically increases with respective to the distance between x; and y.. Thus, the objective function of our spectral clustering can be formulated as follows.
J dd
f
3j
(24)
fE(-1.)
ddz
Because the task only has two classes, we can obtain the solution presented in Eq. (3) by maximiz- ing the objective function.
However, the two graph partitioning algorithms presented in Eq. 23 and Eq. 24 behavior quite different when they are employed to solve the multi-class tasks. clearly show the difference, we suppose there are 4-classes of samples. The i-th class is denoted as C2, and the samples from the same class are listed together. The kernel matrix of two methods are graphically shown in Figure Both methods consider the f Lf which equals sum of values in the dark-colored boxes in Figure Because the tradition model minimizes the value of 1 Lf. so it tends to find dark-colored boxes consisting smaller areas. As for model, it maximizes I' Lf, so it finds the dark-colored boxes having the largest area. In this way, the two classes obtained by the traditional graph partitioning model {C1) and {C2, Cs, C. which indicates that it separates reset classes. When we want to separate the classes, we need 4 vectors {f, 1 to indicates the measermembesship of samples.
class of samples from the
However, as for our model, the obtained classes consist of samples in (Ci, C2 and (Cs, C4), which not consistent the ground-truth of samples.
But, if we further separate C₁ from C2 and C3 from C4, i.e., obtain another partition (Ci, C3} and (C2, C4). we can partition the data set well. Such a partitioning result corresponds to fi [vd1, /dz, -Vd, ddg and fa [vdz,- -Vd2, vdg. -vdjl. Obviously, the two dimensional samples [vd,, vdi). [Vdz,- -v d2))). dd, v dd). -ddd, v di are the centers of the four classes, and they are separated well. Thus, we can employ those two vectors { to extend the bi-class model 4-class model.
And so on, for a C-class task, our model only need to employ = Iog2(C") vectors { to embedding the original samples. Compared with the traditional graph partitioning model needing C' vectors, the embedding of our model is extremely compact.
18

---

Under review as a conference paper at ICLR 2023
(a)
(b)
Figure 8: (a) Traditional spectral graph partitioning. (b) Our designed spectral graph partitioning. The tradi- tional graph partition method needs dimension to code classes of samples, while our method can only use (C) dimension to code C' classes of samples.
Remark 5. Our proposed graph partitioning algorithm can only employ log2(C) dimensions to rep- resent the embeddings of samples. Theoretically, we can employ 64-dimension to code 26a classes of samples. The geometrical illustration of samples obtained by two types of graph partitioning methods are shown in the Figure 8
Now, we employ our proposed graph partitioning algorithm to extract the features. Our mult-class graph partitioning model is formulated
1
f
ΣΣΣ
FF
dd
dg
Tr(F"
SD 1 F)
max
FFF-DD
(25)
max Tr(F" (-D (-D 1/22DD 1/2
FrF=De
[(D11/2SD 11/2) FF
max
F-F-D.
where h = (Cg)(C").
Remark 5. The solution of F in Eq. 25) is the log2(C")-th smallest eign-vectors of D 1D2 1/2
Lemma. If the sample N is large enough, the eigen-vectors of D SDDSSDDD 1/2 are equal to the eigen- vectors of S. Thus, the embedding F can be obtained by performing siguevalue decomposition on S. Proof: Because ii-th element in 1/2 is d,
Because, the S;y is the value of k(x,, x,)
VEN Sij
which is monotonically increase with distance between x; and Xy. If N is large enough, the error ||d,- - dz < e. Under this assumption, there is that dj ss d2 Rs d3 Rs let D 1/22DD 1/2 (d:)²s. Thus, we have that the embedding F can be approximated by the find (Cg)(C")-th smallest eign-vectors S.
dy. Therefore, we can
Theorem 3. When X is the bilinear features, S Xx'"XX becomes the similarity matrix determined by the second-order polynomial kernel function, Let us denote F as the solution in Eq. 25 thus there exists linear projection I. to let F I
Proof. Σ uu FFT
Because
EL= o.n.u"" where a; is the i-th small cigen-value of S, thus, Σl youuf, there
Because S = x"x, there -
=[U1, U2.
The above theorem prove that the bilinear features be reduced extremely low dimensional space with there discriminant information being preserved. This is the theoretical base why method can reduce the bilinear features to 512.
Remark.
19

---

Under review as a conference paper at ICLR 2023
10
RELATION WITH EXISTING METHODS
10.1 PCA BASELINE
PCA baseline is exploited in BCNN and iSQRT-COV for dimension reduction for bilinear features. It adopts a 1 x 1 convolutional layer to reduce the dimension of the local feature from m to l (1 < m). Then the bilinear pooling performed on the 1 dimensional features to I? of-dimensional feature. To be specific, given N local features
= [x1, X2. ,xy] E gT*N, PCA baseline generate the compact local features by Y = X. Thus, the compact bilinear feature is obtained by
= XXU
(26)
According to the theorem the Eq. (6)) actually employs a set of rank matrix bases to reduce high-dimensions bilinear feature XX to a compact one Z. According to our analysis, if the data prefers large rank matrix bases, some discriminant information is lost in the dimension reduction procedure.
To overcome this shortcoming, should employ l projecting matrices {U,} to reduce the local feature X and calculate the compact bilinear feature Z as follows:
U XXU,
(27)
10.2 RANDOM MACLAURIN (RM)
RM is employed in CBP (Gao et al. 2016) for compact bilinear pooling. inspired by the success of CBP, many RM-based algorithms are proposed YYu et al., 2021 Fukui et al. 2016). RM employed two random projecting matrices U and V to reduce the dimension of bilinear features.
Σ X
(28)
where z is the compact bilinear feature, and o is the Hadamard product.
set as an identity matrix, Eq. (28 with Eq.44) are the same. The difference is that U and V in Eq.44 are updated by gradient descent algorithms while those in Eq. (2)) are random variables found by sampling values from the random distributions. may be why methods of Eq.44 outperform methods based on R.M in terms of classification accuracy. However, because the parameters Eq. 4 do not need to update via gradient descent algorithm, involves less computation.
Let us compare Eq 28 with Eq. 4
10.3 TWO-LEVEL KRONECKER-PRODUCE PRODUCT FACTORIZATION (TKPF)
TKPF supposes a projecting matrix P can be decomposed as the Kronecker product presented as follows.
P
(a) C B(a)
(29)
q=1
Then, by further decomposing A (0) and (()
as A = I, & A (g)
and B(a) = I, & TKPF
(4)
formulates the compact bilinear pooling
Σ L 3 (B )Txx""
xx (I, A
(0)
(30)
where A ( and ( are learnable parameters. Because the scale of A (0) and B (0) be adjusted
by the parameter r, TKPF can use very less parameters to reduce the dimension of bilinear feature.
However, find the TKPF has the following shortcoming.
20

---

Under review as a conference paper at ICLR 2023
TKPF assumes that every matrix P can be decomposed as Eq. 29 not true. If we construct the i-th column of as P: Σ u
However, this assumption is where U, [ui.... - uzl.
vr = [Vi Val. Consider U" & V" equals
U, 00 V, jui Vr, U.
(31)
|u 8 vi uf
u
(32)
Compared E1. with Eq.322. we know P Pal can not be decomposed as Eq. 29
Besides, A = I, < (e)
is also a Kronecker product-based decomposition, which may be not hold. This means KTPF ignores lot of feasible projecting directions. If the data prefers those missed feasible projecting directions, the performance is compact is bad.
Let compare TKPF with our bilinear model. For easy comparison, transform bilinear projection y = p" ree X XV.) into a vector-based form presented as follows.
y L
r & U (ee((""))
(33)
Then, the projecting matrix in Eq.33
Q P
T
U
(34)
Obviously, Eq. 299 and Eq 34 look similar. The difference between them is that (3) has matrix L while Eq. 29 does not. L plays the role to select the columns in V @ U to form matrix can not be decomposed by Kronecker product. This makes our can be any matrix. Therefore, the minimal difference makes our proposed projection is an accurate while the projection in Eq. (29 not.
Our proposed bilinear projection is general, it be used analyze the performance of other dimension reduction algorithms for matrix data, such as two-dimensional principal analysis (Zhang & Ren 201T) and two-dimensional linear discriminant analysis (Ye et al. 2004).
As discussed above, TKPF employs some inappropriate matrix decompositions to construct the projection, which means TKPF also can find accurate projecting matrices. Although the per- formance of TKPF looks good on its reported datasets, TKPF may suffer from a great performance reduction in other applications. Thus, the application range of TKPF is limited. At last, compared with our proposed bilinear model, the dimension of the compact bilinear feature is still high, e.g., for best accuracy, the dimension 96 96.
10.4
CODE PERSPECTIVE OF BILINEAR POOLING
FBC Gao al. 2020 proposes general compact bilinear model from the coding perspective.
Given h low rank atoms (V,Uf), FBC Gao et al. (2020) encodes x,y? into f, RRxx1 by solving the following matrix-hased sparse coding optimization problem:
min IXxYT
Σ
f.NN 1 +
(35)
where À is a trade-off between the reconstruction error and the sparsity. U, E Rm and V, E B are two rank-k matrices decomposed from the 1-th rank-k matrix atom. Here, U, and V, are learned by the deep model through the whole set of original samples. The optimization problem in Eq. (35) has a closed-form solution presented as follows:
P( v vP 'P(UTx
x, o VYy,)
(36)
f sign(f)) max(abs(F)) 2.0) 0)
21

---

Under review as a conference paper at ICLR 2023
where U = UU,.... Un] R and V .V..... Val E R=* are the learnable parameters of the dictionary. P E Rhxhk is a fixed binary matrix with only elements in the row columns ((1 ( 1) h) + to (th) being "T", where I E h1.
Because the above formulation involves the matrix inverse operation, it adopts a relaxation strategy ((V"UP" -"vv")))-P((" -P(("X x, V"y, y,) (Ü x, Ye).
r P(U VVV y:) f f= sfgn(f.) max max(abs(t)) - 0)
(37)
However, can prove that the coding model is mathematically equivalent to the traditional model. And we can propose that a new coding-bace based model does not need relaxation. We consider the vector-based coding model miny ||x - w ? where y is the coefficient on of vector x on the dictionary W. There is solution = (WW7) > vec(xy?). ) the i-th column of W as V.U," and as f, we can obtain the formulation (3)
WWx According to FBC, when we replace the x as
is the coefficient Similarly, we can construct another coding-model miny ||x- w (WW")- of the atoms W(W" W)-1. There is the solution y W x which is linear projection. Thus, according to the Eq. 36), if we replace W the i-th column w; as V,Uţ the matrix-based dictionary P((U"UP* v' "VP"))=PPP" -PR"" The i-th column of u, @ V; where u, and v; are the i-th column of and V. This is because R^ vec(x..f) = U o V Yr- Because u, @ v; vec(u,y? r?). so P(UU"UP" v" 'PR"' can also be transformed to a set of matrices. Thus, we have another type of matrix-based coding model, and the bilinear feature can be outputted by f= p= (U" V² y,) which is equivalent to the formulation Eq.(37). And the solution is accurate one.
-P"(UU
This is why directly derive the bilinear pooling model from general bilinear projection f p' v'eeeeU (" "xyy"'V other than the coding based framework.
10.5 FORMULATION OF RK-HFBP
Eq.44) has a matrix P E Rxxh, so can not be very large. Thus, (q. is rank-1 Hadamard product-basee FBiP. As discussed FBC, the rank of projecting matrices are also very important. So we improve Eq. by giving its projecting matrices more rank. By employing the same strategy of our proposed rank-k bilinear projection Z U² (I, & XjV Σ U XV, we obtain new rank-k Hadamard product-brac based bilinear pooling (RK- HFBiP) presented as follows.
P'PT DEC
Uua o Vay!))
(38)
We will RK-HFBiP as a comparison method in our ablation experiments.
11 ABLATION EXPERIMENTS
Table Accuracy (%) with different orders on variance datasets. (a. ... .c) means the feature constructed by concatenating the a-th, o-th..... c-th order statistic information.
T
2
3
4
(2.3)
(2.4)
(34)
(2,3.4)
Indoor
$3.8
85.1
MNIC
83.4
835
83.0
84.3
84.5
83.7
84.2
CUR-200
868
36.9
86.7
87.4
87.2
87.2
87.6
92.5
92.1
92.6
93.8
Number of orders. We can nest several RK-FBP modules together to learn compact representations with high-order information, which we denoted as RK-FBP-M. We evaluate the influence of the order T on RK-FBP-M. We set = l2 h 512, and vary d among (2, 4]. Because model allows us to concatenate the features with different orders, we have types of features. As seen from Table 3 the multi-linear features win the bilinear features by significant margin. Besides,
22

---

Under review as a conference paper at ICLR 2023
with the increase of the order, the result is not always increased. To be specific, the (2,3), (2, 4), and (2.3, 3, 4) features achieve the similar accuracy. Thus, in our paper, we set the order parameter as (2,3) for our the multi-linear model.
Normalization Strategy. In the fully bilinear pooling approaches, the performance is cru- the normalization strategy. Thus, explore how the normalization
cially dependent strategies affect our proposed RK-FBP. Thus, we add two normalization strategies adopted by fully bilinear pooling Lin (2015) and improved bilinear pooling Lin & Maji (2017) be- fore RK-FBP modules. We denote them by 'SgnSqrt' normalization and "SgnSqrt+log' normalization, respectively.
We also compare the result with only 'log' normalization. At last, all bilinear features should be normalized to an 'unit' vector by L2 normalization.
The results are shown in the Table [ As seen from the results, we can find that the results are similar with little variance. This is bit different from Table [ which employs normalization strategies after projection. In Table [ the combination of feature reduction and normalization strategies increases the performance. This may be because the features after projection is still high, there are still much information of large intra-class variance. So the normalization strategies can improve the performance of models by removing the bad information. Because RK-FBP does not use 'SgnSqr'' and 'log' but achieves comparable results with those methods using normalization strategies, it indicates proposed model solve the burstiness problem caused by large intra-class variances.
Table 4: Accuracy (%) with normalization strategies variance datasets.
CUR-2200
Cars-196
RK-FBP
83.8
83.4
86.8
92.5
SpnSqnt
83.7
83.5
86,9
92.3
log
92.7
SpnSyttlog
$4.0
83.5
86.7
92.6
23


Reviews:
[{"review_id": "p3Tymla7Zwt", "reviewer": "Reviewer_MKth", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to study how to effectively achieve compact bilinear representations. To this end, the authors proposed a low-rank factorized bilinear pooling method. Specifically, as shown in Eq. (10), $k$ parallel 1x1 convolutions are used to reduce feature dimension before bilinear pooling, and then $k$ bilinear representations are summed following by a fully-connected layer. As such, size of bilinear representations could be controlled by 1x1 convolution and fully-connected layer. The experiments are conducted on several small-scale fine-grained image benchmarks.", "summary_of_the_review": "I have reviewed this work in previous conference, and the current version achieves clear improvement on writing and experiments. However, discussion on previous works and experimental evaluation could be further strengthened. Particularly, the authors should clarify meaning of ‘Param’ in Table 2. ", "strength_and_weaknesses": "Strength:\n\n+: The idea on parallel 1x1 convolutions for reducing feature dimension before bilinear pooling seems bring clear improvement (Fig.5 and Table 1), which may be a feasible solution to balance performance and model complexity for bilinear pooling methods.\n\n+: On several small-scale fine-grained image benchmarks, the proposed method achieves competitive results with a relatively low representation dimension.\n\n+: The proposed method is clearly written and is easy to implement.\n\nWeaknesses:\n\n-: Could $\\mathbf{U}$ and $\\mathbf{V}$ in Eq. (10) be shared?\n\n-: The idea on combination of 1x1 convolutions and fully-connected layer for obtaining compact covariance representations appeared in [r1] (Section 3.5 and Table 4), where sequential 1x1 convolutions (not parallel) are used to reduce feature dimension before covariance pooling, and then a fully-connected layer is adopted after covariance pooling. It would better discuss relationship between the proposed method and [r1].\n\n[r1] Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization. IEEE T-PAMI, 2021.\n\n-: The concerns on experiments.\n\n(1)\tAre hyper-parameters $k$ and $h$ sensitive to various datasets? Particularly, could the proposed method be flexibly adopted to large-species classification (e.g., iNat2017 [r2]) with the same hyper-parameters?\n\n[r2] The iNaturalist species classification and detection dataset. CVPR, 2018.\n\n(2)\tI wonder what meaning of ‘Param’ in Table 2. Particularly, why Param of iSQRT-COV with backbone of ResNet-50 is 312M? To our best knowledge, iSQRT-COV is parameter-free itself. For dimension reduction, a 1x1 convolution (2048, 256) is used and contain about 0.5M parameters. If ‘Param’ in Table 2 contains one of classifier, it clearly varies for different datasets, while RK-FBP (2+) and RK-FBP (2+,3+) should have different parameters.\n\n(3)\tBesides parameters, FLOPs and running time are important metric to evaluate model complexity.\n\n(4)\tI would like to know how about performance of the proposed method under training-from-scratch setting (e.g., ImageNet-1K).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IEver_lN4a", "reviewer": "Reviewer_xdH1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on bilinear pooling in fine-grained image classification tasks. Bilinear pooling (that generates second-order features) is an effective method to capture details of fine-grained classes and achieves better performance than first-order features. Because bilinear features are high in dimension, many bilinear pooling methods adopt Hadamard product-based bilinear projection to reduce dimensionality. However, the paper argues that this misses \"a lot of possible projecting directions which will significantly harm the performance\". The paper then proposes low-rank factorized bilinear pooling (RK-FBP). The paper argues that RK-FBP \"does not miss any projecting directions\". The paper further extends RK-FBP to pooling features of multiple layers and forms multi-linear features. Experiments show the proposed methods achieves the state-of-the-art.", "summary_of_the_review": "The paper improves over low-rank bilinear pooling (Kong & Fowlkes, 2017) so the novelty is limited. It does not discuss how to address burstiness of bilinear features, particularly multi-linear features. The literature reports that normalizing bilinear features (e.g., using signed square root and L2 normalization) is crucial. Presentation of the paper should be improved further. Therefore, the paper is rated as Reject.\n", "strength_and_weaknesses": "Strength\n- Improving bilinear features is important as standard bilinear features (via outer product) is very high-dimensional.\n- Datasets used in the experiments are good.\n- The way to produce compact multi-linear features is interesting.\n\nBelow are weaknesses.\n\n- The paper does not discuss how to address the burstiness of bilinear/multi-linear features. To note, for bilinear features and low-rank bilinear features (Lin, 2015; Kong & Fowlkes, 2017), it is crucial to apply signed square root transform and L2 normalization. \n\n- It is confusing to state \"(RK-FBP) does not miss any projecting directions\" in the abstract. Authors should clarify, e.g., how to define \"any directions\".\n\n- In Introduction, it is unclear how existing factorized bilinear pooling (FBiP) methods address the burstiness problem. It seems the paper only mentions that these methods aim to reduce dimensionality of bilinear features. Can authors discuss?\n\n- Figure 1 is confusing. In this example, is it just a Fisher Discriminant Analysis? There is not much insight from this figure.\n\n- Figure 2: For methods that use Hadamard product, they learn U and V to optimize classification accuracy, which is the goal of the classification problem. Then, what does it mean by \"missing values\" in Hadamard project.\n\n- Section 4: The proposed Rank-K Factorized Bilinear Pooling (RK-FBP) seems like an extension of low-rank bilinear pooling (Kong & Fowlkes, 2017). This makes the novelty incremental.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DiXLUF64Fpy", "reviewer": "Reviewer_yyVh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Many Factorized Bilinear Pooling (FBiP) uses Hadamard product-based bilinear projection. This paper reveals that the Hadamard product misses a lot of possible projection directions. This paper proposes a general matrix-based bilinear projection based on rank-k matrix base decomposition. This paper uses the proposed bilinear projection to design RK-FBP. To leverage high-order information in local features, the authors nest several RK-FBP modules. ", "summary_of_the_review": "The Rank-k dimension reduction for bilinear pooling seems novel and effective compared with existing bilinear pooling. It appears to be technically correct. Parameter analysis and comparison with state-of-the-art seem to be sufficient. \n\n", "strength_and_weaknesses": "Strengths\n------------\n- The rank-K projection, which performs dimension reduction with a sum of $k$ dimension-reduced vectors, is novel and effective.  \n- The mathematical discussion seems to be correct. \n- This paper shows the connection of the proposed method to existing BiP methods.  \n- This paper shows the effects of parameters and combination with several improved BP (SMSO, MPN, iSQRT-COV). \n- The performance of RK-FBP is higher than state-of-the-art with lower embedding dimensions. \n\nWeakness\n-------------- \n-  Reading this paper is heavy due to many equations. \n-  Some discussions are based on the speculation of authors. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject