Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Many Factorized Bilinear Pooling (FBiP) uses Hadamard product-based bilinear projection. This paper reveals that the Hadamard product misses a lot of possible projection directions. This paper proposes a general matrix-based bilinear projection based on rank-k matrix base decomposition. This paper uses the proposed bilinear projection to design RK-FBP. To leverage high-order information in local features, the authors nest several RK-FBP modules. 
strength_and_weaknesses: Strengths
------------
- The rank-K projection, which performs dimension reduction with a sum of $k$ dimension-reduced vectors, is novel and effective.  
- The mathematical discussion seems to be correct. 
- This paper shows the connection of the proposed method to existing BiP methods.  
- This paper shows the effects of parameters and combination with several improved BP (SMSO, MPN, iSQRT-COV). 
- The performance of RK-FBP is higher than state-of-the-art with lower embedding dimensions. 

Weakness
-------------- 
-  Reading this paper is heavy due to many equations. 
-  Some discussions are based on the speculation of authors. 


clarity,_quality,_novelty_and_reproducibility: Clarity
--------
This paper should improve readability. 

P3 above Eq.(4):  “replace 1 in Eq.(2) by a learnable vector $p_r$”, but Eq.(4) is $P$ and $_r$ is not used for $U$ and $V$. 

P5, 6 In the general bilinear projection in Eq.(9), the dimensions of $P, V, U$ differ from the previous sections. To clarify this fact, it would be better to use different notations. 

Sec.10 
Normalization: The discussion of 2DLDA seems to be irrelevant to the results of the normalization strategy.  

There are typos, eg.,  

P3 Theorem 1. $R^{m\times n}$ seems to be $R^{mn \times 1}$

P7 Rk-FBP 

P8 Figure 6: RK-RBP

P13 $rank(U\otimes V) = l_1 + l_2$ seems to be $l_1 l_2$

Quality
----------
Sec.2.2. argues that the $U \otimes V$ does not cover the entire feature space of bilinear features. I agree that if $U$ and $V$ are fixed, it is correct. However, in deep learning, the dimension reduction and classifier are learned jointly. For Eq.(5), all dimensions in $vec(x_s y_t^T)$ are involved in learning dimension reduction followed by the classifier. 

P4. “Because those auxiliary parameters are implicit, we can not train them as other parameters of our model.”, “However, those free auxiliary parameters probably make the learned projection unsuitable.” are based on the author’s speculation.  

Novelty
----------
The rank-K projections seem to be novel and effective. In the rank-k projection,  each dimension of the output vector is given by $y_p = {\rm Tr}(U_p^TXV_p )$ and the vector is concatenation of $p=1,..h$ values. In contrast to the conventional rank-1 projection $y = U^T X V$, each dimension of rank-K projection is a summation of k projected values, where $K$ is the rank of $U_p, V_p$. Thus, the rank-K projection is unlikely to miss the discriminative information of the original high-dimensional space when the output dimensions are small.

Reproducibility
--------------------
P7: The number of epochs, batch size, and learning rate for SGD are not shown. 


summary_of_the_review: The Rank-k dimension reduction for bilinear pooling seems novel and effective compared with existing bilinear pooling. It appears to be technically correct. Parameter analysis and comparison with state-of-the-art seem to be sufficient.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on bilinear pooling in fine-grained image classification tasks. Bilinear pooling (that generates second-order features) is an effective method to capture details of fine-grained classes and achieves better performance than first-order features. Because bilinear features are high in dimension, many bilinear pooling methods adopt Hadamard product-based bilinear projection to reduce dimensionality. However, the paper argues that this misses "a lot of possible projecting directions which will significantly harm the performance". The paper then proposes low-rank factorized bilinear pooling (RK-FBP). The paper argues that RK-FBP "does not miss any projecting directions". The paper further extends RK-FBP to pooling features of multiple layers and forms multi-linear features. Experiments show the proposed methods achieves the state-of-the-art.
strength_and_weaknesses: Strength
- Improving bilinear features is important as standard bilinear features (via outer product) is very high-dimensional.
- Datasets used in the experiments are good.
- The way to produce compact multi-linear features is interesting.

Below are weaknesses.

- The paper does not discuss how to address the burstiness of bilinear/multi-linear features. To note, for bilinear features and low-rank bilinear features (Lin, 2015; Kong & Fowlkes, 2017), it is crucial to apply signed square root transform and L2 normalization. 

- It is confusing to state "(RK-FBP) does not miss any projecting directions" in the abstract. Authors should clarify, e.g., how to define "any directions".

- In Introduction, it is unclear how existing factorized bilinear pooling (FBiP) methods address the burstiness problem. It seems the paper only mentions that these methods aim to reduce dimensionality of bilinear features. Can authors discuss?

- Figure 1 is confusing. In this example, is it just a Fisher Discriminant Analysis? There is not much insight from this figure.

- Figure 2: For methods that use Hadamard product, they learn U and V to optimize classification accuracy, which is the goal of the classification problem. Then, what does it mean by "missing values" in Hadamard project.

- Section 4: The proposed Rank-K Factorized Bilinear Pooling (RK-FBP) seems like an extension of low-rank bilinear pooling (Kong & Fowlkes, 2017). This makes the novelty incremental.

clarity,_quality,_novelty_and_reproducibility: Clarity and quality of the paper should be improved. Section 2 and 3 are hard to follow. The presentation should be organized in a more structured way. The novelty is incremental as it seems like an extension of low-rank bilinear pooling (Kong & Fowlkes, 2017). It is unclear how to reproduce the results as the paper does not discuss how to address the burstiness of bilinear features and does not provide open-source code.


summary_of_the_review: The paper improves over low-rank bilinear pooling (Kong & Fowlkes, 2017) so the novelty is limited. It does not discuss how to address burstiness of bilinear features, particularly multi-linear features. The literature reports that normalizing bilinear features (e.g., using signed square root and L2 normalization) is crucial. Presentation of the paper should be improved further. Therefore, the paper is rated as Reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work aims to study how to effectively achieve compact bilinear representations. To this end, the authors proposed a low-rank factorized bilinear pooling method. Specifically, as shown in Eq. (10), $k$ parallel 1x1 convolutions are used to reduce feature dimension before bilinear pooling, and then $k$ bilinear representations are summed following by a fully-connected layer. As such, size of bilinear representations could be controlled by 1x1 convolution and fully-connected layer. The experiments are conducted on several small-scale fine-grained image benchmarks.
strength_and_weaknesses: Strength:

+: The idea on parallel 1x1 convolutions for reducing feature dimension before bilinear pooling seems bring clear improvement (Fig.5 and Table 1), which may be a feasible solution to balance performance and model complexity for bilinear pooling methods.

+: On several small-scale fine-grained image benchmarks, the proposed method achieves competitive results with a relatively low representation dimension.

+: The proposed method is clearly written and is easy to implement.

Weaknesses:

-: Could $\mathbf{U}$ and $\mathbf{V}$ in Eq. (10) be shared?

-: The idea on combination of 1x1 convolutions and fully-connected layer for obtaining compact covariance representations appeared in [r1] (Section 3.5 and Table 4), where sequential 1x1 convolutions (not parallel) are used to reduce feature dimension before covariance pooling, and then a fully-connected layer is adopted after covariance pooling. It would better discuss relationship between the proposed method and [r1].

[r1] Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization. IEEE T-PAMI, 2021.

-: The concerns on experiments.

(1)	Are hyper-parameters $k$ and $h$ sensitive to various datasets? Particularly, could the proposed method be flexibly adopted to large-species classification (e.g., iNat2017 [r2]) with the same hyper-parameters?

[r2] The iNaturalist species classification and detection dataset. CVPR, 2018.

(2)	I wonder what meaning of ‘Param’ in Table 2. Particularly, why Param of iSQRT-COV with backbone of ResNet-50 is 312M? To our best knowledge, iSQRT-COV is parameter-free itself. For dimension reduction, a 1x1 convolution (2048, 256) is used and contain about 0.5M parameters. If ‘Param’ in Table 2 contains one of classifier, it clearly varies for different datasets, while RK-FBP (2+) and RK-FBP (2+,3+) should have different parameters.

(3)	Besides parameters, FLOPs and running time are important metric to evaluate model complexity.

(4)	I would like to know how about performance of the proposed method under training-from-scratch setting (e.g., ImageNet-1K).

clarity,_quality,_novelty_and_reproducibility: The paper is generally very written. The idea on parallel 1x1 convolutions for reducing feature dimension before bilinear pooling may be a feasible solution. The proposed method is easy to implement. More evaluation and discussion seem be necessary.  
summary_of_the_review: I have reviewed this work in previous conference, and the current version achieves clear improvement on writing and experiments. However, discussion on previous works and experimental evaluation could be further strengthened. Particularly, the authors should clarify meaning of ‘Param’ in Table 2.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors suggest a type of bilinear projection to deal with second order features.  Two reviewers suggest acceptance, one reviewer suggests rejection.

Strengths:
- strong experimental results in comparison to other bilinear pooling techniques for fixed backbones

Weakness:
- paper is very hard to read, there are language issues and 
- there is no "proof" for the main result but it is based on claims of the authors
- there are no results where this technique is used for end-to-end training
- the authors manipulate quite heavily the style file leading to the fact that in the original submission (and the revised version) the headers of Section 4 and 4.1 are even overlapping - in the revised version this even worse. The plots and tables are very hard to read.

There were also doubts on the novelty of the approach. While this could not be fully resolved, I think that this paper is not ready yet for publication and the authors should invest considerable effort in improving clarity of the presentation, make their argument precise (ideally with a real proof) and add the requested experiment on end-to-end training. 

Details:
- Theorem 1 and the Corollary are straightforward and can be omitted (the corollary is formulated unfortunately, someone could wrongly deduce that the dimension of symmetric matrices is $m^2$ and not $\frac{m(m+1)}{2}$)


justification_for_why_not_higher_score: The presentation and discussion is not ready yet for publication.
Decision: Reject