# Fast Normalization for Bilinear Pooling via Eigenvalue Regularization
## 4 Experiments
### Implementation Details  
**Models**. For the experiments with three fine-grained visual recognition datasets, following the common settings [9; 13; 31], we used the networks pre-trained on the ImageNet1K dataset, and extracted feature matrix \(\mathbf{X}\) after \(relu5\_3\) in the VGG-16 and before global average pooling in other networks. Subsequently, bilinear pooling of Equitation (1) was followed, except for ResNet-50, where we applied the compact bilinear pooling of PSMR [9] because the feature dimension \(D\) here was too large. When training from scratch, we adopted the CNN backbones from [8; 10; 21], where the feature dimension of \(\mathbf{X}\) was reduced to 256 with a \(1\times 1\) convolutional layer.  
**Training and Evaluation.** For fine-grained visual recognition tasks, we adopted the same image pre-processing techniques as those in [20; 50] for a fair comparison. Input images were resized to 448 \(\times\) 448, both for training and inference. Random horizontal flipwas the only data augmentation method applied during training. The training is composed of two stages. At the first stage, we only trained the last FC layer for 100 epochs with Stochastic Gradient Descent (SGD) optimizer, whose learning rate was 1.0, weight decay was \(1\times 10^{-5}\), and the momentum was 0.9. Then, we fine-tuned the whole network for another 100 epochs with the initial learning rate of \(2.7\times 10^{-2}\) and weight decay of \(1\times 10^{-4}\). The batch size was set as 32 and the learning rate was divided by 10 when the training loss remained higher than 0.99 times the current minimum loss value for 10 epochs. At the second stage, we removed the normalization part and continued to fine-tune the network for another 15 epochs. The initial learning rate was 0.01 and was reduced by 10 at the 9 th epoch. For inference, we averaged the classification scores for both the input image and its flipped version as the final prediction. This setting is quite common in the bilinear pooling field [8; 13].  
For training a CNN on large-scale dataset ImageNet1K, according to refs. [10; 20; 21], during training, a random crop was made on the original image and was then resized into \(224\times 224\). A random horizontal flip was also applied. At the first stage, the optimization strategy corresponded to SGD with an initial learning rate of 0.1. The network was trained within 80 epochs for ResNet-18 and 65 epochs for ResNet-50. The learning rate was decayed by 10 in the epochs 30, 60 and 75 for ResNet-18 and was decayed by 10 in the epochs 30, 45 and 60 for ResNet-50. At the second stage, the bilinear normalization was removed and the network was fine-tuned with the learning rate of 0.001, which was decayed by 10 after 9 epochs in 15 epochs in total. For inference, the input image was the center crop of \(224\times 224\) on the resized image whose shorter edge was matched to 256.  
Throughout all the experiments, the loss weight parameters \(\lambda_{i}\) and \(\lambda_{d}\) in Equation (9) changed linearly for every training iteration. For the tasks on the ImageNet1K, \(\lambda_{i}\) increased from \(1\times 10^{-6}\) to \(1\times 10^{-4}\) and \(\lambda_{d}\) increased from \(1\times 10^{2}\) to \(1\times 10^{3}\). For the other tasks, \(\lambda_{i}\) increased from \(1\times 10^{-8}\) to \(1\times 10^{-5}\) and \(\lambda_{d}\) was the same. The studies on loss weight parameters are provided in Section 4.6.