The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes
Authors: Peter Kocsis
Peter Súkeník
Guillem Braso
Matthias Nießner
Laura Leal-Taixé
Ismail Elezi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Low-data Regime
Convolutional Networks

Abstract: Convolutional neural networks were the standard for solving many computer vision tasks until recently, when Transformers of MLP-based architectures have started to show competitive performance. These architectures typically have a vast number of weights and need to be trained on massive datasets; hence, they are not suitable for their use in low-data regimes. In this work, we propose a simple yet effective framework to improve generalization from small amounts of data. We augment modern CNNs with fully-connected (FC) layers and show the massive impact this architectural change has in low-data regimes. We further present an online joint knowledge-distillation method to utilize the extra FC layers at train time but avoid them during test time. This allows us to improve the generalization of a CNN-based model without any increase in the number of weights at test time. We perform classification experiments for a large range of network backbones and several standard datasets on supervised learning and active learning. Our experiments significantly outperform the networks without fully-connected layers, reaching a relative improvement of up to $16\%$ validation accuracy in the supervised setting without adding any extra parameters during inference. 

The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes
Peter Kocsis
Technical University of Munich
peter peter. www.kkossss.md...
Peter Súkeník
Guillem Brasó
Technical University of Munich peter sukeernetttnkkkrtostttt...
Technical University of Munich
gui gum.lem. braso@tum de
Matthias Nießner
Laura Leal-Taixé
Technical University of Munich nies niessner@tum.de
Technical University of Munich
leal. taixeetu..mdee
Ismail Elezi
Technical University of Munich
ismail www.eetum.cccccmmm de
peter-kocsis, http:ii..iittuu...oooeeeeeeeeee htt......//oottttateerraaaaaaaii
Abstract
Convolutional neural networks were the standard for solving many computer vision tasks until recently. when Transformers of MLP-basee architectures have started to show competitive performance. These architectures typically have vast number of weights and need to be trained massive datasets; hence, they are not suitable for their use in low-data regimes. this work, we propose a simple yet effective framework improve generalization from small amounts of data. We augment modern CNNs with fully-connected (FC) layers and show the massive impact this architectural change has in low-data regimes. We further present online joint mnowledge-distillee
method to utilize the extra FC layers at train time but avoid them during test time. This allows us improve the generalization of CNN-based model without any increase in the number of weights at test time. We perform classification experiments for a large range network backbones and several standard datasets on supervised learning and active learning. Our experiments significantly outperform the networks without fully-connected layers, reaching a relative improvement of up to 16% validation accuracy in the supervised setting without adding any extra parameters during inference.
Introduction
Convolutional neural networks (CNNs) [1, 2] have been the dominant architecture in the field of computer vision. Traditionally, CNNs consisted convolutional (often called cross-correlation) and pooling layers, followed by several fully-connected layers [3-5]. The need for fully-connected layers was challenged an influential paper [6], and recent modern CNN architectures [7-11] discarded
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1
PC Sofl
P1
Layer Norn
Zz
daar
djf
Layer-Nora
def
d
dpf
dor
P2
Z3
P3
:
Zp
Lpp
Feature Refiner
Features
Predictions
Figure 1: Feature Refiner architecture. Our network takes the features extracted by the backbone network. We apply comension-rrducctio reduce the model parameters followed by a symmetric 2-layered multi-layer perceptron.
them without a noticeable loss performance and a drastic decrease in the number of trainable parameters.
Recently, the "reign" of all-convolutional networks has been challenged in several papers [12-16], where CNNs were either replaced (or augmented) by vision transformers or replaced by multi-layer perceptrons (MLPs). These methods remove the inductive biases of CNNs, leaving more learning freedom to the network. While showing competitive performance and often outperforming CNNs, these methods come with some major disadvantages. Because their typically large number of weights and the removed inductive biases, they need be trained massive datasets to reach top performance. As consequence, this leads to long training times and the need for massive computational resources. For example, MLPMixer [14] requires thousand TPU days be trained on the ImageNet dataset [17].
In this paper, instead of entirely replacing convolutional layers, we go back to the basics, and augment modern CNNs with fully-connecte neural networks, combining the best of both worlds. Contrary to alternative architectures [14, 15] that usually require huge training sets, we focus study on the opposite scenario: the low-data regime, where the number labeled samples is very-low to moderately low. Remarkably, adding fully-connected layers yields significant improvement in several standard vision datasets. In addition, our experiments show that this is agnostic to the underlying network architecture and find that fully-connected layers required to achieve the best results. Furthermore, we extend our study with two other settings that typically deal with a low-data regime: active and semi-supervised learning. We find that the pattern holds in both cases.
obvious explanation for the performance increase would be that adding fully-connected layers largely increases the number of learnalle parameters, which explains the increase in performance. To disprove this theory, we use knowledge distillation based on gradient gating mechanism that reduces the number of used weights during inference to be equal to the number of weights of the original networks, e.g., ResNet18. We show our experiments that this reduced network achieves the test accuracy the larger (teacher) network and thus significantly outperforms equivalent architecture that does not use our method.
In summary, our contributions are the following:
We show that adding fully-connected layers beneficial for the generalization convolu- tional networks in the tasks working in the low-data regime.
We present a novel online joint knowledge distillation method (OJKD), which allows us to utilize additional final fully-connected layers during training but drop them during inference without noticeable loss in performance. Doing so, we keep the same number of weights during time.
We show state-of-the the-art results in supervised learning and active learning, outperforming all convolutional networks by up to 16% in the low data regime.
2 Methodology
propose a simple yet effective framework for improving the generalization from small amount of data. In our work, bring back fully-connected layers at the end of CNN-based architectures.

---

21
Z/2
PC Softman
Original
Backbone
CE
23
:
Zp
Gradient Gate
Loss
Test time
Train time
Feature
CE
Refiner
Lęn
Figure 2: Online Joint knowledge distillation pipeline. Besides the baseline network's 's classification head, we append an extra head with our Feature Refiner. The network is trained with the composed of the heads' cross-entropy loss. During training, the gradients coming from the single fully- connected layer are blocked. As result, the backbone is updated only by the second head with our Feature Refiner. In test time, this extra head can be neglected without any noticeable performance loss.
We show that by adding as little 0.37% extra parameters during training, we can significantly improve the generalization the low-data regime. Our network architecture consists of two main parts: convolutional backbone network and our proposed Feature Refiner (FR) based on multi-layer perceptrons. Our method task and model-agnostic and can be applied to many convolutional networks.
In our method, we extract features with the convolutional backhone network. Then, we apply our FR followed by task-specific head. More precisely, we first reduce the feature dimension dasf to drrf with single linear layer to reduce the number of extra parameters. Then we apply symmetric two-layer multi-layer perceptron wrapped around by normalization layers. We present the precise architecture of our Feature Refiner in Figure
2.1
Online Joint Knowledge Distillation
One could argue that using more parameters can improve the performance just because of the increased expressivity of the network. To disprove this argument, we develop an online joint knowledge distillation (OJKD) method. Our OJKD enables us to use the exact same architecture as our baseline networks during inference and utilizes our FR solely during training.
We base our training pipeline on the baseline network's architecture. We split the baseline network into two parts, the convolutional backbone for feature extraction and the final fully-connected classification head. We append an additional head with our Feature Refiner. We devise the final loss the sum of the two head's losses, making sure both heads are trained in parallel (online) and enforcing that they share the same network backbone (joint). During inference, we drop the additional head and use only the original one, resulting the exact same test time architecture our baseline. In other words, our FR head is the teacher network that shares the backbone with the student original head, and we distill the knowledge of the teacher head into the student head.
However, the key ingredient of our OJKD the gradient- gating mechanism; we call it Gradient Gate (GG). This gating mechanism blocks the gradient of the original head during training, making the backbone only depend on our FR head. We implement this functionality with a single layer. During the forward pass, GG works as identity and just forwards the input without any modification. However, during the backward pass, it sets the gradient to zero. This way, the original head's ''' gradients are backpropagated only until the GG. While the original head gets optimized, it does not influence the training of the backbone but only adapts to it. Consequently, the backbone is trained only with gradients of our FR head. Furthermore, the original head can still fit to the backbone and reach similar performance as the FR head. We find that we can still improve upon the baseline without our gating mechanism but reach lower accuracy than when we use We show the pipeline of our OJKD in Figure 2.
3

---

Experiments
Validation accurecy (56
Validation sccuracy (%)
60
70
60
50
40
1k
3k
4k
6k
7k
9k
10k
5k
6k
7k
8k
9k:
10k
11k
12k
13k
14k
Dataset size [samples]
Dataset size Isamples]
ResNet18
ResNet18
FR-ResNet18
FR- Nett18
(a) CIFARIO
(b) CIFARIO0
82
Validtion accuracy (%6)
80
90.0
78
87.5
76
85.0
82.5
1k
2k
3k
69
7k
8k
9%
10k
11k
12k
13k
14k
Isamples
Dataset Isamples]
ResNet18
ResNet18
FR-ResNet18
FR-ResNet18
(c) Caltech101
(d) Caltech256
Figure 3: Comparisons with ResNet18. We compare our approach (FR) to the baseline network ResNet18 in supervised learning. Our method significantly outperforms the baseline, especially in the more challenging earlier stages, when we have small amount of data.
In this section, we demonstrate the substantial effectiveness of our simple approach improving the performance of neural networks in low-data regimes.
Datasets the number of labels. For all experiments, we report accuracy as primary metric and use four public datasets: CIFARIO [18]. CIFARI00 [18], Caltech101 [19]. and Caltech256 [19]. We use the predefined train/test split for the CIFAR datasets, while we split the Caltech datasets into 70% training and 30% testing, maintaining the class distribution. In the simpler datasets, CIFARIO and Caltech 101, we start with an initial labeled pool of 1000 images, while in the more complicated, CIFAR 100 and Caltech256, 1000 samples to the labeled pool in each cycle and evaluate the performance with the larger and larger training datasets. We use the active learning terminology for cycle', where cycle is defined as complete training loop.
start with 5000 labeled images. In both cases, incrementally add
CNN backbones. For most experiments, we use ResNet18 [8] as our backbone (dasf 512), and reduced feature size dfrf = 64. We compare the results of our method with those of pure ResNet18 both the supervised and active learning setups. Note that the supervised case (Figure 3) equivalent to random labeling strategy an active learning setup. We compare our method with various active learning strategies. We also compare [14]. Finally, to show the generability of our method, we also experiment with other backbone networks: ResNet34, EfficientNet, and DenseNet. We each experiment times and report the mean and standard deviation. We train each network in single GPU. We summarize the results in plots and refer
non-convolutional network, the MLPMixer
supplementary material for exact numbers and complete implementation details.
Training details. For the CIFAR experiments, we follow the training procedure [20]. More pre- cisely, we train networks for 200 epochs using SGD optimizer with learning rate 0.1, momentum

---

Validation accurscy (86
Valisaaioon accuracy (5%
70
50
30
1k
2k
3k
4k
5k
7k
9k
10k
6k
8k:
10k
18k
12k
13k
14k
Isamples]
[samples]
ResNet18
MLPMixer
ResNet18
MI.PMixer
FR 88-Neee
VIT-B16
FR- 88-Ree
VIT-B16
(a) CIFARIO test accuracy.
(b) CIFARIO0 test accuracy.
Figure 4: Comparison with MLPMixer [14] and VIT. [12] We compare our method with MLPMixer and VIT. the CIFARIO (4a) and on the CIFAR100 (4b) datasets. Our method significantly outperforms both architectures in the low-data regime.
0.9, weight decay 54-4, and divide the learning rate by 10 after 80% epochs. We used cross-entropy loss supervision. For the more complex Caltech datasets, we start with Imagenet-pre-trann preatrindd backbone and reduce the dimensionality in our FR only 256. We use the same training setup for full fine-tuning, except that we reduce the initial learning rate to 1e-3 and train for only 100 epochs.
3.1 Supervised Learning
Comparisons with ResNet18 [8]. We compare the results of our method with those of ResNet18. As shown in Figure 3a, on the first training cycle (1000 labels). our method outperforms ResNet18 by 7.6 percentage points (pp). On the second cycle, we outperform ResNet18 by more than 10pp. We keep outperforming ResNet18 until the seventh cycle, where our improvement is half percentage point. For the remaining iterations, both methods reach the same accuracy.
On the CIFARIOG 100 dataset (see Figure 3a), start outperforming ResNet18 by 5.7pp, and in the second cycle, we are better by almost 7pp. We continue outperforming ResNet18 in all ten cycles, in the last being better by half percentage point. We see similar behavior in Caltech101 (see Figure 3c) and Caltech256 (see Figure 3d).
A common tendency for all datasets is that with increasing number of labeled samples, the gap between our method and the baseline shrinks. Therefore, dropping the fully-connected layers in case of large labeled dataset does not cause any disadvantage, did not analyze this question in the low-data regime, where using FC layers after CNN architectures is clearly beneficial.
found in [6]. However, that work
Comparisons with MLPMixer [14] and VIT [12]. We also compare the results of our method with that of two non-convolutional networks, the MLPMixer and VIT. Similar to us, the power of MLPMixer on strengths of the fully-connected layers. based architecture. Each Transformer block contains attention block and block consisting of fully-connected layers. Unlike these methods, our method uses both convolutional and fully- connected layers to leverage the advantages of both the high-level convolutional features and the global interrelation from the fully-connected layers. In Figure 4a, we compare to MLPMixer and ViT on the CIFAR 10 dataset. On the first cycle, we outperform MLPMixer by 4.4pp and ViT by circa 6pp. We keep outperforming both methods in all other cycles, including the last one where we do see, MLPMixer and ViT do not perform well even in the latest training cycles and require to be trained on massive datasets. We show a similar comparison for CIFARIO0 100 in Figure 4b.
other hand, ViT is transformer-
better than them by circa 13pp. As
Comparisons with Knowledge Distillation baselines. To show that our method's main strength comes from our FR head, compare to several knowledge distillation (KD) methods. DML [2]]
5

---

90
Validation accarncy (%)
Validation accurncy (%)
60
40
40
Isamples]
Dataset Isamples]
ResNet18
KDCL
ResNet18
KDCL
FR- ResNet18
KD Student
FR- FR-ResNet18
KD- Dtuutddent
DML
KD- Teacher
DML
KD- Teacher
(a) CIFARIO
(b) CIFAR100
Figure 5: Comparisons with Knowledge Distillation baselines. We compare to several KD methods.
Our method outperforms the baselines in the earlier iterations with large margin.
trains two networks in an online KD setting. KDCL [22] aims to improve the generalization ability of DML by treating every network students and training them to match the pooled logit distribution. Finally, KD [23] is the original offline method, which trains teacher network, then distills its knowledge into student. In each experiment, the teacher network is ResNet50, the student is ResNet18. As can be seen in Figure 5, our method significantly outperforms the KD methods, by up 8.5pp CIFARIO and 3.6pp CIFARIO0 in the second iteration. In the later iterations, when more data is available and our approach converged to the baseline network's performance, we start to see the benefit of the other KD methods. However, while our method comes with only 0.37pp extra parameters during training, the other KD methods use 210.5pp more parameters.
Comparison with SimSiam [24]. Similarly technique. They train a feature extractor in a self-supervised setting, then fine-tune classifier head in supervised setting. While SimSiam [24] also be directly trained smaller datasets, its behavior on the low data-regime has not been investigated yet. We compare our method against the CIFARIO version of SimSiam
method, SimSiam [24] uses stop-gradient
datasplits and evaluate its performance with kNN and with linear classifier. The training scheme follows SimSiam [24], Appendix D. As it can be seen in Figure 6, our method significantly outperforms SimSiam [24] in all data splits by maximum margin of 25.45pp in the second iteration, and by 8pp in the last iteration.
3.2 Active Learning
We put our method to the test under typical low-data setting, active learning, where instead of randomly choosing samples to label, we choose them based an acquisition score. The results in Section 1.1 can be considered special case of active learning with random acquisition score. We use maximum entropy acquisition score with our network and compare with plain ResNet18 with maximum entropy and two state-of-ther -frte-rft active learning methods, LLAL [20] and core-set [25].
In Figure we summarize the results on all four datasets, showing our method's superior performance. On CIFAR1O, 10, we outperform all methods in early cycles by a large margin, to 7.3pp compared the core-set approach in the second cycle. As more labels become available during training, all the methods tend to converge to the same result. A similar trend can be observed on the CIFAR100 dataset. Here, our approach achieves 5.2pp better accuracy in the second cycle. On the Caltech101 dataset, we have slight advantage (< 0.5pp) compared with the vanilla ResNet18. We have larger advantage on the Caltech256 dataset, where we tend to outperform 1 - 1.5pp. Furthermore, note that core-set and L.ALL approaches perform much worse on these more complex datasets. The final gap between the maximum entropy acquisition and L.AAL is 4.5pp on the Caltech256 and 1.4pp on the Caltech101 dataset.
second-best method by

---

Validation accuracy [96)
40
1k
2k
3k
5k
Datasel size [samples]
ResNet18
SimSiam-KNN
FR-ResNet18
SimSiam--Lnnarr
Figure 6: Comparison with SimSiam [24]. We compare our method against the CIFAR1O version
of SimSiam [24]. Our method significantly outperforms SimSiam in the low-data regime.
3.3 Semi-Supe Learning
We now do experiment where we present preliminary results in semi-supervised learning. We choose to couple MixMatch [26] with our method and compare it with the original version of MixMatch. We use ResNet18 for the experiment and train the methods using only 250 labels. MixMatch reaches 91.07 accuracy, while our method reaches 91.56 accuracy, improving by half a percentage point, and showing that method can be used to improve semi-supervised learning.
3.4 Backbone Agnosticism
We check if our method be used with other backbones than ResNet18. We show the results for ResNet34, DenseNer121, and EfficientNetB3 on CIFAR10 and CIFAR IFARR1011 in Figure 8. The goal of the experiment is show that method is backbone agnostic and generalizes both to different versions ResNet as well as to other types of convolutional neural networks. As we can see, our method significantly outperforms the baselines both datasets and for all three types of backbones.
3.5 Ablation
Feature Refiner We conduct detailed ablation of the architecture of our Feature Refiner, showing the effect of the specific elements. We start with the backbone ResNet18 88 and build our Feature Refiner up step-by-step. Figure 9a shows the results on the CIFARIO dataset. First, we apply only single linear layer without any activation function before the output layer (512x512 w/o Activation). Interestingly, we can already see great improvement of 4.9pp the first cycle, reaching 5.5pp in the second cycle. Second, we change the previously applied linear layer to reduce the dimension, as we did in our Feature Refiner (512x64 w/o Activation). This step further improves the results up to 2.3pp in the third cycle. Third, we add our fully-connected layers with the ReLU activation (FR w/o LayerNorm), which yields an improvement of 3. 1pp in the second cycle. Finally, we use our complete architecture by applying the normalization layers, reaching the best results in the earlier stages, giving an additional performance boost 1.2pp in the second stage.
Online joint knowledge distillation We now evaluate our proposed OJKD, In these experiments, we highlight that during inference, can drop the Feature Refiner head without any noticeable performance loss. We use the same training strategy described in Section 2.1. We train together with the original and Feature Refiner heads. However, during inference, we evaluate both heads. In Figure 9b we show both heads' performance on the CIFAR10 dataset. As we can see, both heads have barely distinguishable performance, showing that the knowledge our teacher properly distilled into the student original head. Furthermore, we ablate the effect of our proposed Gradient Gate. Without our gating mechanism, we can still improve upon the baseline up 4.5pp, but using it can give further 5.6pp improvement in the second stage.
head can be

---

90
5
Vallidie nocunnyy (S)
Wiliidie sanmmiyy (96)
30
20
-00
60
8
-6
70
64
Datrect sind (omple))
Dateet [sampks)
[exmpias)
Dataed (sumpee))
Resteet8-Entroyy
ResNet18- E8-ootopyy
ReaNett&-
HR-ResNett--ntree
PR- 8e Eatropy
FR TRt
LLAL
LLAL
LLAL
LLAL
CoreSct
CoreSct
CoreSet
CoreSct
(a) CIFARIO
(b) CIFARIO0
(c) Caltech101
(d) Caltech256
Figure 7: Active Learning. We compare our approach (FR) in an active learning setup based on maximum entropy. We compare with ResNet18 8 using entropy- based acquisition score, LLAL, [20] and core-set [25]. Our method significantly outperforms all others, especially in the more challenging earlier stages, when we have small amount of data.
The effect of the number of layers. We now study the number of layers with nonlinearities. To do so, we add nonlinear layers in our Feature Refinement. We show the results on the CIFARI0 dataset in Figure 9c. As we can see, adding more nonlinear layers comes with decrease in performance. In fact, the more layers we add, the larger the decrease performance compared to our original model. This makes sense considering that by increasing the number of layers, we increase the number of learnable parameters, and thus we might cause overfitting.
4 Related Work
All-convolutional networks. Convolutional neural networks [1, have made a breakthrough in many computer vision tasks since the pioneering work of [4]. Other works [27, 5] improved over those architectures, typically by increasing the number of convolutional layers but by keeping the same architecture: many convolutional and pooling layers followed by few fully-connected layers. However, this design choice was questioned in the famous Striving for simplicity [6] paper, where was argued that there no need for fully connected layers. Fully convolutionized architectures such as Google-Inceptio [7] or ResNets [8] followed, significantly outperforming the previous state-of- the-art. Some more recent works typically tended to improve the architecture of ResNet architectures either by enlarging the number of convolutional filters [8] or densely connecting convolutional layers [10]. Probably the best solution was found in the EfficientNet [28]. which simultaneously increases the image resolution, number of convolutional channels, and number of convolutional layers. A constant of these approaches that they completely get rid of the fully connected layers. Contrary to them, we find out that adding fully-connected layers comes with a massive benefit in performance when the number of labeled points is limited. By doing so, we are able to improve the performance of several backbones in different learning tasks.
Non-convolutional networks. A parallel line of research has shown that convolutional neural networks can be replaced either by vision transformers [12, 13] or multi-layer perceptrons [14-16]. The main idea behind these works is leave the network as much freedom possible during the learning procedure instead of injecting inductive bias into the network. Indeed, in multi-layer perceptrons, every unit is connected all the units in both the previous and the next layer, allowing it to use all the information of the proceeding layer as well as pass it into the next one. Works like MLPMixer [ [1] and others have shown promising results in several vision tasks. Still, they come with series of limitations, such as the need for pre-training on very large datasets and requiring massive amount of computational resources. Our work similar to ML.Mer [1 [14] in the sense that we also leverage the power of multi-layer perceptrons to solve different vision tasks. However, unlike them,

---

40
70
Valikibin (991
Valideiir autteaay (84)
73
Vilihiiin NcCeary (86)
23
-50
83
100
102
Dornwct stoe (samples)
Darowrt stee [namples]
Dasseee soo [samples)
ResNot34
DenseNet121
EfficienNNeBB3
FR-ResNet3
28
FR-EfficieeNNeeB3
(a) ResNet34 CIFARIO. ResNee34-CIFARIO.
(b) PenseNet121-CIFARR
(c) SrnocientNedB-CIFAARR
80
13
Vilibiiinncc Sesiime (4]
Volddibin 200kers
80
Vahleiion
40
33
35
30
68
86
102
12t
14t
69
7k
88
96
10t
116
12h
13t
14k
5k
7X
96
11k
12k
13k
Dunsect dize [paple))
Dutanet size [pamples]
Datantt size [samllss]
ResNet34
DenseNet121
EfficienNNeB33
FR-ResNet34
-
PR-DemseNet121
3R-EfficienNNer83
(d) ResNet34-CIFARIN
(e) CINeeNNeI21-CFFARIII
(I) EfficiennNeBB--CFFAAA
Figure 8: Backbone agnosticism. We show that our approach is agnostic to the backbone. We evaluate our approach on the CIFARIO and CIFARIO 100 datasets with ResNet34, DenseNet121, and EfficientNet-B3 backbones. In all cases, our method significantly improves over the original network.
we do not get rid of the convolutional layers, but we advocate for hybrid networks. Our networks do not need pre-training and massive computational resources, while it manages significantly improve state-of-the-art results when trained with limited number of labeled points.
Knowledge distillation. With the advancements computational power, larger neural networks are possible to train. However, effective neural networks are required during inference in many practical applications, especially mobile or embedded applications. This requirement motivates the field of Knowledge Distillation (KD). The goal of KD is to reduce the computational demand of a trained network's inference by maintaining its performance. This could be in the form of offline distillation, where one network is used as a teacher to distill its knowledge into another smaller network, the student [23]. However, online distillation has gained more attention lately, thanks to its simpler pipeline [29]. In such setup, the student and teacher models are trained together end-to-end. For more details on this field, we refer the work Gou et al. [29] The published online distillation methodologies are mainly specifically applied for ensemble distillation [30]. On the contrary, we propose an effective yet general online distillation method. We attach separate head during the training (the Feature Refiner) while using gating mechanism that blocks the gradients coming from the original softmax layer of the network. On inference, we can simply drop the Feature Refiner head without noticeable performance drop, thus, using no extra parameters.
Stop-gradient. SimSiam [31] and BYOL [32] train two branches jointly self-supervised setting with different augmentations. In BYOL, the target is provided by the averaged version of the main network, while in SimSiam the same network is used with stop-gradient technique. These methods share similarities with our OJKD: however, they cannot be directly applied to supervised learning, and they are trained on large datasets. Instead, we focus on generalization from low amount of data in supervised setting.

---

Valibain (16)
Waltbaiie accmmiy (1))
Valiiaaton acrmmy (1)
90
60
80
60
30
Dicaret size [samples]
Dogaset size [samples]
Dataser site [samples]]
ResNet18
FR ResNet18 Extaflead
512x512 Activation
2layer
FR-Res R8 OriginalHead
512x64 Activation
FR-3layer
FR- Re OriginalHead Wio GradGate
FR w/o LayerNorm
FR 4layer
FR- FR-ResNet18
FR- FR-5layer
(a) Feature Refiner
(b) OJKD ablation
(c) Number of layers ablation
Figure 9: Feature Refiner 9a: We apply parts of our Feature Refiner step-by-step. First, we use only single linear layer without any extra activation (512x512 w/o Activation), then apply dimension reduction step (5 (512x64 w/o Activation). Finally, we evaluate the effect the LayerNorm layer. OJKD ablation 9b: Our online joint knowledge distillation enables us utilize the advantages of our Feature Refiner without increasing the number of model parameters at the same time. Our method also helps even without the Gradient Gate.
Number of layers ablation 9c: We study the effect of the number of nonlinear fully-connected layers. More layers do not lead better performance.
5 Conclusion
In this paper, we question the long-believed idea that convolutional neural networks not need fully-connected layers. Perhaps surprisingly, show that using hybrid networks that use both convolutional high features and interrelations coming from fully-connected layers improves the generalization performance in the low-data regime, without any drawback in the high-data regime. To show a fair comparison, we also introduce gating mechanism that allows distilling the knowledge from our extra head with the added FC layers to the baseline network. We show that approach model agnostic by evaluating different backbone networks. We show in our experiments that our method yields significant improvement the state-of the-art in supervised and active learning, in a wide range of standard classification datasets without an increase in the number of used parameters.
Broader Impact/ Limitations
shortcoming of our method is that we cannot improve the networks that already contain fully- connected layers, such as the VGG Network [5]. In the supplementary material, we do an experiment showing that our Feature Refiner cannot further improve VGG Network's 's accuracy. However, we were able to decrease the number of learnable parameters by 67%.
Another deficiency of our paper is the lack of theoretical explanation. Although the empirical results underline our methodology, we could not find any mathematical proof for it. Our speculative explanation is that the network has only a single final layer, then the backbone's feature space must be linearly separable. Our intuition is that in the case of low amount of data, the convolutional layers do not get enough supervision to find the right local reasoning, which limits its flexibility. Having additional final fully-connected layers gives global supervision and increases the flexibility of the network to reach better optima. We note that the research in deep learning has been mostly head desirable and ultimately needed.
by empirical results. We believe that theoretical cxplanations At the same time papers without a clear (or a downright wrong) theoretical explanation have had a massive impact deep learning, is the case of the naleh-normalization [33] paper. We hope that our work can open a new interesting line of research, and inspire other researchers to question the 'common knowledge' in deep learning. Furthermore, it would be interesting to see if our results can be generalized in domains where the number of labeled data is very scarce, such as medical imaging.
Acknowledgements This work was supported by Sofja Kovalevskaja Award, a postdoc fellowship from the Humboldt Foundation, the ERC Starting Grant Scan2CAD (804724), and the German Research Foundation (DFG) Research Unit "Learning and Simulation in Visual Computing".
10

---

References
[1] Kunihiko Fukushima and Sei Miyake. Neocognitron: algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognition. 15(6):455-466) 1982. 1,8
[2] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(1 86(11))2278-2322, 1998. 1,8
[3] Dan C. Ciresan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. In CVPR, 2012. 1
[4] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NeurIPS, 2012.
[5] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015. 1, 8, 10
[6] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In ICLRW, 2015. 5, 8
[7] Christian Szegedy, Wei I. Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015. 1.
[8] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 4, 5,
[9] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[10] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 8
[11] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. In CVPR, 2017.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition at scale. In ICLR, 2021. 2, 8.
[13] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In JCML, 2021.
[14] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. In NeurlPS, 2021. 2, 5,
[15] Asher Trockman and J. Zico Kolter, Patches are all you need? arXiv, abs/2201.0977) 2022. 2
[16] Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmlpnet: Hierarchi- cal vision MLP with re-parameterized locality. arXiv, abs/2112. 1081, 2021. 8,
[17] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. Comput. Vis., 115(3):211 1-221 2015.
[18] Alex Krizhevsky. Learning multiple layers features from tiny images. Technical report, 2009.
[19] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
[20] Donggeun Yoo and So Kweon. Learning loss for active learning. CVPR, 2019. 4, 6, 8
[21] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. CoRR, abs/ 703884406,,038, 2017. URL http:://aaxi..or/g//...... 10t.//00.... 00384.0 5
[22] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In CVPR, 2020. 6

---

[23] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015.
[24] Tianyu Hua et al. Simsiam. http:///gggtuuuucoo/a/r///eee/ee https//w.ccc..aaraaaaaaaaaaaaaaa am, 2021. 6,
[25] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-sel approach. In ICLR, 2018. 8,8
[26] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmateh: A holistic approach to semi-supervised learning. In neurips, 2019. 7
[27] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014. 8
[28] Mingxing Tan and Quoe V. Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[29] Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. CoRR, abs/2006.05522 05525, 2020. 9
[30] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In ICLR, 2018. 9
[31]
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. 9
[32]
Jean Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self- supervised learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ticv Lin, editors, NeurIPS, 2020. 9
[33] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015. 10
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "PR_35vcPefv", "reviewer": "Reviewer_DX6o", "summary": "This paper proposes a training method that involves a module plugged into the final features of a backbone as an additional classification head for joint training with the existing classification head. Along with the original head, the newly added head also pass through the softmax for the cross entropy loss; the training is performed with the summation of the two cross-entropy loss. The proposed module is dubbed Feature Refiner (FR), consisting of two fully connected layers followed by the layer normalizations and operating with the gradient gate (GG) right before the classification head. GG acts exactly like a stop-gradient technique, which is widely used in self-supervised learning, to train the original head only based on the output of the frozen backbone, which is the input for the head, and the backbone is trained only with the extra head with FR. At the inference phase, the original backbone with the original classification head is used for the forward propagation. The authors claim that this training method works well on low-data regimes (for me, it is limited to the low-data regimes), which may not be expected. Some experimental results on small datasets, including the CIFAR datasets and the Caltech datasets for supervised learning, active learning (only on CIFAR), and semi-supervised learning (presumably on CIFAR), are provided to show the effectiveness of the proposed method. The authors try to show the universality of the proposed method of training with some backbones, not constrained on ResNets.\n", "questions": "- Why the ImageNet-pretrained model is used only for training on Caltech datasets?\n", "limitations": "- Limitation is provided, but any potential negative social impacts do not seme to be provided.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This paper is easy to follow. \n- The proposed idea looks somewhat interesting.\n\n### Weaknesses\n- The main concern with the proposed method is that it seems to have a very similar training pipeline to SimSiam [1], which showed that using the stop-gradient is a key to training a backbone in self-supervised training. Specifically, Feature Refiner (RF) and the classification heads seem to be the predictor and the projector in the method [1, 2], respectively (the order is reversed but would not be a matter from my standpoint). A difference is at the loss, but as the authors claim (in line 83, p.3), assuming the network is trained in a KD way, the proposed method is a supervised SimSiam (with a single-view ). I hardly agree with the authors' claim that the method performs like a KD,  except for using only a single-view image; the training procedure is very close to SimSiam. Therefore, the authors should argue the difference between the proposed method with SimSiam.\n- There is no intuition why the proposed method has the benefit of training with small data.  \n- The experimental setup is somewhat unconvincing; the setup is inconsistent and does not follow the authors' claim of requiring pre-training for the method (in line 234, page 9). The authors specify that they use an ImageNet-pretrained ResNet for the Caltech datasets training, only providing a seemingly inappropriate reason.\n- All the training dataset is small, so the experimental verification of the claim is limited. Using small data for training and a small dataset is technically different. Therefore, it would be better to justify the proposed method on a larger scale dataset such as ImageNet with a small data regime for a stronger claim.\n- Comparison of the proposed method with MLPmixer and ViT-B16 in Figure 4 seems unfair. MLPMixer and ViT-B16 have the stem of performing non-overlapping patchification for the input, so training them with 32x32 size images in the CIFAR datasets degrades the model accuracy regardless of the size of training data.\n- As aforementioned, the authors claim the proposed method is a joint KD. However, the loss is not a straight KD-based loss (e.g., the KLD loss), and training a backbone with the extra head may not leverage the KD concept, in my opinion. Can the authors elaborate on the concept?\n\n\n### Pre-rebuttal comment\nThis work presents a training method using the stop gradient technique with an extra FC head for model training in a supervised manner. Except for using a single-view image and supervised loss, the overall training pipeline is quite similar to the previous self-supervised methods [1, 2], so the authors should elaborate on the difference and provide any intuition why the proposed method could work well. Another concern is that the experiments are not convincing because of the inconsistent experimental setups and small-scale experiments. Therefore, I am leaning towards rejection but would like to see the authors' response and the other reviewers' comments for my final decision.\n\n[1] Chen et al, Exploring Simple Siamese Representation Learning, CVPR 2021\n[2] Grill et al, Bootstrap your own latent: A new approach to self-supervised Learning, NeuRIPS 2020\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CQQF6n6Op0", "reviewer": "Reviewer_jkGN", "summary": "This paper shows that one can improve accuracy in the low-data regime by adding fully connected layers to CNNs during training and distilling knowledge to a classification head, resulting in a network with the same number of parameters and better generalization performance. ", "questions": "In figure 6f, why does the accuracy of FR-EfficientNet decrease at 10k samples while the accuracy of EfficientNet increases? It seems strange as this effect seems to only happen at this one datapoint. \n\nIn figure 4b, 5(c,d), and 6(c,d,e,f) the FR method has higher accuracy than the baseline for all amounts of data; did you run experiments where these networks are trained on the entire dataset? Is there a general point at which there are diminishing returns in accuracy? \n\n**Update**\n\nAfter the authors provided new experiments to show that their method does in fact outperform knowledge distillation methods in low data regimes, I am satisfied with the evaluation and have changed my review from reject to accept. ", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the method proposed in this paper very interesting and the results showed clear improvements over the base network. Considering how easy this is to implement, I can see this paper having a large amount of impact to the community, and the experiments showing improvements in the active learning and semi-supervised learning setting show the diversity of this approach. However, the lack of knowledge distillation baselines and theoretical backing has me questioning whether this is any different than a typical teach student distillation approach. \n\nFor instance, what would happen if one jointly trained ResNet18 and ResNet50, applied a loss similar to [1]? In this case, the test time network would contain no more parameters, so it seems like a reasonable comparison, and it follows a similar intuition of this paper (train with more parameters and drop them during test-time). Although I am not too familiar with current KD methods, I would be surprised if none of them showed a similar gain in performance, so it would be a necessary baseline to compare this papers method to. \n\nA lesser concern I have is with overfitting. Since this method seems to be training a network with more parameters on a small amount of data, I wonder if performance on robustness benchmarks would be worse than the baseline network. \n\nOverall I think that this paper has the potential to have high impact as it is well written and has a simple, easy to follow method that is effective under several tasks. I am giving it a reject as I don't believe the current experiments are sufficient to prove that this method is more advantageous than other knowledge distillation methods, but I am eager to have the authors quell my doubts with a more thorough evaluation. \n\n[1] Zhang et al. \"Deep Mutual Learning\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7rqP4Wtg7oP", "reviewer": "Reviewer_yAFj", "summary": "The authors investigate the value of fully connected layers at the end of convolutional neural networks in the small data regime. They demonstrate that the addition of these layers significantly improves model quality in this regime.", "questions": "1. I’d like for the authors to include data on the % increase in parameter count with their mechanism in all experiments, as I stated above.\n2. I’m confused by the author’s characterization of ResNet and EfficientNets as fully convolutional. These architectures are typically drawn with a single fully connected layer following the convolutional component of the model.\n3. Caltech101 in Figure 3 appears to be a counterexample. I’m wondering if you can explain this more. Your explanation claims you see similar behavior on this dataset but that is not what I see looking at Figure 3c\n4. One ablation that would be interesting is only training an FC stack on the target task and showing its quality relative to the results in Figure 3. I am curious if it would outperform on very small datasets and then be surpassed as the dataset grows.\n5. I was not able to understand the sentence “We show in our experiments that the network with the train time added fully connected layers still significantly outperforms the original architecture, even if both have an equal number of weights”. I’d like for the authors to clarify this in the text.\n6. Some numbers called out in the text are very specific when I would expect them to vary depending on the target architecture. For example, the “512” and “64” in Figure 1 and the “42k” in section 2. I think the text might be more clear if the authors described these dimensions in abstract and stated their exact parameterization in the experiment details.\n", "limitations": "The above sections detail limitations/questions I’d like to address.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. I found the paper to be very clear and well written. It was easy to understand what the authors were doing and why.\n2. The results seem significant. This is not a phenomenon that I was aware of previously, although I am not an expert in the use of deep learning in the small data regime.\n\nWeaknesses\n1. Distillation is known to improve quality for a constant parameter count so I’m not convinced that the distillation experiments disprove the hypothesis that the quality gains of adding fully connected layers are from increased parameter count. A more convincing argument is that you’re not increasing the parameter count much because of the dimensionality reduction in your FC stack. I’d encourage the authors to include data on the % increase in model parameters with their proposed addition.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept