Improving Neural Network Accuracy by Concurrently Training with a Twin Network
Authors: Benjamin Vandersmissen
Lucas Deckers
Jose Oramas
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Distillation
Representation Learning
Regularization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recently within Spiking Neural Networks, a method called Twin Network Augmentation (TNA) has been introduced. This technique claims to improve the validation accuracy of a Spiking Neural Network simply by training two networks in conjunction and matching the logits via the Mean Squared Error loss. In this paper, we validate the viability of this method on a wide range of popular Convolutional Neural Network (CNN) benchmarks and compare this approach to existing Knowledge Distillation schemes. Next, we conduct a in-depth study of the different components that make up TNA and determine that its effectiveness is not solely situated in an increase of trainable parameters, but rather the effect of the training methodology. Finally, we analyse the representations learned by networks trained with TNA and highlight their superiority in a number of tasks, thus proving empirically the applicability of Twin Network Augmentation on CNN models.

Published as a conference paper at ICLR 2025
IMPROVING NEURAL NETWORK ACCURACY BY CON- CURRENTLY TRAINING WITH A TWIN NETWORK
Benjamin Vandersmissen'
Lucas Deckers"
University of Antwerp. sqIRL/IDLab imec benjamin n. vandersmi wwweesanteeereeeeeeeeeeeeeeeeeee be
University of Antwerp, IDLab, imec
Jose Oramas
University of Antwerp, sqIRL/IDaab imec
ABSTRACT
Recently within Spiking Neural Networks, method called Twin Network Aug- mentation (TNA) has been introduced. This technique claims to improve the val- idation accuracy of a Spiking Neural Network simply by training in conjunction and matching the logits via the Mean Squared Error loss. In this paper, we validate the viability of this method on a wide range popular Convolu- tional Neural Network (CNN) benchmarks and compare this approach to existing Knowledge Distillation schemes. Next, we conduct an in-depth study of the dif- ferent components that make TNA and determine that its effectiveness is not solely situated in an increase of trainable parameters, but rather the effect the training methodology. Finally, analyse the representations learned by networks trained with TNA and highlight their superiority in a number of tasks, thus proving empirically the applicability of Twin Network Augmentation CNN models.
networks
INTRODUCTION
Training a neural network typically follows standard setup where it is optimized via an objective function to align the predictions for a set of inputs to a corresponding set of targets. During this procedure, the internal layers of the neural network learn encode informative features that assist in the accurate prediction of the target labels. To further increase the quality and usefulness of those internal features, different approaches exist such as regularization to prevent overfitting (Ng 2004] Hinton 2012), and data augmentation (Cubuk et al. 2019 2020) to increase input variety.
Beyond aiding in classification, the learned features of the model can have numerous other uses. For example, it has been demonstrated that internal features can be used to cluster and retrieve semantically similar inputs (Babenko et al. 2014),, accelerate the training of smaller models via Knowledge Distillation (KD) (Hinton al. 2015) or Deep Mutual Learning (DML) (Zhang al.,
2018). or even allow the same model to generalize on a different dataset (Weiss et al. 2016). These show the importance of learning strong, usable features, which also explore in this work.
Recently, Deckers et al. 2024) introduced method called Twin Network Augmentation (TNA) which involves optimizing
Spiking Neural Networks (SNNs) concurrently task and aligning their logits during training via an additional Mean Squared Error (MSE) loss term, which results in greatly improved single network test accuracy. While this approach has been shown successfully in the context of SNNs, it has however not been proven that this methodology is applicable across other non opikiin types of neural networks and other different model architectures of machine learning in general.
classification
To this end, we explore the effects of this approach on standard Artificial Neural Networkss (ANNs), and more specifically, a range of benchmark CNN models. Worth-noting is that different from the stateless neuron models that compose standard ANNs; typically characterized by weights, bi- ases and an activation function, such as the ReLU, a spiking neuron integrates its inputs over time
Denotes equal contribution

---

Published as a conference paper at ICLR 2025
and generates binary events depending on its weights, biases and neuronal thresholds. As this is a substantial difference, and considering the wide usage of standard ANNs, we strive to determine whether TNA is valid training strategy for CNNs.
Our contributions are the following.
1. We show the applicability of the recently introduced Twin Network Augmentation (TNA) (Deckers et al., 2024) on Convolutional Neural Networks (CNNs).
2. We corroborate on the superiority of L2 matching the logits w.r.t. using Kullback-
(KL) Divergence on the probabilities (Kim et al.) 2021 in the Online KD setting.
3. We demonstrate that the increased effectiveness of TNA cannot naively be reduced to in- crease in trainable parameters, but rather is related to both network reinforcing each other's predictions, meaning the whole is greater than the sum of its parts.
4. Via an in-depth analysis, we determine that the features learned via TNA exhibit greater predictive capabilities at most levels and show a greater robustness to data corruptions.
The rest of the paper is structured as follows. In section 2 we highlight related methods and ap- proaches. In section 3 we reintroduce Twin Network Augmentation (TNA). Via experiments, we show the impact of applying TNA in section 4 and we conclude this work in section 5
RELATED WORK
2.1 KNOWLEDGE DISTILLATION
Pretrained Teacher. Knowledge Distillation (KD) aims to exploit the learned features within fully trained neural network (a 'teacher network') to accelerate, and improve the training process of a secondary network (a 'student network'). A typical approach uses combination of task- specific loss such as the Cross-Entropy loss for classification, and a distillation loss which allows the student to learn from the teacher. The foundational approach (Hinton et al., (015)) introduces the notion of soft targets, which are the output probabilities of a teacher network given an input, and argue that these soft targets provide more nuanced views of the input compared hard targets, namely ground -rund labels. They introduce a distillation loss based on the Kullback-Leibee (KL) Divergence (Kullback & Leibler 1951) such that the student model learns to mimic the probability distribution of the teacher model. Kim et al. (2021) studies the effect of the temperature parameter r in the KI. divergence and finds that high is equivalent to logit matching, while a close to equivalent to label matching. They furthermore derive a relationship between KI. divergence with a large T and the MSE loss on the logits. A secondary approach (Romero et al. 2015), proposed to match hidden activations, thus allowing the student network to learn similar features to the teacher network. A generalization of the previous approaches is Relational KD (Pakk et al. 2019). Rather than matching the knowledge extracted from single samples, instead relations between the knowl- edge of multiple samples is matched. This can be done in a distance-based manner (second-order matching) or in angle-based manner (third-order matching). Another approach that has shown promise is self-distillation (Zhang et al.) 2201)). in rather than viewing model as monolith, it is stack of submodels, where each submodule feeds in one another. Self disit-
instead considered illation then uses the submodels as students and the full model as teacher. Then, the submodels are updated concurrently with a mix of KD losses, while the full model is training.
Online Knowledge Distillation and Deep Mutual Learning. Rather than using fixed pretrained teacher network, Online Knowledge Distillation (Guo al. 2020) allows for the teacher model to evolve during its operation. This is useful in cases where the data distribution changes throughout time, and allows the teacher to include the new data distribution within its learned features. Deep Mutual Learning (DML) (Zhang et al. 2018) goes step further, rather than using teacher network, instead a set of student networks- called cohort - are trained concurrently. Typically each student model is trained on hard targets via the cross entropy loss and additionally, on soft targets using the KL Divergence Loss.
Comparison with While DML is the field that lies the closest TNA, the key difference is that TNA works directly on the logits of the model, rather than the SoftMax probabilities. This al- lows to use different loss function, namely the MSE Loss, compared to the KI. Divergence Loss,
2

---

Published as a conference paper at ICLR 2025
Lca(Obase) Lca(Ohwin)
LCE( LCE( KL/P1 P2)+ KL/P2 Ps)
LbsseO Otwin)
Twin Network Augmentation (TNA)
Deep Mutual Leamingg (DML)
LCE(01)
Logit Averaging
ICE(@1) KL(P1
LCE(82)
(c) Ensembling w/ Logit Averaging
(d) Knowledge Distillation (KD)
Figure 1: A comparison of TNA to different related approaches. Note that Ensembling employs both models during inference and KD relies frozen pretrained network during training. TNA focuses on matching logits, while DML and KD focus on matching probabilities.
which is only applicable on probability distributions. We state that the MSF-based logit matching allows for a more direct comparison and therefore leads to better generalized latent representations. See fig. for visual comparison.
2.2 SIAMESE NETWORKS AND CONTRASTIVE LEARNING
Siamese networks CCiccoo 202 consist two (or more) identical neural networks that are trained concurrently. Different from TNA. these networks trained in contrastive fashion, with loss such as the Triplet Loss (Schultz Joachims, 2003), and each weight update is applied on all the networks involved, thus all networks have always identical weights. In contrast, TNA starts with differently initialized networks, employs a cross-entropy loss to learn for classification, and has separate weight updates per network.
2.3 ENSEMBLING AND MIXTURE OF EXPERTS
Ensembling Ju et al. 2018) is technique which aggregates the predictions of multiple fully trained networks, either via voting system or via averaging the individual logits. By ensembling, it is pos- sible to produce a classifier that outperforms an individual model, at the cost of additional parameters during inference. Mixture of Experts (MoE) (Jacobs et al. 1991) is similar in this regard, but rather than having classifiers that can recognize all classes, instead the involved models are specialised in limited number of classes. The MoE structure includes a routing mechanism to efficiently determine to which experts specific input should be fed to. In contrast, TNA also involves multiple models that are trained, however while in Ensembling and MoE these models are trained individually. in TNA these models are trained in entangled manner. Additionally, during inference, we only use single model, different to the previously-mentioned methods, which leads to reduced inference costs when compared to those methods.
3 METHODOLOGY
3.1 TWIN NETWORK AUGMENTATION
Contrastive Learning is technique that employs augmented views of the same datapoint to learn co- hesive usable representations. This done by employing two distinct transformation functions and feeding both through encoder network which computes representations. The two representations are transformed via a projection head onto a different space, on which the contrastive loss is applied with the aim of steering feature similarity. After training, then the projection head is discarded.
3

---

Published as a conference paper at ICLR 2025
We hypothesise that similar process be exploited using different (partially) trained neural networks. Rather than generating different views of a datapoint via augmentations, we can feed the same sample through different networks. As both networks have different weights, the views they provide are different from one another. Additionally, we assume that both networks are similarly trained, i.e., have similar validation accuracy, then those views should be of similar quality.
We use this concept to reintroduce Twin Network Augmentation (TNA) from representation learn- ing perspective. TNA is training paradigm that trains multiple networks parallel with the goal of improving the accuracy of the base network. To achieve this, in conjunction with the standard Cross-Entropy losses on each model, an additional Mean Squared Error loss is applied on the logits of both networks. The goal of this term is to force the outputs of the networks to grow closer and incorporate the information of the multiple views of the datapoint in the training process. Having multiple views of the same input allows the model to learn more robust features and reduces the risk of overfitting on certain data points.
Formally, TNA is defined as follows. Given two networks, base network FA and twin network FR with weights A and Öß respectively, we define the output logits of F^ given an input as FA (Z) 0A. This leads us to the following definition of the TNA training procedure.
LINA
C
C
+
|0A -
Base Loss
Twin Loss
Logit Matching Loss
During inference, we can discard the twin network FB, thus only using the hase network FA to make predictions. This means that no additional overhead is incurred when making predictions. An additional option would be to ensemble both models for a minor performance gain at the cost of longer inference time.
3.2 COMPARISON WITH MUTUAL LEARNING
The loss of network FA trained with Deep Mutual Learning (DML) (Zhang et al. 2018 on a dataset with M classes and N samples can be defined as follows:
SiL
L DRi (pallA))
with DKL (PBllPA)) the Kullback-Leilerr (KL) Divergence as defined below.
M M pe log (x.)))
DKL
Σ
(3)
(x;))
with pT,))) the probability of class m for network FA given a sample 2;; computed via SoftMax
(x.) exp(os") m
(4)
Σ @rp(ox"))
Different to DML, we do not aim match probability distributions, but rather the logits. We argue that matching probability distribution, which is generated by applying SoftMax on the logits, discards useful information due to the rescaling properties of SoftMax. We base ourselves on the work Kim et al. (202T) that shows that minimizing KL divergence loss with a temperature + co equivalent minimizing MSE loss with an additional constant. By following the assumption that the teacher's ss logit mean is O, this constant forces the student logit mean to diverge from zero. The authors also prove this empirically.
3.3 EXPERIMENTAL SETUP
We apply the TNA methodology different baselines from the Computer Vision domain, specif- ically focusing on CNNs. We use a simple ConvNet, a ResNet-18 18 (He et al. 2016). and a Mo- bileNetv2 (Sandler al. 2018) datasets such CIFAR-10 and CIFAR-100 (Krizhevsky et al.
4

---

Published as a conference paper at ICLR 2025
Table Results of applying TNA on the different dataset network combinations.
Network
Dataset
Single Train
TNA
Gain
ConvNet
CIFAR-
86.84 + 0.14%
87.07 0.31% %
+0.23%
ResNet-18
CIFAR-10
95.06 + 0.05%
95.38 + 0.18%
+0.32%
MobileNetv2
CIFAR-10
93.94 + 0.22%
94.56 0.32%
+0.62%
ConvNet
CIFAR-100
61.14 + 0.37%
61.83 0.36%
+0.69%
ResNet-18
CIFAR-100
77.66 = 0.28%
79.00 + 0.35%
+1.34%
MobileNetv2
CIFAR-100
76.17 + 0.15%
79.50 + 0.27%
+3.33%
ConvNet
TinylmageNee
37.75 = 0.23%
38.12 = 0.41%
+0.37%
ResNet-18
Tiny ImageNet
59.33 = 0.57%
64.84 = 0.56%
+5.51%
MobileNetv2
60.10 + 0.20%
65.11 + 0.09%
+5.01%
ResNet-50
ImageNet
76.61 = 0.07%
77.65 = 0.01%
+1.04%
ViT-S/16
ImageNet
69.83 + 0.12%
71.76 0.35%
+1.97%
Table 2: The performance of TNA compared to DML..
Network
Dataset
Single Train
TNA
DML
MobileNet-v2
CIFAR-100
0.1
74.74 + 0.16%
77.78 + 0.15%
1.66 + 1.15% 78.08 0.34%
MobileNet-v2
CIFAR-100
0.05
76.17 + 0.15%
79.50 0.27% %
ResNer-18
TinylmageNet
0.1
59.33 + 0.57%
64.84 0.56 %
0.05 + 0.00%
ResNet-18
TinylmageNet
0.05
57.79 + 0.20%
64.34 + 0.29%
62.98 = 0.44%
2009), and TinylmageNer (Le & Yang 2015). More complex scenarios are considered with ResNet- 50 (He et al. 2016). and ViT-S/16 (Beyer et al. 2022) ImageNet (Deng et al.) 2009). The full training configurations, as well the architecture of the ConvNet are listed in the appendix. For each configuration, we apply Stochastic Gradient Descent, and use Cosine Learning Rate annealing during the training process. To ensure that our observations are representative of the trends, each experiment is repeated over three random runs.
4 EXPERIMENTS
4.1 TWIN NETWORK AUGMENTATION
By applying the Twin Network Augmentation methodology on different baselines, we notice a con- sistent increase in accuracy w.r.t. training single network in table [ thus showing that application of TNA valid. However, for the ConvNet architecture, this increase is not alwas statistically sig- nificant. We link this to the limited capacity of the network, and hypothesize that this limits feature space exploration, and as such the performance of TNA. As DML follows similar methodology, we compare with that approach in table We notice that while networks trained with DML. also signif- icantly outperforms the single trained network, applying TNA results in an additional performance good solution, had to tweak the
gain. It should be noted that in order for DML to converge learning rate. In the case of MobileNet-v2 + CIFAR-I00 this tweaking also had a positive effect on the other settings, while in the case of ResNet-18 TinyImageNet, this negatively affects the baseline. This shows additionally that TNA is more robust to different values of hyperparameters. Next we strive to understand why this methodology improves validation accuracy.
4.2 TNA AND NETWORK CAPACITY.
One hypothesis for the increased performance of networks trained with TNA is that the effective capacity of the base network is artificially increased by virtue of the Logit Matching Loss. The reasoning behind this is that when two randomly initialized neural networks are trained separately, they converge to different solutions, encoding different features. By adding the Logit Matching Loss, the output of both networks constrained be similar, i.e., both networks will converge
5

---

Published as a conference paper at ICLR 2025
Table 3: Ensembling of differently trained ResNet-16 18 models on TinyImageNet. in- dicates networks trained with 1r00.55 rather than 1--1.1.
Table 4: Wide ResNet-18 performance with dif- ferent width multipliers compared to ResNet-18 trained with TNA on different datasets.
No Ensemble
Ensemble
Width
Single TNA
59.36 + 0.57%
63.79 + 0.26%
Dataset
x
2x
4x
TNA
64.84 + 0.56%
66.56 + 0.24%
CIFAR100
77.56%
19.12%
79.22%
79.00%
DML* *
62.98 + 0.44%
65.04 0.03%
TinylmageNet
59.08%
60.51%
60.90%
64.65%
a closely similar solution, but by using different initializations, both networks will explore different paths in the feature space. These different paths provide positive reinforcement for each other and lead to general increase in performance.
To test this hypothesis, we conduct three tests. First, we compare TNA with naive method to im- prove the capacity of a neural network, namely by training a single network with a modified width (number of filters of network), thus leading to the same or higher number of trainable parame- ters. Second, we compare TNA with ensembling of regularly trained models, which a technique that increases the capacity of a model during inference by averaging predictions of different mod- els. Third, we conduct a progressive reinitialization test, starting from an identical twin and base network, to determine the impact of twin initialization the TNA algorithm.
Comparison with wider networks. In table 4 we compare the performance of ResNet-18 with TNA to the training of different variants of single WideResNet-IB 18 models. These are variations of ResNet-18 with different capacities, obtained by multiplying the number of filters in each layer by specific factor. In this experiment we choose the multipliers to ResNet-18 18. In the case of CIFAR-100, the validation accuracy of the wider networks and the version with TNA lie well within a standard deviation in both directions. showing no difference of significance. In the case of TinyImageNet, the difference is much more pronounced, with TNA significantly outperforming the wider networks. This shows that the gain of TNA lies not only in a simple increase of the trainable parameters.
(1, 4). with w=l equivalent
Comparison with model ensembling. Next, we study a more advanced approach of increasing the model capacity via model ensembling. We focus on logit averaging, which averages the outputs of a set of trained networks to get a single output. In the case of h: = 2 models, this involves double the computational resources during both training and inference compared to training single network. As such, it has a similar computational cost during training as the TNA technique, but double the inference cost. In table 3 we compare the performance of differently trained ResNet-18 models with that of an ensemble of two ResNet-18 models. We notice that TNA outperforms the ensemble of two standard ResNet-18s, which shows the additional efficiency that encoded within the single network. Remarkably, even though the objective of TNA is to minimize the distance between logits of the base and the twin network, we still achieve a substantial accuracy increase when ensembling the base and the twin network with logit averaging. This suggests that our hypothesis that the twin architecture exploring multiple solutions in the feature space might be correct.
Progressive Reinitialization. A straight-forward assumption to make about TNA is that both net- works need to be initialized differently. Of course, if both networks are totally similar at initializa- tion, each network will produce the same logits given the same input and as such, the matching loss will be zero. However, to determine exactly how different the networks need to for TNA to yield noticeable increase in validation accuracy, we attempted TNA with twin network that was only initialized differently in one layer. We noticed that no matter which layer we initialized differently, this had no significant impact the validation accuracy, thus showing that initialization of the twin network doesn't 't matter for the performance of TNA.
4.3 FEATURES LEARNED BY TNA
Next want assess the quality of the features learned by TNA. In the previous section, discussed that application of TNA can lead to increased validation accuracy on several benchmarks, but this still leaves number of questions unanswered, such as: (1) How different are the learned

---

Published as a conference paper at ICLR 2025
Table 5: Feature Similarity between ResNet-18 models trained on TinyImageNet with different approaches, measured with CKA. Both base and twin networks are trained concurrentyy with TNA, while DML and DML B are trained concurrently with DML.
Single Single
TNA vs Single
Base Twin
DML DML
Block
0.5201 + 0.3226
0.7860 = 0.2863
0.9798 + 0.0132
0.9554 = 0.0231
Block 2
0.8758 + 0.0597
0.9262 0.0714
0,9660 1 0.0209
0,9693 0.0100
Block
0,9292 0.0251
0.9541 = 0.0169
0.9770 0.0147
0,9821 0.0044
Block 4
0.8651 +: 0.0260
0.8496 - 0.0650
0.9749 1 0.0015
0.9384 = 0.0102
Block 5
0.9493 + 0.0058
0.9427 = 0.0158
0,9823 + 0.0007
0.9774 = 0.0002
Block 6
0.8752 = 0.0327
0.8527 0.0426
0.9442 + 0.0026
0.9208 0.0024
Block
0.8746 0.0056
0.8664 = 0.0095
0.9349 0.0010
0.9128 0.0024
Block 8
0.6683 = 0.0016
0.6392 = 0.0029
0.8132 + 0.0023
0.7935 = 0.0020
FC
0.8556 + 0.0009
0.8630 = 0.0026
0.9714 + 0.0001
0.9268 = 0.0002
Typa
0.6
Standard
0.5
Accuncc
0.4
Acouracy
0.4
0.2
0.2
Model Restitt
6.1
Restiet 34
Restiet-50
25
50
75
100 Epoch
125
150
175
200
Layer (block)
Figure 2: Validation accuracy of linear probes inserted at different points a ResNet-18 trained on TinylmageNet.
Figure 3: Convergence behaviour of ResNet
models at different sizes while training on Tiny- ImageNet.
features from those encoded in single trained networks? (2) Are only the deeper-level features better, or is there a distinction already in the shallow-level features? (3) How much difference between the twin and base initialization is required for TNA to work?
Feature Similarity. We use the Centered Kernel Alignment (CKA) metric introduced by Kornblith al. (2019) to directly compare features learned by different networks. Two experimental settings are considered, (1) a comparison networks trained using various methods and (2) comparison of networks learned in conjunction via DML TNA. For each setting, calculate the representation similarity with CKA at the end of each convolutional block a ResNet-18 over the Tiny ImageNet validation set, and list the results in table We notice that applying TNA results in increased feature similarity at each point in the network w.r.t. the baseline. While this is expected at the logit level after the fully connected layer as these are directly matched, this shows that both networks try to mimic each others features. Interestingly, we still notice a considerably larger difference in the high-level features, albeit less pronounced than the case of separately trained network, showing that these networks still consider different information. This indicates that the networks still consider different kinds of information achieve similar logits. Additionally, we notice that the feature similarity of TNA trained networks is higher at the logit level (FC layer) than that of DML-ttrined networks, which indicates better agreement between the networks. Coupled with the results on classification benchmarks, this indicates that the better similarity of the logits achieved by TNA leads to a gain in classification accuracy.
Intermediate feature quality. While it is evident that the features at the last convolutional layer of TNA learned networks are better suited for classification, we next want to determine how dis- criminative the intermediate representations are. For this purpose use linear probing technique introduced by Alain & Bengio (2017). At certain points in the network, we train a linear classifi- cation head the frozen weights of the preceding layers. To avoid large computations, first

---

Published as a conference paper at ICLR 2025
Table ResNet-18 accuracy on different corruptions of the TinyImageNet-C dataset.
Corruption
Single
TNA
Corruption
Single
TNA
Brightness
36.35%
42.45%
Impulse Noise
24.09%
25.29%
Contrast
11.85%
14.57%
JPEG Compression
47.53%
52.33%
Defocus blur
26.57%
28.60%
Motion Blur
32.16%
35.62%
Elastic Transform
38.95%
43.16%
Pixelate
37.65%
41.87%
Fog
27.29%
31.69%
Shot Noise
25.09%
27.56%
Frost
29.69%
31.69%
Snow
27.09%
32.53%
Gaussian Noise
20.82%
22.71%
Zoom Blur
31.02%
33.92%
Glass Blur 21.99%
23.32%
Average
29.88%
32.49%
None
59.33%
64.84%
Table 7: Ablation test of different values for the parameter a. Entries indicated with dash (-) did not converge and remained at random chance.
Network
Dataset
0.0
1E-4
SE-4
1E-3
5E-3
1E-2
SB-2
IE-1
ResNer- 18
Tiny MetgeNee
59.33%
60.98%
62.33%
63.33%
64.65% %
63.79%
ResNet-18
CIFAR-10
95.06%
95.23%
95.25%
95.43%
95.38%
95.33%
MobileNetv2
CIFAR- 100
76.17%
76.54%
77.12%
77.43%
77.86%
78.61%
79.50%
77.91%
apply a dimensionality reduction in the form of Average Pooling across each channel. We show the results in fig. 2 We can see that the main accuracy gain of the networks trained with TNA comes from the deeper features. This lies in line with the results from the CKA analysis, thus showing that the reason for a better performance is a better combination of similar low-level features, resulting in better performing high-level features.
Corruption Robustness. Introduced by Hendrycks & Dietterich 2019). TinylmageNet-C a
dataset that consists of different perturbations applied on the TinylmageNet dataset. The goal of this dataset is to measure the robustness of the model to different types of corruption it might en- counter the wild. The performance on specific corruption is calculated as the mean over different levels of perturbation strength. We list the corruption robustness for ResNet- 18 in table We notice that networks trained with TNA prove more robust to each of the corruptions, thus proving that the representations learned by TNA are stronger than those trained with regular training.
4.4 ABLATION STUDY
The effect of By setting O, the logit matching loss term from LINA (eq. (1) is removed, which in turn severs the dependency between the two networks during training and equates to train- ing both independently. As such, varying the factor a results more less guidance via the twin network. To determine the optimal amount of guidance required, we perform an ablation test on the parameter a, of which we list the results in table
From these results it is clear that higher indicating more guidance of the twin network, results in better performance of the base network. However, at some point, a too large a can inhibit the training procedure, this is evident with MobileNetv2 CIFAR-100, where if a = 0.1 (last column of table 7 the model lags significantly behind the other settings the first epochs of training- An even larger leads to non-convergence and networks that are better than random chance.
Differently sized twin networks. As of now, in our experiments, we used twin networks of the same size as the base network. Le., if the base network is ResNet-18, then the twin network is (differently initialized) ResNet-18 18. 8. In this ablation, we remove that constraint, and study a combi- nation of different ResNet models the TinylmageNet dataset. We list the results in table 8 We notice that for ResNet-50 on this dataset, we require lower value for a to avoid non non-convergence More specifically, for ResNet-50
5E-4 rather than 5B-3 for the other settings.
8

---

Published as a conference paper at ICLR 2025
Table The impact of twin network size on base network accuracy for TinyImageNet.
Twin Network
Base Network
ResNet-18
ResNet-34
ResNet-50
ResNet- 18
64.75 0.47%
64.51 + 0.25%
62.17 0.53%
ResNet-33
66.26 + 0.23%
65.97 + 0.41%
64.69 + 0.48%
ResNet-50
65.08 0.38%
64.70 + 0.24%
64.94 0.77%
3500
(frozon
3000
Logit Motchinn Loss
2500
1500
Type TNA
1000
0.1
500
Standard
0.0
25
50
1.00
125
150
175
200
23
75
1.00
125
150
325
200
Epach
Epoch
Figure 4: The impact of using a frozen fully-trained ResNet-18 model as twin on (left) TinyIma- geNet validation accuracy and (right) training loss.
We notice that applying TNA with a small and large model produces better performing large model, but worse performing small model, than applying TNA with same size models. This shows that Logit Matching seemingly harming the performance of the small model benefit the larger model. We hypothesise that the cause of this lies in the different convergence behaviour of larger networks compared to smaller networks. To illustrate this, we have plotted the validation accuracy curves of standard training on TinyImageNe in fig. We can see that during the initial epochs, the accuracy of the large ResNet-50 model is substantially lower, even though it converges to higher final accuracy. When combining two networks with different convergence behaviours in TNA, this leads to an 'averaging'' behaviour pulling down the smaller model. This phenomenon is not observed in traditional KD methods, in which the knowledge extracted from larger, fully trained model.
Frozen, fully-traines twin networks. One of the disadvantages of TNA is that training two net- works in parallel is more computationally expensive than training a single network, the computa- tional graph is increased drastically. As such, a straightforward application would be to use frozen fully-trained twin network as guidance for the base network. Not only would this vastly speed up the backward pass during training, the base network will already have access to high-quality logits the start of training. This definition coincides with Knowledge Distillation approach following Kim et al. (202T). in which the loss on the hard targets (CCE) is weighted significantly more than the loss on the soft targets (Cmsstching)
In fig. 4 we use ResNet-18 18 Tinylmagenet to show the results. We notice that when using frozen pretrained twin network, the validation accuracy of the base network higher in the earlier epochs, but converges to lower accuracy than the co-training of the twin network and the base network. We hypothesise that this due to the concurrent mutual reinforcement of the twin and base network which provides better guidance within the later epochs than using frozen network. This is evident in the evolution of the Logit Matching loss, where we notice that the logits of the base network and the frozen twin substantially different.
Late matching, Another improvement, which has been proposed by Anil et al. 2018)) is to apply the distillation loss in online KD in later phase. While Anil al. 2018) only studied this with the KL Divergence, we also apply this on the L2 matching loss. The idea behind this approach is that networks at initialization and during early training are too different, and such matching

---

Published as a conference paper at ICLR 2025
Matching
0.7
epochs
0.6
Logit Matching Loxs.
200
300 epachs
350 epochs
0.5
Marching
Accurary
155
0.79
Late Matching None
100
0.2
0:28
spocts
Matching
0.0
25
50
75
100
125
150
175
200
25
75
1000
125
130
175.
2.00
Epoch
Epoch
Figure 5: The accuracy (left) and logit matching loss (right) of ResNet-18 on CIFARI00 for differ- ent settings (at 0, 50, 100 and 150 epochs) of Late Matching.
the features would be counterproductive for their respective training. We conduct experiments for different starting points for the Logit Matching and show the results in fig. 5
Analysing the results on R8 with CIFARI00, we notice that TNA without late matching outperforms the different late matching approaches, albeit with a small margin, and we link later matching with lower performance. Additionally, we notice that when late matching is activated, the validation accuracy jumps sharply in that epoch which coincides with a sharp drop in the logit matching loss, showing that the application of logit matching has immediate and noticeable im- pact. Interestingly, this after a few epochs the validation accuracy drops again, albeit to a higher value. The cause of this is not further explored, but we hypothesise that there is likely mismatch between the new position in the loss landscape and the current learning rate.
CONCLUSION
We have assessed the applicability of the Twin Network Augmentation methodology on set of var- ious CNN benchmarks, demonstrating that TNA consistently improves the baseline by significant margins. To determine the effect of the additional trainable parameters, we conducted an in depth study comparing TNA to similar approaches, and we show that for a given number of trainable pa- rameters, TNA outperforms with an additional bonus of lower inference cost. The performance gain can be boosted even further by ensembling the base and twin network at the cost of no longer having faster inference. Based formance increase when training with TNA is linked to the matching of logits between two equally well-trained models, and that matching needs to occur throughout the training process for best sults. Using a frozen model, or two models with too different sizes, results in less pronounced increases over the baseline, or even decreases.
set of carefully crafted ablations discovered that the per-
Next, we conducted a thorough analysis of the representations learned through TNA to understand the underlying reasons for the superior performance of these networks compared those trained via conventional methods. Our analysis reveals that although the low-level features are similar between standard trained networks and those learned via TNA, the main difference made the high-level features, which are more dissimilar those of regularly trained networks. We link this difference to a higher utility in classification via linear probes.
Next Steps. While we show that TNA induces better high-level features within networks and as such, leads to better validation accuracy, we do not find a specific root cause for this phenomenon. As such, determining the exact cause for the observed better performance might allow training techniques with less overhead. Additionally, we can study different levels of supervision, such as matching intermediate features opposed matching only at the logit level.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is partially supported by the Dehousse Mandate id:54/RECBV007 from the Department of Computer Science the University of Antwerp, the Flemish Government under the "Onderzoek- sprogramma Artificiele Intelligentie (AI) Vlaanderen" programme and a SB grant (1887022N) from Research Foundation Flanders (FWO).
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference on Learning Representations Workshops, 2017.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. Interna- tional Conference on Learning Representations, 2018.
Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. Neural codes for image retrieval. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part I 13, pp. 584-599, Springer, 2014.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- Ik. arXiv preprint arXiv:2205.01580, 2022.
Davide Chicco. Siamese neural networks: overview. Artificial neural networks, pp. 73-94, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Lucas Deckers, Benjamin Vandersmissen, Ing Jyh Tsang, Werner Van Leekwijek, and Latré Steven. Twin network augmentation: A novel training strategy for improved spiking neural networks and efficient weight quantization. arXiv preprint arXiv:2409, 15849, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pallern Recognition, pp. 1 11020-1102)) 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations. 2019.
GE Hinton.
Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.01207.0580 2012.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 103. 2015.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

---

Published as a conference paper at ICLR 2025
Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of applied statistics, 45 (15):2800-2818, 2018.
Taehyeon Kim, Jaehoon Oh, Nak Yil Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback-leibler divergence and mean squared error loss knowledge distillation. arXiv preprint arXiv:2105.08919 2021.
Simon Korblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86, 1951.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Andrew Y Ng. Feature selection, 1 I vs. 12 regularization, and rotational invariance. In Proceedings of the twenty- first international conference on Machine learning, pp. 78, 2004.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. Proceedings of the International Conference on Learning Representations, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Matthew Schultz and Thorsten Joachims. Learning distance metric from relative comparisons. Advances in neural information processing systems, 16, 2003.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3:1-40. 2016.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 07713-77222222 2019.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320 4328, 2018.
A TRAINING CONFIGURATIONS
A.1 TNA AND STANDARD TRAINING
We list the training configurations used for each model table 9 For each model, we used a SGD optimizer with the specified LR, except for the ViT for which used ADAM.
12

---

Published as a conference paper at ICLR 2025
Table 9: Training configurations used for different dataset network combinations.
Network
Dataset
Batch Size
LR
Epochs
ConvNet
CIFAR-10
256
0.1
200
ResNet-18
CIFAR-10
256
0.1
200
MobileNetv2
CIFAR-10
256
0.1
200
ConvNet
CIFAR-100
256
0.1
200
ResNet-18
CIFAR-100
256
0.1
200
MobileNetv2
CIFAR-100
256
0.05
200
ConvNet
TinylmageNet
256
0.1
200
ResNet-18 18
Imagent
256
0.1
200
MobileNetv2
TinylmageNet
256
0.05
200
ResNet-50 ViT-S/16
ImageNet
256
0.2
90
ImageNet
1024
0.001
90
A.2 NETWORK STRUCTURES
CenyNet structure. The ConvNet we use during our experiment with CIFAR-10 10, CIFAR-100 is listed in table 10 After each layer, apply the ReLU activation function, and before the linear classification layer, we flatten the input. For ImaglmageNet, we replace the last pooling layer with adaptive pooling layer to ensure that the input to the linear layer remains the same shape.
Table 10: simple ConvNet for use with CIFAR-10,
Layer
Description
Conv1
3 x Conv layer w/ 16 filters
Conv2
3 X 3 Conv layer w/ 32 filters
Pool
2 x 2 Avg Pooling layer
Conv3
3 x 3 Conv layer w/ 64 filters
Conv4
3 x 3 Conv layer w/ 64 filters
Pool
2 x 2 Avg Pooling layer
Fc
Linear layer w/ 1600 x 10 connections
ResNet-18 structure. Following common practice, we modify the ResNet-18 structure for small- image datasets (CIFAR-10, CIFAR-100, ImageengeNet). We do this by replacing the first convolu- tional layer and omitting the first MaxPool layer. More specifically, rather than convolutional layer with 7 kernel, a padding of 3 and stride 2; instead we employ convolutional layer with 3 x 3 kernel, padding of and stride of 1.
A.3 LINEAR PROBING
To insert the linear probes, we first identified the residual blocks within a ResNet-18 model. Next, freeze the weights up until that point and inserted Downsampling layer via Average Pooling and a linear classification layer. By using this downsampling, we considered a single value per channel avoid excessive computations.
Training the linear probes is done on the full training dataset with reduced learning rate of 0.01 for 10 epochs. Afterwards the accuracy of the linear probe is calculated via the validation set, and this is considered as a proxy of the feature quality at that point in the network.
B COMBINING TNA WITH DIFFERENT AUGMENTATION VIEWS
In the main paper, we have highlighted the similarities between TNA and Contrastive Learning. Both approaches attempt to align two views of the same datapoint. In the case of TNA, this is by
13

---

Published as a conference paper at ICLR 2025
using feeding the datapoint through two distinct networks, while contrastive approaches instead feed two augmented views the same datapoint through a single network.
3.6
275
0.5
3500
Mandhng
0.4
225
390
Thii Augmentations
TWin Augmentaiooe
Strme Different
Same D Offerent
100
125
193
175
Figure 6: The convergence curves of the (left) validation accuracy and (right) Matching loss, for TNA on ResNet-18 Tinyllmaeeeet with, and without different augment
While the effectiveness of both approaches has been shown, one could ask the question what would happen when combining both. This means generating two views of the same datapoint and then feeding each view through a distinct network, and matching the logits as such. In fig. 6 we show the convergence curves of TNA with and without differing augmentations.
We notice that the case where both networks receive different augmented views of the same sample, significantly impacts the convergence of the models. This is caused by severely increased Logit Matching Loss, compared the baseline TNA. Based on this empirical evidence, we hypothesize that the use of augmentations results in logits that are too different from each other, thus leading to an adverse effect on the training phase.
14


Reviews:
[{"review_id": "IiNyyKVrim", "reviewer": "Reviewer_4gAE", "summary": "This paper investigates the Twin Network Augmentation (TNA) method, originally proposed for Spiking Neural Networks, and evaluates its effectiveness on Convolutional Neural Networks (CNNs). The authors show that concurrently training two networks and matching their logits using Mean Squared Error (MSE) loss improves validation accuracy across various CNN benchmarks. They compare TNA to traditional Knowledge Distillation methods, highlighting its superior performance and robustness.", "questions": "1. The application of mean squared error (MSE) loss on logits enhances performance but doubles the training parameters when using twin networks like ResNet-18. It is essential to assess whether ResNet-18 serves as a reasonable baseline, especially in comparison with models that have similar parameter counts. The objective should be to establish a computationally feasible baseline.\n\n2. Although Section 4.2 offers some intuition for the performance improvements, a more rigorous theoretical analysis is required to enhance understanding of the underlying mechanisms.\n\n3. The writing of this paper should be largely improved. For example, lines 58 and 187 must utilize `\\citep` for proper citation formatting. Additionally, both Figure 5 and Table 10 are missing concluding periods, and line 630 must also end with a period for consistency.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper conducts extensive experiments to demonstrate that training two twin networks with the same objective, while simply minimizing the mean squared error of their output logits, can significantly enhance the performance of a single network. The primary contribution of this paper lies in validating the effectiveness of this approach not only for SNNs but also for CNNs.", "weaknesses": "1. This paper simply transfers the method from reference [1] to traditional CNNs, showing some performance improvement based on the experimental results. Although the authors conducted a wide range of experiments from different angles to validate its effectiveness, the overall contribution lacks novelty. In other words, the only modification was replacing SNNs with CNNs, which is trivial.\n\n2. This method is only validated on the classification task and is only compared with the outdated backbones.\n\n*Reference*\n\n[1] Deckers, L. et al. Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0zzDa0sudm", "reviewer": "Reviewer_BRu9", "summary": "This paper first introduce Twin Network Augmentation (TNA) from SNN, then validate the viability of this method on a wide range of popular CNN benchmarks and compare this approach to existing Knowledge Distillation schemes.", "questions": "Why does Eqn. 2 not align with the DML loss described in Figure 1 (b)?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper introduces the TNA method from the SNN field into CNNs for the first time, demonstrating its applicability across a variety of CNN models. The TNA method improves overall performance by simultaneously optimizing two networks and using MSE loss to adjust their logits.", "weaknesses": "1.This paper validates the feasibility of the TNA method in the CNN field. Although the experiments in this paper are comprehensive, they do not sufficiently analyze the reasons why the TNA method is feasible. Considering that this paper does not propose a new method, I doubt whether the work meets the standards of the ICLR conference.\n\n2.Could the author explain why TNA is superior to DML? According to Tables 2 and 7, TNA only surpasses DML under special alpha values.\n\n3.TNA requires training two networks simultaneously, resulting in doubled training costs. How to balance these costs against performance gains?\n\n4.Section 4.3 and Table 5 are confusing. Please clarify what each column in Table 5 represents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6PkbwLSDy", "reviewer": "Reviewer_83Qm", "summary": "The authors implement a new method called Twin Network Augmentation (TNA) for CNNs that improves the downstream performance by training two networks in parallel and using their logits in a combined loss function. The loss function consists of the cross entropy loss of each network's output with a given label and the mean squared error of the network's logits. \nThey compare their method with already established methods like Deep Mutual Training, as well as the standard machine learning approach with similar or larger networks (i.e. wide resnet compared to resnet with TNA).", "questions": "1. In 3.3, describing the network architectures (i.e. ResNet-18, ResNet-50,...) and corresponding datasets: Usually, for small images (e.g. 64x64 in Tiny ImageNet) the first 7x7-conv-layer of a ResNet is swapped for a smaller 3x3-conv-layer without stride. Did you follow this protocol?\n2. Tables 1 and 2 use the terms 'Default Train' and 'Single Train,' respectively. Is there a distinction between the two?\n3. Both the base and twin networks currently receive the same image. Have you considered applying slightly different augmentations to each network, as is commonly done in contrastive learning approaches like SimCLR (https://arxiv.org/abs/2002.05709)?\n4. In 4.4.1, why does a larger alpha result in networks that perform no better than random chance? High values of alpha (much greater than 1) inherently produce larger gradients, necessitating an adjustment in the learning rate. Could this be a contributing factor to poor performance?\n5. In section 4.4.4, you investigated late matching. Have you thought about 'warming up' the alpha (e.g. exponentially)?\n6. In the right-side plot of Figure 5, the curve for late matching at epoch 150 appears to decline independently, even before matching. What does the logit matching loss look like in the absence of any matching?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is written very well and easy to understand. The mathematical concepts are accessible and understandable. All the experiments align closely with the descriptions in the paper, effectively illustrating the theoretival insights. Notably, the paper includes a refreshing variety of experiments in the ablation studies, offering a comprehensive analysis of the TNA approach.", "weaknesses": "Major:\n- Tables and figures could benefit from more detailed captions, specifying, for example, the network and dataset used. While all relevant information is present in the text, including it in the captions would enhance clarity and accessibility.\n- The rationale behind the choice of data sets for specific instances is not always clear.\n\nMinor:\n- Tables 3 and 4 appear to be very small.\n- For Figure 2, a bar plot may be more suitable.\n- Table 7 may be improved by introducing a column with an alpha value of 0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HEt7612QGc", "reviewer": "Reviewer_S9Mr", "summary": "This paper investigates the potential use of Twin Network Augmentation (TNA), originally used to train spiking neural networks, in conventional CNNs trained for image classification in standard benchmarks such as CIFAR and ImageNet variants. The narrative then moves on to in-depth analyses of learning dynamics and ablation studies.", "questions": "- Did the original TNA paper talk about anything in Figure 1? Or is this an original attempt at unifying all the ideas?\n\n- I might have missed this in the paper, but in the conclusion the authors claim that \"the performance increase when training with TNA is linked to the matching of logits\" but isn't matching the softmax pretty much just matching a non-linear function of the same quantity?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is rigorous with a large number of convincing experiments and results. I appreciate the literature review of all similar methods and the summary provided in Figure 1 (even though it seems like they are a little too similar to have such drastically different names).", "weaknesses": "- Table 1 is quite chaotic. I can roughly understand why ResNet-18 is trained on TinyImageNet while ResNet-50 is trained on ImageNet and so on, but this is not rigorous or scientific, and can give the impression that the authors are simply cherry-picking results (I do not think this is the case). It would be better to keep one thing constant (network or dataset) and perform your experiments. (4 networks x 4 datasets is really only 16 cases, perhaps the authors can consider running them all).\n\n- There seems to be a lot of attention placed on DML. Did the original TNA paper also compare DML and TNA for spiking networks? If so, any different observations found for spiking networks vs conventional CNNs? If not, why is so much analysis placed on DML? To be more specific, knowing that TNA is different from DML does not tell me anything more about TNA, which is the focus of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)