dataesr/scientific-paragraphs-categorization huggingface dataset:
Template Attack The template attack relies on the Bayes theorem and considers the features as being dependent. In the state-of-the-art, it is mostly assumed that the noise follows a multivariate normal distribution. Accordingly, each P (X = x|Y = y) ∼ N (µ y , Σ y ) where µ y is the mean vector and Σ y the covariance matrix for each class y. The authors of  [9]  propose to use only one pooled covariance matrix averaged over all classes Y to cope with statistical difficulties and thus a lower efficiency. As a comparison to attacks based on neural networks, we use the pooled template attack for all datasets which do not include a masking countermeasure. In the case of masking, we apply the standard version of the template attack as each Σ y may not only include noise but also information about the class y and therefore the secret key.
The Design Principle: Sample-level CNN We adopted the more specific design strategy introduced in  [25]. This strategy is the 1-dimensional analog to the VGG architecture, characterized by the use of the minimal filter size and highly stacked convolution layers followed by a few fully-connected layers. It is reported that the network based on the strategy indicates better performance in the music related classification tasks which the strategy is originally developed on  [25]. Surprisingly, by grid search and cross validation, the authors in  [18]  reach very similar architecture setup, while this network is developed on the side-channel analysis dataset. It implies that this design is potentially suitable for a various range of problem domains where the input signal is given as the 1-dimensional signal.
ASCAD Neural Network Along with the model we introduce in Section 3. 1, we also test another CNN model introduced in  [18]. For brevity, we refer this model as the ASCAD model in the rest of the paper. This model is developed on the dataset which is also introduced in the paper, by grid search on a number of important model hyperparameters. One of the biggest difference between this model and the above model is the size and depth of the network. Compared to a model developed on the same dataset following the above principle, for instance, ASCAD model has approximately 130 times larger number of parameters, mostly due to the fullyconnected layers with large dimensionality in its hidden units. Still, in terms of depth, the other model (RD network) is deeper, which means that more abstractions can be conducted by the convolution blocks.
After the tuning process, we set the mean and standard deviation of the Gaussian distribution where the noise tensor Ψ is drawn as 0 and 1, respectively. For the ASCAD model, we use the average and standard deviation of the entire trace for each dataset to draw the random noise. Also, since ASCAD model does not have the batch normalization layer, the noise application is executed as the following way: X * = X + αΨ ASCAD d , (5) where Ψ ASCAD d indicates the random noise drawn from the normal distribution whose average and the standard deviation is derived from the dataset d. For all cases, we apply the mixing coefficient α = 0. 5. Experimental Details As already introduced in Section 3. 1, the detailed configuration can differ across datasets since it depends on the spatial shape of the input signal.
The detailed configuration of the models dedicated for each dataset can be found in Appendix A, Table  2. We initialize the weights to small random values and we use the "ADAM" optimizer  [30]. In this work, we ran the experiment with computation nodes equipped with 32 NVIDIA GTX 1080 Ti graphics processing units (GPUs). Each of it has 11 Gigabytes of GPU memory and 3 584 GPU cores. Specifically, we implement the experiment with the PyTorch  [31]  computing framework in order to leverage the GPU computation.
Datasets We consider four datasets covering the main types of SCA scenarios. The first dataset is considered with no countermeasures and has a small level of noise. This dataset represents an ideal attack target (and one not seen often in reality) and is consequently a good indicator of the best possible behavior of the attack technique. The second dataset again does not have any countermeasures but has a high level of noise. Consequently, it represents a difficult target for profiled techniques since the high level of noise makes the boundaries between classes fuzzy. The third dataset implements a random delay countermeasure, which makes it a realistic example of a dataset one could encounter in actual implementations. Finally, the last dataset has a masking countermeasure, which is probably the most widespread technique of SCA protection nowadays.
DPAcontest v4 DPAcontest v4 provides measurements of a masked AES software implementation  [32]. As the masking is found to leak first-order information  [33] , we consider the mask to be known and thus turn the implementation into an unprotected scenario. It is a software implementation with most leaking operation not being the register writing but the processing of the S-box operation and we attack the first round. Accordingly, the leakage model changes to Y (k * ) = Sbox[P i ⊕ k * ] ⊕ M known mask , (6) where P i is a plaintext byte and we choose i = 1. The measured signal to noise ratio (SNR) attains a high maximum value of 5. 8577. The measurements consist of 4 000 features around the S-box part of the algorithm execution. This dataset is available at http://www. dpacontest. org/v4/.
Random Delay Countermeasure (AES RD) As a use case for countermeasures, we use protected (i. e. , with a countermeasure) software implementation of AES. The target smartcard is an 8-bit Atmel AVR microcontroller. The protection uses random delay countermeasure as described by Coron and Kizhvatov  [34]. Adding random delays to the normal operation of a cryptographic algorithm has as an effect on the misalignment of important features, which in turns makes the attack more difficult to conduct. As a result, the overall SNR is reduced. We mounted our attacks against the first AES key byte, targeting the first S-box operation, i. e. , Y (k * ) = Sbox[P i ⊕ k * ], (8) with i = 1. The dataset consists of 50 000 traces of 3 500 features each. For this dataset, the SNR has a maximum value of 0. 0556.
Recently, this countermeasure was shown to be prone to deep learning based side-channel  [3]. However, since its quite often used countermeasure in the commercial products, while not modifying the leakage order (like masking), we use it as a target case study. The trace set is publicly available at https://github. com/ikizhvatov/randomdelays-traces. In the rest of the paper, we denote this dataset as the AES RD dataset.
ASCAD Finally, we test our architecture on the recently available ASCAD database  [18]. The target platform is an 8-bit AVR microcontroller (ATmega8515) running a masked AES-128 implementation and measurements are made using electromagnetic emanation. The dataset follows the MNIST database and provides 60 000 traces, where originally 50 000 traces were used for profiling/training and 10 000 for testing. We use the raw traces and use the pre-selected window of 700 relevant samples per trace corresponding to masked S-box for i = 3. As a leakage model we use the unprotected S-box output, i. e. , Y (k * ) = HW (Sbox[P i ⊕ k * ]). ( 9 ) Interested readers can find more information about this dataset at  [18]. This dataset is available at https://github. com/ANSSI-FR/ASCAD. Note that, the model given in Eq.
(  9 ) does not leak information directly as it is first-order protected, we, therefore, do not state a model-based SNR. The SNR for the ASCAD dataset is ≈ 0. 8 under the assumption we know the mask while it is almost 0 with the unknown mask. In Figure  2 , we depict the most important features for all considered dataset. To investigate the feature relevance, we use the Random Forest classifier. Random Forest is a well-known ensemble decision tree learner  [35]. Decision trees choose their splitting attributes from a random subset of k attributes at each internal node. The best split is taken among these randomly chosen attributes and the trees are built without pruning, RF is a parametric algorithm with respect to the number of trees in the forest. For this experiment, we use the number of trees equal to I = 500.
Notice, for DPAcontest v4 there is a region containing all the relevant features. For the ASCAD dataset, we still observe some regions with more important features while for the AES HD and AES RD datasets, there appears to be no single region where important features are concentrated. Table  1  sums up the details about the datasets as used in our experiments. Note that, while the CNN architectures take as input the complete measurement trace and therefore all features, TA (pooled) needs a pre-selection. For the datasets of DPAcontest v4, AES HD, and AES RD we select 50 features with the highest absolute correlation coefficient between the measurement traces and the corresponding leakage model Y (k * ). As for the ASCAD database the leakage model is not directly correlated with the measurement traces, we follow the approach in  [18]  and perform a PCA and select the 15 most relevant components.
Data Preparation When considering machine learning techniques, we divide the dataset into training, validation, and testing parts. The testing set in our experiments has 25 000  This particular splitting method, often called as repeated random subsampling or Monte Carlo cross-validation  [36] , has an advantage that the number of samples in the validation set is not dependent on the number of trials, since in this case the training and validations sets across the trials are not disjoint as the k-fold cross-validation. Compared to the relatively large number of classes we need to deal with, a number of our benchmark datasets contain limited training sets after holding out 25 000 testing traces. This makes the number of validation traces even smaller to the point where we have only approximately 10 traces per class if 10-fold cross-validation is applied.
Experiments In this section, we start by providing results for our RD neural network, ASCAD neural network, and template attack (pooled) for two noise levels: α = 0 and α = 0. 5. Afterward, we give additional experiments with different levels of noise in order to explore its limitations. As for the training, we run both of the neural networks for 75 epochs per each dataset, except the AES HD dataset, where the networks are trained for 300 epochs. For every training, we monitor the validation accuracy to keep the best model in terms of validation performance during the training even for the case when the validation performance is decreased due to the overfitting.
DPAcontest v4 The results on the first dataset we consider are given in Figure  3  and represent DPAcontest v4, which is the easiest dataset since we consider it without countermeasure and the measurement only have a small level of noise. The RD network is able to reach very good behavior where it requires only a few traces to break this implementation. Next, we see that the behavior after noise addition improved significantly. More precisely, GE results for RD network with noise addition are almost half the results without artificial noise addition. When considering looking at each fold separately, we see that most of the folds with noise are better than the folds without noise. Interestingly, there is one fold that is several times better than the average behavior with noise. For the ASCAD network, we see that DPAcontest v4 is not an easy dataset.
When averaged, we see a slow improvement with the increase in the number of traces but GE stays quite high even for the complete test set. The addition of noise helps only when the number of traces is small and there is a point around 15 000 traces where the behavior is the same for the network with and without noise. A very different behavior is seen for the ASCAD network. Here, we see that for all folds GE reduces with the increase in the number of traces and that artificial noise is beneficial for breaking this dataset with the ASCAD network. When comparing the behavior of the ASCAD network and RD network, we see that the ASCAD network outperform RD. When considering TA, we see that added artificial does not help but GE decreases even in that case with the increase in the number of traces.
AES RD The results for the AES RD dataset are given in Figure  5. This dataset contains a countermeasure in the form of random delays and the behavior of the considered classifiers is very interesting. First, we see that the RD network exhibits superior performance where for all folds GE decreases with the increase in the number of traces and the results with artificial noise are much better than those without noise. When averaged, we see that we require only 2 traces to break the random delay countermeasure implementation. For the ASCAD network, the noise is beneficial since it reduces GE with the increase in the number of traces for all folds, while for the scenario without noise, we see a number of folds that do not improve with the increase in the number of traces. When considering averaged results, to break this implementation we need around 300 traces, which is 150 times more than for the RD network.
Finally, considering TA, we see that the results with and without artificial behave similarly and the required number of traces to be around 20 00 to reach the same performance as RD and ASCAD networks. ASCAD Finally, we give results for the ASCAD dataset in Figure  6. Note that the results given here slightly differ from those given in  [18]. This is natural since we use fewer traces in the training phase. When considering the RD network, we see that all folds improve with the increase in the number of traces but for some folds that improvement is very slow. It is difficult to say whether adding noise helps here since the results are very similar and only after a few hundreds of traces we see that added noise behaves better than no noise. Next, we see that the ASCAD network shows slightly worse performance than RD network if we consider scenarios without added noise.
This is somewhat surprising since the ASCAD network was developed especially for the ASCAD dataset and intuitively one would expect that a network which is thoroughly tuned for a certain    dataset would perform better than a network developed to perform well over a number of datasets. However, we see a substantial increase in the performance of the ASCAD network once the noise is added. Finally, since pooled TA cannot break this implementation, we depict the results for template attack. For a small number of measurements, we see that TA is the most successful method while if considering more than 300 traces, TA and ASCAD perform similarly. What is particularly interesting is that we see that adding artificial noise benefits TA for a small number of measurements. (e) Template attack. Fig. 6 : Guessing entropy results for the ASCAD dataset.
Noise Addition In this section, we present results for the RD neural network and all datasets where we investigate different levels of noise. Besides considering the noise level α = 0. 5 as in the previous section, now we also experiment with α = 0. 25, α = 0. 75, and α = 1. Figure  7  depicts the obtained results for the RD network and all datasets. For completeness, we also include the results when α = 0 for averaged scenarios. For DPAcontest v4, we see that all considered noise levels result in the decrease of GE with the increase in the number of traces. The best result for a fold is obtained with α = 0. 5 but the best averaged result is for α = 1. This result is somewhat expected as DPAcontest v4 is easy to classify and adding extra noise simply makes the model even more distinguishable among different classes.
The noise level α = 0. 75 shows a slight deviation from the other experiments since the decrease in GE is slower when compared with the other noise levels. Finally, when considering no added noise scenario, we can see that actually it works the worst in the case of a very limited number of available traces. Already the second scenario, AES HD shows that α = 1 is a too significant level of noise and there GE actually increases with the number of measurements. Since AES HD is a very noisy dataset, here the smaller level of noise α = 0. 25 behaves the best. The results for noise levels α = 0. 5 and α = 0. 75 are similar and they even overlap at around 20 000 measurements. Here we see the best performance for the smallest noise level and then for each higher noise level, we see a degradation in the results.
The scenario with no noise added performs well for this dataset and the only better case is the smallest added noise equal to α = 0. 25. For AES RD, α = 1 exhibits the worst behavior while the other noise levels behave very similarly (notice that α = 0. 25 and α = 0. 75 behave almost the same). Still, when considering averaged results, we see that α = 0. 5 performs the best with slight advantage over α = 0. 25 and α = 0. 75. Similar to the DPAcontest v4, he scenario with no added noise exhibits the worst performance when the number of available traces is very limited. For the ASCAD dataset, we observe that α = 1 does not work well as it results in most of the folds with an increased GE with increased number of traces. The noise level α = 0. 25 gives the best results here with a small advantage over α = 0.
5 and α = 0. 75. Similar as for the AES HD dataset, we see here that the higher the noise, the worse are the results. The scenario without added noise is positioned in the middle -better than α = 0. 75 and α = 1 but worse than α = 0. 25 and α = 0. 5. As it can be observed, various noise levels can be successfully applied in order to improve the generalizations of a classifier. Additionally, we see that the classifier is sensitive to the changes in the noise level only to a certain extent, which indicates that for most of the applications, it is not needed to tune the noise level to a high precision. The noise level equal to α = 0. 5 shows a stable behavior over all datasets and can be considered as a good choice in our experiments. Naturally, taking α = 0. 25 would be also a viable choice, especially considering the AES HD dataset.
General Findings 1. It seems impossible to develop a single neural network that will show superior performance over all SCA scenarios. When discussing the performance of CNN classifiers, we observe that different neural networks perform well when considered over a number of scenarios. Still, the "No Free Lunch" theorem states that there cannot be a single best supervised classifier for all possible problems  [37]. Since the scenarios we need to consider in SCA are quite diverse with the respect to the number of measurements and features, levels of noise, and applied countermeasures, we consider as the only viable option to develop a suite of classifiers that are tuned to perform excellent for a specific problem. One example is the RD network and its behavior for the Random Delay dataset. Finally, we note that although we used different design principle for the RD network, the end result is similar to the neural network designed in  [18].
Interestingly, despite the similarities in the design, those two neural networks show very different behavior for different datasets. 2. It is possible to enhance the robustness of the model against the noise by adding Gaussian mean-free noise. We show how addition of noise can be actually beneficial in the process of classification since it enhances the robustness of the model in the presence of noise. Although it might sound counter-intuitive to add more noise in order to fight against noise, it is well-known that adding artificial noise in the input data is equivalent to adding regularization term to the objective function. Our results show that the added noise benefited both CNNs and even in certain cases (pooled) template attack. We emphasize the case of the ASCAD network that was tuned for the ASCAD dataset and yet, additional noise improved the results significantly.
3. When conducting the experiments with cross-validation, i. e. , using a number of folds, it is possible that the performance over folds will be extremely varying. Our experiments show that a neural network with good performance for a certain dataset can actually exhibit both good and poor performance when considering separate folds. We hypothesize that the difficulty of given classification task is relatively high, which eventually makes the accuracy stay very close to the random guess. This means that the variability occurred by the other stochastic components in the experimental framework, such as the random split in the cross-validation can have substantial magnitude of the variance. Such variability tends to make the model tuning process difficult, especially the model we use tends not to robust to the randomness.
As we introduced already, VGG is one of the most successful architectures adopted for a number of other application domains including speech, video, and music. Even though one starts their model configuration from such a "template", there is still a substantial amount of freedom to reach the optimal architecture according to the given task and dataset. The common practice is the heuristic search on the hyper-parameters of interest such as the previous works already introduced in the side-channel analysis domain  [17, 18]. On the other hand, one can also find a more specific "instance" that is proven as prominent in other domains especially if there is a considerable commonality on the characteristics of the signal between the two domains. Such instantiation is necessary especially for CNNs, since the details regarding the configurations often depend on the shape of the input signal.
Consequentially, it is not trivial to find a single CNN configuration that works optimally on multiple datasets, unless the CNN is designed specifically to have the timeinvariance by applying causal convolution  [22]. Hence, every instantiation from a certain design principle can be seen as a "new" architecture unless they have used the exactly same architecture that is developed by other, since architecture may be a mere another perspective of "parametrization". Likewise, it does not necessarily mean that another instantiation is always "novel", if the design does not have genuine novelty. On the other hand, such novelty is again possibly irrelevant to the practical performance. Our empirical results show that the two models used here are essentially instantiated from a very similar design principle, which means they might not be novel architectures while their performance varies significantly across the datasets.
Conclusions In this paper, we concentrate on one type of deep learning that showed potential in the context of profiled SCA -Convolutional Neural Networks. We adapt a CNN design principle successfully applied in a different domain but to a same type of signal, i. e. , 1 spatial dimension. With this CNN, we are able to reach high performance over 4 considered datasets where we especially mention the dataset with the random delay countermeasure. There, to break it, we require only 2 traces, which is to the best of our knowledge, a result highly surpassing anything that can be found in the literature. At the same time, we experiment with another CNN that has a similar design but exhibits a significantly different behavior for the investigated dataset. We believe that in the SCA context, we can discuss a general design principles that will work well over a number of datasets but still the specific instantiations need to be tuned for specific scenarios.