Interpreting Emergent Features in Deep Learning-based Side-channel Analysis
Authors: Sengim Karayalcin
Marina Krček
Stjepan Picek
Venue: NeurIPS 2025 poster ()
Keywords: Mechanistic interpretability
Deep learning-based Side-channel analysis
Explainability
Neural networks
Primary area: applications
TL;DR: We use mechanistic interpretability to reverse engineer how neural networks break protected cryptographic implementations via side-channel analysis.

Abstract: Side-channel analysis (SCA) poses a real-world threat by exploiting unintentional physical signals to extract secret information from secure devices. Evaluation labs also use the same techniques to certify device security. In recent years, deep learning has emerged as a prominent method for SCA, achieving state-of-the-art attack performance at the cost of interpretability. Understanding how neural networks extract secrets is crucial for security evaluators aiming to defend against such attacks, as only by understanding the attack can one propose better countermeasures.

In this work, we apply mechanistic interpretability to neural networks trained for SCA, revealing $\textit{how}$ models exploit $\textit{what}$ leakage in side-channel traces. We focus on sudden jumps in performance to reverse engineer learned representations, ultimately recovering secret masks and moving the evaluation process from black-box to white-box. Our results show that mechanistic interpretability can scale to realistic SCA settings, even when relevant inputs are sparse, model accuracies are low, and side-channel protections prevent standard input interventions.

Interpreting Emergent Features in Deep Learning-based Side-channel Analysis
Sengim Karayalçin
Marina Krček Radboud University, The Netherlands marina. krcekeru. nl
Leiden University, The Netherlands s. karayalcin@@iacs leidenuniv....
Stjepan Picek
University of Zagreb Faculty of Electrical Engineering and Computing, Croatia & Radboud University, The Netherlands stjepan picekêru.n)
Abstract
Side-channel analysis (SCA) poses real-world threat by exploiting unintentional physical signals to extract secret information from secure devices. Evaluation labs also use the same techniques to certify device security. In recent years, deep learning has emerged as a prominent method for SCA, achieving state-of- the-art attack performance at the cost of interpretability. Understanding how neural networks extract secrets is crucial for security evaluators aiming to defend against such attacks, as only by understanding the attack can one propose better countermeasures.
In this work, we apply mechanistic interpretability to neural networks trained for SCA, revealing how models exploit what leakage side-channel traces. We focus sudden jumps in performance to reverse engineer learned representations, ultimately recovering secret masks and moving the evaluation process from black- box to white-box. Our results show that mechanistic interpretability can scale to realistic SCA settings, even when relevant inputs are sparse, model accuracies are low, and side-channel protections prevent standard input interventions.
Introduction
Side-channel analysis (SCA) realistic security threat that consists of diverse methods that allow for the extraction and exploitation of unintentionally observable information of internally processed data [23). SCA enables the establishment of a relationship between passively observable information and the internal state of a device under investigation. As such, it poses a major threat to devices that handle sensitive data like keys, private certificates, or intellectual property (see, c.g., [40] 41)). In SCA, sensitive information gets extracted from device by observing its physical characteristics during computation (e.g., power consumption, timing).
Since 2016 [25). deep learning- based side-channel analysis (DLSCA) has received significant atten- tion from the research community [36). The main benefits of using deep learning (DL) over classical techniques are that assumptions about attacker capabilities can be relaxed, leading to better attack performance. Thus, integration of these techniques into evaluation procedures for cryptographic implementations has become standardized [12).
One of the main open challenges for black-box evaluations using DL is interpretability (36] model that can extract the key confirms there is some exploitable leakage, but does not indicate how the network exploits what leakage. Notably, this does allow the evaluator provide any feedback
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Actiation
In this work, we aim to understand what specific side-channel leakage a successful network has learned to exploit. Concretely, we derive features from model outputs, find geometric structures that emerge in principal components (PCs) during sudden jumps in performance (phase transitions), and relate these to the physical leakage. As practical consequence, utilize this emergent structure to PAI Ionn achoddare
Adaabor palciu
302
-22
HW
PC2
IC
180
Tven
103
-200
120
Los HW
High Hil
DWV
CLSE WREWE
ND40 oo/put
Figure 1: The analysis approach used in this study broadly consists of three major steps. After the performance increases are located using the PI metric, we plot logits to extract relevant features. Using these features, we plot the PCs of the activations and find the structure related to the leakage. Finally, we apply activation patching to reverse-engineer the masks.
beyond pass/fail, which complicates the cost-effective implementation of a solution. Understanding how neural networks learn exploit side-channel information can prove crucial for developing robust defenses against these attacks (39]. Thus, several attempts have been made to understand network
behavior. However, these approaches either focus only on input visualization [26] [8], use more
explainable model architectures [52] 53 or require access to masking randomness [4 35) We
discuss this related work in Appendix A
Although interpreting how neural networks perform computations is generally difficult, the algo- rithmic tasks performed in models trained on side-channel data are conceptually relatively simple. Learning to extract leakage information from masked implementations is similar to toy models that learn group operations in the works on grokking [37 30 7 (6). Concretely, for masked implementa- tions, the computations on sensitive value are split into d secret shares s = $2 sd. Then, to learn to extract leakage from side-channel signals, neural network needs combine leakage from each of these shares, often without the knowledge of individual shares even for the training set [27]
The connection between side-channel and grokking models is further motivated by the observation of Masure et al. that the learning curves for models trained against masked targets show an 'initial plateau' (Section 5.2 of (27). After number of training steps where test loss does improve, the models suddenly generalize to the test set and can extract the (sub)key. These sudden increases in performance raise the question of what the model is learning. Indeed, as some models for neural scaling predict neural networks learn in discrete steps (28). we expect that investigating what is learned during these transitions will give reasonable understanding of model behavior. Recent successful results of mechanistic interpretability (MI) investigating sudden generalization in toy and even language models (33) further motivate this direction.
models 30 42
From the point of view of MI, side-channel data provides an interesting test case. The data is often noisy, high-dimensional, characterized by subtle dependencies that difficult capture and interpret, and presents a real-world scenario. Additionally, the masks are hidden values and should not be publicly accessible which further complicates the application of MI as we cannot describe model behavior exhaustively with respect to concrete input features as in 30 or do (automated) input interventions align with a causal model as in [14 8
'Soettmess referred to as phase transitions related works 30
'Even in evaluation contexts with collaboration from developers, this is often impossible; see the introduction of 27 for in-depth discussion.

---

extract input features, i.e., individual shares s; related to device internal randomness, from model activations, providing a path to move from black-box white-box evaluations. The overall analysis process is illustrated in Figure T
To summarize, our main contributions are:
We explore the feasibility of applying MI in a challenging real-world setting where input interventions to features are not possible due to SCA countermeasures.
By investigating the changes in model outputs during sudden jumps in model performance,
we find how networks combine leakage in DLSCA.
We directly retrieve the internal secret share values by applying activation patches to intermediate layer activations across several targets.
We provide more detailed insights into the specific physical leakage that neural networks exploit for widely used (DL)SCA benchmark datasets. Notably, do this without assuming
a priori mask knowledge [54] 35] or requiring custom architectures [52] 53).
We find identical structures emerging during sudden generalizations for models trained on side-channel traces captured on different implementations and in different SCA do-
mains (electromagnetic vs. power), providing further evidence for the weak universality hypothesis 77.
Code to reproduce experiments is available at https://gittubb.co//Sennggg emergence
2 Background
Deep serning-based SCA (DLSCA): The main principle behind SCA that during the execution of an algorithm on a physical device, side-channel information, e.g., power or electromagnetic (EM) measurements, can be influenced by secret-dependent internal computation 23 For example, under standard assumptions, writing 0000 to register will consume less power than 1111. The goal of SCA then to extract this secret from the side-channel observations to establish security bounds for devices that operate in conditions where physical access may be possible for attackers (e.g., bank cards, passports, mobile phones). While many SCA variants exist, common division is into direct and profiling attacks [36]. Direct attacks assume a single device where the attacker uses (classical) statistical techniques to find the most likely key.
In profiled attacks, which are often used by device evalu- ation labs 66. one assumes the attacker can access copy of the device to be attacked. This copy is under the com- plete control of the attacker and is used to build model of the device. The attacker uses that model to attack a different (but similar) device. While profiled attacks are less practical due to the assumption of access to a copy of a device, can be significantly more powerful than direct attacks. Indeed, provided that the model is well- built and there sufficient information in the trace, one could need only single trace from the device under at- tack to obtain the secret key. On the other hand, direct attacks may require significantly more traces to break real-world target [36).One can easily observe a similarity between profiled attacks and the supervised machine learn- ing paradigm (where building a model is training, and the attack is testing). Consequently, in the last decade and more, many machine (deep) learning algorithms have been tested in SCA 36
Figure 2: Example of single trace cap- tured during the execution of the Ad- vanced Encryption Standard (AES) ci- pher
The main workflow for SCA is to take (large) number of traces (see Figure 2 for example) from the profiling device with known key(s). These traces, often containing anywhere from 100-10 000 points per trace") are then labeled using an intermediate value used during the computation that
Activation patching is technique from MI.
'Generally, only small number of points in the trace contains relevant information as the intermediate value is only used during few instructions, complicating classical SCA techniques.
3

---

depends on some public imput (i.e., plaintext) and some sensitive data (i.e., the secret key), The exact mapping between the trace and intermediate values also depends on the assumed leakage model, An evaluator can employ Identity (ID) leakage model, which takes the intermediate values directly, or use leakage models that assume some physical leakage function, e.g., Hamming Weight (HW) or bitwise models. The chosen leakage model then also directly influences the number of labels (e.g., when working with a byte-oriented cipher like AES, the HW leakage model results in classes, while the ID leakage model results in 256 classes). For AES, the label is often the output of the S-Box in the first round (S-Box (p @ k() since it only depends on one plaintext and key byte (p and k). The neural network is then directly trained these labeled traces, eliminating the need for labor-intensive (and often error- proor-rrone feature engineering processes 36).
In the attack phase, the trained model is used to predict intermediate values from the traces of the target device. As models often have accuracies that are only marginally above random guessing, evaluating success is done by accumulating the predictions across larger number of traces and evaluating which of the hypothetical key candidate is the most likely. In evaluation settings, the model is then said to 'break' the device if the correct key is the top candidate within some specified number of traces [44). Other information-theone trace. Perceived Information (PI) (38), easier-to-estimate lower bound on mutual information, is often used in DLSCA settings. Intuitively, a PI of 0.5 means there are 0.5 bits of key-related leakage in single trace.
metrics aim to directly quantify the secret information present a single
To protect against SCA, countermeasures such as hiding or masking are commonly used. In both cases, the goal is to remove the correlation between the observed quantity (traces) and secret information. Hiding countermeasures can happen in the amplitude domain by randonizing/smooting the signal or by adding desynchronizzaion/rando delays in the time domain. Masking (20), on the other hand, divides a secret variable into several shares such that one needs to know all the shares to obtain the secret information. For instance, consider a Boolean masking of secret variable If we combine that secret variable with a random value m, we obtain a new variable y: y = 8 + m. Then, to obtain information about 8, one needs to know both y and m. DLSCA can often automatically circumvent these countermeasures and still result in extremely efficient attacks without requiring additional access assumptions (c.g., the ability disable countermeasures the copy device). For a practical introduction to DLSCA, we refer readers to [22]. and for a broader overview of SCA, see [43].
Mechanistic Interpretability: Mechanistic interpretability (MI) aims to reverse engineer a neural network into human- understandable algorithms (32 31, 48 330. This involves identifying "features", which are directions in internal representations that correspond to concepts, and "circuits", which are subgraphs within the network composed of interconnected neurons and their weights, representing meaningful computations. Generally, the first step in the process of MI is to identify the features. Examples of features include low-level features such as curve or edge detector neurons in vision models [32]. or more high-level features corresponding to the board state in toy models trained on board games [24] 29). J. As features generally correspond to linear directions in the latent space, training linear probes []. i.e., small classifiers, is common for showing the presence of features in the latent space.
After finding features, the goal becomes to determine how these features relate model outputs (or other features). Ideally, we can create a causal abstraction of network behavior based on feature descriptions [14). One method for showing causal effects involves intervening in model activations by performing activation patches [17). Here, we replace (part of the) activations during forward pass with saved activations from another forward pass corresponding to a different feature value to understand the effects on model outputs. This allows for measuring the impact of a specific feature or, eventually, verifying that the circuit is a (faithful) description of the model behavior.
3 Analysis Approach
The analysis process is shown in Figure 1 and detailed in this section. However, additional analysis and MI techniques might have been used depending on the observed behavior and findings of each specific dataset. These additional steps and the reasons for them will be described directly in the experimental results (Section 4h.
$ As target byte time, exhaustively searching over 256 values is easy.

---

Assumptions. In (DL)SCA, the attack typically focuses on extracting a subkey (often single key byte) of the secret key. As we target post-hoc analysis of successful models, our analysis assumes that the attack has already succeeded and the subkey has been recovered. This assumption allows us to label (test) traces by deriving the intermediate value (label) from the input (plaintext) and the key (recall that we target S S-box|p (p @ k)). We do not assume we have access to mask values. The goal is to understand the model's behavior and identify what information it extracts from the traces to make predictions. Additionally, we aim to recover the masks used in the cryptographic algorithm, which enables us to recover the rest of the secret key with (significantly) less effort. Note that when the model fails to recover the correct subkey, suggesting that the underlying masks and feature representations were not properly learned, interpretability methods offer limited insight.
Logit Analysis. Once we have model that successfully recovers a subkey, our primary goal during initial exploratory testing is to understand the factors influencing the network's 's predictions. To achieve this, we analyze models at points directly after generalization and observe changes in the model predictions. As these sudden changes in performance suggest significant changes in the neural network's behavior during training, they are shown to be useful for discerning features (56). We examine the distributions of output logits for different classes, looking for clear separations between classes, indicating distinct patterns in the traces. We aggregate the distribution for model outputs for traces that belong to each class and visualize them to identify commonly confused classes. These insights enable us to formulate hypotheses about higher-level features influencing predictions. Opposed to other recent works that reverse engineer models, see, e.g., [30, 48), where the authors assign features to (or derive features from) model inputs, we rely on output logits as we do not have access to masking randomness. Additionally, the (physical) noise inherent to side-channel traces results in final model accuracies that be only marginally above random guessing, making single trace predictions challenging to analyze from the MI perspective. Note that the analysis becomes easier in white-box SCA settings, where one would assume knowledge of all internal values during computation, including the masking randomness, see 35).
A Activations Analysis. After finding and testing initial hypotheses about the physical leakage used for classification, we can look at activations and how these relate to the predictions. Considering that in SCA only a small number of operations in each trace should be relevant for the classification (i.c., only leakage related to the target value), the number of relevant features should be relatively small. This means PCA will likely reveal the features the model learns during the first phase transitions (42 56). Note that for more complicated tasks where there are more features than dimensions, sparse autoencoders can be an alternative to extract features (10 4
As we expect structure to emerge in the first few PCs [42), we can plot the distribution of attack traces for features we derive from the logit analysis. Still, while we expect specific structure would emerge in the first few PCs, some manual effort in determining the correct (number of) components and subdivisions might be necessary. However, in the tested cases, notice the structure generally emerges with up to the first four components. Ideally, we see clear divisions between groups of traces belonging to certain values. Even if this is not the case due to noise, some regions might contain more/less of certain groups, and the overall distribution should be tied to (noisy) physical leakage. After finding some structure, we should explain how it arises in terms of the physical leakage that is in the trace. For example, if there is a grid-like structure in the PCs, we could assume that (embeddings of) two secret shares correspond to the x y directions of the grid since we have two shares [
Reverse engineering masks with activation patching. When a structure is found, we need to verify that the hypothesized behavior is causally related to the model predictions in the expected way. We do this by fixing the directions that correspond to all but one share to a fixed value.
If possible, we try to fix them to 0 some other value that allows easy descriptions of the output based on the one varying share. Then, we observe how the model outputs relate to the final share. If the hypothesis about model behavior is correct, share from these patched outputs. Finally, after deriving secret shares, we can use Signal-to-Noise Ratio (SNR to plot where in the trace these shares leak to derive which secret share is which, i.e., which of the two shares is the mask and which the masked S-box output. Note that this patching
also directly derive the values for secret
'For higher d, the structure will be higher dimensions. "SNR measures the signal variance versus the noise variance. In SCA, a higher SNR indicates stronger signal compared to noise, making it easier to extract sensitive information.
5

---

35 without requiring a priori knowledge of Epoch 50
75
HIN
HW
HW
HW
536
0.2
+0
6
Cown
25
-25
Output Classes
Epoch
HW
0.4
0.0
-100 1.00
-100 300
Figure 3: Logit analysis (first column) and activation analysis (remaining columns) from models at epoch 50 (top) and epoch 100 (bottom) for CHES CTF. Legends for activation analysis are shared within columns. The difference the number of points between the last two columns due to not plotting the points for classes (HWs) 3, 4, and 5.
setup follows the activation patching method used in secret share values.
4 Experimental Results
This section presents results for three common public SCA targets CHES CTF, ESHARD, and ASCAD (see Appendix for details). The models are Multilayer Perceptron (MIP) neural networks with their hyperparameters taken from (35) for ESHARD and ASCAD (see Appendix Cp. For CHES CTF, we directly train the ESHARD model without additional hyperparameter tuning. Note that we focus MLP and CNN architectures as these are generally sufficient for state-of-the-ar performance in SCA 34 The analyses given here are similar (although somewhat more cumbersome) for CNNs (see Appendix
4.1 CHES CTF Dataset
For the [HES_CTRR target, we see Figure that there are two concrete increases in perceived information during
training. The initial increase starts at epoch 15 and is completed around epoch 40. After another plateau in PI,
Perceived Information
10a
there is a second increase between epochs 70-85, after which there are more significant changes in PI.
10-1
As we aim find what learned during these performance jumps, we show both average logits for different classes and the two main PCs Figure 3 After the first increase, at epoch 50, the predictions on the test set differentiate between high HW values and low HW values. When we use this information to plot PCs in the first layer (middle plot in Figure 3). we see that one diagonal corresponds to high HWs and the other to low HWs. This indicates that the HWs both secret shares mask and masked S-box output leak in the HW leakage model and that these are the features that map onto the PCs. Further details are in Appendix
10-2
Train
Test
50
100
150
200
Epochs
Figure 4: Evolution Perceived Infor- mation for training and test traces of the CHES_ _TF dataset.
Looking at the logits after the second performance increase at epoch 100, Figure shows that in addition the high-low HW divide, the models also separate even-odd HWs. Plotting the same components but separating even-odd HWs shows grid structure of even and odd points. In this grid, the number of changes in even-odd is about nine, corresponding to the nine possible HW values. The even-odd separation also clearly corresponds to learning the parity of target value from HWs

---

SNR retrieved
HWS of Sbox
of
Sbox 67/
-100
-100
10000
15000
100
-100
Trace Samples
PC
PC
Figure 5: SNR plot and PC distributions for mask values using patching experiments for CHES _TF. We set PCO to -20 for both patching experiments, as that resulted in more apparent separation during manual testing.
with Boolean masking (see Appendix H. This leads the ability learn the mask values that the network uses for classification, as discussed next.
Activation Patching: To validate that the PC embeddings are causally related to model outputs, can fix one the components and observe the effects on model outputs. An additional consideration is that when fix the value of one of the Hamming weights should be the HW value of the other share (or 8-output if we fix the first to 8) As such, if the PCs relate mask values, we can patch one share to 0 (or 8) retrieve the value of the other share.
(or 8), the output of the model
To practically extract mask values, we fix the value of one PC to be (near) one of the corners of the grid see in Figure 3 Then, examine the model outputs to verify whether the predicted value has changed as expected. As the model generally predicts HW values between 3-5 (because those occur most), we know the expected number of occurrences for each HW we can take the first 1/256 values to be HW = 0, then the next 8/256 values for HW = 1, and
sort each trace by the difference of logits for high (5-8) and low (0-3) HWs. Since
The resulting mask and masked S- S-box distribution are shown in Figure []] We can see that fixing values of certain PCs to extremes results in the model basing its predictions mainly the other PC, as is expected when one the shares is fixed to 0 (or 8). When we visualize SNR for each share, we observe clear spikes corresponding to the usage of the leaking values. First, we see spikes related the value of Tm. indicating the loading of the mask and some pre-processing before the encryption. Then later, we see leakage related to S box(pp, @ kj] ① T'- Due to the page limit, ESHARD results are in the Appendix E In summary, there only one generalization spike, which results in the ability of the model to distinguish high-low HWs. The results are qualitatively the same as for
4.2 ASCAD Dataset
For the ASCAD target (the main bench- mark for DLSCA research since its in- troduction in 2018 (2), generally the ID leakage model is used as this results in better attack performance [2] (4 As such, for this dataset, we additionally train linear probes for each bit of both the S-box input and output.
1.0
Perceived Information
100
Accurasy
0.8
0.6
20c E8
Irain Test
10-1
0.4
50
100
100
Figure shows sudden transition to positive from epochs 8-12, corre- sponding to increased probe accuracies for the input bits. Immediately after, PI
Fpochs
Fpochs
Figure 6: Evolution of PI and probe accuracies for bits during training for the ASCAD dataset.
still increases marginally until improve- ment stops at epoch 25. This increase is accompanied by the increasing probe accuracies for the two least significant S-box output bits.
$ 'Note that patching one share to be 0 validate that the outputs become directly related the other share has been done before in (35] although by using knowledge of the masking randomness. II the mask values uniformly distributed, which they generally are for the security properties hold 20
7

---

p.9k.8r
00
0.04
00
0.00
-10
-10
-10
-10
PC
PCO
SBox
0..44
Legh
10
0.00
100
200
30
-30
10
10 10
Figure 7: Logit analysis for two LSBs of pj @ at epoch and S-box pp @ kj] at epoch 25 with corresponding actual mask values for ASCAD. Note that for the lower logit plot, we use only traces with pr k; in 00 for clarity, and that extracted mask values are in Figure
Indeed, the two least significant bits (LSBs) for both input and output clearly achieve far higher accuracies than the other bits, which only marginally improve over random guessing.
Looking at the logits in Figure 7 epoch 12, the values are distributed according to the two input bits. When we plot PCs to distinguish the values of these bits in Figure 7 we see an emerging structure in the first two PCs of the activations in the second layer corresponding to the combination of mask values by mapping these on certain axes. Note that the grid structure in both cases follows a 3 x 3 structure over the more ideal 4 x 4 if all four possible 2-bit values of the masks are perfectly distinguished. This is due to the physical leakage two classes for the secret shares (mostly) overlapping, shown in the two rightmost plots.
When we consider the logits at epoch for the output bits 10 the mean values are significantly higher. Additionally, the logits are spread out across fewer values. This aligns with the network's predictions, which now incorporate the information on the output bits. We also observe visually similar structure to the grid at epoch 12 appearing in the 3rd and 4th PCs for the S-box output bits.
The first two PCs remain related to the input bits as in epoch 12. Within the activation patching experiments for ASCAD, we observe causal effects outputs by training probes on the final layer and selectively intervening on key components. However, further refinement is needed to extract mask values accurately. The experiments are presented in Appendix F
Discussion
Recently, several works have characterized the algorithms learned by networks trained on simple al-
gorithmic tasks, e.g., modular arithmetic and more general group operations 33 7 [6]] Furthermore, other studies have identified individual circuits that perform grammatical operations in language models 48 33). These works showcase that interpretable algorithms are learned during discrete phase transitions, aligned with the neural scaling law from (28). which states that network training is collection of discrete 'quanta' that correspond (potentially interpretable) circuits. The eventual goal of interpreting a neural network then becomes to enumerate all of the phase transitions during training. In this work, we showcase that this type of ambitious interpretability can be possible for models trained on real-world datasets in SCA.
While this is positive result, the broader relevance of this is somewhat limited. As mentioned in the introduction, side-channel data poses challenges due to noise and unpredictable physical leakage characteristies, but is also very structured, and the number of relevant features is (expected be) very low. The core task of the networks for SCA, combining secret share leakage to recover the target intermediate value, is very similar to the group operations learned in (30]. Indeed, in the networks we investigate, there are at most two jumps in performance, allowing for more detailed examination
We fix the input bits to 00 increase visibility, for complete description, Appendix L.

---

of each individual case. Additionally, the low number of expected features avoids issues that are prevalent in larger models trained on more general tasks (e.g., superposition (TO).
key difference between our approach and those in previous works on group operations is that assume no knowledge of input features due to threat model. This then prohibits the of standard input interventions, as we cannot replace input features, As such, we need to derive features from the model outputs and find which inputs are grouped together by the model. By finding these groups, we can work backwards from the model outputs to find relevant input features. This still requires domain-specific knowledge as, otherwise, it might not be immediately obvious that the grid structure in Figure 3 relates to the embedding of secret share HWs.
Another consideration is that our analysis is focused model activations and outputs. We do not other available information, like gradients or input visualizations, in this work. Some previous works have focused on visualizing what parts of the input the network is using (see Appendix [A for some examples). However, without access to masking randomness, is difficult to relate this to which shares and intermediate values are (not) being exploited. In contrast, the extraction of secret shares from model activations allows us to generate SNR graphs that match the visualizations in Figure while splitting the individual shares. For automating analyses, gradient information could potentially be used, e.g., to identify phase transitions during training.
One of the most interesting points in our results is that the second performance jump in the CHES [HES_CTF model seems to rely on the first. The high-low HW embeddings of the secret-share features gets progressively refined to be direct embedding of the HW. When these embeddings are sufficiently clean, the model can distinguish even-odd HWs, as described in Appendix HH.2 This indicates that adapting SCA training methodologies first identify 'simpler' leakage and then build upon it with more complex leakage models could be a promising future direction for making models more efficient.
6 Relevance To DLSCA
As DLSCA becomes more common, it is increasingly important to understand how neural networks exploit implementations. This work provides concrete analyses for several common side-channel datasets, showing the possibility of reverse engineering masks from network activations. We show that specific structures occur for different side-channel targets, indicating that building library of common structures could be useful in analyzing future networks. This is especially relevant as masking schemes are often similar across ciphers (i.e., post-quantum ciphers often also use Boolean masking). As networks are often trained to recombine shares, these structures should be shared across different ciphers.
As our main practical result, we can reverse engineer secret shares from a trace by using the structures learned by the neural networks. To our knowledge, only [13] can extract mask values, where this work is focused on a specific implementation using classical side-channel techniques (thus, without considering machine learning approaches), which requires stronger assumptions than DL-based attacks. Extracting mask values substantially benefits evaluations, as we can move from black-box to white-box evaluations. This, in turn, would allow for better feedback to designers of cryptographic implementations.
One might question whether this is relevant for attackers, as we require model that already breaks (one key byte of) the target. When attacks target individual bytes, the difficulty of breaking any individual byte can vary, even for the same device. As such, when masks are shared for all bytes (which is often required for masking non-linear operations, c.g., the S-box in AES), spending significant effort to break one key byte might allow retrieving the shared mask. Subsequent attacks against other key bytes become more straightforward as we can use the retrieved mask during training to effectively move the attack to an unprotected case by including the mask, see the white-box evaluations in (3].
Finally, discovering how neural networks concretely defeat countermeasures can improve evalua- tion/attack methodologies and countermeasure design. On the evaluation/attack side, we can design more effective methods for label distribution that consider common mistakes networks make, which can improve convergence (50). On the defense side, understanding what type of leakage more/less casily exploited could lead to the design of more (cost-) deffectvee countermeasures that enable more

---

robust protections. Concretely, for the Boolean masking schemes we consider in this work, the structures that arise from the model embeddings of the secret share leakages into PCs naturally form the high-low structure. Further, when these embeddings are sufficiently refined, even-odd clusters emerge, indicating that using masking schemes that are less algebraically compatible with practical leakage functions, like prime-field masking 11], could also be beneficial for protecting against DLSCA.
Profiled attacks against real-world targets are often significantly more complex than idealized eval- uation settings, where the same device is used for both profiling and attack. Differences between devices often result in worse performance when models trained on a profiling device are applied to the target [3). In security evaluations, the same device is commonly used for both profiling and attack to represent the worst-case scenario where the device differences are minimal. As such, the attack sets of the considered targets are from the same device as the profiling set, which raises questions about the practical relevance of these results for real-world settings. However, this work considers post-hoe explanations for models that already break a target. Therefore, the experimental evaluations emulate what is possible even for (more realistic) non-profiled adversaries that obtain trained model using techniques like [45), as non-profiled attacks always consider single device.
Doing similar analyses in practice might still be difficult, especially for side-channel evaluators with limited expertise in DL. This work is then aimed at highlighting that ambitious post-hop interpretability in DLSCA is feasible. Future work can build on these results to find more automated approaches to aid evaluators performing these analyses in practice. Notably, automatically finding relevant features by matching novel model outputs to (variations of) features found in this work using, e.g., Kl.-Divergence, seems promising. Variations here can include what leakage model is expected (e.g., which bits leak, using Hamming Distance between operations over HW) and what specific operations leak ge.g., using earlier/later operations in AES).
Conclusions and Future Work
We show that interpreting neural networks trained side-channel models is feasible, without access to random masks. Moreover, we highlight the effectiveness of investigating the structures learned during discrete steps in model performance and find evidence for the weak universality of circuits in side-channel models. Finally, we leverage these insights to reverse engineer the mask values. Automating these analyses represents an interesting direction for future work. Additionally, further work on leveraging the insights into DLSCA models to improve evaluation methods could be useful. For example, using tailored leakage models that consider common structures could help simplify model tuning. Finally, we only focus on MLPs (and CNN in Appendix G as these networks provide ofte-oofthheart attack performance for the tested targets. Extending this Transformer-baat is an interesting direction for future work.
architectures also used in DLSCA [16] 13
Acknowledgments
The authors would like to thank Gorka Abad, Abraham Basurto- Basurto-Becerra, Azade Rezaeezade and anonymous reviewers for valuable feedback which helped improve this work. This work was (in part) supported by the Dutch Research Council (NWO) through the Challenges in Cyber Security (CiCS) project of the Gravitation research program under the grant 024.006.037, and through the PROACT project with grant number NWA. 1215..844...
References
[1] Guillaume Alain and Yoshua Bengio. "Understanding intermediate layers using linear classifier probes". In: 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL: hhttps //openreviee ew meet:forumiid-HJ445
[2] Ryad Benadjila al. "Deep learning for side-channed analysis and introduction to ASCAD database". In: J. Cryptogr: Eng. 10.2 (2020), pp. 163-188 Do1: 10 1007/S13389 019-
00220-8 URL: /ttt:s///ooo.oog///00000///33313
10

---

[3]
Shivam Bhasin et al. "Mind the Portability; A Warriors Guide through Realistic Profiled Side-
mind- htt.a///ww...aaaaa/o//ooo//.o/aa
channel Analysis". In: (2020). URL: //tpppp:://ww.... http://wwwwwdsssyyysysss. http://www..ds....ppp....uoaaaaa pappe
channel-analysiis/
[4]
Trenton Bricken et al. "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning..... In: Transformer Circuits Thread (2023). URL: https::/
sircuits.pub/ 2023 htta://222mmooommmmiiicccc monosemantiic-ffttuueeee www.--faeaeeeneeeeeeeeeeeeeeeeee html
[5]
Olivier Bronchain and François-Xave Xavier Standaert. "Breaking Masked Implementations with
Many Shares on 32-bit Software Platforms or When the Security Order Does Not Matter". In:
IACR Trans. Cryptogr Hardw. Embed. Syst. 2021 (2021), pp. 202 234. DOI: 10. 46586/
TCHES. V2021 13. 202 234
URR: URL: httt:http:///oii org/ / 10. 46586/ v2021. 13. 202
234
[6]
Suresh Chari, Josyula R. Rao, and Pankaj Rohatgi. "Template Attacks". In: Cryptographic Hardware and Embedded Systems CHES 2002, 4th International Workshop, Redwood Shores, CA, USA, August 13-15, 2002, Revised Papers. Ed. by Burton S. Kaliski Jr., Çetin Kaya Koç. and Christof Paar. Vol. 2523. Lecture Notes in Computer Science. Springer, 2002, pp. 13 28.
DO1: 10. --0 564000000300--555
URL: https: org/ /o.g///0./0
5 3
[7]
Bilal Chughtai, Lawrence Chan, and Neel Nanda. "A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations". In: International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. Ed. by Andreas Krause et al. Vol. 202. Proceedings of Machine Learning Research. PMLR, 2023, pp. 6243-6267. URL: attps://prooeeedinga mlr press w202/ http///222////222e/e chtghtaittae html
Arthur Conmy al. "Towards Automated Circuit Discovery for Mechanistic Interpretabilityy. In: Advances in Neural Information Processing Systems. Ed. by A. Oh et al. Vol. 36. Curran Associates, Inc., 2023, pp. 16318 16352.
httt::::/pppccceediiee paper files / http/////aoa////////o////o//att0 Conference pdf
URL: https /
neurips cc/
/ 2023 file 3aertbbebsddcceeaaaf99999bbbbbbe
Paper
[9] Joan Daemen and Vincent Rijmen. The Design of Rijndael: AES The Advanced Encryption Standard. Information Security and Cryptography. Springer, 2002. ISBN: 32540-4 DOI:
10. 10.1007/978-3-660 04722-44
URL:
1007 3
[10] Nelson Elhage et al. "Toy Models of Superposition". In: Transformer Circuits Thread (2022). URL.:
circuits. wtt.rrcuiss..uup///22ee pwwpp//2//aaaaeeeeepeppppe wtt.mmmdllnnde.x pbb//202//too_mmdddl//ddxx http:/didneeeemmmmlmlm
[11] Sebastian Faust et al. "Connecting Leakage- Resileenn Secret Sharing to Practice: Scaling
Trends and Physical Dependencies of Prime Field Masking". In: Advances in Cryptology EUROCRYPT 2024 43rd Anmual International Conference on the Theory and Applications of Cryptographic Techniques, Zurich, Switzerland, May 26-30, 2024, Proceedings, Purt IV.
Ed. by Mare Joye and Gregor Leander. Vol. 14654. Lecture Notes in Computer Science. Springer, 2024, pp. 316-344..00 DOI: 10 1007 / 978 3- 031 58737 5_ 12 URL: https
//doi. 10
[12] Federal Office for Information Security (BSI). Guidelines for Evaluating Machine-Learning based Side-Chunnel Attack Resistance. https : www bai bund de SharedDocs / Downloads Bwwmllads/DD/BBS//2E/
Itter//etttreeeertnnnne//e/ AIS 46 AI _I_guidee pdf
blob:publ.catiioniilee......
Technical Report AIS 46. Feb. 2024.
[13] Si Gao al. "Non--No-roiii profiled Mask Recovery: The Impact of Independent Component Analysis". In: Smart Card Research and Advanced Applications, 17th International Conference, CARDIS 2018, Montpellier, France, November 12- 14,1,, 2018, Revised Selected Papers. Ed. by Begüi Bilgin and Jean-Bernard Fischer. Vol. 11389. Lecture Notes in Computer Science. Springer, URL.: htttp:://doi.ooo org/
2018, pp. 51-64. DOL: 10. 1007/977-330 0-00--333000-0 15462-2442222 2_4
1007/978-3 030-0000000 15462-42222 2_4
[14] Atticus Geiger et al. "Causal Abstractions of Neural Networks". In: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual. Ed. by Marc' Aurelio Ranzato wwwrris.......... cc/ paper/2021 hash/ h213://4522222144444000777222220 htw.s//ppp...000--bborrrrahahaa
al. 2021, pp. 0574-95866 URL: https httpps::::/oocoeiinggg

---

[15] Suvadeep Hajra, Siddhartha Chowdhury, and Debdeep Mukhopadhyay. "EstraNet: An Efficient Shift- Transformer Network for
Analysis". In: IACR Trans. Cryptogr Hardw. Embed. Syst. 2024.1 1 (2024), pp. 336-374. DOI: 10. 46586/TCHES V2024. 316336
374 URL: https: http::::/dd/ooo.oo./...//o org/ 10. 46586/tches v2024. i1. 336
[16] Suvadeep Hajra et al. "TransNet: Shift Invariant Transformer Network for Side Channel Analysis". In: Progress in Cryptology AFRICACRYPT 2022: 13th International Conference on Cryptology in Africa, AFRICACRYPT 2022, Fes, Morocco, July 18-20, 2022, Proceedings. Ed. by Lejla Batina and Joan Daemen. Vol. 13503. Lecture Notes in Computer Science. Springer Nature Switzerland, 2022, pp. 371-396. DOI: 10.
URI.: http::https///oi..... org/ 10. 1100/978-3--00 031 1743-11733-9111
[17] Stefan Heimersheim and Neel Nanda. "How to use and interpret activation patching". In:
CoRR 15255 (2024). DOT: 10. 48550/ARXIV 2404 15255
arXiv: 2404. 15255
URL:: httpthttsp//oooioooo ottp::://doi.org/// 10. 118650/arXi:1 2404 15255
[18] Benjamin Hettwer, Stefan Gehrer, and Tim Güneysu. "Deep Neural Network Attribution Methods for Leakage Analysis and Symmetric Key Recovery". In: Selected Areas in Cryp- tography- - SAC 2019 26th International Conference, Waterloo, ON, Canada, August 12-16, 2019, Revised Selected Pupers. Ed. by Kenneth G. Paterson and Douglas Stebila. Vol. 1 11959. Lecture Notes in Computer Science. Springer, 2019, pp. 645-666. DOI: 10. 00007/978-23
030-38471-5_28 URL: httph:
ooooooog//1 /0.
[19] Yongbo Hu al. "Machine learning and side channel analysis CTF competition". In:
IACR Cryptol. ePrint Arch. (2019), p. 860. URL: http: iacr org/2019/860
[20] Yuval Ishai, Amit Sahai, and David A. Wagner. "Private Circuits: Securing Hardware against Probing Attacks". In: Advances in Cryptology CRYPTO 2003, 23rd Annual International Cryptology Conference, Santa Barbara. California, USA, August 17-21, 2003, Proceedings. Ed. by Dan Boneh. Vol. 2729. Lecture Notes in Computer Science Springer, 2003, pp. 463- org/ 1007/978
481. DOI: 10. 1007 3-49778-------1------ 4-540-45146-44
27
URL:
[21] Akira Ito, Rei Ueno, and Naofumi Homma. "On the Success Rate of Side-Channel Attacks on Masked Implementations: Information-TTeorrtiica Bounds and Their Practical Usage". In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS 2022, Los Angeles, CA, USA, November 7-11, 2022. Ed. by Heng Yin al. URL: https://doi.. org/
ACM, 2022, pp. 1531- DOI: 10. 35565/66666666665779999 10. 05.115/3354866 3560579
[22] Sengim Karayalcin, Marina Kreek, and Stjepan Picek. A Practical Tutorial on Deep Learning- based Side-channel Analysis. Cryptology ePrint Archive, Paper 2025/471. 2025. URT.: https
//eprint.iaet iacrr org/ 2025/4222141
[23] Paul C. Kocher, Joshua Jaffe, and Benjamin Jun. "Differential Power Analysis". In: Advances in Cryptology CRYPTO '99, 19th Annual International Cryptology Conference, Santa Barbara, California, USA, August 19, 1999, Proceedings. Ed. by Michael J. Wiener. Vol. 1666. Lecture Notes in Computer Science Springer, 1999, pp. 397. DOI: 10. 1007/33-5550
48405 1_25 URL: https: / /ooi org/10.. 1007 / 540 48404- 1_25
[24] Kenneth Li et al. "Emergent World Representations: Exploring Sequence Model Trained Synthetic Task" In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
URL: /tttts:://ppereviieeee
net net/forum?id=DedB0 TcZvT
[25] Houssem Maghrebi, Thibault Portigliatti, and Emmanuel Prouff. "Breaking Cryptographic Implementations Using Deep Learning Techniques". In: Security, Privacy, and Applied Cryp- tography Engineering- 6th International Conference, SPACE 2016, Hyderabad, India, Decem- ber 14-18, 2016, Proceedings. Ed. by Claude Carlet, M. Anwar Hasan, and Vishal Saraswat. Vol. 10076. Lecture Notes in Computer Science. Springer, 2016, pp. 3-26. DOT: 10. 1987/978- 00--444455----1
3-919-44 3-119-94445--- 6_1 URL:|nttps:// /toi wtw.://.000/000.00000000000000
[26] Loie Masure, Cécile Dumas, and Emmanuel Prouff. "Gradient Visualization for General Characterization in Profiling Attacks". In: Constructive Side-Channer Analysis and Secure Design 10th International Workshop, COSADE 2019, Darmstudt, Germany, April 3-5, 2019, Proceedings. Ed. by Ilia Polian and Mare Stöttinger. Vol. 11421. Lecture Notes in Computer
Science Springer. 2019, pp. 145-167, DOL: 10. 1007 / 978 - 3- 030 16350 - 1 1_9 URL: 1007 030- 16350-0111661199
12

---

[27] Loic Masure et al. "Don't 't Learn What You Already Know Scheme-Award Modeling for Profiling Side- Channel Analysis against Masking". In: IACR Trans. Cryptogr: Hardw. Embed.
Sys. 2023.1 1 (2023), pp. 32-59. DOr: 10. 46586/ (/086//TCHES. V2023. 11. 32- 59 URL: https:
//doi. , org/10. 10. 46584/ tches.chesseeeeeeeeee v2023. i1 59)
[28] Eric J. Michaud et al. "The Quantization Model of Neural Scaling". In: Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing
Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Ed. by Alice
Oh al. 2023. URL: http : papers nips cc/ hwpe/paper.... files paper 2023 hash/ html
[29] Neel Nanda, Andrew Lee, and Martin Wattenberg. "Emergent Linear Representations in World Models of Self-Suue Sequence Models". In: Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, BluckboxNLP@@ EMNLP 2023, Singapore, December 7, 2023. Ed. by Yonatan Belinkov et al. Association for Computational Linguistics, 2023, pp. 16-30. DOT: 10. 18653/V1/2023 BLACKBOXNLP-1- 1.
URL: https
//doi.org/// 10. 11653/11/20000 2tt.:////..........oooo...m..aa.
[30] Neel Nanda et al. "Progress measures for grokking via mechanistic interpretabillty". In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023. URL: htttts::////t httt::://operreviig/ net forum ? id= 9XFSbDPmdW
[31] Chris Olah. Mechanistic Interrriblttyt.p,ttiliilltttttt Variables, and the Importance of Interpretable Bases. June 27, 2022. URL: https:///ww/ essay (visited on 01/21/2025).
[32]
Chris Olah et al. "Zoom in: An introduction to circuits". In: Distill 5.3 (2020), e00024-001.
[33]
Catherine Olsson et al. "In-context Learning and Induction Heads". In: Transformer Circuits circuits pub / 2022/ context html
Thread (2022). URL: h/ /
learning and- pedductiio--eedd/innne
[34]
Guilherme Perin, Lichao Wu, and Stjepan Picek. "Exploring Feature Selection Scenarios for Deep Learning-based Side-channel Analysis". In: IACR Trans. Cryptogr. Hardw, Embed.
Syst. 2022.4 (2022), pp. 828-861. DOI: 10. 46586/ / TCHES. V2022. 14. 828 861 URL:
https://don org//o// 10. 46586/tches. www.2.2222222222 81448228-6611
[35] Guilherme Perin et al. I Know What Your Layers Did: Layer-wise Explainability of Deep
Learning Side-channel Analysis. Cryptology ePrint Archive, Paper 2022/1087 2022. URL: https://eprints iacr- org/2022/10000
[36] Stjepan Picek et al. "SoK: Deep Learning-based Physical Side-channel Analysis". In: ACM Comput. Surv. 55.11 (2023), 227:1-227:35. DOT: 10. 1145/
URL: http:http:s:/et //doi
org/10. 1145/ 3569577
[37] Alethea Power et al. "Grokking: Generalization Beyond Overfitting on Small Algorithmic
Datasets", In: CoRR abs/2201.02117 (2022). arXiv: 2201 02177
URL: https: //axivv
org/ /t////b//...0.00 02177
[38] Mathieu Renauld et al. "A Formal Study of Power Variability Issues and Side-Channel Attacks for Nanoscale Devices". In: Advances in Cryptology EUROCRYPT 2011 -0tth Annual International Conference on the Theory and Applications of Cryptographic Techniques, Tallinn, Estonia, May 15-19, 2011. Proceedings. Ed. by Kenneth G. Paterson. Vol. 6632. Lecture Notes in Computer Science Springer, 201 pp. 109-128. DOL: 10.
URL: wtt.hhttts//oooo orgo 10. 1007/978-333 2442-24465-448
[39] Jorai Rijsdijk, LichaO Wu, and Guilherme Perin. "Reinforcement Learning- Based Design of Side-Channel Countermeasures" In: Security, Privacy, and Applied Cryptography Engineering 1Ith International Conference, SPACE 2021, Kolkata, India, December 10-13, 2021, Pro- ceedings. Ed. by Lejla Batina, Stjepan Picek, and Mainack Mondal. Vol. 13162. Lecture Notes in Computer Science. Springer, 2021, pp. 168-187, DOL: 10. wt..://01.10000710001110 10w.100079788300 900800000090999.0
95085-90080000 9__
URL: htl:hhttss//dd:1 htte:/aii.oggg///ooo/eee 10.g///0..10
[40] Thomas Roche. EUCLEAK Side Channel Attack the YubiKey 5 Series (Revealing and Breaking Infineon ECDSA Implementationn on the Way). 2024. URL: hhttps://ninjalleeee
13

---

[41] Thomas Roche et al. "A Side Journey To Titan". In: 30th USENIX Security Symposium
(USENIX Security 21). USENIX Association. Aug. 2021, pp. 231-248. ISBN: 178-::77----9911333
3. URL.
https: www. www.useen.... wasemi..org//ooerreeeeeeeeeeee httpa//wwwwa...eeeeeeeeeeeeeeeee
roche
[42] James B. Simon et al. "On the Stepwise Nature of Soff-Supevvied Learning.. In: International
Conference on Machine Learning. JCML 2023, 23-29 July 2023, Honolulu. Hawaii, USA. Ed. by Andreas Krause et al. Vol. 202. Proceedings of Machine Learning Research. PMLR, 2023, URL:: httphttsss//pprcceediiiee François-Xavee Xavier Standaert. Side-Chamner Analysis and Leakage- Resistance. Version 1.2, 2024.
pp. 31852-31876.
mlr wwwpppreasssa. wttpa///v2////mmm//2/ html
[43]
[44]
François-Xaan -avierr Standaert, Tal Malkin, and Moti Yung. "A Unified Framework for the Analysis of Side-Channe' Key Recovery Attacks". In: Advances in Cryptology EURRCRYPP 2009, 28th Annual International Conference on the Theory and Applications of Cryptographic Tech- niques, Cologne, Germany, April 26-30, 2009. Proceedings. Ed. by Antoine Joux. Vol. 5479. Lecture Notes in Computer Science. Springer, 2009, pp. 443-461, DOL: 10 1007/978-3 642-01001 9_ 26 URL: https: :/tp:::://///.. org/ 1007 978 3- 642- 0011-42220001992220
[45] Benjamin Timon. "Non-Profilee Deep Leauminnn-aaseeee
attacks with Sensitivity Analysis". In: IACR Trans. Cryptogr Hardw. Embed. Syst. 2019.2 (2019), pp. 107-131. DOI:
10. 1 TCHES. V2019 I2. 107 131
URL: https: / org/ 10. 205154/tchee
v2019. 12..122.100 107-131
[46]
Daan van der Valk, Stjepan Picek, and Shivam Bhasin. "Kilroy Was Here: The First Step To- wards Explainability of Neural Networks in Profiled Side-Channel Analysis". In: Constructive Aide-Chaneel Analysis and Secure Design 11th International Workshop, COSADE 2020, Lugano, Switzerland, April 1-3, 2020, Revised Selected Papers. Ed. by Guido Marco Bertoni and Francesco Regazzoni. Vol. 12244. Lecture Notes in Computer Science. Springer, 2020, URL: htttp:://dae http:////oi..0r11.000111
pp. 175-199. DOr: 978-3-030-68773-11
[47] Aurélien Vasselle, Hugues Thicbeauld, and Philippe Maurine. "Spatial dependency analysis to extract information from side-channel mixtures: extended version". In: J. Cryptogr: Eng. 13.4 (2023), pp. 409-425, DOI: 10. -0000//013300002000000000 1007/s13389- 000-33399-22---00 022-0899--2220000--99
URL: http:tttt///...rr////
[48]
Kevin Ro Wang et al. "Interpretability in the Wild: Circuit for Indirect Object Identification in GPT-2 Small". In: The Eleventh International Conference on Learning Representattion, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL: htttps:/ net/forum? derrummeA
[49]
Lichao Wu et al. "Ablation Analysis for Multi-Device Deep Learning -easeing-BBaedd Physical Side- Channel Analysis". In: JEEE Trans. Dependable Secur: Comput. 21.3 (2024), pp. 1331-1341. DOI: 10. 1109/ TDSC. 2023. 3278857
URL: battps://ott org / 10. 1109 TDSC 2023
3278857
[50]
Lichao Wu al. "Label Correlation in Deep Learning- Based Side-Channde Analysis". In: IEEE Trans. Inf. Forensics Secur: 18 (2023), pp. 3849 3861. DOI: 10. 1w1111199IISSS 2023FF..2222.....
2023. 2 3287728
URL: https: : //t:///oiioo org. 100g//10//0 10011119/TITT
2023. 32877288
[51] Trevor Yap. Stjepan Picek, and Shivam Bhasin. "OccPols: Points of Interest Based on Neural Network's 's Key Recovery in Aiee-Channel Analysis Through Occlusion". In: Progress in
Cryptology INDOCRYPT 2024 25th International Conference on Cryptology in India, Chennai, India, December 18-21, 2024, Proceedings, Part Ed. by Sourav Mukhopadhyay and Pantelimon Stanica, Vol. 15496. Lecture Notes in Computer Science. Springer, 2024, pp. 3- 28. DOI: 10. 100700079777 3 031 80318-011110 6__
URL: https: :/doi: org/ 10 0007/977773
031-80311-6_1
[52] Trevor Yap et al. "Peek into the Black-Box: Interpretable Neural Network using SAT Equations in Side-Channel Analysis". In: JACR Trans. Cryptogr: Hardw. Embed. Syst. 2023.2 (2023), pp. 23. DOI: 10 46586/ TCHES. V2023. I2. 24 URL:: https: /odii. org/
46586/tches.v200 v2023.12. i..24.33.
[53] Kota Yoshida, Sengim Karayalcin, and Stjepan Picek. "Can KANs Do It? Toward Interpretable Deep Learning-based Side-channel Analysis". In: JACR Cryptol. ePrint Arch. (2024), p. 1570. URI.: hhttps://ept /ettiitt........ iacr.
wwc...rg//...................... org/ 2erg/2024/1500
14

---

[54]
Gabriel Zaid et al. "Conditional Variational AutoEncoder based on Stochastic Attacks". In: IACR Trans. Cryptogr. Hardw. Embed. Syst. 2023.2 (2023), pp. 310-357, DOI: 10. 46586/
TCHES. V2023. I2. 310-357
URI: URI: /sto:s///ooo.org///4446666/eeee v2023. 210
357
[55]
Gabriel Zaid et al. "Methodology for Efficient CNN Architectures in Profiling Attacks". In:
IACR Transactions Cryptographic Hardware and Embedded Systems 2020.1 (Nov. 2019). URL: hattts://tcoee iacr org/
pp. 1-36. DOL: 10. 13154/ v2020 i1 1-36
index. phik pttp:/CES///pppppppppppttttttttt aetaaaiiicc httte///rrtccceee///e//////
[56] Ziqian Zhong et al. "The Clock and the Pizza: Two Stories in Mechanistic Explanation
of Neural Networks". In: Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA.
USA, December 10 16, 2023, Ed. by Alice Oh et al. 2023. URL: bttp: /papers nips. cc/ 2023 hash http:////0000....00//0////000000 Absttaat Abstract. Conference htm
paper_ files/
15

---

Epoch 50
Epoch 100
Ie-6
1c-6
1ec- Epoch
Je- Epoch
10 Spoch
Je Me-sEpochhhh 25
1.0
1.5
1.0
Medan
0.5
2
0.5
0.0
10000 Time Samples
10000 Time Samples
0.0
0.0
1000 Time Samples
1000
2000
Time Samp 0S
Time Samoles
Time Samplcs
(a) CHES CTF
(b) ESHARD
(c) ASCADr
Figure 8: Gradient Visualization Example of MLP models on three datasets.
A Extended Related Work
The first main direction for interpreting what leakage neural networks exploit was the input visual- ization approach. Here, input attribution techniques from the vision domain were adopted to show regions of the traces that influence network predictions [18). Several input attribution methods have been investigated, where examples include gradient- based visualizations in (26), weight-basee visualizations in [55], and occlusion- beeuusion-based approaches (5T). In Figure 8 we show example of gradient-based visualization for the ML.ss used in this work. As we can see, the visualizations do highlight similar similar regions
found using the secret-shares extracted. However, from only these plots is difficult to extract meaningful insights without access to secret-share randomness. In the works proposing these explainability approaches, visualization is often compared with ground-truth leakage information using SNR for known secret-shares (as we do using extracted shares), which can then highlight the leakage from which shares the network exploiting 26 18] However, this assumes knowledge of masking randomness.
More recently, the internals of networks have also been analyzed. Van der Valk et al. [46] compared networks trained on different side-channel datasets, showing that these networks are often very different from each other. Wu al. [49] used ablations evaluate the roles of specific layers in defeating certain countermeasures. In Perin et al. (35). the probes are trained at each layer for several (er)velevant secret shares, showing that information bottleneck forms, resulting in the compression of irrelevant share information while relevant values are maintained.
Finally, there are two works that propose using more interpretable architectures for DLSCA. Yap et al. used a truth table convolutional network to find SAT-equations for important points in the trace (52). Yoshida et al. used Hallmogorov-Amoll networks [53). These works show nice interpretations of network behavior, but these interpretations are only shown on simulations or selected informative features. The interpretability benefits of using these architectures trained full-length traces still an open question, as the additional (non-informative) points result in less 'clean' interpretations. These more interpretable architectures also come an additional computational cost, while resulting in worse attacks, limiting their relevance as practical replacements for standard MLPs and CNNs.
Overall, these works provide limited insights into how certain countermeasures defeated. Al- though some approaches can show which shares are (not) exploited, these require access to masking randomness during training, which is not always possible. Our work provides an approach that allows us to interpret standard neural network architectures, while minimizing the necessary assumptions beyond network that can break target. Besides this, approach is the only approach that allows for the extraction of secret shares.
B Datasets
We utilize publicly available datasets commonly used in SCA literature for benchmarking. These datasets implement AES-128 with Boolean masking protection. The attack set consists 10000 traces for each dataset.
CHES CTF 2018 [19 II
consists of power consumption measurements from an AES- implemen- tation running on ARM Cortex-M4 (32 bits). CHES CTF 2018 raw traces contain 650 000 sample
Referred CHES CTF.
16

---

we take subset of 150 points corresponding to the initial setup and the first AES round and resample to 15 000 samples per trace. The profiling set has 30 000 traces.
points per trace. Following 34
ESHARID-AES12 477 consists of EM measurements from a software-masked AES- 128 imple- mentation running
ARM Cortex-M4 device. The AES implementation is protected with first-order Boolean masking scheme and shuffling of the S-box operations. In this work, we consider trimmed version of the dataset that publicly available and includes the processing of the masks and all S-box operations in the first encryption round without shufffing. This dataset contains 000 measurements with 90000 traces for the profiling set.
ASCAD [2] measures EM emissions from an AES- 128 implementation on AVR RISC (8 bits). We use the version with the variable key in the profiling set. The traces are 250 sample points per trace. Following (35), we take a window of 20 000 points, which are resampled to 2 200 points. 200 000 traces used for profiling-
C Models and Training
The used models MLPs from 35 where model configurations found through random hyperparameter search for ESHARD and ASCAD. Note that, as the ESHARD model performed well directly for CHES __T,, did not do further optimizations.
The model for CHES CHES_CTF and ESHARD is a 4-layer MLP with 40 neurons each layer with he uniform weight initialization. We use relu activations. We use the Adam optimizer with learning rate of 0.0025 and L1 regularization set to 0,000075. The batch size is 400, and we train for 200 epochs for CHES CTF and 100 for ESHARD.
For ASCAD, the model is 6-layer MI.P with 100 neurons in each layer with random_ random_uniform weight initialization. We use the Adam optimizer with learning rate of 0.0005. We use elu activations, and we again train for 100 epochs with batch size of 400.
D Computational Load
Training these models takes under an hour on desktop workstation with 64GB RAM and an NVIDIA 4080 GPU. Producing Pl/probe plots per epoch takes a similar time (mainly because of reloading models for every epoch from disk). All other experiments take negligible compute (seconds, sometimes minutes).
E ESHARD Results
that for ESHARD, only one phase transition occurs for the test set. At epoch 4, the perceived information becomes positive, and the models start to generalize. We note that the main distinction here is again the high-low HWs, similar to the first step in CHES_ CTF. Further analyses are analogous to CHES_ _TF, although the model here can never distinguish between even and odd HWs.
In Figure 9
CHES_CTE
Perceived Information
10-1
Train
10-3
25
50
75
100
Epochs
Figure 9: Evolution Perceived Information for train and test traces of the ESHARD dataset.
PeRerrred to as ESHARD.
sattps://gitaie httpt//..../essrr///eceeeeeeeeee
17

---

In the two rightmost plots in Figure 10
we showcase distributions of the concrete intermediate values the models use. The models are clearly mapping the HWs of secret shares onto specific features.
Logits at Epoch
PCs at Epoch
I' at Epoch 5
10
HW
Logit Value
HW
HW
HW
PCT
0.2
HW
0-
HW
HW
HW
0.0
-10
0
-2
10
55
10
Output
Logits at Epoch
PCs at Epoch
Sbox @ Im at Epoch
0.6
Logit Value
0.2
0.0
-10
-10
0
-10
-5
10
-10
-5
5
10
Output
Figure 10: Logit analysis (first column) and activation analysis (second column) from models at epoch (top row) and epoch 5 (bottom row). The legend is shared among all figures. We also include the PC embeddings for the actual mask of secret shares epoch 5 (third column). The masks extract are in Figure
E.1 Activation Patching
We can do similar patching experiments as done for CHES_CTE in Section 4.1 As the high-low HWs are not the diagonals the PCs epoch we rotate the PC coordinates before patching and then rotate them back before continuing inference to align PCs more with the expected masks. The results we see in Figure IT closely match the actual distributions of secret share HWs as seen in the rightmost plots of Figure 10
SNR for actual shares
SNR for retrieved shares
HWs of
HWs of Sbox
1.25
15
15
1.00
Sbox @
Sbox
10
10
0.75
5
PC
0.50
0
-5
0.00
-10
-10
1000 Trace Samples
1000
-10
D
10
-10
10
Trace Samples
PC
PC
Figure 11: SNR plot and PC component distributions for mask values using patching experiments in ESHARD.
F ASCAD Patching results
As the leakage model for ASCAD is more complicated than the HW models, patching becomes more difficult. First, we train probes on the final layer classify the input and output bits separately. We can directly measure the effects only the input or the output. Patching the input shares in the PC components layer 2, which we show Figure 7 does not work. Then, we find qualitatively similar structure in PCI and PC2 in layer and patch there.
18

---

For the patches on the output shares, we set the first two components, which are related to the input shares, isolate the effects of the patched components. For both experiments, we again rotate the two components by multiplying them with rotation matrix to simplify the patches.
Pi
20
00
10
10
5.0
A
11
2.5
-10
0.0
1000
1500
2000
-20
-10
10
Tace Samples
PC2
PC2
SNR for retrieved shares
Bit 0 & of
Bit 0 & 1 of Sbox 0 r
An
00
Pi B 1/
10
2-
1
-10
-10
Samplas
Figure 12: SNR plot and PC component distributions for mask values using patching experiments in ASCAD.
In Figure 12 we can see that the patches work reasonably well. Clearly, intervening on the found components has some causal effects. Furthermore, as we can see in the SNR plots, the patched outputs of the models are tied to the mask values we expect. However, the SNR values are significantly lower than those for the actual shares, and the r; and S-box O r, shares only result in two or three classes, respectively, where we expect four. Additionally, the reversed shares do not combine to the correct label for the input bits, indicating that while the mask values we retrieve are reasonable clustering. further post-processing is necessary retrieve the actual values.
As aim keep the experiments (somewhat) aligned across all targets, do not tailor the patching methods further for ASCAD. The current experiments show we can intervene in the structures and observe effects on the (probe) outputs. However, refining mask extraction methods in models with more complicated interactions is an interesting direction for future work. We provide further analysis validate model predictions based on the four bits in Appendix I.I
G ASCAD CNN
To whether analyses are feasible for CNNs, consider the CNN used for the ASCAD target we see that the bits that show significantly increased accuracies are the same
from [35. In Figure 13
1.0
Perceived Information
100
Accuracy
0.8
10-1
0.6
St
10-2
Train Test
Sock Bits
0.4
Bit
Bits
50 Epochs
100
50
100
Epochs
Figure 13: Evolution of Perceived Information and probe accuracies for bits during training for the ASCAD dataset for CNN.
19

---

8.8k
p.8k
00
PC
SBox
Figure 14: PC structures for ASCAD CNN after the third convolutional block.
in Figure 6 However, the performance increase (after initial, smaller jumps) is more gradual. This, and the model having significantly more layers to check, results in it being somewhat more difficult find the structures.
As the performance increases steadily, we focus our analysis on the final model at epoch 100 and show these in Figure [4] We note that in this case, shares are still in the first 4 PCs, but the input shares are not the first PCs, but PC 0 and 2 (output shares PC and 3).
H HW Recombination CHES_CTE and ESHARD
Next, we discuss how mask recombination can be done algorithmically for the HW leakage model.
H.1 High-Low HW Distinguishing
For the CHES_CTT and ESHARD targets, we notice that after the first performance increase (for some cases), high and low HWs can be differentiated. These are byte-based implementations protected with Boolean masking with order 2, i.e., the sensitive value æ = 2 @ D2 (# being bitwise xor). When, based prior experience working with these targets, we then choose model the leakage (and therefore the presumed features of the model) as the L = HW x we can consider modeling how occurrences of different classes Y HW(x)) (x) look. In Table I low HWs tend to be the diagonal from top-left to bottom-rige rrghtt, while high HWs tend to on the other diagonal. This (low HWs one diagonal while high HWs the other) matches the PC embeddings for both models in Figure 10 and Figure
H.2 Even-Odd HW Distinguishing CHES CTF
For CHES CTF, we further see that the even and odd HW target classes can be distinguished after the second performance jump. From Table [ it is clear that if the HWs of each secret share can be retrieved accurately enough, there should be a clear separation between even and odd HWs for the resulting point. Indeed, for any point HW
HW(z)) where = + 22 we have that (x1) + HW(x.) (ar2 mod 2 = HW(x) mod This can seen in Table I for two 8-bit shares, but the ability to distinguish the parity of HW (w:) holds for general higher masking orders d (21
I Bitwise Leakage ASCAD
As show in Figure the features the model learns for ASCAD the two least significant bits of both p. ① k; and S box jp, ① k.]. We first note that the way the first two bits S- box(p @ k,] relate to model labels (S box(pp e k.]) is straightforward: if bits 0 and of S-boo/,, @ ks] are 00,
We note that we knew this a priori for ESHARD and it was strongly suspected for THEESCTT However, it also common leakage model in practice.
20

---

(x)
Matrices counting occurrences of HW (a1).
HWW (82)
s.t.
S1
1 from 0-9.
0
0
56
0
56
28
168
0
0
0
0
168
56
280
0
0
0
0
HW =
0
0
0
280
70
280
0
HW =
0
0
280
56
168
0
0
168
28
56
0
0
0
56
8
LO
0
0
0
0
0
0
28
56
0
56
168
168
280
0
28
168
336
840
420
560
0
0
56
168
8410
840
1680
560
560
0
HW =
0
280
420
1680
1120
1680
420
280
HW 33
0
0
560
560
1680
840
840
168
0
0
0
560
420
840
336
168
28
O
D
0
0
280
168
168
56
O
D
0
56
28
O
0
280
280
0
0
0
420
0
1120
420
0
0
O
280
0
1680
1680
0
280
HW 4
1120
2520
1120
70
280
0
680
1680
280
O
420
0
1220
420
0
O
0
280
280
0
0
0
0
70
0
0
28
0
0
0
0
280
168
168
56
0
0
0
560
420
840
336
168
28
0
0
560
560
1680
840
840
168
56
HW 5
280
420
1680
1120
1680
420
280
HW = 6
56
168
840
840
1680
560
560
28
168
336
840
420
560
0
0
56
168
168
280
0
0
28
56
0
O
0
0
0
0
0
0
0
56
8
8
0
0
0
0
168
28
56
0
0
0
280
56
168
HW 7 HW 8
0
0
280
280
0
0
168
56
280
0
0
0
0
0
56
28
168
0
0
0
11
8
56
0
0
0
Table 1: Occurrences of HWs for two 8-bit shares for each of the nine output classes, cell i, in each matrix corresponds to (WW(s1. HW ($2). For the percentage of examples in practice, these values should be divided by 256 As even and odd HW(x (I)) never occur in the same place, we show two HWs in one matrix. Note that any red value (resp. black) is a zero in black (resp. red).
then these correspond to predicting each label mod 4 = 0. For bits 0 and of @ k; we can use the inverse of the S-bo If we define y/ S-Box -1 [y] then if bit 0 and of ps @ kū are 00, we predict y s.t. y mod 4
Combining these, we can divide the output classes into 16 clusters corresponding model predictions.
Practically, we define the outputs that belong to the 16 classes as Y; {vlv mod 4 A y/ =
L mod 4). Here, we set to be a concatenation of bit and of p& @ k; and then bit I and 0 of S S-box(p: ki.)
15e AES b-box is bijective, which simplifies this, but the analysis also works for surjective functions by taking the pre-image.
21

---

Median Logits
1=0
i=1
i=2
i=3
0.050
Logit
0.025
0.000
i=4
i=5
i=6
i=7
0.050
Logit
0.025
0.000
i=8
i=9
1=10
i=11
0.050
Logit
0.025
0.000
1=12
1=13
1414
1=15
0.050
Logit
0.025
0.000
200
200
200
0
200
Figure 15: Median logits for traces belonging to varying Y; classes. The red dots indicate indices in the respective Y,.
We can then train a linear probe on the activations of the final layer to predict these 16 classes. we then transform the probe outputs to evenly distribute the predictions for its i'th output to the values Y,, we can measure the entropy between this resulting distribution and the model outputs. In summary, the probe accuracy is 0.64, and the PI between the probes' transformed distribution and the labels 2.47 vs. 2.58 for the actual model. The entropy (in bits) between probe outputs and model predictions is 0.27, indicating that most of the relevant behavior is explained by using the probe.
L.1 Logits For ASCAD with Classes
In Figure 15
we show the median logits for traces belonging to classes Y, As we can see, the logits corresponding to the expected points in Y; are always the main peaks.
Figure shows how logits change from epoch 12 to epoch 25. When we analyze using only S-box inputs, we see that the logit values are significantly higher before the accuracies for output bits increased. This is explained by the fact that each of these cases combines four plots (vertically) in different
Figure 15 Concretely, as for each trace Figure [6] we combine traces that belong classes of the output bits, we expect the logits for each index that belongs to + k mod = to only be high for 1/4 traces, resulting lower medians. Note that mean values do not show this same trend, as the increase in the Y, class compensated for this decrease.
To verify that the results in Figure 15 are not artifact of selecting traces, visualize the same analysis for bits and in Figure 17 Clearly, the output values are significantly lower than for correct bits, indicating that these bits (mostly) not being used by the model.
22

---

Median Logits for p mod =
i=0
i=1
i=2
0.0150
0.0125
0.0100
Logit
0.0075
0.0050
0.0025
0.0000
0.0150
0.0125
0.0100
Logit
0.0075
0.0050
0.0025
0.0000
200
200
0
200
0
200
Figure 16: Median logits for traces belonging to varying classes of LSBs of S-box input for epoch 12 (top) and 25 (bottom).
Median Logits for Y, with bits 2,3
i=0
i=1
i=2
1=3
0.050
Logit
0.025
0.000
i=4
i=5
i=6
i=7
0.050
Logit
0.025
0.000
i=8
i=9
i=10
i=11
0.050
Logit
0.025
0.000
i=12
i=13
i=14
i=15
0.050
Logit
0.025
0.000
0
200
200
0
200
0
200
Figure 17: Median logits for traces belonging varying Y, for bits 2 and The red dots indicate indices in the respective Y,.
23

---

Median Logits for Y, with bits 0,1
i=2
0.05
Logit
0.00
i=4
i=5
i=6
7
0.05
Lagit
0.00
|=8
1=9
1=10
1=11
0.05
0.00
i=12
i=13
i= 14
15
0.05
Logit
0.00
200
200
200
200
Figure 18: Median logits for traces belonging varying Y, classes for CNN model. The red dots indicate indices the respective Y,
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are supported by experiments against several targets. More speculative claims about implications and generalization discussed.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations when generalizing to different models and targets are discussed in Section 5) The setting and security assumptions are discussed in the introduction and more concretely the start of Section 3
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ycs]
Justification: The (training recipes for) models are described Appendix C and datasets used in the paper public. The methods and analyses are described in the paper. Code is also provided with scripts to simplify reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
25

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is included, there are scripts to download and extract the used datasets from raw traces. We also provided links to download model checkpoints used and extracted datasets to simplify reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.cc/public/gu.dee http://gguuieee/eeeuuubbiiiiiiii wwwpannisssoooooooooico for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
26

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The (training recipes for) models are described in Appendix C (and based on results in (35) and the train/test splits are standard for the used datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments are reverse engineering specific models from previous works..
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The amount of compute and resources is described in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
27

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /thh ctwcieeeeeeeeeeeeennnn
Answer: [Yes]
Justification: The paper works on interpreting models DLSCA. Techniques to extract mask values could potentially be used to improve attacks and broader improvements to DLSCA can improve practical attacks. However, the main purpose is to improve evaluation procedures which should improve the (confidence in the) security of implementations against such attacks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper works advancing deep learning-basee SCA which is dual-use but mainly used in evaluation procedures for testing whether implementations are sufficiently secure. The impacts of this specific work on the SCA domain are discussed in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
28

---

The paper focuses interpreting models that break specific public implementa- tions from previous works. Therefore materials in this work pose no such risks. Guidelines:
Justification:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of the used datasets are publicly available and credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprswii www.wwe.eehd....... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper provides no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy lattps://neuripe http//rrrrp.ccccooofeeeeeeee//// for what should or should not be described.
30


Reviews:
[{"review_id": "nwa1eEQj6L", "reviewer": "Reviewer_8DGR", "summary": "This paper is about Side-channel analysis (SCA) for understanding deep learning (DL) networks. The paper elaborates that SCA can help to understand network learning and can be used to attack the DL networks. The paper used 4-layers MLP network (160 neuron) for the experiments and used Hamming Weight and changes in it to measure the understanding and activities in the network. The experiments used two datasets that then networks' perceived information and learning (accuracy) is measured and related to HW. The discussion mentioned that it is not results are not conclusive but their implication show that there is a possibility of proposed idea.", "questions": "Please look into Strengths and weaknesses mentioned above.", "limitations": "Experiments are limited. They need to clearly support the suggested contributions.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strengths\n- Idea is interesting and novel. \nWeaknesses. \n- article is not well-written.\n- References are not available using the DOIS such as [19],[20],[22],[24] (serious issue)\n- Many references include reports or opinions or archives which not published in peer-reviewed venues such as [4], [27],[37]. This makes the introduction weak.\n- Introduction as well as background are not clear about the contribution. On one hand, it is mentioning that SCA is used as a vulnerability of the DL architectures while on the other hand, it is the reason for explainability. The introduction needs to have clarity so readers can be led to clear contributions\n-  The abbreviation are used before explaining them such as HW\n- The justification of SCA is not clear too as there are no studies clearly mentioning that it can be used well.\n- Profiling attack require copy of the network. If network's copy is available then why SCA is needed as entire network can be studied and interpreted easily.\n- It is not clear how physical characteristics of the network will be accessed to deployed DL network\n- 4-layers MLP is not an optimal selection as state of the art network for experiments. Existing networks are far more complicated like transformers or MAMBA and it is not clear how these results can be extended to recent networks\n- On one hand Discussion clearly mentions that SCA is not conclusive on MLP while on the other hand it is projected as giving inside network information. This shows that there is not clear contribution of the study.\n- Experimental results do not clearly support the contributions\n- discussion is mostly about implications rather than the results", "was_revised": true, "final_justification": "Authors have replied to most of the limitations in the rebuttal. However, if authors embedded all the revisions mentioned in the rebuttal, then it will be a significant change to the original manuscript. I still think that authors need to look into more recent networks other than MLP and CNN.", "rebuttal": "In your summary, the claim is that our paper uses SCA for understanding DL networks, and here we want to emphasize that this is not the case. We use mechanistic interpretability methods and propose a procedure to conduct explainability analysis that aims to explain what DL networks learn when used to perform deep learning-based SCA (DLSCA) on cryptographic implementations. We are interested in understanding how the side-channel leakage of a running crypto system is exploited by the neural network to extract the secret information (key in this case).\nThe recent emergence of DLSCA has resulted in many papers (see [32]) and is becoming standardized for evaluations of cryptographic implementations required for certification (see [9]). Understanding DLSCA and what the models actually learn is therefore an important open question.\nWe discuss the main issues below. \n\nMinor issues:\n- We will add DOIs for the works where these are available. Thanks for pointing this out.\n- Abbreviations:  HW is Hamming weight. We will double-check all abbreviations. Thanks for pointing this out.\n\nWeaknesses:\n - Article is not well written: We will clarify long paragraphs as pointed out by reviewer CmhV. If you have any further actionable concerns about the writing, we are happy to address them.\n - Non-peer-reviewed references in introduction: transformer-circuits is where Anthropic posts many papers ([4,8,27] have 100s of citations) and sometimes provides independent reviews. [37] is meant to show real-world vulnerabilities found by evaluation labs in industry (which are not commonly reported in scientific publications). Other references are preprints or textbooks [40]. We acknowledge that this is not ideal, but as mechanistic interpretability and DLSCA are relatively new fields, this is unavoidable.\n - `Introduction as well as background are not clear about the contribution. On one hand, it is mentioning that SCA is used as a vulnerability of the DL architectures while on the other hand, it is the reason for explainability. The introduction needs to have clarity so readers can be led to clear contributions': We do not mention SCA is used as a vulnerability of DL architectures in the paper, as we are focused on cryptographic implementations. The explainability of DLSCA is a key aspect of performing evaluations (see [9,32]). \n - SCA cannot be used well: This seems to say we do not justify the use of SCA for understanding DL networks, so again, we clarify that this is not what we are doing. We use mechanistic interpretability to understand what networks are learning in the DLSCA use case on crypto systems. However, we argue that we do justify why we investigate explainability for DLSCA. As we discussed in the introduction, deep learning-based SCA has emerged as (one of) the main tools to perform SCA evaluations in industry labs and is becoming standardized [9] (in addition to real-world attacks [37,38]). In performing the evaluations, explainability of DLSCA is a key aspect (see [9,32]). \n - Profiling attacks are a well-established class of attacks for evaluating cryptographic implementations under worst-case assumptions. Profiling SCA requires a clone device running the same cryptographic implementation, not a copy of a neural network.\n - Architectures were taken directly from [31]. Commonly, simple MLPs and CNNs work well in DLSCA; therefore, our work considers relevant networks for DLSCA (note that similar architectures achieve optimal attacks in related works).  Extending the work to other architectures is an interesting direction for future work.\n - `On one hand, the Discussion clearly mentions that SCA is not conclusive on MLP while on the other hand, it is projected as giving inside network information. This shows that there is no clear contribution of the study.': We are not sure what this means. We would be grateful for further clarification on the reviewer's point here.\n - Experimental results do not clearly support the contributions: We respectfully disagree; we provide experiments with various MLPs and a CNN. We can extract masking randomness, indicating that we provide a better understanding of what leakage models exploit in several cases.\n - Discussion is mostly about implications rather than the results: We can expand the discussion on results, which, for now, is mostly contained in Section 4."}, {"review_id": "PKAikknYrM", "reviewer": "Reviewer_LCay", "summary": "This paper uses mechanistic interpretability to understand how deep learning models extract secrets in side-channel attacks. It reveals what leakages models rely on, recovers secret masks, and shows that such interpretability is feasible even in noisy, real-world settings, making security evaluation more transparent.", "questions": "1. Why are only logits and activations chosen for analysis in this work? A clear motivating experiment is needed to show that only logits and activations are enough to analyse to understand the neural networks trained for SCA.\n\n2. It is unclear whether the proposed analysis is effective compared to prior work. A comparison table is required to understand the effectiveness of the proposed analysis.\n\n3. Why is the analysis shown only for one model [31]? To better understand the effectiveness , the proposed analysis should be done on atleast one more model (For eg: model from [32])", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The paper works on an important problem of the interpretability of neural networks trained for SCA.\n\n* By applying mechanistic interpretability to deep learning-based SCA models, the work makes an important and novel contribution in bridging the gap between high-performing yet opaque models and the need for transparency in security evaluations.\n\n* The authors make a compelling case for transitioning from black-box performance benchmarking to white-box understanding of model behavior. This shift is valuable for designing more robust defenses.\n\n* The proposed analysis is evaluated on diverse datasets.\n\nWeaknesses:\n\n* The paper analyses the logits and activations of the neural network models trained for SCA. It is unclear why only these two components are chosen for analysis. A detailed justification/motivation for the same is required.\n\n* It is important to compare with prior works, such as methods that focus on input visualization [22,14] and masking randomness [48,31]. This will establish the effectiveness of the proposed analysis.\n\n* The proposed analysis is evaluated using only one model [31]. It is important to do the same analysis on other models trained for SCA, such as [32].\n\n* While the paper addresses an important topic, it lacks a clear novel theoretical or methodological contribution. The analysis primarily builds on known observations (e.g., [49]) and does not introduce new techniques or frameworks, which limits its overall impact.\n\n* The code and pretrained models do not appear to be available at the time of submission. Consequently, it is not currently feasible to verify the reproducibility of the reported results. Providing access to the implementation, along with detailed setup and usage instructions, would substantially improve the transparency, accessibility, and impact of this work for the broader research community.", "was_revised": true, "final_justification": "Most of my concerns are addressed in the rebuttal and I have increased my score accordingly. However, it is still not clear from Sec.4 why only logits and activations are selected for the analysis. Maybe an analysis/experiment that show that logits and activation are the only features that in a network can be used for this analysis and other features are not relevant.", "rebuttal": "W1Q1: `Why are only logits and activations chosen for analysis in this work? A clear motivating experiment is needed to show that only logits and activations are enough to analyze and understand the neural networks trained for SCA.' \nWe showcase several examples in Section 4 where we can understand what features networks are learning during sharp performance increases. This then enables us to retrieve the secret shares from network activations using activation patching, a first in DLSCA. Overall, we believe that the experiments in Section 4 clearly demonstrate the effectiveness of the approach for understanding (and validating) network behavior. \n\nBesides this, we note that analyzing activations is standard in mechanistic interpretability, as the goal of our work is to understand what is happening inside the models (i.e., what the model is representing in the activations). MI techniques that work with inputs are not possible, as we do not know a priori what features are present in the input.\n\nW2Q2: As the methods for interpretability are qualitative and make different assumptions and/or have different goals, it is difficult to directly compare performance in a table. Input visualization only focuses on inputs and does not split the input attribution to individual secret shares (which we can achieve with SNRs for retrieved shares in Figure 5 (left)). Methods using masking randomness are not comparable, as we do not assume to know this randomness (which is common in practice, see [23]). We can add extended related work with some examples of input visualization to the appendix to provide a more elaborate discussion.\n\nW3Q3: We evaluate using 3 different models from [31] (obtained using random search, which is standard for these targets, see [32]) to improve upon the interpretability from [31] with fewer assumptions (i.e., unknown masking randomness) (CNN in appendix). We note that running searches for model architectures/hyperparameters for new targets is relatively standard in DLSCA. Adding more models from different works is possible, but as the models we use provide state-of-the-art performance for the current targets, other models will (presumably) learn the same features ([32] is a SoK paper and does not contain specific models).\n\nW4 limited novelty: We note that the application of MI techniques to different domains is very limited. In addition, as we do not assume knowledge of input features (i.e., masking randomness), we need to construct features from output logits, which requires adaptation of MI techniques. Furthermore, the setup that we provide for analyzing networks in DLSCA using mechanistic interpretability, the connection of the `initial plateau' [23] to feature emergence as in [26,49], and the application of activation patching in intermediate layers to retrieve masks are novel.\n\nW5: We provide the link to an anonymous repository with code, extracted datasets (and scripts to generate those from raw measurements from original repositories), and pretrained model checkpoints right after summarizing our contributions and before the background section, page 3, line 80-81). We can move the link to the abstract so that it is visible on the first page if that would help."}, {"review_id": "DMt2oC4UQ7", "reviewer": "Reviewer_CmhV", "summary": "This paper applies techniques from mechanistic interpretability (MI) to studying what features deep learning (DL) model exploit in side channel analysis (SCA). It examines features in models trained on three different implementations of AES 128 and uses these to trace back to causal model outputs. The result shows that there is some alignment among features learned by different models.", "questions": "- This is minor, but the title does not match the content of the paper very well. Why not simply call it \"Interpreting Features in Deep Learning-based Side-channel Analysis\"? It wasn't clear from the paper why the phrase \"you have to be realistic\" was included in the title. \n\n- Line 328: \"We show that specific structures can occur for different side-channel targets, indicating that building a library of common structures can be useful in analyzing future networks.\" The current work studies only SCA of AES128, which is a relatively straightforward algorithm. Do you think the methods that you develop here can apply to SCA of other cryptosystems (like CRYSTALS-KYBER https://eprint.iacr.org/2023/1084)?", "limitations": "- The authors could have given more consideration to how features discovered via MI could be leveraged for defense against SCA attacks. Why was this question not explored?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Investigation of an important and interesting problem: what features make DL models so successful at SCA? \n- Creative use of MI tools to study this problem.\n- Evaluation of techniques across several implementations of AES 128 and model types, indicating repeatable results. \n\nWeaknesses: \n- Although universal features are identified across the SCA models evaluated, no defense is proposed (although discussion suggests interpretability techniques could be helpful for designing defenses). \n- Only evaluates a specific cryptosystem (AES 128), leaving open the question of whether these techniques work on other, perhaps more complicated, systems. \n- Only one MI technique (activation analysis) is tested. It would be interesting to explore if other MI techniques provide different insights. \n- Some paragraphs are *very* long and wander, making multiple points. Tightening up the writing and creating paragraphs that address a specific point would improve paper quality.", "was_revised": true, "final_justification": "The rebuttal adequately addressed my concerns. My score remains the same.", "rebuttal": "W1: The analyzed results are mostly aimed at showcasing how to do post-hoc analysis of neural nets that break a target. In our view, understanding how networks exploit leakage is a prerequisite for designing efficient, targeted countermeasures. We can provide more discussion on different masking schemes that avoid the structures present in this work, which (intuitively) may be more difficult for NNs to learn (e.g., prime-field masking [CMM+23]). However, as there are (almost) no publicly available targets (and datasets) that implement such masking schemes, this will remain somewhat speculative and is an interesting direction for future work. \n\nW2Q2: We focus on AES targets as those are common in the related literature, and these provide representative benchmarks of masked implementations. Moreover, AES is a complex target compared to many newer algorithms, e.g., lightweight crypto. The concepts behind the attacks on post-quantum crypto (like Kyber) and symmetric crypto (like AES) are rather different, so one would need to adapt leakage models and consider different operations that leak.\n\nWe also note that how models extract physical leakage and combine secret shares should (broadly) be the same for targets masked with similar (Boolean) schemes. The models are being used to extract and recombine leakage from secret shares to improve predictions for a target value. As such, we expect that mask recombination should look similar for similar masking schemes across ciphers. We can extend the discussion on this.\n\nW3: We emphasize that analyzing activations (and activation patching) are very general approaches where the specifics we use in this paper are only one option (which works well for these cases). It is an interesting direction for future work to use/adapt some more automated techniques from recent MI literature on LLMs, although we note those approaches often assume some knowledge of what input features are relevant a priori, which corresponds to knowledge of masking randomness in SCA. This is not always possible in practice, even in evaluation settings (see [23]). \n\nW4: We will shorten paragraphs and improve the writing.\n\nQ1: The `You have to be realistic' was a reference to extending the work on toy models on group operations to real-world ML problems. We can remove it if it causes confusion.\n\n[CMM+23] Cassiers, G., Masure, L., Momin, C., Moos, T., & Standaert, F.-X. (2023). Prime-Field Masking in Hardware and its Soundness against Low-Noise SCA Attacks. IACR Transactions on Cryptographic Hardware and Embedded Systems, 2023(2), 482-518. https://doi.org/10.46586/tches.v2023.i2.482-518"}]

Author rebuttal:

Meta-review: The paper investigates how neural networks succeed at deep learning-based side-channel analysis (SCA). The authors apply mechanistic interpretability to models trained on cryptographic tasks. They observe that models learn in discrete jumps. These performance gains correspond to the emergence of clear, geometric structures within the network's hidden activations. From a machine learning perspective, the network spontaneously learns a disentangled representation of the hidden causal factors (secret shares) from extremely noisy, high-dimensional input. The authors use this insight to reverse-engineer the model's learned algorithm and successfully recover secret cryptographic masks.

Strengths. The paper applies mechanistic interpretability to a complex, real-world security problem. It provides a compelling case study of a simple MLP learning a multi-step algorithm from scratch. This work demonstrates how a network can discover and separate hidden variables. It is quite impressive that one could causally intervene on the system and recover secret values.

Weaknesses: presentation and limited scope. One reviewer raised significant concerns about the clarity of the writing and the quality of the references, with which AC agrees. The experimental validation is also constrained to the AES cryptosystem and relatively simple model architectures, which raises questions about generalisability.

Rebuttal & discussion. 8DGR criticised the writing quality. Others put more weight on the novelty and significance, trusting the authors that they'll revise the paper appropriately. The final consensus was that the paper's technical contribution is solid and important, provided that the authors undertake a significant revision to improve clarity. **AC strongly urges and requires that the paper go through the promised revision to improve the writing and presentation before the camera-ready.**

Why accept? The decision to accept is based on the novelty and significance of the findings. The paper offers a rare demonstration of successful reverse-engineering of a neural network's learning dynamics. The contribution is valuable both to the security and ML interpretability communities.
Decision: Accept (poster)