Beyond Slow Signs in High-fidelity Model Extraction

Authors: hanna  foerster
robert  mullins
ilia  shumailov
jamie  hayes
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:16:55

Abstract: Deep neural networks, costly to train and rich in intellectual property value, are
increasingly threatened by model extraction attacks that compromise their confiden-
tiality. Previous attacks have succeeded in reverse-engineering model parameters
up to a precision of float64 for models trained on random data with at most three
hidden layers using cryptanalytical techniques. However, the process was identified
to be very time consuming and not feasible for larger and deeper models trained on
standard benchmarks. Our study evaluates the feasibility of parameter extraction
methods of Carlini et al. [1] further enhanced by Canales-Martínez et al. [2] for
models trained on standard benchmarks. We introduce a unified codebase that
integrates previous methods and reveal that computational tools can significantly
influence performance. We develop further optimisations to the end-to-end attack
and improve the efficiency of extracting weight signs by up to 14.8 times com-
pared to former methods through the identification of easier and harder to extract
neurons. Contrary to prior assumptions, we identify extraction of weights, not
extraction of weight signs, as the critical bottleneck. With our improvements, a
16,721 parameter model with 2 hidden layers trained on MNIST is extracted within
only 98 minutes compared to at least 150 minutes previously. Finally, addressing
methodological deficiencies observed in previous studies, we propose new ways of
robust benchmarking for future model extraction attacks.

Main paper: 
 introduction: 
 Training machine learning (ML) models requires not only vast datasets and extensive computational resources but also expert knowledge, making the process costly. This makes models lucrative robbery targets. The rise of ML-as-a-service further amplifies the challenges associated with balancing public query accessibility and safeguarding model confidentiality. This shows the emerging risk of model extraction attacks, where adversaries aim to replicate a model's predictive capabilities from a black box setting where only the input and output can be observed. Previous attacks have approached model extraction either precisely, obtaining a copy of the victim model, or approximately, obtaining an imprecise copy of the victim model. Components of the model that have been the target of extraction include training hyperparameters [3] , architectures [4] [5] [6] , and learned parameters such as weights and biases in deep neural networks (DNNs) [1] [7] .In this paper, we are interested in precise model extraction and use the most recent advances of cryptanalytical extraction of DNNs by Carlini et al. [1] and Canales-Martínez et al. [2] as our starting point. Carlini et al. previously demonstrated that it is feasible to extract model signatures -normalised weights of neural networks -on relatively small models with up to three hidden layers. For 1 hidden layer models the parameter count tested was up to 100,480, however, for 2 and 3 hidden layer models, which are increasingly difficult to extract, the maximum parameter count for models tested was 4,020. While signature extraction was generally determined as straightforward, sign extraction of the weights was identified as a bottleneck with further work needed. Following this, Canales-Martínez et al. [2] improved sign extraction speed from exponential to polynomial time, with their Neuron Wiggle method. However, Canales-Martínez et al. stopped short of evaluating the full extraction pipeline, focusing only on the signs.In our work, we perform a deeper performance evaluation of the signature and sign extraction methods. We first create a comprehensive codebase that integrates Carlini et al. [1] 's signature extraction technique with the sign extraction technique of Canales-Martínez et al. [2] , allowing for systematic and fair benchmarking. We identify inefficiencies in the combined signature-sign method interactions and discover that we can significantly improve on the end-to-end attack efficacy. Importantly, we find that Canales-Martínez et al.'s sign extraction already eliminates the sign extraction bottleneck observed in prior work. Further improving on sign extraction we speed up the process by up to 14.8 times compared to Canales-Martínez et al., so that sign extraction only takes up as little as 0.002% of the whole extraction time for larger models. For larger models due to randomness of the start of the search and the position of neurons in space, the signature extraction can take a long time, highlighting the importance of considering the full pipeline extraction compared to only the sign extraction. For smaller or less complex models, signature extraction is much faster, and then the sign extraction time becomes more significant. For these models we see on average that the whole parameter extraction process is sped up by about 1.2 times and a speed up of up to 6.6 times can be attained when quantizing some extraction sub-routines to float32.We make the following contributions:1. Optimizing Extraction Strategies: We modify the extraction process to only sign extract neurons requiring trivial effort, finding that spending more time in extracting harder to signextract neurons does not lead to higher success in correct sign extraction. This significantly reduces the number of queries needed. We find that these harder to sign-extract neurons' sign extraction can be pipelined with other operations, improving both robustness and speed of sign extraction. An additional deduplication process and a suggestion to quantize some sub-routines speeds up the overall extraction time. 2. Redefining Bottlenecks in Extraction Processes: By optimizing sign extraction, we find that, contrary to earlier studies, extraction is now dominated by signature extraction. This shifts the focus of optimization efforts for achieving scalable high-fidelity extraction. 3. Addressing Methodological Shortcomings: We determine that improvements reported in some prior work come from unfair comparison between (1) standard benchmarks and models trained on random data; (2) models trained with different randomness; (3) models extracted using different randomness; (4) models with more hidden layers or different sizes.Our codebase can be found at https://github.com/hannafoe/cryptanalytical-extraction. 
related work: 
 The analysis of neural network model extraction strategies in Jagielski et al. [8] categorises three types of extraction attacks. Functionally equivalent extraction aims to replicate the target model's output for each input and is the most exact method of extraction. Fidelity extraction seeks to closely mirror the target model's output on specific data distributions, such as label agreement. Task accuracy extraction aims to match or surpass the target model's task performance without replicating errors. Fidelity and accuracy differ in that the goal for fidelity is close replication of the target model, where the target model is considered the benchmark label, whereas for task accuracy the goal is to closely mirror the ground-truth labels of the dataset.Learning-based methods that train a substitute model closely mirroring the target model are used for fidelity and task accuracy extraction. Since knowledge from the target blackbox model is transferred to a simpler model, architecture knowledge is not always necessary [9] , but according to Oh et al. [3] is found to raise fidelity. Moreover, while the training dataset is often not assumed, many works such as Tramèr et al. [10] assume knowledge of the problem domain and training data statistics to recover data distributions, and it has been shown that models trained on non-domain data significantly underperform [9, 11] . For instance, Papernot et al. [12] train models on self-created datasets mimicking MNIST, while others use GANs or other methods to produce data for training (Oliynyk et al. [13] , Truong et al. [14] , Correia-Silva et al. [15] ).Jagielski et al. [8] discuss how learning-based methodologies face challenges due to various nondeterministic factors, such as the random initialization of model parameters, batch formation sequence, and the unpredictable behaviours in GPU processing. In a study where the adversary is assumed to have complete access to both the training data and hyperparameters, and where the original model is used as a labeling oracle to help train a replication model with the same parameters from scratch, 93.4% was the maximum fidelity reached by the replicated model. This is attributed to the non-deterministic elements in the learning processes of both the original and replicated model. More recent work by Martinelli et al. [16] that suggests that high fidelity can actually be reached with learning based methods. However, they assume a slightly different setting, and this only works for smaller networks, and appears to be more expensive.The fidelity limitations of learning-based methods have prompted the development of techniques based on side-channel analysis or cryptanalysis to precisely extract parameters and achieve functional equivalence. Notable methods, such as those introduced by Lowd and Meek [17] , Milli et al. [18] , Batina et al. [19] , Jagielski et al. [8] , while contributing to this area, exhibit limitations in handling standard benchmarks or are confined to neural networks with only two layers, limiting its practicality. Rolnick and Körding [7] and Carlini et al. [  
methodology: 
 In what follows we explain the background methodology of sign extraction in Section 3.1 and its inefficiencies in prior work in Section 3.2, describe improvements to it and analyse its effect on the whole parameter extraction in Sections 3.3, 3.4, and finally outline some other improvements in Section 3.5.Neurons are the most basic components of neural networks. They create decision boundary hyperplanes in the parameter space and our goal is to identify these. These boundaries are defined by weights and the bias associated with the neuron and it is this hyperplane equation that ultimately decides which inputs activate a neuron. Each neuron contributes to the final output of the model by either activating or deactivating with some given input. Locating the decision boundaries involves two main steps: identifying the neuron's value and determining its sign. While the exact scale of the hyperplane's normal vector is not crucial (as the hyperplane can be identified up to a multiplicative factor), the sign is critical as it determines how it partitions the parameter space and on which side of the hyperplane the inputs fall. In other words, the orientation of the decision boundary hyperplane is determined by the sign of the neuron. In a DNN, the sign determines the outcome of the matrix multiplication involving the input and weight matrix and the subsequent application of the ReLU activation. Depending on the sign of a neuron, a positive outcome could change to negative and be turned into 0 through the ReLU activation, turning off the neuron's contribution to the final model output. Carlini et al. [1] introduce the concept of a neuron's signature, a set of normalised weight ratios ( a1 a1 , a2 a1 , . . . ,a d i-1a1 ), where each a i is an individual weight. Each signature signifies the neuron's decision boundary up to sign. They, however, identify the subsequent extraction of the sign as the bottleneck of the whole parameter extraction process. This is why Canales-Martínez et al. [2] develop the Neuron Wiggle method which speeds sign extraction up to polynomial time. In the following, we describe this sign extraction method, in order to describe our contributions on top of this. Please refer to Carlini et al. [1] and Canales-Martínez et al. [2] for an explanation of the whole parameter extraction routine that includes signature extraction.Neuron Wiggle Sign Recovery: A wiggle at layer i is defined as a vector δ ∈ R di-1 of perturbations, where d i-1 is the dimension of layer i -1. The preimage of this wiggle at the input layer is added to witnesses, i.e., inputs x that place a neuron η on the decision boundary of being activated or deactivated. At these 'critical points' on the decision boundary, the output of the neuron is 0 and hence the effect of the perturbation on the neuron can be isolated. We define Âi to be the extracted weight matrix in layer i and Âi k to be the kth row corresponding to the weights of neuron k. Then, when the perturbation is added to the witness, neuron k of layer i changes its value by e k = ⟨ Âi k , δ⟩, i.e., the dot product of the extracted weights of neuron k with the wiggle. Say the vector c = {c 1 , ..., c di } expresses the output coefficients of x in layer i. Then the difference in output that is induced by the wiggle can be expressed as k∈I c k e k , where I contains all indices of active neurons in layer i.We compute the maximal wiggle that affects only one neuron by aligning the wiggle δ parallel to Âi j . This alignment ensures that the absolute value of the dot product |⟨ Âi j , δ⟩| is maximised, making δ effective in activating the target neuron j while minimizing impact on others. The reason for this is because δ being parallel to Âi j has either the same sign or opposite sign to Âi j . So then, all summands in the dot product will have the same sign, preventing cancelling out each others effects and maximizing the summation. If no other row is a multiple of η j then the dot product of the row concerning η j with the wiggle should be bigger than the dot product of the wiggle with any other row of the extracted matrix. Overall, neuron wiggle sign recovery involves the following steps:1. Project Âi j onto the orthogonal basis for the vector space at layer i -1 and scale this projection to a small norm, ε i-1 , so that its influence in the output is minimal and only on the targeted neuron.2. Calculate the preimage of this vector δ to obtain a wiggle in the input dimension(F (i-1) ) -1 (δ) = △. 3. Calculate f (x * ), f (x * + △), f (x * -△).4. Examine whether addition or subtraction of the wiggle activates the neuron.a. If adding △ increases the magnitude of the output more than subtracting it, i.e., |f(x * -△) -f (x * )| < |f (x * + △) -f (x * )|, then η j is positive. So, Âi j = A i j . b.Conversely, if subtracting △ results in a higher magnitude of the output difference, i.e.,|f(x * -△) -f (x * )| > |f (x * + △) -f (x * )|, then η j is negative. So, -Âi j = A i jIn this paper, we focus on further reducing s, the critical parameter affecting sign extraction confidence and performance in Canales-Martínez et al. [2] 's method. Ultimately our optimisations reduce sign extraction to become an insignificant fraction of time and query effort of parameter extraction.Why is Confidence needed? From the above explanation, the output effect of a target neuron is represented as the vector c j e j , where c j is the coefficient and e j is the effect of the target neuron. The combined output effect of all other active neurons ∈ I is expressed as k∈I\{j} c k e k . Incorrect sign re- Raising the number of sign extractions s to more than 15 does not significantly raise the number of correctly recovered neurons. (c) Graph showing confidences of sign recovery when a hard neuron's euclidean distance to its neighbours is manipulated. These results are on hard to sign extract neurons 25 and 26 of an MNIST trained 784-32x8-1 model extracted with seed 42. The confidence metric scales from 1 to 0.5 first on the confidence of false sign recovery, which is equivalent to 0 to 0.5 of confidence in true sign recovery and then from 0.5 to 1 on the confidence of true sign recovery, resulting in the scale going from 1 -0.5 -1.covery occurs under certain conditions: (Necessary Condition) the signs of c j e j and k∈I\{j} c k e k are opposite; (Sufficient Condition:) the magnitude of c j e j is not significant compared to the magnitude of k∈I\{j} c k e k . This leads to a scenario where c j e j + k∈I\{j} c k e k < k∈I\{j} c k e k , meaning that the total activation output when the target neuron is active is less than the output when the target neuron is inactive. This arises when the effect of the target neuron's activation is overwhelmed by the opposing effects of the other neurons. Thus, we refer to "hard" neurons as those for which it is particularly challenging to compute a wiggle that isolates the neuron effectively.To reliably determine the correct sign of the target neuron's effect, it is necessary that |c j e j | > 2 k∈I\{j} c k e k , which implies that the influence of the target neuron's wiggle must be greater than twice the aggregate effect of the other active neurons' wiggles. However, this criterion cannot be checked in practice because we would need to single out the contribution of each of the other neurons to the output given the witness plus perturbation as input. Yet, since the witness is not a witness to any of the other neurons, we cannot single out their effects. This is why Canales-Martínez et al. [2] introduce the concept of confidence. A series of s different sign extractions on s critical points and varying δ are conducted to gather diverse output responses. The confidence of this sign recovery is s majority /s, where s majority is the number of sign extractions supporting the majority sign. The idea is that with more sign extractions, enough Neuron Wiggles will be constructed that are not overwhelmed by the opposing effects of other neurons, so that a confident decision can be made. In fact, according to Canales-Martínez et al., the problem of wrong signs extracted with the Neuron Wiggle method should be "fixable by testing more critical points" [2] . So, after testing on CIFAR10, they decide on fixing s to 200 and recommend rerunning the Neuron Wiggle method on the least-confident 10% of sign recoveries.In practice, many neurons do not exhibit increased confidence even with additional iterations s. After an initially higher confidence on average due to not having seen enough samples, the confidence stabilises after about 10 iterations. Moreover, as illustrated in Figure 1 (b), the number of correctly identified neurons does not increase beyond s = 25 for many models. Furthermore, sign recovery confidence varies notably. Neurons that are correctly recovered typically show a confidence level above 0.75, while those incorrectly recovered average 0.64. This suggests that neurons with lower confidence are more likely to be incorrectly recovered, providing a practical indicator of recovery accuracy.'Easy' and 'Hard' to Sign Extract Neurons: The idea that some neuron signs are "easier" to extract and some are "harder" to extract can be understood by picturing two neurons that are positioned very close to each other within a layer. Creating a "wiggle" that activates one without impacting adjacent neurons is difficult and becomes more complex as the number of tightly clustered neurons increases, hindering the ability to isolate and activate a single target neuron effectively.In Figure 1 (c), the change in confidence and true versus false sign recovery is depicted when a hard neuron's euclidean distance to its neighbours is manipulated. The blue dots depict the confidence of the two originally hard to extract neurons. To manipulate the target neuron to be further away from other neurons, Gaussian noise was added and to manipulate the target neuron to be closer to a cluster of the closest neurons, part of the distance to these closest neurons was added to make the target neuron the midpoint between closest neurons. One can see that if the target neuron is closer to other neurons, with high confidence the wrong sign will be recovered, whereas if the distance to other neurons is further, with high confidence the correct sign will be recovered.The Effect of an Incorrect Sign in the Extraction Process: Prior extraction performance evaluations have primarily utilised models trained on random data originally developed by Carlini et al.. However, observations indicate that the percentage of correctly extracted neurons is significantly higher for the CIFAR10 and MNIST datasets compared to random models, prompting us to test standard benchmarks.Understanding the impact of a sign flip is key to determining the right balance between efficiency and correctness. For example, in a CIFAR10 model with 128 neurons and 8 hidden layers, keeping one hard-to-extract neuron sign flipped in layer 2 decreases the test accuracy to 0.9451 compared to the original model. With 5 sign flips, test accuracy falls between 0.55 and 0.68, and with 28 sign flips, it further declines to 0.268. These results illustrate that the effect of a sign flip is more significant than previously believed, suggesting that all prior methods of sign extraction may not have been adequate for consecutive layers' extractions, as none of them offered perfectly accurate sign extraction. The importance of accurate sign extraction is underscored by the failure of signature recovery in subsequent layers if even a single neuron's sign in a prior layer is incorrect.Following the discovery that the confidence level and the number of correctly recovered neurons stabilises after about s = 15 iterations, we propose that sign extraction can be made more efficient by running it with less iterations. Further, we find that 'hard' to sign extract neurons' signs cannot be extracted with the Neuron Wiggle method. Given that the sign of some neurons cannot be determined, a new robust method for determining them must be sought. Our approach is to test possible values for these hard to extract signs in the next layer. Correct signs can be detected as they will allow signature extraction to complete in the next layer without an error. Conversely, incorrect sign values will result in an error and the signature process can be stopped and restarted with a different combination of neuron signs. This process can be streamlined further in following way:(1) Hard to extract neurons are determined as those with low confidence (between 0.5 and 0.6 of true or false confidence since it is not known if true or false) after 15 iterations (s = 15).(2) The next layer is extracted in parallel for each possible combinations of signs for these low-confidence neurons. If, for instance, there are five such neurons, 2 5 signature extraction processes are initiated. At latest in the sign extraction an error is thrown for an incorrect signature extraction through a check described below. Each process with an error is terminated.Ultimately, only one error-free process should remain, ensuring accurate layer extraction without significantly extending execution time.Note: When extracting one target neuron's sign, errors in the 'sample distance check' lead to having to rerun that iteration of the sign extraction with a new critical point. If an iteration has had to be rerun more than 10 -20 times with high probability it can be deemed that the signature extraction must have been erroneous. Since each iteration in the sign recovery costs 5 queries this will equate to about 100 queries until we have a strong indication that the signature extraction for at least one neuron was wrong. If we wanted to, this could be used to check the correctness of each neuron's signature, by trying sign extraction for each in this way. If the sign extraction was wrong in the previous layer, all neuron signatures will be wrong. If the sign extraction in the previous layer was right but there was an error in the subsequent signature extraction, usually only one or two neuron signatures will be wrong. This check is also a good way to find these one or two erroneous signature extractions early on, so that the signature extraction can be rerun for these neurons before continuing with extraction in subsequent layers.The Sample Distance Check: Specifically, the sample distance check fails continuously if the signature recovery is incorrect. The sign recovery compares the outputs f (x * ), f (x * + ∆), andf (x * -∆) with the subtractions sL = f (x * -∆) -f (x * ) and sR = f (x * + ∆) -f (x * ). If ||sL -sR|| ≤ 10 -13, then the process is aborted and restarts at a new critical point, as such a small difference suggests that the neuron's activation state may not have changed. This indicates a potential error in identifying a critical point. The threshold of 10 -13 is used due to the precision limitations of float64 being at about 10 -15 and correct wiggles typically impacting the output in the order of 10 -9 . Consecutive failures suggest that there must be errors in the neuron's signature since the search for additional critical points in the sign extraction is guided by them.According to Carlini et al. 1. Improvements in Signature Extraction:(a) The process of finding partial signatures was improved to find specific missing partial signatures which diversify the view and help to construct the full signature. (b) The memory and running time was improved by discarding all partial signatures that do not contribute a new view to the full signature (memory deduplication). Please refer to Appendix A for more details on these improvements.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We mention that as in the prior cryptanalytical extraction by Carlini et al. [1] we assume that the DNN uses ReLU activations, that the weights are in high precision, and that the output logits can be obtained. We reiterate how the Neuron Wiggle sign extraction works from Canales-Martínez et al. [2] 's paper and expand on some concepts that we empirically justify since we are working with heuristics. We do not provide any Theorems or Lemmas. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
scalability and accuracy in neuron sign prediction: 
 We assess the accuracy of neuron sign predictions on standard benchmarks by examining MNIST and CIFAR models with various configurations of hidden layers. We find that the number of low confident and incorrectly identified neurons do not exceed 10. In this way, things should remain scalable, as parallelisation does not exceed more than 2 10 signature extractions. More details are in Appendix B.Model Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide the details of the machines the experiments were run on (AMD Ryzen 7 4700U processor with 16GB RAM and Intel's 10th generation Intel Core processor icelake). We also provide information about the datasets models were trained on, the model sizes and accuracies and especially in Table 2 , present all results over a mean of four extractions on different random seeds. We also note the ML framework that was used (jax) and include query numbers in the Appendix in case of extraction time comparability difficulties. The code for producing the 'random' models can also be found in Carlini et al. [1] 's codebase and all our model training on standard benchmarks follows similar training details as can be found in Canales-Martínez et al. [2] 's codebase for training CIFAR models. Furthermore, the main technical contributions we give are all detailed in the main paper and the Appendix and can be added into prior codebases from Carlini et al. [1] and Canales-Martínez et al. [2] . We have now also included a link to our github code base: https://github.com/hannafoe/cryptanalytical-extraction Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
running time of different models: 
 In Table 2 an overview of different model extractions are presented. All extractions in this table were run on a High Performance Cluster with Intel's 10th generation Intel Core processors icelake. In the following, three cases are presented which show the limitations of the signature extraction:Case "Random" vs. MNIST Models: "Random" models were used for testing by Carlini et al. [1] . These are not configured for a specific task but train 100 epochs on randomly generated data. The MNIST models' we additionally analysed have accuracies ranging from 0.67 to 0.94 for the two hidden layer models and is 0.91 for the 8 hidden layer model. In Table 2 one can see that extraction for these "random" models is much faster compared to extraction of MNIST models. A look into the kernel density estimate of the weights visualises that the concentration of weights near 0 is much higher for real world models trained on MNIST or CIFAR10. These models exhibit greater representation sparsity, meaning that a significant number of neurons are rarely activated. As a result, many neurons are grouped together in a compact region of the parameter space where activations are infrequent and close to zero, highlighting their minimal impact on the model's predictions. Consequently, locating specific coordinates for these seldom-activated neurons is challenging, as it is difficult to identify inputs that effectively trigger their activation.Case Deeper Models: Deeper models are harder to extract. As can be seen in Table 2 , comparing the extraction time of the second layer in a two hidden layer model and in an eight hidden layer model makes this apparent. Additionally, extraction of deeper layers is increasingly time consuming -in order to obtain the full signature of a neuron, a set of critical points that activates all previous layer's neurons must be found. Only then can the neuron be looked at from all perspectives, so that the full signature can be obtained. Carlini et al. [1] and Canales-Martínez et al. [2] Here we assume that we been able to find critical points, which are on the decision boundary of activating or deactivating a neuron in our target layer, we describe briefly the idea of how Carlini et al. [1] 's signature extraction works. To understand how the critical point search works in detail, please refer to Carlini et al. [1] .Let us first go into the case of signature recovery for a neuron η j in the first layer with corresponding critical point x * ∈ R d0 . We know that a critical point lies on the decision boundary for η j to turn on or off. So, if we move a tiny amount into some direction this will either turn the neuron active or inactive.Since we are actually in the space of hyperplanes, there are actually d 0 directions we can go to turn the neuron on or off. If we construct input queries such that we can find the gradients in each of these directions then this will reveal the value for the neuron. The gradient in each direction essentially quantifies the rate at which η j 's output changes with respect to small shifts in the input. Therefore, understanding these gradients allows us to infer the weights that η j assigns to each dimension of the input space.Mathematically, we construct the input queries α i,-= ∂f ∂ei (x * -εe i ) and α i,+ = ∂f ∂ei (x * + εe i ) with ε ∈ R a small number and {e 1 , ..., e d0 } a standard basis of R d0 . These are partial derivatives (gradients) in direction e i . Either α i,-or α i,+ will tip η j into an active state. If we subtract α i,- from α i,+ then the gradient information for all neurons except for η j will cancel out, essentially revealing second order partial derivatives of f at x * in direction e i . This gives us a multiple of the value for weight a i of η j . Repeating this procedure in d 0 directions will gives us a complete set of multiples of neuron coordinates a 1 , . . . , a d0 . To obtain a comparable basis between the elements we normalise them by dividing a 1 , . . . , a d0 by some a k , usually a 1 . This now recovers the full signature ( a1 a1 , a2 a1 , . . . ,a d 0 a1). Critical points associated to the same neuron produce identical signatures, whereas those related to neurons in different layers create different signatures. This distinction allows us to identify signatures corresponding to neurons in layer 1.Signatures in consequent layers cannot be as easily found because the direct control of input in the blackbox model is lost, so changing the input in one dimension at a time will not work. Instead of e i a vector δ k of length d 0 in some random direction is sampled from N (0, εI d0 ). Then we find the second order partial derivative{y k } = { ∂ 2 f (x * )∂δ1∂δ k } for k = {1, ..., d i-1 } in a similar fashion as before by adding and subtracting the δ k vector to the critical point, computing its partial derivative and subtracting α i,-from α i,+ . Having found second order partial derivatives in d i-1 directions, we construct a hidden matrix H which denotes the output up to the previous layer i -1 for each input x * + δ k . This hidden matrix also reveals at which places the ReLU in previous layers has set some values to zero, turning neurons off. Now, solving for a in the dot product < H, a >= y with the least squares approximation gives us the signature of a neuron. However, since ReLUs in previous layers might have already set some values of the input to zero, different critical points give varying partial signatures. These must be combined to reconstruct the actual full signature.The output up to layer j, f 1..j (x * ) for some critical point x * , is likely to have some negative values, effectively turning the neuron off, so that only a partial signature can be recovered for neuron η * of layer j + 1. Hence, as already mentioned different partial signatures are combined to unify into one complete signature for a neuron. On average half of the neurons in the previous layer are negative and all points in the neighbourhood of x * will produce entries of zero in the partial signature at those indices. Now, given x 1 , x 2 witnesses to critical points of neuron η * , imagine that f 1..j (x 1 ) produces a partial signature with entries t 1 ⊂ {1, ..., d j } and f 1..j (x 2 ) correspondingly t 2 ⊂ {1, ..., d j }. Then, as long as t 1 ∩ t 2 has at least one element a joint set t 1 ∪ t 2 can be made. Assume r 1 to be the weight vector produced with x 1 with entries t 1 and r 2 the weight vector for x 2 . Then both are solutions for the same matrix row A j i , so should be equivalent at indices r 1 ∩ r 2 up to some multiple:r 1 [t 1 ∩ t 2 ] = c * r 2 [t 1 ∩ t 2 ]. From this one can compute c to recover r 1,2 [t 1 ∩ t 2 ]. This unification procedure at the same time is also a check whether x 1 and x 2 are truly witnesses for the same critical point. Because if this fails then r 1 and r 2 are not scalable by one c and hence not parallel to each other. This is also why it is helpful to have a set t 1 ∩ t 2 ≥ 2 for cross-checking.If only a single reference vector r 1 is used for scaling this can lead to significant imprecisions. The normalization for each partial signature is calculated with Â1 i,j / Â1 i,k , where k is usually set to 1. However, if for example A 1 i,k < 10 -α , where α ≫ 0 and other A i,k values are considerably larger, then normalization will produce precision errors and the further adjustment to other partial signatures through rescaling with a constant c according to a single reference vector will accumulate even more of them.To mitigate these errors, Carlini et al. propose a more robust approach:1. Compute partial signatures r 1 , . . . , r n for all witnesses found for a layer l.2. Cluster these signatures into sets {S j } d l j=1 . Signatures that are the same up to an error tolerance and scale should be grouped together. They are deemed to correspond to the same neuron. 3. For each set, determine the optimal unification constant that normalises the scale of all the different partial signatures within the set.In this way the propagation of errors through scaling should be minimised.We utilise graph clustering to determine subsets S of neuron vectors. Define each vector r m in cluster n by its coordinate r a m . A graph G = (V, E) is constructed where each vertex in V represents a vector r i , and an edge in E connects two vertices if their vectors are sufficiently similar to suggest they belong to the same neuron. Specifically, vertices r i and r j are connected if the sum of indicators k 1[|r k i -r k j | < ϵ] exceeds log d 0 , suggesting close proximity in at least log d 0 dimensions. Carlini et al. [1] suggest that an ϵ of 10 -5 is effective.To obtain the best scaling factor for a partial signature a it is paired with another partial signature b from the same neuron and aligned such that r i a = r i b • C ab for as many indices i as possible. The calculation of all possible scales between two partial signatures can be expressed in a matrix of ratios, where each entry is M i,a,b = r i a /r i b . We then select C ab = median i M i,a,b as the scaling factor between signatures a and b, with the standard deviation e ab = stdev i M i,a,b serving as the error estimate.If unifying row x with a and then combining with b preserves precision, then ideallyC ax • C xb = C ab .If not, and the combined error e ax + e xb is less than e ab , the scaling factor C ab is updated to C ax • C xb , optimizing the scale unification. This process is iterated until no further improvements are found, and the optimal dimension for unification is determined by a = arg min a b e ab , resulting in the finalised vector C a .There is a more targeted critical point search that is described in Carlini et al. [1] . This has been slightly adapted to ensure discovery of a diverse set of critical points for each neuron so that the partial signatures obtained are complementing each other to a full signature and not repeating the same partial signature. For this, the indices of the still missing parts of the full signature are tracked and they are used to guide subsequent critical point searches. Additional debugging now prevents infinite loops that had previously been found in some signature extraction processes. They had been initiated in the targeted critical point search.Furthermore, we discovered that for a whole set of easy to find neurons the full signature is already obtained after few rounds of the graph clustering algorithm. Yet, the general, untargeted critical point search continues until at least three critical points are found for each neuron. This not only accumulates critical points necessary for neurons whose signature was not fully found yet, but also accumulates a lot of critical points for neurons where the full signature was already obtained. However, this is very inefficient for the graph clustering because the number of critical points and partial signatures that must be computed each iteration keeps increasing. Hence, we have implemented a memory deduplication technique that discards of critical points and their partial signatures if for a particular neuron the full signature has already been obtained. Furthermore, in the targeted critical point search, critical points that merely repeat existing partial signatures and do not provide new insights are also discarded. In this way memory is saved to store unnecessary critical points and partial signatures and the extraction process is sped up by not having to graph cluster unnecessary partial signatures. In the signature extraction process small errors can be introduced in the calculation of the parameters which could accumulate to bigger errors in subsequent layers. Additionally, even if extraction is precise up to machine precision, the limited precision of float64 can accumulate an error over time. This is why Carlini et al. [1] develop a method for improving the precision which brings the error from 2 -15 down to 2 -35 or lower. For each neuron η k in layer j, j witnesses are computed by querying the up to layer j extracted model f1...j , so that we obtain a set {x i } dj i=1 [ref. Fig. 2a ]. For each x i , if this is also a witness of the true model then its value V (η; x i ) = 0, if it is imprecise then the value will lie between 0 and some small error ϵ. So, if we find that the value is still imprecise then we can take a random vector θ ∈ R d0 of small magnitude from the input space and find the true witness x i through a binary search in an interval [x i + θ; x i -θ]. Repeating this procedure for each x i will produce a true set of witnesses {x′ i } dj i=1 [ref. Fig 2b]. Subsequently, the improved witness coordinates are put through the model up to layer j -1 to obtain hidden vectors, h ′ i = f1...j-1 (x ′ i ). Since all previous layers are assumed to already be precise at this point, if x ′ i is truly a witness of η k then the output from layer j for η k should be 0, i.e., A j k * h ′ i = 0. Now, we try to fit a line through all critical points by solving with least squares error solution to the system of equations H * x = [1...1], where H is the hidden matrix made up of the hidden vectors [ref. Fig. 2b ]. If the least squares solution error has gone down, the newly calculated neuron coordinate is swapped in for the previous coordinate. This process can be repeated until a certain precision is reached.In practice, it could happen that in the conversion from x i to x ′ i , accidentally the found x ′ i now is a witness to a different neuron. This is the case if a different neuron is closer to 2c ]. Hence, the magnitude of θ becomes very important, as a too big value would contribute to this phenomenon but a too small θ would fail to find a new x ′ i . Carlini et al. [1] set this to 0.1 to start with, as this was a value with which approximately half of the attempts at finding a new witness x ′ i failed. If too many solutions are found, θ is reduced to half and if no solutions are found θ is multiplied byx i than η k [ref. Figure1.1. If the new solution is A j k * h ′ i < A j k * h i ,then the new least squares approximation is set to be the new coordinate of η k .The precision improvement function struggled to identify critical points for MNIST models efficiently and often failed in the precision improvement function. Tackling the first part of the problem which is producing a true set of witnesses {x ′ i } dj i=1 , several changes were made for enhanced correctness and efficiency.The method for discovering critical points was refined. Critical points are calculated with a regression function that uses a loss function based on the magnitude of A j k * h ′ i , where h ′ i = f1...j-1 (x ′ i ). If x ′ i is truly a witness of η k then the output should equate to zero. Originally, the loss was calculated as the sum of all values in the zero vector, which occasionally led to suboptimal points where most values were near zero except one or two outliers. To address this, the maximum value of the vector was added to the loss, i.e., (A j k * h ′ i ) + max(A j k * h ′ i ), ensuring all significant deviations are accounted for in the loss calculation. Additionally, previously identified critical points for η k are now included as starting points in the search for new witnesses. These points aid the optimization of the training process.Refinements have also been made to the criteria for rejecting critical points computed. Initially, a critical point was rejected if the smallest activation in the target layer was smaller or equal to 90% of the smallest activation in the target layer. In practice this, however, only rejected critical points if the minimum activation was equal to 0. This was triggered endlessly for some models. To improve accuracy, the rejection now occurs if the maximum value max(A j k * h ′ i ) exceeds 1e -5, aligning with the approach used in the loss function.Furthermore, in the second step, where a line is fitted through critical points to pinpoint a more precise coordinate for A j k , new coordinates with values near zero were causing inaccuracies and were thus rejected if any value fell below 10 -5 . This filtering process effectively eliminated problematic suggestions that previously impacted correct signature extraction.The improvements in the precision improvement enabled testing of this function for MNIST models, however, even with some changes to enhance efficiency, the precision improvement function turned out to be another bottleneck of the parameter extraction. After working correctly, the initial version took 34.7 seconds on average per neuron for the precision improvement for a layer with 8 neurons from a small MNIST model. Employing some of the tweaks decreased this to about 23.3 seconds. However, this being 33 times more than the signature recovery time and 17 times more than the sign recovery time, still seemed like a major bottleneck. This raises the question: is such high precision necessary?Currently, signature extraction achieves a precision between 10 -8 and 10 -10 , which aligns with float32 standards. The precision improvement function aimed to enhance precision to approximately 10 -15 , venturing into float64 territory. Previously, we needed a float64 accuracy to proceed with the sign extraction, as we assume the blackbox model to be in float64. However, experiments in which the extracted weights are changed to float32 and even float16 precision show that this does not significantly impact the accuracy of the sign recovery. In this setting the target blackbox model is still assumed to be in float64 precision and the only things that are changed in the sign extraction are the precision of the extracted weights and biases and some parameters regarding epsilon values.During sign extraction at these higher quantization levels, faults in the sample distance check start to become more frequent and sometimes the computation of the wiggle fails. Nevertheless, the sign extraction is still able to complete without becoming stuck in an infinite loop of error. Although recalculation of critical points is sometimes necessary due to sample distance check faults, both float32 and float64 showed similar query efficiency, with float16 lagging due to more frequent errors. In terms of processing time, float32 extraction proved faster than using float64, while float16 was slower because of increased fault rates. Ultimately, we do not need to extract up to float64 precision for the subsequent sign recovery and can therefore skip the precision enhancement function, making extraction a lot more efficient. This raises the question of whether subsequent layers can effectively be extracted with prior layers at float32 precision, and whether the following sign extraction processes remain functional. The answer is yes, the precision is often lower than if we had worked with previous layers extracted to float64, but it still extracts a correct signature. For example, for an MNIST model extracted with prior layers at float32 precision the subsequent layer signature still extracts up to precision of between 10 -6 to 10 -8 , where if the prior layer was extracted up to float64 precision the subsequent layer extraction precision was between 10 -7 to 10 -9 . Occasionally, the subsequent sign extraction in float32 does not work and gets stuck on the sample distance check fault. However, employing float16 settings in these instances circumvents these errors, allowing the process to complete correctly.A remaining concern is that the precision will start decreasing more and more from layer to layer, so that deeper layers' signature extractions may start becoming infeasible. Yet, precision improvements for a previous layer can be computed in parallel with the start of extraction for subsequent layers. This approach ensures that once the precision of an earlier layer is enhanced to near float64 levels, it can replace the earlier float32 precision extraction, thus maintaining the integrity and feasibility of the overall extraction process.Prior extraction performance evaluations have primarily utilised models trained on random data originally developed by Carlini et al. [1] . However, observations indicate that the percentage of correctly sign extracted neurons is significantly higher for models trained on standard benchmarks compared to random models. We assess the accuracy of neuron sign predictions on MNIST and CIFAR10 models with various configurations of hidden layers. Specifically, we analyze MNIST models with either 2 or 8 hidden layers, and CIFAR10 models with 8 hidden layers and layer sizes ranging from 2 2 to 2 8 . To handle neurons with potentially incorrect signs, the next layer is extracted with all possible combinations of signs for these low-confidence neurons. The one error free extraction will confirm the correct signs for us (see Section 3.3). We assume that it is reasonable to handle up to 10 low-confidence neurons per layer and up to 2 10 parallel signature extractions.Our findings reveal that MNIST models with only 2 hidden layers exhibit a significantly higher number of incorrectly predicted neuron signs compared to those with 8 hidden layers, despite being trained under identical conditions [ref. Fig. 3(b) ]. Given that most target models possess at least 8 hidden layers, this should be no problem in real life setting. Furthermore, as shown in Figure 3 (c), the number of incorrectly extracted neurons increases for models where adding more neurons to a layer did not significantly improve model accuracy. This plateau in performance indicates that for these models fewer neurons could have been sufficient for effective predictions. If we were to envision the neuron distribution of such a network, it is likely that many neurons would be positioned closely together, contributing to the decision-making process in a similar manner.A structurally pruned network, which eliminates closely spaced redundant nodes, typically avoids these problems, placing neurons distinctly in the parameter space. This placement enhances sign prediction accuracy. Consequently, the described sign extraction method with parallelisation in the subsequent layer's signature extraction should effectively scale with network size, since real-world models are most likely to exhibit these characteristics.As shown in Figure 3 , CIFAR10 and MNIST models with 8 hidden layers have a maximum threshold of incorrectly predicted neurons of 10 in the MNIST model with hidden layer size 256. Yet, this hidden layer size is already unnecessary for the prediction of MNIST as can be seen in the graph since accuracy does not improve much. Taking this model out from our samples we are left with 13 models, where the maximum number of incorrectly recovered neurons is 4 if we run sign extraction s = 15 iterations for each model. These numbers underscore the methods real-world applicability. In fact, Canales-Martinez et al. [2] suggest that in high-dimensional spaces, the likelihood of two random vectors being perpendicular increases with the number of neurons in a layer, thereby enhancing the distinctiveness of the target neuron's wiggle. This property makes neurons predominantly straightforward to extract signs from, further affirming the method's robustness in real-world applications.Finally, we must determine if the incorrectly recovered neurons fall under the low-confidence category, defined by a confidence below 0.6 with s = 15. In the sample of 13 models, most incorrectly extracted neurons were part of the low-confidence group. However, in 4 of the 13 models, there was an additional neuron incorrectly identified outside this group. Therefore, about 30% of the time, another extraction round with s = 15 may be required. Typically, after a subsequent round, these previously misidentified neurons flip sign again and can be identified in this way. These neurons that experience a sign flip can then be included into the parallel signature extraction.Explanation of the runtime numbers in the Abstract: In the abstract we give the extraction time of a whole MNIST model of size 784-64x2-1 with 16,721 parameters. For both the original computation and our improved computation, the precision improvement function is excluded. In the original implementation the precision improvement function did not work for MNIST models and in our implementation the precision improvement function still took multiple times longer than other parts of the extraction. While we show that precision improvement is not necessarily needed, it would have been an unfair comparison to include this computational time only in the original computation. Furthermore, as shown in Section A.2.2, a precision of 10 -8 is already obtained without employing the precision improvement function.Additionally, we run both extraction computations under an optimal scenario, where prior layers' extractions are assumed to be correct. For example, if layer 3 is extracted, layer 1 and 2 are assumed to have already been correctly extracted. In practice, the method is not always that robust. For some extractions up to one or two neurons' signatures are sometimes extracted incorrectly and in the sign extraction both Canales-Martínez et al.'s and our version include some incorrectly sign identified low confident neurons (see Figure 1(b) ). As suggested in Section 3.3, these errors can be found with the 'sample distance check'. However, this also means that in practice if errors are found, the signature extraction needs to be rerun. Similarly, for low confidence in sign extraction, 2 x signature extraction need to be run, where x is the number of low confidence neurons.We tested the extraction time over four extraction seeds. Extraction of layer 1 took approximately 138 minutes and 9,272,363 queries on average in the original Carlini+Canales-Martinez implementation and approximately 90 minutes and 5,157,296 queries on average in our implementation. Extraction of layer 2 took approximately 12 minutes and 4,630,078 queries on average in the original Carlini+Canales-Martinez implementation and approximately 8 minutes and 3,046,048 queries on average in our implementation. The final layer, layer 3, only takes 0.0008 seconds because a more direct method involving a system of linear equations can be used.Idealised Extraction: Out codebase is built to extract weights and biases for one layer at a time following the example of Canales-Martínez et al.'s codebase. This was helpful for understanding the differences in extraction times for different layers. The extraction process becomes much more difficult for deeper layers but becomes very easy again for the last layer. Since extraction is performed layer by layer, previous layers are always assumed to be extracted correctly. Because of this idealised setting in our codebase, we have not actually implemented a parallelised signature extraction, since this only becomes necessary if a prior layer's sign extraction was faulty. To get extraction times and query numbers for the whole extraction we have rerun the code for different layers separately, assuming that all prior layers' extraction were correct. To run the signature extraction on some next layer for all sign configurations of neurons with low confidence, the code for these can be run at the same time on different nodes.Nevertheless, it is important to understand that if one executed all configurations in parallel, then while the extraction time stays on similar levels, the query numbers would be multiple times higher than they are for running signature extraction on just one configuration. In a parallelised version, queries in the general critical point search could be shared, but queries for the faster targeted critical point search could not be shared. Furthermore, while the easiest way to find an error was in the subsequent layer's sign extraction, further analysis of the signature extraction might reveal ways of finding faults earlier on that indicate the sign configuration in the prior layer was incorrect. Additionally, if the implementation considered the whole pipeline extraction, the critical points found that do not belong to the target layer could be saved to use in later layers. This could save us from having to execute some queries in the first part of the general critical point search for later layers. 3 : Extraction Performance Carlini (C), Canales-Martinez (CM) versus Ours on layer 2 of random models. The sign (unified) is the sign extraction time from the unified codebase, which has been used throughout the paper. The sign (original) time is the sign extraction time from the separate original codebase from CM. These are denoted here for completion.When unifying the codebases we noticed that Carlini et al.'s codebase had been written using jax as their machine learning library and their query calls were executed as a jax matrix multiplication of the weights and bias with the input. In contrast, Canales-Martínez et al.'s codebase was using TensorFlow and their query calls were executed through the predict() function in TensorFlow. To unify the codebase and ensure comparability between signature and sign extraction we chose to use jax as the basis machine learning library. This version, is the CM unified version that we use throughout the rest of the paper. The original and unified version do not exhibit any difference in query numbers or theory of the Neuron Wiggle sign extraction. Under Sign (original) in Table 3 Explanation of Figure 5 : Figure 5 shows how signature recovery and sign recovery for the second layer in MNIST models with layer structures ranging from 3072 -4 -4 -1 to 3072 -64 -64 -1 scale. The signature extraction was run with four different random seeds. In 5(a) we can see that the total sign recovery time is a lot smaller compared to the signature recovery time. This is due to two factors: First, the signature recovery takes longer in MNIST models compared to the random models we were looking at earlier. Second, the sign recovery takes a fraction of the time it took before.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide the details of the machines the experiments were run on (AMD Ryzen 7 4700U processor with 16GB RAM and Intel's 10th generation Intel Core processor icelake). Code will also be released.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We do not work with human participants and only consider standard public benchmarks.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The setting considered in the paper is toy and considers models that are rarely used in practice. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work does not yet present a high risk of misuse as methods detailed have only been successful at attacking smaller and not as deep models. 
 
 Appendix:  


Decision: I agree with the unanimous decision of the reviewers to accept the paper. However, there has been some discussion regarding its presentation. I encourage the authors to consider the reviewers' feedback to enhance the paper's readability.

Reviews: review: paper_summary: paper_summary: This paper continues a line of work on cryptanalytically extracting network parameters from (input, logit) pairs. It includes a concise explanation of relevant prior work in the area, a codebase that unifies two key prior works in the area to enable standardized comparisons, and several improvements to these prior methods that enable faster extraction of weight signs and signatures.
strength_weakness: strength_weakness: Moderate issues:
- The main reason I'm not giving this paper a higher score is that it seems to make solid incremental improvements to two prior techniques. It doesn't introduce fundamentally new methods as far as I can tell. However, I think it still adds considerable value to the area, passes the quality bar, and could foster useful discussion at NeurIPS.

Minor issues / suggestions:
- My first question coming into this paper (not having read the prior work) was, "How on earth do people extract the layers one by one when all they have are the inputs and logits?" It took some time to understand this, and I think this is possibly something that could be improved with a well-designed figure.
- Line 85: "In a study where the adversary is assumed to have complete access to both the training data and hyperparameters, 93.4% was the maximum fidelity reached by the replicated model." What dataset is this on? Is 93.4% lower than expected? By how much?

Typos:
- Missing a space in the Figure 3 caption: "(a)The"
questions: questions: Line 48 of the paper states, "Further improving on sign extraction we speed the process up by up to 14.8 times, so that sign extraction only takes up as little as 0.002% of the whole extraction time for larger models." A 14.8x speedup is nice, but doesn't this mean that sign extraction only took ~0.02% of the whole extraction time before? If so, then it was already negligible. Maybe this is part of your point about Canales-Martinez incorrectly estimated the difficulty of full extraction relative to sign extraction alone.

Which of the analyses in Section 3.1 are novel, and which have been touched on before in prior work?

Is there any chance that attacks like this could work for networks using smoother nonlinearities like GELU?
limitations: limitations: Adequately addressed
paper_summary: paper_summary: This paper proposes to perform model extraction attacks against deep neural networks. First, the authors combine two previous proposed methods into a uniformed code-base. Then, they optimize the sign extraction strategy to achieve a speed up in model extraction. Their proposed method can be used for larger models (e.g., a model with 16721 parameters and two hidden layers).
strength_weakness: strength_weakness: - This paper is hard to understand. The main novelty and the insight behind it are unclear to me.
    
    - If Canales-Martinez’s work already eliminates the sign extraction bottleneck (Line 47), why the authors still propose to optimize this process?
        
    - What is the design motivation? Why the authors could do better?
        
    - Carlini’s work is not the only crypt-analytical work, why not compare to other works (e.g., reference 7)?
        
- Too much background in the Methodology part. For example, the whole Sec. 3.1 is about some background.
    
- Section 3.4 is hard to follow.
    
- The cost of parallel computing is exponential.
questions: questions: See above.
limitations: limitations: See above.
paper_summary: paper_summary: This paper explores advanced techniques for high-fidelity model extraction that go beyond simply observing "slow signs" like model outputs or gradients. The authors evaluate and enhance existing parameter extraction methods, particularly those developed by Carlini et al. and further improved by Canales-Martínez et al., applying them to models trained on standard benchmarks. Key contributions include:
1. A unified codebase integrating previous methods
2. Optimizations to improve efficiency in extracting weight signs
3. Identification of weight extraction, rather than weight sign extraction, as the critical bottleneck
4. Significant improvements in extraction speed (e.g., extracting a 16,721 parameter MNIST model in 98 minutes vs. 150+ minutes previously)
5. Proposals for more robust benchmarking of model extraction attacks
strength_weakness: strength_weakness: - The scalability of the approach to larger, more complex models is not fully explored
- Potential ethical implications of improving model extraction techniques are not thoroughly discussed
- The paper could benefit from more discussion on potential defenses against these improved extraction methods
questions: questions: 1. Ethical Considerations: Given the potential misuse of advanced model extraction techniques, could you elaborate on the ethical implications of your work? What safeguards or guidelines do you propose to mitigate potential negative impacts?
Defense Mechanisms: While your paper focuses on improving extraction techniques, have you explored any potential defenses against these advanced model extraction attacks? If so, what were your findings, and if not, what are your thoughts on possible defensive strategies?

2. Defense Mechanisms: While your paper focuses on improving extraction techniques, have you explored any potential defenses against these advanced model extraction attacks? If so, what were your findings, and if not, what are your thoughts on possible defensive strategies?
limitations: limitations: The experiments are only conducted the ViT-B/32 CLIP architecture. It should be at least mentioned that in future it makes sense to justify on  more architectures. 
The authors lowered this limitation by doing evaluation on several datasets.
paper_summary: paper_summary: This paper proposes a unified approach to model parameter extraction by combining two prior works and integrating efficiency optimization. The authors find out that the neuron wiggle sign extraction method proposed in the previous work addresses the bottleneck identified by Calini's paper. Furthermore, the authors present methods to identify hard-to-extract neurons and parallelize the extraction process. Empirical evaluation of different benchmarks shows that the proposed unified model extraction method achieves speed up compared to prior works.
strength_weakness: strength_weakness: This paper has the following weak points: 
- There is no explicit description of the attack model/threat model used in this work; 
- While the proposed unified model extraction method shows speedup compared to the prior works and the results are successful on standardly trained ML models, the benchmarks used in this paper are still small-scale. This makes the intellectual property value of the evaluated models questionable since they might be too simple and do not carry financial value.
questions: questions: Please see the weak points above.
limitations: limitations: Please see the weak points above.

Comments: title: Thank you for your prompt response.
Thanks again for the response.
Thank you for your response
Thanks for the response
comment: Thank you very much for a prompt response!!

1. We will revise the structure to focus more on our technical contributions. We will put the background methodology in a specific background methodology subsection and will move some of it into the appendix. We will instead move some of our technical descriptions for Section 3.4 into the main paper methodologies.

2. We will work on clarifying this further.
1. In my opinion, the background part should be included in the paper but go to the related work or background part. More space should be spent on technical contributions (e.g., Section 3.4).
2. I acknowledge the authors' effort in justifying that sign extraction is no longer a bottleneck; this is indeed important. However, this is not so clear in the paper. I think the author would like to make them clearer in the paper.
1. The first part of Section  3.1 is indeed about one and a half pages of explaining previous works’ methodologies but they were necessary to make our contribution understandable. From reviewer r46i we received positive comments regarding this: “The paper is very well-written and does a great job of summarizing the fairly complex attacks that it builds on. Honestly this paper is one of the best I've ever seen for how well it values the reader's time and presents the relevant background and contributions.”

2. Section 3.4 is a mix of methodological improvements and engineering efforts. We could not go into detail about it in the paper due to limited space but have a full description of what these entail in Appendix A. Especially our insights on how precision improvement is not necessarily needed are not trivial, since this was not possible yet in Carlini et al.’s paper and Canales-Martinez et al. never connected signature and sign extraction.

3. It was not clear at first that sign extraction was not the bottleneck anymore until we unified all methodologies and metrics. In Figure 1 (a), we show with Sign CM original and unified compared to Signature Carlini that sign extraction was more inefficient before but after unifying codebases it is taking a little less time than the signature extraction. However, this does not mean sign extraction time became trivial just using Canales-Martinez et al.’s methods. Only with our improvements in sign extraction, the sign extraction actually becomes trivial compared to the signature extraction. Further, in the full pipeline our improvements in the sign extraction significantly improve the overall extraction time as shown in abstract lines 15-17. 

4. We are not too sure what the reviewer means by contradictory, we followed prior work and improved over the sign extraction performance. By the time our improvements became significant we decided to evaluate the system, including evaluation of the system as a whole, not separately as Canales Martinez. This is where we realized that sign extraction time can become trivial compared to signature extraction, and future efforts should be directed towards signature extraction improvement. Note that everyone working in this field from reading all of the work assumed that sign extraction is the bottleneck and hence iterated over it.
It is convenient in the ML community to put the code after the abstract or introduction.
1. Section 3.1's majority of contents are still backgrounds (about two pages, which occupies about half of the methodology section).
2. Section 3.4 is unclear about the acceleration improvements that have been made. This part seems to focus on engineering efforts rather than systematic methodologies. 
3. According to the introduction and the methodology, the paper focuses most of its effort on sign extraction. Based on the rebuttal, I understand this paper tries to convince others that sign extraction is not a bottleneck. Is that correct? If so, why do the authors still spend effort on improving sign extraction (see abstract lines 12-14)?
4. The authors stated "Hence, the paper is written in a way to highlight that the previously identified bottleneck is no longer relevant and suggests that future improvements need to be made to the weight extraction process." **However, according to the rebuttal, the paper's main contribution is to optimize the sign extraction attack instead of the weight extraction process.** "We identified that some neurons are harder to extract than others and performing the neuron wiggle method on them for more iterations does not help extract more correct signs. So, overall we minimized the neuron wiggle extraction time for easy to extract neurons and performed sign extraction for hard to extract neurons later in the pipeline." This is contradictory.

Thanks again for the authors's response. However, my questions are not well-addressed. I decide to keep my score.
Thank you very much! 

We also do believe sharing code is important, it all lives here: https://anonymous.4open.science/r/anonymized-cryptanalytical-extraction-main-9335/README.md
Thank you for answering my questions. I will leave my review unchanged.

Note: I understand that sharing code in this field is uncommon. However, I believe that having a unified codebase could advance this research toward more realistic models and real-world scenarios.
Dear Reviewer PoZg,

Thank you so much for your insightful feedback on our paper! We hope you found our responses useful. As the discussion period is coming to a close, please feel free to ask any remaining questions you may have. We're happy to provide further clarification!