SlowGuess/NeurIPS_2023_OCR huggingface dataset:
implementation of this as future work and report the performance of partial Gaussian secret recovery, using knowledge of s to validate correctness.
Table 3. Partial and full ternary secret recovery.  $n = {256},{\log }_{2}q = {20}$  . Epoch when secret is recovered.
h	5	10	15	20	21	22	23	24	25
partial recovery	8/10	6/10	6/10	2/10	3/10	2/10	1/10	3/10	1/10
training epoch	0,0,0,0,0,0,1,7	0,0,1,1,1,1	0,0,0,1,2,8	0,2	0,1,3	2,5	9	0,4,7	2
full recovery	8/10	6/10	5/10	1/10	3/10	2/10	0/10	1/10	0/10
training epoch	0,0,0,0,0,0,1,7	1,2,2,6,7,8	1,1,6,9,11	5	4,10,17	8,22	5,12
Table 4. Partial Gaussian secret recovery.  $n = {350},{\log }_{2}q = {27}$  . Epoch when secret is recovered.
h	4	5	6	7	8	9	10
partial recovery	8/10	8/10	7/10	5/10	2/10	2/10	4/10
training epoch	0,0,0,0,1,1,1,1	0,0,0,1,2,3,3,7	0,0,1,2,2,5,10	2,2,3,7,10	0,2	1,9	1,5,6,12
VERDE's performance across secret distributions. VERDE recovers ternary secrets with sparsity up to  $10\%$ , with comparable performance on binary secrets. Table 3 provides details on partial and full ternary secret recovery for  $n = 256$  and  $\log_2q = 20$  and  $h = 5 - - 25$ . For low values of  $h$  ( $h < 20$ ), ternary secrets are partially recovered early during training (i.e. mostly in epoch 0 or 1, during the first pass on preprocessed data), and usually fully recovered in the same epoch or shortly after. As  $h$  increases, more training is required for recovery, and the delay between partial and full recovery increases.
Figure 2. Best  $h$  recovered vs.  $\log_2 q$  and secret distribution.  $n = 256$ .
For small Gaussian secrets, VERDE only implements par
tial recovery (recovery of the nonzero bits). Table 4 presents results for  $n = 350$  and  $\log_2 q = 27$ . Recovered  $h$  are lower than in the binary case, and models require less training for low  $h$ .
Figure 2 compares VERDE's performance across secret distributions for problems with  $n = 256$  and different moduli  $q$ . For each setting, we run 100 recovery experiments and report the highest  $h$  secret recovered in those attempts. Recovery is comparable for binary and ternary secrets. Small Gaussian secrets are significantly harder.
4 Large Dimension
The hardness of LWE increases as  $n$  grows. PICANTE recovered sparse binary secrets for dimension up to  $n = 350$ . VERDE pushes this limit to  $n = 512$ , for sparse binary, ternary, and narrow Gaussian secrets. For  $n = 512$  and  $\log_2 q = 41$ , VERDE recovers binary and ternary secrets with sparsity up to 0.12 (highest  $h = 63, 60$ ) and Gaussian secrets with  $h$  up to 16. The longer VERDE's preprocessing step runs, the higher  $h$  secrets it can recover. Recall that real-world schemes like LIZARD operate in dimension  $n = 608$  [21] and HE in dimension  $n = 1024$  [4]. Those systems use significantly smaller  $q$  where LWE is harder:  $\log_2 q = 10$  and 27, respectively, compared to VERDE's  $\log_2 q = 41$ .
VERDE's performance in large dimension. Table 5 shows VERDE's performance for dimension  $n = 512$ , after 7 to 35 days of preprocessing, on the same set of matrices  $\mathbf{A}_i$ . For each preprocessing time, we measure the quality of lattice reduction via a "reduction factor," computed by taking the ratio of the standard deviation of reduced  $\mathbf{A}_i$  entries to the standard deviation of a random matrix with uniform coefficients  $0 \leq a_i < q$ , i.e.  $\frac{\text{stddev}(\mathbf{A}_{\text{bkz}})}{\text{stddev}(\mathbf{A}_{\text{rand}})}$ , with  $\text{stddev}(\mathbf{A}_{\text{rand}}) = \frac{q}{\sqrt{12}}$ . This metric, used in PICANTE for selecting BKZ parameters, is discussed in §6. As Table 5 demonstrates, the maximum recoverable  $h$  is strongly correlated to the quality of lattice reduction.
Preprocessing adjustments for large  $n$ . Scaling up to  $n = 512$  requires a number of adjustments to our preprocessing methodology. For  $n = 512$ , the first loops in BKZ 2.0 are very slow. To avoid this, we use BKZ with smaller blocksizes than those used for  $n = 256$  and 350 (see §A.1). Also, lattice
Table 5. Data preprocessing vs performance.  $n = 512$ ,  $\log_2 q = 41$ . Highest values of  $h$  recovered, for different reduction factors (lower factor = better reduction).
preprocess time	reduction factor	binary h	ternary h	Gaussian h
7 days	0.519	16,17,17,20	17,20,20,21,21	8,8,8,8,10,10,13
10 days	0.469	21,22,23,28	22,24,24,27,27,29	11,11,11,12,12,12
14 days	0.423	32,32,34,34,35,40	32,34,34,35,35	11,11,11,12,12,12
20 days	0.380	35,35,36,41,49	35,35,37,45,46	13,13,13,16
28 days	0.343	40,43,45,47,50,51,55	40,41,41,44,45,48,48,53	13,13,14,16,16
35 days	0.323	48,48,49,52,57,59,63	45,46,50,55,58,60	14,16
reduction is significantly slower for larger matrices. To mitigate this, for  $n = 512$ , we use 448 LWE samples (instead of 512) when generating  $\mathbf{A}_i$  for lattice reduction, therefore reducing the matrix size from  $1024 \times 1024$  to  $960 \times 960$ . Experimentally, we observe that using slightly fewer samples did not negatively impact our reduction factor or attack performance.
5 Small Modulus
To define real-world parameters for lattice-based cryptography, standardization committees and communities (e.g. [4, 17]) select a small enough modulus  $q$  (for fixed dimension  $n$ ), so that all known attacks are (heuristically) predicted to run in time at least  $2^{128}$ , therefore attaining the U.S. government minimum 128-bit security level. For classical lattice reduction attacks, the smaller the modulus, the more difficult the attack. This is because lattice reduction algorithms such as LLL and BKZ attempt to compute short vectors in Euclidean space, but cryptosystems operate modulo  $q$ . Smaller moduli result in smaller lattice volumes, meaning shorter target vectors are required to break the system. In our ML approach, we also observe that VERDE is less likely to succeed when  $q$  is smaller (see Table 6). Nevertheless, VERDE outperforms the uSVP attack (§A.7).
Figure 3.  $h$  of recovered secrets vs.  $\log_2 q, n = 256$ . 10 random binary secrets attempted for each  $h$ . One green dot represents a successful recovery.
Table 6. Highest  $h$  recovered,  $n = {256},{350}$  . Secret distributions are  $b =$  binary,  $t =$  ternary,g = Gaussian.
n, log2 q	reduction factor	recovered h
b	t	g
256, 20	0.43	33	24	7
256, 18	0.53	18	19	7
256, 16	0.63	12	12	6
256, 14	0.71	9	9	6
256, 12	0.77	6	6	5
350, 27	0.38	36	36	10
350, 21	0.61	12	13	5
PICANTE vs. VERDE's performance on small  $q$ . PICANTE attacks larger moduli: it can recover binary secrets with  $h = 31$  for  $\log_2 q = 23$  and  $n = 256$ , and  $h = 60$  for  $\log_2 q = 32$  and  $n = 350$ . Table 6 presents VERDE's highest recovered  $h$  (in 10 random attempts) for dimensions 256 and 350 and different  $q$  for binary, ternary, and narrow Gaussian secrets. First, note that VERDE recovers binary secrets with  $h = 33$  for  $n = 256$  and  $\log_2 q = 20$ , but also, VERDE succeeds for much smaller  $q$ , as small as  $\log_2 q = 12$ , a near-real-world parameter setting, demonstrating that VERDE significantly outperforms PICANTE. However, for smaller  $q$ , VERDE recovers only secrets with smaller  $h$ . As with classical attacks, the likely culprit is lattice reduction: the reduction factor after preprocessing is 0.71 for  $\log_2 q = 12$  versus 0.43 for  $\log_2 q = 20$ , and  $h$  for recovered binary secrets drops from 33 to 6. Section 6 provides a theoretical explanation of this phenomenon.
Figure 3 visualizes VERDE's success rates for  $n = 256$  with binary secrets. For every value of  $q$  and  $h$ , we run VERDE on 10 binary secrets, using the same preprocessed data. VERDE's success rate decreases as  $h$  increases. Attempting 10 random binary secrets for  $n = 256, \log_2 q = 12$ , VERDE recovers secrets for up to  $h = 6$  but not  $h = 7, 8$  (Table 6). However, with more attempts, VERDE recovers  $1/100$  binary secrets for  $h = 8$ . Experiments with different random seeds (see §A.2) suggest that model initialization alone is not responsible for how success rate trends with  $h$ .
Explaining success for smaller  $q$  via NoMod. As Table 6 indicates, for given  $n$  and  $h$ , secrets are harder to recover for smaller  $q$ . This suggests that the modular operations in the computation of  $b$
Figure 4. Effect of NoMod data on secret recovery for  $n = 256$ , binary secrets. Count = # of experiments.
Table 7. NoMod % before/after permuting the columns of A.  $n = 256, \log_2 q = 14$ , binary secrets. Each column is a random secret for each  $h = 6, 7, 8$ . Entries are the NoMod%. Green = secret recovered; red or black = failure.
h	method	NoMod percentages and recovery success
6	original	56	61	60	61	56	52	67	67	76	67
permuted	57	67	56	67	62	67	60	60	52	56
7	original	60	60	52	49	60	75	55	60	55	56
permuted	67	52	60	75	60	73	60	71	59	66
8	original	60	74	74	66	60	66	55	60	63	55
permuted	55	55	60	52	55	60	55	49	60	60
from  $\mathbf{a}$  might account for the difficulty. To investigate this, we evaluate, for a given known binary secret  $\mathbf{s}$ , the percentage of samples where computing  $b$  did not require the modular operation, for the 4 million samples in our training set. More precisely, we represent the mod  $q$  coordinates of  $\mathbf{a}$  and  $b$  in the interval  $(-q/2, q/2)$ , and compute  $x = \mathbf{a} \cdot \mathbf{s} - b$  without modular operations. If  $b$  was computed without modular operations, then  $x$  is equal to the error for that sample, which is small. Otherwise  $x$  is equal to the error plus a multiple of  $q$ . For each  $(\mathbf{a}, b)$ , if  $|x| < q/2$ , then no modular operation was performed. We define NoMod to be the percentage of such  $x$  in the training set.
Figure 4 shows the distribution of secret recoveries, for varying NoMod, for 210 experiments with dimension  $n = 256$  and  $\log_2 q = 20$ . Clearly, recovery occurs when NoMod exceeds a threshold value, empirically observed to be  $67\%$ . These results confirm our hypothesis that modular arithmetic is the major cause of failure to recover secrets, and help explain the observation ([42, §6.4] and §A.2) that some secrets are never recovered, no matter how many model initializations are tried. For a given secret, NoMod is a property of the training data, not of model initialization, and multiple initializations only seem to help when NoMod is close to the recovery threshold.
A trick for improving attack success. The NoMod percentage can only be calculated if the secret is known, so it cannot aid real-world attack settings. However, our NoMod observations suggest that, if recovery fails for a given secret  $\mathbf{s}$ , the failure may be due to a low NoMod factor in the preprocessed training set. This reveals a strategy for potentially recovering such secrets. If an initial run of VERDE fails for a secret  $\mathbf{s}$ , start over with the (un-preprocessed) matrices  $\mathbf{A}_i$ , sampled from the original LWE samples. For each of these, apply a random permutation  $\Pi$  to the columns and preprocess the permuted matrices. This creates a new dataset of  $\mathbf{A}'$  with corresponding  $b'$ , associated to a permuted counterpart  $\mathbf{s}'$  of the original secret  $\mathbf{s}$ . If the NoMod of  $\mathbf{s}'$ ,  $b'$  and  $\mathbf{A}'$  is higher than that of  $\mathbf{s}$ ,  $b$  and  $\mathbf{A}$  (though this cannot be measured),  $\mathbf{s}'$  can be recovered. If the attack succeeds,  $\mathbf{s}$  can be restored by applying the inverse of permutation  $\Pi$ . Table 7 presents the impact of permutations on NoMod for 10 secrets and  $h = 6, 7, 8$  for  $n = 256$  and  $\log_2 q = 14$ . Some secrets become recoverable after using the permutation trick.
6 A Theoretical Scaling Law for VERDE
Intuition. The NoMod experiments provide a key insight about the hardness of secret recovery in SALSA-based attacks on LWE. They suggest that a secret  $\mathbf{s}$  can be recovered from a training set if over  $67\%$  of the  $\{x = \mathbf{a} \cdot \mathbf{s} - b\}$  are concentrated in the interval of length  $q$ . If the random variable  $x$  is Gaussian (a reasonable assumption for  $h \gg 1$ , since the entries of  $\mathbf{a}$  are random and bounded),  $68\%$  of its values will be within one standard deviation of its mean, i.e. spread over two standard deviations. Therefore, for the secret to be recoverable, the standard deviation of  $x$  should satisfy  $\sigma_x \leq q/2$ . If  $\mathbf{s}$  is a binary secret with Hamming weight  $h$ , and the entries of  $\mathbf{a}$  have standard deviation  $\sigma_a$ , we have  $x \approx \mathbf{a} \cdot \mathbf{s} - e$  and  $\sigma_x \approx \sqrt{h} \sigma_a + \sigma_e \approx \sqrt{h} \sigma_a$ . Therefore,  $\mathbf{s}$  is recoverable if  $\sqrt{h} \sigma_a \leq q/2$ , or  $\sigma_a \leq \frac{q}{2\sqrt{h}}$ .
Scaling Laws. We now apply this insight to the experimental results of ML-based attacks on LWE. Consider the original SALSA attack, which does not utilize data preprocessing. Since the entries of a are uniformly distributed over  $[0,q)$ ,  $\sigma_{a} = \frac{q}{\sqrt{12}}$ . Replacing in  $\sigma_{a} \leq \frac{q}{2\sqrt{h}}$  yields  $h \leq 3$ , the main experimental result of SALSA. If we constrain the entries of a to be in  $[0,\alpha q)$  (Table 4 in [58]), we have  $\sigma_{a} = \frac{\alpha q}{\sqrt{12}}$ , and  $h \leq \frac{3}{\alpha^{2}}$ . Applying this formula for  $\alpha \in [0.6, 0.55, 0.5, 0.45]$ , we obtain maximal recoverable Hamming weights of 8, 10, 12 and 15, which closely match SALSA experimental results.
These results shed light on the role of preprocessing in PICANTE and VERDE. When the standard deviation of  $\mathbf{a}$  is reduced by a factor  $\alpha$ , maximal recoverable  $h$  increases by a factor  $\frac{1}{\alpha^2}$ . However, the formula  $h \leq \frac{3}{\alpha^2}$  underestimates actually recovered  $h$  by a factor of 2. For instance, from the reduction factors from Table 5, we should expect recovered  $h$  to range from 11 to 29 as preprocessing time increases from 7 to 35 days, but actual recovered  $h$  ranges from 20 to 63. As seen in §5, the preprocessing step makes some secrets easier to recover for the same  $h$ , so VERDE performs better than predicted by theory for some secrets (at the expense of other secrets). Finally, note that the formula for the standard deviation of  $x$  is the same for ternary and binary secrets. This accounts for the observation in §3 that ternary secrets are of similar difficulty for VERDE as binary secrets.
7 Model architecture
VERDE's baseline model uses a seq2seq architecture as in SALSA and PICANTE. Here, we compare it to the new encoder-only model discussed in §2. Specifically, this new model is based on the DeBERTa [34] model but replaces its "relative positional embeddings" with rotary word embeddings, in the spirit of rotary positional embeddings (RoPE) [55] but applied to the integer words. It has 4 layers, 4 heads and 512 dimensions. On top of the cross-entropy loss, we add an auxiliary, squared earth mover's distance (EMD) loss to compare model's softmax distribution with the target  $b$ . This encourages the model to make predictions that are at least close to the targets, if not exact matches. Trained with the auxiliary EMD loss, the model also replaces the beam search used in the distinguisher with a novel EMD-based distinguisher that compares the difference between the distributions produced by the model on  $\mathbf{a}_{\mathrm{test}}$  and  $\mathbf{a}_{\mathrm{test}} + K\mathbf{e}_i$ .
Table 8. Performances of seq2seq and encoder-only models.  $n = 256$ ,  $\log_2 q = 12$ ;  $n = 512$ ,  $\log_2 q = 41$ . For binary and ternary secrets, we run 10 secrets per  $h$  and indicate the epochs of full recovery.
n, secret χs	256, binary	256, ternary	512, binary	512, ternary
h	5	6	4	5	6	57	59	63	55	58	60
Seq2seq	7	4,7	0,5,7,18	0,0,1	1	4	8	7	16	11	-
Encoder-only	20,23,27	16,23,28	0,16,27	1,1,2,6	2	2	3	3	-	6	5
Overall, we find that the performance of the two models are comparable (see Table 8). The encoder-only model requires more training epochs before recovery for  $n = 256$ ,  $\log_2 q = 12$ , but requires fewer epochs for  $n = 512$ , and may scale well to larger  $n$ . Furthermore, we observe that the EMD distinguisher still enables full recovery of ternary secrets with high  $h$ .
ML for cryptanalysis. Numerous proposals leverage ML for cryptanalysis, either indirectly or directly. We call approaches which use ML as part of (but not the main element of) the cryptanalysis process indirect approaches. Indirect approaches typically use ML models to strengthen existing cryptanalysis approaches, such as side channel or differential analysis [18]. Most relevant to this work is a recent paper showing the successful use of ML algorithms in side-channel analysis to attack Kyber, one of the NIST standardized PQC proposals [28]. Direct ML-based cryptanalysis schemes train models to directly recover cryptographic secrets from plaintext/ciphertext pairs or similar information. Such approaches have been studied against a variety of cryptosystems, including block ciphers [29, 9, 18, 3, 53, 38, 8], hash functions [30], and substitution ciphers [1, 54, 7, 31]. The two ML-based LWE attacks described in §1, SALSA [58] and PICANTE [42], fall under this heading.
Use of transformers for mathematics. In recent years, language models have been used to solve math problems beyond the cryptanalysis applications explored in this work. Much prior work has considered how well language models can solve written math problems [35, 48]. More recently, [32] showed large transformers could achieve high accuracy on elementary/high school problems, and [59] explored the accuracy of GPT-3 [11] on math word problems. Language models have also been applied to formalized symbolic math problems. After [40] demonstrated that transformers can solve such problems with high accuracy, follow-up work has explored transformers' use in theorem proving [43], dynamical systems [13], SAT solving [51], transport graphs [14], and symbolic regression [10, 23]. Finally, some have proposed customizing model architectures to enable specific arithmetic operations [36, 56, 44].
We present VERDE, a ML-based attack on LWE with sparse small secrets. VERDE improves data preprocessing and secret recovery, enabling significant performance gains over prior work, SALSA [58] and PICANTE [42]. In particular, VERDE can recover secrets with more general distributions (ternary, small Gaussian), larger dimensions  $n$ , smaller moduli  $q$ , and higher  $h$ . In our implementation, VERDE outperforms the uSVP attack, requiring less time but more compute resources (details in Appendix A.7). Most importantly, this work provides key theoretical insights into observed attack performance, paving the way for targeted future work. Note that even if we recover secrets with seemingly low probability, such as one seed out of ten succeeds for that secret (1/10), or if we do not recover all secrets successfully, such as we recover one out of ten secrets with our attack, this is still enough to make these cryptosystems unsafe to use with low weight secrets (unless there is a way to check for vulnerability to our attack without running the attack).
Limitations and broader impact. Despite significantly advancing the state-of-the-art in ML-based LWE attacks, VERDE cannot yet break standardized LWE-based PQC schemes, limiting its real-world impact. Because of this, our paper raises no immediate security concerns. Nevertheless, we have shared a copy of our paper with the NIST PQC group to make them aware of this attack.
Future work. Scaling VERDE to attack real-world systems requires work in several directions: increasing dimension  $n$ , reducing modulus  $q$ , and increasing  $h$ , the number of nonzero entries in recoverable secrets. Continued innovations in model architecture (a la §7) may allow attacks in higher dimension  $n$ , while the theoretical scaling law of §6 provides helpful guidance for improving  $q$  and  $h$ . Given our new insight and analysis of the importance of the NoMod percentage, we conclude that the reason that small  $q$  is hard for the transformers with our current approach is that they are not good at modular arithmetic (not yet, anyway—this is an area for future work and improvement). In addition, our preprocessing is not well-suited to what is precisely needed and relies on existing lattice reduction algorithms which scale poorly with  $n$  and  $q$ .
This suggests two avenues for improvement: first, to develop model architectures that perform modular arithmetic better. Limited existing work has studied the application of ML to modular arithmetic more broadly [44, 33], Second, alternative preprocessing techniques should be developed to directly concentrate the distribution of random vectors, without relying on lattice reduction methods. With the goal of reducing the standard deviation of the training data, around any center, techniques from the broader math community may prove helpful.
Acknowledgements. We thank Jana Sotáková for her contributions to developing the attack on ternary secrets, Hamming reduction techniques, and Section A.3. We also thank Mark Tygert for his helpful input.