SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cathy Yuanchen Li
Emily Wenger
Zeyuan Allen-Zhu
Francois Charton*
Kristin Lauter*
FAIR, Meta
Learning with Errors (LWE) is a hard math problem used in post-quantum cryptography. Homomorphic Encryption (HE) schemes rely on the hardness of the LWE problem for their security, and two LWE-based cryptosystems were recently standardized by NIST for digital signatures and key exchange (KEM). Thus, it is critical to continue assessing the security of LWE and specific parameter choices. For example, HE uses secrets with small entries, and the HE community has considered standardizing small sparse secrets to improve efficiency and functionality. However, prior work, SALSA and PICANTE, showed that ML attacks can recover sparse binary secrets. Building on these, we propose VERDE, an improved ML attack that can recover sparse binary, ternary, and narrow Gaussian secrets. Using improved preprocessing and secret recovery techniques, VERDE can attack LWE with larger dimensions ( $n = 512$ ) and smaller moduli ( $\log_2 q = 12$  for  $n = 256$ ), using less time and power. We propose novel architectures for scaling. Finally, we develop a theory that explains the success of ML LWE attacks.
Language models have been successfully applied to numerous practical science problems in recent years. For example, transformers [57] have been used to solve problems in mathematics [40, 24], theoretical physics [26], chemistry [50], and biology [46]. In this paper, we present an application of transformers to computer security: the cryptanalysis of Learning With Errors (LWE) [45], a hard math problem underpinning leading proposals for post-quantum public key cryptography.
Public-key cryptosystems are the main solution for secure communication over the Internet. Public keys can be used to encode messages or verify digital signatures to or from a user with the corresponding private key. Security relies on the fact that recovering the private key from the public data requires solving a computationally hard math problem. Most currently deployed public-key systems are based on RSA [47], which relies on the hardness of factoring large numbers into products of primes. Unfortunately, large-scale quantum computers will enable implementation of Shor's algorithm [52], which can factor integers in quantum polynomial time and break such systems. As a result, new hard problems are sought to serve as the basis of post-quantum public key cryptography (PQC). The US National Institute of Standards and Technology (NIST) ran a 5 year competition to define future PQC standards [2], and standardized 4 PQC systems in July 2022 [17]. Two of these rely on special cases of the same hard (and Shor-free) problem: Learning with Errors (LWE).
The Learning With Errors problem (LWE) assumes that it is hard to recover a secret vector  $\mathbf{s}$ , given many LWE samples  $(\mathbf{a}, b)$ . In a LWE sample, each  $\mathbf{a}$  is a random vector of  $n$  integers modulo  $q$  ( $n$  is the dimension and  $q$  is the modulus), and  $b$  is a noisy modular inner product of  $\mathbf{a}$  and the secret key  $\mathbf{s}$ —that is,  $b = \mathbf{a} \cdot \mathbf{s} + e \mod q$ , with the error  $e$  drawn from a Gaussian distribution of
*Co-senior authors
width  $\sigma_{e}$  centered at  $0$ ,  $e \sim N(0, \sigma_{e}^{2})$  and  $\sigma_{e} \ll q$ . The hardness of LWE is related to the hardness of well-known lattice problems such as the (approximate) Shortest Vector Problem (SVP).
Most classical attacks on LWE rely on lattice reduction [41, 49, 19]. For example, given  $m$  samples  $(\mathbf{a_i}, b_i)$ , create a matrix  $A_{m \times n}$  whose  $m$  rows are the vectors  $\mathbf{a_i}$ . The unique shortest vector problem (uSVP) attack recovers the secret  $\mathbf{s}$  by finding the shortest vector in a lattice constructed from  $\mathbf{b}$ , the columns of  $A_{m \times n}$ , and other parameters. The best known algorithms for solving SVP run in time exponential in the dimension  $n$ . Somewhat counter-intuitively, the smaller the modulus  $q$ , the harder the LWE problem. Approximate solutions can be computed in polynomial time with LLL [41], but the approximation factor is exponentially bad (exponential in  $n$ ). We compare our results with uSVP attacks on LWE in §A.7.
Machine learning (ML) attacks on LWE. SALSA [58], the seminal ML attack on LWE, uses a large collection of LWE samples  $\{(\mathbf{a}, b)\}$  with the same secret  $s$  to train a transformer [57] that predicts  $b$  from  $\mathbf{a}$ . SALSA presents methods for secret recovery via queries to the trained model, and observes that high model accuracy is not needed: secrets are recovered as soon as the transformer starts to learn (training loss drops). SALSA is a proof of concept, recovering binary secrets with 3 or 4 nonzero bits for problems with dimension up to 128, small instances of LWE solvable via exhaustive search.
PICANTE builds on an observation from SALSA ([58, Table 4]): transformers trained on LWE samples  $\{(\mathbf{a}, b)\}$ , with entries of a drawn from a restricted range instead of all of  $[0, q)$ , can recover binary secrets with larger Hamming weights  $h$  (number of nonzero bits). So PICANTE introduces a data preprocessing phase during which the LWE samples  $(\mathbf{a}, b)$  are processed by BKZ, a lattice reduction algorithm, to obtain LWE samples with the same secret but smaller coordinate variance (and larger error, see §2). In addition to training the transformer on the preprocessed samples, PICANTE reduces the number of LWE samples required for the attack from 4 million in SALSA to  $4n$  (e.g. 1400 for  $n = 350$ ), and improves secret recovery. Overall, PICANTE can recover binary secrets for dimensions up to 350 and Hamming weight up to 60. This is a considerable improvement over SALSA, faster than the uSVP attacks we compare against, and out of reach for exhaustive search.
PICANTE has several limitations. First, it only recovers sparse binary secrets, an important but limited subclass of LWE. Homomorphic encryption (HE) may use binary secrets, but HE and other PQC schemes typically use ternary ( $s_i \in \{-1, 0, 1\}$ ) or small ( $|s_i| < k, k$  small) secrets. Second, PICANTE's preprocessing is costly as dimension increases, making it difficult to scale PICANTE to dimensions larger than 350. For  $n = 512$ ,  $\log_2 q = 45$ , PICANTE's preprocessing approach could not finish in a month with full parallelization. Third, PICANTE only experiments with large modulus  $q$ :  $\log_2 q = 23$  for  $n = 256$ ,  $\log_2 q = 27$  for  $n = 300$ , and  $\log_2 q = 32$  for  $n = 350$ . Practical LWE-based systems use small  $q$ : Lizard [21] recommends  $\log_2 q = 10$  for  $n = 608$  with sparse binary secrets, and the HE standard [4] recommends  $\log_2 q = 25$  for  $n = 1024$  with ternary secrets.
Our work, SALSA VERDE, improves on PICANTE and makes the following contributions:
- We introduce a two-bit distinguisher, a new secret recovery technique for sparse binary, ternary and small Gaussian secrets. VERDE fully recovers binary and ternary secrets equally well (§3).
- We improve data preprocessing techniques, making them forty times faster and  $20\%$  more effective, enabling recovery of binary, ternary and small Gaussian secrets for dimension 512 ( $\S 4$ ).
- We decrease the modulus  $q$ , showing VERDE outperforming uSVP attacks (§5 and §A.7).
- We propose NoMod, a framework for understanding the success of ML-based LWE attacks (§5).
- We present a theoretical analysis to show heuristically that successful secret recovery depends only on  $\sqrt{h}$  and the standard deviation of the distribution of the LWE data ( $\S 6$ ).
- We experiment with encoder-only models and compare their success with seq2seq models (§7).
Key results. Our main finding is that small, sparse LWE secrets are weak. For dimension  $n = 256$ , we can recover binary and ternary secrets with  $10\%$  sparsity  $(h / n)$  for  $\log_2 q = 20$ , and with  $3\%$  sparsity when  $\log_2 q = 12$  (Table 1). For  $n = 512$ ,  $\log_2 q = 41$ , we recover binary and ternary secrets with  $\geq 11\%$  sparsity. Furthermore, VERDE scales well to higher dimensions for small sparse secret recovery. Training VERDE models on  $n = 256 / 350 / 512$  problems takes only 1.5/1.6/2.5 hours per epoch, a small proportional increase compared to the increase in  $n$ . Also, we find that VERDE runs faster than uSVP attacks, at the expense of using more compute resources in parallel (see A.7).
In Table 1, we report the timings for successful secret recoveries, for binary/ternary/Gaussian secrets, with varying  $n$  and  $\log_q$ . The table records the highest  $h$  recovered for each column: for binary secrets,
$h$  is the Hamming weight, and for ternary and Gaussian secrets,  $h$  is the number of non-zero entries. We record the amount of time needed for each stage of the attack, preprocessing (in hours/CPU, assuming full parallelization), model training (hours/epoch  $\cdot$  (# epochs)), and total attack time. For full parallelization, the number of CPU cores required is 4 million divided by  $2n$ . Source code and parameters to reproduce our main experiments are included in the supplementary material. The full code base will be open-sourced.
Table 1. VERDE attack times (preprocessing, model training, and total), for dimension  $n$  and  ${\log }_{2}q$  .  $h = \#$  non-zero entries in recovered secrets,  $b =$  binary,  $t =$  ternary,  $g =$  Gaussian secret distributions.
(n, log2q)	(256, 12)	(256, 20)	(350, 21)	(350, 27)	(512, 41)
secret distribution	b	t	g	b	t	g	b	t	g	b	t	g	b	t	g
highest h	8	9	5	33	24	7	12	13	5	36	36	10	63	58	16
preprocessing (hrs/CPU)	1.5	1.5	1.5	7.5	7.5	7.5	16	16	16	216	216	216	840	840	840
training time (hrs)	1.5	3	12	3	7.5	1.5	1.6	25.6	1.6	1.6	17.6	3.2	17.5	27.5	2.5
total time (hrs)	3	4.5	13.5	10.5	15	9	17.6	41.6	17.6	218	234	220	858	868	843
Scope of results. Instances of cryptographic problems like LWE can be broadly categorized as easy (solvable via exhaustive search), medium-to-hard (requiring significant resources to solve), or standardized (believed secure). VERDE attacks medium-to-hard LWE problems (parameterized by dimension  $n$ , Hamming weight  $h$ , modulus  $q$ ). VERDE does not attack toy problems (like Salsa did), nor does it attack the NIST standard directly. Rather, VERDE demonstrates successful attacks on medium-to-hard LWE problems using tools from AI, improving our understanding of the security of proposed LWE-based cryptosystems.
2 SALSA VERDE Overview
In this section we describe the SALSA VERDE attack and relevant parts of its predecessor PICANTE.
Figure 1. Overview of VERDE's attack methodology
High-level overview. Like PICANTE, VERDE starts with  $4n$  LWE samples with the same secret  $s$ . In practice, this data would be eavesdropped. VERDE then proceeds in three stages: preprocessing, model training and secret recovery (see Figure 1). The preprocessing stage augments the  $4n$  initial  $(\mathbf{a}, b)$  pairs to 2 million, then runs lattice reduction to yield a training set of 4 million samples with the same secret  $s$ . The preprocessed data is used to train a transformer to predict  $b$  from  $\mathbf{a}$ . After each training epoch (2 million LWE samples), the model is queried to form a secret guess. The attack succeeds if the secret guess is correct, tested via statistical methods without knowledge of the actual secret. Otherwise, the model is trained for another epoch, and secret recovery is run again.
Data preprocessing. VERDE's preprocessing is similar to PICANTE's, with several improvements. First, we create  $n \times n$  matrices  $\mathbf{A}_i$  by sampling without replacement  $n$  of the  $4n$  original LWE samples. Then, we apply lattice reduction to the matrices  $\mathbf{A}_i$  to reduce the standard deviation of their entries (initially uniformly distributed over  $[0, q)$ ). This process generates  $2n$  preprocessed samples  $(\mathbf{a}', b')$ , with the same secret and is implemented in parallel to create a training set of 4 million samples.
During lattice reduction, PICANTE applies BKZ (as implemented in  $fppll$  [27]) to the  $2n \times 2n$  matrix:  $\mathbf{\Lambda}_i = \left[ \begin{array}{cc}\omega \cdot \mathbf{I}_n & \mathbf{A}_i\\ 0 & q\cdot \mathbf{I}_n \end{array} \right]$ . BKZ finds a linear transformation  $[\mathbf{R}_i\quad \mathbf{C}_i]$  such that the norms of the  $2n$  rows of  $[\mathbf{R}_i\quad \mathbf{C}_i]$ $\boldsymbol{\Lambda}_{i} = [\boldsymbol {\omega}\cdot \mathbf{R}_{i}\quad \mathbf{R}_{i}\mathbf{A}_{i} + q\cdot \mathbf{C}_{i}]$  are small. Applying  $\mathbf{R}_i$  to  $\mathbf{A}_i$  and  $\mathbf{b}_i$ , PICANTE generates  $2n$  reduced LWE pairs  $(\mathbf{R}_i\mathbf{A}_i,\mathbf{R}_i\mathbf{b}_i)$  (modulo  $q$ ). VERDE instead rearranges the rows of  $\mathbf{\Lambda}_i$  and applies lattice reduction to  $\boldsymbol{\Lambda}'_i = \left[ \begin{array}{cc}0 & q\cdot \mathbf{I}_n\\ \omega \cdot \mathbf{I}_n & \mathbf{A}_i \end{array} \right]$ . This reduces the number of operations needed for lattice reduction and allows BKZ to run with lower floating point precision. These two improvements cut the preprocessing time significantly.
They also result in smaller  $\mathbf{R}_i$  norms (smaller error in  $\mathbf{R}_i\mathbf{b}_i$ ), which can be leveraged to further reduce the norm of  $\mathbf{R}_i\mathbf{A}_i + q \cdot \mathbf{C}_i$  by using a lower  $\omega$  parameter ( $\omega = 10$ ) than PICANTE ( $\omega = 15$ ). Finally, VERDE replaces BKZ with two interleaved algorithms (BKZ 2.0 [19] and the efficient reduction technique introduced in [15]), adaptively increases the blocksize and precision as reduction progresses, and introduces a stopping condition. For instance, for  $n = 256$  and  $\log_2q = 20$ , VERDE's preprocessing is  $45\times$  faster, while improving the quality of the reduction by  $20\%$  (Table 2).
Table 2. Impact of successive improvements to preprocessing.  $n = 256$ ,  $\log_2 q = 20$ . Reduction factor of the standard deviation of entries of  $\mathbf{A}_i$ : lower is better. Time: # of hours to preprocess one matrix on one CPU.
preprocessing technique	reduction factor	time (hrs/CPU)
PICANTE	0.53	338
+ reordered Λi rows	0.47	136
+ reduced floating point precision	0.47	24
+ reduced parameter ω from 15 to 10	0.43	38
VERDE (+ interleaved reduction, adaptive blocksize, early stopping)	0.43	7.5
Quality of reduction is measured by the standard deviation of the entries of  $\mathbf{A}_i$ . Preprocessing time in Table 2 is the hours needed to process one matrix on a single CPU. We list the time for all  $n, q$  attempted in Tables 5 and 19. For each dimension  $n$  and modulus  $q$ , we process 2 million/n matrices in parallel across hundreds of CPUs, see §A.1 for details. Better preprocessing allows VERDE to scale to larger dimensions and retrieve secrets with larger  $h$ .
Transformer training. The 4 million reduced LWE pairs are used to train a transformer to predict  $b$  from a. The values  $b$  and the coordinates of a are integers in  $[0,q)$ . They are represented in base  $B = \lceil q / 8\rceil$  (for  $\log_2q > 30$ ,  $B = \lceil q / 16\rceil$ ) and encoded as sequences of two tokens over a vocabulary of 2,000 (see §A.1 for a discussion of these choices). Model training is framed as a translation task, from a sequence of  $2n$  tokens representing a to a sequence of 2 tokens representing  $b$  (see [40, 12] for similar uses of transformers for mathematical calculations). The model is trained to minimize the cross-entropy between model prediction and the sequence of tokens representing  $b$ , using the Adam optimizer with warmup [39] and a learning rate of  $10^{-5}$ . For  $n = 256, 350$  and 512, each epoch uses 2 million LWE samples and runs for 1.5, 1.6, or 2.5 hours. Time/epoch doesn't vary with  $q$  or secret type. Our models train on one NVIDIA V100 32GB GPU and often succeed in the first epoch for low  $h$ . Number of epochs required are included in many tables throughout, including Tables 3 and 4.
VERDE uses the same architecture as PICANTE: a sequence to sequence (seq2seq) transformer [57], with a one-layer encoder (dimension 1024, 4 attention heads), and a 9-layer decoder (dimension 512, 4 heads). The last 8 layers of the decoder are shared (i.e. they form a Universal Transformer [25]). Iteration through shared loops is controlled by the copy-gate mechanism introduced in [22].
Seq2seq models allow output sequences to be longer than inputs, a useful feature for machine translation but not necessary in our setting. For comparison, we also implement a simpler encoder-only transformer, that is 4-layer BERT-like (dimension 512, 4 heads), together with rotary word embeddings (analogous to the rotary position embeddings [55]) to account for the modular nature of the problem. On top of this, we also add an earth mover's distance (EMD) auxiliary objective. We compare this model's performance to that of the seq2seq model ( $\S 7$ ).
Secret recovery. Secret recovery runs after each epoch (2 million LWE samples). PICANTE used three recovery methods: direct recovery, cross-attention and distinguisher. Direct recovery struggles as dimension increases, because it relies on accurate model evaluations at special  $\mathbf{a}$  values which are out of distribution for the training set. Cross-attention is incompatible with encoder-only architectures and is consistently outperformed by the distinguisher on VERDE-preprocessed data. Thus, VERDE only uses the distinguisher, which works as follows: for any test vector  $\mathbf{a}_{\mathrm{test}}$  and random  $K$ , if the  $i$ -th entry of the secret  $s_i$  is zero, then  $(\mathbf{a}_{\mathrm{test}} + K\mathbf{e}_i)\cdot \mathbf{s} = \mathbf{a}_{\mathrm{test}}\cdot \mathbf{s}$  (where  $\mathbf{e}_i$  is the  $i$ -th standard basis vector). Therefore, for each  $i$ , the distinguisher computes the difference between model predictions on  $\mathbf{a}_{\mathrm{test}}$  and  $\mathbf{a}_{\mathrm{test}} + K\mathbf{e}_i$  for different  $\mathbf{a}_{\mathrm{test}}$ . If differences are small, the corresponding  $s_i$  is likely zero. For ternary and Gaussian secrets, the distinguisher is modified (see §3).
For successful secret recovery, the trained model must generalize well to  $\{\mathbf{a}_{\mathrm{test}}\}$ , the vectors used for testing. In PICANTE, the distinguisher runs on random  $\mathbf{a}_{\mathrm{test}}$ , with coordinates uniform in  $[0,q)$ . However, the model is trained on preprocessed  $\mathbf{a}_{\mathrm{train}}$ , with a non-uniform coordinate distribution. So PICANTE's distinguisher recovery requires that the model generalize outside its training distribution.
This is known to be a difficult ML task. Instead, VERDE runs the distinguisher on a held-out subset of 128 preprocessed vectors  $\mathbf{a}_{\mathrm{test}}$ . Since the test vectors have the same distribution as the training set, the trained model only needs to generalize in-distribution, a much easier task. This change in  $\mathbf{a}_{\mathrm{test}}$  improves the performance of the distinguisher (see Table 14 in §A.4).
In practice, for each secret coordinate, the distinguisher computes the sum of the absolute difference between model predictions at  $\mathbf{a}_{\mathrm{test}}$  and  $\mathbf{a}_{\mathrm{test}} + K\mathbf{e}_i$ , for 128 vectors  $\mathbf{a}_{\mathrm{test}}$  and a random  $K \in (0.3q, 0.7q)$  for each  $\mathbf{a}_{\mathrm{test}}$ . The model makes a secret prediction  $\mathbf{s}'$  by setting the bits to 1 on the  $h$  coordinates with the largest sums, and verifies  $\mathbf{s}'$  by computing  $\mathbf{a} \cdot \mathbf{s}' - b$  on the original  $4n$  LWE samples. If the secret is correctly predicted, this quantity should always be small. The statistical properties of this test are discussed in section A.2 of [42].
Having discussed VERDE's background and described its methodology, we now present VERDE's key results, as summarized in §1. We begin with results on ternary and narrow Gaussian secrets.
3 Secret Distributions
One of VERDE's major contributions is a method to recover secrets from sparse ternary and narrow Gaussian distributions, both of which are being considered for use in real-world cryptosystems [4]. Prior ML-based LWE attacks (PICANTE and SALSA) only recovered sparse binary secrets. Here, we describe VERDE's method for recovering these more general secret distributions, and its performance on ternary and small Gaussian secrets for fixed dimension  $n = 256$  and  $\log_2 q = 20$ . Throughout this paper,  $h$  denotes the number of nonzero entries in a secret, which is equal to the Hamming weight in the binary case. We define secret sparsity as the percentage of nonzero secret entries  $h / n$ .
Recovering ternary and small Gaussian secrets. In lattice-based cryptography, ternary secrets are vectors  $\mathbf{s}$  of dimension  $n$ , with entries equal to 0, 1, or  $-1$  (with equal probability of 1 and  $-1$  in sparse secrets). Gaussian secrets are vectors  $\mathbf{s}$  of dimension  $n$ , with entries drawn from a Gaussian distribution with small standard deviation  $\sigma$ . In this paper, we use  $\sigma = 3$ . The Homomorphic Encryption Standard [4] includes secure parameter choices for Gaussian secrets with  $\sigma = 3.2$ .
Ternary and Gaussian secrets introduce new challenges for ML-based attacks on LWE. The recovery methods in PICANTE distinguish zero from nonzero bits. In the binary case, this produces one guess  $\mathbf{s}_{\mathrm{guess}}$ , which can be verified by checking that  $b - \mathbf{a} \cdot \mathbf{s}_{\mathrm{guess}}$  is small. In the ternary case, PICANTE would produce  $2^h$  secret guesses (about  $12^h$  guesses for Gaussian secrets), due to the additional  $-1$  entries, and verification becomes very expensive as  $h$  increases. VERDE recovers ternary secrets using a two-step approach: first, partial recovery distinguishes zero from nonzero entries, then full recovery guesses the sign of the nonzero bits.
Partial recovery. To identify nonzero bits, VERDE uses the binary secret distinguisher from §2 (after [42, Section 4.3]). For each secret bit, it computes a score from a sample of reduced LWE pairs. The  $h$  bits with the highest scores are candidate nonzero bits. VERDE assumes  $h$  is not known and runs the next step for all reasonable possible values of  $h$ , e.g. from 1 to  $n / 20$ . Partial recovery alone is a major contribution, as we are not aware of existing attacks that can identify nonzero secret bits.
Full recovery (ternary secrets). To determine whether nonzero bits of a ternary secret are 1 or  $-1$ , VERDE introduces a novel two-bit distinguisher, leveraging the following observation. If two nonzero secret bits  $s_i$  and  $s_j$  are equal, then for any  $\mathbf{a}$ , exchanging the coordinates  $a_i$  and  $a_j$  will result in the same  $b$ , and corresponding model predictions will be close. Otherwise, model predictions will be different (if  $s_i \neq s_j$ ). Similarly, if  $s_i = s_j$ , for any  $\mathbf{a}$  and  $c \neq 0$ , changing  $a_i \rightarrow a_i + c$  and  $a_j \rightarrow a_j - c$  yields the same  $b$ , and close model predictions. The two-bit distinguisher uses these techniques to compare each nonzero bit with all others, therefore defining two classes of nonzero bits. Letting one class of bits be 1 or  $-1$ , VERDE produces two secret guesses to be verified.
Full recovery (Gaussian secrets). At present, we implemented full recovery only for binary and ternary secrets. However, full recovery of small Gaussian secrets is possible via the following adaptation of the two-bit distinguisher. The two-bit distinguisher groups nonzero secret bits into  $k$  classes believed to have the same value. In our case, the nonzero bits follow a Gaussian distribution with  $\sigma = 3$ , so we may safely assume that all non-zero secret bits are in  $[-9, 9]$  (within 3 standard deviations) – i.e.  $k = 18$ . Since the secret is Gaussian, we expect the largest classes to correspond to the values  $-1$  and 1, followed by  $-2$  and 2, and so on. Therefore, we can intelligently assign values to classes based on class size, and test the corresponding  $2^{k/2} = 512$  secrets. We leave