# On Pseudorandomness and Deep Learning: A Case Study
## 5 Experiments
### Results  
As mentioned, the validation set is evaluated every 10 epochs, with the best model being chosen to be tested on the test set at the end. This experiment was carried out a total of 10 times. The validation and train accuracies for the last run of the experiment using a binary cross-entropy loss defined by Equation (2) on the Speck dataset are shown in Figure 3.  
As shown in Figure 3a, for this run of the experiment with a dataset built using 200 different samples, the validation accuracies range from 50% to 85%, and converge to 85% within the first 600 epochs, therefore choosing the model with higher validation accuracy as the final model for evaluation on the test set, results in a test accuracy of 83%. However, as shown in Figure 3b, with 300 different samples, the Siamese network outperforms the one using 200 samples. Hence, 300 different samples are our choice for building the datasets. The accuracies of 10 runs of the experiment, for 1000 different one-shot tasks on the test set using the binary cross-entropy loss, are computed according to Equation (4) and are provided in Table 1. As indicated, they resulted in an approximate average accuracy of 94% for both the Speck and Simon ciphers. Based on the above findings, going back to our main goal, we can set up the following distinguisher \(D\) for the experiment Pseudo-R\({}_{D,E}\): \begin{tabular}{|l|} \hline Distinguisher \(D\) \\
1. & Builds the dataset as described before and selects the best DL-Model to distinguish (Simon/Speck) from a random permutation. \\
2. & Sends a query \(x\) to the oracle \(\mathcal{O}(\cdot)\), obtaining in response \(y=\mathcal{O}(x)\). \\
3. & Chooses \(y^{\prime}\) uniformly at random. \\
4. & Constructs the pair \((y,y^{\prime})\) and gives it in input to the DL-Model. \\
5. & If the DL-Model finds \(y\) and \(y^{\prime}\) similar, then \(D\) gives in output 0 (i.e., \(\mathcal{O}(\cdot)\) implements a random permutation); otherwise, \(D\) gives in output 1 (i.e., \(\mathcal{O}(\cdot)\) implements Simon or Speck). \\ \hline \end{tabular}   
The distinguisher \(D\), a one-query distinguisher, succeeds with probability:  
\[Pr[\text{Pseudo-R}_{D,E}=1] = Pr(b=0)\cdot Pr(\mathcal{D}\text{ outputs }0|b=0)+\] \[+Pr(b=1)\cdot Pr(\mathcal{D}\text{ outputs }1|b=1)\] \[= \frac{1}{2}\cdot 0.94+\frac{1}{2}\cdot 0.94>\frac{9}{10}.\]  
As a result, according to Equation (1), \(D\) succeeds with an advantage of at least \(\frac{4}{10}\) over a random guess. Hence, both ciphers do not yield a pseudorandom behavior. Note that a dataset of 5000 sequences is considered for the experiment, however, only 300 samples are chosen to create a training dataset of 10,000 elements for the Siamese network, for each experiment run. Considering 10 different runs of the experiment, at least \(t=3000\approx 2^{11}\) steps are required to set up the best DL-Model. A similar conclusion applies to the cases of the contrastive loss function, presented in the following. The experiment was repeated once more for a total of ten times, using a contrastive loss, as defined by Equation (3) with a margin value of 0.3. Table 2 provides the accuracy results. These findings show that a Siamese network with a contrastive loss outperforms the one with cross-entropy loss, and it reaches an average test accuracy of over 99% for both ciphers. Figure 4 shows the train and validation accuracies for the last run of the experiment on the Speck dataset. As it is clear for the first 100 epochs, accuracies are 100% \begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline
**Cipher** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** & **10** & **Avg** \\ \hline Speck\({}_{2}\) & 0.81 & 0.76 & 0.82 & 0.9 & 0.87 & 0.84 & 0.8 & 0.74 & 0.87 & 0.83 & 0.82 \\ Speck\({}_{3}\) & 0.93 & 0.94 & 0.91 & 0.92 & 0.97 & 0.95 & 0.97 & 0.93 & 0.97 & 0.91 & 0.94 \\ \hline [PERSON] & 0.92 & 0.95 & 0.92 & 0.93 & 0.95 & 0.93 & 0.96 & 0.97 & 0.98 & 0.91 & 0.942 \\ \hline \hline \end{tabular}
\end{table}