# Analyzing Error Rates in Cryptological Programming Assignments

Konstantin Knorr

► To cite this version:

Konstantin Knorr. Analyzing Error Rates in Cryptological Programming Assignments. 15th IFIP World Conference on Information Security Education (WISE), Jun 2022, Copenhagen, Denmark. pp.3-16, 10.1007/978-3-031-08172-9_1. hal-05311510

HAL Id: hal-05311510

https://inria.hal.science/hal-05311510v1

Submitted on 13 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Analyzing Error Rates in Cryptological Programming Assignments

Konstantin Knorr(✉)

Trier University of Applied Sciences, Trier, Germany
knorr@hochschule-trier.de

**Abstract.** Understanding cryptological primitives like encryption, hashing, signatures, and certificates is a central skill when working as an IT security professional or software developer but it is also a major educational challenge. The paper presents a study which measures and compares error rates in cryptological programming assignments. Over a ten-week period, 20 students solved 20 cryptological Java programming assignments checked by 350 tests that were automatically verified using a grader system. The error rate in ~60.000 test results is analyzed: Students made fewer errors in substitutions than transposition ciphers, symmetric ciphers rank lower than asymmetric ones, constructor, exception and padding tests appear easier to solve than signing and its verification. Asymmetric encryption has lower error rates than signing. A discussion of the findings, limitations, and possible future improvements concludes the paper. The approach allows identifying and measuring "hard" and "easy" cryptological assignments in order to improve teaching, which is desirable from an educational perspective.

**Keywords:** Cryptology · Error rate · Grader · Java · JUnit Tests · Programming assignment

## 1 Introduction

Implementing cryptological software is subtle, can easily lead to attackable code, and should be done by experienced developers. Notorious problems are side channel attacks for example when implementing RSA or insufficient pseudo randomness. On the other hand, more and more information is processed and needs protection in our digital world. Recent studies underline the problems developers face when using cryptology in their software. Nadi et al. [1] identify the major obstacle as difficult API usage and missing domain and provider knowledge. Hazhirpasand et al. [2] detect a distinct lack of knowledge in understanding the fundamental cryptological concepts. Secure coding guidelines exist [3] but often developers lack a thorough understanding of cryptology. Typical problems include using ECB instead of GCM or CCM mode in symmetric encryption or confusing the different key types in asymmetric schemes.

This paper addresses these shortcomings by letting students code cryptological assignments in Java. Coding cryptology helps students to better understand the underlying theory [4]. A grader automatically assesses the correctness of the code using pre-defined cryptological Java test cases. Examples for tests include the correct encryption of
a plaintext, the verification of a signature, or a thrown exception when using an incorrect key size. By summing up over all uploads the results can then be used to assess the error rates of the test cases, categories of test cases, assignments, and groups of assignments. This allows investigating in detail which parts of the cryptological assignments cause students most problems.

Students are allowed an unlimited number of uploads of their code and are immediately confronted with the results of the test cases. They can improve their code until the deadline of the assignment. As a side effect, the final upload can also be used for grading the assignment (percentage of positive test cases = grade).

The idea of using automatic graders in the classroom is not revolutionary and has raised the interest of many researchers [5–8]. The authors however do not use graders in a cryptological setting. Braga et al. [9] use test-driven software development in the construction of cryptographic software. They argue that test cases can automate acceptance tests for cryptographic software. Cryptographic test cases are considered good acceptance tests as they meet halfway between cryptologists and developers. However, they do not use Java and utilize a non-educational setting.

Edwards et al. [10] investigate static analysis errors like formatting and JavaDoc style and coding flaws in half a million Java programs. They track changes between computer science students compared to other students over four semesters. Rivers et al. [11] use learning curve analysis on students’ Python program to identify problematic programming concepts. They discover that over time learning some concepts like function definitions and comparisons are easier to learn than e.g. binary operations. Sivasakthi and Rajendran [12] use questionnaires to investigate topics causing most problems when learning Java: concurrent programming, UI programming with Swing, and generic programming are hardest, object orientation and fundamental programming structure easiest. Lahtinen et al. [13] study the difficulties of novice programmers in Java, C++, and Pascal based on questionnaires for students and teachers. Finding bugs in the own code and using recursion, pointers, and references are hardest.

Nadi et al. [1] evaluate Stack Overflow posts and GitHub repositories in order to identify security problems Java developers face. Most problematic are the correct use of symmetric encryption as well as signature generation and verification. Also by assessing the stack overflow web site Hazhirpasand et al. [2] detect a distinct lack of knowledge in understanding the fundamental cryptological concepts (e.g. for OpenSSL, public-key cryptography, or password hashing). Additionally, the usability of cryptological libraries undermines developers to correctly realize a cryptological scenario. Lazar et al. [14] confirm this finding: by analyzing the CVE database for cryptographic problems, they show that just 17% of the bugs are in cryptographic libraries, 83% are misuses of cryptographic libraries.

Knorr describes the didactical framework used in this study in [4]. Preliminary results of the study are published in [15]. This paper extends the study presented in [15] and improves the investigation. To the best knowledge of the author, this is the first attempt to measure error rates with Java tests in cryptological programming assignment in an educational setting.

The remainder of the paper has the following structure: Sect. 2 gives the necessary background information on the grader system used, on Java tests, and on programming
cryptology in Java. Section 3 details on the set-up of the study including a formal definition of the error rate. The results of the study are topic of Sect. 4. Finally, a discussion and future work topics conclude the paper in Sect. 5.

# 2 Background

## 2.1 Automatic Software Evaluation System

Automatic assessment and grading of programming assignments have been in the focus of researchers and practitioners for many years, see [5, 6] for an introduction and overview of current systems. The main advantages of these systems are traceability of the programming progress, immediate feedback, and scalability. Especially for larger cohorts, automated grading is a vital tool in study programs. In addition, the approach is well suited for distance learning.

This paper makes use of the existing and long-running ASE (Automatic Software Evaluation) system, which is described in detail in [16]. ASE has been in use since 2006 at Trier University of Applied Sciences and is mainly used for introductory classes in object-oriented programming in Java. Its architecture is a web based client-server system. Shibboleth provides authentication. TLS secures the network traffic. A detailed privacy policy explains the usage of the students' data, see [4] for a more thorough discussion of privacy issues. ASE's kernel comprises web sockets for managing client-server communication and execution environments for selected programming languages like Java, C++, and Python. Each language can load plugins e.g. unit test runners or modules to check programming style. Tests can be configured using XML files. The result of the assessment of the student's code is stored in a database and written in an XML file that is then presented in the student's browser.

ASE allows defining the maximal execution time for each test case, which is especially important for exhaustive operations like asymmetric key generation or encryption. The parameters were chosen and tested prior to the first upload of the students. The waiting time was less than 10 s per upload. Additionally, ASE uses special Java annotations to configure the tests. For example, @TestFailureMsg provides the opportunity to display a specific message in case of a failed test. See [4, 16] for more information. A typical life cycle for ASE assignments is the following:

* The instructor presents the necessary learning matter in class and prepares the corresponding assignments, a sample solution, the test cases, and optionally a code skeleton for the solution. The test cases are uploaded to ASE.

* Students solve the assignments and submit the solution to ASE. With ASE, they can upload an arbitrary number of solutions within the given time frame.

* ASE verifies each submission by executing the test cases on the submitted code and provides immediate feedback which tests are correct or wrong with error messages defined as part of the test cases.

* After the deadline, the last uploaded solution can be used for grading the assignment. The instructor reviews the assignment results, can comment on common errors in the next lecture, and optionally improves the test cases and assignments for future classes.
## 2.2 Java Tests and Cryptology

Test-driven software development stems from Beck [17]. The fundamental idea is to produce only code that passes predefined test cases. It encourages programmers to always have a running version of their code and to test features and code during implementation thereby avoiding integration problems typically encountered in other development models at the end of a project.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** JUnit test cases for Playfair (left), Elgamal (right) ciphers, and their run time in the Eclipse IDE. Test `encryption3` produced a failure, as an incorrect ciphertext was calculated. Test `illegalCiphertext` raised an exception. All Elgamal tests passed.</figcaption></figure>

For Java, JUnit [18] provides a framework for test cases. It allows for testing classes, methods, and exceptions. Major IDEs like Eclipse provide GUIs to visualize JUnit tests and their results (cf. Fig. 1). A JUnit test can have three different results:

* Passed (colored green), e.g. test `decryption1` in Fig. 1.

* Error (colored red): an unexpected exception has been thrown by the code.

* Failure (colored blue): an incorrect result has been calculated for that test.

ASE maps the result “Passed” to true, “Error” and “Failure” to false. Test cases that expect a certain exception adequately address typical cryptological problems like incorrect key size, erroneous padding, and incorrect private or public key usage. The test case `illegalCiphertext` in Fig. 1 e.g. requires a specific exception to be thrown due to an invalid ciphertext (which the code does not do). A detailed discussion of the methodology can be found in [4], the source code of the tests above in [19].
<table><caption>**Table 1.** Assignments checked by ASE including their group and number of test cases</caption><thead><tr><td>Group</td><td>Abbr.</td><td>Assignments and number of test cases (in brackets)</td></tr></thead><tbody><tr><td>Classical substitution ciphers</td><td>SC</td><td>Playfair cipher (20), FourSquare (23), Porta (24)</td></tr><tr><td>Classical transposition ciphers</td><td>TC</td><td>Rail fence (6), Skytale (19), Bifid (24)</td></tr><tr><td>Classical asymmetric ciphers</td><td>CA</td><td>RSA (22), Rabin (23), Elgamal (25)</td></tr><tr><td>Modern asymmetric ciphers</td><td>MA</td><td>Paillier (17), NTRU (15)</td></tr><tr><td>Signature schemes</td><td>SIG</td><td>Rabin signatures (14), Lamport signatures (21)</td></tr><tr><td>Elliptic curves</td><td>EC</td><td>myEC (16), ECDH (14), ECDSA (13)</td></tr><tr><td>Miscellaneous</td><td>MISC</td><td>BigInt (14), Shamir Secret Sharing (22), Java Key Store (10), Padding Oracle (5)</td></tr></tbody></table>

# 3 Setup of the Study

## 3.1 Background and Assignment Selection

The study makes use of data of the elective Bachelor course “Programming Cryptology” in the winter semester 2020/21 at Trier University of Applied Sciences. Twenty students participated in the course, fifteen regularly uploaded assignments in ASE. As a prerequisite students had to visit an introductory class on object-oriented programming and a class in IT security including an introduction to cryptography in previous semesters. All assignments were in Java as Java is presently the main programming language taught to students in the programming education of the computer science department. Of the 14 weeks of the semester, programming assignments covered ten. ASE checked the assignments of seven weeks, see Table 1. In the last four weeks students had to do a project of their choice related to the topics of the class.

Cryptological concepts comprise but are not limited to encryption/decryption, signature/signature verification, hashing, and pseudo random number generation. The topics covered in the course roughly follow standard cryptological textbooks like [20] for the theoretical part, [21] for the historic ciphers, and [22–24] for Java cryptology. The assignments were chosen based on the recommendations given in [3, 14] and selected for didactical or practical purposes. The assignments covered all current cryptology topics like symmetric ciphers, asymmetric schemes, key exchange (Diffie Hellman), hash functions, pseudo random functions, X.509 certificates, elliptic curves and selected post-quantum schemes (NTRU, Lamport Signatures).

In total there were 34 assignments, ASE checked 20 of them, cf. Table 1. In the 20 ASE assignments students had to code cryptographic primitives from scratch (e.g. RSA encryption). The selection includes the cryptological systems covered in the previously visited IT security course and adds new systems in order to guarantee a broad spectrum of number theoretical concepts. The 20 ASE assignments all follow a similar structure: they require the definition of the Java class, constructor, and cryptological functions following the categories in Table 2. The same person wrote the assignments. A sample can be found in [4, 19].
The remaining 14 assignments were manually checked based on the source code and videos produced by the students. They addressed the correct use of established libraries. Examples for these assignments are the development of a password safe using AES and a command line signature application using ECDSA and X.509 certificates. This split was done to balance between finding difficulties in cryptological assignments on one hand and teaching students not to code their own cryptological code on the other hand. Programmers should use existing and well-established code like OpenSSL or Bouncy Castle [24] for real life scenarios.

<table><caption>**Table 2.** Categories of test cases and its numbers</caption><thead><tr><td>Category and number of test cases</td><td>Explanation</td></tr></thead><tbody><tr><td>Encr (34)</td><td>Encrypt a plaintext with a given cipher and key</td></tr><tr><td>Decr (29)</td><td>Decrypt a ciphertext with a given cipher and key</td></tr><tr><td>EncDec (37)</td><td>Encryption and subsequent decryption using a given and a possibly randomly generated plaintext. Check if result equals the original input</td></tr><tr><td>Sig (8)</td><td>Sign a given plaintext with a given key and signature scheme</td></tr><tr><td>Ver (13)</td><td>Given a signature, public key, and plaintext: verify with the public key that the signature is valid</td></tr><tr><td>SigVer (11)</td><td>Given a signature scheme and possibly random plaintext: Check if the verification of the foremost-generated signature is valid</td></tr><tr><td>Pad (3)</td><td>Check if padding is correctly used e.g. added and removed after use</td></tr><tr><td>Constr (73)</td><td>Check if the constructor works as required in the assignments. Constructors are the basis for cryptological operations in the assignment. The constructor e.g. sets and checks plaintexts, ciphertexts and key(s)</td></tr><tr><td>Excep (52)</td><td>Any test case used for testing exceptions like incorrect key length, unpadded plaintext, or missing private key for signature generation</td></tr><tr><td>Misc (100)</td><td>All remaining test cases not fitting in the above categories. Examples are test cases for the assignment in the Miscellaneous group from Table 1</td></tr></tbody></table>

## 3.2 Formalization

ASE stores the following data per assignment: student ID, name of the test case, time *t* of upload, and the result *r* of the test case. Example: *r* (Rabin, 6137, testEncryption2,
2020-10-29T13:21:35) = true. Formalized:

$$r : A \times S \times T \times Z \rightarrow \{true, false\}$$

$$group : A \rightarrow \{SC, TC, CA, MA, SIG, EC MISC\}$$

$$cat : T \rightarrow \{Encr, Decr, ..., Misc\}$$

* A represents the assignment. The group function clusters assignments into groups, e.g. group(Rabin) = "Classical asymmetric cipher" = CA. Table 1 gives the detailed mapping.

* S stands for the student ID of the participating students, a random number generated by ASE for each student guaranteeing his anonymity.

* T is the set of names of the test cases, typically 10–20 per assignment, and 350 in total. The function cat and Table 2 map the test cases to categories, e.g. cat("testEncryption2") = ENC.

* Z represents the time of the upload. Students had seven days (Wednesday to Tuesday) for solving the assignment, typically 2–3 per week. They could decide when and how often to upload. The number of uploads varied between 2 and 50.

With $|A| = 20$, $|S| = 20$, $|T| = 350$, and the number of uploads ranging from 2–50 there are ~60,000 results in the study. The error rate e measures the percentage of negative test results per assignment, student, and test case, e.g. it divides the number of negative test results by the total number of tests. Example: The results {false, false, false, false, true} for test case testEncryption2 of the Rabin assignment by student 6137 yield an error rate e(Rabin, 6137, testEncryption2) = 0.8. Formalized:

$$e : A \times S \times T \rightarrow [0, 1]$$

Note that the number of results varies per test case and that if the student did not turn in solutions for this assignment, the error rate is undefined. By averaging over the sets A, S, and T using the functions cat and group this setup allows analyzing the following aspects related to solving cryptological programming assignments:

* Error rate per day and hour over all students and assignments.

* Compare error rates within a single assignment in order to identify “easy” and “hard” test cases within this assignment.

* Compare error rates between the assignments or groups of assignments (cf. Table 1) in order to identify “easy” or “hard” assignments or groups.

* Compare error rates between different test cases and their categories, cf. Table 2.

Other aspects like student specific error rates are excluded due to privacy reasons. ASE provides the data in Tab Separated Value format. These files were analyzed with Python. Matplotlib (https:// matplotlib.org) generated the figures. The data files, python scripts, a sample assignment, and sample JUnit tests are available for download, s. [19].
# 4 Results

Figure 2 gives the total and average number of uploads per assignment. It does not take the results of the test cases into account. However, the number of uploads can be seen as a measure for the difficulty the students had with the assignments. The Shamir, Keystore, and Rabin assignment had most tries in total and in average. Porta has the smallest numbers. The differences in Fig. 2 a) and b) are due to the fact that the number of students working on each assignment varied.

The average error rate over all results is 0.297. A dashed line indicates this benchmark in the following figures. Figure 3 gives the total number of true and false results per day. The two days with most correct tests in total were 2020-10-28 and 2020-10-29 due to the popular asymmetric RSA and Rabin assignments. The graph also shows days with zero uploads (gaps in time line). The corresponding assignments were checked manually, not with the help of ASE. Figure 4 shows the error rate per day. The highest peaks are in from 2020-10-23 to 2020-10-27 due the first attempts to solve the Rabin and RSA assignments.

Function *e* can be used to assess the test cases in the different assignments. The Rabin encryption assignment serves for illustration purposes here, see Fig. 5. The encryption and constructor tests have a lower error rate than 1. decryption, 2. encryption with subsequent decryption tests, and 3. tests addressing the Chinese Remainder Theorem (CRT). This is due to the definition of the Rabin cipher: encryption is a simple modular squaring: decryption requires the more complex calculation of modular square roots with the help of the CRT, see [25].

<figure><img src="image_4.png" /><figcaption>Fig. 2. a) Total number of uploads per assignment (left) and b) average number of uploads per assignment and student (right)</figcaption></figure>

Figure 6 shows *e* per assignment. The Padding Oracle assignment [26] has the highest error rate. It is an especially challenging bonus assignment. Solving the challenge only required solving two test cases. The classical symmetric substitution ciphers FourSquare and Porta show low error rate (e < 0.2). The transposition ciphers Rail fence and Sky-tale have average rates (e ≈ 0.3). Playfair was the first assignment in the semester. Most students needed to become accustomed with ASE. The asymmetric encryption assignments RSA, Elgamal, and Rabin have lower error rates than (1) signature related
<figure><img src="image_5.png" /><figcaption>Fig. 3. Total number of uploads per day</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4. Error rate per day (only days with uploads > 0)</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** a) True and false test results (left), b) error rates per test case in the Rabin assignment</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Fig. 6.** Error rates per assignment</figcaption></figure>

assignments (RabinSig, Lamport) and (2) modern asymmetric assignment like Paillier und NTRU. This could be related to the increasing number-theoretical complexity of the newer schemes. Elliptic curve assignments rank closely below average. The high error rate of the Keystore and BigInteger assignment is mentionable: Java Keystores are used to store cryptological keys and certificates. The difficulty lies in the subtle changes in keystore types and the complexity in accessing these stores. Thedesigned assignment is an introduction to working with large numbers (e.g. 4096 bit modules for RSA) in Java
<figure><img src="image_9.png" /><figcaption>Fig. 7. a) Error rates per test case category (left) and b) per assignment group (right)</figcaption></figure>

including modular exponentiation and inversion. The test cases were all independent of each other. Most students solved them chronologically.

Figure 7 a) sums up the results for test case categories. Testing constructors and exceptions has the lowest error rates. Padding, encryption, and decryption rank in mid-field. The categories *EncDec* and *SigVer*, which combine two cryptological operations and are often based on a dynamically randomly generated input have the highest ($e > 0.6$) and second highest error rate. Encryption scores lower than decryption, signing has a lower error rate that verifying a signature. Protecting the confidentiality of data via encryption and decryption appears easier than protecting the integrity of data by signing and verifying the signature. Figure 7 b) addresses assignment groups: Classical substitution ciphers (SC) have a lower error rate than classical transposition ciphers (TC). Modern asymmetric schemes (MA) like NTRU are more error prone than classical schemes (CA) like RSA. Signature schemes (SIG) have an above average error rate, classical asymmetric encryption schemes rank below average. Elliptic curves seem accessible to students. Generally, an increase in the complexity of the cryptological schemes regarding the underlying algebraic structures and operation yields a higher error rate.

# 5 Discussion and Conclusion

The paper presents an approach to identify easy and hard tasks when coding cryptological assignments in Java by assessing the error rate in corresponding JUnit test cases. This approach allows identifying difficult tasks in a more focused way than in many other studies like [1, 2, 12], which assess programming problems using more general approaches like questionnaires. The current study is a first step towards making problems in cryptological programming assignments measurable. This allows for improving the education in teaching programming and cryptology classes in the future.

Coding cryptology is beneficial for understanding and learning cryptology. Only when the concepts are fully understood, it can be implemented correctly and test cases will pass. Sticking to the test cases also helps to split a large assignment into smaller, more conquerable parts. The test cases have been valuable in showing the students their errors. The error rates are a transparent and fair basis for grading assignments. For instructors the approach represents a major time shift from correcting students’ code ex
post to creating sample solutions, test cases, and detailed assignments prior to classes. This approach is scalable and can therefore deal with larger cohorts e.g. in distance learning scenarios. On the downside, the following issues need to be taken into account:

*   **Chronology of assignments:** As students needed to become acquainted with ASE, this can affect the error rates in the first weeks.

*   **Order and dependency of test cases:** ASE tests each upload against all test cases of an assignment and always presents the test results in the same predefined order. Students might tend to solve the test cases in this order. There are also interdependencies between test cases: Constructor tests are often a prerequisite for other operations like encryption or signing. The *EncDec* category requires working implementations of encryption and decryption. The *SigVer* category requires that the signature can be calculated and correctly verified. Padding is often a prerequisite for encryption. The categories used are not all disjunct, e.g. encryption test cases can also raise an exception. Ambiguous test cases have been mapped to the *Misc* category. Labels for the test cases instead of a strict categorization could be used.

*   **Solving time:** Instead of assessing the error rate, the solving time of the first successful test can be analyzed. This has been done in [15] with no significant difference in the results.

*   **Complexity vs. error rate:** The assignments, the test cases, and their underlying cryptology have different complexities. It is e.g. much easier to understand the Skytale than NTRU. Effenberger et al. [27] differentiate between difficulty and complexity of programming assignments. For measuring the difficulty, they recommend the error rate and solving time as done in this study. Measuring the complexity is much harder. The authors recommend the number of line of codes and the type of control flow structure in the reference solution as indicators.

*   **Number of uploads and immediate feedback:** ASE allows for an arbitrary number of uploads per assignment and checks each upload against all the test cases for this assignment. Students receive their result for the test cases immediately. This is inherent to the current didactical concept of ASE: students can learn from their errors and improve their code until the deadline. Contrary, this produces a certain bias towards passed tests.

*   **Plagiarism:** Many assignments have been very challenging and time consuming for the students. They had to solve the assignments individually. Group work was not allowed. As the grade of the course comprised the final error rate per assignment, some students might have been stimulated to share their solutions and cooperate in solving the assignment. ASE does not automatically check uploads for plagiarism. This needs to be done manually based on the uploaded source code.

Besides mitigating the above shortcomings of the study, ASE can be improved in several directions: An automatic feedback for instructors similar to Fig. 5 would help to identify difficult test cases faster. In addition, Python could replace Java as the underlying programming language. Several cryptological primitives like e.g. X.509 certificates are easier to handle in Python due to a more modern architecture for cryptology. The list of ASE assignments can be extended to address more topics, especially those that are
currently still manually checked. These are e.g. assignments addressing modern sym-
metric block ciphers like AES, stream ciphers like ChaCha20, key derivation functions,
and hash functions.

The cryptological world is currently facing tremendous changes. Quantum computers will break most current asymmetric schemes like RSA and ECDSA. So-called post quantum schemes like Rainbow, NTRU, and McEliece are currently specified and elected as future quantum-resistant schemes [28]. Most of these post quantum schemes make use of different algebraic structures like lattices, multi variate polynomials, and error correcting codes instead of modular rings or elliptic curves. These new schemes should be the content of future studies. The NTRU assignment shows first results in the current study ($e \approx 0.4$) in this direction.

References

1. Nadi, S., et al.: Jumping through hoops: why do java developers struggle with cryptography APIs? In: 2016 IEEE/ACM 38th International Conf. on Software Engineering (ICSE), pp. 935–946 (2016)

2. Hazhirpasand, M., et al.: Hurdles for developers in cryptography. In: 37th International Conference on Software Maintenance and Evolution (ICSME), Luxembourg (2021)

3. Long, F., et al.: The CERT Oracle Secure Coding Standard for Java. Addison-Wesley, Boston (2011)

4. Knorr, K.: Learning and grading cryptology via automated test driven software development. In: 13th IFIP WG 11.8 World Conference on Information Security Education (WISE), Maribor, Slovenia, 21–23 September, pp. 3–17 (2022)

5. Desai, C., Janzen, D., Savage, K.: A survey of evidence for test-driven development in academia. ACM SIGCSE Bull. **40**(2), 97–101 (2008)

6. Edwards, S., Pérez-Quiñones, M.: Experiences using test-driven development with an automated grader. J. Comput. Sci. Coll. **22**(3), 44–50 (2007)

7. Isong, J.: Developing an automated program checker. J. Comput. Small Coll. **16**(3), 218–224 (2001)

8. Krusche, S., Seitz, A.: ArTEMiS - an automatic assessment management system for interactive learning. SIGCSE 2018, 21–24 February, Baltimore, MD, USA, pp. 284–289 (2018)

9. Braga, A., Schwab, D., Vannucci, A.: The use of acceptance test-driven development in the construction of cryptographic software. In: 9th International Conference on Emerging Security Information, Systems and Technologies (2015)

10. Edwards, S., et al.: Investigating static analysis errors in student java programs. In: ACM Conference on International Computing Education Research (ICER), Tacoma, pp. 65–73 (2017)

11. Rivers, K., Hardstead, E., Koedinger, K.: Learning curve analysis for programming: which concepts do students struggle with? In: 2016 ACM Conference on International Computing Education Research (ICER), pp. 143–151 (2016)

12. Sivasakthi, M., Rajendran, R.: Learning difficulties of object-oriented programming paradigm using java: students’ perspective. Indian J. Sci. Technol. **4**(8), 983–985 (2011)

13. Lahtinen, E., Ala-Mutka, K., Järvinen, H.: A study of the difficulties of novice programmers. ACM SIGCSE Bull. **37**(3), 14–18 (2005)

14. Lazar, D., et al.: Why does cryptographic software fail? A case study and open problems. In: 5th Asia-Pacific Workshop on Systems, pp. 1–7 (2014)

15. Knorr, K.: Messung der Schwierigkeit von Programmieraufgaben zur Kryptologie in Java. In: Fünfter Workshop Automatische Bewertung von Programmieraufgaben, S, pp. 35–42 (2021)
16. Herres, B., Oechsle, R., Schuster, D.: Der Grader ASB. In: Herausgeber Oliver, J., et al. (ed). Automatisierte Bewertung in der Programmierausbildung, Waxmann-Verlag, pp. 255–271 (2017)

17. Beck, K.: Test-Driven Development: By Example. Addison Wesley, Boston (2002)

18. JUnit Homepage. https://junit.org. Accessed 29 Jan 2022

19. Knorr, K.: Data files and code of the study: https://seafile.rlp.net/d/a22a20689ca1464abd79/. Accessed 27 Mar 2022

20. Stinson, D., Paterson, M.: Cryptography: Theory and Practice. CRC, Boca Raton (2018)

21. Gardner, M.: Codes, Ciphers and Secret Writing. Dover Publications, New York (1984)

22. Hook, D., Eaves, J.: Java Cryptography: Tools and Techniques, eBook (2022). https://leanpub.com/javacryptotoolsandtech

23. Weiss, J.: Java Cryptography Extensions. Morgan Kaufmann, Burlington (2004)

24. Bouncy Castle Homepage. https://www.bouncycastle.org. Accessed 12 Jan 2022

25. Rabin, M.: Digitalized Signatures and Public-Key Functions as Intractable as Factorization. MIT Laboratory for Computer Science (1979)

26. Vaudenay, S.: Security flaws induced by CBC padding — Applications to SSL, IPSEC, WTLS... In: Knudsen, L.R. (ed.) EUROCRYPT 2002. LNCS, vol. 2332, pp. 534–545. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46035-7_35

27. Effenberger, T., Čechák, J., Pelánek, R.: Measuring difficulty of introductory programming tasks. In: 6th ACM Conference on Learning@Scale, pp. 1–4 (2019)

28. NIST Homepage for Post-Quantum Cryptography, https://csrc.nist.gov/Projects/Post-Quantum-Cryptography. Accessed 9 Feb 2022