# A Survey on Adversarial Perturbations and Attacks on CAPTCHAs
## 6 Observation Based on the Analysed Studies  
Based on the reviewed studies, some observations have been noted and summarized as follows:  
* Although new adversarial CAPTCHA generation algorithms have recently been introduced, researchers have not evaluated the robustness of the generated samples against the new sophisticated Google OCR [44]. Thus, it might be interesting to evaluate the robustness of these samples. However, this is out out of the scope of our study and can be a future work.
* The sample size used for evaluating the usability aspects was generally too small. Therefore, it is recommended that the sample size should be taken into consideration when conducting a usability study for developing adversarial CAPTCHAs. Furthermore, since there have been few scientific studies supporting a systematic design or tuning for users, a parametric study may be an interesting focus of future research in analysing adversarial CAPTCHAs at the parameter level by accompanying an experimental study.
* CAPTCHA-solving services that employ human users (e.g., [45]) are still an attack to adversarial CAPTCHAs, since CAPTCHAs are designed to be recognized by humans. Despite the fact that a recent study in [2] proposed a CAPTCHA system against human-based attacks by exploiting the keystroke dynamics authentication system, this should be investigated in the context of adversarial CAPTCHA schemes.  
Figure 3: The sample size used in each study to date: [3; 4; 5; 15; 17; 18; 37; 38].  
There have been few methodologies for generating adversarial CAPTCHAs. Hence, there are still various methods for generating adversarial examples that have not been used yet. These methods are highlighted in the following section.
* The DL algorithms that are most commonly used for breaking CAPTCHAs are the RNN and CNN. Thus, it may be interesting to evaluate other DL algorithms' performance in order to identify the best algorithm.
* The most used domain for adding perturbation is the space domain. However, it appears that this domain is more susceptible to perturbation removal methods. On the other hand, perturbations added in the frequency domain are difficult to remove. This may encourage more investigations to be conducted in terms of inconsistency of the transferability property among various ML models.
* We have observed the potential significance of introducing requirements for generating an adversarial CAPTCHA. Accordingly, the recommended requirements for generating adversarial CAPTCHAs are as follows: \(\odot\) Perturbation: The applied noise should be effective in misleading the targeted system at least 98.5% [3]. \(\odot\) Security: The perturbation should be considered as a global change in order to be resistant to any removable algorithms such as filtering or ML algorithms. \(\odot\) Usability: The perturbation should not affect the readability of the generated CAPTCHA. \(\odot\) Scalability: The CAPTCHA generator should be computationally efficient in terms of generating thousands of CAPTCHAs per second. \(\odot\) Repeatability: The generated adversarial CAPTCHAs should not be repeated. \(\odot\) Predictability: The generated adversarial CAPTCHAs should not be predicted.
* We have observed that most, if not all, attacks against adversarial CAPTCHAs are accomplished based on real CAPTCHA samples downloaded directly from the system. As such, it is highly recommended to investigate a new protection approach, not only for the CAPTCHA itself but also against downloading the generated sample.