# RESTful API Testing Methodologies: Rationale, Challenges, and Solution Directions
## 5 Discussion and Threats to Validity
### General Discussion  
In this SLR, we systematically reviewed the available literature on RESTful APIs' testing methodologies. This study resulted in providing the challenges, problems, and available solutions and approaches associated with the testing domain for RESTful APIs. The presented results intend to classify and categorize the available solutions. The classification is based on the approach being used. This laid forth the path for further work on those approaches and the implementation of further tools based on the presented models and frameworks. One of the core aspects of this review is to find out solutions addressing authentication when it comes to RESTful API testing. This is because most of the RESTful API endpoints are protected and, if not covered by auto-generation of unit tests, code coverage can be affected. This would result in insufficient and unreliable testing and code coverage. We came across no complete solution that falls into the autogenetic unit test-based testing category, which covers authentication, which means that there is a lot of dependence on manual testing, which always has a probability of low code coverage. During the analysis, we found that the selected studies use a common approach to look into the description document of the REST APIs for unit test generation. The open API (a.k.a., swagger) format is one of the most commonly followed formats for describing REST APIs. Because of this, and the fact that authenticated REST APIs endpoints are not covered by any of the approaches, the studies are understood to have the same level of performance for code coverage. Fault detection, on the other hand, is best achieved by RESTTESTGEN: Automated Black-Box Testing of RESTful APIs [32]. The study has tested some of the well-known publicly available REST APIs. The statistics are shown to draw the conclusion clearly. There are many situations in which REST APIs are hosted on-premise and in the cloud, but these are not accessible publicly because they are used for internal systems. On the other hand, several instances of REST APIs are publicly available in terms of use. In either case, access to the source code is usually not available. This is one of the common reasons that black-box testing becomes a viable option. The advantage of black-box testing is that the proposed solution can easily generate and run unit tests by just analyzing the description of REST APIs. The disadvantage is that the exact location or reason for the fault would not be easy to find out, especially when the implementation hides the actual cause of the error in the output. White-box testing has a clear advantage in terms of finding the actual cause and location of the error or bug. However, access to the source code is a must for this kind of testing. The code coverage aspect remains the same in both testing methodologies. One of the main obstacles to achieving a high level of code coverage is to test authenticated REST APIs. A proposed solution can achieve a high level of code coverage if it is able to invoke and call all or max REST APIs' endpoints, using either black-box or white-box testing. There are also additional techniques that can improve search-based test case generation for testing RESTful web services and aim at fully automating the evaluation of testing approaches. [PERSON] et al. [36] proposed a catalog of 10 test coverage criteria for RESTful APIs and a framework for the testing approach evaluation. They used two real-world APIs for the evaluation of their proposal and demonstrated that the coverage levels correlate with measurements of fault detection. [PERSON] [37] investigated how t-wise combinatorial testing works for REST APIs and reported that 1-wise, 2-wise, and 3-wise interaction testing techniques of REST APIs detect the same faults when mutation testing is executed on the test suite. Mutation testing was used to inject faults into the system under test. [PERSON] et al.