# A Methodical Approach to Functional Exploratory Testing for Embedded Systems
## 5 Results  
As a result of nine ET scenario executions, six firmware defects were found, compared with 198 defects found by 983 scripted test cases. This included 361 automated tests run in a continuous integration environment that were responsible for finding five defects in total. Based on an analysis of defects found by exploratory testing, it was concluded that five of those six could not be captured by the test automation or scripted testing being recorded by an ET support tool. However, at the same time they had low occurrence and reproducibility. The severity of those defects met the \"must be fixed\" criteria; therefore, they had to be fixed within the same release they were found in. Figure 2: Exploratory testing framework steps. Based on the track record of executed exploratory testing charters, ten test procedures were created, with one of the charters being split into two test cases. Two of those test cases were introduced as part of the continuous testing scope for subsequent product releases and became automated. One test case was added to the final regression suite, which was executed on release candidate firmware. Effort estimates were accurately reflected during the exploratory testing process, including preparation, execution, and results management. The lightweight approach offered by ET proved to be successful within a rigid schedule. The traditional scripted testing part of the process was suffering from delays caused by underinvested functional requirements and therefore was delayed by over three weeks compared with the estimates provided in Figure 3.  
The summary of the achieved results is presented in Table 1.  
While the exploratory testing scale is much lower than that for scripted testing, and data about the escaped defects are not available, it is still possible to calculate the Test Effectiveness Ratio (TER) coefficient [4] for both the exploratory (1) and scripted testing (2). In principle, they both correspond to the same total product coverage, but were executed in sequential order, with exploratory testing being run in the very late stage of product development. \[TER_{ET}=\frac{\text{Defects Found in Test}}{\text{Total Defects Found}}\times 100\%=\frac{6}{204}\times 100\%\approx 3\% \tag{1}\]  
\[TER_{ST}=\frac{198}{204}\times 100\%\approx 97\% \tag{2}\]  
From the Test Effectiveness Ratio perspective, ET can be thought of as negligible. However, it must be considered that it was executed at the last stages of the development life cycle; therefore, most defects were already found during scripted testing. It still \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{4}{c}{**Test Summary**} \\ \hline
**Type of Testing** & **Test Cases** & **Defects** & **Must be Fixed** & **Weeks** \\ \hline Exploratory Testing & 9 & 6 & 5 & 2 \\ Scripted Testing & 983 & 198 & 79 & 68 \\ \hline Total & 992 & 204 & 84 & 70 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of test results. Figure 3: Original project estimates. managed to detect defects that had significant customer severity aspects and would have otherwise escaped the testing process. The recorded course of actions taken by the test engineers led to developing test procedures to expand the portfolio of available test cases. Some of these have been added to the continuous integration environment or the regression testing suite. The test charter itself can be written in a way to be agnostic to the device under test (DUT). Therefore, it is easily portable between various products that share similar features, but with different kinds of implementation. This makes exploratory testing charters portable and reusable, which is a significant time-saving advantage. In this context, it is extremely valuable to consider the number of tests executed and the time required to test for each testing type [4]. Based on this, exploratory testing provides a comparable, but not as detailed, test coverage, much faster and with fewer test cases that must be developed and maintained.