# A Methodical Approach to Functional Exploratory Testing for Embedded Systems
## 1 Introduction  
Testing is intended to verify and validate a product and is required from the early stages of manufacturing. The earlier a product (and its components) is tested, the cheaper the overall process is, including any changes required to achieve the expected functionality. Agile software development methods distribute the testing effort evenly across iterations and focus on unit testing and verification automation [1]. The complexity of the testing process means that there are a wide range of potential techniques to use. One variation is functional testing, known as black-box testing [2]. Generally, the tester does not know the construction of the program being tested, while tests are used to detect errors in the implementation of the functionality from the requirements specification. In some cases, testing is based on the abstract model, which is known as gray-box testing [2]. Test cases are generated by exploring the search space of the model, but it requires extra effort to create the model. Functional testing requires a special set of metrics to be gathered in order to evaluate the outcome [3; 4; 5].  
The main problem with testing is the complexity of systems, including embedded systems. They are extremely complex as they perform safety-critical functions and require extra treatment during the testing process [6]. The difficulties are compounded by the relationship between producers and testers. Challenges related to the automotive industry were summarized in a taxonomy in [7]. Embedded software can be tested by using a regression testing methodology [8; 9; 10]. This is a retesting activity to gain confidence that the system changes have not introduced unwanted system behavior and that the unchanged parts of the system are still working as before. There is a lot of extra work that testers must put into regression testing, making it expensive. Regression testing includes [11; 12]:  
* Test case selection.
* Test case re-validation.
* Test case execution.
* Failure identification.
* Fault identification and mitigation.  
It is commonly applied within industry but lacks scientific research. Generally, the process of building testing suites is experimental [5; 13].  
It is an interesting problem to run tests on the system during startup and shutdown, or periodically while the system is running [14; 15]. This type of post-production testing has a completely different meaning than classical functional testing.  
Traditional software testing is based on predesigned test cases. Instead, an experience-based approach can be applied, known as exploratory testing (ET) [1; 16; 17]. A unique characteristic of this process is that it extracts the most out of human knowledge and intelligence [18; 19] via simultaneous learning, creating a test design, and performing product verification. It is often confused with ad hoc testing, where the tester does not have a specific objective, apart from interacting with the system during testing. It is usually very effective in finding defects which could escape any formal technique, have a significant severity, but maybe a lower customer occurrence, or are simply harder to reproduce. It is hypothesized that exploratory testing is more efficient than test-case-based testing in recognizing functional failures. This is due to the testers being able to use their personal knowledge in designing tests and recognizing failures on the fly even with low levels of testing experience [20].  
The main contribution of the paper is to provide experience- and case-study-based evidence that ET can be strategically adopted within a complex embedded system during the functional testing process. A methodical approach that can be easily implemented and maintained within an organization is proposed. The conducted experiments prove the suitability of the proposed solution and framework.