# Security Testing of RESTful APIs with Test Case Mutation

Sébastien Salva, Jarod Sue

► To cite this version:

Sébastien Salva, Jarod Sue. Security Testing of RESTful APIs with Test Case Mutation. 19th International Conference on Evaluation of Novel Approaches to Software Engineering, Apr 2024, Angers, France. pp.582-589, 10.5220/0012698600003687. hal-04623765

HAL Id: hal-04623765

https://uca.hal.science/hal-04623765

Submitted on 25 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Security Testing of RESTful APIs with Test Case Mutation

Sébastien Salva, Jarod Sue

► To cite this version:

Sébastien Salva, Jarod Sue. Security Testing of RESTful APIs with Test Case Mutation. 19th International Conference on Evaluation of Novel Approaches to Software Engineering, Apr 2024, Angers, France. pp.582-589, 10.5220/0012698600003687. hal-04623765

HAL Id: hal-04623765

https://uca.hal.science/hal-04623765v1

Submitted on 25 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Security Testing of RESTful APIs With Test Case Mutation

Sébastien Salva¹ and Jarod Sue¹

¹LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, France

²Department of Computing, Main University, MySecondTown, MyCountry
sebastien.salva@uca.fr, jarod.sue@uca.fr

**Keywords:** RESTful APIs; Security; Test Case Generation; Test Case Mutation

**Abstract:** The focus of this paper is on automating the security testing of RESTful APIs. The testing stage of this specific kind of components is often performed manually, and this is yet considered as a long and difficult activity. This paper proposes an automated approach to help developers generate test cases for experimenting with each service in isolation. This approach is based upon the notion of test case mutation, which automatically generates new test cases from an original test case set. Test case mutation operators perform slight test case modifications to mimic possible failures or to test the component under test with new interactions. In this paper, we examine test case mutation operators for RESTful APIs and define 18 operators specialised in security testing. Then, we present our test case mutation algorithm. We evaluate its effectiveness and performance on four web service compositions.

# 1 INTRODUCTION

One of the key motivations for software security is the prevention of attackers exploiting software flaws, which can lead to compromising application security or revealing user data. Despite the continuous growth of the security testing market, there is still an inadequate emphasis on this activity, exposing organisations and end users to unforeseen risks when using vulnerable systems or software. One aspect that may account for this observation, is that selecting security solutions and crafting specific security test cases are two tasks of the software life cycle that demand time, expertise, and experience. Developers often lack the guidance, resources, or skills on how to design, implement secure applications, and test them.

A way to help developers in security testing is the use of test automation, which addresses challenges related to time constraints, complexity, and coverage. In this scope, model based testing (Li et al., 2018) offers the advantage of automating the test case generation. But models are often manually written, and this task is considered as long, difficult and error-prone, even for experts. Numerous others approaches have been proposed to generate test cases without specification, for example by using random testing (Arcuri et al., 2011), graphical user interfaces exploration (Salva and Zafimiharisoa, 2014; Ferreira and Paiva, 2019) or automated penetration testing (Abu-Dabaseh

and Alshammari, 2018). Despite their significant benefits, a recurring limitation observed in employing these approaches is the insufficient understanding of the application business logic and context. As a result, they may fail to identify certain security vulnerabilities that require a deeper understanding of how the application behaves.

Focusing on this background, we propose an intermediate solution based upon the notion of test case mutation. Unlike mutation testing that aims at evaluating the effectiveness of an existing test case set by introducing intentional errors into the original source code of an application under test (Papadakis et al., 2019), test case mutation automatically generates new test cases from an original test case set. As the original test cases should encode some knowledge about the application under test, the mutated test cases should deeper cover the application behaviours and features and hence should detect further defects. A test case mutation operator performs slight test case modifications to mimic possible failures or to experiment the system under test with new interactions. Some test case mutation based approaches have been proposed for detecting bugs or crashes (Xuan et al., 2015; Xu et al., 2010; Arcuri, 2018; Arcuri, 2019; Köröglu and Sen, 2018; Paiva et al., 2020). None of them deals with security testing.

In this paper, we propose a new approach, specifically designed for testing the security of RESTful
Security Testing of RESTful APIs With Test Case Mutation

Sébastien Salva¹ and Jarod Sue¹

¹LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, France

²Department of Computing, Main University, MySecondTown, MyCountry
sebastien.salva@uca.fr, jarod.sue@uca.fr

**Keywords:** RESTful APIs; Security; Test Case Generation; Test Case Mutation

**Abstract:** The focus of this paper is on automating the security testing of RESTful APIs. The testing stage of this specific kind of components is often performed manually, and this is yet considered as a long and difficult activity. This paper proposes an automated approach to help developers generate test cases for experimenting with each service in isolation. This approach is based upon the notion of test case mutation, which automatically generates new test cases from an original test case set. Test case mutation operators perform slight test case modifications to mimic possible failures or to test the component under test with new interactions. In this paper, we examine test case mutation operators for RESTful APIs and define 18 operators specialised in security testing. Then, we present our test case mutation algorithm. We evaluate its effectiveness and performance on four web service compositions.

# 1 INTRODUCTION

One of the key motivations for software security is the prevention of attackers exploiting software flaws, which can lead to compromising application security or revealing user data. Despite the continuous growth of the security testing market, there is still an inadequate emphasis on this activity, exposing organisations and end users to unforeseen risks when using vulnerable systems or software. One aspect that may account for this observation, is that selecting security solutions and crafting specific security test cases are two tasks of the software life cycle that demand time, expertise, and experience. Developers often lack the guidance, resources, or skills on how to design, implement secure applications, and test them.

A way to help developers in security testing is the use of test automation, which addresses challenges related to time constraints, complexity, and coverage. In this scope, model based testing (Li et al., 2018) offers the advantage of automating the test case generation. But models are often manually written, and this task is considered as long, difficult and error-prone, even for experts. Numerous others approaches have been proposed to generate test cases without specification, for example by using random testing (Arcuri et al., 2011), graphical user interfaces exploration (Salva and Zafimiharisoa, 2014; Ferreira and Paiva, 2019) or automated penetration testing (Abu-Dabaseh

and Alshammari, 2018). Despite their significant benefits, a recurring limitation observed in employing these approaches is the insufficient understanding of the application business logic and context. As a result, they may fail to identify certain security vulnerabilities that require a deeper understanding of how the application behaves.

Focusing on this background, we propose an intermediate solution based upon the notion of test case mutation. Unlike mutation testing that aims at evaluating the effectiveness of an existing test case set by introducing intentional errors into the original source code of an application under test (Papadakis et al., 2019), test case mutation automatically generates new test cases from an original test case set. As the original test cases should encode some knowledge about the application under test, the mutated test cases should deeper cover the application behaviours and features and hence should detect further defects. A test case mutation operator performs slight test case modifications to mimic possible failures or to experiment the system under test with new interactions. Some test case mutation based approaches have been proposed for detecting bugs or crashes (Xuan et al., 2015; Xu et al., 2010; Arcuri, 2018; Arcuri, 2019; Köröglu and Sen, 2018; Paiva et al., 2020). None of them deals with security testing.

In this paper, we propose a new approach, specifically designed for testing the security of RESTful
APIs in isolation. This firstly implies that we pro-
pose new specific mutation operators devoted to de-
tecting vulnerabilities. This also means that our ap-
proach generates new executable test cases but also
*mock components*. We recall that a mock component
aims at simulating an existing component, while be-
having in a predefined and controlled way to make
testing more effective and efficient. Mocks are often
used by developers to make test development easier or
to increase test coverage. They may indeed be used
to simplify the dependencies that make testing dif-
ficult (e.g., infrastructure or environment related de-
pendencies). Besides, mocks are used to increase test
efficiency by replacing slow-to-access components.
In summary, the main contributions of this paper in-
clude:

1. a study on mutation operators specialised in the security testing of RESTful APIs, including the definition of 18 operators,

2. an algorithm for the generation of mutated test cases along with test scripts and mock components,

3. the implementation of the approach, along with 4 RESTful API compositions and Log files publicly available in (Sue and Salva, 2024).

The paper is organised as follows: We study and
propose test case mutation operators for RESTful
APIs in Section 2. Our test case mutation algorithm
is presented in Section 3. Section 4 summarises our
contributions and draws some perspectives for future
work.

## 2 TEST CASE MUTATION OPERATORS FOR RESTFUL APIS

This paper focuses on test case mutation operators designed to detect vulnerabilities in RESTful APIs. This testing context introduces specific requirements and a testing architecture, both of which are subsequently presented. From this architecture, we present how test cases are modelled with Input Output Transition Systems (IOTSs) and provide an illustrative example. Then, we study the mutation operators that can be defined within this scope.

2.1 Assumptions

We consider the test architecture depicted in Figure
1, whose attributes are expressed with the following
realistic assumptions:

<figure><img src="image_2.png" /><figcaption>Figure 1: Black-box test architecture for experimenting RESTful API in isolation</figcaption></figure>

*   **Black box testing:** we employ a black box perspective, enabling to interact with a RESTful API, denoted *SUT*, only with HTTP requests or responses. We call them (communication) events;

*   **Event content:** observers are able to get all the events related to a RESTful API under test along with their contents (no encryption). In particular, events include parameter assignments allowing to identify the source and the destination of each event. Besides, an event can be identified either as a request or a response;

*   **Test in isolation:** we consider conducting tests in an isolated environment. If the RESTful API is dependent to other services, the later shall be replaced by mock components. We do not assume that those mock components exist, our approach builds them.

## 2.2 IOTS Test Case Definition

Given the test architecture of Figure 1, we consider
that events have the form *e*(α) with *e* some label,
e.g., a path or a status; "*" is a special notation rep-
resenting any label. α is an assignment of param-
eters in *P* to a value in the set of values *V*. These
parameters allow the encoding of some specific web
service characteristics e.g., if an event is a request,
the receiver and sender of this request, etc. We
write *x* := * the assignment of the parameter *x* with
an arbitrary element of *V*, which is not of interest.
ε denotes the event set. We also use these addi-
tional notations on an event *e*(α) to make our algo-
rithm more readable: *from(e(α)))* (rep. *to(e(α)))*
denotes the source (resp. the destination) of the
event. *isreq(e(α)*), *isresp(e(α)*) are boolean expres-
sions expressing the nature of the event. *body(e(α)*),
*header(e(α)*), *status(e(α)*) are expressions returning
values in α.

We model a test case with a deterministic IOTS
having a tree form and whose terminal states express
test verdicts, e.g., *pass* or *inc*, which stands for in-
conclusive. A test step corresponds to an IOTS tran-
sition *q* $\xrightarrow{e(\alpha),l}$ *q'* with *e(α)* some event and *l* a label
set, which may be empty. Furthermore, we use the
notation *θ* labelled on transitions to represent the ab-
sence of reaction from a service under test (Phillips,
APIs in isolation. This firstly implies that we pro-
pose new specific mutation operators devoted to de-
tecting vulnerabilities. This also means that our ap-
proach generates new executable test cases but also
*mock components*. We recall that a mock component
aims at simulating an existing component, while be-
having in a predefined and controlled way to make
testing more effective and efficient. Mocks are often
used by developers to make test development easier or
to increase test coverage. They may indeed be used
to simplify the dependencies that make testing dif-
ficult (e.g., infrastructure or environment related de-
pendencies). Besides, mocks are used to increase test
efficiency by replacing slow-to-access components.
In summary, the main contributions of this paper in-
clude:

1. a study on mutation operators specialised in the security testing of RESTful APIs, including the definition of 18 operators,

2. an algorithm for the generation of mutated test cases along with test scripts and mock components,

3. the implementation of the approach, along with 4 RESTful API compositions and Log files publicly available in (Sue and Salva, 2024).

The paper is organised as follows: We study and
propose test case mutation operators for RESTful
APIs in Section 2. Our test case mutation algorithm
is presented in Section 3. Section 4 summarises our
contributions and draws some perspectives for future
work.

## 2 TEST CASE MUTATION OPERATORS FOR RESTFUL APIS

This paper focuses on test case mutation operators designed to detect vulnerabilities in RESTful APIs. This testing context introduces specific requirements and a testing architecture, both of which are subsequently presented. From this architecture, we present how test cases are modelled with Input Output Transition Systems (IOTSs) and provide an illustrative example. Then, we study the mutation operators that can be defined within this scope.

2.1 Assumptions

We consider the test architecture depicted in Figure 1, whose attributes are expressed with the following realistic assumptions:

<figure><img src="image_2.png" /><figcaption>Figure 1: Black-box test architecture for experimenting RESTful API in isolation</figcaption></figure>

*   **Black box testing:** we employ a black box perspective, enabling to interact with a RESTful API, denoted *SUT*, only with HTTP requests or responses. We call them (communication) events;

*   **Event content:** observers are able to get all the events related to a RESTful API under test along with their contents (no encryption). In particular, events include parameter assignments allowing to identify the source and the destination of each event. Besides, an event can be identified either as a request or a response;

*   **Test in isolation:** we consider conducting tests in an isolated environment. If the RESTful API is dependent to other services, the later shall be replaced by mock components. We do not assume that those mock components exist, our approach builds them.

## 2.2 IOTS Test Case Definition

Given the test architecture of Figure 1, we consider
that events have the form *e*(α) with *e* some label,
e.g., a path or a status; "*" is a special notation rep-
resenting any label. α is an assignment of param-
eters in *P* to a value in the set of values *V*. These
parameters allow the encoding of some specific web
service characteristics e.g., if an event is a request,
the receiver and sender of this request, etc. We
write *x* := * the assignment of the parameter *x* with
an arbitrary element of *V*, which is not of interest.
ε denotes the event set. We also use these addi-
tional notations on an event *e*(α) to make our algo-
rithm more readable: *from(e(α)))* (rep. *to(e(α)))*
denotes the source (resp. the destination) of the
event. *isreq(e(α)*), *isresp(e(α)*) are boolean expres-
sions expressing the nature of the event. *body(e(α)*),
*header(e(α)*), *status(e(α)*) are expressions returning
values in α.

We model a test case with a deterministic IOTS
having a tree form and whose terminal states express
test verdicts, e.g., *pass* or *inc*, which stands for in-
conclusive. A test step corresponds to an IOTS tran-
sition $q \xrightarrow{e(\alpha),l} q'$ with $e(\alpha)$ some event and $l$ a label
set, which may be empty. Furthermore, we use the
notation $\theta$ labelled on transitions to represent the ab-
sence of reaction from a service under test (Phillips,
1987). Classically, we call a sequence of test steps a test sequence. The label set allows to easily express some knowledge about the event. For instance, "crash" is used when the HTTP status 500 is received. The special label "mock" identifies events performed by some other dependee services. Since we assume testing *SUT* in isolation, the dependee services will have to be replaced with mock components.

An IOTS test case has to meet a few restrictions to avoid indeterministic behaviours while testing. To this end, a test case must allow at most one input event at any state. In reference to (Tretmans, 2008), this last restriction, we say that a test case is *input restricted*. Additionally, still in the context of isolation testing and to keep control of the testing process, a mock component should be deterministic and return at most one response after being invoked with the same event. We say that a test case has to be mock response restricted. This is formulated with:

**Definition 1** A test case *tc* is a deterministic IOTS $(\mathcal{Q}, q_0, \Sigma \cup \{\theta\}, L, \rightarrow)$ where:

* $\mathcal{Q}$ is a finite set of states; $q_0$ is the initial state;

* $\Sigma \subseteq \mathcal{E}$ is the finite set of events. $\Sigma_I \subseteq \Sigma$ is the finite set of input events beginning with "?", $\Sigma_O \subseteq \Sigma$ is the finite set of output events beginning with "!", with $\Sigma_O \cap \Sigma_I = \emptyset$;

* $L$ is a set of labels;

* $\rightarrow \subseteq \mathcal{Q} \times \Sigma \cup \{\theta\} \times L^* \times \mathcal{Q}$ is a finite set of transitions. A transition $(q, e(\alpha), l, q')$ is also denoted $q \xrightarrow{e(\alpha), l} q'$;

* $\mathcal{Q}_f = \{pass, fail, inc\} \subset \mathcal{Q}$ is the set of verdict states; if $q \xrightarrow{e(\alpha), l} \mathcal{Q}_f$ with $q_f \in \mathcal{Q}_f$, then $e(\alpha) \in \Sigma_O \cup \theta$;

* *tc* has no cycles except those in states of $\mathcal{Q}_f$;

* *tc* is input restricted i.e. $\forall q \in \mathcal{Q} : \text{event}(q) = \Sigma_O \cup \{e(\alpha)\}$ for some $e(\alpha) \in \Sigma_I$ or $\text{event}(q) = \Sigma_O \cup \{\theta\}$ with $\text{event}(q) = \{e(\alpha) | \exists q' \in \mathcal{Q} : q \xrightarrow{e(\alpha), l} q'\}$;

* *tc* is mock response restricted i.e. $\forall q \in \mathcal{Q} : \{q \xrightarrow{e(\alpha), l} q' | \text{isResp}(e(\alpha)) \land \text{mock} \in l\} \le 1$.

<figure><img src="image_3.png" /><figcaption>Figure 2: IOTS Test Case example</figcaption></figure>

An IOTS test case example is illustrated in Figure 2. It checks whether a RESTful API AccMan can

be called with "/checkAccountRisk". This service is dependent on another service called CheckRisk. The events related to CheckRisk are labelled by "mock" to express that a mock component has to be built to test AccMan in isolation.

IOTS test cases can be written manually, but this activity may be long and error-prone, especially for un-experimented developers. To solve this problem, we proposed in (Salva and Sue, 2023) an approach and tool for generating IOTS test cases from Log files. The approach also allows to recognise some specific behaviours (authentication, token generation, crash, ) and adds on test steps the following labels "login", "token", "token generation", "crash".

## 2.3 Mutation Operators For Security Testing

The literature does not cover the use of test case mutation operators specialised in assessing the security of web services. Consequently, we initially conducted a literature review to collect relevant data about the security testing of RESTful APIs. We searched for papers indexed in online sources (Scopus, Science Direct, IEEE Xplore, ACM Digital Library, Google Scholar). We identified relevant papers via keyword search by using the terms "web services security vulnerabilities attacks" and then, terms "microservice security vulnerabilities attacks". We found 42 and 35 works between 2006-2023. We isolated 24 papers and 3 surveys by using their abstracts and titles. We then crossed these results with the databases CAPEC (CAPEC, 2024) and CWE (CWE, 2024) of the MITRE organisation in order to classify attacks and avoid duplicates. With regard to our black box test architecture, we kept the attacks related to these domains:

* CAPEC-21: Exploitation of Trusted Identifiers

* CAPEC-22: Exploiting Trust in Client

* CAPEC-63: Cross-Site Scripting (XSS)

* CAPEC-151: Identity Spoofing

* CAPEC-153: Input Data Manipulation

* CAPEC-115: Authentication Bypass

* CAPEC-125: Flooding

* CAPEC-278: Service Protocol Manipulation

* CAPEC-594: Traffic Injection

At this step, we collected a total number of 36 attacks. We finally augmented this compilation, by incorporating 7 recommendations provided in the ENISA good practice guide (Skouloudi et al., 2018).
1987). Classically, we call a sequence of test steps a test sequence. The label set allows to easily express some knowledge about the event. For instance, "crash" is used when the HTTP status 500 is received. The special label "mock" identifies events performed by some other dependee services. Since we assume testing *SUT* in isolation, the dependee services will have to be replaced with mock components.

An IOTS test case has to meet a few restrictions to avoid indeterministic behaviours while testing. To this end, a test case must allow at most one input event at any state. In reference to (Tretmans, 2008), this last restriction, we say that a test case is *input restricted*. Additionally, still in the context of isolation testing and to keep control of the testing process, a mock component should be deterministic and return at most one response after being invoked with the same event. We say that a test case has to be mock response restricted. This is formulated with:

**Definition 1** A test case *tc* is a deterministic IOTS $(\mathcal{Q}, q_0, \Sigma \cup \{\theta\}, L, \rightarrow)$ where:

* $\mathcal{Q}$ is a finite set of states; $q_0$ is the initial state;

* $\Sigma \subseteq \mathcal{E}$ is the finite set of events. $\Sigma_I \subseteq \Sigma$ is the finite set of input events beginning with "?", $\Sigma_O \subseteq \Sigma$ is the finite set of output events beginning with "!", with $\Sigma_O \cap \Sigma_I = \emptyset$;

* $L$ is a set of labels;

* $\rightarrow \subseteq \mathcal{Q} \times \Sigma \cup \{\theta\} \times L^* \times \mathcal{Q}$ is a finite set of transitions. A transition $(q, e(\alpha), l, q')$ is also denoted $q \xrightarrow{e(\alpha), l} q'$;

* $\mathcal{Q}_f = \{pass, fail, inc\} \subset \mathcal{Q}$ is the set of verdict states; if $q \xrightarrow{e(\alpha), l} \mathcal{Q}_f$ with $q_f \in \mathcal{Q}_f$, then $e(\alpha) \in \Sigma_O \cup \theta$;

* *tc* has no cycles except those in states of $\mathcal{Q}_f$;

* *tc* is input restricted i.e. $\forall q \in \mathcal{Q} : \text{event}(q) = \Sigma_O \cup \{e(\alpha)\}$ for some $e(\alpha) \in \Sigma_I$ or $\text{event}(q) = \Sigma_O \cup \{\theta\}$ with $\text{event}(q) = \{e(\alpha) | \exists q' \in \mathcal{Q} : q \xrightarrow{e(\alpha), l} q'\}$;

* *tc* is mock response restricted i.e. $\forall q \in \mathcal{Q} : \{|q \xrightarrow{e(\alpha), l} q' | \text{isResp}(e(\alpha)) \wedge \text{mock} \in l\} \le 1$.

<figure><img src="image_3.png" /><figcaption>Figure 2: IOTS Test Case example</figcaption></figure>

An IOTS test case example is illustrated in Figure 2. It checks whether a RESTful API AccMan can

be called with "/checkAccountRisk". This service is dependent on another service called CheckRisk. The events related to CheckRisk are labelled by "mock" to express that a mock component has to be built to test AccMan in isolation.

IOTS test cases can be written manually, but this activity may be long and error-prone, especially for un-experimented developers. To solve this problem, we proposed in (Salva and Sue, 2023) an approach and tool for generating IOTS test cases from Log files. The approach also allows to recognise some specific behaviours (authentication, token generation, crash, ) and adds on test steps the following labels "login", "token", "token generation", "crash".

## 2.3 Mutation Operators For Security Testing

The literature does not cover the use of test case mutation operators specialised in assessing the security of web services. Consequently, we initially conducted a literature review to collect relevant data about the security testing of RESTful APIs. We searched for papers indexed in online sources (Scopus, Science Direct, IEEE Xplore, ACM Digital Library, Google Scholar). We identified relevant papers via keyword search by using the terms "web services security vulnerabilities attacks" and then, terms "microservice security vulnerabilities attacks". We found 42 and 35 works between 2006-2023. We isolated 24 papers and 3 surveys by using their abstracts and titles. We then crossed these results with the databases CAPEC (CAPEC, 2024) and CWE (CWE, 2024) of the MITRE organisation in order to classify attacks and avoid duplicates. With regard to our black box test architecture, we kept the attacks related to these domains:

* CAPEC-21: Exploitation of Trusted Identifiers

* CAPEC-22: Exploiting Trust in Client

* CAPEC-63: Cross-Site Scripting (XSS)

* CAPEC-151: Identity Spoofing

* CAPEC-153: Input Data Manipulation

* CAPEC-115: Authentication Bypass

* CAPEC-125: Flooding

* CAPEC-278: Service Protocol Manipulation

* CAPEC-594: Traffic Injection

At this step, we collected a total number of 36 attacks. We finally augmented this compilation, by incorporating 7 recommendations provided in the ENISA good practice guide (Skouloudi et al., 2018).
Then, we studied these 43 elements to extract mutation operators. During this process, we applied the following criteria:

* C1: in accordance with our test architecture, we build mutation operators applicable to unencrypted events;

* C2: a mutation operator performs small changes, it is here used to build an attack executed with one test case only. Hence, complex attack scenarios cannot be considered;

* C3: knowledge typically plays a crucial role in performing security attacks. We consider having labels in test steps allowing to recognise authentication processes, token generation and errors. Additional labels allow to recognise the existence of variables acting as tokens or session identifiers;

* C4: an operator can derive new test cases and new mock components.

Using these criteria, we finally wrote 18 mutation operators tailored to testing in isolation the security of black box RESTful APIs. These operators are outlined in Table 1, where column 2 gives short descriptions, columns 3 and 4 give the expected behaviours that should be observed after the execution of mutated test steps and conditions on the application of the operators.

## 3 TEST CASE MUTATION

<figure><img src="image_4.png" /><figcaption>Figure 3: Approach Overview</figcaption></figure>

As illustrated in Figure 3, we propose a test case mutation approach and a tool for RESTful APIs, consisting of three main stages :

1. our approach takes either existing IOTS test cases, or Log files that are used to generate IOTS test cases. As stated previously, the paper (Salva and Sue, 2023) presents algorithms and a tool for performing this step;

2. mutation operators are applied on IOTS test cases to perform slight modifications that aim to mimic security attacks. These modifications may result in numerous mutated test cases. To address this, we suggest strategies to restrict their generation.

Mutation operators are then applied on test cases: we check whether the test steps meet some mutation conditions to restrict the transformations on the relevant steps only; we modify the original test cases and complete them with tests steps and verdicts to get new IOTS mutants;

3. the mutants are finally converted into test scripts and mock components, which will be used to check whether *SUT* is vulnerable.

We formalise those steps in the remainder of this section.

### 3.1 Test Mutation Operator Definition

A mutation operator *M* of an IOTS test case *tc* is made up of three elements. The first is the function *Condition*, which aims at restricting the application of the operator to some events of *tc*. The next function *Change* applies the mutation on *tc* and produces an initial mutant *tc*<sub>m</sub>. Finally, *Expected* is a function that completes *tc*<sub>m</sub> with test sequences finished by verdict states to express the expected observations after the execution of a mutated event.

**Definition 2 (Mutation operator)** *A Mutation operator is the tuple (Condition, Change, Expected) such that :*

* Condition : $Q \times \Sigma \times L^* \times Q \rightarrow \{\text{true, false}\}$ is a function that expresses restrictions on test steps,

* Change : IOTS $\rightarrow$ IOTS is a mutant derivation function,

* Expected : IOTS $\rightarrow$ IOTS is a mutant completion function, such that for any test sequence $q_0 \xrightarrow{(e_1(\alpha_1), l_1), \dots, (e_k(\alpha_k), l_k)} q$ of the IOTS, $q \in Q_f$ is a final state.

The function $\text{Condition}(q \xrightarrow{e(\alpha), l} q')$ of a mutation operator *M* may be used on the label *e*, on the assignments *α*, or on the label list *l*. This function can be used to define a generic operator, for example with a condition of the form $e == *$ supplemented with some conditions on *α*. But, a more specific operator can also be defined with a condition on precise events and parameters. The last column of Table 1 provides several condition examples.

*Change(tc)* applies the mutation operator on a test case *tc* and returns a mutant. We here assume having some transitions marked with the special label "mutation", which targets the transitions to transform. Given under the form of a procedure, *Change* could have the following form :
Then, we studied these 43 elements to extract muta-
tion operators. During this process, we applied the
following criteria:

* C1: in accordance with our test architecture, we build mutation operators applicable to unencrypted events;

* C2: a mutation operator performs small changes, it is here used to build an attack executed with one test case only. Hence, complex attack scenarios cannot be considered;

* C3: knowledge typically plays a crucial role in performing security attacks. We consider having labels in test steps allowing to recognise authentication processes, token generation and errors. Additional labels allow to recognise the existence of variables acting as tokens or session identifiers;

* C4: an operator can derive new test cases and new mock components.

Using these criteria, we finally wrote 18 mutation
operators tailored to testing in isolation the security
of black box RESTful APIs. These operators are out-
lined in Table 1, where column 2 gives short descrip-
tions, columns 3 and 4 give the expected behaviours
that should be observed after the execution of mutated
test steps and conditions on the application of the op-
erators.

3 TEST CASE MUTATION

<figure><img src="image_4.png" /><figcaption>Figure 3: Approach Overview</figcaption></figure>

As illustrated in Figure 3, we propose a test case
mutation approach and a tool for RESTful APIs, con-
sisting of three main stages :

1. our approach takes either existing IOTS test cases, or Log files that are used to generate IOTS test cases. As stated previously, the paper (Salva and Sue, 2023) presents algorithms and a tool for performing this step;

2. mutation operators are applied on IOTS test cases to perform slight modifications that aim to mimic security attacks. These modifications may result in numerous mutated test cases. To address this, we suggest strategies to restrict their generation.

Mutation operators are then applied on test cases: we check whether the test steps meet some mutation conditions to restrict the transformations on the relevant steps only; we modify the original test cases and complete them with tests steps and verdicts to get new IOTS mutants;

3. the mutants are finally converted into test scripts and mock components, which will be used to check whether SUT is vulnerable.

We formalise those steps in the remainder of this
section.

**3.1 Test Mutation Operator Definition**

A mutation operator *M* of an IOTS test case *tc* is made up of three elements. The first is the function *Condition*, which aims at restricting the application of the operator to some events of *tc*. The next function *Change* applies the mutation on *tc* and produces an initial mutant *tc*<sub>m</sub>. Finally, *Expected* is a function that completes *tc*<sub>m</sub> with test sequences finished by verdict states to express the expected observations after the execution of a mutated event.

**Definition 2 (Mutation operator)** *A Mutation operator is the tuple (Condition, Change, Expected) such that :*

*   Condition : $Q \times \Sigma \times L^* \times Q \rightarrow \{\text{true, false}\}$ is a function that expresses restrictions on test steps,

*   Change : IOTS → IOTS is a mutant derivation function,

*   Expected : IOTS → IOTS is a mutant completion function, such that for any test sequence $q_0 \xrightarrow{(e_1(\alpha_1), l_1), \dots, (e_k(\alpha_k), l_k)} q$ of the IOTS, $q \in Q_f$ is a final state.

The function Condition($q \xrightarrow{e(\alpha),l} q'$) of a mutation operator *M* may be used on the label *e*, on the assignments *α*, or on the label list *l*. This function can be used to define a generic operator, for example with a condition of the form *e* == *supplemented with some conditions on *α*. But, a more specific operator can also be defined with a condition on precise events and parameters. The last column of Table 1 provides several condition examples.

*Change(tc)* applies the mutation operator on a test case *tc* and returns a mutant. We here assume having some transitions marked with the special label "mutation", which targets the transitions to transform. Given under the form of a procedure, *Change* could have the following form :
Reach a transition $t := q \xrightarrow{e(\alpha), \{mutation\}} q_1$;

Modify $t$;

(possibly) Keep the next outgoing transitions from $q_1$ to $q_k$ such that $to(q_k \xrightarrow{e_k(\alpha_k)} q_l) = SUT$;

Prune the useless transitions from $q_k$ to a terminal state;

Expected($tc_M$) completes a mutant returned by Change with new test steps such that the last test steps end by a verdict state. Column 3 of Table 1 summarizes the test steps that are added for every mutation operator.

## 3.2 Test Case Generation

The test architecture of Figure 1 emphasises the control and observation logics. The controller parts have the capability to send events to SUT. These events are those that can be modified by mutation operators to send unexpected requests or attacks. The observer parts will be used to collect responses, which are interpreted to decide whether SUT is vulnerable or not.

In this context, we say that a test step $q \xrightarrow{e(\alpha),l} q'$ of a test case is mutable if the recipient of the event is SUT itself and if the operator $M$ may be applied on this test step. Likewise, we use the notation mutable($M$) in $tc$ to get the set of test steps on which the mutation operator $M$ can be applied. It is worth noting that this set may be empty. This is captured by the following definition:

**Definition 3 (Mutable Test Step)** Let $M$ be a mutation operator, $tc$ be an IOTS test case for the service SUT, and $q \xrightarrow{e(\alpha),l} q' \in \mathbf{SUT}$ be a test step.

* $q \xrightarrow{e(\alpha),l} q'$ is mutable$_M$ iff $to(e(\alpha)) = SUT \land M Condition(q \xrightarrow{e(\alpha),l} q') \land ((e(\alpha) \in \sigma_I \lor \text{"mock"} \in l))$.

* mutable($M$) in $tc =_{def} \{q \xrightarrow{e(\alpha),l} q' \in tc | q \xrightarrow{e(\alpha),l} q' \text{ is mutable}_M\}$

Furthermore, we define the IOTS operator mark, which simply adds a label "mutation" on the mutable test steps.

**Definition 4 (IOTS operator mark)** Let $t = q \xrightarrow{e(\alpha),l} q'$ be a test step of a test case $tc = \langle Q, q_0, \Sigma \cup \{\theta\}, L, \rightarrow \rangle$.

mark $t$ in $tc = \langle Q_2, q_0, \Sigma_2 \cup \{\theta\}, L_2, \rightarrow_2 \rangle$ is the IOTS test case derived from the test case $tc$ where $Q_2, \Sigma_2, L_2, \rightarrow_2$ are defined by the following rules:

$$ \begin{array}{rcl} t &=& q \xrightarrow{e(\alpha),l} q' \\ q &\xrightarrow{e(\alpha),l \cup \{\text{"mutable"}\}} & q' \xrightarrow{t_2 \neq t} q' \end{array} $$

### Algorithm 1: IOTS Test Case Mutation

input : Test case set $TC$, Mutation Operator $M$

output: Test case set $TC_M$

1. $TC_M := \emptyset$;

2. **foreach** $tc \in TC$ **do**

3. &nbsp;&nbsp;&nbsp;&nbsp;**foreach** $q \xrightarrow{e(\alpha),l} q' \in \text{mutable}(M)$ in $ts$ such that
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$ts = q_0 \xrightarrow{(e_1(\alpha_1),l_1) \dots (e_k(\alpha_k),l_k)} pass \in tc$ and
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**selection**(TC, $TC_M$) **do**
4. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mark $q \xrightarrow{e(\alpha),l} q'$ in $tc$ such that $q \xrightarrow{e(\alpha),l} q'$
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in mutable($M$) in $ts$ arbitrarily chosen;
5. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$tc_2 := M \text{Change}(tc, q \xrightarrow{e(\alpha),l} q')$;
6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$tc_2 := M \text{Expected}(tc_2)$;
7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;compl $tc_2$;
8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$TC_M := TC_M \cup \{tc_2\}$;

We are now ready to present our test case mutation algorithm given in Algorithm 1: it takes a mutation operator $M$ along with a test case set $TC$. It produces a new test case set, denoted $TC_M$. It covers every mutable test step of a test sequence $ts$ (line 3) starting from the initial state of the test case such that $ts$ is finished by the state *pass*. We choose to only mutate test sequences finished by *pass* to avoid bringing confusion in the test result analysis. Indeed, if we mutate a test sequence finished by *fail* and if we obtain a fail verdict while testing, it is very difficult to deduce whether SUT is faulty on account of the mutation. As the set of mutants may become large, Algorithm 1 calls the function selection(TC, $TC_M$), which returns a boolean value. This function expresses a mutant generation strategy, e.g., "applies $M$ on every test case only once", which stops the mutation of the test cases once some conditions are met. In this case, the function returns false. Algorithm 1 marks the chosen test step with "mutable" to help the mutation operator target the test step to change. A new test case $tc_2$ is built by applying the function $M \text{Change}$ and by completing its branches not finished by a verdict state with $M \text{Expected}$ in order to express the expected behaviour after the execution of the mutated test step. Additionally, the mutant $tc_2$ is completed once more (line 7) with the operator compl: $IOTS \rightarrow IOTS$ to add transitions that express all the behaviours that might be observed and the related test verdicts. The resulting mutant $tc_2$ is stored in $TC_M$. The operator compl is defined by:

**Definition 5 (IOTS operator compl)** $compl: TC_M \rightarrow IOTS$ is the IOTS test case obtained from $tc$ where $Q_2, \Sigma_2, \rightarrow_2$ are defined by the
Reach a transition $t := q \xrightarrow{e(\alpha), \{mutation\}} q_1$;

Modify $t$;

(possibly) Keep the next outgoing transitions from $q_1$ to $q_k$ such that $to(q_k \xrightarrow{e_k(\alpha_k)} q_l) = SUT$;

Prune the useless transitions from $q_k$ to a terminal state;

Expected($tc_M$) completes a mutant returned by Change with new test steps such that the last test steps end by a verdict state. Column 3 of Table 1 summarizes the test steps that are added for every mutation operator.

## 3.2 Test Case Generation

The test architecture of Figure 1 emphasises the control and observation logics. The controller parts have the capability to send events to SUT. These events are those that can be modified by mutation operators to send unexpected requests or attacks. The observer parts will be used to collect responses, which are interpreted to decide whether SUT is vulnerable or not.

In this context, we say that a test step $q \xrightarrow{e(\alpha),l} q'$ of a test case is mutable if the recipient of the event is SUT itself and if the operator $M$ may be applied on this test step. Likewise, we use the notation mutable($M$) in $tc$ to get the set of test steps on which the mutation operator $M$ can be applied. It is worth noting that this set may be empty. This is captured by the following definition:

**Definition 3 (Mutable Test Step)** Let $M$ be a mutation operator, $tc$ be an IOTS test case for the service SUT, and $q \xrightarrow{e(\alpha),l} q' \in \mathbf{SUT}$ be a test step.

* $q \xrightarrow{e(\alpha),l} q'$ is mutable$_M$ iff $to(e(\alpha)) = SUT \wedge M Condition(q \xrightarrow{e(\alpha),l} q') \wedge ((e(\alpha) \in \sigma_I \vee \text{"mock"} \in l))$.

* mutable($M$) in $tc =_{def} \{q \xrightarrow{e(\alpha),l} q' \in tc | q \xrightarrow{e(\alpha),l} q' \text{ is mutable}_M\}$

Furthermore, we define the IOTS operator *mark*, which simply adds a label "mutation" on the mutable test steps.

**Definition 4 (IOTS operator mark)** Let $t = q \xrightarrow{e(\alpha),l} q'$ be a test step of a test case $tc = \langle Q, q_0, \Sigma \cup \{\theta\}, L, \rightarrow \rangle$.

mark $t$ in $tc = \langle Q_2, q_0, \Sigma_2 \cup \{\theta\}, L_2, \rightarrow_2 \rangle$ is the IOTS test case derived from the test case $tc$ where $Q_2, \Sigma_2, L_2, \rightarrow_2$ are defined by the following rules:

$$ \begin{array}{rcl} t &=& q \xrightarrow{e(\alpha),l} q' \\ q &\xrightarrow{e(\alpha),l \cup \{\text{"mutable"}\}} & q' \xrightarrow{t_2 \neq t} q' \end{array} $$

### Algorithm 1: IOTS Test Case Mutation

input : Test case set TC, Mutation Operator M

output: Test case set TC<sub>M</sub>

1. $TC_M := \emptyset$;

2. **foreach** $tc \in TC$ **do**

3. &nbsp;&nbsp;&nbsp;&nbsp;**foreach** $q \xrightarrow{e(\alpha),l} q' \in \text{mutable}(M)$ in $ts$ such that
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$ts = q_0 \xrightarrow{(e_1(\alpha_1),l_1) \dots (e_k(\alpha_k),l_k)} pass \in tc$ and
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**selection**(TC, $TC_M$) **do**
4. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mark $q \xrightarrow{e(\alpha),l} q'$ in $tc$ such that $q \xrightarrow{e(\alpha),l} q' \in \text{mutable}(M)$ in $ts$ arbitrarily chosen;
5. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$tc_2 := M \text{Change}(tc, q \xrightarrow{e(\alpha),l} q')$;
6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$tc_2 := M \text{Expected}(tc_2)$;
7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;compl $tc_2$;
8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$TC_M := TC_M \cup \{tc_2\}$;

We are now ready to present our test case mutation algorithm given in Algorithm 1: it takes a mutation operator $M$ along with a test case set TC. It produces a new test case set, denoted $TC_M$. It covers every mutable test step of a test sequence $ts$ (line 3) starting from the initial state of the test case such that $ts$ is finished by the state pass. We choose to only mutate test sequences finished by pass to avoid bringing confusion in the test result analysis. Indeed, if we mutate a test sequence finished by fail and if we obtain a fail verdict while testing, it is very difficult to deduce whether SUT is faulty on account of the mutation. As the set of mutants may become large, Algorithm 1 calls the function selection(TC, $TC_M$), which returns a boolean value. This function expresses a mutant generation strategy, e.g., "applies $M$ on every test case only once", which stops the mutation of the test cases once some conditions are met. In this case, the function returns false. Algorithm 1 marks the chosen test step with "mutable" to help the mutation operator target the test step to change. A new test case $tc_2$ is built by applying the function M.Change and by completing its branches not finished by a verdict state with M.Expected in order to express the expected behaviour after the execution of the mutated test step. Additionally, the mutant $tc_2$ is completed once more (line 7) with the operator compl: IOTS $\rightarrow$ IOTS to add transitions that express all the behaviours that might be observed and the related test verdicts. The resulting mutant $tc_2$ is stored in $TC_M$. The operator compl is defined by:

**Definition 5 (IOTS operator compl)** $compl: TC_M \rightarrow TC_M$ is the IOTS test case obtained from $tc$ where $Q_2, \Sigma_2, \rightarrow_2$ are defined by the
following rules:

$$r_1 : q_1 \xrightarrow{e(\alpha),l} q_2 \vdash q_1 \xrightarrow{e(\alpha),l} q_2$$

$$r_2 : q_1 \xrightarrow{e(\alpha),l} q_2, q_1 \xrightarrow{!*,\{}} q_3 \notin \vdash_l q_1 \xrightarrow{!*,\{}} inc$$

$$r_3 : q_1 \xrightarrow{!e(\alpha),l} q_2, q_1 \xrightarrow{?e_2(\alpha_2),l} q_3 \notin \vdash_l q_1 \xrightarrow{\theta} q_3 \notin \vdash_l q_1 \xrightarrow{\theta} fail$$

The inference rule $r_1$ takes all the transitions of an IOTS to build a new test case. $r_2$ completes the test case with a new transition to express that any unexpected output leads to the inconclusive verdict. When the test case only expects outgoing transitions labelled by output events, the rule $r_3$ also adds a transition to fail modelling that the absence of reaction is faulty.

The function selection($TC, TC_M$) encodes conditions on the test case sets $TC$ and $TC_M$ to limit the number of mutants by mutation operator. Various conditions and combinations could be considered. Here, we provide some examples:

* No restriction (all mutable test steps are covered);

* Every test case is mutated at most $n$ times

* Every mutable test step of each test case is mutated at most $n$ times

<figure><img src="image_5.png" /><figcaption>Figure 4: IOTS Mutated Test Case example</figcaption></figure>

Figure 4 illustrates an example of mutant obtained from the test case of Figure 2 by applying the operator "Token removal" on the second mutable test step (!/ok). The verdict is pass if a response is observed with a status 403 or a status 401, which encodes that the request has been rejected on account of insufficient permissions.

## 3.3 Generation of Concrete Test Cases

Finally, executable test scripts are generated from IOTS test cases. We have chosen to generate test cases using the frameworks Citrus and Mockserver. Given an IOTS test case $tc \in TC_M$, some parameters may still be assigned to "*". For example, this happens for parameters used to identify sessions. In short, we assign these parameters with stored values collected from Log files or with random values. To

generate a test script, the transitions of $tc$ labelled by "mock" are initially pruned. The resulting IOTS is converted as follows: every request to $SUT$ is converted into code that calls $SUT$ and waits for a response. An example is given in Figure 5. The transitions labelled by responses of this request are used to build assertions. The test script ends with the call of the method "verificationMock", which aims to check whether mock components behave as expected during the test execution. At the moment, we check whether the number of calls to a mocked request matches with the number of time this request is found in $tc$.

To generate mock components, the IOTS transitions of $tc$ labelled by "mock" are used to derive rules of the form `request()...respond()`, which mimic the behaviour of a dependee service. Then, the method "verificationMock" is written according to these rules. Figure 6 shows a rule example written with the language provided by the framework MockServer.

```java
@Test @CitrusTest
public void testAccMan() throwselnException{
    HtrpClient toClient = CitrusEndpoints
        .http().client().requestUrl("http://AccMan/").build();
    $(HTTP)
    .client(toClient).send().("checkAccountRisk").message()
        .header("token",1234).body("""acc""=99")
        .accept(MediaType ALL_VALUE);
    $(receive (toClient)
        .message().type(Technology.PLAINTEXT).name("Response")
        .extract (fromHeaders()
            .header (HttpMessageHeaders.HTTP_STATUS_CODE, "statusCode")
            .header("token","token")));
    variable ("body"," citrus : message (Response. body())");
    variable ("status ", "${statusCode}");
    String status = context. getVariable ("status");
    String t = context. getVariable ("token");
    If (token. equals("1234") && status. equals("403"))  assertTrue (true);
    else Assumptions.assumeTrue(false, "Inconclusive");
    verificationMock ();}
```

<caption>Figure 5: Example of test script for the service AccMan</caption>

```java
mockServer when
    request().withMethod("GET").withPath("evaluateRisk")
        .withHeaders(new Header("acc", "99"), new Header("token", "1234"))
        .Times. exactly (1)
    .respond(response().with Code(200)
        .with Body("LOWRISK"));
```

<caption>Figure 6: Mock component piece of code, which implements the events !/EvaluateRisk and !ok of the test case of Figure 4</caption>
following rules:

<figure><img src="image_5.png" /></figure>

The inference rule $r_1$ takes all the transitions of an IOTS to build a new test case. $r_2$ completes the test case with a new transition to express that any unexpected output leads to the inconclusive verdict. When the test case only expects outgoing transitions labelled by output events, the rule $r_3$ also adds a transition to fail modelling that the absence of reaction is faulty.

The function selection($TC, TC_M$) encodes conditions on the test case sets $TC$ and $TC_M$ to limit the number of mutants by mutation operator. Various conditions and combinations could be considered. Here, we provide some examples:

* No restriction (all mutable test steps are covered);

* Every test case is mutated at most $n$ times

* Every mutable test step of each test case is mutated at most $n$ times

<figure><img src="image_6.png" /><figcaption>Figure 4: IOTS Mutated Test Case example</figcaption></figure>

Figure 4 illustrates an example of mutant obtained from the test case of Figure 2 by applying the operator "Token removal" on the second mutable test step (!/ok). The verdict is pass if a response is observed with a status 403 or a status 401, which encodes that the request has been rejected on account of insufficient permissions.

## 3.3 Generation of Concrete Test Cases

Finally, executable test scripts are generated from IOTS test cases. We have chosen to generate test cases using the frameworks Citrus and Mockserver. Given an IOTS test case $tc \in TC_M$, some parameters may still be assigned to "*". For example, this happens for parameters used to identify sessions. In short, we assign these parameters with stored values collected from Log files or with random values. To

generate a test script, the transitions of $tc$ labelled by "mock" are initially pruned. The resulting IOTS is converted as follows: every request to $SUT$ is converted into code that calls $SUT$ and waits for a response. An example is given in Figure 5. The transitions labelled by responses of this request are used to build assertions. The test script ends with the call of the method "verificationMock", which aims to check whether mock components behave as expected during the test execution. At the moment, we check whether the number of calls to a mocked request matches with the number of time this request is found in $tc$.

To generate mock components, the IOTS transitions of $tc$ labelled by "mock" are used to derive rules of the form `request()...respond()`, which mimic the behaviour of a dependee service. Then, the method "verificationMock" is written according to these rules. Figure 6 shows a rule example written with the language provided by the framework MockServer.

```java
@Test @CitrusTest
public void testAccMan() throwselnException {
    HtrpClient toClient = CitrusEndpoints
        .http().client().requestUrl("http://AccMan/").build();
    $(HTTP)
        .client(toClient).send().("checkAccountRisk").message()
        .header("token",1234).body("""acc""=99")
        .accept(MediaType ALL_VALUE);
    $(receive (toClient)
        .message().type (Type (TypePLAINTEXT).name("Response"))
        .extract (fromHeaders()
            .header (HttpMessageHeaders.HTTP_STATUS_CODE, "statusCode"))
            .header("token","token")));
    variable ("body","" citrus : message (Response. body()));
    variable ("status ", "$ {statusCode}");
    String status = context. getVariable ("status");
    String t = context. getVariable ("token");
    If (token. equals("1234") && status. equals("403"))asserTrue (true);
    else Assumptions.assumeTrue(false,"Inconclusive");
    verificationMock ();}
```

<caption>Figure 5: Example of test script for the service AccMan</caption>

```java
mockServer when
    request().withMethod("GET").withPath("evaluateRisk")
        .withHeaders(new Header("acc", "99"), new Header("token", "1234"))
        .Times. exactly (1)
    .respond(response().with Code (200)
        .with Body("LOWRISK"));
```

<caption>Figure 6: Mock component piece of code, which implements the events !/EvaluateRisk and !ok of the test case of Figure 4</caption>
# 4 CONCLUSION

In this paper, we present an original solution to generate mutated test cases for testing the security of RESTful APIs. We propose a list of 18 mutation operators specialised in the generation of security test cases. Subsequently, we introduced an algorithm allowing to generate concrete test scripts and mock components by means of these operators. A significant contribution of this algorithm is its ability to generate mock components to test a RESTful API in isolation. We provide a preliminary evaluation of our algorithm to study the effectiveness of the mutated test cases and how its efficiency. This part is detailed in (Salva and Sue, 2024). Our results demonstrate its capability to construct hundreds of test cases and mock components within minutes, and show good scalability. Besides, the mutants enable the detection of weaknesses in REST APIs and enhance code coverage.

At the moment, our mutation operators allow to infer mutants that mimic attacks performed by one test step. As part of future work, we aim to define more sophisticated operators that could support the mutation of several steps at a time, thus constructing more complex attack scenarios.

# Acknowledgment

Research supported by the industrial chair on Digital Confidence https://www.ucafondation.fr/chaires/confiance-numerique/

# REFERENCES

Abu-Dabaseh, F. and Alshammari, E. (2018). Automated penetration testing: An overview. In *The 4th International Conference on Natural Language Computing, Copenhagen, Denmark*, pages 121–129.

Arcuri, A. (2018). Test suite generation with the many independent objective (mio) algorithm. *Information and Software Technology*, 104:195–206.

Arcuri, A. (2019). Restful api automated test case generation with evomaster. *ACM Trans. Softw. Eng. Methodol.*, 28(1).

Arcuri, A., Iqbal, M. Z., and Briand, L. (2011). Random testing: Theoretical results and practical implications. *IEEE transactions on Software Engineering*, 38(2):258–277.

CAPEC (2024). Common attack pattern enumeration and classification, https://capec.mitre.org/.

CWE (2024). Common weakness enumeration, https://cwe.mitre.org/.

Ferreira, J. and Paiva, A. C. R. (2019). Android testing crawler. In Piattini, M., da Cunha, P. R., de Guzmán, I. G. R., and Pérez-Castillo, R., editors, *Quality of Information and Communications Technology - 12th International Conference, QUATIC, Ciudad Real, Spain*, volume 1010 of *Communications in Computer and Information Science*, pages 313–326. Springer.

Köröglu, Y. and Sen, A. (2018). TCM: Test Case Mutation to Improve Crash Detection in Android. In *Proceedings of the 21st International Conference on Fundamental Approaches to Software Engineering*, pages 264–280. Springer.

Li, W., Le Gall, F., and Spaseski, N. (2018). A survey on model-based testing tools for test case generation. In Itsykson, V., Scedrov, A., and Zakharov, V., editors, *Tools and Methods of Program Analysis*, pages 77–89, Cham. Springer International Publishing.

Paiva, A., Restivo, A., and Almeida, S. (2020). Test case generation based on mutations over user execution traces. *Software Quality Journal*, 28.

Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Le Traon, Y., and Harman, M. (2019). Mutation testing advances: an analysis and survey. In *Advances in Computers*, volume 112, pages 275–378. Elsevier.

Phillips, I. C. C. (1987). Refusal testing. *Theor. Comput. Sci.*, 50:241–284.

Salva, S. and Sue, J. (2023). Automated test case generation for service composition from event logs. In *38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023 - Workshops, Luxembourg, September 11-15, 2023*, pages 127–134. IEEE.

Salva, S. and Sue, J. (2024). Security testing of restful apis with test case mutation, full paper. https://arxiv.org/abs/2403.03701.

Salva, S. and Zafimiharisoa, S. R. (2014). Model reverse-engineering of Mobile applications with exploration strategies. In *Ninth International Conference on Software Engineering Advances, ICSEA 2014*, Nice, France.

Skouloudi, C., Malatras, A., Naydenov, R., and Dede, G. (2018). Good practices for security of internet of things in the context of smart manufacturing testing.

Sue, J. and Salva, S. (2024). Security testing of restful apis with test case mutation, companion site. https://github.com/JarodSue/Restful-API-test-case-mutation.

Tretmans, J. (2008). *Model Based Testing with Labelled Transition Systems*, pages 1–38. Springer Berlin Heidelberg, Berlin, Heidelberg.

Xu, Z., Kim, Y., Kim, M., Rothermel, G., and Cohen, M. B. (2010). Directed test suite augmentation: Techniques and tradeoffs. FSE ’10, page 257–266, New York, NY, USA. Association for Computing Machinery.

Xuan, J., Xie, X., and Monperrus, M. (2015). Crash reproduction via test case mutation: Let existing test cases help. In *Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering*, ESEC/FSE 2015, page 910–913, New York, NY, USA.
# 4 CONCLUSION

In this paper, we present an original solution to generate mutated test cases for testing the security of RESTful APIs. We propose a list of 18 mutation operators specialised in the generation of security test cases. Subsequently, we introduced an algorithm allowing to generate concrete test scripts and mock components by means of these operators. A significant contribution of this algorithm is its ability to generate mock components to test a RESTful API in isolation. We provide a preliminary evaluation of our algorithm to study the effectiveness of the mutated test cases and how its efficiency. This part is detailed in (Salva and Sue, 2024). Our results demonstrate its capability to construct hundreds of test cases and mock components within minutes, and show good scalability. Besides, the mutants enable the detection of weaknesses in REST APIs and enhance code coverage.

At the moment, our mutation operators allow to infer mutants that mimic attacks performed by one test step. As part of future work, we aim to define more sophisticated operators that could support the mutation of several steps at a time, thus constructing more complex attack scenarios.

# Acknowledgment

Research supported by the industrial chair on Digital Confidence https://www.ucafondation.fr/chaires/confiance-numerique/

# REFERENCES

Abu-Dabaseh, F. and Alshammari, E. (2018). Automated penetration testing: An overview. In *The 4th International Conference on Natural Language Computing, Copenhagen, Denmark*, pages 121–129.

Arcuri, A. (2018). Test suite generation with the many independent objective (mio) algorithm. *Information and Software Technology*, 104:195–206.

Arcuri, A. (2019). Restful api automated test case generation with evomaster. *ACM Trans. Softw. Eng. Methodol.*, 28(1).

Arcuri, A., Iqbal, M. Z., and Briand, L. (2011). Random testing: Theoretical results and practical implications. *IEEE transactions on Software Engineering*, 38(2):258–277.

CAPEC (2024). Common attack pattern enumeration and classification, https://capec.mitre.org/.

CWE (2024). Common weakness enumeration, https://cwe.mitre.org/.

Ferreira, J. and Paiva, A. C. R. (2019). Android testing crawler. In Piattini, M., da Cunha, P. R., de Guzmán, I. G. R., and Pérez-Castillo, R., editors, *Quality of Information and Communications Technology - 12th International Conference, QUATIC, Ciudad Real, Spain*, volume 1010 of *Communications in Computer and Information Science*, pages 313–326. Springer.

Köröglu, Y. and Sen, A. (2018). TCM: Test Case Mutation to Improve Crash Detection in Android. In *Proceedings of the 21st International Conference on Fundamental Approaches to Software Engineering*, pages 264–280. Springer.

Li, W., Le Gall, F., and Spaseski, N. (2018). A survey on model-based testing tools for test case generation. In Itsykson, V., Scedrov, A., and Zakharov, V., editors, *Tools and Methods of Program Analysis*, pages 77–89, Cham. Springer International Publishing.

Paiva, A., Restivo, A., and Almeida, S. (2020). Test case generation based on mutations over user execution traces. *Software Quality Journal*, 28.

Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Le Traon, Y., and Harman, M. (2019). Mutation testing advances: an analysis and survey. In *Advances in Computers*, volume 112, pages 275–378. Elsevier.

Phillips, I. C. C. (1987). Refusal testing. *Theor. Comput. Sci.*, 50:241–284.

Salva, S. and Sue, J. (2023). Automated test case generation for service composition from event logs. In *38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023 - Workshops, Luxembourg, September 11-15, 2023*, pages 127–134. IEEE.

Salva, S. and Sue, J. (2024). Security testing of restful apis with test case mutation, full paper. https://arxiv.org/abs/2403.03701.

Salva, S. and Zafimiharisoa, S. R. (2014). Model reverse-engineering of Mobile applications with exploration strategies. In *Ninth International Conference on Software Engineering Advances, ICSEA 2014*, Nice, France.

Skouloudi, C., Malatras, A., Naydenov, R., and Dede, G. (2018). Good practices for security of internet of things in the context of smart manufacturing testing.

Sue, J. and Salva, S. (2024). Security testing of restful apis with test case mutation, companion site. https://github.com/JarodSue/Restful-API-test-case-mutation.

Tretmans, J. (2008). *Model Based Testing with Labelled Transition Systems*, pages 1–38. Springer Berlin Heidelberg, Berlin, Heidelberg.

Xu, Z., Kim, Y., Kim, M., Rothermel, G., and Cohen, M. B. (2010). Directed test suite augmentation: Techniques and tradeoffs. FSE ’10, page 257–266, New York, NY, USA. Association for Computing Machinery.

Xuan, J., Xie, X., and Monperrus, M. (2015). Crash reproduction via test case mutation: Let existing test cases help. In *Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering*, ESEC/FSE 2015, page 910–913, New York, NY, USA.
<table><caption>Table 1: Test case mutation operators for the security testing of RESTful APIs</caption><thead><tr><td>Mutation</td><td>Description</td><td>Expected Behaviour</td><td>Mutation Condition</td></tr></thead><tbody><tr><td>Event Duplication</td><td>duplicate a request event to SUT</td><td>q <sup>1+(α),J</sup>→ pass with "crash" /∈ l ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>e == *</td></tr><tr><td>HTTP Verb Change</td><td>changing the HTTP verb of a request</td><td>q <sup>1+(α),J</sup>→ pass with status(α) := 405 ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>isReq(e(α)) == true</td></tr><tr><td>XSS attack</td><td>XSS attack</td><td>q <sup>1+(α),J</sup>→ pass with "crash" /∈ l ∧ contains("error", body(α)) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>body(α)! = "" ∨ header(α)! = ""</td></tr><tr><td>Cryptographic failures</td><td>Replay an event using untrusted connexion</td><td>q <sup>1+(α),J</sup>→ pass with contains("ERR_CERT_AUTHORITY_INVARIANT", body(α)) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>AIDD<sup>*</sup> = "*"</td></tr><tr><td>Token Removal</td><td>Delete a token in event</td><td>q <sup>1+(α),J</sup>→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>"token" ∈ l</td></tr><tr><td>Token Removal on the creation</td><td>Delete a token in event</td><td>q <sup>1+(α),J</sup>→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>"token creation" ∈ l</td></tr><tr><td>Token Alteration</td><td>Replacing a token by another one; three types: expired authentication token, token existing but not for this session, and token not existing</td><td>q <sup>1+(α),J</sup>→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>"token" ∈ l</td></tr><tr><td>Stress Testing</td><td>replay events to the tested service a lot of times in a small window</td><td>q <sup>1+(α),J</sup>→ pass with "crash" /∈ l ∧ -contains("error", body(α)) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>e == "*"</td></tr><tr><td>SSRF Enforce "deny by default"</td><td>request or response from an unknown service</td><td>q <sup>1+(α),J</sup>→ pass with "crash" /∈ l ∧ (contains("error", body(α)) ∨ status(α) := 404) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>e == "*"</td></tr><tr><td>Body data manipulation</td><td>replay request using unauthorized data</td><td>q <sup>1+(α),J</sup>→ pass with (status(α) := 400 ∨ status(α) := 422) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>body(α)! = "" ∨ header(α)! = ""</td></tr><tr><td>Cookie manipulation</td><td>change a cookie to inject an attack</td><td>q <sup>1+(α),J</sup>→ pass with status(α) := 400 ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>cookies(α)! = ""</td></tr><tr><td>Failed Login Attempt Duplication</td><td>duplicating login event with wrong credentials</td><td>q <sup>1+(α),J</sup>→ pass with "crash" /∈ l ∧ contains("error: TooManyFailedAttempt", body(α)) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>isReq(e(α)) ∧ "login" ∈ l</td></tr><tr><td>Path manipulation</td><td>change URL to get unauthorised access to data</td><td>q <sup>1+(α),J</sup>→ pass with status(α) := 404 ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>isReq(e(α))</td></tr><tr><td>SQL injection</td><td>manipulate input data to inject SQL code</td><td>q <sup>1+(α),J</sup>→ pass with (status(α) := 400 ∨ contains("error", body(α))) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>body(α)! = ""</td></tr><tr><td>Session management</td><td>add a (long) delay during which no reaction should be observed before the next event</td><td>q <sup>1+(α),J</sup>→ pass with status(α) := 401 ∧ contains("error: sessionterminated", body(α)) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>e == "*"</td></tr><tr><td>Information leakage</td><td>modify a request to get access to sensitive information</td><td>q <sup>1+(α),J</sup>→ pass with status(α) := 401 ∧ "crash" /∈ l ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>isReq(e(α))</td></tr><tr><td>Dependee service shutdown</td><td>shutdown a mock component after requesting it</td><td>q <sup>1+(α),J</sup>→ pass with "crash" /∈ l ∧ (contains("error: connexiontimedout", body(α)) ∨ status(α) := 408) ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>q <sup>1+(α),J</sup>→ with "mock" ∈ l</td></tr><tr><td>Buffer overflow</td><td>overflow input data for trying to crash a server</td><td>q <sup>1+(α),J</sup>→ pass with status(α) := 400 ∧ "crash" /∈ l ∧ from(q <sup>1+(α),J</sup>→ pass) = SUT</td><td>e == "*"</td></tr></tbody></table>
<table><caption>Table 1: Test case mutation operators for the security testing of RESTful APIs</caption><thead><tr><td>Mutation</td><td>Description</td><td>Expected Behaviour</td><td>Mutation Condition</td></tr></thead><tbody><tr><td>Event Duplication</td><td>duplicate a request event to SUT</td><td>q <sup>1+(α),l</sup>→ pass with "crash" /∈ l ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>e == *</td></tr><tr><td>HTTP Verb Change</td><td>changing the HTTP verb of a request</td><td>q <sup>1+(α),l</sup>→ pass with status(α) := 405 ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>isReq(e(α)) == true</td></tr><tr><td>XSS attack</td><td>XSS attack</td><td>q <sup>1+(α),l</sup>→ pass with "crash" /∈ l ∧ contains("error", body(α)) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>body(α)! = "" ∨ header(α)! = ""</td></tr><tr><td>Cryptographic failures</td><td>Replay an event using untrusted connexion</td><td>q <sup>1+(α),l</sup>→ pass with contains("ERR_CERT_AUTHORITY_INVARIANT", body(α)) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>AIDD<sup>*</sup> = "*"</td></tr><tr><td>Token Removal</td><td>Delete a token in event</td><td>q <sup>1+(α),l</sup>→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>"token" ∈ l</td></tr><tr><td>Token Removal on the creation</td><td>Delete a token in event</td><td>q <sup>1+(α),l</sup>→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>"token creation" ∈ l</td></tr><tr><td>Token Alteration</td><td>Replacing a token by another one; three types: expired authentication token, token existing but not for this session, and token not existing</td><td>q <sup>1+(α),l</sup>→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>"token" ∈ l</td></tr><tr><td>Stress Testing</td><td>replay events to the tested service a lot of times in a small window</td><td>q <sup>1+(α),l</sup>→ pass with "crash" /∈ l ∧ -contains("error", body(α)) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>e == "*"</td></tr><tr><td>SSRF Enforce "deny by default"</td><td>request or response from an unknown service</td><td>q <sup>1+(α),l</sup>→ pass with "crash" /∈ l ∧ (contains("error", body(α)) ∨ status(α) := 404) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>e == "*"</td></tr><tr><td>Body data manipulation</td><td>replay request using unauthorized data</td><td>q <sup>1+(α),l</sup>→ pass with (status(α) := 400 ∨ status(α) := 422) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>body(α)! = "" ∨ header(α)! = ""</td></tr><tr><td>Cookie manipulation</td><td>change a cookie to inject an attack</td><td>q <sup>1+(α),l</sup>→ pass with status(α) := 400 ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>cookies(α)! = ""</td></tr><tr><td>Failed Login Attempt Duplication</td><td>duplicating login event with wrong credentials</td><td>q <sup>1+(α),l</sup>→ pass with "crash" /∈ l ∧ contains("error: TooManyFailedAttempt", body(α)) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>isReq(e(α)) ∧ "login" ∈ l</td></tr><tr><td>Path manipulation</td><td>change URL to get unauthorised access to data</td><td>q <sup>1+(α),l</sup>→ pass with status(α) := 404 ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>isReq(e(α))</td></tr><tr><td>SQL injection</td><td>manipulate input data to inject SQL code</td><td>q <sup>1+(α),l</sup>→ pass with (status(α) := 400 ∨ contains("error", body(α))) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>body(α)! = ""</td></tr><tr><td>Session management</td><td>add a (long) delay during which no reaction should be observed before the next event</td><td>q <sup>1+(α),l</sup>→ pass with status(α) := 401 ∧ contains("error: sessionterminated", body(α)) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>e == "*"</td></tr><tr><td>Information leakage</td><td>modify a request to get access to sensitive information</td><td>q <sup>1+(α),l</sup>→ pass with status(α) := 401 ∧ "crash" /∈ l ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>isReq(e(α))</td></tr><tr><td>Dependee service shutdown</td><td>shutdown a mock component after requesting it</td><td>q <sup>1+(α),l</sup>→ pass with "crash" /∈ l ∧ (contains("error: connexiontimedout", body(α)) ∨ status(α) := 408) ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>q <sup>1+(α),l</sup>→ with "mock" ∈ l</td></tr><tr><td>Buffer overflow</td><td>overflow input data for trying to crash a server</td><td>q <sup>1+(α),l</sup>→ pass with status(α) := 400 ∧ "crash" /∈ l ∧ from(q <sup>1+(α),l</sup>→ pass) = SUT</td><td>e == "*"</td></tr></tbody></table>