# Resilience and Resilient Systems of Artificial Intelligence: Taxonomy, Models and Methods
## 5 Models and Methods to Ensure and Assess AIS Resilience
### Methods to Assess AIS Resilience
Resourcefulness and redundancy are closely interrelated, for example, resourcefulness can create redundancies that did not exist before. In addition, resourcefulness and redundancy can affect the speed and time of recovery. Adding resources can reduce the recovery time compared to what would be expected under standard conditions. Theoretically, if infinite resources were available, the recovery time would asymptotically tend to zero. In practice, even with enormous financial and labor resources, there is a certain minimum recovery time. However, recovery time can be quite long even with a large amount of resources due to inadequate planning, organizational failures, or ineffective policies [16]. Resourcefulness and robustness are also interrelated. It can be argued that investing in limiting initial losses (increasing robustness) may in some cases be the best approach to increasing resilience, as this automatically leads to further reductions in recovery time. In order to simultaneously take into account time and performance variables when assessing system resilience, various variants of integral indicators have been developed [17; 18]. These indicators typically characterize the difference or ratio of nominal performance and performance loss over time due to disturbances. For convenience, the integral resilience indicator can be expressed in a normalized form as:  
\[\mathrm{R}\equiv\frac{\frac{1}{|\mathrm{E}|}\sum_{\mathrm{E}}\int_{\mathrm{t= 0}}^{\mathrm{T}_{\mathrm{c}}}\mathrm{C(t)dt}}{\int_{\mathrm{t=0}}^{\mathrm{T}_ {\mathrm{c}}}\mathrm{C^{nominal}(t)dt}}. \tag{6}\]  
\(\mathrm{C(t)}\) is a function of the dependence of the current value of system performance or functionality on time;  
\(\mathrm{C^{nominal}(t)}\) is the value of the system performance in the normal (nominal) functional state, which is entered into the formula to map the values of the integral resilience indicator to the interval [0, 1];  
\(\mathrm{T_{c}}\) is a control period, which is selected based on the results of a preliminary assessment of the average interval between events of disturbance;  
E is a set of disturbance events during the control period. In the case of machine learning, the time axis denotes the amount of training or test data passed through the AIS or the number of iterations, meaning mini-batches of optimal size. Resilience indicators are evaluated in relation to a certain type of perturbation. In the case of neural networks, typical disturbances are adversarial attacks, faults, and concept drift. At the same time, different weights of neural networks have different importance and impact on AIS performance. In addition, an error in the higher bits of the tensor value leads to a greater distortion of the results than an error in the lower bits. Similarly, the effectiveness of adversarial attacks with the same perturbation level can vary greatly depending on the spatial distribution of the perturbed pixels. Therefore, statistical characteristics should be used to evaluate and compare the resilience of the AIS to corrupted tensors or perturbed data. Such statistical characteristics can beobtained from a large number of experiments. For simplicity, we can consider the median value (MED) and interquartile range (IRQ) of the performance and resilience indicators. The TensorFlowI2 library, which is capable of emulating software and hardware failures, has become popular for testing AI for fault tolerance [131]. It has been noted [70] that one of the most difficult types of faults to absorb is the random bit-flip injection into each layer of the model, with a randomly selected fixed fraction of the tensors (failure rate) and one or few randomly selected bit for inversion. In [30; 132], it is proposed not to rely on specific aspects of the model architecture and learning algorithm, such as gradients, to test the model for resistance to noise and adversarial attacks. Instead, testing is based on black box attacks, which expands the family of AIS that can be tested.