# Resilience and Resilient Systems of Artificial Intelligence: Taxonomy, Models and Methods
## 5 Models and Methods to Ensure and Assess AIS Resilience
### Proactivity and Robustness
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Approach** & **Capability** & **Weakness** & **Methods and Algorithms** \\ \hline \multirow{6}{*}{Out-of-domain generalization of disturbance} & & & Domain randomization [93] \\ \cline{3-5}  & & & Adversarial data augmentation [94] \\ \cline{3-5}  & & Less useful for real concept drift & Domain-invariant representation [95] \\ \cline{3-5}  & & & Heterogeneous-domain knowledge propagation [96] \\ \hline \multirow{2}{*}{Ensemble selection} & Absorption of disturbance & Not suitable for large deep neural networks and high-dimensional data & Dynamically weighted Ensemble [97] \\ \hline \multirow{6}{*}{Concept Drift detection} & & & Data distribution-based detection [99] \\ \cline{3-5}  & & & Performance-based detection [100] \\ \cline{3-5}  & Rejection Option in the presence of Concept drift with the subsequent adaptation & Not reliable when exposed to noise, adversarial attacks or faults & Multiple hypothesis-based detection [101] \\ \cline{3-5}  & & & Contextual-based detection [102; 103] \\ \hline \multirow{6}{*}{Out-of-distribution detection} & & & Data and training based epistemic uncertainties estimation [104] \\ \cline{3-5}  & & & Model-based epistemic uncertainties estimation [105] \\ \cline{3-5}  & & & Model-based epistemic uncertainties estimation [105] \\ \cline{3-5}  & & & Post-hoc epistemic uncertainties estimation [106] \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Approaches and algorithms of drift detection and mitigating. Methods based on data and training are aimed at obtaining representations which can produce accurate uncertainty evaluation where necessary [104]. In this category of methods the uncertainties are calibrated using additional data. Additional data can be generated by a generative model, or obtained by perturbing the original data with adversarial attacks, or taken from a separate additional dataset. In model-based methods, the uncertainty evaluator is built into the architecture of the model. Such approaches can take distributions over the model parameters [105]. The uncertainty between the training and test distribution are considered to arise from uncertainties in the model itself. Model-based methods rely on probabilistic forward pass, allowing the weight uncertainties to propagate through the network and giving a probability distribution for the output. These methods can be used with Bayesian neural networks and a hypernetwork which generate the weights for a target neural network. The key advantage of hypernetwork is its flexibility and scalability. Some model-based methods might leverage gradients, ensembles, artefacts of dynamic and stochastic training processes, earlier snapshots of the network and other information to evaluate uncertainty. Post-hoc methods focus on the output of the model and use it to calibrate the predictive uncertainty [106]. Post-hoc methods provide a more accurate reflection of the prediction confidence based on transformation of AIS-output. It can be implemented by a simple temperature scaling or by more complicated means. For example, [107] introduces an auxiliary class which identifies miss-classified samples and explicitly calibrates AI-model on out-of-distribution datasets. Post-hoc methods can be incorporated with any AI architecture and, arguably, any AI model. The analysis of Table 3 shows that it is possible to increase robustness to covariate shift by Out-of-domain generalization and reduce the impact of a certain level of concept drift in the case of a moderately sized models without high dimensionality of the feature space. Concept drift detection is not reliable enough when exposed to noise, adversarial attacks or faults. Whilst there are methods to ensure Out-of-distribution data detection, they are usually computationally expensive.