# A Maturity Model for Trustworthy AI Software Development
## 3 A Maturity Model for Trustworthy AI Software
### \"Pr9\" Case Study: Evaluating the Effectiveness of AI-MM
#### 4.2.2 Assessment Results  
As described in Section 3, AI-MM consists of 5 process areas, 13 processes, and 53 base practices. In addition, the base practices can be divided into common AI practices, fairness-specific practices, and safety-specific practices. In order to confirm the effectiveness of AI-MM in various aspects, we analyzed the assessment results of 13 AI projects with subsets of AI-MM, as follows: \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline
**Process** & **SRA** & **SAD** & **DCO** & **DCL** & **DPR** & **TPM** & **PEA** & **SEA** & **FAM** & **SSE** & **SSP** & **AIN** & **AOM** \\ \hline Software Requirements Analysis (SRA) & 1.000 & & & & & & & & & & & & \\ \hline Software Architecture Design (SAD) & **0.710** & 1.000 & & & & & & & & & & \\ \hline Data Collection (DCO) & 0.127 & 0.399 & 1.000 & & & & & & & & & \\ \hline Data Cleaning (DCL) & 0.307 & 0.354 & 0.347 & 1.000 & & & & & & & & \\ \hline Data Preprocessing (DPR) & 0.583 & 0.596 & 0.464 & **0.760** & 1.000 & & & & & & \\ \hline Training Process Management (TPM) & 0.112 & 0.177 & 0.698 & 0.226 & 0.388 & 1.000 & & & & & \\ \hline Performance Evaluation of AI model (PEA) & **0.736** & 0.581 & 0.356 & 0.372 & **0.741** & 0.443 & 1.000 & & & & \\ \hline Safety Evaluation of AI model (SEA) & 0.446 & 0.539 & 0.164 & 0.191 & 0.465 & 0.134 & 0.680 & 1.000 & & & \\ \hline Final AI model Management (FAM) & **0.803** & **0.763** & 0.308 & 0.608 & 0.674 & 0.363 & 0.656 & 0.444 & 1.000 & & & \\ \hline System Safety Evaluation (SSE) & 0.290 & 0.449 & \(-\)0.025 & 0.193 & 0.449 & \(-\)0.225 & 0.205 & 0.474 & 0.353 & 1.000 & & \\ \hline System Safety Preparedness (SSP) & 0.035 & 0.429 & \(-\)0.051 & \(-\)0.083 & \(-\)0.019 & \(-\)0.181 & \(-\)0.304 & 0.022 & 0.166 & 0.621 & 1.000 & \\ \hline AI Infrastructure (AIN) & \(-\)0.046 & 0.186 & 0.116 & \(-\)0.227 & 0.196 & 0.147 & 0.379 & 0.447 & \(-\)0.132 & 0.436 & 0.156 & 1.000 & \\ \hline AI model Operation Management (AOM) & \(-\)0.187 & 0.006 & 0.401 & 0.107 & 0.253 & 0.528 & 0.289 & 0.446 & 0.007 & 0.141 & \(-\)0.067 & 0.555 & 1.000 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Pearson coefficients among processes of AI-MM based on 13 AI projects (bold font is used for coefficients with an R value above 0.7). \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline
**Process** & **Base Practice** & **Pr1** & **Pr2** & **Pr3** & **Pr4** & **Pr5** & **Pr6** & **Pr7** & **Pr8** & **Pr9** & **Pr10** & **Pr11** & **Pr12** & **Pr13** \\ \hline Software architecture design & User-conscious design & N/A & O & O & O & O & N/A & O & O & N/A & X & O & O & X \\ \hline Data preprocessing result record & O & O & X & X & X & O & X & N/A & O & O & O & X & O \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Part of Fisher’s exact test results among base practices (\(p\)-value is below 0.05). * AI-CMM: A maturity model with 24 Common AI practices
* AI-FMM: A maturity model with 13 Fairness practices
* AI-CFMM: A maturity model with 37 Common and Fairness practices
* AI-MM: A total maturity model with 53 Common, Fairness, and Safety practices  
The language translation service is assessed to have high capability levels with respect to AI fairness processes; accordingly, it shows a high level of fairness as a language translation service for a global IT company. Some results of applying AI-MM (Table 12) are explained as follows:  
* AI-CMM (Common): Processes and base practices commonly applied to AI software development are well established at the organizational level, and the development team of this project also executes and respects the processes and practices well. * AI-FMM (Fairness): This project is assessed to have level 2 or 3 capabilities in most processes. However, organizational preparation and efforts to ensure the fairness of AI are partially insufficient compared to the common maturity model (AI-CMM). Table 13 shows the AI-FMM maturity level in detail.