# A Maturity Model for Trustworthy AI Software Development
## 3 A Maturity Model for Trustworthy AI Software
### \"Pr9\" Case Study: Evaluating the Effectiveness of AI-MM
#### 4.2.2 Assessment Results
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Process Dimension**} & \multicolumn{6}{c}{**Capability Dimension**} & \multicolumn{3}{c}{**Assessment Evidence**} \\ \hline
**Area** & **Process** & **Base Practice** & **L1** & **L2** & **L3** & **L4** & **L5** & **Result** & \\ \hline \multirow{4}{*}{P\&D} & \multicolumn{3}{c}{Fairness risk} & O & & & & \multicolumn{1}{c}{} & Fairness-related matters are managed by AI \\  & \multicolumn{3}{c}{factor analysis} & O & & & & \multicolumn{1}{c}{} & model cards and \\ \cline{2-7}  & \multicolumn{3}{c}{Fairness evaluation metric} & O & & & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & fairness evaluation metrics and stakeholder \\ \cline{2-7}  & \multicolumn{3}{c}{Stakeholder review—} & O & & & & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & fairness reviews are carried out at the project \\  & \multicolumn{3}{c}{fairness} & O & & & & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & level (not organization level) \\ \hline \multirow{2}{*}{SAD} & \multicolumn{3}{c}{Design considering fairness maintenance} & O & O & O & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table} 
Table 13: Assessment results of language translation by AI-FMM (Fairness). \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & & & & & **AI-CFMM** \\
**Process** & **AI-MM** & **AI-CMM** & **AI-FMM** & **AI-FMM** & **(Common \& \\  & & **(Common)** & **(Fairness)** & **(Fairness)** & **(Fairness)** \\ \hline \multicolumn{3}{c}{Software requirements analysis} & 3 & 3 & 2 & 3 \\ \hline \multicolumn{3}{c}{Software architecture design} & 3 & 3 & 3 & 3 \\ \hline \multicolumn{3}{c}{Data collection} & 2 & 3 & 2 & 2 \\ \hline \multicolumn{3}{c}{Data cleaning} & 1 & 1 & 0 & 1 \\ \hline \multicolumn{3}{c}{Data preprocessing} & 3 & 3 & 2 & 3 \\ \hline \multicolumn{3}{c}{Training process management} & 2 & 2 & 2 & 2 \\ \hline \multicolumn{3}{c}{Performance evaluation of the AI model} & 3 & 3 & N/A & 3 \\ \hline \multicolumn{3}{c}{Safety evaluation of the AI model} & 3 & - & - & - \\ \hline \multicolumn{3}{c}{Final AI model management} & 2 & 3 & 0 & 2 \\ \hline \multicolumn{3}{c}{System safety evaluation} & 3 & - & - & - \\ \hline \multicolumn{3}{c}{System safety preparedness} & 2 & - & - & - \\ \hline \multicolumn{3}{c}{AI infrastructure} & 3 & 3 & 3 & 3 \\ \hline \multicolumn{3}{c}{AI model operation management} & 1 & 1 & 1 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Assessment results of language translation by AI-MM. Specifically, to check the practicality of the fairness maturity assessment results, we looked into what fairness issues were detected during the actual language translation service and compared this service with a state-of-the-art commercial language translation service provided by a global IT company. Table 14 shows the comparison results for fairness issues identified in the two services. Both services have similar gender-bias issues, showing that our service has a fairly high and commercial-level quality of fairness, as compared to the global IT company service. Gender issues appear more frequently in Korean-to-English translations; unlike Korean, which uses more words that are not gendered, English often uses words that reveal specific genders. To provide the characteristics of the AI model transparently, an AI model card (Table 15) of this AI project was documented and used intensively to assess capabilities. For example, it was checked by an AI model card that shows fairness policies and design