## 7 Evaluation
### Discussion  
#### 7.4.1 Case Labeling  
From the experimental results in 7.3, we can see that our MParser had better results. We also calculated each case match rate (MR) for all words (\(N\) = 505) between experts and MParser outputs as the ratio of _MatchedCase_ to _TotalCase_. From the results shown in Table 10, we found that PRE and GEN cases had extremely high MRs, which were 0.986 and 0.959, respectively. ADV, ACC, and LIN cases came next. The MR of DAT was relatively low because of the differences in the judgment of the infinitive. To our surprise, the MR of the NOM case was relatively low. Through one-to-one analysis of sentences, we found that when nouns were under the COM (\(\text{COM}_{\text{n}}/\text{COM}_{\text{v}}\)) structure, some experts still labeled the COM case for nouns, and our MParser identified the nouns as NOM case. For COM, \(\text{COM}_{\text{n}}\), and \(\text{COM}_{\text{v}}\) cases, the MR was not very high because the experts had different labels on which COM case to use for prepositions. However, if the COM case was considered a general COM case, \(\text{COM}_{\text{all}}\), the average of the MR achieved a very high score, which was 0.920, indicating a consensus on the COM case. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Group**} & **Inter-Annotator** & \multirow{2}{*}{\(\kappa\) *} & \multirow{2}{*}{\(\kappa_{\text{avg.}}\)} & **Intra-Annotator** & \multirow{2}{*}{\(\kappa\) *} & \multirow{2}{*}{\(\kappa_{\text{avg.}}\)} \\  & **(Expert 2, Expert 3)** & & & & \\ \hline \multirow{2}{*}{\begin{tabular}{c} Group **A** \\ _(N = 15)_ \\ \end{tabular} } & _(Expert 1, Expert 2)_ & _0.688_ & & _(Expert 1, MParser)_ & _0.637_ & \\  & _(Expert 2, Expert 3)_ & _0.831_ & _0.741_ & _(Expert 2, MParser)_ & _0.853_ & _0.753_ \\  & _(Expert 1, Expert 3)_ & _0.703_ & & _(Expert 3, MParser)_ & _0.768_ & \\ \hline \multirow{2}{*}{\begin{tabular}{c} Group **B** \\ _(N = 15)_ \\ \end{tabular} } & _(Expert 1, Expert 2)_ & _0.597_ & & _(Expert 1, MParser)_ & _0.537_ & \\  & _(Expert 2, Expert 3)_ & _0.766_ & _0.663_ & _(Expert 2, MParser)_ & _0.685_ & _0.668_ \\  & _(Expert 1, Expert 3)_ & _0.627_ & & _(Expert 3, MParser)_ & _0.781_ & \\ \hline \multirow{2}{*}{\begin{tabular}{c} Group **C** \\ _(N = 15)_ \\ \end{tabular} } & _(Expert 1, Expert 2)_ & _0.648_ & & _(Expert 1, MParser)_ & _0.603_ & \\  & _(Expert 2, Expert 3)_ & _0.673_ & _0.642_ & _(Expert 2, MParser)_ & _0.779_ & _0.734_ \\  & _(Expert 1, Expert 3)_ & _0.605_ & & _(Expert 3, MParser)_ & _0.821_ & \\ \hline \multirow{2}{*}{\begin{tabular}{c} Group **D** \\ _(N = 15)_ \\ \end{tabular} } & _(Expert 1, Expert 2)_ & _0.694_ & & _(Expert 1, MParser)_ & _0.613_ & \\  & _(Expert 2, Expert 3)_ & _0.775_ & _(Expert 2, MParser)_ & _0.835_ & _0.724_ \\  & _(Expert 1, Expert 3)_ & _0.593_ & & _(Expert 3, MParser)_ & _0.724_ & \\ \hline \multirow{2}{*}{\
\begin{tabular}{c} Group **E** \\ _(N = 15)_ \\ \end{tabular} } & _(Expert 1, Expert 2)_ & _0.686_ & & _(Expert 1, MParser)_ & _0.616_ & \\  & _(Expert 2, Expert 3)_ & _0.837_ & _0.730_ & _(Expert 2, MParser)_ & _0.749_ & _0.706_ \\  & _(Expert 1, Expert 3)_ & _0.668_ & & _(Expert 3, MParser)_ & _0.753_ & \\ \hline \multirow{2}{*}{
\begin{tabular}{c} **Avg.** \\ \end{tabular} } & _Substantial_ & _0.693_ & _Substantial_ & & _0.717_ \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Kappa agreement between experts and MParser.