## 4 Experiments and Results
### Case Study  
To show the capabilities of the noise-detection strategy, we selected four representative instances in the training process for a case study. As shown in Table 3, each instance had a corresponding _sml_ calculated in Section 3.3. If the _sml_ exceeds the threshold \(\phi\), the label of instance will be dynamically modified to NA during training, \(r\) is the original relation label r and \(r^{\prime}\) is the new label modified by Noise Detection Strategy. The correct relation of the current instance is marked in blue, and the wrong relationis marked in red. From the table, we can see that: 1) Except for the first instance, none of the instances were true positive instances. The second and third instances were detected to have no relation in the instance because of their _sml_. The second instance was noisy and its label was correctly modified to NA according to the noise-detection strategy. 2) The original label of the fourth instance was _people/person/place_of_birth_, but its correct label was _people/person/place_lived_. The noise was not detected as the head entity \"Peggy\" had a relation _people/person/place_of_birth_ with \"Miami\". Thus, noise is hard to detect when there are multiple relations in an instance.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**MODE** & \multicolumn{3}{c}{**Single**} & \multicolumn{3}{c}{**Multiple**} & \multicolumn{3}{c}{**Whole**} \\ \hline
**Model** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]** & **AUC** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]** & **AUC** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]** & **[EMAIL_ADDRESS]** & **AUC** \\ \hline REDS2 & 69.1 & 61.1 & 57.4 & 37.5 & 82.4 & 79.6 & 76.6 & 57.59 & 75.3 & 70.4 & 65.5 & 44.65 \\ NDRE(-Label) & 69.4 & 67.6 & 58.1 & 38.2 & 86.4 & 82.4 & 78.7 & 61.40 & 76.5 & 71.7 & 67.6 & 46.72 \\ NDRE(-MHA) & 73.5 & 68.0 & 60.3 & 39.3 & 88.6 & 87.5 & 82.4 & 62.90 & 80.9 & 74.9 & 71.3 & 47.89 \\
**NDRE** & **83.9** & **75.3** & **63.8** & **42.0** & **98.6** & **90.9** & **86.4** & **66.06** & **89.0** & **85.9** & **76.6** & **50.94** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results of precision at recall (P@) and the area under the curves (AUC) under different testing modes.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Instances** & \(\mathbf{sml}\) (\(\mathbf{\phi}=0.16\)) & \(\mathbf{r}\) & \(\mathbf{r^{\prime}}\) & **Noise** & **Correct** \\ \hline [PERSON], one of the world ’s & & & & & \\ top goalkeepers, and & & & & & \\
**[[PERSON]]**, & & & people/person/ & & \\
**[Italy]**’s captain, were & & & nationality & & No & Yes \\ interviewed by the & & & & & \\ authorities & & & & & & \\ The director, & & & & & & \\
**[[PERSON]\_miller]**, had made & & & & & \\ the unflinching & & & & & \\ documentary “The cruise” & & & & & \\ in 1998, which followed the & & 0.68 & & & \\ gradual descent of a & & & & & \\ near-homeless **[New\_York city]** & & & & & \\
**city]** tour guide. & & & & & & \\ Then he needs to be able to relate to the poor when he & & & & & \\ goes to places like **[Brazil]** & & & & & \\ or Morocco or & & & & & & \\
**[Burkina Faso]**. & & & & & & \\
**[[PERSON]]** was born in Miami, & & & & & & \\ Florida and lived in & & & & & & \\
**[New\_York city]** for & & & & & & & \\
52 years. & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Case study: a real example in our dataset for the noise-detection strategy. “Noise” represents whether the instance was detected as a noisy one, and “Correct” represents whether the final label of the instance was correct.