# An Open Relation Extraction System for Web Text Information
## 4 Experimental Results and Experiment Analysis
### Experiments
Adding a topic model was beneficial to diversity, but it slightly reduced the accuracy. The problem can be further studied by optimizing the topic model and the appropriate setting of topic weights. Table 3 shows the experimental results from 15-shot setting. Continuing the previous trend, the recall rate of ORES was improved, but the precision and F1 value were lower than with Neural Snowball. Compared with Topic Snowball, the performance of ORES was improved. This shows that injecting external knowledge is effective. To further analyze the iterative process of ORES, we find that the classifier obtained from the three experimental settings showed similar trends in performance. The performance of the ORES was not good enough in the 10-seed and 15-seed settings. However, few-shot focuses on scenarios with fewer instances. For the five-shot setting, the performance of the ORES improved significantly. In general, the model's performance can be improved by increasing the number of seed examples. However, the low-resource setting that is stronger for small-sample relation extraction can better test the model's generalization ability. To further evaluate the performance of ORES with few instances, we set up experiments with fewer instances. As shown in Figure 8, ORES has higher precision and recall than Neural Snowball (NS) with fewer shots. This indicates that injecting external knowledge and introducing topic models are effective for few-shot relation learning. As the number of seed instances increases, the accuracy of NS begins to be higher than that of ORES after five shots. This is related to the accumulation of noise in the data mined by distant supervision. When the data noise increases, it is easy to overfit. As shown in Figure 9, when there were fewer seed instances, ORES's accuracy was higher, indicating that the topic similarity of the instances is more valuable than the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Precision (\%)** & **Recall (\%)** & **F1 (\%)** \\ \hline BREDS & 28.29 & 17.02 & 21.25 \\ Fine-tuning (CNN) & 47.58 & 38.36 & 42.28 \\ Distant Supervision (CNN) & 42.48 & 48.64 & 45.35 \\ Fine-tuning (BERT) & 5.87 & 55.19 & 57.43 \\ Distant Supervision (BERT) & 38.45 & 76.12 & 51.09 \\ Neural Snowball (BERT) & 60.50 & 62.20 & 61.34 \\ \hline Our Topic Snowball (tBERT) & 49.28 & 78.18 & 56.33 \\ Our ORES (t-KBERT) & 51.47 & 75.01 & 61.05 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The 10-shot experimental results on various previous baseline models. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Precision (\%)** & **Recall (\%)** & **F1 (\%)** \\ \hline BREDS & 25.24 & 17.96 & 20.99 \\ Fine-tuning (CNN) & 74.70 & 48.03 & 58.46 \\ Distant Supervision (CNN) & 43.70 & 54.76 & 48.60 \\ Fine-tuning (BERT) & 81.60 & 58.92 & 68.43 \\ Distant Supervision (BERT) & 35.48 & 80.33 & 49.22 \\ Neural Snowball (BERT) & 78.13 & 66.87 & 72.06 \\ \hline Our Topic Snowball (tBERT) & 50.08 & 79.88 & 58.16 \\ Our ORES (t-KBERT) & 56.23 & 78.15 & 66.74 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The 15-shot experimental results on various previous baseline models. semantic similarity. ORES worked better when there were few samples, but as the number of seed instances increased, the impact of semantic similarity on classifier performance increased. If the topic weight is too large, it will affect the further improvement of the classifier performance.