## 7 Indicator System and Effectiveness Verification
### Validation of Method Effectiveness  
Validating the effectiveness of different models and choosing the best methods to conduct topic identification and prediction is also significant. The methods for verifying effectiveness can generally be divided into the following three categories, on the basis of the indicator system summarized above:  
* Time series models for prediction based on quantitative indicators. The prediction indicators involved in such tasks are mostly the above basic scalar indicators, such as scale, word frequency, popularity intensity or novelty, which are essentially regression tasks. Such tasks also use common error statistics indicators used in time series analysis and prediction to evaluate effectiveness. For example, [PERSON] and [PERSON] used LSTM neural networks to characterize the time series characteristics of the evolution of subject topic popularity, and built a subject topic popularity prediction model based on LSTM [9]. [PERSON] et al. used the ARIMA model and the exponential smoothing method to predict the trends of topics [88], characterized by topic intensity and novelty. [PERSON] et al. identified frontier themes with topic emergence degree and an attention index, and studied the lag effect of topic diffusion evolution using an autoregressive distribution lag model [46]. Most of these use root means square error (RMSE), mean absolute error (MAE), relative error (RE) and other indicators to evaluate the accuracy of prediction, and RMSE is the square root of the ratio of the sum of squares of the deviations between the predicted value and the actual value to the predicted times. MAE is the ratio of the absolute deviation between the predicted value and the actual value to the number of predictions, and is the average value of the absolute error. RE is the ratio of the D value between the predicted value and the actual value to the actual value. The smaller the value of these indicators, the higher the prediction accuracy of the model. Some studies also use R2 to represent the prediction effect of the model. The closer R2 is to 1, the greater the goodness of fit of the model. * Link prediction model based on complex networks  
Such tasks are mostly based on multi-relationship and multi-type networks, such as co-words/citations. Topics are identified or predicted through the common indicators of centrality or link prediction tasks, such as the common neighbor indicators and the Jaccard coefficient. For such models, the evaluation standard AUC (area under the ROC curve) is the most commonly used precision measurement indicator. For example, [PERSON] took research in the field of deep learning as the object, took related topics as the nodes, and predicted the possibility of emerging topics through link prediction [47]. [PERSON] took the field of virus nucleic acid detection technology as an example, constructed a weighted co-occurrence network of technology subject fields, and proposed an improved technology prediction method based on the link prediction algorithm [84]. [PERSON] et al. introduced the method of complex network link prediction combined with a neural network algorithm to build a dynamic prediction network for emerging technology development based on the patent data of Derwent [83]. The AUC is the area under the ROC curve. The ROC curve is a characteristic curve based on a confusion matrix, wherein the abscissa is the false positive rate and the ordinate is the true rate. When evaluating the effectiveness of different methods, the area under the ROC curve is the most intuitive basis for measuring the accuracy of each index in the link prediction algorithm as a whole. * Classification model of topic recognition based on deep learning. The process of identifying topics is also a classification problem in essence. Whether it is the classification of topics into corresponding fields, or whether a topic is emerging/subversive/breakthrough in nature or not, it can be included in this column.