# A Machine Learning Model to Predict Citation Counts of Scientific Papers in Otology Field

Authors: Yousef A. Alohali; Mahmoud S. Fayed; Tamer Mesallam; Yassin Abdelsamad; Fida Almuhawas; Abdulrahman Hagr
Journal: BioMed Research International
 (2022)

---

## Abstract

One of the most widely used measures of scientific impact is the number of citations. However, due to its heavy-tailed distribution, citations are fundamentally difficult to predict but can be improved. This study was aimed at investigating the factors and parts influencing the citation number of a scientific paper in the otology field. Therefore, this work proposes a new solution that utilizes machine learning and natural language processing to process English text and provides a paper citation as the predicted results. Different algorithms are implemented in this solution, such as linear regression, boosted decision tree, decision forest, and neural networks. The application of neural network regression revealed that papers’ abstracts have more influence on the citation numbers of otological articles. This new solution has been developed in visual programming using Microsoft Azure machine learning at the back end and Programming Without Coding Technology at the front end. We recommend using machine learning models to improve the abstracts of research articles to get more citations.


---

## Body

## 1. Introduction

In the research world where researchers publish the results of their work through research papers, one can consider the paper citations as one of the common indicators of the paper’s quality, importance, and relevance. This is in contrast to the software world where its success is measured by the number of downloads or to the social media world, like Facebook posts and YouTube videos, where the number of views/interactions is the major Key Performance Indicator (KPI) beside the scientific content of research articles; other factors influence the paper citations like social effects, author’s name, and the journal rank [1, 2].Most academic papers are scarcely cited while a few others are highly cited. Some factors such as the paper’s quality, journal impact, number of authors, visibility, and international cooperation are stronger predictors than others such as authors’ gender, age, and race and characteristics of results and discussion [3]. Moreover, as citations demonstrate a heavy-tailed distribution, with most publications receiving few citations, these simple measures are exceedingly difficult to estimate using traditional regression analysis [4, 5].Citation prediction of scholarly papers is of great significance in guiding funding allocations, recruitment decisions, and rewards. Models use multifeatures predictive through author-based, journal-based, and citation [6]. Funding agencies and researchers with limited time and resources increasingly seek metrics and models to quantify the potential impact of a collaboration or a proposal [7–9].The remainder of this paper is organized as follows. Section2 describes related works. Section 3 illustrates the dataset. Section 4 demonstrates using machine learning to implement the different models. Section 5 presents experimental results and analysis, while Section 6 demonstrates the Ring programming language and the Programming Without Coding Technology tool to build the citation prediction application and a user interface. Finally, we present the discussion, future work, and the conclusion in Section 7.


## 2. Related Work

In [10, 11], Newman conducted a study based on finding the relationship between the publication date, topic, and an early number of citations. He identified several papers that could have a high impact in the future. In a dataset of 2000 papers, he expected that 50 papers will do the best. After five years, on average these papers received 23 times as many citations as the initial count and 15 times as many as the average paper in a randomly drawn control group that started with the same number of citations.In [12], Dong et al. used statistical methods to know if the paper will increase the h-index. They studied the correlation between the citations and many factors related to the paper’s author, content, venue, social, and references.In [13], the authors used a neural network to predict the citations based on features like paper ID, title, author score, number of published papers by the author, average download rates, and average number of citations for the author.In [14], the authors presented a study on biomedical research papers, they built a model using support vector machines (SVMs) using features like title, abstract, number of articles (for the first author), number of citations (for the first author), number of articles (for the last author), number of citations (for the last author), publication type, number of authors, number of institutions, and journal impact factor.In [15], the authors developed a machine learning model and a web-based h-index predictor using the author h-index, total publications, and the absolute year of the first publication by the author. Also, the application support prediction uses paper information like title, authors, year, and abstract. The dataset contains 1,712,433 authors with 2,092,356 papers from computer science venues held until 2012. They used logistic regression (LRC), support vector machine (SVM), naive Bayes (NB), radial basis function network (RBF), bagged decision trees (BAG), and random forest (RF).In [16], a dataset containing 1086 papers from the Bioinformatics journal was used. The authors used Bayesian networks (naive Bayes and K2), logistic regression, decision trees, and the K-nearest neighbor (K-NN) algorithm to predict the citations. The accuracy of naive Bayes and logistic regression supervised classification methods was 89.4% and 91.5%, respectively.In [17], the authors used a dataset containing 8 million bibliographic entries spanning over 3 million unique authors. They used Shannon entropy and Jensen-Shannon divergence to model the effects of each author’s influence and the words in the title of the paper. They used naive Bayes, logistic regression, support vector machine (SVM), random forest, and boosted trees and achieved an accuracy of 88%.In [18], the authors used multivariate analyses in three journals in the field of social-personality psychology. They discovered that the author’s gender and nationality, collaboration, and university prestige do not predict the impact. But the first author’s eminence, journal prestige, and article length predict the impact.Research about the impact of scientific articles mainly focuses on two interrelated questions: how to assess the past impact of an article and how to accurately predict its future impact. This includes using techniques like quantile regression, multivariate analysis, multivariate analysis random forest classifier long term, correlation analysis, and linear regression analysis [19]. Some of the open challenges could include sleeping beauty, multidimensional prediction, and rising star prediction [20, 21].In [22], the authors used a dataset of 38 million papers and 19 million authors. They focus on publications dated from 2000 to 2005 in seven key domains (12.7 million papers and 3 million authors). The domains are CS, biology, chemistry, medicine, engineering, mathematics, and physics. They used features like authors, institutions, venues, reference network (citations), and content similarity. They used a support vector regression machine (SVR) model, and the achieved accuracy varies between 17% and 39% based on the domain. The accuracy of the prediction is calculated using the R-squared of the predictions with actual citations. The R-squared (coefficient of determination) evaluates the scatter of the data points around the fitted regression line.In [23], the authors discovered the association between two variables and the paper citations. These variables are the sum of repetition of keywords in abstract divided by abstract length and the frequency of paper’s keyword per journal. These results are based on using a dataset of 5875 papers from 12 journals in the education.In [24], the authors did a study that proves that journals that publish papers with shorter titles receive more citations per paper. In [25], the authors did another study that proves that articles with short titles describing the results are cited more often.In [26], the authors did a study using a dataset of 6122 papers related to environmental modeling. They used features like citation count, year, page count, author count, author name, journal, abstract length, title length, and special issue. They discovered that the number of citations could be predicted with no knowledge about the paper quality.Scientific breakthroughs are rare events. In [27], the authors developed methods that combine curve fitting and thresholding strategies for the early detection of candidate breakthrough papers.In [28], the authors discovered that the BP neural network significantly outperformed the other six baselines (XGBoost, RF, LR, SVR, KNN, and RNN).In [29, 30], the authors showed that a wide range of descriptors is necessary as an input to the machine learning algorithms, such as decision forest and neural networks, for improved accuracy. These studies [29, 30] used input descriptors to describe the chemical molecular in 3D space (i.e., molecular descriptors). In this study, since the input is a text written in the English language, we used natural language processing as a stage that processes the text and produces such descriptors.From the previous studies, we notice that the paper citation prediction results are different based on the following:(i)
The dataset used (domain and size): very large datasets are more general but lead to low prediction accuracy compared to small and specialized datasets.(ii)
The change in the features used in the prediction will lead to different results, and feature selection plays an important role.(iii)
Many machine learning models could be used, and the performance of each model is different based on the dataset and used features.(iv)
The user who will benefit from the citation prediction applications could be the following:(i)
Paper author: who wants to improve his/her paper(ii)
Journal editor: who wants to accept the best papers(iii)
Researcher: who wants to select which papers to read.So developing a custom solution for each domain could provide the best benefit for the interested researchers. This process should include using a custom dataset, doing the right feature selection, and testing different machine learning models to use the best one that provides the highest level of accuracy.


## 3. The Dataset

Our dataset contains 500 research papers (500 rows)—we have information about each paper like the title, authors, abstract, and total citations.For the total citation (TC) column, the minimum value is 57 citations, while the maximum value is 579 citations. So we have a range of579‐57+1 citations, i.e., 523 citations.The dataset is available as a PDF file, and we saved the file as a text file (using the “Save As” feature from the Acrobat PDF reader); then, we converted the TXT file to a CSV file using a program written in the Ring programming language [31]. This Ring program is generated using the Programming Without Coding Technology (PWCT) software which is considered a general-purpose visual programming language [32–34]. PWCT is a popular visual programming language that is used in many applications and systems including the development of the Supernova language and the critical node application for the LASCNN algorithm [35, 36].


## 4. Algorithms and Machine Learning Models

### 4.1. Algorithms

This study uses the next algorithms for regression. We picked some of the popular machine learning algorithms in the literature [37–39].
(i)
Linear regression(ii)
Boosted decision tree regression(iii)
Decision forest regression(iv)
Neural network regressionThe next tools are used for development.(i)
Microsoft Azure machine learning: we selected this tool because it is a visual tool that supports many machine learning models and reduces the development time [40–42].(ii)
The Ring programming language: we selected this language because it is a simple and dynamic programming language like Python but comes with integrated GUI tools like Visual Basic(iii)
Programming Without Coding Technology (PWCT): we selected this tool because it is a visual programming language that reduces development timeSteps:(i)
Prepare and analyze the dataset(ii)
Preprocess the text(iii)
Split the data (training data and test data)(iv)
Extractn-gram features(v)
Select columns(vi)
Select the algorithm(vii)
Train the models(viii)
Score and evaluate (calculate the root mean squared error)(ix)
Compare the results between the different algorithms


### 4.2. Natural Language Processing

Preprocess text: in this stage, the text is processed before usage by our machine learning model.(i)
Remove stop words(ii)
Preform lemmatization(iii)
Detect sentences(iv)
Normalize case to lowercase(v)
Remove numbers(vi)
Remove special characters(vii)
Remove duplicate characters(viii)
Remove email addresses(ix)
Remove URLs(x)
Expand verb contraction(xi)
Split tokens on special charactersSplit data: 70% of our data is used for training while 30% is used for testing.


#### 4.2.1. Extractn-Gram Features

There are many weighting functions like binary weight, TF weight, IDF weight, TF-IDF weight, and graph weight. In this stage, we used the TF-IDF weighting function. The minimum word length is three (3) while the maximum word length is 25. The minimumn-gram document absolute frequency is five (5). The maximum n-gram document frequency ratio is 80%. There are many feature scoring methods like Pearson correlation, mutual information, Kendall correlation, Spearman correlation, chi-squared, fisher score, and count based. The feature scoring method used in our experiments is chi-squared.


## 4.1. Algorithms

This study uses the next algorithms for regression. We picked some of the popular machine learning algorithms in the literature [37–39].
(i)
Linear regression(ii)
Boosted decision tree regression(iii)
Decision forest regression(iv)
Neural network regressionThe next tools are used for development.(i)
Microsoft Azure machine learning: we selected this tool because it is a visual tool that supports many machine learning models and reduces the development time [40–42].(ii)
The Ring programming language: we selected this language because it is a simple and dynamic programming language like Python but comes with integrated GUI tools like Visual Basic(iii)
Programming Without Coding Technology (PWCT): we selected this tool because it is a visual programming language that reduces development timeSteps:(i)
Prepare and analyze the dataset(ii)
Preprocess the text(iii)
Split the data (training data and test data)(iv)
Extractn-gram features(v)
Select columns(vi)
Select the algorithm(vii)
Train the models(viii)
Score and evaluate (calculate the root mean squared error)(ix)
Compare the results between the different algorithms


## 4.2. Natural Language Processing

Preprocess text: in this stage, the text is processed before usage by our machine learning model.(i)
Remove stop words(ii)
Preform lemmatization(iii)
Detect sentences(iv)
Normalize case to lowercase(v)
Remove numbers(vi)
Remove special characters(vii)
Remove duplicate characters(viii)
Remove email addresses(ix)
Remove URLs(x)
Expand verb contraction(xi)
Split tokens on special charactersSplit data: 70% of our data is used for training while 30% is used for testing.


### 4.2.1. Extractn-Gram Features

There are many weighting functions like binary weight, TF weight, IDF weight, TF-IDF weight, and graph weight. In this stage, we used the TF-IDF weighting function. The minimum word length is three (3) while the maximum word length is 25. The minimumn-gram document absolute frequency is five (5). The maximum n-gram document frequency ratio is 80%. There are many feature scoring methods like Pearson correlation, mutual information, Kendall correlation, Spearman correlation, chi-squared, fisher score, and count based. The feature scoring method used in our experiments is chi-squared.


## 4.2.1. Extractn-Gram Features

There are many weighting functions like binary weight, TF weight, IDF weight, TF-IDF weight, and graph weight. In this stage, we used the TF-IDF weighting function. The minimum word length is three (3) while the maximum word length is 25. The minimumn-gram document absolute frequency is five (5). The maximum n-gram document frequency ratio is 80%. There are many feature scoring methods like Pearson correlation, mutual information, Kendall correlation, Spearman correlation, chi-squared, fisher score, and count based. The feature scoring method used in our experiments is chi-squared.


## 5. Experimental Results and Analysis

### 5.1. Prediction Using the Title

Concerning the maximumn-grams in the model parameters, we allowed 2000 n-grams. In practice, the model uses 165 columns including 164 n-grams. The other column is the total citations. Table 1 provides some of the n-grams used by the model and their weight.Table 1
Some of the titlen-grams with positive weight.
FeatureWeightPreprocessed TI.[ganglion]915.36Preprocessed TI.[speak_language]188.11Preprocessed TI.[chronic]180.89Preprocessed TI.[ear]175.04Preprocessed TI.[implantation]141.14Preprocessed TI.[acoustic_stimulation]138.86Preprocessed TI.[implication_cochlear]110.82Preprocessed TI.[perception_cochlear]94.83Preprocessed TI.[adult_use]86.15Preprocessed TI.[affect]78.60Preprocessed TI.[auditory_nerve]73.73Preprocessed TI.[development]72.84Preprocessed TI.[language_development]70.51Preprocessed TI.[use_cochlear]66.22Preprocessed TI.[deafness]55.75Preprocessed TI.[skill]53.03Preprocessed TI.[electrical]47.46Preprocessed TI.[depth]45.88Preprocessed TI.[speak]42.75Some of then-gram have positive weight, while other n-gram have negative weight as demonstrated in Table 2.Table 2
Some of the titlen-grams with negative weights.
FeatureWeightPreprocessed TI.[electrode_insertion]-90.47Preprocessed TI.[assessment]-90.87Preprocessed TI.[profound]-92.07Preprocessed TI.[stimulation_auditory]-93.67Preprocessed TI.[study]-94.31Preprocessed TI.[ganglion_neuron]-96.39Preprocessed TI.[implant_patient]-99.88Preprocessed TI.[nerve]-103.80Preprocessed TI.[congenital]-106.46Preprocessed TI.[early_cochlear]-161.79Preprocessed TI.[cochlear_implantation]-164.26Preprocessed TI.[child_use]-172.33Preprocessed TI.[implant_user]-172.34Preprocessed TI.[spiral]-453.88Preprocessed TI.[spiral_ganglion]-453.88In Figure1, the word art visualizes the n-gram features. From this figure, we notice that some words come with big weight (more importance) like ganglion, speak the language, and chronic. The figure uses the font size, location, and colors to demonstrate the importance of the word.Figure 1
Titlen-gram word art.Table3 provides the results when predicting the total citations using the title.Table 3
Using different models to predict the total citations using the paper title.
AlgorithmMean absolute errorRoot mean squared errorRelative absolute errorRelative squared errorCoefficient of determinationLinear regression58.7380.431.211.41-0.41Boosted decision tree regression48.6770.151.001.07-0.07Decision forest regression46.1269.450.951.05-0.05Neural network60.2387.511.241.67-0.67In this experiment, the decision forest regression provides the minimum root mean squared error (69.45); then, we have the boosted decision tree regression providing the root mean squared error (70.15) while the linear regression provides 80.43 as the root mean squared error, and finally, the neural network provides 87.51 as the root mean squared error. So, in this case, the best algorithm is the decision forest regression.The dataset contains 500 papers; out of these papers, we have 350 papers used for training and 150 papers used for testing (using the decision forest regression). The citation range is 523 citations.Table4 demonstrates the error percentage while predicting the citations for 150 papers during the testing stage.Table 4
Error in citation count.
ErrorPercentage of citation range (523 citations)Papers countPercentage of testing papers (150 papers)≤10 citations1.9%29 papers19.33%≤40 citations7.6%98 papers65.33%≤80 citations15.29%125 papers83.33%≤100 citations19.12%131 papers87.33%Table4 is a good indicator of the model’s accuracy. If we considered that the error in citation prediction should be less than 40 citations (7.6% of the citation range), then we have 65.33% of papers passing this condition. Considering that the error should be less than or equal to 100 citations (19.12% of citation range), then 87.33% of the papers in the testing stage pass this condition.Figure2 demonstrates the root mean squared error for different models using the title n-grams.Figure 2
RMSE for different models that use the titlen-grams.


### 5.2. Prediction Using the Abstract

Concerning the maximumn-grams in the model parameters, we allowed 2000 n-grams. In practice, the model uses 1715 columns including 1714 n-grams. The other column is the total citations.Table5 provides some of the n-grams used by the model and their weight.Table 5
Some of the abstractn-grams with positive weights.
FeatureWeightPreprocessed AB.[specimen]215.01Preprocessed AB.[chronic]201.60Preprocessed AB.[expression]188.20Preprocessed AB.[individual]180.01Preprocessed AB.[largely]150.27Preprocessed AB.[direct]137.41Preprocessed AB.[age]135.98Preprocessed AB.[refer]130.23Preprocessed AB.[language_development]128.61Preprocessed AB.[hear_aid]128.15Preprocessed AB.[detection]123.85Preprocessed AB.[place]119.44Preprocessed AB.[base]117.42Preprocessed AB.[point]116.75Preprocessed AB.[listen]115.41Preprocessed AB.[excellent]110.98Preprocessed AB.[widely]110.75Preprocessed AB.[English]110.69Preprocessed AB.[psychological]109.72Table6 demonstrates that some of the n-grams have positive weight, while other n-grams have negative weight.Table 6
Some of the abstractn-grams with negative weights.
FeatureWeightPreprocessed AB.[amplitude]-77.38Preprocessed AB.[regard]-77.63Preprocessed AB.[world]-77.78Preprocessed AB.[occur]-78.17Preprocessed AB.[normal]-81.00Preprocessed AB.[aid_condition]-82.22Preprocessed AB.[profound_deafness]-82.57Preprocessed AB.[child_use]-82.95Preprocessed AB.[potential_record]-85.43Preprocessed AB.[overall]-87.01Preprocessed AB.[child_implant]-90.88Preprocessed AB.[outcome]-93.81Preprocessed AB.[month_implantation]-95.26Preprocessed AB.[receptive]-95.41Preprocessed AB.[frequency_information]-96.82Preprocessed AB.[treat]-96.91Preprocessed AB.[distort]-102.61Preprocessed AB.[achieve]-107.08Preprocessed AB.[implant_year]-111.90Preprocessed AB.[post]-117.16Preprocessed AB.[old]-117.89Preprocessed AB.[site]-124.55In Figure3, the word art visualizes the n-gram features; from this figure, we notice that some words like specimen and chronic have higher weight and are more important.Figure 3
Abstractn-gram word art.Table7 presents the next results when predicting the total citations using the abstract.Table 7
Using different models to predict the total citations using the paper abstract.
AlgorithmMean absolute errorRoot mean squared errorRelative absolute errorRelative squared errorCoefficient of determinationLinear regression51.4968.561.251.30-0.30Boosted decision tree regression47.8766.001.161.21-0.21Decision forest regression42.7563.531.041.12-0.12Neural network40.4862.760.981.09-0.09In this experiment, the neural network provides the minimum root mean squared error (62.76); then, we have the decision forest regression providing the root mean squared error (63.53) while the boosted decision tree regression provides 66 as the root mean squared error, and finally, the linear regression provides 68.56 as the root mean squared error. So, in this case, the best algorithm is the neural network.The dataset contains 500 papers; from these papers, we have 18 papers that come without abstracts. We have 337 papers used for training and 145 papers used for testing (using the neural network regression). The citation range is 523 citations.Table8 demonstrates the error percentage while predicting the citations for 145 papers during the testing stage.Table 8
Error in citation count.
ErrorPercentage of citation range (523 citations)Paper countPercentage of testing papers (145 papers)≤10 citations1.9%46 papers31.72%≤40 citations7.6%93 papers64.13%≤80 citations15.29%127 papers87.58%≤100 citations19.12%135 papers93.1%Table8 is a good indicator of the model’s accuracy. If we considered that the error in citation prediction should be less than 40 citations (7.6% of the citation range), then we have 64.13% of papers passing this condition. Considering that the error should be less than or equal to 100 citations (19.12% of citation range), then 93.1% of the papers in the testing stage pass this condition.Figure4 demonstrates the root mean squared error for different models using the abstract n-grams.Figure 4
RMSE for different models that use the abstractn-grams.


### 5.3. Prediction Using the Authors

For the maximumn-grams in the model parameters, we allowed 2000 n-grams. In practice, the model uses 95 columns including 94 n-grams. The other column is the total citations.Some of then-grams have positive weight, while other n-gram has negative weight.Table9 provides the results when predicting the total citations using the authors.Table 9
Using different models to predict the total citations using the paper authors.
AlgorithmMean absolute errorRoot mean squared errorRelative absolute errorRelative squared errorCoefficient of determinationLinear regression50.1269.581.0491.12-0.12Boosted decision tree regression45.3465.790.9491.00-0.00Decision forest regression46.9467.360.981.05-0.05Neural network49.8470.191.041.14-0.14In this experiment, the boosted decision tree regression provides the minimum root mean squared error (65.79); then, we have the decision forest regression providing the root mean squared error (67.36) while the linear regression provides 69.58 as the root mean squared error, and finally, the neural network provides 70.19 as the root mean squared error. So, in this case, the best algorithm is the boosted decision tree regression.The dataset contains 500 papers; one paper comes without the authors. We have 349 papers used for training and 150 papers used for testing (using the boosted decision tree regression). The citation range is 523 citations. Table10 demonstrates the error percentage while predicting the citations for 150 papers during the testing stage.Table 10
Error in citation count.
ErrorPercentage of citation range (523 citations)Paper countPercentage of testing papers (150 papers)≤10 citations1.9%23 papers15.33%≤40 citations7.6%90 papers60%≤80 citations15.29%130 papers86.66%≤100 citations19.12%136 papers90.66%Table10 is a good indicator of the model’s accuracy. If we considered that the error in citation prediction should be less than 40 citations (7.6% of the citations range), then we have 60% of papers passing this condition. Considering that the error should be less than or equal to 100 citations (19.12% of citations range), then 90.66% of the papers in the testing stage pass this condition.Figure5 demonstrates the root mean squared error for different models using the author n-grams.Figure 5
RMSE for different models that use the authorn-grams.


### 5.4. Web Services

We published a web service for each trained model (boosted decision tree, decision forest, and neural networks).


## 5.1. Prediction Using the Title

Concerning the maximumn-grams in the model parameters, we allowed 2000 n-grams. In practice, the model uses 165 columns including 164 n-grams. The other column is the total citations. Table 1 provides some of the n-grams used by the model and their weight.Table 1
Some of the titlen-grams with positive weight.
FeatureWeightPreprocessed TI.[ganglion]915.36Preprocessed TI.[speak_language]188.11Preprocessed TI.[chronic]180.89Preprocessed TI.[ear]175.04Preprocessed TI.[implantation]141.14Preprocessed TI.[acoustic_stimulation]138.86Preprocessed TI.[implication_cochlear]110.82Preprocessed TI.[perception_cochlear]94.83Preprocessed TI.[adult_use]86.15Preprocessed TI.[affect]78.60Preprocessed TI.[auditory_nerve]73.73Preprocessed TI.[development]72.84Preprocessed TI.[language_development]70.51Preprocessed TI.[use_cochlear]66.22Preprocessed TI.[deafness]55.75Preprocessed TI.[skill]53.03Preprocessed TI.[electrical]47.46Preprocessed TI.[depth]45.88Preprocessed TI.[speak]42.75Some of then-gram have positive weight, while other n-gram have negative weight as demonstrated in Table 2.Table 2
Some of the titlen-grams with negative weights.
FeatureWeightPreprocessed TI.[electrode_insertion]-90.47Preprocessed TI.[assessment]-90.87Preprocessed TI.[profound]-92.07Preprocessed TI.[stimulation_auditory]-93.67Preprocessed TI.[study]-94.31Preprocessed TI.[ganglion_neuron]-96.39Preprocessed TI.[implant_patient]-99.88Preprocessed TI.[nerve]-103.80Preprocessed TI.[congenital]-106.46Preprocessed TI.[early_cochlear]-161.79Preprocessed TI.[cochlear_implantation]-164.26Preprocessed TI.[child_use]-172.33Preprocessed TI.[implant_user]-172.34Preprocessed TI.[spiral]-453.88Preprocessed TI.[spiral_ganglion]-453.88In Figure1, the word art visualizes the n-gram features. From this figure, we notice that some words come with big weight (more importance) like ganglion, speak the language, and chronic. The figure uses the font size, location, and colors to demonstrate the importance of the word.Figure 1
Titlen-gram word art.Table3 provides the results when predicting the total citations using the title.Table 3
Using different models to predict the total citations using the paper title.
AlgorithmMean absolute errorRoot mean squared errorRelative absolute errorRelative squared errorCoefficient of determinationLinear regression58.7380.431.211.41-0.41Boosted decision tree regression48.6770.151.001.07-0.07Decision forest regression46.1269.450.951.05-0.05Neural network60.2387.511.241.67-0.67In this experiment, the decision forest regression provides the minimum root mean squared error (69.45); then, we have the boosted decision tree regression providing the root mean squared error (70.15) while the linear regression provides 80.43 as the root mean squared error, and finally, the neural network provides 87.51 as the root mean squared error. So, in this case, the best algorithm is the decision forest regression.The dataset contains 500 papers; out of these papers, we have 350 papers used for training and 150 papers used for testing (using the decision forest regression). The citation range is 523 citations.Table4 demonstrates the error percentage while predicting the citations for 150 papers during the testing stage.Table 4
Error in citation count.
ErrorPercentage of citation range (523 citations)Papers countPercentage of testing papers (150 papers)≤10 citations1.9%29 papers19.33%≤40 citations7.6%98 papers65.33%≤80 citations15.29%125 papers83.33%≤100 citations19.12%131 papers87.33%Table4 is a good indicator of the model’s accuracy. If we considered that the error in citation prediction should be less than 40 citations (7.6% of the citation range), then we have 65.33% of papers passing this condition. Considering that the error should be less than or equal to 100 citations (19.12% of citation range), then 87.33% of the papers in the testing stage pass this condition.Figure2 demonstrates the root mean squared error for different models using the title n-grams.Figure 2
RMSE for different models that use the titlen-grams.


## 5.2. Prediction Using the Abstract

Concerning the maximumn-grams in the model parameters, we allowed 2000 n-grams. In practice, the model uses 1715 columns including 1714 n-grams. The other column is the total citations.Table5 provides some of the n-grams used by the model and their weight.Table 5
Some of the abstractn-grams with positive weights.
FeatureWeightPreprocessed AB.[specimen]215.01Preprocessed AB.[chronic]201.60Preprocessed AB.[expression]188.20Preprocessed AB.[individual]180.01Preprocessed AB.[largely]150.27Preprocessed AB.[direct]137.41Preprocessed AB.[age]135.98Preprocessed AB.[refer]130.23Preprocessed AB.[language_development]128.61Preprocessed AB.[hear_aid]128.15Preprocessed AB.[detection]123.85Preprocessed AB.[place]119.44Preprocessed AB.[base]117.42Preprocessed AB.[point]116.75Preprocessed AB.[listen]115.41Preprocessed AB.[excellent]110.98Preprocessed AB.[widely]110.75Preprocessed AB.[English]110.69Preprocessed AB.[psychological]109.72Table6 demonstrates that some of the n-grams have positive weight, while other n-grams have negative weight.Table 6
Some of the abstractn-grams with negative weights.
FeatureWeightPreprocessed AB.[amplitude]-77.38Preprocessed AB.[regard]-77.63Preprocessed AB.[world]-77.78Preprocessed AB.[occur]-78.17Preprocessed AB.[normal]-81.00Preprocessed AB.[aid_condition]-82.22Preprocessed AB.[profound_deafness]-82.57Preprocessed AB.[child_use]-82.95Preprocessed AB.[potential_record]-85.43Preprocessed AB.[overall]-87.01Preprocessed AB.[child_implant]-90.88Preprocessed AB.[outcome]-93.81Preprocessed AB.[month_implantation]-95.26Preprocessed AB.[receptive]-95.41Preprocessed AB.[frequency_information]-96.82Preprocessed AB.[treat]-96.91Preprocessed AB.[distort]-102.61Preprocessed AB.[achieve]-107.08Preprocessed AB.[implant_year]-111.90Preprocessed AB.[post]-117.16Preprocessed AB.[old]-117.89Preprocessed AB.[site]-124.55In Figure3, the word art visualizes the n-gram features; from this figure, we notice that some words like specimen and chronic have higher weight and are more important.Figure 3
Abstractn-gram word art.Table7 presents the next results when predicting the total citations using the abstract.Table 7
Using different models to predict the total citations using the paper abstract.
AlgorithmMean absolute errorRoot mean squared errorRelative absolute errorRelative squared errorCoefficient of determinationLinear regression51.4968.561.251.30-0.30Boosted decision tree regression47.8766.001.161.21-0.21Decision forest regression42.7563.531.041.12-0.12Neural network40.4862.760.981.09-0.09In this experiment, the neural network provides the minimum root mean squared error (62.76); then, we have the decision forest regression providing the root mean squared error (63.53) while the boosted decision tree regression provides 66 as the root mean squared error, and finally, the linear regression provides 68.56 as the root mean squared error. So, in this case, the best algorithm is the neural network.The dataset contains 500 papers; from these papers, we have 18 papers that come without abstracts. We have 337 papers used for training and 145 papers used for testing (using the neural network regression). The citation range is 523 citations.Table8 demonstrates the error percentage while predicting the citations for 145 papers during the testing stage.Table 8
Error in citation count.
ErrorPercentage of citation range (523 citations)Paper countPercentage of testing papers (145 papers)≤10 citations1.9%46 papers31.72%≤40 citations7.6%93 papers64.13%≤80 citations15.29%127 papers87.58%≤100 citations19.12%135 papers93.1%Table8 is a good indicator of the model’s accuracy. If we considered that the error in citation prediction should be less than 40 citations (7.6% of the citation range), then we have 64.13% of papers passing this condition. Considering that the error should be less than or equal to 100 citations (19.12% of citation range), then 93.1% of the papers in the testing stage pass this condition.Figure4 demonstrates the root mean squared error for different models using the abstract n-grams.Figure 4
RMSE for different models that use the abstractn-grams.


## 5.3. Prediction Using the Authors

For the maximumn-grams in the model parameters, we allowed 2000 n-grams. In practice, the model uses 95 columns including 94 n-grams. The other column is the total citations.Some of then-grams have positive weight, while other n-gram has negative weight.Table9 provides the results when predicting the total citations using the authors.Table 9
Using different models to predict the total citations using the paper authors.
AlgorithmMean absolute errorRoot mean squared errorRelative absolute errorRelative squared errorCoefficient of determinationLinear regression50.1269.581.0491.12-0.12Boosted decision tree regression45.3465.790.9491.00-0.00Decision forest regression46.9467.360.981.05-0.05Neural network49.8470.191.041.14-0.14In this experiment, the boosted decision tree regression provides the minimum root mean squared error (65.79); then, we have the decision forest regression providing the root mean squared error (67.36) while the linear regression provides 69.58 as the root mean squared error, and finally, the neural network provides 70.19 as the root mean squared error. So, in this case, the best algorithm is the boosted decision tree regression.The dataset contains 500 papers; one paper comes without the authors. We have 349 papers used for training and 150 papers used for testing (using the boosted decision tree regression). The citation range is 523 citations. Table10 demonstrates the error percentage while predicting the citations for 150 papers during the testing stage.Table 10
Error in citation count.
ErrorPercentage of citation range (523 citations)Paper countPercentage of testing papers (150 papers)≤10 citations1.9%23 papers15.33%≤40 citations7.6%90 papers60%≤80 citations15.29%130 papers86.66%≤100 citations19.12%136 papers90.66%Table10 is a good indicator of the model’s accuracy. If we considered that the error in citation prediction should be less than 40 citations (7.6% of the citations range), then we have 60% of papers passing this condition. Considering that the error should be less than or equal to 100 citations (19.12% of citations range), then 90.66% of the papers in the testing stage pass this condition.Figure5 demonstrates the root mean squared error for different models using the author n-grams.Figure 5
RMSE for different models that use the authorn-grams.


## 5.4. Web Services

We published a web service for each trained model (boosted decision tree, decision forest, and neural networks).


## 6. Citation Prediction Application

We developed an application that can accept the title, authors, and abstract to predict the total citations (demonstrated in Figure6). The application is developed using the Ring programming language where the source code is generated using the Programming Without Coding Technology (PWCT) software. The main window in our application provides a data entry form that we can use to enter the paper details. We need at least to determine the title, author, or abstract. Then, we click the “Predict” button to get the prediction results. Using the “Select” button, we get another window that contains our dataset rows, where we can quickly select any of these rows and use them for testing our application.Figure 6
Citation prediction application: main window.Figure7 presents the dataset rows; each row in our dataset contains the three features (title, authors, and abstract) and one label (total citations). The title, authors, and abstract are textual data while the total citations are numeric data.Figure 7
Citation prediction application: dataset window.We can select a row and then click on the “Select” button to insert the row data in our main window as demonstrated in Figure8.Figure 8
Inserting data from the dataset window to the main window.


## 7. Discussion, Future Work, and Conclusion

### 7.1. Discussion and Future Work

The results of these research and case studies demonstrate that we can use different machine learning algorithms to build models that predict the paper citations using different features. We detected the best algorithm for the different features like the title, authors, and abstract. The difference in RMSE between the algorithms when using the same feature is not so big, but the difference in RMSE when using the different features could be notable. The best result could be achieved when using the paper abstract in the prediction.Table11 provides each feature, the best algorithm, and the root mean squared error achieved in our experiments while predicting the total citations.Table 11
The best algorithms and the corresponding RMSE.
FeatureBest algorithmRoot mean squared errorTitleDecision forest regression69.45AbstractNeural network62.76AuthorsBoosted decision tree regression65.79Prediction using the abstract and the neural network provides the minimum root mean squared error (62.76) as demonstrated in Figure9.Figure 9
Graph demonstrates the RMSE achieved by each algorithm.Table12 provides the used feature and the number of n-grams. The graph in Figure 10 presents these results; we notice that when using the abstract feature in the prediction, we have a huge number of n-grams.Table 12
Number ofn-grams for each feature.
The feature used in predictionn-gramsTitle164Abstract1714Authors94Figure 10
Graph demonstrated the number ofn-grams for each feature.The predicted paper citation is just an indicator that can be used by the reviewers on the journal side to pick the paper that could be more attractive to the readers. Also, it can be used by the authors of the research to improve the paper’s impact by rewriting the paper title and abstract until getting the higher possible prediction of the paper citations.In the future, we will extend our experiments; for example, we will try more neural networks with different scripts that set the layer count, nodes in each layer, and different activation functions. Also, we plan to try different weight functions in the text processing stage. We plan also to use ensemble learning and use many different models together in the prediction process to get higher accuracy. An improvement that we plan to do too is developing a tool that provides a simple GUI to analyze the prediction output and provide suggestions about which words to keep and which words to change. We plan also to replace our desktop front end application with web-based solution to quickly deliver new updates and a mobile application to have more accessible software.


### 7.2. Conclusion

The use of models that can predict which citations an article will receive after publication can be a useful tool in the publisher’s evaluation process. Also, it can help the research authors to improve the paper content to get more citations.In this paper, we presented a machine learning model to predict the total number of citations of the research papers using different algorithms like boosted decision tree, decision forest, and neural networks. We did many experiments to evaluate the performance of each model and determine which one provides the best results. Our results demonstrate that using neural networks and the paper abstract provides the minimum root mean squared error compared to using other algorithms like the boosted decision tree or the decision forest. We developed the model using the Microsoft Azure machine learning tool and also developed an application using the Programming Without Coding Technology that displays the dataset and predicts the paper citations using different algorithms.The quality of the research papers could be improved through the adoption of machine learning models by more researchers Also, these models could become more suitable in the future when different machine learning methods and specific datasets could be used for each scientific domain.


## 7.1. Discussion and Future Work

The results of these research and case studies demonstrate that we can use different machine learning algorithms to build models that predict the paper citations using different features. We detected the best algorithm for the different features like the title, authors, and abstract. The difference in RMSE between the algorithms when using the same feature is not so big, but the difference in RMSE when using the different features could be notable. The best result could be achieved when using the paper abstract in the prediction.Table11 provides each feature, the best algorithm, and the root mean squared error achieved in our experiments while predicting the total citations.Table 11
The best algorithms and the corresponding RMSE.
FeatureBest algorithmRoot mean squared errorTitleDecision forest regression69.45AbstractNeural network62.76AuthorsBoosted decision tree regression65.79Prediction using the abstract and the neural network provides the minimum root mean squared error (62.76) as demonstrated in Figure9.Figure 9
Graph demonstrates the RMSE achieved by each algorithm.Table12 provides the used feature and the number of n-grams. The graph in Figure 10 presents these results; we notice that when using the abstract feature in the prediction, we have a huge number of n-grams.Table 12
Number ofn-grams for each feature.
The feature used in predictionn-gramsTitle164Abstract1714Authors94Figure 10
Graph demonstrated the number ofn-grams for each feature.The predicted paper citation is just an indicator that can be used by the reviewers on the journal side to pick the paper that could be more attractive to the readers. Also, it can be used by the authors of the research to improve the paper’s impact by rewriting the paper title and abstract until getting the higher possible prediction of the paper citations.In the future, we will extend our experiments; for example, we will try more neural networks with different scripts that set the layer count, nodes in each layer, and different activation functions. Also, we plan to try different weight functions in the text processing stage. We plan also to use ensemble learning and use many different models together in the prediction process to get higher accuracy. An improvement that we plan to do too is developing a tool that provides a simple GUI to analyze the prediction output and provide suggestions about which words to keep and which words to change. We plan also to replace our desktop front end application with web-based solution to quickly deliver new updates and a mobile application to have more accessible software.


## 7.2. Conclusion

The use of models that can predict which citations an article will receive after publication can be a useful tool in the publisher’s evaluation process. Also, it can help the research authors to improve the paper content to get more citations.In this paper, we presented a machine learning model to predict the total number of citations of the research papers using different algorithms like boosted decision tree, decision forest, and neural networks. We did many experiments to evaluate the performance of each model and determine which one provides the best results. Our results demonstrate that using neural networks and the paper abstract provides the minimum root mean squared error compared to using other algorithms like the boosted decision tree or the decision forest. We developed the model using the Microsoft Azure machine learning tool and also developed an application using the Programming Without Coding Technology that displays the dataset and predicts the paper citations using different algorithms.The quality of the research papers could be improved through the adoption of machine learning models by more researchers Also, these models could become more suitable in the future when different machine learning methods and specific datasets could be used for each scientific domain.



---

*Source: 2239152-2022-07-20.xml*