# Natural Language Processing for the Turkish Academic Texts in the Engineering Field: Key-Term Extraction, Similarity Detection, Subject/Topic Assignment

Bora Kat

► To cite this version:

Bora Kat. Natural Language Processing for the Turkish Academic Texts in the Engineering Field: Key-Term Extraction, Similarity Detection, Subject/Topic Assignment. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.411-424, 10.1007/978-3-031-34107-6_33. hal-05498066

HAL Id: hal-05498066

https://inria.hal.science/hal-05498066v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Natural Language Processing for the Turkish Academic Texts in the Engineering Field: Key-Term Extraction, Similarity Detection, Subject/Topic Assignment

Bora Kat¹,²(✉) ID

¹ Middle East Technical University, Ankara, Turkey
borakat@gmail.com

² The Scientific and Technological Research Council of Turkey (TÜBİTAK), Ankara, Turkey

**Abstract.** The information retrieved from texts plays crucial roles in many aspects. Although there are significant attempts on natural language processing for various types of texts in Turkish, none of them deals with academic texts. This study mainly aims to retrieve precise key terms from Turkish academic texts in the field of engineering and develops algorithms for similarity detection and automatic classification based on these key terms. In the first step of this study: a library and customized templates, that can transform the n-grams into structured forms, are created by considering the features of engineering terminology and the grammar of the Turkish language. Then, a customized similarity detection algorithm is developed. Finally, the Naïve Bayes Classifier is used to assign the documents to the appropriate engineering sub-fields. The project proposals submitted to The Scientific and Technological Research Council of Turkey (TÜBİTAK) Academic Research Funding Program Directorate (ARDEB) are analyzed as a case study. The results indicate that the proposed similarity algorithm correctly detects almost all of the re-submitted proposals while the accuracy of the classifier is 83.3% in the first prediction and reaches up to 96.4% in the first three predictions over a sample of 1255 proposals.

**Keywords:** Key term extraction · Feature extraction · Natural language processing (NLP) · Supervised machine learning · Naïve Bayes classifier · TÜBİTAK · Conceptual similarity · subject/topic assignment

## 1 Introduction

The era of rapid digitalization and the concomitant expansion in the size of the data make it essential to develop mechanisms that will support the management of big data and the retrieval of valuable information. At this point, natural language processing (NLP) studies provide support to many processes in different fields and for different languages. Academic texts also take its share from this transformation, e.g., feature extraction from
unstructured texts, similarity detection between the documents and automatic classification or clustering of the documents. Especially machine learning-based text classification studies [1] have gained momentum in recent years, e.g., sentiment analysis, indexing news texts, grouping social media posts, classification of legal decisions, etc. In line with the increasing amount of funds allocated to R&D, the number of project proposals and the data related to these applications and outputs of funded research have increased significantly. As a result, the need for effective monitoring of data on proposals and their outputs has emerged. To satisfy this need; TÜBİTAK has developed several business software for different phases of funding, i.e., application, evaluation, and monitoring. On the other hand, the need for decision support systems (DSS) that will help the processes for these applications to be carried out in a more effective, efficient and qualified manner is increasing day by day. An example of such a DSS, PaneLIST, in which the problem of identifying the most appropriate reviewers for a set of proposals, was demonstrated in [2].

Considering the Turkish language, there is an important gap in the classification of academic texts according to their subjects. Current study aims to fill this gap with algorithms developed based on a thorough analysis carried out on the project proposals submitted to TÜBİTAK ARDEB. In the current study, as a complement to PaneLIST, new and customized algorithms have been developed to provide solutions to the following three essential issues:

I) Automatic key term extraction from the proposals

II) Similarity level determination between the proposals

III) Automatic subject/topic assignment to the proposals

The study focuses on the project proposals submitted to ARDEB, however, the proposed algorithms can be used (without an extensive revision) for all Turkish academic texts in closer scientific domains.

The rest of the paper is organized as follows. First, the literature review is presented. Then, the methodology is explained in detail with dedicated sub-sections for each step and each problem. Next come the details related to the case study of ARDEB project proposals. Finally, the paper ends with the conclusion and recommendations for future research.

## 2 Literature Review

The performance of the first step, (I) key term extraction, is very crucial since its outputs are also inputs to the algorithms those are developed in (II) and (III), i.e., similarity level determination and subject/topic assignment. The most critical issue in key-term extraction is the preprocessing procedures that will help transforming the words and phrases in the proposals into structured forms. At this phase, in addition to the to “noise” cleaning (removing stop words such as conjunctions, prepositions and pronouns, along with punctuation marks), “stemming” to obtain the basic roots of words, and “lemmatization” to make the words simple by considering their morphology, are used. For the Turkish language, there are studies [3, 4] indicating that stemming has a limited effect on the subject classification while root analysis algorithms have a positive effect on information
extraction [5]. In the current study, besides the standard preprocessing steps, the main distinguishing contribution is the identification of frequently used terms within the scope of the engineering field and the construction of templates that will transform these terms into common key-term structures.

Investigating the similarity of two texts or vectorial structures obtained from these texts, on the other hand, is another well-known problem in the literature and practice. The topic is studied in two general frameworks: algorithm/software development for plagiarism detection [6–8] and studies for detecting conceptual similarity [9]. Moreover, different metrics have been defined to determine the level of similarity [6–9]. Within the scope of the current study, a customized similarity algorithm has been developed in which both textual and conceptual similarity can be detected. Analyzes have shown that the proposed algorithm provides almost 100% accuracy in detecting re-submissions.

Classification of a text according to its subject (III) emerges as a research area that has been frequently studied in recent years, especially employing machine learning-based algorithms [1, 10–15]. It can be in the form of assignment to a pre-defined set of classes, or it can be in the form of dividing the texts into a certain number of groups without any pre-defined class set. For the latter (also in the current study) generally the supervised machine learning algorithms are used. Classification for the Turkish texts have gained importance in recent years [16–23]. Although, these studies include movie [20–24], hotel [20] and restaurant [25] reviews, website content [23], tweet contents [20–24, 26], e-mail contents [27] and fairy tales [22] etc.; the data sets in the classification studies in Turkish is mostly the news data sets [3, 5, 13, 14, 18, 21, 27–29]. On the other hand, while there are extensive studies on the classification of academic texts in English [11, 30–32], no study has been found for the Turkish academic texts. In the present study, the Naïve Bayes [33, 34] algorithm is used as the classifier. This algorithm is frequently used in the literature for similar purposes and generates successful results. Moreover, it has been extensively examined in many aspects (preprocessing, training set size etc.) and significant inferences have been obtained [35–37].

## 3 Methodology

This section explains all of the steps followed in the study. First, preprocessing procedure is summarized. Next, the algorithm to retrieve keywords and key-terms is illustrated. Then, the postprocessing procedure is explained. Last two sub-sections are devoted to the similarity detection and subfield classification, respectively. Note that the parameters and sets used in the proposed algorithms are summarized below in Table 1.
<table><caption>**Table 1.** Sets and parameters.</caption><thead><tr><th>Symbol</th><th>Definition</th><th>Symbol</th><th>Definition</th></tr></thead><tbody><tr><td>D, d</td><td>Set of proposals &amp; elements of the set</td><td>δ</td><td>Scaling parameter 2 for n-grams</td></tr><tr><td>D<sub>e</sub></td><td>Training set</td><td>θ</td><td>Similarity score threshold 1</td></tr><tr><td>S, s</td><td>Set of classes and each element of the set</td><td>τ</td><td>Similarity score threshold 2</td></tr><tr><td>K<sup>d</sup>, k<sup>d</sup></td><td>Set of key-terms and elements of the set</td><td>σs</td><td>Penalty parameter for lower overlaps</td></tr><tr><td>GRAM<sup>d</sup><sub>1</sub></td><td>Average of frequencies of top five one-grams in proposal d</td><td>π, C, μ</td><td>Scaling and correction parameters for the similarity score</td></tr><tr><td>GRAM<sup>d</sup><sub>n</sub></td><td>Set of n-grams in proposal d; n = 1,2,3,4</td><td>sim(x, y)</td><td>Similarity score between x and y - [0,1]</td></tr><tr><td>xGRAM<sup>d</sup><sub>n</sub></td><td>Highest frequency value of n-gram in d</td><td>s<sub>ML</sub></td><td>Class with the highest likelihood</td></tr><tr><td>tf<sup>d</sup>(k)</td><td>Term frequency of key-term k in d</td><td>m(s)</td><td>Number of proposals in class s</td></tr><tr><td>atf<sup>d</sup>(k)</td><td>Term frequency of key-term k in d – adj</td><td>α</td><td>Correction parameter</td></tr><tr><td>γ<sub>n</sub></td><td>Scaling parameter 1 for n-grams</td><td>V</td><td>Set of all key-terms in the training set</td></tr></tbody></table>

## 3.1 Preprocessing

Some of the preprocessing stages are carried out with the standard methods, and in some of them specific arrangements are made considering the scope of the documents and the features of the Turkish language, i.e., academic engineering terminology, common words/phrases specific to TÜBİTAK application forms and distinguishing structure of key-terms in Turkish. The preprocessing steps performed are presented below:

- **Punctuation marks removal**

- **Lower casing**

- **Stop-words removal**: In addition to the standard stop-words in Turkish, some common words and terms, e.g., “prof” (“prof”), “profesör” (“professor”), “araştırmacı” (“researcher”), “proje ekibi” (“project team”), “öğrenci” (“student”), “lisansüstü” (“graduate”), “TÜBİTAK”, “bursiyer” (“scholar”), etc., used in engineering proposals are appended to the stop-words library.

- **Tokenization**: Vectors of uni-grams, bi-grams, 3-g and 4-g are extracted.

- **Tailored stemming**: The templates that convert n-grams into standard forms considering the fact that the single-word key-terms are in nominative form and the multi-word key-terms are generally in the form of inflexional suffixes in Turkish.
## 3.2 Keyword and Key-Term Extraction - KeyEx

In order to extract key-terms after the preprocessing stage, the texts must first be separated into its tokens. As a result of this tokenization process, sequential arrays consisting of n (n = 1, 2, 3, ...) elements called grams are obtained. In the present study, after creating single (uni-gram), binary (bi-gram), triple (tri-gram) and quadruple (4-g) sequences, a term-frequency vector is created. In addition, if user-defined keywords exists in the proposal, the developed code can distinguish these terms and assign the highest frequency coming from other n-grams. The terms in the final vector are then converted into a word cloud image for further analysis by an expert's eye (see Fig. 2). At this stage, a scaling was carried out on the frequencies in order to obtain a more understandable structure in the cloud image and to get more meaningful results in the similarity algorithm to be run later. The scaling function (for uni-grams and n-grams, respectively) is as follows:

$$ \text{atf}^d(k^d|k^d \in \text{GRAM}_1^d) = \left[ \frac{tf^d(k^d)}{\overline{\overline{\text{GRAM}_1^d}}} \right]^{\gamma_1} ; \forall d \in D \quad (1) $$

$$ \text{atf}^d(k^d|k^d \in \text{GRAM}_n^d) = \left[ \frac{tf^d(k^d)}{\max\{x\text{GRAM}_n^d, \delta \cdot |\text{GRAM}_1^d|\}} \right]^{\gamma_n} ; \forall d \in D, n = 2, 3, 4 \quad (2) $$

The primary purpose in these equations is to normalize the frequency values over 0–1 interval. In the first equation, the frequency values of individual grams are divided by the mean of the top five values. In the second equation, the frequency of multi-grams is divided by the highest frequency in the relevant gram set or a certain ratio of the total number of words in the text (to represent the impact of the proposal length). After this calculation, all values greater than 1 are set to 1. Another purpose is to eliminate the incompatibility caused by the frequency difference of single and multiple strings. Single grams can have high frequency by combining with different words and word groups in expressions written for different purposes in a text. The fact that their frequency is high may not be sufficient to conclude that those words are descriptive or distinctive for the relevant text. However, the repetition of multiple sequences in the text strengthens the possibility that these sequences are certain concepts describing the project.

## 3.3 Postprocessing

The operations carried out in the preprocessing stage are aimed at bringing the text to a standard structure as much as possible. In the final processing stage, updates are made not only in the text, but also in the term-frequency vector obtained from the text. The purpose of using this postprocessing step, which is not often encountered in NLP studies, is to make the retrieved concepts capable of indexing texts as much as possible without the need for an additional expert evaluation process. Then, as a result of detailed
preliminary analysis, the concepts in the following three groups were removed from the term-frequency vector:

- Uni-grams that do not make sense on their own. These words may either represent very general meanings, e.g., “amaç” (“purpose”), “ortak” (“common”), “deney” (“experiment”), etc., or they are often part of multiple sequences, e.g., “kendiliğinden” (“spontaneous”), “etkin” (“efficient”), etc.

- Multi-grams with a first word that does not allow a grammatically correct phrase. They usually have a possessive suffix and are located in the later part of the concept in multi-grams, e.g., “dayalı” (“based”), “yapısı” (“structure of ...”), etc.

- Multi-grams with a last word that does not allow a grammatically correct phrase. These last words are generally adjective or adjective in noun and are expected to take place in different positions in other grammatically correct sequences, e.g., “yeni” (“new”), “bölgesel” (“regional”), etc.

## 3.4 Similarity Detection - SimDet

Within the scope of the study, a tailored similarity algorithm was developed, in which both conceptual and textual similarity are considered together, taking into account the common definitions of similarity, i.e., cosine similarity and jaccard similarity. Let $atf^x$ and $atf^y$ represent adjusted term-frequency vectors created for document $x$ ($x \in D$) and $y$ ($y \in D$). The similarity calculated over these vectors are shown in eqns. (3)-(9) presented below. First, an intersection set of terms with $atf$ values above a certain threshold level is created in key-term sets, see Eq. (3). In Eq. (4)-(6); the number of terms scoring above the threshold levels and the number of terms in the intersection set are calculated. In Eq. (7), a similarity factor is calculated which takes into account how much of the terms in each document are covered by the intersection set.

$$INT(x, y) = \text{set}(x|atf^x(k^x) > \theta) \cap \text{set}(y|atf^y(k^y) > \theta) \quad (3)$$

$$len(x) = |\text{set}(x|atf^x(k^x) > \theta)| \quad (4)$$

$$len(y) = |\text{set}(y|atf^y(k^y) > \theta)| \quad (5)$$

$$lenINT(x, y) = |INT(x, y)| \quad (6)$$

$$\pi = \left[ C \cdot \frac{lenINT(x, y)}{len(x)} \cdot \frac{lenINT(x, y)}{len(y)} \right]^\mu \quad (7)$$

To create a score for similarity detection; first, the geometric mean of the $atf$ values in each document was taken for the terms in the intersection set, see Eq. (8). Then, in Eq. (9), the arithmetic mean of the geometric means is found. However, as can be seen in the related equation; when calculating the arithmetic mean, the sum of the geometric mean is divided by a fixed number if the intersection set is below a certain value. This adaptation, based on observations made as a result of preliminary studies, prevents misleading results from very few but high-scoring matches. Similarly, previously calculated $\pi$ factor is used
to reflect the impact of how much the terms in the texts overlap (considering proposal lengths). In addition, the power term added for the calculated arithmetic mean was designed to increase the effect of relatively high frequency overlaps and especially to facilitate the identification of re-submissions. The parameters used in the algorithm were determined as a result of a comprehensive preliminary study and are presented in Table 2.

$$ \mathrm{atf}^z(k^z) = \sqrt{\mathrm{atf}^x(k^z) \cdot \mathrm{atf}^y(k^z)}; \forall k^z \in \mathrm{INT}(x, y) \qquad (8) $$

$$ \mathrm{sim}(x, y) = \pi \cdot \left[ \frac{\sum_{\forall k^z \in \mathrm{set}U(x,y)} \mathrm{atf}^z(k^z)}{\max\{\sigma, \mathrm{lenINT}(x, y)\}} \right]^{\frac{2}{(1+|\mathrm{INT}(x,y)|\mathrm{atf}^z(k^z)>\tau|)}} \qquad (9) $$

<table><caption>**Table 2.** Values of parameters in SimDet.</caption><tr><td>θ = 0, 60</td><td>σ = 50</td><td>C = 100</td><td>τ = 0, 65</td><td>μ = 0, 50</td></tr></table>

## 3.5 Subfield Classification by Machine Learning – SubCla

Naïve Bayes (NB) classifier is employed for automatic assignment of the proposals into the current classification scheme used in ARDEB (see Table 3.). NB classifier, as a supervised machine learning classifier, calculates the probability that a text whose class is to be predicted belongs to each class, by using the probabilities determined over the texts in the training set. In order to use the NB classifier, first of all, a training set, $D_{training}$, is needed. Equation (10) shows the probability of document $d$ belonging to class $s$ according to Bayes theory. It shows the product of the prior probability and the conditional probability divided by the evidence value. As a result, the relationship given in the intermediate stage, Eq. (11) turns into Eq. (12). This equation identifies the class that maximizes the probability of the class that document $d$ belongs to. In order to determine the target class, the $p(s)$ and $p(k|s)$ values in Eq. (12) must be calculated where the former is obtained by dividing the number of documents belonging to class $s$ in the training set by the total number of training documents as represented in Eq. (13). When a text is known to belong to class $s$, the probability that the term $k$ is in the related text is presented in Eq. (14) where the expression in the numerator gives the total frequency of term $k$ in all documents in the relevant class, and the denominator gives the total number of all terms in all documents in class $s$. $\alpha$ values added to the numerator and denominator are for correction to prevent the product in Eq. (12) from being 0 and to improve the estimation performance [33, 34]. When $\alpha = 1$, it is called the Laplace correction, and when it is smaller, the Lidstone correction [34].

$$ p(s|d) = \frac{p(d|s) \cdot p(s)}{p(d)} \qquad (10) $$

$$ s_{ML} = \underset{s \in S}{\operatorname{argmax}} p(s|d) = \underset{s \in S}{\operatorname{argmax}} \frac{p(d|s) \cdot p(s)}{p(d)} \qquad (11) $$
$$s_{ML} = \operatorname{argmax}_{s \in S} \left\{ \prod_{k^d \in K^d} p(k^d|s) \cdot p(s) \right\} \quad (12)$$

$$p(s) = \frac{m(s)}{|D_e|} \quad (13)$$

$$p(k|s) = \frac{[\sum_{d \in s} tf^d(k)] + 1}{\sum_{v \in V, d \in s} [tf^d(v) + 1]} = \frac{[\sum_{d \in s} tf^d(k)] + 1}{\sum_{v \in V, d \in s} [tf^d(v)] + |V|} \rightarrow \frac{[\sum_{d \in s} tf^d(k)] + \alpha}{\sum_{v \in V, d \in s} [tf^d(v)] + \alpha \cdot |V|} \quad (14)$$

Two adaptations have been made to ensure the functioning of the algorithm and increase both speed and prediction performance. First, in Eq. (12), where there are many terms, sum of their natural logarithms instead of the product of these terms, are used to overcome the problem of the product of these terms, which have very small values, converge to 0. Second, the terms with a frequency of 3 or more are taken into account for each text; frequency values are also scaled by taking their natural logarithms.

# 4 Case Study

## 4.1 The Data

When submitting a project proposal to ARDEB, the applicants mark the support group and the panel areas under this support group for which their projects are suitable. The engineering fields and subfields in the ARDEB's Engineering Research Grant Committee (MAG) are presented in Table 3.

As can be seen, Industrial Eng., Aviation & Space Eng., Petroleum & Natural Gas Eng. Consist of only one main field, while detailed sub-fields and “other” option are defined for the remaining fields of engineering. The list consists of 45 sub-fields. As a result of expert experiences and preliminary analyzes on the subject; Aviation & Space Eng., Petroleum & Natural Gas Eng., and Architecture, which have only several proposals within the selected time period, together with the “other” options, are excluded from the scope. Textile Eng., on the other hand, is considered as a single field. For convenience, a Turkish abbreviation consisting of 4 letters has been defined for the each sub-field used in the case study, as seen in Table 3. In the study, 1255 project proposals submitted to MAG within the scope of “1001 Program” between 2015–2021 are analyzed. The break-down of these proposals to sub-fields is presented in Fig. 1.

## 4.2 Performance of the Algorithms

**KeyExt:** The performance of the key-term extraction mostly depends on the extent to which the engineering concepts in the texts are standardized. For this reason, a detailed preliminary study is carried out and templates that transform words into similar forms were applied to cover almost all terms in the field as much as possible. In addition, different from the preceding studies, the postprocessing step customized specifically for this study made a significant contribution.
<table><caption>**Table 3.** Engineering fields and sub-fields in MAG.</caption><tr><td>Industrial Engineering (ENDU)</td><td>Chemical Engineering</td><td>Material Sci. & Engineering</td></tr><tr><td><b>Aviation & Space Engineering</b></td><td>Separation Processes (KMAI)</td><td>Battery/Storage Technologies (MZBP)</td></tr><tr><td><b>Civil Engineering</b></td><td>Biotech.-Bioeng. (KMBB)</td><td>Metallurgy/Met. All. & Comp (MZMM)</td></tr><tr><td>Geotechnics (INGE)</td><td>Polymers (KMPL)</td><td>Polymer Composites (MZPK)</td></tr><tr><td>Hydraulics (INHD)</td><td>Tissue Engineering (KMDM)</td><td>Ceramics/Glass (MZSC)</td></tr><tr><td>Hydrology (INHJ)</td><td>Energy Technologies (KMET)</td><td>Applied Physics (MZUF)</td></tr><tr><td>Coastal and Ocean (INKL)</td><td>Catalysts (KMKT)</td><td>Other</td></tr><tr><td>Architecture</td><td>Other</td><td>Pet. And Nat. Gas Engineering</td></tr><tr><td>Transportation (INUL)</td><td><b>Mechanical Engineering</b></td><td><b>Textile Engineering</b> (TEKS)</td></tr><tr><td>Structures & Earthquake (INYD)</td><td>Energy Systems (MEES)</td><td>Apparel Technology</td></tr><tr><td>Materials of Construction (INYM)</td><td>Manufacturing Tech. (MEIM)</td><td>Technical Textiles</td></tr><tr><td>Building Mechanics (INYK)</td><td>Thermo-Fluid (MEIA)</td><td>Textile Machinery</td></tr><tr><td>Construction Management (INYY)</td><td>Mach. Theo., Dyn. & Des (MEMT)</td><td>Textile Materials</td></tr><tr><td>Other</td><td>Mechanics (MEMK)</td><td>Textile Technologies</td></tr><tr><td><b>Mining Engineering</b></td><td>Mechatronics-Robotics (MEMR)</td><td>Textile Finishing & Chemistry</td></tr><tr><td>Mineral Preparation (MDCH)</td><td>Combustion (MEYN)</td><td>Other</td></tr><tr><td>Mining (MDIS)</td><td>Other</td><td></td></tr><tr><td>Other</td><td></td><td></td></tr></table>

The word clouds showing the high-frequency terms are presented in Fig. 2 for randomly selected projects from two different engineering fields. These key-terms are precise enough to be used without requiring an additional expert examination.

**SimDet:** Determining the similarity for proposals serves two important purposes: identifying re-submitted (revised) proposals and identifying conceptually similar proposals. A pilot study was conducted in MAG to determine the threshold levels of the similarity scores for a proposal to be attributed as a re-submission or as a conceptually similar one. It is found appropriate to evaluate the proposal pairs with a similarity score above 0.90
<figure><img src="image_4.png" /><figcaption>Fig. 1. Break-down of the proposals in the case study by sub-fields: number</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. Word cloud representation of key terms for randomly selected project proposals.</figcaption></figure>

as re-submissions, and those with a similarity score between 0.70–0.90 as conceptually close proposals. The re-submission status information (in the ARDEB database) of 1255 proposals covered in the case study was identified and it was determined that there were 148 re-submitted pairs, 12 re-submitted triples and 1 proposal with 4 versions. Considering all pairwise combinations between re-submissions, there is a total of 190 revision pairs, and the number of proposals with any re-submission is 336. In the case study, similarity scores were calculated for pairwise combinations of 1255 proposals (C(1255,2) = 786,885 pair-wise comparisons). After running the algorithm, 10 more re-submissions (not marked as re-submission in the database) are identified.

<table><caption>Table 4. Performance indicators of SimDet for identifying revised manuscripts.</caption><thead><tr><td>Indicator</td><td>over proposals</td><td>over proposal pairs</td></tr></thead><tbody><tr><td>Accuracy</td><td>97.131%</td><td>99.998%</td></tr><tr><td>Precision</td><td>93.956%</td><td>94.146%</td></tr><tr><td>Recall</td><td>96.067%</td><td>96.500%</td></tr><tr><td>F1 Score</td><td>95.000%</td><td>95.309%</td></tr></tbody></table>

The results show that 342 proposals out of 356 re-submissions are correctly predicted. There are 11 pairs that are not actually re-submissions but have a similarity score above 0.90. However, when these proposals are examined in more detail, 7 of these pairs were identified to be submitted by the same research team. Remaining 4 pairs, on the other
hand, were found to be very close in terms of subject, content and even the project title. SimDet calculated similarity score below 0.90 for 7 pairs which are originally re-submissions. However, it has been observed that significant changes have been made for these proposals in line with the issues highlighted in the review reports. The performance indicators of the algorithm are presented in Table 4.. As can be seen, the accuracy rate is close to 100% in the analysis performed on pairs.

**SubCla:** The 1255 proposals used in the case study were allocated 70%-30% to the training and test clusters. As a result, 895 of 1255 project proposals were used in the training set and 360 in the test set. The distribution of the number of proposals and percentages in the training and test sets by sub-field is given in Table 5..

<table><caption>**Table 5.** Break-down of the training & test data by sub-fields: number of proposals.</caption><thead><tr><th></th><th>Train</th><th>Test</th><th></th><th>Train</th><th>Test</th><th></th><th>Train</th><th>Test</th><th></th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>ENDU</td><td>70</td><td>30</td><td>INYM</td><td>40</td><td>16</td><td>MDCH</td><td>14</td><td>5</td><td>MEYN</td><td>18</td><td>7</td></tr><tr><td>INGE</td><td>22</td><td>9</td><td>INYY</td><td>10</td><td>4</td><td>MDIS</td><td>16</td><td>6</td><td>MZBP</td><td>19</td><td>7</td></tr><tr><td>INHD</td><td>7</td><td>2</td><td>KMAI</td><td>24</td><td>9</td><td>MEES</td><td>38</td><td>16</td><td>MZMM</td><td>47</td><td>19</td></tr><tr><td>INHJ</td><td>7</td><td>2</td><td>KMBB</td><td>45</td><td>19</td><td>MEIA</td><td>62</td><td>26</td><td>MZPK</td><td>54</td><td>21</td></tr><tr><td>INKL</td><td>4</td><td>1</td><td>KMDM</td><td>38</td><td>15</td><td>MEIM</td><td>35</td><td>14</td><td>MZSC</td><td>33</td><td>13</td></tr><tr><td>INUL</td><td>23</td><td>9</td><td>KMET</td><td>10</td><td>2</td><td>MEMK</td><td>38</td><td>17</td><td>MZUF</td><td>68</td><td>31</td></tr><tr><td>INYD</td><td>20</td><td>8</td><td>KMKT</td><td>42</td><td>19</td><td>MEMR</td><td>7</td><td>2</td><td>TKST</td><td>44</td><td>17</td></tr><tr><td>INYK</td><td>15</td><td>6</td><td>KMPL</td><td>9</td><td>2</td><td>MEMT</td><td>16</td><td>6</td><td></td><td></td><td></td></tr></tbody></table>

Performance indicators of the SubCla algorithm on the first estimates over the all sub-fields can be seen in Table 6.. It shows that the SubCla algorithm is quite successful in estimating such a large number of defined sub-fields. For some of the relatively underperforming sub-fields (INHD, INYK, MEMR), the dataset in the relevant field is limited, for some (MZSC, MZUF), interdisciplinary projects are intense as a subject or actually divided into separate sub-fields. When an evaluation is made on the basis of main areas (see Fig. 3), the performance of the algorithm is significantly higher. The results indicate a perfect performance for the fields with distinct and focused proposals, e.g., industrial/textile/mining eng., while a moderate performance for those with overlapping research avenues, e.g., chemical/civil/mechanical eng. and material science.

# 5 Results and Discussion

Rapid growth in the R&D funds and the number of stakeholders in the science and technology ecosystem have brought along a significant increase in the total number of applications. TÜBİTAK has developed a number of information management systems in order to manage the increasing workload and effectively meet the needs of all stakeholders. However, the significant increase in the volume of transactions makes it essential to
<table><caption>**Table 6.** Classifier performance of SubCla algorithm.</caption><thead><tr><th></th><th>Accuracy</th><th>Recall</th><th>F1</th><th></th><th>Accuracy</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>ENDU</td><td>100,0%</td><td>96,8%</td><td>98,4%</td><td>MDCH</td><td>100,0%</td><td>100,0%</td><td>100,0%</td></tr><tr><td>INGE</td><td>100,0%</td><td>90,0%</td><td>94,7%</td><td>MDIS</td><td>100,0%</td><td>100,0%</td><td>100,0%</td></tr><tr><td>INHD</td><td>0,0%</td><td>-</td><td>-</td><td>MEES</td><td>87,5%</td><td>82,4%</td><td>84,8%</td></tr><tr><td>INHJ</td><td>100,0%</td><td>66,7%</td><td>80,0%</td><td>MEIA</td><td>92,3%</td><td>72,7%</td><td>81,4%</td></tr><tr><td>INKL</td><td>100,0%</td><td>100,0%</td><td>100,0%</td><td>MEIM</td><td>78,6%</td><td>91,7%</td><td>84,6%</td></tr><tr><td>INUL</td><td>77,8%</td><td>100,0%</td><td>87,5%</td><td>MEMK</td><td>88,2%</td><td>75,0%</td><td>81,1%</td></tr><tr><td>INYD</td><td>87,5%</td><td>77,8%</td><td>82,4%</td><td>MEMR</td><td>50,0%</td><td>100,0%</td><td>66,7%</td></tr><tr><td>INYK</td><td>33,3%</td><td>50,0%</td><td>40,0%</td><td>MEMT</td><td>50,0%</td><td>50,0%</td><td>50,0%</td></tr><tr><td>INYM</td><td>87,5%</td><td>100,0%</td><td>93,3%</td><td>MEYN</td><td>71,4%</td><td>100,0%</td><td>83,3%</td></tr><tr><td>INYY</td><td>100,0%</td><td>100,0%</td><td>100,0%</td><td>MZBP</td><td>71,4%</td><td>83,3%</td><td>76,9%</td></tr><tr><td>KMAI</td><td>77,8%</td><td>77,8%</td><td>77,8%</td><td>MZMM</td><td>73,7%</td><td>73,7%</td><td>73,7%</td></tr><tr><td>KMBB</td><td>78,9%</td><td>88,2%</td><td>83,3%</td><td>MZPK</td><td>71,4%</td><td>71,4%</td><td>71,4%</td></tr><tr><td>KMDM</td><td>100,0%</td><td>71,4%</td><td>83,3%</td><td>MZSC</td><td>69,2%</td><td>90,0%</td><td>78,3%</td></tr><tr><td>KMET</td><td>50,0%</td><td>100,0%</td><td>66,7%</td><td>MZUF</td><td>74,2%</td><td>88,5%</td><td>80,7%</td></tr><tr><td>KMKT</td><td>94,7%</td><td>81,8%</td><td>87,8%</td><td>TKST</td><td>100,0%</td><td>89,5%</td><td>94,4%</td></tr><tr><td>KMPL</td><td>50,0%</td><td>100,0%</td><td>66,7%</td><td></td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Classifier performance of SubCla algorithm: over main fields.</figcaption></figure>

develop intelligent DSSs that can assist users and decision makers at every stage. This study, as a continuation of PaneLIST [2], develops NLP-based algorithms for proposals, i.e., extracting key terms, similarity determination, and automatic subject/topic assignment. This study fills an important gap in the literature in the context of these issues for the Turkish academic texts. As mentioned in the literature section; although there are studies on the classification of Turkish texts (comments on movie/hotel/restaurant websites, website/tweet/e-mail contents or news texts), no study has been found on the classification of academic texts. A detailed version of this study is available as a peer-reviewed article in Turkish [38]. In the current study, the proposals submitted to MAG
are examined, and a normalization library is created based on the words and terms used
in the field of engineering. Then, a data set containing 1255 project proposals is pre-
pared, and in the first stage, key terms are extracted from the proposals and a similarity
algorithm was developed over these key-term vectors. Afterwards, the Naïve Bayes clas-
sification approach is employed to predict the category that each proposal belongs to.
It has been shown that the proposed algorithm fully meets the need in the detection of
re-submission or similar proposals. Moreover, for the classification algorithm, a success
rate of 83.3% in the first estimation, 92.5% in the first two estimations and 96.4% in the
first three estimations is achieved.

References

1. Khan, A., Baharudin, B., Lee, L., Khan, K.: A review of machine learning algorithms for text-documents classification. J. Adv. Inf. Technol. **1**(1), 4–20 (2010)

2. Kat, B.: An algorithm and a decision support system for the Panelist assignment problem: the case of TÜBİTAK. J. Fac. Eng. Archit. Gazi Univ. **36**(1), 69–88 (2021)

3. Çagtayli, M., Çelebi, E.: The effect of stemming and stop-word-removal on automatic text classification in Turkish language. In: Arik, S., Huang, T., Lai, W.K., Liu, Q. (eds.) Neural Information Processing. LNCS, vol. 9489, pp. 168–176. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26532-2_19

4. Deniz, A., Kiziloz, H.E.: Effects of various preprocessing techniques to Turkish text categorization using n-gram features. UBMK **2017**, 655–660 (2017)

5. Öztürkmenoğlu, O., Alpkoçak, A.: Comparison of different lemmatization approaches for information retrieval on Turkish text collection. INISTA **2012**, 1–5 (2012)

6. Kat, B.: Analysis of the tools used for identifying similarities of scientific texts: addressing the road map and the pertinent approach for ARDEB project proposals. TÜBİTAK (2015)

7. Vrbanec, T., Mestrovic, A.: The struggle with academic plagiarism: approaches based on semantic similarity. MIPRO **2017**, 870–875 (2017)

8. Chong, M., Specia, L., Mitkov, R.: Using natural language processing for automatic detection of plagiarism. In: IPC-2010 (2010)

9. Gomaa, W.H., Fahmy, A.A.: A survey of text similarity approaches. Int. J. Comput. Appl. **68**(13), 975–8887 (2013)

10. Dharmadhikari, S.C., Ingle, M., Kulkarni, P.: Empirical studies on machine learning based text classification algorithms. Adv. Comput. Int. J. **2**(6), 161–169 (2011). https://doi.org/10.5121/acij.2011.2615

11. Kandimalla, B., Rohatgi, S., Wu, J., Giles, C.L.: Large scale subject category classification of scholarly papers with deep attentive neural networks. Front. Res. Metrics Anal. **5**, 600382 (2021)

12. Kadhim, A.I.: Survey on supervised machine learning techniques for automatic text classification. Artif. Intell. Rev. **52**(1), 273–292 (2019). https://doi.org/10.1007/s10462-018-09677-1

13. Gurcan, F.: Multi-class classification of Turkish texts with machine learning algorithms. ISMSIT **2018**, 1–5 (2018)

14. Koksal, O.: Tuning the Turkish Text Classification Process Using Supervised Machine Learning-based Algorithms. INISTA (2020)

15. Küçük, D., Arici, N.: A literature study on deep learning applications in natural language processing. UYBİSBBD **2**(2), 76–86 (2018)

16. Kilimci, Z.H., Akyokus, S.: The evaluation of word embedding models and deep learning algorithms for turkish text classification. UBMK **2019**, 548–553 (2019)
17. Kilimci, Z.H., Akyokus, S.: Deep learning- and word embedding-based heterogeneous classifier ensembles for text classification. Complexity **2018**, 1–10 (2018). https://doi.org/10.1155/2018/7130146

18. Aydin, G., Hallaç, İR.: Automatic topic detection on Turkish text. Fırat Univ. J. Eng. Sci. **33**(2), 599–606 (2021)

19. Güran, A., Akyokuş, S., Güler Bayazıt, N., Gürbüz, M.Z.: Turkish Text Categorization Using N-Gram Words. INISTA, pp. 369–373 (2009)

20. Erşahin, B., Aktaş, Ö., Kilinç, D., Erşahin, M.: A hybrid sentiment analysis method for Turkish. Turkish J. Elect. Eng. Comput. Sci. **27**, 1780–1793 (2019)

21. Kaya, M., Fidan, G., Toroslu, I. H.: Sentiment analysis of Turkish political news. In: IEEE/WIC/ACM International Conference on Web Intelligence, pp. 174–180 (2012)

22. Boynukalın, Z.: Emotion Analysis of Turkish texts by using machine learning method. METU (2012)

23. Kaşıkçı, T., Gökçen, H.: Determination of e-Commerce sites by text mining. J. Inf. Technol. **7**(1), 25–32 (2014)

24. Kaynar, O., Görmez, Y., Yıldız, M., Albayrak, A.: Sentiment Analysis with Machine Learning Techniques. IDAP (2016)

25. Salur, M., Aydın, I., Jamous, M.: An ensemble approach for aspect term extraction in Turkish texts. Pamukkale Univ. J. Eng. Sci. **28**(5), 769–776 (2021)

26. Coban, O., Ozyer, B., Ozyer, G.T.: Sentiment analysis for Turkish Twitter feeds. In: SIU, pp. 2388–2391 (2015)

27. Uysal, A.K., Gunal, S.: The impact of preprocessing on text classification. Inf. Process. Manage. **50**(1), 104–112 (2014)

28. Yıldırım, S., Yıldız, T.: A comparative analysis of text classification for Turkish language. Pamukkale Univ. J. Eng. Sci. **24**(5), 879–886 (2018)

29. Aydin, G., Hallac, I.R.: Document Classification Using Distributed Machine Learning. arXiv preprint:1802.03597, 166–169 (2018)

30. Yau, C.-K., Porter, A., Newman, N., Suominen, A.: Clustering scientific documents with topic modeling. Scientometrics **100**(3), 767–786 (2014). https://doi.org/10.1007/s11192-014-1321-8

31. Kim, S.-W., Gil, J.-M.: Research paper classification systems based on TF-IDF and LDA schemes. HCIS **9**(1), 1–21 (2019). https://doi.org/10.1186/s13673-019-0192-7

32. Suominen, A., Toivanen, H.: Map of science with topic modeling: comparison of unsupervised learning and human-assigned subject classification. JASIST **67**(10), 2464–2476 (2016)

33. Kılınç, D., Borandağ, E., Yüçalar, F., Tunali, V., Şimşek, M., Özçift, A.: Classification of scientific articles using text mining with KNN algorithm and R language. Marmara J. Pure Appl. Sci. **28**(3), 89–94 (2016)

34. Raschka, S.: Naive Bayes and Text Classification I - Introduction and Theory. arXiv preprint:1410.5329 (2014)

35. Huang, Y., Li, L.: Naive Bayes classification algorithm based on small sample set. In: IEEE CCIS2011, pp. 34–39 (2011)

36. Chandrasekar, P., Qian, K.: The impact of data preprocessing on the performance of a Naïve Bayes classifier. IEEE COMPSAC **2**, 618–619 (2016)

37. Noyan, T., Kuncan, F., Tekin, R., Kaya, Y.: A new content-free approach to identification of document language: angle patterns. J. Fac. Eng. Archit. Gazi Univ. **37**(3), 1277–1292 (2022)

38. Kat, B.: Natural language processing for the Turkish academic texts in the engineering field and development of a decision support system: the case of TUBITAK project proposals. J. Fac. Eng. Archit. Gazi Univ. **38**(3), 1879–1892 (2023)