IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024

1

Evaluation of Similarity Analysis of Newspaper Article
Using Natural Language Processing
Ayako Ohshiro 1†, Takeo Okazaki 2††, Takashi Kano3†††, and Shinichiro Ueda4††††

Department of Business Administration, Okinawa International University,
Ginowan, 0kinawa 901-2701, Japan
2††
Faculty of Engineering, University of the Ryukyus, Okinawa 903-0213, Japan
3†††
Graduate School of Economics Hitotsubashi University, Naka, Kunitachi Tokyo, Japan
4††††
Department of Clinical Research and Quality Management Graduate School of Medicine University of the Ryukyus
Nishihara, 0kinawa 903-0215, Japan
1†

Abstract

Comparing text features involves evaluating the ”similarity”
between texts. It is crucial to use appropriate similarity measureswhen comparing similarities. This study utilized various
techniques to assess the similarities between newspaper articles,
including deep learning and a previously proposed method: a
combination of Pointwise Mutual Information (PMI) and Word
Pair Matching (WPM), denoted as PMI+WPM. For performancecomparison, law data from medical research in Japan were utilized
as validation data in evaluating the PMI+WPM method. The
distribution of similarities in text data varies depending on theevaluation technique and genre, as revealed by the comparative
analysis. For newspaper data, non-deep learning methods
demonstrated better similarity evaluation accuracy than deeplearning methods. Additionally, evaluating similarities in law data
is more challenging than in newspaper articles. Despite deeplearning being the prevalent method for evaluating textual
similarities, this study demonstrates that non-deep learning
methods can be effective regarding Japanese-based texts.

Keywords:

Pointwise Mutual Information, Simpson coefficient, Doc2vec,
BERT, Newspaper.

1. Introduction
The development of computer functions has facilitatedaccess to previously inaccessible data, thereby expanding
the possibilities of data analysis and enabling diverse
perspectives in interpreting data that differ from traditional
approaches. For instance, by quantitatively analyzing timeseries text data from newspapers that document changes insocietal conditions over time, researchers have visualized
the daily evolution of topics [1] and compared article
characteristics across different newspapers [2]. Comparing
text features involves evaluating the “similarity” between
texts. It is crucial to use appropriate similarity measureswhen comparing similarities. In natural language
processing, the extraction of structured or similar
documents [3] has been crucial in comprehending andmanaging them more efficiently. For example, local
Manuscript received June 5, 2024
Manuscript revised June 20, 2024
https://doi.org/10.22937/IJCSNS.2024.24.6.1

government ordinances have similar content, but theirdifferences across regions are compared manually byprofessionals [4]. Adopting a computer-based language
processing approach is expected to reduce working costs forthese tasks [5][6][7][8]. A study has been conducted thatapplies text mining methods directly to law documents. The
study aims to interpret law documents using word
frequency as an index [9]. Recently, comparative studies forlaw documents using deep learning algorithms such as a
document vector feature from BERT [10] and Doc2vec [11]have increased. Our study has thus far concentrated onclinical research laws including, three guidelines,
ministerial ordinances, and the law regarding clinicalresearch. Visualizing relationships can help clinicalresearchers busy with clinical practice to understand them.
Our research has successfully predicted similarities
between these laws and visualized them. First, the study ofthe possibility of interpreting the relationship between laws
related to clinical research using word2vec and topic model
has been considered [12][13]. In addition, each law wasanalyzed by generating a co-occurrence network ofincluded words. The network was used to interpret the law
at the word level, by identifying pairs of words with strong
co-occurrence relationships, commonly occurring words,
and comparing their co-occurring words [14]. Furthermore,
the study predicted the similarity between three laws
using ”word-matching” under the hypothesis that sentenceswith more common words are more similar [15]. The
verification indicated that while the method correctly
identified the ”high similarity/match” combination, it also
incorrectly
evaluated
some
pairs
as
”low
similarity/mismatch,” presenting issues to be addressed.
Subsequently, a new scale for predicting similarity was
proposed by [16], which combines ”Word Group Matching”
with ”Pointwise Mutual Information” to evaluate thesimilarity of laws based on the probability of co-occurrenceof multiple words. The evaluation experiment result
suggests that this approach has the potential to improveaccuracy compared to conventional methods depending on

2

IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024

the combination used. In cases where a similarity evaluation
method is constructed based on the text data used in theexperiment, it becomes necessary to validate the method forgeneralization by applying it to different text data andmethods. For instance, comparing the accuracy of similarity
evaluations using text data from different genres can further
assess the property of the proposed method and lead tomeasures suited to specific text genres. Therefore, this study
aims to evaluate the similarity of newspaper articles using
multiple similarity evaluation methods in natural language
processing, including the previously proposed PMI+WPM.
The structure of this study is as follows: Section 2 describes
the similarity evaluation methods used, and Section 3
outlines the experimental setup. Section 4 presents theexperimental data, the results, and discussion, and Section
5 concludes the study.

2. Similarity Evaluation Method
The objective of this study is to evaluate thesimilarities between newspaper articles, as depicted in
Figure 1.

Furthermore, if the combinations of a specific word group
have a high degree of matching, the documents are
considered highly similar. Considering the proportional
relationship between the occurrence frequency of word
groups and the length of the document, it is essential toconsider the ratio of the total number of occurrences ofword groups represented as 𝐹𝐹{𝑥𝑥, 𝑦𝑦} when determining thesimilarity between documents based on the degree of matchof word groups. In this study, the degree of word groupmatching represented as 𝑤𝑤𝑥𝑥,𝑦𝑦 in texts 𝐴𝐴 and 𝐵𝐵 is denoted as
WGM{𝑤𝑤𝑎𝑎 (𝑥𝑥, 𝑦𝑦), 𝑤𝑤𝑏𝑏 (𝑥𝑥, 𝑦𝑦)}, and the similarity between thetwo documents using the total number of occurrences ofword groups in the compared texts, 𝐹𝐹{𝑥𝑥, 𝑦𝑦}, is defined as
PMI + WGM according to Equation (2).

𝑃𝑃(𝑥𝑥, 𝑦𝑦) + WGM{𝑤𝑤𝑎𝑎 (𝑥𝑥, 𝑦𝑦), 𝑤𝑤𝑏𝑏 (𝑥𝑥, 𝑦𝑦)}𝐹𝐹(𝑥𝑥, 𝑦𝑦)

When the words in documents A and B are consideredsets A and B, the similarity between documents A and B iscalculated as in (3) using the Simpson coefficient.

|𝐴𝐴 ∩ 𝐵𝐵|
𝑚𝑚𝑚𝑚𝑚𝑚(|𝐴𝐴|, |𝐵𝐵|)
Fig 1: Similarity between articles (example)

This section describes the natural language processing
methods and evaluation index used for assessing similarity
and the experimental procedures.

2-1. Similarity Evaluation Methods without
Deep LearningThis section describes the similarity evaluation
methods that utilize set and information theories rather thandeep learning. If a specific combination of words appears
frequently, it infers a high similarity between the documents.
When two words x and y appear within a single text, their
Pointwise Mutual Information (PMI), PMI(x,y), can be
expressed as indicated in Equation (1).

𝑃𝑃𝑃𝑃𝑃𝑃 (𝑥𝑥, 𝑦𝑦) = log

𝑃𝑃(𝑥𝑥,𝑦𝑦)𝑃𝑃(𝑥𝑥)𝑃𝑃(𝑦𝑦)

(1)

(2)

(3)

Despite the availability of related coefficients such as
Dice and Jaccard, the Simpson coefficient was selected forcomparison in this study due to its superior performance inpreliminary experiments.

2-2. Similarity Evaluation Methods with
Neural Network
In the Doc2vec algorithm, text features are converted
vector and calculated similarities with unsupervised
learning. It was developed after Word2Vec with a neuralnetwork that generates hundreds of dimensional vectors foreach word based on surrounding words and numerical
relationships between words. By extending worddistributed representation to Paragraph Vector [17],document similarity could be calculated using a documentdistributed representation.
The BERT algorithm can accurately infer from thecontextual information of sentences. The accuracy of word
prediction based on context, which was previously
challenging to improve, has been enhanced by combining
bidirectional learning with conventional unidirectional

IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024

3

learning and completing fill-in-the-blank questions.
Consequently, the study aims to correlate similar law
documents [10][11].

The procedure for evaluating the similarity of newspaper
articles is outlined below and depicted in Figure 2.

3. Experiment

Step1

Obtain benchmark data by requesting anexpert in economics to rate each combinationof newspaper articles on a four-level ordinalscaleof”match/highsimilarity/low
similarity/mismatch.”

Step2

Calculate the similarity scores for all possiblecombinations of the target text data.

Step3

Divide the distribution of similarity scoresobtained in Step 2 into quarters, with the range
defined as (maximum value - minimum value)
/ 4.

Step4

Convert the quartile measures derived in Step
3 into the four-tiered ordinal scale defined in
Step 1.

Step5

Calculate the rank correlation coefficients
between each quartile measure and thebenchmark data to determine the estimation
accuracy.

This section outlines the procedure for determining
article similarity in newspapers using the previously
mentioned evaluation methods.
The rank correlation coefficient [0,1] is utilized as
preprepared benchmark data to evaluate the performance ofthe calculated similarities. The benchmark data werecollected by soliciting evaluations from economics expertson the similarity between newspaper articles, using a fourlevel ordinal scale of “match/high similarity/low
similarity/mismatch,” as presented in Table 1.

Table 1: Benchmark data for similarity between articles (EXAMPLE)𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝒊𝒊𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝟏𝟏
𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝟏𝟏:
𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝟏𝟏𝟏𝟏

𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝒋𝒋
𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝟐𝟐
𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝟑𝟑:
𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝟓𝟓𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺
mismatchmatch:
low similarity

Furthermore, to calculate the rank correlation
coefficient with the benchmark data, the similarity scoresobtained using the similarity evaluation methods must be
converted into ordinal scales. In this study, the range of eachsimilarity score was evenly divided into four levels, and,similar to the benchmark data, the scores were converted
into ”identical/high similarity/low similarity/no similarity”
based on their rank of similarity.

Step 1 corresponds to the procedure for generating
benchmark data, Steps 2 to 4 correspond to the procedurefor generating test data using the similarity assessment scale,and Step 5 corresponds to the procedure for evaluating theperformance of the test data.

4

IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024

Fig 2: Procedure of investigation (Steps 1- Step4)

IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024

5

Table 2: Distribution of similarity with each method applied to each data

Method

WGM+PMI
Simpson’s
Coefficient
Doc2vec
BERT

Max

L1・L2

Min

SD

Max

0.783

0

0.124

0.522
0.728

-0.356
0.561

0.621
0.032

0.707

0

0.009

Law
L2・L3

Min

SD

Max

0.270

0

0.034

0.450
0.755

-0.436
0.556

0.627
0.032

0.638

0

0.150

L1・L3

Newspaper

Min

SD

Max

Min

SD

0.199

0

0.039

0.973

0.014

0.150

0.476
0.769

-0.466
0.538

0.666
0.033

1.000
0.829

-0.355
0.556

0.216
0.033

0.544

0

0.016

0.825

0.002

0.077

Table 3: Ordinal correlation coefficient between similarity calculated by each method for each data and benchmark data

Method
WGM+PMI
Simpson’s
Coefficient
Doc2vec
BERT

L1・L2
0.454

Law
L2・L3
0.320

L1・L3
0.205

0.627

0.311

0.355

0.218

0.662

-0.0180.205-0.019
0.405

0.012
0.397

0.368
0.262

4. Results and Discussion
This section describes the data used in the experimentand the results obtained following the procedure outlined in
the previous section.

4-1. Experimental Data
Newspaper articles widely reflect the social landscapeand are read by a broad spectrum of society. In the field of
language processing, these are frequently processed. They
must be written in a manner that is easily comprehensible toanyone who has completed compulsory education. The aim
is to convey content briefly to readers, with the main points
presented at the beginning and details following later. This
structure is a characteristic feature [20]. In addition to thecontent [21], the sentence structures and their organization
have been observed to change over time. In this study, we
analyzed articles related to ”economics” from the ”Ryukyu
Shimpo,” a local newspaper, to ensure that our research was
regionally relevant. The data covers two weeks from
September 2023.

4-2. Discussion
In this section, accuracy is defined as the ordinalcorrelation coefficient between benchmark data and the
proposed similarity evaluation scale. Additionally, thedifferences between methods and text data were explored.
The similarity evaluation performance was also comparedwith the traditional method. To compare the differences in
Manuscript received June 5, 2024
Manuscript revised June 20, 2024
https://doi.org/10.22937/IJCSNS.2024.24.6.1

Newspaper

results depending on the text data, the results of similarity
evaluation between each article of the three types of law
data that is validation, data for the previously proposed
method as WGM+PMI, was utilized. First, to show thesimilarity distribution of each method used for each text
data, the maximum value (Max), the minimum value (Min),and the standard deviation (SD) of similarity are presented
in Table 2.
The minimum value is 0 when using WGM + PMI orSimpson coefficient, indicating that ”the degree ofsimilarity at the word level is 0” from the result of using law
data. However, the minimum value of the BERT model
exceeds 0.5, indicating that it is significantly higher than theminimum value of other methods. Additionally, thestandard deviation of Doc2vec exceeds 0.6 and the variation
is significantly higher than other methods, owing to thesimilarity of WGM+PMI, Simpson coefficient, and BERT,
which all fall in the range of [0,1]. This aspect may be
because Doc2vec is [-1,1]. Subsequently, when using
newspaper data, the Simpson coefficient and Doc2vec
exhibit maximum values exceeding 0.95, while WGM+PMI
and BERT demonstrate values below 0.85. Additionallysimilar to the law data, the minimum values of WGM+PMI,
Simpson coefficient, and Doc2vec are below 0.015,
indicating a very low degree of similarity. However, theminimum value of BERT is above 0.5, which suggests ahigher degree of similarity. Consequently, BERT exhibits alower standard deviation of similarity degree compared toother methods Regardless of the method used, themaximum similarity value is higher when using newspaper
data. In addition, the standard deviation is particularly low

6

IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024

when using Doc2vec with newspaper data. From the resultof BERT, no significant changes were observed in any of
the indicators, including maximum value, minimum value,or standard deviation, depending on the data genre.
Applying WGM + PMI, Simpson coefficient, and Doc2vec
to newspaper data resulted in a higher maximum value.
Additionally, the distribution of similarity measures was not
significantly affected by differences in data genre by thecase using BERT.
Subsequently, the ordinal correlation coefficient
between the similarities calculated by each method and thebenchmark data is presented in Table 3. This coefficient
represents the accuracy of each method.
First, the similarities of the data used were compared. When
analyzing law data, negative values were observed in somecases, indicating that the accuracy of Doc2Vec wassignificantly lower than that of the proposed method WGM+ PMI, Simpson coefficient, and BERT. WGM+PMI or
BERT were the preferred methods, but the best results
achieved were less than 0.5. Consequently, identifying anappropriate method was challenging. However, when using
newspaper data, the proposed method and the Simpson
coefficient achieved an accuracy of over 0.6, while deeplearning methods such as Doc2Vec and BERT resulted inan accuracy below 0.4. Classical similarity measure may be
more effective than deep learning in indicating thesimilarity of newspaper data, as demonstrated by the results.
Next, we compare the similarities among different methods.
When using WGM + PMI, Simpson coefficient, andDoc2vec, the accuracy is higher with newspaper data. Whenusing BERT, the accuracy varies depending on the
combination, but it is still higher with law data. Whenfocusing on the characteristics of law data and newspaper
article data, due to the limited vocabulary and semantics oflaw data, BERT, which is bidirectional is more suitable forsimilarity evaluation than WGM+PMI and Doc2vec, which
predict specific words from surrounding words. Newspaper
article data are expected to exhibit a higher frequency of
relevant words in proximity to the target word than law data.
Thus, the use of similarity measures such as WGM+PMI
and the Simpson coefficient, which are based on word cooccurrence probabilities and sets, is more appropriate.
Consequently, this demonstrates the potential for
selecting appropriate mining methods tailored to thecharacteristics of each genre within textual data.

5. Conclusion
In this study, the similarities between newspaper articles
were evaluated using various similarity evaluation methods
in natural language processing. These methods included
approaches based on deep learning, incorporating set theory

and information theory. For performance comparison, in
addition to newspaper article data, law document data that
had been used to verify the previously proposed
WGM+PMI were also utilized. Results of the similarity
distribution comparison for all data combinations indicatethat the similarity distribution varies with the text data andthe methods used. Furthermore, even when using the same
method, the accuracy of similarity evaluation performancevaries depending on the genre of text data handled.
Many similarity evaluation methods have been
proposed in the field of language processing. However, thisstudy demonstrates the potential for better similarity
evaluation by selecting methods appropriate to the genre of
text data being analyzed.

References[1] James Allan, Rahul Gupta, and Vikas Khandelwa, ”
Temporal Summaries of News Topics.”, In
Proceedings of the 24th annual international ACM
SIGIR conference on Research and development in
information retrieval，2001.
[2] Yusuke Hoshino, “A Preliminary Analysis ofNewspaper Editorials on COVID-19 Using Natural
Language Processing Technologies : Differencesamong Newspapers and Further Research”, Musashino
University Management Journal (5), 113-148, 2022
[3] Larry M. Manevitz, Malik Yousef, ”One-Class SVMs
for Document Classification”, Journal of Machine
Learning Research 2 ,139-154, 2001
[4] Fujii, Machiko, “The Present Condition and Issues ofMunicipal Ordinances in Merger of Nuncipalities :Case of the Ordinance Making of Koka City ”, Thebulletin of the Graduate School of Law, Ryukoku
University, 181-214, 2007
[5] KAKUTA Tokuyasu, “An analysis of regulations oflocal governments using a supercomputer and theapplication to a regulation database”, Nagoya
University Journal of Law and Politics, Vol. 246, 6991, 2012[6] TAKENAKA YOICHI, WAKAO TAKESHI,“Automatic Generation of Article Correspondence
Tables for the Comparison of Local Government
Statutes”, Journal of natural language processing, Vol.
19 No. 3, pp.194- 212, 2012.
[7] Gaitake.K, Tomoya.S, Youiti.T, “Meizi minpou seitei
zi ni okeru nitihutu minpou zyoubun no sansyou kankei
sai suitei ” (in Japanese) The 25th Annual Meeting of
the Association for Natural Language Processing,pp.398-401, 2019.
[8] Gaitake.K, Tomoya.S, Youiti.T, ”Meizi minpou tokakukokumin hou to no zyoubun ruizi kankei ni
motozuku rikkyakuten no kaiseki” (in Japanese), The
26th Annual Meeting of the Association for Natural
Language Processing, pp93-96, 2020

IJCSNS International Journal of Computer Science and Network Security, VOL.24 No.6, June 2024
[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina
Toutanova,” Bidirectional Encoder Representations
from Transformers”, In Proceedings of the 2019
Conference of the North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short
Papers), 4171–4186, 2019[10] Quoc V. Le, Tomas Mikolov, ”Distributed
Representations of Sentences and Documents”,
proceedings of the 31st International Conference on
Machine Learning, PMLR 32(2):1188-1196, 2014[11] Ayako OHSHIRO, Shinichiro UEDA. “ Feature
extraction of each laws for clinical research and theirrelation“ , Institute of Electronics, Information andCommunication Engineers. 2019; D-5-4[12] Ayako OHSHIRO, Shinichiro UEDA. ”Interpretabilityof laws related to clinical research with text mining.”,
THE
INSTITUTE
OF
ELECTRONICS,
INFORMATIONANDCOMMUNICATIONENGINEERS. 2019 ; NLC2019-33 (320)NLC201933:35-40[13] Ayako OHSHIRO, Takeo OKAZAKI, Shinichiro
UEDA. “Visualization of clinical research-related laws
using co-occurrence network”, The Japanese Society ofClinicalPharmacology
andTherapeutics(JSCPT).2024; 55(1): 57-62.
[14] Ayako OHSHIRO, Takeo OKAZAKI, Shinichiro
UEDA. “ Study on relationship visualization of clinicalresearch-related laws using word- matching“ , The
Japanese Society of Clinical Pharmacology andTherapeutics (JSCPT). 2023 ; 54(1): 43–48.
[15] Ayako OHSHIRO, Takeo OKAZAKI, Shinichiro
UEDA. ziko sougo zyouhou ryou to tango gun itti do
wo ku mi a wase ta rinsyou kenkyuu kanren hourei noruizisei hyouka no kentou (in Japanese). The 29th
Annual Meeting of the Association for Natural
Language Processing, pp1216-1219, 2023
[16] Mikolov, T.; Le, Q. Distributed representations ofsentences and documents. Proceedings of the 31st
International Conference on Machine Learning, ICML
2014, 2014, p.1188-1196.
[17] Hiroki T, Makoto N, ”BERT wo moti i ta hikakuhou
kenkyuu ni okeru ruizi zyoukou no taiou zu ke” (inJapanese), The 28th Annual Meeting of the Association
for Natural Language Processing, pp948-951, 2022
[18] Kaito Koyama, Tomoya Sano, Yoichi, ”The legislative
study on Meiji civil code by machine learning”,
Fifteenth International Workshop on Juris-informatics(JURISIN 2021)[19] Sinryou.H, ”「wakariyasusa」wo mezasi te ka ka re ta
sinbun kizi no buntai teki tokutyou syakai gengogaku(in Japanese)”, The Japanese journal of language insociety, pp.43-54, vol.15(2015).
[20] Yuta Ichikawa, “PRELIMINARY STUDY ON
DETECTION OF NEWSPAPER TREND AMONG

7THEIR PUBLISHERS USING TEXT-MINING
APPROACH”, Bulletin of graduate studies.
Engineering Hosei University, Vol.57(2016)
Ayako Ohshiro received the B.S. and
M.S. degrees from University of the
Ryukyus in 2009 and 2011, respectively.
She took Ph.D. from University of
Ryukyus in 2017 and she has been an
associate
Professor
atOkinawa
International University. Her researchinterests are data analysis at the field of Intelligent
information engineering.

Takeo Okazaki received B.Sc. andM.Sc.
degreesfrom
KyushuUniversity in 1987 and 1989,
respectively. He earned his Ph.D. from
University of the Ryukyus in 2014. He
is currently a professor at the
University of the Ryukyus. His
research interests are statistical datanormalization for analysis, statistical analysis, data analysis,genome informatics, tourism informatics, geographic
information systems, and data science. He is a member of
JSCS, IEICE, JSS, GISA, and BSJ Japan.
Takashi Kano received his Ph.D.
from the University of British
Columbia in 2003 and has been aprofessor at Hitotsubashi University
since 2016. His research interests
includemacroeconomics,
international finance, and applied
econometrics.
Shinichiro Ueda received the M.D in
1985 and took Ph.D. from Yokohama
City University in 1997 and he has been
professor at University of the Ryukyus.
His research interests are ClinicalPharmacology and Therapeutics, andClinical Research education.