# Summary and Evaluation of the Application of Knowledge Graphs in Education 2007–2020

Authors: Yanmei Mao
Journal: Discrete Dynamics in Nature and Society
 (2021)

---

## Abstract

Since 2007, knowledge graphs, an important research tool, have been applied to education and many other disciplines. This paper firstly overviews the application of knowledge graphs in education and then samples the knowledge graph applications in CSSCI- (Chinese Social Sciences Citation Index-) indexed journals in the past two years. These samples were classified and analyzed in terms of research institute, data source, visualization software, and analysis perspective. Next, the situation of knowledge graph applications in education was summarized and evaluated in detail. Furthermore, the authors discussed and assessed the normalization of knowledge graph applications in education. The results show that in the past 15 years, knowledge graphs have been widely used in education. The academia has reached a consensus on the paradigm of the research tool: examining the hotspots, topics, and trends in the related fields from the angles of keyword cooccurrence network (KCN), time zone map, clustering network, and literature/author cocitation, with the aid of CiteSpace and other visualization software and text analysis. However, there is not yet a thorough understanding of the limitations of the visualization software. The relevant research should be improved in terms of scientific level, normalization level, and quality.


---

## Body

## 1. Introduction

Knowledge graphs provide an extensively applied research tool. Since 2007, many domestic scholars have successfully introduced this tool to study the cooperative research models, hotspots, topics, and trends in their research domains. On June 7th, 2021, our research team found 6,277 Chinese papers with “knowledge graphs” in their titles on CNKI and 3,342 foreign papers with the same words in their titles on the Engineering Index Database. However, the research tool and its supporting software [1] were developed in foreign countries and have not been applied for a very long time. Therefore, the application of knowledge graphs in education and other disciplines generally faces problems like poor research quality and low scientific and normalization levels.Literature research shows that a handful of scholars discussed the effectiveness [2, 3] and normalized use of knowledge graphs, and some put forward suggestions for improving and innovating the paradigm of CiteSpace research in Chinese journals [4, 5]. But very few Chinese researchers have systematically reviewed or evaluated the scientific and normalization levels of the domestic papers on the application of knowledge graphs. Tang [6] published “Review and Evaluation of the Empirical Research Essays in Domestic Knowledge Mapping Areas,” which is one of the few papers that deal with the said issue. There is virtually no report on the summary and evaluation of knowledge graph applications in education.Inspired by Kuhn’s [7] paradigm theory and other scholars’ discoveries [8, 9], following the basic requirements [10–13] on empirical papers of social sciences [14, 15], this paper summarizes the academic papers and graduation theses in CNKI, as well as the published books, which report the application of knowledge graphs in education, under the analysis framework of Tang [6]. On this basis, the authors evaluated the scientific and normalization levels of the knowledge graph applications in CSSCI- (Chinese Social Sciences Citation Index-) indexed journals in 2019–2020. The purpose is to systematically review the evolution, application state, and future trends of knowledge graph applications in education in China and promote the healthy implementation of the research tool in the field of education.


## 2. Overview of Knowledge Graph Applications in Education

This paper queries for each of the three types of literature, namely, academic papers, graduation theses, and books, and carries out a comprehensive analysis in terms of the annual distribution of literature quantity, distribution of high-yield institutions, distribution of prolific authors, and distribution of research topics.


### 2.1. Knowledge Graph Applications in Academic Papers

The authors queried for the existing papers on CNKI about knowledge graph applications in education (query date: June 7th, 2021) and obtained 1,073 records after discrimination and screening. Figure1 shows the annual distribution of literature quantity.Figure 1
Annual distribution of the quantity of academic papers.As shown in Figure1, Chinese education researchers first studied knowledge graph applications in 2007. The earliest published paper was authored by Peng et al. at the Dalian University of Technology, which is titled “Knowledge Graph Analysis on the Research State of International Entrepreneurship University” [16]. This is the earliest application of knowledge graphs in education. It is only two years later than the first domestic attempt to apply knowledge graphs [17].From the annual distribution of the quantity of academic papers, it can be seen the number of knowledge graph applications in education has been exploding since 2011, especially in the past three years. More than 200 academic papers were published in each of these three years.The top 10 high-yield institutions are listed as follows: Shaanxi Normal University published the most academic papers (56), Beijing Normal University (32), Nanjing Normal University (29), Central China Normal University (27), Wenzhou University (23), Henan University (21), Southwest University (20), Liaoning Normal University (19), East China Normal University (17), and Capital Normal University (16). The top-ranking institution published over three times more papers than the institution ranking in the 10th place (as shown in Figure2).Figure 2
Papers published by high-yield institutions.The top 9 prolific authors are listed as follows: Chen Yulin at Jiaying University (10), Cai Jiandong at Henan University (9), Cai Wenbo at Shihezi University (8), Chang Qinghui at Tiangong Technology (7), Li Yubin at Liaoning Normal University (6), Qi Zhanyong at Shaanxi Normal University (5), Yuan Liping at Shaanxi Normal University (5), Sun Furong at Wenzhou University (5), and Tang Jianmin at Zhejiang Shuren University (5) (as shown in Figure3).Figure 3
Papers published by prolific authors.Finally, the research of knowledge graph applications in education mainly focuses on the following topics: using the knowledge graphs provided by CiteSpace to examine the hotspots (e.g., massive online open course, MOOC, and entrepreneurship education), research frontiers, research status, research topics, research trends, and development trends in the field of education, through visualized analysis, coword analysis, or cluster analysis. Table1 lists the high-frequency keywords.Table 1
High-frequency keywords.
KeywordFrequencyKnowledge graphs826Research hotspots199CiteSpace167Visualized analysis132Coword analysis63Literature metrology58Visualization56Cluster analysis35Bibliometrics32Hotspots31College students29Flipped classroom28Entrepreneurship education28Research frontiers27Scientific knowledge graph27Trends26Research status24Higher education23MOOC21Social network analysis21Research topics20Colleges20Research trends18Development trends17


### 2.2. Knowledge Graph Applications in Graduation Theses

The authors queried for the existing graduation theses on CNKI about knowledge graph applications in education (query date: June 7th, 2021), using the Full Text Database of China’s Excellent Master’s Theses and Full Text Database of China’s Doctoral Dissertations. A total of 96 samples were obtained after discrimination and screening, including 94 master’s theses and 2 doctoral dissertations. Figure4 shows the annual distribution of the graduation theses.Figure 4
Annual distribution of the number of graduation theses by masters and PhDs.As shown in Figure4, the earliest graduation thesis written by education masters/PhDs about the application of knowledge graphs was published in 2009. It is a master’s thesis authored by Qu Tianpeng at the Dalian University of Technology. The title reads Knowledge Graphs of the Distribution and Cooperative Network for Natural Science Disciplines in Colleges of Liaoning Province Based on SCI. Since then, the number of graduation theses has continued to increase. In the last two years, the annual number stabilized at about 20.The 96 graduation theses were written by masters and PhDs from 57 colleges around China. Minzu University of China and Central China Normal University contributed 6, respectively; Chongqing University and Northwest Normal University contributed 5, respectively; Sichuan Normal University contributed 4; Beijing University of Posts and Telecommunications and Shaanxi Normal University contributed 3, respectively; Bohai University, University of Electronic Science and Technology of China, Northeast Normal University, Harbin Institute of Technology, Henan University, Henan Normal University, Hunan Normal University, Tsinghua University, Shanghai Normal University, Wenzhou University, Xi’an University of Technology, Yunnan Normal University, Changsha University of Science and Technology, and Zhengzhou University contributed 2, respectively; every other college contributed only 1.On research topics, the collected samples mainly utilize software like CiteSpace for visualized analysis on knowledge graphs through coword analysis, citation analysis, and cooperative network, and discuss the research hotspots, frontiers, and progresses of the following fields: individualized learning, education technology, discipline construction, data structure, secondary school students, higher education, learning diagnosis, ontology, education economics, and MOOC.


### 2.3. Knowledge Graph Applications in Books

The authors queried with the formal title name of the knowledge graph in the Online Public Access Catalog (OPAC), National Library of China (query date: June 7th, 2021). A total of 336 relevant books were found. Through manual screening, 7 books were confirmed to be related to the field of education (Table 2).Table 2
Books on knowledge graph applications in education.
Lan Guoshuai. Development of International Education Technology Based on Knowledge Graphs (M). Beijing: China Science Publishing and Media Ltd. (CSPM), 2020Qi Zhanyong. Knowledge graphs of China’s educational policies: 1985–2015. (M) Beijing: CSPM, 2019Deng Guomin. Knowledge map of international educational technology Research; Theory, Technology and Practice. Shanghai: Fudan University Press, 2018Lu Dan. Knowledge graphs of China’s News communication education research: 2000–2014. Wuhan: Wuhan University Press, 2016Chen Yulin. Construction of visualized knowledge graphs for China’s education technology. Chengdu: Sichuan Fine Arts Publishing House, 2016Jiang Fei. Knowledge graphs of China’s curriculum and teaching theories in the New century. Wuhan: Central China Normal University Press, 2015Huang Wei. A study on the mapping knowledge domains of educational economics in China from 1980 to 2010 based on Multi-methods combination. Beijing: Economic Science Press, 2012The seven books cover multiple fields, namely, international education technology, China’s education technology, China’s education policies, China’s journalism and communication education, China’s curriculum and teaching theories, and China’s educational economics. Overall, there are too few books about knowledge graph applications in education.It is convenient to search for useful information online. But the search results are not always valuable. To quickly pinpoint the desired information, it is necessary to locate information according to user interests and build a user interest model. To a certain extent, the keyword-based data search and query meet the interests and needs of actual users. Therefore, user preference- or keyword-based data search and query could greatly contribute to the application of knowledge graphs in education, in addition to the above three types of literature.During the application of knowledge graphs in education, the purpose of forming knowledge graphs is to facilitate discovery, understanding, communication, and education and to visualize the education discipline. Knowledge graphs in education could provide a panorama of the booming education sector. Through the summary of knowledge graph applications in academic papers, graduation theses, and books, it is concluded that the application of knowledge graphs in China can be divided into an exploratory stage in 2020–2016 and a developmental stage from 2017 till now. In general, knowledge graphs in education are mostly applied to four aspects: intelligent search, in-depth questions and answers (Q&A), social networks, and recommendation systems. The knowledge graphs in education display the search results in the form of knowledge cards, answer user questions in natural languages, and connects people, locations, and things together to support intuitive and precise query. In addition, it is easy to recommend another entity closely related to the target entity with the aid of knowledge graphs.


## 2.1. Knowledge Graph Applications in Academic Papers

The authors queried for the existing papers on CNKI about knowledge graph applications in education (query date: June 7th, 2021) and obtained 1,073 records after discrimination and screening. Figure1 shows the annual distribution of literature quantity.Figure 1
Annual distribution of the quantity of academic papers.As shown in Figure1, Chinese education researchers first studied knowledge graph applications in 2007. The earliest published paper was authored by Peng et al. at the Dalian University of Technology, which is titled “Knowledge Graph Analysis on the Research State of International Entrepreneurship University” [16]. This is the earliest application of knowledge graphs in education. It is only two years later than the first domestic attempt to apply knowledge graphs [17].From the annual distribution of the quantity of academic papers, it can be seen the number of knowledge graph applications in education has been exploding since 2011, especially in the past three years. More than 200 academic papers were published in each of these three years.The top 10 high-yield institutions are listed as follows: Shaanxi Normal University published the most academic papers (56), Beijing Normal University (32), Nanjing Normal University (29), Central China Normal University (27), Wenzhou University (23), Henan University (21), Southwest University (20), Liaoning Normal University (19), East China Normal University (17), and Capital Normal University (16). The top-ranking institution published over three times more papers than the institution ranking in the 10th place (as shown in Figure2).Figure 2
Papers published by high-yield institutions.The top 9 prolific authors are listed as follows: Chen Yulin at Jiaying University (10), Cai Jiandong at Henan University (9), Cai Wenbo at Shihezi University (8), Chang Qinghui at Tiangong Technology (7), Li Yubin at Liaoning Normal University (6), Qi Zhanyong at Shaanxi Normal University (5), Yuan Liping at Shaanxi Normal University (5), Sun Furong at Wenzhou University (5), and Tang Jianmin at Zhejiang Shuren University (5) (as shown in Figure3).Figure 3
Papers published by prolific authors.Finally, the research of knowledge graph applications in education mainly focuses on the following topics: using the knowledge graphs provided by CiteSpace to examine the hotspots (e.g., massive online open course, MOOC, and entrepreneurship education), research frontiers, research status, research topics, research trends, and development trends in the field of education, through visualized analysis, coword analysis, or cluster analysis. Table1 lists the high-frequency keywords.Table 1
High-frequency keywords.
KeywordFrequencyKnowledge graphs826Research hotspots199CiteSpace167Visualized analysis132Coword analysis63Literature metrology58Visualization56Cluster analysis35Bibliometrics32Hotspots31College students29Flipped classroom28Entrepreneurship education28Research frontiers27Scientific knowledge graph27Trends26Research status24Higher education23MOOC21Social network analysis21Research topics20Colleges20Research trends18Development trends17


## 2.2. Knowledge Graph Applications in Graduation Theses

The authors queried for the existing graduation theses on CNKI about knowledge graph applications in education (query date: June 7th, 2021), using the Full Text Database of China’s Excellent Master’s Theses and Full Text Database of China’s Doctoral Dissertations. A total of 96 samples were obtained after discrimination and screening, including 94 master’s theses and 2 doctoral dissertations. Figure4 shows the annual distribution of the graduation theses.Figure 4
Annual distribution of the number of graduation theses by masters and PhDs.As shown in Figure4, the earliest graduation thesis written by education masters/PhDs about the application of knowledge graphs was published in 2009. It is a master’s thesis authored by Qu Tianpeng at the Dalian University of Technology. The title reads Knowledge Graphs of the Distribution and Cooperative Network for Natural Science Disciplines in Colleges of Liaoning Province Based on SCI. Since then, the number of graduation theses has continued to increase. In the last two years, the annual number stabilized at about 20.The 96 graduation theses were written by masters and PhDs from 57 colleges around China. Minzu University of China and Central China Normal University contributed 6, respectively; Chongqing University and Northwest Normal University contributed 5, respectively; Sichuan Normal University contributed 4; Beijing University of Posts and Telecommunications and Shaanxi Normal University contributed 3, respectively; Bohai University, University of Electronic Science and Technology of China, Northeast Normal University, Harbin Institute of Technology, Henan University, Henan Normal University, Hunan Normal University, Tsinghua University, Shanghai Normal University, Wenzhou University, Xi’an University of Technology, Yunnan Normal University, Changsha University of Science and Technology, and Zhengzhou University contributed 2, respectively; every other college contributed only 1.On research topics, the collected samples mainly utilize software like CiteSpace for visualized analysis on knowledge graphs through coword analysis, citation analysis, and cooperative network, and discuss the research hotspots, frontiers, and progresses of the following fields: individualized learning, education technology, discipline construction, data structure, secondary school students, higher education, learning diagnosis, ontology, education economics, and MOOC.


## 2.3. Knowledge Graph Applications in Books

The authors queried with the formal title name of the knowledge graph in the Online Public Access Catalog (OPAC), National Library of China (query date: June 7th, 2021). A total of 336 relevant books were found. Through manual screening, 7 books were confirmed to be related to the field of education (Table 2).Table 2
Books on knowledge graph applications in education.
Lan Guoshuai. Development of International Education Technology Based on Knowledge Graphs (M). Beijing: China Science Publishing and Media Ltd. (CSPM), 2020Qi Zhanyong. Knowledge graphs of China’s educational policies: 1985–2015. (M) Beijing: CSPM, 2019Deng Guomin. Knowledge map of international educational technology Research; Theory, Technology and Practice. Shanghai: Fudan University Press, 2018Lu Dan. Knowledge graphs of China’s News communication education research: 2000–2014. Wuhan: Wuhan University Press, 2016Chen Yulin. Construction of visualized knowledge graphs for China’s education technology. Chengdu: Sichuan Fine Arts Publishing House, 2016Jiang Fei. Knowledge graphs of China’s curriculum and teaching theories in the New century. Wuhan: Central China Normal University Press, 2015Huang Wei. A study on the mapping knowledge domains of educational economics in China from 1980 to 2010 based on Multi-methods combination. Beijing: Economic Science Press, 2012The seven books cover multiple fields, namely, international education technology, China’s education technology, China’s education policies, China’s journalism and communication education, China’s curriculum and teaching theories, and China’s educational economics. Overall, there are too few books about knowledge graph applications in education.It is convenient to search for useful information online. But the search results are not always valuable. To quickly pinpoint the desired information, it is necessary to locate information according to user interests and build a user interest model. To a certain extent, the keyword-based data search and query meet the interests and needs of actual users. Therefore, user preference- or keyword-based data search and query could greatly contribute to the application of knowledge graphs in education, in addition to the above three types of literature.During the application of knowledge graphs in education, the purpose of forming knowledge graphs is to facilitate discovery, understanding, communication, and education and to visualize the education discipline. Knowledge graphs in education could provide a panorama of the booming education sector. Through the summary of knowledge graph applications in academic papers, graduation theses, and books, it is concluded that the application of knowledge graphs in China can be divided into an exploratory stage in 2020–2016 and a developmental stage from 2017 till now. In general, knowledge graphs in education are mostly applied to four aspects: intelligent search, in-depth questions and answers (Q&A), social networks, and recommendation systems. The knowledge graphs in education display the search results in the form of knowledge cards, answer user questions in natural languages, and connects people, locations, and things together to support intuitive and precise query. In addition, it is easy to recommend another entity closely related to the target entity with the aid of knowledge graphs.


## 3. Normative Evaluation

The overview of development reveals the scale, prosperity, and evolution speed of knowledge graph applications in education in China. To understand the internal structure and normative level of these applications, this paper further examines the papers about knowledge graph applications in education, which were published in CSSCI-indexed journals in the past two years.


### 3.1. Perspectives

Research Institutions. The Chinese colleges offering education courses are either normal colleges or comprehensive colleges. In this paper, the research institutions are divided into two classes: (1) normal colleges; (2) comprehensive colleges. Any college with “normal university” in its name was categorized to class (1), and the other colleges were allocated to class (2).Data Sources. The data of the academic papers on knowledge graph applications are usually from standard paper databases. In this paper, these databases are categorized into two types: (1) foreign databases and (2) Chinese databases. The former mainly refers to the Web of Science of Information Sciences Institute (ISI), The ProQuest Dissertation and Theses Global (PQDT) database of graduation theses of doctors and masters, OADDS database of graduation theses, and EI. The latter mainly includes CNKI and CSSCI.Data Analysis Units. According to the purposes of most papers, this paper defines three data analysis units: (1) the title or keyword of the paper; (2) the authors or their institutions; (3) citations.Visualization Software. According to the status quo of domestic research, the visualization software fall into three classes: (1) CiteSpace, capable of reflecting the dynamic evolution process; (2) UCInet or Pajek, capable of presenting the internal structure; (3) SPSS or BICOMB, capable of drawing matrix graphs and multidimensional analysis graphs.Normative Requirements for Empirical Research Papers on Knowledge Graphs. Considering the research purpose, this paper evaluates the normative level of knowledge graph applications in education papers under the analysis framework proposed by Tang [6].


### 3.2. Data Collection

The authors queried for the papers on the application of knowledge graphs in education 2019–2020 on the website of CSSCI. The query was carried out in the following steps: input “knowledge graphs” into the field of “keywords”, “education” into the field of “discipline type” and “2019–2020” into the field of “years.” A total of 45 records was obtained. After reading each record, the authors found that 18 records are not about knowledge graph applications. Therefore, the remaining 27 papers were adopted for comprehensive analysis and evaluation (Table3).Table 3
Papers on knowledge graph applications in education 2019–2020 from CSSCI database.
Serial numberTitleJournalAuthorsR1Summary of the research on Educational Internationalization in Recent Decade: Quantitative Analysis of CNKI and WOS Journals, 2010–2019Comparative Education ReviewYi Hongjun; Zeng LinqinR2Progress and Hotspot Comparison of Entrepreneurship Curriculum Research in Colleges: Analysis of Knowledge Graphs Based on CNKI and WOSUniversity Education ScienceYuan Liping; Yang YangR3Research Methods and Topic Evolution of International Higher Education of EngineeringResearch in Higher Education of EngineeringHan Shuangmiao; Xie Jing; Xiao XiaofeiR4Characteristics and Trends of Global Interdisciplinary Education: Data Analysis Based on CitespaceResearch in Higher Education of EngineeringZhang Wei; Wei Lina; Qu ChenR5Historical Development and Contemporary Mission of Learning Sciences in Chinese Context: Analysis Based on Scientific Paradigm Formation and Knowledge Graph EvidenceJournal of East China Normal University (educational sciences)Zhao Jian; Yang XiaozheR6Hot Topics and Frontier Trends of International Hybrid Learning: Empirical Analysis Based on Scientific Knowledge GraphsJournal of East China Normal University (educational sciences)Ma Jing; zhou QianR7Academic Right of Speech, Hotspots, and Development Trends of International Teacher Assessment Research: Analysis Based on the Scientific Knowledge Graphs and Contents of SSCI-Index JournalsTeacher Education ResearchZhang Nan; Wang Guangming; Li YepingR8Hotspot Distribution and Development Trend of Multimedia Learning Research for Students with Hearing Disorder in China: Visualized Analysis Based on CNKI Literature 2010–2019Educational research and experimentRen QiangR9Retrospect and Prospect of China's Subjective Education Research in the 40 Years of Reform and Opening-Up: Analysis Based on Knowledge Graphs and Literature VisualizationEducational Research and ExperimentYue Wei; Xu YuanyuanR10Research Progress in China's Education Historiography in the 40 Years of Reform and Opening-Up: Knowledge Graph Analysis Based on CiteSpaceEducational Research and ExperimentLiu Laibing; Yang RongR11Diversification of Teacher Education: Review and Prospect after 40 Years of Reform and Opening-UpEducational Research and ExperimentWu Yuanyue; Yang YingyingR12Hotspots and Development Trends of International STEAM Teachers Research: Bibliometric Analysis Based on VOSviewerOpen Education ResearchZhang Nan; Song Naiqing; Huang Xin; Li YepingR13New Century Landscape of PISA Research in ChinaSocial Sciences of Inner Mongolia (Chinese)Tan Jisheng; zhu JingR14Hotspots of Quality Research on China’s Preschool Education Based on Scientific Knowledge GraphsSocial Sciences in NingxiaGuo Liping; Cai GuoyingR15Review of the Research Development for Graduate Education Economics in China: Visualized Analysis Based on CSSCI-Indexed Literature, 1998–2019Tsinghua Journal of EducationLi Fengliang; Wang YuqiR16Hot Topics and Evolution Trends of Chinese Education Research: Bibliometric and Knowledge Graph Analysis Based on CSSCI (1998–2017)Journal of Xiamen University (Arts and Social Sciences)Zhu Yu; Cai WuR17Topic Areas and Structural Evolution of International Core Competence ResearchJournal of Shandong Normal University (Humanities and Social Sciences)Bi Hualin; Ye JianqiangR18Dynamic Tracking of Frontier Research on Ideological and Political Education in China: Bibliometric Analysis of CSSCI-Indexed Papers from CNKI since 1998Studies in Ideological EducationZheng BinR19Key Authors of Chinese Education Research Papers in International English Journals and Their Coauthor NetworkModern University EducationLi Mei; Su ShuliR20Knowledge Graphs of Innovation and Entrepreneurship Education in China's Colleges in Recent Two DecadesModern University EducationWu Yiying; Yang DongR21Differences and Similarities between Domestic and Foreign Smart Campus Research in Research Hotspots and Development TrendsModern Educational TechnologyLi Yiyu; Chen JinhuaR22Research Hotspots and Education Application Trends of Deep Learning from the Perspective of Artificial Intelligence: Knowledge Graph Analysis of 20,708 Literature from WOS Database in 2006–2019Modern educational technologyZhang hai; cui Yulu; Yu luyao; Ji Mengxue; Wang YiningR23Research Landscape and Trends of EEG Technology Education: Knowledge Graph Analysis Based on International Literature 2000–2019Modern Distance Education ResearchZheng Xudong; Ma YunfeiR24Retrospect and Reflection of Rural Education Research in the 70 Years after the Founding of the New ChinaModern Distance Education ResearchOuyang xiujunR25Review of First-Class Discipline Research: Hotspots, Trends, and Outlook: Visualized Analysis Based on CNKI KeywordsAcademic Degrees and Graduate EducationLi Aibin; Du XiaohongR26Evolution and Trend of Smart Teaching Research in China in the Recent DecadeDistance Education in ChinaZhang Xiumei; Tian Tian; Tian Mengmeng; Gao Liizhi; Zhang XueboR27Hotspots and Future Trends in China's Educational Equality Research in the 21st century: Knowledge Graph Analysis Based on Co-Word MatrixDistance Education in ChinaDing Xueyang; Cheng Tianjun


### 3.3. Data Analysis

#### 3.3.1. Simple Classification and Analysis of Sample Structure

As shown in Table3, the 27 sample papers were published in the following journals: 4 on Educational Research and Experiment; 2 on Research in Higher Education of Engineering, Journal of East China Normal University (Educational Sciences), Modern University Education, Modern Educational Technology, Modern Distance Education, and Distance Education in China, respectively; 1 on Academic Degrees and Graduate Education, Comparative Education Review, University Education Science, Teacher Education Research, Open Education Research, Tsinghua Journal of Education, Studies in Ideological Education, Social Sciences of Inner Mongolia (Chinese), Social Sciences in Ningxia, Journal of Xiamen University (Arts and Social Sciences), and Journal of Shandong Normal University (Humanities and Social Sciences), respectively. 11 were published in 2020 and 16 in 2019.Following the classification criteria in the research design, the papers in Table3 were read through and classified (Table 4). The following conclusions can be drawn from the paper contents and the results in Table 4.(1)
60% of the authors are from normal colleges, and 40% are from comprehensive colleges. The normal colleges mainly include Central China Normal University, East China Normal University, Tianjin Normal University, Northeast Normal University, South China Normal University, etc., and the comprehensive colleges mainly include Zhejiang University and Southwest University.(2)
2/3 of the papers are indexed in Chinese databases, and 1/2 in foreign databases (some papers are indexed in both Chinese and foreign databases, such as R1 [18], R2 [19], R5 [20], and R21 [21]). The Chinese databases mainly refer to CNKI or CSSCI. 18 papers are indexed in ISI’s Web of Sciences (13) and Scopus (1).(3)
On data analysis units, 16, 26, 16, and 7 papers were involved by selecting annual distribution of literature quantity and journal, keywords, authors/institutions/regions, and citations. Hence, keyword analysis is the most important analysis perspective, followed by the annual distribution of literature quantity and journal. On average, each journal has 2.30 analysis perspectives. That is, there are more than two (two to four) perspectives to analyze these papers, namely, R7 [22], R22 [23], and R23 [24].(4)
On visualization software, 27 papers utilize CiteSpace, 4 utilize UCInet, 4 utilize SPSS/BICOMB, and 3 utilize VOSviewer. CiteSpace is obviously the most widely used software, taking up 81%. This reflects the immense popularity of CiteSpace among domestic researchers engaging in knowledge graph application in education.Table 4
Classification statistics of sample paper.
Serial numberInstituteDatabasePerspectiveVisualization softwareYear of publicationNormal collegeInstituteChineseForeignAnnual distribution of literature quantity and journalKeywordsAuthors/institutions/regionsCitationsCiteSpaceUnicent/UcinetSPSS/BicombVosviewerR1√√√√√√√2020R2√√√√√2020R3√√√√√√√√2020R4√√√√√√√2020R12√√√√√√√2020R14√√√√2020R19√√√√√√2020R21√√√2020R22√√√√√√√2020R23√√√√√√√2020R26√√√√√2020R5√√√√2019R6√√√√√√√2019R7√√√√√√√2019R8√√√√2019R9√√√√√√2019R10√√√√√2019R11√√√√√2019R13√√√√√2019R15√√√√√2019R16√√√√√√2019R17√√√√√√√2019R18√√√√√√√2019R20√√√√2019R24√√√√√2019R25√√√√2019R27√√√√√√2019Note. If a paper can be classified by more than one perspective, only the most important perspective is adopted for classification.


#### 3.3.2. Detailed Classification and Analysis of Sample Structure

To fully understand the research paradigm of the sample papers, this paper further classifies and discusses their structure and trends with an orthogonal view. In other words, the sample papers were observed from two or more angles at the same time. The combined angles include database and research perspective, database and visualization software, year of publication, and visualization software.(1)
From the angle of database and research perspective, citation analysis is not applicable to knowledge graph visualization of the literature exported directly from domestic databases because the literature thus obtained does not generally contain any reference. As shown in Table4, citation analysis has been adopted to visualize the knowledge graphs of the sample papers, all of which are exported from foreign databases.(2)
From the angle of database and visualization software, the papers indexed in domestic or foreign databases both utilize an average of 1.1 visualization software. Detailed analysis shows a certain difference in the use frequency of different software facing different databases. Among the three papers adopting VOSviewer, two utilize foreign databases. Among the four papers adopting SPSS/BICOMB, three utilize domestic databases (R1 utilizes both domestic and foreign databases simultaneously). Among the five papers without adopting CiteSpace, four utilize a domestic database. Among the four papers adopting UCInet, three utilize domestic databases. Relatively speaking, CiteSpace is often coupled with foreign databases, while other software like UCInet is often coupled with domestic databases.(3)
From the year of publication and visualization software, the number of CiteSpace-based knowledge graph analyses is even throughout the period (11 in 2019 vs. 11 in 2020); the papers using BICOMB for knowledge graph analysis were all published in 2019, so were those using UCInet. The papers using VOSviewer were all published in 2020. To a certain extent, the data reflect the preference for visualization software of researchers engaging in knowledge graph applications in education. Overall, CiteSpace and VOSviewer are the favorite choices of the researchers.


### 3.4. Normative Evaluation

Another important purpose of this paper is to evaluate the scientific and normative levels of papers. Table5 shows the normative evaluation criteria and questions of the sample papers. With these questions in mind, the researchers carefully read each paper and evaluated each paper against every question. The evaluation results are recorded in Table 6. Each positive answer is denoted as Y, each negative answer is denoted as N, and each ambiguous answer (the paper only partially conforms to the criterion) is denoted as C; the number of papers with Y, N, and C is denoted as EY, EN, and EC, respectively.Table 5
Evaluation criteria and questions.
Serial numberQuestionReasons for setting this criterionQ1Does the paper clearly specify the visualization software?There are various visualization software programs, each of which has its unique features.Q2Does the paper clearly specify the size and features of samples?The size and intrinsic structure of the samples directly determine the feasibility of knowledge graphs.Q3Does the paper clearly specify the thresholds?Different thresholds lead to different graphs.Q4Does the paper clearly specify the basis for determining the thresholds?Different thresholds lead to different graphs. This calls for scientific determination of thresholds.Q5Does the paper take the knowledge graphs as the main strategy?Knowledge graph provides only one research tool. Normally, it should be combined with other methods to draw comprehensive conclusions.Q6Does the paper clearly specify the limitations of conclusions?Knowledge graphs have many limitations, namely, the unstable and incomplete clustering and the conflict between the large data volume and the limited space.Data source: Tang [5].Table 6
Normative evaluation results.
Serial numberQ1Q2Q3Q4Q5Q61YYNNNN2YYNNNN3YYNNNN4YYNNNN5YYNNNN6YYNNNN7YYNNNN8YYNNNN9YYNNNN10YYNNNN11YYNNNN12YYNNNN13YYNNNN14YYNNNNEN202227272615YYYNNN16YYNNNN17YYNNNN18YYYNNN19YYNNNY20YYYNNN21NYNNNN22YYNNNN23YYYNNN24NYNNNN25YYYNNN26YYNNNN27YYNNNNEY25275001EC000000The following conclusions can be drawn from Table6.(1)
The predominant majority (25 out of 27) of the sample papers clearly specify the visualization software.(2)
Only five sample papers specify the threshold of each component of the similarity vector for knowledge graph plotting. 80% of the samples do not mention “threshold” during the preparation of knowledge graphs.(3)
None of the papers derive conclusions solely from the plotted knowledge graphs. Instead, knowledge graphs are combined with bibliometric methods, such as quantitative text analysis or qualitative word frequency statistics. Some papers cross-validate the knowledge graphs drawn by multiple software programs to ensure research accuracy (R3) [25] or verify the bibliometric results based on knowledge graphs through detailed qualitative tests (R5) [20]. Of course, the qualitative text analysis is not well integrated with the empirical and quantitative knowledge graphs.(4)
Only five papers (R15 [26], R18 [27], R20 [28], R23 [24], and R25 [29]) clearly specify the threshold values, yet without providing the basis. Compared with Tang’s work [6] in 2013, the lack of a detailed explanation of threshold setting is a long-lasting problem. The main reason for the problem is that most education scholars mainly engage in the research of liberal arts. They know how to apply knowledge graphs to their research domains but do not know the mechanism behind the application of the tool.(5)
All papers introduce the sample collection process and roughly report the internal features of the samples. This is a major progress compared with Tang’s work [6] in 2013. The researchers must have noticed the significant influence of internal features on the overall state and bibliometric results of the knowledge graphs.(6)
Only one paper (R19) [30] clearly specifies the limitations of conclusions. From the perspective of research normalization, this is not at all surprising. It is a must for a normalized research paper to summarize its limitations along with the conclusions. Most education experts only know how to use knowledge graphs in research. They are, after all, not designers of the bibliometric methods. It is impossible for them to make professional reflections on the research tool, not to mention providing improvement suggestions.


## 3.1. Perspectives

Research Institutions. The Chinese colleges offering education courses are either normal colleges or comprehensive colleges. In this paper, the research institutions are divided into two classes: (1) normal colleges; (2) comprehensive colleges. Any college with “normal university” in its name was categorized to class (1), and the other colleges were allocated to class (2).Data Sources. The data of the academic papers on knowledge graph applications are usually from standard paper databases. In this paper, these databases are categorized into two types: (1) foreign databases and (2) Chinese databases. The former mainly refers to the Web of Science of Information Sciences Institute (ISI), The ProQuest Dissertation and Theses Global (PQDT) database of graduation theses of doctors and masters, OADDS database of graduation theses, and EI. The latter mainly includes CNKI and CSSCI.Data Analysis Units. According to the purposes of most papers, this paper defines three data analysis units: (1) the title or keyword of the paper; (2) the authors or their institutions; (3) citations.Visualization Software. According to the status quo of domestic research, the visualization software fall into three classes: (1) CiteSpace, capable of reflecting the dynamic evolution process; (2) UCInet or Pajek, capable of presenting the internal structure; (3) SPSS or BICOMB, capable of drawing matrix graphs and multidimensional analysis graphs.Normative Requirements for Empirical Research Papers on Knowledge Graphs. Considering the research purpose, this paper evaluates the normative level of knowledge graph applications in education papers under the analysis framework proposed by Tang [6].


## 3.2. Data Collection

The authors queried for the papers on the application of knowledge graphs in education 2019–2020 on the website of CSSCI. The query was carried out in the following steps: input “knowledge graphs” into the field of “keywords”, “education” into the field of “discipline type” and “2019–2020” into the field of “years.” A total of 45 records was obtained. After reading each record, the authors found that 18 records are not about knowledge graph applications. Therefore, the remaining 27 papers were adopted for comprehensive analysis and evaluation (Table3).Table 3
Papers on knowledge graph applications in education 2019–2020 from CSSCI database.
Serial numberTitleJournalAuthorsR1Summary of the research on Educational Internationalization in Recent Decade: Quantitative Analysis of CNKI and WOS Journals, 2010–2019Comparative Education ReviewYi Hongjun; Zeng LinqinR2Progress and Hotspot Comparison of Entrepreneurship Curriculum Research in Colleges: Analysis of Knowledge Graphs Based on CNKI and WOSUniversity Education ScienceYuan Liping; Yang YangR3Research Methods and Topic Evolution of International Higher Education of EngineeringResearch in Higher Education of EngineeringHan Shuangmiao; Xie Jing; Xiao XiaofeiR4Characteristics and Trends of Global Interdisciplinary Education: Data Analysis Based on CitespaceResearch in Higher Education of EngineeringZhang Wei; Wei Lina; Qu ChenR5Historical Development and Contemporary Mission of Learning Sciences in Chinese Context: Analysis Based on Scientific Paradigm Formation and Knowledge Graph EvidenceJournal of East China Normal University (educational sciences)Zhao Jian; Yang XiaozheR6Hot Topics and Frontier Trends of International Hybrid Learning: Empirical Analysis Based on Scientific Knowledge GraphsJournal of East China Normal University (educational sciences)Ma Jing; zhou QianR7Academic Right of Speech, Hotspots, and Development Trends of International Teacher Assessment Research: Analysis Based on the Scientific Knowledge Graphs and Contents of SSCI-Index JournalsTeacher Education ResearchZhang Nan; Wang Guangming; Li YepingR8Hotspot Distribution and Development Trend of Multimedia Learning Research for Students with Hearing Disorder in China: Visualized Analysis Based on CNKI Literature 2010–2019Educational research and experimentRen QiangR9Retrospect and Prospect of China's Subjective Education Research in the 40 Years of Reform and Opening-Up: Analysis Based on Knowledge Graphs and Literature VisualizationEducational Research and ExperimentYue Wei; Xu YuanyuanR10Research Progress in China's Education Historiography in the 40 Years of Reform and Opening-Up: Knowledge Graph Analysis Based on CiteSpaceEducational Research and ExperimentLiu Laibing; Yang RongR11Diversification of Teacher Education: Review and Prospect after 40 Years of Reform and Opening-UpEducational Research and ExperimentWu Yuanyue; Yang YingyingR12Hotspots and Development Trends of International STEAM Teachers Research: Bibliometric Analysis Based on VOSviewerOpen Education ResearchZhang Nan; Song Naiqing; Huang Xin; Li YepingR13New Century Landscape of PISA Research in ChinaSocial Sciences of Inner Mongolia (Chinese)Tan Jisheng; zhu JingR14Hotspots of Quality Research on China’s Preschool Education Based on Scientific Knowledge GraphsSocial Sciences in NingxiaGuo Liping; Cai GuoyingR15Review of the Research Development for Graduate Education Economics in China: Visualized Analysis Based on CSSCI-Indexed Literature, 1998–2019Tsinghua Journal of EducationLi Fengliang; Wang YuqiR16Hot Topics and Evolution Trends of Chinese Education Research: Bibliometric and Knowledge Graph Analysis Based on CSSCI (1998–2017)Journal of Xiamen University (Arts and Social Sciences)Zhu Yu; Cai WuR17Topic Areas and Structural Evolution of International Core Competence ResearchJournal of Shandong Normal University (Humanities and Social Sciences)Bi Hualin; Ye JianqiangR18Dynamic Tracking of Frontier Research on Ideological and Political Education in China: Bibliometric Analysis of CSSCI-Indexed Papers from CNKI since 1998Studies in Ideological EducationZheng BinR19Key Authors of Chinese Education Research Papers in International English Journals and Their Coauthor NetworkModern University EducationLi Mei; Su ShuliR20Knowledge Graphs of Innovation and Entrepreneurship Education in China's Colleges in Recent Two DecadesModern University EducationWu Yiying; Yang DongR21Differences and Similarities between Domestic and Foreign Smart Campus Research in Research Hotspots and Development TrendsModern Educational TechnologyLi Yiyu; Chen JinhuaR22Research Hotspots and Education Application Trends of Deep Learning from the Perspective of Artificial Intelligence: Knowledge Graph Analysis of 20,708 Literature from WOS Database in 2006–2019Modern educational technologyZhang hai; cui Yulu; Yu luyao; Ji Mengxue; Wang YiningR23Research Landscape and Trends of EEG Technology Education: Knowledge Graph Analysis Based on International Literature 2000–2019Modern Distance Education ResearchZheng Xudong; Ma YunfeiR24Retrospect and Reflection of Rural Education Research in the 70 Years after the Founding of the New ChinaModern Distance Education ResearchOuyang xiujunR25Review of First-Class Discipline Research: Hotspots, Trends, and Outlook: Visualized Analysis Based on CNKI KeywordsAcademic Degrees and Graduate EducationLi Aibin; Du XiaohongR26Evolution and Trend of Smart Teaching Research in China in the Recent DecadeDistance Education in ChinaZhang Xiumei; Tian Tian; Tian Mengmeng; Gao Liizhi; Zhang XueboR27Hotspots and Future Trends in China's Educational Equality Research in the 21st century: Knowledge Graph Analysis Based on Co-Word MatrixDistance Education in ChinaDing Xueyang; Cheng Tianjun


## 3.3. Data Analysis

### 3.3.1. Simple Classification and Analysis of Sample Structure

As shown in Table3, the 27 sample papers were published in the following journals: 4 on Educational Research and Experiment; 2 on Research in Higher Education of Engineering, Journal of East China Normal University (Educational Sciences), Modern University Education, Modern Educational Technology, Modern Distance Education, and Distance Education in China, respectively; 1 on Academic Degrees and Graduate Education, Comparative Education Review, University Education Science, Teacher Education Research, Open Education Research, Tsinghua Journal of Education, Studies in Ideological Education, Social Sciences of Inner Mongolia (Chinese), Social Sciences in Ningxia, Journal of Xiamen University (Arts and Social Sciences), and Journal of Shandong Normal University (Humanities and Social Sciences), respectively. 11 were published in 2020 and 16 in 2019.Following the classification criteria in the research design, the papers in Table3 were read through and classified (Table 4). The following conclusions can be drawn from the paper contents and the results in Table 4.(1)
60% of the authors are from normal colleges, and 40% are from comprehensive colleges. The normal colleges mainly include Central China Normal University, East China Normal University, Tianjin Normal University, Northeast Normal University, South China Normal University, etc., and the comprehensive colleges mainly include Zhejiang University and Southwest University.(2)
2/3 of the papers are indexed in Chinese databases, and 1/2 in foreign databases (some papers are indexed in both Chinese and foreign databases, such as R1 [18], R2 [19], R5 [20], and R21 [21]). The Chinese databases mainly refer to CNKI or CSSCI. 18 papers are indexed in ISI’s Web of Sciences (13) and Scopus (1).(3)
On data analysis units, 16, 26, 16, and 7 papers were involved by selecting annual distribution of literature quantity and journal, keywords, authors/institutions/regions, and citations. Hence, keyword analysis is the most important analysis perspective, followed by the annual distribution of literature quantity and journal. On average, each journal has 2.30 analysis perspectives. That is, there are more than two (two to four) perspectives to analyze these papers, namely, R7 [22], R22 [23], and R23 [24].(4)
On visualization software, 27 papers utilize CiteSpace, 4 utilize UCInet, 4 utilize SPSS/BICOMB, and 3 utilize VOSviewer. CiteSpace is obviously the most widely used software, taking up 81%. This reflects the immense popularity of CiteSpace among domestic researchers engaging in knowledge graph application in education.Table 4
Classification statistics of sample paper.
Serial numberInstituteDatabasePerspectiveVisualization softwareYear of publicationNormal collegeInstituteChineseForeignAnnual distribution of literature quantity and journalKeywordsAuthors/institutions/regionsCitationsCiteSpaceUnicent/UcinetSPSS/BicombVosviewerR1√√√√√√√2020R2√√√√√2020R3√√√√√√√√2020R4√√√√√√√2020R12√√√√√√√2020R14√√√√2020R19√√√√√√2020R21√√√2020R22√√√√√√√2020R23√√√√√√√2020R26√√√√√2020R5√√√√2019R6√√√√√√√2019R7√√√√√√√2019R8√√√√2019R9√√√√√√2019R10√√√√√2019R11√√√√√2019R13√√√√√2019R15√√√√√2019R16√√√√√√2019R17√√√√√√√2019R18√√√√√√√2019R20√√√√2019R24√√√√√2019R25√√√√2019R27√√√√√√2019Note. If a paper can be classified by more than one perspective, only the most important perspective is adopted for classification.


### 3.3.2. Detailed Classification and Analysis of Sample Structure

To fully understand the research paradigm of the sample papers, this paper further classifies and discusses their structure and trends with an orthogonal view. In other words, the sample papers were observed from two or more angles at the same time. The combined angles include database and research perspective, database and visualization software, year of publication, and visualization software.(1)
From the angle of database and research perspective, citation analysis is not applicable to knowledge graph visualization of the literature exported directly from domestic databases because the literature thus obtained does not generally contain any reference. As shown in Table4, citation analysis has been adopted to visualize the knowledge graphs of the sample papers, all of which are exported from foreign databases.(2)
From the angle of database and visualization software, the papers indexed in domestic or foreign databases both utilize an average of 1.1 visualization software. Detailed analysis shows a certain difference in the use frequency of different software facing different databases. Among the three papers adopting VOSviewer, two utilize foreign databases. Among the four papers adopting SPSS/BICOMB, three utilize domestic databases (R1 utilizes both domestic and foreign databases simultaneously). Among the five papers without adopting CiteSpace, four utilize a domestic database. Among the four papers adopting UCInet, three utilize domestic databases. Relatively speaking, CiteSpace is often coupled with foreign databases, while other software like UCInet is often coupled with domestic databases.(3)
From the year of publication and visualization software, the number of CiteSpace-based knowledge graph analyses is even throughout the period (11 in 2019 vs. 11 in 2020); the papers using BICOMB for knowledge graph analysis were all published in 2019, so were those using UCInet. The papers using VOSviewer were all published in 2020. To a certain extent, the data reflect the preference for visualization software of researchers engaging in knowledge graph applications in education. Overall, CiteSpace and VOSviewer are the favorite choices of the researchers.


## 3.3.1. Simple Classification and Analysis of Sample Structure

As shown in Table3, the 27 sample papers were published in the following journals: 4 on Educational Research and Experiment; 2 on Research in Higher Education of Engineering, Journal of East China Normal University (Educational Sciences), Modern University Education, Modern Educational Technology, Modern Distance Education, and Distance Education in China, respectively; 1 on Academic Degrees and Graduate Education, Comparative Education Review, University Education Science, Teacher Education Research, Open Education Research, Tsinghua Journal of Education, Studies in Ideological Education, Social Sciences of Inner Mongolia (Chinese), Social Sciences in Ningxia, Journal of Xiamen University (Arts and Social Sciences), and Journal of Shandong Normal University (Humanities and Social Sciences), respectively. 11 were published in 2020 and 16 in 2019.Following the classification criteria in the research design, the papers in Table3 were read through and classified (Table 4). The following conclusions can be drawn from the paper contents and the results in Table 4.(1)
60% of the authors are from normal colleges, and 40% are from comprehensive colleges. The normal colleges mainly include Central China Normal University, East China Normal University, Tianjin Normal University, Northeast Normal University, South China Normal University, etc., and the comprehensive colleges mainly include Zhejiang University and Southwest University.(2)
2/3 of the papers are indexed in Chinese databases, and 1/2 in foreign databases (some papers are indexed in both Chinese and foreign databases, such as R1 [18], R2 [19], R5 [20], and R21 [21]). The Chinese databases mainly refer to CNKI or CSSCI. 18 papers are indexed in ISI’s Web of Sciences (13) and Scopus (1).(3)
On data analysis units, 16, 26, 16, and 7 papers were involved by selecting annual distribution of literature quantity and journal, keywords, authors/institutions/regions, and citations. Hence, keyword analysis is the most important analysis perspective, followed by the annual distribution of literature quantity and journal. On average, each journal has 2.30 analysis perspectives. That is, there are more than two (two to four) perspectives to analyze these papers, namely, R7 [22], R22 [23], and R23 [24].(4)
On visualization software, 27 papers utilize CiteSpace, 4 utilize UCInet, 4 utilize SPSS/BICOMB, and 3 utilize VOSviewer. CiteSpace is obviously the most widely used software, taking up 81%. This reflects the immense popularity of CiteSpace among domestic researchers engaging in knowledge graph application in education.Table 4
Classification statistics of sample paper.
Serial numberInstituteDatabasePerspectiveVisualization softwareYear of publicationNormal collegeInstituteChineseForeignAnnual distribution of literature quantity and journalKeywordsAuthors/institutions/regionsCitationsCiteSpaceUnicent/UcinetSPSS/BicombVosviewerR1√√√√√√√2020R2√√√√√2020R3√√√√√√√√2020R4√√√√√√√2020R12√√√√√√√2020R14√√√√2020R19√√√√√√2020R21√√√2020R22√√√√√√√2020R23√√√√√√√2020R26√√√√√2020R5√√√√2019R6√√√√√√√2019R7√√√√√√√2019R8√√√√2019R9√√√√√√2019R10√√√√√2019R11√√√√√2019R13√√√√√2019R15√√√√√2019R16√√√√√√2019R17√√√√√√√2019R18√√√√√√√2019R20√√√√2019R24√√√√√2019R25√√√√2019R27√√√√√√2019Note. If a paper can be classified by more than one perspective, only the most important perspective is adopted for classification.


## 3.3.2. Detailed Classification and Analysis of Sample Structure

To fully understand the research paradigm of the sample papers, this paper further classifies and discusses their structure and trends with an orthogonal view. In other words, the sample papers were observed from two or more angles at the same time. The combined angles include database and research perspective, database and visualization software, year of publication, and visualization software.(1)
From the angle of database and research perspective, citation analysis is not applicable to knowledge graph visualization of the literature exported directly from domestic databases because the literature thus obtained does not generally contain any reference. As shown in Table4, citation analysis has been adopted to visualize the knowledge graphs of the sample papers, all of which are exported from foreign databases.(2)
From the angle of database and visualization software, the papers indexed in domestic or foreign databases both utilize an average of 1.1 visualization software. Detailed analysis shows a certain difference in the use frequency of different software facing different databases. Among the three papers adopting VOSviewer, two utilize foreign databases. Among the four papers adopting SPSS/BICOMB, three utilize domestic databases (R1 utilizes both domestic and foreign databases simultaneously). Among the five papers without adopting CiteSpace, four utilize a domestic database. Among the four papers adopting UCInet, three utilize domestic databases. Relatively speaking, CiteSpace is often coupled with foreign databases, while other software like UCInet is often coupled with domestic databases.(3)
From the year of publication and visualization software, the number of CiteSpace-based knowledge graph analyses is even throughout the period (11 in 2019 vs. 11 in 2020); the papers using BICOMB for knowledge graph analysis were all published in 2019, so were those using UCInet. The papers using VOSviewer were all published in 2020. To a certain extent, the data reflect the preference for visualization software of researchers engaging in knowledge graph applications in education. Overall, CiteSpace and VOSviewer are the favorite choices of the researchers.


## 3.4. Normative Evaluation

Another important purpose of this paper is to evaluate the scientific and normative levels of papers. Table5 shows the normative evaluation criteria and questions of the sample papers. With these questions in mind, the researchers carefully read each paper and evaluated each paper against every question. The evaluation results are recorded in Table 6. Each positive answer is denoted as Y, each negative answer is denoted as N, and each ambiguous answer (the paper only partially conforms to the criterion) is denoted as C; the number of papers with Y, N, and C is denoted as EY, EN, and EC, respectively.Table 5
Evaluation criteria and questions.
Serial numberQuestionReasons for setting this criterionQ1Does the paper clearly specify the visualization software?There are various visualization software programs, each of which has its unique features.Q2Does the paper clearly specify the size and features of samples?The size and intrinsic structure of the samples directly determine the feasibility of knowledge graphs.Q3Does the paper clearly specify the thresholds?Different thresholds lead to different graphs.Q4Does the paper clearly specify the basis for determining the thresholds?Different thresholds lead to different graphs. This calls for scientific determination of thresholds.Q5Does the paper take the knowledge graphs as the main strategy?Knowledge graph provides only one research tool. Normally, it should be combined with other methods to draw comprehensive conclusions.Q6Does the paper clearly specify the limitations of conclusions?Knowledge graphs have many limitations, namely, the unstable and incomplete clustering and the conflict between the large data volume and the limited space.Data source: Tang [5].Table 6
Normative evaluation results.
Serial numberQ1Q2Q3Q4Q5Q61YYNNNN2YYNNNN3YYNNNN4YYNNNN5YYNNNN6YYNNNN7YYNNNN8YYNNNN9YYNNNN10YYNNNN11YYNNNN12YYNNNN13YYNNNN14YYNNNNEN202227272615YYYNNN16YYNNNN17YYNNNN18YYYNNN19YYNNNY20YYYNNN21NYNNNN22YYNNNN23YYYNNN24NYNNNN25YYYNNN26YYNNNN27YYNNNNEY25275001EC000000The following conclusions can be drawn from Table6.(1)
The predominant majority (25 out of 27) of the sample papers clearly specify the visualization software.(2)
Only five sample papers specify the threshold of each component of the similarity vector for knowledge graph plotting. 80% of the samples do not mention “threshold” during the preparation of knowledge graphs.(3)
None of the papers derive conclusions solely from the plotted knowledge graphs. Instead, knowledge graphs are combined with bibliometric methods, such as quantitative text analysis or qualitative word frequency statistics. Some papers cross-validate the knowledge graphs drawn by multiple software programs to ensure research accuracy (R3) [25] or verify the bibliometric results based on knowledge graphs through detailed qualitative tests (R5) [20]. Of course, the qualitative text analysis is not well integrated with the empirical and quantitative knowledge graphs.(4)
Only five papers (R15 [26], R18 [27], R20 [28], R23 [24], and R25 [29]) clearly specify the threshold values, yet without providing the basis. Compared with Tang’s work [6] in 2013, the lack of a detailed explanation of threshold setting is a long-lasting problem. The main reason for the problem is that most education scholars mainly engage in the research of liberal arts. They know how to apply knowledge graphs to their research domains but do not know the mechanism behind the application of the tool.(5)
All papers introduce the sample collection process and roughly report the internal features of the samples. This is a major progress compared with Tang’s work [6] in 2013. The researchers must have noticed the significant influence of internal features on the overall state and bibliometric results of the knowledge graphs.(6)
Only one paper (R19) [30] clearly specifies the limitations of conclusions. From the perspective of research normalization, this is not at all surprising. It is a must for a normalized research paper to summarize its limitations along with the conclusions. Most education experts only know how to use knowledge graphs in research. They are, after all, not designers of the bibliometric methods. It is impossible for them to make professional reflections on the research tool, not to mention providing improvement suggestions.


## 4. Conclusions

Fifteen years has passed since knowledge graphs were introduced to the field of education in 2007. During these years, the research tool has been applied to increasingly extensive domains. After reviewing the development and analyzing the current state, the sample papers were classified and normatively evaluated. Based on the results, the following conclusions can be drawn:(1)
With the elapse of time, knowledge graphs are being increasingly applied to education, indicating the applicability of the tool to education.(2)
The research paradigm has already taken shape. By carefully deconstructing the research samples, this paper finds that academia has reached a consensus on the paradigm of the research tool: examining the hotspots, topics, and trends in the related fields from the angles of keyword cooccurrence network (KCN), time zone map, clustering network, and literature/author cocitation, with the aid of CiteSpace and other visualization software and text analysis.(3)
The research quality is yet to be improved. Firstly, there are relatively few high-quality papers, as evidenced by the papers indexed in the CSSCI database and master/PhD’s graduation thesis databases and the published books on the relevant field. Second, the existing studies are defected in scientific and normalized levels. According to our normative evaluation, most papers ignore the importance of threshold settings to the plotting of knowledge graphs and do not have a widely recognized and feasible standard for threshold setting. Besides, few papers reflect on the limitations of research.In the future, education researchers should try to master the principle of knowledge graphs and carry out refined research on the application of this tool. The improvement of knowledge graph applications will surely promote the research level in the field of education.



---

*Source: 6304109-2021-09-28.xml*