TECHKG: A LARGE-SCALE CHINESE TECHNOLOGY- ORIENTED KNOWLEDGE GRAPH
Year: 2019

Abstract:
Knowledge graph is a kind of valuable knowledge base which would benefit lots of AI-related applications. Up to now, lots of large-scale knowledge graphs have been built. However, most of them are non-Chinese and designed for general purpose. In this work, we introduce TechKG, a large scale Chinese knowledge graph that is technology-oriented. It is built automatically from massive technical papers that are published in Chinese academic journals of different research domains. Some carefully designed heuristic rules are used to extract high quality entities and relations. Totally, it comprises of over 260 million triplets that are built upon more than 52 million entities which come from 38 research domains. Our preliminary experiments indicate that TechKG has high adaptability and can be used as a dataset for many diverse AI-related applications.

heading: 1 INTRODUCTION
text: Generally, a knowledge graph (KG) stores factual knowledge in the form of structural triplet like <h, r, t>, in which h and t are entities (called head entity and tail entity respectively) and r denotes the relation type from h to t. KG is a kind of valuable knowledge base and is important for many different kinds of AI-related applications. Nowadays, great achievements have been made in building large scale KGs. And there are many available KGs like FreeBase Bollacker et al. (2008), WordNet Miller (1995), YAGO Suchanek et al. (2007), etc. Usually there are millions of entities and billions of relational facts in these KGs. With them, great progress have been made in lots of AI-related applications, such as knowledge graph embedding (KGE), distantly supervised relation extraction (RE), knowledge based question and answering (KBQA), etc. However, there are still following two challenges inherited in these KG-dependent applications.
First, almost all of these KGs can only be used in English-related applications. But the fact is that each language has its own characteristics. It is uncertain whether a model that performs well in an English-related KG-dependent task could still perform well in the corresponding task of other language. Thus there is a pressing need to build KGs of different languages other than English.
Second, most of existing KGs are designed for general purpose. It is also uncertain whether a model that performs well in an application of general domain could still perform well in applications of specific domains, such as in the technical domains liking medicine, biology, electronics, etc. It is necessary to evaluate whether there are some new kinds of characteristics hidden in KGs that haven’t been noticed by researchers up to now.
To address these issues, this work introduces TechKG, a large scale Chinese KG built from massive technical papers that are published in Chinese academic journals of different research domains. Compared with existing KGs, TechKG has following important characteristics.
First, TechKG is a large scale Chinese KG that is technology-oriented. Totally, it comprises of over 280 million triplets that are built upon more than 60 million entities which come from more than 30 research domains. To the best of our knowledge, this is the first Chinese KG that is built by using multi-domain technical papers.
Second, TechKG is a high quality KG. Both of its entities and relations are acquired in two ways. One is directly taking some components of a paper as entities and building relations between entities by simply re-organizing a paper’s components. The other is taking carefully selected domain
terminologies as entities and building relations between them by some carefully designed heuristic rules. These two ways guarantee TechKG have high accuracies at both entity and relation levels.
Third, TechKG can provide more kinds of information associated with the original papers besides common triplets. This makes it can be easily taken as a benchmark dataset for many diverse AIrelated applications, such as KGE, distantly supervised RE, NER, KBQA, text classification(TC), machine translation(MT), among others.
Fourth, TechKG can be continually updated as long as there are new available published papers. As the needed data source is readily available in almost all languages, researchers can easily build similar KGs for other languages.
In the rest of this paper, we first introduce the building process of TechKG. Then we analyze the characteristics of TechKG in detail. We also conduct some primary experiments using TechKG as a dataset to three diverse tasks (see the Appendix part for detail). Both our analyses and experimental results show that lots of characteractics distinguish TechKG from existing KGs and it raises many new challenges for lots of existing applications.

heading: 2 RELATED WORK
text: At present, there are many available KGs, such as SUMONiles and Pease (2001), Cyc Matuszek et al. (2006), FreeBase Bollacker et al. (2008), WordNet Miller (1995), YAGO Suchanek et al. (2007), YAGO2 Hoffart et al. (2013), YAGO3 Mahdisoltani et al. (2015), etc. Among them, FreeBase, WordNet, YAGO and some of their subsets are the most widely used KGs in many diverse applications. We will introduce them in the following.
FreeBase Bollacker et al. (2008) is a database system designed to be a public repository of the word’s knowledge. Its design is inspired by Semantic Web and Wikipedia. It tries to merge the scalability of structured databases with the diversity of collaborative wikis into a practical and scalable database of structured general human knowledge. Totally, it contains more than 100 million asserts about topics spread over 4000 types, including people, media, locations, and many others.
FB15k Bordes et al. (2013) is a subset of Freebase. It is one of the most widely used datasets for many AI-related tasks. Totally, it contains 14,951 entities, 592,213 triplets, and 1,345 different relations. A large fraction of the triplets in this KG describe facts about movies, actors, awards, etc.
WordNetMiller (1995) is an electronic lexical database that is considered to be the most important resource available to researchers in computational linguistics, text analysis, and many related areas. Its design is inspired by psycholinguistic and computational theories of human lexical memory. In WordNet, English nouns, verbs, adjectives, and adverbs are organized into synonym sets, each represents one underlying lexicalized concept, and semantic relations link the synonym sets. Specifically, WordNet includes some kinds of semantic relations like synonymy, antonymy, hyponymy, memonymy, troponymy, entailment, etc.
WN18 Bordes et al. (2013) is a subset of WordNet. Like FB15k, it is widely used in many AI-related tasks. Totally, it consists of 18 kinds of relations, 40,943 entities, and 151,442 triplets. Most of its triplets consist of hyponym and hypernym relations.
YAGOSuchanek et al. (2007) is a light-weight and extensible ontology with high coverage and quality. It contains more than 1 million entities and 5 million facts. These facts are expressed by relations and are automatically extracted from Wikipedia and unified with WordNet by a carefully designed combination of rule-based and heuristic methods. In YAGO, the relations include the Is-A hierarchy as well as non-taxonomic relations like HasWonPrize, BornInYear, DiedInYear, etc.
YAGO2 Hoffart et al. (2013) is an extension of YAGO knowledge base, in which entities, facts, and events are anchored in both time and space. YAGO2 is built automatically from Wikipedia, GeoNames, and WordNet. It contains 447 million facts and about 9.8 million entities.
YAGO3 Mahdisoltani et al. (2015) is another extension of the YAGO knowledge base. It combines the information from the Wikipedia in 10 languages including English, German, French, Dutch, Italian, Spanish, Romanian, Polish, Arabic, and Farsi. The choice of languages was determined by the proficiency of the authors, so as to facilitate manual evaluation. Totally, there are 8,936,324 triplet facts built upon 4,595,906 entities.
YAGO3-10 Mahdisoltani et al. (2015) is a subset of YAGO3 and is also widely used in many applications. It consists of entities which have a minimum of 10 relations each. Totally, it has 123,182 entities and 37 kinds of relations. Most of the triplets deal with descriptive attributes of people, such as citizenship, gender,profession, etc.
NOTE that here we copy these KGs’ statistics directly from their corresponding reference papers. There may be some differences for these statistics because almost all of these KGs are continually updating.

heading: 3 TECHKG BUILDING
text: Every day, there are massive Chinese technical papers published in journals of different research domains. These papers often have following characteristics.
First, there is a clear structure for each of these papers. For example, each paper often has such components like: a title, an author list, an authors’ affiliation list, keyword list, abstract, etc.
Second, each technical paper can be accurately classified into a proper research domain according to the journal where this paper is published. This is because that each academic journal has its own scope for submissions. Taking “Chinese Journal of Computers” for example, it only focuses on the latest research in computer science domain. And it would never accept a paper in chemistry or artist domain. In other word, as long as we know the focused research domain of a journal, we can confidently classify all of its accepted papers into this research domain. There may be some exceptions, but the proportion would be very small.
These characteristics inspire us that we can use technical papers as data source to build a large scale Chinese KG that is technology-oriented.

heading: 3.1 PAPER COLLECTION
text: We take technical papers as data source to build TechKG. These papers were mainly collected from journals’ websites. According to the submission scopes of the journals1, we classify all of them into 38 types: each of which corresponds to a specific research domain.
For simplicity2, we omit the body part of a paper and only collect its: title, author list, authors’ affiliation list, keyword list, and abstract. If a paper’s these components also have English translations, the corresponding English parts will also be collected.
Totally, over 70 million technical papers are collected.

heading: 3.2 ENTITY SELECTION
text: In TechKG, each component of a paper except abstract is taken as an entity. That is to say, title, author, keyword, and affiliation, either in Chinese or English, are all taken as entities. And the name of each research domain is also taken as an entity.
Among the obtained entities, keywords account for a very large proportion. We had expected that all of them were high quality domain terminologies, but there are lots of exceptions. For example, in some papers of the computer science domain, there are keywords like “three phases”, “two steps”, “generation”, etc. Obviously, these keywords are not good domain terminologies for they have less to do with computer science. So we use atf*idf method to filter such kind of keywords. Specifically, we concatenate all the keyword lists in each research domain to form a long domain-related document. Then the tf value of a keyword in each domain can be computed. Different domain-related documents are viewed as the counterpart of each other, thus the idf value of a keyword can be computed accordingly. The higher a keyword’s tf *idf value, the more possible this keyword is a good domain terminology.
1Here we also use CNKI’s(http://www.cnki.net/) categories for reference, but we merge some similar research domains in CNKI and reduce the amount of research domains.
2It is difficult to effectively analyse the body part of a paper.

heading: 3.3 RELATION EXTRACTION
text: The role of relations is to describe the attributes of entities and the relationships between entities. In TechKG, keywords and authors are two kinds of entities with the largest amounts. So we define relations mainly centered upon them. Besides, papers themselves contain much valuable factual knowledge. So we define some paper-centered relations to describe the attributes of papers themselves. Totally, 16 relations are defined in TechKG.
Author-centered relations include: author of, first author, second author, co author, research interest, and affiliation.
Keyword-centered relations include: belonged domain, co-occurrence with, and hierarchical.
Paper-centered relations include: published year, contained chn keywords, contained eng keywords, other author, all authors, and published journal.
Many technical papers also have English translations for their titles, affiliations, keywords, etc., so we define a new relation named translation of.
Among these defined relations, co author, translation of and co occurrence are a kind of symmetrical relations and their directions are not taken into consideration.
In TechKG, a set of heuristic rules are employed to assign relations between entities.
First, for each keyword ki, if it occurs in the keyword list of a paper and this paper is published in a journal that has been categorized into a research domain dj, we assign a “belonged domain” relation from ki to dj.
Second, we notice that most Chinese researchers have such a writing habit: in a keyword list, they are more likely to place high-level (more abstract) keywords ahead of low-level (more concrete) keywords. For example, we are more likely to see such a keyword list “natural language processing, neural machine translation, attention mechanism” than a list of “neural machine translation, natural language processing, attention mechanism”. In other word, there is a natural hierarchical structure inherent in the keyword list of a paper. Based on this finding, we design a simple heuristic rule to mine the “hierarchical” relations between keywords. Specially, for each keyword pair “k1-k2”, we count the number of papers that:1) both k1 and k2 occur in their keyword lists; and 2) k1 occurs ahead of k2 in their keyword lists. If this number is larger than a predefined threshold, we assign a “hierarchical” relation from k1 to k2.
Third, for any two keywords k1 and k2, if they co-occurrence in a paper’s keyword list, we assign a “co-occurrence with” relation between them. We also record the number of papers that k1 and k2 co-occurrence. This number is taken as a confidence factor to evaluate the reliability of this “co-occurrence with” relation between k1 and k2.
Fourth, if there is an English part for a paper’s title, authors, or affiliations, we assign a “translation of ” relation to the corresponding bilingual entity pairs. It should be noted that there may be multi authors or affiliations in a paper. In such case, we will assign the “translation of ” relation to two entities according to their occurrence orders in the corresponding bilingual parts. Taking authors for example, a “translation of ” relation will be assigned to the first Chinese author and the first English author; to the second Chinese author and the second English author, and so on.
Fifth, if there are bilingual keyword lists in a paper, we first construct candidate entity pairs according to two entities’ occurrence orders in their corresponding bilingual parts. For example, a possible entity pair “ci-ej” may be: the first Chinese keyword with the first English keyword, the second Chinese keyword with the second English keyword, and so on. Then for each candidate entity pair “ci-ej”, we count the number of papers that can generate it. If this number is larger than a predefined threshold, we assign a “translation of ” relation between ci and ej. Here we take different extraction rules for the “translation of ” relation due to the reason that the numbers of keywords in Chinese keyword list and English keyword list are not always the same.
For other kinds of relations, we can precisely obtain them by re-organizing the meta-items of the original papers.

heading: 3.4 BUILDING RESULTS
text: Finally, more than 52 million entities and more than 260 million triplets are extracted. Some basic statistics of TechKG are shown in Table 1. It should be noted that entities maybe overlapped in different kinds of relations. So in Table 1, the total entity number is less than the sum number of entities in all relations. Statistics of the “hierarchical” relation are not listed in Table 1 because they depend on the predefined threshold. One can see the detailed statistics about each relation (including the “hierarchical” relation) in the Appendix part.
Currently, we don’t mine complex relations for TechKG because we put more emphasis on its accuracy. We can see that most of the triplets in TechKG are generated by re-organizing the meta-items of the original papers. Thus, TechKG is a large-scale KG that has high accuracy for both its entities and relations. Besides, the tf*idf method in entity selection and the threshold method in relation extraction can further guarantee TechKG’s accuracy.

heading: 4 TECHKG CHARACTERISTICS
text: We find there are three main characteristics that distinguish TechKG from most of existing KGs: the duplicate name issue, the imbalance issue, and the good application adaptability.

heading: 4.1 DUPLICATE NAME ISSUE
text: In TechKG, duplicate name is a very common phenomenon: not only some authors share some common names, but also some research domains also share some common terminologies. For example, Ning Li, a researcher name that appears in 34 research domains. That is to say, there is a researcher named Ning Li in almost all of research domains. We call this kind of duplicate names as cross-domain duplicate name issue.
In Table 2, we report some statistics for the duplicate name issue of authors. Here dpa means the average appeared domains per duplicated author. It should be noted that here we ignore nature science and social science during making statistics. Both of them are comprehensive research domains and it is very likely to generate some false duplicated author names if we take them into consideration. For example, a researcher may publish his paper in both computer science domain and in nature science domain.
However, when we make statistics according to research domain, we find both the duplicate ratio and the dpa increase greatly (see the Appendix part for detail). These statistics mean that there are a very small number of duplicated names that appear in many different research domains. Or most of
Table 2: Cross-domain duplicate name statistics for authors. total # duplicate # duplicate ratio dpa
1,826,217 654,884 35.86% 3.97
the duplicated names appear in a relative smaller research domain set. To verify this hypothesis, we further make statistics of the authors’ domain distribution. The results are shown in Table 3.
From Table 3 we can see that more than 80% of the duplicated authors appear in less than 5 research domains. There are 7,769 duplicated authors that appear more than 20 research domain. More extremely, we find that there are 167 author names (such as: Lin Zhang, Hua Liu, Yang Wang, Lei Zhang, etc) that appear in all research domains. These results well verify our hypothesis and can explain the reason of why both the duplicate ratio and the dpa increase greatly in separated research domains.
There is not only cross-domain duplicate name issue, but also another kind of in-domain duplicate name issue. For example, in the research domain of computer science, there will be many duplicated authors that are in different affiliations. Among the in-domain authors that have several affiliations, some of them are real duplicated names, some are not. Although the entity linkage (or entity resolution) issue has been well studied, it is so hard that it is almost impossible for us to accurately pick up the true in-domain duplicate names due to the following two reasons.
First, there may be some researchers that work in different affiliations in different times.
Second, some authors provide their affiliations with different granularity in different papers. For example, an author may write his affiliation as “Northeastern University”, “School of Computer Science and Engineering, Northeastern University”, or even “Natural Language Processing Lab, Northeastern University”, etc.
Here we try to provide a rough extreme analysis of this in-domain duplicate name issue.
For the worse case, as long as an author name appears in different affiliations, we take it as a duplicated name.
For the better case, for an author name appears in different affiliations, we take it as a duplicated name only when there is no inclusion relationship between these affiliations.
In Table 4, we report the in-domain duplicate name statistics for metallurgical industry, computer science, electric industry, and traffic transportation, 4 randomly selected research domains. Here apa means the average appeared affiliations per duplicated author.
From Table 4 we can see that the in-domain duplicate name issue is very serious. In both cases, more than 50% authors will have multi affiliations. An interesting finding is that the duplicate ratio of the better case is lower than the worse case, but the apa value of the better case in larger than the worse case. The former results are in line with our expectations, but the latter are not. This means the filtered authors have fewer affiliations. In other words, authors are more likely to use fewer numbers of affiliations with different granularity.
Here we are keenly aware that both of our two rules for the extreme analysis can be criticized for there are lots of exceptions for each of them. We leave the accurate statistics of this issue as an open issue and to solve it in the future.
For keywords, they will only involve the cross-domain duplicate name issue. The number of duplicated keywords will be affected by the predefined tf*idf threshold. We make statistics under different tf*idf settings and the results are shown in Table 5. Here dpk means the average appeared domains per duplicated keyword.
From Table 5 we can see that the number of duplicated items decreases as the tf*idf increases. However, both the duplicate ratio and the dpk don’t change too much. Even only remaining the keywords whose tf*idf values rank in top-10%, there are still 17.51% of the keywords appearing in average 2.62 research domains.
We also make statistics of the keywords’ domain distribution as we made for authors. But different from authors’ statistics, keywords’ statistics will depend on the tf*idf settings. Here report the domain distribution of keywords under top-10% tf*idf setting in Table 6 for metallurgical industry, computer science, electric industry, and traffic transportation, 4 randomly selected research domains.
From Table 6 we can see that compared with the domain distribution of authors, the keywords’ domain distribution is more balanced: most keywords appear less than 5 research domain. Generally, the more strict a tf*idf required, the less possible for a keyword appears in different research domains. For example, under the tf*idf setting of above top-80%, more than 90% keywords appear less than 5 domains (see the Appendix part for detail). But there are some keywords that appear in many research domains. For example, under the tf*idf setting of top-10%, there are still many keywords appearing in more than 20 research domains, such as genetic algorithm, numerical simulation, finite element method, preventive measure, etc.

heading: 4.2 IMBALANCE ISSUE
text: TransH3 classifies a KG’s relations into 4 types: 1-1, 1-n, m-1, and m-n. Following this definition, we make statistics for the proportion related triplets in different type of relations under different tf*idf settings. For the latter 3 types of relations, we also make statistics of the average value of n and the average value of m/n. The statistics results are shown in Table 7.
From Table 7 we can see that the triplet distribution for these 4 relation types is extremely imbalanced. In each tf*idf setting, the m-n relations account for more than 60% triplets. On the other hand, the 1-n relations account for so small a proportion that they can even be ignored. The proportion is almost unchanged for different type of relation under different tf*idf settings.
When we look into the average number of tails per head (ñ in Table 7) or the average number of head per tail (m̃ or ū in Table 7), the difference among different types of relations is far huge: in m-1 relations, the average value of m is more than 10, but both the average value of n in 1-n relations and the average value of m/n in m-n relations is relative smaller. These average values are also almost unchanged for different type of relation under different tf*idf settings.
The imbalance issue absolutely will have some impacts on the performance of some applications like knowledge graph embedding, etc. In the experiment section, we will further discuss these impacts.

heading: 4.3 APPLICATION ADAPTABILITY
text: In Table 8, we make a comparison between TechKG and some other KGs. TechKG can support more kinds of applications because it can provide other kinds of information as listed below.
Bilingual Translation Pairs. These pairs include bilingual titles, bilingual keywords, and bilingual abstracts. These bilingual translation pairs usually have very high quality for they are written by experts in the corresponding research domain. They can provide high quality training data for a machine translation task.
3For each relation r, we compute averaged number of tails per head (tphr), averaged number of heads per tail (hptr). If tphr<1.5 and hptr< 1.5, r is treated as 1-1. If tphr≥ 1.5 and hptr≥ 1.5, r is treated as m-to-n. If hptr< 1.5 and tphr≥ 1.5, r is treated as 1-n. If hptr≥ 1.5 and tphr< 1.5, r is treated as m-1.
Domain Categorizations. For each triplet or a paper’s abstract, it has a pre-organized domain category. With these categories, one can use the collected abstracts as a dataset for a text classification task. TechKG can also be easily divided into different subsets according to the research domains. Thus different kinds of transfer learning methods can also be evaluated by these subsets.
Domain Terminologies. In TechKG, a large proportion of its entities are domain terminologies. Thus a new kind of named entity recognition (NER) task that aims to recognize domain terminologies from text can be evaluated with these domain terminologies. In such a NER task, the required training data can be obtained by the distantly supervision based method as used in Yang et al. (2018).
Cross-language KG based Applications. The translation of relation in TechKG provides a possibility to link KGs of different languages together. This would benefit lots of cross-language KG based applications like cross-language Q&A, cross-language information retrieval, etc.
Finally, we also construct following kinds of knowledge bases (KBs) together with TechKG. Each of these KBs can be viewed as a by-product of TechKG and can be taken as a dataset for some applications.

heading: 4.3.1 TECHTERM/TECHBITERM
text: They are two KBs that consist of Chinese domain terminologies and Chinese-English terminology pairs respectively.
For the former one, we randomly select 10,000 terminologies with the highest tf*idf values for each research domain.
For the latter one, we randomly select 10,000 bilingual translation pairs with the highest cooccurrence values from each research domain.

heading: 4.3.2 TECHABS
text: It is a KB that stores papers’ abstracts. It can be used as a dataset for a text classification task. Here we randomly select 100,000 abstracts from each research domain to construct TechAbs.

heading: 4.3.3 TECHQA
text: TechQA is a dataset designed for the KBQA task. It is constructed based on TechKG-10. All the questions in it are generated by some predefined patterns. Currently, it focuses on 4 kinds of questions which are “who”, “what”, “when”, and “where”.
We list the used question patterns and detailed statistics of TechQA in the Appendix part.

heading: 4.3.4 TECHRE/TECHNER
text: They are two KBs designed mainly for relation extraction (RE for short) and domain terminology recognition tasks respectively. The former one is constructed based on TechKG-10 (a subset of TechKG, see the Appendix part for detail) and the latter one is constructed based on TechTerm.
RE aims to extract semantic relations between entity pairs from plain text. It is a typical classification task when the entity pairs and their possible relations are specified. To achieve high quality results, a large-scale training corpus is required. However, it is very time-consuming and expensive to construct such a corpus manually. To address this issue, distant supervision is proposed Mintz et al. (2009) to automatically generate training data via aligning KBs and texts. Its basic idea is that: if two entities have a relational fact in KBs, all the sentences that contain these two entities will express the corresponding relation. Accordingly, these sentences can be taken as positive samples for training this relation. Distant supervision provides an effective strategy to automatically construct large-scale training data for a RE task. Here TechKG-10 is used as the required supervision KB and papers’ abstracts are used as the required text corpus. It should be noted that except keywords, it is little possible for other types of components occur in the abstract part of a paper. So in TechRE, only hierarchical relation is considered. We set the co-occurrence threshold to 10 for the extraction of hierarchical relation. Specially, for each two terminologies in a co-occurrence with type of triplet, we first pick up all the sentences that both of these two terminologies appear. We call each
terminology pair along with its mentioned sentence set as a bag. If the co-occurrence value of a bag’s terminology pair is greater than 10, we assign a hierarchical relation to this bag and take it as a positive hierarchical training sample. Otherwise a NA relation is assigned to this bag and it is taken as a negative hierarchical training sample. By this way, we extract more than 14M training bags. Averagely, there are about 42 sentences per bag and 5 entities per sentences. The whole dataset is too large. So we randomly select at most 200,000 training bags for each research domain to construct TechRE. Besides, we limit the average sentences per bag in TechRE to 6 at most.
For TechNER, we also adopt a distant supervision method as TechRE, but TechTerm is used as the required KB. Papers’ abstracts are taken as the required text corpus. If a sentence contains a terminology, this sentence will be regarded as a positive training sample for recognizing this terminology. This distant supervision method for constructing NER training corpus is also used in Yang et al. (2018). By this way, we extract more than 210M training sentences. Averagely, there are about 3 entities per sentences. We randomly select 30,000 training samples in each research domain to construct TechNER. There are averagely 3 entities per training sample. In TechNER, if there are nested domain terminologies in a sentence, we will only remain the longest one.
The detailed and complete statistics of TechRE and TechNER can be found in the Appendix part.

heading: 5 CONCLUSIONS AND FUTURE WORK
text: In this paper, we introduce TechKG, a large-scale Chinese KG. To the best of our knowledge, TechKG is: both the first KG that is constructed from massive technical papers and the first KG that is technology-oriented and can be divided into different subsets according to the research domains.
We analyze the main characteristics of TechKG. Lots of statistics show that there are many new characteristics hierent in TechKG distinguish it from other existing KGs. We also conduct some primary experiments (see the Appendix part for detail) to evaluate the adaptability of TechKG. Experiments show that: 1)TechKG provides a new choice for lots of applications; and 2)a lot of new characteristics hierent in TechKG raise new challenges for many existing methods.
Currently, we generate TechKG mainly by re-arranging the meta-items in the collected technical papers. In the future, we will explore to extract more types of relations such as <method, solve, problem>, etc.

heading: A EXPERIMENTS AND ANALYSES
text: To evaluate the adaptability of TechKG, we apply it to three completely different applications: knowledge graph embedding, distantly supervised RE, and distantly supervised NER. For each of them, we select one of its state-of-the-art methods as baseline and use these baselines to conduct experiments on TechKG10 4, TechRE, and TechNER respectively. For each application, we report its experimental results on 4 randomly selected research domains that are metallurgical industry, computer science, electric industry, and traffic transportation.
NOTE that the aim of our experiments is not to tell whether a model is good or not. On the contrary, we just want to find whether our TechKG could show some new characteristics hidden in KGs that haven’t been noticed up to now.
A.1 KNOWLEDGE GRAPH EMBEDDING
KGE projects each item (entities and relationships) of a KG into continuous low-dimensional space(s). It aims to solve two major challenges that prohibit the availability of KGs. One is the incompleteness issue: although most of existing KGs contain large amount of triplets, they are far from complete. The other is the computation issue: most of existing KGs are stored in symbolic and logical formations while applications often involve numerical computing in continuous spaces. Link prediction, which is to predict the missing h or t for a correct triplet <h, r, t>, i.e., predict t given <h, r> or predict h given <r, t>, is often used to evaluate the performance of a KGE method. Rather than requiring one best answer, link prediction emphasizes more on ranking a set of candidate entities from the KG. Hits@10, the proportion of correct entities ranked in top 10, is often used as the evaluation metric for link prediction.
In this paper, we select ConvEDettmers (2018), one of the latest state-of-the-art KGE methods as baseline. We want to find: 1) whether the duplicate name issue would affect the performance; 2) whether the imbalance issue would affect the performance; and 3) whether TechKG also suffers from the test set leakage issue that is severe in WN18 and FB15k Dettmers (2018).
For these aims, we report the experimental results on following kinds of datasets.
1) The original TechKG10 dataset.
4The whole TechKG is a huge and growing KB, which is almost impossible for many applications to directly experiment on. Thus, we create TechKG10: a subset of it. We construct TechKG10 by selecting the subset of entities that meet following two requirements. First, if the entity is a keyword, its tf*idf value must rank in top-10%. Second, each selected entity has at least 10 mentions. Some basic statistics of TechKG10 are shown in Table 9.
2) A revised TechKG10 dataset that no cross domain author duplicate names included. 3) A revised TechKG10 dataset that no in domain author duplicate names included. 4) A revised TechKG10 dataset that no cross domain keyword duplicate names included. 5) A revised TechKG10 dataset that no any duplicate names included. 6) A revised TechKG10 dataset that no relations of co ocurrence or research interest. 7) A revised TechKG10 dataset5 that no relations of co ocurrence or research interest and no
any duplicate names included.
Even in TechKG10, the triplet number of its each domain is far larger than FB15k. So in experiments, we randomly select 30,000 triplets for each of above 7 datasets. The number of triplets used here is less than that of FB15k: there are about 48,000 triplets in FB15k. However, for each of these 7 datasets, its involved parameter size is far larger than FB15k. FB15k has 14951 entities and 1345 relations, but the average number of contained entities in these 7 datasets is about 140,000. This means it needs more time for training and testing on these 7 datasets.
The datasets 2-5 are used to evaluate the effects of duplicate names for the KGE task. The dataset 6 is used to evaluate the effects of imbalance issue. In TechKG10, both the co ocurrence relation and the research interest relation account for a large proportion in TechKG10, so we remove them to generate a more balanced dataset.
In experiments, the proportion of training, development and test sets is 8:1:1. The experimental results are shown in Table 10 and Figure 1, from which we can find following observations. During training, we iterate 300 rounds for each experiment.
First, compared with FB15k or WN18, the performance on TechKG10 is far lower. We think this is caused mainly by the data sparsity issue. Compared with other datasets, there are more parameters needed to be trained, but the number of training triplets is less.
Second, there is almost no the test set leakage issue at all in TechKG, which means it is impossible for a simple rule-based inverse model to achieve state-of-the-art results on TechKG.
Third, both the cross-domain and in-domain author duplicate name issues have a heavily negative effect on the performance. When there are no any duplicated names in the data set, each domain’ performance increases greatly. Taking metallurgical industry for example, the Hits@10 value on the original dataset is only 0.051, but this value increases to 0.389 when all the duplicated items are removed from the dataset. What’s more, the cross-domain keyword duplicate name issue has more negative effect on the performance, which is not in line with our original expectation. The reason of this result should be further studied.
Fourth, the imbalance issue has a heavy negative effect on the performance. When we remove two types of relations from the dataset, the performance obtains a huge improvement. This indicates that the more balanced a dataset’s relation types’ distributions, the better performance could be obtained.
Fifth, there are obvious performance gaps between different kinds of research domains. For example, when there is no any duplicated names, the Hits@10 value of the metallurgical industry domain is 0.389, which is much higher than that of the computer science domain, which is only 0.308.
Sixth, when there are no any duplicated names and the two special relations are removed, the best results are obtained, which is similar with that of on the datasets of WN18RR and FB15k-237.
Based on these observations we can roughly draw following conclusions. First, TechKG10 is a more challenging dataset than traditional WN18 and FB15k. Second, both the duplicate name issue and the imbalance issue are two inherent characteristics in TechKG.
A.2 DISTANT SUPERVISION RELATION EXTRACTION
In this task, we select the “PCNN+ATT” Lin et al. (2016) model as the baseline method. Following two main factors are considered for this selection. First, it is a state-of-the-art RE method and many current RE methods are based on it. Second, there are available source codes for it, which is very important for building a RE system quickly.
In almost of all of current RE research, Freebase is used as the distant supervision KB and the threeyear NYT corpus from 2005 to 2007 is used as the text corpus. In NYT dataset, there are about 281,270 training bags and 96,678 test bags. The size of TechRE is a little smaller than that of NYT. However, if taking the factor of sentences per bag into consideration, we can find that the parameter sizes of our datasets are far larger than NYT.
The average number of sentences per training bag in NYT corpus is about two, while this number is more 60 in TechRE. To evaluate whether this factor will affect the performance, we report the experimental results under different average number of sentences per training bag. There is also a huge difference between the proportion of NA relations in TechRE and NYT. To evaluate whether this factor will affect the performance, we also report the experimental results under different proportions of NA relations. During training, we iterate 100 rounds for each experiment. Finally, the p@m results are shown in Table 11, Figure 2, and Figure 3.
From these results we can find following observations.
First, both the average sentence per bag and the NA relation proportion have effects on the performance of RE task. For some domains, the higher the average sentence per bag, the poorer performance would be obtained. But for other domains, there is an opposite conclusion. For the factor of the NA relation proportion, we find that when we reduce the proportion of NA relations,
the performance decrease accordingly. Here not all of the experimental results are in line with our expectation. The reasons should be further studied.
Second, there is a big performance gap between NYT dataset and TechRE in both the precision/recall curves and the p@m: the performance on TechRE is far lower than that of on NYT. We find that in TechRE, there are almost no any cues in the training sentences that can imply a kind of relations. In NYT, however, there are words like “caused”, “because”, “contain”, “have”, etc. Obviously, these words are very helpful for the relation decision. In fact, we think TechRE is a more challenging dataset for the RE task: there are not always useful cues available.
Third, we find that there are also huge performance gaps between different kinds of domains. For example, under the setting of 2 sentences per bag, the performance of computer science domain is far higher than the results of other domains.
A.3 NAMED ENTITY RECOGNITION
In this study, we select LSTM-CRF Lample et al. (2016), one of the state-of-the-art NER methods as baseline and evaluate its performance on recognizing domain terminologies.
In each domain of TechNER, the number of training number is far larger than CoNll2003. So in experiments, we randomly select 21,000 training samples for each selected domain. By this way, there is a similar sample size between CoNll2003 (also about 21,000) and TechNER. In experiments, we select 15,000 samples for training, 3,500 samples for both the development set and the test set. During training, we iterate 300 rounds for each experiment. Finally, the experimental results are shown in Table 12 and Figure 4, from which we can find following observations.
First, there is a big performance gap between CoNll2003 and TechNER. In fact, the terminology recognition task is a more challenging task than the traditional NER task. The main reason is that lots of useful features that work well in a traditional NER task cannot work here. For example, capitalizations, prefix and suffix feature, collocations, all of these features are very useful in a tradition NER task. But none of them can be used here. What’s worse, as observed in the previous RE task, there are also almost no any cues in the training sentences that can imply the boundaries of a terminology.
Second, there is also an obvious performance difference between different kinds of domains. But compared with the previous KGE or RE tasks, the performance gap between different kinds of domains here is far smaller.
One may wonder whether we could obtain better performance by increasing the number of training samples. To answer this question, we select 25,000 training sentences for each domain and conduct the experiments again. Other experiment settings remain unchanged. The new results are denoted as large in Table 12. We can see that it is true that there is a minor improvement on the F1 values for each domain by increasing the number of training samples. But we find the performance gaps between different kinds of domains couldn’t be eliminated by this way.

heading: B DETAILED STATISTICS FOR TECHKG
text: Supposing there is a paper whose basic information is listed as:
Then the corresponding relations obtained are as listed.
Other statistics information about TechKG is shown in the following tables.
all authors 299221 162368 112354 59153 60325 36682 113688 60093 publi journal 300694 304045 72694 72821 190705 191887 112151 113069 translation of 738605 463472 320236 192795 150664 84112 309387 190816 Total 1602651 6556905 601142 2748591 580138 2052750 673566 2750582
MathRelated Military Machinery ArchitectureRelated Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri#
author of 85298 97359 17287 19647 317186 451634 378001 488228 first author 70164 47958 12936 8013 245344 163764 305670 209767 second author 48289 32372 8137 4860 193264 132597 209722 140807 co author 34183 60632 7788 21704 143882 445428 147138 422327 research inter 109831 318530 19267 56058 348790 1575060 394395 1726944 affiliation 51141 137190 6270 16530 90596 193492 105555 296834 belged domain 84561 84560 45274 45274 271134 271133 380489 380488 cooccurrence with 84334 303192 45040 215970 270817 1414790 380032 2586344 published year 59760 60059 50490 50750 289942 292878 530158 536351 contained cky 106065 53583 64307 32532 476547 240539 810061 409713 contained eky 72836 36589 6474 3254 242015 121259 312200 156465 other author 24575 17035 6265 6774 166201 155290 153569 137709 all authors 84109 47980 14929 8016 308244 163825 382513 209896 publi journal 59766 59912 50485 50582 290041 292305 530370 537169 translation of 270735 167022 30459 17530 736666 449162 941452 585890 Total 468289 1524016 162185 557500 1620300 6363473 2321365 8825463
AgricultureRelated ElectricIndustry Forestry MaterialScience Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri#
author of 454171 875559 321536 465500 111149 163046 66045 97167 first author 335916 235246 259346 173736 78978 52202 42734 26317 second author 290594 207008 189222 132736 62782 41477 37712 24415 co author 211507 1198644 135927 460054 55719 206904 39170 132590 research inter 432607 2942049 359189 1579019 120707 564252 77469 340315 affiliation 200113 732908 125865 347644 30375 65086 46816 128112 belged domain 371021 371021 324776 324776 113026 113026 49322 49322 cooccurrence with 370559 3208824 324348 1875036 112852 660413 49271 197269 published year 705112 718518 400724 406143 140191 141419 37363 37664 contained cky 1018196 516386 581225 294232 208406 104973 62587 31489 contained eky 399977 200569 284228 142586 91247 45698 45868 23018 other author 315185 433346 152853 159046 65013 69377 44179 46435 all authors 450364 235326 320083 173835 99092 52224 50837 26330 publi journal 705422 722127 400873 405742 140251 141260 37348 37413 translation of 1048945 684803 834585 520957 258882 161649 145708 90958 Total 2854050 13282530 1912996 7461378 654388 2583059 295413 1288937
ArtRelated PowerIndustry Mechanics Aerospace Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri#
author of 51753 39151 242128 334368 37258 47743 127097 168772 first author 48269 31699 192127 129725 26885 17255 100993 67146 second author 8962 5391 139685 94340 23155 15241 71143 48484 co author 8391 14382 101355 320363 19463 45339 54175 151348 research inter 47206 91295 249826 1152985 52518 178325 144054 554489 affiliation 1983 1699 111773 321825 25503 64698 39971 89938 belged domain 125637 125637 214309 214309 35907 35907 128325 128325 cooccurrence with 125292 1289926 214040 1246853 35894 127426 128172 577962
published year 254324 259120 274742 277813 19766 19811 120960 122302 contained cky 272725 140982 421708 213552 36737 18448 175622 88603 contained eky 13831 6935 201966 101368 31070 15581 89499 44824 other author 2598 2070 112516 110354 18710 15249 58155 53148 all authors 50464 31764 240252 129859 32463 17259 121975 67168 publi journal 254455 258339 274840 278110 19746 19751 120996 121511 translation of 98210 53865 588788 366697 112914 68574 308614 183263 Total 593371 2352315 1357767 5292736 193028 706631 661511 2467501
Bioscience AnimalHusbandry EnergyRelated MiningIndustry Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri#
author of 220383 374054 112335 199300 199497 295852 171235 211116 first author 149691 98705 75338 49653 149836 101651 130207 84686 second author 134009 90038 66178 44665 113533 75287 92642 59149 co author 120131 565274 61213 312094 90241 355897 77261 206023 research inter 241546 1298293 117775 661520 206738 1185271 200331 763546 affiliation 148883 492152 42939 106813 99350 350778 50093 96712 belged domain 152943 152943 144486 144485 158772 158771 174923 174923 cooccurrence with 152754 743139 144286 1165478 158635 1052945 174782 825785 published year 136963 138050 269495 277489 171945 173624 161687 163021 contained cky 231780 116802 341725 174647 283221 142712 288575 145345 contained eky 154771 77745 83811 42076 159958 80159 135982 68140 other author 150487 185321 78441 105005 112089 118949 77402 67287 all authors 188665 98731 95842 49687 190843 101739 158881 84728 publi journal 137006 137439 269574 280387 172004 172952 161736 162769 translation of 501006 320686 234783 149316 485747 307204 426639 260175 Total 1016197 4889510 873713 3762703 1025518 4673947 943136 3373454
Astronomy CultureRelated LightIndustry SocialScience&Philos ophy
Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri# author of 358797 632889 106430 85699 256322 352752 800417 763626 first author 282323 203774 99818 66896 201018 140177 733745 536810 second author 228664 165890 24425 15661 140552 96315 264352 165644 co author 147335 742940 19338 28906 104138 335546 178099 292637 research inter 402486 2243002 87407 187610 257723 1162402 769605 2636381 affiliation 103486 329883 5001 4938 90172 245136 163685 301436 belged domain 311435 311434 210159 210159 272246 272245 794635 794635 cooccurrence with 311083 1674665 209775 2967853 271629 1906011 793704 6250869 published year 306976 310215 483035 505601 373662 378913 1351764 1363281 contained cky 515453 259807 491900 254748 542107 275366 2144814 1084329 contained eky 318143 159516 44204 22132 203655 102299 926805 465458 other author 216826 263252 4236 3157 116655 116307 85617 61183 all authors 381349 203848 106413 66954 250632 140330 839278 536933 publi journal 307168 308563 483330 511041 373822 379262 1352491 1367087 translation of 974101 610861 183399 108123 607602 377253 2364399 1530996 Total 1918791 8420905 1101328 5039721 1577973 6280999 5482685 18151694
Education TrafficTransportation IndustrialTechnology Finance Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri#
author of 309337 260059 270505 323174 132743 152571 366453 365594 first author 279552 183452 223218 149307 94819 58153 323869 227102 second author 94165 56786 140807 92900 69201 42856 150355 97077 co author 75746 98772 102908 253687 67728 163072 101406 177472 research inter 282404 807177 283403 1117265 166966 534338 321837 1135657
affiliation 45382 51812 73780 167084 67516 137114 65015 107884 belged domain 459339 459341 290242 290242 188271 188270 544014 544013 cooccurrence with 458409 6487553 289732 1928432 187940 849757 542917 6446286 published year 1387391 1435873 410641 415921 174508 176551 1761391 1785618 contained cky 1759244 910876 609960 309770 253069 127808 2029112 1037511 contained eky 265118 132834 224883 112727 87151 43654 311787 156461 other author 29941 19824 97981 80988 60791 51567 58789 41426 all authors 307335 183556 270693 149382 110228 58169 378969 227176 publi journal 1388214 1450263 410810 415172 174555 175862 1762158 1795718 translation of 712538 447052 665226 408883 317372 189344 890097 553170 Total 3439775 12985548 1708950 6215738 836742 2949164 4196406 14698754
ComputerScience Chemistry Ent# Tri# Ent# Tri#
author of 317782 443985 529270 921386 first author 255881 169619 414458 294746 second author 198351 140089 338716 240198 co author 138764 391289 222034 1031983 research interest 350332 1561345 500015 3019792 affiliation 133206 358802 247832 836399 belged domain 294829 294828 368382 368382 cooccurrence with 294450 1717881 367962 2363407 published year 353439 356711 485128 493364 contained cky 526722 266678 773325 392785 contained eky 275255 138010 472653 238015 other author 150728 134301 315932 386539 all authors 318746 169666 545942 295031 publi journal 353530 356221 485342 492794 translation of 829241 520559 1295177 836286 Total 1818503 7020316 2734418 12211721
Physics 166374 63948 38.44% 3.45 153886 48161 31.30% 3.25 Traffic 391979 176284 44.97% 3.96 366150 138097 37.72% 3.62 History& 154324 84683 54.87% 4.16 135382 60282 44.53% 3.89 Agriculture 555514 266312 47.94% 3.51 517525 223289 43.15% 3.3 Machinery 401522 203016 50.56% 3.6 345031 141293 40.95% 3.44 Chemistry 582921 192445 33.01% 3.71 519895 130998 25.20% 3.74 Law 123017 55986 45.51% 3.82 107890 38195 35.40% 3.51 Energy 242753 100449 41.38% 3.88 222868 72442 32.50% 3.63 Astronomy 517269 182781 35.34% 3.43 488177 147766 30.27% 3.21 Education 544084 247013 45.40% 3.85 496295 190482 38.38% 3.7 Power 311758 132881 42.62% 3.93 266076 83308 31.31% 3.83 Art-related 115208 70250 60.98% 4.43 105339 55755 52.93% 4.1 Material 73431 43548 59.30% 3.95 64075 31637 49.37% 3.72 Metal 369741 149003 40.30% 3.78 327094 99341 30.37% 3.64 Electronic 461533 179554 38.90% 3.48 429959 138569 32.23% 3.3 Politics 317923 170370 53.59% 4.02 296393 139571 47.09% 3.79 Military 42421 33484 78.93% 4.82 37223 27337 73.44% 4.04 Medicine* 1357307 193647 14.27% 3.7 1357307 179792 13.25% 3.42 Bioscience 233339 137138 58.77% 3.22 231246 132610 57.35% 2.99 Mining 253300 116607 46.04% 3.8 229435 85991 37.48% 3.52 Culture 210510 120672 57.32% 4.12 188244 91374 48.54% 3.96 Sport 116087 54913 47.30% 4.41 104234 39707 38.09% 4.04 Total 7978805 1676176 21.01% 2.84 7883539 1363444 17.29% 2.68 Top-60% tf*idf Top-50% tf*idf domain total # dup # ratio(%) dpk total # dup # ratio(%) dpk Compute 330618 91361 27.63% 3.46 330618 86579 26.19% 3.39 Mechanics 49233 17425 35.39% 3.45 40677 11492 28.25% 3.73 Power 127891 41333 32.32% 4.02 125292 37402 29.85% 3.94 Publishing 108383 38477 35.50% 4.25 96591 28518 29.52% 4.38 Forestry 120953 36503 30.18% 3.87 100416 24788 24.69% 4.2 Industrial 169796 73709 43.41% 4.16 163843 60740 37.07% 4.18 Math 105123 23238 22.11% 3.19 105123 22369 21.28% 3.12 Environme 164196 56997 34.71% 4.12 162204 50640 31.22% 4.11 Fisheries 46247 19162 41.43% 3.95 39154 15122 38.62% 3.99 Animal 148401 60725 40.92% 3.59 125118 47054 37.61% 3.71 Architectur 411955 86091 20.90% 4.06 411955 80319 19.50% 4.01 Aerospace 139229 45166 32.44% 3.91 116244 30815 26.51% 4.14 LightIndu 284236 87248 30.70% 4.02 234687 57906 24.67% 4.4 Finance 519083 170558 32.86% 3.71 519083 160350 30.89% 3.63 Physics 130950 27847 21.27% 3.57 130950 27139 20.72% 3.49 Traffic 334847 87609 26.16% 3.91 280680 59434 21.18% 4.2 History& 113698 36864 32.42% 4.16 94730 29792 31.45% 4.28 Agriculture 426907 132544 31.05% 3.63 426907 126279 29.58% 3.59 Machinery 345031 103216 29.91% 3.65 274464 71234 25.95% 3.94 Chemistry 519895 112394 21.62% 3.71 519895 106713 20.53% 3.65 Law 98168 26158 26.65% 3.57 81650 18476 22.63% 3.7 Energy 181397 36103 19.90% 4.33 177900 31577 17.75% 4.18 Astronomy 418406 74062 17.70% 3.72 418406 69803 16.68% 3.69 Education 413007 128199 31.04% 3.85 413007 122616 29.69% 3.73 Power 228137 45596 19.99% 4.45 225262 40841 18.13% 4.38 Art-related 91793 38101 41.51% 4.21 72759 28137 38.67% 4.39 Material 54982 22371 40.69% 3.9 53657 20451 38.11% 3.82 Metal 279246 55826 19.99% 4.16 279246 52324 18.74% 4.12 Electronic 365397 76880 21.04% 3.72 365397 73182 20.03% 3.65 Politics 266360 103506 38.86% 3.83 220783 80911 36.65% 3.89 Military 31705 20459 64.53% 3.68 27319 15218 55.70% 3.54
Medicine* 1357307 150415 11.08% 3.41 1357307 137004 10.09% 3.41 Bioscience 178399 84968 47.63% 3.21 178399 83517 46.81% 3.16 Mining 209757 49421 23.56% 3.97 171103 32061 18.74% 4.35 Culture 169734 70008 41.25% 3.96 141133 57728 40.90% 3.95 Sport 86908 22425 25.80% 4.26 73779 14738 19.98% 4.37 Total 7494900 820490 10.95% 2.87 7234018 691549 9.56% 2.91 Top-40% tf*idf Top-30% tf*idf domain total # dup # ratio(%) dpk total # dup # ratio(%) dpk Compute 330618 85447 25.84% 3.35 160492 77130 48.06% 3.25 Mechanics 40677 11381 27.98% 3.7 21928 8787 40.07% 3.6 Power 125292 36903 29.45% 3.9 66560 28818 43.30% 3.77 Publishing 80443 21413 26.62% 4.43 80443 20771 25.82% 4.33 Forestry 100416 24569 24.47% 4.1 100416 23925 23.83% 3.92 Industrial 129906 45490 35.02% 4.31 129906 44178 34.01% 4.17 Math 65727 22136 33.68% 3.08 52563 13028 24.79% 2.99 Environme 125698 38074 30.29% 4.28 125698 36534 29.06% 4.14 Fisheries 39154 14930 38.13% 3.91 23036 12253 53.19% 3.76 Animal 125118 46644 37.28% 3.62 65635 39093 59.56% 3.44 Architectur 411955 78870 19.15% 3.96 411955 76538 18.58% 3.83 Aerospace 116244 30364 26.12% 4.1 116244 29082 25.02% 4.01 LightIndu 234687 56958 24.27% 4.32 234687 54453 23.20% 4.17 Finance 519083 157255 30.29% 3.56 237796 120491 50.67% 3.53 Physics 130950 26836 20.49% 3.45 130950 25750 19.66% 3.34 Traffic 280680 58669 20.90% 4.15 280680 57342 20.43% 4.01 History& 90999 26010 28.58% 4.18 90999 25580 28.11% 4.07 Agriculture 426907 124177 29.09% 3.51 204111 108968 53.39% 3.39 Machinery 274464 70529 25.70% 3.9 274464 68873 25.09% 3.76 Chemistry 519895 103942 19.99% 3.61 218944 68892 31.47% 3.6 Law 81650 18312 22.43% 3.62 81650 17740 21.73% 3.54 Energy 177900 31246 17.56% 4.13 177900 30129 16.94% 4.02 Astronomy 418406 68308 16.33% 3.64 418406 66308 15.85% 3.53 Education 413007 120116 29.08% 3.63 413007 114874 27.81% 3.54 Power 225262 40380 17.93% 4.33 225262 39313 17.45% 4.18 Art-related 69365 24796 35.75% 4.3 69365 24508 35.33% 4.18 Material 37175 20308 54.63% 3.77 30025 13479 44.89% 3.83 Metal 279246 51588 18.47% 4.07 279246 49229 17.63% 3.94 Electronic 365397 72171 19.75% 3.6 365397 70338 19.25% 3.49 Politics 220783 80104 36.28% 3.78 220783 78053 35.35% 3.67 Military 21811 10929 50.11% 3.49 15800 7495 47.44% 3.66 Medicine* 621737 111247 17.89% 3.31 570727 76733 13.44% 3.32 Bioscience 117523 75619 64.34% 3.06 98653 55237 55.99% 3.07 Mining 171103 31557 18.44% 4.32 171103 30962 18.10% 4.18 Culture 141133 57227 40.55% 3.83 77019 56576 73.46% 3.69 Sport 73779 14484 19.63% 4.28 73779 13955 18.91% 4.18 Total 6357230 662029 10.41% 2.88 5224408 594194 11.37% 2.84 Top-20% tf*idf Top-10% tf*idf domain total # dup # ratio(%) dpk total # dup # ratio dpk Compute 107083 41106 38.39% 3.35 52679 23121 43.89% 2.94 Mechanics 14378 4455 30.98% 3.9 13109 2782 21.22% 3.54 Power 42362 12931 30.52% 3.98 42362 10213 24.11% 3.46 Publishing 80443 18323 22.78% 4.06 18899 5757 30.46% 3.55 Forestry 100416 20812 20.73% 3.71 20338 8676 42.66% 3.35 Industrial 56644 25755 45.47% 4.15 32172 9659 30.02% 3.96 Math 41059 7187 17.50% 3.07 16902 4213 24.93% 2.72 Environme 55104 32477 58.94% 3.9 28190 12582 44.63% 3.61 Fisheries 15467 5983 38.68% 3.9 7703 3498 45.41% 3.61
Animal 44513 20591 46.26% 3.49 21635 11337 52.40% 3.2 Architectur 132244 62937 47.59% 3.58 82356 26109 31.70% 3.25 Aerospace 44489 18194 40.90% 3.91 28764 7763 26.99% 3.71 LightIndu 87820 44482 50.65% 3.95 54445 17972 33.01% 3.57 Finance 175871 77142 43.86% 3.43 80437 34048 42.33% 3.01 Physics 46272 13871 29.98% 3.16 37124 8311 22.39% 2.9 Traffic 280680 52391 18.67% 3.72 54608 18863 34.54% 3.44 History& 90999 22250 24.45% 3.81 19364 8568 44.25% 3.38 Agriculture 135481 56395 41.63% 3.44 68732 31123 45.28% 3.1 Machinery 98038 53510 54.58% 3.63 48717 20694 42.48% 3.5 Chemistry 186654 50161 26.87% 3.54 73432 26187 35.66% 3.22 Law 81650 15430 18.90% 3.34 15890 6279 39.52% 2.87 Energy 177900 27563 15.49% 3.79 35818 11903 33.23% 3.29 Astronomy 124643 37559 30.13% 3.41 62318 22069 35.41% 3.1 Education 136691 68097 49.82% 3.45 68097 29567 43.42% 3.01 Power 225262 34860 15.48% 3.92 44511 12622 28.36% 3.65 Art-related 29950 18012 60.14% 3.97 14451 7378 51.06% 3.51 Material 19590 6944 35.45% 4.23 19590 6032 30.79% 3.52 Metal 90972 41917 46.08% 3.76 56900 16731 29.40% 3.48 Electronic 117859 46219 39.22% 3.35 57358 23912 41.69% 3 Politics 79945 52325 65.45% 3.51 39743 24345 61.26% 3.03 Military 15767 6912 43.84% 3.4 6397 3002 46.93% 3.03 Medicine* 309239 50200 16.23% 3.23 155681 21741 13.97% 2.9 Bioscience 68539 31521 45.99% 3.26 29217 19270 65.95% 3.01 Mining 171103 28667 16.75% 3.93 31666 12476 39.40% 3.6 Culture 53358 35120 65.82% 3.61 25639 17156 66.91% 3.11 Sport 73779 12220 16.56% 3.94 16694 4220 25.28% 3.4 Total 2870181 412436 14.37% 2.8 1153819 202060 17.51% 2.62
Education 219439 88219 59976 18463 3076 240547 66863 17246 955 4 Power 101378 43924 40005 16295 3038 117158 35801 13058 1039 4 Art- 39538 26654 22101 9391 2706 50313 29400 6959 288 4 Material 32894 12718 10319 4878 1555 36766 12941 3768 249 1 Metal 126442 48233 41187 16217 3055 140198 36777 12740 985 4 Electronic 170228 53877 42948 16699 3050 186580 37361 11772 935 4 Politics 128661 61094 42208 14438 3027 146014 51200 12424 562 4 Military 16503 9423 10789 6638 2342 20057 14777 5316 350 4 Medicine 235742 66734 50723 17867 3074 197769 40031 12813 934 4 Bioscie 149574 27733 17390 7759 2456 152119 20227 4566 248 0 Mining 94202 35394 33282 14191 2931 106863 30312 11194 925 3 Culture 80185 42999 30772 11507 2910 96446 39512 9069 342 4 Sport 32544 19804 20979 10316 2869 41189 23612 6999 431 4 Total 1508200 201467 85567 19461 3076 1604384 174265 28770 1113 4
Top-80% tf*idf Top-70% tf*idf Domain [1,5) [5,10) [10,20) [20,30) [30,36) [1,5) [5,10) [10,20) [20,30) [30,36) Computer 178551 18490 5311 290 1 144694 13198 3471 162 0 Mechanis 25445 3456 1010 73 0 18657 2276 632 50 0 Power 73520 12170 4750 301 1 51769 7367 2877 164 0 Publising 63339 16731 3983 231 1 46943 11947 2251 114 0 Forestry 68862 11681 3626 302 1 53059 7655 2017 147 0 Industrial 119043 25138 8195 395 1 90643 16247 5113 222 0 Math 42315 3983 1083 103 0 32918 2686 716 62 0 Environe 111366 19714 6020 362 1 83606 13507 3639 197 0 Fisheries 30416 7402 1748 119 1 24240 4905 943 57 0 Animal 83788 13368 2949 195 1 70944 9922 1801 108 0 Architecr 174971 25065 8875 409 1 138368 18672 5819 226 0 Aerospace 76395 13512 4514 307 1 54559 8682 2890 163 0 LightIndu 140034 26722 8871 408 1 102813 18263 5597 223 0 Finance 278183 45069 8578 347 1 228419 33115 5247 193 0 Physics 55639 6368 1819 122 0 42653 4277 1154 77 0 Traffic 143385 24236 8254 408 1 115716 16821 5337 223 0 History& 66317 15232 3029 104 1 47757 10821 1657 47 0 Agricue 229104 29071 7730 406 1 196048 21981 5034 226 0 Machiner 173589 22321 6770 335 1 121333 15311 4461 188 0 Chemistry 162082 22878 7127 357 1 109643 16445 4708 202 0 Law 46273 7780 1813 119 1 32487 4788 862 58 0 Energy 84042 10704 5343 359 1 61357 7321 3562 202 0 Astronoy 159056 17989 5389 346 1 130638 13404 3525 199 0 Education 201937 37268 7451 356 1 158275 27549 4467 191 0 Power 109614 16374 6500 392 1 68112 10768 4216 212 0 Art-relatd 52387 14955 2819 88 1 42654 11469 1592 40 0 Material 35674 6042 1741 91 0 26533 3976 1074 54 0 Metal 125018 17215 6397 372 1 83499 11493 4139 210 0 Electronic 155845 17881 5491 336 1 121852 12853 3676 188 0 Politics 135302 29686 5176 205 1 113926 22562 2975 108 0 Military 23253 8484 1633 113 1 21793 4725 761 58 0 Medicine 163118 24004 6159 365 1 155809 19548 4232 203 0 Bioscie 122389 12387 2264 98 0 121259 9798 1495 58 0 Mining 98533 12142 5575 356 1 73768 8295 3726 202 0 Culture 94265 22559 3730 117 1 72183 17062 2076 53 0 Sport 41589 10381 2786 156 1 30878 7209 1547 73 0 Total 1574965 87502 13294 414 1 1292004 62879 8333 228 0 Top-60% tf*idf Top-50% tf*idf Domain [1,5) [5,10) [10,20) [20,30) [30,36) [1,5) [5,10) [10,20) [20,30) [30,36) Computer 79805 9156 2317 83 0 76488 8019 2007 65 0
Mechanis 15188 1721 483 33 0 9798 1291 374 29 0 Power 33787 5401 2058 87 0 30880 4693 1758 71 0 Publising 28979 8075 1362 61 0 21149 6322 1012 35 0 Forestry 29909 5295 1223 76 0 19655 4215 870 48 0 Industrial 58174 11821 3591 123 0 47958 9722 2972 88 0 Math 21008 1691 505 34 0 20421 1491 425 32 0 Environe 44751 9801 2344 101 0 39948 8673 1945 74 0 Fisheries 15329 3279 531 23 0 12074 2638 398 12 0 Animal 51843 7630 1199 53 0 39883 6249 890 32 0 Architecr 68930 13068 3967 126 0 64933 11915 3381 90 0 Aerospace 37008 6027 2045 86 0 24803 4379 1572 61 0 LightIndu 69236 13980 3906 126 0 44072 10734 3010 90 0 Finance 143460 23667 3331 100 0 137240 20365 2677 68 0 Physics 24143 2882 773 49 0 23797 2628 667 47 0 Traffic 71213 12521 3754 121 0 46868 9485 2995 86 0 History& 28320 7530 986 28 0 22632 6429 717 14 0 Agricue 113301 15837 3280 126 0 109111 14299 2779 90 0 Machiner 87234 12539 3334 109 0 58550 9905 2701 78 0 Chemistry 94955 13800 3522 117 0 91084 12508 3035 86 0 Law 22323 3286 520 29 0 15700 2397 364 15 0 Energy 28257 5274 2460 112 0 25178 4305 2010 84 0 Astronoy 62282 9335 2334 111 0 59190 8559 1970 84 0 Education 105898 19411 2786 104 0 103757 16529 2258 72 0 Power 34834 7709 2939 114 0 31581 6636 2539 85 0 Art- 28729 8414 932 26 0 20813 6642 668 14 0 Material 18579 2998 764 30 0 17190 2600 636 25 0 Metal 44478 8357 2875 116 0 42049 7648 2543 84 0 Electronic 65234 9041 2507 98 0 62923 8005 2178 76 0 Politics 85095 16523 1824 64 0 66576 12962 1337 36 0 Military 17081 2950 393 35 0 12959 2007 234 18 0 Medicine 131366 15874 3059 116 0 120185 14188 2547 84 0 Bioscie 76524 7439 979 26 0 75875 6812 810 20 0 Mining 40310 6269 2732 110 0 25213 4657 2110 81 0 Culture 55891 12826 1259 32 0 46514 10290 910 14 0 Sport 16779 4683 918 45 0 10939 3157 616 26 0 Total 768736 45996 5632 126 0 648186 38705 4568 90 0
Top-40% tf*idf Top-30% tf*idf Domain [1,5) [5,10) [10,20) [20,30) [30,36) [1,5) [5,10) [10,20) [20,30) [30,36) Computer 75925 7586 1876 60 0 69126 6404 1553 47 0 Mechanis 9745 1251 359 26 0 7556 954 255 22 0 Power 30613 4551 1675 64 0 24130 3349 1288 51 0 Publising 15831 4789 762 31 0 15614 4447 683 27 0 Forestry 19729 4011 787 42 0 19627 3578 685 35 0 Industrial 35478 7478 2456 78 0 35010 6964 2140 64 0 Math 20279 1435 390 32 0 11981 779 242 26 0 Environe 29457 7026 1528 63 0 28776 6382 1323 53 0 Fisheries 12067 2496 357 10 0 10020 1961 263 9 0 Animal 39947 5879 791 27 0 34036 4456 581 20 0 Architecr 64296 11385 3109 80 0 63318 10457 2697 66 0 Aerospace 24561 4240 1509 54 0 23800 3897 1341 44 0 LightIndu 43877 10242 2759 80 0 42737 9238 2412 66 0 Finance 136361 18477 2356 61 0 104157 14501 1785 48 0 Physics 23648 2509 637 42 0 22919 2234 562 35 0 Traffic 46640 9150 2801 78 0 46332 8535 2411 64 0 History& 20045 5407 546 12 0 20150 4931 487 12 0 Agricue 108219 13356 2522 80 0 96095 10768 2039 66 0 Machiner 58368 9549 2542 70 0 57914 8710 2193 56 0
Chemistry 89130 11920 2815 77 0 58814 8057 1963 58 0 Law 15794 2191 312 15 0 15417 2029 279 15 0 Energy 25067 4208 1896 75 0 24445 3919 1704 61 0 Astronoy 58331 8106 1796 75 0 57316 7374 1556 62 0 Education 103116 14946 1990 64 0 99890 13209 1722 53 0 Power 31466 6444 2393 77 0 31093 6049 2107 64 0 Art- 18573 5691 522 10 0 18909 5134 455 10 0 Material 17211 2479 596 22 0 11191 1831 440 17 0 Metal 41713 7400 2400 75 0 40324 6746 2097 62 0 Electronic 62425 7611 2067 68 0 61557 6919 1808 54 0 Politics 67276 11661 1135 32 0 66580 10466 978 29 0 Military 9356 1422 138 13 0 6265 1116 105 9 0 Medicine 98540 10722 1913 72 0 67893 7445 1339 56 0 Bioscie 69108 5859 639 13 0 50304 4462 461 10 0 Mining 24942 4537 2006 72 0 24802 4314 1787 59 0 Culture 47127 9330 758 12 0 47713 8221 630 12 0 Sport 10911 3018 531 24 0 10672 2782 479 22 0 Total 622623 35235 4091 80 0 560547 30150 3431 66 0 Top-20% tf*idf Top-10% tf*idf Domain [1,5) [5,10) [10,20) [20,30) [30,36) [1,5) [5,10) [10,20) [20,30) [30,36) Computer 36424 3762 901 19 0 21463 1372 278 8 0 Mechanis 3664 621 158 12 0 2425 278 74 5 0 Power 10496 1763 650 22 0 8876 1014 314 9 0 Publising 14417 3419 475 12 0 4894 786 76 1 0 Forestry 17561 2754 480 17 0 7693 897 82 4 0 Industrial 20221 4171 1334 29 0 7839 1406 405 9 0 Math 6536 491 147 13 0 3975 181 52 5 0 Environe 26552 4987 912 26 0 10725 1649 201 7 0 Fisheries 4784 1076 119 4 0 2909 553 34 2 0 Animal 17813 2499 271 8 0 10217 1062 57 1 0 Architecr 53639 7506 1761 31 0 23276 2333 491 9 0 Aerospace 14920 2428 824 22 0 6562 936 258 7 0 LightIndu 35884 6942 1625 31 0 15382 2154 427 9 0 Finance 67528 8587 1007 20 0 31595 2269 180 4 0 Physics 12507 1079 269 16 0 7761 441 102 7 0 Traffic 43929 6815 1616 31 0 16376 2019 459 9 0 History& 18194 3729 321 6 0 7553 970 45 0 0 Agricue 49578 5796 990 31 0 28649 2224 241 9 0 Machiner 45609 6416 1457 28 0 18007 2222 456 9 0 Chemistry 43218 5652 1263 28 0 23545 2230 403 9 0 Law 13703 1523 197 7 0 5951 305 23 0 0 Energy 22832 3428 1273 30 0 10521 1030 343 9 0 Astronoy 32857 3885 788 29 0 20116 1710 235 8 0 Education 59670 7501 903 23 0 27506 1887 171 3 0 Power 28327 5032 1470 31 0 10648 1532 433 9 0 Art- 14347 3384 278 3 0 6396 939 43 0 0 Material 5450 1205 281 8 0 5279 638 110 5 0 Metal 35017 5391 1478 31 0 14518 1761 443 9 0 Electronic 40886 4230 1080 23 0 22102 1453 349 8 0 Politics 45394 6343 575 13 0 22615 1631 97 2 0 Military 6013 821 72 6 0 2765 227 10 0 0 Medicine 44932 4463 781 24 0 20291 1311 133 6 0 Bioscie 28305 2947 266 3 0 17897 1310 62 1 0 Mining 23563 3752 1324 28 0 10720 1329 418 9 0 Culture 29856 4932 330 2 0 15754 1362 40 0 0 Sport 9663 2207 341 9 0 3660 508 50 2 0 Total 390134 20104 2167 31 0 194842 6612 597 9 0
Finance 14.75 0 \ 27.61 29.08 57.64 1.3 14.88 0 \ 27.84 28.8 57.28 1.32 Physics 13.52 0 \ 13.67 7.32 72.81 1.16 13.59 0 \ 13.75 7.15 72.66 1.17 Traffic 16.56 0 \ 23.06 12.83 60.38 1.3 16.81 0 \ 23.44 12.65 59.75 1.32 History& 12.23 0.5 2.2 32.83 15.97 54.47 0.99 12.91 0.5 2.2 34.25 15.52 52.33 0.99 Agricue 12.85 0 \ 15.93 13.58 71.22 1.28 12.91 0 \ 15.98 13.38 71.11 1.29 Machiner 15.9 0 \ 18.77 9.34 65.33 1.29 16.3 0 \ 18.14 8.48 65.55 1.33 Chemistry 14.83 0 \ 15.65 10.29 69.53 1.32 14.97 0 \ 15.21 9.67 69.82 1.35 Law 16.61 0.1 1.8 36.33 16.28 46.92 1.03 17.45 0.2 1.8 37.31 15.48 45.08 1.04 Energy 14.01 0 \ 14.55 8.85 71.44 1.25 14.13 0 \ 14.66 8.67 71.22 1.26 Astronoy 15.34 0 \ 14.44 9.99 70.22 1.32 15.4 0 \ 14.53 9.84 70.07 1.33 Education 13.99 0 \ 27.37 25.03 58.65 1.08 14.4 0 \ 26.13 22.94 59.46 1.09 Power 16.03 0 \ 19.62 10.68 64.35 1.25 16.33 0 \ 19.99 10.35 63.68 1.29 Art- 10.04 0.1 2.4 30.83 23.81 59.03 1.06 10.38 0.1 2.4 31.56 23.47 57.96 1.06 Material 13.54 0 \ 13.94 6.47 72.53 1.1 13.58 0 \ 14.09 6.27 72.33 1.11 Metal 15.73 0 \ 17.3 9.83 66.97 1.28 15.92 0 \ 17.49 9.56 66.6 1.31 Electronic 16.12 0 \ 18.74 10.63 65.14 1.24 16.24 0 \ 18.92 10.47 64.84 1.25 Politics 12.94 0.1 1.6 34.5 26.91 52.47 1.07 13.3 0.1 1.6 34.04 25.66 52.58 1.07 Military 13.15 0 \ 30 29.21 56.85 1.06 13.94 0 \ 31.12 28.28 54.94 1.07 Medicine 12.14 0 \ 14.28 15.63 73.58 1.48 12.14 0 \ 14.28 15.63 73.58 1.48 Bioscie 12.78 0 \ 12.48 7.29 74.75 1.13 12.8 0 \ 12.5 74.71 1.13 Mining 17.58 0 \ 19.13 8.61 63.28 1.24 17.88 0 \ 19.47 62.64 1.26 Culture 8.35 0.1 1.6 27.88 22.71 63.7 1.04 8.59 0.1 1.6 28.32 63.01 1.04 Sport 16.29 0 \ 24.11 15.61 59.6 1.37 16.86 0 \ 24.48 58.66 1.41 Total 14.6 0 1.9 19.83 13.13 65.56 1.25 14.77 0 1.9 19.78 65.44 1.27
Top-60% tf*idf Top-50% tf*idf 1-1 1-n m-1 m-n 1-1 1-n m-1 m-n
Domain % % ñ % % ū(m/n) % % ñ % % ū(m/n) Compute 16.75 0 \ 19.16 9.38 64.09 1.28 16.75 0 \ 19.16 9.38 64.09 1.28 Mechanis 17.84 0 \ 16.54 6.45 65.63 1.09 18.09 0 \ 16.9 6.17 65.02 1.11 Power 15.2 0 \ 17.14 7.8 67.67 1.18 15.29 0 \ 17.22 7.76 67.49 1.18 Publising 17.34 0 \ 29.54 19.48 53.12 1.33 18.42 0 \ 28.38 17.44 53.2 1.37 Forestry 15.47 0 \ 20.75 11.06 63.77 1.3 15.63 0 \ 21.17 10.79 63.2 1.33 Industrial 16.14 0 \ 23.63 9.78 60.23 1.14 16.37 0 \ 23.93 9.69 59.7 1.14 Math 22.77 0 \ 20.26 7.99 56.96 1.18 22.77 0 \ 20.26 7.99 56.96 1.18 Environe 15.32 0 \ 18.82 8.41 65.86 1.33 15.38 0 \ 18.89 8.39 65.72 1.33 Fisheries 13.17 0 \ 18.91 13.42 67.92 1.13 13.29 0 \ 19.18 13.14 67.53 1.15 Animal 11.82 0 \ 22.02 18.95 66.17 1.21 11.92 0 \ 22.38 18.58 65.69 1.23 Architecr 17.33 0 \ 18.66 9.77 64.01 1.37 17.33 0 \ 18.66 9.77 64.01 1.37 Aerospace 17.47 0 \ 19.65 8.04 62.88 1.3 17.86 0 \ 20.16 7.77 61.99 1.34 LightIndu 15.97 0 \ 19.71 11.66 64.32 1.32 16.32 0 \ 20.14 11.32 63.54 1.36 Finance 15.18 0 \ 27.65 27.43 57.17 1.37 15.18 0 \ 27.65 27.43 57.17 1.37 Physics 13.62 0 \ 13.91 6.89 72.47 1.19 13.62 0 \ 13.91 6.89 72.47 1.19 Traffic 17.01 0 \ 23.82 12.45 59.16 1.34 17.35 0 \ 24.43 12.12 58.22 1.39 History& 13.63 0.6 2.2 35.86 15.01 49.96 1 14.08 0.6 2.2 36.58 14.43 48.76 1.02 Agricue 13.04 0 \ 15.54 12.46 71.42 1.33 13.04 0 \ 15.54 12.46 71.42 1.33 Machiner 16.3 0 \ 18.14 8.48 65.55 1.33 16.53 0 \ 18.53 8.15 64.94 1.38 Chemistry 14.97 0 \ 15.21 9.67 69.82 1.35 14.97 0 \ 15.21 9.67 69.82 1.35 Law 17.86 0.2 1.8 38.36 15.23 43.62 1.05 18.24 0.2 1.8 39.83 14.86 41.76 1.07 Energy 14.2 0 \ 14.82 8.35 70.98 1.3 14.31 0 \ 14.95 8.32 70.74 1.31 Astronoy 15.51 0 \ 14.48 9.35 70.01 1.37 15.51 0 \ 14.48 9.35 70.01 1.37 Education 14.78 0 \ 25.54 21.36 59.68 1.11 14.78 0 \ 25.54 21.36 59.68 1.11 Power 16.53 0 \ 20.3 10.08 63.17 1.32 16.61 0 \ 20.41 10.06 62.98 1.32 Art- 10.87 0.1 2.4 32.59 23.02 56.43 1.08 11.43 0.1 2.4 33.87 22.42 54.58 1.1 Material 13.61 0 \ 14.13 6.05 72.26 1.12 13.65 0 \ 14.19 6.02 72.16 1.12 Metal 16.12 0 \ 17.45 9.13 66.43 1.34 16.12 0 \ 17.45 9.13 66.43 1.34 Electronic 16.4 0 \ 19.24 10.16 64.36 1.29 16.4 0 \ 19.24 10.16 64.36 1.29 Politics 13.57 0.1 1.6 34.55 25.23 51.79 1.09 14.11 0.1 1.6 34.75 23.88 51.04 1.12 Military 14.77 0 \ 32.22 27.33 53.01 1.08 15.54 0 \ 33.29 26.55 51.18 1.09
Medicine 12.14 0 \ 14.28 15.63 73.58 1.48 12.14 0 \ 14.28 15.63 73.58 1.48 Bioscie 12.74 0 \ 12.61 6.97 74.66 1.16 12.74 0 \ 12.61 6.97 74.66 1.16 Mining 18.1 0 \ 19.71 8.19 62.19 1.27 18.48 0 \ 20.22 7.88 61.3 1.32 Culture 8.78 0.1 1.6 28.73 21.88 62.42 1.05 9.01 0.1 1.6 29.42 21.4 61.49 1.07 Sport 17.64 0 \ 24.74 14.2 57.62 1.47 18.43 0 \ 24.95 13.39 56.62 1.52 Total 14.91 0 1.9 19.8 12.39 65.28 1.29 14.99 0 1.9 19.84 12.27 65.17 1.3
Top-40% tf*idf Top-30% tf*idf 1-1 1-n m-1 m-n 1-1 1-n m-1 m-n
Domain % % ñ % % ū(m/n) % % ñ % % ū(m/n) Compute 16.75 0 \ 19.16 9.38 64.09 1.28 16.86 0 \ 19.67 8.75 63.48 1.42 Mechanis 18.09 0 \ 16.9 6.17 65.02 1.11 18.76 0 \ 16.28 5.2 64.96 1.19 Power 15.29 0 \ 17.22 7.76 67.49 1.18 15.46 0 \ 17.5 7.14 67.05 1.26 Publising 19.67 0 \ 28.31 16.05 52.01 1.42 19.67 0 \ 28.31 16.05 52.01 1.42 Forestry 15.63 0 \ 21.17 10.79 63.2 1.33 15.63 0 \ 21.17 10.79 63.2 1.33 Industrial 17.01 0 \ 24.74 9.26 58.24 1.17 17.01 0 \ 24.74 9.26 58.24 1.17 Math 23.11 0 \ 20.36 7.37 56.53 1.26 24.02 0 \ 21.08 7.11 54.91 1.3 Environe 15.73 0 \ 18.95 7.91 65.32 1.38 15.73 0 \ 18.95 7.91 65.32 1.38 Fisheries 13.29 0 \ 19.18 13.14 67.53 1.15 13.4 0 \ 19.18 12.26 67.42 1.2 Animal 11.92 0 \ 22.38 18.58 65.69 1.23 11.9 0 \ 22.83 17.59 65.27 1.3 Architecr 17.33 0 \ 18.66 9.77 64.01 1.37 17.33 0 \ 18.66 9.77 64.01 1.37 Aerospace 17.86 0 \ 20.16 7.77 61.99 1.34 17.86 0 \ 20.16 7.77 61.99 1.34 LightIndu 16.32 0 \ 20.14 11.32 63.54 1.36 16.32 0 \ 20.14 11.32 63.54 1.36 Finance 15.18 0 \ 27.65 27.43 57.17 1.37 16.33 0 \ 24.76 21.34 58.91 1.56 Physics 13.62 0 \ 13.91 6.89 72.47 1.19 13.62 0 \ 13.91 6.89 72.47 1.19 Traffic 17.35 0 \ 24.43 12.12 58.22 1.39 17.35 0 \ 24.43 12.12 58.22 1.39 History& 14.5 0.6 2.2 37.46 14.28 47.44 1.01 14.5 0.6 2.2 37.46 14.28 47.44 1.01 Agricue 13.04 0 \ 15.54 12.46 71.42 1.33 12.85 0 \ 15.27 11.4 71.88 1.42 Machiner 16.53 0 \ 18.53 8.15 64.94 1.38 16.53 0 \ 18.53 8.15 64.94 1.38 Chemistry 14.97 0 \ 15.21 9.67 69.82 1.35 15.19 0 \ 14.65 8.22 70.16 1.49 Law 18.24 0.2 1.8 39.83 14.86 41.76 1.07 18.24 0.2 1.8 39.83 14.86 41.76 1.07 Energy 14.31 0 \ 14.95 8.32 70.74 1.31 14.31 0 \ 14.95 8.32 70.74 1.31 Astronoy 15.51 0 \ 14.48 9.35 70.01 1.37 15.51 0 \ 14.48 9.35 70.01 1.37 Education 14.78 0 \ 25.54 21.36 59.68 1.11 14.78 0 \ 25.54 21.36 59.68 1.11 Power 16.61 0 \ 20.41 10.06 62.98 1.32 16.61 0 \ 20.41 10.06 62.98 1.32 Art- 11.76 0.1 2.4 34.71 22.31 53.4 1.11 11.76 0.1 2.4 34.71 22.31 53.4 1.11 Material 13.58 0 \ 14.29 5.77 72.14 1.16 13.83 0 \ 14.5 5.56 71.67 1.17 Metal 16.12 0 \ 17.45 9.13 66.43 1.34 16.12 0 \ 17.45 9.13 66.43 1.34 Electronic 16.4 0 \ 19.24 10.16 64.36 1.29 16.4 0 \ 19.24 10.16 64.36 1.29 Politics 14.11 0.1 1.6 34.75 23.88 51.04 1.12 14.11 0.1 1.6 34.75 23.88 51.04 1.12 Military 16.46 0 \ 34.52 25.59 49.02 1.11 17.3 0 \ 35.84 24.7 46.86 1.16 Medicine 11.96 0 \ 13.86 14.22 74.18 1.63 12.14 0 \ 13.26 13.33 74.6 1.64 Bioscie 12.75 0 \ 12.59 6.54 74.66 1.2 12.89 0 \ 12.76 6.39 74.36 1.21 Mining 18.48 0 \ 20.22 7.88 61.3 1.32 18.48 0 \ 20.22 7.88 61.3 1.32 Culture 9.01 0.1 1.6 29.42 21.4 61.49 1.07 8.88 0.1 1.6 29.65 20.59 61.39 1.17 Sport 18.43 0 \ 24.95 13.39 56.62 1.52 18.43 0 \ 24.95 13.39 56.62 1.52 Total 14.99 0 1.9 19.87 12 65.13 1.32 15.15 0 1.9 19.65 11.48 65.19 1.36
Top-20% tf*idf Top-10% tf*idf 1-1 1-n m-1 m-n 1-1 1-n m-1 m-n
Domain % % ñ % % ū(m/n) % % ñ % % ū(m/n) Compute 17.52 0 \ 20.37 8.41 62.12 1.49 18.08 0 \ 21.09 8.13 60.82 1.57 Mechanis 19.42 0 \ 16.86 4.93 63.71 1.23 19.78 0 \ 17.19 4.88 63.03 1.24 Power 16.33 0 \ 18.31 6.8 65.36 1.31 16.33 0 \ 18.31 6.8 65.36 1.31 Publising 19.67 0 \ 28.31 16.05 52.01 1.42 23.61 0 \ 32.29 14.16 44.1 1.8 Forestry 15.63 0 \ 21.17 10.79 63.2 1.33 16.36 0 \ 22.78 9.64 60.86 1.52 Industrial 18.14 0 \ 26.14 8.51 55.72 1.26 19.79 0 \ 27.81 8.08 52.4 1.3 Math 24.63 0 \ 21.57 6.91 53.79 1.33 25.29 0 \ 22.17 6.58 52.54 1.43 Environe 15.63 0 \ 19.23 7.51 65.14 1.49 16.76 0 \ 20.47 7.2 62.76 1.55
Question Type Patterns
Who
Who published a paper titled B in journal C? With whom A published a paper titled B in journal C? With whom A published a paper titled B? With whom A published a paper in journal C?
When
When A published a paper titled B in journal C? When A published a paper titled B with C in journal D? When A published a paper titled B? When A published a paper in journal C? When A published a paper titled B with C?
What
What is/are the research interests of A? What are the papers’ titles that A published with B in journal C? What are the papers’ titles that A published with B? What are the papers’ titles that A published in journal C?
Where
Where A published a paper titled B with C? Where A published a paper titled B? Where A published a paper with C?
all authors 1044 622 129 77 113 61 1027 573 publi journal 110816 112490 47952 47978 6846 6981 82985 84539 translation of 5869 7180 2218 2394 275 271 3724 4352 Total 248236 2977132 122410 1376827 35077 402398 193004 2218082
Math-related Military Machinery Architecturerelated
Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri# author of 50457 64322 7189 11380 208939 350641 216669 345047 first author 36614 24230 4257 2710 150981 104146 157830 110878 second author 35835 24732 4148 2654 149407 107603 156400 110076 co author 23990 48284 4244 17599 89431 353023 93073 342245 research inter 29960 113592 5906 20623 115023 757508 140007 945441 affiliation 25131 79075 2980 8077 50380 104554 60345 162820 belged domain 9424 9423 3624 3624 29674 29673 52173 52172 cooccurrence with 9318 67567 3585 35216 29531 425569 52082 1000942 published year27933 28091 3408 3467 84763 85741 29059 29332 contained cky 45 26 7 6 99 62 164 99 contained eky 0 0 11 10 4 2 7 6 other author 21609 15366 5006 6016 140709 138908 131650 124143 all authors 278 180 3 2 537 308 1499 870 publi journal 25139 25229 3402 3405 115594 116343 119422 121800 translation of 1029 1640 236 231 5124 5833 5857 7066 Total 65557 501757 11975 115020 247582 2579914 283294 3352937
Agriculturerelated
Electric industry
Forestry Material science
Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri# author of 367390 792108 212926 367206 77392 132902 55966 87833 first author 261789 186313 155437 106513 51430 34903 34542 21506 second author257138 187965 153910 112630 50427 34875 33157 21964 co author 169794 1113293 90428 387883 38493 176327 32759 121330 research inter202184 1872518 123461 802381 48104 285167 36972 165228 affiliation 129799 445648 73760 208663 16711 32580 30109 78613 belged domain47621 47621 40001 40001 12486 12486 6570 6570 cooccurrence with 47574 1414785 39921 655854 12421 185877 6462 46256 published year83455 84388 83793 85321 40146 40464 23019 23098 contained cky 257 181 229 140 32 22 33 26 contained eky 2 1 21 13 0 0 0 0 other author 293233 417871 135632 148081 55749 63134 41114 44363 all authors 448 275 1052 663 131 67 69 52 publi journal 185995 188968 118211 119924 35667 35861 23272 23297 translation of 9247 13648 10807 12282 843 971 1484 1633 Total 432620 6765583 264637 3047555 93130 1035636 65759 641769
Art-related Power Industry Mechanics Aerospace Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri#
author of 4171 3978 92715 154519 29192 40268 77208 123739 first author 2111 1273 59953 39363 19759 12847 54999 37038 second author 2005 1220 58746 39558 19306 13102 54055 38465 co author 2908 9172 50214 218643 15071 39334 35253 122117 research inter 13016 35047 61792 294510 16752 58734 43249 209388 affiliation 265 258 44098 117659 14122 36477 23854 56031 belged domain14033 14033 15261 15260 3461 3461 12136 12136 cooccurrence with 14030 455486 15021 133627 3395 19252 12001 118672 published year 1767 2623 32003 32290 14558 14546 16775 17020 contained cky 269 209 35 19 2 1 40 26
contained eky 0 0 0 0 0 0 0 0 other author 1713 1494 66117 75609 17115 14321 50411 48242 all authors 338 206 248 134 33 27 130 72 publi journal 3199 4095 44098 44392 10571 10565 40894 41050 translation of 144 140 1724 1887 422 585 927 958 Total 22483 529234 114666 1167470 34549 263520 93319 824954
Bioscience Animal Husbandry
Energyrelated
Mining Industry
Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri# author of 186930 339841 89700 178260 143319 246040 101033 151265 first author 120526 79421 57976 38999 97315 66082 69664 46699 second author118260 80497 56369 39339 96722 65572 67909 45768 co author 103173 532748 47741 286241 74251 330765 48093 162094 research inter116001 693164 57385 378149 92585 734045 65591 337779 affiliation 93517 296214 26312 59685 64177 241310 24135 46338 belged domain17459 17459 18040 18039 19601 19600 19216 19216 cooccurrence with 17379 210251 18028 445930 19527 379263 19115 227004 published year69801 70082 40104 42168 57474 58093 37938 38336 contained cky 93 56 116 77 31 23 15 10 contained eky 2 1 11 10 0 0 14 13 other author 142871 179933 71255 99945 105059 114421 63771 58803 all authors 220 161 169 93 341 188 413 223 publi journal 81627 81757 41938 44295 68590 69005 53721 54195 translation of 4966 6389 1832 2106 2749 3493 1434 1577 Total 216087 2587974 111528 1633336 172372 2327900 126860 1189320
Astronomy Culture-related Light Industry
Social Science & Philosophyrelated
Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri# author of 261764 537706 12507 12166 156382 262230 250650 295782 first author 196980 143830 8452 5344 111030 77888 188027 126276 second author192263 144006 7322 4572 108961 77797 180403 119382 co author 108822 671164 6038 16275 70892 281892 99977 199917 research inter142647 1223155 22498 70632 98356 647939 221173 1430544 affiliation 65533 211438 1005 1112 50855 133089 72759 143853 belged domain38159 38158 24306 24306 35396 35395 106131 106131 cooccurrence with 38070 550541 24294 1338615 35273 652721 106042 3209705 published year33294 33581 8864 13914 25522 26117 2102 2399 contained cky 188 116 427 327 162 105 282 194 contained eky 9 6 0 0 4 2 11 7 other author 197176 249897 2815 2265 101604 106589 66827 50134 all authors 517 337 1289 770 1348 830 6744 3843 publi journal 150371 150845 11924 22092 84539 86691 152525 158459 translation of 6515 8016 1225 1204 4972 5814 20690 37479 Total 310418 3962796 43321 1513594 202297 2395099 402514 5884105
Education Traffic Transportation
Industrial Technology
Finance
Ent# Tri# Ent# Tri# Ent# Tri# Ent# Tri# author of 76464 80324 145027 215219 81973 110204 131243 165185 first author 53592 34260 105535 71575 50893 31919 97488 67219 second author 50198 31895 103962 72269 50234 32639 94155 64920 co author 30647 51924 62605 195227 44528 129752 50958 115497 research inter 75003 359370 92660 549797 48267 160936 101293 595651 affiliation 11306 14084 40018 89779 34034 68532 27167 46638 belged domain61161 61162 36162 36162 13151 13150 71913 71912
cooccurrence with 61148 3769343 36031 684482 12903 124291 71903 3304046 published year12011 15824 85806 88376 34538 35107 5562 6733 contained cky 1663 1249 130 90 49 27 590 393 contained eky 0 0 2 1 2 1 2 1 other author 20260 14171 82256 71396 51233 45651 44480 33057 all authors 3575 2241 1130 628 148 85 2926 1645 publi journal 53197 76178 84185 85901 32994 33252 79535 87870 translation of 5071 6902 4487 5025 1071 1138 5415 6601 Total 157465 4518927 192028 2165927 101779 786684 220977 4567368
Computer science
Chemistry
Ent# Tri# Ent# Tri# author of 216180 348282 397169 795356 first author 159209 109004 295670 212199 second author158493 116608 291230 211992 co author 89562 314808 173179 943688 research interest117962 793333 212420 1881048 affiliation 75704 216821 153608 527827 belged domain34291 34290 43944 43944 cooccurrence with 34230 574754 43861 892135 published year102943 103891 113915 116198 contained cky 138 90 310 206 contained eky 21 15 4 2 other author 131788 122691 293307 371261 all authors 515 289 2394 1516 publi journal 126325 127415 221285 224906 translation of 9415 11325 11978 17717 Total 262255 2873616 462361 6239995

Review summaries:
This paper introduces a large-scale knowledge graph database called TechKG, which is constructed from a massive repository of academic papers in Chinese. The authors have described in details the process and heuristics in use for constructing such knowledge base, and also reported important statistics and characteristics of the database, including duplicate name, imbalance issue etc. The main contribution is to provide a KG in Chinese, which wasn’t widely available compared to popular languages such as English.