Title: TECHKG: A LARGE-SCALE CHINESE TECHNOLOGY-ORIENTED KNOWLEDGE GRAPH

Abstract: Knowledge graph is a kind of valuable knowledge base which would benefit lots of AI-related applications. Up to now, lots of large-scale knowledge graphs have been built. However, most of them are non-Chinese and designed for general purpose. In this work, we introduce TechKG, a large scale Chinese knowledge graph that is technology-oriented. It is built automatically from massive technical papers that are published in Chinese academic journals of different research domains. Some carefully designed heuristic rules are used to extract high quality entities and relations. Totally, it comprises of over 260 million triplets that are built upon more than 52 million entities which come from 38 research domains. Our preliminary experiments indicate that TechKG has high adaptability and can be used as a dataset for many diverse AI-related applications.
Under review as a conference paper at ICLR 2020 YAGO3-10 Mahdisoltani et al. ( 2015) is a subset of YAGO3 and is also widely used in many applications. It consists of entities which have a minimum of 10 relations each. Totally, it has 123,182 entities and 37 kinds of relations. Most of the triplets deal with descriptive attributes of people, such as citizenship, gender,profession, etc.
NOTE that here we copy these KGs' statistics directly from their corresponding reference papers. There may be some differences for these statistics because almost all of these KGs are continually updating.
Every day, there are massive Chinese technical papers published in journals of different research domains. These papers often have following characteristics.
First, there is a clear structure for each of these papers. For example, each paper often has such components like: a title, an author list, an authors' affiliation list, keyword list, abstract, etc.
Second, each technical paper can be accurately classified into a proper research domain according to the journal where this paper is published. This is because that each academic journal has its own scope for submissions. Taking "Chinese Journal of Computers" for example, it only focuses on the latest research in computer science domain. And it would never accept a paper in chemistry or artist domain. In other word, as long as we know the focused research domain of a journal, we can confidently classify all of its accepted papers into this research domain. There may be some exceptions, but the proportion would be very small. These characteristics inspire us that we can use technical papers as data source to build a large scale Chinese KG that is technology-oriented.
We take technical papers as data source to build TechKG. These papers were mainly collected from journals' websites. According to the submission scopes of the journals 1 , we classify all of them into 38 types: each of which corresponds to a specific research domain.
Under review as a conference paper at ICLR 2020
The role of relations is to describe the attributes of entities and the relationships between entities. In TechKG, keywords and authors are two kinds of entities with the largest amounts. So we define relations mainly centered upon them. Besides, papers themselves contain much valuable factual knowledge. So we define some paper-centered relations to describe the attributes of papers themselves. Totally, 16 relations are defined in TechKG.

1. INTRODUCTION:

Generally, a knowledge graph (KG) stores factual knowledge in the form of structural triplet like <h, r, t>, in which h and t are entities (called head entity and tail entity respectively) and r denotes the relation type from h to t. KG is a kind of valuable knowledge base and is important for many different kinds of AI-related applications. Nowadays, great achievements have been made in building large scale KGs. And there are many available KGs like FreeBase Bollacker et al. (2008) , WordNet Miller (1995) , YAGO Suchanek et al. (2007) , etc. Usually there are millions of entities and billions of relational facts in these KGs. With them, great progress have been made in lots of AI-related applications, such as knowledge graph embedding (KGE), distantly supervised relation extraction (RE), knowledge based question and answering (KBQA), etc. However, there are still following two challenges inherited in these KG-dependent applications.

First, almost all of these KGs can only be used in English-related applications. But the fact is that each language has its own characteristics. It is uncertain whether a model that performs well in an English-related KG-dependent task could still perform well in the corresponding task of other language. Thus there is a pressing need to build KGs of different languages other than English.

Second, most of existing KGs are designed for general purpose. It is also uncertain whether a model that performs well in an application of general domain could still perform well in applications of specific domains, such as in the technical domains liking medicine, biology, electronics, etc. It is necessary to evaluate whether there are some new kinds of characteristics hidden in KGs that haven't been noticed by researchers up to now.

To address these issues, this work introduces TechKG, a large scale Chinese KG built from massive technical papers that are published in Chinese academic journals of different research domains. Compared with existing KGs, TechKG has following important characteristics.

First, TechKG is a large scale Chinese KG that is technology-oriented. Totally, it comprises of over 280 million triplets that are built upon more than 60 million entities which come from more than 30 research domains. To the best of our knowledge, this is the first Chinese KG that is built by using multi-domain technical papers.

Second, TechKG is a high quality KG. Both of its entities and relations are acquired in two ways. One is directly taking some components of a paper as entities and building relations between entities by simply re-organizing a paper's components. The other is taking carefully selected domain terminologies as entities and building relations between them by some carefully designed heuristic rules. These two ways guarantee TechKG have high accuracies at both entity and relation levels.

Third, TechKG can provide more kinds of information associated with the original papers besides common triplets. This makes it can be easily taken as a benchmark dataset for many diverse AIrelated applications, such as KGE, distantly supervised RE, NER, KBQA, text classification(TC), machine translation(MT), among others.

Fourth, TechKG can be continually updated as long as there are new available published papers. As the needed data source is readily available in almost all languages, researchers can easily build similar KGs for other languages.

In the rest of this paper, we first introduce the building process of TechKG. Then we analyze the characteristics of TechKG in detail. We also conduct some primary experiments using TechKG as a dataset to three diverse tasks (see the Appendix part for detail). Both our analyses and experimental results show that lots of characteractics distinguish TechKG from existing KGs and it raises many new challenges for lots of existing applications.

2. RELATED WORK:

At present, there are many available KGs, such as SUMONiles and Pease (2001), Cyc Matuszek et al. (2006) , FreeBase Bollacker et al. (2008 ), WordNet Miller (1995) , YAGO Suchanek et al. (2007) , YAGO2 Hoffart et al. (2013) , YAGO3 Mahdisoltani et al. (2015) , etc. Among them, FreeBase, Word-Net, YAGO and some of their subsets are the most widely used KGs in many diverse applications. We will introduce them in the following.

FreeBase Bollacker et al. ( 2008) is a database system designed to be a public repository of the word's knowledge. Its design is inspired by Semantic Web and Wikipedia. It tries to merge the scalability of structured databases with the diversity of collaborative wikis into a practical and scalable database of structured general human knowledge. Totally, it contains more than 100 million asserts about topics spread over 4000 types, including people, media, locations, and many others.

FB15k Bordes et al. (2013) is a subset of Freebase. It is one of the most widely used datasets for many AI-related tasks. Totally, it contains 14,951 entities, 592,213 triplets, and 1,345 different relations. A large fraction of the triplets in this KG describe facts about movies, actors, awards, etc.

WordNetMiller (1995) is an electronic lexical database that is considered to be the most important resource available to researchers in computational linguistics, text analysis, and many related areas. Its design is inspired by psycholinguistic and computational theories of human lexical memory. In WordNet, English nouns, verbs, adjectives, and adverbs are organized into synonym sets, each represents one underlying lexicalized concept, and semantic relations link the synonym sets. Specifically, WordNet includes some kinds of semantic relations like synonymy, antonymy, hyponymy, memonymy, troponymy, entailment, etc.

WN18 Bordes et al. (2013) is a subset of WordNet. Like FB15k, it is widely used in many AI-related tasks. Totally, it consists of 18 kinds of relations, 40,943 entities, and 151,442 triplets. Most of its triplets consist of hyponym and hypernym relations. 2007) is a light-weight and extensible ontology with high coverage and quality. It contains more than 1 million entities and 5 million facts. These facts are expressed by relations and are automatically extracted from Wikipedia and unified with WordNet by a carefully designed combination of rule-based and heuristic methods. In YAGO, the relations include the Is-A hierarchy as well as non-taxonomic relations like HasWonPrize, BornInYear, DiedInYear, etc.

YAGOSuchanek et al. (:

YAGO2 Hoffart et al. (2013) is an extension of YAGO knowledge base, in which entities, facts, and events are anchored in both time and space. YAGO2 is built automatically from Wikipedia, GeoNames, and WordNet. It contains 447 million facts and about 9.8 million entities.

YAGO3 Mahdisoltani et al. (2015) is another extension of the YAGO knowledge base. It combines the information from the Wikipedia in 10 languages including English, German, French, Dutch, Italian, Spanish, Romanian, Polish, Arabic, and Farsi. The choice of languages was determined by the proficiency of the authors, so as to facilitate manual evaluation. Totally, there are 8,936,324 triplet facts built upon 4,595,906 entities.

For simplicity2 , we omit the body part of a paper and only collect its: title, author list, authors' affiliation list, keyword list, and abstract. If a paper's these components also have English translations, the corresponding English parts will also be collected.

Totally, over 70 million technical papers are collected.

3.2. ENTITY SELECTION:

In TechKG, each component of a paper except abstract is taken as an entity. That is to say, title, author, keyword, and affiliation, either in Chinese or English, are all taken as entities. And the name of each research domain is also taken as an entity.

Among the obtained entities, keywords account for a very large proportion. We had expected that all of them were high quality domain terminologies, but there are lots of exceptions. For example, in some papers of the computer science domain, there are keywords like "three phases", "two steps", "generation", etc. Obviously, these keywords are not good domain terminologies for they have less to do with computer science. So we use atf*idf method to filter such kind of keywords. Specifically, we concatenate all the keyword lists in each research domain to form a long domain-related document. Then the tf value of a keyword in each domain can be computed. Different domain-related documents are viewed as the counterpart of each other, thus the idf value of a keyword can be computed accordingly. The higher a keyword's tf *idf value, the more possible this keyword is a good domain terminology.

3.4. BUILDING RESULTS:

Finally, more than 52 million entities and more than 260 million triplets are extracted. Some basic statistics of TechKG are shown in Table 1 . It should be noted that entities maybe overlapped in different kinds of relations. So in Table 1 , the total entity number is less than the sum number of entities in all relations. Statistics of the "hierarchical" relation are not listed in Table 1 because they depend on the predefined threshold. One can see the detailed statistics about each relation (including the "hierarchical" relation) in the Appendix part.

Currently, we don't mine complex relations for TechKG because we put more emphasis on its accuracy. We can see that most of the triplets in TechKG are generated by re-organizing the meta-items of the original papers. Thus, TechKG is a large-scale KG that has high accuracy for both its entities and relations. Besides, the tf*idf method in entity selection and the threshold method in relation extraction can further guarantee TechKG's accuracy.

4. TECHKG CHARACTERISTICS:

We find there are three main characteristics that distinguish TechKG from most of existing KGs: the duplicate name issue, the imbalance issue, and the good application adaptability.

4.1. DUPLICATE NAME ISSUE:

In TechKG, duplicate name is a very common phenomenon: not only some authors share some common names, but also some research domains also share some common terminologies. For example, Ning Li, a researcher name that appears in 34 research domains. That is to say, there is a researcher named Ning Li in almost all of research domains. We call this kind of duplicate names as cross-domain duplicate name issue. 52,716,813 260,763,332 In Table 2 , we report some statistics for the duplicate name issue of authors. Here dpa means the average appeared domains per duplicated author. It should be noted that here we ignore nature science and social science during making statistics. Both of them are comprehensive research domains and it is very likely to generate some false duplicated author names if we take them into consideration.

For example, a researcher may publish his paper in both computer science domain and in nature science domain.

However, when we make statistics according to research domain, we find both the duplicate ratio and the dpa increase greatly (see the Appendix part for detail). These statistics mean that there are a very small number of duplicated names that appear in many different research domains. Or most of the duplicated names appear in a relative smaller research domain set. To verify this hypothesis, we further make statistics of the authors' domain distribution. The results are shown in Table 3 .

From Table 3 we can see that more than 80% of the duplicated authors appear in less than 5 research domains. There are 7,769 duplicated authors that appear more than 20 research domain. More extremely, we find that there are 167 author names (such as: Lin Zhang, Hua Liu, Yang Wang, Lei Zhang, etc) that appear in all research domains. These results well verify our hypothesis and can explain the reason of why both the duplicate ratio and the dpa increase greatly in separated research domains. There is not only cross-domain duplicate name issue, but also another kind of in-domain duplicate name issue. For example, in the research domain of computer science, there will be many duplicated authors that are in different affiliations. Among the in-domain authors that have several affiliations, some of them are real duplicated names, some are not. Although the entity linkage (or entity resolution) issue has been well studied, it is so hard that it is almost impossible for us to accurately pick up the true in-domain duplicate names due to the following two reasons.

First, there may be some researchers that work in different affiliations in different times.

Second, some authors provide their affiliations with different granularity in different papers. For example, an author may write his affiliation as "Northeastern University", "School of Computer Science and Engineering, Northeastern University", or even "Natural Language Processing Lab, Northeastern University", etc.

Here we try to provide a rough extreme analysis of this in-domain duplicate name issue.

For the worse case, as long as an author name appears in different affiliations, we take it as a duplicated name.

For the better case, for an author name appears in different affiliations, we take it as a duplicated name only when there is no inclusion relationship between these affiliations.

In Table 4 , we report the in-domain duplicate name statistics for metallurgical industry, computer science, electric industry, and traffic transportation, 4 randomly selected research domains. Here apa means the average appeared affiliations per duplicated author.

From Table 4 we can see that the in-domain duplicate name issue is very serious. In both cases, more than 50% authors will have multi affiliations. An interesting finding is that the duplicate ratio of the better case is lower than the worse case, but the apa value of the better case in larger than the worse case. The former results are in line with our expectations, but the latter are not. This means the filtered authors have fewer affiliations. In other words, authors are more likely to use fewer numbers of affiliations with different granularity.

Here we are keenly aware that both of our two rules for the extreme analysis can be criticized for there are lots of exceptions for each of them. We leave the accurate statistics of this issue as an open issue and to solve it in the future. For keywords, they will only involve the cross-domain duplicate name issue. The number of duplicated keywords will be affected by the predefined tf*idf threshold. We make statistics under different tf*idf settings and the results are shown in Table 5 . Here dpk means the average appeared domains per duplicated keyword. We also make statistics of the keywords' domain distribution as we made for authors. But different from authors' statistics, keywords' statistics will depend on the tf*idf settings. Here report the domain distribution of keywords under top-10% tf*idf setting in Table 6 for metallurgical industry, computer science, electric industry, and traffic transportation, 4 randomly selected research domains. From Table 6 we can see that compared with the domain distribution of authors, the keywords' domain distribution is more balanced: most keywords appear less than 5 research domain. Generally, the more strict a tf*idf required, the less possible for a keyword appears in different research domains. For example, under the tf*idf setting of above top-80%, more than 90% keywords appear less than 5 domains (see the Appendix part for detail). But there are some keywords that appear in many research domains. For example, under the tf*idf setting of top-10%, there are still many keywords appearing in more than 20 research domains, such as genetic algorithm, numerical simulation, finite element method, preventive measure, etc.

4.2. IMBALANCE ISSUE:

TransH3 classifies a KG's relations into 4 types: 1-1, 1-n, m-1, and m-n. Following this definition, we make statistics for the proportion related triplets in different type of relations under different tf*idf settings. For the latter 3 types of relations, we also make statistics of the average value of n and the average value of m/n. The statistics results are shown in Table 7 .

From Table 7 we can see that the triplet distribution for these 4 relation types is extremely imbalanced. In each tf*idf setting, the m-n relations account for more than 60% triplets. On the other hand, the 1-n relations account for so small a proportion that they can even be ignored. The proportion is almost unchanged for different type of relation under different tf*idf settings. When we look into the average number of tails per head (ñ in Table 7 ) or the average number of head per tail ( m or ū in Table 7 ), the difference among different types of relations is far huge: in m-1 relations, the average value of m is more than 10, but both the average value of n in 1-n relations and the average value of m/n in m-n relations is relative smaller. These average values are also almost unchanged for different type of relation under different tf*idf settings.

The imbalance issue absolutely will have some impacts on the performance of some applications like knowledge graph embedding, etc. In the experiment section, we will further discuss these impacts. Thus different kinds of transfer learning methods can also be evaluated by these subsets.

4.3. APPLICATION ADAPTABILITY:

Domain Terminologies. In TechKG, a large proportion of its entities are domain terminologies.

Thus a new kind of named entity recognition (NER) task that aims to recognize domain terminologies from text can be evaluated with these domain terminologies. In such a NER task, the required training data can be obtained by the distantly supervision based method as used in Yang et al. (2018) .

Cross-language KG based Applications. The translation of relation in TechKG provides a possibility to link KGs of different languages together. This would benefit lots of cross-language KG based applications like cross-language Q&A, cross-language information retrieval, etc.

Finally, we also construct following kinds of knowledge bases (KBs) together with TechKG. Each of these KBs can be viewed as a by-product of TechKG and can be taken as a dataset for some applications.

4.3.1. TECHTERM/TECHBITERM:

They are two KBs that consist of Chinese domain terminologies and Chinese-English terminology pairs respectively.

For the former one, we randomly select 10,000 terminologies with the highest tf*idf values for each research domain.

For the latter one, we randomly select 10,000 bilingual translation pairs with the highest cooccurrence values from each research domain.

4.3.2. TECHABS:

It is a KB that stores papers' abstracts. It can be used as a dataset for a text classification task. Here we randomly select 100,000 abstracts from each research domain to construct TechAbs.

4.3.3. TECHQA:

TechQA is a dataset designed for the KBQA task. It is constructed based on TechKG-10. All the questions in it are generated by some predefined patterns. Currently, it focuses on 4 kinds of questions which are "who", "what", "when", and "where".

We list the used question patterns and detailed statistics of TechQA in the Appendix part.

4.3.4. TECHRE/TECHNER:

They are two KBs designed mainly for relation extraction (RE for short) and domain terminology recognition tasks respectively. The former one is constructed based on TechKG-10 (a subset of TechKG, see the Appendix part for detail) and the latter one is constructed based on TechTerm.

RE aims to extract semantic relations between entity pairs from plain text. It is a typical classification task when the entity pairs and their possible relations are specified. To achieve high quality results, a large-scale training corpus is required. However, it is very time-consuming and expensive to construct such a corpus manually. To address this issue, distant supervision is proposed Mintz et al. (2009) to automatically generate training data via aligning KBs and texts. Its basic idea is that: if two entities have a relational fact in KBs, all the sentences that contain these two entities will express the corresponding relation. Accordingly, these sentences can be taken as positive samples for training this relation. Distant supervision provides an effective strategy to automatically construct large-scale training data for a RE task. Here TechKG-10 is used as the required supervision KB and papers' abstracts are used as the required text corpus. It should be noted that except keywords, it is little possible for other types of components occur in the abstract part of a paper. So in TechRE, only hierarchical relation is considered. We set the co-occurrence threshold to 10 for the extraction of hierarchical relation. Specially, for each two terminologies in a co-occurrence with type of triplet, we first pick up all the sentences that both of these two terminologies appear. We call each terminology pair along with its mentioned sentence set as a bag. If the co-occurrence value of a bag's terminology pair is greater than 10, we assign a hierarchical relation to this bag and take it as a positive hierarchical training sample. Otherwise a NA relation is assigned to this bag and it is taken as a negative hierarchical training sample. By this way, we extract more than 14M training bags. Averagely, there are about 42 sentences per bag and 5 entities per sentences. The whole dataset is too large. So we randomly select at most 200,000 training bags for each research domain to construct TechRE. Besides, we limit the average sentences per bag in TechRE to 6 at most.

For TechNER, we also adopt a distant supervision method as TechRE, but TechTerm is used as the required KB. Papers' abstracts are taken as the required text corpus. If a sentence contains a terminology, this sentence will be regarded as a positive training sample for recognizing this terminology. This distant supervision method for constructing NER training corpus is also used in Yang et al. (2018) . By this way, we extract more than 210M training sentences. Averagely, there are about 3 entities per sentences. We randomly select 30,000 training samples in each research domain to construct TechNER. There are averagely 3 entities per training sample. In TechNER, if there are nested domain terminologies in a sentence, we will only remain the longest one.

The detailed and complete statistics of TechRE and TechNER can be found in the Appendix part.

5. CONCLUSIONS AND FUTURE WORK:

In this paper, we introduce TechKG, a large-scale Chinese KG. To the best of our knowledge, TechKG is: both the first KG that is constructed from massive technical papers and the first KG that is technology-oriented and can be divided into different subsets according to the research domains.

We analyze the main characteristics of TechKG. Lots of statistics show that there are many new characteristics hierent in TechKG distinguish it from other existing KGs. We also conduct some primary experiments (see the Appendix part for detail) to evaluate the adaptability of TechKG. Experiments show that: 1)TechKG provides a new choice for lots of applications; and 2)a lot of new characteristics hierent in TechKG raise new challenges for many existing methods.

Currently, we generate TechKG mainly by re-arranging the meta-items in the collected technical papers. In the future, we will explore to extract more types of relations such as <method, solve, problem>, etc.

A EXPERIMENTS AND ANALYSES:

To evaluate the adaptability of TechKG, we apply it to three completely different applications: knowledge graph embedding, distantly supervised RE, and distantly supervised NER. For each of them, we select one of its state-of-the-art methods as baseline and use these baselines to conduct experiments on TechKG104 , TechRE, and TechNER respectively. For each application, we report its experimental results on 4 randomly selected research domains that are metallurgical industry, computer science, electric industry, and traffic transportation.

NOTE that the aim of our experiments is not to tell whether a model is good or not. On the contrary, we just want to find whether our TechKG could show some new characteristics hidden in KGs that haven't been noticed up to now. A.1 KNOWLEDGE GRAPH EMBEDDING KGE projects each item (entities and relationships) of a KG into continuous low-dimensional space(s). It aims to solve two major challenges that prohibit the availability of KGs. One is the incompleteness issue: although most of existing KGs contain large amount of triplets, they are far from complete. The other is the computation issue: most of existing KGs are stored in symbolic and logical formations while applications often involve numerical computing in continuous spaces.

Link prediction, which is to predict the missing h or t for a correct triplet <h, r, t>, i.e., predict t given <h, r> or predict h given <r, t>, is often used to evaluate the performance of a KGE method. Rather than requiring one best answer, link prediction emphasizes more on ranking a set of candidate entities from the KG. Hits@10, the proportion of correct entities ranked in top 10, is often used as the evaluation metric for link prediction.

In this paper, we select ConvEDettmers (2018), one of the latest state-of-the-art KGE methods as baseline. We want to find: 1) whether the duplicate name issue would affect the performance; 2) whether the imbalance issue would affect the performance; and 3) whether TechKG also suffers from the test set leakage issue that is severe in WN18 and FB15k Dettmers (2018) .

For these aims, we report the experimental results on following kinds of datasets.

1) The original TechKG10 dataset.

2) A revised TechKG10 dataset that no cross domain author duplicate names included.

3) A revised TechKG10 dataset that no in domain author duplicate names included. 4) A revised TechKG10 dataset that no cross domain keyword duplicate names included. 5) A revised TechKG10 dataset that no any duplicate names included. 6) A revised TechKG10 dataset that no relations of co ocurrence or research interest. 7) A revised TechKG10 dataset5 that no relations of co ocurrence or research interest and no any duplicate names included.

Even in TechKG10, the triplet number of its each domain is far larger than FB15k. So in experiments, we randomly select 30,000 triplets for each of above 7 datasets. The number of triplets used here is less than that of FB15k: there are about 48,000 triplets in FB15k. However, for each of these 7 datasets, its involved parameter size is far larger than FB15k. FB15k has 14951 entities and 1345 relations, but the average number of contained entities in these 7 datasets is about 140,000. This means it needs more time for training and testing on these 7 datasets. In experiments, the proportion of training, development and test sets is 8:1:1. The experimental results are shown in Table 10 and Figure 1 , from which we can find following observations. During training, we iterate 300 rounds for each experiment.

First, compared with FB15k or WN18, the performance on TechKG10 is far lower. We think this is caused mainly by the data sparsity issue. Compared with other datasets, there are more parameters needed to be trained, but the number of training triplets is less.

Second, there is almost no the test set leakage issue at all in TechKG, which means it is impossible for a simple rule-based inverse model to achieve state-of-the-art results on TechKG.

Third, both the cross-domain and in-domain author duplicate name issues have a heavily negative effect on the performance. When there are no any duplicated names in the data set, each domain' performance increases greatly. Taking metallurgical industry for example, the Hits@10 value on the original dataset is only 0.051, but this value increases to 0.389 when all the duplicated items are removed from the dataset. What's more, the cross-domain keyword duplicate name issue has more negative effect on the performance, which is not in line with our original expectation. The reason of this result should be further studied.

Fourth, the imbalance issue has a heavy negative effect on the performance. When we remove two types of relations from the dataset, the performance obtains a huge improvement. This indicates that the more balanced a dataset's relation types' distributions, the better performance could be obtained.

Fifth, there are obvious performance gaps between different kinds of research domains. For example, when there is no any duplicated names, the Hits@10 value of the metallurgical industry domain is 0.389, which is much higher than that of the computer science domain, which is only 0.308.

Sixth, when there are no any duplicated names and the two special relations are removed, the best results are obtained, which is similar with that of on the datasets of WN18RR and FB15k-237.

Based on these observations we can roughly draw following conclusions. First, TechKG10 is a more challenging dataset than traditional WN18 and FB15k. Second, both the duplicate name issue and the imbalance issue are two inherent characteristics in TechKG.

A.2 DISTANT SUPERVISION RELATION EXTRACTION:

In this task, we select the "PCNN+ATT" Lin et al. ( 2016) model as the baseline method. Following two main factors are considered for this selection. First, it is a state-of-the-art RE method and many current RE methods are based on it. Second, there are available source codes for it, which is very important for building a RE system quickly.

In almost of all of current RE research, Freebase is used as the distant supervision KB and the threeyear NYT corpus from 2005 to 2007 is used as the text corpus. In NYT dataset, there are about 281,270 training bags and 96,678 test bags. The size of TechRE is a little smaller than that of NYT. However, if taking the factor of sentences per bag into consideration, we can find that the parameter sizes of our datasets are far larger than NYT.

The average number of sentences per training bag in NYT corpus is about two, while this number is more 60 in TechRE. To evaluate whether this factor will affect the performance, we report the experimental results under different average number of sentences per training bag. There is also a huge difference between the proportion of NA relations in TechRE and NYT. To evaluate whether this factor will affect the performance, we also report the experimental results under different proportions of NA relations. During training, we iterate 100 rounds for each experiment. Finally, the p@m results are shown in Table 11 , Figure 2 , and Figure 3 . From these results we can find following observations.

First, both the average sentence per bag and the NA relation proportion have effects on the performance of RE task. For some domains, the higher the average sentence per bag, the poorer performance would be obtained. But for other domains, there is an opposite conclusion. For the factor of the NA relation proportion, we find that when we reduce the proportion of NA relations, the performance decrease accordingly. Here not all of the experimental results are in line with our expectation. The reasons should be further studied.

Second, there is a big performance gap between NYT dataset and TechRE in both the precision/recall curves and the p@m: the performance on TechRE is far lower than that of on NYT. We find that in TechRE, there are almost no any cues in the training sentences that can imply a kind of relations. In NYT, however, there are words like "caused", "because", "contain", "have", etc.

Obviously, these words are very helpful for the relation decision. In fact, we think TechRE is a more challenging dataset for the RE task: there are not always useful cues available.

Third, we find that there are also huge performance gaps between different kinds of domains. For example, under the setting of 2 sentences per bag, the performance of computer science domain is far higher than the results of other domains. 4 , from which we can find following observations.

First, there is a big performance gap between CoNll2003 and TechNER. In fact, the terminology recognition task is a more challenging task than the traditional NER task. The main reason is that lots of useful features that work well in a traditional NER task cannot work here. For example, capitalizations, prefix and suffix feature, collocations, all of these features are very useful in a tradition NER task. But none of them can be used here. What's worse, as observed in the previous RE task, there are also almost no any cues in the training sentences that can imply the boundaries of a terminology.

Second, there is also an obvious performance difference between different kinds of domains. But compared with the previous KGE or RE tasks, the performance gap between different kinds of domains here is far smaller.

One may wonder whether we could obtain better performance by increasing the number of training samples. To answer this question, we select 25,000 training sentences for each domain and conduct the experiments again. Other experiment settings remain unchanged. The new results are denoted as large in Table 12 . We can see that it is true that there is a minor improvement on the F1 values for each domain by increasing the number of training samples. But we find the performance gaps between different kinds of domains couldn't be eliminated by this way.

B DETAILED STATISTICS FOR TECHKG:

Supposing there is a paper whose basic information is listed as: Then the corresponding relations obtained are as listed. ,5) [5,10) [10,20) [20,30) [30,36) [1,5) [5,10) [10,20) [20,30) 

-70% tf*idf 1-1 1-n m-1 m-n 1-1 1-n m-1 m-n Domain % % ñ % % ū(m/n) % % ñ % % ū(m/n) Compute

-50% tf*idf 1-1 1-n m-1 m-n 1-1 1-n m-1 m-n Domain % % ñ % % ū(m/n) % % ñ % % ū(m/n) Compute

-10% tf*idf 1-1 1-n m-1 m-n 1-1 1-n m-1 m-n Domain % % ñ % % ū(m/n) % % ñ % % ū(m/n) Compute

:

Here we also use CNKI's(http://www.cnki.net/) categories for reference, but we merge some similar research domains in CNKI and reduce the amount of research domains.

It is difficult to effectively analyse the body part of a paper.

For each relation r, we compute averaged number of tails per head (tphr), averaged number of heads per tail (hptr). If tphr<1.5 and hptr< 1.5, r is treated as 1-1. If tphr≥ 1.5 and hptr≥ 1.5, r is treated as m-to-n. If hptr< 1.5 and tphr≥ 1.5, r is treated as 1-n. If hptr≥ 1.5 and tphr< 1.5, r is treated as m-1.

The whole TechKG is a huge and growing KB, which is almost impossible for many applications to directly experiment on. Thus, we create TechKG10: a subset of it. We construct TechKG10 by selecting the subset of entities that meet following two requirements. First, if the entity is a keyword, its tf*idf value must rank in top-10%. Second, each selected entity has at least 10 mentions. Some basic statistics of TechKG10 are shown in Table9.

Because the duplicated names and the two selected relations account for a large proportion in TechKG10, there are only about 235,000 and 224,000 triplets for the traffic transportation and metallurgical industry domains respectively. For the rest two research domains, there are about 290,000 triplets in this dataset.

Figures and Tables:

Figure 1: Hits@10 results of KGE. The ID numbers in the bottom line correspond to the datasets.

Figure 2: Experimental results of RE (part 1).

Figure 3: Experimental results of RE (part 2).

Figure 4: F1 results of NER.

Some basic statistics of TechKG.

Table content: <table><tr><td>Relation Types</td><td colspan="2">EntityNum TripletNum</td></tr><tr><td>author of</td><td>7,397,886</td><td>15,403,800</td></tr><tr><td>first author</td><td>6,662,907</td><td>5,588,103</td></tr><tr><td>second author</td><td>4,645,848</td><td>3,858,252</td></tr><tr><td>co author</td><td>1,764,270</td><td>17,188,520</td></tr><tr><td>research interest</td><td>5,179,265</td><td>50,748,054</td></tr><tr><td>affiliation</td><td>1,900,697</td><td>10,775,303</td></tr><tr><td>belonged domain</td><td>4,636,780</td><td>9,494,972</td></tr><tr><td>co-occurrence with</td><td>4,628,635</td><td>72,101,139</td></tr><tr><td>published year</td><td>15,628,610</td><td>16,367,393</td></tr><tr><td colspan="2">contained chn keyword 22,345,343</td><td>11,697,138</td></tr><tr><td colspan="2">contained eng keyword 8,338,872</td><td>4,304,324</td></tr><tr><td>other author</td><td>3,651,823</td><td>5,958,552</td></tr><tr><td>all authors</td><td>9,403,013</td><td>5,591,189</td></tr><tr><td>published journal</td><td>15,636,388</td><td>16,413,111</td></tr><tr><td>translation of</td><td>19,681,543</td><td>15,263,050</td></tr><tr><td>Total</td><td/><td/></tr></table>

Cross-domain duplicate name statistics for authors.

Table content: <table><tr><td>total #</td><td>duplicate #</td><td>duplicate ratio</td><td>dpa</td></tr><tr><td colspan="2">1,826,217 654,884</td><td>35.86%</td><td>3.97</td></tr></table>

Distribution of domain numbers for cross-domain duplicated authors.

Table content: <table><tr><td/><td colspan="2">Domain distribution Duplicate #</td><td>Duplicate ratio</td></tr><tr><td/><td>[30, 36]</td><td>1,390</td><td>0.21%</td></tr><tr><td/><td>[20, 30)</td><td>6,369</td><td>0.97%</td></tr><tr><td/><td>[10, 20)</td><td>25,976</td><td>3.97%</td></tr><tr><td/><td>[5, 10)</td><td>73,082</td><td>11.16%</td></tr><tr><td/><td>[1, 5)</td><td>548,067</td><td>83.69%</td></tr><tr><td>Total</td><td>[1,36]</td><td>654,884</td><td>100%</td></tr></table>

Extreme analysis of in-domain duplicate author name.

Table content: <table><tr><td/><td/><td>Worse/Better</td><td/><td/></tr><tr><td>domain</td><td>total #</td><td>duplicate #</td><td>ratio(%)</td><td>apa</td></tr><tr><td>metal</td><td>92637</td><td colspan="3">56160/54065 60.62/58.36 4.38/4.46</td></tr><tr><td>computer</td><td>95163</td><td colspan="3">54333/52719 57.09/55.40 4.25/4.31</td></tr><tr><td>electric</td><td>92230</td><td colspan="3">51040/49055 55.34/53.19 4.19/4.26</td></tr><tr><td>traffic</td><td>53354</td><td colspan="3">29661/28843 55.59/54.06 3.44/3.48</td></tr></table>

Cross-domain duplicate name statistics for keywords.

Table content: <table><tr><td colspan="2">tf*idf setting total #</td><td>duplicate #</td><td>duplicate ratio</td><td>dpk</td></tr><tr><td>Top-100%</td><td colspan="2">8,003,868 1,817,771</td><td>22.71%</td><td>4</td></tr><tr><td>Top-90%</td><td colspan="2">8,003,293 1,808,536</td><td>22.60%</td><td>3.32</td></tr><tr><td>Top-80%</td><td colspan="2">7,978,805 1,676,176</td><td>21.01%</td><td>2.84</td></tr><tr><td>Top-70%</td><td colspan="2">7,883,539 1,363,444</td><td>17.29%</td><td>2.68</td></tr><tr><td>Top-60%</td><td colspan="2">7,494,000 820,490</td><td>10.95%</td><td>2.87</td></tr><tr><td>Top-50%</td><td colspan="2">7,234,018 691,549</td><td>9.56%</td><td>2.91</td></tr><tr><td>Top-40%</td><td colspan="2">6,357,230 662,029</td><td>10.41%</td><td>2.88</td></tr><tr><td>Top-30%</td><td colspan="2">5,224,408 594,194</td><td>11.37%</td><td>2.84</td></tr><tr><td>Top-20%</td><td colspan="2">2,870,181 412,436</td><td>14.37%</td><td>2.8</td></tr><tr><td>Top-10%</td><td colspan="2">1,153,819 202,060</td><td>17.51%</td><td>2.62</td></tr></table>

Distribution of domain numbers for keywords.

Table content: <table><tr><td/><td/><td colspan="3">Duplicate num /duplicate ratio(%)</td><td/></tr><tr><td>domain</td><td>[1, 5)</td><td>[5, 10)</td><td>[10, 20)</td><td>[20, 30)</td><td>[30, 36]</td></tr><tr><td>metal</td><td colspan="2">14518/86.77 1761/10.53</td><td>443/2.65</td><td>9/0.05</td><td>0/0</td></tr><tr><td>computer</td><td colspan="2">21463/92.83 1372/5.93</td><td>278/1.20</td><td>8/0.03</td><td>0/0</td></tr><tr><td>electric</td><td colspan="2">22102/92.43 1453/6.08</td><td>349/1.46</td><td>8/0.03</td><td>0/0</td></tr><tr><td>traffic</td><td colspan="2">16376/86.82 2019/10.70</td><td>459/2.43</td><td>9/0.05</td><td>0/0</td></tr></table>



Table content: <table><tr><td/><td colspan="4">: Cross-domain duplicate name statistics for keywords.</td><td/></tr><tr><td>tf*idf setting</td><td>1-1(%)</td><td>1-n(%)/ ñ</td><td>m-1(%)/ m</td><td>m-n(%)/</td><td>ū</td></tr><tr><td>Top-100%</td><td>13.61</td><td>0.27/1.51</td><td>20.64/14.99</td><td colspan="2">65.47/1.21</td></tr><tr><td>Top-90%</td><td>14.36</td><td>0.01/1.87</td><td>19.93/13.69</td><td colspan="2">65.70/1.23</td></tr><tr><td>Top-80%</td><td>14.6</td><td>0.01/1.87</td><td>19.83/13.13</td><td colspan="2">65.56/1.25</td></tr><tr><td>Top-70%</td><td>14.77</td><td>0.01/1.87</td><td>19.78/12.71</td><td colspan="2">65.44/1.27</td></tr><tr><td>Top-60%</td><td>14.91</td><td>0.01/1.87</td><td>19.80/12.39</td><td colspan="2">65.28/1.29</td></tr><tr><td>Top-50%</td><td>14.99</td><td>0.01/1.87</td><td>19.84/12.27</td><td colspan="2">65.17/1.3</td></tr><tr><td>Top-40%</td><td>14.99</td><td>0.01/1.87</td><td>19.87/12</td><td colspan="2">65.13/1.32</td></tr><tr><td>Top-30%</td><td>15.15</td><td>0.01/1.87</td><td>19.65/11.48</td><td colspan="2">65.19/1.36</td></tr><tr><td>Top-20%</td><td>15.43</td><td>0.01/1.87</td><td>19.88/10.73</td><td colspan="2">64.68/1.46</td></tr><tr><td>Top-10%</td><td>16.32</td><td>0.01/1.87</td><td>20.74/10.14</td><td colspan="2">62.93/1.58</td></tr></table>

Comparisons of different KGs.Domain Categorizations.For each triplet or a paper's abstract, it has a pre-organized domain category. With these categories, one can use the collected abstracts as a dataset for a text classification task. TechKG can also be easily divided into different subsets according to the research domains.

Table content: <table><tr><td/><td colspan="3">WordNet FreeBase YAGO</td><td>YAGO2</td><td>YAGO3</td><td>TechKG</td></tr><tr><td>Language</td><td>English</td><td>English</td><td colspan="2">English English</td><td>MultiLingual</td><td>Chinese</td></tr><tr><td/><td/><td/><td/><td/><td>(noChn)</td></tr><tr><td>Purpose</td><td colspan="2">General General</td><td colspan="2">General General</td><td>General</td><td>Technology</td></tr><tr><td>DataSource</td><td>words</td><td colspan="4">Wikipedia Wikipedia Wikipedia Wikipedia</td><td>TechPapers</td></tr><tr><td>EntityNum</td><td colspan="2">117,597 ≈ 80M</td><td>≥1M</td><td>2,648,387</td><td>4,595,906</td><td>162,521,245</td></tr><tr><td>TripletNUM</td><td colspan="3">207,016 ≈1.2KM ≥5M</td><td colspan="2">124,333,521 8,936,324</td><td>258,554,528</td></tr><tr><td colspan="2">SupportedTasks KGE</td><td>KGE,RE,</td><td>KGE,RE,</td><td>KGE,RE,</td><td>KGE,RE,</td><td>KGE,RE,TC,</td></tr><tr><td/><td/><td>KBQA</td><td>KBQA</td><td>KBQA</td><td>KBQA</td><td>KBQA,NER,MT</td></tr><tr><td colspan="7">In Table 8, we make a comparison between TechKG and some other KGs. TechKG can support</td></tr><tr><td colspan="7">more kinds of applications because it can provide other kinds of information as listed below.</td></tr><tr><td colspan="7">Bilingual Translation Pairs. These pairs include bilingual titles, bilingual keywords, and bilingual</td></tr><tr><td colspan="7">abstracts. These bilingual translation pairs usually have very high quality for they are written by</td></tr><tr><td colspan="7">experts in the corresponding research domain. They can provide high quality training data for a</td></tr><tr><td colspan="2">machine translation task.</td><td/><td/><td/><td/></tr></table>

Some basic statistics of TechKG10.

Table content: <table><tr><td/><td>EntityNum</td><td>TripletNum</td></tr><tr><td>author of</td><td>4701043</td><td>12107175</td></tr><tr><td>first author</td><td>3826827</td><td>3210759</td></tr><tr><td>second author</td><td>3808524</td><td>3240132</td></tr><tr><td>co author</td><td>1291222</td><td>15324905</td></tr><tr><td>research interest</td><td>1955144</td><td>29705592</td></tr><tr><td>affiliation</td><td>1055761</td><td>6428984</td></tr><tr><td>belonged domain</td><td>780466</td><td>1160626</td></tr><tr><td>co-occurrence with</td><td>777960</td><td>30553582</td></tr><tr><td>published year</td><td>1713143</td><td>1793187</td></tr><tr><td>contained chn keyword</td><td>9377</td><td>6800</td></tr><tr><td>contained eng keyword</td><td>426</td><td>308</td></tr><tr><td>other author</td><td>3342604</td><td>5657370</td></tr><tr><td>all authors</td><td>36462</td><td>22328</td></tr><tr><td>published journal</td><td>3332185</td><td>3547271</td></tr><tr><td>translation of</td><td>146204</td><td>281555</td></tr><tr><td>Total</td><td>5750579</td><td>113040574</td></tr></table>

Hits@10 results of KGE.

Table content: <table><tr><td>ConvE</td><td>InverseModel</td></tr></table>

Experimental results of RE.

Table content: <table><tr><td>p@300</td><td>p@300</td><td>p@300</td><td>mean</td></tr></table>

F1 results of NER.In this study, we select LSTM-CRFLample et al. (2016), one of the state-of-the-art NER methods as baseline and evaluate its performance on recognizing domain terminologies.

Table content: <table><tr><td/><td/><td>F1</td></tr><tr><td>CoNll2003</td><td/><td>90.94</td></tr><tr><td/><td>Metal</td><td>76.86</td></tr><tr><td>TechNER</td><td colspan="2">Computer 74.36 Electric 77.21</td></tr><tr><td/><td>Traffic</td><td>78.19</td></tr><tr><td/><td>Metal</td><td>79.01</td></tr><tr><td>TechNER (large)</td><td colspan="2">Computer 80.33 Electric 79.31</td></tr><tr><td/><td>Traffic</td><td>79.52</td></tr><tr><td colspan="3">In each domain of TechNER, the number of training number is far larger than CoNll2003. So in</td></tr><tr><td colspan="3">experiments, we randomly select 21,000 training samples for each selected domain. By this way,</td></tr><tr><td colspan="3">there is a similar sample size between CoNll2003 (also about 21,000) and TechNER. In experiments,</td></tr><tr><td colspan="3">we select 15,000 samples for training, 3,500 samples for both the development set and the test set.</td></tr><tr><td colspan="3">During training, we iterate 300 rounds for each experiment. Finally, the experimental results are</td></tr><tr><td>shown in Table 12 and Figure</td><td/></tr></table>

Basic Information of a Paper

Table content: <table><tr><td colspan="2">Title Authors</td><td colspan="2">Affiliation Keywords</td><td>Keywords</td><td>domain</td><td>Published</td><td>Published</td></tr><tr><td/><td/><td/><td>(chn)</td><td>(eng)</td><td/><td>year</td><td>journal</td></tr><tr><td>p</td><td>a 1 , a 2 ,...,</td><td>o 1 , o 2 , ...,</td><td>ck 1 , ck 2 ,</td><td>ek l , ek 2 , ...,</td><td>d i</td><td>y</td><td>j</td></tr><tr><td/><td>a m</td><td>o s</td><td>..., ck n</td><td>ek l</td><td/><td/><td/></tr></table>

Samples of Extracted Relations , other author, p>, < a 4 , other author, p>, ..., < a m , other author, p>, all authors < p, all authors, a 1 + a 2 +...+ a m > published journal < p, published journal, j> translation of <ck 1 , translation of, ek 1 >, < ck 2 , translation of, ek 2 >, ..., < ck s , translation of, ek s >. (Threshold requirement should be met.)Other statistics information about TechKG is shown in the following tables.

Table content: <table><tr><td>relations</td><td>Extracted relations</td></tr><tr><td colspan="2">author of &lt;a co-&lt;ck 1 , co-occurrence with, ck 2 &gt;, ...,&lt;ck 1 , co-occurrence with, ck n &gt;,</td></tr><tr><td>occurrence with</td><td>...,&lt;ck n-1 , co-occurrence with, ck n &gt;</td></tr><tr><td>published year</td><td>&lt;p, published year, y&gt;</td></tr><tr><td>contained cky</td><td>&lt; p, contained cky, ck 1 + ck 2 +...+ ck n &gt;</td></tr><tr><td>contained eky</td><td>&lt; p, contained eky, ek 1 + ek 2 +...+ ek l &gt;</td></tr><tr><td>other author</td><td>&lt; a 3</td></tr></table>

Detailed Statistics of TechKG

Table content: <table><tr><td/><td>Law</td><td/><td colspan="2">MedicineRelated</td><td colspan="2">PoliticsRelated</td><td colspan="2">NatureScience</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td>Related</td><td/></tr><tr><td/><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td></tr><tr><td>author of</td><td>52763</td><td colspan="2">42219 1614774</td><td>4074249</td><td>123342</td><td>94895</td><td>790242</td><td>1120957</td></tr><tr><td>first author</td><td>48051</td><td colspan="2">32563 1297282</td><td>990282</td><td>112876</td><td>73466</td><td>632613</td><td>437183</td></tr><tr><td>second author</td><td>14199</td><td colspan="2">7917 1154040</td><td>890725</td><td>30285</td><td>16851</td><td>470032</td><td>320782</td></tr><tr><td>co author</td><td>11312</td><td>11921</td><td>609337</td><td>5899008</td><td>25958</td><td>27953</td><td>326878</td><td>1147280</td></tr><tr><td>research inter</td><td>61760</td><td colspan="3">137344 1114364 12239418</td><td>120958</td><td>283638</td><td>883893</td><td>3920870</td></tr><tr><td>affiliation</td><td>5295</td><td>5770</td><td>583627</td><td>2514451</td><td>10602</td><td>9605</td><td>388660</td><td>1252779</td></tr><tr><td colspan="2">belged domain 101466</td><td>101466</td><td>841224</td><td>841224</td><td>296130</td><td>296130</td><td>688301</td><td>688301</td></tr><tr><td>co-</td><td>101293</td><td>594562</td><td>838379</td><td>8703065</td><td colspan="2">295547 2922678</td><td>687361</td><td>3472226</td></tr><tr><td>occurrence with</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">published year 147795</td><td colspan="2">149318 2346223</td><td>2385304</td><td>835976</td><td>849274</td><td>710377</td><td>715292</td></tr><tr><td colspan="2">contained cky 180710</td><td colspan="2">91544 3805573</td><td>1946078</td><td>928248</td><td colspan="2">476598 1166195</td><td>588306</td></tr><tr><td>contained eky</td><td>46719</td><td colspan="2">23425 1441121</td><td>725911</td><td>97514</td><td>48927</td><td>777421</td><td>390670</td></tr><tr><td>other author</td><td>2356</td><td colspan="2">1740 1189796</td><td>2193664</td><td>6797</td><td>4591</td><td>372259</td><td>363014</td></tr><tr><td>all authors</td><td>52282</td><td colspan="2">32566 1883060</td><td>991031</td><td>120923</td><td>73499</td><td>807775</td><td>437306</td></tr><tr><td>publi journal</td><td>147880</td><td colspan="2">149268 2347191</td><td>2389266</td><td>836394</td><td>861521</td><td>710725</td><td>714789</td></tr><tr><td>translation of</td><td>153835</td><td colspan="2">93709 3295408</td><td>2142785</td><td>295830</td><td colspan="2">181573 2229279</td><td>1418314</td></tr><tr><td>Total</td><td colspan="8">475944 1475362 9308021 48928685 1870480 6221255 4378579 16988413</td></tr><tr><td/><td colspan="2">SportRelated</td><td colspan="2">PublishingRelated</td><td colspan="4">EnvironmentRelated FisheriesRelated</td></tr><tr><td/><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td></tr><tr><td>author of</td><td>87482</td><td>99025</td><td>73092</td><td>68834</td><td>192889</td><td>272905</td><td>41081</td><td>68587</td></tr><tr><td>first author</td><td>75790</td><td>53981</td><td>64202</td><td>43271</td><td>141195</td><td>92267</td><td>28857</td><td>19450</td></tr><tr><td>second author</td><td>40008</td><td>26217</td><td>27577</td><td>17103</td><td>109566</td><td>72367</td><td>23867</td><td>16258</td></tr><tr><td>co author</td><td>26775</td><td>67321</td><td>20556</td><td>33123</td><td>93953</td><td>318459</td><td>20714</td><td>90368</td></tr><tr><td>research inter</td><td>71352</td><td>318380</td><td>72235</td><td>221583</td><td>200140</td><td>961108</td><td>46784</td><td>230432</td></tr><tr><td>affiliation</td><td>22495</td><td>55028</td><td>8409</td><td>8987</td><td>48544</td><td>94239</td><td>27160</td><td>86679</td></tr><tr><td>belged domain</td><td>87841</td><td>87841</td><td>138747</td><td>138747</td><td>163699</td><td>163698</td><td>49305</td><td>49304</td></tr><tr><td>co-</td><td>87648</td><td>754956</td><td>138468</td><td>1032810</td><td>163507</td><td>910402</td><td>49250</td><td>323270</td></tr><tr><td>occurrence with</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">published year 158179</td><td>160267</td><td>260025</td><td>262332</td><td>193861</td><td>196357</td><td>66708</td><td>68089</td></tr><tr><td colspan="2">contained cky 195815</td><td>100011</td><td>302989</td><td>153703</td><td>287999</td><td>145679</td><td>93183</td><td>47365</td></tr><tr><td>contained eky</td><td>70894</td><td>35550</td><td>51663</td><td>25916</td><td>129718</td><td>65010</td><td>28167</td><td>14122</td></tr><tr><td>other author</td><td>22320</td><td>18836</td><td>11947</td><td>8460</td><td>105294</td><td>108288</td><td>26711</td><td>32881</td></tr><tr><td>all authors</td><td>91601</td><td>54012</td><td>73250</td><td>43282</td><td>174118</td><td>92329</td><td>36878</td><td>19456</td></tr><tr><td>publi journal</td><td>158231</td><td>159533</td><td>260121</td><td>263231</td><td>193928</td><td>195690</td><td>66705</td><td>68240</td></tr><tr><td>translation of</td><td>201234</td><td>127884</td><td>157651</td><td>95069</td><td>392207</td><td>239751</td><td>99742</td><td>62492</td></tr><tr><td>Total</td><td colspan="2">556452 2118866</td><td>705674</td><td>2416572</td><td colspan="2">961988 3928619</td><td>278473</td><td>1197019</td></tr><tr><td/><td colspan="4">MetallurgicalIndustry PhysicsRelated</td><td colspan="4">History&amp;Geography PowerEngineering</td></tr><tr><td/><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td><td>Ent#</td><td>Tri#</td></tr><tr><td>author of</td><td>303394</td><td>465580</td><td>129777</td><td>225201</td><td>63257</td><td>54465</td><td>130051</td><td>186652</td></tr><tr><td>first author</td><td>236332</td><td>162211</td><td>90422</td><td>59140</td><td>55766</td><td>36680</td><td>93176</td><td>60050</td></tr><tr><td colspan="2">second author 173593</td><td>118676</td><td>77858</td><td>52303</td><td>16265</td><td>9623</td><td>72376</td><td>47147</td></tr><tr><td>co author</td><td>128389</td><td>499336</td><td>69062</td><td>383499</td><td>15502</td><td>34318</td><td>65074</td><td>240627</td></tr><tr><td>research inter</td><td colspan="2">307348 1507161</td><td>160426</td><td>727998</td><td>79819</td><td>161376</td><td>153163</td><td>645826</td></tr><tr><td>affiliation</td><td>129811</td><td>385070</td><td>60320</td><td>173495</td><td>8296</td><td>10481</td><td>76361</td><td>195820</td></tr><tr><td colspan="2">belged domain 244813</td><td>244812</td><td>107366</td><td>107365</td><td>147371</td><td>147370</td><td>120266</td><td>120265</td></tr><tr><td>co-</td><td colspan="2">244494 1391266</td><td>107217</td><td>394796</td><td>147128</td><td>956521</td><td>120139</td><td>560521</td></tr><tr><td>occurrence with</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">published year 300589</td><td>304862</td><td>72691</td><td>73636</td><td>190612</td><td>192141</td><td>112125</td><td>113461</td></tr><tr><td colspan="2">contained cky 465548</td><td>235547</td><td>127151</td><td>64104</td><td>224319</td><td>113625</td><td>178792</td><td>90362</td></tr><tr><td colspan="2">contained eky 253867</td><td>127440</td><td>96534</td><td>48424</td><td>30352</td><td>15195</td><td>92333</td><td>46362</td></tr><tr><td>other author</td><td>160783</td><td>184747</td><td>85364</td><td>113764</td><td>9639</td><td>8162</td><td>72243</td><td>79466</td></tr></table>

Detailed cross-domain duplicate name statistics for keywords in TechKG

Table content: <table><tr><td/><td colspan="2">Top-100% tf*idf</td><td/><td colspan="2">Top-90% tf*idf</td><td/></tr><tr><td>domain</td><td>total #</td><td>dup #</td><td>ratio(%) dpk</td><td>total #</td><td>dup #</td><td>ratio(%) dpk</td></tr><tr><td>Compute</td><td>526348</td><td>300981</td><td>57.18% 6.79</td><td/><td/><td/></tr></table>

Detailed triplet statistics under different tf*idf setting in TechKG

Table content: <table><tr><td/><td colspan="3">Top-100% tf*idf</td><td/><td/><td/><td/><td colspan="3">Top-90% tf*idf</td><td/></tr><tr><td/><td>1-1</td><td>1-n</td><td/><td>m-1</td><td/><td>m-n</td><td/><td>1-1</td><td>1-n</td><td/><td>m-1</td><td>m-n</td></tr><tr><td>Domain</td><td>%</td><td>%</td><td>ñ</td><td>%</td><td/><td>%</td><td colspan="2">ū(m/n) %</td><td>%</td><td>ñ</td><td>%</td><td>%</td><td>ū(m/n)</td></tr><tr><td colspan="2">Computer 15.55</td><td>0</td><td>\</td><td colspan="5">18.78 11.05 65.68 1.19 16.22</td><td>0</td><td>\</td><td colspan="2">18.38 10.1 65.39 1.21</td></tr><tr><td colspan="2">Mechanis 16.85</td><td>0</td><td>\</td><td>15.3</td><td colspan="4">7.34 67.85 1.05 17.3</td><td>0</td><td>\</td><td colspan="2">15.75 7.03 66.95 1.06</td></tr><tr><td>Power</td><td>14.05</td><td>0</td><td>\</td><td>16.51</td><td colspan="4">9.32 69.44 1.12 14.5</td><td>0</td><td>\</td><td>16.43</td><td>8.7</td><td>69.06 1.13</td></tr><tr><td colspan="2">Publising 13.48</td><td>0</td><td>\</td><td>30</td><td colspan="4">26.7 56.52 1.2 14.55</td><td>0</td><td>\</td><td colspan="2">28.6 23.27 56.85 1.23</td></tr><tr><td>Forestry</td><td>14.07</td><td>0</td><td>\</td><td colspan="5">18.95 12.47 66.97 1.2 14.51</td><td>0</td><td>\</td><td colspan="2">19.45 12.07 66.04 1.22</td></tr><tr><td colspan="2">Industrial 14.17</td><td>0</td><td>\</td><td colspan="5">21.76 11.73 64.07 1.09 14.72</td><td>0</td><td>\</td><td colspan="2">22.13 11.09 63.15 1.1</td></tr><tr><td>Math</td><td>21.03</td><td>0</td><td>\</td><td>18.71</td><td colspan="4">9.06 60.26 1.11 21.76</td><td>0</td><td>\</td><td>19.31</td><td>8.7</td><td>58.93 1.12</td></tr><tr><td colspan="2">Environe 13.77</td><td>0</td><td>\</td><td colspan="5">18.34 10.08 67.88 1.23 14.21</td><td>0</td><td>\</td><td colspan="2">18.76 9.79 67.04 1.25</td></tr><tr><td colspan="2">Fisheries 11.94</td><td>0</td><td>\</td><td>18.5</td><td>16</td><td colspan="3">69.56 1.08 12.26</td><td>0</td><td>\</td><td colspan="2">18.88 15.55 68.86 1.09</td></tr><tr><td>Animal</td><td>11.02</td><td>0</td><td>\</td><td>21.18</td><td>20.8</td><td colspan="3">67.8 1.16 11.37</td><td>0</td><td>\</td><td colspan="2">21.2 19.87 67.43 1.17</td></tr><tr><td colspan="2">Architecr 15.4</td><td>0</td><td>\</td><td colspan="5">20.45 13.38 64.15 1.26 16.12</td><td>0</td><td>\</td><td colspan="2">19.29 11.78 64.59 1.28</td></tr><tr><td colspan="2">Aerospace 15.51</td><td>0</td><td>\</td><td colspan="5">19.78 10.41 64.71 1.2 16.39</td><td>0</td><td>\</td><td colspan="2">18.39 8.87 65.21 1.22</td></tr><tr><td colspan="2">LightIndu 14.22</td><td>0</td><td>\</td><td colspan="5">20.17 14.59 65.61 1.23 14.65</td><td>0</td><td>\</td><td colspan="2">20.31 13.93 65.04 1.25</td></tr><tr><td>Finance</td><td>13.7</td><td>0</td><td>\</td><td colspan="5">30.27 35.02 56.03 1.26 14.43</td><td>0</td><td>\</td><td colspan="2">28.09 30.49 57.48 1.28</td></tr><tr><td>Physics</td><td>13.21</td><td>0</td><td>\</td><td>13.3</td><td>7.9</td><td colspan="3">73.5 1.13 13.4</td><td>0</td><td>\</td><td colspan="2">13.51 7.61 73.08 1.14</td></tr><tr><td>Traffic</td><td>15.75</td><td>0</td><td>\</td><td colspan="5">21.95 13.55 62.3 1.25 16.22</td><td>0</td><td>\</td><td colspan="2">22.54 13.17 61.24 1.27</td></tr><tr><td colspan="13">History&amp; 10.81 0.4 2.2 32.12 18.34 56.67 0.99 11.68 0.4 2.2 31.65 16.36 56.23 0.99</td></tr><tr><td>Agricue</td><td colspan="8">7.17 5.1 1.5 16.97 15.71 70.73 1.23 12.72</td><td>0</td><td>\</td><td colspan="2">15.78 13.85 71.51 1.26</td></tr><tr><td colspan="2">Machiner 15.29</td><td>0</td><td>\</td><td>18.12</td><td>9.89</td><td colspan="3">66.6 1.24 15.7</td><td>0</td><td>\</td><td colspan="2">18.54 9.58 65.76 1.27</td></tr><tr><td colspan="2">Chemistry 14.38</td><td>0</td><td>\</td><td colspan="5">15.48 10.98 70.14 1.27 14.71</td><td>0</td><td>\</td><td colspan="2">15.51 10.53 69.79 1.3</td></tr><tr><td>Law</td><td colspan="3">14.65 0.1 1.8</td><td>34.4</td><td colspan="8">18.31 50.83 1.02 15.77 0.1 1.8 34.74 16.78 49.36 1.02</td></tr><tr><td>Energy</td><td>13.44</td><td>0</td><td>\</td><td>14.61</td><td colspan="4">9.82 71.95 1.21 13.81</td><td>0</td><td>\</td><td colspan="2">14.37 9.15 71.82 1.23</td></tr><tr><td colspan="2">Astronoy 14.6</td><td>0</td><td>\</td><td colspan="5">15.44 11.84 69.95 1.27 14.97</td><td>0</td><td>\</td><td colspan="2">15.55 11.29 69.48 1.3</td></tr><tr><td colspan="2">Education 13.03</td><td>0</td><td>\</td><td colspan="5">29.62 29.99 57.35 1.06 13.46</td><td>0</td><td>\</td><td colspan="2">28.6 27.61 57.95 1.06</td></tr><tr><td>Power</td><td>15.29</td><td>0</td><td>\</td><td>18.8</td><td colspan="4">11.26 65.91 1.21 15.76</td><td>0</td><td>\</td><td colspan="2">19.31 10.92 64.93 1.23</td></tr><tr><td>Art-</td><td colspan="12">8.79 0.1 2.4 31.78 28.52 59.34 1.04 9.33 0.1 2.4 31.37 26.22 59.2 1.05</td></tr><tr><td>Material</td><td>13.28</td><td>0</td><td>\</td><td>13.6</td><td colspan="4">6.95 73.12 1.08 13.47</td><td>0</td><td>\</td><td>13.79</td><td>6.7</td><td>72.74 1.08</td></tr><tr><td>Metal</td><td>15.05</td><td>0</td><td>\</td><td colspan="5">17.31 10.85 67.64 1.24 15.48</td><td>0</td><td>\</td><td colspan="2">17.45 10.34 67.07 1.26</td></tr><tr><td colspan="2">Electronic 15.13</td><td>0</td><td>\</td><td colspan="5">19.35 12.32 65.53 1.2 15.84</td><td>0</td><td>\</td><td colspan="2">18.42 10.94 65.74 1.21</td></tr><tr><td>Politics</td><td colspan="5">11.62 0.1 1.6 36.31 32.26</td><td>52</td><td colspan="6">1.05 12.48 0.1 1.6 34.47 28.19 52.97 1.06</td></tr><tr><td>Military</td><td>11.84</td><td>0</td><td>\</td><td colspan="5">27.75 31.07 60.42 1.05 12.43</td><td>0</td><td>\</td><td colspan="2">28.86 30.14 58.71 1.05</td></tr><tr><td colspan="2">Medicine 11.85</td><td>0</td><td>\</td><td colspan="5">15.33 17.49 72.83 1.44 12.08</td><td>0</td><td>\</td><td colspan="2">14.53 16.09 73.39 1.47</td></tr><tr><td>Bioscie</td><td>12.51</td><td>0</td><td>\</td><td>12.63</td><td colspan="4">8.03 74.86 1.11 12.78</td><td>0</td><td>\</td><td colspan="2">12.41 7.51 74.81 1.12</td></tr><tr><td>Mining</td><td>16.5</td><td>0</td><td>\</td><td>19.12</td><td colspan="4">9.83 64.38 1.19 17.2</td><td>0</td><td>\</td><td>18.7</td><td>8.92</td><td>64.1 1.21</td></tr><tr><td>Culture</td><td colspan="7">7.63 0.1 1.6 29.01 26.37 63.29 1.03</td><td>8.1</td><td colspan="4">0.1 1.6 27.48 23.26 64.35 1.03</td></tr><tr><td>Sport</td><td>14.94</td><td>0</td><td>\</td><td colspan="5">23.03 16.91 62.02 1.3 15.54</td><td>0</td><td>\</td><td colspan="2">23.86 16.5</td><td>60.6 1.33</td></tr><tr><td>Total</td><td colspan="8">13.61 0.3 1.5 20.64 14.99 65.47 1.21 14.36</td><td>0</td><td colspan="3">1.9 19.93 13.69 65.7 1.23</td></tr><tr><td/><td colspan="3">Top-80% tf*idf</td><td/><td/><td/><td/><td>Top</td><td/><td/><td/></tr></table>

Completed and detailed statistics for TechRE, TechNER, and TechQA.

Table content: <table><tr><td/><td>17.52</td><td>0</td><td>\</td><td>20.37</td><td colspan="2">8.41 62.12 1.49 18.08</td><td>0</td><td>\</td><td colspan="4">21.09 8.13 60.82 1.57</td></tr><tr><td colspan="2">Mechanis 19.42</td><td>0</td><td>\</td><td>16.86</td><td colspan="2">4.93 63.71 1.23 19.78</td><td>0</td><td>\</td><td colspan="4">17.19 4.88 63.03 1.24</td></tr><tr><td>Power</td><td>16.33</td><td>0</td><td>\</td><td>18.31</td><td>6.8</td><td>65.36 1.31 16.33</td><td>0</td><td>\</td><td>18.31</td><td>6.8</td><td colspan="2">65.36 1.31</td></tr><tr><td colspan="2">Publising 19.67</td><td>0</td><td>\</td><td colspan="3">28.31 16.05 52.01 1.42 23.61</td><td>0</td><td>\</td><td colspan="3">32.29 14.16 44.1</td><td>1.8</td></tr><tr><td>Forestry</td><td>15.63</td><td>0</td><td>\</td><td colspan="3">21.17 10.79 63.2 1.33 16.36</td><td>0</td><td>\</td><td colspan="4">22.78 9.64 60.86 1.52</td></tr><tr><td colspan="2">Industrial 18.14</td><td>0</td><td>\</td><td>26.14</td><td colspan="2">8.51 55.72 1.26 19.79</td><td>0</td><td>\</td><td colspan="2">27.81 8.08</td><td>52.4</td><td>1.3</td></tr><tr><td>Math</td><td>24.63</td><td>0</td><td>\</td><td>21.57</td><td colspan="2">6.91 53.79 1.33 25.29</td><td>0</td><td>\</td><td colspan="4">22.17 6.58 52.54 1.43</td></tr><tr><td colspan="2">Environe 15.63</td><td>0</td><td>\</td><td>19.23</td><td colspan="2">7.51 65.14 1.49 16.76</td><td>0</td><td>\</td><td>20.47</td><td>7.2</td><td colspan="2">62.76 1.55</td></tr></table>

Detailed distribution of domain numbers for cross-domain duplicated authors in TechKG10.

Table content: <table><tr><td/><td>Ratio(%)</td><td>number</td></tr><tr><td>domain</td><td>[1</td></tr></table>

Detailed cross-domain duplicate name statistics for keywords in TechKG10. (It should be noted that in TechKG10, all of its keywords rank in top-10% tf*idf values).

Table content: <table><tr><td>[30,36)</td></tr></table>

Peer review (iclr2020):
I have read the author response, thank you for responding. Original review: This paper presents the extraction of a bibliographic database of Chinese technical papers. This database could potentially be a valuable resource for the community. However, the paper is mis-targeted to the ICLR conference, as it does not discuss learning representations or using deep learning (the 36-page appendix does include some material on knowledge graph embedding, but this is not covered in the main body of the paper). Also, for important tasks like name deduplication, the paper does not discuss or compare against techniques from previous work, and instead proposes a small set of heuristics, which is a sensible approach for building a resource but will not be of interest to the ICLR audience. The paper refers to their resource as a knowledge graph, but I would say it is more accurate to call it a bibliographic database (it consists primarily of paper titles, authors, and keywords). This is very different from the broader KBs and KGs discussed in the related work. YAGO, Freebase, Cyc, and so on capture a much wider variety of semantic relationships and entity types. I think re-positioning this submission as being aimed at building a bibliographic database, rather than a Knowledge Graph, would help ensure it reaches the right audience. Also, targeting a different venue like a digital libraries conference and making a strong case that this set of bibliographic data offers advantages in coverage or accuracy over other comparable resources, if any, would help. Finally, one of the most valuable features of a bibliographic database is the citation graph, it would be exciting if TechKG could be extended to include citation information. Minor: Section 2, most of these citations should not be shortcites but should be regular full cites. For example in the first paragraph, Miller (1995) should be (Miller, 1995).