# Analysis and Research Based on the Crowdsourcing Corpus System in Guangdong-Hong Kong-Macao Greater Bay Area (GBA)

Authors: Zheyu Zhu; Mingyang Xu; Ying Jiang; Jing Yang
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Objective. To sort out the application and status quo of some domestic crowdsourcing models, explore the factors affecting multilingual manual annotation through experiments and offer suggestions. Methodology. Crawling the government news texts in Mandarin, Cantonese, English, and Portuguese in Guangdong, Hong Kong, and Macao, and enter them into the database. Combine it with corpus tagging uses the established web platform to practice crowdsourcing and collect a large number of annotation results and behavior data. Results. Made assumptions about factors that may affect the quality of manual annotation, used SPSS and other data analysis software to evaluate the degree of interpretation of the assumptions, provided a regression formula for calculating the accuracy, and provided constructive suggestions for the corpus annotation quality assurance projects. Limitations. More corpus information in more languages and more professional annotators are needed. Conclusions. The study found that the accuracy of annotation is strongly related to the attributes of the corpus itself, such as the total number of vocabularies, the number of rare words, the complexity of parts of speech, etc., and the condition of languages are different from each other.


---

## Body

## 1. Introduction

With the rapid development of science and technologies and the increasingly developed social economy, great changes have taken place in the times. The Internet has become the most solid force for innovation, and the era of big data is coming. Based on the big data resource sharing, service object concretization and cloud informatization, what corresponds to the growing demands of the tasks, the application and optimization of the new crowdsourcing model in various fields of society.As one of the most open and active economy regions in China, Guangdong-Hong Kong-Macao Greater Bay Area (GBA) places an important strategic role in the overall national development. At the same time, there are rich and diverse corpora in GBA, and crowdsourcing has not been widely used in the field of text language corpus there. Furthermore, it is also a dynamic economy that is very suitable for the application of the crowdsourcing model, which has a great scientific research value and application value. At present, GBA and its radiation areas have multiple languages which leads to the problems of complex and diverse language types, layered and intertwined language distribution, and complexity of foreign relations. It is the region with the most developed language and writing diversity, the most usage, and the longest coexistence time [1] from ancient China to modern times, and those problems require the support of multilingual corpora.According to the “Report on language conditions in the Guangdong-Hong Kong-Macao Greater Bay Area (2021)” published by Ministry of Education of the People’s republic of China, the establishment of the GBA into a world-class bay area and a world-class city cluster, construction of an international language environment, and the pool of multilingual talents are essential. Guangdong Province, Hong Kong, and Macao plan to strengthen the coordinated planning of language talents training, in which Hong Kong will play the role of English-speaking countries in radiating English-speaking countries, Macao will play the role of Portuguese-speaking countries radiating Portuguese-speaking countries, and strengthen the role of English- and Portuguese-speaking talent training bases, and cities such as Guangzhou, Shenzhen, etc., Will be responsible for strengthening the training and reserve of relevant multilingual talents to the countries along the “Belt and Road.” As the three languages of the first tier of language use in GBA, Mandarin, Cantonese, and English cover most of the regions and populations in GBA. Although Portuguese is ranked third in the level of use, it is the official language of Macao, which has certain representativeness [2].This paper applies the crowdsourcing model to experiment, analyze, and summarize the four languages mentioned above, i.e., Mandarin, Cantonese, English, and Portuguese, in the field of text corpus with the self-developed GBA multilingual annotation system and corpus. Through empirical analysis, this paper attempts to find the factors that affect the artificial accuracy rate, provides new ideas for quality assurance engineering, and deepens the application of crowdsourcing in the field of language research.This paper mainly solves the following problems: to integrate and improve the existing accuracy evaluation index calculation methods and compare the results; to list the factors that may affect the accuracy in a single corpus annotation crowdsourcing task, analyze the collected multilingual data through correlation analysis, multiple linear regression analysis, etc.,; to summarize the differences in the influence degree of various factors among different languages with the data analysis results, and further provide suggestions for improving labor efficiency and optimizing process after combining crowdsourcing mode with multilingual annotation.


## 2. Literature Review

At present, there is a relatively rich research on the application of crowdsourcing at home and abroad, mainly focusing on the following five aspects:(1)
business practice: Vuculescu et al. [3] believe that organizations can complete a series of tasks by utilizing the public, including evaluation (i.e., Amazon, Netflix), content creation (i.e., Threadless and trouble-shooting) (i.e., OpenIDEO, InnoCentive).(2)
Application in the field of computers: McDuff et al. [4] applied crowdsourcing to the process of collection of human facial expression data, and proposed for the first time to collect dynamic, natural and spontaneous facial response data in the form of an online media reader; Tong et al. [5] proposed a crowdsourcing-based data cleaning system for solving the problems of high computational difficulty and low accuracy of existing data cleaning methods.(3)
Application in the field of library and information: Ridge [6] believes that crowdsourcing can provide a powerful platform for public participation in museum work, on which participants can build a deep and valuable connection with cultural heritage through online collaboration with common goals or resources; Zhao [7] analyzed the connotation of archives crowdsourcing, and proposes the construction methods of the archive crowdsourcing platform of “policy of prior prosperity and common prosperity” and “all-in-one” style.(4)
Application in the field of transportation: Yang [8] applied crowdsourcing to traffic monitoring, and proposed the concept of collecting very large-scale sensory data with a large number of personal smartphones, which subverts the traditional real-time road condition information collection technology. Manikonda et al. [9] developed a travel plan generation system named AI-MIX.(5)
Applications in scientific research: the domestic scholar Zhang [10] first proposed the concept of scientific research crowdsourcing, that is, gather the wisdom of global researchers to collaborate on scientific research and solve scientific research problems together. Pang et al. [11] analyzed the particularity of participants and processes in scientific research crowdsourcing. Zhang et al. [12] discussed the composition, utility and realization path of the innovation model of scientific research crowdsourcing network.Generally, although in-depth theoretical discussions on crowdsourcing applications in many popular scientific research fields has been conducted by scholars, crowd sourced applied research in the field of language and literature is still not sufficient. GBA as an economic zone with great corpus value, it is unrealistic to rely solely on individuals or small groups to collect large-scale data and complete a series of word segmentation tasks there. The model of crowdsourcing can make a difference of the characteristics of “many hands make light work” and can quickly gather the power of talents in various fields to complete the corpus research work in each link. The GBA multilingual corpus used in this experiment is a corpus annotation function developed based on HanLp natural language processing technology that supports simplified Chinese, traditional Chinese, Cantonese, English, and Portuguese. HanLp has been popular since its release in 2014. At present, HanLp has become one of the most popular NLP technologies in the world. Tens of thousands of artificial intelligence projects have been developed using it, including Baidu and Xiaomi. In this study, we use its rich API provided to conduct multilingual word segmentation and model construction. HanLp greatly improves the efficiency of the study.The online system preliminarily performs word segmentation and annotation processing on the uploaded corpus, and then, the annotators check, proofread, and revise the corpus, and cross-validate the corpus while submitting the annotation results, and further to output high-quality corpus as shown in Figure1 screenshot of the corpus annotation system. Due to the huge amount of data in the corpus, there have been more than 400 annotators who have used the system for corpus annotation since 2019, and crowdsourcing is needed to optimize the process of the system to improve the efficiency of the word segmentation task allocation of the system and the quality of annotation.Figure 1
Screenshot of the corpus annotation system.


## 3. Data

### 3.1. Source of Raw Corpus and Its Pre-processing Steps

Life cycle of corpus is shown in Figure2. The corpus of this study comes from domestic authoritative news websites that have columns related to the GBA or are mainly aimed at the interior of the Bay Area. The main fields that need to be saved when the crawler acquires data are the title, URL, body, publication time, language, website name, and specific section name of the article, among them, website and section can be combined into one field by “_,” and then save each field as a record and save it in a the local in the file format of Excel or Txt.Figure 2
Life cycle of corpus.Figure 3
Irrelevant information processing rules.A series of the following pre-processing steps are required before placing the locally stored corpus into a relational database.(1)
Noise reduction: media reports may contain information including links, forwarding, users of “@,” etc., and the presence of such information can interfere with the results of the analysis task, but the information itself is irrelevant to the task. Therefore, it is necessary to replace such information with unified marks with regular expression. The detailed regular expression and replaceable marks are shown in Figure3:(2)
Sentence segmentation: it can be found through the statistics on the localized storage files that the content of an article is usually between 200 and 5000 words, and if all the annotation tasks are assigned to one user of annotation, it will lead to uneven distribution and excessive task volume, therefore, it is necessary to split the tasks to make them suitable for reading, the cutting regular expression is as follows:(3)
Repeat detection: if the content of a corpus after the segmented is detected and found to already exist in the database, this corpus does not need to be entered to avoid repeated annotation.(4)
Machine word segmentation: after the sentence is segmented, the segmented corpus will be pre-segmented and annotated, and the multilingual tokenizer provided by HanLp and NLTK is used to implement this step, and the precision of pre-annotation can generally reach 85% to 95%, which can appropriately reduce the workload of manual annotation and improve efficiency.(5)
Data storage: the article is stored in a separate table ARTICLE, the segmented corpus is stored in the table ORIGIN, and the results of expert user and non-expert users are stored in the table LABEL_RESULT. Classify the users with user_id. The table structure is shown in Figure4.(6)
Task assignment: using its id as the unique index in the database, the segmented corpus is extracted from the table ORIGIN to generate a sequence to be annotated and added to the Redis cache to wait for the users of annotation to extract and complete the task of annotation. The principle of task assignment is that a corpus will not be annotated more than once by the same user; if a corpus has been assigned to a sequence, it means that it is being processed and cannot be assigned to other users.Figure 4
Primary storage table design.


### 3.2. Corpus Annotation

In order to ensure the quality of the corpus, the annotation process can be divided into three phases. The first phase is the machine word segmentation in Section3.1. The second phase is the crowd sourced annotating phase that non-expert users review the original sentences and deal with 5% to 15% of system annotating errors, and a corpus needs to be annotated by at least n (n > 1) different users for quality assurance measures such as cross-validation. In the third phase, expert annotating phase, the experts need to focus on processing the “Difficult Corpus” left over from the second phase, and also check and revise the results of non-experts. We believe that the generated corpus after three-phases of strict annotation, is credible and can be used as the basis for corpus construction.All the text data can be classified into four categories, Mandarin, Cantonese, English, and Portuguese. The annotated data statistics is shown as follows:Table1 only shows part of corpus sources, and the percentage of the number of manual annotations R is calculated as:(1)R=QfQt+Qf×100%,Qt is the unverified number, Qf is the verified number. The corpus should be supplemented in time before the average R value of a language reaches 100% to avoid affecting the progress of the overall manual annotation.Table 1
Statistics of articles and corpora in the four languages.
LanguageSourceNumber of articlesNumber of corpusesNumber of manual annotations (R)MandarinPress releases of the government of the Hong Kong special administrative region4,31957,89948,787 (84%)News of Hong Kong Education Bureau3,52759,00631,088 (52%)Department of Justice89216,99916,999 (100%)…(7)CantoneseMingpao News OL_ instant entertainment2,98329,20128,655 (98%)Mingpao News OL_CULTURE & LEISURE4717,4607,310 (97%)Mingpao News OL_CULTURE & BEAUTY2261,6091,570 (97%)…(4)EnglishSouth China Morning Post _Politics2,86339,00434,323 (88%)South China Morning Post _Education1,48417,31915,587 (90%)South China Morning Post _Law and Crime1,07213,59612,067 (89%)…(2)PortugueseRevista Macau_Macau2978,0046,201 (77%)Revista Macau_Cultura1914,2973,310 (77%)Revista Macau_China1723,9923,000 (75%)…(30)Total5523,030274,194330,872(80%)


## 3.1. Source of Raw Corpus and Its Pre-processing Steps

Life cycle of corpus is shown in Figure2. The corpus of this study comes from domestic authoritative news websites that have columns related to the GBA or are mainly aimed at the interior of the Bay Area. The main fields that need to be saved when the crawler acquires data are the title, URL, body, publication time, language, website name, and specific section name of the article, among them, website and section can be combined into one field by “_,” and then save each field as a record and save it in a the local in the file format of Excel or Txt.Figure 2
Life cycle of corpus.Figure 3
Irrelevant information processing rules.A series of the following pre-processing steps are required before placing the locally stored corpus into a relational database.(1)
Noise reduction: media reports may contain information including links, forwarding, users of “@,” etc., and the presence of such information can interfere with the results of the analysis task, but the information itself is irrelevant to the task. Therefore, it is necessary to replace such information with unified marks with regular expression. The detailed regular expression and replaceable marks are shown in Figure3:(2)
Sentence segmentation: it can be found through the statistics on the localized storage files that the content of an article is usually between 200 and 5000 words, and if all the annotation tasks are assigned to one user of annotation, it will lead to uneven distribution and excessive task volume, therefore, it is necessary to split the tasks to make them suitable for reading, the cutting regular expression is as follows:(3)
Repeat detection: if the content of a corpus after the segmented is detected and found to already exist in the database, this corpus does not need to be entered to avoid repeated annotation.(4)
Machine word segmentation: after the sentence is segmented, the segmented corpus will be pre-segmented and annotated, and the multilingual tokenizer provided by HanLp and NLTK is used to implement this step, and the precision of pre-annotation can generally reach 85% to 95%, which can appropriately reduce the workload of manual annotation and improve efficiency.(5)
Data storage: the article is stored in a separate table ARTICLE, the segmented corpus is stored in the table ORIGIN, and the results of expert user and non-expert users are stored in the table LABEL_RESULT. Classify the users with user_id. The table structure is shown in Figure4.(6)
Task assignment: using its id as the unique index in the database, the segmented corpus is extracted from the table ORIGIN to generate a sequence to be annotated and added to the Redis cache to wait for the users of annotation to extract and complete the task of annotation. The principle of task assignment is that a corpus will not be annotated more than once by the same user; if a corpus has been assigned to a sequence, it means that it is being processed and cannot be assigned to other users.Figure 4
Primary storage table design.


## 3.2. Corpus Annotation

In order to ensure the quality of the corpus, the annotation process can be divided into three phases. The first phase is the machine word segmentation in Section3.1. The second phase is the crowd sourced annotating phase that non-expert users review the original sentences and deal with 5% to 15% of system annotating errors, and a corpus needs to be annotated by at least n (n > 1) different users for quality assurance measures such as cross-validation. In the third phase, expert annotating phase, the experts need to focus on processing the “Difficult Corpus” left over from the second phase, and also check and revise the results of non-experts. We believe that the generated corpus after three-phases of strict annotation, is credible and can be used as the basis for corpus construction.All the text data can be classified into four categories, Mandarin, Cantonese, English, and Portuguese. The annotated data statistics is shown as follows:Table1 only shows part of corpus sources, and the percentage of the number of manual annotations R is calculated as:(1)R=QfQt+Qf×100%,Qt is the unverified number, Qf is the verified number. The corpus should be supplemented in time before the average R value of a language reaches 100% to avoid affecting the progress of the overall manual annotation.Table 1
Statistics of articles and corpora in the four languages.
LanguageSourceNumber of articlesNumber of corpusesNumber of manual annotations (R)MandarinPress releases of the government of the Hong Kong special administrative region4,31957,89948,787 (84%)News of Hong Kong Education Bureau3,52759,00631,088 (52%)Department of Justice89216,99916,999 (100%)…(7)CantoneseMingpao News OL_ instant entertainment2,98329,20128,655 (98%)Mingpao News OL_CULTURE & LEISURE4717,4607,310 (97%)Mingpao News OL_CULTURE & BEAUTY2261,6091,570 (97%)…(4)EnglishSouth China Morning Post _Politics2,86339,00434,323 (88%)South China Morning Post _Education1,48417,31915,587 (90%)South China Morning Post _Law and Crime1,07213,59612,067 (89%)…(2)PortugueseRevista Macau_Macau2978,0046,201 (77%)Revista Macau_Cultura1914,2973,310 (77%)Revista Macau_China1723,9923,000 (75%)…(30)Total5523,030274,194330,872(80%)


## 4. Assessment Indicators and Methods

### 4.1. Gold Standard

GS, Gold Standard, was originally coined in the field of economics to denote a monetary system in which the value of a monetary unit is based on the quantity of physical gold. Although this system is no longer in use, it is still deemed as an extremely solid financial system. Subsequently, this term has gradually to been used to denote a scientific procedure or collection of accepted standards. GSC, Gold Standard Corpus, in NLP is a manually annotated text collection. GSC with high quality requires multiple experts to independently check the data and calculate the inter-annotator agreement to ensure its quality, which made the creation of GSC a very time-consuming and expensive process. To reduce the cost of creation of GSC, several methods are available, such as reducing the total number of corpus annotations, reducing the number of expert reviews, using non-expert annotators through crowdsourcing, replacing or supplementing GSC with the automatically created Silver Standard Corpus, SSC [13].Nowak and Rüger [14] compared the annotation results between the expert users and the no-expert users, found that the consistency among expert users exceeds 90%, concluding that expert users do not need multiple reviews. And it is found in this same study that non-experts have the same consistency as experts, both at 92%. And another similar study reported 51% consistency for non-experts and 61% consistency for experts. These data show that non-expert users achieve nearly the same level of consistency of results as experts. [15].Based on the comprehensive consideration of the above study results and cost reduction, GSC is built by the method of circular estimation with the combination of the idea of “cross-checking” in modeling application. The main process is shown in Figure5. During the process, experts will review at most once and give priority to reviewing the questionable corpus, and if a corpus has been reviewed by an expert, the result will be accepted and the annotation of that corpus will be terminated. For those that has not had time to be reviewed by experts and its n non-expert results, their result set is L=l1,l2,l3⋯ln, n is the number of uses of this corpus, ln is the annotation result of the user un, n cannot be exceeded the threshold and should be determined by the administrator in advance. In l=s1,s2,s3⋯si, s contains the attribute word and pos. Word refers to the words, pos refers to the corresponding part of speech. The process of cross-validation is similar to voting. Whenever a new annotation result is submitted, it traverses and records the number of votes for word, and the final result can be represented by a one-dimensional array as shown in Figure 6. The third line is the new annotation result, and the array subscript in the fourth line indicates the position of the character in l, and the value at each position indicates the number of words in which the character should appear according to the result of the vote, and the “1, 1” at the beginning of the array indicates that “Tim” and “Kai” should be the first word of GS. After word is confirmed, just keep the corresponding pos with the highest number of votes.Figure 5
Cross-validation flowchart.Figure 6
Example of cross-validation results.If the GS cannot be determined by the votes and the number of annotations of the current corpus has exceeded the threshold, then there is a dispute over the annotation result of this corpus that requires intervention of experts, and experts should also take the lead in resolving the annotation results of this type of corpus, which helps reduce the total number of annotations, as well as to rationally schedule the energy of various users, thereby reducing the cost of building GSC.


### 4.2. Precision of Manual Annotation

According to a previous study, precision, recall rate, and F-score are generally adopted for evaluating the word segmentation effect, in which the annotation effect of port of speech is generally evaluated by the annotation Accuracy [16–22]. The above indicators are usually used to evaluate the large-scale annotation results generated by machines rather than a single record generated manually. And since word segmentation and annotation were carried out at the same time in this study, for the convenience of comparison, the calculation methods of the above four indicators are now integrated and improved.Previous studies comparedln with lgold, that is GS, to evaluate the annotating effect of un on ln. Take HanLP’s comparison function as an example, it puts ln and lgold into an algorithm which realizes the process of comparing the user’s annotation and standard results by indexes and maintains a variable inc to calculate the result coverage. Coverage is an integer variable using to record the degree of overlap of annotation in ln and lgold. When word and pos are complete overlapped, the increment is inc (generally inc is set to 1), other cases will not be handled. This method is obviously not objective enough in evaluating the manual annotation results. When word is consistent but pos are inconsistent, this evaluation of the actual annotation of pos is unavailable.Take the sentence “點解收咁貴” (Tim Kai Sau Kam Kwai) in Figure6 as example, it means “why is it so expensive”. There are 2 imprecise annotation results which are l1=點解,r,收,v,咁,r,貴,a,l2=點解,v,收,v,咁,r,貴,a, the correct annotation should be lgold=點解,ry,收,v,咁,r,貴,a, The word “點” is an interrogative word in Cantonese and it equals to “how” in English. For example, “你最近點啊” means “how are you doing recently?,” and “解” has the meaning of “explain,” therefore, “點解” means “why” or “how to explain”. The part of speech of “點” is interrogative word, which is the main contradiction of the annotation of it. According to the previous evaluation methods, the scores of the above two imprecise annotation results are the same, for the coincidence of word and pos is also 3 times, accounting for 75% of the number of s (i = 4), so this result is not reasonable. Pos of s1 in l1 is “r” (pronoun), pos of s1 in l2 is “v” (verb). Figure 7 shows the part of the content after the parts of speech are arranged in the form of a tree according to the hierarchical relationship. Pos of s1 in lgold is “ry” (interrogative pronouns) that is a node of part of speech “r”. It is assumed that the length of each line segment in Figure 7 is 1, the distance between “ry” and “r” is 1, and the distance between “ry” and “v” is 3, the distance between parts of speech reflects the correlation between parts of speech, thus, the shorter the distance, the higher the correlation. Apparently, the score of u1 should be higher than u2 for the annotation of the former is closer to the standard result.Figure 7
Part-of-speech tree.In summary, whenever the comparison function is used to calculate coverage and inc, the following formula is adopted:(2)inci=1,sicompletely overlapped,1−distance2.h+1,si.pos non−overlapped,0,si.word non−overlapped,coverage=coverge+inci.Distance represents the distance between the standard result part-of-speech and the user-marked part-of-speech,h refers to the maximum height of part-of-speech relation tree. Due to the specifications of different languages and part-of-speech sets are different, the value of h also needs to be determined according to the specific condition (The height of part-of-speech relation tree in this study is 3). At this time, the precision, recall, and harmonic mean need to be recalculated, the results are in Table 2:Table 2
Evaluation results.
UP_1R_1F_1P_2R_2F_2u175.0075.0075.0096.4296.4296.42u275.0075.0075.0089.2889.2889.28P_2, R_2, F_2 introduced the determination of the degree of overlap between the word and pos attributes ins and the standard result compared with P_1, R_1, F_1. There is a certain difference in the evaluation results of two non-expert users, which is in line with expectations and also in line with reality. The adjusted F will be adopted as the evaluation index below.


## 4.1. Gold Standard

GS, Gold Standard, was originally coined in the field of economics to denote a monetary system in which the value of a monetary unit is based on the quantity of physical gold. Although this system is no longer in use, it is still deemed as an extremely solid financial system. Subsequently, this term has gradually to been used to denote a scientific procedure or collection of accepted standards. GSC, Gold Standard Corpus, in NLP is a manually annotated text collection. GSC with high quality requires multiple experts to independently check the data and calculate the inter-annotator agreement to ensure its quality, which made the creation of GSC a very time-consuming and expensive process. To reduce the cost of creation of GSC, several methods are available, such as reducing the total number of corpus annotations, reducing the number of expert reviews, using non-expert annotators through crowdsourcing, replacing or supplementing GSC with the automatically created Silver Standard Corpus, SSC [13].Nowak and Rüger [14] compared the annotation results between the expert users and the no-expert users, found that the consistency among expert users exceeds 90%, concluding that expert users do not need multiple reviews. And it is found in this same study that non-experts have the same consistency as experts, both at 92%. And another similar study reported 51% consistency for non-experts and 61% consistency for experts. These data show that non-expert users achieve nearly the same level of consistency of results as experts. [15].Based on the comprehensive consideration of the above study results and cost reduction, GSC is built by the method of circular estimation with the combination of the idea of “cross-checking” in modeling application. The main process is shown in Figure5. During the process, experts will review at most once and give priority to reviewing the questionable corpus, and if a corpus has been reviewed by an expert, the result will be accepted and the annotation of that corpus will be terminated. For those that has not had time to be reviewed by experts and its n non-expert results, their result set is L=l1,l2,l3⋯ln, n is the number of uses of this corpus, ln is the annotation result of the user un, n cannot be exceeded the threshold and should be determined by the administrator in advance. In l=s1,s2,s3⋯si, s contains the attribute word and pos. Word refers to the words, pos refers to the corresponding part of speech. The process of cross-validation is similar to voting. Whenever a new annotation result is submitted, it traverses and records the number of votes for word, and the final result can be represented by a one-dimensional array as shown in Figure 6. The third line is the new annotation result, and the array subscript in the fourth line indicates the position of the character in l, and the value at each position indicates the number of words in which the character should appear according to the result of the vote, and the “1, 1” at the beginning of the array indicates that “Tim” and “Kai” should be the first word of GS. After word is confirmed, just keep the corresponding pos with the highest number of votes.Figure 5
Cross-validation flowchart.Figure 6
Example of cross-validation results.If the GS cannot be determined by the votes and the number of annotations of the current corpus has exceeded the threshold, then there is a dispute over the annotation result of this corpus that requires intervention of experts, and experts should also take the lead in resolving the annotation results of this type of corpus, which helps reduce the total number of annotations, as well as to rationally schedule the energy of various users, thereby reducing the cost of building GSC.


## 4.2. Precision of Manual Annotation

According to a previous study, precision, recall rate, and F-score are generally adopted for evaluating the word segmentation effect, in which the annotation effect of port of speech is generally evaluated by the annotation Accuracy [16–22]. The above indicators are usually used to evaluate the large-scale annotation results generated by machines rather than a single record generated manually. And since word segmentation and annotation were carried out at the same time in this study, for the convenience of comparison, the calculation methods of the above four indicators are now integrated and improved.Previous studies comparedln with lgold, that is GS, to evaluate the annotating effect of un on ln. Take HanLP’s comparison function as an example, it puts ln and lgold into an algorithm which realizes the process of comparing the user’s annotation and standard results by indexes and maintains a variable inc to calculate the result coverage. Coverage is an integer variable using to record the degree of overlap of annotation in ln and lgold. When word and pos are complete overlapped, the increment is inc (generally inc is set to 1), other cases will not be handled. This method is obviously not objective enough in evaluating the manual annotation results. When word is consistent but pos are inconsistent, this evaluation of the actual annotation of pos is unavailable.Take the sentence “點解收咁貴” (Tim Kai Sau Kam Kwai) in Figure6 as example, it means “why is it so expensive”. There are 2 imprecise annotation results which are l1=點解,r,收,v,咁,r,貴,a,l2=點解,v,收,v,咁,r,貴,a, the correct annotation should be lgold=點解,ry,收,v,咁,r,貴,a, The word “點” is an interrogative word in Cantonese and it equals to “how” in English. For example, “你最近點啊” means “how are you doing recently?,” and “解” has the meaning of “explain,” therefore, “點解” means “why” or “how to explain”. The part of speech of “點” is interrogative word, which is the main contradiction of the annotation of it. According to the previous evaluation methods, the scores of the above two imprecise annotation results are the same, for the coincidence of word and pos is also 3 times, accounting for 75% of the number of s (i = 4), so this result is not reasonable. Pos of s1 in l1 is “r” (pronoun), pos of s1 in l2 is “v” (verb). Figure 7 shows the part of the content after the parts of speech are arranged in the form of a tree according to the hierarchical relationship. Pos of s1 in lgold is “ry” (interrogative pronouns) that is a node of part of speech “r”. It is assumed that the length of each line segment in Figure 7 is 1, the distance between “ry” and “r” is 1, and the distance between “ry” and “v” is 3, the distance between parts of speech reflects the correlation between parts of speech, thus, the shorter the distance, the higher the correlation. Apparently, the score of u1 should be higher than u2 for the annotation of the former is closer to the standard result.Figure 7
Part-of-speech tree.In summary, whenever the comparison function is used to calculate coverage and inc, the following formula is adopted:(2)inci=1,sicompletely overlapped,1−distance2.h+1,si.pos non−overlapped,0,si.word non−overlapped,coverage=coverge+inci.Distance represents the distance between the standard result part-of-speech and the user-marked part-of-speech,h refers to the maximum height of part-of-speech relation tree. Due to the specifications of different languages and part-of-speech sets are different, the value of h also needs to be determined according to the specific condition (The height of part-of-speech relation tree in this study is 3). At this time, the precision, recall, and harmonic mean need to be recalculated, the results are in Table 2:Table 2
Evaluation results.
UP_1R_1F_1P_2R_2F_2u175.0075.0075.0096.4296.4296.42u275.0075.0075.0089.2889.2889.28P_2, R_2, F_2 introduced the determination of the degree of overlap between the word and pos attributes ins and the standard result compared with P_1, R_1, F_1. There is a certain difference in the evaluation results of two non-expert users, which is in line with expectations and also in line with reality. The adjusted F will be adopted as the evaluation index below.


## 5. Empirical Analysis

### 5.1. Factors

The data in this section is based on the long-term statistics of this experiment, and combined with the main problems of this experiment and long-term experience and observations. This paper classifies the corpus by language first, and then conducts multiple linear regression analysis with the following six factors as input to explore the degree of influence on the precision of users’ annotation and the relationship between them.(1)
WC: Word Count, an integer variable. It is the number of words inln;(2)
OC: Obscure Word Count, an integer variable. The initial value of OC is 0, traverseln, and take s.word. If its word frequency in the corpus is less than or equal to 1, the value of OC increases by 1 until the end of the traversal;(3)
PC: Pos Count (non-repetitive), an integer variable. Traverseln and take s.pos, then save it in a hash set, PC is the size of the set;(4)
AL: Article Location, the location of corpus, a float variable. It indicates the location of the current corpus in the whole article, ranging from 0 to 1. The closer AL is to 1, the closer it is to the end of the article.(5)
CF: Change Fluctuation, a float variable. Calculate the similarity between users’ annotation results and GS using the Levenstein distance, ranging from 0 to 1. The closer the CF is to 1, the smaller the change.(6)
MD: Mark Date, an integer variable. From 00 : 00 to 23 : 59, each 2 hours is regarded as a time period. A day can be divided into 12 time periods in total, represented from 1 to 12.Table3 shows the format of the data.Table 3
Data formats.
IDOIDUIDWCOCPCALCFMDF14049903306430.09.00.251.012100.024049903310430.09.00.251.012100.034049923310660.014.00.3750.499082577100.044049993310420.011.00.81250.49859947100.054050023310740.012.00.00.4992548210100.0…………………………


### 5.2. Description of Statistics

The annotation results can be classified as Mandarin, Cantonese, English, and Portuguese according to the languages. Through Table4, as the native language of the annotating users in this study, Mandarin occupies the highest F-mean and the lowest standard deviation, indicating that Mandarin has a good annotation effect and the very little fluctuation of the precision of the annotation results in the studied languages. Portuguese, as a language unfamiliar to annotators, has the lowest F-mean and a standard deviation of 12.72, its annotation effect is not as good as other language, and different annotators generally have deviations in the understanding of the corpus. According to F-mean, the overall effect of the annotation is sorted by language from high to bottom as Mandarin > English > Cantonese > Portuguese.Table 4
Statistics of descriptions in the four languages.
LanguageFactorsMeanStandard deviationNumber of casesMandarinWC30.7628.6853470OC0.220.73PC10.083.41AL0.510.80CF0.970.12MD6.060.552F99.434.695CantoneseWC26.428825.0166337105OC0.82291.5555PC9.09584.04348AL0.490.29CF0.970.10MD8.6192.438F99.28674.821EnglishWC28.147820.8018663142OC0.1370.45646PC11.71953.71311AL0.470.29CF0.980.087MD8.5052.706F99.31135.327PortugueseWC32.720133.711327475OC0.21320.94596PC10.84833.71377AL0.490.29CF0.870.16MD8.5122.96F97.492412.30536


### 5.3. Correlation Analysis

Based on Table5, the six factors listed in this paper do have a certain correlation with the F, which enables the further regression analysis to be carried out. In statistics, a significant correlation indicates that there is a significant relationship between the two factors in the regression analysis, and only on this basis can the next step of the analysis be carried out. However, a significant correlation is not equivalent to a strong correlation, but the strength of the correlation is determined by the absolute value of the correlation coefficient. The closer the absolute value of the correlation coefficient is to 0, the weaker the correlation, and the closer to 1, the stronger the correlation. Therefore, the data in Table 5 shows that the factors with strong correlation between languages are WC, OC, CF.Table 5
Correlation Analysis of the four languages.
LanguageFactorWCOCPCALCFMDFMandarinWC1OC0.303∗∗1PC0.640∗∗0.195∗∗1AL−0.035−0.029∗∗−0.0601CF−0.275−0.105∗∗0.013∗∗−0.0021MD−0.007−0.001−0.003−0.0040.0081F−0.083∗∗−0.035∗∗−0.0030.0060.311∗∗0.011∗∗1CantoneseWC1OC0.568∗∗1PC0.741∗∗0.423∗∗1AL−0.150∗∗−0.078∗∗−0.285∗∗1CF−0.237∗∗−0.143∗∗−0.015∗∗0.0031MD0.019∗∗−0.0010.017∗∗0.0050.0031F−0.133∗∗−0.075∗∗−0.027∗∗−0.0070.320∗∗−0.0051EnglishWC1OC0.147∗∗1PC0.558∗∗0.085∗∗1AL−0.083∗∗−0.020∗∗−0.041∗∗1CF−0.304∗∗−0.069∗∗−0.0070.016∗∗1MD0.010∗−0.0030.005−0.006−0.0071F−0.180∗∗−0.051∗∗−0.028∗∗0.035∗∗0.406∗∗−0.015∗∗1PortugueseWC1OC0.132∗∗1PC0.575∗∗0.167∗∗1AL0.006−0.013−0.052∗∗1CF−0.329∗∗−0.028∗−0.154∗∗0.029∗1MD0.0110.068∗∗0.012−0.0050.0111F−0.060∗∗−0.038∗∗−0.051∗∗0.027∗0.113∗∗−0.0191Note. ∗ and ∗∗ indicate significant correlation at Person <0.05 level and the extremely significant correlation at Person <0.01 level respectively.


### 5.4. Multiple Linear Regression Analysis

According to Tables6 to 8, each factor is significantly correlated with the precision, but the degree of explanation is generally not high, indicating that there may be more factors that affect the F value that have not been discovered. In the analysis of these four languages, Portuguese has the smallest Durbin-Watson value of 1.897, indicating that the samples are highly independent of each other, and the minimum Variance Inflation Factor, VIF for Portuguese is 1.018, suggesting that the variables in the model have no multidisciplinary. Generally, this study has certain statistical significance. The regression formulas for each language are as follows:(3)FMandarin=98.18−0.032∗WC−0.083∗OC+0.175∗PC+0.084∗MD,FCantonese=98.678−0.069∗WC+0.039∗OC+0.271∗PC−0.0007∗MD,FEnglish=99.363−0.081∗WC−0.35∗OC+0.215∗PC−0.027∗MD,FPortuguese=99.359−0.024∗WC−0.336∗OC−0.03∗PC−0.077∗MD.Table 6
Summary of models in the four languages.
LanguageRR SquareAdjustedR squareError in standard estimationDurbin-Watson valueMandarin0.311a0.0960.0964.463501.920Cantonese0.325a0.1060.1064.559522.025English0.411a0.1690.1694.856421.983Portuguese0.120a0.0140.01412.218371.897Table 7
ANOVAa of the four languages.
LanguageSum of squaresDegree of freedomMean squareFSignificanceMandarinRegression113769.507337923.1691903.4990.000bResidual1065196.3955346619.923Sum1178965.90353469CantoneseRegression91362.407330454.1361464.9010.000bResidual771300.7513710120.789Sum862663.15837104EnglishRegression302657.4703100885.8234277.5820.000bResidual1489119.4746313923.585Sum1791776.94463142PortugueseRegression16392.46435464.15536.6010.000bResidual1115333.9797471149.288Sum1131726.4437474a. Dependent variable:f, b. Predictive variable: (constant) WC, OC, CF.Table 8
Regression coefficients for various languages.
LanguageFactorNonstandard coefficientStandardization coefficienttSignificanceCollinearity statisticsBStandard errorBetaToleranceVIFMandarin(Constant)98.180.26377.520WC−0.0320.001−0.172−29.9100.5571.795OC−0.0830.033−0.011−2.5340.0110.9081.101PC0.1750.0090.11320.18600.591.695MD0.0840.0410.0092.040.04111Cantonese(Constant)98.6780.123802.950WC−0.0690.002−0.311−37.3100.3722.69OC0.0390.0220.0111.7960.0730.6771.478PC0.2710.010.19826.23200.4512.216MD−0.0070.012−0.003−0.6020.5470.9991.001English(Constant)99.3630.107928.430WC−0.0810.001−0.279−59.3500.6791.474OC−0.350.052−0.026−6.77100.9781.022PC0.2150.0080.13228.39300.6891.452MD−0.0270.009−0.012−3.1310.00211Portuguese(Constant)99.3590.633157.030WC−0.0240.005−0.065−4.59500.6681.498OC−0.3360.158−0.025−2.1270.0330.9661.036PC−0.030.049−0.009−0.620.5360.6611.514MD−0.0770.05−0.018−1.540.1240.9951.005


### 5.5. Comparative Analysis

This paper calculates the average precision rate of each user after aggregating the data according to user ID as shown in Figure8. It can be found that the results of the four languages are concentrated in the interval of 99% to 100%, indicating that it is in a relatively good overall annotation condition. There are no users with 100% of precision in the annotations of Portuguese, which refers that in the case that the annotator has mastered a certain language in a lower degree, the precision rate cannot reach a satisfactory level even though training and machine annotation results are provided for assistance in advance. The comparative analysis is aimed to locate the distribution of the precision of annotation users in these four languages, and further obtain the level of differentiation between different annotators.Figure 8
Distribution of the average precision rate of users in the four languages.


### 5.6. Cluster Analysis

According to Tables9 to 10, the cases of Mandarin, English, Portuguese and Cantonese were clustered analyzed by K-means fast clustering method, and the final clustered results shows that the total number of cases classified as 1 and 3 in Mandarin is 4624, accounting for about 8.65% of the whole; the total number of cases classified as 1 and 2 in English is 30, accounting for about 0.05%; the number of cases classified as 1 and 3 in Portuguese was 14, accounting for about 0.19%; and the number of cases classified as 1 and 3 in Cantonese was 3893, accounting for about 10.49%. There are no obvious categorical features among the four language cases. Cluster analysis is a basic statistical method in statistics. Generally, if each group has a more obvious characteristics, the number of each group will be relatively uniformly distributed during classification. The basic principle of cluster analysis is that birds of a feather fly together, that is, classify the same type of things into the same category to find out the features of annotation users, and the analysis here suggests that there is not obvious characteristics between users.Table 9
Clustering and sub-centers of the four languages.
Final cluster center (CNS)Final cluster center (ENG)Final cluster center (POT)Final cluster center (YUE)ClusterClusterClusterCluster123123123123WC440.7524.7689.312006.00381.8627.93438.5031.82947.0020.4776.901356.00OC4.330.180.630.001.960.140.000.210.000.602.7030.00PC19.299.6214.9610.0018.9611.7215.4210.8417.008.3915.1518.00AL0.29546701060.51401991670.44688561100.740.670.470.86906302840.48959103640.90981735150.49505270830.42572337170.1025641030CF0.75754857350.97983107640.87505186250.010.800.980.90239541000.86980411400.49994146500.97275258740.90844607810.1665387700MD6.006.066.0612.009.368.518.928.5110.008.618.729.00Table 10
Number of clustered cases in the four languages.
Number of cases in each cluster (CNS)Number of cases in each cluster (ENG)Number of cases in each cluster (POT)Number of cases in each cluster (YUE)Cluster163.0002.00012.00033212.000248844.00028.0007461.0003892.00034563.00063113.0002.0001.000Effective53470.00063143.0007475.00037105.000Missing0.0000.0000.0000.000


## 5.1. Factors

The data in this section is based on the long-term statistics of this experiment, and combined with the main problems of this experiment and long-term experience and observations. This paper classifies the corpus by language first, and then conducts multiple linear regression analysis with the following six factors as input to explore the degree of influence on the precision of users’ annotation and the relationship between them.(1)
WC: Word Count, an integer variable. It is the number of words inln;(2)
OC: Obscure Word Count, an integer variable. The initial value of OC is 0, traverseln, and take s.word. If its word frequency in the corpus is less than or equal to 1, the value of OC increases by 1 until the end of the traversal;(3)
PC: Pos Count (non-repetitive), an integer variable. Traverseln and take s.pos, then save it in a hash set, PC is the size of the set;(4)
AL: Article Location, the location of corpus, a float variable. It indicates the location of the current corpus in the whole article, ranging from 0 to 1. The closer AL is to 1, the closer it is to the end of the article.(5)
CF: Change Fluctuation, a float variable. Calculate the similarity between users’ annotation results and GS using the Levenstein distance, ranging from 0 to 1. The closer the CF is to 1, the smaller the change.(6)
MD: Mark Date, an integer variable. From 00 : 00 to 23 : 59, each 2 hours is regarded as a time period. A day can be divided into 12 time periods in total, represented from 1 to 12.Table3 shows the format of the data.Table 3
Data formats.
IDOIDUIDWCOCPCALCFMDF14049903306430.09.00.251.012100.024049903310430.09.00.251.012100.034049923310660.014.00.3750.499082577100.044049993310420.011.00.81250.49859947100.054050023310740.012.00.00.4992548210100.0…………………………


## 5.2. Description of Statistics

The annotation results can be classified as Mandarin, Cantonese, English, and Portuguese according to the languages. Through Table4, as the native language of the annotating users in this study, Mandarin occupies the highest F-mean and the lowest standard deviation, indicating that Mandarin has a good annotation effect and the very little fluctuation of the precision of the annotation results in the studied languages. Portuguese, as a language unfamiliar to annotators, has the lowest F-mean and a standard deviation of 12.72, its annotation effect is not as good as other language, and different annotators generally have deviations in the understanding of the corpus. According to F-mean, the overall effect of the annotation is sorted by language from high to bottom as Mandarin > English > Cantonese > Portuguese.Table 4
Statistics of descriptions in the four languages.
LanguageFactorsMeanStandard deviationNumber of casesMandarinWC30.7628.6853470OC0.220.73PC10.083.41AL0.510.80CF0.970.12MD6.060.552F99.434.695CantoneseWC26.428825.0166337105OC0.82291.5555PC9.09584.04348AL0.490.29CF0.970.10MD8.6192.438F99.28674.821EnglishWC28.147820.8018663142OC0.1370.45646PC11.71953.71311AL0.470.29CF0.980.087MD8.5052.706F99.31135.327PortugueseWC32.720133.711327475OC0.21320.94596PC10.84833.71377AL0.490.29CF0.870.16MD8.5122.96F97.492412.30536


## 5.3. Correlation Analysis

Based on Table5, the six factors listed in this paper do have a certain correlation with the F, which enables the further regression analysis to be carried out. In statistics, a significant correlation indicates that there is a significant relationship between the two factors in the regression analysis, and only on this basis can the next step of the analysis be carried out. However, a significant correlation is not equivalent to a strong correlation, but the strength of the correlation is determined by the absolute value of the correlation coefficient. The closer the absolute value of the correlation coefficient is to 0, the weaker the correlation, and the closer to 1, the stronger the correlation. Therefore, the data in Table 5 shows that the factors with strong correlation between languages are WC, OC, CF.Table 5
Correlation Analysis of the four languages.
LanguageFactorWCOCPCALCFMDFMandarinWC1OC0.303∗∗1PC0.640∗∗0.195∗∗1AL−0.035−0.029∗∗−0.0601CF−0.275−0.105∗∗0.013∗∗−0.0021MD−0.007−0.001−0.003−0.0040.0081F−0.083∗∗−0.035∗∗−0.0030.0060.311∗∗0.011∗∗1CantoneseWC1OC0.568∗∗1PC0.741∗∗0.423∗∗1AL−0.150∗∗−0.078∗∗−0.285∗∗1CF−0.237∗∗−0.143∗∗−0.015∗∗0.0031MD0.019∗∗−0.0010.017∗∗0.0050.0031F−0.133∗∗−0.075∗∗−0.027∗∗−0.0070.320∗∗−0.0051EnglishWC1OC0.147∗∗1PC0.558∗∗0.085∗∗1AL−0.083∗∗−0.020∗∗−0.041∗∗1CF−0.304∗∗−0.069∗∗−0.0070.016∗∗1MD0.010∗−0.0030.005−0.006−0.0071F−0.180∗∗−0.051∗∗−0.028∗∗0.035∗∗0.406∗∗−0.015∗∗1PortugueseWC1OC0.132∗∗1PC0.575∗∗0.167∗∗1AL0.006−0.013−0.052∗∗1CF−0.329∗∗−0.028∗−0.154∗∗0.029∗1MD0.0110.068∗∗0.012−0.0050.0111F−0.060∗∗−0.038∗∗−0.051∗∗0.027∗0.113∗∗−0.0191Note. ∗ and ∗∗ indicate significant correlation at Person <0.05 level and the extremely significant correlation at Person <0.01 level respectively.


## 5.4. Multiple Linear Regression Analysis

According to Tables6 to 8, each factor is significantly correlated with the precision, but the degree of explanation is generally not high, indicating that there may be more factors that affect the F value that have not been discovered. In the analysis of these four languages, Portuguese has the smallest Durbin-Watson value of 1.897, indicating that the samples are highly independent of each other, and the minimum Variance Inflation Factor, VIF for Portuguese is 1.018, suggesting that the variables in the model have no multidisciplinary. Generally, this study has certain statistical significance. The regression formulas for each language are as follows:(3)FMandarin=98.18−0.032∗WC−0.083∗OC+0.175∗PC+0.084∗MD,FCantonese=98.678−0.069∗WC+0.039∗OC+0.271∗PC−0.0007∗MD,FEnglish=99.363−0.081∗WC−0.35∗OC+0.215∗PC−0.027∗MD,FPortuguese=99.359−0.024∗WC−0.336∗OC−0.03∗PC−0.077∗MD.Table 6
Summary of models in the four languages.
LanguageRR SquareAdjustedR squareError in standard estimationDurbin-Watson valueMandarin0.311a0.0960.0964.463501.920Cantonese0.325a0.1060.1064.559522.025English0.411a0.1690.1694.856421.983Portuguese0.120a0.0140.01412.218371.897Table 7
ANOVAa of the four languages.
LanguageSum of squaresDegree of freedomMean squareFSignificanceMandarinRegression113769.507337923.1691903.4990.000bResidual1065196.3955346619.923Sum1178965.90353469CantoneseRegression91362.407330454.1361464.9010.000bResidual771300.7513710120.789Sum862663.15837104EnglishRegression302657.4703100885.8234277.5820.000bResidual1489119.4746313923.585Sum1791776.94463142PortugueseRegression16392.46435464.15536.6010.000bResidual1115333.9797471149.288Sum1131726.4437474a. Dependent variable:f, b. Predictive variable: (constant) WC, OC, CF.Table 8
Regression coefficients for various languages.
LanguageFactorNonstandard coefficientStandardization coefficienttSignificanceCollinearity statisticsBStandard errorBetaToleranceVIFMandarin(Constant)98.180.26377.520WC−0.0320.001−0.172−29.9100.5571.795OC−0.0830.033−0.011−2.5340.0110.9081.101PC0.1750.0090.11320.18600.591.695MD0.0840.0410.0092.040.04111Cantonese(Constant)98.6780.123802.950WC−0.0690.002−0.311−37.3100.3722.69OC0.0390.0220.0111.7960.0730.6771.478PC0.2710.010.19826.23200.4512.216MD−0.0070.012−0.003−0.6020.5470.9991.001English(Constant)99.3630.107928.430WC−0.0810.001−0.279−59.3500.6791.474OC−0.350.052−0.026−6.77100.9781.022PC0.2150.0080.13228.39300.6891.452MD−0.0270.009−0.012−3.1310.00211Portuguese(Constant)99.3590.633157.030WC−0.0240.005−0.065−4.59500.6681.498OC−0.3360.158−0.025−2.1270.0330.9661.036PC−0.030.049−0.009−0.620.5360.6611.514MD−0.0770.05−0.018−1.540.1240.9951.005


## 5.5. Comparative Analysis

This paper calculates the average precision rate of each user after aggregating the data according to user ID as shown in Figure8. It can be found that the results of the four languages are concentrated in the interval of 99% to 100%, indicating that it is in a relatively good overall annotation condition. There are no users with 100% of precision in the annotations of Portuguese, which refers that in the case that the annotator has mastered a certain language in a lower degree, the precision rate cannot reach a satisfactory level even though training and machine annotation results are provided for assistance in advance. The comparative analysis is aimed to locate the distribution of the precision of annotation users in these four languages, and further obtain the level of differentiation between different annotators.Figure 8
Distribution of the average precision rate of users in the four languages.


## 5.6. Cluster Analysis

According to Tables9 to 10, the cases of Mandarin, English, Portuguese and Cantonese were clustered analyzed by K-means fast clustering method, and the final clustered results shows that the total number of cases classified as 1 and 3 in Mandarin is 4624, accounting for about 8.65% of the whole; the total number of cases classified as 1 and 2 in English is 30, accounting for about 0.05%; the number of cases classified as 1 and 3 in Portuguese was 14, accounting for about 0.19%; and the number of cases classified as 1 and 3 in Cantonese was 3893, accounting for about 10.49%. There are no obvious categorical features among the four language cases. Cluster analysis is a basic statistical method in statistics. Generally, if each group has a more obvious characteristics, the number of each group will be relatively uniformly distributed during classification. The basic principle of cluster analysis is that birds of a feather fly together, that is, classify the same type of things into the same category to find out the features of annotation users, and the analysis here suggests that there is not obvious characteristics between users.Table 9
Clustering and sub-centers of the four languages.
Final cluster center (CNS)Final cluster center (ENG)Final cluster center (POT)Final cluster center (YUE)ClusterClusterClusterCluster123123123123WC440.7524.7689.312006.00381.8627.93438.5031.82947.0020.4776.901356.00OC4.330.180.630.001.960.140.000.210.000.602.7030.00PC19.299.6214.9610.0018.9611.7215.4210.8417.008.3915.1518.00AL0.29546701060.51401991670.44688561100.740.670.470.86906302840.48959103640.90981735150.49505270830.42572337170.1025641030CF0.75754857350.97983107640.87505186250.010.800.980.90239541000.86980411400.49994146500.97275258740.90844607810.1665387700MD6.006.066.0612.009.368.518.928.5110.008.618.729.00Table 10
Number of clustered cases in the four languages.
Number of cases in each cluster (CNS)Number of cases in each cluster (ENG)Number of cases in each cluster (POT)Number of cases in each cluster (YUE)Cluster163.0002.00012.00033212.000248844.00028.0007461.0003892.00034563.00063113.0002.0001.000Effective53470.00063143.0007475.00037105.000Missing0.0000.0000.0000.000


## 6. Results

Based on the experiments in Sections5.2 to 5.6, it is found that:(1)
The six factors that this paper tries to present can significantly affect the annotation accuracy of languages, among them, WC, OC, CF affect the F value in the multilingual annotation, while PC has no significant effect on Mandarin, AL is not significant for Mandarin and Cantonese, and MD is not significant for Cantonese and Portuguese;(2)
Among the models of all languages, the multiple linear regression model of Portuguese is generally not highly explanatory but has a high significance, indicating that although the factors in this paper are reasonable, more factors that affect the accuracy of manual annotation have not been discovered and need to be further explored;(3)
According to the regression results, OC is more influential in English and Portuguese than in Mandarin and Cantonese, which may be because that the annotation users of English and Portuguese in this study are not native speakers and are still in the learning stage, while the basic morpheme of these two languages is “word,” therefore, the degree of mastery of vocabulary and understanding of corpus greatly determines the accuracy of annotation. To ensure the quality of the corpus in a multilingual environment, native speakers or long-term users of such languages should be the first options for the annotation crowdsourcing task. Secondly, considering the factor WC, the corpus segmentation should be reasonable to avoid too long sentences or too many uncommon words which will result in the problem of large fluctuations in task difficulty. If there are many uncommon words in the corpus to be annotated, the relevant information should be checked before annotating it. The crowdsourcing system itself should also provide users of certain support means, such as dictionaries, part-of-speech annotating sets, and sample corpora, for reference;(4)
Observe the standardized coefficients, CF accounts for a higher proportion in these four languages, indicating that excessive revision to the results of machine annotation will reduce the accuracy of the annotation. It involves the annotation rules, which can be solved by negotiating the annotation granularity in advance, and promptly raising and giving feedback on annotation details;(5)
. PC can reflect the complexity of the parts of speech in the corpus, and it can also reflect the difficulty of the corpus itself, which has the highest impact on the annotation of Cantonese, the reason of which may be that many words in Cantonese have broader meanings than Mandarin. For example, the word “纸” (Zhi) in Mandarin only represents various kinds of papers, but in Cantonese it refers to various meanings such as banknotes, forms, certificates, documents, etc., in addition, Cantonese still retains a lot of ancient expressions and words in both oral and written Cantonese, and its parts of speech are more flexible. In order to ensure the quality of annotation, corpus with large part-of-speech complexity can be preferentially assigned to annotation users with higher evaluations, and expert users also need to pay more attention to this type of corpus to ensure that the annotation results are correct before entering them into the database;(6)
MD has minimal impact among the four factors, which indicates that the working hour period has little effect on the accuracy of the results in the multilingual corpus annotation crowdsourcing task. When making work arrangements, users can choose to work in any efficient and comfortable time according to their own wishes to finally complete the due total workload, boredom should be avoided to reduce the quality of annotation.


### 6.1. Limitations

(1)
Classification of corpus sources according to languages can further classify accuracy of corpus annotation for different subjects with observing the influence of various factors based on the content;(2)
In terms of corpus selection, it does not cover all media websites in GBA, and the web page also has a certain anti-crawling mechanism due to the sensitive information in some corpora. Therefore, even the same website cannot achieve full coverage of the section;(3)
The degree of interpretation of the model is not high, and more factors are needed to be considered, however, the data collected through the system at this stage is relatively limited;(4)
In terms of the selection of non-expert users, since most of the annotators are college students, only Mandarin and Cantonese annotators are native speakers, while English and Portuguese annotators are still in the learning stage, which has a certain impact on the accuracy of annotation.


## 6.1. Limitations

(1)
Classification of corpus sources according to languages can further classify accuracy of corpus annotation for different subjects with observing the influence of various factors based on the content;(2)
In terms of corpus selection, it does not cover all media websites in GBA, and the web page also has a certain anti-crawling mechanism due to the sensitive information in some corpora. Therefore, even the same website cannot achieve full coverage of the section;(3)
The degree of interpretation of the model is not high, and more factors are needed to be considered, however, the data collected through the system at this stage is relatively limited;(4)
In terms of the selection of non-expert users, since most of the annotators are college students, only Mandarin and Cantonese annotators are native speakers, while English and Portuguese annotators are still in the learning stage, which has a certain impact on the accuracy of annotation.


## 7. Conclusion

The construction of multilingual word segmentation and corpus are of great significance to language research in the GBA. This paper explores the factors that affect the actual accuracy of users of crowdsourcing annotation with tools such as big data analysis and SPSS based on the experiment of crowdsourcing system in corpus annotation, and study the practical problems arising from the combination of crowdsourcing and corpus. The selection of annotators in different languages, the schedule, and the cutting methods, the distribution methods, and complexity of the corpus itself will affect the annotation results to a certain extent, which should be taken into consideration in conjunction with the actual situation to ensure the quality. In the future, we will further expand the range of languages including Hakka Chinese, and continue to explore how to better integrate the crowdsourcing system with language research issues.



---

*Source: 4815254-2022-09-30.xml*