Addressing Linguistic Bias through a Contrastive Analysis of Academic Writing in the NLP Domain
Authors: Robert Ridley
Zhen Wu
Jianbing Zhang
Shujian Huang
Xinyu Dai
Venue: EMNLP 2023 Main (2023)
Keywords: contrastive analysis
linguistic bias
lexis
morphology
syntax
cohesion

Abstract: It has been well documented that a reviewer’s opinion of the nativeness of expression in an academic paper affects the likelihood of it being accepted for publication. Previous works have also shone a light on the stress and anxiety authors who are non-native English speakers experience when attempting to publish in international venues. We explore how this might be a concern in the field of Natural Language Processing (NLP) through conducting a comprehensive statistical analysis of NLP paper abstracts, identifying how authors of different linguistic backgrounds differ in the lexical, morphological, syntactic and cohesive aspects of their writing. Through our analysis, we identify that there are a number of characteristics that are highly variable across the different corpora examined in this paper. This indicates potential for the presence of linguistic bias. Therefore, we outline a set of recommendations to publishers of academic journals and conferences regarding their guidelines and resources for prospective authors in order to help enhance inclusivity and fairness.

Addressing Linguistic Bias through Contrastive Analysis of Academic Writing in the NLP Domain
Xinyu Dai National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Robert Ridley
Zhen Wu* Jianbing Zhang Shujian Huang
Collaborative Innovation Center of Novel Software Technology and Industrialization, China
robertr@smate www...jju...... edu. nju. {wuz, zjb, huangsj daixinyujenju.edu.. edu.ccn
Abstract
even if the quality of the content is sufficient and communicabillty isn't impacted. This is supported by Strauss (2019); Yen and Hung (2019), who con- clude that reviewers are more likely to accept pa- pers whose language is more nativelike.
It has been well documented that a reviewer's opinion of the nativeness of expression in an academic paper affects the likelihood of it be- ing accepted for publication. Previous works have also shone a light on the stress and anxiety authors who are non-native English speakers experience when attempting to publish in in- lernational venues. We explore how this might be concern in the field of Natural Language Processing (NLP) through conducting a com- prehensive statistical analysis of NLP paper abstracts, identifying how authors of different linguistic backgrounds differ in the lexical, mor- phological, syntactic and cohesive aspects of their writing. Through our analysis, we identify that there a number of characteristics that are highly variable across the different corpora examined this paper. This indicates potential for the presence of linguistic bias. Therefore, we outline a set of recommendations to pub- lishers of academic journals and conferences regarding their guidelines and resources for prospective authors in order to help enhance inclusivity and fairness.
To make steps towards addressing the issue of linguistic bias, we conduct comprehensive con- trastive statistical analysis of paper abstracts from the NLP domain, where we investigate how writing differs between authors from different linguistic backgrounds. We choose this domain due to its value the NLP community, where computational methods are used to study language and linguistic expression is seen as important.
In our analyses, we explore aspects of Orga- nizational Competence, outlined by Bachman et al. (1990). More specifically, we focus on the sub-components of Grammatical Competence and Textual Competence, within which we analyse how lexical, morphological, syntactic and cohesion usage varies for authors from different geographi- cal locations. Through our analyses, we identify a number of charactcristics that vary widely between writers from different linguistic backgrounds. For instance, we identify that in the China, Japan and India corpora from dataset, there are varying preferences for specific lexical bundles, and all cor- pora in our dataset have vast differences in the discourse connectors that are preferred.
1 Introduction
Artificial Intelligence (AI) research has experi- enced boom in recent years, seeing a significant increase in published works from across the globe. With the international language of academia being English, a significant number of publications written by non-native speakers of English. This phenomenon brings about the issue of linguistic bias (Politzer-Ahles et al., 2016; Hanauer et al., 2019; Flowerdew, 2019; Politzer-Ahles et al., 2020; Soler, 2021).
Therefore, to support writers whose usage pat- terns may impact the expression of the ideas pre- sented in their work and enhance their chances of being evaluated fairly in international academic publications, we also outline dations for conferences and journals regarding the author guidelines, resources and tools they make available.
of recommen-
Described by Politzer-Ahles et al. (2020), lin- guistic bias, also known as linguistic injustice, is the case of academic writing being judged more harshly by reviewers and editors if it doesn't 't meet the standards of international academic English,
In summary, the contributions of our work are as follows:
To explore how linguistic bias might be present NLP, we are the first to perform
Conresponning author

---

result of comprehensive contrastive analysis of aca- demic NLP writing from authors of different native-language backgrounds.
Syntax Comparisons of syntax are typically made through the lens of complexity at different levels, such as phrasal, clausal and sentence. Pre- vious works have identified developmental stages for syntactic complexity in second-language learn- crs of English. For example, Biber et al. (2011) proposed that complement clauses are developed first and then phrasal modifiers acquired later, Other research, however, has found that these de- velopmental stages are not always observed. For instance, Lu and Ai (2015) recognized that Chinese, Japanese and Russian writers use far less subordi- nation than both native English speakers and other non-native English speakers. They also recognized that Chinese writers use fewer coordinating clauses than writers from other backgrounds. The authors suggested that these observations native-language influence.
Through our analysis, identify how writers of different nationalities differ in their expres- sion across the dimensions of lexis, morphol- ogy, syntax and cohesion, and conclude that there are many aspects of writing that may give rise to linguistic bias.
We determine that action is required to address the issue of linguistic bias, and detail set of recommendations to help alleviate the issue.
2
Related Work
Some of the earliest work related to differences in language output based on one's native language comes from Lado (1957), who posited that features of the target language that differed greatly from a learner's ss native language would be more difficult for the learner to learn. Conversely, features of the target language that similar a learner's native language would be easier for the learner learn.
Cohesion is common for researchers to com- pare the usage of discourse connectors different groups of writers. For instance, Milton and Tsang (1993) compared native English speakers and Can- tonese speakers in their use of 25 connectors in the categories of additive, adversative, causal and sequential. They found that the Cantonese speak- ers had a considerably higher usage for a number of connectors such as firstly, secondly, lastly, be- sides, moreover and therefore; and lower usage for connectors such as likewise and previously.
More recently, with the support of learner cor- pora, such as Granger et al. (2020), researchers have sought to identify how writers of different native languages differ in their writing when com- pared with native English speakers. We summarize findings of works focusing on lexis, morphology, syntax and cohesion below.
Existing works on contrastive analysis in second-language setting generally focus on analysing writing by language learners under- graduate students. Moreover, they limited in their scope, typically focusing on one area of lan- guage e.g., syntax (Wong and Dras, 2009) or cohesion (Milton and Tsang, 1993). This study differs from previous research in that it is the first to perform a contrastive analysis of professional academic writing in the NLP domain. Thus, we are able to analyze characteristics specific aca- demic writing in target domain and can give more relevant recommendations. A further point of difference is that we carry out more compre- hensive analysis than previous research, focusing on wider range of language features.
Lexis Existing works on lexis have identified that measures of lexical diversity, sophistication and density are higher for non-native English-speaking students with higher proficiency (Memari, 2021; Prados, 2010).
Non-native English speakers also tend more lexical bundles and use them more frequently than native English speakers (Hyland, 2008a; Wei and Lei, 2011; Pan et al., 2016; Bychkovska and Lee, 2017) in order to produce text that appears more native-like.
Morphology It has been found that it is possible for morphological complexity to reach native-like levels for second-language speakers (Brezina and Pallotti, 2019). However, it is common for com- plexity to be lower for those whose native language is less morphologically complex than the target lan- guage, as they may have yet to acquire the target structure.
Data Collection
Our data' are acquired from the following aca- demic NLP venues: ACL 2018-2280 10211 EMNLP
'Our data and analysis code are available at https:// github. ctw/roberttteeeeoeoooooooeeeecec http:/bberrrrdddyyyeele hideerrrrrrdey/nnisssiici...

---

Abstracts
Word Count
Location
ACL
EMNLP
arXiv
All Data
ACL
EMNLP
arXiv
China
487
2.442
3,575
69.286
92.755
389.081
551,122
United States
257
1.492
2,105
34,269
49,303
226,036
309,608
Germany
106
603
833
13,877
16,692
89,555
120,124
Japan
90
305
503
10.857
13,853
44,957
69,667
India
59
338
452
8,753
7,193
53,839
69,785
Total
999
1,289
5,180
7,468
137,042
179,796
803,468
1,120,306
Table 1: Dataset Statistics (punctuation removed)
2017-2021 and "Computation and Language' tagged papers from arXiv.org published between January 2015 and April 24 2022. After collect- ing abstracts from these venues, we apply set of heuristics (detailed in Appendix A) to determine the nationality of the authors.
Location
Token- Type Ratio
Chain Length
Chima
0.61
1.36
United States
0.63
1.39
Germany
0.63
1.36*
Japan
1.33*
India
0.61*
1.34
Table 2: Token-Type Ratio (calculated as unique tokens divided by total tokens) and Average Lexical Chain Length. A pairwise t-test is carried out for the United States corpus against each the p-value set to 0.05, statistically significant results are marked with *
In order to ensure we have sufficient quantity of text on which to perform our analyses, we limit analyses to the five most common geographic locations in the dataset. An overview of our data is shown in Table
other corpora. With
4 Analyses
of connectors that are used to much higher extent than in any other corpus.
As outlined in Section 1, we perform contrastive analyses across the lexical, morphological, syn- tactic, and cohesion dimensions of the different geographic locations in our dataset.
The details of our analyses and findings are out- lined in the following subsections.
4.1 Lexical
In summary, we find the following:
We analyze the lexical characteristics of each lo- cation at two different levels. First, we look at lexical diversity at token level through calculating the token-type ratio. Then, we explore lexis at a multi-word level through a lexical-bundle analysis, where we identify commonly used lexical chunks.
Lexis In a number of corpora in our dataset, syn- onyms, hypernyms and hyponyms are used at a lower rate, resulting in lower lexical complexity. In addition, in these corpora there is an increase in usage of fixed lexical expressions. This is evident in the China, India and Japan corpora, for exam- ple, where the usage of fixed lexical expressions is roughly double that of the United States corpus.
4.1.1 Lexical Diversity
Analysis Methodology To inspect lexical diver- sity, we aim to capture the proportion of unique words each abstract. there is higher propor- tion of unique words, this will indicate that a larger range of vocabulary is used and thus a higher de- gree of lexical diversity. We measure this through calculating each abstract's token-type ratio, which is carried out by dividing the number of unique words by the number of total words.
Morphology The distributions of verb tense us- age are generally similar across the corpora. How- ever, we notice that in the Japan corpus past tense verbs used far more frequently and non-third person singular present verbs far less frequently than in all other corpora.
Syntax The United States corpus typically ex- hibits less syntactic complexity than all other cor- pora at phrase level. However, it is found that there is more complexity at clause level and sentence level.
In addition, we also aim quantify the usage of synonyms, hypernyms and hyponyms. To do this we perform a lexical chain analysis. First, we extract all nouns from each abstract. Then, utilizing the WordNet interface provided by the python NLTK package?, words that are considered
Cohesion In the United States corpus, there is greater usage of discourse connectors. However, each of the other corpora has its own specific set
hhtts://www.ltt..org

---

Location
Bundles per MW
Unique Bundles
NP
PP
Clause
Conj
Other
China
10,422.37
9
19
United States
4,521.85
3
10
Germany
5,527.62
4
12
0
Japan
9,344.45
5
India
8,999.07
13 19
16
Table 3: Four-wort Bundle Statistics: Bundles per MW refers the number of four-word lexical bundles per million words, Unique Bundles to the number of unique bundles, NP noun phrase-based bundles, PP preposition phrase-based bundles, verb phrase- based bundles, Clause clause-based bundles, and Conj conjunctions.
Bundle
China
United States
Germany
India
Japan
this paper we this work we this paper proposes a this paper presents a in this study we this study focuses on this paper describes our this paper describes the Total
2866.85 888
1343.63
1215.41
2221.11
1794.25
593.34
807.47
907.4
673.5
315.79
143.34
83,98
272.73
79.84
71.65
86.12
83,98
83.25
602.87
114.83
186.29
86.12
143.3
3683,4
2319.06
2206.6
3295.85
3272.71
Table
Bundle counts (per million words) for function of introducing main idea of paper.
synonyms, hypernyms or hyponyms are combined into the same lexical chain. Once these lexical chains have been generated, we calculate the av- erage chain length. Longer chains indicate more usage of synonyms, hypernyms and hyponyms.
and appearing in at least one percent of abstracts for the specific corpus. A description regarding our selection of these criteria and additional processing is covered in Appendix B.
Table shows the normalized bundle counts (number of bundles per million words) for the dif- ferent geographic locations our dataset. also displays the total number of unique bundles, with each unique bundle being classified according to whether is noun paarase-based, preposition phrase- based, verb phrase-based, clause-based, conjunc- tion other. In classifying each bundle, use the categories provided by Lu and Deng (2019); we additionally add category Other for bundles that do not fit the predefined categories. For explana- tions of each category and corresponding examples, please refer to Table 8 in Appendix B.
Findings The results of our analysis played in Table 2. For results from each of the pub- lication venues in the dataset, we refer the reader to Table in Appendix C.1.1. From looking at the Token- Type Ratio column, we can see that the United States and Germany corpora have the high- est ratios, which indicates a higher degree of lex- ical diversity. We suggest that this is a result of increased usage of synonyms, hypernyms and hy- ponyms. This is supported by lexical chain analysis, which is displayed in the Chain Length column of Table 2. The average lexical chain length in the United States corpus is the longest all cor- pora, meaning that on average there is the use of a wider range of vocabulary to describe similar con- cepts. In contrast, the Japan and India corpora have the shortest average chain lengths, meaning that decreased use of semantically-relater terms is likely factor in their lower token-type ratio scores.
dis-
Findings From looking at the number of bundles per million words in Table 3, we see that in three corpora -Chiaa, Japan and India lexical bundle usage is more than double that of the United States corpus. Additionally, while not to the same degree, the Germany corpus also exhibits higher lexical bundle usage than the United States corpus, with an increase of over 1,000 more bundles per million words. As can also be observed, not only are bundles used more frequently in the corpora from non-native English speaking locations, but the range of bundles used is also higher. We can see that the number unique bundles for the China, Japan and India corpora (Unique Bundles column in Table 3) are all at least twice that of the United
4.1.2 Lexical Bundles
Analysis Methodology We perform lexical- bundle analysis with the goal of capturing com- monly used lexical sequences for different groups of writers. In our analysis, we collect all instances of four-word bundles that meet the criteria of oc- curring 20 times per million words in the corpus

---

States corpus.
Distributions of Verb Forms
Our findings are consistent with a number of previous lexical-bundle analysis studies, which observe that writers using their second language use lexical bundles more frequently than writers who are writing in their native language (Hyland, 2008a; Wei and Lei, 2011; Pan al., 2016; By- chkovska and Lee, 2017). As described in By- chkovska and Lee (2017), writers using their sec- ond language rely on using fixed expressions in order to help them produce more academic-like texts and to avoid producing expressions that could be perceived as unconventional. We examine this phenomenon more closely by investigating bundle usage for specific function within the abstract, which is that of introducing the main idea of the pa- per. The normalized bundle counts for this function are displayed in Table 4.
VBD
VEN
VB2
VBG
VB
0
0.2
0.3
0.4
0.5
0.6
Japan
India
China
United States
Germany
Figure I: Distribution of Verb Forms for each corpus: VBD denotes past tense form (e.g.. they thought), VBN past participle (e.g,, a sunken ship), VB verb's 's base form (e.g., think), VBZ hird--ersoon singular present (e.g., she thinks), VBP non-third person singular present (e.g., I think) and VBG gerund (e.g., thinking is fun).
Here, is apparent that in the China, Japan and India corpora, there is a higher reliance on using fixed expressions to carry out the function of intro- ducing the main idea of the paper, with the total number of bundles used by these nationalities be- ing between 41% % (for the Japan corpus) and 69% (for the China corpus) greater than in the United States corpus. Interestingly, however, these three corpora differ vastly in the bundles they use. For instance, the China corpus has the highest propor- tion of bundle counts for this function, but these are distributed across only four different bundle types (in this paper we, in this work we, this paper pro- poses a and this paper presents a). From looking at the results, the high use of the single bundle in this paper we seems to be the main reason why bun- dle usage high for this particular function, with usage being more than double that of the United States corpus.
times more frequently than any other corpus. Like- wise, it is the only corpus that registers usages for this focuses on.
In contrast, the Germany corpus features less bundle usage for this particular function, with total usage slightly below that of the United States cor- pus. This likely means that on average there is less usage of fixed expressions in order to carry out the function of introducing the main idea of a paper.
4.2 Morphological
Analysis Methodology To analyze the morpho- logical dimension, we investigate the distributions of different verb forms by writers from each geo- graphic location. Specifically, for each sentence in the respective corpus, we identify the main verb and classify3 the verb form according to whether it is past tense, past participle, base form, third-person present, non-third person present gerund. The distribution results are displayed in Figure 1. For results of each publication venue in the dataset, we refer the reader to Figure in Appendix C.1.2
While the Japan and India corpora both have preference for using the bundle in this paper we, is to a lesser extent than in the China corpus. In contrast to the China corpus, the high bundle use in these two corpora appears to be as a result of using a wider range of bundles, with both corpora recording usages of five and seven bundles respec- tively. Moreover, through observing the types of bundles used, can also identify differences in terms of lexical preferences. For example, in the Japan corpus, there is preference for the use of the other corpora. The bundle in this study we is used over 600 times per million words in the Japan corpus, more than
Findings One key point of salience that can be observed is that in the Japan corpus the past tense is used significantly more frequently than in each of the other corpora in our dataset; 12.4% of all main verbs in the Japan corpus are in the past tense form. This is more than double that of the India corpus (5. (5.2%), which is the next highest. The increased use of the past tense in the Japan corpus appears
study when compared
We utilize the python package spaCy: https://spcc.iio/

---

be a result of a decreased use of non-third person present tense (only 45.5% in comparison to 56% in the United States corpus).
exhibiting high phrasal complexity is shown here:
This article describes an efficient training method for online streaming attention-bar based encoder (AED) automatic speech recognition (ASR) sys- tems.
4.3 Syntactie
Analysis Methodology In our analyses of syntax, we explore the complexity phrase-level, clause- level and sentence-level. To do this, we employ a variety of measurements: the number of noun- phrase modifiers, the number of clauses per sen- tence, the average parse-tree depth, and the average sentence length.
The sentence contains long noun phrase (high- lighted in red). The head noun of this phrase is systems, which is modified by three phrases preced- ing it (online streaming, attention-base based encoder- decoder and automatic speech recognition). This sentence could also be expressed through the use of subordinate clauses. For example, as follows:
Location NP
Cl per
PT
Sent Length
Mods
Sent
Depth
China
1,75
1.88
6,24*
24.56*
This article describes an efficient train- ing method for online streaming au- tomatic speech recognition (ASR) sys- tems based on allention-based encoder- decoder (AED) architectures.
United States
1.62
2.04
6.82
26.33
Germany
1.63
1.88*
6,64*
25.06*
Japan
1.67*
1.92
6.51*
24.64*
India 1.70
1.69
6.62*
24.03*
Table 5: Syntactic Complexity Measures (NP Mods is the average number of noun-phrase modifiers per noun phrase, per Sent the average number of subordinat- ing clauses per sentence, PT Depth is the average parse tree depth for sentence, and Sent Length is the average number of tokens per sentence). Pairwise t-tests are used to compare the United States corpus with all other corpora. Statistically significant results with a p-value of 0.05 are notated with an asterisk (*).
As we can see, re-writing the sentence in this way would decrease the phrasal complexity, as now the constituents are broken up into smaller, less complex phrases (highlighted in red and blue). The sentence length and clause counts have also been increased as a result. This appears go some way to explaining why this corpus has slightly less com- plexity at sentence level and clausal level but more complexity a phrasal level when compared with the United States corpus. When we look at the aver- age parse-tree depth and sentence length measure- ments, we also observe overall sentence complexity to be higher in the United States corpus and lower for the three corpora with the highest measures of phrasal complexity. This indicates that the com- plexity sentence level in the United States corpus is likely as a result of increased clausal complexity.
Findings The results of our analyses are dis- played in Table 5. Results across each publica- tion venue in the dataset can be found in Table 10 from Appendix C.1.3. In Table 5, we can see that the United States corpus has the fewest number of noun-phrase modifiers on average. In contrast, the location with the highest phrasal complexity by this metric is China, which, along with India and Japan, possesses a significantly higher degree of phrasal complexity than the United States corpus.
4.4 Cohesion
Our findings have lot in common with those of Lu and Ai (2015), who analyse papers written by English language learners of a variety of nation- alities and find that Chinese writers exhibit higher degrees of phrasal complexity than they do clausal complexity. Likewise, they observe that Japanese writers also possess lower degree of clausal com- plexity. However, in contrast to our findings, they find that Japanese writers express lower level of phrasal complexity than native English speakers.
Analysis Methodology As with previous works which have explored cohesion in the writing of second-language speakers (Milton and Tsang. 1993; Narita et al., 2004; Field and Oi, 1992; Goldman and Murray, 1992), we analyse cohesion through the use of cohesive devices and discourse connectors. We record all uses of the connectors provided by Kalajahi et al. (2017). In this work, the authors provide list of 632 discourse connec- tors, which are divided into eight categories and a further 17 sub-categories. Summaries of these cate- gories and sub-categories can be found in Table 12 in Appendix D.
We explore the preference for phrasal complex- ity as opposed to clausal complexity in the Japan corpus in our dataset. An example of a sentence

---

Location
Connectors per Sentence
between sentences. For example, Zhou and Xue (2012) discover that 82% of tokens in their Chinese Treebank exhibit implicit relations, as opposed around 54% in the English PDTB 2.0.
China
1,89*
United States
1.98
Germany
1.98
Japan
1.82*
India
1.81
We also recognised that there are a number of cases where certain connectors have high usage rates across the different corpora, illustrated by Table 7. For instance, in the China corpus, the con- nector with the highest usage ratio (in comparison to the United States corpus) is firstl, whose usage 1.49 times more frequent than in the United States corpus.
Table 6: Average number discourse connectors. pairwise 1-test carried out for the United States corpus against all other corpora. Statistically significant results are denoted with when the p-value is set to 0.05.
Location
Connector
Usage Ratio
firstly
11.49
besides usually
10.60
China
5.53
The connector firstly is often used either to give an argument or introduce point for non-sequential ideas. However, it appears that in the China corpus there is preference for using it as sequential connector. To examine this, calculated how frequently then is used in the sentence following the use of firstly, as these two terms are used logether to describe event sequences. the China corpus, 33% of the time firstly appears, the next sentence contains the connector then. Here is an example:
meanwhile
5.18
matching
3.45
the basis of
12.25
accordingly
8.91
Japan
considering
4.45
simultaneously
4.28
hence hence
3.77
7.18
in comparison
4.44
India
namely in all right the sume time
4.11
3.85
3.33
6.87
to this end
3.25
Firstly, we extract syntactic indicators under the guidance of syntactic knowl- edge. Then we construct neural net- work to [...]
Germany
mostly thereby
3.15
3.09
hence
2.97
Table 7:
Five discourse connectors with highest usage ratio when compared with United States corpus
In this case, the two contributions of the work are introduced as two sequential items. While the order in which the contributions were completed may be sequential, presenting them sequentially is a style choice more common in the China corpus than in others; in all other corpora (except for one instance in the United States corpus) there are no cases of firstly being used with then. This pattern in the China corpus is in contrast with how these sequential adverbs are used in the other corpora. For example, in the United States corpus, firstly often used with secondly to introduce linked ideas, rather than to introduce ideas or actions as a sequence:
For each corpus, we calculate the average num- ber of discourse connectors per sentence. These results are summarized in Table Results for each publication venue can be found in Table 11 in Appendix C.1.4. Additionally, to gain a better understanding of connector usage, in Table 7 we display the five connectors in each corpus with the highest ratio of usage in comparison the United States corpus.
Findings From Table 6, we can see that discourse connectors are used more frequently per sentence in the United States and Germany corpora, with the other three corpora showing less frequent us- ages. There are some interesting characteristics of other languages that might give indication as to why fewer connectors are used. For instance, number of works (Wang, 2011) have identified that in Chinese, connectors are used less frequently than in English. This is a result of English expressing cohesion in a more explicit manner than Chinese, which relies more on context to imply cohesion
The purpose of this paper is two-fold; firstly, we propose novel attention model [...] Secondly, we study the inter- action between attention and syntactic structures [...]
Another connector that has a high usage ratio in the China corpus is besides. We discovered that a common usage pattern for this connector in our dataset for linking ideas either in a descriptive or

---

analytical context. This leads to besides being pre- ferred over addition or additionally in the China corpus, whereas besides appears far less frequently in other corpora of the dataset. Here an example from the China corpus:
how different pragmatic functions of the paper can be expressed. For instance, we found that different corpora our dataset had different preferences for bundle usage as a means for introducing a paper's s main idea. The author guidelines should provide multiple examples for how authors can introduce the main idea of their work, in addition to examples for other common pragmatic functions.
Results show that models outper- forms existing methods on multi-domain dialogue, giving the state-of-the-an in the literature. Besides, with little train- ing data, we show its transferability by outperforming prior best model by 13.9% on average.
Morphology As have identified, there are dif- ferent preferences for grammatical tense across the different corpora in our dataset. In academic NLP pupers, there are a number of common functions expressed, such as introducing one's contributions, citing related work, describing methods, introduc- ing experiments, discussing results, etc. Each func- tion carries its own set of expectations regarding the grammatical tense to be used (for example, which tense to use when citing literature in Related Work section). To aid writers regarding usage of grammatical tense, we suggest that clear guidelines accompanied by sufficient examples be provided regarding tense usage for specific functions.
In this short extract, the first and second sen- tences are describing two separate contributions of the research paper, rather than the second sentence being an additional point in an argument.
A connector with high usage ratios in the Japan, India and Germany corpora is hence. This seems to be as result of each of these three corpora having preference for consequential type connectors. We observe that these three corpora contain the highest proportion of consequential connectors. Moreover, there are a number of other consequential connec- tors that are used significantly more frequently than in the United States corpus. For example, thereby is the connector in the Germany corpus with the fourth highest ratio and has the sixth highest ratio in the Indian corpus; and therefore has the seventh highest ratio in the Germany corpus and the sixth highest in the Japan corpus.
Syntax From our analysis, we can see that in some of the corpora investigated, there is prefer- ence for expressing complex points through com- plex phrase structures. Increased complexity with regards phrasing can impact understanding and communicability. We suggest that author guide- lines include clear examples of how to increase readability through splitting phrases containing multiple modifiers. Journals and conferences could also provide free access to automated writing tools capable of paraphrasing, thus providing authors with ideas how rewrite complex content.
5 Recommendations
order to make progress on the issue of linguis- tic bias and based on the findings of our analyses, we outline a set of recommendations to academic journals and conferences regarding how their au- thor guidelines could support prospective authors from around the globe. Our recommendations focused on the four areas of analysis performed in this study and are summarized as follows:
Cohesion As we have shown in this study, the preference for different discourse connectors across many of the corpora investigated are highly vari- able. We recommend providing a list of common connectors and examples of how they can be used. This can also be accompanied by list of frequently overused misused connectors along with alterna- tive options for connectors.
Lexis We found variation in lexical diversity across each of the investigated corpora due to dif- ferences in the use of synonyms, hypernyms and hyponyms. We recommend that author guidelines include lists of commonly used terms in NLP re- search and include examples of suitable synonyms.
Conclusions and Future Work
This paper seeks to address the issue of linguistic bias in academic publishing. Our comprehensive contrastive analysis of academic writing in the NL.P domain identifies number of characteristics that are highly variable across writers from different
We also found that there was higher use of lex- ical bundles in number of the corpora studied. To help avoid over-reliance on lexical bundles, we pro- pose that author guidelines include suggestions for

---

nationalities. These findings highlight the poten- tial risk of linguistic bias. To mitigate this risk, we outline a set of recommendations to academic publishers about how their resources could better support writers in presenting their ideas and contri- butions and thus ensuring that their work is fairly evaluated.
writing. As such, our work seeks identify some of these differences and to provide suitable recom- mendations regarding author guidelines to assist writers with the expression of their ideas and thus to better enable inclusivity and fairness.
The findings and recommendations outlined Sec- tion 5 may be sensitive to some. Both in this sec- tion and throughout the paper, are careful to present our analyses across the different corpora objectively, terming differences as variations in production, or preferences for different production patterns, rather than to emphasize or evaluate differ- ences of proficiency between native English speak- ers and non-native English speakers.
Regarding possible directions for future work, we propose that more work be done analyze the writing of wider range of author backgrounds. Moreover, our analyses could provide a basis for integration into writing correction and sugges- tion tools. With current solutions typically focus- ing on providing corrections for grammatical or convention- based errors, there is scope to extend this to suggestions based on writing characteristics where expression may impact an author's 's chance of being assessed fairly, but not necessarily be in- correct grammatically or according to conventions.
Variations in English production can result from number of factors, such as linguistic backgrounds, educational systems, language acquisition op- portunities. It is our firm belief that such variations should not be the cause for discrimination and not be the basis for determining the value or quality of academic contributions.
Limitations
The analyses in this paper are conducted on ab- stracts of academic papers published in conferences and on pre-prin pre-print server. We ensure that all data used in our analysis were obtained legally and ethi- cally. Regarding licensing for our dataset, all data from the ACL and EMNLP conferences are cov- ered by the Creative Commons Attribution 4.0 In- ternational License, and the arXiv data are covered by Creative Commons CCO 1.0 Universal Public Domain Dedication. In order to respect the rights and privacy of the authors in the dataset, we are careful to remove any personally identifialle infor- mation.
In order to ensure that we have sufficient data for each of the geographic locations in our analysis, we limit our focus to the five most common locations in our dataset. For future research, there is value in extending the analysis to include writers from wider range of backgrounds.
Moreover, while we perform analyses on a range of language features, our focus on the grammati- cal and textual components of writing. Exploring other areas of language, such as sociolinguistic as- pects (e.g., use of register, figures of speech, and cultural references) could also be explored in future work.
We recognize the importance of responsible and ethical conduct in AI research and will continue prioritize these values in our work.
Finally, also acknowledge that, while at- tempt to identify the location of where the writers originate, we cannot determine the native language or the variety of the language spoken by each writer. For instance, in India, there are many languages spoken, and in China, while the predominant lan- guage is Mandarin, there are many varying dialects spoken across the nation.
A Acknowledgements
We thank all anonymous reviewers for their feed- back and suggestions. This work is supported by the National Natural Science Foundation of China (No. 619761 14, 622061 266 and 61 61936012).
Ethics Statement
English is considered a global language and it is natural that different preferences and patterns of usage exist for speakers from different geographi- cal locations. In this work, we do not consider any variation be correct or incorrect, but it is conceiv- able that differences in expression may impact how ideas are understood and perceived in academic
References
Annelie Adel and Britt Erman. 2012. Recurrent word combinations in academic writing by native and non- native speakers of english: A lexical bundles ap- proach. English for specific purposes, 31(2):81-44
Lyle F Bachman et 1990. Fundamental considera- lions in language testing. Oxford university press.

---

Douglas Biber, Susan Conrad, and Viviana Cortes. 2004, If you look at...: : Lexical bundles in university teach- ing and textbooks. Applied linguistics, 25(3):371- 405.
Ken Hyland. 2008b. As can be seen: Lexical bundles and disciplinary variation. English for specific pur- poses, 27( (712)24211
Seyed Ali Rezvani Kalajahi, Steve Neufeld, and Ain Nadzimah Abdullah. 2017. The discourse con- nector list: a multi-genre cross-cultural corpus analy- sis. Text & Talk, 37(3)228--31.
Douglas Biber, Bethany Gray, and Kornwipa Poonpon. 2011. Should we use characteristics of conversa- tion to measure grammatical complexity in 12 writing development? Tesol Quarterly, 45(1):5-35
Robert Lado. 1957. Linguistics across cultures: Ap- plied linguistics for language teuchers. University of Michigan press.
Douglas Biber. Stig Johansson, Geoffrey Leech, Susan Conrad, Edward Finegan, and Randolph Quirk. 1999. Longman grammar of spoken and written English, volume 2. Longman London.
Xiaofei Lu and Haiyang Ai. 2015. Syntactic complexity in college-level english writing: Differences among writers with diverse 11 backgrounds. Journal of sec- ond language writing, 29:16-27 27.
Vaclav Brezina and Gabriele Pallotti. 2019. Morpholog- ical complexity in written 12 texts. Second language research, 35(1) 1119999911
Xiaofei Lu and Jinlei Deng. 2019. With the rapid de- velopment: A contrastive analysis of lexical bundles in dissertation abstracts by chinese and english doctoral students. Journal of English for Academic Purposes, 39:21-36.
Tetyana Bychkovska and Joseph J Lee. 2017. At the same time: Lexical bundles in ll and 12 university student argumentative writing. Journal of English for Academic Purposes, 30:38-52.
Pat Byrd and Averil Coxhead. 2010. On the other hand: Lexical bundles in academic writing and in the teach- ing of eap. University of Sydney Papers in TESOL, 5(5):31-64.
Zainab Memari. 2021. Educational policies revisited: Measuring the lexical complexity of english language users in academic and technical contexts: A compar- ative study. 2021 Sustainable Leadership and Aca- demic Excellence International Conference (SLAE). pages 26-31 IEEE
Yu-Hur Chen and Paul Baker. 2010. Lexical bundles and 12 academic writing. Language learning technology, 14(2):30-99
John CP Milton and Elza Shuk-ching Tsang. 1993. A corpus-based study of logical connectors in efl stu- dents' writing: Directions for future research. In Studies in lexis. Proceedings of a seminar on lexis organized by the Language Centre of the HKUST, Hong Kong (Language Centre, HKUST, Hong Kong, 1993).
Viviana Cortes. 2004. Lexical bundles in published and student disciplinary writing: Examples from history and biology. English for specific purposes, 23(4):397 423.
Yvette Field and Yip Lee Mee Oi. 1992. comparison of internal conjunctive cohesion in the english essay writing of cantonese speakers and native speakers of english. RELC journal, 23(1):11-22)
Masumi Narita, Chicko Sato, and Masatoshi Sugiura. 2004. Connector usage in the english essay writing of japanese efl learners. In LREC, volume 27, pages 1171-1174.
John Flowerdew. 2019. The linguistic disadvantage of scholars who write in english as an additional language: Myth or reality. Language Teaching, 52(2):249-26) 260..
Fan Pan, Randi Reppen, and Douglas Biber. 2016. Com- paring patterns of versus 12 english academic pro- fessionals: Lexical bundles in telecommunications research journals. Journal of English for Academic Purposes, 21:60-71.
Susan R Goldman and John D Murray. 1992. Knowl- edge of connectors as cohesion devices in text: A comparative study of native-english and english-as- d-second-Languee speakers. Journal of Educational Psychology, 84(4):004
Stephen Politzer-Ahles Teresa Girolamo, and Saman- tha Ghali. 2020. Preliminary evidence of linguistic bias in academic reviewing. Journal of English for academic purposes, 47:1008.5.
Sylviane Granger, Maité Dupont, Fanny Meunier, Hu- bert Naets, and Magali Paquot. 2020. The interna- tional corpus of learner english. version 3.
Stephen Politzer-Ahles, Jeffrey J Holliday, Teresa Giro- lamo, Maria Spychalska, and Kelly Harper Berkson. 2016. Is linguistic injustice a myth? a response to hyland (2016). Journal of second language writing. 34:3.
David I Hanauer, Cheryl L Sheridan, and Karen Eng- lander. 2019. Linguistic injustice in the writing of re- search articles in english as a second language: Data from taiwanese and mexican researchers. Written Communication, 36( 136-1144
Mercedes Diez Prados. 2010. Gender and II influence efl learners' lexicon. In Gender perspectives on vocabulary in foreign and second languages, pages 44-73. Springer.
Ken Hyland. 2008a. Academic clusters: Text patterning published and postgraduate writing. International journal of applied linguistics, 18(1):41-62

---

Josep Soler. 2021. Linguistic injustice in academic pub- lishing in english: Limitations and ways forward in the debate. Journal of English for Research Publica- tion Purposes, 2(2):160-171
Pat Strauss. 2019. Shakespeare and the english poets: The influence of native speaking english reviewers on the acceptance of journal articles. Publications, 7(1):20.
Jianxin Wang. 2011. Contrastive connectors English and Chinese: A corpus-based study. Ph.D. thesis.
Yaoyu Wei and Lei Lei. 2011. Lexical bundles in the academic writing of advanced chinese efi learners. RELC journal,
Sze-Meng Jojo Wong and Mark Dras. 2009. Contrastive analysis and native language identification. In Pro- ceedings of the Australasian Language Technology Association Workshop 2009, pages 53-61.
Chun-Ping Yen and Tzu- Wei Hung. 2019, New data on the linguistic diversity of authorship in philosophy journals. Erkenntnis, 84(4)9953-974.
Yuping Zhou and Nianwen Xue. 2012. Potb-style dis- course annotation of chinese text. In Proceedings of the 50th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers),
pages 69-77.

---

A Appendix
B Appendix
A.1 Data Acquisition: Nationality
Determining Author
B.1 Lexical Bundles: Bundle Processing
To capture meaningful lexical sequences, we need to consider three criteria, lexical-bundle size, fre- quency and dispersion (as specified in Bychkovska and Lee, 2017).
First, attempt to determine the nation of the institution from which the paper is published. To do this, we extract the listed email address of the first author. Then, we check whether the email matches a listed university domain' and thus we can extract the nation. For emails addresses which are not matched, we extract the top-level domain and match it with a nationality if possible. Papers without email addresses or in instances where the nationality of the institution cannot be determined are discarded.
B.1.1 Bundle Size
For bundle size, follow existing works (Biber et al., 2004; Hyland, 2008b; Byrd and Coxhead, 2010; Ådel and Erman, 2012; Pan al., 2016; By- chkovska and Lee, 2017; Lu and Deng, 2019) and extract four-word lexical bundles. This is because, as described by Cortes (2004), three-word struc- tures are often contained in four-word structures (for example, Cortes, 2004 notes that result is contained in the structure tion, four-word bundles are far more common than five-word bundles, thus providing wider range of structures analyse.
Once we have collected list of papers for which we know the nationality of the publishing institu- tion, we determine whether the first author origi- nates from the same nation as the publishing institu- tion. To do this, we utilize a name origin "atabase If the nation of origin of author's family name is listed the same that of the publishing institution, we assume the author originate from that nation. All papers for which this criterion is not met are discarded. As a further post-processing step, since many papers are published on preprint servers be- fore being submitted to conferences, we are careful to de-duplicate papers in our dataset.
result of). In addi-
B.1.2 Bundle Frequency
The frequency criterion allows us to capture lexical chunks that occur frequently enough to be repre- sentative of the target register. Following previous works (Biber et al., 2004; Hyland, 2008b; Byrd and Coxhead, 2010; Ädel and Erman, 2012: Pan et al., 2016; Bychkovska and Lee, 2017; Lu and Deng, 2019), which set frequency thresholds be- tween 20 and 40 occurrences per million words, we choose a threshold of 20 occurrences per million words. Initially, we experimented with a few cut- off frequencies. Through manual inspection, we generally found that setting the frequency to 20 per 1 million words was sufficient for preventing the capture of rare cow-frequency bundles. At higher found that there was limited dif- ference in the bundles that were captured, as bun- dles that met our dispersion threshold (discussed in Appendix Section 1.33) typically occurred more frequently than 40 instances per million words.
It is not impossible that in some cases, native English speakers could appear non-first authors for papers in the non-native English speaker cor- pora. However, determined by the heuristics outlined above, we do not observe a high instance count for this being the case. addition, is gen- erally considered that first authors are the main contributors to scientific papers and we thus make the assumption that they are the writers in our cor- pora. We thus believe that the small quantity of native English speaker non-first authors in non- native English speaker corpora coupled with first authors generally making the largest contributions will mean that only limited amount of noise could be added by native English speakers these cor- pora.
frequencies,
B.1.3 Bundle Dispersion In order to ensure that tendencies of an entire group are collected and not just the quirks of particular author, dispersion threshold is required. Disper- sion is handled in number of different ways in existing research. For example. Chen and Baker (2010); Ädel and Erman (2012) require each bun- dle to appear in at least three separate texts; and Biber et al. (1999, 2004); Cortes (2004); Pan et al. (2016) require each bundle to appear in at least five
Shtps//wwww.famllyyccc..rrr

---

Category NP-based
Subcategory
Example
ol-phrase fragment
the performance of
other post-modifier fragment
the research on
Other
more more attention
PP-based
PP with embedded ol-phrase
in the form of with respect to the
Other PP fragment
VP-based
copula be + NP/adjective phrase
is one of the
VP with active verb
play an important role
VP with infinitive verb
better understand the
VP with passive verb beginning with past participle
can be used to
based on the above
cause-based
PP copula be
this thesis
NP copula be
this thesis is
Anticipatory it copula be adjective phrase
it possible
Anticipatory passive verb that
found that
NP/complementizer passive verb NP + active verb
little is known about
this thesis focuses on
NP active verb that
we find that the well the
Conjunctions
Table 8: Lexical bundle categories with examples, as outlined in Lu and Deng (2019).
separate texts. Due the imbalance of our five different corpora and the shorter length of each of our texts in comparison to previous works, we re- quire each bundle to appear in at least one percent of abstracts in the corpus of interest.
Appendix
C.1
Publication Venue Specific Results
C.1.1 Lexical Complexity
Publication Venue
Location
Token-Type Ratio
Chain
B.1.4 Handling Overlapping Bundles
Length
China
0.62
1.36
After collecting our list of four-word bundles, there is another processing step that is required and that is handle overlapping bundles. For instance, in the China corpus there are two overlapping bundles we propose novel and propose a novel method. this case, these two bundles are fact part of a five-word bundle we propose a novel method. In order to avoid inflating the bundle counts, we sub- sume the later bundle into the former, becoming propose a novel (method). In deciding which of the overlapping bundles to preserve, keep the bun- dle with the highest instance count and discard the bundle with the lower count. In the China corpus, we propose a novel occurred more frequently than propose a novel method and thus was the bundle that was retained.
United States
0.64
1.38
ACL
Germany
0.64
1.35
Japan
0.62*
1,32*
India
0.61*
1.30*
China
0.62
1.35
United States
0.64
1.40
EMNLP
Germany
0.64
1.35
Japan
0.62
1.31*
India
0.63
1.31
China
0.60*
1.36
United States
0.63
1.39
arXiv
Germany
0.63
1.37
Japan
0.60
1.34*
India
0.60*
1.36*
Table 9: Token-Type Ratio (calculated unique tokens divided by total tokens) and Average Lexical Chain Length for each publication venue. Values that are sta- tistically significant when compared with the United States corpus are indicated with (p-value of 0.05).

---

C.1.2 Morphological
C.1.3 Syntactic Complexity
Distributions Verb Forms (ACL abstracts)
Pub Venue
Location
#NP Mod
CT/
PT
Sent
Sent
Dep
Len
China
1,74
1.87*
6.26*
24.29
US
1.61
1.99
6.69
25.46
VIN
ACL
Germany
1.59
2,00
6.71
25.19
Japan
1.65
1.91
6.59
24.22*
VBP
India
1.73*
1.66*
6.51
23.66"
China
1.74*
1,89*
24.18*
2.10
6.86
VBG
EMNLP
Germany
1.64
1.88*
6.71*
24.33*
Japan
1.67*
1.91*
6.51*
23,91
India
1.76*
1.62*
6.61*
23.32"
China
1.75*
1,87*
6.24*
24.71
US
1.63
2.04
7.01
26.50
Japan Indin
China
United States
Germany
arXiv
Germany
1.64
1.86*
6.62*
25.17
(a) Verb form distributions for ubstracts published in ACL
Japan
1.67*
1.93*
6.49*
24.98*
India
1,68*
1.70
6,67*
24.19
Distributions Verb Forms (EMNLP abstracts)
Table 10: Syntactic Complexity Measures for each pub- lication venue in the dataset (#NP Mod is the average number of noun noun-phrase modifiers per phrase, Cl / Sent is the average number of subordinating clauses per sentence, PT Dep the average parse tree depth for sentence, and Sent Len is the average number of tokens per sentence). Values that are statistically signifi- cant when compared with the United States corpus are marked (p-value of 0.05).
VBN
VIZ
VBG
Japan
India
China
United Germany
C.1.4 Cohesion
(b) Verb form distributions for abstracts published in EMNLP
Publication Venue
Location
Connectors per Sentence
Distriburions of Verb Forms (arXiv abstracts)
China
VBD
United States
ACL
Germany
1.92
VBN
Japan
1.77
India
China
1.88*
V82
United States
2.00
EMNLP
Germany
1.99
VBG
Japan
1.72*
India
1.75
VB
China
1.90
United States
1.98
0.1
0.2
0.3
0.4
0.5
0.6
arXiv
Germany 1.99
Japan
India
China
United States
Germany
Japan
1.87*
(c) Verb form distributions for abstracts published in arXiv
India
1.82*
Figure 2: Distribution of Verb Forms for each corpus in each publication venue: VBD denotes past tense form (e.g., they thought), VBN past participle (e.g., sunken ship). VB verb's base form (e.g., think), VBZ third- person singular present (e.g., she thinks), VRP non-third person singular present (e.g., think) and VBG gerund (e.g., thinking is fun).
Average number of discourse connectors for each publication venue in our dataset. Statistically significant values are denoted with (p-value set 0.05) when compared with the United States corpus.
Table 11:
D Appendix
In this section, we outline the discourse connector categories and sub-categories that are adopted in our cohesion analyses. There are a total cight cat- egories: additive, apposition, consequential, com-

---

Category Additive
Description
Sub-categories
Examples
add new information
Equative
equally
Reinforcing
furthermore
Apposition
to give examples or explanation
Exemplification
for example
Restatement
in other words
Consequential
to give cause and effect relationships
Causative
due to
Resultive
because
Conditional
whether
Comparison
point out imply resemblances between objects introduce surprising contrasting information
like
Contrastive
Antithetic
then
Concessive
still
Reformulatory
better
Replacive
rather
Clarification
clarify point view, attitude statement
Equative
indeed
Emphasizing
in fuct generally
Generalization
Sequential
list main points and signal the sequence
Ordering
fist, then after, immediately
Timing
Transitional
by the way overall
Summative
to conclude or sum up preceding information
Table 12: Discourse connector categories and sub-categories, as outlined in Kalajahi al. (2017).
parison, contrastive, clarification, sequential and summative. These categories are further divided into the following sub-categories: equative, rein- forcing, exemplification, restatement, causative, re- sultive, conditional, antithetic, concessive, refor- mulatory, replacive, equative, emphasizing, gener- alization, ordering, timing and transitional.


Reviews:
[{"review_id": "L0L91YJ9Yn", "reviewer": "Reviewer_Nr5L", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation regarding the level of detail, comments regarding the robustness of our dataset creation procedure and the interest of our paper to the NLP community. We value your concerns regarding (1) the statistical significance of the results, (2) distribution of paper venues, and (3) guidelines outlined in Section 5, and address them as follows:\n\n__1__: For instances where numerical deviation between different corpora appears to be small, we include statistical significance results from a pairwise t-test for each corpus compared with the United States corpus, with the p-value set at 0.05 (statistically significant results denoted with *) and individual samples being either for each abstract (e.g., token-type ratio) or for each sentence (e.g., clauses per sentence):\n\n&nbsp;\n\n__Table__ __2:__\n|\t\t\t| Token-Type Ratio\t| Chain Length |\n| --------------- | :---------------------: | :---------------: |\n| China |\t*\t| * |\n| Germany |\t| * |\n| Japan |\t*\t| * |\n| India |\t*\t| * |\n\n&nbsp;\n\n__Table__ __5:__\n|\t\t\t| #NP Mods\t| #Clause per sent | #Parse Tree Depth | #Sent Length |\n| --------------- | :--------------: | :------------------: | :-------------------: | :---------------: |\n| China |\t*\t| * | * | * |\n| Germany |\t| * | * | * |\n| Japan |\t*\t| * | * | * |\n| India |\t*\t| * | * | * |\n\n&nbsp;\n\n__Table__ __6:__\n|\t\t\t| #Logical Connectors per Sentence |\n| --------------- | :---------------------------------------: |\n| China |\t*\t|\n| Germany |\t|\n| Japan |\t*\t|\n| India |\t*\t|\n\n&nbsp;\n\nIn the revised version of this paper, we will include statistical significance results where relevant.\n\n&nbsp;\n\n__2__: Regarding venue distributions, we list the paper counts in our dataset for each venue:\n\n| Venues | #Abstracts |\n| ----- | :-----: |\n| ACL | 999 |\n| EMNLP | 1289 |\n| ARXIV | 5180 |\n\nWhile we appreciate the concern that arxiv papers do not exhibit the same writing quality of those from ACL and EMNLP, we add that the distributions of papers from the different geographical locations are almost identical across the venues; therefore, we believe it to be unlikely that differences between the different locations can be attributed to the different publication venues. \n\nAdditionally, we have also carried out a number of experiments on papers from each individual publication venue, and have found the phenomena observed to be consistent with what is published in our paper for all three venues combined. For instance, the usage of logical connectors from each venue and for each country are displayed in the following table:\n\n&nbsp;\n\n__#Logical Connectors per Sentence__\n|\t\t\t| ACL | EMNLP | ARXIV |\n| --------------- | :----: | :-------: | :------: |\n| China |\t1.88\t| 1.88 | 1.90 |\n| United States | 1.95 | 2.00 | 1.98 |\n| Germany | 1.92 | 1.99 | 2.00 |\n| Japan | 1.77 | 1.72 | 1.86 |\n| India |\t1.82\t| 1.75 | 1.81 |\n\n\n__3__: Regarding the concern that the recommendations outlined in Section 5 may be interpreted as discriminatory, our intention is to provide assistance to non-native speakers of English based on the findings of our analysis. We will make some wording changes to Section 5 in our revised version in order to address this issue. The suggestion for reviewers to focus on the content rather than the writing patterns is good and is the ideal scenario for academic publishing; this should certainly also be recommended by publication venues. As an additional note, however, research has shown that reviewers often judge work more harshly when it is less nativelike, even when the communicability is unaffected by non-native writing patterns (examined by work cited in the second paragraph of the introduction). Thus, we believe that providing writers with ways to assist them in being evaluated on the quality of their research is highly valuable to the research community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The level of detail the authors considered for their study is very impressive. Lots of insights, some of which reiterate or contradict the findings in the literature, were mentioned. In addition to that, numerous new linguistic insights were presented, which will be a delight to read by a linguist/computational linguist. \n\n2. The heuristics considered in this study for paper selection are pretty robust. This ensures one of the founding pillars for quality dataset creation for this study. \n\n3. This study will interest to a broader NLP community especially those who like the linguistic aspect in the NLP.", "reasons_to_reject": "There are several fundamental weaknesses I find in this study. \n\n1. Authors have showcased many results showcasing the linguistic difference between native and non-native writers. Most of the time, the numerical deviation in the values is not very high. In this scenario, examining statistical significance is a primary requirement, and unfortunately, the authors have not done that for a single outcome. WIth out this, I am taking the results with a pinch of salt. \n\n2. Authors have not given any statistical distribution on paper venues. How many papers are from ACL and EMNLP vs Arxiv? Everyone knows Arxiv papers are not peer-reviewed and are hardly considered formal publications. My intuition is that it's the arxiv papers that showcase this bias. If this is the case, I would not like to consider the results presented in this paper as significant. Also, I have severe reservations about including of Arxiv papers in this study. Most of Arxiv's papers are junk and are not considered as severe writing at all. \n\n3. Most of the recommendations mentioned in section 5 are guidelines for publishers to encourage non-native users to use different writing patterns by showcasing examples. This advisory is very racist and discriminatory towards non-native writers. Instead, authors should propose guidelines for reviewers on ‘Look into the contributions, not writing patterns’. Research papers stand for their technical and empirical contributions. They are not literary contributions for which they should be advised on ‘how to write’.  Also, in the era of AI tools like Grammarly, why do the authors think that there is a need for such recommendations?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an exploratory study of linguistic variation in academic writing across geographical locations. Authors have explored linguistic variation across four dimensions, (i) Lexical, (ii) Morphological, (iii) Syntax, and (iv) Cohesion. The geographical locations they focused on are (i) China, (ii) US, (iii) Germany, (iv) Japan, and (v) India. For their study, they considered abstracts from two prominent NLP research venues, ACL and EMNLP, for papers published between 2018-2021 and 2017-2021, respectively. In addition, they considered abstracts from Arxiv papers tagged with ‘Computation and Language’ published between 1st Jan 2015 to 24th April 2022. They followed various heuristics (Appendix A) to identify the origin country of the considered papers (First author, to be precise), which sounds robust. \n\nVarious well-known approaches, such as token-type ratio, lexical chan length, lexical bundles per million words etc., were calculated to showcase the linguistic difference across the papers from the considered geographical locations. Finally, they have proposed some guidelines for the journal and conference publishers to prepare instructions for non-Native writers to reduce the bias in their writing.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Section 5 needs to pass the ethical test."}, "extra_text": {}}, {"review_id": "lrE4ZhFkFU", "reviewer": "Reviewer_QejG", "summary": "", "questions": "Please make it clear whether you are making general statements about L1/L2 or about native English vs. non-native English. The fact that this not always done by the authors you cite doesn’t make it more acceptable,\nIt would be important to discuss the linguistic supremacy of US-based academic writing and point out that English is not monolithic. For instance, why the “native” corpus is labelled “US”.  We might also expect differences between US/UK/Australian usage. If not, why not?\n\nWhy not suggest that US-based writers adopt international usage? For instance, you note that the Japanese corpus shows “under-use of first-person present tense (only 45.5% of main verbs are first-person present, in comparison to 56% in the United States corpus).” This could also be considered as over-use of first person present by US writers.  Who decides the US usage has to be the norm?\n\nThe recommendations in section 5 will be very useful for editors, reviewers, journals and conferences. But why insist that authors should display a range of variations? Formulaic language (lexical bundles) can actually be useful for both writers and readers.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on our paper regarding its importance to fairness in academic reviewing. Regarding your concerns and questions, we address them as follows:\n\n__QA__: Distinction between L1/L2 and native English / Non-native English  \n__Response__: We agree that where ‘native’ and ‘non-native’ are used it should be clearly defined as to which language is being referred to. We will clarify each usage of these terms in the revised version of this paper. Regarding the US corpus being adopted as the native corpus, we will also add clarification that while US English is dominant in academic publishing, it is not the only variation of English and thus our analyses do not necessarily extend to other locations where English is also a first language. It is possible that other varieties of English (UK, AUS, etc.) could exhibit differences in comparison to the US, particularly with word choices. For future research, it would certainly be of value to perform analyses across more locations where English is used as a first language.\n\n__QB__: Why not suggest US adopt international standards for academic writing?  \n__Response__: The suggestion for US writers to adopt international standards is something that could be considered as an ideal scenario, and is something that could potentially be explored in the long term. One of the barriers that our work attempts to make progress on is to aid writers according to existing publishing conventions, whereby a culture has developed of nativelike English being considered prestige and many publishing venues often adopting US-based style guides.\n\n__QC__: Range of variations suggestion in Section 5  \n__Response__: Lexical bundles can certainly be of benefit to writers. However, through our analysis, we have identified that there are certain bundles that are used more frequently by writers from certain geographic locations. High frequency use of lexical bundles is likely to impact how the reader evaluates the quality of the writing, as text may feel repetitive and perhaps even tedious. It is likely the case that the degree of diversity that is observed in the US corpus is a factor in how the quality of a text is perceived. Therefore, bundle diversity is likely to be useful for non-native English writers, too.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "See above", "reasons_to_reject": "None regarding the content of the paper. But the text must be made clearer regarding the terms “native speakers” and non-native speakers” and whether specific remarks apply only to writers who do not have English as a first language, or to any writing in a non-native language.\nThe authors must ensure that everywhere the terms \"native\" and \"non-native\" are used it is clear what language they are referring to. Almost everyone is a native speaker of at least one language and the generic term \"non-native speaker\", as often used to refer to non-native English speakers, is insulting and demeaning.\nThis is especially necessary in the abstract (\"non-native authors\") and the conclusion (\"native or non-native\"), but also throughout the text. There are many places discussing research in second language writing where it is not clear whether it is actually EFL (English as a foreign language) or L2 (any second language) that is being discussed.\n", "questions_to_authors": "Please make it clear whether you are making general statements about L1/L2 or about native English vs. non-native English. The fact that this not always done by the authors you cite doesn’t make it more acceptable,\nIt would be important to discuss the linguistic supremacy of US-based academic writing and point out that English is not monolithic. For instance, why the “native” corpus is labelled “US”.  We might also expect differences between US/UK/Australian usage. If not, why not?\n\nWhy not suggest that US-based writers adopt international usage? For instance, you note that the Japanese corpus shows “under-use of first-person present tense (only 45.5% of main verbs are first-person present, in comparison to 56% in the United States corpus).” This could also be considered as over-use of first person present by US writers.  Who decides the US usage has to be the norm?\n\nThe recommendations in section 5 will be very useful for editors, reviewers, journals and conferences. But why insist that authors should display a range of variations? Formulaic language (lexical bundles) can actually be useful for both writers and readers.", "paper_topic_and_main_contributions": "This is an excellent paper, which addresses a very important problem, that of linguistic bias in academic reviewing. The issue of how reviewers can be influenced by the perceived native language of authors and by prejudices against non-native English speakers/writers in an environment dominated by native English speakers must be tackled and this is a very good start. The authors not only provide examples of divergences between the lexical choices and grammatical constructions of authors of different nationalities, but also make useful suggestions for publishers and journals and conference editors. The linguistic analyses are detailed and well-documented, with good examples.\nThe data should be replicable when the authors make their data available.\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A\n\nBut (First et al.2019) should be (Education First, 2019)", "Typos_Grammar_Style_and_Presentation_Improvements": "p.8, bus -> but", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "HpmtgNP0Q4", "reviewer": "Reviewer_5giR", "summary": "", "questions": "1. How did the authors address overlapping bundles? In Sec. B.1.4, \"we propose a novel\" is selected rather than \"propose a novel method\", but how about \"a novel method for\"? What kind of algorithm was applied to this selection?\n2. In l.197, it is said that Japanese writers use a lower proportion of first-person singular present verbs. I wonder whether first-person singular present verbs are used in *ACL papers because I have rarely seen them. Could it be possible to raise some examples or number of occurrence?\n3. For syntactic analysis what kind of tool did the authors used?\n\n(all answered)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on our work. We address your concerns and questions as follows:\n\n__R1:__ Reasons why the NLP field was targeted.  \n__Response:__ In terms of performing our analysis in the NLP domain as opposed to universal domain, we believe that there are certain characteristics of writing that are specific to each domain. For example, in the NLP domain, writers are often required to write with a high degree of technical detail, often outlining algorithm details; this may impact the syntactic complexity and the sentence structures that are used for instance. We felt that performing our analysis on universal domain data might reduce the impact of some of the phenomena observed in our study. Additionally, regarding why the NLP domain was selected specifically, it is the field we work in and are engaged in, and we are committed to contributing to help the NLP community improve and thrive. \n\n__R2:__ It is difficult to recommend something about writing an article based only on the abstract analysis.  \n__Response:__ In our study, we chose to analyze paper abstracts due to the fact that abstracts are clean and often carefully written; additionally, they are all written with the same function – that is to summarise the contributions of the paper. These characteristics make abstracts easily comparable and thus well-suited for a comparative analysis. Performing an analysis of whole papers is also of value, however, with many potential new insights to be gained, and thus we reserve this for future work.\n\n__R3:__ The authors claim that lexical diversity in cn/in/ja is less than that in us/de, but there is only 0.02 point difference, which amounts to about 2-3 words in each abstract.  \n__Response:__ While the lexical diversity scores do not appear to be hugely different between the cn/in/ja corpora and the us/de corpora, we support our results with a pairwise t-test, comparing each corpus with the US corpus. In this case, each sample is the score from one abstract. The p-value is set to 0.05. The results are as follows, with statistically significant results denoted by *.\n\n__Table__ __2:__\n|\t\t\t| Token-Type Ratio\t| Chain Length |\n| --------------- | :---------------------: | :---------------: |\n| China |\t*\t| * |\n| Germany |\t\t| * |\n| Japan |\t*\t| * |\n| India |\t*\t| * |\n\nWhile lexical diversity could be down to preferences for singular nouns or certain prepositions, we believe that our lexical-chain length analysis supports the hypothesis that the higher token-type ratio score in the US corpus is as a result of more synonym, hypernym and hyponym usage.\n\n__R4:__ In the past many EAP studies used four-word lexical bundles for analysis, but as Swales (2019) criticises, they contain non-sense bundles like \"and of the\".  \n__Response:__ With regards to using four-word bundles, we found that this was the most beneficial bundle size to obtain useful insights, as three-word bundles extract many common non-sense bundles, as highlighted in the review; and five-word bundles are too infrequent. There are some extra post-processing steps we took to ensure that our bundles were informative. For instance, any non-sense bundles were removed by hand (these were very infrequent), as were any bundles that contained technical language or jargon that was relevant only for the domain in which it appears (e.g., ‘aspect based sentiment analysis’). Other methods for extracting lexical bundles such as mutual information are definitely worth exploring, and we reserve this for future research.\n\n__QA__: How are overlapping bundles handled?  \n__Response__: In the case of overlapping bundles, when choosing which of ‘we propose a novel’ and ‘propose a novel method’ to record, the bundle with the highest instance count is chosen. For ‘a novel method for’, it will also be recorded as a separate bundle if it is not part of another overlap with a higher instance count (e.g., if ‘propose a novel method’ has a higher instance count, then ‘a novel method for’ will not be recorded). We will add the algorithm details in our revised version.\n\n__QB__: Any examples or instance counts for first-person singular?  \n__Response__: Below we have included a few examples of first-person singular instances extracted from our dataset:\n\n“I present an end-to-end probabilistic model …”  \n“I collect a novel dataset …”  \n“I compare the model-based …”  \n\nIn addition, we have included the most common first-person singular verbs for each corpus and the proportions they occupy within each corpus.\n\n|\t\t\t| 1\t|  2 |  3 | 4 |\n| --------------- | :---: | :---: | :---: | :---: |\n| China |\tpropose (0.21)\t| show (0.13) | demonstrate (0.05) | introduce (0.03) |\n| United States |\tshow (0.10)\t| propose (0.08) | present (0.06) | demonstrate (0.05) |\n| Germany |\tshow (0.11)\t| propose (0.08) | present (0.07) | demonstrate (0.04) |\n| Japan |\tpropose (0.19)\t| show (0.14) | demonstrate (0.06) | introduce (0.04) |\n| India |\tpropose (0.13)\t| show (0.08) | present (0.08) | demonstrate (0.04) |\n\n\n__QC__: Which tool was used for syntactic analysis?  \n__Response__: For our syntactic analyses, a range of tools were used. For word and sentence tokenization, we used nltk, for tree parsing, we used the CoreNLP package from Stanford, and for clause extraction, we used clausie (https://spacy.io/universe/project/spacy-clausie).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work presents comprehensive analyses of English usage in *ACL abstracts, in that it includes multiple linguistic perspective and writers' native languages unlike previous work in English for Academic Purposes.\n\nIt reveals the differences of English usage among native languages of authors of research articles quantitatively.", "reasons_to_reject": "- The authors give reasons why the NLP field was targeted: its value in the NLP community, where computational methods are used to study language and linguistic expression is seen as important, which I think does not make sense. As authors mention in the paper, there are many papers working on genre analysis of research articles in different domains. As scientific research, more generalised work is preferred because many discipline-specific (but not comprehensive) studies have been published such as Cunningham (2015) for mathematics and Esfandiari & Barbary (2017) for psychology.\n- So, even if focusing only on the NLP domain, comparison with previous findings should be conducted to show the findings are universal or NLP-specific. The paper does not fully compare their findings with past work.\n\n- The work is only focused on abstracts. I do not find any technical/scientific reasons for this. There exists a full-text corpus of ACL Anthology (e.g. ACL Anthology Sentence Corpus), and the methodologies the authors adopted are computer-based ways (counting words, pos taggings, etc.) that do not require much manual labour or many of GPUs. Analyses of abstracts are okay, but in my opinion, it is difficult to recommend something about writing an article based only on the abstract analysis.\n- The authors claim that lexical diversity in cn/in/ja is less than that in us/de, but there is only 0.02 point difference, which amounts to about 2-3 words in each abstract. There may be preferences for singular nouns or specific prepositions.\n\n- Although in the past many EAP studies used four-word lexical bundles for analysis, but as Swales (2019) criticises, they contain non-sense bundles like \"and of the\". Simpson-vlach & Ellis (2010) adopted mutual information, and several methods to extract lexical bundles have been proposed such as Liu et al. (2016) and Brooke et al. (2017). The method should be more carefully considered.\n\n- After reading the review by Nr5L, I find the suggestion in Section 5 goes against recent ACL's efforts. In recent conferences reviewers have been asked not to reject papers for poor English. Academic writing and writing assistance are important but forcing non-native authors to use/not use specific wording unfairly impose a burden.", "questions_to_authors": "1. How did the authors address overlapping bundles? In Sec. B.1.4, \"we propose a novel\" is selected rather than \"propose a novel method\", but how about \"a novel method for\"? What kind of algorithm was applied to this selection?\n2. In l.197, it is said that Japanese writers use a lower proportion of first-person singular present verbs. I wonder whether first-person singular present verbs are used in *ACL papers because I have rarely seen them. Could it be possible to raise some examples or number of occurrence?\n3. For syntactic analysis what kind of tool did the authors used?\n\n(all answered)", "paper_topic_and_main_contributions": "This paper provides a genre analysis of abstracts of research articles in the NLP field and some pieces of advice to make their usage of English closer to native speakers' one.\nDifferent from many other similar past papers, the paper presents the comprehensive analyses from four different viewpoints: lexical, morphological, syntactic, and cohesive perspectives.\nSome of the findings are consistent with some previous EAP work. The findings are as follows:\n\n- Lexical diversity of cn/in/jp is lower than us/de\n- Lexical bundles appear more frequently in cn/in/jp/de than us, which is consistent with Hyland (2008a) and others\n- Past tense is more used in jp than others\n- First-person singular present verbs are less used in jp than others\n- Complex phrases are used in in/cn/jp than us, which is partly consistent with Lu and Ai (2015)\n- More clauses are used in us than others.\n- Discourse connectors are less used in us/de than cn/in/ja, which is partly consistent with Wang (2011)\n- \"Firstly\" appears in cn ten times more than us.\n- \"Besides\" also frequently appears in cn, which is consistent with Yeung (2009)\n- \"Hence\" is overly used in de/in/ja\n\nMethodologies:\n- lexis: calculating unique word ratio, using wordnet to identify synonyms/hypernyms\n- lexical bundles: 4-grams extracted\n- morphology: using spaCy to tag POS\n- cohesion: parse-tree\n\nThe authors recommend that ACL's guidelines should include some useful expressions.\n\nAlso, they claims that many aspects of writing give indication as to whether an author is a native English speaker or not.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a linguistical analysis of how NLP paper abstracts vary by nationality of the first author. The topic addressed, linguistic bias, is an important one, and overall, the reviewers are positive. However, the reviewers also raise a number of concerns related to presentation that undermine the impact of the authors' work, such as referring to "native speakers" rather than "native English speakers". Some of the assumptions made in the text (i.e. that Indian authors are not native English speakers) seem easily avoidable by discussing between-country differences rather than categorizing authors from specific countries as native or non-native English speakers. Bias towards differences between, for example, Indian English and American English are just as pertinent to the broader discussion as bias against non-native English speakers.

In summary, the reviewers seem to agree that the topic is important and the work is technically sound; one reviewer even recommends the paper for an award. Yet all reviewers also agree that some of the paper's language needs revision.
Decision: Accept-Main