CASSI: Contextual and Semantic Structure-based Interpolation Augmentation for Low-Resource NER
Authors: Tanmay Surana
Thi-Nga Ho
KYAW ZIN TUN
EngSiong Chng
Venue: EMNLP 2023 Findings (2023)
Keywords: Named Entity Recognition
Text Augmentation
Low-Resource
Structure-Based Augmentation
Language Model
Context Diversity

Abstract: While text augmentation methods have been successful in improving performance in the low-resource setting, they suffer from annotation corruption for a token-level task like NER. Moreover, existing methods cannot reliably add context diversity to the dataset, which has been shown to be crucial for low-resource NER. In this work, we propose Contextual and Semantic Structure-based Interpolation (CASSI), a novel augmentation scheme that generates high-quality contextually diverse augmentations while avoiding annotation corruption by structurally combining a pair of semantically similar sentences to generate a new sentence while maintaining semantic correctness and fluency. To accomplish this, we generate candidate augmentations by performing multiple dependency parsing-based exchanges in a pair of semantically similar sentences that are filtered via scoring with a pretrained Masked Language Model and a metric to promote specificity. Experiments show that CASSI consistently outperforms existing methods at multiple low resource levels, in multiple languages, and for noisy and clean text.

CASSI: Contextual and Semantic Structure-based Interpolation Augmentation for Low-Resource NER
Tanmay Surana, Ho Thi Nga, Kyaw Zin Tun, Eng Siong Chng
Nanyang Technological University, Singapore tanmay603@e ntu. www.ntt..uuuuu.... {ngaht, ztkyaw, aseschngJ@ntu.cm @dw.:eeutttuuuuuu....
Abstract
domain expertise. Therefore, NER datasets tend to be small, especially for narrow domains and low-resource languages.
While text augmentation methods have been successful in improving performance in the low-resource setting, they suffer from annota- tion corruption for a token-level task like NER. Moreover, existing methods cannot reliahly add context diversity to the dataset, which has been shown to be crucial for low-resource NER. In this work, we propose Contextual and Se- mantic Structure-based Interpolation (CASSI), novel augmentation scheme that generates high-quality contextually diverse augmenta- tions while avoiding annotation corruption by structurally combining a pair of semantically similar sentences to generate a new sentence while maintaining semantic correctness and flu- ency. To accomplish this, we generate candi- date augmentations by performing multiple de- pendency parsing-based exchanges in pair of semantically similar sentences that are filtered via scoring with pretrained Masked Language Model and metric promote specificity. Ex- periments show that CASSI consistently out- performs existing methods at multiple low re- source levels, in multiple languages, and for noisy and clean text.!
Text Augmentation has been a successful method for improving performance in the low- resource setting for various tasks like Natural Lan- guage Inference, Sentiment Analysis, etc. in- volves creating additional text using the existing training text while generally preserving the labels?. Typical methods for text augmentation include ran- dom distortions such as random deletions, inser- tions, substitutions, (Wei and Zou, 2019), word replacement (Wu al., 2019; Kobayashi, 2018; Kim al., 2022), backtranslation (Sennrich et al., 2016; Yu al., 2018; Coulombe, 2018), paraphras- ing (Madnani and Dorr, 2010; Kumar et al., 2019), and label-conditioned text generation through Lan- guage Models (Anaby-Tavor et al., 2020; Yoo et al., 2021), GANs (Sun and He, 2020), VAEs (Qiu et al.. 2020), etc.
Preserving the token-level entity tags in NER challenging task and text augmentation meth- ods often cause annotation corruption, i.c., assign- ment of incorrect token-level tags. Dai and Adel 2020 avoid the annotation corruption problem by performing random replacements of the same en- tity type. However, this creates limited diversity in the dataset and generates noisy sentences since the entity replacements don't take sentence con- text into account. Methods like backtranslation and paraphrasing show limited improvements and re- quire an additional retagging step, which tends to be erroneous (Sharma et al., 2022: Kyaw, 2022). More effective works attempt to solve the label- preservation problem for low low-resource NER by per- forming augmentations using label-conditioned language model. The LM is conditioned on the tags by explicitly including the tags the training
1 Introduction
Named Entity Recognition (NER) is the token-level task of classifying terms in text that belong pre- defined entities like 'location', 'person', 'organi- zation', etc. It is an important task in NLP that is used in recommendation systems, search and con- tent classification systems for healthcare, academia, news organizations, etc., and as an integral part of other tasks like text summarization (Nallapati et al., 2016), question answering (Zhou et al., 2018; Fab- bri al., 2020), topic modeling (Krasnashchok and Jouili, 2018), etc. Annotating data for token-level task like NER can be expensive, often requiring
Not all text augmentation methods necessarily preserve lahels. For example, Yoon et al. 2021 and Harel-Canade et al. 2022.
'Our code is available at httpp:///githut com/ tanmaysurana/CASSI

---

(irput Sentence)
Subtree Exchanges
BERT Score
Training Set
Ranking
X00X
XXX 2001 XX X00
by LM Score
Jaccard Similerity
XOX XOOX
XXXX XXXX XXX
XX XOX XOU XOX XOXX
XXX XOOX
(Selected Seniercce)
XXX SO0X XXX X000X
1006 X00X 005
200 3000X X XXXX
XOXK 30 X0XX
100
Scoring
XXX XOOX X000
X00X XXXX XX XXX
XXX XXOX X0X
XX XXX XXXX
XXXX
XXX XOOX XOOX X XX
300X
XXXL X0XX XX. XXXX
XXXX
X XXX NoX X5S0
XX XXX BOXX XXXX
XOX X00X XXXXX
KX JOX XXXK JOOK X00X
JOK X00O
XX XXX xoox 200L X000X [Ouriaa Senitance
(b)
(C)
Figure 1: The Overall Pipeline of Our Method. Best viewed in color
sentences, in process called label linearization. DAGA (Ding et al., 2020) uses this process to gen- erate new sentences including entity tags. However, these sentences tend to be ungrammatical and dis- fluent in nature when trained a small number of examples. MELM (Zhou al., 2022) uses this process to perform entity replacements through masked generation using a pretrained Masked Lan- guage Model, such that the replacements take sen- tence context into account, but does not introduce new context patterns into the dataset. Both of these methods are also able to add new entities to the training set. These methods outperform simpler methods like random substitution, random deletion, etc. However, these methods involve training lan- guage model on limited data and are therefore still prone to annotation corruption, generating noisy training examples. This also severely limits the number of augmentations that can be added.
for English text by replacing 'O' tags. However, this requires additional in-domain data. Moreover, MASS is only pretrained on 4 languages: English, German, French, and Romanian. Additionally, Shi et al. 2021 alter the content while maintaining sen- tence structure by performing random substitutions of same-label spans for few-shot structured predic- tion tasks where annotations contain structural in- formation like part-of-speech tagging, dependency parsing, and constituency parsing.
In this work, propose Contextual and Se- mantic Structure-based Interpolation (CASSI) text augmentation scheme for low-resource NER. With the goal of increasing contextual diversity and avoiding annotation corruption, we use depen- dency parse trees to identify subtrees containing the subject, object, or complement between pairs of semantically similar sentences. We then generate candidate augmentations by performing exchanges between all pairs of subject subtrees and all pairs of object/complement subtrees between the two sen- tences. To ensure semantic correctness and fluency, we filter the resulting candidates by scoring them with a language model, similar to rescoring meth- ods used in Automatic Speech Recognition (Xu et al., 2018; Chan et al., 2016) and Neural Machine Translation (Gülçehre et al., 2015). We leverage existing pretrained Masked Language Model to avoid training language model limited data. Furthermore, take steps to reduce the bias from the language model against specific names and to- wards shorter sentences and high- frequency generic terms. Specifically, we pick the top scored sen-
Moreover, Lin et al. 2020 show that, in the low- resouree setting, decreasing contextual diversity in the training set more significantly impacts NER per- formance than reducing entity diversity. Therefore, contextual diversity is more important than entity regularity knowledge to generalize to unseen exam- ples in the low-resource setting. While transformer language models are pretrained on large corpora, they not able to sufficiently capture context pat- terns for NER when fine-tuned small number of examples. Contextual augmentation has been applied to the token-level task of Aspect Term Ex- traction by Li et al. 2020, who finetune MASS (Song et al., 2019) to generate new partial contexts

---

Subject Exchange(s)
delays decision on relugee volling. OSCE rejects German call to boycott Britsh lamb.
amood
auk
amoof
ObjectiComplemett
Exchanges
rejects
German
call
to
boycott
British
lamb.
OSCE delays
boycott
OSCE delays Brilish lamb.
pobp
call boycott Britsh lamb. OSCE delays decision on Brrtish lamb.
OSCE dalays
nouby
prep
compouer
rejects decision refugen voting.
rejects relugee voting.
V:
OSCE
delays
decision
on
relugee
voting.
rejects German call boycot dedision on relugee voling.
rejects German boyoot: retugee vosing.
Figure 2: An example of subtree exchanges to generate candidate augmentations. Every colored box is sub- jectoobjectcoomplemnn subtree. The green arrow indicates the candidate chosen as the final augmentation by our method. Best viewed in color.
tences and further filter them using a simple metric based on Jaceard scores between the candidate aug- mentations and the original sentences.
sentence. We use BERTScore (Zhang et al., 2020) as the sentence similarity metric. BERTScore is token- wise metric used to evaluate text generation for tasks like machine translation and image caption- ing. It calculates Precision (P) and Recall (R) by matching tokens in the source and target sentences by cosine similarities between their embeddings retrieved from an MLM like BERT (Devlin et al., 2019). 1). For a source sentence with normalized embeddings {ai,.., ap} and a target sentence with normalized embeddings (bi,..,by))
Our contributions can be summarized as follows: (I) we present a novel text augmentation scheme that improves context diversity in the dataset while avoiding annotation corruption, (2) show that method consistently and significantly outper- forms existing methods in multiple languages for monolingual and cross-lingual NER, and (3) show that our method outperforms existing meth- ods on noisy social media text.
2 Proposed Method
Our proposed method (shown in Figure 1) gener- ates augmentations by combining a pair of semanti- cally similar sentences to generate a new sentence while ensuring high degree of fluency, and contex- tual and semantic correctness. For every sentence in the training set, we first rank the remaining sen- tences by sentence similarity (Figure la: Section 2.1). Next, we generate multiple candidate augmen- tations by performing structure-based exchanges between the given sentence and selected semanti- cally similar sentence using their dependency parse trees (Figure bb; Section 2.2). We then filter out subset of the candidate augmentations by scoring them with a pretrained Masked Language Model. We select the final augmentation from this subset using a simple metric to promote specificity and diversity (Figure 1c; Section 2.3).
P= P
P
Σ
max a by: R b
Σ byeB
max ag bb a.A
O.CA
The final score the score (F1 = 2 PR
P+R We choose BERTScore because is able to use existing MLMs directly, without additional modifi- cations like calculating sentence-level embeddings (Reimers and Gurevych, 2019). Moreover, pect the token-wiss nature of BERTScore make more robust noisy social media for token- level task like NER.
2.2 Generating Candidate Augmentations
Figure 2 shows an example of subtree exchanges to generate candidate augmentations. For a given sentence U and a selected sentence V, we use dependency parsing to identify subject subtrees and object/complement subtrees. A subject subtree is the subtree of verb that contains the verb's 's subject, and object/complement subtree is the subtrec of verb or preposition that contains the verb preposition's object complement. To generate candidate augmentations, we perform exchanges within all possible pairs of subject subtrees between U and V. Similarly, we perform exchanges within
2.1 Sentence Selection
Since sentence semantics and context clues im- portant for detecting entities, for given sentence in the training set rank the remaining sentences using a sentence similarity metric. Therefore, to generate n augmentations, we use the top n sen- tences according to their similarity to the given

---

all possible pairs of object/complement subtrees between U and V. Therefore, each such exchange generates a pair of candidate augmentations.
Sentence
LM Score
I will be staying at the hotel
-0.74
I will be staying at the Hilton
-1.29
We choose subjects, objects, and complements for exchanges as they are likely to contain entities and context clues for entities. We use subtrees as the structures to exchange they contain more information than just the subject, object, com- plement or the noun chunks containing them, thus generating more diverse candidate augmentations.
John will be staying at the hotel John Doe will be staying at the hotel
-1.75
-2.19
Table I:
Example of LM bias against specificity. The LM score decreases for more specific sentences. LM scores are
calculated using XLM-RRASE following Salazar et al. 2020. The scores are normalized by the number of words the sentence.
Since we consider all exchanges between and V as candidate augmentations, we avoid selecting duplicate pairs of sentences. Therefore, if V is selected for U, U is skipped when selecting sen- tences for V. For sentences where the dependency parse cannot find least subject we replace entities in the sentence with random entities of the same type avoid simply oversam- pling these sentences.
2.3.2 Filtering for Specificity and Diversity
As shown in Table LM scoring is inherently bi- ased against specificity. Generic terms are assigned higher probabilities than specific names since they oceur more frequently, which is detrimental to a task like NER. To reduce this bias and promote diversity in the augmentations, we pick the top k LM-scored candidates and further filter them with the geometric mean of the Jaccard similarities be- tween the words in the candidates and the words in the original pair of sentences. Given a sentence U, sentence V, and candidate augmentation C sets of words:
object,
2.3 Filtering Candidates
To select the final augmentation, filter the can- didate augmentations in two steps: (1) scoring with a Language Model, and (2) filtering for specificity and diversity. We select just one augmentation for every pair of sentences to increase the diversity of the augmentations.
Vncl jUCCIVC)
A.SooreUU,VOC
2.3.1 Language Model Scoring
Language Model scoring is used in Automatic Speech Recognition and Machine Translation to ascertain the correctness of a sentence (transcrip- tion or translation) by calculating the probability of occurrence of each token in the sentence. We use LM scoring to score the candidate augmentations for every pair of sentences. We calculate the LM score using a pretrained MLM following Salazar al. 2020. The score calculated as the sum of the log- liceliodsss of each token in the sentence given all other tokens. Also following Salazar et al. 2020, we normalize the score by the number of words in the sentence (ignoring punctuation) to counter the effects of sentence length on the score. Formally, given sentence S with tokens (t1....;mm) the LM score is calculated as:
The selected augmentation is therefore biased to- wards containing more words from both original sentences, thus increasing the overall specificity of the augmentation. For the same reason, the selected augmentation is likely to be more diverse than the other k - 1 candidates (since the selected augmen- tation is biased against being disproportionately more similar one of the two original sentences).
2.4 Post-Processing
In the event that multi-word entity gets split by de- pendency parsing (for example, only the last name is picked from a name containing both first and last names), the entity's s tag is fixed according to the 1OB scheme to ensure that all entities (1) start with a B- tag, (2) end with an I- tag, and (3) only contain I- tags in between.
LMScore(S) M
log P(L S (4)
3 Experiments
We evaluate our augmentation scheme on low- resource monolingual and cross-lingual NER multiple languages. We also evaluate our method
where M the number of words in the sentence, and θ represents the MLM's parameters.

---

Gold
Method
En
De
Es
NI
Avg
Gold Only
07.11/1.99
39.52 / 30.40
57.93 / 49.75
41.58 33.51
49.03 38.89
DAGA
60.30/ 56.52
92.11/45.0
60.82 / 56,41
45,44/ 40.06
54.67 / 49,55
100
MELM
67.36 / 62.06
53.88 / 49.12
62.61 / 58.62
47.32 / 43.38
57.79 / 53.30
CASSI (Ours)
75.04/ 70.04
65.96 58.54
69.93 65.35
64.09 / 58.13
68.75 / 63.02
Gold Only
75.64 / 70.52
64.37 58.19
69.38 65.15
/ 56.94
67.56 / 62.70
DAGA
74.87 / 69.89
59.85 / 54.16
68.75 / 64.75
/
65.82 / 61,51
250
MELM
77.48 / 72.47
63.50/ 57.23
70.46 / 66.36
64.08 / 60.22
68. 88.88 64.07
CASSI (Ours)
82.02 / 77.08
72.79 / 65.26
75.89 / 71.79
72.90 / 68.23
75.90 / 70.59
Gold Only
79.98 / 75.02
68,69 62.42
72.91 68.81
67.18 64.90
72.19 / 67.79
DAGA
80.60 75.51
71.42 65.48
76.43 / 72.64
69.51 / 66.34
74.49 69.99
500
MELM
81.84 / 76.43
69.44 64.16
75.66 72.53
71.29 / 68.68
74.56 / 77.45
CASSI (Owrs)
85.03 / 80.24
76.69 70.47
79.60 76,42
77.17 / 74.13
79.62 / 75.31
Gold Only
83.69 / 78,82
75.07 / 69.17
76.88 73.55
71.41
77.52 73.17
DAGA
84.35 / 79.58
73.81 68.45
77.90/4..
72.09 / 69.96
77.04 73.14
750
MELM
84.06 / 78.84
73.87 / 68.01
78.50 75.26
74.29 72.40
77.68 73.63
CASSI (Ours)
86,39 / 81.83
77.88 / 72.19
80.51 77.62
78.82 75.77
80,90 / 76.85
Gold Only
85.50 / 80.86
77.00 71.82
78.55 / 75.78
75.88 / 73.26
79.23 / 75.43
DAGA
85.21 / 80.62
76.64 / 70.98
79.72 76.90
75.97 / 74.23
79.38 75.68
1k
MELM
85.56 / 80.56
76.74/71.00
80,20 77.33
76.08 73.92
79.66 69.64/ 75..72
CASSI (Ourx)
87.24 / 82.86
79.48 / 73.71
81.25 / 78.82
80.08 / 76.90
82.01 78.07
Gold Only
8 83,94
80.42 75.00
82.77 / 80,26
77.85
82.94 / 79.26
DAGA
88.53 / 84,27
79.45 74.08
82,70/ 80.16
79.97 77.668
82.66 79.05
2k
MELM
88.39 83.79
79.00 73.42
82.67 80.26
79.97 / 78.65
82.51 79.03
CASSI (Owrs)
88.68 / 84,49
80.78 75.200
83.06 / 80.66
82.55 / 79.83
83.77 / 80.05
Gold Only
90.01 / 85,90
82.32 / 77.40
84.60 / 82.23
/
84.58 81.29
DAGA
89.82/ / 85, 70
82.00 / 76.96
84.23 / 81,85
77.31
83.50 / 80.45
MELM M
89.58 / 85.20
82.00 / 76.65
84,10 81.74
79.56 79.40
83.81 80.75
CASSI (Ours)
90.02 / 85.97
82.45 77.20
84.97 82.75
83.51 81.10
85.24 / 81.75
Table 2: Results of Monolingual NER on subsets of CoNLL 2002/2003,
Results are reported as Micro-F1 /
Macro-F1. Numbers in bold indicate best performance
on noisy social media datasets. We compare our method against MELM (Zhou al., 2022) and DAGA (Ding et al., 2020).
ing set, WNUT-2017 (Derczynski et al., 2017) an English tweets dataset containing 3.3k sentences in the training set, and ReLDI-NormTagNEN 2.0 (Miličević and Ljubešié, 2016) Croatian tweets dataset.
3.1 Datasets
For monolingual NER, we perform experiments the CoNLL 2002/2003 (Tjong Kim Sang, 2002: Tjong Kim Sang and De Meulder, 2003) datasets in four languages: English (En), German (De), Span- ish (Es), and Dutch (NI). For each language, we generate subsets of the training set of size 100, 250, 500, 750, 1k, 2k, and 4k. To concretely study the effects of adding data, we sample the subsets pro- gressively, i.c., each smaller subset is sampled from the next larger subset.
For all datasets except ReL.D--NoTTTaNNEER-
2.0, we use the standard dev and test sets. For WLLDI-NorTTagNEEN
2.0, we randomly split the dataset into 4.8k training sentences, 1.6k dev sentences, and 1.6k test sentences since the dataset source does contain standard splits. All datasets follow the IOB annotation scheme.
3.2 Experimental Setting
3.2.1 Augmentation
For zero-shot cross-lingual NER, we evaluate models trained using the subsets of the English training set and the standard English dev set on the German, Spanish, and Dutch test sets.
Baseline Methods For DAGA and MELM, we follow augmentation settings as described in Ding et al. 2020 and Zhou et al. 2022, respectively.
Additionally, we evaluate the performance of our method in conditions where dependency parsing is likely be error- pronn. For this, we perform exper- iments on the following social media datasets: Wei- boNER (Peng and Dredze, 2015)- a Chinese social media dataset containing 1. 1.4k sentences in the train-
CASSI We use XLM-RBASE (Conneau et al., 2020) for both caleulating the pairwise BERTScore and LM scoring. For dependency pars- ing, we use the following language-specific pipelines from SpaCy (Honnibal et al., 2020): en_core_web_te trf for English, de_dep_news_trr

---

Gold
Method
De
+
Avg
Gold Only
40.65 / 29.13
38.60/30.34
35.04/25.70
38.10/28.39
DAGA
38.61 34.18
39.27 / 35,88
36,96/ 36.68
38.28 / 35,58
100
MELM
40.95 / 35.14
45.10/37.67
34.36 33.67
40.14 / 35.49
CASSI (Ours)
54.83 42.83
54.94/
54.65 / 48.68
54.81 / 45.64
Gold Only
47.77 39.30
50.91 45.06
45.24 43.79
47.97 42.72
DAGA
50,99 /3392
51.95 / /45.5
49.89 44.65
50.94 / 44.70
250
MELM
/ 50.64
57.54 50.43
55.60 51.54
57.31 / 50.87
CASSI (Ours)
64.90/53.37 53.37
61,65 / 53.25
63.63 / 58.13
63.39 / 54.92
Gold Only
42.97
$4.52 47.66
50.81 48.75
52.42 46.36
DAGA
60.71 52.24
59.95 / 53.37
57.98 53.15
59.55 52,92
500
MELM
63.34 / 55.86
62.34/ 55.97
59.25 / 55.83
61.64 55.89
CASSI (Ours)
67.18 / 56.57
64,45 / 56,50
61.84/ 58.28
64,49 / 57.12
Gold Only
S2.70
60.61 / 53.78
57.76
61.79 / 54.75
DAGA
66.09 / 56.12
62.97 / 56.06
62.03 57.79
63.70 56.66
750
MELM
65.17 / 56.47
63.43 56.63
58.87 55.64
62.49 / 56.25
CASSI (Ours)
68.60 58.57
64.77 57.79
63.09 59.51
65,49 / 58.62
Gold Only
66.02 / 56.60
63.44 / 57.02
60.67 / 58.05
63.38 / 57.22
DAGA
66.21 / 58.32
65.11 58.29
62.20/ 59.36
64.51 58.66
1k
MELM
67.47 59,42
64.97 57.81
61.99 58,94
64.81 58.72
CASSI (Ours)
69.64 / 60.15
66.68 / 59.22
66.90 / 63.47
67.74 60.95
Gold Only
62.36
68.02 62.01
68.19 65.78
(8.98/ 63.38
DAGA
71,52 / 62.66
68.40 61.45
67.71 / 64.84
69.37 62.98
2k
MELM
68.86 61.61
67.96 61.87
67.28 64.03
68.03 62.50
CASSI (Ours)
71.54/ 62.52
68.22 61.59
68.74/ 66.76
69.50 / 63.62
Gold Only
72.01 / 63.65
70.00 / 64.02
68,30/ 66.23
70.10 64.63
DAGA
71.95 / 63.98
69.44 63.25
69.86 67.69
70.42 / 64.97
MELM
71.48 63.87
68.92 63.20
67.09 / 66.09
69. 64.39
CASSI (Ours)
72.47 / 64.29
69.83 / 63.97
69.25 / 67.50
70.52 / 65.25
Table 3: Results of Cross-lingual NER on subsets of CoNLL 2002/2003. Results are reported as Micro-F1 / Macro-F1. Numbers in bold indicate best performance. Underlined numbers indicate competitive performance.
for German, es_dep_news_trf for Spanish, nl_core_news, for Dutch, zh_core_web_ trf for Chinese, and hr_core_news _g for Croatian. the number of candi-
generate ranges from N to 10N. For gold train- ing sets of size 1k and larger, also generate 0.25N, 0.5.,, and 0.75.N augmentations, by domly sampling from the set of augmentations of size N. For each method, we stop adding augmen- tations if there is no improvement the best dev set Micro-Fl for three consecutive increments the number augmentations. For this purpose, we track the average dev set Micro-Fl across the three runs, as to report single augmentation size in each case.
For all experiments, dates selected for J-Score filtering (k) to 5 based on preliminary experiments.
3.2.2 NER Training
We perform all experiments using mBERTBASE BASE-
(Vasantharajan et al., 2022a,b). We train with batch size of 32 for 50 epochs using the AdamW optimizer. All train- ing sets up to size 500 (including augmentations) are trained with learning rate of 2e 4 with the number of warm-up steps set to 200. For train- ing larger than 500 sentences, decrease the learning rate and increase the number of warm-up steps proportionately with the increase in training set size. For the CoNLL subsets, we average the performance on three randomly sampled subsets of the same size. For the social media NER datasets, we average the performance on three runs. We report both Micro-Fl and Macro-Fl in each case. For all augmentation methods, given a gold train- ing set of size N, the number of augmentations we
3.3 Results
Monolingual NER Table summarizes the re- sults on monolingual NER. Our method consis- tently outperforms the baseline methods for all lan- guages. There is significant improvement across languages for low-resource levels up to 1k. The improvement becomes marginal for subset sizes of 2k and 4k for English, German, and Spanish. This is consistent with observations from Lin al. 2020 that increasing context diversity in a clean text regular NER dataset has a limited effect on the performance for over 2k sentences when fine- tuning a pretrained transformer. However, the per-

---

Method
WNUT-2017
WeiboNER
ReI.DI-Hr 2.0
30-
Gold Only
41.81/36.00
53.78
72.58/ 52.45
DAGA
31.67
64.45
55.99
71.70/ 52.88
MELM
42.12 36.27
62.80/ 54.21
72.49 / 51.99
% Relative Impessmmnnt Catt-res
CASSI (Ours)
43.22 37.31
65.02 / 56.08
72.90 / 53.19
20
CASSI Ik
Table 4: Results of NER on Social Media Datasets.
MELM
Results are reported Micro-FI Macro-F1.
formance continues to significantly improve for Dutch. We discuss this further in Section 4.1. On average, MELM and DAGA marginally improve performance for 1k sentences but marginally re- duce performance for 2k and 4k sentences. Given that performance improvements from augmenta- tions diminish as dataset size increases, we attribute this to the noise in the augmentations outweighing any diversity introduced by them.
Language
Figure 3: Relative Improvement (%) of Micro-F1 over Gold-Only, calculated as (F1 FI FlG))// F Float.) Best viewed in color.
Discussion
4.1 Performance on Dutch
Cross-lingual NER Table summarizes the re- sults on zero-shot cross-lingual NER with English the source language. On average, our method outperforms both MELM and DAGA for all sub- set sizes. Interestingly, DAGA becomes competi- tive in the Micro-F1 for resource levels of 2k and 4k sentences. With enough examples, DAGA can produce better context patterns. While these hurt performance in the monolingual setting, they seem to help generalize to unseen entities in other lan- guages. However, it more notably underperforms in the Macro-F1 compared to our method. We posit that this is because the auto-regressive generation of entity tags along with the sentences causes it to disproportionately generate
Figure shows the language-wise relative improve- ment of CASSI and MELM over the Gold-Only baseline for 500 and 1 sentences. For CASSI, across both resource levels, the relative improve- ments for English, German, and Spanish are very similar whereas the improvement for Dutch sig- nificantly higher. Given that the datasets are from the same domain (news articles), the most salient difference in the experiment settings for these lan- guages is the amount of pretraining they received in mBERT. mBERT is pretrained on the entire Wikipedia dump of the top hundred languages with the largest Wikipedias The English, German, and Spanish Wikipedias contain 3.3B, 1.1B, and 800M words respectively, however, the Dutch Wikipedia considerably smaller at 300M words*. We sus- peet that Dutch gains more from contextual diver- sity in the augmentations due to lower represen- tation during pretraining. This is consistent with observations from Lin et al. 2020 who conclude that pretraining helps capture context patterns when finetuning for NER. This implies that context diver- sity in the augmentations more important for low- resource languages with low representation during pretraining. Figure also shows that MELM, an entity replacement method that does not introduce new context patterns, does not follow this pattern.
frequent tags. In the case MELM, the lack of new context patterns and annotation corruption cause the performance to drop for 2k and 4k sentences.
NER on Noisy Text Table 4 summarizes the re- sults of our experiments on social media datasets. Our method consistently outperforms baseline methods even for noisy social media text where dependency parsing is likely to be erroneous. More- over, these datasets tend to contain a considerable number of mislabeled examples and a stilted dis- tribution of tags (for example, the 'person' tag oc- curs lot more frequently than other tags in these datasets), which can make learning reliable dis- tribution of labels difficult for label-conditioning methods.
/https://githbb cowm//oolle-rreeerc//bettt/t///e master master/multilimme /multilinuall
"A of March/April 2019. Collected using https: //en. wikipedia, org/wiki /iitto/sstooooo Wikiipeiia and https://

---

Method
IK
Gold
Method
Avg
Randren Solection
87.43/83.000 78 72.64 81.29/ 78.81 79.16/ 76665
DAGA
7
8
8
BERTScore
$7.38 83,09
73.47 81.4479.02 /7.30
100
MELM
5
2
3
3.75
CASSI (Ours)
10
9
10
8
9.25
Table 5: BERTScore Ablation. Results are reported as
DAGA
2
3
Micro-F1 Macro-F1.
250
MELM
3.75
CASSI (Ours)
8
9
9
8.25
DAGA
1.5
Best LM Score 87.26/ 82,90 79.51/74.1
80.95
78.99/ 76.42
500
MELM
2
1.5
Random (Top 5) 86.88 82,70 29.46/74.08 80,73/ 78.09 79.05 76.72 77.72
CASSI (Ours)
6.25
J-Seore (Top 87.38 83.09 79.10/73447 81.44/79.0 77.00/77.34
DAGA
1
1
1
750
MELM
2
1
1,5
Table 6: J-Score Ablation. Results are reported as Micro-FI Macro-FI.
CASSI (Ours)
4
4
10
6
6
DAGA
0.5
0.25
0.75
0.63
1k
MELM
0.25
0.75
0.75
0.56
CASSI (Ours)
5
6
5
4.2 Ablations
DAGA
0.25
0.25
0.25
0.25
0.25
2k
MELM
0.25
0.75
0.5
0.25
0.44
We perform ablation studies on the subsets of CoNLL 2002/2003 to evaluate the effect of (1) matching sentences by similarity, and (2) using the J-Score filter for specificity. To assess the quality the augmentations, we compare the performance on the test set upon adding 10.V augmentations (where N 1k). As mentioned in Section 3.2.2, for each language, average the results on three sampled subsets of size 1k.
CASSI (Ours)
0.25
1.31
DAGA
0.25
0.25
0.25
0.44
4k
MELM
0.25
0.25
0.5
0.25
0.31
CASSI (Ours)
0.75
2
1.44
Table 7: Number of augmentations (xN, where is the number of Gold samples) added for each method. The higher quality of our augmentations allows us to add more augmentations than the baseline methods.
4.2.1 BERTScore
performance-improving augmentations than base- line methods. Note that we do not go beyond 10N augmentations in any case.
Table 5 compares using BERTScore to select se- mantically similar sentences with selecting sen- tences at random. On average, matching sentences by similarity outperforms random selection.
Additionally, we evaluated all methods on all augmentation increments up to 10N on the 500 sentence subsets of CoNLL 2002/2003 to compare the dev set Micro-F1 for every method. The results are summarized in Figure 4. Our method outper- forms the baseline methods on all augmentation increments for all languages.
4.2.2 J-Score Filter
We compare the performance of selection from the top l candidates using the J-Score metric for speci- ficity against (1) selecting the candidate with the best LM score, and (2) randomly selecting can- didate from the top k. Table 6 shows these results. On average, filtering using the J-score metric out- performs the other methods. It notably outperforms random selection from top k, showing that using the J-score metric does indeed select sentences with useful properties.
4.3.2 Naturalness
We performed LM scoring the 1k subsets of CoNLL 2002/2003, and 1 augmentations gener- ated from them. We scored the sentences using uncased mBERT BASE BAstucceedd (we do not use XLM-R BLM--BASE since the augmentations were selected using XLM- REASE) following Salazar et al. 2020 and normal- ized them by the number of words (ignoring punc- tuation). The augmentations are scored separately, i.e., they are not added to the gold-only set. The LM Scores pscudo- socudo-lo-llikllhooodt with greater values being better. We present the mean and stan- dard deviation of the LM scores over the sets. Table shows that the augmentations score very similarly (albeit marginally lower on average) to the gold sentences.
4.3 Quality of Augmentations
Here, we consider the quality of our augmenta- tions by (1) comparing the number of performance- improving augmentations added by each method, and (2) comparing the 'naturalness' (via LM scor- ing) of our augmentations the gold sentences.
4.3.1 Number of Augmentations
Table 7 summarizes the number of augmentations added for each method on the CoNLL 2002/2003 subsets. We are able to add considerably more

---

For high-resource languages, increasing context diversity provides limited performance improve- ments for clean text regular NER outside of the low-resource setting.
Es MELM
Dey Mcco-11
DAGA
A Acknowledgements
De CASSI
De MELM
70-
This research/project is supported by the National Research Foundation, Singapore under its AT Sin- gapore Programme (AISG Award No: AISG2-GC- 2022-005). We would like to acknowledge the High Performance Computing Centre of Nanyang Technological University Singapore, for providing the computing resources, facilities, and services that have contributed significantly to this work.
Fx CASSI
65-
Is DAGA
00-
N8 MELM
DAGA
Angmentations
Figure 4: Dev set Micro-FI Number of Augmenta- tions for different methods on the 500 sentence subsets CoNI.L. Best viewed in color.
References
Method
En lk
De lk
Es IK
NI
Avg
Gold-Only -1002
0.64 0.51
0.52
0.58
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, N. Tep- per, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In AAAI Conference on Artificial Intelligence.
Augs
-1.05 0.70 -0.7 0,53 -0.58 0.48 -1.11 0.84 40.863 0.64
Table 8: LM Scores on 1k gold sentences and 1k aug- mentations. LM scores are pseudo-log-liellhoood lielliikss cal- culated using BABERTBASE-COE 2020. The scores are normalized by the number of words in the sentence.
following Salazar et al.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 1960049644 IEEE.
5 Conclusion
In this work, propose CASSI as text aug- mentation scheme for low-resource NER. CASSI produces contextually diverse sentences while pre- serving entity tags by structurally combining a pair of semantically similar sentences through the gener- ation of candidate combinations that are filtered for semantic correctness and specificity. It leverages dependency parsers for structural information and pretrained MLM for scoring. Through extensive experiments, show that it outperforms baseline methods for multiple languages in monolingual and cross-lingual settings. Furthermore, we show that is also able to outperform baseline methods under noisy text conditions where dependency parsing is likely to be unreliable.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Claude Coulombe, 2018. Text data augmentation made simple by leveraging NLP cloud apis. CoRR, abs/1812.0471k
Xiang Dai and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. Proceedings of the 28th International Conference Computational Linguistics, pages 3861-3867, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Limitations
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy User generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguis- tics.
The proposed method requires a dependency parser in the target language which might not be available for some extremely low-resource languages. Cur- rently, Universal Dependencies contains treebanks for 141 languages* (Nivre al., 2020) and SpaCy only supports 24 languages.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
"tnnss//miivraaldepdddenicr.r

---

deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4171-4188, Minneapolis, Minnesota. Association for Computational Linguistics.
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhan- dari, and Partha Talukdar. 2019. Submodular optimization-base diverse paraphrasing and its ef- fectiveness in data augmentation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pages 3609-3619, Minneapolis, Min- nesota. Association for Computational Linguistics.
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kru- engkrai, Thien Hai Nguyen, Shafiq Joty. Luo Si, and Chunyan Miao. 2020. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (E.,LP)), pages 6045-6057, Online, Association for Computational Linguistics.
Zin Tun Kyaw. 2022. Data augmentation for name entity recognition.
Kun Li, Chengbo Chen, Xiaojun Quan, Qing Ling, and Yan Song. 2020. Conditional augmentation for aspect term extraction via masked sequence-to- sequence generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7056-7066, Online. Association for Computational Linguistics.
Alexander Fabbri, Patrick Ng. Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. Template-based question generation from retrieved sentences for im- proved unsupervised question answering. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4508 4513, On- line. Association for Computational Linguistics.
Hongyu Lin, Yaojie Lu, Jialong Tang. Xianpei Han, Sun, Zhicheng Wei, and Nicholas Jing Yuan. 2020. A rigorous study on named entity recognition: Can fine-tuning pretrained model lead to the promised land? Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7291-7300, Online. Association for Computational Linguistics.
Çaglar Güllçhrre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huci-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine trans- lation. CoRR, abs/ 150.003535.
Nitin Madnani and Bonnie J. Dorr. 2010. Generat- ing phrasal and sentential paraphrases: survey of data-driven methods. Computational Linguistics. 36(3):341-387.
Fabrice Harel-Canada, Muhammad Ali Gulzar, Nanyun Peng, and Miryung Kim. 2022. Sibylvariant transfor- mations for robust text classification. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1788- Dublin, Ireland. Association for Computational Linguistics.
Maja Miličević and Nikola Ljubešić. 2016. Tviterasi, tviteraši or twitteraši? producing and analysing normalised dataset of croatian and serbian tweets. Slovenščina 2.0: empirical, applied and interdisci- plinary research, 462):156-188
Matthew Honnibal, Ines Montani, Sofie Van Lan-
deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Hazel H Kim, Daecheol Woo, Seong Joon Oh, Jeong- Won Cha, and Yo-Sub Han. 2022. Alp: Data augmen- tation using lexicalized pefgs for few-shot text clas- sification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10894- 10902.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulgehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to-sequeeec RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Sosuke Kobayashi. 2018 Contextual augmentation:
Data augmentation by words with paradigmatic re- lations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 4577, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Joakim Nivre,
de Marneffe, Filip Gin- ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034 4043, Marseille, France. European Language Resources Association.
Katsiaryna Krasnashchok and Salim Jouili, 2018 Im- proving topic quality by promoting named entities in topic modeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 247-253, Melbourne, Australia. Association for Computational Linguistics.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 548-554, Lisbon, Portugal. Asso- ciation for Computational Linguistics.

---

Siyuan Qiu, Binxia Xu, Jie Zhang, Yafang Wang, Xi- aoyu Shen, Gerard de Melo, Chong Long, and Xiao- long Li. 2020. Fasyaug: An automatic textual data augmentation platform for classification tasks. Com- panion P Proceedings of the Web Conference 2020.
named entity recognition. In COLING-O2: The 6th Conference on Natural Language Learning 2002 (CoNLI-20022..
Frik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: ienepeeeinddeennenttt named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics.
Charangan Vasantharajan, Kyaw Zin Tun, Lim Zhi Hao. and Chng Eng Siong. 2022a. NERP. GitHub.
Charangan Vasantharajan, Kyaw Zin Tun, Ho Thi-Nga, Sparsh Jain, Tong Rong, and Chng Eng Siong. 2022b, Medbert: A pre-trained language model for biomed- ical named entity recognition. In 2022 Asia-Pacifte Signal and Information P Processing Association An- nual Summit and Conference (APSIPA ASC), pages 1482-1488
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Pupers), pages 86-96, Berlin, Germany. Association for Computational Lin- guistics.
Jason Wei and Kai Zou. 2019. EDA: Easy data augmen- tation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Con- serence on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 6382-6388 Hong Kong, China. As- sociation for Computational Linguisties.
Saket Sharma, Aviral Joshi, Namrata Mukhija, Yiyun Zhao, Hanoz Bhathena, Prateek Singh, Sashank San- thanam, and Pritam Biswas. 2022. Systematic re- view of effect of data augmentation using paraphras- ing on named entity recognition. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contex- tual augmentation. In Computational Science-ICCS 2019: 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part IV 19, pages 84 95. Springer.
Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmenta- tion for NLP. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 3499-3500 Online. Association for Computational Linguisties.
Hainan Xu, Tongfei Chen, Dongji Gao. Yiming Wang. Ke Li, Nagendra Goel, Yishay Carmiel, Daniel Povey, and Sanjeev Khudanpur. 2018. A pruned mnlm auttice-rescoing algorithm for automatic speech recognition. In 2018 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 5929-5922999 5933. IEEE
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2019. Mass: Masked sequence to sequence pre- training for language generation. In International Conference on Machine Learning.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Sang-Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Lever- aging large-scale language models for text augmen- tation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 2225-2239, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xiao Sun and Jiajin He. 2020. A novel approach to generate large scale of supervised data for short text sentiment analysis. Multimedia Tools and Appli- cations, (99--10:5439-5455
Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021. Ssmix: Saliency-based span mixup for text classi- fication. Findings of the Association for Com- putational Linguistics: ACU/UCNLPP 2021, Online Event, August 1-6, 2021, volume ACL/ICCNLP 2021 of Findings of ACL., pages 3225-3234. Association for Computational Linguistics.
Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation diversity in natural language genera- tion. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, EAC1.2021, Online, April 19 23, 2021, pages 326-346. Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLLL2002 shared task: Langugge-independdnt

---

global self-attention for reading comprehension. In 6th International Conference on Learning Represen- nations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Conference Track Proceedings. OpenReview.net.
En
De IK
Es 1k
Gold-Only
0.591 0.067
0.622 0.038
0.625 0.055 0,22 = 0.049
J-Scure
0.599 = 0.054
0.624 0.034 0.633 = 0.048 0.623 = 0.042
Best LM 0.601 0.054 0.629 0.036 a 0048
1004
Table 9: BeffoesTSsoree calculated upon adding 5x augmentations to Ik gold-only sentences. Lower scores imply greater diversity.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenRe- view.net.
may be clear from the following example: given two gold sentences X and Y, 10 words long cach with no overlap in words, and two candidate aug- mentations C1. which contains 5 words from X and 5 words from Y, and C2. which contains 7 words from X and 3 words from Y: intuitively, out of the two, C1 would be more diverse. This is re- flected in the J-Scores: C₁ has a J-Score of 0.333, whereas C2 has a J-Score of 0.309. Morcover, if the sentences in the original dataset are already well formed which is the case in CoNLL- LM Scoring is likely to rank C2 over C1.)
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2018. Neural question generation from text: A preliminary study. In Nat- ural Language Processing and Chinese Computing: 6th CCF International Conference, NLPCC 2017, Dalian, China, November 8-12, 2017, Proceedings 6. pages 662-671. Springer.
Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cam- bria, Luo Si. and Chunyan Miao. 2022. MELM: Data augmentation with masked entity language model- ing for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), pages 2251-2262, Dublin, Ireland. Association for Compu- tational Linguistics.
Note that the Seff-BERTScores of the augmen- tations are marginally higher than the gold-only sentences. This is expected for two reasons: (1) we pick the most semantically similar sentences for our augmentations, and (2) our augmentations consist of parts of the gold sentences used directly, i.e., the same words in the same order.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41 International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 1097-1100. ACM.
B Alternatives to Pairwise BERTScore
While pairwise BERTScore (calculation of BERTScore for every pair of sentences in the dataset) is efficient enough for small datasets, we recognize that using pairwise BERTScore becomes time-consuming for larger datasets since it involves token-wiin operations for every pair of tokens in the two sentences. For this reason, we suggest the following alternatives for larger datasets: (1) pairwise cosine similarity using sentence embed- dings (Reimers and Gurevych, 2019), and (2) ran- domly sampling fixed number of sentences for each source sentence, and selecting the most mantically similar sentences from these random samples using BERTScore or sentence similarity.
A Measuring Semantic Diversity of the Augmentations
We measure the semantic diversity of the augmen- tations when added to the gold sentences using Self-BERT Score, diversity metric similar to Self- BLEU (Zhu et al., 2018)6. We score the 1k sets of CoNLL 2002/2003 using mBERT cased in three cases: (i) Gold-Only, (ii) Gold-Only along with 5k augmentations filtered using the J-score filter, and (iii) Gold-Only along with 5k augmen- tations picked using the Best LM Score. Table 9 shows that augmentations selected using the J- Score metric are more diverse than those selected using the Best LM Score. (The intuition behind this
C Examples of Generated Augmentations
"Given the nature of our augmentations, the embedding- based BERTScore is a better metric than an n-gram based met- rie like BLEU. Furthermore, BERTScore outperforms BLEU at measuring diversity. though it still significantly underper- forms on human evaluation (Tevet and Berant, 2021). More- over, our augmentations essentially serve as an edge case for automated metrics since we do not formulate entirely new sentences and instead copy exact phrases from the source sentences.
Table 10 shows some example augmentations taken from a single 1k subset of CoNLL 2003 English. The examples show sentence interpolation, in- stances where the augmentation process leads to simple short phrase or word replacement, and in- stances of grammatical incorrectnes/incoeeenee

---

in the augmentations. (Note that in the examples of simple phrasc/word replacements the augmenta- tions can still introduce entities with different tags or introduce an entity in place of a non-entity.)

---

Gold Sentence Pair
Angmentation
The European Commissiion said on Thunday it disagreed with German advive consumers shon Brinish lamb until seientists determine whether disease can he transmitedd to sheep.
The Europcan Commission suid on Thursay humans could contract an illness similar to mad cow disease by eating contaminated beef.
Bonn has efforts protect public health after consumer confidence col- lapsed in March after a British report suggested humans could contract an illnews similar mad cow discase by cating contornizated beef.
Opel AG together General Motoes came second place with 49,269 registrations, 16.4 perceni of the overall figure.
Opel AG together with General Motors rose 0.04 percent 903.09 and the LISPI index rose 0.02 percent 81.58.
The BLOM Stock Index which covers buth markets rose 0.04 pervent to 903.09 and the I LISPI index roso 0.02 porcent to 81.58.
Defence Minister Volker Ruehe Bosnin next year as part of an intorational force to ensure the establishment peace, newspaper reported Saturday.
German troops would stay on
Defence Minister Volkee Ruche said that German rroops would stay on in Roania next year part to maintain friendship and cooperation with Moscow a newspaper reported on Saturday.
Iragi President Saddam Husscin has told visiting Russian ultra-tationalist Vladimir Zhirinvvsky that Raghdad wanted maintain friendship and coop- eration with Moscow, official tragi newspupers said on Thursday.
They said Zhirinovsky told Saddam hefore he left Raghdad on Wednesday that Liberal Democratic party and Russian Duma (parfiament) are calling for an immediate lifting of the emburgo imposed on Irag after its 1990 invasion of Kuwait.
They said German troops lold Suddan before Baghdad Wednesday that his Liberal Democratic party and the Russian Duma (parlian (pallimment) are calling for an immediate lifting of the embargo imposed Irag after its 1990 invasion of Kuwait.
Defence Minister Volleer Ruche said that German troops would stay on in next yeur as part interational to ensure establishment of peace, a newspaper reported on Saturday.
Lebed, who has urranged militury with sepuratist rebels the southem Russia region, was Moscow this week socking support for a deal on Chech- nya's political status.
Lehed, who has arranged a military truce with separatist rehels the southern Russia region, was in Muscow this week seeking attend the referendum held on Iraq's presidencyy which extended Saddam''s is term for seven more years.
Last October he was invited attend the referendum held on Iraq's presidency. which extended Saddam's 's term for seven more years.
Maronite Patriarch Sfeir expressed sorrow over the violations Sunday's elections.
Maronite Patriarch Steir expressed sorrow over the redistribution of Syrian troops' locations Lebanon.
Fears an Israeli operation conses redistribution Syrian troops' loca- Lions in Lebanon.
ohtain work and residence riphts say Prime Minister Alain Juppe's 's propusals are insuflicient as hunger strike enters 49th day Paris church and Wednesday raily attracts 8,000 sympathisers.
Africans sooking
Africans seaking to renew or obtain work and residence rights was asked news conference India's is decision block adopsion the accord in Geneva would lead to an anns moc with Pakistan and China.
Foreign Minister I.K. Gujial was asked a news conference if India's 's decision block adoption of the accord in Geneva would lead an race with Pakistan and China.
American world champion Gwen Torrence, the bronze medallist in Atlania, was second 11.00.
American world champion Gwen Torrence, the bronze medallist in Atlanta, second in Longchamp.
Hever Golf Rose (11-4), last year's Prix de I' A'baaay winner at Longchamp. Einished third, further one a quaster lengths away with favourite
Mind Games in Tourth.
European champions Juventus will face English league and cup double win ners Manchester United in this season's is European C Champions' League.
European champions Juventus will face new four-year contract with English league F.A. Cup champions Manchester United this season's Eurapean Champions" League.
Ireland midfielder Roy Keune has signed new four year contract with English league and F.A. Cup champions Manchester United.
The Netherlands govemment has naled out paying mansom money for a Dutch couple kidnapped from their farm, while Costa Rican authorilies on Tuesday they had no leads in the cuse.
The Netherfands government his ruled out it expeets to take an unspocified time charge to pay for the merger, while Costa Rican authoriries said Tuesday they had no leads the case.
Advanced Medical said expects take an unspecified one-rime charge pay for the merger.
Table 10: Example augmentations generated from a subset of CoNLL 2003 English. Instances of grammatical invorrectnes/insscerence are underlined in red.


Reviews:
[{"review_id": "JADAKj1d0a", "reviewer": "Reviewer_JhLS", "summary": "", "questions": "- Question A: Compared to the original sentences, how much lower/higher is the naturalness (LM score or other) of the augmented sentences on average? Relevant to justifying the “high quality” claim in line 011. \n- Question B: Compared to the original sentences, how much lower/higher is the diversity (self-BLEU or other) of the augmented sentences on average? Relevant to justifying the “contextual diversity” claim in line 011. In general, one can intuitively see how the proposed approach increases diversity, but it would be helpful to quantify it for the community. \n- Question C: What efficiency improvements have you considered for the various steps in CASSI? The time required scales as a function of dataset size due to the BERTScore similarity selection, which limits the feasibility of its use in larger datasets.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your favourable review of our work. We appreciate your detailed assessment and based on your comments we have decided to make the following additions to the paper. We discuss these in greater detail below. \n1. Include the scores with regards to Question A and Question B in Section 4 (Discussion) of the paper\n2. Clarify how J-Score filtering improves diversity\n3. Indicate that not all augmentations preserve labels in the Introduction\n4. Include a discussion of efficient alternatives to BERTScore in the Appendix\n5. If space allows, change Figure 1 to include a natural language example (space constraints did not allow us to do so earlier)\n\n## Quantifying “High Quality” \n>_(Includes responses to Question A, Question B, Reasons to Reject, and “Presentation Improvements - Difficult to Follow”)_\n\nWhile naturalness is a key part of our “high quality augmentations” claim, our claim  of “high quality” encompasses label preservation, context diversity, and naturalness/semantic correctness. In the paper, apart from the main results and ablations, we attempt to support the high quality claim via the number of augmentations (Section 4.3) in two ways: (i) by showing that we are able to add significantly more augmentations than baseline methods, and (ii) by showing that our method outperforms baselines at all augmentation levels. However, we agree that separately addressing the different properties of our augmentations would be very helpful. We discuss this further below:\n\n### 1. Naturalness \n>_(Response to Question A -  Compared to the original sentences, how much lower/higher is the naturalness (LM score or other) of the augmented sentences on average? Relevant to justifying the “high quality” claim in line 011.)_\n\nWe performed LM scoring on the 1k subsets of CoNLL 2002/2003, and 1k augmentations generated from them. We scored the sentences using `mBERT-Base-cased` (we do not use `XLM-R-Base` since the augmentations were selected using `XLM-R-Base`) following Salazar et al. [1] and normalized them by the number of words (ignoring punctuation). The augmentations are scored separately, i.e, they are not added to the gold-only set. The LM Scores are log-likelihoods, with greater values being better. We present the mean and standard deviation of the LM scores over the set. The table below shows that the augmentations score very similarly (albeit marginally lower on average) to the gold sentences.\n\n| Method    | En         | De         | Es         | Nl         | Avg         |\n|-----------|------------|------------|------------|------------|-------------|\n| Gold-Only | -1.02 ± 0.64 | -0.72 ± 0.51 | -0.63 ± 0.52 | -1.05 ± 0.65 | -0.855 ± 0.58 |\n| Augs      | -1.05 ± 0.70 | -0.71 ± 0.53 | -0.58 ± 0.48 | -1.11 ± 0.84 | -0.863 ± 0.64 |\n\n\n### 2. Context Diversity\n>_(Includes responses to:_\n>- _Question B - Compared to the original sentences, how much lower/higher is the diversity (self-BLEU or other) of the augmented sentences on average? Relevant to justifying the “contextual diversity” claim in line 011. In general, one can intuitively see how the proposed approach increases diversity, but it would be helpful to quantify it for the community._ \n>- _“Difficult to Follow; Line 276” - Unclear how filtering with Jaccard similarity increases diversity if you’re trying to find similar sentences. This filtering seems to merely offset the genericization of LM-scoring in the prior step and actually discourages diversity by keeping the augmentations that are more similar to existing samples in the dataset.)_\n\nContext Diversity is difficult to directly quantify in the case of NER, since it has to do with entities in new settings, to help the model find entities in unseen examples. The new settings have to be semantically consistent with the entity. Therefore, two sentences can have different contexts (with respect to entities) but similar semantics. This is certainly the case with our augmentations because we select the most semantically similar sentences. Therefore, to be reliably measured, context diversity needs to be decoupled from the semantic “topic” diversity of the sentences. Most sentence similarity/diversity metrics like BLEU, BERTScore, etc., are inherently biased towards measuring semantic similarity. Nonetheless, we scored our augmentations and the gold dataset, as shown below, using “Self-BERTScore”, calculated in a similar way as Self-BLEU (BLEU was not a suitable metric given the nature of our augmentations, since BLEU scores via matching n-grams, which leads to  5x - 6x higher scores when gold-only and augmentations are combined. BERTScore is more suitable since it measures cosine similarities between word embeddings.)\n\n| Set                                 | En                | De                | Es                | Nl                |\n|-------------------------------------|-------------------|-------------------|-------------------|-------------------|\n| Gold-Only (1k)                      | **0.593 ± 0.057** | **0.622 ± 0.038** | **0.625 ± 0.055** | **0.621 ± 0.049** |\n| _Gold-Only (1k) + J-Score (5k)_     | _0.599 ± 0.054_   | _0.624 ± 0.034_   | _0.633 ± 0.048_   | _0.623 ± 0.042_   |\n| Gold-Only (1k) + Best LM Score (5k) | 0.601 ± 0.054     | 0.629 ± 0.036     | 0.637 ± 0.048     | 0.625 ± 0.044     |\n\n\nWe score the 1k sets of CoNLL 2002/2003 using mBERT-Base-cased in three cases: (i) Gold-Only, (ii) Gold-Only along with 5k augmentations filtered using the J-score filter, and (iii) Gold-Only along with 5k augmentations picked using the Best LM Score (to address “Difficult to Follow; Line 276”). \n\nSelf-BERTScore is scored between -1 and 1, with lower scores indicating higher diversity. Along with the Self-BERTScore, we also include the standard deviation of the calculated BERTScores in each case.\n\nThe table above shows that the augmentation largely maintains the semantic diversity in the dataset. While the augmentation diversity scores are marginally higher, they fall well within the first standard deviation of the gold-only scores.\n\nFurthermore, the table also shows that using J-Score filtering does consistently result in higher diversity as compared to using the Best LM Score. (Relevant to “Presentation Improvements: Difficult to Follow - Line 276”. The intuition is that given two gold sentences $X$ and $Y$, $10$ words long each with no overlap in words, and two candidate augmentations $C_1$, which contains $5$ words from $X$ and $5$ words from $Y$,  and $C_2$, which contains $7$ words from $X$ and $3$ words from $Y$; intuitively, out of the two, $C_1$ would be more diverse. This is reflected in the J-Scores: $C_1$ has a J-Score of $0.333$, whereas $C_2$ has a J-Score of $0.309$. Moreover, if the sentences in the original dataset are already well formed - which is the case in CoNLL - LM Scoring is likely to rank $C_2$ over $C_1$)\n\n\n### 3. Label Preservation\n>_(Response to second half of “Reasons to Reject” - Authors could use an approach like cleanlab to support the claim that CASSI does not introduce any new label issues brought on by shifts in augmented semantics.)_\n\nAs suggested, we used CleanLab to find potential label issues in the augmentations. CleanLab inputs a datasets labels and prediction probabilities over the labels for each instance, and outputs outliers based on incorrect predictions with high confidence. We performed inference using the gold-only 1k English set on 5k augmentations generated from the set (we only used English to be able to manually detect semantic shifts). For incorrect classifications from the gold-only model, upon manual inspection, we found no incorrect labelling explicitly introduced by the augmentations. Specifically, almost all cases of incorrect classifications were that the entities had been classified as “I-MISC”, which is the most underrepresented entity class in the dataset. The remaining label issues originated from incorrect labelling in the original training set.\n\n\n## Alternatives to BERTScore\n>_(Response to Question C -  What efficiency improvements have you considered for the various steps in CASSI? The time required scales as a function of dataset size due to the BERTScore similarity selection, which limits the feasibility of its use in larger datasets.)_\n\nWhile pairwise BERTScore (calculation of BERTScore for every pair of sentences in the dataset) is efficient enough for the sizes of the low-resource datasets we work with in the paper, we recognize that using pairwise BERTScore becomes time consuming for larger datasets (taking 6 hours for a 15k sentence dataset on an Nvidia PV100 PCIe GPU). We suggest considering the following alternatives to the pairwise BERTScore calculation:\n- Cosine Similarity using Sentence Embeddings\n- For larger datasets, a constant number of sentences can be randomly sampled for each source sentence, and the most semantically similar sentence(s) can be selected from these random samples using BERTScore/Sentence Similarity.\n\n\n## Conclusion\n\nThis concludes our response. As mentioned in the beginning, we will revise the paper to accommodate your suggestions. We believe this will help improve the quality of our paper. Once again, thank you for all your valuable feedback on our work!\n\n## References\n\n[1] Salazar, Julian, Davis Liang, Toan Q. Nguyen and Katrin Kirchhoff. 2019. “Masked Language Model Scoring.” Annual Meeting of the Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and easy to follow. Frequently, when doubts would arise, they were often addressed in the next sentence or paragraph. Well done. \n- CASSI explicitly considers both diversity and naturalness of the generations whereas many existing techniques yield ungrammatical, implausible, or incoherent texts. Some would argue that low-quality synthetic texts are acceptable because they still improve downstream model performance and generalization. However, in the real world, people often place significant weight on readability and having a technique that improves performance while being natural can help overcome objections from human decision makers and increase the likelihood of use. \n- The evaluation setup acknowledges the limitations of the augmentation to low resource settings and sufficiently demonstrates its benefits relative to existing baselines.", "reasons_to_reject": "- In general, the claims of “high-quality” and increased “contextual diversity” are left unquantified, at least directly. The high-quality claim may potentially be substantiated via improvements to model performance but given the gist of the filtering criteria suggests that a quality == naturalness argument is being made. Authors could use an approach like cleanlab (https://github.com/cleanlab/cleanlab) to support the claim that CASSI does not introduce any new label issues brought on by shifts in augmented semantics.", "questions_to_authors": "- Question A: Compared to the original sentences, how much lower/higher is the naturalness (LM score or other) of the augmented sentences on average? Relevant to justifying the “high quality” claim in line 011. \n- Question B: Compared to the original sentences, how much lower/higher is the diversity (self-BLEU or other) of the augmented sentences on average? Relevant to justifying the “contextual diversity” claim in line 011. In general, one can intuitively see how the proposed approach increases diversity, but it would be helpful to quantify it for the community. \n- Question C: What efficiency improvements have you considered for the various steps in CASSI? The time required scales as a function of dataset size due to the BERTScore similarity selection, which limits the feasibility of its use in larger datasets.", "paper_topic_and_main_contributions": "This paper proposes an augmentation scheme for NER tasks that takes the following steps. Given a particular source text, (1) search the remainder of the dataset for the N most similar texts, (2) extract dependency parses from the selected texts and construct all combinations of subject and object/complement subtrees swaps, (3) filter the candidates via LM score for “correctness” / naturalness, (4) further filter the most natural candidates by Jaccard similarity with the source texts, which simultaneously increase specificity and offsets the genericizing effect of LM score filtering, and finally (5) some post-processing to fix potential IOB issues. Experiments on several mono- and cross-lingual NER datasets show the benefits of CASSI, especially for the lower resource settings and for languages like Dutch with lower representation in pretraining datasets. To highlight that CASSI is robust to imperfect dependency parses, the authors also explore augmentation on noisier social media datasets where CASSI still outperforms existing baselines. \n\nContributions include a new, high-performing augmentation pipeline that generates contextually diverse texts suitable for NER tasks as well as appearing general enough to be broadly applicable in other NLP task settings. Different elements of that pipeline, like LM scoring and Jaccard similarity filtering, can be adopted by other existing augmentation to improve the overall quality of the generations.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Regarding the statement in line 047, these are not necessarily “missing references” but are examples of augmentations that do not preserve the original label. SSMix represents an example of an augmentation that also aims to enhance contextual diversity like CASSI, though the result is not as natural. \n\n- Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021. SSMix: Saliency-Based Span Mixup for Text Classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3225–3234, Online. Association for Computational Linguistics.\n- Fabrice Harel-Canada, Muhammad Ali Gulzar, Nanyun Peng, and Miryung Kim. 2022. Sibylvariant Transformations for Robust Text Classification. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1771–1788, Dublin, Ireland. Association for Computational Linguistics.\n- Qi Liu, Matt Kusner, and Phil Blunsom. 2021. Counterfactual Data Augmentation for Neural Machine Translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 187–197, Online. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- Line 343 - is the Base supposed to be subscripted like that? \n\nPresentation:\n- Figure 1 - a missed opportunity to show your approach more clearly using a natural language example\n- Line 047 - not all augmentations preserve the label, e.g. mixup augmentations, sibylvariant transforms, and counterfactual augmentations all alter semantics and derive new targets for the augmented data. See “missing references” for example works. Can clarify via “... while generally preserving the labels.” \n\nDifficult to Follow:\n- Line 276 - Unclear how filtering with Jaccard similarity increases diversity if you’re trying to find similar sentences. This filtering seems to merely offset the genericization of LM-scoring in the prior step and actually discourages diversity by keeping the augmentations that are more similar to existing samples in the dataset."}}, {"review_id": "x3BdsnaBT9", "reviewer": "Reviewer_vXhU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your favourable review of our work.\n\n>_(The following is our response to \"Reasons to Reject - It would be beneficial if the authors could demonstrate the effectiveness of the method on few-shot cross-domain NER, rather than only conducting experiments on cross-lingual settings, which are more common.\")_\n\nWhile we agree that evaluation on cross-domain NER would be interesting, it is currently not feasible due to the lack of availability of cross-domain datasets that share tag-sets. Recent works on cross-domain NER [1-5] use cross-domain datasets with mismatched tag-sets, and attempt to design methods that address the issue of mismatching tag-sets. However, this would be out of the scope of this work. On the other hand, we evaluate on cross-lingual NER since it is more feasible because all 4 languages come from the CoNLL datasets and thus, share the same tag-set.\n\n\n\n\n## References\n\n[1] Hu, Jinpeng, DanDan Guo, Yang Liu, Zhuo Li, Zhihong Chen, Xiang Wan, and      Tsung-Hui Chang. 2023. “A Simple Yet Effective Subsequence-Enhanced Approach for Cross-Domain NER”. Proceedings of the AAAI Conference on Artificial Intelligence 37 (11):12890-98.\n\n[2] Hu, Jinpeng, He Zhao, Dan Guo, Xiang Wan and Tsung-Hui Chang. 2022. “A Label-Aware Autoregressive Framework for Cross-Domain NER.” NAACL-HLT.\n\n[3] Jia, Chen and Yue Zhang. 2020. “Multi-Cell Compositional LSTM for NER Domain Adaptation.” Annual Meeting of the Association for Computational Linguistics.\n\n[4] Jia, Chen, Liang Xiao and Yue Zhang. 2019. “Cross-Domain NER using Cross-Domain Language Modeling.” Annual Meeting of the Association for Computational Linguistics\n\n[5] Lin, Bill Yuchen and Wei Lu. 2018. “Neural Adaptation Layers for Cross-domain Named Entity Recognition.” Conference on Empirical Methods in Natural Language Processing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation to augment data based on dependency parse tree is well-founded and interesting.\n2. The proposed method outperforms previous data augmentation methods in monolingual NER.", "reasons_to_reject": "It would be beneficial if the authors could demonstrate the effectiveness of the method on few-shot cross-domain NER, rather than only conducting experiments on cross-lingual settings, which are more common.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a data augmentation method for low-resource NER. The proposed method first generates candidate augmentations based on exchanging subtrees. The subtrees are predicted using an external dependency parser. Then, the generated candidates are filtered using (i) LM scoring or (ii) diversity and specificity creterions.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tvy69nsa6K", "reviewer": "Reviewer_sAiY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your favourable review of our work.\n\n>_(The following is the response to \"Presentation Improvement - In Section 3.2.1, line 339, you say \"[...] we set k to 5 [...]\". I can assume that k is the top-k most similar sentences based on the LM-scoring. Am I right? I suggest to better specify what k is here, I feel like the reference to previous paragraph might be too weak.\")_\n\nYou are correct, the $k$ on line 339 does refer to the top-$k$ sentence selection after LM Scoring. We will revise Section 3.2.1 to clarify this. Thank you for pointing this out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A novel technique that aims to improve the performance of NER systems in low-resource settings.\n- The technique is well-explained and backed by a solid experimental setup that includes numerous experiments in various settings, including monolingual and cross-lingual settings.\n- Section 4 offers interesting insights, including a more in-depth discussion about the results on a specific language and an ablation study that goes through various hyper-parameters of the text augmentation pipeline.\n- Since the authors have included the code as supplementary material, I assume that it will be open-sourced.\n", "reasons_to_reject": "None", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper proposes a new text augmentation technique aimed at addressing the challenges posed by low-resource Named Entity Recognition (NER) tasks. he authors introduce a novel approach (CASSI, Contextual and Semantic Structure-based Interpolation) which centers on the augmentation of NER-tagged text using dependency parsing-based exchange in pairs of semantically similar sentences. A key advantage of their method is its ability to generate fluent, contextually diverse, and semantically coherent text without corrupting annotations. The paper compares CASSI with two similar techniques (DAGA (Ding et al., 2020)) and MELM (Zhou et al., 2022)) in both low-resource monolingual and cross-lingual NER settings, demonstrating its superior effectiveness and improved performance compared to previous works.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Section 3.2.1, line 339, you say \"[...] we set k to 5 [...]\". I can assume that k is the top-k most similar sentences based on the LM-scoring. Am I right? I suggest to better specify what k is here, I feel like the reference to previous paragraph might be too weak."}}]

Author rebuttal:

Meta-review: The paper introduces an NER augmentation method, which involves finding similar texts in the dataset, generating various subtree swaps, and filtering candidates based on language model scores and Jaccard similarity. Experiments demonstrate its effectiveness, particularly in low-resource settings and for languages like Dutch. The method remains robust even with imperfect dependency parses, outperforming existing baselines on noisy social media datasets. Notwithstanding the inquiries or apprehensions raised by the reviewers, the authors adeptly resolved them in their rebuttal.
Decision: Accept-Findings