Building sequencetagging approach toGrammatical Error Correction andText Simplification
Oleksandr Skurzhanskyi, Kostia Omelianchuk, Jan 28, 2022

Outline of the talk●Introduction○ Few words about us○ GEC task overview

●
●
●
●

GEC task●

Q&A

Sequence tagging approach
GECToR for GEC
Reusing GECToR on Text
Simplification Task

Who we are?

Who we are?
Oleksandr Skurzhanskyi, Applied
Research Scientist, Grammarly

Kostia Omelianchuk, Applied
Research Scientist, Grammarly

What is Grammarly?
Grammarly’s AI-powered writingassistant helps you make yourcommunication clear and eﬀective,
wherever you type.

GEC task
overview

GEC: Grammatical Error Correction
The goal of GEC task is to produce the grammatically correctsentence from the sentence with mistakes. Here we’re talking
about English language speciﬁcally.

Example:Source: He go at school.
Target: He goes to school.

GEC: Data

GEC: Shared task
-CoNLL-2014 (the test set is composed of 50 essays written
by non-native English students; metric - M2Score)-BEA-2019 (new data; the test set consists of 4477
sentences; metric - Errant)

GEC: ProgressDominant approach in 2019●

GEC was treating as machine translation problem mostly●

Seq2Seq models are status quo for most sentence-leveltransduction tasks
The cat sat●on mat

Seq2Seq
ModelThecat

Drawbacks○

Require large amounts of annotated parallel data - slow and expensive○

Autoregressive decoding - not performant○

Not controllable or explainable - unﬁt for real-world products

satonthematSequenceTagging

This work was done in fall 2019
This part of the presentation is based on the paper written by:-Kostiantyn Omelianchuk
Oleksandr Skurzhanskyi
Vitaliy Atrasevych
Artem ChernodubSequence Tagging
We approach the GEC task as a sequence tagging problem.
In this formulation for each token in the source sentence a GEC
system should produce a tag (edit) which represent a required
correction operation for this token.
For solving this problem we use non-autoregressive
transformer-based model.

Basic transformations

g-transformations

KEEP

keep the current token
unchanged

CASEchange the case of thecurrent token

DELETE

delete current token

MERGE

merge the current andthe next token

APPEND_ti

append new token ti

SPLIT

split the current token
into two

REPLACE_ti

replace the current token
with token ti

NOUN
NUMBER

convert singular nouns
to plurals and vice versa

VERB
FORM

change the form of
regular/irregular verbs

Token-level transformations
A ten years old boy go school

A ⇨ A:$KEEPten ⇨ ten, -:$KEEPyears ⇨ year, -:old ⇨ old:go ⇨ goes, to:school ⇨ school, .:,$MERGE_HYPH$NOUN_NUMBER_SINGULAR$MERGE_HYPH$KEEP$VERB_FORM_VB_VBZ$KEEP$APPEND_{to}$APPEND_{.}

A ten years old boy go school
A ten years old boy go school
A ten-year-old boy goes to school.

Sequence Tagging: Pros
1.

Generating tags is fully independent and easy to parallelize
operation.

2.

Smaller output vocabulary size compare to seq2seq
models.

3.

Less usage memory and faster inference compare to
encoder-decoder models.

Sequence Tagging: Cons1.

Independent generating of tags relies on assumption thaterrors are independent between each other.

2.

Word reordering is tricky for this architecture.

First baselines

Academia vs Industry-

diﬀerent data (both training/evaluation)-

latency/throughput matters-aggressive deadlinesBaseline architecture
Predicted tags
Error
replacement
linear layer

Error insertion
linear layer(s)Errordetection
linear layer
Postprocess

Encoder

Embeddings

Input sentence

CorrectedsentenceBaseline hyperparameters-

Emedings (trainable random, glove, bert, distill-bert)-

Encoder (cnn, lstm, stacked-lstm, pass_through, transformers)-Output layers (1-2 for insertions, 1 for replacement, 1 fordetection)-

Output vocabulary size (100-50000)-

Other (dropouts, training schedule, tp/tn ratio, etc)

Baseline results
Baseline/m2 scoreCoNLL-2014 (test)

Stacked LSTM (vocab_size=1000)

30.5

Stacked LSTM (vocab_size=1000; + g-transformations)

35.6

Stacked LSTM (vocab_size=1000; + g-transformations; +BERT embeddings)

46

Academic SOTA (single model) [2019]

61.3

Insights-

Increasing size of output vocabulary did not help

-Adding BERT as emdebbings helped a lot-

Training BERT with small batch_size failed (didn’t converge);
training with bigger batches required gradient accumulating

Similar approach

PIE paper

Our SOTA on NUCLE: 46

Transformers

TransformerAttention Is All You Need(Vaswani et al. 2017)

Multihead attention

BERT-like architecture

BERT (Devlin et al. 2018)

Trainingtransformers

huggingface / transformers

Transformers recipe
Thing that made our transformers work:●small learning rate (1e-5)●

big batch size (at least 128 sentences)or use gradient accumulation●

freeze BERT encoder ﬁrst and train only linear layer,then train jointly

BERTology works
LSTM

35.6

LSTM + BERT
embeddings

46.0

DistillBERT

52.8

BERT-base57.3*on CoNLL-2014 (test)

Additional tricks

Iterative Approach
Sequence

A ten years old boy go school (zero corrections)

Iteration
Iteration 11: A ten-years old boy goes school (2 total corrections)
Iteration 22 A ten-year-old boy goes to school (5 total corrections)
Iteration
Iteration 33: A ten-year-old boy goes to school. (6 total corrections)
Iteration

GECToR model: iterativepipeline
Predicted tags
Error correction
linear layer

Error detection
linear layer

Corrected

Postprocesssentence
Encoder (transformer-based)

Input sentence
Repeat N times

Staged training●Training is splitted in N stages●

Each stage has its own data●

Each stage has its own hyperparameters●On each stage model is initialized by the best weights ofprevious stage

Training stages
I.

II.
III.

Pre-training on synthetic
errorful sentences as in(Awasthi et al., 2019)
Fine-tuning on errorful-onlysentencesFine-tuning on subset oferrorful and errorfree
sentences as in(Kiyono et al., 2019)

Dataset# sentences% errorfulsentencesPIE-synthetic

9,000,000

100.0%

Lang-8947,344

52.5%

NUCLE

56,95838.0%

FCE

34,49062.4%

W&I+LOCNESS

34,304

67.3%

Training stages
I.

II.
III.

Pre-training on synthetic
errorful sentences as in(Awasthi et al., 2019)
Fine-tuning on errorful-onlysentences(new!)Fine-tuning on subset oferrorful and errorfree
sentences as in(Kiyono et al., 2019)

Dataset# sentences% errorfulsentencesPIE-synthetic

9,000,000

100.0%

Lang-8947,344

52.5%

NUCLE

56,95838.0%

FCE

34,49062.4%

W&I+LOCNESS

34,304

67.3%

Training stages
I.

II.
III.

Pre-training on synthetic
errorful sentences as in(Awasthi et al., 2019)
Fine-tuning on errorful-onlysentencesFine-tuning on subset oferrorful and errorfree
sentences as in(Kiyono et al., 2019)

Dataset# sentences% errorfulsentencesPIE-synthetic

9,000,000

100.0%

Lang-8947,344

52.5%

NUCLE

56,95838.0%

FCE

34,49062.4%

W&I+LOCNESS

34,304

67.3%

Training details●●●Adam optimizer (Kingma and Ba, 2015);
Early stopping after 3 epochs of 10K
updates each w/a improvement;
Epochs & batch sizes:○ Stage I (pretraining):batch size=256, 20 epochs○ Stages II, III (ﬁnetuning):batch size=128, 2-3 epochs* Results are given for GECToR (XLNet).

Training
stage #CoNLL-2014(test)*, F0.5

BEA-2019(dev)*, F0.5

I

49.9

33.2

II

59.7

44.4

III

62.5

50.3

III + Inf.
tweaks

65.3

55.5

Inference tweaks 1
By increasing probability of $KEEP tag we can force model to makeonly conﬁdent actions.
In such a way, we can increase precision by trading recall.

Inference tweaks 2
We also compute the minimum probability of incorrect class acrossall tokens in the sentence.
This value (min_erorr_probability) should be higher than threshold inorder to run next iteration.

BERT family

Varying encoders frompretrained transformers
CONLL-2014 (test)

BEA-2019 (test)

Encoder
PR

F0.5

PR

F0.5

LSTM**

51.6

15.3

35.0

---

ALBERT

59.5

31.0

50.3

43.8

22.3

36.7

BERT

65.6

36.9

56.8

48.3

29.042.6

GPT-2

61.0

6.3

22.2

44.5

5.0

17.2

RoBERTa

67.5

38.3

58.6

50.3

30.5

44.5

XLNet

64.6

42.6

58.6

47.1

34.2

43.8

* Training was performed on data from training stage II only. ** Baseline.

Varying encoders frompretrained transformers
CONLL-2014 (test)

BEA-2019 (test)

Encoder
P

R

F0.5

PR

F0.5

LSTM**

51.6

15.3

35.0---ALBERT

59.5

31.0

50.3

43.8

22.3

36.7

BERT

65.6

36.9

56.8

48.3

29.0

42.6

GPT-2

61.0

6.3

22.2

44.5

5.0

17.2

RoBERTa

67.5

38.358.6

50.3

30.5

44.5

XLNet

64.6

42.6

58.6

47.1

34.2

43.8

* Training was performed on data from training stage II only. ** Baseline.

Model inferencetime

Speed comparison● NVIDIA Tesla V100
● CoNLL-2014 (test)● single model● batch size=128

Speed comparison

Final results

Big picture

Results [2019]

Results [2022; CoNLL-2014 (test)]

Source: nlpprogress.com [link]Results [2022; BEA-2019 (test)]

Source: nlpprogress.com [link]

Text
Simpliﬁcation

This work was done in 2020
This part of the presentation is based on the Text Simpliﬁcation
by Tagging paper written by:-

Kostiantyn Omelianchuk-

Oleksandr Skurzhanskyi-

Vipul Raheja

Text Simpliﬁcation●consists of modifying the content and structure of a text in order to make iteasier to read and understand, while preserving its main idea andapproximating its original meaning●

could beneﬁt low literacy readers, English learners, children, and people withreading disabilities●

most commonly-used automatic metrics are:○○○SARIFKGLBLEU

Text Simpliﬁcation

SourceTarget

Simplification

All students when attending a university must
adhere to these guidelines.

All students when going into a university must
follow these rules.

GECShe see Tom is catched by policeman in parkat last night.

She saw Tom caught by a policeman in the park
last night.

Challenges●Training data: only 2 publicly available training datasets:○

WikiLarge (300k) is a set of automatically aligned complex-simple sentence pairs
from English Wikipedia○

Newsela includes thousands of news articles professionally leveled to diﬀerent
reading complexities. Has legal constraints to use it for public research.

●Evaluation: unreliable metrics (SARI, FKGL), like for almost
every text generation task

●Adapt GECToR approach from GEC to Text Simpliﬁcation

TST: GECToR for Simpliﬁcation●

Edit-Tag Vocabulary○●

Data Preprocessing○●

Tags overlap between tasks 92% allow to use GEC tags

Tried special preprocessing for Simpliﬁcation task which was beneﬁcial

GEC Pretraining○

Explored GEC initialization for Text Simpliﬁcation○

GECToR codebase is outdated (transformers 2.*) -> updated the code○

Tokenization in GECToR was incorrect -> ﬁxed●

Data Augmentations (details in next slide)●Tagging models○

Used RoBERTa-BASE for Text Simpliﬁcation (vs. an ensemble of BERT, XLNET andRoBERTa used by GECToR)●Inference Tweaks○$DELETE is highly important tag -> designed a new inference tweak for itTST: Data Augmentations●

Standard Train/Test sets for Text Simpliﬁcation●WikiAll: 384k pairs collected from English Wikipedia-Simple
Wikipedia○

WikiSmall (88k Pairs)○WikiLarge (296k Pairs)●WikiBT: Back-translated WikiAll (en-de and en-fr)●WikiEns: Ensemble Distillation of 3 models●○

TST on WikiAll (randomly initialized)○

TST-GEC on WikiAll (TST ﬁne tuned on GEC task)○

TST on WikiAll + WikiBT

Final training set: WikiAll (384k Pairs) + WikiEns (384k Pairs)

Results: Evaluation Metrics●Metrics depend a lot on tokenization. 1+ points could be achieved by simply
changing tokenization method●SARI is calculated diﬀerently for the corpus-level:○

It’s not just averaged of the sentence-level scores: statistics should begathered on
the whole corpus, then calculated○

F1 is used for deletion operation●BLEU is bad for the Text Simpliﬁcation evaluation●EASSE is a great evaluation package

Results: Simpliﬁcation Quality
SOTA on WikiSmall, near-SOTA on TurkCorpus and ASSET
TurkCorpus

ASSET

WikiSmallResults: Readability
SOTA on ASSET, Near-SOTA on TurkCorpus and WikiSmall
TurkCorpus

ASSET

WikiSmallResults: Ablation Study
All adaptation steps progressively enhance the system

Average SARI and FKGL scores

Reusable artifacts

Repository & models

Repository & models

Links● GEC paper● Text Simpliﬁcation paper● The code

Questions?