SlowGuess/NAACL_2022_OCR huggingface dataset:
Currently, the number of linearly separable Boolean functions is known only up to 9 variables (Gruzling, 2007). However, in comparison with IBM's expressivity (Table 8), we know that ESC has more expressive power until at least the combination of 19 base systems, since  $8^{18} < 7.2 \times 10^{16}$ .
Table 8: IBM's upper bound of expressive power and ESC's lower bound of expressive power, both in terms of possible logic function expressions.  $n$  denotes the number of base systems.
No. of base systems	BEA-2019
P	R
T5-Large + G. XLNet	80.20	61.11 75.48
+ GECToR Roberta	84.69	59.02 77.91
+ Riken&Tohoku	86.15	59.41 79.04
+ UEDIN-MS	86.16	61.06 79.62
+ Kakao&Brain	86.65	60.91 79.90
Table 9: The performance of GEC system combination with different number of base systems on the BEA-2019 test set.
6.3 Comparison with GEC-IP
Proposition 2. ESC is a generalized form of GEC-IP.
GEC-IP works by selecting one hypothesis for each edit type. We can model GEC-IP's final decision with ESC Equation 2 by setting one of the weights to one, and the others to zero, i.e.,  $w_{\bar{s}} = 1$ ,  $w_{j \neq \bar{s}} = 0$ $\forall j \in \{1, \dots, k\}$ , where  $\bar{s}$  denotes the optimal hypothesis from GEC-IP training, for each edit type. Thus, this shows that GEC-IP is a special case of ESC.
7.1 Effect of the Number of Base Systems
In this section, we discuss the effect of using different number of GEC base systems in our combination. We conduct an experiment of GEC system combination with increasing number of base systems from 2 to 6 (Table 9). In this experiment, we always pick the best-performing base systems first, i.e., we combine T5-Large with GECToR XLNet in the 2-system combination, then add GECToR Roberta in the 3-system combination, and so on.
We find that even with combining only three systems, our method already outperforms the state of the art (75.9). We also find that adding more base systems, even though an additional system is weaker than the already combined base systems, is still beneficial to improve the  $\mathrm{F}_{0.5}$  score. We find that adding more base systems allows the model to have more information in picking the correct edits, hence improving the precision.
7.2 Ensemble Alternative
We run system combination experiments with base systems that are very similar to each other to evaluate whether our system combination can be an alternative to the conventional ensemble. We run this experiment on two sets of models, a set of sequence
Different iteration trees may result in different sets of learned functions, and the user needs to choose one.
Model	BEA-2019
P	R	F0.5
(1) Transformer-big 1	68.25	58.73	66.11
(2) Transformer-big 2	68.49	59.51	66.48
(3) Transformer-big 3	68.45	60.02	66.58
(4) Transformer-big 4	68.38	60.02	66.53
1+2+3+4 Ensemble	69.88	59.71	67.57
1+2+3+4 ESC	75.86	52.63	69.70
Table 10: Comparison of ESC with conventional ensemble on sequence-to-sequence models.
Model	BEA-2019
P	R	F0.5
(1) GECToR XLNet	79.20	53.90	72.40
(2) GECToR Roberta	77.20	55.10	71.50
(3) GECToR BERT	71.50	55.70	67.60
1+2 Ensemble	79.40	57.20	73.70
1+2 ESC	83.94	50.44	74.10
1+2+3 Ensemble	78.90	58.20	73.60
1+2+3 ESC	84.67	51.50	75.01
Table 11: Comparison of ESC with conventional ensemble on sequence tagging models.
to-sequence models and a set of sequence-tagging models. By conventional ensemble, we mean the method of averaging the output probabilities from multiple models during the prediction step, i.e., when predicting the tag for each token in sequence tagging models and when predicting the next token during beam search in sequence-to-sequence models.
The sequence-to-sequence experiment combines 4 transformer-big models we train ourselves. The four models only differ in the random seeds, pretraining data, and batch sizes during fine-tuning. 2 of them are pre-trained with 50M sentences of synthetic data, and the other 2 are pre-trained with 100M sentences of synthetic data before being fine-tuned with BEA-2019 training data. For the sequence tagging experiment, we use GECToR's (Omelianchuk et al., 2020) models<sup>5</sup>, which differ in the pre-trained masked language model, and then fine-tune with GEC parallel data.
We find that even though the base systems are relatively similar, our system combination can still improve the  $\mathrm{F}_{0.5}$  score by at least 1.7 points. Furthermore, our model yields higher  $\mathrm{F}_{0.5}$  scores than the conventional ensemble (Tables 10 and 11). This
suggests that using ESC instead of the conventional ensemble is better when combining multiple models.
In this work, we present a novel GEC system combination method using logistic regression by formulating the task as binary classification. Our method increases the  $\mathrm{F}_{0.5}$  score by 4.17 points from the highest base GEC system on CoNLL-2014 and 7.24 points on BEA-2019. This shows that our method does not overfit to the dataset even though it is trained on the BEA-2019 development data. In addition, our combined system outperforms the best published GEC system by 4.0 points on the BEA-2019 test set, 1.21 points on the CoNLL-2014 test set with original annotation, and 3.41 points on the CoNLL-2014 test set with alternative annotation.
We also compare our system combination method to the other GEC system combination methods, including MEMT (Heafield and Lavie, 2010), IBM (Kantor et al., 2019), and GEC-IP (Lin and Ng, 2021), and show that our method has better expressive power compared to IBM and GEC-IP. We also show that our method produces a higher  $F_{0.5}$  score than the conventional ensemble.
This work highlights the importance of edit type information in system combination for grammatical error correction. In recent GEC research, most systems just employ end-to-end learning without making use of any syntactic knowledge. We show that syntactic information, such as edit type, can also be useful to improve neural grammatical error correction models. Even though we only evaluate this method on English GEC, we believe our method can also be applied to other languages when a similar error type annotation toolkit is available.
We thank Ammar Fathin Sabili and Hannan Cao for helpful comments on this paper. This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-RP-2019-014). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nssc.sg).
$^{5}$ https://github.com/grammarly/gector/tree/feas1532608
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel iterative edit models for local sequence transduction. In Proceedings of EMNLP, pages 4260-4270.
Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of BEA, pages 52-75.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of ACL, pages 793-805.
Yo Joong Choe, Jiyeon Ham, Kyubyong Park, and Yeoil Yoon. 2019. A neural grammatical error correction system built on better pre-training and sequential transfer learning. In Proceedings of BEA, pages 213-227.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of NAACL 2012, pages 568-572.
Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In Proceedings of BEA, pages 252-263.
Nicolle Gruzling. 2007. Linear separability of the vertices of an  $n$ -dimensional hypercube. Ph.D. thesis, University of Northern British Columbia.
Wenjuan Han and Hwee Tou Ng. 2021. Diversity-driven combination for grammatical error correction. In Proceedings of ICTAI.
Kenneth Heafield and Alon Lavie. 2010. CMU multi-engine machine translation for WMT 2010. In Proceedings of WMT, pages 301-306.
Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceedings ACL, pages 4248-4254.
Yoav Kantor, Yoav Katz, Leshem Choshen, Edo Cohen-Karlik, Naftali Liberman, Assaf Toledo, Amir Menczel, and Noam Slonim. 2019. Learning to combine grammatical error corrections. In Proceedings of BEA, pages 139-148.
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In Proceedings of EMNLP, pages 1236-1242.
Ruixi Lin and Hwee Tou Ng. 2021. System combination for grammatical error correction based on integer programming. In Proceedings of RANLP, pages 829-834.
Wolfgang Macherey and Franz J. Och. 2007. An empirical study on computing consensus translations from multiple machine translation systems. In Proceedings of EMNLP, pages 986-995.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of CoNLL, pages 1-14.
Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of CoNLL, pages 1-12.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR - grammatical error correction: Tag, not rewrite. In Proceedings of BEA, pages 163-170.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035.
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of ACL, pages 702-707.
Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of BEA, pages 37-47.
Raymond Hendy Susanto, Peter Phandi, and Hwee Tou Ng. 2014. System combination for grammatical error correction. In Proceedings of EMNLP, pages 951-962.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
A Base System URLs
The URLs of these base systems are given in Table 12.
B Computing Budget
We run our experiments on a single NVIDIA A100 GPU. The training time takes an average of 0.6 minutes for all experiments. Even though we use GPU in our experiments, our method takes less than 2 minutes to train on a CPU.
Model name	URLs
1. T5-Large	T5: https://github.com/google-research/text-to-text-transfer-transformerFine-tuning dataset (cLang-8):https://github.com/google-research-datasets/clang8
2. GECToR XLNet	https://github.com/grammarly/gector/tree/fea1532608
3. GECToR Roberta	https://github.com/grammarly/gector/tree/fea1532608
4. Riken&Tohoku	https://github.com/butsugiri/gec-pseudodata
5. UEDIN-MS	https://github.com/grammatical/pretraining-bea2019/
6. Kakao&Brain	https://github.com/kakaobrain/helo_word/
Table 12: Base system sources.
#	BEA-2019 Dev	CoNLL-2014 Test
P	R	F0.5	P	R	F0.5
1	72.16	40.96	62.62	80.75	44.56	69.47
2	71.62	41.38	62.49	80.36	44.76	69.33
3	73.25	39.46	62.54	81.83	43.11	69.37
4	72.58	40.37	62.59	81.52	44.11	69.69
5	72.86	40.37	62.76	81.48	43.78	69.51
Table 13: CoNLL-2014 combination results with 5 runs. The results are from experiments with random seeds 0, 17, 171, 999, and 8888 respectively.
#	BEA-2019 Dev	BEA-2019 Test
P	R	F0.5	P	R	F0.5
1	74.04	38.88	62.70	87.36	59.32	79.82
2	73.36	40.26	63.00	86.41	60.98	79.76
3	74.02	39.28	62.89	87.37	59.69	79.95
4	73.63	40.12	63.09	86.65	60.91	79.90
5	73.69	40.02	63.08	86.68	60.78	79.87
Table 14: BEA-2019 combination results with 5 runs. The results are from experiments with random seeds 0, 17, 171, 999, and 8888 respectively.
C Complete Experimental Results
The complete 5-run experiments can be seen in Table 13 for our CoNLL-2014 results and Table 14 for our BEA-2019 results. The experiment with the highest BEA-2019 dev set  $\mathrm{F}_{0.5}$  score for the CoNLL-2014 experiments is from the one with random seed 8888, and the experiment with the highest BEA-2019 dev set  $\mathrm{F}_{0.5}$  score for the BEA-2019 experiments is from the one with random seed 999.
D Expressive Power of ESC
D.1 Definition
Suppose  $f_{\theta}(x_1, \ldots, x_n)$  is a combination function which takes  $x_1, \ldots, x_n$  as input and returns a binary value using the parameter value  $\theta \in \Omega$  where  $\Omega$  is the parameter space. Let  $\mathcal{EP}(f)$  be the expressive power of  $f$ , which is defined as the number of unique functions of  $f$ , obtained from all possible parameter values, formally given as follows:
$$
\mathcal {E P} (f) = | \{f _ {\theta} | \theta \in \Omega \} |
$$
D.2 Calculating the Number of Possible Functions<sup>6</sup>
With  $n$  Boolean variables, there are  $2^{n}$  possible combinations of the variable values. For example, for  $n = 2$ , there are  $2^{2}$  possible values:
$$
\begin{array}{l} \bullet a = 0, b = 0 \\ \bullet a = 0, b = 1 \\ \bullet a = 1, b = 0 \\ \bullet a = 1, b = 1 \\ \end{array}
$$
From these 4 possible values, we can generate  $2^{4}$  functions, based on the combination of the returned values for each input combination. The possible functions are shown in Table 15. If we set the output of  $a = 1, b = 1$  to a fixed value such as in Table 16, the number of semantically different functions is reduced by half, as F0 becomes the same function as F1, F2 becomes the same function as F3, and so on. This reduces the expressive power of the function to half from 16 to 8.
$^{6}$ This explanation is inspired from https://math.stackexchange.com/a/698086
a b	F0	F1	F2	F3	F4	F5	F6	F7	F8	F9	F10	F11	F12	F13	F14	F15
0 0	0	0	0	0	0	0	0	0	1	1	1	1	1	1	1	1
0 1	0	0	0	0	1	1	1	1	0	0	0	0	1	1	1	1
1 0	0	0	1	1	0	0	1	1	0	0	1	1	0	0	1	1
1 1	0	1	0	1	0	1	0	1	0	1	0	1	0	1	0	1
Table 15: Possible input & output mappings of logic functions with two Boolean variables.
a b	F0	F1	F2	F3	F4	F5	F6	F7	F8	F9	F10	F11	F12	F13	F14	F15
0 0	0	0	0	0	0	0	0	0	1	1	1	1	1	1	1	1
0 1	0	0	0	0	1	1	1	1	0	0	0	0	1	1	1	1
1 0	0	0	1	1	0	0	1	1	0	0	1	1	0	0	1	1
1 1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Table 16: Possible logic functions with two Boolean variables, with the output of  $a = 1$ ,  $b = 1$  fixed to 0.
n	all	Linearly separa-ble	ESC's bound	lower-
2	\( 2^{2^2} \)	14	7
3	\( 2^{2^3} \)	104	52
4	\( 2^{2^4} \)	1882	941
5	\( 2^{2^5} \)	94572	47286
6	\( 2^{2^6} \)	15028134	7514067
7	\( 2^{2^7} \)	8.3780708e+9	4.189035e+9
8	\( 2^{2^8} \)	1.7561539e+13	8.7807698e+12
9	\( 2^{2^9} \)	1.4413053e+17	7.2065266e+16
Table 17: The number of possible Boolean functions.  $n$  denotes the number of parameters, which is the same as the number of hypotheses.
ESC is a generalized linear function, so it can only express linearly separable functions. Furthermore, we do not consider any edits that do not appear in any of the hypotheses, making the output of the function with the input of  $(x_{1} = 0,x_{2} = 0,\ldots ,x_{k} = 0)$  always resulting in 0. Thus, the number of semantically different logic functions that can be expressed is at least half of linearly separable Boolean functions. With the list of number of linearly separable Boolean functions from (Gruzling, 2007), we can get the lower bound of ESC's expressive power in Table 17.