Improving Grammatical Error Correction with Multimodal Feature Integration
Year: 2023

Abstract:
Grammatical error correction (GEC) is a promising task aimed at correcting errors in a text. Many methods have been proposed to facilitate this task with remarkable results. However, most of them only focus on enhancing textual feature extraction without exploring the usage of other modalities’ information (e.g., speech), which can also provide valuable knowledge to help the model detect grammatical errors. To shore up this deficiency, we propose a novel framework that integrates both speech and text features to enhance GEC. In detail, we create new multimodal GEC datasets for English and German by generating audio1 from text using the advanced text-to-speech models. Subsequently, we extract acoustic and textual representations by a multimodal encoder that consists of a speech and a text encoder. A mixture-of-experts (MoE) layer is employed to selectively align representations from the two modalities, and then a dot attention mechanism is used to fuse them as final multimodal representations. Experimental results on CoNLL14, BEA19 English, and FalkoMERLIN German show that our multimodal GEC models achieve significant improvements over strong baselines and achieve a new stateof-the-art result on the Falko-MERLIN test set.

heading: 1 Introduction
text: Grammatical error correction (GEC) is one of the promising applications in natural language processing (NLP), aiming to correct sentences containing grammatical errors. GEC has attracted substantial attention in the past few decades owing to its importance in writing assistance for language learners (Rothe et al., 2021; Zhao and Wang, 2020; Qorib et al., 2022; Wan et al., 2020; Chollampatt and Ng, 2018; Tarnavskyi et al., 2022; Kaneko et al., 2020;
∗Equal Contribution †Co-corresponding Author
1https://github.com/NLP2CT/MultimodalGEC
Zhang et al., 2022a; Fang et al., 2023a; Zhang et al., 2023a; Fang et al., 2023b; Zhang et al., 2023b).
In recent years, pre-trained Transformer-based models have proven effective in many NLP tasks (Hu et al., 2022a,b; Clinchant et al., 2019; Liu and Lapata, 2019; Hu et al., 2023b; Zhong et al., 2022; Liu et al., 2021; Li et al., 2022), including GEC (Gong et al., 2022; Li et al., 2023), because these models consist of multiple-layer multi-head attention and are trained with massive language data so that they are more powerful in feature extraction than other counterpart models. For example, Kaneko et al. (2020) first proposed to fine-tune BERT with the GEC corpus and then use the output of BERT as additional features to enhance GEC. Rothe et al. (2021) used the T5 structure (Raffel et al., 2020) to refine the GEC corpus (i.e., CLang8) and obtained promising results in GEC for different languages. Furthermore, Qorib et al. (2022); Tarnavskyi et al. (2022) employed binary classification or majority votes on span-level edits to ensemble multiple Transformer-based models.
Although these methods have achieved considerable improvements, they may focus on the bet-
9328
ter use of textual data while failing to take other modalities into consideration (e.g., speech). Many studies have shown that other modality data (e.g., speech) can effectively enhance feature extraction and thus promote model performance, such as risk forecasting (Sawhney et al., 2020), semantic matching (Huzaifah and Kukanov, 2022), etc. For example, Huzaifah and Kukanov (2022) studied a joint speech-text embedding space through a semantic matching objective and achieved better results in downstream tasks. Kim and Kang (2022) proposed to learn the cross-modality interaction between acoustic and textual information for emotion classification, which outperformed unimodal models. These works illustrate that audio signals can be regarded as complementary information and provide valuable features to promote text processing. Besides, intuitively, the audio with grammatical errors can be easily captured by the native speakers according to their spoken language experiences, which can implicate that speech should be effective in helping the model to distinguish whether the text contains ungrammatical elements.
Therefore, in this paper, we propose to integrate speech and text features to promote GEC, with an example shown in Figure 1. Firstly, owing to the lack of multimodal datasets for GEC, we adopt advanced text-to-speech (TTS) models to automatically generate audio for each instance in GEC datasets. Afterward, we extract acoustic and textual representations by a multimodal encoder that consists of pre-trained speech and text encoders. Furthermore, we propose to utilize an MoE layer to selectively align features from speech and text modalities, and then simple dot attention is applied to fuse them as final multimodal representations, which are then input to a pre-trained decoder to generate corrected sentences. Experimental results on English and German benchmarks illustrate the effectiveness of our proposed model, where our model achieves significant improvements over strong unimodal GEC baselines. Further analysis shows that our multimodal GEC model demonstrates significant improvements in most POS-based fine-grained error types, as well as in the major Operation-Level error types such as word substitutions, missing words, and unnecessary words.
The contributions are concluded as follows:
• To the best of our knowledge, this paper is the first to utilize a multimodal model to combine audio and text features to facilitate GEC.
• This paper constructs multimodal GEC datasets for English and German, where each sample in the dataset is a triple (ungrammatical text, audio, grammatical text). • This paper proposes to use a mixture-of-experts module to dynamically align text and speech pairs for multimodal GEC. • This paper reveals the gains and losses of incorporating speech modality into GEC on error types, providing clues for future research.

heading: 2 Data Construction
text: Owing to the lack of speech data in the GEC task, we need to construct multimodal GEC datasets for multimodal GEC tasks. Therefore, in this section, we give the details of dataset construction. Speech processing has achieved promising improvement over the past few decades, including converting sentences in the text into utterances (Ren et al., 2019; Qi et al., 2023). Therefore, we employ the advanced speech synthesis system to convert each piece of source side of GEC data (i.e., the ungrammatical side) into audio data to construct GEC multimodal data. As a result, each example in the GEC dataset is expanded into a triplet consisting of the ungrammatical sentence, the audio generated from the corresponding ungrammatical sentence, and the grammatical sentence.

heading: 2.1 English GEC Multimodal Data
text: For constructing the English GEC multimodal dataset, we adopt the FastSpeech22 text-to-speech model (Wang et al., 2021) to produce audio data from the source side of English GEC data. Specifically, to construct GEC multimodal training data, we convert the distilled English CLang8 GEC data (Rothe et al., 2021) into audio data. For constructing development and test sets, we select the widelyused CoNLL14 (Ng et al., 2014) and BEA19 (Bryant et al., 2019) English GEC benchmarks. For the CoNLL14 benchmark, the CoNLL13 (Ng et al., 2013) and the official-2014.combined.m2 version of CoNLL14 are used for constructing multimodal development and test sets, respectively. For the BEA19 benchmark, we use the BEA19 development and test sets to construct audio data.
2https://github.com/facebookresearch/fairseq/ tree/main/examples/speech_synthesis

heading: 2.2 German GEC Multimodal Data
text: For building German multimodal GEC datasets, we employ gTTS (Google Text-to-Speech) toolkit3 to generate audio data from the source side of German GEC training, development and test data. We build multimodal training data from German CLang8 and the official Falko-MERLIN (Boyd et al., 2014) training data. As for the multimodal development and test sets, we produce the audio data from FalkoMERLIN German validation and test sets.
3https://gtts.readthedocs.io/en/latest/

heading: 2.3 Data Processing
text: To prepare the text GEC datasets for audio generation, we first remove duplicate instances from the English CLang8 dataset, while keeping the other datasets unaltered. Additionally, we follow Katsumata and Komachi (2020) to use Moses script (Koehn et al., 2007) to detokenize GEC data for English and German. The statistics of the final multimodal datasets are shown in Table 1.

heading: 3 Method

heading: 3.1 Problem Definition
text: Existing approaches mainly utilize an encoderdecoder framework to address the GEC problem. In detail, the input is a sentence with grammatical errors X = x1, x2, · · · , xN , where N is the number of tokens, and the goal of this task is to correct the input sentence and generate a right one Y = y1, y2, · · · , yL, where L is the length of target sentence. Motivated by the success of multimodal in other tasks (Li et al., 2018; Sawhney et al., 2020), in this paper, we propose a novel multimodal GEC task and take a text-audio pair (X,S) as input (text and audio, respectively), aiming to integrate acous-
tic and textual features to enhance GEC. Therefore, the generation process for the multimodal GEC problem can be formulated as:
p(Y |X,S) = L∏
t=1
p(yt | y1, . . . , yt−1, X, S). (1)
Moreover, we utilize the negative conditional loglikelihood of Y given the pair (X,S) to train the model:
θ∗ = argmax θ
L∑
t=1
log p (yt | y1, ..., yt−1, X, S; θ) ,
(2) where θ is the trainable parameters of the model. An overall structure of our proposed method is presented in Figure 2.

heading: 3.2 Multimodal Encoder
text: The multimodal encoder in our model consists of two main feature extractors: speech encoder and text encoder, respectively.
Speech Encoder We utilize a pre-trained Transformer-based model (e.g., wav2vec2 (Baevski et al., 2020)) as our speech encoder, which can learn powerful representations from speech audio and achieve promising results in many downstream tasks. [c1, c2, · · · , cP ] = fae(S), (3) where c is the features extracted from speech, fae refers to speech encoder and P is length of acoustic features.
Text Encoder We adopt a pre-trained model (e.g., T5 encoder (Raffel et al., 2020)) as our text encoder to capture textual features z from X:
[z1, z2, · · · , zN ] = fte(X), (4) where zi is high dimensional vector for representing token xi and fte refers to the text encoder.

heading: 3.3 Multimodal Alignment and Fusion
text: On the one hand, it is intuitive that the speech should be semantically close to the corresponding text if they are in one pair since they actually represent similar meanings through different modalities. On the other hand, audio is used to provide complementary information instead of completely consistent information to help the model to better recognize and detect grammatical errors. As a result, we should allow some variance between features extracted from different modalities during multimodal alignment.
Therefore, we adopt a mixture-of-experts (MoE) to dynamically select semantically similar information from acoustic features, which is used to align with textual representation. The MoE layer in our model consists of M experts, denoted as E1, E2, · · · , EM , and each expert is a simple MLP with ReLU. Note that although these experts have identical structures, they have separate parameters instead of shared ones. We first obtain the overall representation of the speech S and text X by mean pooling, which can be formulated as:
c̄ = Mean([c1, c2, · · · , cP ]), (5) z̄ = Mean([z1, z2, · · · , zN ]), (6)
We utilize the MoE to further extract the features from c̄ that should be close to z̄. Specifically, the output of i-th expert is denoted as Ei(c̄) and we follow Shazeer et al. (2017) to generate a gate Gi(c̄) for each expert. The output of the MoE module can be written as:
b =
M∑
i=1
Gi(c̄)Ei(c̄), (7)
where b should be the information that is semantically close to the text. We utilize a simple mean squared error (MSE) objective to constrain this process and align these textual and acoustic features, which can be formulated as:
Lmse = MSE(b, z̄), (8) After dynamic alignment between audio and text, we utilize dot attention to fuse these two features. In detail, we first compute the attention weight with the softmax function:
ai = Softmax(zicT). (9)
Herein, ai can be viewed as a probability distribution and used to produce a weighted sum over the visual patch representations:
zci = P∑
k=1
ai,kck. (10)
Finally, we sum the zc and z as final multimodal representation h.

heading: 3.4 Decoder
text: The multimodal representation h is input to the pretrained decoder (e.g., T5) to generate the correct sequence:
yt = fde(h, y1, · · · , yt−1) (11)
This process is repeated until the complete sentence is obtained.
As for training, the final objective is the linear combination of losses from the sequence generation and multimodal alignment:
L = Lge + λLmse, (12) where Lge is the basic sequence-to-sequence loss and λ is the weight to control the MSE loss.

heading: 4 Experimental Settings and Results

heading: 4.1 Data and Evaluation
text: The multimodal GEC data used for training is presented in Table 1 in section 2. With respect to English, we follow Rothe et al. (2021) and use only the English CLang8 multimodal data for training as they reported that further fine-tuning on highquality English datasets, such as FCE v2.1 (Yannakoudakis et al., 2011) and W&I (Yannakoudakis et al., 2018), led to a drop in performance. For validation, we use the CoNLL13 multimodal data and the BEA19 multimodal development data when testing on the CoNLL14 and BEA19 English test sets, respectively. In terms of German, we first train our models on the German CLang8 multimodal data as Rothe et al. (2021), and then finetune the models on the official Falko-MERLIN German multimodal training data. For the development and test data, we use the official Falko-MERLIN German benchmark. Additionally, to establish a stronger baseline, we follow Katsumata and Komachi (2020) to use the same 10M synthetic data (Náplava and Straka, 2019)4 to pre-train T5/mT5Large model for English and German.
For evaluation, we use the M2 scorer (Dahlmeier and Ng, 2012) to evaluate the model performance on the CoNLL14 English test and the official FalkoMERLIN German benchmark. The BEA19 English test is evaluated by ERRANT (Bryant et al., 2017). We employ the T -test method to test the significance of the results, except for the BEA19 English test, which is a blind test set.

heading: 4.2 Implementation Details and Training
text: In our experiments, we adopt Huggingface5 library to build our multimodal GEC model. Specifically, for the basic experiments, we utilize T5Large (Raffel et al., 2020) and mT5-Large (Xue
4https://github.com/ufal/ low-resource-gec-wnut2019/tree/master/data
5https://github.com/huggingface/transformers
et al., 2020) as our text backbone models (including both text encoder and decoder), with the former being used for English and the latter for German. We follow their default setting, which uses 24 layers of self-attention with 16 heads. For the experiments with stronger baselines, we use our T5-Large and mT5-Large models fine-tuned on 10M synthetic data as text backbone models for English and German, respectively. The details of the training settings can be found in Appendix A.1. For the speech encoder, we adopt Hubert Large pre-trained model (Hsu et al., 2021) to extract features for English audio and wav2vec2-xls-r-300m pre-trained model (Babu et al., 2021) for German speech. We also follow the default settings for these speech models. As for training, we utilize Adafactor (Shazeer and Stern, 2018) to optimize all trainable parameters in our model. We set the number of experts to 6. The weight hyper-parameter λ is set to 0.1 for both English and German experiments. The other settings for training the multimodal GEC models are reported in Appendix A.2.

heading: 4.3 Baselines
text: To explore the effect of the proposed multimodal model for GEC, we compare our model with the following baselines: • LRGEC (Náplava and Straka, 2019): it pre-
trains a Transformer seq2seq model on synthetic data and then fine-tunes on authentic data. • TAGGEC (Stahlberg and Kumar, 2021): the model improves GEC performance by data augmentation (e.g., generating synthetic data with the guidance of error type tags). • GECTOR (Omelianchuk et al., 2020), TMTC (Lai et al., 2022), EKDGEC (Tarnavskyi et al., 2022): these models utilize the sequence tagging approach to improve GEC performance with multiple stage training, where they firstly pre-train on errorful-only sentences and further fine-tune on a high-quality dataset. • SADGEC (Sun et al., 2021), gT5 XXL and T5/MT5 LARGE/XXL (Rothe et al., 2021): these GEC models borrow knowledge from pretrained language models, where SADGEC is based on the BART (Lewis et al., 2020) pretrained model, gT5 XXL is a large teacher model for distilling Lang8 data, which is first pre-trained from scratch on a large amount of synthetic data followed by fine-tuning on high-quality data. T5/MT5 LARGE/XXL adopt
T5/mT5 as the backbone structure and fine-tune on the corresponding distilled CLang8 data for GEC tasks in different languages.

heading: 4.4 Experimental Results
text: Results on English dataset To illustrate the effectiveness of our proposed model, we compare our model with existing studies with the results reported in Table 2. We obtain several observations from the results. First, the comparison between OURS and other baselines illustrate the ef-
fectiveness of our design in the GEC task, where our model achieves much better performance even though these competitors utilize many ways (e.g., data augmentation) to enhance feature extraction in GEC. The reason might be that compared to pure textual information, audio can provide complementary information to help the model better grasp the grammatical error in the sentence, and our model can selectively align these features from speech and text by the MoE module. It is easy to follow that a native speaker can distinguish whether the audio is grammatically correct. Second, compared to the sequence tagging method (e.g., GECTOR), sequence-to-sequence based models (e.g., T5-LARGE) perform better in recall score but are inept at precision. Especially it is found that the strength of our proposed model lies in its high recall compared to other baselines. Third, continuing training T5-Large on 10M synthetic data can further improve the model performance, illustrating that synthetic data can alleviate the gap between GEC data and pre-training corpus. Appendix A.3 shows some examples generated by the unimodal and multimodal GEC models.
Results on German dataset To further demonstrate the validity of our model, we also conduct experiments on the German dataset, with the results reported in 3. We can obtain similar trends as in English GEC, where our proposed mode out-
performs other baselines and achieves a superior F0.5 score. Especially, by further fine-tuning the models on the official data, we achieve a new stateof-the-art result (i.e., 76.3 F0.5). This result further demonstrates that audio can provide valuable benefits in GEC tasks regardless of language type. Additionally, even though the German dataset is much smaller compared to the English dataset, our model still achieves significant improvements, which highlights its effectiveness in low-resource settings.

heading: 5 Analyses

heading: 5.1 Ablation Study
text: To explore the effectiveness of our proposed method, we conduct the ablation studies with the following settings: a) removing the MoE layer (−MOE) and retaining the dot attention module to fuse acoustic and textual features. b) removing the speech encoder (−SPEECH ENC.), which degenerates our multimodal GEC model into a text-only unimodal GEC model. As shown in Table 4, when we remove the MoE layer, the results of the multimodal GEC model show a decrease in all settings, demonstrating the validity of MoE in the multimodal feature fusion. Moreover, if we discard the speech encoder, the results of the reverted text-only unimodal GEC baseline models are significantly lower than the multimodal model for both English and German, which illustrates the effectiveness of our proposed multimodal GEC models.

heading: 5.2 Error Type Performance
text: To investigate the ability of GEC systems to correct different error types, we used the ERRANT toolkit (Bryant et al., 2017) to analyze the evaluation results on the CoNLL14 test set with respect to both POS-based fine-grained error types and OperationLevel error types.
Fine-grained Error Types Figure 3 shows the performance of the POS-based fine-grained error types. We can observe that while multimodal GEC is inferior to text-only unimodal GEC systems in certain error types (i.e., PUNCT, ADV, CONJ, and PREP), our model obtains better results in most types of errors, including ADJ, NOUN, NOUM: NUM, PRON, VERB, VERB: TENSE, DET, MORPH, ORTH, and PART, which further confirms the effectiveness of multimodal feature integration in the GEC task. In fact, adverb and conjunction error types account for a relatively small percentage of all grammatical errors (not more than 1.6%). In other words, multimodal GEC can improve the performance of common errors in GEC and thus bring considerable improvements overall.
Operation-Level Error Types We evaluate the performance of Operation-Level error types using the ERRANT toolkit, which categorizes them into three categories: Replacement, Missing, Unnecessary. Considering that word order (WO) is a sub-type of Replacement, which is different from other types of errors, we manually separate into a separate category. As shown in Table 5, compared to text-only unimodal GEC baseline models, our multimodal GEC models are better at correcting the major operation-level error types, such as word substitutions (64.3%), missing words (17.9%), and unnecessary words (17.0%), demonstrating that the corresponding speech information is beneficial to GEC. However, the multimodal GEC model does not perform well in correcting word order, even if it is a minor issue (0.8%). We hypothesize that correcting word order requires sentence structure information (Zhang et al., 2022b), but the speech may not provide such information to GEC models.

heading: 6 Related work
text: Grammatical Error Correction (GEC) is the task of automatically identifying and correcting grammatical errors in a text (Ng et al., 2013). Previous research in this field has primarily focused on strengthening the representations of text data through data augmentation techniques, such as using the back-translation method (Sennrich et al., 2016) for the GEC task (Kasewa et al., 2018; Xie et al., 2018; Kiyono et al., 2019), and injecting noise with specific rules into grammatical sentences (Lichtarge et al., 2019; Zhao et al., 2019; Xu et al., 2019; Stahlberg and Kumar, 2021). More recently, pre-trained language models (PLMs) have
been demonstrated to be effective in improving the performance of GEC tasks. Studies such as Choe et al. (2019) have leveraged sequential transfer learning to adapt pre-trained Transformer models to the GEC domain. Kaneko et al. (2020) initialized an encoder-decoder GEC model with pre-trained BERT weights to enhance GEC performance. Katsumata and Komachi (2020) utilized the pre-trained BART model as a generic pre-trained encoder-decoder model for GEC, and Rothe et al. (2021) adopted a pre-trained T5 model to distill GEC corpus and used the pre-trained structure as part of the network for distilled GEC training, achieving promising results. However, to date, no previous work has attempted to incorporate multimodal information (e.g., speech modality) into the GEC task. Our work is the first to explore the use of multimodal information for GEC.
Multimodal Many studies have demonstrated the potential of incorporating multimodal information in improving the performance of single-modal tasks in the NLP domain. For example, Schifanella et al. (2014) and Cai et al. (2019) integrated image modality into the Twitter sarcasm detection task and found that incorporating image information can enhance the performance of this text-only task. Hu et al. (2023a) proposed to integrate radiology
images and textual findings to improve impression generation. Additionally, Zheng et al. (2021) fused acoustic and text encoding to jointly learn a unified representation, thereby improving speech-to-text translation tasks. Li et al. (2017) demonstrated that fusing speech modality can enhance the readability of text summarization tasks. Huzaifah and Kukanov (2022) studied a joint speech-text embedding space through a semantic matching objective, achieving improved results in downstream tasks. Kim and Kang (2022) proposed a method for learning the cross-modality interaction between acoustic and textual information, which outperformed the unimodal models in emotion classification. In this work, we are the first to attempt to fuse acoustic and text to improve the GEC task.

heading: 7 Conclusion
text: This paper presents a novel approach to the task of multimodal GEC that integrates speech and text features to improve grammatical error correction. Due to the scarcity of speech data in GEC, we expand the original GEC data to create new multimodal GEC datasets for English and German, where each sample in our datasets is a triple (grammatically incorrect text, audio, and corrected text). Our approach utilizes a speech and text encoder to extract
acoustic and textual features from the speech and input text, respectively. Then, we employ an MoE approach to selectively extract audio features that align with the textual features and use a dot attention layer to fuse the features from different modalities as the final representation. This fused representation is input to the decoder to generate the corrected sentence. Our experimental results on widely-used benchmarks demonstrate the effectiveness of our proposed model, achieving significant improvements compared to existing studies.
Limitations
Our proposed multimodal Grammatical Error Correction (GEC) model is based on a Seq2Seq generative framework, which utilizes different encoders to extract information from each modality, and then fuses them to provide input to an autoregressive decoder. However, in this work, we did not explore the use of a sequence tagging framework, which may be a consideration for future research, as it has the advantage of faster decoding speed. Additionally, this study focuses on the use of audio representations of the source-side of GEC data, rather than the target-side, to construct multimodal GEC data. Our further analysis concludes that our proposed multimodal GEC model has limitations in correcting certain minor error types (e.g., ADV, CONJ, PUNCT, and word order) when compared to text-only GEC models.

heading: Acknowledgments
text: This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCT/060/2022/AFJ, FDCT/0070/2022/AMJ), and the Multi-year Research Grant from the University of Macau (Grant No. MYRG2020-00054-FST), and the Shenzhen Science and Technology Program (JCYJ20220818103001002), and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen. This work was performed in part at SICC which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University of Macau.

heading: A Appendix
text: A.1 Pre-training Settings for T5/mT5-Large Model
The settings of hyper-parameters for pre-training T5/mT5-Large models for English and German are listed in Table 6.
A.2 Settings of Training Multimodal GEC Models
Table 7 presents the settings of hyper-parameters for training English and German multimodal GEC models.
A.3 Case Study Table 8 shows some examples generated by the textonly unimodal GEC model and multimodal GEC model. Our multimodal GEC model is better at correcting common error types (e.g. VERB) while exhibiting inferior performance in correcting word order errors.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section
A2. Did you discuss any potential risks of your work? Not applicable. There are no potential risks associated with this paper because all tasks we used are public ones that have been verified for years.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract section, and Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: We use ChatGPT AI writing assistants to check some spelling errors and polish some sentences of our work (i.e., sections 4.1, 4.2, and 7)
B 3 Did you use or create scientific artifacts? Section 2, Section 4, and Section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All datasets and models we used here are public without restriction for research purposes.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? All datasets and models we used here are public without restriction for research purposes.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we used in our paper do not have such issues according to the claims in the original paper.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The datasets we used in our paper do not have such issues according to the claims in the original paper.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2, and Section 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 2, Section 4, Section 4.1, Section 4.4, Section 5.1, Appendix A.1, A.2, Table 3, and Table 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.1, A.2

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4, Appendix A.1, A.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4, Section 5.1, Table 3, Table 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2, Section 4, and Appendix A.1, A.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: However, in this work, we did not explore the use of a sequence tagging framework, which may be a consideration for future research, as it has the advantage of faster decoding speed. Additionally, this study focuses on the use of audio representations of the source-side of GEC data, rather than the target-side, to construct multimodal GEC data. Our further analysis concludes that our proposed multimodal GEC model has limitations in correcting certain minor error types (e. g. , ADV, CONJ, PUNCT, and word order) when compared to text-only GEC models