Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods
Year: 2023

Abstract:
Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks. However, applying prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks and their controllability remains underexplored. Controllability in GEC is crucial for real-world applications, particularly in educational settings, where the ability to tailor feedback according to learner levels and specific error types can significantly enhance the learning process. This paper investigates the performance and controllability of promptbased methods with GPT-3 for GEC tasks using zero-shot and few-shot setting. We explore the impact of task instructions and examples on GPT-3’s output, focusing on controlling aspects such as minimal edits, fluency edits, and learner levels. Our findings demonstrate that GPT-3 could effectively perform GEC tasks, outperforming existing supervised and unsupervised approaches. We also showed that GPT-3 could achieve controllability when appropriate task instructions and examples are given.

heading: 1 Introduction
text: Grammatical Error Correction (GEC) is an essential application of Natural Language Processing (NLP) in educational settings, as it significantly enhances learners’ language skills and writing performance (Kaneko et al., 2022). In real-world applications, controlling specific GEC settings, such as minimal and fluency edits and learner level-based corrections, is crucial to address diverse learning needs and scenarios (Napoles et al., 2017; Bryant et al., 2019; Flachs et al., 2020). Although recent GEC approaches based on supervised learning have achieved remarkable progress, they heavily rely on large training datasets comprising both genuine and pseudo data (Xie et al., 2018; Ge et al., 2018; Zhao et al., 2019; Lichtarge et al., 2019; Xu et al., 2019; Choe et al., 2019; Qiu et al., 2019; Grundkiewicz
et al., 2019; Kiyono et al., 2019; Grundkiewicz and Junczys-Dowmunt, 2019; Wang and Zheng, 2020; Zhou et al., 2020; Wan et al., 2020; Koyama et al., 2021a). Collecting such data for each specific setting is challenging and time-consuming, which limits the scalability of these methods in various learning situations.
Prompt-based methods utilize large-scale pretrained language models (PLMs), such as GPT-3, and have demonstrated promising results in numerous NLP downstream tasks. These tasks include natural language inference, question answering, and summarization (Brown et al., 2020; Radford et al., 2019). Given the demand for control in GEC tasks across various settings, prompt-based methods are appealing because they deliver exceptional performance without needing extensive labeled data. Despite the success of prompt-based methods in multiple NLP tasks, their application to GEC remains under-explored. Although Coyne and Sakaguchi (2023) and Fang et al. (2023) have recently assessed prompt-based methods on select GEC benchmarks, a comprehensive analysis has yet to be conducted. This study aims to bridge this gap by concentrating on in-depth analyses of prompt-based methods and their controllability, aspects that have not been thoroughly investigated in previous research.
Our research seeks to address the following questions: 1) To what extent can PLMs using promptbased methods solve GEC tasks? and 2) Is it possible to control GEC settings with prompts written in natural language using prompt-based methods?
In this work, we demonstrate that prompt-based methods with GPT-3 (Brown et al., 2020) achieve outstanding performance in GEC tasks (Section 3). In addition, the approach provides better control over the GEC process using task instructions and examples (Section 5). We conduct analyses to examine the impact of different types of task instructions on GPT-3’s performance in both zero-shot
205
and few-shot setting, which emphasizing the importance of appropriate task instructions for GEC tasks (Section 4.1). Additionally, we investigate the effect of varying the number of examples in few-shot setting, and reveal that performance improves as the number of examples increases, albeit not strictly linearly (Section 4.2).
Furthermore, we explore the model’s controllability in various GEC scenarios, more specifically, its ability to concentrate on either minimal or fluency aspects (Section 5.1) and edits based on learner levels (Section 5.2). Experimental results indicate that task instructions alone may be sufficient to control editing without examples. However, we found that combining task instructions with examples resulted in more effective controlling performance. This indicates the importance of both task instruction and examples for better control of GEC settings using prompt-based methods, although the example set tends to have more importance.

heading: 2 Overall Experimental Settings
text: In this study, we designed a series of experiments using the prompt-based method with GPT-3 to evaluate the performance in GEC tasks. We utilized the GPT-3 model (text-davinci-003) through the API provided by OpenAI1. Our experiments were conducted in two settings: zero-shot and few-shot.
Zero-shot In the zero-shot setting, we assessed GPT-3’s ability to perform GEC tasks without any prior examples. We employed the following template for prompts in the zero-shot setting:
{task instruction}: {input text}; output:___
Few-shot For the few-shot setting, we implemented in-context learning as described by Brown et al. (2020). We provided the model with a few examples to guide its understanding of the GEC task. We randomly sampled pairs of examples from the training (or validation) sets of each experimental setting to serve as examples for the model. Details on the number and source of examples used in each experiment are described in the corresponding sections below. The template for prompts in the few-shot setting is as follows:
{task instruction}
1https://openai.com/blog/openai-api
{example 1} ... {example N} {input text}; output:___
Prompt We used natural language text prompts for all our experiments. The task instruction within the prompt serves as a directive that informs the model about the desired outcome of each task. We varied the task instructions in both zero-shot and few-shot setting to examine the model’s adaptability to different phrasings (refer to Section 4.1). The instruction candidates employed in our prompt analyses are listed in Appendix A. Examples of task instructions include: Correct the grammatical errors in the following sentence, Revise mistakes in this text, and Rewrite the following text with proper grammar.

heading: 3 General Performance
text: To address research question 1) mentioned in Section 1, we investigated the overall performance of the prompt-based method with GPT-3 in GEC tasks. This investigation is particularly relevant given the increasing prevalence of GPT-3 in various NLP applications and the need to assess its potential capabilities for GEC tasks specifically.

heading: 3.1 Settings
text: We evaluated the performance of GPT-3 on three GEC test sets: JFLEG (Napoles et al., 2017), CoNLL2014 (Ng et al., 2014), and W&I+LOCNESS (Bryant et al., 2019; Granger, 1998) using both zero-shot and few-shot settings with 16 examples. We used examples from the training set of JFLEG, NUCLE (Dahlmeier et al., 2013), and W&I+LOCNESS as examples in the few-shot setting when evaluating with JFLEG, CoNLL2014, and W&I+LOCNESS test sets, respectively.
We compared our prompt-based methods to baselines, including supervised and unsupervised approaches. For the supervised approach, we trained a Transformer (big) using the settings described in Vaswani et al. (2017) and employed annotated data from multiple training sets. These sets included W&I+LOCNESS, FCE corpus (Yannakoudakis et al., 2011), Lang-8 Corpus of Learner English (Mizumoto et al., 2012), and NUCLE. After removing uncorrected sentence pairs, the train-
ing data used to train the Transformer model was approximately 600K pairs. For unsupervised approach, we compared our methods to previous work in the literature including Grundkiewicz and Junczys-Dowmunt (2019) and Grundkiewicz et al. (2019) where models were pre-trained with synthetic data. We also compared with the result of ChatGPT performance in zero-shot with chain-ofthought (CoT) reported in Fang et al. (2023).

heading: 3.2 Results
text: Table 1 shows the GLEU scores for JFLEG, F0.5 scores for CoNLL2014, and W&I+LOCNESS. From the table, GPT-3 performed competitively in the GEC tasks in both zero-shot and few-shot settings, outperforming the Transformer model in all test sets. In the zero-shot setting, GPT-3 surpassed the Transformer, with gains of about 11, 5, and 2 percentage points on JFLEG, CoNLL2014, and W&I+LOCNESS, respectively. The few-shot setting with 16 examples further improved GPT-3’s performance, indicating the model’s capability to adapt to the task with minimal examples quickly.
When comparing GPT-3 to unsupervised methods, we observe that GPT-3 outperforms other approaches in all test sets consistently. This comparison demonstrates the advantage of GPT-3 over existing unsupervised methods, even in the zeroshot setting. When comparing the performance of ChatGPT in the zero-shot setting with CoT, GPT-3 outperforms ChatGPT CoT in all three test sets. These results indicate GPT-3 is a more effective model for GEC tasks, especially in unsupervised settings.

heading: 4 Investigation on Prompt
text: In this section, we analyze the impact of different factors in prompt on the performance of GPT-3 in
GEC tasks. We focus on two factors: (1) the type of task instructions used and (2) the number of examples used in the few-shot settings. Our primary objective is to comprehend the influence of various factors in prompts to the models’ output, which will enable us to optimize GPT-3 more effectively for GEC tasks.

heading: 4.1 Effect of Task Instruction
text: In this section, we examine the effect of various types of task instructions on GPT-3’s performance in GEC tasks. We conduct evaluations using different task instructions in both zero-shot and few-shot settings.

heading: 4.1.1 Settings
text: We created three types of task instructions, with ten candidates per type, following related work on natural language inference task (Webson and Pavlick, 2022). The types of task instructions are as follows (See Appendix A for details). We used the JFLEG validation set in this experiment.
Instructive instructions explicitly request the model to correct grammatical errors in the given text, such as Correct grammatical errors in this sentence and Revise grammatical mistakes in the following text.
Misleading instructions do not directly ask for grammar correction but instead require paraphrasing or rewriting, such as Paraphrase the following sentence and Rewrite the following text to make it clearer.
Irrelevant instructions are unrelated to grammar correction, such as Translate the following sentence and Write a news headline about this sentence.

heading: 4.1.2 Result
text: Figure 1 shows the summary of the results when using different types of instructions in both zero-shot and few-shot settings. The findings reveal that task instructions significantly affect the performance of GPT-3 in GEC tasks.
In the zero-shot setting, instructive instructions produced the highest average score (65.54), while irrelevant instructions resulted in the lowest average score (17.05), clearly demonstrating that the type of task instruction impacts the model’s performance. Misleading instructions fell in the middle, with an average score of 43.45.
In few-shot settings, instructive instructions still outperformed the other two types, but the performance gap between instructive and misleading instructions decreased as the number of examples increased. The variance of the scores decreased with an increasing number of examples, suggesting that the model’s performance becomes more consistent as it receives more examples.
When comparing the different few-shot settings, we observed a clear trend of increasing performance as the number of examples increased. The standard deviation also decreased as the number of examples increased, indicating that the model’s performance became more consistent with more examples.

heading: 4.2 Effect of Number of Examples
text: In this section, we examine the impact of the number of examples used in few-shot settings on GPT3’s performance. Our objective is to understand
how providing varying numbers of examples to the model influences its performance. By maintaining a fixed instruction and focusing solely on varying the number of examples, we aim to better comprehend their effect on the model’s performance.

heading: 4.2.1 Settings
text: We conducted experiments on the JFLEG test set to examine the effect of the number of training examples on the model’s performance. The task instruction was kept consistent across all experiments. To perform the experiments, we randomly sampled examples from the training set of the JFLEG dataset. We tested the model with 2, 4, 8, 16, 32, and 64 examples, limiting the maximum number of examples to 64 due to the maximum input length of the model employed in our study.

heading: 4.2.2 Result
text: The results obtained from each experimental setting are presented in Figure 2. Our experiments revealed a clear trend: performance improved as the number of examples increased. Our analysis further indicated that the models benefit from having more examples during the few-shot learning process. The highest score of 69.25 was achieved with 64 examples, suggesting that providing more examples can offer better guidance and context for the models to understand and effectively perform the task.
However, it is important to note that performance improvement is not strictly linear with the increase in the number of examples. For instance, the score slightly dipped from 67.11 to 66.67 when the number of examples increased from 16 to 32. This
deviation from linearity could be attributed to the quality of the examples or the inherent variability in the models’ performance. Further investigation is required to understand better the factors contributing to these fluctuations and identify the optimal number of examples needed to maximize performance.

heading: 5 Controllability through Prompt
text: In this section, we explore GPT-3’s controllability for GEC tasks through prompt-based methods. Our experiments focus on two settings: (1) comparing the model’s performance when instructed to make minimal edits versus emphasizing fluency, and (2) tailoring the editing to different learner levels, including beginner, intermediate, advanced, and native speakers. We aim to gain insights into GPT-3’s flexibility and controllability under various conditions. We also analyze the relative influence of task instruction and examples to identify the factor that significantly impacts the model’s output.

heading: 5.1 Minimal vs. Fluency Edits

heading: 5.1.1 Settings
text: We evaluated controllability for minimal and fluency edits using the CoNLL2014 and JFLEG test sets, respectively. CoNLL2014 is a widely-used benchmark for GEC tasks, while JFLEG focuses on fluency-based evaluation. We conducted experiments in zero-shot and 16-shot settings. We used different task instructions to control the settings in the prompts, such as ’Revise the following sentence with proper grammar’ for minimal edits and ’Revise the following sentence to improve fluency’ for fluency edits.
We assessed the models using performancebased evaluation and edit distance-based evaluation. Performance-based evaluation measures the model’s error correction or fluency improvement ability, while edit distance-based evaluation quantifies the difference between original and revised sentences, offering insights into the extent of editing performed.

heading: 5.1.2 Results
text: Performance-based Evaluation Figure 3 compares scores in performance-based evaluation for minimal and fluency edit instructions. In the zeroshot setting, minimal edit instructions perform better on the CoNLL2014 test set, while both instructions yield comparable scores on the JFLEG set. In the few-shot setting, higher scores are observed when using corresponding task instructions for each test set, emphasizing the effectiveness of text prompts in controlling editing settings. The discrepancy between zero-shot and few-shot settings might be due to the model’s limited understanding of the task in the zero-shot setting. Additional examples in the few-shot setting enable the model to comprehend the task’s objective better and adjust its output accordingly.
Additionally, we also compared the promptbased method with a supervised controlling method that uses special tokens as in Johnson et al. (2017), where different special tokens were used to control target languages in multilingual translation. We trained a Transformer (Big) encoder-decoder with annotated data tagged with special tokens indicating minimal and fluency edits settings. Despite using more training data, this supervised method failed to control specific settings while achieving higher scores on both test sets with fluency edit
tokens, as in Figure 3. This finding highlights the potential advantages of the prompt-based approach.
Edit Distance-based Evaluation Figure 4 presents edit distance distributions for each setting as part of edit distance-based evaluation. A shift to the right indicates more edits performed with fluency edit instructions. In the few-shot setting, the difference in edit distance distributions between minimal and fluency edits is smaller than in the zero-shot setting, which can be attributed to the influence of the examples presented in the prompt. The model’s ability to generalize from examples in the few-shot setting may diminish the difference in edit distance between the two settings, further emphasizing the importance of carefully selected examples.
In summary, the prompt-based method using GPT-3 can effectively control GEC task outputs for either minimal or fluency edits. Controllability is more evident in few-shot settings, where additional examples help the model adapt its behavior according to the given instructions. The edit distance-based evaluation further supports the model’s ability to adjust its editing behavior based on the prompt, showcasing its potential for practical applications.

heading: 5.2 Learner Level-based Correction

heading: 5.2.1 Settings
text: In this section, we examine GPT-3’s adaptability to diverse GEC task requirements and contexts by analyzing the impact of varying additional information in task instructions. We conducted experiments in both zero-shot and few-shot (16-shot) settings. We utilized the W&I+LOCNESS validation sets, comprising text from various CEFR levels (A: Beginner, B: Intermediate, C: Advanced) and native speakers (N) as evaluation sets. We devised an experiment with three settings based on the following types of additional information (refer to Appendix B):
No Info: No extra information is provided.
Positive Info: Information that supports the input sentence’s characteristics, such as the number of errors to be revised. Example: "Revise mistakes in the following text written by a beginner learner with a lot of mistakes."
Negative Info: Information that contrasts with the input sentence’s characteristics, e.g., a text written by a beginner learner with many errors but described as having few. Example: "Revise mistakes in the following text written by an advanced learner with only a few mistakes."

heading: 5.2.2 Results
text: Figure 5 shows the results of controlling task instruction with additional information on learner levels. In the zero-shot setting, positive information improved performance, while negative information adversely impacted output across most learner levels. This demonstrates the influence of additional
information in task instructions. In the few-shot setting, task instructions without additional information (No Info) achieved comparable scores to cases with Positive Info, suggesting that the model effectively utilizes examples to understand the desired correction level. However, with Negative Info, performance dropped for most learner levels compared to No Info and Positive Info cases.

heading: 5.3 Effect of Task Instruction vs. Examples
text: In this section, we present an experiment to examine the relative effect of task instruction and examples on GPT-3’s performance in controllability, in few-shot settings. Our primary objective is to determine which of these two components, task instruction and example, has a more significant impact on the model’s outputs. Moreover, we extend our investigation to explore the influence of examples on the editing process of the output, providing a more comprehensive understanding of the interplay between these variables in the context of few-shot learning.

heading: 5.3.1 Settings
text: To investigate the relative influence of task instructions and examples independently, we designed two experiments, each featuring distinct conditions:
Varied Task Instruction with Fixed Examples (VIFE) We modified the task instructions while maintaining a constant set of examples. This approach allows us to assess the influence of task instructions on the model’s performance.
Fixed Task Instruction with Varied Examples (FIVE) We utilized a single task instruction and altered the set of examples. This condition helps us evaluate the impact of examples on the model’s performance.
In this experiment, we employed the JFLEG and CoNLL2014 test sets. We assessed the performance using F0.5 score for CoNLL2014 and GLEU for JFLEG. For the VIFE condition, we prepared a fixed set of examples and a varied set of task instructions for each dataset, similar to the approach in Section 5.1. We used task instructions that requested the model to perform minimal edits on the CoNLL2014 test set and fluency edits on the JFLEG test set. For the FIVE condition, we prepared fixed task instructions and varied examples from the training sets of NUCLE and JFLEG, which correspond to minimal and fluency edits,
respectively. We conducted experiments in this section with 16-shot setting.

heading: 5.3.2 Results
text: Figure 6 summarizes the results regarding the performance scores. In both CoNLL2014 and JFLEG, we observed performance gaps between the two settings, minimal and fluency edits. However, the gaps were more drastic when changing the example set compared to varying the task instruction. These results suggest that examples play a more critical role in controlling the model’s behavior than task instructions, as changing the example set leads to more significant differences in achieving the desired output. This is likely because examples provide specific and contextual information, while task instructions can be abstract and open to interpretation. This highlights the importance of carefully selecting examples to optimize model performance.
We further investigated the example set’s impact on model output, using Jensen-Shannon distance to compare edit distance distributions in both minimal
and fluency edits settings. Lower Jensen-Shannon distance indicates a more similar edit distribution between the example set and model output. Results in Table 2 show lower distances in diagonal entries, signifying closer alignment between the model output and corresponding example set. This highlights the importance of carefully selecting examples to guide the model in generating outputs with desired characteristics.

heading: 6 Related Work
text: Supervised learning approaches have predominantly driven GEC research, resulting in state-ofthe-art performance. Encoder-decoder models are commonly employed in GEC using supervised learning. Yuan and Briscoe (2016) first applied an encoder-decoder model to GEC, inspiring subsequent researchers to propose various encoderdecoder-based GEC models (Ji et al., 2017; Chollampatt and Ng, 2018; Junczys-Dowmunt et al., 2018; Zhao et al., 2019; Kaneko et al., 2020; Yamashita et al., 2020). These methods typically rely on large training datasets containing parallel sentences with and without grammatical errors (Kiyono et al., 2019). However, scalability remains challenging, as labeled data is required for each specific situation, such as grammar correction style or input text domain.
Unsupervised GEC approaches aim to reduce dependency on labeled data by leveraging unsupervised learning techniques, including PLMs, handcrafted rules, denoising autoencoders, or unsupervised machine translation (Grundkiewicz et al., 2019; Grundkiewicz and Junczys-Dowmunt, 2019; Flachs et al., 2019; Solyman et al., 2021; Koyama et al., 2021b). However, these methods necessitate creating large-scale pseudo data for model training, making it difficult to generate pseudo-data and train models for different learning scenarios. Some studies have proposed unsupervised GEC methods using PLMs (Alikaniotis and Raheja, 2019; Yasunaga et al., 2021), but they have not focused on prompt-based methods with PLMs.
Recently, the GPT-3 model (Brown et al., 2020) has demonstrated remarkable performance across various NLP tasks, although its GEC performance remains limited. Schick et al. (2022) employed a simple zero-shot prompt for GEC, while DwivediYu et al. (2022) conducted a more comprehensive analysis using diverse zero-shot prompts. Coyne and Sakaguchi (2023) and Fang et al. (2023) com-
pared the latest GPT-3 model’s performance (textdavinci-003) and ChatGPT against GEC leaderboard models and reference edits, finding that these prompt-based methods exhibited strong GEC performance. However, automatic metrics and human evaluations occasionally disagreed on the relative quality of corrections.
Controlling GEC model generation is crucial but remains underexplored. Hotate et al. (2019) proposed a GEC method that controls the degree of correction by tagging input with the correction level, but it requires supervised learning with parallel data. Additionally, Hotate et al. (2020) suggested a beam search method to control GEC correction diversity by dynamically updating search tokens within the beam based on the likelihood of predicting source sentence tokens. While this method enables model control without additional training, it falls short in accommodating specific learner requests, such as minimal and fluency edits.
GEC model evaluation methods have been proposed based on learner levels and correction styles. To account for differences in correction styles and domains, Maeda et al. (2022) introduced a method to train evaluation models using only parallel data. Takahashi et al. (2022) created proficiencyannotated data to train evaluation models and developed an evaluation method that considers proficiency by fine-tuning PLMs (Yoshimura et al., 2020).

heading: 7 Conclusion
text: In conclusion, this study demonstrates the potential of using prompt-based methods with GPT-3 for GEC tasks, achieving competitive performance compared to traditional supervised and unsupervised methods. By carefully crafting task instructions and examples, we show that GPT-3 can be effectively controlled to focus on different aspects of the GEC process and adapt to diverse learning needs. Our findings highlight the importance of optimizing task instructions and example selection to enhance the performance and controllability of GPT-3, paving the way for further research on refining prompt engineering techniques and exploring their applicability to other NLP tasks and language models.

heading: 8 Educational Implications and Community Benefits
text: Our study provides valuable implications for education. The controllability of large-scale language models in GEC tasks can be leveraged to design personalized language instruction. It allows educators to provide feedback that matches individual students’ proficiency levels and focuses on specific areas for improvement. For learners, instant, tailored feedback can enhance their language learning process. Moreover, our findings can improve intelligent tutoring systems, making them more responsive to individual needs. Beyond education, our research can enhance language-based interfaces and AI communication systems, offering more accurate and context-specific language corrections. This study lays the groundwork for future exploration into how large language models can improve language education and literacy.

heading: 9 Limitation
text: While our study provides valuable insights into the use of prompt-based methods with GPT-3 for GEC tasks and its controllability, several limitations should be acknowledged.
Focus on GPT-3: This study exclusively examines GPT-3 as the language model for GEC tasks. While GPT-3 has shown remarkable performance in various NLP tasks, other pre-trained language models, such as GPT-4, may offer different results. A broader investigation that includes other language models would provide a more comprehensive understanding of the applicability of promptbased methods in GEC tasks.
Limited evaluation metrics: The evaluation of GPT-3’s performance and controllability in our experiments mainly relies on quantitative metrics, such as edit distance and task scores. These metrics may not fully capture the nuances of grammatical error correction or the model’s ability to adapt to different learning scenarios. Additional qualitative analysis, along with more diverse evaluation metrics, could provide a richer understanding of the model’s performance and controllability.
Variability in examples: While our study highlights the importance of example selection in fewshot settings, we do not thoroughly explore the impact of example quality or diversity. The effect
of using different types of examples or a more diverse set of examples remains to be investigated, which could further inform the design of effective example sets for prompt-based GEC tasks. By addressing these limitations in future research, we can further advance our understanding of the performance and controllability of prompt-based methods with GPT-3 and other language models in GEC tasks and beyond.
Potential fine-tuning on test data: There is a possibility that GPT-3 has been fine-tuned (instruction tuning) on the test data we are using, which might explain the higher evaluation scores compared to previous research. As this information has not been disclosed, we are unable to verify it at this time. This point should be taken into consideration when interpreting our results.

heading: Acknowledgements
text: These research results were obtained partially from the commissioned research (No. 225) by National Institute of Information and Communications Technology (NICT), Japan.

heading: A Prompts for Investigation on Instruction Effect
text: All instructions used for experiments described in Section 4.1 are listed in Table 3.

heading: B Prompts for Learner’s Level-based Control
text: All instructions and additional information used for experiments described in Section 5.2 are listed in Table 4.

heading: Future Work
text: By addressing these limitations in future research, we can further advance our understanding of the performance and controllability of prompt-based methods with GPT-3 and other language models in GEC tasks and beyond. Potential fine-tuning on test data: There is a possibility that GPT-3 has been fine-tuned (instruction tuning) on the test data we are using, which might explain the higher evaluation scores compared to previous research. As this information has not been disclosed, we are unable to verify it at this time. This point should be taken into consideration when interpreting our results