Mixture of Small and Large Models for Chinese Spelling Check
Authors: Ziheng Qiao, Houquan Zhou, Zhenghua Li
Accepted at: acl25

Mixture of Small and Large Models for Chinese Spelling Check

Ziheng Qiao, Houquan Zhou, Zhenghua Li*
School of Computer Science and Technology, Soochow University, China
{zhqiao,hqzhou}@stu.suda.edu.cn, zhli13@suda.edu.cn

Abstract
In the era of large language models (LLMs), the Chinese Spelling Check (CSC) task has seen various LLM methods developed, yet their performance remains unsatisfactory. In contrast, fine-tuned BERT-based models, relying on high-quality in-domain data, show excellent performance but suffer from edit pattern overfitting. This paper proposes a novel dynamic mixture approach that effectively combines the probability distributions of small models and LLMs during the beam search decoding phase, achieving a balanced enhancement of precise corrections from small models and the fluency of LLMs. This approach also eliminates the need for fine-tuning LLMs, saving significant time and resources, and facilitating domain adaptation. Comprehensive experiments demonstrate that our mixture approach significantly boosts error correction capabilities, achieving state-of-the-art results across multiple datasets. Our code is available at https://github.com/zhqiao-nlp/MSLLM.

1 Introduction
The Chinese Spelling Check (CSC) task focuses on identifying and correcting spelling errors in given sentences, as demonstrated in Figure 2. Such errors may lead to comprehension difficulties and adversely affect various natural language processing applications, including machine translation and information retrieval. Given its practical significance, CSC has gained substantial research attention in recent years.

In the era of pre-trained language models, BERT-based approaches (Devlin et al., 2019) have emerged as the dominant solution for the CSC task (Zhang et al., 2020; Cheng et al., 2020; Li et al., 2022). Based on the characteristic where the input and output are of equal length for the CSC task, they effectively treat CSC as a character-level classification problem. That is, for each character in the input sentence, they predict whether it needs to be corrected. During the fine-tuning process using in-domain data, either artificial or real-world, these BERT-based models can adeptly capture intricate relationships between edit pairs. However, this process sometimes leads to overfitting specific edit pairs and generating erroneous sentences.

On the other hand, the primary objective of the CSC task is to generate fluent and accurate sentences. As a result, generative models are particularly well-suited for this task. With the advent of large language models (LLMs), which boast extensive parameter sizes and vast training datasets, these models exhibit strong cross-domain generalization abilities. Researchers have explored various strategies for utilizing LLMs in the CSC task, focusing on whether to fine-tune these models (Li et al., 2023; Dong et al., 2024). Nevertheless, LLMs often over-polish the text for fluency, choosing expressions they consider better, which leads to inconsistencies between the prediction and the input lengths. Li et al. (2024) attempted to address this issue by employing a character-level tokenization

![Overview of our approach. The correct sentence is “开车忘带驾驶证，被查到不要慌” (If you forgot to bring the driver’s license while driving, don’t panic when being checked).](page_1012_370_388_312.png)

Figure 1: Overview of our approach. The correct sentence is “开车忘带驾驶证，被查到不要慌” (If you forgot to bring the driver’s license while driving, don’t panic when being checked).
and supervised fine-tuning (SFT) technique. However, this approach requires significant time and resources.

Unlike previous LLM strategies, Zhou et al. (2024) fully utilize the language modeling capabilities of LLMs. They treat open-source LLMs as pure language models and manually design a distortion model to ensure faithfulness between inputs and outputs by leveraging phonetic and glyph similarities. This approach is particularly effective in zero-shot scenarios.

In general, compared to fine-tuned small models, the correction performance of LLM approaches remains unsatisfactory. Liu et al. (2024a) tried using an unfine-tuned small model as an arbiter to choose between predictions from both small and large models, yet only achieved minimal improvements. Indeed, due to their different inference mechanisms, BERT-based models and LLMs inherently excel in different aspects of error correction: precision and domain adaptability for BERT-based models, and fluency for LLMs. We believe that a deeper integration of these two models at the inference stage could be a more effective strategy.

Motivated by these insights, this paper presents a novel dynamic mixture approach that strategically integrates a BERT-based model with an LLM. Specifically, we incorporate the probability distribution from the BERT-based model into the LLM’s beam search process, thereby preserving the correction capabilities of both models while mitigating the small model’s overfitting tendencies through the LLM’s robust language modeling. Furthermore, by fine-tuning the small model instead of the LLM, we significantly cut down on the resources and time needed for domain adaptation.

Our contributions are summarized as follows:

• We propose a novel and straightforward approach that combines a BERT-based model with an LLM, leveraging their complementary strengths to further enhance error correction performance.

• Our approach does not require fine-tuning the LLM, significantly reducing time and resource costs while preserving its strong generalization capability.

• Extensive experiments on multiple mainstream public benchmarks show that our mixture approach substantially boosts correction performance, achieving SOTA results on several datasets.

<table>
  <tr>
    <th>Type</th>
    <th>Probability</th>
  </tr>
  <tr>
    <td>Identical</td>
    <td>0.962</td>
  </tr>
  <tr>
    <td>Same Pinyin</td>
    <td>0.023</td>
  </tr>
  <tr>
    <td>Similar Pinyin</td>
    <td>0.008</td>
  </tr>
  <tr>
    <td>Similar Shape</td>
    <td>0.004</td>
  </tr>
  <tr>
    <td>Unrelated</td>
    <td>0.003</td>
  </tr>
</table>

Table 1: The distribution of the different distortion types extracted from Zhou et al. (2024).

2 The Basic Approaches

Given an input sentence comprising \( n \) characters, denoted as \( x = x_1 \cdots x_i \cdots x_n \), the objective of a CSC model is to generate a corresponding corrected sentence, represented as \( y = y_1 \cdots y_i \cdots y_n \), in which all erroneous characters in \( x \) are replaced with the correct ones. In other words, CSC models aim to find an optimal sentence \( y \) that maximizes \( \operatorname{score}(x, y) \). Currently, there exist two representative CSC models, i.e., the generative LLM-based models, and the classification-based models.

2.1 The LLM-based Approach

Recently, Zhou et al. (2024) proposed a novel prompt-free training-free LLM-based approach for the CSC task. The key is treating LLM as a pure language model. They designed a distortion model (DM) to model the relationships between \( x \) and \( y \), and more precisely to ensure \( y \) is faithful to \( x \).

\[
\operatorname{score}(x, y) = \log p_{\text{LLM}}(y) + \log p_{\text{DM}}(x \mid y)
\]

\[
p_{\text{LLM}}(y) = \prod_{i=1}^n p_{\text{LLM}}(y_i \mid y_{<i})
\]

\[
p_{\text{DM}}(x \mid y) = \prod_{i=1}^n p_{\text{DM}}(x_i \mid y_i)
\]

\[
p_{\text{DM}}(x_i \mid y_i) = p(\operatorname{type}(x_i, y_i))
\]

The LLM component generates a sentence in an auto-regressive manner, and gives us the probability, i.e., \( p_{\text{LLM}}(\cdot) \).

The DM component first classifies each character pair (e.g., \((c_1, c_2)\)), into five types, and then obtains the pre-defined corresponding probability, as shown in Table 1.

2.2 The Classification Approach

In the pre-trained model era, most mainstream CSC models follow a BERT-based approach (Zhang et al., 2020; Xu et al., 2021; Li et al., 2022; Liu et al., 2024b). These models treat CSC as a local
classification problem, i.e., for each character, they determine whether it needs to be corrected and, if so, which character it should be modified to:

\[
p_{SM}(y \mid x, i) = \operatorname{softmax}(\text{MLP}(h_i))[y]
\]

where \( h_i \) represents the contextual representation of the \( i \)-th character obtained from the BERT-based encoder, and SM is the abbreviation for the small classification model. By selecting the character with the highest probability at each position, i.e., \( y^* = \arg\max_{y \in \mathcal{V}} p(y \mid x, i) \), where \( \mathcal{V} \) represents the vocabulary, we can obtain the final correction result \( y \). This classification-based approach demonstrates strong fitting capabilities in specific domains with high-quality training datasets, and it also offers fast decoding speed. However, it lacks a global score\((x; y)\), which results in the model tending to memorize specific edit pairs, leading to locally optimal solutions.

3 Our Mixture Approach

In this work, we propose a straightforward mixture approach to integrate the power of both small and large models. On the one hand, the training-free LLM-based approach of Zhou et al. (2024) exhibits remarkable ability in domain generalization. On the other hand, the small classification models can be effectively trained on in-domain labeled data, if available, and thus can dramatically improve in-domain performance.

\[
\begin{align*}
\operatorname{score}(x, y) &= \log p_{LLM}(y) + \log p_{DM}(x \mid y) \\
&\quad + \log p_{SM}(y \mid x) \\
p_{SM}(y \mid x) &= \prod_{i=1}^{n} p_{SM}(y_i \mid x, i)
\end{align*}
\]

3.1 Incremental Decomposition

In the inference phase, our model follows the LLM component, and produces \( y \) from left to right in an auto-regressive manner. Thus, we give an incremental decomposition of a partial output sentence as follows:

\[
\begin{align*}
\operatorname{score}(x, y_{\leq i}) &= \operatorname{score}(x, y_{<i}) \\
&\quad + \log p_{LLM}(y_i \mid y_{<i}) \\
&\quad + \log p_{DM}(x_i \mid y_i) \\
&\quad + \log p_{SM}(y_i \mid x, i)
\end{align*}
\]

3.2 Token-based Generation and Beam Search Decoding

Current LLMs usually generate a sentence token by token, i.e., using tokens as the basic units. Therefore, the output sentence can also be denoted as \( y = t_1 \cdots t_k \cdots t_o \), where a token is composed of \( \ell \geq 1 \) characters, i.e., \( t_k = y_{i-\ell+1} \cdots y_i \).

Moreover, we follow Zhou et al. (2024) and employ their proposed faithfulness reward to further encourage that \( y \) retains the same meaning as \( x \).

The full model. Combining the above two factors, we give the incremental decomposition of our full model.

\[
\begin{align*}
\operatorname{score}(x, y_{\leq i} = t_{\leq k}) &= \operatorname{score}(x, t_{<k}) \\
&\quad + \log p_{LLM}(t_k \mid t_{<k}) \\
&\quad + (1 + H_{LLM}(\cdot)) \times \left( \frac{\alpha \times \log p_{DM}(x, i \mid t_k)}{} + \frac{\beta \times \log p_{SM}(t_k \mid x, i)}{} \right)
\end{align*}
\]

where \( \alpha \) and \( \beta \) are the weights of the distortion component in the generative LLM-based model and the small BERT-based model, respectively; \( H_{LLM}(\cdot) \) corresponds to the faithfulness reward of Zhou et al. (2024), and represents the entropy of \( p_{LLM}(t_k \mid t_{<k}) \), i.e., the probability distribution of the LLM component regarding the generation of \( t_k \). Higher entropy means the LLM is more uncertain about the selection of the next token, and thus the other two components obtain higher weights.

The token-level formulations for the DM component and the small model are defined as follows:

\[
\begin{align*}
p_{DM}(x, i \mid t_k) &= \prod_{j=1}^{\ell} p_{DM}(x_{i-\ell+j} \mid t_k[j-1]) \\
p_{SM}(t_k \mid x, i) &= \prod_{j=1}^{\ell} p_{SM}(t_k[j-1] \mid x, i - \ell + j)
\end{align*}
\]

For multi-word tokens, the value of \( p_{DM}(\cdot) \) is computed as the product of the probabilities of all characters within the token. In the method of Zhou et al. (2024), the weight \( \alpha \) of the distortion component is set to 1. The computation of \( p_{SM}(\cdot) \) involves segmenting a LLM token into multiple small-model tokens and using the probability distribution from the small model to derive the score of the corresponding multi-word token, thereby achieving token-level alignment.

Beam search. During inference, we follow Zhou et al. (2024) and employ beam search, in order to explore a larger search space. At each decoding step, we retain the top \( K \) candidates with the highest scores, and based on them build candidates for the next step.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">rSIGHANs</th>
    <th colspan="3">CSCD-NS</th>
    <th colspan="3">MCSCSet</th>
    <th colspan="3">ECSpell</th>
    <th colspan="3">LEMON</th>
  </tr>
  <tr>
    <th>S-F<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
    <th>FPR</th>
    <th>S-F<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
    <th>FPR</th>
    <th>S-F<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
    <th>FPR</th>
    <th>S-F<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
    <th>FPR</th>
    <th>S-F<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
    <th>FPR</th>
  </tr>
  <tr>
    <td colspan="15"><b>Previous SOTAs</b></td>
  </tr>
  <tr>
    <td>BERT<sup>†</sup></td>
    <td>70.8</td>
    <td>81.9</td>
    <td>12.1</td>
    <td>77.4</td>
    <td>79.3</td>
    <td>10.9</td>
    <td><b>87.6</b></td>
    <td><b>94.0</b></td>
    <td>1.8</td>
    <td>91.0</td>
    <td>94.6</td>
    <td><b>3.8</b></td>
    <td>48.1</td>
    <td>49.3</td>
    <td>13.1</td>
  </tr>
  <tr>
    <td>ReLM<sup>†</sup></td>
    <td>71.8</td>
    <td>81.8</td>
    <td>11.5</td>
    <td>77.3</td>
    <td>79.7</td>
    <td>10.5</td>
    <td>87.0</td>
    <td>93.7</td>
    <td>1.6</td>
    <td><b>92.3</b></td>
    <td><b>94.9</b></td>
    <td>7.4</td>
    <td><b>50.6</b></td>
    <td><b>51.6</b></td>
    <td><b>11.7</b></td>
  </tr>
  <tr>
    <td colspan="15"><b>LLMs (Zhou et al., 2024)</b></td>
  </tr>
  <tr>
    <td>Baichuan2</td>
    <td><b>59.1</b></td>
    <td><b>70.1</b></td>
    <td>15.4</td>
    <td><b>62.7</b></td>
    <td><b>65.7</b></td>
    <td><b>16.8</b></td>
    <td><b>67.2</b></td>
    <td><b>78.0</b></td>
    <td>2.0</td>
    <td><b>85.4</b></td>
    <td><b>90.2</b></td>
    <td><b>5.1</b></td>
    <td><b>53.2</b></td>
    <td><b>56.7</b></td>
    <td><b>9.9</b></td>
  </tr>
  <tr>
    <td>Qwen2.5</td>
    <td>55.6</td>
    <td>68.6</td>
    <td>17.9</td>
    <td>58.6</td>
    <td>62.6</td>
    <td>23.5</td>
    <td>63.3</td>
    <td>74.1</td>
    <td>3.2</td>
    <td>81.3</td>
    <td>88.1</td>
    <td>6.6</td>
    <td>48.6</td>
    <td>53.5</td>
    <td>12.8</td>
  </tr>
  <tr>
    <td>IL2.5</td>
    <td>52.5</td>
    <td>66.2</td>
    <td>20.1</td>
    <td>55.3</td>
    <td>59.1</td>
    <td>26.5</td>
    <td>51.9</td>
    <td>62.8</td>
    <td>5.3</td>
    <td>80.3</td>
    <td>87.0</td>
    <td>6.9</td>
    <td>45.3</td>
    <td>50.0</td>
    <td>15.6</td>
  </tr>
  <tr>
    <td colspan="15"><b>Ours</b></td>
  </tr>
  <tr>
    <td>BERT + BC2</td>
    <td>72.5</td>
    <td>83.4</td>
    <td>7.4</td>
    <td>78.1</td>
    <td>79.1</td>
    <td>10.1</td>
    <td>91.2</td>
    <td>96.0</td>
    <td>1.4</td>
    <td>94.4</td>
    <td>96.3</td>
    <td><b>1.7</b></td>
    <td>56.4</td>
    <td>58.2</td>
    <td>6.9</td>
  </tr>
  <tr>
    <td>BERT + QW2.5</td>
    <td>73.6</td>
    <td><b>83.9</b></td>
    <td><b>7.0</b></td>
    <td>76.8</td>
    <td>78.6</td>
    <td>11.3</td>
    <td>91.6</td>
    <td>95.9</td>
    <td>1.6</td>
    <td>94.0</td>
    <td>95.5</td>
    <td>2.7</td>
    <td>56.1</td>
    <td>58.7</td>
    <td>10.1</td>
  </tr>
  <tr>
    <td>BERT + IL2.5</td>
    <td>72.8</td>
    <td>83.8</td>
    <td>7.9</td>
    <td>77.3</td>
    <td>78.3</td>
    <td>11.5</td>
    <td>90.7</td>
    <td>95.7</td>
    <td>1.7</td>
    <td>94.3</td>
    <td>94.9</td>
    <td>2.2</td>
    <td>53.8</td>
    <td>55.1</td>
    <td>6.7</td>
  </tr>
  <tr>
    <td>ReLM + BC2<sup>†</sup></td>
    <td>73.9</td>
    <td>83.0</td>
    <td>9.2</td>
    <td><b>79.9</b></td>
    <td><b>81.8</b></td>
    <td><b>8.4</b></td>
    <td><b>91.7</b></td>
    <td><b>96.2</b></td>
    <td><b>1.1</b></td>
    <td><b>97.1</b></td>
    <td><b>98.3</b></td>
    <td><b>2.1</b></td>
    <td><b>61.0</b></td>
    <td><b>61.6</b></td>
    <td><b>5.6</b></td>
  </tr>
  <tr>
    <td>ReLM + QW2.5</td>
    <td>73.9</td>
    <td>82.6</td>
    <td>9.1</td>
    <td>78.9</td>
    <td>79.8</td>
    <td>10.0</td>
    <td><b>92.1</b></td>
    <td><b>96.3</b></td>
    <td>1.3</td>
    <td>96.6</td>
    <td>97.8</td>
    <td>3.0</td>
    <td>60.5</td>
    <td>61.4</td>
    <td>7.9</td>
  </tr>
  <tr>
    <td>ReLM + IL2.5</td>
    <td><b>74.4</b></td>
    <td>83.6</td>
    <td>9.5</td>
    <td>79.3</td>
    <td>81.0</td>
    <td>10.1</td>
    <td>90.4</td>
    <td>95.4</td>
    <td>1.5</td>
    <td>96.7</td>
    <td>98.0</td>
    <td>2.6</td>
    <td>58.9</td>
    <td>60.2</td>
    <td>7.9</td>
  </tr>
</table>

Table 2: Sentence- and character-level results on the mainstream CSC test sets. \( F_1 \) and FPR scores are reported (%). BC2 stands for Baichuan2, QW2.5 for Qwen2.5, and IL2.5 for InternLM2.5. All LLMs use the 7B model size version. Note that the performance metrics for rSIGHANs, ECSpell, and LEMON are presented as macro averages. “†” indicates that the models are pre-trained on 34 million synthetic data and fine-tuned with MFT strategy (LEMON cannot be further fine-tuned due to the lack of in-domain training data).

4 Experimental Setup

4.1 Datasets

Chinese Learner Texts. Following the conventions of previous works, we employ the SIGHAN datasets (Wu et al., 2013; Yu et al., 2014; Tseng et al., 2015) as our benchmarks, which are derived from Chinese learner texts. We utilize a revised version of SIGHAN (Yang et al., 2023b), which was manually annotated for errors and noise present in the original SIGHANs. We refer to this version as rSIGHANs. In the training stage, we use Wang271K (Wang et al., 2018) + SIGHANs as our training set.

Chinese Native-speaker Texts. Due to the lack of domain diversity in the SIGHAN datasets, we further conduct experiments on more diverse datasets, including LEMON (Wu et al., 2023), ECSpell (Lv et al., 2023), CSCD-NS (Hu et al., 2024), and MCSCSet (Jiang et al., 2022), all of which were written by native speakers. Notably, LEMON includes test sets from seven different domains but does not provide in-domain training sets, making it an ideal benchmark for evaluating a model’s cross-domain generalization capability. Detailed information about all datasets can be found in Appendix A.

4.2 Baseline Models
We select several BERT-based models and LLMs as our baselines.

Small BERT-based Models. For benchmarks across various domains, we select BERT (Devlin et al., 2019) and ReLM (Liu et al., 2024b) for experiments. Additionally, on the rSIGHAN15 test set, we select several mainstream SOTA models for experiments and report the results, such as ReLiSe (Xu et al., 2021) and SCOPE (Li et al., 2022). Detailed information about all baselines can be found in Appendix B.

Open-source LLMs. We use Baichuan2 (Yang et al., 2023a), Qwen2.5 (Bai et al., 2023), and InternLM2.5 (Cai et al., 2024) as our open-source LLMs for experiments. All LLMs use the base version. In the main experiments, we fix the model size to 7B. For comprehensive and robust experiments, in Section 6.1, we select LLMs with different sizes ranging from 0.5B to 20B.

Mixture Strategy. ARM (Liu et al., 2024a) attempts to make trade-offs between the correction results of small models and GPT-3.5-Turbo.1 Due to the lack of open-source code and differences in experimental settings on LEMON, we only list their results in Table 10 and Table 13.

1 https://platform.openai.com
Input: 水饺和新鲜的空气都很重要。
Dumplings and fresh air are both important.

Reference: 水饺 → 睡觉 (shuìjiào → shuìjiào, sleep)
ReLM: 水饺 → 水饺 (shuìjiào → shuìjiào, water sleep)
LLM: NONE
ReLM+LLM: 水饺 → 睡觉 (shuìjiào → shuìjiào, sleep)

(a) Correct errors from ReLM and LLM

Input: 开车忘带驾驶证，被查到不要慌，...
If you forgot to bring the driver’s license while driving, don’t panic when being checked...

Reference: 驾驶证 → 驾驶证 (zhèng → zhèng, driver’s license)
ReLM: 驾驶者 → 驾驶证 (zhèng → zhèng, driver’s license)
LLM: NONE
ReLM+LLM: 驾驶者 → 驾驶证 (zhèng → zhèng, driver’s license)

(b) Correct errors from LLM

Input: 不仅缺账，还提无力要求！
Not only do they refuse to pay, but they also make powerless demands!

Reference: 无力 → 无理 (wúlǐ → wúlǐ, unreasonable)
ReLM: 提 → 是 (tī → shì, are)
LLM: 无力 → 无理 (wúlǐ → wúlǐ, unreasonable)
ReLM+LLM: 无力 → 无理 (wúlǐ → wúlǐ, unreasonable)

(c) Correct errors from ReLM

Figure 2: Cases from rSIGHAN15 and LEMON-New test sets. The LLM used is BaiChuan2(7B).

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th rowspan="2">Size</th>
    <th colspan="2">rSIGHAN15</th>
    <th colspan="2">ECSpell-Odw</th>
    <th colspan="2">LEMON-Nov</th>
  </tr>
  <tr>
    <th>S-F1</th>
    <th>C-F1</th>
    <th>S-F1</th>
    <th>C-F1</th>
    <th>S-F1</th>
    <th>C-F1</th>
  </tr>
  <tr>
    <td>BC2</td>
    <td>7B</td>
    <td>79.6</td>
    <td>85.4</td>
    <td>96.5</td>
    <td>98.2</td>
    <td>50.0</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td></td>
    <td>13B</td>
    <td>77.9</td>
    <td>84.6</td>
    <td>96.7</td>
    <td>98.3</td>
    <td>50.8</td>
    <td>50.9</td>
  </tr>
  <tr>
    <td></td>
    <td>0.5B</td>
    <td>78.4</td>
    <td>84.2</td>
    <td>95.7</td>
    <td>97.3</td>
    <td>45.4</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td>QW2.5</td>
    <td>3B</td>
    <td>78.2</td>
    <td>83.3</td>
    <td>95.7</td>
    <td>97.6</td>
    <td>48.1</td>
    <td>49.0</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>79.3</td>
    <td>83.7</td>
    <td>96.7</td>
    <td>98.1</td>
    <td>48.8</td>
    <td>49.9</td>
  </tr>
  <tr>
    <td></td>
    <td>14B</td>
    <td>77.9</td>
    <td>84.2</td>
    <td>95.9</td>
    <td>97.7</td>
    <td>49.4</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td></td>
    <td>1.8B</td>
    <td>79.0</td>
    <td>84.5</td>
    <td>95.5</td>
    <td>97.6</td>
    <td>45.1</td>
    <td>46.7</td>
  </tr>
  <tr>
    <td>IL2.5</td>
    <td>7B</td>
    <td>79.5</td>
    <td>85.3</td>
    <td>96.9</td>
    <td>98.3</td>
    <td>47.7</td>
    <td>48.8</td>
  </tr>
  <tr>
    <td></td>
    <td>20B</td>
    <td>76.8</td>
    <td>83.6</td>
    <td>97.1</td>
    <td>98.3</td>
    <td>48.1</td>
    <td>48.6</td>
  </tr>
</table>

Table 3: Sentence- and character-level \( F_1 \) scores of different model families and sizes. All LLMs are combined with ReLM.

5 Main Results

Table 2 presents experimental results across five benchmark datasets: rSIGHANs, CSCD-NS, MC-SCSet, ECSpell, and LEMON. When integrated with LLMs, nearly all models exhibit substantial enhancements across sentence- and character-level correction metrics (Precision, Recall, and \( F_1 \)). We also provide the results of the original SIGHAN15 dataset in Appendix C.1.

To evaluate cross-domain generalization capability, we specifically test on ECSpell and LEMON datasets. Our approach demonstrates consistent performance gains on both test sets, with detailed sub-domain results provided in Appendix C.2. Notably, ReLM integrated with BaiChuan2 achieves remarkable improvements: a 10.4% average increase in sentence-level \( F_1 \) across all seven LEMON domains, and 4.8% enhancement on ECSpell. During our experiments, we discovered that ECSpell has a serious issue of target sentence leakage, leading to overly high performance of fine-tuned small models. We will explain this issue and present the results on the cleaned ECSpell dataset in Appendix C.3.

Due to different experimental setups, we also list another mixture strategy, ARM (Liu et al., 2024a), in Appendix C. In comparison, our method achieves greater improvements across all datasets.

4.3 Evaluation Metrics

Following the mainstream evaluation metrics for CSC tasks, we report the Precision (P), Recall (R), and \( F_1 \) scores of the correction subtask at both the sentence- and character-level, denoted as S-P/R/F and C-P/R/F, respectively. To comprehensively evaluate the model’s correction capability, we also include the false positive rate (FPR) as an additional metric.

4.4 Implementation Details

For ReaLiSe, we employ the pre-trained checkpoint available from its official GitHub repository.2 For SCOPE, we adopt their official implementation for fine-tuning.3 Both BERT and ReLM utilize the framework from Liu et al. (2024b), with their pre-trained models (trained on a corpus of 34 million synthetic sentences) being fine-tuned in our experiments.4 During training, we set the batch size to 128 and the learning rate to \( 3 \times 10^{-5} \), while incorporating the MFT strategy (Wu et al., 2023). All experiments are conducted on an NVIDIA A100-PCIE-40GB GPU.

2https://github.com/DaDaMrX/ReaLiSe
3https://github.com/jiahaozhenbang/SCOPE
4https://github.com/gingasan/lemon
Figure 3: Model performance (sentence-level \( F_1 \)) of ReLM + LLMs on rSIGHAN15, ECSpell, and LEMON with different \( \alpha \) and \( \beta \). The x-axis is \( \alpha \) and y-axis is \( \beta \). The red cells \( \blacksquare \) denote superior performance of ReLM + LLM compared to ReLM. The blue cells \( \blacktriangle \) represent inferior performance. The darker the color, the larger the performance gap.

<table>
  <tr>
    <th>Model</th>
    <th>S-P*</th>
    <th>S-R*</th>
    <th>S-F*</th>
    <th>C-P*</th>
    <th>C-R*</th>
    <th>C-F*</th>
    <th>FPR</th>
  </tr>
  <tr>
    <th colspan="8">rSIGHAN15</th>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>76.8</td>
    <td>73.8</td>
    <td>75.2</td>
    <td>87.9</td>
    <td>78.0</td>
    <td>82.6</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>67.1</td>
    <td>55.8</td>
    <td>61.0</td>
    <td>78.7</td>
    <td>61.0</td>
    <td>68.7</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>83.5</b></td>
    <td><b>76.1</b></td>
    <td><b>79.6</b></td>
    <td><b>93.8</b></td>
    <td><b>78.3</b></td>
    <td><b>85.4</b></td>
    <td><b>4.1</b></td>
  </tr>
  <tr>
    <td>- DM</td>
    <td>-4.3</td>
    <td>-0.2</td>
    <td>-2.1</td>
    <td>-4.2</td>
    <td>+1.3</td>
    <td>-1.1</td>
    <td>+3.4</td>
  </tr>
  <tr>
    <td>- FR</td>
    <td>-3.6</td>
    <td>-0.9</td>
    <td>-2.1</td>
    <td>-3.1</td>
    <td>+0.3</td>
    <td>-1.2</td>
    <td>+2.3</td>
  </tr>
  <tr>
    <td>- both</td>
    <td>-4.7</td>
    <td>-0.5</td>
    <td>-2.5</td>
    <td>-4.6</td>
    <td>+1.1</td>
    <td>-1.4</td>
    <td>+3.8</td>
  </tr>
  <tr>
    <th colspan="8">ECSpell-Odw</th>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>89.4</td>
    <td>91.5</td>
    <td>90.4</td>
    <td>93.1</td>
    <td>95.7</td>
    <td>94.4</td>
    <td>5.8</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>92.0</td>
    <td>89.1</td>
    <td>90.5</td>
    <td>95.0</td>
    <td>92.6</td>
    <td>93.8</td>
    <td>1.6</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>97.2</td>
    <td>95.7</td>
    <td>96.5</td>
    <td><b>98.7</b></td>
    <td><b>97.7</b></td>
    <td><b>98.2</b></td>
    <td><b>0.8</b></td>
  </tr>
  <tr>
    <td>- DM</td>
    <td>-5.2</td>
    <td>-1.5</td>
    <td>-3.4</td>
    <td>-3.2</td>
    <td>-0.8</td>
    <td>-2.0</td>
    <td>+3.3</td>
  </tr>
  <tr>
    <td>- FR</td>
    <td><b>+0.4</b></td>
    <td><b>+0.4</b></td>
    <td><b>+0.4</b></td>
    <td>-0.2</td>
    <td><b>+0.3</b></td>
    <td><b>+0.0</b></td>
    <td><b>+0.0</b></td>
  </tr>
  <tr>
    <td>- both</td>
    <td>-4.4</td>
    <td>-0.8</td>
    <td>-2.7</td>
    <td>-3.0</td>
    <td>-0.5</td>
    <td>-1.7</td>
    <td>+3.3</td>
  </tr>
  <tr>
    <th colspan="8">Lemon-Nov</th>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>46.3</td>
    <td>32.2</td>
    <td>38.0</td>
    <td>48.9</td>
    <td>31.0</td>
    <td>37.9</td>
    <td>17.6</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>49.8</td>
    <td>37.0</td>
    <td>42.4</td>
    <td>53.5</td>
    <td>42.0</td>
    <td>47.1</td>
    <td>14.9</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>64.0</b></td>
    <td><b>41.0</b></td>
    <td><b>50.0</b></td>
    <td><b>67.2</b></td>
    <td><b>40.3</b></td>
    <td><b>50.4</b></td>
    <td><b>9.8</b></td>
  </tr>
  <tr>
    <td>- DM</td>
    <td>-15.8</td>
    <td>-4.2</td>
    <td>-8.2</td>
    <td>-16.1</td>
    <td>-4.3</td>
    <td>-8.1</td>
    <td>+7.8</td>
  </tr>
  <tr>
    <td>- FR</td>
    <td>-6.2</td>
    <td>+3.1</td>
    <td>+0.0</td>
    <td>-6.2</td>
    <td><b>+4.5</b></td>
    <td><b>+1.3</b></td>
    <td>+4.3</td>
  </tr>
  <tr>
    <td>- both</td>
    <td>-16.4</td>
    <td>-3.0</td>
    <td>-7.8</td>
    <td>-16.5</td>
    <td>-2.6</td>
    <td>-7.1</td>
    <td>+8.6</td>
  </tr>
</table>

Table 4: Ablation results of distortion model (DM) and faithfulness reward (FR) on ReLM + Baichuan2(7B). “ReLM” and “BC2” represent using ReLM and Baichuan2(7B) model alone respectively. “-both” represents that we remove the intervention of both DM and FR on the results.

model in Eq. (1), which allows the language model to play a greater role. Figure 2(b) illustrates a typical under-correction scenario. Here, the LLM’s under-correction of “驾驶者” (driver) is successfully revised to “驾驶证” (driver’s license) through ReLM’s complementary intervention. The third case in Figure 2(c) reveals an over-correction scenario where ReLM introduces an erroneous modification, which is effectively identified and corrected by the LLM through beam search decoding.

In summary, LLMs demonstrate superior fluency preservation, while small models are better at making accurate corrections. The synergistic integration of both capabilities through our framework yields optimal correction results.

6 Discussion

Following Zhou et al. (2024), we evaluate our approach on three benchmark datasets: rSIGHAN15, ECSpell-Odw, and LEMON-Nov.

6.1 Impact of Different LLM Families and Model Sizes

To systematically analyze how LLM families and model sizes affect correction performance, we conduct comprehensive experiments with multiple open-source LLMs scaled from 0.5B to 20B parameters, including Baichuan2 (7B-13B), Qwen2.5 (0.5B-14B), and InternLM2.5 (1.8B-20B). All models are integrated with ReLM and evaluated across all three datasets. Table 3 reveals two key findings:
• Within the same LLM family, larger models do not necessarily lead to better performance. For example, on rSIGHAN15, the correction performance of all LLMs does not show an increasing trend with the growth of model size.
• Different LLM families exhibit varying strengths across different domains. For example, InternLM2.5 outperforms Qwen2.5 on the rSIGHAN15 dataset, whereas the opposite is true for LEMON-Nov. Overall, the combination of ReLM and Baichuan2 shows more stable correction performance.

6.2 Impact of Hyperparameters

We conduct an analysis of two critical hyperparameters in Eq. (1): the distortion model weight \( \alpha \) and the small BERT-based model weight \( \beta \). Figure 3 demonstrates their effects on three 7B-scale LLM families evaluated on rSIGHAN15. For \( \alpha \), most LLMs achieve optimal performance around
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">rSIGHAN15</th>
    <th rowspan="2">FPR</th>
  </tr>
  <tr>
    <th>S-P<sup>†</sup></th>
    <th>S-R<sup>†</sup></th>
    <th>S-F<sup>†</sup></th>
    <th>C-P<sup>†</sup></th>
    <th>C-R<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
  </tr>
  <tr>
    <td colspan="8">LLMs (Zhou et al., 2024)</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>67.1</td>
    <td>55.8</td>
    <td>61.0</td>
    <td>78.7</td>
    <td>61.0</td>
    <td>68.7</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td colspan="8">Previous SOTAs & Ours</td>
  </tr>
  <tr>
    <td>RealLiSe</td>
    <td>75.7</td>
    <td>70.2</td>
    <td>72.9</td>
    <td>83.4</td>
    <td>73.9</td>
    <td>78.4</td>
    <td>8.1</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>80.5</td>
    <td>72.0</td>
    <td>76.0</td>
    <td>92.9</td>
    <td>74.6</td>
    <td>82.7</td>
    <td>5.2</td>
  </tr>
  <tr>
    <td>BERT<sup>†</sup></td>
    <td>75.3</td>
    <td>74.9</td>
    <td>75.1</td>
    <td>86.2</td>
    <td><b>79.6</b></td>
    <td><b>82.8</b></td>
    <td>10.8</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>82.6</td>
    <td>74.4</td>
    <td>78.3</td>
    <td>93.3</td>
    <td>78.4</td>
    <td><b>85.2</b></td>
    <td>3.5</td>
  </tr>
  <tr>
    <td>ReLM<sup>†</sup></td>
    <td>76.8</td>
    <td>73.8</td>
    <td>75.2</td>
    <td>87.9</td>
    <td>78.0</td>
    <td>82.6</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>83.1</td>
    <td><b>75.6</b></td>
    <td>79.2</td>
    <td>93.8</td>
    <td>78.1</td>
    <td><b>85.2</b></td>
    <td>4.1</td>
  </tr>
  <tr>
    <td>SCOPE<sup>†</sup></td>
    <td>78.7</td>
    <td>73.5</td>
    <td>76.0</td>
    <td>83.9</td>
    <td>76.7</td>
    <td>80.1</td>
    <td>7.0</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td><b>84.9</b></td>
    <td>75.4</td>
    <td><b>79.9</b></td>
    <td><b>93.8</b></td>
    <td>77.2</td>
    <td>84.7</td>
    <td><b>3.3</b></td>
  </tr>
</table>

Table 5: Ablation results of SOTA BERT-based models combined with Baichuan2(7B).

0.5. Compared with Baichuan2, both Qwen2.5 and InternLM2.5 require higher \( \beta \) values for peak performance. This difference likely stems from their weaker performance on rSIGHAN15 compared to Baichuan2. This hypothesis is further supported by experiments on different test sets. On ECSpell-Odw, where the small and large models demonstrate closer correction capabilities, we observe greater flexibility in \( \beta \) selection. Conversely, for LEMON, the superior performance of LLMs reduces the significance of small models.

It is important to note that in Table 2, the weights are fixed (\( \alpha = 0.5, \beta = 0.9 \)). In practice, tuning these weights can further enhance model performance. Importantly, our approach consistently surpasses the ReLM baseline across all hyperparameter settings, highlighting its robustness.

6.3 Impact of Distortion Model and Faithfulness Reward

Table 4 investigates the individual contributions of the distortion model (DM) and faithfulness reward (FR) through an ablation study. The results show that while the removal of DM leads to performance degradation, our approach still outperforms the ReLM baseline. This demonstrates that even when stripped of LLM-specific correction strategies, an LLM retaining core language modeling capability can still provide effective corrective guidance to the small model.

Moreover, removing the DM and FR has a larger impact on Precision than on Recall. This is because the DM’s similarity information and FR’s constraints help the model avoid unnecessary modifications.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">S-P<sup>†</sup> S-R<sup>†</sup> S-F<sup>†</sup> C-P<sup>†</sup> C-R<sup>†</sup> C-F<sup>†</sup></th>
    <th rowspan="2">FPR</th>
  </tr>
  <tr>
    <th>S-P<sup>†</sup></th>
    <th>S-R<sup>†</sup></th>
    <th>S-F<sup>†</sup></th>
    <th>C-P<sup>†</sup></th>
    <th>C-R<sup>†</sup></th>
    <th>C-F<sup>†</sup></th>
  </tr>
  <tr>
    <th colspan="8">rSIGHAN15</th>
  </tr>
  <tr>
    <td>BC2</td>
    <td>67.1</td>
    <td>55.8</td>
    <td>61.0</td>
    <td>78.7</td>
    <td>61.0</td>
    <td>68.7</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>BC2*</td>
    <td>72.3</td>
    <td>45.1</td>
    <td>55.6</td>
    <td>88.7</td>
    <td>45.2</td>
    <td>59.9</td>
    <td>2.7</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td colspan="6"></td>
    <td></td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>83.5</td>
    <td><b>76.1</b></td>
    <td><b>79.6</b></td>
    <td>93.8</td>
    <td><b>78.3</b></td>
    <td><b>85.4</b></td>
    <td>4.1</td>
  </tr>
  <tr>
    <td>+ BC2*</td>
    <td><b>83.9</b></td>
    <td>65.0</td>
    <td>73.3</td>
    <td><b>97.4</b></td>
    <td>66.6</td>
    <td>79.1</td>
    <td><b>1.2</b></td>
  </tr>
  <tr>
    <td>ECSpell-Odw</td>
    <td colspan="6"></td>
    <td></td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>92.0</td>
    <td>89.1</td>
    <td>90.5</td>
    <td>95.0</td>
    <td>92.6</td>
    <td>93.8</td>
    <td>1.6</td>
  </tr>
  <tr>
    <td>BC2*</td>
    <td>85.4</td>
    <td>59.1</td>
    <td>69.9</td>
    <td>96.7</td>
    <td>60.5</td>
    <td>74.4</td>
    <td>1.2</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td colspan="6"></td>
    <td></td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td><b>97.2</b></td>
    <td><b>95.7</b></td>
    <td><b>96.5</b></td>
    <td><b>98.7</b></td>
    <td><b>97.7</b></td>
    <td><b>98.2</b></td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>+ BC2*</td>
    <td>96.3</td>
    <td>91.8</td>
    <td>94.0</td>
    <td><b>99.2</b></td>
    <td>93.6</td>
    <td>96.3</td>
    <td><b>0.4</b></td>
  </tr>
  <tr>
    <th colspan="8">Lemon-Nov</th>
  </tr>
  <tr>
    <td>BC2</td>
    <td>49.8</td>
    <td>37.0</td>
    <td>42.4</td>
    <td>53.5</td>
    <td><b>42.0</b></td>
    <td>47.1</td>
    <td>14.9</td>
  </tr>
  <tr>
    <td>BC2*</td>
    <td>61.0</td>
    <td>19.0</td>
    <td>29.0</td>
    <td>64.5</td>
    <td>19.6</td>
    <td>30.0</td>
    <td>4.5</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td colspan="6"></td>
    <td></td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>64.0</td>
    <td><b>41.0</b></td>
    <td><b>50.0</b></td>
    <td>67.2</td>
    <td>40.3</td>
    <td><b>50.4</b></td>
    <td>9.8</td>
  </tr>
  <tr>
    <td>+ BC2*</td>
    <td><b>83.3</b></td>
    <td>23.3</td>
    <td>36.4</td>
    <td><b>84.0</b></td>
    <td>21.8</td>
    <td>34.6</td>
    <td><b>1.5</b></td>
  </tr>
</table>

Table 6: Performance of Baichuan2(7B). “*” means using the chat version of the LLM.

6.4 Impact of Small BERT-based Models

To evaluate the impact of small models on experimental outcomes, we conducted experiments with several classic and high-performing BERT-based models in combination with Baichuan2(7B). As demonstrated in Table 5 and Table 10, the integration with LLMs leads to substantial performance improvements across all small models. Notably, this enhancement is reflected in both Precision and Recall metrics, while simultaneously reducing the FPR. Furthermore, our analysis reveals a strong positive correlation between the performance of the selected small models and the overall correction performance of our mixture approach.

6.5 Conflicts Between Our Method and In-context Learning (ICL) Technique

Our proposed method treats the LLM as a pure language model, utilizing its ability to generate correct and fluent text to assist the small model in error correction. When using ICL technique, the output of the LLM itself would become biased, and the settings for fluent text generation would be disrupted, leading to reduced performance. Therefore, the LLMs we use are the base version, meaning they have only undergone pre-training and have not been subject to Instruction Fine-Tuning (IFT), so they possess almost no conversational abilities. As a result, our approach conflicts with ICL technique. We attempted to replace the LLM with the chat version of Baichuan2(7B), and the results are presented in Table 6.
<table>
  <tr>
    <th>Model</th>
    <th>SaP</th>
    <th>SiP</th>
    <th>SiS</th>
    <th>Others</th>
    <th>Total</th>
  </tr>
  <tr>
    <th colspan="6">rSIGHAN15</th>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>85.2</td>
    <td>84.6</td>
    <td>83.4</td>
    <td>55.3</td>
    <td>82.6</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>74.5</td>
    <td>67.7</td>
    <td>68.8</td>
    <td>12.7</td>
    <td>68.7</td>
  </tr>
  <tr>
    <td>ReLM+BC2</td>
    <td><b>87.2</b></td>
    <td><b>85.8</b></td>
    <td><b>83.5</b></td>
    <td><b>60.3</b></td>
    <td><b>85.4</b></td>
  </tr>
  <tr>
    <th colspan="6">ECSpell-Odw</th>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>94.2</td>
    <td>86.3</td>
    <td>96.2</td>
    <td>50.8</td>
    <td>90.6</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>96.1</td>
    <td>88.1</td>
    <td>96.3</td>
    <td>48.0</td>
    <td>93.8</td>
  </tr>
  <tr>
    <td>ReLM+BC2</td>
    <td><b>98.8</b></td>
    <td><b>98.2</b></td>
    <td><b>99.2</b></td>
    <td><b>85.7</b></td>
    <td><b>98.3</b></td>
  </tr>
  <tr>
    <th colspan="6">Lemon-Nov</th>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>49.7</td>
    <td>34.7</td>
    <td>43.8</td>
    <td>13.6</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>60.5</td>
    <td>38.9</td>
    <td>54.0</td>
    <td>1.9</td>
    <td>47.1</td>
  </tr>
  <tr>
    <td>ReLM+BC2</td>
    <td><b>61.2</b></td>
    <td><b>45.2</b></td>
    <td><b>54.2</b></td>
    <td><b>14.2</b></td>
    <td><b>50.4</b></td>
  </tr>
</table>

Table 7: Character-level \( F_1 \) scores of different error types. “SaP”, “SiP”, and “SiS” denote the three types of spelling errors: same pinyin, similar pinyin, and similar shape, respectively.

The results show that ICL technique is not compatible with our approach. We found that the performance loss primarily affects Recall, while Precision sometimes even improves, which aligns with our previous analysis. ICL biases the LLM’s output towards error correction, weakening its ability to model fluency, thus preserving Precision but significantly reducing Recall. The prompt used in our experiments is shown in Figure 5.

6.6 Performance of Different Error Types

We further broke down the character-level correction metrics with the error types categorization strategy from the distortion model. This allowed us to obtain specific performance results for each error type, such as same pinyin, similar pinyin, similar shape, and others, as shown in Table 7.

The results show that our mixture approach improves the F1 score across all error types, with simultaneous improvements in both Precision and Recall, which demonstrates the robustness of our method. It is important to note that the “Others” error type is the most challenging to correct in the CSC task. However, our method achieves significant improvements in this category, demonstrating that it effectively leverages the complementary strengths of LLMs and BERT-based models.

6.7 Impact of Beam Size

During beam search inference, beam size determines the number of candidate sequences maintained during decoding. While larger beam sizes enhance output diversity and better approximate the global optimal Score\((x, y)\), this comes at the expense of decoding speed. Beam search can alleviate the limitation where generative models can only access previously generated tokens. In our approach, incorporating BERT-based models supplies additional contextual information to the LLMs, thereby reducing the dependence on beam search compared to using LLMs alone. As shown in Figure 4, as beam size increases, the performance of all models generally shows an upward trend.

<table>
  <tr>
    <th>Model</th>
    <th>Speed (ms/sent)</th>
    <th>Slowdown</th>
  </tr>
  <tr>
    <td>BC2</td>
    <td>1,276.0</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>13.7</td>
    <td>–</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>1,470.7</td>
    <td>1.15×</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>14.1</td>
    <td>–</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>1,498.1</td>
    <td>1.17×</td>
  </tr>
</table>

Table 8: The decoding time per sentence on rSIGHAN15. The LLM used is Baichuan2(7B).

6.8 Inference Speed

We compare the inference speeds of our approach and baseline models in Table 8, with all experiments conducted on a single NVIDIA A100-PCIE-40GB GPU. The evaluation uses a batch size of 1 and a beam size of 12 for LLMs. The results reveal that our approach exhibits marginally slower inference speed compared to using LLMs alone, which can be attributed to the computational overhead introduced by integrating the small model’s probability distribution at each beam search step.

7 Related Work

7.1 BERT-based Approaches

Since the advent of BERT, BERT-based CSC models have shown strong correction capabilities. To better capture relationships between misspelled and correct characters, phonetic and glyph similarities have been incorporated into the encoder. Techniques include using confusion sets for data augmentation (Liu et al., 2021) and employing neural networks to encode phonetic and glyph features (Xu et al., 2021; Huang et al., 2021). Additionally, new training objectives like pinyin prediction tasks have been developed to boost model performance (Liu et al., 2021; Li et al., 2022; Liang et al., 2023).

Other strategies involve modifying the model pipeline, for example, by adding a detection layer to enhance detection (Zhang et al., 2020; Huang et al., 2023); and using decoding intervention strategies to improve result selection (Wang et al., 2019; Bao et al., 2020; Lv et al., 2023; Qiao et al., 2024).
Figure 4: \( F_1 \) scores of different LLMs (model size is 7B) with varying beam sizes. All models are combined with ReLM. Solid lines show sentence-level results, while dashed lines show character-level results.

7.2 LLM Approaches
In the era of LLMs, researchers have been actively exploring their potential in CSC tasks. A key consideration in these studies is whether LLMs require fine-tuning. Li et al. (2023) pioneered the exploration of prompt-based methods for correction tasks. They also experimented with SFT techniques, though the results were not satisfactory. Dong et al. (2024) enhanced the correction capabilities of LLMs by incorporating pinyin and radical information of Chinese characters into prompts. However, these prompt-based strategies struggle to maintain consistency between the prediction and input lengths.

Further advancements were made by Li et al. (2024), who replaced mixed tokenization with character-level tokenization. They fine-tune the LLMs to perform corrections in a character-by-character way, thereby resolving over 99% of the length inconsistency issues.

In contrast to these methods, Zhou et al. (2024) introduced a novel approach that is both prompt-free and training-free, treating LLMs as pure language models. This innovative approach opens a new research direction, aiming to fully leverage the intrinsic language modeling potential of LLMs.

7.3 Mixture Approaches
In the field of Grammatical Error Correction (GEC), model ensemble is commonly used. Multiple small models vote on edits to produce a final prediction (Zhang et al., 2022). Zhou et al. (2023) employed a language model, GPT-2, and a Grammatical Error Detection (GED) model, BART, as critics to dynamically guide the token selection in a Seq2Seq GEC model. They did not use an LLM due to the absence of a token-alignment method.

In the CSC domain, Liu et al. (2024a) introduced an ARM method. Similar to the model ensemble technique, they replaced the voting process with an unfine-tuned small model that selects between the outputs of the LLM and a fine-tuned small model. Unlike their method, which focuses on post-processing predictions, our approach integrates both models during the beam search decoding process.

8 Conclusions
In this paper, we propose a novel dynamic mixture approach that effectively combines small models and LLMs during the beam search decoding phase. By leveraging the strong error correction capabilities of fine-tuned BERT-based models and the language modeling strengths of LLMs, we achieve significant performance improvements. The advantage of not requiring fine-tuning of LLMs enhances the domain adaptability of our method. Experiments on mainstream public datasets demonstrate that our mixture approach achieves SOTA performance across multiple datasets.

Limitations

Different Tasks. Our method is primarily designed for the CSC task. However, this mixture approach can also be adapted to other tasks, such as GEC, which is another type of text error correction, and even other areas that can be modeled simultaneously as classification and generation tasks.

Different Languages. Our method is currently limited to Chinese. However, for other languages, such as English, Korean, and Japanese, by modifying the token alignment between small models and LLMs, the same mixture approach can also achieve collaborative error correction.

Acknowledgements
We sincerely thank the anonymous reviewers for their insightful comments and valuable efforts,
which have significantly improved the quality and clarity of our work.

This work was supported by National Natural Science Foundation of China (Grant No. 62336006 and 62176173) and Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.

References

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report. ArXiv preprint, abs/2309.16609.

Zuyi Bao, Chen Li, and Rui Wang. 2020. Chunk-based Chinese Spelling Check with Global Optimization. In Findings of EMNLP, pages 2031–2040, Online.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jia, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zenan Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruilang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaidu Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahu Lin. 2024. InternLM2 Technical Report. ArXiv preprint, abs/2403.17297.

Xingyi Cheng, Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, and Yuan Qi. 2020. SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check. In Proceedings of ACL, pages 871–881, Online.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pages 4171–4186, Minneapolis, Minnesota.

Ming Dong, Yujing Chen, Miao Zhang, Hao Sun, and Tingting He. 2024. Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7372–7383, Bangkok, Thailand. Association for Computational Linguistics.

Yong Hu, Fandong Meng, and Jie Zhou. 2024. CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers. In Proceedings of ACL, pages 146–159, Bangkok, Thailand.

Haojing Huang, Jingheng Ye, Qingyu Zhou, Yinghui Li, Yangning Li, Feng Zhou, and Hai-Tao Zheng. 2023. A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check. In Findings of EMNLP, pages 11514–11525, Singapore.

Li Huang, Junjie Li, Weiwei Jiang, Zhiyu Zhang, Minchuan Chen, Shaojun Wang, and Jing Xiao. 2021. PHMOSpell: Phonological and Morphological Knowledge Guided Chinese Spelling Check. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5958–5967, Online. Association for Computational Linguistics.

Wangjie Jiang, Zhihao Ye, Zijing Ou, Ruihui Zhao, Jianguang Zheng, Yi Liu, Bang Liu, Siheg Li, Yuju Yang, and Yefeng Zheng. 2022. MCSCSet: A Specialist-annotated Dataset for Medical-domain Chinese Spelling Correction. In Proceedings of CIKM, pages 4084–4088.

Jiahao Li, Quan Wang, Zhendong Mao, Junbo Guo, Yanyan Yang, and Yongdong Zhang. 2022. Improving Chinese Spelling Check by Character Pronunciation Prediction: The Effects of Adaptivity and Granularity. In Proceedings of EMNLP, pages 4275–4286, Abu Dhabi, United Arab Emirates.

Kunting Li, Yong Hu, Liang He, Fandong Meng, and Jie Zhou. 2024. C-LLM: Learn to Check Chinese Spelling Errors Character by Character.

Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao Zheng, and Qingyu Zhou. 2023. On the (In)Effectiveness of Large Language Models for Chinese Text Correction.

Zihong Liang, Xiaojuan Quan, and Qifan Wang. 2023. Disentangled Phonetic Representation for Chinese Spelling Correction. In Proceedings of the 61st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 13509–13521, Toronto, Canada. Association for Computational Linguistics.

Changchun Liu, Kai Zhang, Junzhe Jiang, Zirui Liu, Hanqing Tao, Min Gao, and Enhong Chen. 2024a. ARM: An Alignment-and-Replacement Module for Chinese Spelling Check Based on LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10156–10168, Miami, Florida, USA. Association for Computational Linguistics.

Linfeng Liu, Hongqiu Wu, and Hai Zhao. 2024b. Chinese Spelling Correction as Rephrasing Language Model. In Proceedings of AAAI, pages 18662–18670, Vancouver, Canada.

Shulin Liu, Tao Yang, Tianchi Yue, Feng Zhang, and Di Wang. 2021. PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction. In Proceedings of ACL-IJCNLP, pages 2991–3000, Online.

Qi Lv, Ziqiang Cao, Lei Geng, Chunhui Ai, Xu Yan, and Guohong Fu. 2023. General and Domain-adaptive Chinese Spelling Check with Error-consistent Pre-training. TALLIP, pages 1–18.

Ziheng Qiao, Houquan Zhou, Yumeng Liu, Zhenghua Li, Min Zhang, Bo Zhang, Chen Li, Ji Zhang, and Fei Huang. 2024. DISC: Plug-and-Play Decoding Intervention with Similarity of Characters for Chinese Spelling Check.

Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to SIGHAN 2015 Bake-off for Chinese Spelling Check. In Proceedings of SIGHAN, pages 32–37, Beijing, China.

Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018. A Hybrid Approach to Automatic Corpus Generation for Chinese Spelling Check. In Proceedings of EMNLP, pages 2517–2527, Brussels, Belgium.

Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided Pointer Networks for Chinese Spelling Check. In Proceedings of ACL, pages 5780–5785, Florence, Italy.

Hongqiu Wu, Shaohua Zhang, Yuchen Zhang, and Hai Zhao. 2023. Rethinking Masked Language Modeling for Chinese Spelling Correction. In Proceedings of ACL, pages 10743–10756, Toronto, Canada.

Shih-Hung Wu, Chao-Lin Liu, and Lung-Hao Lee. 2013. Chinese Spelling Check Evaluation at SIGHAN Bake-off 2013. In Proceedings of SIGHAN, pages 35–42, Nagoya, Japan.

Heng-Da Xu, Zhongli Li, Qingyu Zhou, Chao Li, Zizhen Wang, Yunbo Cao, Heyan Huang, and Xian-Ling Mao. 2021. Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking. In Findings of ACL-IJCNLP, pages 716–728, Online.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. 2023a. Baichuan 2: Open Large-scale Language Models.

Liner Yang, Xin Liu, Tianxin Liao, Zhenghao Liu, Mengyan Wang, Xuezhi Fang, and Erhong Yang. 2023b. Is Chinese Spelling Check ready? Understanding the correction behavior in real-world scenarios. AI Open, pages 183–192.

Liang-Chih Yu, Lung-Hao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. 2014. Overview of SIGHAN 2014 Bake-off for Chinese Spelling Check. In Proceedings of CIPS-SIGHAN, pages 126–132, Wuhan, China.

Shaohua Zhang, Haoran Huang, Jicong Liu, and Hang Li. 2020. Spelling Error Correction with Soft-Masked BERT. In Proceedings of ACL, pages 882–890, Online.

Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022. MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction. In Proceedings of NAACL, pages 3118–3130, Seattle, United States.

Houquan Zhou, Zhenghua Li, Bo Zhang, Chen Li, Shaopeng Lai, Ji Zhang, Fei Huang, and Min Zhang. 2024. A simple yet effective training-free prompt-free approach to Chinese spelling correction based on large language models. In Proceedings of EMNLP, pages 17446–17467, Miami, Florida, USA.

Houquan Zhou, Yumeng Liu, Zhenghua Li, Min Zhang, Bo Zhang, Chen Li, Ji Zhang, and Fei Huang. 2023. Improving Seq2Seq Grammatical Error Correction via Decoding Interventions. In Findings of EMNLP, pages 7393–7405, Singapore.

A Details of Datasets

SIGHANs. The SIGHAN collection comprises three Chinese learner corpora (SIGHAN 13/14/15) (Wu et al., 2013; Yu et al., 2014; Tseng et al., 2015), which serve as standard benchmarks for CSC research. To address the noise and annotation errors in the original datasets, we adopt the revised version (Yang et al., 2023b) (rSIGHANs), which was manually re-annotated. Following prior setups, we combine SIGHAN datasets with Wang271K (Wang
et al., 2018) (containing 271K synthetic training instances) to form the composite training set.

LEMON. Containing over 22K sentences across seven distinct domains (game, encyclopedia, contract, medical care, car, novel, and news) (Wu et al., 2023), LEMON provides a cross-domain evaluation framework for CSC systems. Due to the absence of in-domain training sets, LEMON is employed to assess the cross-domain generalization capability of CSC models.

ECSpell. ECSpell consists of three small-scale datasets from the domains of law, medical treatment, and official document writing (Lv et al., 2023), offering domain-specific evaluation scenarios for CSC models.

CSCD-NS. CSCD-NS (Hu et al., 2024) contains 40K annotated samples sourced from real posts on Sina Weibo, effectively reflecting the real-world error correction performance of CSC models.

MCSCSet. Developed for medical domain applications (Jiang et al., 2022), this large-scale corpus contains approximately 200K professionally annotated sentences. The significant domain gap between medical texts and open-domain datasets makes MCSCSet particularly valuable for evaluating domain adaptation capabilities in CSC systems. Detailed statistics are presented in Table 11.

B Details of Baselines

In our experiments, we selected four BERT-based models as baselines: BERT, ReaLiSe, SCOPE, and ReLM.

• BERT: The vanilla BERT architecture implemented using the bert-chinese-base5 pre-trained weights.
• ReaLiSe: This model employs GCN and CNN respectively to encode the pinyin and visual features (font images) of each Chinese character. These additional representations help the model capture the intrinsic phonetic and glyph relationships between characters. It uses chinese-roberta-wwm-ext6 as its backbone.
• SCOPE: As one of the current SOTA CSC models, SCOPE incorporates an extra training task for character pronunciation prediction (CPP). The model is initialized from ChineseBERT-base7.
• ReLM: ReLM reframes the CSC task as a rephrasing problem rather than a tagging task. In practice, it still adopts non-autoregressive inference and remains essentially a BERT-based model, pre-trained and fine-tuned based on bert-chinese-base.

C More Experiments

In this section, we present detailed results across different domains and the results of some other CSC models, such as ARM (Liu et al., 2024a) and C-LLM (Li et al., 2024).

C.1 Experiments on Original SIGHAN15

Following the experimental settings of previous studies, we use the original SIGHAN15 test set for comparison.8 In line with rSIGHAN15, we also adopt ReaLiSe and SCOPE, as shown in Table 10. Evidently, our method significantly improves the performance of all baseline models, surpassing the

<table>
  <tr>
    <th>Source</th>
    <th colspan="9">SIGHAN15</th>
  </tr>
  <tr>
    <th></th>
    <th>S-P*</th>
    <th>S-R*</th>
    <th>S-F*</th>
    <th>C-P*</th>
    <th>C-R*</th>
    <th>C-F*</th>
    <th>FPR</th>
  </tr>
  <tr>
    <td colspan="8">LLMs (Zhou et al., 2024)</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>61.2</td>
    <td>58.2</td>
    <td>59.7</td>
    <td>69.0</td>
    <td>65.4</td>
    <td>67.2</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td colspan="8">Previous SOTAs & Ours</td>
  </tr>
  <tr>
    <td>ReaLiSe</td>
    <td>75.9</td>
    <td>79.9</td>
    <td>77.8</td>
    <td>83.4</td>
    <td>83.8</td>
    <td>83.6</td>
    <td>12.0</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>80.7</td>
    <td>82.4</td>
    <td>81.5</td>
    <td>87.2</td>
    <td>85.3</td>
    <td>86.3</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>BERT**</td>
    <td>75.2</td>
    <td>83.4</td>
    <td>79.1</td>
    <td>81.9</td>
    <td>89.2</td>
    <td>85.4</td>
    <td>14.1</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>81.7</td>
    <td>84.1</td>
    <td>82.9</td>
    <td>86.8</td>
    <td><b>89.6</b></td>
    <td><b>88.2</b></td>
    <td>7.9</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>76.8</td>
    <td>83.9</td>
    <td>80.2</td>
    <td>83.2</td>
    <td>88.6</td>
    <td>85.8</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>82.8</td>
    <td><b>85.9</b></td>
    <td>84.3</td>
    <td>87.6</td>
    <td>88.7</td>
    <td>88.1</td>
    <td><b>7.8</b></td>
  </tr>
  <tr>
    <td>SCOPE**</td>
    <td>78.6</td>
    <td>85.5</td>
    <td>81.0</td>
    <td>85.3</td>
    <td>86.6</td>
    <td>84.9</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td><b>83.8</b></td>
    <td>85.0</td>
    <td><b>84.4</b></td>
    <td><b>88.1</b></td>
    <td>87.6</td>
    <td>87.9</td>
    <td>7.9</td>
  </tr>
  <tr>
    <td>+ ARM</td>
    <td>79.5</td>
    <td>83.1</td>
    <td>81.3</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
</table>

Table 9: Examples of ECSpell-Law.

<table>
  <tr>
    <th>Model</th>
    <th>S-P*</th>
    <th>S-R*</th>
    <th>S-F*</th>
    <th>C-P*</th>
    <th>C-R*</th>
    <th>C-F*</th>
    <th>FPR</th>
  </tr>
  <tr>
    <td colspan="8">LLMs (Zhou et al., 2024)</td>
  </tr>
  <tr>
    <td>BC2</td>
    <td>61.2</td>
    <td>58.2</td>
    <td>59.7</td>
    <td>69.0</td>
    <td>65.4</td>
    <td>67.2</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td colspan="8">Previous SOTAs & Ours</td>
  </tr>
  <tr>
    <td>ReaLiSe</td>
    <td>75.9</td>
    <td>79.9</td>
    <td>77.8</td>
    <td>83.4</td>
    <td>83.8</td>
    <td>83.6</td>
    <td>12.0</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>80.7</td>
    <td>82.4</td>
    <td>81.5</td>
    <td>87.2</td>
    <td>85.3</td>
    <td>86.3</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>BERT**</td>
    <td>75.2</td>
    <td>83.4</td>
    <td>79.1</td>
    <td>81.9</td>
    <td>89.2</td>
    <td>85.4</td>
    <td>14.1</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>81.7</td>
    <td>84.1</td>
    <td>82.9</td>
    <td>86.8</td>
    <td><b>89.6</b></td>
    <td><b>88.2</b></td>
    <td>7.9</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>76.8</td>
    <td>83.9</td>
    <td>80.2</td>
    <td>83.2</td>
    <td>88.6</td>
    <td>85.8</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td>82.8</td>
    <td><b>85.9</b></td>
    <td>84.3</td>
    <td>87.6</td>
    <td>88.7</td>
    <td>88.1</td>
    <td><b>7.8</b></td>
  </tr>
  <tr>
    <td>SCOPE**</td>
    <td>78.6</td>
    <td>85.5</td>
    <td>81.0</td>
    <td>85.3</td>
    <td>86.6</td>
    <td>84.9</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>+ BC2</td>
    <td><b>83.8</b></td>
    <td>85.0</td>
    <td><b>84.4</b></td>
    <td><b>88.1</b></td>
    <td>87.6</td>
    <td>87.9</td>
    <td>7.9</td>
  </tr>
  <tr>
    <td>+ ARM</td>
    <td>79.5</td>
    <td>83.1</td>
    <td>81.3</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
</table>

Table 10: Sentence- and character-level results on the SIGHAN15 test set. ARM represents the latest approach that integrates the results of small models and GPT-3.5-Turbo (Liu et al., 2024a).

5https://huggingface.co/bert-base-chinese
6https://huggingface.co/hfl/chinese-roberta-wwm-ext
7https://huggingface.co/ShannonAI/ChineseBERT-base
8Since the dataset includes traditional characters, we use the preprocessed version by Xu et al. (2021).
<table>
  <tr>
    <th>Training Sets</th>
    <th>SIGHANs</th>
    <th>Wang271K</th>
    <th>CSCD-NS</th>
    <th>MCSCSet</th>
    <th colspan="3">ECSpell</th>
  </tr>
  <tr>
    <th>Subsets</th>
    <th>–</th>
    <th>–</th>
    <th>–</th>
    <th>–</th>
    <th><i>Law</i></th>
    <th><i>Med</i></th>
    <th><i>Odw</i></th>
  </tr>
  <tr>
    <td>#Sent.</td>
    <td>6,479</td>
    <td>271,329</td>
    <td>30,000</td>
    <td>157,194</td>
    <td>1,960</td>
    <td>3,000</td>
    <td>1,720</td>
  </tr>
  <tr>
    <td>Avg. Length</td>
    <td>42.1</td>
    <td>42.6</td>
    <td>57.4</td>
    <td>10.9</td>
    <td>30.7</td>
    <td>50.2</td>
    <td>41.2</td>
  </tr>
  <tr>
    <td>Avg. Error/Sent.</td>
    <td>1.0</td>
    <td>1.4</td>
    <td>0.5</td>
    <td>0.9</td>
    <td>0.9</td>
    <td>0.8</td>
    <td>0.9</td>
  </tr>
</table>

<table>
  <tr>
    <th>Test Sets</th>
    <th>rSIGHANs</th>
    <th>CSCD-NS</th>
    <th>MCSCSet</th>
    <th>ECSpell</th>
    <th>LEMON</th>
  </tr>
  <tr>
    <th>Subsets</th>
    <th>15</th>
    <th>14</th>
    <th>13</th>
    <th>–</th>
    <th>–</th>
    <th><i>Law</i></th>
    <th><i>Med</i></th>
    <th><i>Odw</i></th>
    <th>–</th>
  </tr>
  <tr>
    <td>#Sent.</td>
    <td>1,100</td>
    <td>1,062</td>
    <td>1,000</td>
    <td>5,000</td>
    <td>19,650</td>
    <td>500</td>
    <td>500</td>
    <td>500</td>
    <td>22,252</td>
  </tr>
  <tr>
    <td>Avg. Length</td>
    <td>30.6</td>
    <td>50.0</td>
    <td>74.3</td>
    <td>57.6</td>
    <td>10.9</td>
    <td>29.7</td>
    <td>49.6</td>
    <td>40.5</td>
    <td>35.4</td>
  </tr>
  <tr>
    <td>Avg. Error/Sent.</td>
    <td>0.8</td>
    <td>0.9</td>
    <td>1.5</td>
    <td>0.5</td>
    <td>0.9</td>
    <td>0.8</td>
    <td>0.7</td>
    <td>0.8</td>
    <td>0.5</td>
  </tr>
</table>

Table 11: Detailed statistics of all datasets used in our experiments.

<table>
  <tr>
    <th>Datasets</th>
    <th colspan="3">ECSpell (cleaned)</th>
  </tr>
  <tr>
    <th>Subsets</th>
    <th><i>Law</i></th>
    <th><i>Med</i></th>
    <th><i>Odw</i></th>
  </tr>
  <tr>
    <td colspan="4">Previous SOTA</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td>71.0</td>
    <td>70.3</td>
    <td>77.7</td>
  </tr>
  <tr>
    <td colspan="4">LLMs (<b>Zhou et al., 2024</b>)</td>
  </tr>
  <tr>
    <td>Baichuan2</td>
    <td>83.7</td>
    <td><b>82.0</b></td>
    <td><b>90.5</b></td>
  </tr>
  <tr>
    <td>Qwen2.5</td>
    <td>83.7</td>
    <td>73.1</td>
    <td>89.1</td>
  </tr>
  <tr>
    <td>IL2.5</td>
    <td><b>85.8</b></td>
    <td>66.2</td>
    <td>89.0</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td colspan="3"></td>
  </tr>
  <tr>
    <td>ReLM + BC2</td>
    <td>86.4</td>
    <td><b>87.9</b></td>
    <td>93.1</td>
  </tr>
  <tr>
    <td>ReLM + QW2.5</td>
    <td><b>89.0</b></td>
    <td>86.7</td>
    <td>93.5</td>
  </tr>
  <tr>
    <td>ReLM + IL2.5</td>
    <td>86.4</td>
    <td>87.1</td>
    <td><b>93.7</b></td>
  </tr>
</table>

Table 12: Experiments on cleaned ECSpell.

System and User Prompts for LLMs

System Prompt:
你是一个优秀的中文拼写纠错模型。中文拼写纠错模型即更正用户输入句子中的拼写错误。
User Prompt:
请输出句子包含错别字的文本，你需要输出正确的文本，请不要输出预测文本以外的任何输出。
示例：
输入：消费者对事物的安全性问题越来越重视，消费有机食材在经济上已不再光顾
输出：消费者对食物的安全性问题越来越重视，消费有机食材在经济上已不再光顾
输入：它是我们曾经是朋友，但是现在却是对手
输出：它是我们曾经是朋友，但是现在却是对手
输入：近日办案民警终于分别在东莞、深圳两地抓获两人
输出：近日办案民警终于分别在东莞、深圳两地抓获两人
输入：[待纠错文本]
输出：

Figure 5: Prompt template.

current SOTA performance. We also show the latest approach that ensembles the results of small models and GPT-3.5-Turbo (SCOPE + ARM) (Liu et al., 2024a), compared with which our method achieves a more effective performance improvement through deep integration of the small and large models during the inference stage.

C.2 Detailed Results

Due to space limitations, we present the detailed results of the subsets of rSIGHANs, ECSpell, and LEMON in Table 13, Table 14, and Table 15. We also provide the results of ARM and C-LLM in the corresponding tables. The results show that our method outperforms another mixture approach and the current SOTA SFT-based LLM strategy.

C.3 Experiments on Cleaned ECSpell

Our analysis reveals that the strong performance of fine-tuned small models on ECSpell originates from substantial data leakage caused by homogeneous sentence pairs. Although ECSpell avoids including identical source-target sentence pairs, it introduces different synthesized errors for the same correct sentences that appear in both the training and test sets. These overlapping sentences account for 52.7%, 19.3%, and 28.2% of the ECSpell-Law/Med/Odw training sets, respectively. For example, the first sentence in the ECSpell-Law test set appears five times in the training set as the same sentence, as illustrated in Table 9.

Undoubtedly, the fine-tuning process of small models leads to results that are significantly higher than those of the LLMs without fine-tuning. Therefore, we removed the leaked sentences from the training sets and reorganized the experiments in Table 12. It is worth noting that all hyperparameter settings remain identical to those in the main experiments. Clearly, our approach continues to achieve significant and stable improvements. In fact, due to the significantly reduced size of the training set, the performance of fine-tuned small models lags behind that of LLMs, and appropriately lowering the weight of the small model \( \beta \) can yield even more significant performance improvements.
<table>
  <tr>
    <th rowspan="2">Datasets</th>
    <th colspan="3">rSIGHANs</th>
    <th colspan="3">ECSpell</th>
    <th colspan="7">LEMON</th>
  </tr>
  <tr>
    <th>Subsets</th>
    <th>15</th>
    <th>14</th>
    <th>13</th>
    <th>Law</th>
    <th>Med</th>
    <th>Odw</th>
    <th>Car</th>
    <th>Cot</th>
    <th>Enc</th>
    <th>Gam</th>
    <th>Med</th>
    <th>New</th>
    <th>Nov</th>
  </tr>
  <tr>
    <td colspan="14">Previous SOTAs</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td></td><td>75.1</td><td>65.6</td><td>71.6</td><td>95.9</td><td>88.0</td><td>89.1</td><td>52.0</td><td>63.8</td><td>45.3</td><td>32.9</td><td>50.8</td><td>56.0</td><td>35.8</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td></td><td>75.2</td><td>65.9</td><td>74.4</td><td>96.2</td><td>90.2</td><td>90.4</td><td>53.6</td><td>67.7</td><td>47.7</td><td>34.6</td><td>53.9</td><td>58.8</td><td>38.0</td>
  </tr>
  <tr>
    <td>MDCSpell*</td>
    <td></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>34.1</td><td>49.2</td><td>32.8</td><td>14.8</td><td>29.5</td><td>34.4</td><td>14.3</td>
  </tr>
  <tr>
    <td colspan="14">LLMs (Zhou et al., 2024)</td>
  </tr>
  <tr>
    <td>Baichuan2</td>
    <td></td><td><b>61.0</b></td><td><b>53.1</b></td><td><b>63.1</b></td><td>83.7</td><td>82.0</td><td>90.5</td><td><b>54.2</b></td><td><b>63.3</b></td><td><b>51.4</b></td><td><b>36.9</b></td><td><b>60.6</b></td><td><b>63.9</b></td><td><b>42.4</b></td>
  </tr>
  <tr>
    <td>Qwen2.5</td>
    <td></td><td>58.3</td><td>50.7</td><td>57.9</td><td>83.7</td><td>73.1</td><td>89.1</td><td>48.2</td><td>59.9</td><td>46.5</td><td>34.7</td><td>54.2</td><td>59.5</td><td>37.1</td>
  </tr>
  <tr>
    <td>IL2.5</td>
    <td></td><td>55.6</td><td>45.9</td><td>56.0</td><td><b>85.8</b></td><td>66.2</td><td>89.0</td><td>44.4</td><td>54.7</td><td>45.2</td><td>33.0</td><td>50.2</td><td>57.2</td><td>32.1</td>
  </tr>
  <tr>
    <td colspan="14">Mixture</td>
  </tr>
  <tr>
    <td>MDCSpell + ARM*</td>
    <td></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>37.1</td><td>52.7</td><td>35.2</td><td>15.3</td><td>33.0</td><td>36.4</td><td>15.6</td>
  </tr>
  <tr>
    <td>BERT + BC2*<sup>**</sup></td>
    <td></td><td>78.3</td><td>67.1</td><td>72.1</td><td>95.7</td><td>93.8</td><td>93.6</td><td>56.0</td><td>68.0</td><td>49.4</td><td>41.2</td><td>63.9</td><td>68.7</td><td>47.6</td>
  </tr>
  <tr>
    <td>BERT + QW2.5</td>
    <td></td><td>77.7</td><td><b>69.5</b></td><td>73.7</td><td>96.5</td><td>94.1</td><td>91.5</td><td>57.7</td><td>67.0</td><td>52.4</td><td>37.9</td><td>65.2</td><td>67.5</td><td>44.7</td>
  </tr>
  <tr>
    <td>BERT + IL2.5</td>
    <td></td><td>78.4</td><td>67.7</td><td>72.3</td><td>96.3</td><td>94.7</td><td>92.0</td><td>55.8</td><td>67.4</td><td>51.6</td><td>38.3</td><td>55.6</td><td>64.8</td><td>43.1</td>
  </tr>
  <tr>
    <td>ReLM + BC2*<sup>**</sup></td>
    <td></td><td><b>79.6</b></td><td>67.6</td><td>74.6</td><td><b>98.3</b></td><td><b>96.6</b></td><td><b>96.5</b></td><td><b>62.7</b></td><td><b>74.5</b></td><td><b>56.5</b></td><td><b>45.1</b></td><td><b>66.3</b></td><td><b>72.0</b></td><td><b>50.0</b></td>
  </tr>
  <tr>
    <td>ReLM + QW2.5</td>
    <td></td><td>77.9</td><td>67.7</td><td><b>76.0</b></td><td>98.1</td><td>95.9</td><td>95.9</td><td><b>63.6</b></td><td><b>71.2</b></td><td><b>57.4</b></td><td>44.0</td><td><b>66.7</b></td><td>71.4</td><td>49.4</td>
  </tr>
  <tr>
    <td>ReLM + IL2.5</td>
    <td></td><td>79.5</td><td>68.3</td><td>75.3</td><td>97.3</td><td>95.9</td><td><b>96.9</b></td><td>61.3</td><td>72.0</td><td>56.1</td><td>39.5</td><td>64.7</td><td>71.0</td><td>47.7</td>
  </tr>
</table>

Table 13: Sentence-level \( F_1 \) (S-F) scores across different domains. “*” indicates that the results are extracted from Liu et al. (2024a) and the small model is trained on Wang271K + SIGHANs.

<table>
  <tr>
    <th rowspan="2">Datasets</th>
    <th colspan="3">rSIGHANs</th>
    <th colspan="3">ECSpell</th>
    <th colspan="7">LEMON</th>
  </tr>
  <tr>
    <th>Subsets</th>
    <th>15</th>
    <th>14</th>
    <th>13</th>
    <th>Law</th>
    <th>Med</th>
    <th>Odw</th>
    <th>Car</th>
    <th>Cot</th>
    <th>Enc</th>
    <th>Gam</th>
    <th>Med</th>
    <th>New</th>
    <th>Nov</th>
  </tr>
  <tr>
    <td colspan="14">Previous SOTAs</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td></td><td>82.8</td><td>77.3</td><td>85.7</td><td>97.4</td><td>93.3</td><td>93.2</td><td>52.7</td><td>65.3</td><td>46.1</td><td>35.6</td><td>52.0</td><td>57.4</td><td>36.3</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td></td><td>82.6</td><td>76.7</td><td>86.1</td><td>93.6</td><td>94.4</td><td>94.4</td><td>54.3</td><td>67.4</td><td>48.1</td><td>37.9</td><td>54.9</td><td>60.5</td><td>37.9</td>
  </tr>
  <tr>
    <td colspan="14">LLMs (Zhou et al., 2024; Li et al., 2024)</td>
  </tr>
  <tr>
    <td>Baichuan2</td>
    <td></td><td><b>68.7</b></td><td><b>64.9</b></td><td><b>76.8</b></td><td><b>88.0</b></td><td><b>93.8</b></td><td><b>93.8</b></td><td><b>58.8</b></td><td><b>65.3</b></td><td><b>56.3</b></td><td><b>40.9</b></td><td><b>61.6</b></td><td><b>66.8</b></td><td><b>47.1</b></td>
  </tr>
  <tr>
    <td>Qwen2.5</td>
    <td></td><td>66.1</td><td>64.7</td><td>75.0</td><td>81.6</td><td>93.0</td><td>93.0</td><td>54.1</td><td>62.5</td><td>52.4</td><td><b>41.8</b></td><td>56.6</td><td>63.2</td><td>43.8</td>
  </tr>
  <tr>
    <td>IL2.5</td>
    <td></td><td>64.6</td><td>60.6</td><td>73.4</td><td>78.3</td><td>92.4</td><td>92.4</td><td>49.9</td><td>58.1</td><td>51.6</td><td>37.5</td><td>53.5</td><td>61.0</td><td>38.6</td>
  </tr>
  <tr>
    <td>C-LLM</td>
    <td></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>57.5</td><td>60.4</td><td><b>56.5</b></td><td>38.0</td><td><b>65.3</b></td><td>64.5</td><td>43.9</td>
  </tr>
  <tr>
    <td colspan="14">Ours</td>
  </tr>
  <tr>
    <td>BERT + BC2</td>
    <td></td><td>85.2</td><td>78.3</td><td>86.6</td><td>96.2</td><td>95.6</td><td>95.6</td><td>58.3</td><td>68.4</td><td>51.8</td><td>43.9</td><td>64.4</td><td>70.6</td><td>50.1</td>
  </tr>
  <tr>
    <td>BERT + QW2.5</td>
    <td></td><td>84.6</td><td><b>80.1</b></td><td>87.0</td><td>95.4</td><td>93.4</td><td>93.4</td><td>60.5</td><td>67.3</td><td>55.5</td><td>43.4</td><td>65.7</td><td>70.0</td><td>48.7</td>
  </tr>
  <tr>
    <td>BERT + IL2.5</td>
    <td></td><td>84.8</td><td>79.3</td><td><b>87.4</b></td><td>92.7</td><td>94.4</td><td>94.4</td><td>57.4</td><td>68.0</td><td>52.3</td><td>41.5</td><td>56.0</td><td>66.2</td><td>44.0</td>
  </tr>
  <tr>
    <td>ReLM + BC2*<sup>**</sup></td>
    <td></td><td><b>85.4</b></td><td>78.0</td><td>85.7</td><td><b>97.8</b></td><td><b>98.2</b></td><td><b>98.2</b></td><td><b>62.5</b></td><td><b>74.2</b></td><td><b>56.7</b></td><td><b>47.8</b></td><td><b>66.5</b></td><td><b>73.1</b></td><td><b>50.4</b></td>
  </tr>
  <tr>
    <td>ReLM + QW2.5</td>
    <td></td><td>84.2</td><td>77.1</td><td>86.6</td><td>96.9</td><td>97.7</td><td>97.7</td><td><b>64.0</b></td><td><b>69.8</b></td><td><b>57.8</b></td><td><b>48.2</b></td><td><b>66.8</b></td><td><b>72.7</b></td><td><b>50.6</b></td>
  </tr>
  <tr>
    <td>ReLM + IL2.5</td>
    <td></td><td>85.3</td><td>78.7</td><td>86.8</td><td>97.5</td><td><b>98.3</b></td><td><b>98.3</b></td><td>62.2</td><td>71.8</td><td>57.4</td><td>44.1</td><td>65.1</td><td>72.1</td><td>48.8</td>
  </tr>
</table>

Table 14: Character-level \( F_1 \) (C-F) scores across different domains.

<table>
  <tr>
    <th rowspan="2">Datasets</th>
    <th colspan="3">rSIGHANs</th>
    <th colspan="3">ECSpell</th>
    <th colspan="7">LEMON</th>
  </tr>
  <tr>
    <th>Subsets</th>
    <th>15</th>
    <th>14</th>
    <th>13</th>
    <th>Law</th>
    <th>Med</th>
    <th>Odw</th>
    <th>Car</th>
    <th>Cot</th>
    <th>Enc</th>
    <th>Gam</th>
    <th>Med</th>
    <th>New</th>
    <th>Nov</th>
  </tr>
  <tr>
    <td colspan="14">Previous SOTAs</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td></td><td>10.8</td><td>13.0</td><td>12.5</td><td>3.7</td><td>5.1</td><td>2.5</td><td>12.2</td><td>7.8</td><td>13.8</td><td>22.4</td><td>8.5</td><td>9.4</td><td>17.3</td>
  </tr>
  <tr>
    <td>ReLM</td>
    <td></td><td>8.3</td><td>13.6</td><td>12.6</td><td>6.5</td><td>9.8</td><td>5.8</td><td>12.0</td><td>4.9</td><td>12.7</td><td>20.6</td><td>5.8</td><td>8.4</td><td>17.6</td>
  </tr>
  <tr>
    <td colspan="14">LLMs (Zhou et al., 2024)</td>
  </tr>
  <tr>
    <td>Baichuan2</td>
    <td></td><td><b>8.3</b></td><td><b>15.7</b></td><td><b>22.3</b></td><td>4.9</td><td><b>8.7</b></td><td><b>1.6</b></td><td><b>6.9</b></td><td><b>7.8</b></td><td><b>10.2</b></td><td><b>19.8</b></td><td><b>3.4</b></td><td><b>6.0</b></td><td><b>14.9</b></td>
  </tr>
  <tr>
    <td>Qwen2.5</td>
    <td></td><td>10.2</td><td>19.4</td><td>24.2</td><td><b>4.1</b></td><td>12.9</td><td>2.9</td><td>10.7</td><td>6.7</td><td>13.2</td><td>23.9</td><td>5.4</td><td>9.2</td><td>20.7</td>
  </tr>
  <tr>
    <td>IL2.5</td>
    <td></td><td>13.3</td><td>21.4</td><td>25.5</td><td><b>4.1</b></td><td>14.0</td><td>2.5</td><td>13.0</td><td>10.1</td><td>15.2</td><td>28.3</td><td>7.9</td><td>9.8</td><td>24.8</td>
  </tr>
  <tr>
    <td colspan="14">Ours</td>
  </tr>
  <tr>
    <td>BERT + BC2</td>
    <td></td><td><b>3.5</b></td><td>10.0</td><td><b>8.7</b></td><td><b>1.6</b></td><td><b>2.6</b></td><td><b>0.8</b></td><td>6.0</td><td>2.3</td><td>7.5</td><td>14.5</td><td>2.4</td><td>4.2</td><td>11.4</td>
  </tr>
  <tr>
    <td>BERT + QW2.5</td>
    <td></td><td>3.7</td><td><b>8.7</b></td><td><b>8.7</b></td><td>1.6</td><td>4.4</td><td>2.1</td><td>8.4</td><td>2.5</td><td>10.9</td><td>22.5</td><td>3.1</td><td>6.7</td><td>16.9</td>
  </tr>
  <tr>
    <td>BERT + IL2.5</td>
    <td></td><td>5.6</td><td>9.3</td><td><b>8.7</b></td><td>2.0</td><td>3.3</td><td>1.3</td><td>5.7</td><td>2.2</td><td>7.2</td><td>14.1</td><td>2.5</td><td>3.5</td><td>11.9</td>
  </tr>
  <tr>
    <td>ReLM + BC2*<sup>**</sup></td>
    <td></td><td><b>4.1</b></td><td>10.0</td><td>13.4</td><td><b>2.4</b></td><td><b>3.1</b></td><td><b>0.8</b></td><td><b>4.5</b></td><td><b>1.3</b></td><td><b>6.6</b></td><td><b>11.6</b></td><td><b>2.3</b></td><td><b>3.1</b></td><td><b>9.8</b></td>
  </tr>
  <tr>
    <td>ReLM + QW2.5</td>
    <td></td><td>5.4</td><td>11.0</td><td>10.8</td><td>3.3</td><td>4.5</td><td>1.2</td><td>6.4</td><td>2.9</td><td>9.2</td><td>17.3</td><td>2.2</td><td>5.1</td><td>12.2</td>
  </tr>
  <tr>
    <td>ReLM + IL2.5</td>
    <td></td><td>5.2</td><td>10.6</td><td>12.7</td><td>3.3</td><td>3.8</td><td><b>0.8</b></td><td>6.8</td><td>2.2</td><td>9.1</td><td>15.7</td><td>3.3</td><td>4.4</td><td>13.9</td>
  </tr>
</table>

Table 15: False positive rate (FPR) scores across different domains.

Reviewer paper summaries: This paper presents a novel approach for Chinese Spelling Check (CSC) that combines BERT-based models with large language models (LLMs). The method integrates probability distributions from both model types during beam search decoding, leveraging the strengths of each. Key advantages include no need for LLM fine-tuning, enabling efficient domain adaptation, and incorporating a distortion model and faithfulness reward for output consistency. Experiments show significant performance improvements across multiple datasets, achieving state-of-the-art results with strong cross-domain generalization. The approach's effectiveness, efficiency, and adaptability make it a notable contribution to CSC research.
Summary of strengths: 1. The paper introduces an innovative approach that synergizes BERT-based models with LLMs for Chinese Spelling Check, leveraging their complementary strengths. This concept of using small models to provide distortion signals for controllable data generation by LLMs could potentially benefit other NLP tasks.
2. The motivation is well-grounded and intuitive: BERT-based models excel at capturing intricate relationships between edit pairs, while LLMs demonstrate superior fluency in content generation.
3. The experimental design is robust, with comprehensive ablation studies that thoroughly examine the contribution of each component.
4. The proposed method achieves significant performance improvements across multiple datasets without requiring LLM fine-tuning, enhancing efficiency and domain adaptability.
Summary of weaknesses: 1. The performance advantage of LLMs over BERT-based models is not consistently substantial across all datasets. A comparison of inference times between LLMs and BERT models could provide valuable insights into the trade-offs between performance and computational efficiency.
2. The training process for BERT-series models did not utilize the CSCD-NS training set, which potentially offers higher quality data. Incorporating this dataset might further improve the performance of the BERT-based component.
3. The specific parameters of the distortion model used in LLMs are not clearly detailed. Providing this information would allow for a more comprehensive understanding and comparison of the approaches.
Comments, suggestions and typos: Previous research has demonstrated that in-context learning (ICL) can significantly enhance error correction performance of LLM. The paper could benefit from exploring potential synergies or conflicts between the proposed method and ICL techniques.
Overall assessment scores: 3.5