Constructing Word-Context-Coupled Space Aligned with Associative Knowledge Relations for Interpretable Language Modeling
Year: 2023

Abstract:
As the foundation of current natural language processing methods, pre-trained language model has achieved excellent performance. However, the black-box structure of the deep neural network in pre-trained language models seriously limits the interpretability of the language modeling process. After revisiting the coupled requirement of deep neural representation and semantics logic of language modeling, a Word-Context-Coupled Space (W2CSpace) is proposed by introducing the alignment processing between uninterpretable neural representation and interpretable statistical logic. Moreover, a clustering process is also designed to connect the wordand context-level semantics. Specifically, an associative knowledge network (AKN), considered interpretable statistical logic, is introduced in the alignment process for word-level semantics. Furthermore, the context-relative distance is employed as the semantic feature for the downstream classifier, which is greatly different from the current uninterpretable semantic representations of pre-trained models. Our experiments for performance evaluation and interpretable analysis are executed on several types of datasets, including SIGHAN, Weibo, and ChnSenti. Wherein a novel evaluation strategy for the interpretability of machine learning models is first proposed. According to the experimental results, our language model can achieve better performance and highly credible interpretable ability compared to related stateof-the-art methods.1

heading: 1 Introduction
text: Machine learning has recently been democratized in various domains, such as search engines, conversational systems, and autonomous driving(Gao et al., 2018; Grigorescu et al., 2020; Wang et al., 2022). However, despite AI technologies signifi-
*Corresponding author 1https://github.com/ColeGroup/W2CSpace
cantly facilitating industrial processes and improving work experiences, the uninterpretable logic of machines leads to distrust, which hinders further development of AI. Explainable Artificial intelligence (XAI), proposed to bridge the block between humans and machines, has been increasingly attracting attention recently, where "explanation" is described as abductive inference and transferring knowledge (Josephson and Josephson, 1996; Miller, 2019). Calling for explaining and understanding the machine learning process, researchers aim to interpret the methods for system verification, compliance with legislation, and technology improvement.
Computational linguistics, which serves as the theoretical foundation for NLP, aims to promote communication between humans and machines (Khan et al., 2016). However, during the recent development, the uninterpretable NLP methods have raised concerns. The decreased transparency and increased parameter complexity adversely affect the model explainability and controllability, even if the performance of the language models has significantly improved, such as BERTs (Devlin et al., 2019; Liu et al., 2019a; Clark et al., 2020), GPTs (Radford et al., 2018, 2019; Brown et al., 2020) and so on. The performance advantage of the black-box methods is attractive while the researchers investigate interpretable algorithms. Therefore, the existing works mainly focus on (1) explaining the blackbox methods and (2) using interpretable models (Ribeiro et al., 2016; Li et al., 2022a).
In order to explain over-parameterized language models, the model-agnostic analysis is investigated on recent deep neural methods. Without modifying the black-box models, the researchers analyze the immediate feature of the neural layers, attention distribution, and so on (Clark et al., 2019; Vig, 2019; Rogers et al., 2020). Quantitative experiments and visual analysis is able to partially reveal the behaviors of the key components and
8414
the overall response of the methods to certain patterns (Hewitt and Manning, 2019; Kovaleva et al., 2019). However, without any interpretable optimization of the model, the analysis is unable to provide enough detail for understanding (Rudin, 2019), which indicates a completely faithful explanation from black-box components or deep neural methods is impossible.
Different from model-agnostic analysis, methods that integrate interpretable algorithms enable more comprehensive interpretability. Two different types of structures are adopted in these approaches, including (1) a black-box backbone with an interpretable bypass for implicit informing and (2) a transparent backbone with an interpretable algorithm for direct interpreting (Beckh et al., 2021). For implicitly informed methods, the introduced interpretable knowledge regulates the immediate feature or embedding (Liu et al., 2019b; Rybakov et al., 2020). Within the interpretable bypass, the performance of the backbone is maximally preserved, which is the main reason these structures are often opted for over-parameterized models (Jang et al., 2021; Chen et al., 2020). However, the integrated knowledge is unable to decisively change the structure of the backbones for a transparent decision process, which adversely affects the generalization ability of the approaches and limits the application of the methods to specific tasks. In contrast, approaches with interpretable backbones exhibit a more integrated relationship between the components, enabling better explanations than the above two types of approaches. The interpretable algorithms serve as word embedding, immediate feature, or the classifier to realize transparency decision process (Onoe and Durrett, 2020; Lee et al., 2022; Kaneko et al., 2022). But the performance of existing interpretable models remains incomparable to the most advanced language models.
In this work, we address the aforementioned obstacles by developing a novel interpretable language modeling method by constructing a WordContext-Coupled Space (W2CSpace) aligned with statistical knowledge2, which enables (1) effective interpretation of BERT representations (Devlin et al., 2019) by introducing the interpretable statistical logic, (2) reasonable context abstraction
2Chinese characters and English words are typically considered at the same processing level in pre-trained language models, so we refer to Chinese characters as "Chinese word" in this paper to avoid ambiguity.
with the coupled word-level semantics, and (3) interpretable modeling for the given text with the context-relative distance. W2CSpace serves as the key component in the backbone of our language modeling method, which realizes a decisive transparency increasing compared with the modelagnostic and implicitly informed methods. The structure of our method is illustrated in Figure 1. Specifically, our main contributions can be summarized as follows:
• Word-level semantics in W2CSpace is originated from BERT immediate representation with the help of a mapping network, preserving the language modeling ability of the deep neural methods (Section 2.1.1).
• An associative matrix sampled from associative knowledge network (AKN, Li et al., 2022c) is introduced for aligning with the semantic distances (Section 2.1.2 and 2.1.3).
• Based on the linguistic concept, the contexts are abstracted using k-means clustering on neighboring word elements. (Section 2.2.1).
• The context-relative distance, computed between the input text and the context clusters in W2CSpace, serves as the semantic feature to describe the text semantics (Section 2.3.1).
• The experiments on different NLP tasks demonstrate the effectiveness of W2CSpace. Additionally, an interpretable analysis is designed to verify the interpretability of the our method (Section 3).

heading: 2 Methodology

heading: 2.1 Initialization of W2CSpace
text: Since current researchers opt for high-dimensional representations in their methods, it is widely believed that over-parameterization advances language modeling performance. With respect to the standard language modeling process, the words in the given text are modeled based on their meaning under different contexts. Regardless of the attributes of the words themselves, greater representation dimensions enable better performance in distinguishing the words with similar semantics. However, different from the deep neural methods, the linguistic attributes of the text, such as co-occurrence rules, word shape, and so on, serve
as the basis for NLP tasks, which fit the understanding and deduction processes of humans.
In order to unify the high-dimensional representations and interpretable statistical knowledge, we design a mapping network to transfer the semantic representation from BERT encoder to lowdimensional elements in W2CSpace and introduce a statistical alignment with AKN during the training of the mapping network. Within the above processing, the mapped elements are distributed in W2CSpace according to their corresponding wordlevel semantics.

heading: 2.1.1 Representation Mapping from BERT
text: The mapping network is a neural network with a backbone of a convolution network, which enables dimension reduction process. The mapped elements in smaller dimensions are regarded as the coordinates of the word-level semantics in W2CSpace. Besides, the introduction of the convolution network is able preserve semantic information from BERT for maintain the performance advantages of pre-trained models. For BERT representations FB of given sentence S = {x1, x2, . . . , xd}, the corresponding elements C = {c1, c2, . . . , cd} in W2CSpace are obtained accord-
ing to:
C = Tanh{LN[Convs(FB) + Res(FB)]} (1)
where FB ∈ Rn×h and C ∈ Rn×k, h is the hidden sizes of BERT encoder and k is the coordinate size of W2CSpace. Tanh(·) and LN(·) are Tanh and layer normalization operations. A convolution network Convs(·) with different filter sizes added a residual connection Res(·) is used in the mapping network.

heading: 2.1.2 Statistical Alignment with AKN
text: AKN, a statistical network based on phrase cooccurrence, is introduced to sample to an associative matrix reflected the associative relations among the given sentence, which is also adopted in previous work (AxBERT, Wang et al., 2023). Within the original AKN is conducted on phrase-level, we modify AKN to word-level to fit the processing of BERT and opt for the construction and sampling methods of the AKN (A) and associative matrix (MS) similar to AxBERT as:
Ai,j = ∏
sent
SR ∑
sent
1
distance⟨i,j⟩ (2)
MSi,j = σ Ai,j
Avg(Ȧi:) − 0.5 (3)
where A ∈ Rv×v, MS ∈ Rd×d, v is the length of the word list, d is the length of the given sentence, and σ(·) and Avg(·) are functions of sigmoid and average. For the word pair ⟨i, j⟩, distance⟨i,j⟩ = |i− j| is the word-distance between i-th and j-th word in sentence. Ȧi: is the association score of i-th word under current sentence, and MSi: is the i-th row of MS . 3
Since we compute the cosine distances matrix in the given sentence, the associative matrix is aligned with the word-level distance matrix to integrate the statistical logic into W2CSpace. We introduce a mean square indicator IMS to indicate the alignment result. Specifically, for the word pair ⟨i, j⟩, the indicator IMSi,j is defined as:
IMSi,j = MnSqr[CosDis(ci, cj),MSi,j ] (4)
where IMS ∈ Rn×n and n is the length of the given sentence. MnSqr(·) and CosDis(·, ·) are the mean square and cosine distance functions.

heading: 2.1.3 Training of Mapping Network
text: The objective LM of the mapping network is composed of LMS and LRec, which correspond to the mean square error loss and the reconstruction loss. With respect to the statistical alignment process, the mapping network is trained under the alignment objective LM . Besides, we introduce a reconstruction loss by reversing the structure of mapping network to reconstruct the representation of BERT immediate feature FB . The objective of the mapping network is calculated according to:
LM = LMS + LRec (5)
LMS = Mean(IMS) (6)
LRec = MAE(C − FB) (7) where Mean(·) is the average function, MAE(·) is the mean absolute error operation (Choi et al., 2018). The introduction of reconstruction loss aims to guarantee that the mapped word elements preserve the semantics of BERT representations.

heading: 2.2 Abstraction of Context-level semantics
text: Humans are able to recognize emotion from language, action, and so on (Barrett et al., 2007). Specifically, in linguistics, humans recognize emotion with the context in the given sentences. However, humans are able to feel the emotion rather than explicitly describe it, because that context is
3The same shrink rate SR = 0.95 as AxBERT.
an abstract concept in linguistics and is hard to simply quantify.
While the sentence is composed of words, the corresponding context is established from the word semantics, which can be realized in W2CSpace. Therefore, we employ the k-means clustering on the word-level semantics to abstract the context semantics. The context is able to be extracted based on the common semantics among the words located adjacently in W2CSpace.

heading: 2.2.1 Context Clustering Based on Word Semantics
text: With the help of the mapping network and the statistical alignment, the word elements are reasonably distributed in W2CSpace according to their semantics, where the neighbors in W2CSpace refer to similar semantics. k-means clustering is an algorithm based on the distances (Hartigan and Wong, 1979), which is introduced to abstract the word semantics to k classes according to their semantics (distance). The clustering process is defined as:
X{xi|Context(xi)} = KMCosDis(c1, c2, . . . , cn) (8) where xi is the i-th classes of context and KMCosDis(·) is the k-means clustering algorithm based on cosine distance. The word semantics in W2CSpace is clustered into k classes, which represents the process of abstraction of context.

heading: 2.2.2 Reasonable Cluster Merging for Context Clustering
text: When the clustering is executed, especially in kmeans algorithm, the appropriate k number is hard to determine (Hamerly and Elkan, 2003). As for the clustering process for context, the number k additionally represents the types of the contexts in W2CSpace. A small number of k possibly decreases the performance of language modeling with a rough context environment, but the large number is contrary to the logic of humans as humans cannot precisely distinguish the detailed emotion behind the text as a machine does. Besides, the large number of k increase the time-costing of the language modeling process. Choosing the right number k relate to the reasonability of the context subspace.
Serving as the part of W2CSpace, the context clusters are used for language modeling. Therefore, we introduce a merge matrix to the top of the clustering results, which is optimized under the
downstream task. With the guidance of the downstream tasks, the merge matrix dynamically adjusts context semantics for the inter-communication between different context clusters, which reflects a gradual clustering process and realizes a reasonable context clusters for the downstream tasks. The clustering is able to be balanced with the computation process of the merging is defined as:
X̄{x̄i|Context(x̄i)} = MM × [x1, x2, . . . , xk] (9) where x̄i indicates the context semantics after merging, MMerge is the merge matrix and MMerge ∈ Rn×k. n is the size of coordinates in W2CSpace and k is the presetting number of the clustering. 4

heading: 2.3 Interpretable Language Modeling via W2CSpace
text: As the standard methodology for language modeling in deep neural methods, the semantic representation is gradually modeled through containing neural networks, which is a simulation of the neural processing of the brain. However, the structural simulation is unable to realize the interpretable on logical level. The decision process through neural networks still remains in the black box.
From the perspective of humans, emotion recognition is significant in daily (Barrett et al., 2011), which is also an important ability for machines to interact with humans (Kosti et al., 2017). By simulating the recognition process of humans, we introduce a context-relative distance computed between the given text and the contexts in W2CSpace, which enables the interpretable language modeling process with the cooperation of the semantics on word- and context-level.

heading: 2.3.1 Computation of Context-Relative Distance
text: The context-relative distance based on the cosine distance is also adopted in Formula 4. Compared with the euclidean distance, the equation of the cosine distance is more efficient in time-costing and storage. The context-relative distance D is computed according to:
D = CosDis(X̄,C) (10)
where X̄ is the context clusters, C is the mapped word elements from BERT encoder, D is the context-relative distance and D ∈ Rd×k. n is the
4However, the number k will influence the performance of language modeling, which is discussed in Section 3.4.
word length of the given text and k is the number of the context clusters.

heading: 2.3.2 Training of Interpretable Language Modeling Method
text: The context-relative distance is able to directly connect with the downstream classifier, which is similar to the traditional encoding structure. The interpretable language modeling component is regarded as the standard BERT-based encoder for downstream tasks, where the standard objectives in transformer package5 are employed.

heading: 3 Experiments

heading: 3.1 Experimental Settings
text: We conduct our work on NVIDIA Tesla A100 and AMD EPYC 7742 64-Core CPU. During the interpretable language modeling process, BERT-baseChinese pre-trained model is used, and the original parameters are opted 6. Additionally, we opt for the rate of 0.3 for all the dropout layers, a learning rate of 2e-5 for 10-epoch-training of BERT encoder in Fig. 1a, a learning rate of 1e-5 for 3-epoch-training of the mapping network in Fig. 1b.

heading: 3.2 Datasets
text: The detailed information of the datasets is exhibited in Table 1. CLUE 7, an open-ended, communitydriven project, is the most authoritative Chinese natural language understanding benchmark (Xu et al., 2020), where the news dataset is used to initialize associative knowledge network; SIGHAN15 is benchmark for traditional Chinese spelling check evaluation (Tseng et al., 2015), which is widely adopted in simplified Chinese spelling check evaluation by converting to simplified Chinese (Cheng et al., 2020; Liu et al., 2021); Hybird is a massive dataset for Chinese spelling correction (Wang et al., 2018), which is used the training of the correction methods (Wang et al., 2019; Cheng et al., 2020); Weibo 8 and ChnSenti 9 sentiment dataset is constructed with the comments from the largest Chinese social community (Sina Weibo) and Chinese hotel reservation websites, which are adopted in the previous work for sentiment classification evaluation (Li et al., 2020, 2022b).
5https://pytorch.org/hub/huggingface_pytorchtransformers/
6https://huggingface.co/bert-base-chinese 7https://github.com/CLUEbenchmark/CLUE 8https://github.com/pengming617/bert_classification 9https://github.com/SophonPlus/ChineseNlpCorpus/

heading: 3.3 Comparison Approaches
text: We fine-tune our method with standard classifiers of BERT Masked LM and sequence classification for spelling correction and sentiment classification with training of 10 epoch, 1e-5 learning rate and 3 epoch, 1e-5 learning rate. SoftMask is a BERT-based spelling correction method with a soft-mask generator, where the softmasked strategy is similar to the concept of error detection (Zhang et al., 2020). FASPell conducts the Seq2Seq prediction by incorporating BERT with additional visual and phonology features (Hong et al., 2019). SpellGCN incorporates BERT and the graph convolutional network initialized with phonological and visual similarity knowledge for Chinese spelling correction (Cheng et al., 2020). PLOME integrates the phonological and visual similarity knowledge into a pre-trained masked language model with a large pre-train corpus consisted of one million Chinese Wikipedia pages. And it is the SOTA in previous work (Liu et al., 2021). HeadFilt is an adaptable filter for Chinese spell check, which introduce a hierarchical embedding according to the pronunciation similarity and morphological similarity (Nguyen et al., 2021). HLG is a Chinese pre-trained model for word representation by aligning the word-level attention with the word-level distribution with a devised pooling mechanism (Li et al., 2020). MWA introduces a heterogeneous linguistics graph to pre-trained language model. The graph-based structure integrates the linguistics knowledge in the neural network and achieves the SOTA performance in language modeling (Li et al., 2022b).
HLG and MWA is employed on various pretrained language model, such as vanilla BERT (Devlin et al., 2019), BERT-wwm (Cui et al., 2021), and ERNIE (Sun et al., 2019). We use the evaluation results on different pre-trained language models in their original paper.

heading: 3.4 Main Experiments
text: The efficacy of our interpretable language modeling method is evaluated on different tasks, including Chinese spelling correction and sentiment classification. Chinese spelling correction requires the advanced language model for token classification, where every word in the given text is classified into a single class. And the sequence classification is needed in Chinese sentiment classification, where the given text is classified into positive and negative sentiment. The token and sequence classification tasks are able to cover most of the current classification scenario, which enable the efficient demonstration of the language modeling performance of our method.

heading: 3.4.1 Results of Chinese Spelling Correction
text: Similar with the past works (Cheng et al., 2020; Liu et al., 2021), the correction experiment is employed on word- and sentence-level. Within a more comprehensive perspective, the sentence-level evaluation is wider adopted and more convincing, so we use the same evaluation matrix with the past works (Liu et al., 2021; Nguyen et al., 2021).
As shown in Table 2, the evaluation on wordand sentence-level composed of different indexes, including detection precision (DP), correction precision (CP), detection recall (DR), correction recall (CR), detection F1 score (DF1) and correction F1 score (CF1). Besides, we assess the influence of the number choosing of n (the size of the coordinates in W2CSpace) and k (the context number in k-means clustering algorithm).
From the correction results in Table 2, our method outperforms the baselines on both wordand sentence-level. Specifically, at sentence-level, our method respectively advances 0.2 and 0.7 points in DF1 and CF1; at word-level, our method is not able to achieve comparable performance than PLOME (Liu et al., 2021), but advances than the SpellGCN (Cheng et al., 2020) with a 0.5 and 0.4 point improvement, and we think the massive train-
ing dataset of PLOME significantly enhance the correction performance, where the size of dataset composed of per-train and fine-tune dataset is 600 times larger than ours.
With respect to the changes of the parameters of W2CSpace, the correction performance are various. W2CSpace cannot achieves best performance on a specific parameter combination, but the overall performance is comparable attractive. Besides, we notice that the W2CSpace with a larger size of the coordinates performs better than the smaller one, where the larger W2CSpace is advanced in 8 indexes. The advantages of enlarging the number k are not obvious, where W2CSpace with n = 100 and k = 2000 is the most advanced combination for the correction task. However, W2CSpace with k = 1500 and k = 3000 is also a good choice for correction. Generally, we think the introduction of the merge matrix balances the difference of k.

heading: 3.4.2 Results of Chinese Sentiment
text: Classification
We evaluate the sentiment classification performance with the classification accuracy. In Table 3, the results of the sentiment classification are illustrated. Specifically, even the other pre-training models partly perform better than BERT, our method achieves improvements on both ChnSenti and Weibo100K datasets as 0.04 and 0.09 points.
For the choice of n and k numbers, our method achieves advanced performance with the combi-
†While the other comparison methods are trained on HybirdSet, PLOME additionally pre-trained on a 600 times larger dataset compared with HybirdSet. We uniquely highlight the advanced index of PLOME with bold italic font.
nation of n = 100 and k = 800 for sentiment classification, which is different from the correction experiment. Besides, similar to the tendency in the correction experiment, a larger n number enables a small improvement in performance.

heading: 3.5 Interpretable Analysis
text: The interpretable machine learning methods are considered with a transparent decision process (Molnar et al., 2020; Verhagen et al., 2021). However, the method interpretability is hard to define. The rule-based approaches, widely regarded as the interpretable methods, are especially advanced in the controllability that is one of the most significant characteristics of interpretability (Lee et al., 2017; Tian et al., 2019; Tripathy et al., 2020).
The context-relative distance, the key in our interpretable language modeling process, originated from the relativity between the input sentences and the context in W2CSpace. Therefore, we design an analysis focused on the interpretability of context-relative distance by means of Chinese sentiment classification task. The procedure of interpretable analysis is exhibited in Figure 2. Ideally, the context-relative distance correlates with sentiment, e.g., the shorter distance indicates stronger relativity between the context and the labeled sentiment. If sentiment prediction results change correspondingly after reversing the context space, it convincingly shows that (1) the interpretable knowledge from AKN is integrated into W2CSpace, (2) the feature mapping and context abstraction pro-
cesses are conducted reasonably, and (3) the distance within W2CSpace is interpretable and associated with the emotion conveyed by the input text.
The interpretable analysis result on Weibo100K is exhibited in Table 4. OA is the original accuracy, CA is the sentiment classification accuracy after modification, and RA is the reversing accuracy for the sentences that successfully reverse the sentiment labels after modification. Because language modeling mainly relies on context semantics, the predicted sentiments after modification should be reversed compared with the original prediction. From the interpretable results, the values of RA approximate to 100%, which indicates that the predicted sentiments are mostly reversed and matches our expectation. The predictable changes of the accuracy reflects the controllability of our method and the interpretability of W2CSpace. Besides, from the perspective of model structure, the ideal transparent method enables a completely controllable decision process from input to output. And in our method, even though some parts are still in black-box, but RA reflects the interpretability between the decision processes from W2CSpace to the output and from input to the output. The interpretable contexts are consistent with the linguistics logic in input articles and serves as the agent to cooperate with the articles to realize the controllable process. The value of RA does not directly indicate the interpretability of our method, but but the more approximate to 1, the more semantically explainable of W2CSpace and its context.

heading: 4 Conclusion
text: An interpretable language model is proposed by constructing a word-context-coupled space in this study. Within W2CSpace, (1) the uninterpretable neural representation in BERT is regulated by interpretable associative knowledge relations, (2) an intermediate representation space with reasonable interpretable semantics is designed, (3) an interpretable semantic feature is introduced based on intermediate representation space for downstream classifiers. Obviously, the above strategies bring a strong generalization ability for the interpretable pre-trained language modeling process. Besides, for the potential risk preventing, the interpretable machine learning method is introduced for migrating the adverse affects from the black-box structure. Moreover, in our method, the controllable decision process realize the regulation for the illegal language inputting by controlling the related context, and the strong cooperation between pretrained models and W2CSpace can protect the parameter privacy from the data stealing.
Nevertheless, W2CSpace is unable to directly handle high-level semantics, including sentences, paragraphs, and so on. Even the word-level language models act as the mainstream methods in NLP, the above limitation should be further considered in the future. Besides, restricted by our knowledge and efforts, the main experiments cannot cover all common tasks and all pre-trained models in NLP. Relatively, the token- and sequence-level classifications have demonstrated attractive experimental performance on most NLP tasks. Next, we also plan to extend W2CSpace to more NLP tasks and find its more specific value.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62272201, and 61872166; in part by the
Six Talent Peaks Project of Jiangsu Province under Grant 2019 XYDXX-161.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In Section 3.4, we discuss the interpretability of our method from the perspectives of model structure and controllability; In the conclusion, we discuss the limitation of our experiments.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Conclusion.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract; Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The code of our work is available and will be presented after acceptance.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We use the open-sourced dataset, and cite them in Section 3.2.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: The license of our code will be presented together with our code after acceptance.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We conduct our experiments on the same task as the existing works, which is completely applicapable.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use the dataset which widely adopted in past works, which is checked before.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The datasets are constructed for Chinese text evaluation, which is claimed in Section 3.2.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 1.
C 3 Did you run computational experiments? Section 3.4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3.1; Section 3.2.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3.4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.1.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Even the word-level language models act as the mainstream methods in NLP, the above limitation should be further considered in the future. Besides, restricted by our knowledge and efforts, the main experiments cannot cover all common tasks and all pre-trained models in NLP. Relatively, the token- and sequence-level classifications have demonstrated attractive experimental performance on most NLP tasks. Next, we also plan to extend W2CSpace to more NLP tasks and find its more specific value