Improving Language Models’ Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary
Authors: Myeongjun Erik Jang
Thomas Lukasiewicz
Venue: EMNLP 2023 Main (2023)
Keywords: Language Model
Consistency
Conceptual Role Theory
TL;DR: This paper propose a method that learns precise meaning of language through conceptual role theory

Abstract: The non-humanlike behaviour of contemporary pre-trained language models (PLMs) is a leading cause undermining their trustworthiness. 
A striking phenomenon of such faulty behaviours is the generation of inconsistent predictions, which produces logically contradictory results, such as generating different predictions for texts delivering the same meaning or violating logical properties. Previous studies exploited data augmentation or implemented specialised loss functions to alleviate the issue. However, their usage is limited, because they consume expensive training resources for large-sized PLMs and can only handle a certain consistency type. To this end, we propose a practical approach that alleviates the inconsistent behaviour issue by fundamentally improving  PLMs' meaning awareness. Based on the conceptual role theory, our method allows PLMs to capture accurate meaning by learning precise interrelationships between concepts from word-definition pairs in a dictionary. Next, we propose an efficient parameter integration technique that updates only a few additional parameters to combine the learned interrelationship with PLMs' pre-trained knowledge. Our experimental results reveal that the approach can concurrently improve multiple types of consistency, enables efficient knowledge integration, and easily applies to other languages.

Improving Language Models' Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary
Myeongjun Erik Jang' Thomas Lukasiewicz
1 Department of Computer Science, University of Oxford, UK
Institute of Logic and Compulation, Vienna University of Technology, Austria
myeonggun.jang www.://wnnnn....ccc...o ac..k Noomas.luksseewic..c. tuwien.accat
Abstract
The suggestive evidence revealing PLMs' inca- pability of behaviours (Mitchell ct al., 2022). Unlike humans, they exhibit logically contradictory behaviours many respects (Jang et al., 2022a), such as gener- ating different predictions for semantically identi- cal texts (Ravichander et al., 2020; Elazar et al., 2021) violating logical properties, e.g., the logi- cal negation property (Kassner and Schütze, 2020; Ettinger, 2020; Asai and Hajishirzi, 2020; Jang et al., 2022b), symmetry (Wang et al., 2019c: Li et al., 2019; Kumar and Joshi, 2022), or transitiv- ity (Asai and Hajishirzi, 2020; Lin and Ng. 2022). Such inconsistent behaviours undermine models' trustworthiness and limit their practical usefulness, particularly in risk-sensitive fields where even a minor undesirable action could lead to catastrophic consequences.
is inconsistent
The non-humanlike behaviour of contempo- rary pre-trained language models (PLMs) is a leading cause undermining their trustworthi- ness. A striking phenomenon of such faulty hehaviours is the generation of inconsistent pre- dictions, which produces logically contradic- tory results, such as generating different predic- tions for texts delivering the same meaning or violating logical properties. Previous studies exploited data augmentation implemented specialised loss functions to alleviate the issue. However, their usage is limited, because they consume expensive training resources for large- sized PLMs and can only handle a certain con- sistency type. To this end, we propose a prac- tical approach that alleviates the inconsistent behaviour issue by fundamentally improving PLMM' meaning awareness. Based the con- ceptual role theory, our method allows PLMs to capture accurate meaning by learning pre- cise interrelationships between concepts from word-definttion pairs in a dictionary. Next, we propose an efficient parameter integration tech- nique that updates only few additional param- eters to combine the learned interrelationship with PLMs' pre-trained knowledge. Our ex- perimental results reveal that the approach can concurrently improve multiple types of consis- tency, enables efficient knowledge integration, and easily applies to other languages.
Previous works tried to alleviate the incon- sistency issues through data augmentation (Ray et al., 2019) or introducing specialised loss func- tions (Elazar et al., 2021; Kim et al., 2021). These approaches, however, have limitations, as they can only address a specific type of consistency and are unable to handle others. For instance, consis- tency regularisation loss, which enforces a model's predictive distribution on the original and its para- phrased inputs be similar (Elazar al., 2021), cannot resolve the violation of the logical negation property or symmetry. Morcover, these approaches require expensive resources, which practically inefficient, i.c., using a lot of computing resources for training large PLMs is impractical, and col- lecting auxiliary data (e.g., paraphrased texts) is demanding for low-resource languages.
1 Introduction
AI systems that behave like humans can be more reliable and trustworthy (De Visser et al., 2016; Jung et al., 2019). However, despite striking ad- vances in various language tasks that modern PLMS have made, recent evidence of their oonhhumanlike behaviours (Hossain al., 2020; Kassner and Schüitze, 2020; Hosseini et al., 2021; Gupta al.. 2021; Sinha et al., 2021b) casts doubts on their trustworthiness, suggesting that PLMs' language understanding ability is far below that of humans.
To this end, propose comprchensive solu- tion that can concurrently enhance multiple types of consistency in resource-efficient manner. Based on previous studies that demonstrated the limita- tion of the distributional hypothesis (a fundamental theory underlying modern PLMs) in learning the

---

meaning of natural language (Sinha al., 2021b; Jang et al., 2022b), we hypothesise that a leading cause contributing to the inconsistent behaviour is the incapability of PLMs to capture the precise meaning of language. Thus, our proposed approach is centred around mitigating PLMs' deficiencies by leveraging an auxiliary model that is trained to imitate human cognition, thereby enhancing the awareness of semantic meaning. Our underpinning assumption is the conceptual role theory, a con- vincing theory that accounts for human cognition. Based on the theory, by which the interrelation- ship between concepts predominantly determines a word's meaning (Deacon, 1998; Santoro et al.. 2021; Piantadosi and Hill, 2022), our approach first learns abundant symbol meanings by track- ing more precise interconnections through word- definition pairs in a dictionary. Next, we propose a training-efficient parameter integration method that combines the learned parameters with those of ex- isting PLMs. This enables the fine-tuning process to update only a small number of parameters. Our contributions can be briefly summarised as follows: 1. We verify that enhancing PLMs' meaning awareness can improve multiple types of con- sistency concurrently.
the conceptual role theory (Piantadosi and Hill, 2022). PLMs are the contemporary NLP tech- niques that exploit the distributional hypothesis as their learning objective, i.e., masked language modelling (Sinha et al., 2021a)' This hypothesis assumes that semantically analogous words will appear in similar contexts (Harris, 1954). Hence, PLMs learn to define the words' meaning through other words that co-occur in similar contexts. That is, they capture the interrelationship of concepts frequently appearing in analogous contexts. Al- though the distributional hypothesis is an efficient assumption allowing self-supervised training, it cannot perfectly capture the interrelation between concepts because the concepts can deliver different meanings even though they often appear in simi- lar contexts. For example, it is difficult to capture semantic antonymy through the distributional hy- pothesis (Jang al., 2022b).
To this end, we aimed to improve PLMs' under- standing of meaning by making them learn more precise interrelationships. Recent studies revealed that the training data play a critical role in deciding the inductive bias rather than the model's 's structure or learning objective (Furrer et al., 2020; Wang et al., 2022; Jang et al., 2022a). Therefore, we designed learning task that provides training in- stances where concept and other closely intercon- nected concepts are presented together to a model having a language modelling objective. To achieve this, we used worl-definition pairs from a dictio- nary, because definition is composition of words explaining the target word, and thereby, it has been the most commonly used and very effective tool for vocabulary learning, especially for second and foreign language learners (Takahashi, 2012: Zhang et al., 2020). A target word and its definitions were concatenated as a single text and used as a training instance for language modelling, allow- ing model to determine word's meaning based on highly related concepts rather than those that appear in similar contexts. For example, when it comes the words "happy" and "unhappy", dis- tributional models normally generate high similar- ity, because they are both emotional expressions and appear similar contexts frequently 2. How- ever, our proposed task enables capturing precise interconnection between concepts: "unhappy" with "not happy", "sad", and "not pleased". An exam-
2. We propose a training-efficiee parameter in- tegration, which allows practical fine-tuning for large-sized PLMs.
3. The proposed approach is readily applicable to low-resource languages.
4. The results suggest that the proposed approach allows further consistency improvements in simple but effective way by aggregating the parameters of PLMs possessing different in- ductive biases.
2 Methodology
2.1 Learning Conceptual Roles from Dictionary
an effort to imitate the human language un- derstanding process, we employed the conceptual role theory, a convincing theory assuming that is a concept's is role in some greater mental theory that primarily determines its meaning (Piantadosi and Hill, 2022), where the key is the interrelation between concepts (Deacon, 1998; Santoro et al., 2021). For example, the meaning of "water" can be defined by other interlinked concepts like "liquid", "without smell", "hydrogen", and "oxygen".
'See Appendix for details.
FFor instance, Skip-gram vectors (Mikolov et al., 2013) trained with English Wikipedia: [link].
The recent success of large-size PLMs supports

---

ple of our training data is provided in Table 6 in Appendix C.
As a result, W new becomes the addition of fixed matrices Wp, W and a learned matrix AW, Rdxi,
We introduced the intermediate training tech- nique (Phang et al., 2018; Wang et al., 2019a; Liu et al., 2019a; Pruksachatkun et al., 2020; Vu et al., 2020) that first trains PLMs on an intermediate task and uses it for other downstream tasks. Hence, we retrained PLMs on our new dataset and named it conceptual role model (CRM). A leading cause for using the intermediate training that the number of word word-definition pairs was not sufficient enough to train large PLMs from scratch, and their weights can be used as good initial values. The CRM has practical advantages in that (1) it can be applied to any PLM trained with language modelling ob- jectives, and (2) it is readily applicable to other languages due to the relative ease of collecting dic- tionary data.
Updating the whole parameters of AW, , is ex- acty the same as the fine-tuning with the pre- trained weights of Wp + We. which is impracti- cal particularly for large-sized PLMs. Also, fine- tuning contains the risk of catastrophic forgetting, as studied in many previous works (Pruksachatkun et al., 2020; Wallat et al., 2020). To compen- sate for this, we introduce a low-rank adaptation technique (Hu et al.. 2022) for the parameter in- tegration, which fixes the pre-trained weights and updates only a few number of parameters based on PLMs intrinsic dimension (Aghajanyan et al., 2021). Specifically,
transform AW, to multi- plication of two matrices A Rd and B R where r min(d, I):
2.2 Training-efficient Parameter Integration
W new W + W. + AB.
(4)
We propose parameter integration method to ef- fectively incorporate the previous knowledge ob- tained by PLM with enhanced meaning awareness CRM.
As consequence, the number of trainable pa- rameters is considerably reduced, enabling effi- cient fine-tuning for large-sized PLMs. Also, com- pared to other adapter modules (Houlsby et al., 2019; Pfeiffer et al., 2021) )) that introduce additional adapter modules between layers, the approach only adds up AB without increasing the number of pa- rameters. As a result, avoids the use of any addi- tional time or resources during the inference phase, which enables practical inference (Hu et al., 2022).
2.2.1 Problem Statement
Let Wp E Raxt and We E Rddx be the parameter matrices of the PLM and CRM, respectively, Note that the PLM and CRM share the same model ar- chitecture, i.e., having a parameter matrix of the same size. We aim to learn Wnew E R i.e., an integrated parameter matrix, from Wp and W. dur- ing fine-tuning. Thus, the process can be defined follows:
Aggregating W, and W.. The addition of W, and Wa in Eq. causes the amplification of the weight scale, which prevents from using searching training hyperparameters based on val- ues used in prior studies. Thus, we used the simple averaging aggregation method (Wortsman et al., 2022):
W W.
W new Wo
(1)
where Wo E Rddx2d is learnable parameter matrix while Wp and W. remain fixed during fine-tuning. By decomposing W, into W, E Rdxd and W. E Raxd, Eq. can be rewritten as follows:
W + W. + A'B'.
W new
(5)
2
W
W new
[W1 W2]
W
The reason behind leveraging the simple aver- aging technique to show the efficacy of CRM, i.e., achieving remarkable improvements in consis- teney through the most straightforward aggregation method. However, alternative aggregation meth- ods, such as Fisher-Weight Average (Matena and Raffel, 2022) or RegMean (Jin et al., 2023), can also be employed.
(2)
W, Wp + W.W
W
W
where W' and W denote the matrices after fine- tuning. Finally, decomposing W' with fixed (W) and updated part (AW) allows us to reformulate Eq. 2 as follows:
W New (Wp + AWW) + (W. + AW.)
Additional Knowledge Add-up. Note that the aggregation further allows us to integrate the
= W, + W. + AW,

---

weights of any other models if they share the same structure with the PLM and CRM. Thus, Eq. 5 can be rewritten as follows:
2019b) for measuring basic NLU task performance and BECEL (Jang et al., 2022a) for evaluating the consistency.
W New = +2
W. W AE(D.C,)
Σ
(6)
3.2.1 Experiments on the GLUE Benchmark We trained the proposed model to GLUE down- stream tasks to ascertain whether the approach per- forms well on widely used NLU tasks. The models that only leverage PLM weights (i.e., zero weights for the CRM) were also trained for comparison. We repeated the experiments five times for tasks with training set having more than 100K data points (e.g., MNLI, QNLI, QQP, and SST2) and ten times for the other tasks. Detailed training hyperparame- ters are presented Appendix B. The evaluation metric is Matthew's correlation for COLA and ac- curacy for the others. The results of the validation set are summarised in Table 1. For RoBERTa-base our approach performs better than using only the PLM weights in three tasks, i.e., COLA, MRPC, and RTE, and shows a comparable performance in other tasks. For RoBERTa-large, two approaches produce a similar performance most tasks apart from COLA, where our approach performs sig- nificantly better. It is captivating that integrating CRM weights can significantly improve the per- formance of the COLA task, which is designed to check grammatical errors, i.c., requires linguistic knowledge. Also, the improvements are more sig- nificant in RoBERTa base, where the PLM stores less pre-trained knowledge. The experimental re- sults suggest that CRM weights provide more abun- dant linguistic knowledge, which can help enhance the understanding of text meaning.
where S is the set of additional PLMs' weights that are going to be added. Note that, as the parameters are integrated through an addition, no additional training/inference resources required during fine-tuning, which is computationally beneficial.
Applying the approach to PLMs. As modern PLMs have transformer (Vaswani et al., 2017) backbone architecture, we followed Hu et al. (2022) to apply the low-rank adaptation technique, i.e., limiting its usage self-attention weights and excluding MLP weights from the scope. We also used the same hyperparameters found by Hu al. (2022), i.c., for a hidden representation æ, A 2 is scaled by & respectively.
y. where a and r are set to 16 and 8,
3 Experiments
3.1 Training the CRM model
Dataset. To collect word-defanition pairs, we first collected English words from the List of English Words and Wikipedia Word Frequency data 4, and extracted word meanings through the Wikipedia English Dictionary API. As result, 455K word- definition pairs were collected for training the CRM model.
Model and training details. We used RoBERTa (Liu et al., 2019b) as a backbone model for our experiments. Both base- and large-size models were used to observe the influence of the model's size. The base- and large-size models were trained for 500K and 700K steps, respectively. We used the AdamW optimiser (Loshchilov and Hutter, 2017) with a learning rate of Ie-6 and a linear learning rate scheduler decaying from le-2. The 31. 32, and e values were set to 0.9, 0.98, and 1e-6, respectively. Four GeForce GTX TITAN XP GPUs were used for the training of the CRMs.
3.2.2 Experiments on the BECEL Dataset
Next, we assessed our approach on the BECEL dataset that allows the evaluation of multiple con- sistency types over various downstream tasks. For each task, model was fine-tuned based on the original training set and evaluated on the test sets, specially designed to measure various consistency types. We refer to the original paper for more de- tailed information. We did not consider the additive consistency in this work, because it was reported that most PLMs performed well in the additive consistency (Jang et al., 2022a). Two downstream tasks, SST and AG-News, were omitted from our evaluation scope, because they contain evaluation sets for semantic consistency only. To ascertain the influence the CRM and parameter integration technique, we investigated four evaluation scenar-
3.2 Evaluation on the English Dataset
We evaluated the proposed approach on two ex- isting benchmark datasets: GLUE (Wang et al.,
"[Link], accessed 20th January, 2023.
"[Link], accessed 20th January, 2023.

---

Model
COLAA
MRPC
RTH
QQP
SST2
MNII QNLI Avg
P Ponly
62.2
89.1
77.1
89.8
94.2
87.0
92.6
84.5
RoB
P+C
63.2*
89.5
79.1*
89.8
94.5
87.0
92.6
85.1
P Ponny
67.3
90.5
86.6
90.9
95.9
90.4
94.8
88.0
RoB,
P4C
68.4*
90.5
86.9
91.11
96.0
90.4
94.8
88.3
Table 1: Average GLUE performance of RoBERTTa-ass base (RoBg) 8) and RoBERTa-large (RoBr) models when the parameter integration is applied. The best performance is in bold. The figures of the proposed approach (P+C) show a significant difference compared to our counterpart (P only) with p-value < 0.05 (*) and p-value < 0.1 (f) using the t-test.
BoolQ
MRPC
RTE
SNLI
WiC
Model
Tsem
Thep
Taem
Tney
Taym
Teem
Thep
Taym
Taem
They
Tagm
Tere
Taem
Taym
Tten
FT
P-only 11.3 43.2
7.4
82.2
2.7
11.9
22.9
1.0
10.8
8.3
12.0
3.8
14.8
4.0
13.6
(125M)
P+C 10.7f 35.4*
6.7
78.91
2.7
11.4
18.3t
0.5
10.6
7.4
11.2f
3.7
13.8
4.2
12.9
RoBg
PL
P-only 12.2 59.9
10.0
82.2
6.5
12.8
30.9
1.4
9.8
8.3
11.8
4.1
15.1
3.5
13.9
(0.8M) P+C 11,4* 50.7*
7,8
71.1"
6.3
11.9f
26.4f
1.7
10.0
7.3
12.1
4.0
11.9
3.5
13.5
FT P-only 8.4 45.9
6.2
78.9
2.6
7.8
7.0
2.1
9.2
7.8
9.2
3.3
10.6
5.9
11.5
C-only 8.1 45.4
6.2
72.8
2.2
7.2*
7.7
1.4f
9.2
6.0
9.1
3.2
11.0
4.7
10.8
(355M)
P+C 8.0
45.8
6.2
75.2
2.0
6.2
7.0
1.3
9.0
5.9*
8.8
3.2
8.91
4.8
9.61
RoB,
P-only 9.6 43.3
6.8
77.3
4.7
7.7
7.0
2.2
8.6
7.3
7.3
3.2
11.7
3.0
10.6
Pl
C-only 8,1*
41.9
6.4
74.8
2.0
6.6
7.5
2.1
8.8
5.3*
7.3
3.2
12.2
3,8
9.5t
(2.6M)
P+C
8.7*
37.8*
6.7
77.0
4.4
6.5t
7.2
1.6
8.6
5.2
7.1
3.0*
10.9
2.8
9.5t
Table 2: Average BECEL performance of RoBERTa-base (RoBg) and RoBERTa- (RoB2). Tsems Treg- Taym-
and Teen denote semantic, negational, symmetric, and transitive inconsistency, respectively. All evaluation metrics are lower the better. For each evaluation scenario, the best figure is highlighted in bold. The values of P+C and C-only show a significant difference compared to P-only with v-vuue < 0.05 (*) and p-value < 0.1 (t) using the t-test.
ios determined by (1) whether to only use the PLM (P-only) exploit the CRM (P+C), and (2) fine- tuning the whole parameters (FT) or applying the parameter integration (PI). Regarding RoBERTa- large models, we additionally introduced models that exclusively employ CRM (C-only) for more detailed analysis. For clarification, P+C with FT de- notes model that fine-tunes the whole parameters where the initial weights are set to the aggregation of the PLM and CRM weights, and P-only with PI refers to a model that only introduces a low-rank adaptation technique to the PLM, i.e., the work of Hu et al. (2022). Detailed information regarding the training hyperparameters is provided in Ap- pendix B. For the SNLI task, having large train- ing set, we repeated the experiments five times for each model and ten times for the other tasks. When basically
this approach entails risk of misestimation, be- cause it includes instances where the model makes incorrect decisions. Hence, to avoid the risk, we adopted more conservative evaluation metric that exclusively considers instances in which the model makes accurate predictions, i.e., the metrics are cal- culated based on the model's belief. The average performance is provided in Table 2.
Influence of the CRM. Experimental results show that leveraging the CRM weights can im- prove multiple types of consistency across many downstream tasks. Specifically, among 15 consis- tency test cases, the models using both PLM and CRM exhibit lower inconsistencies than their coun- terparts using only PLM. RoBERTa-base models with the CRM weights exhibit statistically signifi- cant improvements in 7 cases for FT and PI. The figures for the RoBERTa-large models are 6 and 7, respectively. It is captivating that the improve- ments are very significant in consistency types re- quiring
it comes to the evaluation metrics, followed the metrics used by Jang al. (2022a) but made a marginal modification in negation and symmetric consistency metrics. The two consis- tency types are conditional and, therefore, apply to examples having specific labels. Jang et al. (2022a) used gold labels for assessing the condition, yet
i.e., semantic and negational consistency. The experimental results support our hypothesis, stating that learning pre-

---

BoolQ
MRPC
RTE
SNLI Taom Tneg Tayon Tirn
WIC
Model
A
Term Ting
Tarm
Fneg
Taym
A
Tarm Tueg Taym
Teem Taym Tern
(P+C)
85.6* 8.7* 37.8F 90.5
6.71 77.0
4.4"
86.9
6.5
7.2
1.6*
93.0*
8.6
5.2 7.1 3.0"
73.0
10.9*
2.8
9.5*
PI (I+CC+M 85.7
8.3
33.9: 90.3
6.3
69.3*
4.3
87.6*
6.5
5.3
1.8*
93.0*
8.3
5.2%
7.2
2,9
74.2
8.6
3.0
8.9
DictBERT
74.6
8.7
68.1 87.6
11.2 86.5
5.7
74.6
15.0
38.0
2.5
90.7
10.5
10.8
10.0
3.7
67.3
13.3
3.4
16.6
Sem-Aug 84.6
8.2
44.1
90.3
5.3
80.4
1.2
86.2
6.3
12.1
3.7
56.3
7.6
17.9 10.2
1.1
72.5
8.6
4.7
11.2
Sem-CR
84.3 7.8
67.6 90.7
5.8
78.7
2.7
83.3
7.1
13.8
2.8
55.5
6.6 16.9 11.8
69.9
3.8
11.6
Table 3: Performance comparison of our proposed approaches and baseline models. M denotes a RoBERTa- Ta-lage model trained on the meaning-matching task. Teems Tneg Teym. and Ttrn denote semantic, negational, symmetric, and transitive inconsistency, respectively. A refers to the validation accuracy. The best figure is highlighted in bold. The values of the proposed approaches show a significant difference compared to the perffrrffom baseline with p-value < 0.05 (*) and p-value 0.1 (f) using the t-test.
cise conceptual roles can improve PLMs' meaning awareness and, therefore, can improve consisten- cies. However, the improvements in consistency types requiring the understanding of logical rela- tions, i.e., symmetric and transitive consistency, are relatively marginal, particularly in small-sized mod- els. This indicates that enhancing meaning aware- ness has a limitation to improving logical consis- tencies, and another solution is required for further improvements. We leave this as future work.
bility and achieve more significant improvements in consistency compared to the use of only CRM. This underscores the advantage of incorporating pre-trained knowledge from PLM and the enhanced meaning understanding of CRM.
Fine-tuning vs. parameter integration. The leading cause of introducing PI is enhance the training/inference efficiency of large-size PLMs. Applying PI considerably decreases the num- ber of training parameters, from 125M to 0.8M for RoBERTa-base and from 355M to 2.6M for RoBERTa- laggage,,, respectively. For RoBERTa-base, we observe several cases where introducing PI in- creases the inconsistency metrics in both P-only and P+C models, 4.5 over 15 test cases average under p-value 0.1. The aggravation mostly oc- curred in downstream tasks having small training set, i.e., BoolQ, MRPC, and RTE, while the per- formance marginally changed in other tasks with a large training set. However, this is not a severe de- merit, because fine-tuning the base-size model on a small training set is not a resource-consuming job. In contrast, when considering RoBERTa-large, the degradation in inconsistency metrics is observed in of 15 test cases on average. with a p-value < 0.1, which is considerably lower compared RoBERTa-base. Additionally, the inconsistency is improved in 5.5 out of 15 test cases on aver- age, which stands in contrast to RoBERTa-base, where no statistically significant improvements are observed with the use of PI. The results suggest that leveraging PI is more beneficial to large-size models, because it not only considerably reduces the number of training parameters but also gener- ally produces consistencies greater than or equal those of FT. This a great advantage that perfectly matches the objective of applying PI.
The effect of using both CRM and PLM. In or- der to ascertain whether the combined use of both CRM and PLM offers greater benefits compared to utilising only CRM, we conducted training with RoBERTa-large models that exclusively rely on CRM for both FT and PI. First, we observe that C-only models achieve an enhanced consistency in general compared P-only models, but the im- provements are less significant compared to P+C models. Under p-value < 0.1, C-only models ex- hibit statistically significant improvements in 4 and 5 out of 15 test cases in FT and Pl, respectively, while the figures of P+C models are 5 and 7, respec- tively. Also, it is confirmed that P+C models gen- erally produce a significantly lower inconsistency than C-only models in several test cases under p- value < 0. These cases involve semantic and tran- sitive consistency in the WiC task and negational consistency in the RTE task regarding FT. For PI, negational consistency in the BoolQ, semantic con- sistency in the WiC, transitive consistency in the SNLI task, and symmetric consistency in the RTE and WiC tasks correspond to these cases. More- over, the inconsistency of C-only models is even higher than P-only models in some cases, while P+C models do not exhibit a statistically signifi- cant performance degradation. The experimental findings suggest that P+C models exhibit higher sta-

---

KB- BB-BoolQ KB- BB-COPA KB. WIC KB- wellaSwag KB SentiNeg
Model
Acest
Atest 1
Ateat 1
Auest t
Tucg +
FT
P-only
78.2
60.4
79.1
78.2
8.5
(11IM)
P+C
78.7
65.3*
81.3"
78.1
7.4
R
Pl
P-only
77.2
74.1
79.4
77.1
7.1
(1.2M) P+C
78.4
75.0f
80.7
77.4
6.5
FT P-only
86.5
75.7
85.1
83.4
5.5
(338M)
86.9
82.5*
86.3
4.6
xoRoBr,
Pl
P-only
86.7
83.2
85.5
84.8
5.5
(2.6M)
P+C
87.6*
83.91
85.7
85.1
4.5
Table Average KoBEST performance of KoRoBERTa- (KoRoB p) and KoRoBERTa-large (KoRoBr).. Anst and Tneg represent the accuracy and negational consistency on the test set, respectively. The best performance is in bold. The figures of the proposed approach (P+C) show a significant difference compared to our counterpart (P-only) with p-value < 0.05 (*) and p-value 0.1 (t) using the t-test.
Comparison with baseline models. We com- pared the performance of our models with three baseline approaches. The first model is Dict- BERT (Yu et al., 2022), a model that employs the definition of rare words from the dictionary dur- ing the pre-training stage. The second approach is paraphrased data augmentation (Sem-Aug), which is also known adversarial training (Yoo and Qi, 2021), which additionally leverages paraphrased versions of the original data for training. The last approach semantic consistency regularisa- tion (Sem-CR) (Elazar et al., 2021), which trains a model to generate nearly identical predictive distri- butions for both the original and paraphrased data instances through consistency regularisation term. Referring to prior studies (Elazar et al., 2021; Ku- mar and Joshi, 2022), defined the loss function of Sem-CR approach as follows:
negation expressions and antonyms. We hypoth- esised that adding the parameters trained with a different inductive bias could further enhance con- sistencies, particularly in negational consistency in our experiment, while maintaining overall perfor- mance.
The results are summarised in Table 3. First, observe that approach outperforms Dict- BERT in terms of both accuracy and consistency by large margin. As the performance gap could be attributed to differences model size, we also compared the performance between DictBERT and RoB B-P--PP)(P++ in Table 2. Regarding the accu- racy, the latter exhibited higher performance than the former in all five downstream tasks, with sta- tistical significance at a p-value 0.05. When it comes consistency, the latter produced better results across 9 out of 15 cases, which was statistically significant with a p-value 0.05, while the former outperformed the latter only in test cases. The results demonstrate that our proposed approach can achieve a better inductive bias com- pared to DictBERT.
C= Cao + XJS(ollp),
where Zee is the cross-entropy loss, and JS refers to the Jensen-Shannon divergence. o and p are the original and corresponding paraphrased data points, respectively. Similarly to Elazar et al. (2021), we conducted tuning of parameters (A E 0.1, 0.5, 1) pick the best model. To generate paraphrased data for Sem-Aug and Sem-CR, we employed the EMBEDDING method of TextAttack (Morris et al., 2020), which replaces a certain portion of words (15% in our experiment) with their neighbours in the counter-fiie fitter embedding space. Sem-Aug and Sem-CR were applied to Roberta-large To verify the impact of additional knowledge add-up. in- corporated the weights of a RoBERTa-large model trained on the 2022b). The model employed dictionary data in a distinct manner compared to our approach and showed an improved performance in understanding
We ascertained that both Sem-Aug and Sem-CR produced lower semantic inconsistency compared to our approach in most test cases out of on average). However, they were substantially worse in other consistency types. These findings provide evidence supporting our claim that data augmenta- tion and consistency regularisation are restricted in addressing a specific consistency type, whereas our approach can improve multiple consistency types coneurrently. Furthermore, both methods exhibited lower accuracy across all downstream tasks and encountered a complete failure in SNLI, which off- sets their lower transitive inconsistency. We believe that a primary factor is the imperfect performance of the paraphrasing method, which can confuse
matching task (Jang et al..

---

models during training. Several examples of such cases are presented in Table in Appendix C. Fi- nally, we confirmed that the P+C+M model exhib- ited a similar or better performance than the P+C model. Specifically, the former showed statistically significant improvements in accuracy for 2 out of 5 tasks and achieved lower inconsistency in 5 out of 15 test cases. These improvements were predomi- nantly observed in negational consistency out of test cases). No statistically significant differences were observed in other test cases. These results verify our aforementioned hypothesis regarding the effect of integrating additional parameters trained with distinct inductive bias.
in English experiments. First, exploiting the CRM weights positively influences both fine-tuning and parameter integration cases, recording statistically significant performance improvements in out of 5 tasks on average. A marginal difference is that for large-size model, i.c., RoBERTa-large, us- ing CRM shows almost similar performance to using PLM-only in the GLUE benchmark experi- ments, while the former outperforms the latter in the KoBEST experiments. We presume that the leading causes are that (1) we collected more data (about four times higher) for training Korean CRM than the English counterpart, and (2) KoBEST downstream tasks require higher language under- standing ability, where the effects of CRM can be more pronounced. Second, although we examined only one test case due to the nonexistence of the Korean dataset for evaluating multiple consistency types, the negational consistency is significantly improved after applying CRM, which in line with our results on English datasets. It would be in- teresting future work to expand the BECEL dataset to Korean and investigate the effect of CRM. The experimental results support that our proposed ap- proach is easily applicable and works well in other languages beyond English.
3.3 Experiments on the Korean Dataset
As mentioned earlier, flexible applicability to other languages is one of the key advantages of our ap- proach. Hence, applied our method to the Ko- rean language to verify this.
3.3.1 Training the Korean CRM Model
We collected 1.6M word-definition pairs from the Korean Dictionary made by the National Institute of Korean Language. Regarding the backbone PLM, we used Korean RoBERTa models (Park et al., 2021). The same computing resources and training hyperparameter settings described in Sec- tion 3.1 were used for training.
4 Related Work
Analysis on consistency. Most widely conducted studies regarding consistency based seman- lic equivalence, i.e., model should generate the same predictions for text inputs conveying identi- cal meanings. Several works observed that when PLMs perform masked language modelling, they generate different predictions for queries where an object is replaced with its plural form (Ravichan- der et al., 2020) or for paraphrased queries (Elazar et al.. 2021). Elazar et al. (2021) introduced a con- sistency regularisation method to alleviate the in- consistent issue that forces PLM to produce similar predictive distributions for the original and para- phrased queries. Zhou et al. (2022) proposed a novel consistency regularisation technique known as "prompt consistency", designed for zero-shot learning. This method encourages consistent pre- dictions across various sets of prompts. Another line of work investigated PLMs' violation of log- ical properties, such symmetry (Wang al., 2019c: Li al., 2019; Kumar and Joshi, 2022) and transitivity (Li et al., 2019; Asai and Ha- jishirzi, 2020; Lin and Ng, 2022). Recently, Jang et al. (2022a) proposed the definition of language
3.3.2 Experiments on the KoBEST Dataset
We assessed the proposed approach on the KoBEST dataset (Kim et al., 2022), a recently proposed chal- lenging benchmark similar to the Korcan-GLUE benchmark (Park et al., 2021). The dataset con- sists of five downstream tasks: Boolean Ques- tion (KB- KKB-BoolQ.. Choice of Plausible Alternative (KB-COPA). Words in Context (KB-WiC), Hel- laSwag (KB-HellaSwag), and Sentiment Negation (KB-SentiNeg). Each task is designed to evaluate the model's 's understanding of context information, causality, word's meaning, time flow, and negation expressions, respectively. The SentiNeg task is a sentiment analysis task, but the test set is composed of negated training sentences where adding the negation expressions changes the polarity. There- fore, it supports the evaluation of negational con- sistency. Detailed hyperparameter settings for each task are described in Appendix B.
Table 4 presents the experimental results. Over- all, the results exhibit the analogous trend observed

---

model's consistency based on behavioural consis- tency and classified previous works into three large categories: semantic, logical, and factual consis- tency. Semantic consistency contains works based on semantic equivalence, which should always be satisfied regardless of the tasks and data. Logical consistency only applies to tasks where certain property holds and consists of four subcategories: negational, symmetric, transitivity, and additive consistency. Unlike the abovementioned studies that concentrated on the prediction inconsistency language models (LMs), several research delved into the inconsistency of explanations generated by PLMs and large language models (LLMs) (Jung et al., 2022; Wang et al., 2023; Jang et al., 2023).
sistency types, hardly apply to low-resource lan- guages, and require expensive training resources for large PLMs.
In this paper, we proposed an approach that im- proves overall consistencies in practical man- ner. We hypothesised that compelling reason for inconsistent behaviours is the lack of meaning- understanding ability and, hence, attempted to en- hance PLMs' meaning awareness based on the con- ceptual role theory. We designed a learning task that leverages word-definition pairs in a dictionary to capture more precise interconnection between concepts and named the learned model CRM. Next, we employed a parameter integration technique that combines the weights of CRM and PLM to maximally exploit previously- and newly-leamne knowledge. Our findings suggest that improved meaning awareness largely contributes to enhane- ing multiple consistency types concurrently. addition, we observed that further improvements are achievable through incorporating additional pa- rameters possessing distinct inductive bias. Finally, verified that our approach can readily apply languages beyond English. Our experimental sults suggest that investigating human cognition and transplanting it to learning objectives could be a key to trustworthy language Al.
Adaptation technique. Adapter modules in- vented to efficiently fine-tune PLMs by main- taining pre-trained weights fixed and introduc- ing only few trainable parameters. This is es- pecially efficient when training multiple down- stream tasks, as it allows training a few new pa- rameters per task without re-updating previously trained weights. Many prior works inserted low-rank residual adapter between layers (Rebuffi et al., 2017; Houlsby et al.. 2019; Lin al., 2020). Karimi Mahabadi et al. (2021) proposed the COM- PACTER method, which exploits the Kronecker product for a low-rank parameterisation, unlike the previous approaches that use down- and up- projections. Recently, Hu et al. (2022) proposed an approach named LORA. A distinctive difference between LoRA and the aforementioned methods is that the learnable weights of the former method are aggregated with pre-trained weights during the inference phase and, hence, do not increase the latency.
Limitations
For Korean dataset experiments, we were only able to measure negational consistency due the lack of a publicly available consistency evaluation dataset in Korean. In our English experiments, we col- lected 455K word-definition pairs to train CRM. Despite our attempts to collect more data by lever- aging the Oxford Dictionary API were unable to obtain approval for the API key. Considering the experimental results in Korean experiments, where CRM was trained with 1.6M word-definition pairs, we could anticipate further performance improve- ments in the English experiments with more word- definition pairs.
5 Summary and Outlook
Trustworthiness is highly recommended prop- erty that good natural language processing (NLP) model should satisfy, especially in risk-sensitive fields, such as the legal or medical domain. How- ever, recent studies have revealed that modern PLMs make so many mistakes that humans rarely commit and, hence, are not perfectly reliable. One such faulty phenomenon is PLMs' inconsistent be- haviours. Many studies proposed a remedy to re- solve the issue by using data augmentation and designing loss functions that penalise inconsistent actions. However, such methods have disadvan- tages in that they can only deal with specific con-
A Aknnowledgements
This work was partially supported by the Alan Tur- ing Institute under the EPSRC grant 10//510129/1 and by the AXA Research Fund. We also ac- knowledge the use of Oxford's ARC facility, of the EPSRC-funded Tier 2 facility JADE II (EP/
hhtp://awvooprrattttittttneeeeeo

---

T022205/I), and of GPU computing support by Scan Computers International Ltd.
language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Daniel Furrer, Mare van Zee. Nathan Scales, and
Nathanael Schärli. 2020. Compositional generaliza- tion in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint arXiv:2007.08970
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettle- moyer. 2021. Intrinsic dimensionality explains the effectiveness of language model mine-tunng. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 11th Inter- national Joint Conference on Natural Language Pro- cessing (Volume I: Long Papers). pages 7319-7328, Online. Association for Computational Linguistics.
Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT family eat word salad: Experiments with text understanding, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12946-12954.
Zellig S. Harris. 1954, Distributional structure. Word, 1662-2):
Akari Asai and Hannaneh Hajishirzi. 2020. Logic- guided data augmentation and regularization for con- sistent question answering. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Länguistics, pages 5642-5650 Online, Asso- ciation for Computational Linguistics.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language infer- ence benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9106-9118, Online, Association for Computa- tional Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R De- von Hjelm, Alessandro Sordoni, and Aaron Courville. 2021. Understanding by understanding not: Model- ing negation in language models. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1301-1312, Online. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efffcient transfer learning for NLP, In ternational Conference on Machine Learning, pages 2790-2799, PMLR.
Ewart J. De Visser, Samuel S. Monfort, Ryan McK- endrick. Melissa A. Smith, Patrick E. McKnight, Frank Krueger, and Raja Parasuraman. 2016. Almost human: Anthropomorphism increases trust resilience in cognitive agents. Journal of Experimental Psy- chology: Applied, 22(3):331.
Edward J. Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Terrence William Deacon. 1998. The Symbolic Species: The Co-evolution of Language and the Brain, WW Norton & Company.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171 4181.
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022a. BECEL: Benchmark for con- sistency evaluation of language models. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 3680-3696, Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander. Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Erratum: Measuring and im- proving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9: 407-1407.
Myeongjun Jang. Bodhisattwa Prasad Majumder, Ju- lian McAuley. Thomas Lukasiewicz, and Mana-Maria Camburu. 2023. KNOW how to make up your mind! ad adversarially detecting and alleviating inconsisten- cies in natural language explanations. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 540-553, Toronto, Canada, Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT not: Lessons
from a new suite of psycholinguistic diagnostics for

---

Myeongjun Jang. Frank Muumbuka, and Thomas Lukasiewicz. 2022b. Beyond distributional bypoth- esis: Let language models learn meaning-text corre- spondence. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 2030- 2042, Seattle, United States. Association for Compu- tational Linguistics.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Sriku- mar. 2019. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL pages 3924-3935, Hong Kong, China, Association for Computational Linguistics.
Xisen Jin, Xiang Ren. Daniel Preotiuc-Piete PPettr,, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In Interna- tional Conference on Learning Representations.
Ruixi Lin and Hwee Tou Ng. 2022. Does BERT know that the IS-a relation is transitive? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 94-99, Dublin, Ireland. Association for Com- putational Linguistics.
Eun-Soo Jung, Yeo- Dong, and Soo- YoonY Lee. 2019. Neural correlates of variations in human trust in human-like machines during non-reciprocal inter- actions. Scientific Reports, 11).
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020. Exploring versatile generative language model via parameter-efficies ings of the Association for Computational Linguistics: EMNLP 2020. pages 441-459, Online. Association for Computational Linguistics.
transfer learning. In Find-
Jachun Jung. Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1266-1279, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguisties.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters. and Noah A. Smith. 2019a. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Techmologies, Volume (Long and Short Pa- pers), pages 1073-1094.
Rabech Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, volume 34, pages 1022 1035. Curran Associates, Inc.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.1100 11692.
Nora Kassner and Hinrich Schützee 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational L Linguistics, pages 781 1-7818. Online. Asso- ciation for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017.
Fixing weight decay regularization in Adam. ArXiv,
abs/1711.05101.
Dohyung Kim, Myeongjun Jang, Deuk Sin Kwon, and Eric Davis. 2022. KoBEST: Korean balanced eval- uation of significant tasks. In Proceedings of the 29th International Conference on Computational Lin- guistics. pages 3697-3708, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Michael Matena and Colin A Raffel. 2022. Merg- ing models with fisher-weighted averaging. In Ad- vances in Neural Information Processing Systems, volume 35, pages 17703-17716. Curran Associates, Inc.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc.
Gangwoo Kim, Hyunjae Kim. Jungsoo Park, and Jae- woo Kang. 2021. Learn resolve conversational dependency: A consistency training framework for conversational question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 6130 6141, Online. Association for Computational Linguistics.
Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Arm- strong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. 2022. Enhancing self- consistency and performance of pretrained language models with NLI. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing ( (EMNLP). Association for Computational Linguistics.
Ashutosh Kumar and Aditya Joshi. 2022. Striking a bal- ance: Alleviating inconsistency in pre-trained mod- els for symmetric classification tasks. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1887-1895. Dublin, Ireland. Association for Computational Linguistics.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: A frame- work for adversarial attacks, data augmentation, and

---

adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126, Online. Association for Computa- tional Linguistics.
outside?! Improving answer consisteney in VQA through entailed question generation. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIA) pages 5860-5865, Hong Kong, China. Association for Computational Linguistics.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Jun- seong Kim, Youngsook Song. Taehwan Oh, Joohong Lee. Jubyun Oh, Sungwon Lyu, Younghoon Jeong. Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee. Seongbo Jang, Seungwon Do. Sunkyoung Kim. Kyungtae Lim. Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Lucy Park, Alice Oh, Jung-Wan Woo Ha (NAVER AI Lab), Kyunghyun Cho, and Kyunghyun Cho. 2021. KLUE: Korean language understanding eval- uation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks, volume 1. Curran.
Sylvestre-Alviee Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural In- formation Processing Systems, volume 30. Curran Associates, Inc.
Adam Santoro, Andrew Lampinen, Kory Mathewson, Timothy Lillicrap, and David Raposo. 2021. Sym- bolic behaviour in artificial intelligence. arXiv preprint arxiv:2102.03466.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes. Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional hy- pothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2888 2913, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Lin- guistics.
Koustuv Sinha, Prasanna Parthasarathi. Joelle Pineau. and Adina Williams. 202 UnNatural Language Inference. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 7329-7346, Online. Association for Computa- tional Linguistics.
Jason Phang. Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders stilts: Supplementary training on intermediate laheled-data tasks. CoRR, abs/1811.0100
Steven Piantadosi and Felix Hill 2022. Meaning with- out reference in large language models. In NeurlPS 2022 Workshop on Neuro Causal and Symbolic Al (nCSI).
Chikako Takahashi. 2012. Impact of dictionary use skills instruction on second language writing. Studies in Applied Linguistics and TESOL, 12(2).
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang. Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances Neural Information Process- ing Systems, 30.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessan- dro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji. and Mohit lyyer. 2020. Ex- ploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 7882-7926.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18)):
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypemyymy in BERT. In Proceedings of the Ninth Joint Conference on Lex- ical and Compurational Semantics, pages 88-102, Barcelona, Spain (Online). Association for Computa- tional Linguistics.
Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2020. BERTnesia: Investigating the capture and forgetting of knowledge in BERT. In Proceedings of the Th Third BlackboxNLP Workshop on Analyzing and Interpret- ing Neural Networks for NLP, pages 174-183.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappa- gari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard
Arijit Ray. Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. 2019. Sunny and dark

---

Grave, Ellie Pavlick, and Samuel Bowman. 2019a. Can you tell me how to get past Sesame Street? Sentence-level pretraining beyond language model- ing. In Proceedings the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1465-4477
Songshan Zhang, Hai Xu, and Xian Zhang. 2020. The effects of dictionary use on second language vocab- ulary acquisition: A meta-analysis. International Journal of Lexicography, 34(1):1-38.
Chunting Zhou, Junxian He, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Prompt con- sistency for zero-shot task generalization. In Find- ings the Association for Computational Linguistics: EMNLP 2022, pages 2613-2626, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019b. GLUE: multi-task benchmark and analysis plat- form for natural language understanding. In Interna- tional Conference on Learning Representations.
Haohan Wang, Sun, and Eric P. Xing. 2019c. What if we simply swap the two text fragments? straight- forward yet effective way test the robustness of methods to confounding signals nature language inference tasks. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 33, pages 7136-7143.
Peifeng Wang, Zhengyang Wang. Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023. SCOTT: Self- consistent chain-of-thougle distillation, In Proceed- ings of the 6l Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pages 5546-5558, Toronto, Canada. Association for Computational Linguistics.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3170-3179, 3179, Dublin, Ireland. Association for Computational Lin- guistics.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rehecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Car- mon, Simon Kornblith, and Ludwig Schmidt. 2022. Model soups: Averaging weights of multiple fine- tuned models improves accuracy without increasing inference time. In Proceedings of the 39th Interna- tional Conference Machine Learning, volume 162 Proceedings of Machine Learning Research, pages 29965-23998. PMLR.
Jin Yong Yoo and Yanjun Qi. 2021. Towards improv- ing adversarial training of NLP models. In Find- ings of the Association for Computational Linguis- ties: EMNLP 2021, pages 945-956, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu. Shuohang Wang. Yichong Xu, Michael Zeng, and Meng Jiang. 2022. model pre-training with dictionary. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1907-1918. Dublin, Ireland. Association for Computational Linguistics.
Enhancing language

---

A PLMs and Distributional Hypothesis
modelling objective, e.g., GPT-2 (Radford et al., 2019) and (Brown al., 2020), are also based on the distributional hypothesis by simply changing (tt) in Step 2 as follows:
Sinha et al. (2021a) showed four steps that train- ing BERT (Devlin et al., 2019) with the masked language modelling objective is not much different from training word2vec (Mikolov et al., 2013), representative method based on the distributional hypothesis. We briefly recall the material here. We recall the parameterisation of skip-gram word vectors (Mikolov al., 2013):
C(t) { {wo, w1,... Wy 1
where wt denotes a word that appears right be- fore the word t.
B Hyperparameters
eft.ww))
p(||w; Evev ef(t
This section presents the hyperparameter values used for the downstream task experiments.
where of is a dot product, V is the set of all possible words, and & and w are vectors of the target word and a word in the context, respectively. During training, negative sampling is used within window size, and the loss function, which is log (w + k E log (-wwwwww- w 1'), is optimised over the con- text C(w))) {w, (w: W4-1, Wi+1, ... Wi++) for a word index i, a window size of 2k, and unigram probability distribution P.
B.1 Experiments on English datasets
For the GLUE (Wang al., 2019b) and BECEL datasets (Jang et al., 2022a), we used the same hyperparameter settings. We set the maximum sequence length to 256, trained all models for 15 epochs, and chose the model with the best validation accuracy. The batch size was set to 32. All models are trained with the AdamW opti- miser (Loshchilov and Hutter, 2017) with linear learning rate scheduler decaying from 12 and warm-up ratio of 0.06, The learning rate was set to - for the FT models and 5e -4 for the PI models, respectively.
Computing the full softmax prob- ability entails a huge matrix multiplication with large vocabulary V. Instead, BERT exploits sub- word units that guarantee a smaller total vocabulary U in the softmax denominator.
Step 1: BPE:
B.2 Experiment on the KoBEST dataset
Step 2: Defenestration: Next, instead of using the local context window, BERT uses the entire sen- tence while making the target word: C(t) - {w E S A t}, where S' is the sentence including the word w.
Table shows the hyperparameter values used for training the PI models on KoBEST downstream tasks. For training the FT models, small changes made for the learning rate, le for the KB- WiC and KBB-HellSSagg and -6 for the others.
Step Non-linearity: The pairwise word-level dot product f(w, t) is replaced with a non- non-linear function. We can imagine using a sequence of multi-head self-attention layers g(t, C(t) in BERT. Now, we get the parameterisation of BERT as fol- lows:
BoolQ COPA
WiC HellaSwag SentiNeg
max-len epochs b-size
256
128
128
176
128
10
10
10
10
10
Ir
5e
5e -4
5e
5e-4
5e
weight decay
0.1
0.1
0.1
0.1
0.1
es((CC()
Table 5: Hyperparameters used for training the PI mod- els on KoBEST downstream tasks.
p(t()());)) Erev (O())
Now, the model is trained with tremendous amounts of documents. After the pre-training, the parameters of the model g are also updated during the fine- tuning on downstream tasks.
Step 4: Sprinkle data and compute:
The same method can be applied any PLMs trained with a masked language modelling objec- tive. We can also show a similar way that genera- tive PLMs trained with an regressgeee language

---

C
Examples
Raw Dictionary Data Definition
Word
happy
fecling showing pleasure; pleased
2) giving or causing pleasure
unhappy
1) not happy; sad
2) not pleased or satisfied with somebody
or something
Training Instances
happy feeling showing pleasure; pleased giving or causing pleasure unhappy happy; sad not pleased satisfied with somebody something
Table 6: Examples of word-definition pairs a dictionary and their concatenated version for training the CRM.
PREMISE: An older man drinking orange juice at restaurant.
ORIGINAL HYPOTHESIS: A man is drinking juice.
PARAPIRASED HYPOTHESIS: A man is alcohol juice. LABEL: Entailment
LaBEL: Entailment
PREMISE: A person on skis rail at night.
ORIGINAL HYPOTHESIS: They are fantastic skiiers. LaBEL: Neutral
PARAPHRASED HYPOTHESIS: They are terrific skiiers.
LaBEL: Neutral
PREMISE:
High fashion ladies wait outside beside crowd of people in the city.
ORIGINAL HYPOTHESIS: Women waiting by tram.
PARAPHRASED HYPOTHESIS: Women are waiting by streetcar.
LABEL:: Entailment
LABEL: Entailment
Table 7: Examples of imperfect paraphrased hypotheses of the SNLI task generated by TextAttack.


Reviews:
[{"review_id": "Jo8cDs9Xfc", "reviewer": "Reviewer_37Br", "summary": "", "questions": "- Maybe I misunderstood your ablations but did you also train i) Roberta alone on your data (so P-only but fine-tuned on your data) and ii) continued training Roberta for equally many steps on Wikipedia data? The first would be a sanity check for the more elaborate architecture you are using and the second would be a sanity check that it is not simply more data that improves performance but it is actually the type of data you are adding that is better than other data.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Please find our answers to your questions below:\n\nA1: By \"related work fusing additional knowledge into PLMs\", you mean integrating knowledge bases with PLMs? If so, note that these approaches are not our proper baselines. The term “knowledge add-up” that we used implies integrating parameters of multiple models having different inductive biases to improve the understanding of textual meaning, not fusing knowledge bases with PLMs. We will choose an alternative terminology to avoid confusion.\n\n\nA2: Note that, in Section 3.2, we are not arguing that the introduction of CRM achieves a higher textual meaning understanding based on the GLUE results. Instead, we claim that the CRM model contains more linguistic knowledge than the original PLM. As illustrated in lines #337-343, the experimental results show that the employment of the CRM model generates an improved performance in the COLA task, which is designed to check grammatical errors and requires linguistic knowledge. Also, the improvements were more significant in RoBERTa-base that stores less pre-trained knowledge. Based on these results, we demonstrated that “The experimental results suggest that CRM weights provide more abundant linguistic knowledge”. \n\nWhen it comes to the rest of the statement in lines #345-346, we are not arguing at this stage that our proposed approach achieved a better textual meaning understanding based on the GLUE results. Our intention is that the abundant linguistic knowledge provided by CRM weights might help the model to improve the understanding of textual meaning and, thereby, enhance consistency. We will state this more clearly to avoid confusion.\n\n\nA3: By “P-only but fine-tuned on our data”, you mean the CRM-only model? We tested this model and observed that the consistency is generally improved than P-only, but the improvements were less significant than P+C approaches. We will add this to the final manuscript to provide more comprehensive information. Regarding the second ablation model that you mentioned, we did not perform experiments, but we do believe that the result can highlight the importance of data type. We will conduct this experiment and add the results to the final manuscript.\n\nA4: As for what you write under “Typos, Grammar, Style, and Presentation Improvement”, we did not include references for the statement that you mentioned, because it is demonstrated in the abstract, but we did add references regarding that statement in lines #33-34. Could you let us know the other vague references/statements that we made? We will fix this in our final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Performance on the consistency dataset BECEL is better than their Roberta baseline model.\n- Performance on GLUE is on par with Roberta as is, meaning that their method does not degrade general performance.", "reasons_to_reject": "- They test their method on Roberta only. I doubt that their method is generally useful specifically in the context of more recent larger models where these definitions seem like a drop in the ocean. \n- There is related work fusing additional knowledge into PLMs. I wonder how their method's performance differs from other forms of continued training. (See also my question.)\n- The authors formulate conclusions which are not backed by their experimental findings. In section 3.2 their results are not showing systematic improvements over GLUE. The only thing one can take away from their GLUE results is that it does not degrade general performance. They could have presented it as that but instead they conclude that section with \"The experimental results suggest that CRM weights provide more abundant linguistic knowledge, which can help enhance the understanding of text meaning.\".", "questions_to_authors": "- Maybe I misunderstood your ablations but did you also train i) Roberta alone on your data (so P-only but fine-tuned on your data) and ii) continued training Roberta for equally many steps on Wikipedia data? The first would be a sanity check for the more elaborate architecture you are using and the second would be a sanity check that it is not simply more data that improves performance but it is actually the type of data you are adding that is better than other data.", "paper_topic_and_main_contributions": "Core idea: This paper addresses the problem of  inconsistent behavior by improving “PLMs’ meaning awareness”. They state that existing methods addressing inconsistency  i) require expensive resources and ii) are only able to address a specific type of consistency and are unable to handle others. They hypothesis that insufficient meaning awareness is the root cause of inconsistency.\n\nMethod:They continue training a PLM (Roberta) on a corpus consisting of a collection of word-definition pairs (455k) which they call conceptual role model (CRM). They then fine-tune an efficient integration of the parameters of the original PLMs and CRM on downstream tasks. \n\nEvaluation: They evaluate on English GLUE and BECEL (a consistency dataset) and Korean KoBEST (similar to Korean-GLUE). They compare to English/Korean Roberta model and baseline methods targeting consistency.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are quite a number of places where you made broader references/statements which were very vague. E.g., 'The non-humanlike behaviour of contemporary pre-trained language models (PLMs) is a leading cause undermining their trustworthiness.'"}}, {"review_id": "Nvv4QSzwXj", "reviewer": "Reviewer_DNHZ", "summary": "", "questions": "A. How was CRM model trained ? Did you use MLM for this model as well ? How many steps was the model trained for ?\nB. Line 399 - which figures ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review. Please find our answers to your comments and questions below.\n\nAnswer to 1.i)+ii): We tested the C-only model and observed that it showed a better consistency than P-only, but the improvements were less significant than P+C. We will add this to the manuscript. When it comes to the second baseline, we did not test the approach, because continued training on the dictionary data can cause catastrophic forgetting of the original corpus, which can overestimate the advantage of the CRM parameter integration. However, we agree that it would be more helpful to understand the benefits of the parameter integration, if our concerns are verified with the performance of the second baseline. We will perform the experiments and add them to the final manuscript.\n\nAnswer to 2.i) : The additive consistency was not reported, mainly because previous experiments in BECEL paper showed that every PLM is highly consistent on the additive consistency. We also ascertained through pilot tests that the models that employed the CRM model (both FT and PI) were also highly consistent on the additive consistency, so it is excluded from the experimental scope of the paper. We will mention this in the final manuscript.\n\nAnswer to 2.ii): The inclusion of inaccurate predictions can prevent precise evaluations, because logical consistencies, such as negation consistency and symmetric consistency, are conditional consistency types. For example, negation consistency applies when the label is ``Equivalent'' for the STS task. Consider the below examples of the MRPC task:\n\nS1: In the evening, he asked for six pepperoni pizzas and two six-packs of soft drinks, which officers delivered.\n\nS2: In the evening, he asked for six pizzas and soda, which police delivered.\n\nS2-neg: In the evening, he asked for six pizzas and soda, which police did not deliver.\n\nThe gold label of the S1-S2 pair is “Equivalent”, and provided the model generates a correct answer for the S1-S2 pair, predicting the relation between S1-S2-neg as “Equivalent” is a violation of negation consistency. However, assume that the model believes that the answer of the S1-S2 pair is ``Not Equivalent''. In this case, according to the original metric, it is considered as the violation of negation consistency if the model generates “Not Equivalent” as the answer of the S1-S2-neg pair despite it is the correct answer. On the other hand, if the model generates “Equivalent” as the answer of the S1-S2 pair, it is considered as a consistent prediction despite it is a wrong answer. This example shows that the condition of logical consistencies should be applied based on the model’s belief not the gold-label of data instances. Therefore, we modified the metrics to consider only the instances where the models make correct predictions. We will add this to the final manuscript with an additional page.\n\nAnswer to \"How was CRM model trained?\": The CRM model is trained by using the MLM objective. The base and large models are trained for 500K and 700K steps, respectively, as outlined in lines #303-304.\n\nAnswer to \"which figures?\": The word ‘figure’ denotes the number of test cases where the RoBERTa-large model showed statistically significant improvements in FT and PI. We will change the terminology to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting idea to leverage easily available dictionary data for improved contextual information during pre-training\n2. Parameter efficient model can enable improving the model in a resource efficient manner.\n3. Generalizable method - helps improve multiple consistency types with the same training setup. Additionally easily extendable to new languages also - dictionary data is easily available in most languages.", "reasons_to_reject": "1. Missing baselines - some important baselines are missing which are necessary for a complete picture of the model. These suggested experiments are not parameter efficient, but will provide the trade-off details wrt performance.\n\n   i) C-only model - how well does CRM model do the task is PI necessary ?\n\n   ii) Continued training on P instead of PI - again to ablate whether PI is necessary, continuing to train the original on the original corpus for same steps and additionally on the dictionary corpus for same steps will help understand benefits of current weight integration setup.\n\n\n2. Experiment and Evaluation details are a bit hard to follow - there are many decision choices on data (some datasets included but others aren’t) and in evaluation (modifying evaluation metric) which are mentioned but in a high-level manner. Without giving details, it’s hard to understand why certain experimental setups were chosen and hard to even reproduce using details in the paper.\n\ni) 354-360 Can you give more details on why additive consistency was not reported. Yes PLMs perform well, but does this model perform well as well ?\n\nii) Line 388 - Why was evaluation metric modified (exclusively considers instances in which the model makes accurate predictions) - why is including inaccurate predictions problematic ? maybe more details on misestimation might help ?", "questions_to_authors": "A. How was CRM model trained ? Did you use MLM for this model as well ? How many steps was the model trained for ?\nB. Line 399 - which figures ?", "paper_topic_and_main_contributions": "The paper aims to develop more robust and consistent LMs by incorporating word definition knowledge into LMs by leveraging external dictionary data. Current LMs have inconsistent behaviour like different predictions for semantically identical text or displaying logical contradictions. This work hypothesizes that current pretraining mechanisms rely on co-occurance contexts which is insufficient and that using word definitions provides a more reliable ‘meaning’ based context which is more reliable. They train an auxiliary model (CRM) using dictionary data and propose parameter integration methods to more efficiently incorporate the model with existing one. Their experiments show that the model has comparable or better NLU performance on GLUE and better performance on consistency settings like BECEL. Additionally, they aslo show that this model/technique can be easily extended beyond English by showing similar experiments on Korean datasets. \n\nThe main contributions of this paper are:\n\n1. Leveraging dictionary data to improve robustness and consistency\n\n2. Parameter efficient training and methods to incorporate new knowledge with existing models\n\n3. Method applicable to new languages with readily available dictionary data.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GVkx1nQrZk", "reviewer": "Reviewer_zTcT", "summary": "", "questions": "Q1: Lines #252-#255: Where are these previous studies from? Please cite them.\n\nQ2: Lines #296-297 There are 455K collected word-definition pairs for training the CRM model. Will the performance be improved if the training pair increases? It is suggested to make a comparison across different training size scales.\n\nQ3: Table 3: How to integrate the P+C+M model? By Eq(6)?\n\nQ4: Lines #539-#540. Did the author try multilingual Roberta PLM (instead of Korean RoBERTa)? If you did, are there differences in the results between them?\n\nQ5: Lines #667-#670: If there is a problem with the collection of dictionaries, how about using the synset strings (from WordNet) as training samples?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review. Please find our answers to your questions below:\n\nAnswer to Q1: The prior studies refer to the work of Hu et al. (2022) for parameter integration and the RoBERTa paper (Liu et al., 2019). We will mention these studies in lines #252-#255.\n\nAnswer to Q2: We do believe that more training pairs can improve the performance, because (1) it is a generally observed phenomenon in the NLP field that more data increase the performance, and (2) the experiments on the Korean dataset, which employed more training pairs, demonstrate a larger performance gain. We will conduct experiments with smaller training pairs and add the results to the final manuscript.\n\nAnswer to Q3:  The weights of P, C, and M are integrated through the averaging aggregation method. We will add more detailed explanations and update the equation to avoid possible confusion.\n\nAnswer to Q4: We did not compare the performance of the Multilingual RoBERTa model, as our primary intent for the Korean dataset experiments was to ascertain our approach’s applicability to low-resource languages. The usage of multilingual models can prevent precise evaluation, because such models can benefit from the extensive datasets in foreign languages. Hence, to minimise the influence stemming from external factors, multilingual models are not considered. However, we agree that it would be interesting to see how the consistency performance would be affected by the usage of multilingual models. We will conduct additional experiments regarding this.\n\nAnswer to Q5: We tried to also use WordNet synsets but observed that the words were highly overlapping with the words that we collected. Hence, we did not use the WordNet dataset for our experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposed a simple but effective learning framework to enhance the meaning-understand capability of PLMs. The authors conducted experiments on two/one English/Korean public datasets, and the results supported the claims and proved the effectiveness. Furthermore, the proposed parameter integration approach (by aggregating additional weights learned from various resources) not only integrates knowledge from different sources to enhance the understanding capability of PLMs but also avoids the risk of catastrophic forgetting. The related experiments, findings, and observations will benefit further research.", "reasons_to_reject": "According to the experimental results, the model size and training approach of the PLM will affect the improved performance of the proposed method. The authors use RoBERTa as the backbone model. This work does not discuss whether other PLMs (such as BERT) will have similar results. Furthermore, large language models' (LLM) training parameters (i.e., the stored knowledge) are significantly larger than those of PLMs. There is no relevant experiment and discussion on whether the inconsistency issues in PLM also occur in LLM in this work.", "questions_to_authors": "Q1: Lines #252-#255: Where are these previous studies from? Please cite them.\n\nQ2: Lines #296-297 There are 455K collected word-definition pairs for training the CRM model. Will the performance be improved if the training pair increases? It is suggested to make a comparison across different training size scales.\n\nQ3: Table 3: How to integrate the P+C+M model? By Eq(6)?\n\nQ4: Lines #539-#540. Did the author try multilingual Roberta PLM (instead of Korean RoBERTa)? If you did, are there differences in the results between them?\n\nQ5: Lines #667-#670: If there is a problem with the collection of dictionaries, how about using the synset strings (from WordNet) as training samples?\n", "paper_topic_and_main_contributions": "To address the inconsistent issues of PLMs, this paper introduces a learning framework that leverages word-definition pairs in a dictionary to capture the interconnection between concepts to enhance the capability of meaning-understanding for PLMs. The authors further integrate learned parameters (from multiple knowledge sources) by an aggregation method to improve performance and reduce inconsistency. Especially the proposed approach can concurrently enhance multiple types of consistency simultaneously. Experimental results demonstrate the effectiveness of both English and Korean Language datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method to improve the consistency of pretrained LMs predictions by leveraging structured semantic information extracted from a dictionary. The augmented model is able to outperform baseline ones on the GLUE and BECEL datasets. Though the method is not particularly innovative, there is consensus about the methodological soundness of the paper, which makes it a valuable contribution. A possible limitation is the fact that the method is only applied to RoBERTa, lacking a larger experimentation with other LMs. However, this does not negatively affect the general strength of the paper.
Decision: Accept-Main