Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding
Authors: Caoyun Fan
Jidong Tian
Yitian Li
Wenqing Chen
Hao HE
Yaohui Jin
Venue: EMNLP 2023 Main (2023)
Keywords: masked language models
multi-step reasoning
Chain-of-Thought
natural language understanding
TL;DR: We propose Chain-of-Thought Tuning (CoTT), a two-step reasoning framework based on prompt tuning to implement step-by-step thinking for MLMs on NLU tasks.

Abstract: Chain-of-Thought (CoT) is a technique that guides Large Language Models (LLMs) to decompose complex tasks into multi-step reasoning through intermediate steps in natural language form. Briefly, CoT enables LLMs to think step by step. However, although many Natural Language Understanding (NLU) tasks also require thinking step by step, LLMs perform less well than small-scale Masked Language Models (MLMs). To migrate CoT from LLMs to MLMs, we propose Chain-of-Thought Tuning (CoTT), a two-step reasoning framework based on prompt tuning, to implement step-by-step thinking for MLMs on NLU tasks. From the perspective of CoT, CoTT's two-step framework enables MLMs to implement task decomposition; CoTT's prompt tuning allows intermediate steps to be used in natural language form. Thereby, the success of CoT can be extended to NLU tasks through MLMs. To verify the effectiveness of CoTT, we conduct experiments on two NLU tasks: hierarchical classification and relation extraction, and the results show that CoTT outperforms baselines and achieves state-of-the-art performance.

Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding
Caoyun Fan', Jidong Tian', Yitian Li', Wenqing Chen', Hao He' Yaohui Jin" MoE Key Lab of Artificial Intelligence, AT Institute, Shanghai Jiao Tong University {fcy3649, frank92, yitian_li, hehao, jinyh]es edu. 2 School of Software Engincering, Sun Yat-sen University chenwg95email. sysu. edu. cn
edu. cn
Abstract
Test Whes SASAC nasiliched 2000 tamberr of SCEa wus 195.
mochtn lourning tcchiggs
Mhain-of-Thoughe (CoT) is technique that guides Large Language Models (LLMs) to de- compose complex tasks into multi-step rea- soning through intermediate steps in natural language form. Briefly, CoT enables LLMs to think step by step. However, although many Natural Language Understanding (NLU) tasks also require thinking step by step. LLMs perform less well than small-scale Masked Language Models MML.s). To migrate CoT from LLMs to MLMs, propose Chain-of- Thought Tuning (CoTT), a two-step reasoning framework based on prompt tuning, imple- ment step-by-step thinking for MLMs on NLU tasks. From the perspective COT, CoTT's two-step framework enables MLMs imple- ment task decomposition; CoTT's prompt tun- ing allows intermediate steps to used in natu- ral language form. Thereby, the success of CoT can be extended to NLU tasks through MLMs. To verify the effectiveness of CoTT, we conduct experiments two NLU tasks: hierarchical classification and relation extraction, and the results show that CoTT outperforms baselines and achieves state-of-the-an performance.
Domain
Subyess Eatay, ergonianive Wettty
O
Labet Areic ML.
Label
Relation: feawske!
(a) Hierarchical Classification
(b) Relation Extraction
Figure I:
In NLU tasks, various forms of intermediate steps (drawn in gray) would exist as reasoning evidence between Text and Label.
(Zhao et al., 2023), Yang et al. (2023); Kocon et al. (2023); Lai et al. (2023) found that LLMs were less competitive than small-scale Masked Language Models (MLMs) (Devlin al., 2019; Liu et al., 2019; Lan et al., 2020) in many traditional Natural Language Understanding (NLU) tasks (e.g. tasks in GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a)). The reasons for this are man- ifold: on the one hand, LLMs are typically autore- gressive models and are not well-suited for NLU tasks (Liu et al., 2021; Raffel et al.. 2020); on the other hand, NLU tasks usually involve rich domain knowledge (Fan al., 2023a; Zhang et al., 2023) and require fine tuning to master such knowledge (Fan et al., 2023b.c; Wang et al., 2023). Therefore, we wonder whether the success of CoT in LLMs can be transferred to MLMs in NLU tasks.
1 Introduction
Mohins-of-Thought (CoT) (Wei et al., 2022b; Fu et al., 2023; Zhang et al., 2022) is a technique to help language models think step by step (Kojima et al., 2022). Through intermediate steps in natural language form, CoT can guide language models to decompose complex tasks into multi-step reason- ing processes. Currently, COT mainly employed in Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023; Zeng et al., 2023; OpenAI, 2023), as LLMs demon- strate impressive complex reasoning capabilities (Wei et al., 2022a: Zhao et al., 2023).
In fact, many NLU tasks also require thinking step by step. For example, in hierarchical classifi- cation (Jr. and Freitas, 2011: I; Kowsari et al., 2017), each text should follow a pre-defined taxonomic hierarchy to generate multiple labels in turn, so that high-level labels can be considered as the in- termediate step, as shown in Fig. 1(a); in relation extraction (Zhang et al., 2017; Alt et al., 2020; Sto- ica al., 2021), the entity types of the subject and the object in each instance need be determined in advance of annotating the relation (Zhou and Chen, 2021), so the entity types can be considered as the intermediate step, as shown in Fig. 1(b). Although
However, although LLMs achieved state-of-the- art performance on a wide range of NLP tasks
*Coreesponiigg author.

---

previous studies (Chen et al., 2020b; Han al.. 2021; Zhou and Chen, 2021) have also attempted to incorporate intermediate steps into language mod- els, in the perspective of CoT, these methods lack both a decomposition process for tasks (not multi- step reasoning) and explicit use of intermediate steps (not in natural language form).
tion, and the experimental results demonstrate the effectiveness of CoTT.
2
Related Work
2.1
Chain-of-Thought
(CoT) (Wei et al., 2022b) is prompt technique that elicits LLMs to produce intermediate reasoning sleps leading to the final answer. Recent studies (Zhou et al., 2023; Fu et al., 2023; Zhang et al., 2022) have confirmed that CoT can substantially improve the reason- ing ability of LLMs. Studies (Zhang et al., 2022) have shown that LLMs can perform CoT reason- ing with zero-shot scenarios (Kojima et al., 2022) or manually written few-shot scenarios (Wei et al., 2022b). In zero-shot scenarios, Kojima et al. (2022)
In this study, we propose Chain-of-Thought Tun- ing (CoTT), two-step reasoning framework based on prompt tuning (Sun et al., 2022; Liu et al., 2022), to implement step-by-step thinking for MLMs on NLU tasks. The two steps of CoTT are:
Step I: MLM generates the intermediate step i based on the text z.
Step II: MLM predicts the final result y based the text and the intermediate step f.
CoTT effectively addresses the shortcomings of previous methods: on the one hand, CoTT's two- step framework enables MLM to implement task decomposition; the other hand, CoTT is based on prompt tuning, which allows intermediate steps be used natural language form (Schick and Schütze, 2021b; Gu et al., 2022). In order to in- ject/generate intermediate steps flexibly in both steps, we propose convertible slot [C], a new type of template slot (Petroni et al., 2019; Liu et al., 2022) in prompt tuning.
showed that LLMs can generate decent interme- diate steps, by adding certain magic phrases like "Let's think step by step". Zelikman et al. (2022) employed LLM to generate many intermediate steps and those intermediate steps that can lead to the final answer are chosen. In few-shot scenar- ios, LLM's reasoning ability can be improved by few effective demonstrations on multi-step rea- soning tasks. Some studies discussed how select demonstrations efficiently: Fu al. (2023) con- sidered prompts with higher reasoning complexity (more intermediate steps) to be efficient demonstra- tions, Rubin et al. (2022) automatically constructed demonstrations based on the semantic similarity of texts. However, it is still unknown whether CoT can be applied to small-scale language models.
To evaluate the effectiveness of CoTT, we con- duct extensive experiments on two traditional NLU tasks: hierarchical classification and relation extrac- tion, and the experimental results reveal that CoTT obtains superior performance compared with base- lines and achieves state-of-the-art performances. Furthermore, due to the introduction of intermedi- ate steps, CoTT is no longer end-to-end method but can display the reasoning process. Therefore, we can further improve the ability of CoTT by mon- itoring the reasoning process. We summarize our contributions as follows:
2.2 Prompt Tuning
Since the advent of GPT-3 (Brown et al., 2020), prompt tuning (Sun et al., 2022; Liu et al., 2022) has received considerable attention. Prompt tuning (Schick et al., 2020; Gu et al., 2022) aims to trans- form the downstream tasks into the pre-training tasks of Pre-traines Language Models (PLMs) with appropriate manual prompts, which can bridge their gap and better utilize PL.Ms (Han et al., 2021; Chen et al., 2022a). Although the origin of prompt tuning is large-scale autoregressive language models, the following studies (Schick and Schütze, 202 1a,b)
Based on the philosophy of CoT, we propose two-step reasoning framework to enable MLMs to think step by step in NLU tasks, and we call it Chain-of-Thoughe Tuning (CoTT).
We propose convertible slot [C], new type of slot in prompt tuning, which can flexibly in- ject or generate intermediate steps depending on scenarios.
found that small-scale Masked Language Models (MLMs) (Devlin et al., 2019; Liu al., 2019; Lan et al., 2020) can also achieve competitive perfor- mance using prompt tuning. In practice, ML.S can implement prompt tuning in the form of cloze- style tasks (Devlin et al., 2019; Liu et al., 2019).
We evaluate our CoTT on two NLU tasks: hierarchical classification and relation extrac-

---

With MLMs, prompt tuning has been applied large variety of tasks such as factual probing (Perez et al., 2021), text classification (Gao al., 2021 aa; Hambardzumyan et al., 2021), relation extraction (Chen et al., 2022b), commonsense reasoning (Et- tinger, 2020) and question answering (Khashabi et al., 2020; Jiang et al., 2021), etc.
in templates, to introduce intermediate steps flex- ibly in both steps (Section 4.1). Finally, we fuse the information from both steps to rectify MLM's prediction (Section 4.4).
4.1 Convertible Slot in Template
In Section 3, the traditional template contains two types of slots ([T) and [A]) for injecting texts and generating predictions, respectively. However, this template cannot flexibly introduce intermedi- ate steps in natural language form. To overcome this problem, propose new type of slot- con- vertible slot [C], which can be converted between injecting and generating intermediate steps based on scenarios.
3 Preliminaries of Prompt Tuning
Formally, text classification dataset can be de- noted D {X' }/. where X' the text set and y is the class set. For each instance x X, it is made up of several words x = { [11, w / and is annotated with a label E y.
To bridge the gap between pre-training tasks and downstream tasks (Schick et al., 2020; Gu et al., 2022), prompt tuning is proposed as cloze-style task to tune MLMss Prompt tuning consists of template T, verbalizer Φy(+) and a MLM M.
Specifically, we can incorporate the convertible slot [C] at the appropriate position of the tem- plate based on semantics. Here, we notate I as the intermediate step set and prepare a verbalizer oz I + Vy to establish connection between I and intermediate step word set V/- When the inter- mediate step is unknown, MLM needs generate the intermediate step. In this case, we simply fill in [C] with <M>, allowing MLM to make cloze- style prediction about Vi to get the intermediate step. [C] is analogous to [A]. When a specific in- termediate step 1 E I is given, MLLM needs to be provided with such information through the prompt. In this case, we can directly fill the intermediate step word 27 = +I([I) into [C] to inject the ap- propriate information. [C] is analogous to [T]. brief, the flexibility of convertible slot [C] to con- vert between [T] and [A] allows MLM to combine intermediate steps to make predictions.
The template (Petroni al., 2019; Liu et al., 2022) is a textual string with two slots: a text slot [T] for text x and an answer slot [A] (a <MASK> token) for the cloze-style prediction. The verbalizer is an injective mapping function Φy y + Vy that bridges the class set J' and the label word set Vy. Specifically, when the text is injected into the text slot, we get the prompt T,, Then, we can formalize the label probability by feeding the prompt T3 into MLM M as:
p(y|z) (MMSS> (y((TT
expleøy(u) HLUSK)))
(I)
Every expler h
where heusk> is the hidden vector of answer slot, and e, is the embedding of each label word E Vy. Hereafter, we abbreviate <MASK> <M> In the training process, we can maximize the learning objective Evex log ((<p((<>>>> (T) to tuning MLM M.
4.2 Step Generate Intermediate Step
The purpose of step I is to generate the intermediate step by MLM. Specifically, as shown in Fig. 2(a), we fill the text z into [T] and fill an additional <M> into [C] to get the prompt. We still notate such prompt as Tz. Then, the probability of I can be obtained by feeding Tr to MLM M as:
4 Chain-of-Thought Tuning
p([[>) (MC (T))T))
(2)
We propose Chain-of-Thought Tuning (CoTT), a two-step reasoning framework for Masked Lan- guage Models (MLMs) to think step by step, as shown in Fig. 2. The core of CoTT is interme- diate steps in natural language form. MLM first generates the intermediate step in step (Section 4.2), and then uses the generated intermediate step to predict the final result in step (Section 4.3). We design convertible slot [C], a new type of slot
Eq. 2 implements the first step of CoT: generate the intermediate step based on the text (x: I). Here, denote MLM's prediction of the intermediate step in Eq. f.
Predict Label in Parallel
Due to the introduction of the convertible slot, the generation of intermediate steps and labels can be

---

MOK 2010 soin (Toul Proubilly Thewen)
MM
NLM
MIM
[T]
[C)
[A]
M
[C]
[A]
(a) Step
(b) Step II
(c) Probability Rectification
Figure 2: Overview of Chain-of- Thought Tuming (CoTT), CoTT two-step reasoning framework: generate the intermediate step (step I) and use the intermediate step (step II). Then, probability rectification is proposed to rectify MLM's prediction based on the information from both steps. Here, T denotes the prompt, available information is drawn blue, and generated information is drawn in orange.
simultaneous (multiple <M>s in prompt). Therefore, using the same prompt Tr, MLM can predict the label in parallel as:
intermediate step i to perform logical reasoning. However, this process of information integration lacks explicit guidance. Therefore, we propose counterfactual-based contrastive learning in step II, to guide MLM's information integration by con- trasting the hidden vectors obtained from the fac- tual/counterfactual intermediate step.
x))lx) (IM[ (y))))T>)
We denote MLM's label prediction in Eq. as ya- which is independent of It is worth noting that Eq. 3 does not follow the generation process of CoT: MLM predicts the label in one step without any intermediate steps (rr - y). Due to the lack of step-by-step reasoning process, we consider Úz be an intuitive prediction.
In most cases, contrastive learning (Gao et al., 202 1b) requires an anchor as well as positive and negative samples for each instance. In CoTT, it is natural to consider the hidden vector h. of [A] in step the anchor, and the hidden vector h of [A] step II the positive sample. To construct negative sample, we sample the counterfactual intermediate step f based on the probability distri- bution of intermediate steps in Eq. 2 as:
4.3 Step II: Use Intermediate Step
In step II, since the intermediate step i is available, MLM can predict the label using both the text and the intermediate step, which consistent with the second step of CoT (r + + I).
~ Pej (1x) -
p(Ix) {{ 1-p(1z)
I I
As shown in Fig. 2(b), we inject x and i into the proper slots of the template to obtain the prompt, and we notate such prompt Tr,,- Similar to Eq. 1, the label probability with the intermediate step be expressed as:
(5)
0 I I-f,
where Pej (1(x) refers to the normalized probabil- ity after masking the prediction of the intermediate step f. Then, similar to step II, we inject counter- factual intermediate step 1 as well as the text x to the template to obtain the counterfactual prompt Ta, and feed Ta, to MLM to get the hidden vector h of [A] as the negative sample.
p(y|æ, kr I) - PMM([A] (v.)))
(4)
We denote MLM's label prediction in Eq. 4 as Jz./- Compared to Eq. 3, MLM can perceive and com- bine the text and the intermediate step in Eq. 4, which makes sophisticated reasoning process pos- sible. Therefore, /z.1 is considered to be rational prediction.
Following (Chen et al., 2020a), we design a small neural network projection head g(-) that maps each hidden vector into the projection space. In this study, we employ MLP with one hidden layer obtain the projection vector as:
Counterfactual Contrastive Learning
The core of Step II is that MLM needs to inte- grate the information from both the text x and the
2 = gh) = ( o(W(1) h),
(6)

---

where o is ReLU nonlinearity, W(L) and W(2) are the learnable parameters. We take the cosine similarity of projection vectors as the similarity function because it implicitly normalizes each vec- tor. The similarity between two projection vectors and 2j can be described as:
estimating the label probability utilizing the infor- mation from two steps. Specifically, we assume that when + f. we have:
(pylpl Dllpl a)) < Den(plule, D1lp D). (10)
where DKL, refers to the Kullback-Leibeer Diver- gence. The assumption implies that, compared to using an inconsistent intermediate step, the estima- tion without using intermediate steps is relatively more accurate. This consistent with the human perception of intermediate steps. Therefore, we replace p(y) x I) in Eq. with p(y) x)) for all cases satisfying * I, then the label probability can be estimated more exactly as:
2 2j
sim 24 2z)
||
Then, the counterfactual contrastive loss Ce is defined as:
cimm
Le = log (Sim(z,,5s + 11
(8)
pr(vlz) p(1 x) plyla,
[(((s) p(yla,
where r denotes the temperature parameter. By contrasting the similarity of hz and (hr,,, h-,1-} in the projection space, MLM learns distinguish whether æ and match: when they do, the hidden vectors of two steps are similar, and vice versa. Overall, counterfactual-based con- trastive learning forces MIM to perceive the rela- tionship between texts and intermediate steps more fine-gre nneegrained, and integrates the perceived informa- tion into the hidden vector of [A].
zo r(l x) plylar + (1 r((p)) p(ulo))
(11)
Eq. 11 is the rectified label probability. Essentially, the probability rectification is an adaptive weighted probability based on the probability of the inter- mediate step p( [())))) shown in Fig. 2(c): when p( [[)
is high, we consider that i is more trustwor- thy, so py (z) is closer to the prediction step II p(yl xr Σ). and vice versa, pr (w|xz) is closer to the prediction in step p(y] Probability rectifica- tion is efficient and does not introduce additional computational complexity, but rather judiciously integrates known information from both steps.
4.4 Probability Rectification
In the two-step process mentioned in Section 4.2 & 4.3, we combine and Í to obtain the label probability p(y] xz 2). However, this is actually an estimation of the label probability. From the per- spective of the total probability theorem, with the consideration of intermediate steps, the exact label probability should be expressed as:
4.5 Training Details
During the training process, each prediction made in two steps should be optimized, therefore, the loss of CoTT consists of three prediction losses as well as a contrastive loss fe. We employ the Cross-Entropy loss to calculate the prediction loss. The loss of CoTT L is denoted as:
pr(v)x)= = Σp([To) I). IEI
In step 11, pr(()z in Eq. 9 is estimated to be equal to p(ylæ, 7) in Eq. 4. The meaning of this estima- tion is that even if I 4 i. p(y) I) x still holds true. There a drawback this estima- tion: large estimation errors will occur when MLM is ambiguous about intermediate steps (p( (1)x) is relatively low), which is known as exposure bias (Yang et al., 2018).
log p(1[z) log p(y|z)
X
SEX
Siep
(12)
Le log loll
Suep
where 0 are the weights to balance each loss respectively, and | represents the number of in- stances in the dataset.
However, it is computationally costly to strictly follow Eq. 9 to calculate the exact label probabil- ity, as requires MLM to repeat [Z] times for step II. Therefore, our proposed probability rectifica- tion method aims to simplify Eq. 9 by efficiently
5 Experiments
5.1
Datasets and Evaluation Metrics
To verify the effectiveness of CoTT, we conducted extensive experiments on two traditional NLU

---

Dataset
train
dev
test
# label
method with two vanilla prompt tuning methods: HardPrompt and SoftPrompt'. For RE task, as fine tuning of MLMs achieved promising results, we fine tumed two traditional MLMs: BERT (Devlin et al., 2019) and Roberta (Liu et al., 2019), as well as two knowledge-enhaneed (Joshi et al., 2020) and KnowBERT (Peters et al., 2019) as baselines. Since CoTT is based on prompt tuning, also employed two prompt tuning meth- ods HardPrompt and PTR (Han et al., 2021) as baselines. Experimental details of LLMs can be found in Appendix A.
WOS
30070
7518
9397
141
TACRED
68124
22631
15509
42
TACREV
68124
22631
15509
42
ReTACRED
58465
19584
13418
40
MLMs: SpanBERT
Table I: The statistic details of the four datasets, where # represents the number of instances in each set.
tasks: Hierarchical Classification (HC) and Rela- tion Extraction (RE). For HC task, conducted our experiments on Web-of-Science (WOS) dataset (Kowsari et al., 2017). WOS contains abstracts of published papers from Web of Science, and the labels WOS have two hierarchies: 7 domains and 134 areas. We treated the domain label as the intermediate step of the reasoning process. We measured the results with Micro-Fi and Macro- F1. For RE task, we experimented on three rela- tion classification datasets: TACRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Stoica et al., 2021). TACRED was a large-scale sentence-level relation extraction dataset, which was obtained via crowd-sourcing. TACREV cor- rected the errors in TACRED, and ReTACRED addressed some shortcomings of TACRED, refac- toring its training set, development set and test set. ReTACRED also modified few relation types. For all these datasets, we considered the NER types of subject and object as the intermediate step (refer Han et al. (2021)), and we adopted Fj score as the metric for evaluation. The statistic details of datasets are illustrated in Table 1.
5.3 Implementation Details
Following the setting of the previous study, we adopted bert-base-uncased and roberta-base in Hugging Face library (Wolf al., 2019) the base architecture for HC task and RE task, respec- tively. We set the batch size to 8, and used Adam (Kingma and Ba, 2015) as the optimizer, the learn- ing rate was initially le-5 and decayed by 0.5 every 2 epochs. The weight decay was set to le-2. After training models for 10 epochs, we selected the best checkpoint on the training set for evaluation. For weighting coefficients, set o 0.1, , 0.1. We r 1.0. The manual templates we designed for HC task and RE task are as follows:
HC: [T], the domain [C], the area is [A]. RE: [T], the SUBJ is [A] of the 0BJ [C].
where SUBJ and OBJ refer the subject and the object of each instance, respectively. Since it was a challenge to design appropriate label words to distinguish different labels in the verbalizer (Han et al., 2021), following (Chen et al., 2022a; Wang et al., 2022b), we created the learnable virtual label word By for each label y E y
5.2 Baselines
For comparison with LLMs in NLU tasks, we em- ployed two state-of-the-art LLMs in both tasks: davinci-text-eet and 5-tturbo via the OpenAl API. For HC task, the focus of the re- cent methods is to exploit label semantics: HiAGM (Devlin al., 2019) introduced hierarchy-aware structure encoders for modeling label dependen- cies; HTCInfoMax (Deng et al., 2021) improved HiAGM by text-label mutual information maxi- mization and label prior matching; HiMatch (Chen et al., 2021) matched the text semantics and the la- bel semantics in joint embedding space; HGCLR (Wang et al., 2022a) directly embedded the hierar- chy into text encoder; HPT (Wang et al., 2022b) handled this task from a multi-label MLM perspec- tive based on prompt tuning. To better compare the performance of CoTT, we also compared our
Results and Analysis
6.1 Main Results
Table 2 & 3 exhibit the experimental results of CoTT as well as other compared baselines on HC and RE. It can be observed that our proposed CoTT achieved state-of-the-art results on all datasets. This reflected the superiority of CoTT in NLU tasks. In addition, we found that the performance of LLMs was much worse than MLMs on both tasks, which demonstrated that LLMs still cannot master some NLU tasks well.
'The concepts of HardPrompt and SoftPrompt originate from Wang et al. (2022b).

---

Methods
Micro-Fi
Macro-Fi
Methods
E.D.
TACRED TACREV ReTACRED
with LLMs
with LLMS
text-003
40.50
22.25
text-003
w/o
48.1
51.0
55.3
ChatGPT
48.50
31.83
CharGPT
w/o
44,9
43.9
52.1
tuning MLMs
Fine tuning MLMs
BERT
w/o
68.4
77.2
87.7
Vanilla Fine Tuning
85.63
79.07
Roberta
w/o
68.9
77.0
87.3
HiAGM
80.19
SpanBERT
W/
70.8
78.0
85.3
HTCInfoMax
86.30
79.97
KnowBERT
W/
71.5
79.3
89.1
HiMatch
86.70
81.06
Prompt tuning MLMS
HGCLR
87.11
81.20
HardPrompt
w/o
70.1
79.5
89.0
Prompt tuning Ms.Ms
FTR
w/o
70.8
80.2
89.1
HardPrompt
86.39
80.43
CaTT
w/o 71.8
80.6
90.0
SoftPrompt 86.57
80.75
HPT
87.16
81.93
Table 3: Experiment results of different methods TACRED, TACREV and ReTACRED. 'E.D.' refers to 'Extra Data', 'wfo" means that no additional data used for pre-training and fine-tuning, and "w/" means that extra data are used for data augmentation.
CoTT
87.46
82.49
Table 2: Experiment results of different methods WOS dataset. text-003 and ChatGPT refer to and gpt-3. 5-turbb, respectively, The best performance would be bold.
Models
Micro-Fi Fi
Macro- Fi
Vanilla Prompt Tuning
86.39
80.43
Results on HC
CoTT
87.46
82.49
PR
87.37
82.24
As shown in Table CoTT outperformed all base- lines on WOS in both metries. Compared with vanilla fine tuning, CoTT achieved 1.83% and 3.42% improvement on Micro-Fi and Macro- Fi respectively. Besides, vanilla HardPrompt and Soft- Prompt can exceed many fine tuning baselines, which revealed that the superiority of prompt tun- ing remains in small-scale MLMs. On the basis of prompt tuning, CoTT further improved the performance of MLM, which shows the effective- ness of introducing intermediate steps.
CCL
87.30
82.28
PR&CCL
87.17
82.08
Table Ablation results of CoTT on WOS dataset. PR and CCL refer Probability Rectification and Counterfactual-bant Contrastive Learning. \ denotes the removing operation.
The basis of CoTT is the introduction of inter- mediate steps. On this basis, also proposed Probability Rectification (PR) and Counterfactual- based Contrastive Learning (CCL) improve the performance of MLM. When we introduced only intermediate steps () (R&CCL) the performance of MLM clearly improved. Micro-Fi and Macro- Fi were improved by 0.78% and 1.65%, respec- tively. compared to vanilla prompt tuning. This implied the importance of intermediate steps in the reasoning process. As added PR and CCL, the performance of MLM continued improve, which demonstrated the effectiveness of both modules.
Results RE
Table 3 shows that our CoTT continued to perform better than other baselines on three RE datasets. Specifically, CoTT exceeded fine tuning Roberta by 2.9%, 3.6% and 2.7% on F1 score on TA- CRED, TACREV and ReTACRED, respectively. Note that although no extra data were introduced, CoTT could achieve better performance than those knowledge-enhaneee MLMs with fine tuning. This demonstrated that even if task-specific informa- tion was already contained in knowledge-chhanced MLMs, fine tuning struggled to take full use of (Wang et al., 2022b).
6.3 Case Study
To concretely demonstrate the performance of CoTT using intermediate steps, selected sev- eral cases from WOS dataset for evaluation, shown in Table 5. Here, recorded only the top three labels by probability in each prediction. The complete texts and the full names labels can be found Appendix B.
6.2 Ablation Study
We conducted an ablation study to investigate the independent effect of each module CoTT, and the experimental results are illustrated in Table 4.

---

Teat
Domain
Area
x61+)
plaje. /)
pv(y:x)
90
nomo nemprrn noteer qpe-
(0.87)
Come (0.83)
Gene (0.36)
Gree (0.85)
80
dies satiered Sigiificnt con
Clv (0.06)
BS (0,02)
MB (003)
MB (0.03)
70
1.0
3.
13
ination is soographizd...
Pay. (0.06)
PCR (0.02)
PCR (0.02)
PCR (0,02)
60
69
0.0
inhorssieiis is ano of the mor
Med. (0.53)
PCR (0.64)
HIA 10)
KCR 00.15)
50
thiss, whule delayed paliasi....
connon infections diseoses
BKS (0.13)
C5 (003))
Gene (0.07)
BIA (0.94)
PCR (0.04)
(1:0.12)
HBA (017)
E1 (1.0))
40
30
1.
19
5.
intahis mangnrec and cama-
Big (0.55)
PCR (0.25)
Gime (0.54)
Gene: (0,57)
20
1.
nne shores the pocific....
MAll (0.24)
Gene
(0.19)
PCR (0.33)
10
grnes....
Mis
Micro-33
Macro-F5
Hermiing los
Table 5: Three cases of CoTT on WOS dataset. The prediction probability of each result in parentheses, and the ground truth in each case is drawn in orange.
Figure 3: Performance CoTT in monitoring the rea- soning process. M refers to the Monitor.
indicates that MLM may have a misunderstanding of the intermediate step I, making the predictions relatively less reliable.
In Table 5, the first row shows how CoTT would behave under ideal conditions: in step I, MLM's prediction of intermediate steps Bio. (0.87) was correct with high confidence, which implied that MLM clearly understood this case, so the label prediction Gene (0.86) in step II more trust- worthy. In this case, probability rectification can be disregarded. However, MLM was sometimes ambiguous about intermediate steps. In the second row, the probabilities of Med. (0.52) and Bio. (0.43) were close, according to the analysis in Section 4.4, label prediction H/A (0.30) can be significantly biased. Therefore, probability rectification is nec- essary here. In this case, the label prediction was rectified PCR (0.33). The situation in the third row was akin to that in the second row, with the main difference being that the rectified label predic- tion Gene. (0.37) remained unchanged. Essentially, p( [1=)), p(y)x)), and p(yl ir, i) contain abundant in- formation, while probability rectification can adap- tively integrate the information embedded in these probabilities.
Monitor 2: Correctness of Intermediate Steps
When the true intermediate step is provided, it be- comes an option to monitor whether the prediction of intermediate steps in step is correct. Reason- ably, we consider the incorrect prediction of in- termediate steps as an anomaly: If MLM cannot predict the correct intermediate step, then the sub- sequent label prediction is less credible.
Following these two monitors, we evaluated the performance of CoTT in monitoring the rea- soning process on WOS dataset. Hamming loss (Tsoumakas and Katakis, 2007) was introduced an additional metric. As shown in Fig. 3, these monitors (MI & M2) can determine a more reason- able decision scope for CoTT, which can signifi- cantly improve the performance of COTT. In prac- tice, this ability to monitor the reasoning process can be of great use when faced with risk-sensitive tasks (e.g., medical (Quinn et al., 2020), judicial (Laptev, 2021)), since reliability is even more im- portant in these scenarios.
6.4 Reasoning Process Monitoring
Because of the introduction of intermediate steps, CoTT is no longer an end-to-end method, but can additionally monitor the reasoning process (the dis- tribution of the reasoning process can be found in Appendix C). Therefore, we can detect anomalies from the reasoning process, thus further improving the performance of CoTT.
Conclusion
In this study, we propose Chain-of-Though Tun- ing (CoTT), a two-step reasoning framework based on prompt tuning implement step-by-step think- ing for MLMs on NLU tasks. Specifically, the two-step framework of CoTT enables MLM to im- plement task decomposition; and CoTT based on prompt tuning, which allows intermediate steps to be used in natural language form. Experiments demonstrated that CoTT achieved state-of-the-arr performance on two NLU tasks. In the future, we aim to scale up MLMs fundamentally improve their language ability. In addition, using LLMs to implement multi-step reasoning in NLU tasks is also a possible research direction.
Monitor 1: Caff-Consistecy of Predictions
In step and step II of CoTT, ML. should make the label prediction with/without the intermedi- ate step f, respectively. Ideally, the two predic- tions (intuitive prediction d² and rational prediction (r.1) should be self-consistent. Therefore, we con- sider the inconsistency of label predictions as an anomaly: if the two predictions are contradictory, it

---

Limitations
Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brenman Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR.
There are two main limitations in CoTT. Firstly, the application of CoTT is relatively narrow, as COTT can only handle NLU tasks with intermediate steps. Secondly, compared to LLMs, CoTT struggles to implement interaction and cooperation between intermediate steps and to handle reasoning tasks with more steps.
Acknowledgments
This work was supported by the Shanghai Mu- nicipal Science and Technology Major Project (2021SHZDDZX0122), and the Fundamental Re- search Funds for the Central Universities.
References
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. Tacred revisited: A thorough evalua- tion of the tacred relation extraction task. ACL.
Zhongfen Deng, Hao Peng, Dongxiao He, Jianxin Li, and Philip Yu. 2021. HTCInfoMax: A global model for hierarchical text classification via information maximization. NAACI.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. NeurIPS.
Jacob Devlin. Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL.
A Allyson Ettinger. 2020. What BERT is not: Lessons from new suite of psycholinguistic diagnostics for language models. TACL.
Caoyun Fan, Wenging Chen, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. 2023a. Accurate use of label dependency in multi-label text classification through the lens of causality. Applied Intelligence.
Haibin Chen, Qianli Ma. Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-awan label semantics matching network for hierarchical text classification. In ACL.
Caoyun Fan, Wenqing Chen, Jidong Tian, Yitian Li.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020a. A simple framework for contrastive learning of visual representations. In ICML.
Hao He, and Yaohui Jin. 2023b. Improving the out-of-distribution generalization capability of lan- guage models: not enough. In ICASSP.
wwwterrfcttualll--aaeettte data is
Wenqing Chen, Jidong Tian, Liqiang Xiao, Hao He, and Yaohui Jin. 2020b. Exploring logically depen- dent multi-lask learning with causal inference. In EMNLP.
Caoyun Fan, Wenqing Chen, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. 2023c. Unlock the po- tential of counterfictually-aagmmeeedd data in out-of- distribution generalization. Expert Systems With Ap- plications.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng. Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022a. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. In WWW.
Yao Fu, Hao Peng. Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-basee prompting for multi-step reasoning, ICLR.
Yulong Chen, Yang Liu, Li Dong, Shuohang Wang. Chenguang Zhu, Michael Zeng. and Yue Zhang. 2022b. Adaprompt: Adaptive model training for based NLP. In Findings of EMNLP.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021 Making pre-trained language models better few-shot learners. ACI..
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021b. Simese: Simple contrastive learning of sentence em- beddings. In EMNLP.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts,

---

Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. PPT: pre-trained prompt tuning for few-shot learning. In ACI.
Vasiliy Andreevich Laptev, 2021. Artificial intelligence in court (judicial ai): the legal basis and prospects for its work. Russian justice.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: word-level adversarial reprogramming. In ACL.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2022. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CSUR.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: prompt tuning with rules for text classification, CoRR.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. CoRR.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know When language models know? on the calibration of language models for question answering. TACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanhert: Improving pre-training by representing and predict- ing spans. TACL.
OpenAL. 2023. GPT-4 technical report. CoRR.
Carlos Nascimento Silla Jr. and Alex Alves Freitas. 2011. survey of hierarchical classification across different application domains. Data Min. Knowl. Dis-
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. NeurlPS.
Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contex- tual word representations. In EMNLP.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- nanch Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with single QA system. Findings of EMNLP.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. Lewis, Anton Bakhtin. Yuxiang Wu. and Alexander H. Miller. 2019. Language models as knowledge bases? In EMNLP.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR.
T. Quinn. Manisha Senadeera. Stephan Jacobs, Simon Coghlan, and Vuong Le. 2020. Trust and medical ai: The challenges we face and the expertise needed to overcome them. JAMIA.
Jan Kocon, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydlo. Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocon, Bartlomiej Koptyra, Wik- toria Mieleszezenko- Kowszzewzz.zzziizz Piotr Milkowski, Marcin Oleksy, Maciej Piasecki, Lukasz Radlinski, Konrad Wojtasik, Stanislaw Wozniak, and Przemys- law Kazienko. 2023. Chatgpt: Jack of all trades, master of none. CoRR.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NeurIPS.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant, 2022. Learning to retrieve prompts for in-context learning. In NAACL.
Kamran Kowsari, Donald F. Brown, Mojtaba Hei- darysafa, Kiana Jafari Meimandi, Matthew S. Gerber, and Laura Barnes. 2017. Hdltex; Hierarchical deep learning for text classification. In JCMLA.
Timo Schick, Helmut Schmid, and Hinrich Schüizee 2020. Automatically identifying words that can serve as labels for few-shot text classification. In COLING.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL.
Viet Dac Lai. Nghia Trung Ngo, Amir Pouran Ben Vey- seh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards comprehensive evaluation of large lan- guage models in multilingual learning. CoRR.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few-shot learners. In NAACL.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR.
George Stoica, Emmanouil Antonios Platanios, and
Barnab's as P'oczos. 2021. Re-tacred: Addressing shortcomings of the tacred dataset. In AAAI.

---

Tianxiang Sun, Xiangyang Liu, Xipeng Qiu, and Xuan- Jing Huang, 2022. Paradigm shift in natural language processing. Int. J. Autom. Comput.
Pengcheng Yang. Xu Sun, Wei Li. Shuming Ma, Wei Wu, and Houfeng Wang. 2018. SGM: sequence gen- eration model for multi-label classification. In COL- ING.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models. CoRR.
E. Zelikman, Yuhuai Wu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In NeurIPS.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM- 130B: an open bilingual pre-trained model. ICLR.
Grigorios Tsoumakas and Ioannis Manousos Katakis. 2007. Multi-label classification: An overview. Int. J. Data Warehous. Min.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. In NeurIPS.
Kai Zhang, Bemal Jiménez Gutiérrez, and Yu Su. 2023. Aligning instruction tasks unlocks large language models as zero-shot relation extractors. ACL.
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In EMNIP.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In ICLR.
Zhuosheng Zhang. Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. CoRR.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. GPT-NER: named entity recognition via large language models. CoRR.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR.
Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022a. Incorporating hierarchy into text encoder: contrastive learning approach for hierarchical text classification. ACI.
Zihan Wang, Pciyi Wang. Tianyu Liu, Yunbo Cao, Zhi- fang Sui, and Houfeng Wang. 2022b. HPT: hierarchy- aware prompt tuning for hierarchical text classifica- tion. EMNLP.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Olivier Bousquet, Quee Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. ICLR.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. Trans. Mach. Learn. Res.
Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence-level relation extraction. CoRR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022b. Chain-of-thoul thought prompt- ing elicits reasoning in large language models. In NeuriPS.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of the-offf-hh-rr-ee art natural language processing. CoRR.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of IIms in prac- tice: A survey on chatgpt and beyond. CoRR.

---

A Experimental Details of LLMs
Text
We tested LLMs in the 4-shot scenario due to text length limitations, and we sampled 500 texts in each dataset for evaluation. To enable LLMs to im- plement NLU tasks, we added the task description and label set to the prompt as:
the parden deemouse eliomys quercions, a native cumpein rodent spacies, sufforod signilicant contraction its googaphical range in the last few decades, species disappeared from large parts cental and castern curupe and is considered estinct in some countrios. reviewed the infor- mation available the occurience distritution of the specics 26 countrics where was previously reported. present and past introduction outside its native range were also summarised. the gunten dormouse considered extinct in liturania, finland, and slovakia, probably extinct helarns, and present with singl populaation in the notherands, poland, and slovenia; in slovakis, however, monitoring is necessary verify rocent records, the spacies is rare and localised austris, ukriine romanis, croatia and is in regression in gemanny flandess (belgium), czech republic. latvia, and estonia. 2015, the garden doemouse occupied 4915 of its 1978 geographical range and 67% of its 2008 runge. south-wsserm curope the strougheld of the speeies, is still commen in large parts of portugal, are indications that
Based on the abstract of the paper, choose the do- main and area of the paper. The optional domains are: {domains, : domaing, ".ddmmin...,dmmiin...).....
domain The optional areas are: {areal, areag,.... areag) The abstract of the paper is "x1", ", the domain is "d "domaini", ", the area is "area1".
spain, frunce, and italy. however, countries, the species is locally declining. present knowledge cannot es- plan the extensive regression of the garden dormouse range central and castern eumope probably. is resall of interaction of different facturs, acting locally and ai lang scale, and related specific ccological roguiremens the spacies. there is string the reusons for the dramatic population and geographical range contraction of the ganlen dormonse meanelille it impor tant to monicee this spocios idensify appropriate conservaiion messures.
these
for research determine
The abstract of the paper is "x4", the domain is "domains", ", the area is "areas". The abstract of the paper is "x",
2
tuberculosis is one of the most cammon infectious disesses in chinn, while delayed patien: finding obstructed discase control espocially for negative pitients. the current stuly was undertsken to evalusic the diagnce- tie accuracy of genexpert mibtii compared with comeetional methods in the detection pulmonury tuherculosis patiests. total of 295 spet sputum samples from confirmed palmonary tuberculonis pasients were evaluated from september 2014 to june 2015. each sample was examined by acid- bacillus smear mictoscopy. culture genespert mib/rif. sputum cultree onfowenttiinjeree
B Supplementary Information in Case Study
To demonstrate the performance of CoTT, we se- lected several cases from WOS dataset? to evaluate in case study section. We display the full names of labels, along with the complete texts of the selected cases in Tables & 7.
[-j]ucs considerd as the gokd-stiddand after testing by smear. 68.81% (203 295) negutive and 31.19% (92/295) was positive. is Ihegolf-stnddrll cuiture detected 37.97% (1 (112/295) positive of all specimens, while the positivity for genexpert miblii was 46 44% (133/205). compared l-j cullure, specifiiity, posltive predictive value (pps) and negative predictive value (upy) for gesexpent milivs were 94 44.645, 82.97% 77. 37% and 96, 18% respectively. smear negutive specimens, the sensitivity, specificity, ppy and new for gesespent mil rif were 96 96.009 83 05% 44 44.4% and 99, 32%; while for mesiiioosiivve specimens, the cortopoodingg accuracy values were 94.25% s0005, 98 80% and 44.445 the findings of study indiested that genexpert mibiti aaszy demonstrated a high sensitivity in detecting mycobaciciumm tuberculosis compared smear method a high npv among smear negative patients. (c) 2017 chevierr hd. all rights reserved.
combised sensiiivity
C Decision Distribution of CoTT
With the introduction of intermediate steps, CoTT is longer an end-to-end approach, so it is feasible to monitor the reasoning process. We counted the reasoning process of CoTT on WOS dataset, and calculated the decision distribution, as shown in Fig. 4.
3 this study cramined the ply logy of the barraele fistulobalaemss albicostatus, which inhabits mangroves and estuarine shares pacific. differentition in the mitcchodrial cytochrome e exiduse subunit (12s)
west
(col) and 12s ribosomal was crsmined in samples from 16 locations in west pacific, ring from honshu to souther china. our results reveuled that t.albicontatss comprises mujor clades exhibiting 1.25% to 2.8% clade demonstrated the widest distribution, ranging from japan chins, and was divided into throe subcludes occurring the south china sea (al), olinawa (a2), and hondha, kores and qingdan (23). clade was determined to he ondemic to okinawa: i.. two endemmi lincaees occur this island. thus, falbicostatas resembles several inter tidal species in having clades that are endemic to okinzwan waters. neverthlless, in contrast to the rocky inser-iddal bamacles tetrahii spp. and chthamalus mullaeemis, wultti and aceanic populazions, but instcad is divided inco noethem and southern clades, probably because of the yangtar river discharge, which limits gone flow between the northeen southem populations.
of 401 specimens fallbicostatus
divergenae ranging from
Sy Pu
not found be sepiratial into cominental
Saalies Blagrnn
100%
2:42 arigld
9J%
GI swliddd
Table 6: The complete texts of the cases in case study.
Figure 4: Decision distribution of CoTT
Full Name
Abbreviation
Biochemistry
Bio.
The statistical results show that the decision de- tails provided by the two-phase reasoning frame- work can help CoTT adaptively adjust the decision scope and thus improve performance. This cannot be achieved by end-to-end methods.
Genetics
Gene.
Civil
Civ.
Remote Sensing
RS
Molecular Biology
MB
Psychology
Psy.
Polymerase Chain Reaction
PCR
IIVAAIDS
II/A
Fungal Infection
Computer Science
CS
MAE
MAE
'It is available at web_of_ _scee__s
Table 7: The full names of labels in case study.


Reviews:
[{"review_id": "Rrfp5Tl7qS", "reviewer": "Reviewer_VXXs", "summary": "", "questions": "1.\tThe chain-of-though style technique requires the task to be solved in a step-by-step manner, stressing the order of subparts after decomposition. But solution flows are not only a flat chain of thoughts but also others like tree-like style or complex combination. \n\n2.\tSince this work mentions the task decomposition and degrade the importance of P(y|x,I) in Line370 - 372, there is a lack of comparison between CoTT with a parallel multi-task framework. As we know, there are many works with multi-task frameworks targeting intermediate output for further better final prediction.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts and constructive suggestions on our paper. We will further revise our paper based on the reviewer's comments. Here is our response, and we mark the $\\color{#660000}{\\text{CRITICAL}}$ issues in red that may influence the reviewer's judgments. \n\n_Q1_: Solution flows are not only a chain but also others like tree-like style or complex combination. ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A1_: I agree with the reviewer that a generic solution flow is not in the form of a chain but is much more complex. There are several reasons why our research (CoTT) uses a chain-based solution flow: \n- Language perspective. The motivation for the original CoT is to explicitly represent the reasoning process through language. And language is linear and seems to be best suited for a chain-based solution flow. For LLMs, CoT can perform tree-like solution flow (e.g., complex reasoning, theorem proving), which we attribute to the stronger reasoning ability of LLMs. MLMs do not have the reasoning ability of LLMs, and so we restrict the solution flow to a chain.\n- Flow perspective. Theoretically, any solution flow can be regarded as a combination of chains. So, we consider it is also important to study the basic unit (chain) of the solution flow, especially when current techniques are not sufficient to solve the generic solution flow. \n- Effectiveness Perspective. Our proposed CoTT significantly improves the ability of MLMs in some NLU scenarios. In line with the EMNLP philosophy, we believe that empirical methods (even with limitations) are valuable to the NLP community. \n\n_Q2_: Lack of baselines with a parallel multi-task framework. \n\n_A2_: We chose the baselines based on previous researches. For the HC task, the intermediate labels themselves are included in the evaluation metrics, so the intermediate labels are included in the training objectives. Therefore, some baselines of HC (Vanilla Fine Tuning, HardPrompt, SoftPrompt) can all be considered as parallel multi-task frameworks. For the RE task, the intermediate labels are independent of the evaluation metrics. This may have led previous researches to ignore the baselines of parallel multi-task frameworks in RE, so we do not include this type of baselines. We consider the reviewer's comment of parallel multi-task framework to be constructive, and we will include baselines for parallel multi-task frameworks in the RE task in subsequent versions of the paper.\n\nAdditionally, we would like to express our gratitude to the reviewer for pointing out the presentation problems in our paper. This feedback is valuable not only for this paper but also for our future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis work shows a comprehensive understanding of task background and challenges. \n\n2.\tCoTT is a straightforward but novel solution to study traditional natural language understanding tasks that can be decomposed into two sequential sub-tasks.\n\n3.\tThis work provides a clear and solid introduction to the model framework,  a rigorous experimental setup, and a plentiful and noteworthy discussion.", "reasons_to_reject": "1.\tThe chain-of-though style technique requires the task to be solved in a step-by-step manner, stressing the order of subparts after decomposition. But solution flows are not only a flat chain of thoughts but also others like tree-like style or complex combination. \n\n2.\tSince this work mentions the task decomposition and degrade the importance of P(y|x,I) in Line370 - 372, there is a lack of comparison between CoTT with a parallel multi-task framework. As we know, there are many works with multi-task frameworks targeting intermediate output for further better final prediction.", "questions_to_authors": "1.\tThe chain-of-though style technique requires the task to be solved in a step-by-step manner, stressing the order of subparts after decomposition. But solution flows are not only a flat chain of thoughts but also others like tree-like style or complex combination. \n\n2.\tSince this work mentions the task decomposition and degrade the importance of P(y|x,I) in Line370 - 372, there is a lack of comparison between CoTT with a parallel multi-task framework. As we know, there are many works with multi-task frameworks targeting intermediate output for further better final prediction.", "paper_topic_and_main_contributions": "This work proposes a chain-of-thought tuning on mask language models to solve the natural language understanding tasks in a step-by-step manner. Inspired by the success of the large language model, decomposing the complex task into multiple intermediate subtasks, the authors design a step-by-step tuning on the mask language model to seek achievement on traditional natural language understanding tasks. The proposed framework consists of a two-step tuning process, where the task is sequentially divided into two stages, with the final prediction being conditioned on the intermediate output. The proposed model shows impressive new SOTA results on hierarchical classification and relation extraction tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "For Section 5, you may use simple present tense rather than past tense, because we usually use simple present tense to introduce facts."}, "extra_text": {}}, {"review_id": "c1nFR80m8N", "reviewer": "Reviewer_xtiA", "summary": "", "questions": "- A. You introduced contrastive learning in Step II to explicitly guide the integration of information from input text and intermediate step (ll.288-296), but is it not already explicitly guided by the natural language text parts of the template other than the slot? \n- B. I am not convinced that introducing contrastive learning. You explain that the prediction in Step I is an intuitive prediction and the prediction in Step II is a rational prediction. Why learning to make the hidden states for them similar leads to integrating information from the input text and the intermediate step?\n- C. CoTT seems to require multiple forward and backward processes in training phase. How much the cost compared to the baseline methods? Is the inference cost about twice as much?\n- D. You selected the best checkpoints on the training set for evaluation (ll.465-466). Is it natural to use the training set instead of the dev set for the HC and RE tasks?\n- E. Can CoTT be combined with knowledge-enhanced MLMs to better exploit their potential?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts and constructive suggestions on our paper. We will further revise our paper based on the reviewer's comments. Here is our response, and we mark the $\\color{#660000}{\\text{CRITICAL}}$ issues in red that may influence the reviewer's judgments. \n\n_Q1_: The descriptions in Section 4 are difficult to understand, especially the explanation of the convertible slot. (Reasons To Reject 1)\n\n_A1_: While writing this paper, we also found this problem - how to logically and smoothly introduce *convertible slot* [C] (Section 4.1). Our writing logic is to pre-define *text slot* [T] and *answer slot* [A] in Preliminaries (Section 3). Then in Section 4.1, [C] is treated as a kind of slot for converting between [T] and [A], and in order to help readers understand [C] more conveniently, the notation and expressions in Section 4.1 are referred to Section 3. We do not provide concrete examples in Section 4.1 because we consider the examples to be task-based, and we describe the tasks we use to evaluate CoTT (HC and RE) in Section 5, so the corresponding templates are also organized in Section 5 (lines 469-471). \n\n_Q2_: Why does CoTT introduce contrastive learning to explicitly integrate text and intermediate steps? This integration should be implemented via prompt. (Question A) ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A2_: The motivation for introducing contrastive learning is to __prevent MLMs to learn the correlation shortcut__. Due to space constraints, we only briefly introduce the role of contrast learning in the paper. Since the reviewer has some questions about our motivation for introducing contrast learning, we will explain it in detail. \n- What is the correlation shortcut. A correlation shortcut [1,2] is that a deep learning model may make decisions based on the statistical correlation of partial information, even if it is provided with complete information for causal inference. The essence of this phenomenon is that the optimization process of deep learning is unable to distinguish between correlation and causality of data [3]. A common example of this is that a model distinguishes between a cow and a camel based on the context (grass/desert) rather than the object (cow/camel). \n- Why does the correlation shortcut exist in CoTT? Because there is often a strong statistical correlation between intermediate steps and the final results. For example, there is clearly a high statistical correlation between the intermediate step (Biochemistry) and the final result (Genetics). However, it is unreasonable to predict Genetics solely based on Biochemistry (ignoring the text), even though it may seem effective. However, due to the inability to distinguish between correlation and causality, MLMs are likely to learn this correlation shortcut while ignore the correct reasoning process (integrating text and intermediate steps to predict the final result). \n- Why prompt cannot block the correlation shortcut. The main role of prompt in CoTT is to provide the complete information required for causal inference to MLMs through slots. Specifically, prompt makes text and intermediate steps available to MLMs in the most straightforward manner. But as mentioned before, even provided with complete information, MLMs may make decisions based on the statistical correlation of partial information (the correlation shortcut). Therefore, it is necessary to explicitly guide MLMs to integrate text and intermediate steps. (__for Question A__)\n- Why contrastive learning can block the correlation shortcut. To explicitly integrate text and intermediate steps, we introduce Counterfactual-based Contrastive Learning (CCL). Specifically, there are three hidden vectors in CCL: hidden vector $h_x$ without intermediate steps (in step I), hidden vector $h_{x, I}$ with correct intermediate step (in step II), hidden vector $ h_{x, I^*}$ with counterfactual intermediate step (in step II). The goal of CCL is that $h_{x, I}$ is close to $h_x$, and $h_{x, I^*}$ is far away from $h_x$. Intuitively, by contrasting the similarity of $h_x$ and {$h_{x, I}, h_{x, I^*} $}, MLMs have to distinguish whether $x$ and $I$ match: when they do, the hidden vectors of two steps are similar, and vice versa. Overall, CCL forces MLMs to perceive the relationship between text and intermediate steps more fine-grained, and integrates the perceived information into the hidden vector $h$, thereby restricting MLMs from exploiting the correlation shortcut (text cannot be ignored). (__for Question B__)\n\n_Q3_: Why make hidden vectors similar in contrastive learning to integrate information from text and intermediate step? (Question B) ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A3_: We explain in detail the reasons for introducing contrastive learning in _A2_. Our interpretation of this question is in the last point in _A2_.\n\n_Q4_: CoTT seems to require multiple forward and backward processes in training phase. How much the cost compared to the baseline methods? Is the inference cost about twice as much? \n\n_A4_: Yes, CoTT requires two forward and backward processes in the training phase and two forward processes in the inference phase. Compared to the most basic baseline (Vanilla Prompt Tuning), the computational effort is about twice as high. However, some of the advanced baselines we have chosen also significantly increase the computational effort. In addition, CoTT allows for more information to be obtained from the two forward processes, which can increase the interpretability of CoTT (Table 5), as well as detecting anomalies to determine a reasonable decision scope (Section 6.4). Overall, CoTT trades increased computational effort for better performance, as well as increased interpretability and reliability. \n\n_Q5_: Is it natural to use the training set instead of the dev set to select the best checkpoints? (Question D)\n\n_A5_: We agree with the reviewer that using dev set to select checkpoints is a more rigorous approach according to the principles of machine learning. However, we did find many studies on HC and RE tasks that used the TRAINING SET to select checkpoints. After our verification, in most cases the two results are consistent. However, we consider that we should follow the principles of machine learning and adopt a more rigorous approach, and we will make corresponding changes in the subsequent versions of the paper. \n\n_Q6_: Can CoTT be combined with knowledge-enhanced MLMs to better exploit their potential? (Question E)\n\n_A6_: We believe that this potential combination is feasible because the proposed CoTT is theoretically applicable to any MLMs. Our study only used traditional MLMs (BERT, Roberta) in order to allow for clearer comparisons with the baselines. We consider the reviewer's comments to be constructive, and we will include the experimental results of CoTT combined with knowledge-enhanced MLMs in the subsequent version of the paper.\n\n[1] Shah, Harshay, et al. \"The pitfalls of simplicity bias in neural networks.\" NeurIPS (2020).\n\n[2] Feder, Amir, et al. \"Causalm: Causal model explanation through counterfactual language models.\" Computational Linguistics (2021).\n\n[3] Fan, Caoyun, et al. \"Accurate use of label dependency in multi-label text classification through the lens of causality.\" Applied Intelligence (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposed Chain-of-Thought Tuning (CoTT), an interesting framework allowing MLMs to solve NLU tasks by two-step inference.\n- CoTT achieves state-of-the-art performance on two NLU tasks, HC and RE.\n- Conducted appropriate ablation studies.\n- Introducing an intermediate step may allow us to monitor the inference process of the model and increase the explainability of the model.", "reasons_to_reject": "- Description of the proposed method (Section 4) is not friendly \n    - Concrete examples of prompt template are not provided for explaining the method.\n    - Especially, it is difficult to understand the explanation of the convertible slot in Section 4.1 without a concrete example because it is before the explanation of steps I and II.\n- I do not understand the motivation to introduce contrastive learning (Question A, B)\n- The costs of training and inference are not stated (Question C)\n- Only two tasks were used in the evaluation, and it is unclear how effective CoTT is for other NLU tasks, especially tasks in GLUE or SuperGLUE.", "questions_to_authors": "- A. You introduced contrastive learning in Step II to explicitly guide the integration of information from input text and intermediate step (ll.288-296), but is it not already explicitly guided by the natural language text parts of the template other than the slot? \n- B. I am not convinced that introducing contrastive learning. You explain that the prediction in Step I is an intuitive prediction and the prediction in Step II is a rational prediction. Why learning to make the hidden states for them similar leads to integrating information from the input text and the intermediate step?\n- C. CoTT seems to require multiple forward and backward processes in training phase. How much the cost compared to the baseline methods? Is the inference cost about twice as much?\n- D. You selected the best checkpoints on the training set for evaluation (ll.465-466). Is it natural to use the training set instead of the dev set for the HC and RE tasks?\n- E. Can CoTT be combined with knowledge-enhanced MLMs to better exploit their potential?", "paper_topic_and_main_contributions": "Natural language understanding (NLU) tasks are still difficult for LLMs to solve, and smaller masked language models (MLMs) still outperform them.\nThen, as an analogy to Chain-of-Thought (CoT), which improves the performance of LLMs, this paper introduces a step-by-step reasoning method for MLMs.\nSpecifically, this paper proposes Chain-of-Thought Tuning (CoTT), a framework that implements two-step reasoning by introducing an intermediate step to cloze-style prompt tuning in MLMs, and decomposes NLU tasks and performs step-by-step reasoning.\n\nCoTT outperforms existing fine-tuning and prompt-tuning methods in two NLU tasks, Hierarchical Classification (HC) and Relation extraction (RE), and achieves state-of-the-art performance.\nFurthermore, the introduction of an intermediate step allows the reasoning process of the model to be monitored and could increase the explainability of the model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Text in Table 5 is too small"}, "extra_text": {}}, {"review_id": "kTrEhe6fsZ", "reviewer": "Reviewer_AVcC", "summary": "", "questions": "A. What is the performance of different CoT variants [1,2,3] of LLMs for the two evaluated tasks? \n\nB. Why is SoftPrompt not included in Table 3?\n\nC. The interpretation of Figure 3 seems incomplete. For instance, how to determine reasonable decision scope for CoTT based on Figure 3 (Line 592-594)?\n\nD.For CCL, what is the size of the counterfactual step I* (Line 304)? Is there only a single negative sample for CCL? Since CL benefits from larger batch sizes and more negative samples, it would make more sense to enlarge I* for CCL.\n\n[1] Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. \n\n[2] Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.\n\n[3] Fu et al., Complexity-Based Prompting for Multi-step Reasoning. ICLR 2023.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts and constructive suggestions on our paper. We will further revise our paper based on the reviewer's comments. Here is our response, and we mark the $\\color{#660000}{\\text{CRITICAL}}$ issues in red that may influence the reviewer's judgments. \n\n_Q1_: CoTT has marginal performance gains? (Reasons To Reject 1)\n\n_A1_: Firstly, we consider that a performance gain of around 1% is also acceptable on traditional NLU tasks like HTC and RE. Second, we value the insights that the research provides to the NLP community more than the performance gains, and in this research, the insight we want to provide is to migrate the CoT technique used for LLMs to MLMs.\n\n_Q2_: The major components of CoTT are CCL and PR. (Reasons To Reject 3) ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A2_: The core of CoTT is a __two-stage reasoning framework__. Our motivation for proposing CoTT is to introduce the idea of CoT into MLMs so that MLMs can predict the final result more directly using intermediate steps. __Both CCL and PR are auxiliary modules of CoTT__ rather than the main components. From the ablation experiments in Table 4, we can also see that the two-stage reasoning framework of CoTT is significantly better than the baseline even without the inclusion of the two modules, CCL and PR. \n\n_Q3_: The scope of the study is under-specified. More CoT-based baselines are needed. (Question A) ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A3_: This question is very important and relates to the motivation of our research, so we will explain it in detail. \n- First, we emphasize the motivation of our research. Currently, LLMs have brought the field of NLP into a whole new era, but essentially the way LLMs solve NLP tasks is by transforming everything into a text generation task. However, NLU tasks (which require matching text with a large number of labels) are not suitable for this transformation. So, given that MLMs are more suitable for solving NLU tasks, we would like to transfer the CoT technique from LLMs to MLMs. \n- The scope of our research is to transfer the CoT technique originally used in LLMs to MLMs. The essential difference between these two language models is autoregressive/masked, rather than the scale of the model (although the scale of LLMs is usually much larger than that of MLMs). An autoregressive language model predicts the next word based on the previous text, whereas a masked language model predicts all <MASK> in parallel. The solution in our study is to transfer the CoT technique used for autoregressive language models to MLMs using a __two-step reasoning framework__ (in Fig. 2).\n- We did not employ more LLMs+CoT baselines because LLMs are not able to efficiently transform NLU tasks into text generation tasks, which results in LLMs performing much less well than MLMs on NLU tasks, so we do not think that the introduction of more LLMs+CoT baselines would have brought more insight to our research. However, we agree with the reviewer's comments and will include more LLMs+CoT baselines in the subsequent version of our paper, although we believe that the performance of these baselines will be much lower than that of MLMs. \n\n_Q4_: Why is SoftPrompt not included in Table 3? (Question B)\n\n_A4_: In fact, HTC and RE are two different NLU tasks with different baseline approaches. SoftPrompt is not widely regarded as a baseline approach in studies on RE, so we do not include SoftPrompt in Table 3. However, we believe that SoftPrompt can solve the RE task, and we will add the result of SoftPrompt on RE in a subsequent version of the paper.\n\n_Q5_: The interpretation of Fig. 3 seems incomplete. How to determine reasonable decision scope for CoTT? (Question C)\n\n_A5_: We propose two monitoring methods in Section 6.4 to improve the performance of CoTT by determining a reasonable decision scope. The two methods are __Monitor 1: Self-Consistency of Predictions__ (Line 570-580) and __Monitor 2: Correctness of Intermediate Steps__ (Line 581-588). And Fig. 3 shows CoTT's performance under these two monitoring methods. The reviewer may have overlooked that M1 and M2 in Fig. 3 are abbreviations for Monitor 1 and Monitor 2. \n\n_Q6_: For CCL, what is the size of the counterfactual intermediate step $I^*$? (Question D)\n\n_A6_: We sample only one counterfactual intermediate step in CCL. On the one hand, more sampling would bring extra computation, and on the other hand, since the counterfactual intermediate steps are sampled from $p_{\\notin \\hat{I}}(I|x)$, and this probability distribution is likely to be not very uniform, but distributed over certain confusing intermediate steps (e.g., Biochemistry and Genetics). Our aim in CCL is to help MLMs distinguish between these confusing intermediate steps, so the gain from sampling more counterfactual intermediate steps may not be as significant as in CL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The overall structure of the work as well as motivations for the integration of CoT and individual components is  well written.\n\n2. The adaptation of CoT to NLU tasks is quite interesting and provides fresh perspectives for NLU tasks.", "reasons_to_reject": "1. The reported performance gain of the proposed framework is marginal when compared to the improvements introduced by simple Prompt Tuning approaches. For instance,for Table 3, out of 2.7% gain over Roberta backbone on ReTACRED, prompting tuning (i.e. HardPrompt) already achieves the gain of 1.7%. \n\n2. The scope of the study is under-specified. It seems that the work focuses on injecting CoT- based approach to small-scale Language Models. If that is not the case, additional relevant CoT baselines for in-context learning of Large Language Models (for text-003 and ChatGPT) are missing in Table 2 and 3 (See Question A).\n\n3. The major components of the proposed frameworks are CCL and PR. Both of them are incremental over the previous methods with minor adaptation for CoT-based prompting proposal.", "questions_to_authors": "A. What is the performance of different CoT variants [1,2,3] of LLMs for the two evaluated tasks? \n\nB. Why is SoftPrompt not included in Table 3?\n\nC. The interpretation of Figure 3 seems incomplete. For instance, how to determine reasonable decision scope for CoTT based on Figure 3 (Line 592-594)?\n\nD.For CCL, what is the size of the counterfactual step I* (Line 304)? Is there only a single negative sample for CCL? Since CL benefits from larger batch sizes and more negative samples, it would make more sense to enlarge I* for CCL.\n\n[1] Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. \n\n[2] Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.\n\n[3] Fu et al., Complexity-Based Prompting for Multi-step Reasoning. ICLR 2023.", "paper_topic_and_main_contributions": "The authors propose Chain-of-Thought Tuning framework in an attempt to unify the benefits of Masked Language Modeling (MLM) and Chain-of-thought (CoT) in two NLU tasks: hierarchical classification and relation classification. Specifically, the framework is divided into 2 steps (1) Intermediate Step Generation, (2) Target Outcome Prediction via Intermediate Step generated in step (1). Additional Counterfactual-based Contrastive Learning (CCL) and Probability Rectification (PR) are also proposed to effectively integrate intermediate steps into the overall framework.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 135:  the final chosen answers\n\nLine 519 and 524: , -> .\n\nTense consistency across the manuscript is recommended."}, "extra_text": {}}, {"review_id": "e1yYasfJEF", "reviewer": "Reviewer_bKM7", "summary": "", "questions": "1) Line 42-48. The paper mentions that LLM needs to be improved for NLU tasks such as GLUE and SuperGLUE NLU datasets. However, the paper only evaluates the proposed approach in two tasks, HTC and RE. Can the paper evaluates the proposed approach in more NLU tasks?\n\n2) While this paper primarily discusses integrating CoT into MLM, it would also be useful to demonstrate the effectiveness of open-source LLMs like LLaMa and ChatGLM using the CoT tuning approach in two steps.\n\n3) The original CoT consists of natural language sentences that make up a reasoning process. However, the proposed CoT uses intermediate labels without detailed explanations. My question is, why are these intermediate labels referred to as CoT?\n\n4) Which dataset did this paper use for the experiment shown in Figure 3?\n\nI will update the final score according to the authors responses.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts and constructive suggestions on our paper. We will further revise our paper based on the reviewer's comments. Here is our response, and we mark the $\\color{#660000}{\\text{CRITICAL}}$ issues in red that may influence the reviewer's judgments. \n\n_Q1_: Can we evaluate CoTT in more NLU tasks, such as GLUE and SuperGLUE? ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A1_: CoTT emphasizes the introduction of intermediate steps to MLMs in NLU tasks, therefore, CoTT imposes some limitations on NLU tasks (__intermediate steps in suitable forms__). We explain this from both task and model perspectives: \n- Task perspective. Not all NLU tasks require intermediate steps. In fact, the process of many NLU tasks in GLUE and SuperGLUE are so simple that introducing intermediate steps is redundant. This is not a situation specific to MLMs, and the original CoT technique is not universal for LLMs, as many text generation tasks do not require intermediate steps (think step by step). \n- Model perspective. MLMs limit the form of intermediate steps. Since MLMs can only perform cloze-style tasks, the intermediate steps cannot be natural language forms of sentences, but only some predefined set of intermediate steps.\n\nWe chose HTC and RE to evaluate CoTT also because these two tasks require intermediate steps and intermediate steps can be defined as a finite set. Although CoTT imposes some restrictions on NLU tasks, it is capable of handling much more NLU tasks than HTC and RE, for example, it also has potential in NLU tasks such as information retrieval and common-sense reasoning. \n\n_Q2_: Can CoTT be introduced in open-source LLMs? ($\\color{#660000}{\\text{CRITICAL}}$)\n\n_A2_: This question is quite important and relates to the motivation of our research -- to improve the ability of MLMs (__not LLMs__) on NLU tasks, so we will explain it in detail. \n- Task perspective: LLMs are not suitable for NLU tasks. Although LLMs have brought the NLP field into a whole new era, essentially the way LLMs solve NLP tasks is to transform everything into text generation tasks. However, many NLP tasks do not suit this transformation, for example, text classification tasks require matching text with a large number of labels, and a more typical task is information retrieval, which requires finding the most relevant content from a huge amount of information. If these NLP tasks are not transformed into text generation tasks and language models are considered only as a textual feature extractors, MLMs (e.g. DeBERTaV3[1]) are still a better choice than LLMs (e.g. LLaMa, ChatGLM). \n- Model perspective: LLMs do not require a two-step process. LLMs are usually autoregressive language models, i.e., they predict the next word based on the previous text. Therefore, LLMs do not need a two-step process like CoTT because autoregressive models automatically use the generated intermediate steps in the generation process to predict the final result. But MLMs predict all <MASK> in parallel, not autoregressively. This is the reason why we introduced a two-step process: we want MLMs to also use the intermediate steps for predicting the final result as LLMs do, so in the first step MLMs predict the intermediate steps and then fill the templates with the intermediate steps for the second step. As for the reason why we force MLMs to do what it is not good at, you can refer to our explanation from the task perspective (LLMs cannot solve all NLP tasks). \n- Cost perspective. LLMs cost more. As academic research, cost does not seem to be the most critical reason. However, it has to be said that the cost of training and use of LLMs far exceeds that of MLMs. so, currently most fine-tuned LLMs are also so-called domain LLMs -- i.e., have the ability to solve a range of tasks in a specific domain. Whereas fine-tuned MLMs are much less costly and can therefore be adapted for a specific single task. So, I think a comparison between prompting LLMs and fine-tuning MLMs is a much more reasonable choice. \n\n_Q3_: Why intermediate labels referred to as CoT?\n\n_A3_: The motivation for CoT is to express the reasoning process explicitly[2]. First, do intermediate labels express the reasoning process explicitly? We believe it does, because the intermediate labels we chose for both HTC and RE are a necessary component of the reasoning process, even humans (implicitly) make use of these intermediate labels. Second, why can we only use the form of intermediate labels and not the form of natural language? As stated in _A1_, since MLMs can only perform cloze-style tasks, generating natural language is not something that MLMs are good at. Overall, intermediate labels are a form that fits the reasoning process of MLMs. \n\n_Q4_: Which dataset did we use for the experiment shown in Figure 3?\n\n_A4_: We thank the reviewer for pointing out the oversight in our manuscript. The experimental results presented in Fig. 3 were obtained on the WOS dataset, and we will add this information in subsequent versions of the manuscript. \n\n\n\n[1] He, Pengcheng et al. “DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing.” ICLR 2023.\n\n[2] Wei, Jason et al. “Chain of Thought Prompting Elicits Reasoning in Large Language Models.” NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The proposed CoTT technique introduces a two-step reasoning framework based on prompt tuning to facilitate step-by-step thinking in MLMs. The proposed approach is novel.\n\n2) The paper provides a strong motivation for exploring the transferability of CoT from LLMs to MLMs.\n\n3) The paper is well written and easy to follow.", "reasons_to_reject": "1) The evaluation only covers two NLU tasks: hierarchical classification and relation extraction. Although these tasks are useful for evaluation, a wider evaluation of diverse NLU tasks would strengthen the generalizability of the proposed CoTT framework.\n\n2) Based on the results presented in Tables 2 and 4, the proposed method and its various components do not show significant benefits.", "questions_to_authors": "1) Line 42-48. The paper mentions that LLM needs to be improved for NLU tasks such as GLUE and SuperGLUE NLU datasets. However, the paper only evaluates the proposed approach in two tasks, HTC and RE. Can the paper evaluates the proposed approach in more NLU tasks?\n\n2) While this paper primarily discusses integrating CoT into MLM, it would also be useful to demonstrate the effectiveness of open-source LLMs like LLaMa and ChatGLM using the CoT tuning approach in two steps.\n\n3) The original CoT consists of natural language sentences that make up a reasoning process. However, the proposed CoT uses intermediate labels without detailed explanations. My question is, why are these intermediate labels referred to as CoT?\n\n4) Which dataset did this paper use for the experiment shown in Figure 3?\n\nI will update the final score according to the authors responses.", "paper_topic_and_main_contributions": "The paper proposes an approach called \"Chain-of-Thought Tuning (CoTT)\" that aims to enable MLMs to perform step-by-step thinking in NLU tasks. The CoTT framework involves two steps: Step I, where the MLM generates an intermediate step and label based on the input text, and Step II, where the MLM predicts the final result based on the text and the intermediate step. The paper presents experimental results on hierarchical classification and relation extraction tasks, demonstrating that CoTT outperforms baseline methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 33: process -> processes\n2) Line 39: despite -> although"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This journal paper presents a novel framework called Chain-of-Thought Tuning (CoTT) for solving natural language understanding (NLU) tasks. 
CoTT is designed to decompose NLU tasks into two sequential sub-tasks and incorporates prompt tuning to enable step-by-step reasoning in masked language models (MLMs). It also includes ablation studies and highlights the potential for increased explainability of the model by introducing an intermediate step. The integration of CoTT to NLU tasks is well-motivated and provides fresh perspectives for the field. 
However, the significant components of the proposed frameworks are minor adaptations from previous methods. The evaluation only covers two types of tasks and uncovers how effective the chain-of-thought style technique is for other NLU tasks. 
Moreover, the reported performance gain of the proposed framework is marginal compared to simple Prompt Tuning approaches. 
Therefore, a more comprehensive evaluation of diverse NLU tasks would strengthen the proposed framework's generalizability.

Overall, the paper is well-written, easy to follow, and demonstrates the transferability of CoT concepts from language models to masked language models.
Decision: Accept-Main