SlowGuess/NAACL_2022_OCR huggingface dataset:
Haode Zhang $^{1}$  Haowen Liang $^{1}$  Yuwei Zhang $^{2}$
Liming Zhan $^{1}$  Xiaolei Lu $^{3}$  Albert Y.S. Lam $^{4}$  Xiao-Ming Wu $^{1*}$
University of California, San Diego<sup>2</sup> Nanyang Technological University, Singapore<sup>3</sup>
Fano Labs, Hong Kong S.A.R.
haode.zhang,michaelhw.liang,lmzhan.zhan}@connect.polyu.hk,zhangyuwei.work@gmail.com
xiao-ming.wu@polyu.edu.hk, xiaolei.lu@ntu.edu.sg, albert@fano.ai
It is challenging to train a good intent classifier for a task-oriented dialogue system with only a few annotations. Recent studies have shown that fine-tuning pre-trained language models with a small amount of labeled utterances from public benchmarks in a supervised manner is extremely helpful. However, we find that supervised pre-training yields an anisotropic feature space, which may suppress the expressive power of the semantic representations. Inspired by recent research in isotropization, we propose to improve supervised pretraining by regularizing the feature space towards isotropy. We propose two regularizers based on contrastive learning and correlation matrix respectively, and demonstrate their effectiveness through extensive experiments. Our main finding is that it is promising to regularize supervised pre-training with isotropization to further improve the performance of few-shot intent detection. The source code can be found at https://github.com/fanolabs/isoIntentBert-main.
Intent detection is a core module of task-oriented dialogue systems. Training a well-performing intent classifier with only a few annotations, i.e., few-shot intent detection, is of great practical value. Recently, this problem has attracted considerable attention (Vulić et al., 2021; Zhang et al., b; Dopierre et al., b) but remains a challenge.
To tackle few-shot intent detection, earlier works employ induction network (Geng et al., 2019), generation-based methods (Xia et al., a), metric learning (Nguyen et al., 2020), and self-training (Dopierre et al., b), to design sophisticated algorithms. Recently, pre-trained language models (PLMs) have emerged as a simple yet promising solution to a wide spectrum of natural language processing (NLP) tasks, triggering the surge of PLM
based solutions for few-shot intent detection (Wu et al., 2020; Zhang et al., a,b; Vulic et al., 2021; Zhang et al., b), which typically fine-tune PLMs on conversation data.
A PLM-based fine-tuning method (Zhang et al., a), called IntentBERT, utilizes a small amount of labeled utterances from public intent datasets to fine-tune PLMs with a standard classification task, which is referred to as supervised pre-training. Despite its simplicity, supervised pre-training has been shown extremely useful for few-shot intent detection even when the target data and the data used for fine-tuning are very different in semantics. However, as will be shown in Section 3.2, IntentBERT suffers from severe anisotropy, an undesirable property of PLMs (Gao et al., a; Ethayarajh, 2019; Li et al., 2020).
Anisotropy is a geometric property that semantic vectors fall into a narrow cone. It has been identified as a crucial factor for the sub-optimal performance of PLMs on a variety of downstream tasks (Gao et al., a; Arora et al., b; Cai et al., 2020; Ethayarajh, 2019), which is also known as the representation degeneration problem (Gao et al., a). Fortunately, isotropization techniques can be applied to adjust the embedding space and yield significant performance improvement in many tasks (Su et al., 2021; Rajaee and Pilehvar, 2021a).
Hence, this paper aims to answer the question:
- Can we improve supervised pre-training via isotropization for few-shot intent detection?
Many isotropization techniques have been developed based on transformation (Su et al., 2021; Huang et al., 2021), contrastive learning (Gao et al., b), and top principal components elimination (Mu and Viswanath, 2018). However, these methods are designed for off-the-shelf PLMs. When applied on PLMs that have been fine-tuned on some NLP task such as semantic textual similarity or intent classification, they may introduce an adverse effect,
arXiv:2205.07208v3 [cs.CL] 15 Sep 2024
Figure 1: Illustration of our proposed regularized supervised pre-training. SPT denotes supervised pre-training (fine-tuning an off-the-shelf PLM on a set of labeled utterances), which makes the feature space more anisotropic. CL-Reg and Cor-Reg are designed to regularize SPT and increase the isotropy of the feature space, which leads to better performance on few-shot intent detection.
as observed in Rajae and Pilehvar (2021c) and our pilot experiments.
In this work, we propose to regularize supervised pre-training with isotropic regularizers. As shown in Fig. 1, we devise two regularizers, a contrastive-learning-based regularizer (CL-Reg) and a correlation-matrix-based regularizer (CorReg), each of which can increase the isotropy of the feature space during supervised training. Our empirical study shows that the regularizers can significantly improve the performance of standard supervised training, and better performance can often be achieved when they are combined.
- We present the first study on the isotropy property of PLMs for few-shot intent detection, shedding light on the interaction of supervised pre-training and isotropization.
- We improve supervised pre-training by devising two simple yet effective regularizers to increase the isotropy of the feature space.
- We conduct a comprehensive evaluation and
analysis to validate the effectiveness of the proposed approach.
2.1 Few-shot Intent Detection
With a surge of interest in few-shot learning (Finn et al., 2017; Vinyals et al., 2016; Snell et al., 2017), few-shot intent detection has started to receive attention. Earlier works mainly focus on model design, using capsule network (Geng et al., 2019), variational autoencoder (Xia et al., a), or metric functions (Yu et al., 2018; Nguyen et al., 2020). Recently, PLMs-based methods have shown promising performance in a variety of NLP tasks and become the model of choice for few-shot intent detection. Zhang et al. (c) cast few-shot intent detection into a natural language inference (NLI) problem and fine-tune PLMs on NLI datasets. Zhang et al. (b) propose to fine-tune PLMs on unlabeled utterances by contrastive learning. Zhang et al. (a) leverage a small set of public annotated intent detection benchmarks to fine-tune PLMs with standard supervised training and observe promising perfor
mance on cross-domain few-shot intent detection. Meanwhile, the study of few-shot intent detection has been extended to other settings including semi-supervised learning (Dopierre et al., b,a), generalized setting (Nguyen et al., 2020), multi-label classification (Hou et al., 2021), and incremental learning (Xia et al., b). In this work, we consider standard few-shot intent detection, following the setup of Zhang et al. (a) and aiming to improve supervised pre-training with isotropization.
2.2 Further Pre-training PLMs with Dialogue Corpora
Recent works have shown that further pre-training off-the-shelf PLMs using dialogue corpora (Henderson et al., b; Peng et al., 2020, 2021) are beneficial for task-oriented downstream tasks such as intent detection. Specifically, TOD-BERT (Wu et al., 2020) conducts self-supervised learning on diverse task-oriented dialogue corpora. ConvBERT (Mehri et al., 2020) is pre-trained on a 700 million open-domain dialogue corpus. Vulic et al. (2021) propose a two-stage procedure: adaptive conversational fine-tuning followed by task-tailored conversational fine-tuning. In this work, we follow Zhang et al. (a) to further pre-train PLMs using a small amount of labeled utterances from public intent detection benchmarks.
2.3 Anisotropy of PLMs
Isotropy is a key geometric property of the semantic space of PLMs. Recent studies identify the anisotropy problem of PLMs (Cai et al., 2020; Ethayarajh, 2019; Mu and Viswanath, 2018; Rajaee and Pilehvar, 2021c), which is also known as the representation degeneration problem (Gao et al., a): word embeddings occupy a narrow cone, which suppresses the expressiveness of PLMs. To resolve the problem, various methods have been proposed, including spectrum control (Wang et al., 2019), flow-based mapping (Li et al., 2020), whitening transformation (Su et al., 2021; Huang et al., 2021), contrastive learning (Gao et al., b), and cluster-based methods (Rajaee and Pilehvar, 2021a). Despite their effectiveness, these methods are designed for off-the-shelf PLMs. The interaction between isotropization and fine-tuning PLMs remains under-explored. A most recent work by Rajaee and Pilehvar shows that there might be a conflict between the two operations for the semantic textual similarity (STS) task. On the other hand, Zhou et al. (2021) propose to fine-tune PLMs with
Dataset	BERT	IntentBERT
BANKING	.96	.71(.04)
HINT3	.95	.72(.03)
HWU64	.96	.72(.04)
Table 1: The impact of fine-tuning on isotropy. Fine-tuning renders the semantic space notably more anisotropic. The mean and standard deviation of 5 runs with different random seeds are reported.
isotropic batch normalization on some supervised tasks, but it requires a large amount of training data. In this work, we study the interaction between isotropization and supervised pre-training (fine-tuning) PLMs on intent detection tasks.
Before introducing our approach, we present pilot experiments to gain some insights into the interaction between isotropization and fine-tuning PLMs.
3.1 Measuring isotropy
Following Mu and Viswanath (2018); Bió et al. (2021), we adopt the following measurement of isotropy:
$$
\mathbf {I} (\mathbf {V}) = \frac {\operatorname* {m i n} _ {\mathbf {c} \in C} \mathbf {Z} (\mathbf {c} , \mathbf {V})}{\operatorname* {m a x} _ {\mathbf {c} \in C} \mathbf {Z} (\mathbf {c} , \mathbf {V})}, \tag {1}
$$
where  $\mathbf{V} \in \mathbb{R}^{N \times d}$  is the matrix of stacked embeddings of  $N$  utterances (note that the embeddings have zero mean),  $C$  is the set of unit eigenvectors of  $\mathbf{V}^{\top} \mathbf{V}$ , and  $\mathbf{Z}(\mathbf{c}, \mathbf{V})$  is the partition function (Arora et al., b) defined as:
$$
\mathrm {Z} (\mathbf {c}, \mathbf {V}) = \sum_ {i = 1} ^ {N} \exp \left(\mathbf {c} ^ {\top} \mathbf {v} _ {i}\right), \tag {2}
$$
where  $\mathbf{v}_i$  is the  $i_{\mathrm{th}}$  row of  $\mathbf{V}$ .  $\operatorname{I}(\mathbf{V}) \in [0,1]$ , and 1 indicates perfect isotropy.
3.2 Fine-tuning Leads to Anisotropy
To observe the impact of fine-tuning on isotropy, we follow IntentBERT (Zhang et al., a) to fine-tune BERT (Devlin et al., 2019) with standard supervised training on a small set of an intent detection benchmark OOS (Larson et al., 2019) (details are given in Section 4.1). We then compare the isotropy of the original embedding space (BERT) and the embedding space after fine-tuning (IntentBERT) on target datasets. As shown in Table 1, after fine-tuning, the isotropy of the embedding space is notably decreased on all datasets. Hence, it can be
seen that fine-tuning may render the feature space more anisotropic.
Figure 2: The impact of contrastive learning on IntentBERT with experiments on HWU64 and BANKING77 datasets. The performance (blue) drops while the isotropy (orange) increases.
3.3 Isotropization after Fine-tuning May Have an Adverse Effect
To examine the effect of isotropization on a fine-tuned model, we apply two strong isotropization techniques to IntentBERT: dropout-based contrastive learning (Gao et al., b) and whitening transformation (Su et al., 2021). The former fine-tunes PLMs in a contrastive learning manner<sup>1</sup>, while the latter transforms the semantic feature space into an isotropic space via matrix transformation. These methods have been demonstrated highly effective (Gao et al., b; Su et al., 2021) when applied to off-the-shelf PLMs, but things are different when they are applied to fine-tuned models. As shown in Fig. 2, contrastive learning improves isotropy, but it significantly lowers the performance on two benchmarks. As for whitening transformation, it has inconsistent effects on the two datasets, as shown in Fig. 3. It hurts the performance on HWU64 (Fig. 3a) but yields better results on BANKING77 (Fig. 3b), while producing nearly perfect isotropy on both. The above observations indicate that isotropization may hurt fine-tuned models, which echoes the recent finding of Rajaee and Pilehvar.
The pilot experiments reveal the anisotropy of a PLM fine-tuned on intent detection tasks and the
(a) HWU64.
(b) BANKING77.
Figure 3: The impact of whitening on IntentBERT with experiments on HWU64 and BANKING77 datasets. Whitening transformation leads to perfect isotropy but has inconsistent effects on the performance.
challenge of applying isotropization techniques on the fine-tuned model. In this section, we propose a joint fine-tuning and isotropization framework. Specifically, we propose two regularizers to make the feature space more isotropic during fine-tuning. Before presenting our method, we first introduce supervised pre-training.
4.1 Supervised Pre-training for Few-shot Intent Detection
Few-shot intent detection targets to train a good intent classifier with only a few labeled data  $\mathcal{D}_{\mathrm{target}} = \{(x_i,y_i)\}_{N_t}$ , where  $N_{t}$  is the number of labeled samples in the target dataset,  $x_{i}$  denotes the  $i_{\mathrm{th}}$  utterance, and  $y_{i}$  is the label.
To tackle the problem, Zhang et al. (a) propose to learn intent detection skills (fine-tune a PLM) on a small subset of public intent detection benchmarks by supervised pre-training. Denote by  $\mathcal{D}_{\mathrm{source}} = \{(x_i,y_i)\}_{N_s}$  the source data used for pre-training, where  $N_{s}$  is the number of examples. The fine-tuned PLM can be directly used on the target dataset. It has been shown that this method
We refer the reader to the original paper for details.
(a) CL-Reg.
(b) Cor-Reg.
Figure 4: Illustration of CL-Reg (contrastive-learning-based regularizer) and Cor-Reg (correlation-matrix-based regularizer).  $x_{i}$  is the  $i_{\mathrm{th}}$  utterance in a batch of size 3. In (a),  $x_{i}$  is fed to the PLM twice with built-in dropout to produce two different representations of  $x_{i}$ :  $\mathbf{h}_i$  and  $\mathbf{h}_i^+$ . Positive and negative pairs are then constructed for each  $x_{i}$ . For example,  $\mathbf{h}_1$  and  $\mathbf{h}_1^+$  form a positive pair for  $x_{1}$ , while  $\mathbf{h}_1$  and  $\mathbf{h}_2^+$ , and  $\mathbf{h}_1$  and  $\mathbf{h}_3^+$ , form negative pairs for  $x_{1}$ . In (b), the correlation matrix is estimated from  $\mathbf{h}_i$ , feature vectors generated by the PLM, and is regularized towards the identity matrix.
can work well when the label spaces of  $\mathcal{D}_{\mathrm{source}}$  and  $\mathcal{D}_{\mathrm{target}}$  are disjoint.
Specifically, the pre-training is conducted by attaching a linear layer (as the classifier) on top of the utterance representation generated by the PLM:
$$
\mathrm {p} (y | \mathbf {h} _ {i}) = \operatorname {s o f t m a x} \left(\mathbf {W h} _ {i} + \mathbf {b}\right) \in \mathbb {R} ^ {L}, \tag {3}
$$
where  $\mathbf{h}_i\in \mathbb{R}^d$  is the representation of the  $i_{\mathrm{th}}$  utterance in  $\mathcal{D}_{\mathrm{source}}$ ,  $\mathbf{W}\in \mathbb{R}^{L\times d}$  and  $\mathbf{b}\in \mathbb{R}^{L}$  are the parameters of the linear layer, and  $L$  is the number of classes. The model parameters  $\theta = \{\phi ,\mathbf{W},\mathbf{b}\}$ , with  $\phi$  being the parameters of the PLM, are trained on  $\mathcal{D}_{\mathrm{source}}$  with a cross-entropy loss:
$$
\theta = \underset {\theta} {\arg \min } \mathcal {L} _ {\mathrm {c e}} \left(\mathcal {D} _ {\text {s o u r c e}}; \theta\right). \tag {4}
$$
After supervised pre-training, the linear layer is removed, and the PLM can be immediately used as a feature extractor for few-shot intent classification on target data. As shown in Zhang et al. (a), a parametric classifier such as logistic regression can be trained with only a few labeled samples to achieve good performance.
However, our analysis in Section 3.2 shows the limitation of supervised pre-training, which yields a anisotropic feature space.
4.2 Regularizing Supervised Pre-training with Isotropization
To mitigate the anisotropy of the PLM fine-tuned by supervised pre-training, we propose a joint training objective by adding a regularization term  $\mathcal{L}_{\mathrm{reg}}$  for isotropization:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {c e}} \left(\mathcal {D} _ {\text {s o u r c e}}; \theta\right) + \lambda \mathcal {L} _ {\mathrm {r e g}} \left(\mathcal {D} _ {\text {s o u r c e}}; \theta\right), \tag {5}
$$
where  $\lambda$  is a weight parameter. The aim is to learn intent detection skills while maintaining an appropriate degree of isotropy. We devise two different regularizers introduced as follows.
Contrastive-learning-based Regularizer. Inspired by the recent success of contrastive learning in mitigating anisotropy (Yan et al., 2021; Gao et al., b), we employ the dropout-based contrastive learning loss used in Gao et al. (b) as the regularizer:
$$
\mathcal {L} _ {\text {r e g}} = - \frac {1}{N _ {b}} \sum_ {i} ^ {N _ {b}} \log \frac {e ^ {\operatorname* {s i m} \left(\mathbf {h} _ {i} , \mathbf {h} _ {i} ^ {+}\right) / \tau}}{\sum_ {j = 1} ^ {N _ {b}} e ^ {\operatorname* {s i m} \left(\mathbf {h} _ {i} , \mathbf {h} _ {j} ^ {+}\right) / \tau}}. \tag {6}
$$
In particular,  $\mathbf{h}_i\in \mathbb{R}^d$  and  $\mathbf{h}_i^+ \in \mathbb{R}^d$  are two different representations of utterance  $x_{i}$  generated by the PLM with built-in standard dropout (Srivastava et al., 2014), i.e.,  $x_{i}$  is passed to the PLM twice with different dropout masks to produce  $\mathbf{h}_i$  and  $\mathbf{h}_i^+$ .  $\mathrm{sim}(\mathbf{h}_1,\mathbf{h}_2)$  denotes the cosine similarity between  $\mathbf{h}_1$  and  $\mathbf{h}_2$ .  $\tau$  is the temperature parameter.  $N_{b}$  is the batch size. Since  $\mathbf{h}_i$  and  $\mathbf{h}_i^+$  represent the same utterance, they form a positive pair. Similarly,  $\mathbf{h}_i$  and  $\mathbf{h}_j^+$  form a negative pair, since they represent different utterances. An example is given in Fig. 4a. By minimizing the contrastive loss, positive pairs are pulled together while negative pairs are pushed away, which in theory enforces an isotropic feature space (Gao et al., b). In Gao et al. (b), the contrastive loss is used as the single objective to fine-tune off-the-shelf PLMs in an unsupervised manner, while in this work we use it jointly with supervised pre-training to fine-tune PLMs for few-shot learning.
Correlation-matrix-based Regularizer. The above regularizer enforces isotropization implicitly.
Here, we propose a new regularizer that explicitly enforces isotropization. The perfect isotropy is characterized by zero covariance and uniform variance (Su et al., 2021; Zhou et al., 2021), i.e., a covariance matrix with uniform diagonal elements and zero non-diagonal elements. Isotropization can be achieved by endowing the feature space with such statistical property. However, as will be shown in Section 5.3, it is difficult to determine the appropriate scale of variance. Therefore, we base the regularizer on correlation matrix:
$$
\mathcal {L} _ {\text {r e g}} = \| \boldsymbol {\Sigma} - \mathbf {I} \|, \tag {7}
$$
where  $\|\cdot\|$  denotes Frobenius norm,  $\mathbf{I} \in \mathbb{R}^{d \times d}$  is the identity matrix,  $\boldsymbol{\Sigma} \in \mathbb{R}^{d \times d}$  is the correlation matrix with  $\boldsymbol{\Sigma}_{ij}$  being the Pearson correlation coefficient between the  $i_{\mathrm{th}}$  dimension and the  $j_{\mathrm{th}}$  dimension. As shown in Fig. 4b,  $\boldsymbol{\Sigma}$  is estimated with utterances in the current batch. By pushing the correlation matrix towards the identity matrix during training, we can learn a more isotropic feature space.
Moreover, the proposed two regularizers can be used together as follows:
$$
\begin{array}{l} \mathcal {L} = \mathcal {L} _ {\mathrm {c e}} \left(\mathcal {D} _ {\text {s o u r c e}}; \theta\right) + \lambda_ {1} \mathcal {L} _ {\mathrm {c l}} \left(\mathcal {D} _ {\text {s o u r c e}}; \theta\right) \tag {8} \\ + \lambda_ {2} \mathcal {L} _ {\text {c o r}} \left(\mathcal {D} _ {\text {s o u r c e}}; \theta\right), \\ \end{array}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are the weight parameters, and  $\mathcal{L}_{\mathrm{cl}}$  and  $\mathcal{L}_{\mathrm{cor}}$  denote CL-Reg and Cor-Reg, respectively. Our experiments show that better performance is often observed when they are used together.
To validate the effectiveness of the approach, we conduct extensive experiments.
Datasets. To perform supervised pre-training, we follow Zhang et al. to use the OOS dataset (Larson et al., 2019) which contains diverse semantics of 10 domains. Also following Zhang et al., we exclude the domains "Banking" and "Credit Cards" since they are similar in semantics to one of the test dataset BANKING77. We then use 6 domains for training and 2 for validation, as shown in Table 2. For evaluation, we employ three datasets: BANKING77 (Casanueva et al., 2020) is an intent detection dataset for banking service. HINT3 (Arora et al., a) covers 3 domains, "Mattress Products Retail", "Fitness Supplements Retail", and "Online
Training	Validation
“Utility”, “Auto com-mute”, “Work”, “Home”, “Meta”, “Small talk”	“Travel”, “Kitchen din-ing”
Table 2: Split of domains in OOS.
Dataset	#domain	#intent	#data
OOS	10	150	22500
BANKING77	1	77	13083
HINT3	3	51	2011
HWU64	21	64	10030
Gaming". HWU64 (Liu et al., 2019a) is a large-scale dataset containing 21 domains. Dataset statistics are summarized in Table 3.
Our Method. Our method can be applied to fine-tune any PLM. We conduct experiments on two popular PLMs, BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). For both of them, the embedding of  $[CLS]$  is used as the utterance representation in Eq. 3. We employ logistic regression as the classifier. We select the hyperparameters  $\lambda, \lambda_1, \lambda_2$ , and  $\tau$  by validation. The best hyperparameters are provided in Table 4.
Table 3: Dataset statistics.
Method	Hyperparameter
CL-Reg	λ = 1.7, τ = 0.05
Cor-Reg	λ = 0.04
CL-Reg + Cor-Reg	λ1 = 1.7, λ2 = 0.04, τ = 0.05
(a) BERT-based.
Method	Hyperparameter
CL-Reg	λ = 2.9, τ = 0.05
Cor-Reg	λ = 0.06
CL-Reg + Cor-Reg	λ1 = 2.9, λ2 = 0.13, τ = 0.05
(b) RoBERTa-based.
Table 4: Hyperparameters selected via validation.
Baselines. We compare our method to the following baselines. First, for BERT-based methods, BERT-Freeze freezes BERT; CONVBERT (Mehri et al., 2020), TOD-BERT (Wu et al., 2020), and DNNC-BERT (Zhang et al., c) further pre-train BERT on conversational corpus or natural language inference tasks. USE-ConveRT (Henderson et al., a; Casanueva et al., 2020) is a transformer-based dual-encoder pretrained on conversational corpus. CPFT-BERT
Method	BANKING77	HINT3	HWU64	Val.
2-shot	10-shot	2-shot	10-shot	2-shot	10-shot	2-shot	10-shot
BERT-Freeze	57.10	84.30	51.95	80.27	64.83	87.99	74.20	92.99
CONVBERT¶	68.30	86.60	72.60	87.20	81.75	92.55	90.54	96.82
TOD-BERT¶	77.70	89.40	68.90	83.50	83.24	91.56	88.10	96.39
USE-ConveRT¶	-	85.20	-	-	-	85.90	-	-
DNNC-BERT¶	67.50	89.80	64.10	87.90	73.97	90.71	72.98	95.23
CPFT-BERT	72.09	89.82	74.34	90.37	83.02	93.66	89.33	97.30
IntentBERT¶	82.40	91.80	80.10	90.20	-	-	-	-
IntentBERT-ReImp	80.38(.35)	92.35(.12)	77.09(.89)	89.55(.63)	90.61(.44)	95.21(.15)	93.62(.38)	97.80(.18)
BERT-White	72.95	88.86	65.70	85.70	75.98	91.26	87.33	96.05
IntentBERT-White	82.52(.26)	92.29(.33)	78.50(.59)	90.14(.26)	87.24(.18)	94.42(.08)	94.89(.21)	98.07(.12)
CL-Reg	83.45(.35)	93.66(.22)	79.30(.87)	91.06(.30)	91.46(.15)	95.84(.12)	94.43(.22)	98.43(.02)
Cor-Reg	83.94(.45)	93.98(.26)	80.16(.71)	91.38(.55)	90.75(.35)	95.82(.14)	95.02(.22)	98.47(.07)
CL-Reg + Cor-Reg	85.21(.58)	94.68(.01)	81.20(.45)	92.38(.01)	90.66(.42)	95.84(.19)	95.41(.25)	98.58(.01)