Cross-View Consistency Regularisation for Knowledge Distillation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:21:51

Abstract: Knowledge distillation (KD) is an established paradigm for transferring privileged knowledge from a cumbersome model to a more lightweight and efficient one. In recent years, logit-based KD methods are quickly catching up in performance with their feature-based counterparts. However, existing research has pointed out that logit-based methods are still fundamentally limited by two major issues in their training process, namely overconfident teacher and confirmation bias. Inspired by the success of cross-view learning in fields such as semi-supervised learning, in this work we introduce within-view and cross-view regularisations to standard logit-based distillation frameworks to combat the above cruxes. We also perform confidence-based soft label selection to improve the quality of distilling signals from the teacher, which further mitigates the confirmation bias problem. Despite its apparent simplicity, the proposed Consistency-Regularisation-based Logit Distillation (CRLD) significantly boosts student learning, setting new state-of-the-art results on the standard CIFAR-100, Tiny-ImageNet, and ImageNet datasets across a diversity of teacher and student architectures, whilst introducing no extra network parameters. Orthogonal to on-going logit-based distillation research, our method enjoys excellent generalisation properties and, without bells and whistles, boosts the performance of various existing approaches by considerable margins. Our code and models will be released.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) have achieved tremendous success across a plethora of computer vision, natural language processing, and multimedia tasks [17, 21, 44] . Behind their widespread applications, high-performance DNNs are often associated with larger if [10, 24, 28, 31, 36, 60, 69] . (b) Methods optimising and mimicking contrastive relations [54] . (c) Our CRLD which involves within-view and crossview logit transfer.not prohibitive model sizes and computational overheads, which render them hard to implement on resource-constrained devices and platforms. Towards computation-efficient, storage-friendly, and real-time deployment of DNNs, a viable solution is knowledge distillation (KD), which was first proposed by Hinton et al. [24] for model compression. KD works by transferring the advanced capability of a larger, cumbersome teacher model to a more lightweight and efficient student model. Since its proposal, KD has witnessed significant advancements in the past decade as a range of featurebased [7, 8, 20, 23, 34, 38, 41, 49, 50, 59, 61, 64] and response-based (logit-based) [10, 24, 28, 36, 54, 60, 69] KD algorithms are proposed for diverse tasks and applications [24, 25, 35, 52, 58, 71] . Stateof-the-art KD methods have largely reduced the teacher-student performance gap. For instance, top-performing methods [10, 28, 69] are capable of training students that are on par with or even surpass their corresponding teacher models on smaller datasets such as CIFAR-100 (see Table 1 ), and are not far behind on larger datasets [16] (Tables 3 and 4 ).In this paper, our goal is to further advance the capability of knowledge distillation by addressing two long-standing problems in existing KD methods. Previous research has reported that stronger teacher models and more accurate teacher predictions do not necessarily lead to better distilled students [11, 24, 36, 62] . This counterintuitive observation points to a prominent and fundamental problem in knowledge distillation -overconfident teacher. In the pioneering work of KD [24] , Hinton et al. argued that valuable information is hidden in teacher's predictions of the non-target classes. These predictions, known as the "dark information", are however largely suppressed when the teacher make predictions with an overly-high confidence. Hence, regularisation of teacher predictions is essential to distilling knowledge with greater generalisation capabilities to the student [18, 24, 37] .In their work [24] , Hinton et al. propose to mitigate the overconfidence problem by softening the predicted probabilities after Softmax using the temperature hyperparameter. This practice is inherited by many later works [10, 28, 31, 47, 60, 69] . Some methods [36] produce smoothed teacher predictions by introducing auxiliary teacher networks with smaller capacity. More straightforward techniques have also been investigated, including label smoothing [37] and early stopping [37] . These works also highlighted overfitting as another detrimental phenomenon closely related to overconfident teacher. These efforts motivate us to look at consistency regularisation via view transformation -another viable solution to combat overconfidence and overfitting. Although widely explored in the semi-supervised learning (SemiSL) literature [45, 53] , consistency regularisation and view transformation have received little attention in knowledge distillation research. According to [57] , strong augmentation amplifies the dark information that is insignificant in the weak view. As such, in this paper we reframe these techniques for KD by designing a novel set of within-view and cross-view consistency regularisation objectives and achieve state-of-the-art KD performance.On the other hand, teacher's predictions are not always correct. Confirmation bias [2] arises when erroneous pseudo-labels predicted by the teacher is used to teach the student. In existing logit-based methods [10, 24, 31, 60, 69] , the student is designated to faithfully learn whatever supervision the teacher has to provide. Such blind mimicking neglects a key fact that the teacher's predictions may be erroneous and misleading, thereby exacerbating the confirmation bias phenomenon. It has been pointed out in recent research [57] that strong perturbation helps mitigate such confirmation bias, which also supports our introduction of a strongly augmented view of the input image. To further mitigate confirmation bias, we draw inspiration from state-of-the-art SemiSL frameworks whose success is partially attributed to their confidence-aware pseudo-labelling [45, 53, 57] . As such, we propose to selectively pick the more reliable predictions made by the teacher for the student to learn, which is proven beneficial in our experiments.The considerations and designs described above altogether lead to a novel Consistency-Regularisation-based Logit Distillation framework, dubbed "CRLD". By drawing inspiration and reaping the fruits from orthogonal research on semi-supervised learning (SemiSL), CRLD presents a simple yet highly effective and versatile solution to knowledge distillation. It easily boosts advanced KD methods, including DKD [69] , MLLD [28] , and NormKD [10] , without introducing extra network parameters. When applied to existing top-performing methods [10, 28] , CRLD establishes new state-ofthe-art results across different datasets. A schematic comparison of CRLD against prior logit-based approaches from a cross-view learning perspective is presented in Figure 1 In summary, the contributions of this paper include:(1) We introduce extensive within-view and cross-view consistency regularisation to combat the overconfident teacher and over-fitting problems common in KD.(2) We design a reliable pseudo-label mining module to avoid the negative impact of unreliable and erroneous supervisory signals from the teacher, thereby mitigating the confirmation bias in KD. (3) We present the simple, versatile, and highly effective CRLD framework. CRLD achieves new state-of-the-art results on multiple benchmarks across diverse network architectures and readily boosts existing logit-based methods by considerable margins. 
related work 2.1 knowledge distillation: 
 Knowledge distribution (KD) is first proposed in [24] as a model compression technique. It transfers advanced knowledge from a larger, cumbersome "teacher" model to a smaller, lightweight "student" model. Following its initial success in image classification [24] and object detection [52, 58, 70] , KD has quickly swept over many more challenging tasks in 3D scene understanding [19, 25, 66, 71] .Existing KD methods are primarily divided into feature-based and logit-based distillation according to where in the network knowledge transfer takes place -the feature space or the logit space.Feature-based Distillation. As its name suggests, feature-based distillation transfers knowledge in the intermediate feature space of the teacher and student models. A most straightforward way is to simply let the student mimic the features of the teacher, as is done in many early works [1, 23, 41, 49] . Some methods also mine and transfer higher-order information from the teacher's feature maps to the student, including inter-channel [34] , interpixel [50] , inter-layer [61] , inter-class [27] , intra-class [27] , and intersample [38] [39] [40] correlations, inter-sample distance [38] , as well as the teacher network's attention [20, 64] . Generative modelling has also been leveraged for feature-based distillation [59] , where randomly masked student features are required to re-generate full teacher features. In addition, cross-stage distillation paths [7, 8] and one-to-all pixel paths [32] have also been proposed for improved feature-based distillation.Logit-based Distillation. Logits are the prediction output by a neural network before its final Softmax layer. Distillation methods that perform knowledge transfer in the prediction space are referred to as logit-based or response-based distillation. Pioneering methods KD [24] and DML [68] directly transfer the teacher's predictions to the student by minimising the Kullback-Leibler (KL) divergence between their predictions. Similar to advances in feature-based distillation, intra-sample and inter-sample relations are also exploited for transfer within the logit space in several works [28, 54] . Instead of treating all logits indifferently, DKD [69] and NKD [60] decompose all logits into target-class and non-target class logits and treat them separately and demonstrate higher knowledge transfer performance. More recent methods such as CTKD [31] and NormKD [10] adopt dynamic Softmax temperature, as opposed to a de facto fixed temperature value in previous works [24, 36, 68, 69] , reporting state-of-the-art performance. Another branch of methods [36, 46] introduce assistant networks between the teacher and the student to aid the former's imparting of logit-space knowledge to the latter. Consistency regularisation is at the core of the recent success of state-of-the-art semi-supervised learning algorithms [3, 4, 30, 42, 45, 48, 53] . It involves enforcing invariant representations across different views of the same unlabelled input image to improve the generalisation of learnt representations on unseen data and distribution. The different views of an input image are generated by semantic-preserving transformations, from simple operations such as random crop, horizontal flip, and MixUp [65] as weak transformations, to more sophisticated [13, 22] or even adaptive [3, 12] augmentation strategies for producing strongly augmented views. Given these artificially generated views, representation consistency can be enforced across two stochastical weak views as is done in [4, 53] , or a pair of strong and weak views as in [3, 45] . To our best knowledge, beyond SemiSL, the idea of consistency regularisation has not been explored within the context of knowledge distillation.Data augmentation has served as a main pillar to the triumph of deep learning. It transforms training samples into augmented versions whilst preserving their semantic connotation. This results in an abundant if not unlimited amount of extra training data to improve the generalisation of deep neural networks. In the context of knowledge distillation, data augmentation is yet to receive considerable attention, and only a handful of preliminary studies are conducted [5, 14, 15, 51] . Specifically, Das et al. [15] studies the effect of data augmentation in training the teacher model. SSKD [54] and HSAKD [55] incorporate elements of contrastive learning. They apply simple rotation to establish a self-supervised pretext task for improved student learning. Wang et al. [51] and IDA [14] design data augmentation strategies tailored to to boost KD. In contrast, our focus is not the design of a data augmentation strategy itself but to leverage the idea of consistency regularisation to improve student's learning. Data augmentation is simply our tool to allow the formulation of consistency regularisation objectives, as is done in advanced SemiSL methods, and we simply use RandAugment as our strong view transformation following them. 
methodology 3.1 knowledge distillation: 
 Knowledge distillation (KD) involves the student model learning from both ground-truth labels and distillation signals from a pretrained teacher. For the image classification task, ground-truth supervisions are widely enforced via a cross-entropy minimisation objective L ; the distillation objective L is enforced by minimising the distance between either the intermediate features or the final predictions of the teacher and the student. Thus, KD in its simplest form has L = L + L as its objective, where is a balancing scalar. In this paper, we investigate logit-based distillation, where L minimises the discrepancy between predicted probabilities by the teacher and the student, and is commonly implemented as the Kullback-Leibler (KL) divergence loss.Consistency regularisation has been widely employed in SemiSL research [3, 4, 42, 45, 48] . It involves creating different views of the same unlabelled image, which are separately fed into a neural network to obtain a pair of network predictions. Consistency regularisation is enforced between the pair of predictions given the prior knowledge that both views fundamentally represent the same high-level information such as the object category.In CRLD, we employ one weak view and one strong to set the stage for our set of within-view and cross-view consistency criterion. Specifically, we adopt RandAugment [13] with random magnitude alongside random crop, random horizontal flip, and Cutout as our strong data augmentation policy. A full list of RandAugment's transformation operations is provided in Supplementary Material. For the weak augmentation, we simply apply random crop and random horizontal flip, which is the standard data augmentation in previous logit-based KD methods [10, 24, 60, 69] . We denote our weak and strong view transformation functions by (•) and (•), respectively.Concretely, given a batch of training samples x = { : ∈ (1, ..., )}, we separately apply (•) and (•) to each sample to obtain a weakly augmented and a strongly augmented view of , denoted as and , respectively. Next, we feed both views of the input image individually to the teacher and the student, obtaining four network predictions, namely p , p , p , and p , where we drop subscript for brevity.We define within-view consistency regularisation as the consistency criterion between teacher's and student's predictions of the same weak or strong view. The within-view consistency objective is therefore computed as:EQUATIONNext, we design a novel cross-view consistency regularisation. It demands the teacher and student to receive differently augmented views of an image and yet produce logit predictions as similar as possible. Formally, this cross-view objective is given by:EQUATIONThe overall KD objective is a sum of the within-view and crossview consistency losses, that is, L = L + L . Note that although one may tune the weighting of each of the four constituting objective terms, for simplicity we choose to let them share the same weighting in this work. A schematic diagram of the pipeline is provided in Figure 2 .Furthermore, a previous work [5] reported that teacher and student shall receive an identical view of the same input using the same image transformation for maximal knowledge distillation performance. With our specific design, however, we will demonstrate that the teacher and student receiving different views of an input using different view transformations leads to optimal performance. In Section 4.4, we conduct extensive ablation experiments to verify whether and when the proposed cross-view learning really works. As we will show, cross-view consistency regularisation using different views of the same input image is key to the strong performance of the proposed CRLD framework.Confirmation bias harms distillation when the student learns from erroneous soft labels provided by the teacher. By introducing the more challenging strongly augmented views, we are also increasing the likelihood that the well-trained teacher produces misleading predictions that undermine student learning. We experimentally observe that strongly-augmented samples generated by our strong view transformation policy can sometimes be almost unintelligible (refer to Appendix for examples), with false predictions made by the teacher. Therefore, we are motivated to refrain unreliable teacher predictions from forming the consistency regularisation pairs. To this end, we propose a simple thresholding mechanism by considering the highest class probability in teacher's per-instance prediction as an indicator of teacher's uncertainty about this prediction. Teacher predictions whose highest class probability is below a given threshold are discarded.In practice, we apply two thresholds and for teacher's predictions of the weak and strong views, respectively. Different from the common practice in SemiSL [45, 53] , we do not convert the preserved predictions into hard, one-hot pseudo-labels. This is due to the nature of the KD task, whose success hinges on the dark knowledge carried within the non-target class predictions [24, 60, 69] . Instead, we keep teacher's soft predictions as they are as supervision. As an example, objective term KLD(p , p ) with EQUATIONwhere 1(•) is the indicator function. Other objectives are defined like-wise and are omitted for brevity.The overall training objective for CRLD is a weighted combination of previously described loss terms, namely a ground-truth supervision loss L and L .EQUATIONwhere is a balancing weight. L is computed between student's predictions of the weakly-augmented image and the groundtruth label using the cross-entropy loss, which is the standard practice in all previous logits-based distillation methods. L is the sum  
implementation details: 
 We evaluate our method by conducting knowledge distillation across various teacher-student pairs of common DNN architecture families: ResNet [21] , WRN [63] , VGG [44] , MobileNet [26, 43] , and ShuffleNet [67] . In all experiments, we strictly adhere to standardised training configurations of previous knowledge distillation methods [7-10, 20, 23, 24, 31, 38-41, 49, 50, 55, 56, 60, 64] . All reported experimental results are averaged over 3 independent runs. CIFAR-100 & Tiny-ImageNet. We train our method (and also when reproducing others) for a total of 240 epochs, with an initial learning rate of 0.025 for MobileNet [43] and ShuffleNet [67] students and 0.05 for others. The learning rate decays by a factor of 10 after the 150th, 180th, and 210th epochs; the SGD optimiser is used, with a momentum of 0.9, a weight decay of 5 × 10 -4 , and a batch size of 64.ImageNet. We conduct 100-epoch training with a batch size of 512, with an initial learning rate of 0.2 that decays by a factor of 0.1 at the 30th, 60th, and 90th epochs. Other parameters, unless otherwise stated, follow CIFAR-100 and Tiny-ImageNet experiments.Our method is implemented in the mdistiller codebase, using NormKD [10] as the baseline. Our code and models will be made publicly available for reproducibility.Distillation performance. We present extensive experimental results on CIFAR-100, Tiny-ImageNet, and ImageNet datasets using a diversity of teacher-student pairs in Tables 1 to 4 . Specifically, the proposed CRLD outperforms all existing methods on all evaluated datasets across teacher-student pairs of both homogeneous (Tables 1, 3 , and 4) and heterogeneous (Tables 2 and 4 ) architectures. When using MLLD's [28] training configurations (marked with " †"), our method achieves further performance gains and leads MLLD by a considerable margin. Generalisation capabilities In addition to NormKD [10] , we also apply the proposed CRLD to state-of-the-art logit-based knowledge distillation frameworks [24, 28, 60, 69] and report the results in Table 5 . Note that for a fair comparison, we report our reproduced results for compared methods, using official implementations and specifications. The experimental results cogently validate the generalisation capability of our method. The proposed CRLD works orthogonally to existing knowledge distillation methods and can be easily incorporated to significantly boost knowledge transfer performance without introducing any extra network parameter or any additional inference overhead.Design of consistency regularisation. We break down our full training objective and investigate the play of each individual term in CRLD's overall effectiveness. A set of ablation experiments are conducted with results presented in Table 7 . First, we observe that within-view losses are individually effective and consistency within the strong view alone is more effective compared to weak view alone (Expt.0-1). Intriguingly, cross-view consistencies are harmful when used individually (Expt. 1-2), but are rather beneficial when applied in concert with within-view consistencies (Expt. 4-9). Finally, our ablation experiments (Expt. 6-9) demonstrate that each individual consistency objective in our full objective plays a non-negligible part and their joint play leads to the optimal performance. Note that our experiments also highlight that the effectiveness of CRLD does not stem from a mere increase in the diversity of training samples, as a notable +1.56% accuracy gain is achieved compared to when the exact same set of strong view augmentation policies are applied in a naive manner (i.e., Expt. 1 →9).Self-supervision of student. A common technique to exploit the potential of unsupervised learning is to enforce a self-supervision criterion on the student. Specifically, the student can receive two different views of the same input and learns to minimise the distance between its predictions of these two views. We investigate whether Method Acc. (%) Latency (ms) # Param. FitNets [41] 73.50 21 16.8k RKD [38] 71.90 22 0 OFD [23] 74.95 26 86.9k CRD [49] 75.51 32 12.9M ReviewKD [8] 76.12 36 1.8M KD [24] 73.33 20 0 CTKD [31] 73.79 34 52.2k TAKD [36] 73.81 -270k MLLD [28] 74.64 21 0 NormKD [10] 76.26 20 0 DKD [69] 76.32 20 0 NKD [60] 76. such self-supervision consistency enforce upon the student may bring further improvements to the proposed framework. From the results in Tables 7, we observe that student's self-supervision tends to degrade its own learning. In particular, more severe degradation occurs when less powerful external regularisation i.e., (those imposed by the teacher) are enforced on the student. The same set of ablation experiments on Tiny-ImageNet manifest the same behaviours and are reported in Supplementary Material.Our postulation for such degradation is that the less restricted student, when trained from scratch, is able to find a shortcut in the learnable network parameter space that maximally minimises the distance between its predictions of both views but with poor generalisation. Experimentally, we observe a rapidly converging self-supervision loss quickly dropping to extremely small values, which supports our conjecture. As such, we deem self-supervision regularisation of the student unnecessary and harmful and do not include it in our final optimisation objective.Training curves. For further insights into the training profile of different methods, in alleviated overfitting and improved generalisation brought about by CRLD. In addition, we also notice less oscillatory test accuracy curves of CRLD, which is likely due to improved generalisation and mitigated confirmation bias of our method. t-SNE visualisation. We visualise the feature space learnt by the student using different logit-based distillation methods. As seen in Figure 4 , features learnt using the proposed CRLD are significantly more seperable in the feature space, with more tightly clustered class-wise features and greater inter-class feature variations. These observations imply greater generalisation of the learnt model and substantiate the superiority of the proposed distillation method.Speed & efficiency. In Table 8 , we benchmark the computational efficiency of our method against existing algorithms in terms of average training latency per batch (including data preprocessing) and extra network parameters incurred. Our method has a training latency on par with CTKD [31] and ReviewKD [8] and comparable to some of the fastest methods, and a significant portion of it stems from the on-the-fly data augmentation operations. Besides, our method does not introduce any extra network parameters during training and inference. Note that TAKD [36] involves training additional intermediate assistant networks from scratch and therefore has training costs orders of magnitude higher than others. Consistency regularisation in the feature space. Thus far, we have strictly followed the definition of logit-based distillation by enforcing all consistency regularisation at the network's logit prediction stage. By extension, we are curious about to what extent can our proposed regularisation schemes be extended into the feature space. In theory, closer to the network's input end, features' level of abstraction lowers, and the discrepancy between their representations grows. Forcing feature maps to match can therefore hurt student training within its intermediate stages. Therefore, intuitively we expect degraded student performance as the point of application of consistency regularisation moves towards shallower layers. Shown in Table 9 , our experimental results corroborate our intuition. As consistency regularisation moves upper stream -from the logit layer to the pooled feature layer (pool-feat), and then to the 3rd and 2nd feature layers (feats-3 & feats-2), an overall decrease in performance is observed (detailed descriptions found in Supplementary Material). Interestingly, the optimal performance is reached when consistency regularisation is applied to pool-feat instead of the logit space. Nevertheless, the largest performance gains are yielded when it is applied in the logit space, suggesting that the capacity of consistency regularisation is maximally utilised when working with logits. Motivated by these results, we leave the question of how to unleash the potential of the proposed method within the feature space for future work. Distillation without ground-truths. As consistency regularisation is a widely successful technique in semi-supervised learning, one may naturally ponder whether the proposed CRLD would work with unlabelled training samples. Herein, we formally propose a slightly deviating KD task coined "label-free knowledge distillation (LFKD)", which forbids the use of ground-truth labels during knowledge transfer. LFKD is highly relevant and practical scenario -it is often the case that we have access to only a pre-existing, pre-trained teacher model, but the annotations used to train the teacher are no longer accessible (annotations can be costly not made public especially in industrial contexts). In Table 10 , we reevaluate several methods under the LFKD setting. We find that feature-based methods fail completely under LFKD, which is due to a vague causal linkage between feature mimicking and the downstream classification task. In contrast, logit-based methods deliver comparable performance. CRLD remains as the top-performing method, which signifies its robustness to the absence of annotations. Besides, their performance may be even stronger when using a more knowledgeable teacher with high-accuracy predictions that can largely supersede the role of ground-truth labels. Note that logit-based methods such as DKD and NKD do not support LFKD due to the involvement of ground-truth labels in their objective formulation. 
conclusion: 
 In this paper, we presented a novel logit-based knowledge distillation framework named CRLD. The motivation of CRLD lies in revamping popular ideas found in the semi-supervised learning literature, such as consistency regularisation and pseudo-labelling, to combat the overconfident teacher and confirmation bias problems in knowledge distillation. Our design of within-view and cross-view consistency regularsations, enabled by weak and strong image transformations and coupled with a confidence-based soft label selection scheme, leads to a highly effective and versatile CRLD framework.Extensive experiments demonstrate that CRLD can boost existing logit-based methods by considerable margins and sets new records on different datasets and under different knowledge distillation configurations. 
 
 Appendix:  


Decision: This work introduces within-view and cross-view regularisations and confidence-based soft label selection to standard logit-based distillation frameworks to address issues, resulting in a significant boost in student learning and setting new state-of-the-art results on multiple datasets. Overall, the paper is technically solid and will have high impact on multimodal perception research. Given the resolution of raised concerns and the unanimous positive reviews, the paper is accepted for publication.

Reviews: review: paper_summary: paper_summary: The paper introduces a new knowledge distillation method called Consistency-Regularisation-based Logit Distillation (CRLD). Existing logit-based Knowledge Distillation (KD) methods have rapidly caught up with feature-based methods in terms of performance, but they still suffer from two main issues during the training process: overconfident teachers and confirmation bias. CRLD addresses these issues by introducing intra-view and inter-view consistency regularization, and by improving the quality of signals transferred from the teacher through confidence-based soft label selection, further mitigating the confirmation bias problem. Despite the seemingly simple design of CRLD, it significantly enhances the learning effect of the student model, achieving new state-of-the-art results on CIFAR-100, Tiny-ImageNet, and ImageNet datasets, and performs well across different teacher and student architectures without introducing additional network parameters.
strength_weakness: strength_weakness: 1. The paper proposes the CRLD framework, which innovatively addresses some issues in knowledge distillation through intra-view and inter-view consistency regularization as well as confidence-based soft label selection.
2. CRLD has achieved new state-of-the-art results on multiple datasets, indicating that this method can effectively enhance the performance of student models.
3. The CRLD method does not require the introduction of additional network parameters, which allows it to improve performance while maintaining model size.
limitations: limitations: 1.The article employs view consistency to address a range of issues and attempts to enhance the reliability of the teacher model. However, it could attempt to better elucidate the source of the improved reliability, further clarifying the motivations behind this research.
2.The implementation of CRLD may be more complex than traditional knowledge distillation methods, as it involves multi-view data processing and regularization. There is a need for further discussion on the relationship between the performance gains and the increase in computational complexity.
3.The language quality of the article could be improved; for example, the correct spelling in the title should be "Regularization" instead of "Regularisation."
4. The models the author is comparing are rather outdated, it's necessary to include the latest models for comparison.
5. The model baseline used by the author appears to have certain issues, which may be unfavorable for comparison. Further examination is advised.
paper_summary: paper_summary: This paper proposed a knowledge distillation method with cross-view consistency regularisation. Based on a weak view and a strong view augmentation, the proposed method aims to solve the overconfident teacher and confirmation bias. Experiments on 3 datasets show that the proposed CRLD method slightly outperforms the previous methods.
strength_weakness: strength_weakness: 1. The proposed method is simple and easy to implement.
2. Extensive results have shown the effectiveness of the proposed method.
limitations: limitations: 1. Overconfidence is an important issue claimed in this paper. Why does data augmentation (within-view and cross-view) help to reduce this problem? If only the weak or strong augmentation is used, how about the performance? Tables 6 and 6 do not illustrate this point well. 
2. In Section 3.3, two thresholds tau_t and tau_s are used for soft label mining. How to determine these two parameters?
3. In Equation 4, how to set the parameter lambda? Is the proposed method sensitive to this parameter?
paper_summary: paper_summary: This paper proposes CRLD, a method for knowledge distillation that employs multiple views of a single sample. Specifically, each input sample is transformed into a weak view and a strong view using different data augmentations, and both views are used for predictions by the teacher and student models. Cross-view and within-view distillation are then performed. Experiments on CIFAR-100 and Tiny-ImageNet demonstrate the effectiveness of the method.
strength_weakness: strength_weakness: 1. The motivation is clear: to solve the problem of overconfident teachers and confirmation bias.
2. The paper is well-organized and easy to follow.
limitations: limitations: 1. The proposed method has been partially explored in existing works. For example, paper [1] uses different samples from the same class for distillation, which is somewhat similar to using different views of the same sample.
2. Paper [2] points out that identical augmentations for inputs of both the teacher and the student outperform using different augmentations, but the authors' conclusion is contradictory. Additionally, paper [3] notes that performance improvements on small-scale datasets do not necessarily transfer to large-scale ones. Since the experiments in this paper are conducted on CIFAR-100 and Tiny-ImageNet, which are relatively small compared to the ImageNet dataset used in paper [2], results on larger-scale datasets should be provided to further prove the effectiveness of the proposed method. A method that only works on toy datasets cannot support real-world scenarios.

[1]. Regularizing Class-wise Predictions via Self-knowledge Distillation, CVPR 2020.

[2]. Knowledge Distillation: A Good Teacher is Patient and Consistent, CVPR 2022.

[3]. Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale, NeurIPS 2023.
paper_summary: paper_summary: This work proposes CRLD, a framework to further advance the capability of knowledge distillation by addressing two long-standing problems in existing KD methods: overconfident teacher and confirmation bias. It introduces extensive within-view and cross-view consistency regularisation to combat the overconfident teacher and over-fitting problems common in KD and a reliable pseudo-label mining module to avoid the negative impact of unreliable and erroneous supervisory signals from the teacher. The proposed framework archives good performance in CIFAR100, TinyImageNet, and ImageNet with numerous teacher-student architectures.
strength_weakness: strength_weakness: 1. The writing is clear and sophisticated.
2. The experiments are well-designed, with extensive visualizations, ablations, and settings demonstrated.
limitations: limitations: 1. In Section 3.3, this work tries to solve confirmation bias by setting a threshold for confidence-based soft label mining. The proposed metric is termed as $L_{KD}$ in Equation 3, which appears again in Equation 4. However, $L_{KD}$ in Equation 4 is the combination of KLD in Section 3.2, which seems different from it. Would the authors further clarify this?

2. Again in Section 3.3, it would be better if the authors could further clarify how Equation 3 works to be part of the process of knowledge distillation. This part seems a bit confusing.