Hilbert Distillation for Cross-Dimensionality Networks
Authors: Dian Qin
Haishuai Wang
Zhe Liu
HONGJIA XU
Sheng Zhou
Jiajun Bu
Venue: NeurIPS 2022 Accept (2022)
Keywords: knowledge distillation
cross-dimensionality networks
video recognition
medical imaging

Abstract: 3D convolutional neural networks have revealed superior performance in processing volumetric data such as video and medical imaging. However, the competitive performance by leveraging 3D networks results in huge computational costs, which are far beyond that of 2D networks. In this paper, we propose a novel Hilbert curve-based cross-dimensionality distillation approach that facilitates the knowledge of 3D networks to improve the performance of 2D networks. The proposed Hilbert Distillation (HD) method preserves the structural information via the Hilbert curve, which maps high-dimensional (>=2) representations to one-dimensional continuous space-filling curves. Since the distilled 2D networks are supervised by the curves converted from dimensionally heterogeneous 3D features, the 2D networks are given an informative view in terms of learning structural information embedded in well-trained high-dimensional representations. We further propose a Variable-length Hilbert Distillation (VHD) method to dynamically shorten the walking stride of the Hilbert curve in activation feature areas and lengthen the stride in context feature areas, forcing the 2D networks to pay more attention to learning from activation features. The proposed algorithm outperforms the current state-of-the-art distillation techniques adapted to cross-dimensionality distillation on two classification tasks. Moreover, the distilled 2D networks by the proposed method achieve competitive performance with the original 3D networks, indicating the lightweight distilled 2D networks could potentially be the substitution of cumbersome 3D networks in the real-world scenario.

Hilbert Distillation for Cross-Dimeesionality Networks
Dian Qin Haishuai Wang! Zhe Liu' Hongjia Xu' Sheng Zhou'd 1.2 Jiajun Bult
"Zhejiang Provincial Key Laboratory of Service Robot, College of Computer Science, Zhejiang University, Hangzhou, China
Schooo of Software Technology, Zhejiang University, Ningbo, China
{qindian, haishual .aaii.aagg wang, zheliu, xu_hj, zhousheng_zju, bjj]ezju.edd edu. en
Abstract
3D convolutional neural networks have revealed superior performance in processing volumetric data such as video and medical imaging. However, the competitive performance by leveraging 3D networks results in huge computational costs, which are far beyond that of 2D networks. In this paper, we propose novel Hilbert curve- based pross-dimensionality distillation approach that facilitates the knowledge of 3D networks to improve the performance of 2D networks. The proposed Hilbert Distillation (HD) method preserves the structural information via the Hilbert curve, which maps high-dimensional (>=2) representations to one-dimensional continuous space- filling curves. Since the distilled 2D networks are supervised by the curves converted from dimensionally heterogeneous 3D features, the 2D networks are given informative view in terms of learning structural information embedded in cellntrained digh-iiiensional representations. We further propose a Variable- length Hilbert Distillation (VHD) method dynamically shorten the walking stride of the Hilbert curve in activation feature areas and lengthen the stride context feature areas, forcing the 2D networks to pay more attention to learning from activation features. The proposed algorithm outperforms the current state- of-the-art distillation techniques adapted to cross-dimensionality distillation two classification tasks. Moreover, the distilled 2D networks by the proposed method achieve competitive performance with the original 3D networks, indicating the lightweight distilled 2D networks could potentially be the substitution of cumbersome 3D networks in the real-world scenario.
Introduction
Knowledge distillation aims to transfer knowledge from cumbersome models to lightweight models. The vanilla knowledge distillation [11] collects the logits that contain the cognizance of wrong classes in the cumbersome model. Forcing the lightweight model to mimic the logits can obviously strengthen its parsing ability. Recent efforts further devote to improve the distillation effectiveness by
adopting intermediate representations (9 18 32 35 36) and the relation knowledge 232 33 of samples as new distillation mediums. In some real-world scenarios (e.g., video analysis and medical imaging processing), 3D models typically present overwhelming performance compared with common models, yet the 3D models suffer from huge computational costs. One way to mitigate this problem is by leveraging knowledge distillation to encapsulate the parsing ability of 3D models into 2D models. However, the cross cross-dimensoonaiity distillation problem, especially 3D-to-2D distillation, is largely unaddressed in the literature.
Most existing methods that utilize intermediate features for distillation are designed by employing a metric function (e.g., mean square error (MSE) and Dallback-Leiblee (KL) divergence) between
"qqual contribution. Corresponniig authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the pair of feature maps extracted from teacher and student networks. Unfortunately, they are not applicable in dinensiiiens distillation tasks due to the inconsistent dimensionality. To get rid of the constraint, several dimensionality reduction methods such as pooling and convolution that can completely condense the extra dimension maybe feasible by converting the 3D feature maps (with the size of D x W x H) to reduced 2DD-from-3D representations (with the size of 1 x W x H). However, the reduction tends to dropout the high-dimensional structural information. As a result, they have limited applicability to deal with volumetric data in the case of lacking the structural information.
In this paper, we propose a new method, namely Hilbert Distillation (HD), to explicitly distill structural information embedded in intermediate features of 3D models. Our approach preserves both 3D and 2D features and maps them into mne-dimensional representations based on the intrinsic rules of Hilbert curve [10] in the original feature space. The distilled models with Hilbert curve retain the relative position information in the high-dimensional space, i.e., the mapped data points that are close to each other in the 1D space are also adjacent in the original high-dimensional space, thereby the structural information is highly preserved via the proposed HD method. Hence, the distilled lightweight 2D models obtain significant improvements by learning the converted one-dimensional representations from the 3D feature maps. Furthermore, we propose Variable- length Hilbert Distillation (VHD) to more efficiently transfer the structural knowledge by dynamically shortening the walking stride when constructing Hilbert curves in activation areas of feature maps (37). To address the optimization problem in the dynamic process, we also present approximation approach by calculating self-adaptive weights when adopting mapping functions of the Hilbert curve. We conduct extensive cross poos--dmmnsiionaliy distillation experiments in activity and medical image classification domains, where 3I) models are typically applied. The proposed method further narrows the gap between student and teacher networks comparing with the current state-of-the -he in the distillation benchmark. More importantly, the distilled 2D models by our method achieve competitive performance with the 3D models, suggesting that the structural information learned by the proposed method helps promote the generalizability of the 2D models. Hence, the distilled lightweight models could potentially be the substitution of cumbersome models for analyzing the volumetric data.
2 Related Work
Knowledge Distillation. The vanilla knowledge distillation [II] focuses on transferring knowledge embedded in the logits, i.e., the model outputs scores before applying Softmax. It also provides the idea that coarse information would be beneficial to the distillation, and proposes the Softmax with temperature should be adopted before the calculation of distillation loss. Succeeding efforts [6 18 26 32 35 36) try to utilize intermediate representations further. For example, the Attention Transter 36 calculates attention maps by accumulating values in feature maps along with the channel dimension. The attention maps set out the most informative features for the student model and tackle the problem of different channel sizes between participated models. Some other works (22 23 25 33
try to extract the relation knowledge among samples to enable the distillation between models. For instance, SP (33) calculates the relationships between intermediate feature maps of different inputs. Conducting the distillation between student and teacher relationship maps results in decent improvements to the student model. Recent works tend to explore deeper knowledge and transfer the knowledge in more complicated ways. Ji et al. (14) propose to use feature maps of all layers through the integration with self-adaptive weights learned by attention mechanism. HKD method (38) designs graph-based scheme to distill holistic knowledge, thus the student model is able to learn individual knowledge and relational knowledge simultaneously.
Dimensionality Reduction. Most of the mentioned methods are not applicable for cross- dimensionality distillation networks due to the inconsistent dimension. A straightforward way to cope with this issue is to adopt dimensionality reduction methods. For examplc, PCA (12) and
LDA 66 are commonly used to obtain low-dimensional representation by projecting data points to few principal components. Some non-linear dimensionality reduction methods such as ISOMAP (31) and LLE 3 are available for more complicated sample distributions. Recent efforts about trans- former 4 27 provide valuable ideas for learning position coding for complex representations such as continuous large-scale feature maps. The reduction can also be performed by following the learned position coding. However, these methods either lose the important structural information for the cross-dimensionaliyy distillation to a certain extent or divide the original high-dimensional space into much smaller space where the problem of dimensional inconsistency still exists. This work refers to

---

Context Aren
(b)
(c)
(d)
Figure 1: Examples of the Hilbert curve (a-c) and the proposed Variable-length Hilbert curve (d).
continuous space-filling dimensionality reduction approach, namely Hilbert curve, address the above issues. Details are described in the next section.
Distillation. Recently, cross ponss-diinnsionality distillation has also been studied in [3] 19] that have considered adopting 3D models as the distillation participants. However, their methods still follow the 2D-to-2D distillation by applying customized tricks in order to transfer 3D representations into the 2D form, resulting the 2D models are improved only marginally after cross-dimensionality
Cross-dimensionaal
performing the severely diluted knowledge. To this end, propose distillation method by explicitly extracting and transferring the structural knowledge embedded in high-dimensional feature maps.
3
Method
In this section, first introduce the Hilbert Distillation for doossdiimensoonality scenarios based on the Hilbert curve. Then we elaborate the proposed Variable-length Hilbert Distillation by dynamically adjusting the stride of Hilbert curve deduction. Since encoding and optimizing the dynamic progress is challenging, we further present an approximation solution by mapping the conventional Hilbert curve into Variable-length Hilbert curve using self-adaptive weights.
3.1 Hilbert Curve
The Hilbert curve [10] is a continuous space-filling curve in the Euclidean space, which provides mapping between ID and high dimension space while preserves space structure fairly well. The principle behind the Hilbert curve that the mapped data points which are close to each other ID space also close to each other in the original high dimension space.
The construction [ of the Hilbert curve can be different. this paper, we adopt Lindenmayer- System [17] (L-System) to describe the construction that produces walking guides recursively. The system consists components: 1) two Variables, i.e., and B; 2) three Constants, i.e., denotes move forward, ( denotes turn left 90" and @ denotes turn right 90%; 3) the Axiom A, i.e., the starting point of recursion; 4) two Production Rules:
Rule 1 A + +B GA
D
D
BO Aθ
Rule 2 > +OA L + B
B
(2)
We denote that order p controls the times of recursion well as the scale of the curve. When p we can generate the walking guides as @ Đ e & e >> @ (note that the Variables and B are only the placeholder for embedding the rules) based on the Production Rule 1 (Eq. [ from the starting point Axiom A. Thus, the simplest Hilbert curve for a 2x2 square space illustrated in Fig. Ta) can be constructed by looking right at the start. If 2, the Production Rules 21 are embedded into BAAB allocated in Production Rule separately. Then we can get the walking guides as O O >> D & O O e D e O D O D O D O D e e @. It important to note that the contiguous Constants @ and O will be cancelled out and removed from the guides.
Hence, the final walking guides will be > B D C b e D e D e b @ D + > a D e & P θ t> @ > @ D. Fig. [b) illustrates the Hilbert curve of order p = 2 for a larger square space based on the guides. Similarly, the Hilbert curve for higher order can be easily constructed via this method. It can also be extended and applied in higher dimensional space as shown in Fig, Tc).

---

3.2 Hilbert Distillation
Since the above mentioned curve is able to well preserve the structural information its original space, we facilitate the curve to distill knowledge from dimensionally heterogeneous features. Specifically. we aim to supervise the 2D student models by efficiently learning the knowledge that they have never seen in feature maps extracted from the 3D teacher models.
Fig. aaa) illustrates the pipeline of the proposed Hilbert Distillation framework. Let (De, We, H.) and (W,, H)) denote the size of the 3D feature map 7ad and 2D feature map 7/2d respectively. We first leverage the mapping function of Hilbert curve Ha.p (refer Algorithm 1 to rearrange and stretch the feature maps. Let parameter n denotes the dimension (we use 2 or 3 in this paper, but theoretically it could be larger), and p denotes the order that controls the curve length as described in We define the following rule to determine the value of p using the ceiling function [.]
S.c3.1
Pn=3 = [lgg, max (D. Wr, H.)1 Pn= = [log, max W., HH,]
(3)
(4)
After determining the values of and p. we adopt the Lindenmayer-System described in See 3.1 to
generate the Hilbert curve with desired scale, and Algorithm hhelps deduce the mapping function Hn=2.p(1. (t. = v that gives the pixel-level surjective mapping rule from spatial feature map 722 to a one-dimensional representation Îzzd- Although our algorithm describes the case of n = 2, it can synergistically be expanded to n 3 to get the 3D mapping function Hz=3,p(t, j. k) = v with the construction of Hilbert curve for 3D space. The output value v is the new position index of the original pixel-level feature the one-dimensional representation with the length 2nxp. Thus, the one-dimensional representations f/3d, 1/2a of the feature maps yad, 72d can be calculated by
ĀBa(v)) = j, where Hn=3,p(i,j.h = (1) (2) where (i.j) =
(5)
(6)
At this point, we can define the Hilbert Distillation between dross-dimensionaliiy features by calculat- ing the loss between the converted one-dimensional representations
R(rssa)
R(iraa)
hnd =
(7)
(1R()a))
I (f2()
where R(-) is the nearest rescaling function that scales the length of iys from 23xP to 22xp before the calculation, and I|-1|2 and ||-1|ı are the L2 and LI normalization. The division between representations and its L2 norm is intended to climinate the distribution difference between features from different modality. As knowledge distillation method, the distillation process is commonly performed along with the training process. The final loss function for the end-to-end training scheme is given by
C = Lov + aLnd
where LoE is the cross entropy loss corresponding to the original task of the student network, and the hyperparameter a is a balance weight between training from distillation and hard labels.
As claimed in the interpretability 15 28 37 and attention (36) research areas, only partial feature
maps are activated and crucial for the final task. This conclusion implies the reason why our method works. In other words, the superiority of the proposed distillation method is that the 2D models can directly learn the activation features that preserve the 3D structural information as much as possible. As illustrated in Fig. a), the activated "dog features" in 3D feature maps (leftmost) still gathered together after the mapping, as shown in the black area of the bar. Theoretically, our method could be extended to conduct distillation between networks of any dimensionality since the Hilbert curve-based mapping approach can be applied in any Euclidean space.
In addition, we also provide several tricks to address the following possible minor obstacles when applying our method in real-world scenarios:
a. What if the activation features in the converted one-dimensional representations of 2D and 3D models are not relatively aligned. Fig. demonstrates an ideal case that both dogs are located at the left bottom so that the relative positions of the activation features are well aligned. In reality, in the medical imaging task that 3D models are commonly applied, the spatial distribution of human organs are always fixed regardless of the data modality. The activation features from 2D and 3D

---

Context Features
Activation Features
Amal
Hilbert Curvee
Curve
(a) Hilbert Distilation
Context Features
Activation Features
Aocal
Variable-lenge Hilber Clave
Vanable length Hilbert Curve 2D-a-1D
3D-w-II
(b) Variable-ength Hilbert Distilation
Figure 2: The pipeline of the proposed Hilbert Distillation and Variable-length Hilbert Distillation between cross-dimensionality feature maps.
Algorithm 1: The Mapping Function of Hilbert Curve Hn=22.p
Initialize = =,
Save the map (i.))
while Algorithu 1 is running do // Algorithm starts with looking right
if > is activated then if look up / down then i i= - else if look right/ / left then
i=i-1
j=j+1 / j j -
11
Save the map Hn=2,p(i. =
12
else
13
Waiting for D>
Output Hn=2,p
feature maps are well aligned in the final stage of our method. For another task using 3D models, such video recognition, simple length-preserving fully connection layer (FC) for any one-dimensional representation before the distillation loss calculation able to address this issue.
b. What if the feature maps do not exactly fulfill square cube space. As the vanilla Hilbert curve is only applicable in the space with a side length equals a power of 2, we always construct Hilbert curve for minimum space that can hold the feature maps according to the side length which can be calculated by Eqs. 3 and e.g., constructing curve for x 8 space according to a x 7 x feature map. The mapping function H6 only considers the areas where contains features.
3.3 Variable-length Hilbert Distillation
The scale of activation features relatively small compared with that context features in most cases, as the ratio of black and gray parts illustrated in Fig. 2 a). The proportion of activation features in 3D feature maps may even lower than in 2D feature maps. To make the distillation more focus on the activation areas where the features are significantly related to semantics, we further propose improvement for the construction process Hilbert curve called Variable-length Hilbert Distillation.
Specifically, we dynamically change the walking stride when constructing the Hilbert curve. As demonstrated Fig. (b), we lengthen the walking stride areas that consist context features to draw sparser curves, and shorten the stride in areas that consist of activation features to draw denser curves. Those skipped context features will represented by the nearest valid features located on the curves. Thus, the proportion of activation features in the calculation of the final distillation loss is
5

---

greatly amplified, as depicted in the figure. To this end, we propose to alter the Production Rules (Eqs. and 2 the described L-system into the formation as
A + + Blo(D)I(T >J e (d))) [=]{ e 1T [ B + @(p(A)ll D-] @ [ [B] @ 1%6 D ([)o
(9)
(10)
where () is the decision function that decides whether to perform recursion, which is given by
A|B, if T'upcoming walking routes that follow the Hilbert curve contain activation feature,
6(A|B)=
Null, otherwise.
(11)
In order to figure out whether there is an activation feature, we define the Activation Mapping (AM) for given convolutional layer 1. We first compute the activation weight for single feature map in the layer I. For 2D and 3D models, the computations are separately defined as
3ad
ΣΣ FEN
dy
22
1 22d
By T (1,5)
(12)
23d X
where in and n denote the numbers of feature maps, #3d and 32d are the numbers of features in the By is the gradient of the output score yf of class e with respect to feature
3D and 2D feature maps, Bh maps 17 Different from the Class Activation Mapping (CAM) proposed by [28), we compute the gradients for all the objective classes and accumulate the results because the knowledge distillation method requires preserving informative knowledge as much as possible, The activation features for wrong classes may also be helpful the distilled model. To this end, we calculate the AM by weighted sum of the feature maps, which is given by
AM3d = Σ
AM2d = Σ
72d m
(13)
The AM generally presents the importance of features corresponding positions, and the size of AM is the same the size of the feature map in the layer l. Moreover, get the activation if the feature's s Sigmoid value AM at the corresponding position is larger than a threshold The threshold is user defined based on the required degree of activation.
Fig. (d) illustrates a case when order p For the first round of rule generation with Eq. 9 the inferred walking guides can be #/> D>] e A(7g A e hho D-]e. Only the two areas marked "A" where contain the activation feature are preserved after the execution of Φ. The additional factor To in the guides controls the multiple compensation for the walking steps, which doubles the execution number of the following >> whenever neighboring & results in Null. Hence, the actual walking guides of the first round are ① P > θ A > A &
P @ In the next round of generation, we can get the final walking guides as @ D D D e D e D @ D @ D e O i D @ by replacing with Eq. recursivve. This idea is the desired Variable-length Hilbert curve depicted in Fig. (d). The Variable-length Hilbert Distillation (VHD) can be hereby designed by following the framework of HD and substituting the curve construction process, as the illustrated pipeline in Fig. 2b)..
However, unlike the vanilla Hilbert curve that offers the hard mapping for space of fixed scale, the Variable-length Hilbert curve is dynamically changed in terms of different activation maps from different inputs, however, it is hardly be performed forward propagation and optimized in backpropagation. In this paper, we propose an approximation approach to achieve the variable-length by calculating self-adaptive weights when adopting the mapping function H. We rewrite the original mapping process in Eqs. 5 and 6 using the product of the features and the values in AM, as follows,
(r)(v) = (s j. k) AMaa(i.j.k) where (i.j,,( =
(14)
(2)(()) (i)(i. AMza(i.j) where Hn (f,j) =
(15)
Finally, the optimizable distillation loss function of the proposed approximation scheme is given as
R(73))
R(Aza)
Lond
(16)
[
(Taa)
Similar with HD, we define the end-to-end training of VHD by replacing Lnd with Cokd in Eq
Note that we only keep the distilled 2D model during the inference phase.

---

645
87
avg
avg
61.0
86
Top- Accuracy (%)
CTTV
Top- 1 Acouracy (%)
635
maK
85
ITEI
630
81
62.5
83
62.0
82
61.5
81
61.0
80.
47
FUNK
SKD
IAVD
AFD
RKD
AT
FONce
SKD
IRVD
ADD
FKD
RKD
HD
viD
(a) ActivityNet
(b) Large-COvIDE 19
Figure 3: The performance of distillation methods using feature maps the ActivityNet (graph a)
and Corge-COVIDD-9 (graph b) datasets. The white point centered in the box is the mean value.
4 Experiments
4.1 Datasets
3D models present overwhelming performance in some specific tasks, such as video recognition and medical imaging processing. Therefore, we demonstrate the effectiveness of the proposed cross- dimensionality distillation method on two datasets: ActivityNet [5] dataset for activity classification and Large-COVVD-19 20) dataset for medical imaging classification.
ActivityNet, which is benchmark video dataset for untrimmed human activity classification. We adopt the AetivityNet--I version that contains 100 human activity classes. We randomly split the 7000 videos into 2:1 for training and testing.
Large-CoVID-19, which is lung Computed Tomography (CT) dataset of COVID-19 for Commu- nity Acquired Pneumonia (CAP) classification. This dataset contains 7593 COVID-19 images, 2618 CAP images, and 6893 normal images. Most images make up the 3D CT sean by combining their neighboring images. We randomly select 20% images as the test set.
4.2 Experiment Setting
Distillation Setup. Different from existing distillation works that consider the same dimensionality in the teacher and student models, we adopt ResNet-50 (8) and VGG16 (30) as the 2D student models and well-trained 3D ResNet-50 the 3D teacher model. In the training process, we feed the randomly selected images into the 2D models, and the volumetric data that consists of 16 consecutive frames into the 3D model. The selected pair of 2D and 3D inputs are restricted in the same category. Since the category of the entire video (activity) is hard to be classified using only one frame, the 2D training scheme video processing commonly adopts the combination of 2D CNN L.STM (2) or 2D CNN with voting strategies. We follow the former scheme to train the 2D student model by sequentially feeding one of the 16 consecutive frames that are prepared for the 3D teacher model into the 2D student model. The distillation method is applied between 3D CNN and 2D CNN before performing the voting. We train and test models on an NVIDIA GeForee RTX 3090 GPU (24GB).
Network Architectures. For the activity classification task, we randomly crop the input frame into 256 256. The batch size is set to 16. The training process lasts 40 epochs. The hyperparameter in Eq. 8 is set to 103. For the medical imaging classification task, we resize the input image into 224 x 224 horizontally. The batch size for training set to 32. The number training epochs is 60. Different from the activity classification task, we set to 10. The selection and performance of the fluctuation of a are also discussed our experiments. The models are trained with initial learning rate 0.01, and the Adam optimization algorithm is employed for both tasks.

---

Table 1: Top-1 accuracy (%) of the student models the ActivityNet and Large-COVID-19 with distillation. The bold values are the best distillation performance.
ActivityNet
Large-CovID-II
Teacher Student
3DResNet-50 ResNet-50
3DResNet-50 VGG16
3DResNet-50 ResNet-50
3DResNet-50 VGG16
ARI(%)
ARI(%)
Teacher
71.34
71.34
90.15
90.15
Student
61.42 = 0.18
60.22 = 0.09
79.92 = 0.02
77.4 = 0.12
KD
11
62.30 = 0.29
61.45 + 0.38
184.59
82.08 = 0.53
110.51
SP
33
62.88 = 0.36
62.27
0.51
71.11
83.69 = 0.32
82.55 = 0.46
47.79
PKT
23
62.73 1733:0.44 0.48
62.70
64.02
83.20 0.26
82.69
61.15
RKD
22
62.14 #.67
61.23 0.46
247.15
83.02
0.45
69.87
CCKD
[25]
62.78 = 0.49
01.88
98.65
83.19 = 0.41
82.70 = 0.58
61.27
HD
63.55
0.28
63.46 0.49 0.49
12.40
85.05
0.58
84.63 + 0.38
9.99
VHD
63.71 - 0.63
64.02 = 0.85
85.55 = 0.72
85.37 = 0.86
0
4.3 Results and Analysis
To demonstrate the performance and stability of the commonly used distillation methods, we separate the experiments into two parts based their applicability in the zross-dimensionality distillation problem. We first present the performance of the distillation methods using feature maps, i.c.
AFD 14). SKD [8]. FitNet (26). IFVD [34], HKD (2). RKD 77. and AT [36]. Since they not to be applied directly due to the dimensionality difference problem, we employ three alignment funetions, i.e., average pooling ("avg" for short in the figure), max pooling ("max" for short in the figure), and convolution with kernel size D x x 1 ("cnv" for short the figure), to align the 3D feature maps into 2D representations by compacting the values along the dimension D. Fig. 3 shows the Top-I accuracy (%) of baselines and the proposed methods for the activity and medical imaging classification tasks. The range values in the boxes collected from the last ten epochs of all folds. The distillation performance is more stable if the box is shorter. The proposed methods HD and VHD outperform all the baselines in terms of all the tasks. Compared with existing methods, the accuracy of the proposed HD method improves about 2% on the Large-COVID-I9 19 data and 1.5% on the ActivityNet using both homogeneous architecture (3DResNet to ResNet) and heterogencous architecture (3DResNet to VGG). The better performance of the proposed VHD method implies that VHD is able to transfer the activation knowledge more efficiently. We can also conclude that the max pooling and convolution alignment functions should be prioritized when the conventional distillation approaches are not able to deal with the dimensionality inconsistency in the cooss-dimensionality distillation problem.
The goal of many distillation methods (i.c., RKD 22 PKT 23 CCKD (25). and SP (33] is to
excavate the relation knowledge among the data samples. Theoretically, they can be directly applied dinensiimenio problems because the calculation between the feature maps of the data can be performed inside the model. The scale of calculated relation representations for distillation is only related to the number of samples. The premise be effective of those methods is that the inputs of the distillation models are consistent, and the student models can learn relational knowledge from teacher models. However, this condition does not hold in the cross- dimensionality scenario. Table provides the performance of commonly adopted methods. We can observe that the proposed HD/VHD methods achieve a remarkable improvement compared with the benchmark methods. The main reason why the baselines do not perform well is that the data prepared for 3D and 2D models are usually different. Note that the vanilla KD [IT] is always feasible if distillation is applied in classification tasks because the number of logits is only related to the number of classes. The performance of the vanilla KD can also be regarded as benchmark of distillation methods for cross-dimensionality problems.
As shown in Table I the fluctuation range of VHD is greater than HD. The main reason might be the Variable-length Hilbert curve is not directly obtained in this work. We approximately achieve the curve by the products of the corresponding values in Activation Mapping, which is the same as the self-adaptive weights used for the vanilla Hilbert curve. Although the values deduced by the gradients of output scores do not perfectly indicate the activation features, VHD always outperforms others in all experiments. As demonstrated in Fig. Bb), only VHD can reach over 86% Top-1 I accuracy. To show the ad advattages of our method more clearly, calculate the Average Relative Improvement
8

---

(ARI) by
1 ACCyHD ACC ARI
100%
(17)
M
ACCERD -AcosTU
where is the number of different architecture combinations, and BKD and STU refer to the baseline methods and student network, respectively. As shown in Table ARI presents the magnitude of the improvement of the proposed VHD compared with the method located in that row.
4.4
Hyperparameter Tuning
65
88
65
88
Top-1 Aocumay (%9
64
VHD
190 A00000 (%0)
86
VHD
Top Acauasy (%%
64
VHD
Too Accuuma (9%)
86
VHD
HDD
HDD
63
84
63
84
62
82
62
82
61
80
61
80
60
78
60
78
32
61
128
256
32
61
123
296
11
12
13
14
11
12
13
14
Channel
Channel
Layer
Layer
(a) ActivityNet
(b) COVID-19
(c) ActivityNet
(d) Large- COVID-19
Figure 4: Performance of proposed methods HD and VHD with respect various hyperparameters. Results are from the ResNet-50 model distilled from 3D ResNet-50.
In addition to discussing the effectiveness of our methods, we also conduct tuning experiments to demonstrate the influence the hyperparameters. We follow the setup described above, and adopt 3D ResNet-50 as the teacher model to train a ResNet-50 as the devised distillation method. Unlike range values shown above, here we only present the mean Top-I accuracy (%) of the distilled student networks to compare the fluctuation more clearly.
Hyperparameters in the loss functions. As shown in Eq. the hyperparameter con- trols the tradeoff of the original task and distillation during training. Fig. S illustrates the performance with respect to various for both two tasks. The value of could be initial- ized with relatively large value, i.e., about 10 for the medical imaging classification and 10 for the activity classification task. Extensive experiments demonstrate that our method can perform well in the dinensiiinns problems given any value within reasonable range.
The number of feature maps for distillation. Sec. 3ec.3imm iittroduces the distillation using a pair of feature maps extracted from 3D and 2D models. In most cases, a given layer can generate the same number of feature maps as the convolutional kernel. The distillation loss Eq. [ can also be calculated in batches between the same number of dimensionally heterogeneous feature maps. For architecture ho- mogeneous models such as 3D ResNet and ResNet, the number of feature maps naturally the same as in the symmetrically positioned layer. For architecture heterogeneous models, we suggest applying extra 1x convolutional layers align the channel dimension, i.e.,
Toccuryyy (%
64
63
62
VHD
61
10-10"10* 108 10 10*
86
Top 1 AAccracy
84
the number of feature maps. Figs. a) and (b) show the performance
82
VHD
of our method with respect to various number of feature maps. The highest score in both figures is reached when using 128 feature maps for distillation.
80
HD
10'10"10' '0' 10 10 10
Figure 5: Performance with respect to various a on the ActivityNet (top) and Large- COVID-19 (bottom).
Layer selection for distillation. Finally, we show the effectiveness of our methods with respect various convolutional layers. Figs. 4ic) and (d) illustrate that the distillation on the second layer (res- block) contributes to the most improvement for the student model. It is important to note that the layer distribution (from layer to layer
4) in our experiments is from ResNet, thereby this result shown for reference only, if a different model architecture is applied. In conclusion, rear layers whose feature

---

maps are not too small should have priority to be distilled because very small feature maps usually do not have explicit structural knowledge.
Conclusion
Although considerable improvement has been obtained using cumbersome models when analyzing volumetric data, the success of cumbersome models comes primarily from the availability of comput- ing power. Our goal is distill the cumbersome models into lightweight models while preserving the structural information the high-dimensional space. To this end, we propose a Hilbert curve-based approach, which aims transfer structural information as much as possible from 3D to 2D models. Furthermore, we design the Variable-length Hilbert Distillation to force the student to focus more on learning the structural knowledge from activation features. Extensive experiments demonstrate the effectiveness of the proposed methods.
Acknowledgements
This work is supported by the National Natural Science Foundation of China under Grants 61972349, 62202422 and 62106221.
References
[1] Alber, J., Niedermeier, R.: On multidimensional curves with hilbert property. Theory of Computing Systems 33(4), 295-312 (2000)
[2] Cai, J.X., Mu, T.J., Lai, Y.K., Hu, S.M.: Linknet: 2d-3d linked multi-modal network for online
semantic segmentation of rgb-d videos. Computers & Graphics 98, 37-47 (2021)
[3] Donoho, D.L., Grimes, C.: Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences 100(10), 5591-5596 (2003)
[4] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1999 (2020)
[5] Fabian Caba Heilbron, Victor Escorcia, B.G., Niebles, J.C.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-976 (2015)
[6] Fisher, R.A.: The statistical utilization of multiple measurements. Annals of eugenies 8(4), 376-386 (1938)
[7] Gao, M., Shen, Y., Li, Q., Loy, C.C.: Residual knowledge distillation. arXiv preprint arXiv:2002.09168 (2020)
[8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[9] He, T., Shen, C., Tian, Z., Gong, D., Sun, C., Yan, Y.: Knowledge adaptation for efficient semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 578-587 (2019)
[10] Hilbert, D.: Über die stetige abbildung einer linie auf ein flächenstück. In: Dritter Band:
Analysis- Grundlagen der Mathematik- Physik Verschiedenes, pp. 1-2. Springer (1935)
[11] Hinton, G., Vinyals, O., Dean, J., al.: Distilling the knowledge in a neural network. arXiv preprint 1eXivv15533.22:
2(7) (2015)
[12] Hotelling, H.: Analysis a complex of statistical variables into principal components. Journal of educational psychology 24(6), 417 (1933)
[13] Hu, W., Zhao, H., Jiang. I.., Jia, J., Wong, T.T.: Bidirectional projection network for cross dimension scene understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14373-14382 (2021)
10

---

[14] Ji, M., Heo, B., Park, S.: Show, attend and distill: Knowledge distillation via attention- based feature matching. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 7945-7952 (2021)
[15] Jiang, P.T., Zhang, C.B., Hou, Q. Cheng, M.M., Wei, Y.: Layercam: Exploring hierarchical
class activation maps for localization. IEEE Transactions on Image Processing 30, (2021)
[16] Li, X., Li, S., Omar, B., Wu, F.. Li. X.: Reskd: Residual-guidee guided knowledge distillation. IEEE Transactions on Image Processing 30, 4735 4774 (2021)
[17] Lindenmayer, A.: Mathematical models for cellular interactions in development filaments with one-sided inputs. Journal of theoretical biology 18(3), 280-299 (1968)
[18] Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang. J.: Structured knowledge distillation for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2604-2613 (2019)
[19] Liu, Z., Qi, X., Fu, C.W.: 3d-to-2d distillation for indoor scene parsing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4464 4474 (2021)
[20] Maftouni, M., Law, A.C.C., Shen, B., Grado, Z.J.K., Zhou, Y., Yazdi, N.A.: A robust ensemble- deep learning model for covid- diagnosis based integrated scan images database. In: Institute Industrial and Systems Engineers
IIE Annual Conference. Proceedings. pp.
(ISE) (2021)
[21] Moon, B., Jagadish, H.V., Faloutsos, C., Saltz, J.H.: Analysis of the clustering properties of the hilbert space-filling filling curve. IEEE Transactions on knowledge and data engineering 13(1), (1),
124-141 (2001)
[22] Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3967-3976 (2019)
[23] Passalis, N., Tefas, A.: Learning deep representations with probabilistic knowledge transfer. In:
Proceedings of the European Conference Computer Vision (ECCV). pp. 268-284 (2018)
[24] Passalis, N., Tzelepi, M., Tefas, A.: Heterogeneous knowledge distillation using information flow modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2-399-348 (2020)
[25] Peng. B., Jin, X., Liu, J., Li, D., Wu, Y., Liu, Y., Zhou, S., Zhang, Z.: Correlation congruence for knowledge distillation. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 5007-5016 (2019)
[26] Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for (2014)
thin deep nets. arviv preprint
[27] Ryoo, M.S., Piergiovanni, A., Arnab, A., Dehghani, M., Angelova, A.: Tokenlearner: What can
8 learned tokens do for images and videos? arXiv preprint arXiv:2106.1 1297 (2021)
[28] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-basee localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618-626 (2017)
[29] Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.e.: Convolutional Istm network:
A machine learning approach for precipitation nowcasting. Advances in neural information
processing systems 28 (2015)
[30] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arx: 1409.1456666 (2014)
[31] Tenenbaum, J.B., Silva, V.d., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. science 290(5500), 2319-2323 (2000)
[32] Tian, Y., Krishnan, D.. Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910. 10699 (2019)
[33] Tung, F., Mori, G.: Similarity-preservinn knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1365-1374 (2019)
[34] Wang, Y., Zhou, W., Jiang, T., Bai, X., Xu, Y.: Intra-class feature variation distillation for semantic segmentation. In: European Conference on Computer Vision. pp. 346-362, Springer (2020)
11

---

[35] Xu, G., Liu, Z., Li, X., Loy, C.C.: Knowledge distillation meets self-supervision. In: European Conference on Computer Vision. pp.
Springer (2020)
[36] Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 16122.39289 (2016)
[37] 7hou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.:
Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921-2929 (2016)
[38] Zhou, S., Wang, Y., Chen, D., Chen, J., Wang, X., Wang, C.,, Bu, J.: Distilling holistic knowledge with graph neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10387-10396 (2021)
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Sec. 4.3 describes the limitation
that the VHD is not directly applicable in our work. The proposed approximation
approach may results in a larger fluctuation range of the distillation performance.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] There
no potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] In the supplemental material, we release the source codes as a URL.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please refer to Secs. 4.1 and 4.2
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We have reported all error bars the experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer the experiment setting in Sec.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have properly cited all the creators such as (5)
(b) Did you mention the license of the assets? [N/A] We only use publicly available assets,
and we have properly cited all the assets.
(c) Did you include any new assets either the supplemental material as URL? [Yes] The supplemental material includes the source codes of this work that will be provided
as a URL. There no other asset be released.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All the data used in this work are publicly available, and we have properly cited such as (5)2
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] There is no personally identifiable information
or offensive content in the data we are using.
5. If you used crowdsourcing conducted research with human subjects...
12

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "kjkM-F58RDj", "reviewer": "Reviewer_iQbj", "summary": "The paper studies 3D knowledge distillation for video analysis and medical imaging via Hilbert curve. The authors propose a Hilbert Distillation (HD), to explicitly distill structural information in intermediate features of 3D models. The authors further design Variable-length Hilbert Distillation (VHD) to dynamically shorten the walking stride of the Hilbert curve. The authors demonstrate the robustness of the method compared with some state-of-the-art methods.\n", "questions": "- Although the overall architecture is novel, its individual components are largely inspired by previous works [1 - 4]. In my humble view, the work seems to be very similar to the previous work. The technical novelty should be emphasized.\n- Line # 169-170 “In reality, in the medical imaging task that 3D models are commonly applied, the spatial distribution of human organs are always fixed regardless of the data modality”. Why is the spatial distribution always fixed? Based on my understanding, the distribution of organs, especifically tumors,  should share a high variance [5 - 6, 12]. The authors should explore more medical examples instead of showing the special case.\n- Line # 156 - 158 “only partial feature maps are activated and crucial for the final task”. Could you show some visualization to demonstrate these.\n- As shown in Tables, the magnitude of the improvement of the proposed distillation-based methods remain unclear.\n- I am wondering whether the authors can provide a more clear picture of how the Hibert distillation works. Although the results and the experiments look good, this is still the heart of the model, and it makes it hard for me to reason about the importance of the result and whether it is exhaustively defended? Do you have any theoretical proofs?\n- Lack of comparing current state-of-the-art knowledge distillation based methods [5 - 8].\n- More benchmarks in computer vision and medical imaging domains should be included to demonstrate the robustness of the proposed method.\n\nReference\n[1] Self-Distillation Amplifies Regularization in Hilbert Space\n\n[2] Overcoming the Curse of Dimensionality in Neural Networks\n\n[3] Does knowledge distillation really work?\n\n[4] Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation\n\n[5] Tumor‐infiltrating lymphocytes are a marker for microsatellite instability in colorectal carcinoma\n\n[6] Integrative genomic profiling of large-cell neuroendocrine carcinomas reveals distinct subtypes of high-grade neuroendocrine lung tumors\n\n[7] Contrastive representation distillation\n\n[8] Heterogeneous knowledge distillation using information flow modeling\n\n[9] Contrastive multiview coding\n\n[10] Residual knowledge distillation\n\n[11] Reskd: Residual-guided knowledge distillation\n\n[12] Novel transfer learning approach for medical imaging with limited labeled data\n\n", "limitations": "See “Questions” above.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "##### Strengths\n+ The paper proposes Hilbert Distillation (HD) to distill structural information from intermediate 3D feature maps.\n+ The authors introduce Variable-length Hilbert Distillation (VHD) to efficiently transfer structural knowledge via Hibert curves.\n+ The results are promising compared to some previous baselines.\n\n##### Weaknesses\n- The technical novelty of the method could be emphasized.  \n- The usefulness and motivation of the Hilbert curve is not clear.\n- There are some claims that are not well supported with references, empirical or theoretical evidence (See “Question” - Q3)\n- Results should include state-of-the-art methods from the literature.\n- More benchmarks should be included to demonstrate the robustness of the proposed method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SzLSXU_uQYJ", "reviewer": "Reviewer_eP7P", "summary": "This paper presents a cross-dimensionality distillation approach based on the use of Hilbert curves. This facilitated the distillation of 3D network to 2D networks, with a variable-length method to encourage the 2D network to attend to activation features. Experiments were performed on two distinct datasets, compared to a set of existing knowledge distillation methods. Clear margins of improvement were observed in both datasets.", "questions": "A quick comment on the computation cost would be appreciated. ", "limitations": "There did not seem to be discussion of the limitation of the presented work. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presented adoption of Hilbert curves to enable cross-dimensionality distillation is interesting, elegant, and well rationalized. The novelty is high.\n\nThe experimentation was thorough, and the baseline models are representative of existing approaches. The results show significant improvements over all baseline models in both datasets. The ablation analyses were also thorough.\n\nI do not see notable weakness in the paper. One curiosity is regarding the computational cost of the construction of the Hilbert curve, and how does that affect the training of the model.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NAgjGsmOj_R", "reviewer": "Reviewer_ELut", "summary": "In this paper the authors present a cross-dimensional distillation approach. More specifically, they proposed to use the knowledge of 3D networks to improve the performance of 2D networks. TO this end, they propose to present the structural information that exists in the teacher model using Hilbert curves, which can map the high dimensional representations to one dimensional continuous filling curves. The student network is then supervised using this information. To further improve the performance of the proposed method they also propose a variable length variant of the proposed method which can dynamically shorten the walking stride of the Hilbert curve. The authors demonstrate that the proposed method outperforms the current distillation methods that can be used for cross-dimensionality distillation using two datasets.", "questions": "Generally I am positive regarding this paper, since it introduces a novel idea on how cross-dimensional distillation can be performed. However, there is clear room for improvements, e.g., compare with regular distillation from larger 2D networks to smaller ones, evaluate the effect of using different layers - different layer matching, discuss the computational complexity, extend the evaluation to other larger datasets. ", "limitations": "Authors discuss limitations of their work. However, there is no dedicated limitations section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nPositive aspects:\n\n- The idea proposed in this paper is interesting and makes sense. The paper is well written and easy to follow.\n- The experimental evaluation clearly demonstrates the improvements obtained using the proposed method. Ablation studies are also provided.\n\nNegative aspects:\n- There are no comparisons with regular distillation approaches from larger 2D networks to smaller 2D networks. Only cross dimensional distillation is evaluated - if I have understood the experiments correctly.\n- It is not clear how the layer that is used for extracting the features can affect the distillation. How should the layers be matched to each other?\n- There is no discussion on the computational complexity of generating the Hilbert-based matching during the training.\n- Only two -relatively small - datasets were used for the evaluation. As far as I understand the proposed method can be used for any video dataset.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9kCkd04JV9h", "reviewer": "Reviewer_qHTA", "summary": "The paper propose a Hebert curve-based approach to perform distillation of the large models into lightweight models. In specific, the paper proposes Variable-length Hilbert curve that if flexible for different activation maps from different inputs. This is different than vanilla Hilbert curve that offers hard mapping for a space of fixed scale. The efficacy of the methods is demonstrated in two separate benchmarking datasets where the proposed method achieve improved performance over existing methods. \n", "questions": "Check weakness section above.", "limitations": "Check weakness section above.", "rating": 7, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper proposes a novel approach based on Hibert curve to perform distillation to facilitate the knowledge of 3D networks to improve the performance of 2D networks. The idea is interesting as it doesn't require computational scale as the existing methods.\n- The proposed Hilbert Distillation method is extended as Variable-length Hilbert Distillation (VHD) to provide more flexibilityin activation feature map and stride length. \n- The experimental evaluation demonstrate improved performance. \n\nWeakness:\n- The major weakness is the poor writing and presentation of the paper. This not only made reading paper difficult but also made it impossible to understand the results and outcomes of the proposed method. Some highlights of this include:\na. Poor and unclear figures: Fig 3, Fig 4 and Fig 5 is missing legends, X- and Y-axis. It is impossible to understand what's happening. \nb. Grammatical issues in the writing. \nOverall, the submission quality is far below the quality of NeurIPS submission. I strongly encourage authors to carefully proofread their paper before submission. \n\n- The experimental setup also is confusing. \na. Is ActivityNet also splitted randomly? The wordings in the description of two datasets raises the question.\nb. Also why different ratio of train-test split is done for two dataset?\nc. How are hyperparameter selected? e.g., alpha, epochs number, batch-size, etc.\n\n\n######### update ##########\nThere was technical issue on my end which impacted my initial reviews. After resolving this technical issue, I was able to understand the paper more clearly. With this understanding, I am increasing my score to 7. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept