Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies 3D knowledge distillation for video analysis and medical imaging via Hilbert curve. The authors propose a Hilbert Distillation (HD), to explicitly distill structural information in intermediate features of 3D models. The authors further design Variable-length Hilbert Distillation (VHD) to dynamically shorten the walking stride of the Hilbert curve. The authors demonstrate the robustness of the method compared with some state-of-the-art methods.

strengths_and_weaknesses: ##### Strengths
+ The paper proposes Hilbert Distillation (HD) to distill structural information from intermediate 3D feature maps.
+ The authors introduce Variable-length Hilbert Distillation (VHD) to efficiently transfer structural knowledge via Hibert curves.
+ The results are promising compared to some previous baselines.

##### Weaknesses
- The technical novelty of the method could be emphasized.  
- The usefulness and motivation of the Hilbert curve is not clear.
- There are some claims that are not well supported with references, empirical or theoretical evidence (See “Question” - Q3)
- Results should include state-of-the-art methods from the literature.
- More benchmarks should be included to demonstrate the robustness of the proposed method.

questions: - Although the overall architecture is novel, its individual components are largely inspired by previous works [1 - 4]. In my humble view, the work seems to be very similar to the previous work. The technical novelty should be emphasized.
- Line # 169-170 “In reality, in the medical imaging task that 3D models are commonly applied, the spatial distribution of human organs are always fixed regardless of the data modality”. Why is the spatial distribution always fixed? Based on my understanding, the distribution of organs, especifically tumors,  should share a high variance [5 - 6, 12]. The authors should explore more medical examples instead of showing the special case.
- Line # 156 - 158 “only partial feature maps are activated and crucial for the final task”. Could you show some visualization to demonstrate these.
- As shown in Tables, the magnitude of the improvement of the proposed distillation-based methods remain unclear.
- I am wondering whether the authors can provide a more clear picture of how the Hibert distillation works. Although the results and the experiments look good, this is still the heart of the model, and it makes it hard for me to reason about the importance of the result and whether it is exhaustively defended? Do you have any theoretical proofs?
- Lack of comparing current state-of-the-art knowledge distillation based methods [5 - 8].
- More benchmarks in computer vision and medical imaging domains should be included to demonstrate the robustness of the proposed method.

Reference
[1] Self-Distillation Amplifies Regularization in Hilbert Space

[2] Overcoming the Curse of Dimensionality in Neural Networks

[3] Does knowledge distillation really work?

[4] Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation

[5] Tumor‐infiltrating lymphocytes are a marker for microsatellite instability in colorectal carcinoma

[6] Integrative genomic profiling of large-cell neuroendocrine carcinomas reveals distinct subtypes of high-grade neuroendocrine lung tumors

[7] Contrastive representation distillation

[8] Heterogeneous knowledge distillation using information flow modeling

[9] Contrastive multiview coding

[10] Residual knowledge distillation

[11] Reskd: Residual-guided knowledge distillation

[12] Novel transfer learning approach for medical imaging with limited labeled data


limitations: See “Questions” above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a cross-dimensionality distillation approach based on the use of Hilbert curves. This facilitated the distillation of 3D network to 2D networks, with a variable-length method to encourage the 2D network to attend to activation features. Experiments were performed on two distinct datasets, compared to a set of existing knowledge distillation methods. Clear margins of improvement were observed in both datasets.
strengths_and_weaknesses: The presented adoption of Hilbert curves to enable cross-dimensionality distillation is interesting, elegant, and well rationalized. The novelty is high.

The experimentation was thorough, and the baseline models are representative of existing approaches. The results show significant improvements over all baseline models in both datasets. The ablation analyses were also thorough.

I do not see notable weakness in the paper. One curiosity is regarding the computational cost of the construction of the Hilbert curve, and how does that affect the training of the model.

questions: A quick comment on the computation cost would be appreciated. 
limitations: There did not seem to be discussion of the limitation of the presented work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper propose a Hebert curve-based approach to perform distillation of the large models into lightweight models. In specific, the paper proposes Variable-length Hilbert curve that if flexible for different activation maps from different inputs. This is different than vanilla Hilbert curve that offers hard mapping for a space of fixed scale. The efficacy of the methods is demonstrated in two separate benchmarking datasets where the proposed method achieve improved performance over existing methods. 

strengths_and_weaknesses: Strength:
- The paper proposes a novel approach based on Hibert curve to perform distillation to facilitate the knowledge of 3D networks to improve the performance of 2D networks. The idea is interesting as it doesn't require computational scale as the existing methods.
- The proposed Hilbert Distillation method is extended as Variable-length Hilbert Distillation (VHD) to provide more flexibilityin activation feature map and stride length. 
- The experimental evaluation demonstrate improved performance. 

Weakness:
- The major weakness is the poor writing and presentation of the paper. This not only made reading paper difficult but also made it impossible to understand the results and outcomes of the proposed method. Some highlights of this include:
a. Poor and unclear figures: Fig 3, Fig 4 and Fig 5 is missing legends, X- and Y-axis. It is impossible to understand what's happening. 
b. Grammatical issues in the writing. 
Overall, the submission quality is far below the quality of NeurIPS submission. I strongly encourage authors to carefully proofread their paper before submission. 

- The experimental setup also is confusing. 
a. Is ActivityNet also splitted randomly? The wordings in the description of two datasets raises the question.
b. Also why different ratio of train-test split is done for two dataset?
c. How are hyperparameter selected? e.g., alpha, epochs number, batch-size, etc.


######### update ##########
There was technical issue on my end which impacted my initial reviews. After resolving this technical issue, I was able to understand the paper more clearly. With this understanding, I am increasing my score to 7. 
questions: Check weakness section above.
limitations: Check weakness section above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper the authors present a cross-dimensional distillation approach. More specifically, they proposed to use the knowledge of 3D networks to improve the performance of 2D networks. TO this end, they propose to present the structural information that exists in the teacher model using Hilbert curves, which can map the high dimensional representations to one dimensional continuous filling curves. The student network is then supervised using this information. To further improve the performance of the proposed method they also propose a variable length variant of the proposed method which can dynamically shorten the walking stride of the Hilbert curve. The authors demonstrate that the proposed method outperforms the current distillation methods that can be used for cross-dimensionality distillation using two datasets.
strengths_and_weaknesses: 
Positive aspects:

- The idea proposed in this paper is interesting and makes sense. The paper is well written and easy to follow.
- The experimental evaluation clearly demonstrates the improvements obtained using the proposed method. Ablation studies are also provided.

Negative aspects:
- There are no comparisons with regular distillation approaches from larger 2D networks to smaller 2D networks. Only cross dimensional distillation is evaluated - if I have understood the experiments correctly.
- It is not clear how the layer that is used for extracting the features can affect the distillation. How should the layers be matched to each other?
- There is no discussion on the computational complexity of generating the Hilbert-based matching during the training.
- Only two -relatively small - datasets were used for the evaluation. As far as I understand the proposed method can be used for any video dataset.


questions: Generally I am positive regarding this paper, since it introduces a novel idea on how cross-dimensional distillation can be performed. However, there is clear room for improvements, e.g., compare with regular distillation from larger 2D networks to smaller ones, evaluate the effect of using different layers - different layer matching, discuss the computational complexity, extend the evaluation to other larger datasets. 
limitations: Authors discuss limitations of their work. However, there is no dedicated limitations section.
review_title: null

Meta-review: metareview: This submission was reviewed by four reviewers. All reviewers provided detailed and informative reviews. During rebuttal, the authors actively submitted detailed rebuttals, which lead to improved evaluations by the reviewers with improved scores. Overall, this is an interesting paper and an accept is recommended.
confidence: Certain
Decision: Accept