Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this work the authors propose to transfer knowledge between teacher and student networks trained on separate datasets, and claim to overcome challenges in availability of data annotations for challenging semantic segmentation in medical imaging domain.

Strengths
The proposed model is simple to follow and is targeted towards a significant problem in medical imaging analysis domain. 

Comments
The authors have used five segmentation networks, it is suggested that the selection of these five algorithms is further justified. 
One of the major concern in medical imaging domain is the black-box nature of the DL algorithms used, authors should comment on how relying on this black-box nature for knowledge transfer would effect the interpretability of these results. 
The method relies on three datasets and three models to come up with the final target segmentation, what are the requirements on the size of these datasets, in general the authors should discuss the effect of this on the overall performance.  
review_title: The authors present a network agnostic framework for student teacher training paradigm. The experiments and results are presented for medical imaging datasets, where annotations are hard to achieve. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes to use student-teacher training as a way of knowledge transfer between neural networks with different architectures without access to the source data. Instead the authors propose to use a separate dataset to transfer the knowledge of the teacher network and a potential different dataset for fine-tuning. The paper evaluates their method with various segmentation architectures by pretraining a DeepLab v3+ on an internal breast lesion dataset and testing transfer and fine-tuning using different medical datasets. The authors find that knowledge transfer performs similar to regular transfer learning in most combinations of datasets.

I believe that the paper tackles an interesting scenario of transferring knowledge from a fixed pre-trained network to a potentially different application without access to the original training data and sets up an extensive set of combinations of target tasks, student network architectures and transferral dataset (called dataset agent in the paper).

However, I cannot recommend the paper for acceptance in its current form. Most importantly the paper seems to be lacking proper positioning within the space of knowledge distillation and student-teacher training which leads to an unclear message about the novelty of the paper. Student-teacher training for knowledge distillation is not novel and the message that it is possible to transfer knowledge using student-teacher training  is unsurprising given that it has been shown before that you can transfer knowledge without any observed data (e.g. KegNet: Knowledge Extraction with No Observable Data. Yoo et al. NeurIPS 2019).
Further, the experiments do not contribute any new insights about how to chose the best student network nor which transferral dataset to use even though the introduction refers to unsuitability of certain pretraining tasks for a different target task. The first example (Table 2) seems to be using an internal dataset that randomly has been split into train/val/test splits and therefore resembles no real transfer learning task and it seems unsuprising that the 'direct learning' approach yields the best results. The second example (Table 3) using the Baheya breast lesion dataset seems to tackle the problem of unsupervised domain adaptation rather than transfer learning: the teacher and target dataset both tackle breast lesion segmentation on ultrasound images. Here it could be that using the target dataset as transferral dataset might help to adjust batch statistics for potential normalisation layers to improve the performance. This leaves the last two examples as only real transfer learning experiments. Lastly, the whole setup assumes that the input and output space of the student and teacher network are always the same, while it is argued that this approach allows for flexibility in difference of network architectures between the student and teacher network. However, semantic segmentation tasks in medical imaging often appear with various numbers of classes and 'input channels' requiring more advanced knowledge distillation -- this could be an interesting problem to tackle in a later version of this work.

Further comments:
- Data preprocessing: What preprocessing are you using for the training of the networks? How are you handling different shapes of the images? Are the segmentation algorithms trained on the full images or on patches?
- Transferral datasets: Looking at the images in Fig. 2, it seems that even the same modality ultrasound images show different sorts of image artefacts - do you clean those at all? Do you think domain shift might be something that's interacting with your setup?
- Do you have any theoretical or intuitive justification why you would want to perform knowledge transfer using unrelated data (skin lesion / different anatomy etc)? Why should this be better than using no data at all or regular computer vision datasets? Do you think the number of training examples for transfer matters?
- What do example segmentations look like? Are there similar shapes for different datasets? Does the network also learn some sort of shape prior? (see Oktay, et al. Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. 2017)
- Which loss did you use: CE / Dice-loss or a combination of the two?
- The term dataset agent is already used in the abstract and is not very clear - I'd personally find something like 'transfer[ral] dataset' easier to grasp.
- Introduction, first paragraph: 'black-[space]box' -> 'black-box'
- Introduction: 'network(teacher)' -> 'network (teacher)'
- What's a latent dataset? I would rather simply refer to 'learned representations' or 'knowledge'
- 'XXX' is already used in the introduction and not explained - I would simply refer to 'internal / in-house datasets'. Also, note the comment on the breast lesion dataset only being a single dataset with different splits.
- I have not seen the term 'educated' in reference to neural networks before - it would be more common to say 'trained'.
- Section 5.3.2): You mention that the networks trained from scratch have poor performance because of the small tuning dataset - I guess you are referring to the small training set?
- Another potential reference for knowledge transfer for medical imaging could be Kuzina, et al. Bayesian Generative Models for Knowledge Transfer in MRI Semantic Segmentation Problems. 2019
- As this work is relatively application-specific it might be better suited for one of the more medically inclined venues like MIDL, MICCAI, ...


review_title: Interesting direction but lacks positioning in related work and clearer experimentation

final_score: rating: 3: Clear rejection
review_content: The paper describes a  knowledge transfer technique based on  training a student network using annotation creating by  a teacher network. This is actually not a summary of the method but  the method itself. Most the the rest of the paper is devote do describe experiment details. 

The idea is  well known in machine learning community see e.g. Distilling the Knowledge in a Neural Network by Hinton. where is used to transfer knowledge from a huge network to a small network. Hence, there is not much novelty in the paper.

Although the method is very simple it is difficult the follow the experimental results. It is written in a very unclear way.
Do you use step 3 in the experiments? 

what is your conclusion regarding parameter fine tuning vs. your approach?

Over all the paper is more suitable for a medical imaging conference than  fro a general deep learning conference. 
review_title: The proposed idea is not novel

Meta-review: A majority of the reviewers find the paper lacks novelty and provides an insufficient discussion of the state-of-the-art in knowledge distillation and student teacher training to warrant publication.
The approach is quite narrow to the application domain and the paper does not provide novel insights on how to chose a good network.
A subset of the experiments performed on an internal data set with random train-test-splits do not evaluate a realistic transfer setting.


Decision: Reject