Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a method to do zero-shot ICD coding, which comes down to determining which elements from a set of natural language labels (ICD codes) apply to a given text (diagnostic summary). The main problem is that many codes have zero or very few examples. The proposed solution for this problem is to learn a feature-space generator for examples which can be conditioned on a code. This generator is trained using a GAN, which moves the few-/zero-shot problem to training the discriminator. Here, the tree structure of the ICD codes is used, by using examples with sibling labels as approximate examples for codes with few or on labelled data points. Additionally, a keyword reconstruction loss is used, based on the idea that the keywords of the corresponding ICD code can be reconstructed from a good feature vector.

The paper is written and executed well; the setup, which involves a fair number of components, is described and motivated clearly. The experiments include comparisons to state of the art results, finding that applying the GAN-based technique they choose (introduced in Xian et al., 2018 for computer vision problems) produces significant gains in recall of low-data classes (few or zero examples). The precision decreases, probably due to a shift from false negatives to false positives, while the AUC shows modest gains.

The main potential issue with the paper is the degree of (effective) novelty. The bulk of the gain relative to the SotA seems to be achievable by applying the GAN-based method, which is described in the Xian et al. (2018) reference, or by applying the label-distribution-aware margin, which are known techniques even if they have not been applied to this particular dataset yet. The additional elements introduced by the authors - the use of sibling codes and the keyword reconstruction loss - are good ideas, and it is worthwhile having a documented test of the benefit they provide, but they don’t seem to have a major influence on the quality of the model.

All in all, I would argue for the paper to be accepted. The work done here is valuable to have on record, and the presentation and execution are well done.

Two questions for the authors:

1. Could the model benefit from having a self-attention model, like a transformer? This applies mostly to the diagnostic text encoder, as interpreting the ICD code descriptions seems not to depend strongly on structure or context. From the text of the paper it appears that a 1D convolution was used to process the diagnostic texts, but I would expect that longer-distance links between words can be quite relevant there.

2. Could the precision-recall curves be added to the supplementary material?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper addresses the problem of zero-shot prediction for ICD codes for medical notes. The main idea of the paper is to use GANs to generate latent features for the text conditioned on the ICD codes and taking into account the tree-like structure of the ICD codes themselves. The classification of ICD codes is trained on the GAN-extracted features.

The input clinical document is represented as a concatenation of its word embeddings. The representation of an ICD code is realized by averaging the embeddings of the words in its corresponding description. The feature extraction model takes the word embeddings from the input text and passes them through a CNN; in addition, attention is used for the encodings of all the ICD codes.

For further encoding the ICD codes, the tree-like organization of the codes is exploited by using GRNNs. The generated features together with the graph-encodings of the ICD codes are used to classify the clinical text.

Some suggestions for improvement:

* It was not clear the relationship with previous work by Rios and Kavuluru. I had to refer to the paper to be able to appreciate the differences. The feature extraction network seems to be similar (Rios & Kavuluru used GCNNs instead of GRNNs to encode the ICD codes). The GAN-based feature generation seems to be new in this paper. 

* I did not understand the comment regarding the independent training of the zero-shot cases vs the rest.

* Siblings ICD codes mean codes that share an immediate parent?

* Table 1. seems to cite the related work results, but it doesn't seem to include the results of the proposed method.

* In Table 2, for the cases where precision and recall are 0, are the AUC numbers correct? Table 2 also shows the results for the ablation studies that are discussed later in the section. It took me a while to understand the difference between Table 2 and Table 4.

* For Table 4, you could add the explanation on what you considered few-shot cases for those results. 

I think the paper could improve the explanation of the components of the model and the presentation of the evaluation results.

I think the idea of using GANs to generate features that can help with the classification + taking advantage of the relationship/structure among the ICD codes are both interesting ideas.
review_title: Official Blind Review #2571

final_score: rating: 3: Weak Reject
review_content: The paper deals with the problem of text classification when the number of class is large (17000) and most of the classes do not have examples in the training set. This problem is known as Zero-shot learning. The paper proposes to use adversarial methods and the hierarchical organisation of the classes to improve current models.

The paper lacks a clear description of the complete system : Figure 1 seems to be the one but there is no mention of the classification part. The description of each bloc is clear enough independently  but many question remains on the global picture. 

The author should for example emphasis the fact that in their problemn the classes come with a short text  description. This is somehow unususal for a text classification problem where usually the classes are not defined by a description. In the figure, the difference of the processing of the text from the clinical document and from the class description is not clear. 

3.1 is related to feature extraction for the clinical document, but it is also said that this processing is also apply to class description.

3.2 label encoder : if the notations were the same as on figure A, it would help understanding.

Results : the claim "our methods improve the F1 score from nearly 0 to 20.91% for the zero-shot codes" is not true : state-of-the-art models such as Xian2018 and Felix2018 are already over 20% F1. 
 
When looking at Figure 3 and the analysis, WGAN-Z seem to provide better representation that WGAN with it has almost no impact on the classification results (F1 20.48 versus F1 20.30)

In conclusion, it seems that the proposed methods make the model more complex without bringing a significant improvements.
review_title: Official Blind Review #1

Meta-review: This paper proposes a method to do zero-shot ICD coding, which involves assigning natural language labels (ICD codes) to input text. This is an important practical problem in healthcare, and it is not straightforward to solve, because many ICD codes have none or very few training examples due to the long distribution tail. The authors adapt a GAN-based technique previously used in vision to solve this problem. All of the reviewers agree that the paper is well written and well executed, and that the results are good. However, the reviewers have expressed concerns about the novelty of the GAN adaptation step, and left this paper very much borderline based on the scores it received. Due to the capacity restrictions I therefore have to recommend rejection, however I hope that the  authors resubmit elsewhere. 
Decision: Reject