Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors have proposed a data augmentation framework which helps in improving the performance of existing disease normalization models. The framework is compared against two existing methodologies and in smaller dataset setting as well.
strength_and_weaknesses: Strength: 
1. The authors have focused on an important problem of disease normalization.
2. The proposed framework improves the performance of the existing normalization models. 

Weaknesses [Questions to the authors]: 
1. The paper needs some proof-reading and formatting. Some common formatting issues are highlighted below:
    - The starting double quotes in latex need to be `` as opposed to " since they appear inverted during pdf built. 
    - All the tables would look a bit better with the top and bottom margins would look a bit cleaner.
    - Para 1 of the introduction consists of multiple sentences with grammatical errors as well. 
2. The authors have mentioned the work of Falis et al [2022] and Abdollahi et al [2021] for data augmentation using UMLS but have not compared their methodologies for data augmentation.
3. The information regarding the NER tool are completely missing from the paper. On which data was the NER model from section 4.1 trained and what was its performance? Since the performance of the framework heavily depends on the model as well. 
4. Is there a validity study done to see if the axis rotation method would not create diseases which are not anatomically or medically correct or significant?
5. Section 4.2 needs some more details regarding the training procedure. Were the experiments done using multiple seeds? If yes, could you please share the average and std for the results? 
6. Details in Section 5.2 are confusing. Para 2 of the section mentions accuracy as the evaluation metric but also mentions that F1-score was used for evaluation according to the previous work. Also, was macro or micro F1-score used for evaluation? 
7. Table 1 mentioned that the result is of devset accuracy. Are all the experiments performed on devset? And is the testset accuracy or F1-score not reported?

-------------------------------------
Recommendation to the authors
-------------------------------------
I believe that the work is important and necessary but I would also like to add that it would be much more suitable for a clinically oriented workshop or conference such as ML4H, MLHC, LOUHI, BioNLP or clinicalNLP. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow but needs some more work for refining the language and presentation. The work is novel but needs some more consideration for the validity of pre-training before the task fine-tuning. Currently, it would be quite hard to reproduce the results. 
summary_of_the_review: Currently, the work needs some more effort on the presentation and proof-reading. The experimentation and training procedure needs to be more clearly written and experiments need to be a bit more thorough in terms of number of runs. Some more baselines would also improve the quality of work.
As I mentioned earlier, I believe that the work is important and necessary but I would be much more suitable for a clinically oriented workshop or conference such as ML4H, MLHC, LOUHI, BioNLP or clinicalNLP.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes authors work related to set of data augmentation techniques that work together as an augmented training task for disease diagnosis normalization.Various techniques are described which show incremental improvement in disease diagnosis normalization against ICD-10 classification using CHIP-CDN dataset. The key to their work are described methods which augment the labels.
strength_and_weaknesses: Strengths:
- Authors work does demonstrate incremental benefit(3%) over prior related work.
- Clearly,this is valuable works there is significant variability in disease diagnosis as documented by clinicians.

Weakness: 
- Incremental benefit is limited.
- Some of the grammar could be improved.
- Unclear if the Chinese language have the exact translation for comparison.
clarity,_quality,_novelty_and_reproducibility: Quality: The paper is of reasonable quality.
Clarity: Table 6 is unclear.The comparison between different diagnosis codes goes from ankle to Knee and the label in the DA-AR is duplicated.So, not sure how to assess.Not sure if the challenge is different or even worse as there is a language component in it too(Chinese vs English).
Originality: While the authors show incremental benefit of their technique over prior related work, not sure if the gains can be generalized.
summary_of_the_review: The paper addresses an important challenge faced in healthcare data,especially with disease diagnosis. Abbreviations,different standards, different languages and different coding nomenclature(ICD vs others), make it difficult to find the correct diagnosis in the text. This is a challenge from not just label point of view but also from any other modeling aspect.
The authors have shown through their work that their novel techniques can show incremental benefit in overcoming this challenge.
But it is not clear, whether this is generalizable or not.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The proposed and evaluated data augmentation techniques for normalizing disease names in Chinese. The techniques were derived from disease corpora extracted from an existing dataset of authentic clinical text and the standardized ICD-10 coding system. Experimental results demonstrated a fair 3% performance gain with potential for further gains on smaller datasets.
strength_and_weaknesses: * Experimental results demonstrated a fair 3% performance gain with potential for further gains on smaller datasets.
* Medical natural language processing is societally important.
* Although the evaluation results demonstrated performance gains, no statistical significance testing, confidence intervals, or effect sizes were reported in the paper. However, the authors reported on an ablation study whose merit should be noted.
* The novelty and clarity of the study - or minimally their clear and convincing argumentation - can be questioned.
* Ethical considerations of this study were insufficiently addressed.
* Unfortunately, the paper was formatted using an incorrect template; it did not look like an ICLR 2023 paper and hence, should be desk rejected. 

clarity,_quality,_novelty_and_reproducibility: The paper addressed a fundamental task in medical natural language processing: disease name normalization. However, although this task is extensively studied, also in Chinese, the paper did not seem to properly acknowledge and appreciate this prior work. To illustrate, its first two sections (Introduction and Background) did not use peer-reviewed publications to support its argumentation (in total, one paper (arXiv preprint from 2019) was cited in the entire first section and none of the related literature was cited in the second section) and this problem of insufficient positioning of this paper in the prior work was also evident in the last section (Conclusion) of the paper; no discussion section or paragraph to position the contributions of this paper compared to the prior work was included here to close the study. Finally, the third section (Related Work) was hard to understand because the scope of the included studies was not communicated; I would have wanted to learn if, for example, the included papers were targeting or at least applicable to medical natural language processing in Chinese. Consequently, the novelty and clarity of the study - or minimally their clear and convincing argumentation - can be questioned.

Given the societal importance of medical natural language processing, I would have expected to see a broader impact statement in this paper. Also, although the CHIP-CDN - Chinese disease normalization dataset used in this study seems to have been collected and released before, the authors should have discussed relevant ethical considerations, minimally describing how the authors analyzed and evaluated that the dataset was created ethically and that their use of its for the purposes of this study was ethical.

Unfortunately, the paper was formatted using an incorrect template; it did not look like ICLR 2023 papers and hence, should be desk rejected. Also, further care should have been demonstrated in writing; for instance, the paper title was inconsistently capitalized.
summary_of_the_review: The paper has more weaknesses than strengths and hence, I recommend its rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper:  This paper introduces a new data augmentation method for Chinese disease normalization dataset after analysing the unnormalized disease name and standard disease name. The main contribution is a novel data augmentation method adapted to a new dataset, which consists of axis-word replacement and multi-grain aggregation.


strength_and_weaknesses: ## Strength:
- The paper proposes an interesting method to augment dataset, it achieave good performance compared with other augmentation method.
- It demonstrated that EDA and BT methods are harmful  on CDN datasets.

## Weakness:
- It would be nice if the authors could basically check how many ratios of  wrong labels brought by the proposed DA and existing one, either using expert annotation  or automatic evaluation. This could be tested by using only a few examples.

- There are no details about how many augmented examples used for baseline DA methods including EDA and BT. This should be carefully compared for a fair comparison.

- Only a small-scaled Chinese dataset was used. It would be better if the authors could evidences their method across many datasets

- The task is only for Chinese language, making it too narrow to fit ICLR.

- The authors should detail the motivaiton to use EDA and BT as the baselines. Since there are many other  DA methods in NLP, see https://arxiv.org/abs/2110.01852 




## minor issue:
- The font and the reference format seems wrong
clarity,_quality,_novelty_and_reproducibility: The paper seems a little bit redundant, e.g. two assumptions (structural invariance, transitivity) are repeated by many times.  -- it seems the naming of such two properties is not that appropriate. 

Questions:
- Why don't the authors also benchmark the DA method in English disease normalization dataset (if there is any). From my understanding, some of the DA methods are specific to Chinese language, right? If yes, this should be clearly stated. Otherwise English disease normalization dataset should be considered.

- Do you think it is more challenging for Chinese language than English to deal with  disease normalization dataset?  If so, please illustrate a little bit morel,

summary_of_the_review: The paper focus a intereting problem although the problem seems not a core problem in ICLR community. The method seems sound but I think it is not properly compared with various existing DA methods
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: SUMMARY

The paper proposes and evaluates data augmentation
techniques for normalizing disease names in Chinese. The
data used are derived from disease corpora extracted from
ICD-10 and clinical text.  Experiments demonstrate a 3%
performance gain with potential for further gains on smaller
datasets.

STRENGTHS

Important problem

Improved performance compared to prior work

Helpful ablation study

Linguistic contribution: the problem of normalization has
not been studied as extensively for Chinese as for English

WEAKNESSES

The work is rather incremental and lacks novelty.

The paper should explain more clearly how normalization
differs in English and Chinese (i.e., why it's not just a
matter of applying existing English techniques to Chinese).

Lack of comparison to important baselines

Experiments not described in sufficient detail, e.g.,
NER. Partially, but not sufficiently addressed in the response.

Experimental results vary depending on seed, but variation/confidence is not measured.

Evaluation on dev set only, lack of test set

Small-scale evaluation

A qualitative manual evaluation on a small sample is highly desirable.

Writeup is unclear and has poor quality.


justification_for_why_not_higher_score: Lack of novelty
justification_for_why_not_lower_score: no lower score possible
Decision: Reject