What's behind the mask? Enhancing Large Language Models with Pseudo Data
Authors: Paul Landes, Aaron J Chaise, Tarak Nath Nandi, Ravi Madduri


Reviewer paper summaries: The paper investigates the effects of training large language models on masked (redacted) text versus realistic pseudo data, in the context of clinical notes. The authors pretrain several state-of-the-art language models (RoBERTa, XLM-RoBERTa) on a MIMIC-III dataset with masked tokens intact versus with tokens replaced by pseudo data generated from realistic sources. They then fine-tune these models for the protected health information (PHI) de-identification task. Surprisingly, certain masked models outperform their pseudo counterparts, though XLM-RoBERTa benefits more from pseudo data. The best performing models surpass previous baselines by a large margin on the de-identification task. The paper analyzes these results, provides open-source resources, and discusses recommendations for language model pretraining based on their findings.
Summary of strengths: 1. Open-source contributions: The paper makes available the pretrained embeddings, de-identification models, pseudo dataset generation code, and other resources, facilitating reproducibility and future research in this area.

2. The paper is well written and clearly organized. Pseudo data generation limitations: The bias in syntheitc data generation could be challenge for this approach as not many models cana generate high quality domain specific synthetic samples. How is the synthetic data quality measured and what are the steps for poor sample removal ?
Summary of weaknesses: 1. Pseudo data generation limitations: The bias in syntheitc data generation could be challenge for this approach as not many models cana generate high quality domain specific synthetic samples. How is the synthetic data quality measured and what are the steps for poor sample removal ?

2. Lack of qualitative analysis: Examining samples of the model outputs, particularly for error cases, could give better insights into the differences between masked and pseudo models' behaviors.

3. Limited model types: The paper focuses on variants of RoBERTa and XLM-RoBERTa models. Evaluating the proposed approaches on other state-of-the-art language model architectures like llmama Mistral, etc. could provide a more generalizable understanding of the findings.

4. Dataset limitations: The paper uses the MIMIC-III dataset, which, despite being a popular clinical corpus, may not be fully representative of the diversity of clinical notes and healthcare systems across different regions and institutions.

5. The paper mentions using the MIMIC-III dataset for creating the masked and pseudo data used for pre-training and fine-tuning their models. However, the details on how this dataset was constructed are not provided. What were the instructions given to the human annotators ?
Comments, suggestions and typos: NA
Overall assessment scores: 2.5