SlowGuess/NAACL_2022_OCR huggingface dataset:
Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333-389.
Tanveer Syeda-Mahmood. 2010. Similarity retrieval of cardiac reports. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, pages 1135-1141. IEEE.
Tanveer F. Syeda-Mahmood, Ken C. L. Wong, Joy T. Wu, Ashutosh Jadhav, and Orest B. Boyko. 2020. Extracting and learning fine-grained labels from chest radiographs. In AMIA 2020, American Medical Informatics Association Annual Symposium, Virtual Event, USA, November 14-18, 2020. AMIA.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. 2017. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106.
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia. Association for Computational Linguistics.
Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019a. Applying bert to document retrieval with birch. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 19-24.
Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019b. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3481-3487.
A Appendix: Model training details
Here we provide more details on the models used in Section 3. We used the Huggingface "BERT-base-uncased" model (pre-trained on BookCorpus and English Wikipedia, availabel at: https://huggingface.co/bert-base-uncased) as our BERT model for comparison. The BERT (fine-tuned) model was fine-tuned on the EHR text (Indiana or NIH dataset) using MLM for 5 epochs based on the "BERT-base-uncased" model. The pre-trained BioClinicalBERT (Alsentzer et al., 2019) (availabel at: https://github.com/EmilyAlsentzer/clinicalBERT) was initialized with BioBERT (Lee et al., 2020) and fine-tuned on clinical notes.
Our SBERT model was initialized with the BioClinicalBERT. We fine-tuned SBERT using the triplet loss for 10 epochs for all datasets in this study. We used AdamW optimizer with the learning rate 2e-5, weight decay 0.01 and a linear learning rate warm-up of 100 steps.
The SBERT model used as comparison was pre-trained on 500K (query, answer) pairs from the MS MARCO dataset. This pre-trained model (msmarco-bert-base-dot-v5) was one of the recommended sentence embedding models from the official SBERT webpage (https://www.sbert.net/docs/pretrained_models.html).
Among all the pre-trained models, we picked this one because it is the only pre-trained model based on "BERT-base" model, to be consistent with all the other models (all based on "BERT-base") in our experiments. Since this model was tuned to be used with dot-product, we used dot-product to calculate similarity scores only for this model in the retrieval experiments in Table 2. For all the other models, cosine-similarity was used to calculate scores. However, for the embedding separation analysis in Table 3, cosine-similarity was used for SBERT (MS MARCO) as well so that the scale of the similarity scores is comparable to the others.