MDC at SemEval-2023 Task 7: Fine-tuning transformers for textual entailment prediction and evidence retrieval in clinical trials
Year: 2023

Abstract:
We present our entry to the Multi-evidence Natural Language Inference for Clinical Trial Data task at SemEval 2023. We submitted entries for both the evidence retrieval and textual entailment sub-tasks. For the evidence retrieval task, we fine-tuned the PubMedBERT transformer model to extract relevant evidence from clinical trial data given a hypothesis concerning either a single clinical trial or pair of clinical trials. Our best performing model achieved an F1 score of 0.804. For the textual entailment task, in which systems had to predict whether a hypothesis about either a single clinical trial or pair of clinical trials is true or false, we fine-tuned the BioLinkBERT transformer model. We passed our evidence retrieval model’s output into our textual entailment model and submitted its output for the evaluation. Our best performing model achieved an F1 score of 0.695.

heading: Future Work
text: In future work we’d like to include sequences that exceed the token limit - perhaps by applying the model in a strided fashion and averaging its outputs, or using a model with a longer attention span. While the sequential pipeline model is simple and effective, a single model - trained to perform both tasks - would be more efficient and possibly more robust to mislabelled evidence data as the model could learn to identify relevant evidence using the textual entailment signal in addition to the relevant line indices. We plan to investigate this in future work. References Hilda Bastian, Paul Glasziou, and Iain Chalmers. 2010. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLOS Medicine, 7(9):1–6. Catherine Blake and Ana Lucic. 2015. Automatic endpoint detection to support the systematic review process. Journal of Biomedical Informatics, 56:42–56. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Jay DeYoung, Eric Lehman, Benjamin E. Nye, Iain James Marshall, and Byron C. Wallace. 2020. Evidence inference 2. 0: More data, better models. CoRR, abs/2005. 04177. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. CoRR, abs/2007. 15779. Mael Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, Donal Landers, and André Freitas. 2023. Semeval-2023 task 7: Multi-evidence natural language inference for clinical trial data. In Proceedings of the 17th International Workshop on Semantic Evaluation. Svetlana Kiritchenko, Berry de Bruijn, Simona Carini, Joel Martin, and Ida Sim. 2010. Exact: Automatic extraction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10:56. Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. CoRR, abs/1904. 01606. Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711. 05101. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 197–207, Melbourne, Australia. Association for Computational Linguistics. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912. 01703. Rodney L. Summerscales, Shlomo Argamon, Shangda Bai, Jordan Hupert, and Alan Schwartz. 2011. Automatic summarization of results from clinical trials. In 2011 IEEE International Conference on Bioinformatics and Biomedicine, pages 372–377. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910. 03771. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links