Year: 2023

Abstract:
The “Causal Medical Claim Identification and 2 Extraction from Social Media Posts” task at SemEval 3 2023 competition focuses on identifying and validating 4 medical claims in English, by posing two subtasks on 5 causal claim identification and PIO (Population, 6 Intervention, Outcome) frame extraction. In the context 7 of SemEval, we present a method for sentence 8 classification in four categories (claim, experience, 9 experience_based_claim or a question) based on 10 BioBERT model with a MLP layer. The website from 11 which the dataset was gathered, Reddit, is a social news 12 and content discussion site. The evaluation results show 13 the effectiveness of the solution of this study (83.68%). 14

heading: 1 Introduction 15
text: One of the most brainstorming tasks facing natural 16 language processing (NLP) is the information 17 extraction (Khurana, D. et al., 2023; Landosi, M.Y. 18 et al., 2023) with important applicability in the 19 medical field (Zhang, T. et al., 2021). In fact, an 20 essential step for various medical decision-making 21 processes is the identification and automatic 22 verification of specific claims from unstructured 23 user-generated text data (Doppalaudi, S. et al., 24 2022). It refers to the idea of cause and effect. The 25 causal relation extraction has been mostly treated 26 as a subtask of relation extraction, being helpful in 27 the context of personalized healthcare (Khetan, V. 28 et al., 2022). Extracted causal information from 29 clinical notes, as a crucial step within medical 30
1 SemEval-2023 Task 8 overview is available at:
https://causalclaims.github.io/
decision-making, can and have to be combined 31 (Yang, J. et al., 2022). The aim of this paper is 32 focused on identification of causal claims1, 33 experience, etc., in a provided multi (or single) 34 sentence text snippet. The rest of the paper is 35 organized as follows: section 2 briefly presents 36 studies related to claim identification, section 3 37 provides information about the system designed to 38 classify text sentences, section 4 describes the 39 experimental setups. Section 5 resumes the results 40 of the conducted experiments, with their 41 interpretations, followed by section 6 with the 42 conclusions. 43
44
2 Background 45
This topic has attracted significant attention in 46 recent years, evidenced by increasing number of 47 workshops (e.g., Workshop on Curative Power of 48 MEdical Data - MEDA 2017; 2018; 2020, 49 Workshop on Events and Stories in the News 2018) 50 (Cohen, K.B et al., 2020., Gifu, D. et al., 2019, 51
Tommaso, C. et al., 2018). 52 Competitions such as SemEval-2023 Task 8: 53 Causal Medical Claim Identification and 54 Extraction from Social Media Posts are attractive, 55 especially since the problem of labeled data is 56 somewhat solved, since the automatic 57 identification and automatic verification of medical 58 claims depends on them. 59
913
In order to automatically identify and validate the 60 four main classes (claim, personal experience, 61 question and claim by personal experience), in text 62 snippets extracted from the Reddit platform (this 63 representing the first task within the competition), 64 we researched similar studies. In fact, there are 65 many approaches based on NLP techniques trying 66 to solve this (Sumner, P. et al., 2014, Mausam, 67 2016), while there is still a considerable number of 68 challenges along the way. 69 Our solution implies the usage of BioBERT 70 (Bidirectional Encoder Representations from 71 Transformers for Biomedical Text Mining). In fact, 72 it is a BERT-based model pre-trained on large-scale 73 biomedical corpora which outperforms BERT on 74 biomedical named entity recognition (0.62% F1 75 score improvement), biomedical relation extraction 76 (2.80% F1 score improvement) and biomedical 77 question answering (12.24% MRR improvement). 78 Starting from an existing model-generating code 79 base2 (Yu, B. et al., 2019), changes3 have been 80 brought in the label naming and implementations 81 have been added in the direction of preprocessing 82 and postprocessing of the datasets of interest. 83 SemEval-2023 Task 8 implies that for a provided 84 user-generated English-language text, it is 85 requested to identify the span of text that is either a 86 claim, personal experience, personal experience-87 based claim, or a question (Figure 1.1, Figure 1.2). 88 The dataset that this competition provided us 89 with consists of 7121 Reddit posts. As input, the 90 post IDs and specific annotations are given, to be 91 further used in constructing the final datasets. 92
93
Social media Post Claim(s) Personal
Experience(
s)

heading: 2 BioBERT model-generating code base is available at:
text: https://github.com/junwang4/causal-
language-use-in-science 3 The solution code repository is available at: https://github.com/dinosaph/SPLN_Tog edemaru_Semeval_Task_8
Figure 1.1 – Causal claim identification - The given 94 input (social media post as a text snippet) and the 95 requested classes to be identified for the first subtask: 96 claim(s), personal experience(s) (…) 97
Question(s) Claim(s) based on
Personal Experience(s)
Figure 2.2 – Causal claim identification - question(s), 98 claim(s) based on personal experience(s) (cont.) 99
An extracting script4 shared by the task’s 100 organizers was used to obtain a Reddit collection of 101 around 5696 posts (for training) and 1425 posts (for 102 testing) which, after preprocessing and splitting by 103 annotations, implied around 21652 sentences (for 104 training) and 13567 sentences (for testing). The 105 initial labels have been changed into integers in the 106 interest of an easier manipulation. (0 – “claim”, 1 – 107 “per_exp”, 2 – “question”, 3 – “claim_per_exp”). 108 As output, the predicted labels for a test dataset, 109 consisting of sentences, were shared between the 110 words of the text snippet of interest (Figure 2.3 - 111 each word in the sentence occupying a different 112 row, but sharing the same label as the whole 113 sentence, as required by the task organizers). 114
115
0 post_id, subreddit_id, stage1_labels
s1jpia,t5_2s23e,"[{""crowd-entity-
annotation"":{""entities"":[{""endOffset"":858," "label"":""per_exp"",""startOffset"":661},{""en dOffset"":2213,""label"":""per_exp"",""startOff
set"":1861}, …}}
1 post_id, subreddit_id, stage1_labels, text
(Same as ‘0’, but followed by the post text
itself)
2 sentence, label
It seems like so long ago now,1
but one morning I woke up and my left side wasnt responding as fast as my right side,1
It felt heavy,1
4 The script used to extract the Reddit posts: https://drive.google.com/file/d/10D5 VKvdKcIJvtC47vE7IcQQl_2f9qvG4/view
Causal research can help improve existing 116 medical decision-making processes. There are 117 many studies focused on the causal relation 118 extraction, leaving a nice starting point for further 119 training, and testing of CSC tasks (Causal Sentence 120 Classification; Tan, F.A. et al., 2021, Yu, B. Y et al., 121 2019) that provides a good base for BioBERT 122 embeddings fed through one layer of MLP, which 123 serves as a classifier. The solution described in this 124 paper also describes the results obtained by 125 choosing BioBERT as this model proved to 126 outperform BERT in several medical related tasks 127 which fits our task’s purpose. Yu B and colleagues 128 (Yu, B. Y et al., 2019) provided an open-source 129 code that was used as a starting point in this 130 development. It was required to also set up a 131 sklearn5 related repository dealing with imbalanced 132 classes for BERT. (Pedregosa et al., 2011) 133
134
3 System overview 135
The strategy picked for this task represents a causal 136 approach to BERT (BioBERT + MLP (Figure 3) as 137 found in the architecture (Tan, F. A et al., 2021). 138 The starting point in the development of this 139 solution was the model-generating code made 140 public by Yu, B., Li, Y. and Wang, J. (2019), on 141 Github. The only significant changes were done to 142 the labels, from 0 – “none”, 1 – “causal”, 2 – 143 “cond”, 3 – “corr” to 0 – “claim”, 1 – “per_exp”, 2 144 – “question”, 3 – “claim_per_exp”. 145
The datasets were constructed from the given 150 collection of post ids and annotations by automatic 151 extraction from the Reddit platform, preprocessed 152 and further used for training the BioBERT + MLP 153 model. The training process took around 2.5 hours 154 and it implied a combination of 5 folds and 5 155 epochs. 156 The subtask provided an extraction script, 157 mentioned in the previous sections, to be used for 158 extracting the required training social posts/text 159 snippets. The given input files for training and 160 testing were only files describing the path from 161 which we had to extract the text by post id with 162 preset Reddit credentials, process that took around 163 3 hours for the posts required by the task of interest 164 (Figure 4). The purpose was to extract the text and 165 match the substrings by the labels specified by 166 ranges of “<startOffset>” and “<endOffset>” 167 (Figure 5). 168
169
The resulted texts still needed to pass through a 174 preprocessing class that would prepare them for the 175 model training (Figure 6). Further development of 176 the solution is shown in the implementation of such 177 class that modifies the input dataset in the preferred 178 way. The text has been split into sentences sharing 179 the initial text snippet label, the unnecessary 180 symbols and the empty spans were removed and 181 finally, the labels were converted into integers, 182 leaving a clean input dataset (Figure 7). 183
A main class called “CausalExtractor” has been 195 implemented which handles all model related 196 actions and loads the pretrained model by 197 initialization. This class provides a baseline 198 algorithm description, but also uses our chosen 199 BioBERT algorithm to perform the predictions on 200 the required test data (Figure 8). One of its 201 functions, “generate_submission_st1” further 202 controls the predicted output and generates the 203 required results format for the subtask (Figure 9). It 204 splits the sentences into words, each word sharing 205 the same label, while the labels are converted back 206 into the initial string format. 207
4 Experimental setups 208
The datasets used in the development of this 209 solution have gone through some changes, before 210 the training of the model and after the generation of 211 the predictions. The following tables describe the 212 training dataset going through different changes, 213 from its initial state to its preprocessed form: 214
INITIAL DATASET: 215
post_id, subreddit_id, stage1_labels
s1jpia,t5_2s23e,"[{""crowd-entity-
annotation"":{""entities"":[{""endO ffset"":858,""label"":""per_exp""," "startOffset"":661},{""endOffset"": 2213,""label"":""per_exp"",""startO ffset"":1861},{""endOffset"":2407," "label"":""per_exp"",""startOffset" ":2255},{""endOffset"":3254,""label "":""claim_per_exp"",""startOffset" ":2697},{""endOffset"":3620,""label "":""claim_per_exp"",""startOffset" ":3294},{""endOffset"":3751,""label "":""claim_per_exp"",""startOffset" ":3621},{""endOffset"":4480,""label "":""per_exp"",""startOffset"":3752 },{""endOffset"":4759,""label"":""q
uestion"",""startOffset"":4482}]}}]
"
(…)
216
INPUT DATASET AFTER EXTRACTION: 217
post_id, subreddit_id, stage1_labels, text
s1jpia,t5_2s23e,"[{""crowd-entity-
annotation"":{""entities"":[{""endOffse t"":858,""label"":""per_exp"",""startOf fset"":661},{""endOffset"":2213,""label "":""per_exp"",""startOffset"":1861},{" "endOffset"":2407,""label"":""per_exp"" ,""startOffset"":2255},{""endOffset"":3 254,""label"":""claim_per_exp"",""start Offset"":2697},{""endOffset"":3620,""la bel"":""claim_per_exp"",""startOffset"" :3294},{""endOffset"":3751,""label"":"" claim_per_exp"",""startOffset"":3621},{ ""endOffset"":4480,""label"":""per_exp" ",""startOffset"":3752},{""endOffset"": 4759,""label"":""question"",""startOffs
et"":4482}]}}]","De-Nial
“I wrote this a few years ago and just
found it again. I thought I'd share...
&#x200B;
When I was 17, it was a very good year. Like the
opening line of the old Frank Sanatra song, when I was 17, it truly was a very good year. I was getting ready to graduate high school, I had deeply bonded
friends, I loved where I lived, and I had amazing
parents. (…)”
218
PREPROCESSED DATASET: 219
6 Sklearn dummy classifier details available at:
The provided code repository contains a class 221 called “PreprocessExpert” that deals with the 222 preprocessing of both training and testing datasets. 223 Each function in this class serves the purpose of 224 cleaning the texts, organizing the data in such a way 225 that aids the training or the testing. The training 226 data is split by label ranges (“<endOffset> - 227 <startOffset>”, such as in Figure 10), then split by 228 sentence and finally filtered (keeping only 229 sentences with more than 3 tokens/words) and 230 cleaned (removing messy symbols, empty spaces). 231 Finally, this dataset’s columns consist of only 232 “sentence” and “label”. 233
Similar cleaning and splitting processes take 234 place for the testing dataset, but this time the 235 columns of interest remain “post_id”, 236 “subreddit_id” and “text” (Figure 11). 237
The main run of the solution is executed through 238 Python by calling the main script, “main.py”, 239 which prepares the datasets (train and test), creates 240 a “CausalExtractor” instance (previously described 241 in Section 3) and generates scores and predictions 242 as requested (Figure 12). 243
5 Results 244
In order to explore the efficacy of the chosen 245 solution, tests have been run on the input data 246 within a sklearn dummy classifier6 through 247 multiple modes which returned the accuracies 248 shown in Figure 13. 249
250
learn.org/stable/modules/generated/s klearn.dummy.DummyClassifier.html
By comparing the scores of our BioBERT model 255 with the dummy classifier, the chosen model is a 256 good classifier for the given data (Figure 14). 257
Due to choosing a model outperforming Bert in 261 the medical tasks, the solution’s model performs 262 better than the classic baseline methods for the 263 testing data, though improvements can be brought 264 in the preprocessing stage and in studying the 265 impact of specific tokens in the final classification. 266 We managed to create a light solution that fulfills 267 the need of automatically identifying and 268 validating medical claims in social media posts. By 269 studying the results, we can see that the best 270 performing combination of k-fold on 5 epochs was 271 reached at the 3rd epoch with an F1 accuracy score 272 of around 0.899 (Figure 15). 273
BERT follows an interesting technique and due 274 to its complexity, we might need to further review 275 the way in which we have preprocessed the text and 276 the edits made on the base code for BioBERT, as 277 the results are not 100% correct and we are aware 278 of the need of their improvement. The fulfillment 279 of the next subtask is what interests us in the future. 280 We will be posting updates on the task’s repository 281 on which we will reorganize our thoughts and ideas 282 into a better development of the results for both 283 subtasks. 284
6 Conclusion 285
For performing automatic span identification from 286 textual documents (a Reddit collection) that is 287 either a claim, experience, experience-based claim, 288 or a question in an unstructured user-generated 289 English text, we developed a BioBERT model with 290 a MLP layer. While the solution for the subtask of 291 interest received the 7th place out of 7 entries, the 292 results show that it generated better scores than the 293 classic baseline methods, so it provides a good start 294 in this study, while being light and flexible for 295 improvement. In future work, we would like to 296 develop new approaches based on current rule-297 based systems with rules tailored to the linguistic 298 features associated with medical claim. 299
300
301
References 302
Khurana, D., Koli, A., Khatter, K, Singh, S. 2023. 303 Natural language processing: state of the art, current 304 trends and challenges. Multimed Tools Appl 82, pages 305 3713–3744. https://doi.org/10.1007/s11042-022-13428-306 4. 307
Landolsi, M.Y., Hlaoua, L. & Ben Romdhane, L. 2023. 308 Information extraction from electronic medical 309 documents: state of the art and future research directions. 310 Knowl Inf Syst 65, pages 463–516. 311 https://doi.org/10.1007/s10115-022-01779-1. 312
Zhang, T., Lin, H., Tadesse, M.M. et al. Chinese medical 313 relation extraction based on multi-hop self-attention 314 mechanism. Int. J. Mach. Learn. & Cyber. 12, 355–363 315 (2021). https://doi.org/10.1007/s13042-020-01176-6. 316
Doppalapudi, S., Wang, T. and Qiu, R. 2022. 317 Transforming unstructured digital clinical notes for 318 improved health literacy, Digital Transformation and 319 Society, Vol. 1 No. 1, pages 9-28. 320 https://doi.org/10.1108/DTS-05-2022-0013. 321
Khetan, V., Rizvi, M.I., Huber, J., Bartusiak, P., 322 Sacaleanu, B., and Fano, A. 2022. MIMICause: 323 Representation and automatic extraction of causal 324 relation types from clinical notes. In Findings of the 325 Association for Computational Linguistics: ACL 2022, 326 pages 764–773, Dublin, Ireland. Association for 327 Computational Linguistics. 328
Yang, J., Wan, Y., Ni, Q., Zuo, J., Wang, J., Zhang, X., 329 and Zhou, L. 2022. Quantifying causal effects from 330 observed data using quasi-intervention. BMC Med 331 Inform Decis Mak 22, 337 (2022). 332 https://doi.org/10.1186/s12911-022-02086-z. 333
Cohen, K.B., Gîfu, D., Li, Y., Ripple, A. and Xia, J., 334 2020, August. MEDA 2020: The curative power of 335 medical data. In Proceedings of the ACM/IEEE Joint 336 Conference on Digital Libraries in 2020 (pp. 575-576). 337
Gîfu, D., Trandabăț, D., Cohen, K. and Xia, J., 2019. 338 Special Issue on the Curative Power of Medical 339 Data. Data, 4(2), p.85. 340
Tommaso, C., Miller, B., van Erp, M., Vossen, P., 341 Palmer, M., Hovy, E., Mitamura, T., Caswell, D., 342 Brown, S. W., Bonial, C. 2018. Proceedings of the 343 Workshop Events and Stories in the News 2018, 344 Association for Computational Linguistics, New 345 Mexico, US. 346
Sumner, P., Vivian-Griffiths, S., Boivin, J., Williams, A., 347 Venetis, C. A., Davies, A., Ogden, J., Whelan, L., 348 Hughes, B., Dalton, B., Boy, F., Chambers, C. D. 2014. 349 The association between exaggeration in health-related 350
science news and academic press releas-es: retrospective 351 observational study. BMJ, 349, g7015. 352
Tan, F. A., Hazarika, D., Ng, S.-K., Poria, S. and 353 Zimmermann, R. 2021. Causal Augmentation for Causal 354 Sentence Classification. Proceedings of CI+NLP: First 355 Workshop on Causal Inference and NLP, ACL, pages 1–356 20. https://doi.org/10.18653/v1/2021.cinlp-1.1. 357
Yu, B., Li, Y., and Wang, J. 2019. Detecting Causal 358 Language Use in Science Findings, Proceedings of the 359 2019 Conference on Empirical Methods in Natural 360 Language Processing, and the 9th International Joint 361 Conference on Natural Language Processing, ACL, 362 pages 4664–4674. https://doi.org/10.18653/v1/D19-363 1473. 364
Pedregosa et al. 2011. Scikit-learn: Machine Learning 365
in Python, JMLR 12, pp. 2825-2830 366
A. Appendices 367
368
375
Figure 15 – Chosen solution accuracies.
Figure 16 – Task submission results sample

heading: Future Work
text: The fulfillment 279 of the next subtask is what interests us in the future. 280 We will be posting updates on the task’s repository 281 on which we will reorganize our thoughts and ideas 282 into a better development of the results for both 283 subtasks