Clinical Contradiction Detection
Authors: Dave Makhervaks
Plia Gillis
Kira Radinsky
Venue: EMNLP 2023 Main (2023)
Keywords: clinical contradiction detection
medical ontologies
clinical distant supervision
TL;DR: Clinical contradiction detection benefits from using distant supervision paired with a medical ontology.

Abstract: Detecting contradictions in text is essential in determining the validity of the literature and sources that we consume. Medical corpora are riddled with conflicting statements. This is due to the large throughput of new studies and the difficulty in replicating experiments, such as clinical trials. 
Detecting contradictions in this domain is hard since it requires clinical expertise. 
We present a distant supervision approach that leverages a medical ontology to build a seed of potential clinical contradictions over 22 million medical abstracts. 
  We automatically build a labeled training dataset consisting of paired clinical sentences that are grounded in an ontology and represent potential medical contradiction. The dataset is used to weakly-supervise state-of-the-art deep learning models showing significant empirical improvements across multiple medical contradiction datasets.

Clinical Contradiction Detection
Dave Makhervaks
Plia Gillis
Kira Radinsky
Technion Israel Institute of Technology
Tel Aviv University
Technion
Tel Aviv, Israel stiggillis@maan com
Israel Institute of Technology
Haifa, Israel davemêcs. www.settchhhinoccc ac..onn.cc.....
Haifa, Israel
il
Abstract
clinical research found that 16% of established in- terventions had their outcome refuted (Ioannidis, 2005). Extending this to PubMed, over million articles would disagree with a previous finding.
Detecting contradictions in text is essential in determining the validity of the literature and sources that we consume. Medical corpora are riddled with conflicting statements. This is due to the large throughput of new studies and the difficulty replicating experiments, such as clinical trials. Detecting contradictions in this domain is hard since it requires clinical exper- tise. We present distant supervision approach that leverages medical ontology to build seed of potential clinical contradictions over 22 million medical abstracts. We automatically build labeled training dataset consisting of paired clinical sentences that are grounded in ontology and represent potential medical contradiction. The dataset is used to weakly- supervise state-of-the-ar deep learning models showing significant empirical improvements across multiple medical contradiction datasets.
The problem of contradiction detection has been studied in the task of natural language inference (NLI) sentence level. NLI aims to determine whether a pair of sentences tailing, or neutral. Deep learning approaches reach impressive results for this task. Specifically, large language models (LLMs) such as DeBERTaV3 (He et al., 2021) and BioELECTRA (raj Kanakarajan et al., 2021), are considered the state-of-the-art (SOTA) for NLI. However, in medical research, de- tecting a contradiction is more difficult. Sometimes more context is needed to detect contradiction due to the difficulty of the material. Consider the exam- ple below (both interventions are ACE inhibitors): "However, in the valsartan group, significant improvements left ventricular hypertro- phy and mieroalbuminuria were observed." 2. "Although bedtime dose of doxazosin can significantly lower the blood pressure, it can also increase left ventricular diameter, thus increasing the risk of congestive heart fail- ure."
contradictory,
1 Introduction
Determining whether pair of statements is contra- dictory foundational fields including science, politics, and economics. Detecting that statements contradict sheds light on fundamental issues. For instance, mammography is an integral routine in modern cancer risk detection, but there is contro- versy regarding if certain mammography patterns indicate cancer risk (Boyd et al., 1984). Rec- ognizing that a certain topic has opposing view- points, signifies that the issue may deserve further investigation. Medicine an interesting domain for contradiction detection, as it is rapidly devel- oping, of high impact, and requires an in-depth understanding of the text. Per the National Li- brary of Medicine, the PubMed (Canese and Weis, 2013) database averaged 900k citations for the years 2018-2021, with a growing trajectory (med, 2022). The publication of contradictory papers is not uncommon in scientific research, as it is part the process of validating or refuting hypotheses and advancing knowledge. A study on high impact
Detecting that this pair contradicts requires knowing that improvements in left ventricular hy- pertrophy is a positive outcome, whereas an in- crease [in] left ventricular diameter is a negative outcome in the context of heart failure.
Deep learning methods for NLI require large datasets (Conneau al., 2017). However, few datasets exist train such algorithms in the clinical contradiction domain. Time and cost of labeling complex medical corpora, could be potential rea- son for this. The MedNLI dataset (Romanov and Shivade. 2018) for instance, required the expert labeling of clinicians over the course of 6 weeks Yet, MedNLI is fabricated since each of the clin-
MIMIC-III certified users can find the McdNLI dataset

---

icians was given clinical description of a patient and came up with contradicting, entailing, and neutral sentence to pair up with that description. this work, we are interested in naturally occur- ring sentences in clinical literature as opposed to manually curated texts. We focus on sentences con- taining clinical outcomes and attempt to identify whether outcomes contradict. We will show that even this step, which does not address the inter- vention and population, yields an initial system for identifying medical contradictions.
models when fine-tuned on the ontology-driven datasct. We make our code publicly available 2.
2 Related Work
NLI primarily focuses on textual entailment, start- ing with the RTE challenges proposed by Dagan et al. (2013) and Dagan et al. (2005). The task in- volves determining if one sentence can be inferred from another. Over time, data and classification criteria were introduced, including the labeling of contradictions in the third challenge (Giampiccolo et al.. 2007). However, the medical domain brings additional challenges requiring clinical expertise.
One approach to overcome the lack of large data is distant supervision (Mintz et al., 2009). Distant supervision uses existing knowledge sources to au- tomatically label large amount of data. The labels can be noisy, so the goal is train robust models that learn meaningful patterns. We propose a novel methodology leveraging distant supervision and a clinical ontology, the Systematized Nomenclature of Medicine Clinical Terms (SNOMED) (Stearns et al., 2001). SNOMED is developed by a large and diverse group of medical experts (Donnelly et al.. 2006) and it contains extensive information about clinical terms and their relations. Our method uses knowledge extracted from SNOMED to classify pairs of "naturally occurring", potentially contra- dictory sentences. PubMed's abstract database is our source for naturally occurring sentences.
Despite the complexity of medical literature and the reality of contradictions in publications, there is surprisingly little work in this area. Large NLI corpora contain relatively easy contradiction pairs, partly due to the cost of annotating complex con- tradictions. The contradiction is often a negation through words like 'not'. An example from a large NLI corpus, MultiNLI (Williams et al., 2017) is: 1. "Met my first girlfriend that way."
2. "I didn't meet my first girlfriend until later." Romanov and Shivade (2018) 8) introduce MedNLI, a curated medical NI.I dataset, and experiment with baselines, finding InferSent (Conneau et al., 2017) to be the best model. They explore retrofitting (Faruqui et al., 2014) and knowledge-directte atten- tion but observe worse results with retrofitting and minimal improvement with mnoweedge-directer at- tention (0.2%)0.3%) compared to their InferSent baseline. In contrast, our approach differs by not using retrofitting or ontology term distances. We also keep the attention mechanism in the models unchanged. Instead, we utilize ontological fea- tures specific to clinical findings (Section 3.1) to construct fine-tuning dataset. We include their methodology as bascline in Table 3.
We fine-tune SOTA deep learning models on the aforementioned ontology-driven created dataset, We test the results using manually labeled clinical contradiction datasets. The results demonstrate that our suwrrattssperviiiiooooo based methodology yields statistically significant improvements of the models for contradiction detection. The average results of 9 different models see an improvement on our main evaluation set (Section 1.1.1) over previous SOTA. Specifically, we find that the improvement is consistent across both small models and those that are considered to be SOTA on task,,,, which is the closest task to that of contradiction detection.
Scientific fact-checking is a related task, where a claim is verified against evidence (Wadden et al., 2020; Sarrouti et al., 2021). The work in this field deals with popular claims justified by evidence from a corpus, as opposed to naturally occurring sentences in medical literature. Kotonya and Toni (2020)'s data comes from popular media sources such as the Associated Press and Reuters News.
The contribution of our work is threcfold: (1) We present the novel problem of contradiction analysis of naturally occurring sentences in clinical data. (2) We create clinical contradiction dataset by using distant supervision over clinical ontology, yield- ing improvements of SOTA deep learning models when fine-tuning on it. (3) We empirically evalu- ate on numerous manually labeled clinical contra- diction datasets and show improvements of SOTA
Alamri and Stevenson (2016) developed a la- beled dataset, ManConCorpus, for contradictory cardiovascular research claims in abstracts. This
'nttps:/ http:///g......cccco/////aaaaaa. medical-come
here: http:://hhiiinettmngoooomemmmmmm

---

corpus has pairings between fabricated query and a naturally occurring claim, thus not dealing with naturally occurring sentences. It is annotated by domain experts. There are works which address contradiction of a clinical query and a claim. Taw- fik and Spruit (2018) use hand-crafted features to build classifier. whereas Yazi et al. (2021) use deep neural networks (see baseline in Table 3). Un- like their approaches, classify any sentence-pair representing a clinical outcome. To our knowledge, no work evaluates contradiction detection between naturally occurring sentences in clinical literature. Semantic predications from SemMedDB (Kil- icoglu al., 2012) are commonly used in clinical contradictions. Alamri (2016) uses B construct AutoConCorpus by querying subject- predicate-object tuples. Predicates are 'incompati- ble' if they belong to different groups. The dataset is in the same query-claim format as ManConCor- pus and the 'retricval' stage is manual. Rosemblat et al. (2019) also use SemMedDB by finding rele- vant semantic predications with their corresponding opposing predicate pairs. Unlike these approaches, we do not use predicate logic. In addition, our dataset creation fully automated. We show its efficacy by fine-tuning models on it, which is not done by either of the SemMedDB datasets.
cusing on ontology parts related clinical find- ings, which represent the outcomes of clinical ob- servations (Rory, 2023). Unlike mutually exclusive nouns like 'cat' or 'dog' the terms we encounter in this context may have conflicting clinical findings, suggesting difference in clinical outcome.
3.1 SNOMED CT Ontology
SNOMED an ontology containing over 350,000 clinical terms (Stearns et al., 2001). It has informa- tion about a plethora health concepts, containing attributes such as relationships to other terms and interpretations. The structure of SNOMED allows grouping terms based on their relationships. We hy- pothesize that this structure coupled with synonyms and antonyms, enables to create corpora of po- tentially contradicting and non-contradicting clini- cal terms. We use the 2022 edition of SNOMED.
3.1.1 SNOMED Node Attributes
Each term in the SNOMED ontology is node in tree-like structure designating a clinical finding. These findings have attributes used in determining their relationships. Each node belongs parented by the group root. In addition, they have a simple interpretation which is a defined attribute within the ontology. In Figure 1, the group consists of nodes describing the group root cardiac oulput. The green (right) node has the interpretation in- creased. Groupings of terms with these attributes have logical connection, resulting in pairings of contradicting and non-contradicting phrases within the group context. Determining the relationship be- tween a pair of terms done partially through com- paring their interpretations. In Figure the nodes have the interpretations decreased and increased respectively. We assign the pair an attribute label (A:g) of contradiction where and j are SNOMED terms. In Algorithm 1, Aij is assigned on Line 12. Group size can get large. For instance, the group root Cardiia function has 275 children. Larger groups may contain child terms that are less closely related compared to smaller groupings. For exam- ple aortic valve regurgitation due to dissection and dynamic subaortic stenosis both pertain to impair- ments of cardiac function, but it would be unfair to consider them related outcomes. Though these large groupings yield many pairings, they may also lead to less accurate ones. Section 5.2 investigates the effects of group sizes.
group
Mintz et al. (2009) introduced distant super- vision, which was later extended to knowledge bases like YAGO by Nguyen and Moschitti (2011) 1). Neural networks became popular for distantly- supervised relation extraction (Zeng et al., 2015; Zhang et al., 2019). We apply distant supervision to detect contradictions between sentence pairs representing clinical outcomes. We use weak su- pervision during fine-tuming with SOTA models and leverage the relational knowledge of a clinical ontology. Unlike common approaches (Smirnova and Cudré-Mauroue 2018; Purver and Battersby, 2012), we infer contradictions using the structure and attributes of a clinical ontology instead of rely- ing on known relation labels. Our approach yields positive and negative term relations, distinguish- ing it from traditional distant supervision models (Smirnova and Cudré-Mauroux, 2018). To our knowledge,
first use distant supervision for detecting contradictions in the clinical domain.
3 Methods
Our method creates a dataset for training or fine- tuning using ontology knowledge, specifically fo-
Below are pairings contradictions in various medical domains that our methodology yields:

---

suppressed urine secretion ++ polyuria
is contradiction. The same holds for St.- We define the logic in Lines 15 through 17. We reach 79% aceuracy through this heuristic on the human- labeled SNOMED term-pairs with a Cohen's kappa coefficient of 0.853 for inter-annotator agreement.
clevation of SaO2 + oxygen saturation within
reference range
joint stable + chronic instability of joint
Algorithm 1 SNOMED Traversal
1: function TRAVERSE(rOOt)
3.2 Ontology-Driven Distant Supervision
2: for rool. rool.children do
Using the relational knowledge extracted from SNOMED, we weakly-supervise naturally occur- ring sentences in PubMed to build our SNOMED dataset. We fine-tune on this dataset to achieve significant improvements over existing baselines. Algorithm summarizes the procedure. We search PubMed for sentences containing the phrase-pairs discussed in Section 3.1, resulting in corpus of pairs of sentences, The sentence-pairs are then la- beled through distant supervision as explained be- low. For given pair of SNOMED terms (p1, p2), we label sentences ($1, s2) as formalized in Eq.1, where label E
3: if n.num childs S group size pairs <- DET.RELATION(11)
4:
5: end if
6: end for
7: return pairs
8: end function
9: function DET RELATION(())
pairs t 0
for G, Cj E.child_pairs do
12:
Ar,j + GET.AITR.LABELCC 0j)
13:
Sig <- GEN (C,
{ {contradiction, non-contradiction)
14:
labeli,j + Aij
15:
if St,j contra or Arj contra
(pi E (() A (p2 s2) A ((P1.2) E label) (1) This methodology focuses outcomes, since the SNOMED terms we use are clinical findings. Although there is chance that there may be dif- fering interventions or participants, we show that even with focusing on outcomes we improve per- formance on our evaluation sets. Concretely, ps may be a subset of 8;, so there may be information loss (statistics on sizes of 8; are reported in Table 2). Given that the purpose of the SNOMED dataset is to increase the amount of training or fine-tuning data of model, we find that this introduced noisc is acceptable and still yields positive results.
16:
labeli, < contra end if
17:
pairs + pairs U {labelli Ci Cy) end for return pairs
20:
21: end function
22: SNOMED - TRAVERSE(rOot)
23: (OCTUUE
3.1.2 Synonyms
After leveraging structure, we examine synonyms and antonyms, which offer a strong signal in an ontology. Grouped clinical terms share context, thereby allowing the use of simpler indicators to de- termine their relationship. We word-tokenize each clinical phrase, removing the intersection of the sets of tokens, leaving each with its unique tokens. We assign a SNOMED pair - í.j a synonym label (S),)). A visualization is found in Appendix B.1.
Group Rost
Cardiac output
Constrictive pericarditis
Increased cardiac output
Children
Interpretations
Decreased
contradiciin
Increased
Figure I: The group with Cardiac output as its root. The children depicted have contradicting interpretations.
3.1.3 Combining Attributes and Synonyms
To combine Ai,j and Sij to form final label, we build validation set of the publicly available SNOMED term-pairs. Two annotators with domain knowledge label 149 SNOMED phrase-pairs. 70 contradictory and 79 non-contradictory. More de- tails are in Appendix A.1. We find that when Ar,j indicates contradiction, then it's likely that labela.
3.3 Filtration
Naively, we pair all sentences satisfying Lgq. 1, even if they do not share context. Yet, two sentences containing matching clinical SNOMED terms, may be unrelated. The example below exhibits this:

---

1. "The present results suggest that the upstream changes in blood flow are transmitted by the velocity pulse faster than by the pressure pulse in the microvasculature."
Table 1: Cardiology Dataset Breakdown
Split
Total
Contra
Non-Contra
Train
1347
571
776
2. "His chest wall was tender and his pulse slow but the remainder of his physical examination was normal."
Dev
198
100
98
Test
227
55
172
The bolded clinical terms are central to the meaning of the sentences and are contradictory their own. However, when placed in context they may be less relevant each other in the example above. We stricter criteria for filtering sentence matches, namely using MeSH (Medical Subject Headings) terms (Lipscomb, 2000) and cosine similarity.
Empirical Evaluation
4.1 Evaluation Datasets
4.1.1 Cardiology Dataset
To assess the quality of the SNOMED dataset, we tweak ManConCorpus. The corpus consists of question-claim pairs. Each question has opposing 'yes', 'no' claims. The claims naturally occur in PubMed and the questions are generated by experts. We convert the corpus by pairing up the sentence claims from PubMed. pair contradicts if con- stituent claims answer the question differently. We coin this dataset as Cardio (Table 1).
McSH terms categorize articles within PubMed. We hypothesize that sentences drawn from articles with related McSH terms likely discuss the same topic. Eq. 2 is our formulation for filtering via MeSH terms. McSH; and MeSHi are the sets of MeSH terms for articles containing sent, and sent, respectively. Let t be chosen threshold.
4.1.2 Hard Cardiology Dataset
if (MSSHjuMMSH,] min([MeSH.), McSH,) 2
We observe that models tend to classify sentences as contradictory if negations appear. For example:
1
(2)
0
otherwise
"Our results indicate that atorvastatin therapy significantly improves BP control in hyperlipi- demic hypertensive patients."
Using MeSH terms is not perfect. The example below achieves a score of 0.4 per Eq. 2.
1. "In dogs challenged with endotoxin, the inhi- bition of nitric oxide production decreased cardiac index and did not improve survival."
2. "Administration of a statin in hypertensive pa- tients in whom blood pressure is effectively re- duced by concomitant antihypertensive treat- ment does not have an additional blood pres- sure lowering effect."
2. "Intra-aortic balloon pumping increased car- diac index and aortic distensibility by 24% and 30%, respectively, and reduced myocar- dial oxygen demand by 31% (P < .001 for all alterations)."
We construct a version of Cardio through removing negation words. This version removes model re- liance on such words to make a correct prediction. We coin this dataset as Hard-Cardio.
Despite overlap in MeSH terms, they are very dif- ferent one discusses dogs and the other humans.
4.1.3 MedNLI Datasets
The second filtration method measures the co- sine similarity between one-hot vectors. Topically related sentences should have a higher one-hot vec- tor cosine similarity. Let o, and oy be the respective one-hot vectors of sent; and senty. Vector lengths are equal to the number of unique words spanning the sentence-pair. We compute the similarities be- tween the vectors as shown in Eq. 3. The dog example above, yields similarity score of 0.2.
Inspired by SNLI (Bowman al., 2015), MedNLI was created with focus the clinical domain (Romanov and Shivade, 2018). It was curated for 6 weeks, borrowing the time of doctors. MedNLI consists sentence- pairs which are grouped into triples a contradictory, entailing, and neutral pair. The sentences are not naturally occurring in exist- ing medical literature. The premise is shared across the three pairs, but each have different hypothesis, yielding different label. Since MedNLI deals with 3-class problem, we relabel the dataset by making fentailment, neutral} map to non-contradicctoon. Our focus is to show that the SNOMED dataset is as powerful as the MedNLI dataset, without the
{
if cosine(o,, o,) 2
1
otherwise
We experiment with t, ultimately choosing t 0.35 based on an external validation set.

---

need for manual intervention. The baseline on the relabeled version of MedNLI gives high results (Appendix G), so adding additional data makes lit- tle change. The largest labeled datasets containing naturally occurring sentences are at most hundreds of sentences. Thus, we sample 100 instances from MedNLI's train-split and report results on that. To explore fields outside of cardiology, we cre- ate versions of MedNLI focused on gynecology (GYN), endocrinology (ENDO), obstetrics (OB), and Surgery. The annotators introduced Sec- tion 3.1.3 filter the data. We sample from the train- split in the same fashion as explained above. These datasets also have 2-class label structure as ex- plained in Section 4.1.3 (Appendix A.2 for details),
Table SNOMED Dataset
Sentence length:
NLTK token count
25.1
BioGPT token count
29.4
BioELECTRA token count
30.7
BERT Base token count
36.8
Total Dataset Statisties:
SNOMED term matches in PubMed
4.99M
Number of articles
2.87M
Number of qualifying pairs in SNOMED
0.63M
forward network. We use the same hyperparame- ters the authors replicate their results.
We evaluate an additional LLMs and compare their performance when they are fine-tuned on the SNOMED dataset versus without. The task of clas- sifying contradiction a subtask of NLI, thus we use models that top leaderboards for the MNLI and MedNLI datasets namely SeBERTaV3-Basee ALBERT (Lan et al., 2019), and BioELECTRA. ELECTRA (Clark et al., 2020) and BERT (Devlin et al., 2018) are also included as they are perfor- mant architectures. Additionally, success on small models important, as they require less comput- ing resources and may allow the SNOMED dataset to have a stronger influence during fine-tuning. Thus, we include BERT-Smale (Turc et al., 2019), ELECTRA-Small, and DeBERTaV3-Smmn et al., 2021). Finally, include BioGPT (Luo et al., 2022) for completeness, it has decoder and also pre-trained on biological data. Table 3 breaks down the number of parameters per model.
4.2 SNOMED Dataset Analysis
Table 2 presents statistics of the SNOMED dataset used for weak supervision, including the number of analyzed articles and the sentence counts in PubMed containing SNOMED ontology terms. See Appendix D for qualitative examples of our dataset.
4.2.1 Phrase Matching Noise
The proposed matching introduces noise when p; is not central to the meaning of 8;. To approximate this noise, we sample 100 sentences from the SNOMED dataset. A human annotator was asked to evaluate p: contributes to the central message of 8;- We observe 91 % accuracy.
(He
4.2.2 SNOMED Labeling Noise
The LLMs have the same high-level architecture, so we use the functionalities of HuggingFFace (Wolf et al., 2019) and the Sentence-Tramee Transformer library (Reimers and Gurevych, 2019). We pass sentence- pairs input into the network and add binary classification head the model body. Hyperparam- eters come from the Tranenncce-Taaafommrr library. except for training batch size- for models above 30M parameters and 16 for models under 30M pa- rameters. Each model is tuned with the SNOMED dataset. It uses group size of 25, sampling 10 sentence-pait painn from PubMed for every SNOMED term-pair. These hyperparameters are determined through ablation tests the Cardio validation set.
The SNOMED dataset may contain noise. We sam- ple 100 instances from it and manually label the sentence-pains paires similar to Mintz al. (2009). The annotators label each pair as containing contradic- tory or con-contradictory elements. The gold label is compared to the distantly-supervised label. We extract positive and negative relations from our ontology, thus reporting accuracy to indicate the method's effectiveness. We achieve 82% accuracy, surpassing previous noise analyses (Mintz al., 2009), possibly due rich ontology information.
4.3 Baseline Models
Romanov and Shivade (201 8) use InferSent with knowledge-directed attention to achieve their top results on MedNLI. Yazi et al. (2021) achieve the SOTA on the ManConCorpus, which we turn into the Cardio corpus (Section 4. 1.). They con- catenate BERT embeddings for their question and claim, feeding the input into a Siamese-lik feed
Empirical Results
5.1 Main Result
Table 3 summarizes our findings. Base denotes fine-tuning a model only on the training split the corresponding evaluation dataset. Ours de-

---

Table 3: Performance Models tuned with SNOMED vs. Without
Algorihm (Number of Params)
Ditesct
Method
ALBERT
ELECTRA
BERT Small (28.8M)
ELECTRA
BERT
Bio- ELECTRA
DeBERTss Small (141.9M)
DeBERTa
Bio-
(Yazi
(Romanov
Base (11.7M)
Small (13.M)
Base (109.5M)
Base (109.5M)
Base (184.4M)
GPT (3468M)
st
al.
and Shis- sck: 2018)
(109.5M)
2021)
Cardio
Base
0.911
0,877
0,888
0.863
0.914
0.890
0.885
0.861
0.858
0.858
0.824
Ours
0.928
0.947
0.958
0.892
0.878
0.925
0.931
0.942
0.930
Hard-
Rase
0.785
0.717
0.850
0.842
0.845
0.762
0.687
0.688
Cardio
Ours
0.925
0,853*
0.794
0,773
0.791
0.925
0.917
0.936'
0
MedNLI
Base
0.541
0.516
0.492
0.537
0.661
0.539
0.551
0.565
0.725
0.529
0.643
General
Ours
0.682
0.636
0.605*
0.754
0.759
0.779
0.695
0.774
0.800*
M-
Base
0,749
0.542
0.553
0.000
0.759
0.597
0.589
0.672
0.840
0.557
0.738
Cardio
Ours
0.785v
0.964
M
0.533
0.558
0.592
0.508
GYN
Ours
0.608
0.617
0.767
0.758
0.792
0,808
0,683
0.825
0.783
M.dNLI
Base
0.09%
0.525
0.567
0.584
0.639
0.601
0.522
0.601
0.840
0.560
0.707
ENDO
Ours
0.860
0.690
0.725
0.793v
0.867
0.860
0.852*
0.883
0.878
M
Base
0.532
0.502
0,513
0.505
0.557
0.549
0.502
0.579
0.507
0.505
0.538
0.616
0.542
0.702
0.618
0.740
L Surgery
Hase
0,708
0.502
0.555
0.681
0.842
0.609
0.576
0.6991
0.925
0.602
0.898
Ours
0.892
0.668
0,807
0.912
0,903
0.925
0.808
0.884
0.940
notes fine-tuning on both the SNOMED dataset and the training split of the corresponding evaluation dataset. Both Base and Ours are evaluated on the same test sets, We measure the area under the ROC curve of each baseline and verify statistical signifi- cance through Delong's test (DeLong et al., 1988). Significant differences marked with an asterisk Across most mataset-model combinations, the weak supervision of the SNOMED dataset reaches superior results compared to fine-tuning only on the original dataset. The result holds for Romanov and Shivade (2018)'s InferSent model and Yazi et al. (2021)'s SOTA model (Appendix F).
5.2
Ablation Studies
5.2.1 Group and Sentence Samples Size
We explain SNOMED term grouping in Section 3.1 and illustrate in Figure 1. Group size and pairing quality may be closely related. Larger groupings tend have more terms which are less related each other as explained in Section 3. 1.1.1. Thus, we experiment with SNOMED datasets based on terms belonging to groups of at most 6, 12, 25, and 50.
During dataset creation, we choose how many sentence-paid pairss to sample per SNOMED pairing. In Figure 2, each line with different color/marker represents different number of samples averaged across the models. The ablations we perform in- clude 10, 25, and 50 samples per pairing.
Cardio and Hard-Cardio are difficult datasets of potentially contradicting, naturally occurring pairs in PubMed. The performance on Hard-Cardio drops relative to Cardio as expected. This veri- fies our hypothesis that removing negations makes the problem more difficuul.. Fine-tuning the SNOMED dataset improves the baselines for 8 out of models we evaluate for both datasets.
Figure shows 10 samples outperforms higher sampling numbers for most group sizes. Increased sampling results in over-saturation of certain term- pairs. The best group size is 25 for small models and 12 for large models. These numbers strike the balance of a large amount of SNOMED phrase- pairings and accurate relationships (as discussed in Section 3.1.1). Smaller models may benefit more from larger group sizes, because they have more limited base knowledge than those of large models.
On synthetically created common datasets, such MedNLI, our methodology improves over all baselines for this corpus. Improvements are also consistent for sub- speccaliis when fine-tuning on SNOMED. This shows the scalability of our meth- ods for clinical contradiction detection through dif- ferent fields within healthcare.
5.2.2 Filtering Based on Similarity
To enhance the relevance of sampled sentences. we explore the use of high McSH term or cosine simi- larity for filtering (Section 3.2). Figure shows the relation between the filtration methods. Continuing the ablation analysis from Figure 2, we set the num-
Further, see a trend where small models may be more affected by fine-tuning on SNOMED. All the evaluation datasets improve over the baseline on every model under 30 million parameters.

---

100 ctal RomStor & SiioOd sasapiic
Small Model Large Models
5.3.2 Similar Articles
To find similar articles, use the PubMed Related Articles metric (Lin and Wilbur, 2007). Castro et al. (2015) find that this is an effective metric for semantic simalrity between articles when compared to others like BM25 (Robertson et al., 1980).
Seriutes
10 Stmipees
ROC
0.00
5.3.3 Finding Contradictions
0.92
We sample 850 abstracts. 51 are flagged as poten- tially contradictory by BERT-Small model fine- tuned the SNOMED dataset. After filtering sen- tences without annotator-identiffed outcomes, we have 24 remaining. Among them, 9 are labeled contradictory by our annotators. 12 false positives are attributed to mismatch in intervention or ex- perimental design, despite contradictory outcomes. For example the abstract discussed different drug types, but the sentences had opposite findings of a patient's blood pressure. 3 were not contradictory. In future work we intend to improve our methodol- ogy by also focusing on the intervention component and making this application more robust. We attach examples of contradictory abstracts in Appendix E.
0.82
comp
Figure 2: Small and large model performance across group sizes and sample numbers. Reported on Cardio.
ber of samples at 10 and the group size at 25, The cosine similarity outperforms both no-filtering and McSH. Although McSH terms are usefuul, they are tagged on the article-level, thereby not providing the same topic granularity as one-hot vectors.
Conclusions
091
BLOCTRIA
B ENGLECTRA
Contradiction detection is central to many fields, but is especially important in medicine due to hu- man impact and rapid clinical research growth. Though contradictions are a subfield of NLI, there is less work in the clinical domain. Medical con- tradictions require context and domain knowledge, making them complex. Labeling datasets which produce effective deep learning models is costly. We introduce a novel method of using clinical ontology to weakly-superve tentially contradicting dataset with naturally occur- ring sentences. We coin it the SNOMED dataset. Empirical results suggest that fine-tuning on the SNOMED dataset results in consistent improve- ment across SOTA models over diverse evaluation datasets spanning multiple medical specialties. We find balance between term group size and the number of PubMed sentence samples per pairing. Additionally, better results achieved by filtering the PubMed sentences included in our dataset.
0.00
0.99
1009
MA2H
Figure 3: Performance across filtration methods. Number of samples is 10 and group size is 25, Reported on Cardio.
the creation of po-
5.3 Qualitative Application on Real Abstracts
In an extrinsic experiment, we aim to show the value of our method in more realistic clinical set- ting. Thus far, evaluated our methodology the sentence level. We now present preliminary system see if our method can extract contradic- tory abstracts. Results are evaluated by clinicians.
For future exploration we suggest investigating more robust sentence filtration methods, such as topic modeling or sentence embedding similarity. Looking into how other clinical ontologies can be paired with SNOMED may also be fruitful.
5.3.1 Representative Sentences
Our models are trained on sentence-level data, so we find a sentence representative of the abstract. This is done automatically by finding the sentence with the largest cosine similarity to the title.

---

Limitations
Abdulaziz Alamri. 2016. The detection contradic- tory claims in biomedical abstracts. Ph.D. thesis, University of Sheffield.
We believe in the novelty of our work and the im- pact it may provide, but there are some limitations. This methodology is limited to SNOMED terms, many of which do not appear within PubMed. Due the evolving nature of knowledge bases, termi- nology and information changes, potentially alter- ing relations between terms. In addition, the struc- ture we extract from the clinical ontology is not ground-truth, yielding noise during dataset creation as discussed in Section 4.2.
Abdulaziz Alamri and Mark Stevenson. 2016. A corpus of potentially contradictory research claims from car- diovascular research abstracts. Journal of hiomedical semantics, 7(1):1-9.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 550.00536.
Norman F Boyd, Brian O' O'Sullivan, Eve Fishell, Imre Simor, and Gabriel Cooke. 1984. Mammographic patterns and breast cancer risk: methodologic stan- dards and contradictory results. Journal of the Na- tional Cancer Institute, 1256):25331129.
In the creation of the SNOMED dataset we fo- contradicting outcomes. Incorporating in- tervention and participant understanding into our methodology is important and we leave this for future work. The final qualitative application on real abstracts (Section 5.3) a preliminary sys- tem which can be improved in finding the most representative sentence of an abstract and only in- cluding those sentences with outcomes reflecting the main finding. Our system is also limited to single sentences which is constraint, especially when wanting to compare full abstracts.
Kathi Canese and Sarah Weis. 2013. Pubmed: the bibli- ographic database. The NCBI handbook, 2(1).
Leyla Jacl Garcia Castro, Rafael Berlanga, and Alexan- der Garcia. 2015. In the pursuit of a semantic simi- larity metric based on umls annotations for articles in pubmed central open access. Journal of biomedical informatics, 57:204-218.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555.
Ethical Considerations
Whenever working within the clinical domain, eth- ical considerations are crucial. The data that work with is all rooted in already publicly available corpora and PubMed. To the best of our knowledge the data we use does not contain any personal in- formation of any humans involved in clinical trials. There is potential risk of over representing com- mon diseases and outcomes in our dataset, thereby not including enough data about other outcomes. If our approach were be implemented in real medical environment, there could also be repercus- sions. Due to the imperfect nature the methodology, any label predicted by our model would need be used as a guide for leading the researcher or doc- tor, as opposed to the ground truth. An example scenario may be choosing to use drug 1 to treat disease B since model does not find any con- tradictory evidence regarding the effectiveness of drug A. However, the decision of the physician should take into account that the model may not always be accurate.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint 1005v:1705.02:55
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop. pages 177-190. Springer.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Mas- simo Zanzotto. 2013. Recognizing textual entail- ment: Models and applications. Synthesis Lectures on Human Language Téchnologies, 6(4):1-220.
Elizaheth R DeL.nng, David M Del.ngg, and Daniel I. Clarke-Pearsonn 1988. Comparing the areas under two more correlated receiver operating character- istic curves: a nonparametric approach. Biometrics. pages 837-845.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1100v.1810.04805.
References
Kevin Donnelly et al. 2006. Snomed-ct: The advanced terminology and coding system for ehealth. Studies in health technology and informatics, :21:
2022. Medline@ citation counts by year of publication (as of january 2022).

---

Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2014. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv: 141::::111... 1.666
Matthew Purver and Stuart Battersby. 2012. Experi- menting with distant supervision for emotion classift- cation. In Proceedings of the 13th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages 482-491.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9.
Kamal raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. 2021. Bioelectra: pre- trained biomedical text encoder using discriminators. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 143-154.
Pengcheng He. Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with diaffentdddsettagleeddd embedding shar- ing. arxiv preprint arXiv:2111.0954
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese nett-networks. arXiv preprint prXiv:9908 10084.
John PA loannidis. 2005. Contradicted and initially stronger effects in highly cited clinical research. Jama, 294(2)218-22)))
Stephen Robertson, Cornelis van Rijshergen, and Martin F Porter. 1980. Probabilistic models of in- dexing and searching. In SIGIR, volume 80, pages 35-56.
Halil Kilicoglu, Dongwook Shin, Marcelo Fiszman, Graciela Rosemblat, and Thomas C Rindflesch. 2012. Semmeddb: a pubmed-scale repository biomedical semantic predications. Bioinformatics, 28(23):3158- 3160.
Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference the clini- cal domain. arXiv preprint arXiv: 06752.
Davidson Rory. 2023. Snomed et editorial guide.
Neema Kotonya and Francesca Toni, 2020. Explain- able automated fact-checking for public health claims. arXiv preprint arXiv:2010.09926.
Graciela Rosemblat, Marcelo Fiszman, Dongwook Shin, and Halil Kilicoglu. 2019. Towards a characteriza- tion of apparent contradictions in the biomedical lit- erature using context analysis. Journal of hiomedical informatics, 98:103275.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv:1909.1 11942.
Mourad Sarrouti, Asma Ben Abacha, Yassine M'rabet, Comner-Fushhaan Fushman. 2021. Evidence-basee fact-che of health helatee claims. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 3499-3512.
and Dina Demner-
Jimmy Lin and W John Wilbur. 2007. Pubmed related articles: probabilistic basee model for content similarity, BMC bioinformatics, 8(1):1-14.
Alisa Smirnova and Philippe Cudré-Maurone 2018. Re- lation extraction using distant supervision: A survey. ACM Computing Surveys (CSUR), 51(5):1-35.
Carolyn E. Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265.
Michael Q Stearns, Colin Price, Kent A Spackman, and Amy Y Wang. 2001. Snomed clinical terms: overview of the development process and project sta- tus. In Proceedings the AMIA Symposium, page 662, American Medical Informatics Association.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang. Hoifung Poon. and Tie Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6).
Noha S Tawfik and Marco Spruit. 2018. Automated contradiction detection biomedical literature. International Conference on Machine Learning and Data Mining in Patlern Recognition, pages 138-148. Springer.
Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf- sky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings the Joint Con- ference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003- 1011.
Iulia Ture. Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint (00Xx::998.08000000
Truc- Vien T Nguyen and Alessandro Moschitti. 2011. End-to- sod--to-nd relation extraction using distant supervi- sion from external semantic repositories. In Proceed- ings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 277-282.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact fiction: Verifying scientific claims. arXiv preprint arXiv:2004.1410 14974.

---

Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2017. A coveraee challenge corpus for sentence understanding through inference. arXiv
B Additional Methodology Details
B.1 Synonym Extraction
preprint arX:::1104.0555
Synonym extraction is part of our methodology which is explained in Section 3.2. Figure 4 pro- vides a depiction of this for the clinical terms short- ened p wave and prolonged p wave. The respective unique tokens are shortened and prolonged. Since the unique tokens antonyms, the synonym label for the pair is a contradiction. In Algorithm 1, the synonsm label (St.) is assigned on Line 13. Sim- ilarly, if the respective tokens are synonyms, then S.g would be a non-contradiction.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019, Huggingface's transformers: State-of- the-art natural language processing, arXiv preprint arXiv:1910.03771
Fatin Syafiqah Yazi, Wan-Tze Vong, Valliappan Ra- man, Patrick Hang Hui Then, and Mukulraj Lunia. 2021. Towards automated detection contradictory research claims in medical literature using deep learn- ing approach. In 2021 Fifth International Conference on Information Retrieval and Knowledge Manage- ment (CAMP), pages 116-121. IEFE.
wave feature
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceed- ings of the 2015 conference empirical methods in natural language processing, pages 1753-1762.
Shortened P wayé
...
Prolonged PWayé
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen. 2019. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. arXiv preprint arXiv:1903.01.00
shortened
antonyms
prolonged
Figure 4: The terms shortened p wave and prolonged wave are simplified to just shortened and prolonged after their common words are removed. The remaining words antonyms.
A Annotation
As mentioned in Section 3.1.3, we work with an- notators with domain knowledge. The annotators were used due to their expertise in the field.
C Additional Dataset Details
We include some additional details to the break- down of the evaluation datasets. In particular, re- garding the token lengths of the datasets. Ta- ble we include the sentence length breakdown according to various tokenizers of the SNOMED dataset. In the appendix we also include the break down of the Cardio dataset well the McdNLI dataset (Tables 4 and 5 respectively). Although our SNOMED dataset and Cardio dataset contain roughly the same number of tokens per sentence, the MedNLI dataset contains roughly half the num- ber of tokens per sentence. This may serve as an indicator to the decreased difficulty of MedNLI as well as evidence that that sentences are not natu- rally occurring.
A.1 SNOMED Term-Pairs
The annotators labeled 149 SNOMED term-pairs as either contradictory or non-contradietory. They were provided with list of pairs, without any ad- ditional information about the ontological structure they came from. This was done in order to preserve fairness and integrity during the labeling process. The instructions were come up with a binary label for each of the pairs.
A.2 Filtering MedNLI
A human annotator also helped with coming up with list of sub-words which served as indicators for particular fields of medicine. For example, the sub-words vulv and gyno, are indicative of gyne- cology. These word lists were used to create the variations of MedNLI discussed in Section 4. 1.3. You can find the lists of words in the code that is released with the paper.
D SNOMED Dataset Examples
We include several randomly sampled examples from the SNOMED dataset. The data is also pub- licly available.

---

Table 4: Cardio Dataset Additional Details
Sentence 1: hearing loss tinnitus hyper- acusis and difficulty hearing in noise re- main persistent and in some cases pro- gressive complaints for patients.
Sentence length:
NLTK token count
26.7
BioGPT token count 30.6
BieELECTRA token count
31.8
Sentence 2: chief complaints were long- standing localized pain and hearing diffi- culty.
BERT- Base token count
37.2
Table 5: MedNLI Additional Details
E
Wild Contradictory Abstracts
Sentence length:
The numbers are the PMID of the articles. More ex- amples can be found in the data dircctory attached with the paper submission. Example
NLTK token count
13.2
BioGPT token count
16.3
LECTRA token count
17.1
BERT- Basse token count
19.1
Abstract 27045229: This meta analy-snalysis investigated the effects of preoperative prophylactic intraaortic balloon pump placement postoperative renal func- tion and short-term death of high-risk pa- tients undergoing coronary artery bypass grafting. We found that preoperative pro- phylactic intraaortic balloon pump sup- port reduced the incidence of coronary artery bypass grafting-associinn kidney injury and short-term death and dramatically decreased the incidence of postoperative renal replacement therapy by 82% compared with high-risk patients without the procedure. This is the first meta-analysis demonstrate significant beneficial effects preoperative prophy- lactic intraaortic balloon pump on renal function in high-risk patients undergoing coronary artery bypass grafting.
D.1 Contradiction Examples
Example:
the plasma cck and luminal content of lerf were measured by spe- cific rddioimmunnassyyybiil--pppaannee juice diversion significantly increased pancreatic secretion plasma cck and lcrf levels.
Sentence 1:
acute
Sentence 2: blockade of the cek receptor results in decreased pancreatic secretion and atrophy.
Example:
Sentence although the mutant does not swim still is able to move and perform photobehavior.
Abstract 29863415: Background Urgent or emergency coronary artery bypass grafting in patients with acute coronary syndrome is associated with increased morbidity and mortality. We investi- gated the effects of preoperative intraaor- tic balloon pump support in this high- risk patient cohort. Methods Our in- stitutional database was retrospectively reviewed for patients with acute coro- nary syndrome and an urgent emer- gency indication for coronary artery by- pass from April 2010 to December 2016. Data of 1066 patients were analyzed. We assessed the impact of preopera- tive intraaortic balloon pump therapy on postoperative mortality and major ad- verse cardiovascular and cerebrovascular
Sentence 2: whereas the chev mutants still produced both types of flagella and were able swim and swarm.
D.2 Non-Contradiction Examples Example
Sentence 1: computed tomography (ct) seans showed bilateral contracted kid- neys with mass projecting from the lower pole of the right kidney.
Sentence 2: ultrasonography and com- puted tomography revealed a masslike expansion involving the upper polc of an otherwise small right kidney.
Example

---

events, and performed propensity-score matching. Results Intraaortic balloon pump support was implemented 223 (20.9%) patients: 55 (5.2%) preopera- tively and 168 (15.8%) intra- or post- operatively. Overall hospital mortality was 8.8%. Patients with preoperative intraaortic balloon pump had increased mortality (11/55, 20%) compared con- trols (p= 0.006). After propensity-scor matching, all-cause mortality (20.0% vs. 18.2%, = 0.834), cardiac mortality (18.2% vs. 14.5%, = 0.651), and major adverse cardiovascular and cerebrovascu- lar events (29.1% vs. 27.3%, p 0.855) were comparable between groups. Con- clusions Preoperative intraaortic balloon pump support does confer any addi- tional clinical benefit on patients under- going coronary artery bypass grafting for acute coronary syndrome.
tant. Among the mosaic of assumed pres- sor mechanisms, distinct Na+ retention almost invariably involved diabetes mellitus, while sympathetic activation tends to occur in essential hypertension, particularly in association with obesity. Insulin resistance may develop con- sequence of intracellular excess of Ca2+ decrease in Mg2+, impaired iesulin-mediated rise skeletal muscle blood flow, increased sympathetic activ- ity or excess body weight. Acute hyper- insulinemia causes arterial vasodilation on one hand and increases sympathetic activity and renal Na+ reabsorption the other. Chronically, hyperinsulinemia may promote cardiovascular muscle cell proliferation and atherogenesis, while in- sulin resistance may be associated with certain transmembraneous cation trans- porters, leading to an increase in cytoso- lic Ca2+. Hyperinsulinemia and/or in- sulin resistance may also be associated with an increased blood pressure sensi- tivity high salt intake. In the mosaic of many different blood pressure-raising mechanisms, insulin resistance and/or hyperinsulinemia is likely represent an amplifying slow or very slow pressor factor.
Example
Abstract 8576790: GENETIC PREDIS- POSITION: Insulin resistance and re- active hyperinsulinemia occur not only with obesity, impaired glucose tolerance or non-insulin-dependent (type 2) dia- betes mellitus, but also in many non- obese, non-diabetic patients with essen- tial hypertension and their currently nor- motensive, lean, young offspring, as well as in some other conditions known promote hypertension. Insulin resistance impairs glucose tolerance, while insulin resistance and/or hyperinsulinemia pro- mote dyslipidemia, body fat deposition and probably atherogenesis. Therefore, the common coexistence of genetic pre- disposition for hypertension with insulin resistance helps to explain the frequent, although temporally often dissociated, occurrence of hypertension together with dyslipidemia, obesity and type 2 diabetes in given patient. INSULIN RESIS- TANCE AND HYPERINSULINEMII AS SLOW PRESSOR MECHANISMS: In the pathogenesis of hypertension, inap- propriate vasoconstriction (due an im- balance of vasoactive substances and/or raised cytosolic calcium) and/or struc- tural vasculopathy is particularly impor-
Abstract 7700881: Resistance to the metabolic effects of insulin and compen- satory hyperinsulinemia have been pos- tulated to mediate human essential hy- pertension, especially when associated with obesity. Evidence supporting this hypothesis has come mainly from epi- demiological studies showing correla- tions between insulin resistance, hyper- insulinemia, and blood pressure, and from short-term studies suggesting that insulin has renal and sympathetic effects that could raise blood pressure the ef- fects were sustained. However, there have been no studies demonstrating di- rect causal relationship between chronic hypertension and insulin resistance hy- perinsulinemia humans. The few long- term studies that have been conducted dogs and humans do not support the hypothesis that hyperinsulinemia causes

---

hypertension or potentiates the hyperten- sive effects of other pressor agents such as angiotensin or increased adrenergic tone. To the contrary, multiple studies dogs and in humans suggest that the va- sodilator action of insulin tends to reduce blood pressure. Although resistance to insulin's metabolic effects has been sug- gested be essential for hyperinsuline- mia cause hypertension, chronic in- creases in plasma insulin concentrations do not cause hypertension in dogs or hu- mans even in the presence of insulin re- sistance. Also, recent studies have also shown that the blood pressune-lowering effects of antihyperglycemie agents, ini- tially believed lower blood pressure by decreasing insulin resistance, may be unrelated to their effects on insulin sensi- tivity. Obesity appears to be a key factor in accounting for correlations between insulin resistance, hyperinsulinemia, and hypertension, but increased blood pres- sure in obesity does not appear to be me- diated by insulin resistance and hyperin- sulinemia. Although insulin resistance and hyperinsulinemia may not be directly linked hypertension, there increas- ing evidence that metabolic abnormal- ities associated with insulin resistance may increase the risk of cardiovascular disease (e.g., coronary artery disease) as- sociated with hypertension and Type II diabetes. For this reason, further stud- ies of the long-term effects of insulin resistance on cardiovascular, renal, and metabolic functions are needed.
We developed strategy to identify in- leracting candidate genes/loci for CRC risk that utilizes both linkage and RNA- seq data from mouse models combina- tion with allele-specific imbalance (ASI) studies in human tumors. We applied our strategy to three previously identi- fied CRC susceptibility loci in the mouse that show evidence of genetic interaction: Scc4, Sce5 and Sec13. 525 SNPs from genes showing differential expression in the mouse and/or a previous role in can- cer from the literature were evaluated for allele-specific imbalance in 194 paired human normal/tumor DNAs from CIN- related CRCs. One hundred three SNPs showing suggestive evidence of ASI (31 variants with uncorrected p values 0.05)
were genotyped in validation of 296
paired DNAs. Two variants in SNX10 (SCC13) showed significant evidence of allelic selection after multiple compar- isons testing. Future studies will evaluate the role of these variants in combination with interacting genetic partners in colon cancer risk in mouse and humans.
Abstract 21314996:
Common single- nucleotide polymorphisms (SNPs) in ten chromosomal loci have been shown to predispose to colorectal cancer (CRC) in genome-wide association studies. A plausible biological mechanism CRC susceptibility associated with genetic variation has so far only been proposed for three loci, each pointing to vari- ants that affect gene expression through distant regulatory elements. In this study, we aimed to gain insight into the molecular basis of seven low-penetrance CRC loci tagged by rs4779584 at 15q13, 1810795668 at 10p14, rs3802842 at 11q23, rs4444235 at 14q22, 1s9929218 at 16q22, rs1041121 1210 at 19q13, and rs961253 20p12.
Example
Abstract 25973956: Colorectal cancer (CRC) can be classified into different types. Chromosomal instable (CIN) colon cancers are thought to be the most common type of colon cancer. The risk of developing CIN-related CRC is due in part to inherited risk factors. Genome- wide association studies have yielded over 40 single nucleotide polymorphisms (SNPs) associated with CRC risk, but these only account for subset of risk alleles. Some of this missing heritabil- ity may be due to gene-gene interactions.
Possible somatic gain of the risk allele loss of the protective allele stud- ied by analyzing allelic imbalance in tu- mour and corresponding normal tissue samples of heterozygous patients. Fune- tional variants were searched from sil- ico predicted enhancer elements locat-

---

ing inside the CRC-associating linkage- disequilibrium regions.
No allelic imbalance targeting the SNPs was observed at any of the seven loci. Altogether, 12 SNPs that predicted to disrupt potential transcription factor binding sequences were genotyped in the same population-based case-control se- ries as the seven tagging SNPs originally. None showed association with CRC.
The results of the allelic imbalance anal- ysis suggest that the seven CRC risk vari- ants are somatically selected for in the neoplastic progression. The bioinfor- matic approach was unable to pinpoint cancer-causing variants at any of the seven loci. While it is possible that many of the predisposition loci for CRC are involved in control of gene expression by targeting transcription factor binding sites, also other possibilities, such as reg- ulatory RNAs, should be considered.
F SNOMED Tuning Baselines
We decide to also fine-tune the baselines of Ro- manov and Shivade (2018) and Yazi et al. (2021) on the SNOMED dataset. Table 6 summarizes these results. Almost all mataset-model combina- tions improve with the addition of the SNOMED dataset. This result consistent with our main results.
Full MedNLI Dataset
For completeness, we also report results on the full MedNLI dataset. Results be see in Table 7. Notably, there are statistically significant results. Although fine-tuning with the SNOMED dataset yields better results in majority of the models, we see that results are not statistically significant in any of the cases. Therefore, we hypothesize that there is over-saturation which occurs at this stage.

---

Table 6: Performance Models tuned with SNOMED vs. Without
Algorithm
Dataset
Method
(Yazi
(Romanov and Shiv-
al.
2021)
ade,
2018)
Cardio
Base
0.858
0.824
Ours
0.917
0.881
Hard-
Buse
0.687
0.688
Cardio
Ours
0.728
MedNLI-
Base
0.529
0.643
General
Ours
0.643
0.659
MedNLI-
Base
0.557
0.738
Cardio
Ours
0.686
0.738
MedNLI-
Base
0,508
0.708
GYN
Ours
0.617
0.775
MedNLI-
Base
0.560
0.707
ENDO
Ours
0.771
0.748
MedNLI
Base
0.505
0.538
OB
Ours
0.595
0.605
ModNLI
Base
0.602
0.898
Surgery
Ours
0.895
0.855
Table 7: Performance of Models tuned with SNOMED Without
Algorithm Parami)
ALBERT
ELECTRA
ELECTRA
DeBERTs Small (141.9M)
DeBERTa
CYazi
Base
Small (13,5M)
Base (109.5M)
Base (109.5M)
ELECTRA (109.50
Base (184-4M)
GPT (346.8M)
al.
and Shiv-
(11.7M)
(28.8))
2021)
ade.
2018)
McdNLI
Base
0.946
0994
0.936
0.962
0.951
0.973
0.968
0.977
0.962
0.934
0.930


Reviews:
[{"review_id": "YXrxY1IesP", "reviewer": "Reviewer_ufkc", "summary": "", "questions": "* L373: Why not use medical semantic textual similarity methods instead of one-hot vectors with cosine? \n* L408: What is the added scientific value of the Hard-Cardio dataset? It seems some portions of the Appendix might be more valuable for the space taken.\n* Why not also evaluate with fine-tuning on only naturally-occurring samples (i.e. the SNOMED dataset)?\n* Why not annotate and reserve enough of the SNOMED dataset to do testing on it?\n* L559: What exactly does \"samples averaged across the models\" mean?\n* L596: Why not use medical semantic textual similarity methods instead of the PubMed Related Articles metric?\n* L599: simalrity -> similarity", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your meaningful feedback!\n> L373: Why not use medical semantic textual similarity methods instead of one-hot vectors with cosine?\n\nWe did some initial testing with medical semantic textual similarity, but empirically did not find a significant difference with the more naive and quickly computable one-hot vector cosine methodology. We will add this experiment to the appendix.\n\n> L408: What is the added scientific value of the Hard-Cardio dataset? It seems some portions of the Appendix might be more valuable for the space taken.\n\nThe added scientific value of the Hard-Cardio dataset is that it is a semantically equivalent dataset to Cardio, but we empirically found that models perform worse on it due to the removal of negation phrases. We address this in Section 4.1.2, but will emphasize the added scientific value. \n\n> Why not also evaluate with fine-tuning on only naturally-occurring samples (i.e. the SNOMED dataset)?\n\nWe are interested in showing the added value of fine-tuning on the SNOMED dataset. We thought that the best way to show this would be through reporting the relative improvement via additional fine-tuning on the dataset, as fine-tuning on the training set is a common practice reported in baseline papers (Alamri and Stevenson, 2016; Romanov and Shivade, 2018).\n\n> Why not annotate and reserve enough of the SNOMED dataset to do testing on it?\n\nThe primary reason for this was because we wanted to evaluate against existing datasets instead of introducing a new evaluation benchmark. Our primary goal in the creation of the SNOMED dataset was for training/fine-tuning as opposed to evaluation.\n\n> L559: What exactly does \"samples averaged across the models\" mean?\n\nThe wording here is a bit confusing. In Figure 2, there are lines for 10, 25, and 50 (green, blue, and red respectively) samples for a given group size. For instance, the blue solid line represents the average ROC AUC of all the small models. The blue dashed line represents the average ROC AUC of all the large models. Small models are defined to be those under 30 million parameters (L543-L546). We will clarify it in the final version.\n\n> L596: Why not use medical semantic textual similarity methods instead of the PubMed Related Articles metric?\n\nSince this was an extrinsic experiment, we decided to proceed with what was established as working well (Castro et al., 2015). This would be an interesting exploration for future work. It would be also interesting to see whether semantic textual similarity methods degrade as context increases.\n\n> All Typos, Grammar, Style And Presentation Improvements\n\nThe point about the title is a great one and we will consider it strongly. Thank you for catching all of the typos and other presentation issues, we will make sure to fix them in the next iteration of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper adds depth to the discussion on NLI by thoroughly considering the subtask of medical contradiction detection. It provides SotA results and has exemplary writing, meaningful comparison with established literature, and excellent evaluation.", "reasons_to_reject": "None.", "questions_to_authors": "* L373: Why not use medical semantic textual similarity methods instead of one-hot vectors with cosine? \n* L408: What is the added scientific value of the Hard-Cardio dataset? It seems some portions of the Appendix might be more valuable for the space taken.\n* Why not also evaluate with fine-tuning on only naturally-occurring samples (i.e. the SNOMED dataset)?\n* Why not annotate and reserve enough of the SNOMED dataset to do testing on it?\n* L559: What exactly does \"samples averaged across the models\" mean?\n* L596: Why not use medical semantic textual similarity methods instead of the PubMed Related Articles metric?\n* L599: simalrity -> similarity", "paper_topic_and_main_contributions": "This article introduces a dataset and weakly supervised method -- using domain knowledge through SNOMED -- for contradiction detection from naturally occurring statements in biomedical literature. Extensive evaluations showing SotA results are reported, including formative evaluations to justify design choices, dataset analyses, meaningful comparison with SotA methods and benchmark datasets, ablation studies, and broader evaluations. Overall, the writing is clear, the topic is well-aligned, and the methodology and experimentation sound.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Title: I would argue that this should be considered \"medical contradiction detection\" rather than \"clinical\" because, although it uses SNOMED (which has clinical care in view) for the weak supervision, the evaluation is done on biomedical literature. Statements in biomedical literature are general statements about a disease or treatment; statements in clinical text are statements about a patient. These sociolinguistic settings differ significantly. \"Medical\" was also used in MedNLI and is sufficiently broad to cover both types of texts/settings.\n* L058: \"Consider the highly domain-specialized example below\" rather than talking about ACE inhibitors\n* L521: Give the legend for the asterisk in the caption or just below Table 3.\n"}}, {"review_id": "ZUE65Z9ydp", "reviewer": "Reviewer_Kpwr", "summary": "", "questions": "- In Section, 3.1.3 and in Appendix A.1, it is specified that 149 phrase pairs from SNOMED were labeled by annotators? How were these 149 phrase pairs selected? And is there any justification for this specific number? \n- In Section 4.1.2, it is mentioned that negation words were removed. How was this performed? \n- In Section 4.2.2, it is mentioned that 100 instances were sampled. Can more details be provided about the sampling? Also, how many instances of each label were present in this sample set?\n- Can this method of distant supervision be applied towards building datasets for other healthcare-related NLP tasks (beyond contradiction)? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions, we hope to address them below!\n\n> In Section, 3.1.3 and in Appendix A.1, it is specified that 149 phrase pairs from SNOMED were labeled by annotators? How were these 149 phrase pairs selected? And is there any justification for this specific number?\n\nAs can be seen in Figure 2 and stated on Line 578, we set the group size to be 25 with 10 samples. To narrow down the phrase pairs, we focused on cardiology related phrase pairs, yielding 750 pairs. We decided to label 20% of these pairs, while filtering phrases which had numbers as part of their phrase. This left us with 149 phrase pairs to be labeled.\n\n> In Section 4.1.2, it is mentioned that negation words were removed. How was this performed?\n\nThis was performed manually by our annotators. The objective was to rephrase the sentence as to keep semantic meaning, while not using negation words.\n\n> In Section 4.2.2, it is mentioned that 100 instances were sampled. Can more details be provided about the sampling? Also, how many instances of each label were present in this sample set?\n\nThe instances were sampled entirely at random. 32 instances were non-contradictory and 68 were contradictory. We will make sure to add this information in the paper.\n\n> Can this method of distant supervision be applied towards building datasets for other healthcare-related NLP tasks (beyond contradiction)?\n\nOne potential application of distant supervision could be towards creating a medical NER dataset. Many of the terms within SNOMED are named entities which can be grouped by their parents’ roots for example. This would allow for a faster way to create NER datasets. In addition, since the nodes contain many attributes - such as synonyms or alternate labels - these could be used to create even larger datasets. For example the sentence “The patient had a heart attack” would have the word ‘heart attack’ distantly labeled as a named entity. Simultaneously, the sentence “The patient had a myocardial infarction” could be synthetically created through substituting the named entity with its synonym. We will add this in the conclusion when discussing future work.\n\n> Reproducibility, Typos, Grammar, Style, And Presentation Improvements\n\n The code is shared via the link at the bottom of Page 2. We realize that we were not able to upload all of the data due to the zip file size limit in the submission, we will make all data publicly available. Thank you for catching all of the typos and other presentation issues, we will make sure to fix them in the next iteration of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an innovative and relatively inexpensive method of data augmentation for detection of clinical contradiction by leveraging the existing SNOMED medical database\n\n- The paper is well-explained and evidence is provided for most assumptions and modeling decisions. ", "reasons_to_reject": "No reasons to reject but I do have a few questions on the analysis (as specified in the \"Questions For The Authors\" section)", "questions_to_authors": "- In Section, 3.1.3 and in Appendix A.1, it is specified that 149 phrase pairs from SNOMED were labeled by annotators? How were these 149 phrase pairs selected? And is there any justification for this specific number? \n- In Section 4.1.2, it is mentioned that negation words were removed. How was this performed? \n- In Section 4.2.2, it is mentioned that 100 instances were sampled. Can more details be provided about the sampling? Also, how many instances of each label were present in this sample set?\n- Can this method of distant supervision be applied towards building datasets for other healthcare-related NLP tasks (beyond contradiction)? ", "paper_topic_and_main_contributions": "- This paper introduces an approach of using publicly available clinical ontology towards creating a dataset of contradictions using sentences taken from medical abstracts.\n- In particular, this paper employs a distant supervision method that utilizes attributes of clinical terminology defined in SNOMED to create pairs of contradicting terms across different medical domains\n- Supervised fine-tuning of a number of transformer models (derivatives of BERT and GPT) augmented by the dataset created improves the SOTA performance on a number of labeled datasets for clinical contradiction. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "- Under section 2, \"Related Work\", the term RTE is not defined / introduced. "}, "extra_text": {}}, {"review_id": "lJm106YaGk", "reviewer": "Reviewer_G3pb", "summary": "", "questions": "Question A: Is the label at Algorithm 1 Line 13 manually assigned? If automated, how?\nQuestion B: Is Mintz et al’s 2009 work a valid comparison in the context of section 4.2.2? Please justify.\nQuestion C: Are you able to publicly share the raw annotations based on which the 91% and 82% accuracies are claimed in 4.2.1 and 4.2.2?\nQuestion D: Were the section 3.1.3’s 149 manually verified pairs the only ones used in training the NLI models? What is the difference between the 79% accuracy in 3.1.3 versus the 82% accuracy in in 4.2.2?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews and careful read of our paper, we hope to address your comments below!\n\n> Many of the proposed heuristics are crude (e.g., the example at line 940 does not clearly differentiate the subject “the mutant” vs “the chev mutants”), which cast doubt on the promising results in Table 3 with regard to the validity of the sentence-based evaluation. Determining contradictions in medical research is very context-sensitive, and even minor misalignment of the settings can make a detected pair useless. A more realistic test scenario is probably what is reported in section 5.3.3, with a precision of 0.375 (9/24).\n\nThank you for your points, we believe that they are valid concerns. Our research falls under the umbrella of distant supervision where noisier sources of supervision are used to create larger training sets quickly. Although they are noisy, when coupled during fine-tuning the results are promising, as seen in Table 3. Similar results of distant supervision using noisier data have produced similar trends in other fields (Purver and Battersby, 2012; Zeng et al., 2015).\nAlthough the experiment in 5.3.3 is an interesting one, in this work we focus on sentence-level contradiction and the main benchmarks in the medical NLI field. We include Section 5.3.3 to show the merit on broader tasks such as document contradiction. We are hopeful that with the success on the sentence-level which we presented in our paper, we will be able to more rigorously define and extend to the document-level task through additional research. \n\n> Is the label at Algorithm 1 Line 13 manually assigned? If automated, how? \n\nThe label at Algorithm 1 Line 13 is not manually assigned. The 2 phrases are composed of their respective words, but we ignore the words which are present in both phrases (i.e. we focus on the unique words). The label is assigned based on whether the unique words are synonyms. This check is done via WordNet, since most of the unique words are non-medical parts of speech like adjectives.\n\n> Is Mintz et al’s 2009 work a valid comparison in the context of section 4.2.2? Please justify. \n\nWe believe that Mintz et al.’s 2009 work is a valid comparison in the context of Section 4.2.2. In Section 7.2 of Mintz et al.’s work they explain that they manually evaluate 100 samples for a given relation. Similarly, we manually evaluate 100 samples for our contradiction relation. We also follow their evaluation criteria, by assigning “truth” or “falsehood” to our contradiction relationship.\n\n> Are you able to publicly share the raw annotations based on which the 91% and 82% accuracies are claimed in 4.2.1 and 4.2.2?\n\nYes, we will share those publicly with the rest of the data. Some example are included in Appendix D, and here are a couple more: \n\n*Correctly Labeled Contradiction Example*\n\n**“46 patients (66%) had difficulty talking and 36 (51%) reported difficulty eating.”**\n\n---\n\n**“only 4 (8%) of 49 patients reported mouth pain and all patients reported being able to eat well.”**\n\n*Correctly Labeled Non-Contradiction Example*\n\n**“we report the third case of persistent polyclonal b-cell lymphocytosis in a male.”**\n\n---\n\n**“he manifested a lymphocytic leukemoid reaction characterized by marked lymphocytosis and the presence of lymphoblasts in his circulation.”**\n\n> Were the section 3.1.3’s 149 manually verified pairs the only ones used in training the NLI models? What is the difference between the 79% accuracy in 3.1.3 versus the 82% accuracy in in 4.2.2?\n\nThe manually verified pairs in Section 3.1.3 are SNOMED phrase terms, where each phrase can be thought of as a node in the SNOMED graph. The 79% accuracy is a reflection of Algorithm 1’s performance. \nSection 4.2.2 on the other hand discusses the samples from our SNOMED dataset, where each instance is a pair of naturally-occurring sentences from PubMed. The 82% accuracy reported is a reflection of the quality of the SNOMED dataset.\nThe 149 manually verified phrases from Section 3.1.3 were not used in training any of the NLI models, but rather used in the creation of the SNOMED dataset. The SNOMED dataset was in turn used for fine-tuning the NLI models.\n\n\n> Reproducibility, Typos, Grammar, Style, And Presentation Improvements\n\n The code is shared via the link at the bottom of Page 2. We realize that we were not able to upload all of the data due to the zip file size limit in the submission, we will make all data publicly available. Thank you for the input on improving the readability of the paper, catching typos, and other presentation issues, we will make sure to fix them in the next iteration of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this is a well-chosen, meaningful challenge in terms of matching what the NLI technical framework could be conveniently adapted to tackle in the medical domain. Despite the underdeveloped methods, the genuine thought process and ambition to solve a hard problem are appreciated.", "reasons_to_reject": "Many of the proposed heuristics are crude (e.g., the example at line 940 does not clearly differentiate the subject “the mutant” vs “the chev mutants”), which cast doubt on the promising results in Table 3 with regard to the validity of the sentence-based evaluation. Determining contradictions in medical research is very context-sensitive, and even minor misalignment of the settings can make a detected pair useless. A more realistic test scenario is probably what is reported in section 5.3.3, with a precision of 0.375 (9/24).", "questions_to_authors": "Question A: Is the label at Algorithm 1 Line 13 manually assigned? If automated, how?\nQuestion B: Is Mintz et al’s 2009 work a valid comparison in the context of section 4.2.2? Please justify.\nQuestion C: Are you able to publicly share the raw annotations based on which the 91% and 82% accuracies are claimed in 4.2.1 and 4.2.2?\nQuestion D: Were the section 3.1.3’s 149 manually verified pairs the only ones used in training the NLI models? What is the difference between the 79% accuracy in 3.1.3 versus the 82% accuracy in in 4.2.2?", "paper_topic_and_main_contributions": "The study introduced a weakly-supervised approach by using the SNOMED-CT ontology to supplement the training data in developing deep learning classifiers that identify contradicting medical sentences. An antonym lexicon was created by leveraging the opposite SNOMED-CT interpretation values associated with each clinical finding concept and the concept terms under specified subtrees. The lexicon was then used with compositional heuristics to assign pairs of PubMed abstract sentences as contradictory or non-contradictory, which were used to train deep learning natural language inference (NLI) models and evaluated on several existing medical NLI datasets. Major improvements were observed with using these ontology-supplemented training data. Choosing proper subtree sampling scope and pre-denoising irrelevant sentence pairs were also found beneficial.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think the “contradiction” and “contradictory” are used too liberally where they probably refer to antonym and antonymous in some context; please revisit.\nThere are many scattered heuristics throughout the methods that also require referring back and forth with the appendix, making it very hard for the reader to track which is which in the end.\nEnglish needs editing."}}]

Author rebuttal:

Meta-review: Quality:
Pros with regards to quality are noted by two of the reviewers as follows:
-	“This paper adds depth to the discussion on NLI by thoroughly considering the subtask of medical contradiction detection. It provides SotA results and has exemplary writing, meaningful comparison with established literature, and excellent evaluation.”
-	“The paper is well-explained and evidence is provided for most assumptions and modeling decisions.”

Cons with regards to quality are noted by one reviewer as follows:
-	“Many of the proposed heuristics are crude (…),which cast doubt on the promising results in Table 3 with regard to the validity of the sentence-based evaluation. Determining contradictions in medical research is very context-sensitive, and even minor misalignment of the settings can make a detected pair useless. A more realistic test scenario is probably what is reported in section 5.3.3, with a precision of 0.375 (9/24).”
The authors rebut this point by stating “Our research falls under the umbrella of distant supervision where noisier sources of supervision are used to create larger training sets quickly. Although they are noisy, when coupled during fine-tuning the results are promising, as seen in Table 3.”
The reviewer further notes the mention of WordNet in the rebuttal but not in the paper, which exacerbates concerns around reproducibility. The authors note that the full experiments are fully reproducible via the shared code, i.e. “If you navigate to our code repo (bottom of Page 2), in the file "snomed/snomedct_arr.ipynb", in the function "use_synonym_heuristic", you will see the use of WordNet. All details can be found and reproduced via the code.”
I feel that the point about reproducibility has been well addressed by having the code shared. The point about the crude heuristics is a more challenging one, however, the authors do a good job in pointing this out as a limitation of weak supervision and highlight the added value despite the presence of minor misalignments.

Clarity:
The reviewers disagree with regards to the clarity of the paper. Reviewer 1 states that the paper “has exemplary writing, meaningful comparison with established literature, and excellent evaluation”, while Reviewer 3 notes that “There are many scattered heuristics throughout the methods that also require referring back and forth with the appendix, making it very hard for the reader to track which is which in the end. English needs editing.”

Originality and Significance:
-	“The paper presents an innovative and relatively inexpensive method of data augmentation for detection of clinical contradiction by leveraging the existing SNOMED medical database.”
-	“a well-chosen, meaningful challenge in terms of matching what the NLI technical framework could be conveniently adapted to tackle in the medical domain.”

Overall, I believe that the points raised by reviewer 3 have been meaningfully addressed by the authors, with reproducibility further guaranteed through the shared code.
Decision: Accept-Main