JMIR MEDICAL INFORMATICS

Senior et al

Original Paper

Evaluation of SNOMED CT Grouper Accuracy and Coveragein Organizing the Electronic Health Record Problem List by
Clinical System: Observational Study
Rashaud Senior1, MMCi, MD; Timothy Tsai2, MMCi, DO; William Ratliff3, MBA; Lisa Nadler1, MD; Suresh
Balu3, MS, MBA; Elizabeth Malcolm4, MSHS, MD; Eugenia McPeek Hinz1, MS, MD
1

Duke University Health System, Durham, NC, United States
Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States
3
Duke Institute for Health Innovation, Durham, NC, United States
4
Division of General Internal Medicine, Duke University School of Medicine, Durham, NC, United States

2

Corresponding Author:
Rashaud Senior, MMCi, MD
Duke University Health System
2424 Erwin Rd
Durham, NC, 27705
United States
Phone: 1 9542950076
Email: rashaudsenior@gmail.com

Abstract
Background: The problem list (PL) is a repository of diagnoses for patients’ medical conditions and health-related issues.
Unfortunately, over time, our PLs have become overloaded with duplications, conflicting entries, and no-longer-valid
diagnoses. The lack of a standardized structure for review adds to the challenges of clinical use. Previously, our default
electronic health record (EHR) organized the PL primarily via alphabetization, with other options available, for example,
organization by clinical systems or priority settings. The system’s PL was built with limited groupers, resulting in many
diagnoses that were inconsistent with the expected clinical systems or not associated with any clinical systems at all. As aconsequence of these limited EHR configuration options, our PL organization has poorly supported clinical use over time,
particularly as the number of diagnoses on the PL has increased.
Objective: We aimed to measure the accuracy of sorting PL diagnoses into PL system groupers based on Systematized
Nomenclature of Medicine Clinical Terms (SNOMED CT) concept groupers implemented in our EHR.
Methods: We transformed and developed 21 system- or condition-based groupers, using 1211 SNOMED CT hierarchal
concepts refined with Boolean logic, to reorganize the PL in our EHR. To evaluate the clinical utility of our new groupers,
we extracted all diagnoses on the PLs from a convenience sample of 50 patients with 3 or more encounters in the previous
year. To provide a spectrum of clinical diagnoses, we included patients from all ages and divided them by sex in a deidentified
format. Two physicians independently determined whether each diagnosis was correctly attributed to the expected clinicalsystem grouper. Discrepancies were discussed, and if no consensus was reached, they were adjudicated by a third physician.
Descriptive statistics and Cohen κ statistics for interrater reliability were calculated.
Results: Our 50-patient sample had a total of 869 diagnoses (range 4-59; median 12, IQR 9-24). The reviewers initiallyagreed on 821 system attributions. Of the remaining 48 items, 16 required adjudication with the tie-breaking third physician.
The calculated κ statistic was 0.7. The PL groupers appropriately associated diagnoses to the expected clinical system with asensitivity of 97.6%, a specificity of 58.7%, a positive predictive value of 96.8%, and an F1-score of 0.972.
Conclusions: We found that PL organization by clinical specialty or condition using SNOMED CT concept groupers
accurately reflects clinical systems. Our system groupers were subsequently adopted by our vendor EHR in their foundation
system for PL organization.
JMIR Med Inform 2024;12:e51274; doi: 10.2196/51274

https://medinform.jmir.org/2024/1/e51274

JMIR Med Inform 2024 | vol. 12 | e51274 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Senior et al

Keywords: electronic health record; problem List; problem list organization; problem list management; SNOMED CT;
SNOMED CT Groupers; Systematized Nomenclature of Medicine; clinical term; ICD-10; International Classification of
Diseases

Introduction
The electronic health record (EHR) problem list (PL) isa dynamic repository of a patient’s current and historicalconditions as well as other health-related issues. As such,it supports communication across a wide range of potential caregivers and clinical environments. An accurate PL
serves as a foundation for clinical care and population healthmanagement, with multiple derivative secondary processes,
including phenotype extraction and disease prediction.
Understanding the history of the PL helps to illustrate why
this construct has become the default format for summarizing
patients’ clinical history. In the 1960s, Lawrence L Weed,
MD, proposed the concepts of the problem-oriented medical
record; the PL; and the Subjective, Objective, Assessment,
and Plan (SOAP) notes for documentation [1]. The idea wasto colocate clinical problems with clinical results to focus
on systematically addressing all of a patient’s diagnoses [2].
Although the SOAP note became the standard format forclinical notes, the PL has encountered more inconsistent use,
struggling with problems of inaccuracy, missing diagnoses,

not being updated, and bloating [3]. In 2009, the HITECH(Health Information Technology Economic and Clinical
Health) Act codified the requirement for an up-to-date PL
for meaningful use [4]. Until recently, our vendor EHR had
relied on relatively ineffective organization strategies for thePL.
With no one owner, the PLs have become disorganized and cluttered with duplications, conflicting entries,
and no-longer-valid diagnoses that contribute to information
overload and bloat, obscuring the patient’s clinical picture[5]. In its former state, our EHR PL was organized primarilyalphabetically, with other options based on primary specialty
or priority, all of which have limited clinical utility, especiallyas the number of diagnoses on the PL increases (Figure 1) .
For example, for one patient, we found active diagnoses oflung nodule (Respiratory System), then lung cancer (Oncology System), and then lung cancer with brain metastases(Oncology System). These diagnoses were all related to thesame problem but were added sequentially with previous
diagnoses that were no longer clinically relevant and were
not removed.

Figure 1. Appearance of a problem list before and after grouping algorithm application. Items were reorganized into 21 system groupers using
Boolean logic with the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) codes; they were then translated into the International Classification of Diseases, Tenth Version (ICD-10) codes. Groupers were based on a combination of traditional medical specialty categories,
clinically relevant care coordination, and procedure-based groupings, some of which were themselves combined due to overlapping diagnostic
coverage. The final order of the problem list items was determined by Epic System’s base hierarchy. CMS: The Centers for Medicare andMedicaid Services; HCC: Hierarchical Condition Category; HHS: US Department of Health and Human Services; FEN/GI: Fluids, Electrolytes,
Nutrition/Gastrointestinal; GFR=Glomerular Filtration Rate.

There are several major terminology standards that capture patient diagnoses, symptoms, and other health-related
conditions, two of which are the International Classification
https://medinform.jmir.org/2024/1/e51274

of Diseases (ICD) [6] and the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) [7]. The
World Health Organization maintains ICD codes, which are
JMIR Med Inform 2024 | vol. 12 | e51274 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Senior et al

designed to classify diseases, conditions, and other healthrelated issues [8,9]. Organized into 21 chapters, they usean alphanumeric classification format to identify diseases,
injuries, and factors influencing health. The United Statesuses an additional Clinical Modifier for further specificity [10]. The ICD codes are used in various clinical andnonclinical settings, including disease description, treatment
selection, billing, and research applications [11]. There are
78,044 total codes in the 2024 code set, according to theCenters for Medicare and Medicaid Services [12].
Currently managed by Systematized Nomenclature of
Medicine (SNOMED) International (previously known as
the International Health Terminology Standards Development
Organization [IHTSDO]), SNOMED CT was designed to be a
US standard for health information exchange. It functions as
a highly granular ontology used to describe clinical observations and findings [13]. SNOMED CT uses a polyhierarchal (parent-child) format organized around a generalroot concept (eg, a clinical finding, procedure, situation
with explicit context, or event) with increased granularity
achieved by differentiating more specific descriptions of thatroot concept. This process allows for the representation of
specific clinical content in a machine-readable format [14,15].
Updated monthly, the total number of concepts is 512,087 (asof April 1, 2024 [16]) and continues to increase over time.
Though required for billing, the ICD, Tenth Revision(ICD-10) terminology is not used directly in clinical care,as many code names are not consistent with clinical vernacular. For example, code Z91.038 “Allergy status tounspecified drugs, medicaments and biological substances”
is not as intuitive as “Allergy to insect stings.” For thisexample, our third-party vendor, Intelligent Medical Objects(IMO), transforms the ICD-10 codes into clinically relevant
human-readable concepts. IMO additionally maps at least
1 SNOMED CT concept for each diagnosis, attached as
metadata, upon which the PL groupers can be organized.
Although SNOMED CT concepts are mapped to ICD-10
codes [17,18], as with most ontologies, there are gaps inclinical concept coverage.
PL organization and cleanup is challenging for manyreasons, including there being no single owner of a patient’s
PL [19] and its maintenance and cleanup being secondary toother direct clinical care priorities. The tools in the EHR forcleanup are limited to a single patient and do not allow for
automated processing or opportunities to categorize or define
the state of the PL or large-scale maintenance at the population level. Multiple interventions, including reconfiguration of
the EHR PL and re-education, have been met with limited
success [20]. Despite these attempts, PL bloat and inaccuracy
are widely recognized as issues affecting clinical care andsecondary downstream uses of the data [3]. We have cometo recognize that curating a clinically relevant and updated
PL is a difficult challenge; our primary option for improving
its organization was to extend and improve SNOMED CT
groupers.
In this paper, we present a PL reorganization developedaround clinical specialty groupings using SNOMED CT
codes and Boolean logic. We describe the evaluation of the
new PL groupers for clinical accuracy and efficiency using a
convenience set of patients and their diagnoses. This systemallows for future characterization of the PLs at the patientand population levels; it also provides potential for automatedcleanup options in the future.

Methods
SNOMED CT Grouper Development and
Evaluation
Author EMH extended and extensively modified 19
previously defined groupers initially developed by Heidi
Twedt, MD, and added 2 newly defined system- or condition-based groupers, one for pediatric and one for transplant-specific conditions (Table 1). System groupers included
traditional medical specialty categories as well as clinically
relevant care coordination and procedure-based groupings.
Some specialties were combined due to overlapping diagnosis
domains (eg, “Respiratory and Allergy” and “Orthopedic andMusculoskeletal” domains). The primary focus was for thesystem grouper diagnoses to be organized around clinicaluse. For example, “acute myocardial infarction” and “venous
thromboembolism” were sorted into the “Cardiovascular andPeripheral Vascular” grouper, while addiction issues, suchas “alcohol use disorder,” were sorted into the “Behavioral
Health” grouper.

Table 1. List of system groupers with example diagnoses.
Condition or specialty grouper
1. Care Coordination

Notable deviations
Includes health-related social needs

2. Oncology

Example diagnosis
Physical deconditioning, food insecurity, risk
for falls
Malignancies and radiation therapy diagnoses

3. Cardiovascular and Peripheral VascularAtrial fibrillation and deep vein thrombosis

Excludes cerebral vascular diagnoses

5. Endocrine

Diabetes mellitus, gout, and hypothyroidism

4. Respiratory and Allergy
6. Behavioral Health
7. Transplant

8. Infectious Disease, Immune, or Lymphatic
https://medinform.jmir.org/2024/1/e51274

Asthma and peanut allergy

Schizophrenia and opioid use disorder

Living-related kidney transplant and graft
versus host disease
Pneumonia and immune deficiencies

Excludes dermatology cancers and includes
treatment complications—a——Includes transplant complications—
JMIR Med Inform 2024 | vol. 12 | e51274 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Senior et al

Condition or specialty grouper

Example diagnosis

Notable deviations

9. Blood

Anemia

—11. Ears, Nose, Throat (ENT)

Nasal polyps, cleft palate, and hearing loss—

10. Neurology or Sleep

12. Fluids, Electrolytes, Nutrition, andGastrointestinal
13. Obstetrics and Gynecology

14. Genitourinary and Nephrology

Seizure and sleep disorders

Hyponatremia and Crohns disease

Excludes chronic pain—

Ovarian cysts; hemolysis, elevated liver
enzymes, low platelet count (HELLP)
syndrome; and dense breast tissue

Female-specific diagnoses

Atopic dermatitis and melanoma

Includes all dermatology-specific cancers (eg,squamous or basal cell carcinoma)

Ureteral calculus and prostatitis

Includes male-specific genitourinary issues

16. Rheumatology

Rheumatoid arthritis—

18. Ophthalmology or Eye

Uveitis15. Dermatology

17. Orthopedic and Musculoskeletal
19. Genetics

20. Pediatrics

21. Surgery, Trauma, Wound, and Pain
22. Other

Hip fracture

Includes complications of eye from other diseases

Trisomy 21

28-week prematurity

Includes all nonspecific system genomic issues

Gunshot wound and complex regional pain
syndrome
Edema and medication management

aNot applicable.

Due to its polyhierarchal framework, all child-related
SNOMED CT concepts include all related downstream
concepts, unless excluded by the Boolean logic. In thisformat, fewer SNOMED CT concepts can represent many
derivative ICD-10 codes more comprehensively than could
be achieved by directly curating ICD-10 codes. For example,
167 SNOMED CT concepts within the Neurology grouper
were mapped to 9243 IMO ICD-10 diagnoses. Our default
EHR PLs were reorganized according to this system-based
methodology in the order presented in Table 1.
Using our EHR vendor’s built-in tools for grouper build,author EMH iteratively refined the groupers to be consistent
with clinical systems using 1211 SNOMED CT concepts.
System groupers included the highest parent concept that
was appropriate with logic to exclude child-related SNOMED
CT concepts not clinically appropriate for a system. For
example, squamous cell carcinoma and skin cancers ingeneral are managed clinically by dermatology. In thebuild for the Oncology grouper, therefore, all dermatologic
cancers were excluded and instead added to the Dermatology
grouper. As another example, our Cardiovascular grouper
includes peripheral vascular diseases like “deep venous
thrombosis” but excludes cerebral vascular concepts. This
allows diagnoses like “cerebral avascular malformation” tobe presented within our Neurology grouper. Both examples
highlight the focus of this grouper organization to support
clinical specialty coordination of diagnoses.
Multisystem disorders were grouped according to thespecialty that would typically manage each disease entity. For
example, systemic lupus erythematosus was grouped under“Rheumatology.” If a diagnosis’s SNOMED CT concept
was too broad to be captured by one of the 21 groupers, it
defaulted into the “Other” category. For example, “edema” ishttps://medinform.jmir.org/2024/1/e51274—

Includes developmental disorders—

Includes any diagnosis that does not fit intoanother groupera clinical finding that can be reasonably attributed to multiple
diseases. As such, it does not have a specific condition or
specialty and instead falls into the “Other” category.
To evaluate the effectiveness of specialty sorting, we useda convenience sample of 50 patients randomly identified in
January 2022. These patients had at least 3 encounters in
the previous year and were selected across all age groups,
ranging from newborn to geriatric patients, with an equal
ratio of sexes (Table 2). The encounter criteria ensured
that identified patients had multiple recent opportunities tohave their PLs updated. The PLs for these patients wereextracted through screen capture software by author EMH
to develop a cohort with no patient identifiers. Standard
EHR PL functionality included system grouper name, timeframe since the problem was added to the PL, and a limited
free-text overview if included with the entry. These study PL
entries were reconfigured into a study document with labels
indicating sequential patient number, patient age, and patientsex.
Two of the authors, both family medicine physicians(TT and RS), independently examined each patient’s PL todetermine the clinical accuracy of system groupings for all
diagnoses (Table 3). For any items whose system attribution they questioned, the reviewers identified the SNOMED
CT code attached to the ICD-10 code. Diagnoses thatwere deemed correctly grouped into the appropriate systemgrouper were considered true positives, while those thatwere incorrectly grouped were considered false positives. All
diagnoses in the dropout “Other” category were examined bytheir associated SNOMED CT code for options for attribution to a defined system grouper. A diagnosis for which the
SNOMED CT code was too vague or not specific enough tobe grouped was considered a true negative. Any diagnosis
JMIR Med Inform 2024 | vol. 12 | e51274 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Senior et althat had a SNOMED code that could have been placed in a
relevant system grouper but was not was considered a false
negative.
Table 2. Patient demographics and baseline descriptive statistics. A total of 50 patients, subdivided by age and sex, with descriptive statistics, werereported for each age range.
Age ranges (years)

Gender<1
1-17

Total, n (%)
6 (12)
7 (14)

Male, n (%)
3 (50)
3 (42.9)

Female, n (%)
3 (50)
4 (57.1)≥65
All ages

13 (26)50 (100)

8 (61.5)
25 (50)5 (38.5)
25 (50)18-64

24 (48)11 (45.8)

13 (54.2)

Problems
Total, n
72
157

Mean
12.0
22.4

Median10
26

Min-Max
6-20
4-35

298
869

22.9
17.4

21
12

4-59
4-59

342

14.3

10

4-43

Table 3. Description of metrics used to determine the effectiveness of automated system grouping. Two reviewers examined individual problem list
items and their assigned grouping, placing each into a category.
Assessment category

Definition

ExampleTrue positive (correct system association)

Diagnosis falls into the right disease system—the SNOMEDa grouper is specific andattributable.
Diagnosis falls in the wrong system grouper.

“Community-acquired pneumonia” in the
Infectious Disease system

True negative (Other—correct systemassociation)

The SNOMED grouper associated with adiagnosis is not specific enough to be inanything but the Other category.

False negative (Other—incorrect systemassociation)

Diagnosis belongs to a specified systemgrouper but falls into the Other category due
to logic deficits in the grouper.

“Anticoagulated” placed with the SNOMED
grouper “Drug therapy finding”. This is not
specific enough to be attributed to justanticoagulation status.
“Genetic disorder” falling into the “Other”
category until the VCG Grouper is corrected.

False positive (incorrect system association)“Diaphragmatic stimulation by cardiac
pacemaker” grouped under “Central
Hypoventilation Syndrome”.

aSNOMED: Systematized Nomenclature of Medicine.

Each diagnosis was independently categorized according
to the scheme in Table 3; then the reviewers compared
their determinations. A third independent clinician (author
LN) served as a tie-breaker for those PL items for whichan agreement was not reached. We calculated descriptive
statistics to summarize the volume of diagnoses for the 50
test patients and performance metrics to assess the accuracy
and validity of the groupers. The correlation coefficient (κstatistic) was calculated for the degree of agreement between
reviewers and for SNOMED CT grouper attributions.
This work was performed using Epic Systems (version
May 2021; Verona, WI) initially deployed with ambulatory
applications in July 2012 and inpatient applications in June2013 within the Duke University Health System.

Ethical Considerations
All patient data were anonymized with all demographic
identifiers removed except for age. This study was approvedby the Duke University Internal Review Board for exempt
status (IRB #PRO-00108903).

of 12 diagnoses per patient. Table 2 includes the breakdownof the volume of PL entries across age and sex.
After their independent evaluations of the PLs, thereviewers initially agreed on 821 (94.4%) of the 869 total
problems (Cohen κ coefficient of 0.7, indicating moderate agreement [21]). Of the remaining 48 diagnoses, they
subsequently agreed on 32 for a revised agreement rate of
98.2%. The remaining 16 were adjudicated by author LN for
attribution.
Based on the definitions presented in Table 3, Figure 2
describes our results. Our final attribution evaluation foundthat the diagnoses were correctly attributed to a systemgrouper (ie, sensitivity) in 97.6% of cases, and the nonspecificdiagnoses were correctly placed in the “Other” category (ie,specificity) in 58.7% of cases. The positive predictive value,or the correct grouper accuracy rate, was 96.8%. We found37 (4.3%) true negatives, representing concepts without a
SNOMED CT code or diagnoses too general to be attributed
to a clinical system. The calculated F1-Score was 0.972.

Results
Across the 50 patients, aged 14 days to 93 years, there were atotal of 869 (range 4-59) diagnoses identified, with a medianhttps://medinform.jmir.org/2024/1/e51274

JMIR Med Inform 2024 | vol. 12 | e51274 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Senior et al

Figure 2. Two clinicians’ review of problem list sorting algorithm. FN: false negative; FP: false positive; PPV: positive predictive value; NPV:
negative predictive value; TN: true negative; TP: true positive; Sn: sensitivity; SNOMED CT: Systematized Nomenclature of Medicine ClinicalTerms; Sp: specificity.

Discussion
Overview
The PL is the repository of medical diagnoses intended toreflect the patient’s clinical conditions. Without groupings
reflective of the larger specialty formats of clinical care, thePL can become overloaded and difficult to use as a tool tocommunicate a patient’s clinical status across encounters. We
developed 21 SNOMED CT groupers for system concepts tostandardize the organization of our EHR PL based on 1211
concepts (Table 1). We chose to evaluate these PL groupers
across all ages and sexes to provide a more representative
sample of diagnoses across our EHR patient population,
recognizing that this is only a subset of the total potential
diagnoses. Taking advantage of the hierarchal logic of the
SNOMED CT concepts refined with Boolean logic allowed
for more than 95% of the diagnoses to be attributed to asystem grouper [22,23].
We established the effectiveness of these SNOMED CT
groupers in organizing the PL by clinical system related toclinical specialty or condition. We propose that this standardized format for PL organization permits the sharing andreproduction of concepts across other health systems andEHRs.

Comparison to Prior Work
Other groups have used conceptually similar methods with
SNOMED CT codes for clinical phenotyping [22]. However, those code sets are typically more narrow in scopefor a more specific clinical description. The United Medical Language System Clinical Observations Recordings andEncoding Problem List Subset is meant to “facilitate theuse of SNOMED CT as the primary coding terminology
for PLs or other summary level clinical documentation”[24]. Compared to these other SNOMED CT code sets,
our implementation includes broader clinical coordination
https://medinform.jmir.org/2024/1/e51274

groupings (eg, surgical, transplant, care coordination, andinfectious disease) that are more reflective of the PL clinicalcare needs within our institution. Our work here builds upon
those efforts and applies them at the system level, which ismore accessible for clinical use.

Limitations
We noted some limitations and challenges in using SNOMED
CT concepts for this build. Despite ongoing international
mapping efforts [17,18], SNOMED CT concepts are not fully
representative of all the ICD-10 codes because of differences in original intended uses [8,13] and baseline granularity [25,26]. For example, the ICD-10 code “Encounter for
pre-transplant evaluation for chronic liver disease” is mapped
to the SNOMED CT concept “patient encounter status,” asthere is no other comparable SNOMED CT coding option.
Estimates for the proportions of completely mapped concepts
or codes are found in studies reviewing the automationof mapping SNOMED CT and ICD-10 codes; one study
estimated the proportion of complete mappings to ICD-10Clinical Modification (ICD-10-CM) at 74% in 2012 [25], andanother one estimated the proportion of complete mappings
to ICD-10-Procedure Coding System (ICD-10-PCS) (used tocapture inpatient procedures) to be about 86% in 2017 [27].
There were many ICD-10 diagnoses that were too broadto easily match a SNOMED CT system grouper. “Fatigue”
is a good example of an inherently vague constitutionalor multisystemic symptom that does not have a clearly
identifiable system-level grouper in our schema. For these
diagnoses, the “Other” category was used to capture theremaining nonspecific diagnoses. It is important to note that
this category is not the same as the ICD-10 options for “Not
Otherwise Specified“ (NOS) or “Not Elsewhere Classifiable” (NEC) codes for lesser defined diagnoses. For example, “Pneumonia due to other infectious organisms, NEC”
still falls into our “Infectious Disease, Immune, Lymphatic”
grouper.
JMIR Med Inform 2024 | vol. 12 | e51274 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
We also note that the mappings are not completely
represented across all specialties in terms of the breadthof coverage of concepts. For example, we found more
SNOMED CT cardiology-specific concepts and fewer
pediatric-specific concepts. These differences may reflect
the relative volume of cardiology diagnoses in the general population. The more specific diagnosis of “Encounter for assessment of implantable cardioverter-defibrillator”
was mapped to an appropriate SNOMED CT concept andwas correctly placed into our cardiovascular system grouper.
However, the pediatric diagnosis “Concern about growth”
was only mapped to the SNOMED CT code “Finding
reported by subject or history provider,” which was too broad
to be added to the Pediatric grouper only, consequently falling
into the “Other” category. Specialties such as pediatrics also
require greater levels of specificity for their diagnoses thanis always possible with the SNOMED CT concepts currently
available.
There were also multiple ICD-10 codes mapped to thesame SNOMED CT code that made attribution to a systemgrouper challenging. For example, the diagnoses “Diaphragmatic stimulation by pacemaker” and “Disorder of cardiac
pacemaker system” mapped to the same SNOMED CT codeof “Disorder of cardiac pacemaker system,” placing theminto the Cardiovascular grouper, although the former would
ideally be attributed to the Pulmonary grouper.
As we consider the future challenges of algorithm-based
PL sorting, it will be important to investigate the implicationsof updating ontologies as the World Health Organization has

Senior et alalready published the 11th edition of ICD (ICD-11) with 35
countries now implementing it [28]. We do not suspect that
ICD-11 will replace SNOMED CT as an ontology organization method, as SNOMED CT maintains greater flexibility
for clinical use. Health systems are always evolving, and it
will be important to consider how such algorithms and theirapplications will evolve within them.

Conclusions
We leveraged a PL sorting algorithm based on the clinicalsystem–based SNOMED CT groupers to create a standardized PL format in our EHR, reorganizing the diagnoses,
symptoms, and medical problems for better clinical utility.
We found subjective positive outcomes for our clinical users
who reported streamlining their clinical review processes andeasier ability to identify similar and duplicate diagnoses. Thismay be especially helpful for patients with complex issues
and many associated diagnoses. A structured PL also enables
a shift from patient-level evaluation to potentially populationlevel assessments and cleanup automation.
As with improvements in the provider experience,
automated PL maintenance may also impact researchers
leveraging PL diagnoses for machine learning and othersimilar research. Such possibilities underscore the need for
accurate and updated PL diagnoses to achieve and maintainhigh-fidelity outputs. It will be important to further evaluate
methods to automate the maintenance of accurate Pls and best
influence care delivery.

Acknowledgments
The authors would also like to acknowledge the support of Tres Brown from the Duke Health Technology System’s Maestro
Care Electronic Health Records (EHR) Application Team. This work was done while authors RS and TT were Clinical
Informatics Fellows at Duke University.
This work was partially funded by a grant from the Duke Institute for Health Innovation.
Data Availability
The authors have uploaded the groupers to GitHub and will share if emailed directly.
Authors’ Contributions
All the authors of this manuscript participated in and contributed equally to the conceptualization, design, and evaluation of the
program list (PL) grouper categorization. EHM used the previous work of Heidi Twedt, MD, at Stanford (172 concepts across
19 systems and conditions) to develop the final set by extending 8 systems' coverage (eg, "Pulmonary" to "Pulmonary andAllergy"), adding 2 system groupers, and extending them to a total of 1211 concepts with the Boolean logic. Authors RS and
EMH primarily authored the manuscript with other authors contributing to editing. Epic Systems incorporated these groupers
into their standard development for PL organization by system as default in November of 2022. EMH will continue to makeyearly updates to the groupers.
Conflicts of Interest
None declared.
References
1.
Weed LL. Medical records that guide and teach. N Engl J Med. Mar 14, 1968;278(11):593-600. [doi: 10.1056/
NEJM196803142781105] [Medline: 5637758]
2.
Wright A, Sittig DF, McGowan J, Ash JS, Weed LL. Bringing science to medicine: an interview with Larry Weed,inventor of the problem-oriented medical record. J Am Med Inform Assoc. Dec 2014;21(6):964-968. [doi: 10.1136/
amiajnl-2014-002776] [Medline: 24872343]

https://medinform.jmir.org/2024/1/e51274

JMIR Med Inform 2024 | vol. 12 | e51274 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.

Senior et al

Poulos J, Zhu L, Shah AD. Data gaps in electronic health record (EHR) systems: an audit of problem list completenessduring the COVID-19 pandemic. Int J Med Inform. Jun 2021;150:104452. [doi: 10.1016/j.ijmedinf.2021.104452][Medline: 33864979]
Henricks WH. “Meaningful use" of electronic health records and its relevance to laboratories and pathologists”. J Pathol
Inform. Feb 11, 2011;2:7. [doi: 10.4103/2153-3539.76733] [Medline: 21383931]
Wright A, McCoy AB, Hickman TT, et al. Problem list completeness in electronic health records: a multi-site study andassessment of success factors. Int J Med Inform. Oct 2015;84(10):784-790. [doi: 10.1016/j.ijmedinf.2015.06.011][Medline: 26228650]
International statistical classification of diseases and related health problems (ICD). World Health Organization. URL:
https://www.who.int/standards/classifications/classification-of-diseases [Accessed 2023-02-23]
SNOMED international recognizes entity linking challenge winners. SNOMED. URL: https://www.snomed.org/[Accessed 2023-02-23]
Moriyama IM, Loy RM, Robb-smith AHT, Rosenberg HM, Hoyert DL. History of the Statistical Classification of
Diseases and Causes of Death. National Center for Health Statistics; 2011.
International Classification of Diseases, Tenth Revision (ICD-10). Centers for Disease Control and Prevention. URL:
https://www.cdc.gov/nchs/icd/icd10.htm [Accessed 2023-02-23]
2.05: ICD-10-CM. MedicalBillingandCoding.org. URL: https://www.medicalbillingandcoding.org/icd-10-cm/ [Accessed2023-03-14]
Alharbi MA, Isouard G, Tolchard B. Historical development of the statistical classification of causes of death anddiseases. Cogent Med. Jan 1, 2021;8(1):1893422. [doi: 10.1080/2331205X.2021.1893422]
2024 ICD-10-CM. Centers for Medicare and Medicaid Services. URL: https://www.cms.gov/medicare/coding-billing/icd-10-codes/2024-icd-10-cm [Accessed 2024-04-20]
Overview of SNOMED CT. National Library of Medicine. URL: https://www.nlm.nih.gov/healthit/snomedct/snomed_
overview.html [Accessed 2023-03-14]
Davidso D, Rawson M. SNOMED CT: why it matters to you. Wolters Kluwer. URL: https://www.wolterskluwer.com/
en/expert-insights/snomed-ct-why-it-matters-to-you [Accessed 2023-02-23]
5-step briefing. SNOMED International. URL: https://www.snomed.org/five-step-briefing [Accessed 2023-02-24]
Release summary. SNOMED CT Release Statistics 2024-04-01. URL: https://browser.ihtsdotools.org/qa/#/
SNOMEDCT/release-summary [Accessed 2024-04-20]
SNOMED CT to ICD-10-CM map. US National Library of Medicine. URL: https://www.nlm.nih.gov/research/umls/
mapping_projects/snomedct_to_icd10cm.html [Accessed 2023-03-02]
March 2021 - SNOMED CT managed service - US edition (US). SNOMED Confluence. 2021. URL: https://confluence.
ihtsdotools.org/pages/viewpage.action?pageId=121867053 [Accessed 2023-03-02]
Klappe ES, de Keizer NF, Cornet R. Factors influencing problem list use in electronic health records-application of theunified theory of acceptance and use of technology. Appl Clin Inform. May 2020;11(3):415-426. [doi: 10.1055/s-00401712466] [Medline: 32521555]
Kreuzthaler M, Pfeifer B, Vera Ramos JA, et al. EHR problem list clustering for improved topic-space navigation. BMC
Med Inform Decis Mak. Apr 4, 2019;19(Suppl 3):72. [doi: 10.1186/s12911-019-0789-9] [Medline: 30943968]
McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. Oct 2012;22(3):276-282. [doi: 10.11613/BM.2012.
031] [Medline: 23092060]
Willett DL, Kannan V, Chu L, et al. SNOMED CT concept hierarchies for sharing definitions of clinical conditions
using electronic health record data. Appl Clin Inform. Jul 2018;9(3):667-682. [doi: 10.1055/s-0038-1668090] [Medline:
30157499]
Chu L, Kannan V, Basit MA, et al. SNOMED CT concept hierarchies for computable clinical phenotypes fromelectronic health record data: comparison of Intensional versus extensional value sets. JMIR Med Inform. Jan 16,
2019;7(1):e11487. [doi: 10.2196/11487] [Medline: 30664458]
The CORE Problem List Subset of SNOMED CT®. US National Library of Medicine. URL: https://www.nlm.nih.gov/
research/umls/Snomed/core_subset.html [Accessed 2023-03-18]
Fung KW, Xu J. Synergism between the mapping projects from SNOMED CT to ICD-10 and ICD-10-CM. AMIA Annu
Symp Proc. 2012;2012:218-227. [Medline: 23304291]
McGlothlin S. SNOMED and ICD: aligning to standards. J2 Interactive. 2021. URL: https://www.j2interactive.com/
blog/snomed-and-icd/ [Accessed 2023-03-02]
Fung KW, Xu J, Ameye F, Gutierrez AR, D’Have A. Achieving logical equivalence between SNOMED CT and
ICD-10-PCS surgical procedures. AMIA Annu Symp Proc. Apr 16, 2018;2017:724-733. [Medline: 29854138]

https://medinform.jmir.org/2024/1/e51274

JMIR Med Inform 2024 | vol. 12 | e51274 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
28.

Senior et al

ICD-11 2022 release. World Health Organization. 2022. URL: https://www.who.int/news/item/11-02-2022-icd-11-2022release [Accessed 2023-03-15]

Abbreviations
EHR: electronic health record
HITECH: Health Information Technology Economic and Clinical Health
ICD: International Classification of Diseases
IHTSDO: International Health Terminology Standards Development Organization
IMO: Intelligent Medical Objects
NEC: Not Elsewhere Classifiable
NOS: Not Otherwise Specified
PL: problem list
SNOMED: Systematized Nomenclature of Medicine
SNOMED CT: Systematized Nomenclature of Medicine Clinical Terms
SOAP: Subjective, Objective, Assessment, and Plan

Edited by Christian Lovis; peer-reviewed by Christophe Gaudet-Blavignac, Ling Chu, Triep Karen; submitted 27.07.2023;
final revised version received 01.12.2023; accepted 22.02.2024; published 09.05.2024
Please cite as:
Senior R, Tsai T, Ratliff W, Nadler L, Balu S, Malcolm E, McPeek Hinz E
Evaluation of SNOMED CT Grouper Accuracy and Coverage in Organizing the Electronic Health Record Problem List by
Clinical System: Observational StudyJMIR Med Inform 2024;12:e51274
URL: https://medinform.jmir.org/2024/1/e51274
doi: 10.2196/51274© Rashaud Senior, Timothy Tsai, William Ratliff, Lisa Nadler, Suresh Balu, Elizabeth Malcolm, Eugenia McPeek Hinz.
Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 09.05.2024. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in
JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on
https://medinform.jmir.org/, as well as this copyright and license information must be included.

https://medinform.jmir.org/2024/1/e51274

JMIR Med Inform 2024 | vol. 12 | e51274 | p. 9
(page number not for citation purposes)