## 5 Discussion  
We developed an NLP algorithm that extracts ECOG PS scores from unstructured EHR documents with high performance compared to manually abstracted data. In contrast to prior research, which largely trained and applied ECOG extraction efforts to smaller single-site cohorts, this study applies a rule-based information extraction approach on a large nationwide EHR-derived de-identified RWD source to extract ECOG PS across a multitude of cancer types [44]. We showed that this strategy can achieve high performance, extracting ECOG PS at the time of treatment initiation, with a transparent and explainable algorithm deployed onto a large database that originated from real-world EHR data sources. The underlying study network from which the training and testing cohorts were selected includes approximately 280 cancer clinics (\(\sim\)800 sites of care) distributed nationwide (US-based), including academic and community practices, with a wide range of sizes [13,49]. The Flatiron Health database has been compared to The Surveillance, Epidemiology, and End Results Program (SEER) program and the National Program of Cancer Registries (NPCR)--both authoritative sources for population cancer surveillance and research in the US--and found general similarities in demographic and geographic distribution. However, patients from the Flatiron Health database appeared to be diagnosed with later stages of disease and their age distribution differs from the other datasets [49]. Another unique strength of this study is the availability of a patient sub-cohort with manually abstracted ECOG PS scores as the internal reference for quality, together with access to the primary EHR data sources, both structured and unstructured. This infrastructure enabled the verification of discrepancies and the identification and investigation of potential errors in both directions (of both the NLP algorithm-extracted and the manually-abstracted data). Another strength of this study includes our improvement upon data completeness for EHR-derived study cohorts in oncology. In our database of interest (N = 480,825 patient-lines), use of this algorithm increased overall ECOG PS completeness from 60.4% to 73.2%. Guidelines from health authorities have highlighted completeness as a key component of data integrity and  
Figure 2: rwOS in patients with aNSCLC stratified according to their ECOG PS score for the subcohort with ECOG PS scores available as structured data, and for the subcohort with ECOG PS scores extracted via algorithm. a common shortcoming of observational data [51, 52], spurring investigation in this area. NLP has been proven useful for the extraction of information from clinical notes in oncology EHRs; for example, date or site of disease recurrence, presence of symptoms or adverse events, date of disease onset, etcetera [28, 30, 31, 32, 33, 34, 53, 54]. NLP techniques can be deployed to derive variables from EHR data, increasing data completeness and quality, boosting analytic robustness, and therefore improving the utility of RWE [18]. For our work, we considered NLP to be the best suited to extract explicit assessments of a patient's ECOG PS recorded in the EHR, rather than relying on contextual inferences made from EHR documents. The fact that NLP is traceable and auditable to source documentation is another benefit of the approach. Furthermore, a strength of this study was our RWE focus in algorithmic evaluation. First, we stratified performance by ECOG score to allow researchers to understand the implications of using different ECOG PS-based criteria in their studies. Additionally, we assessed how NLP-extracted ECOG PS scores correlate with overall survival. Ensuring that extracted ECOG PS scores align with clinical expectations (the observation that patients with worse ECOG PS scores do not live as long [55, 56, 57, 58, 59]) provides reassurance about the prognostic value of the data extracted by the algorithm.