2_use_case_name: Nediser reports QA
3_agency: Department of Veterans Affairs
3_abr: VA
4_bureau: VHA: Veterans Health Administration
8_topic_area: Health & Medical
10_commercial_ai: None of the above.
11_purpose_benefits: Nediser is a continuously trained artificial intelligence “radiology resident” that assists radiologists in drafting their reports. Using computer vision/ML, Nediser selects normal templates (for example, the knee, shoulder joint), detects absence or presence of hardware in the image, evaluate patella alignment and leg length and angle discrepancy, measure Cobb angles, and evaluate hip exams for osteoarthritis. The output is directed to a QC field in the report that the Radiologist may use. The output helps standardize reports and improve accuracy of reporting.
12_outputs: The system classifies arthritis grade and performs measurements which are output to a QC field in dictation software as a draft that the Radiologist may or may not incorporate into their report templates. This is similar to a Radiology resident who provides draft reports that the Attending Radiologist reviews. The system can also detect objects and landmarks to assist in making measurements and identify abnormality. The output are landmark annotations and measurements. However, these outputs are not sent to the production PACS. They are stored in a secure directory which the Radiologists can refer to and do not become part of the medical record. All original production DICOM images are deleted after processing. Nediser saves de-identified JPG images and reports (no PHI) for audit and review of system performance, and to further train the model is necessary.
16_dev_stage: Operation and Maintenance
17_impact_type: Both
18_date_initiated: 12/1/2020
19_date_acq_dev_began: 12/1/2020
20_date_implemented: 1/1/2021
22_dev_method: Developed in-house.
24_hisp_support: No
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: 
Radiology images within VISN 21 are used for training and fine-tuning. The original data set for training the system comes from Stanford University through IRB-approved research.

34_data_docs: Documentation is widely available: Documentation is not only complete, but is widely accessible within the agency, and has an owner and a regular update cadence.
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: No
41_system_name: Nediser
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: No
45_compute_request: No
47_timely_resources: Yes
49_existing_reuse: None: This use case does not re-use any internally developed tooling or managed infrastructure from any other AI development efforts within the agency.
50_internal_review: Documentation has been published: Complete documentation has been published to a repository or data catalog within the agency and is made accessible to other data science teams for review and feedback.
51_extension_request: No – Agency did not request an extension for this use case.
52_impact_assessment: Yes
53_real_world_testing: Performance evaluation in operational environment: The AI use case has been tested in an operational environment before being fully implemented as a solution, or has been tested in simulated or controlled environment using operational or synthetic data.
54_key_risks: Nediser is a system that receives Radiology images from the PACS. The system uses AI/computer vision/CNNs to perform several tasks including: identify landmarks on the images to perform measurements of lengths and angles of the lower legs and spine, identifies the absence or presence of hardware, and classifies the grade of hip arthritis. Before the models were deployed in operation, their performance were evaluated with testing and validation, and performance have been published in peer reviewed scientific conferences and/or publications:   Ha, AY., Do, BH, Bartret, AL, Fang, CX, Hsiao, A., Lutz, AM, Banerjee, I., Riley, GM., Rubin, DL., Stevens, KJ, Wang, E., Wang, W., Beaulieu, CF., Hurt, B. Automating scoliosis measurements in radiographic studies with machine learning: Comparing artificial intelligence and clinical reports. Journal of Digital Imaging, Feb 2022. https://link.springer.com/article/10.1007/s10278-022-00595-x   Larson, N., Nguyen, C., Do, B., Kaul, A., Larson, A., Wang, S., Wang, E., Bultman, E., Stevens, K., Pai, J., Ha, A., Boutin, R., Fredericson, M., Do, L., Fang, C. Artificial Intelligence System for Automatic Quantitative Analysis and Radiology Reporting of Leg Length Radiographs. J Digit Imaging July 2022. https://doi.org/10.1007/s10278-022-00671-2.   Ha, A., Fang, C., Baig, S., Do, BH. Automatic Detection of Aortic Aneurysm on CT Exams Using Deep Convolutional Neural Networks. Scientific presentation. AMIA 2020 Annual Symposium, Virtual, November, 2020.   Aryan Kaul1, Jason Pai2, Charles Fang2, Ed Boas3, Kathryn Stevens2, Jason Saleh2, Vananh Nguyen2, Constance Chu, Jamie Schroder2, Michelle Nguyen2, Joshua Reicher2, Woon Teck Yap2, Amelie Lutz2, Bao H. Do. Automatic Diagnosis of Knee Patella Malalignment on X-ray Using Artificial Intelligence: Performance Comparison with Physicians. Scientific presentation. AMIA 2020 Annual Symposium, Virtual, November, 2020.  Audrey Ha, John Vorhies, Andrew Campion, Charles Fang, Michael Fadell II, Steve Dou, Safwan Halabi, David Larson, Emily Wang, YongJin Lee, Joanna Langner, Japsimran Kaur, Bao Do. Automatic Extraction of Skeletal Maturity from Whole Body Pediatric Scoliosis X-rays Using Regional Proposal and Compound Scaling Convolutional Neural Networks. Scientific presentation, Virtual. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), December 2020.  The goal is to deploy this system VISN wide in VISN21. However, the past 3 years have been spent testing in a limited, "shadow" deployment. Limited = Palo Alto VA only. "Shadow" deployment = the system is never deployed in an autonomous workflow, requires human in the loop, and outputs of the system (annotations of measurements) are never sent back to the PACS or EHRM. Instead, the model's output is sent as text to a marge field/QC field in the Radiologist dictation software called Powerscribe. This does not go to VistA or CPRS or the patient medical record. The merge field is then used by Radiologists if they desire, imported into their draft report. They can review this output, validate visually, and agree or disagree. The program deletes original DICOM data. All annotations of the program, predictions are saved with final radiologist report for audit.  Because the AI system requires human in the loop, never sends output to the ERHM/VistA, and is an optional QC/merge field in the report draft, the technical risks are minimized. Clinically, the model performs orthopedic measurements (patella, leg lengths) and arthritis. The models are not involved in real-time decision support of clinical diseases such as stroke, lung cancer.   However, all systems have risk. We identify the key risk of the Nediser system is that there may be discrepancy between the system draft output and the final report if the Radiologist does not read the AI draft report and inadvertently saves the report.
55_independent_eval: TRUE
56_monitor_postdeploy: Intermittent and Manually Updated: A plan for monitoring the AI use case is in place, and requires data science teams to work with DevOps engineers to manually update models at scheduled intervals, and create metrics to detect data distribution shifts between the operational environment and the training data for the model. 
57_autonomous_impact: No - Some individual decisions or actions require direct human oversight.
59_ai_notice: N/A - individuals are not interacting with the AI for this use case
61_adverse_impact: No
62_disparity_mitigation: This AI performs computer vision to detect and classify hip arthritis based on the images.
63_stakeholder_consult: None of the above
65_appeal_process: No – it is not operationally practical to offer this.
66_no_appeal_reason: In Radiology, if patients or providers have a question or concern about a Radiology report, they contact the Radiologist on the report who can discuss and amend the report for errors if necessary.
67_opt_out: Other
67_question_type: The final radiology report does not change whether the radiologist uses the AI to perform a measurement with validation by the radiologist vs performing the measurement manually.