Title: Mixed Reality Imaging of Congenital Heart Disease to Improve Surgical Decision-Making
Funder: Department of Health and Social Care

Abstract: Aim: to develop novel technology for holographic display and quantification 3D time-resolved ultrasound images of the beating heart; and to prove that this technology will improve surgical planning and communication between clinicians and patients. Project summary: We will develop a holographic system for visualisation and interrogation of 4D cardiac images. The system will have a mobile, clinically compatible physical component (hardware) and an application ecosystem (software) for cardiac quantification, described below. The hardware will consist of a clinical trolley (Bytec MIT-Cart), fitted with a workstation and a commercially available holographic display (zSpace); and a mobile visualization device. The software will have two components: a fully featured holographic application in the workstation; and a virtual reality (VR) application running on the mobile device which will reproduce in real time the holographic view from the workstation. The clinical impact of this proposal is three-fold. Firstly, the proposed system will enable improved visualisation and quantification of the patient's heart. Secondly, it will enhance treatment planning by facilitating visualisation of the hologram by multiple clinicians simultaneously, with the aim of enhancing discussion and communication. Thirdly, the system will provide an intuitive way of communicating with patients and training staff about congenital heart defects. Team: we have a team of world-leading academics and young researchers with a balanced expertise in cardiology (Prof. Simpson, Dr Pushparajah, from Evelina London Children’s Hospital), computational imaging (Prof. Schnabel, Dr Gomez, from KCL) and software engineering (Dr. Toussaint from KCL). Two recruits will complete the team: a Software Developer (SW) and a Research Associate (RA) Outline: the project is organised in four work-packages (WP) over 24 months as follows, with WP leaders in parenthesis: WP1: Design and development of the holographic software (SW, Toussaint, Schnabel), by integrating our current software prototype (in Unity) with standard measuring, slicing and planar representation of medical images. - Month 1-9: migration of prototype to suitable platform, using the zSpace C++ API. - Month 10-15: Incorporation of a standard multi-planar reformatting (MPR) representation. - Month 16-21: Development of interrogation tools, which will include 3D landmarking, linear and curvilinear measurement tools, annotating and labelling and predefinition of planes and views. WP2: Shared holographic visualisation (RA, Gomez, Schnabel). Development of a mobile application, largely based on our current Unity prototype, which will receive from the WP1 software the necessary information in real time to reproduce interaction and annotations on the heart representation in the hologram. - Month 1-3: analysis of VR vs Augmented Reality (AR) software (e.g. Vuforia) and hardware (e.g. Google Cardboard). - Month 4-12: Adaption of Unity prototype to VR/AR platform. - Month 13-21: Synchronized communication between mobile device and main software. WP3: System validation (SW, Pushparajah). Assessment of the accuracy of the system towards prospective CE marking for clinical use upon completion. - Month 1-9: images from ultrasound phantom with calibrated points and surfaces will be annotated by experts as reference (accuracy, precision and observer variability) for validation. - Month 10-18: measurement on patient data will be compared to standard software (Philips QLab). WP4: Clinical translation (RA, Simpson). Assessment of the potential impact in surgery planning and communication between users. - Month 10-18: selection of cases for surgical re-planning using the hologram and comparison with original plan. - Month 19-24: Test the ability of the system to communicate patient's cardiac condition between clinicians. WP4 will also target patient and public engagement by obtaining feedback from them on the use of the proposed system as a tool for communicating clinical aspects of the disease. The Gantt chart in the supporting documentation summarizes the above and includes the milestones associated to each WP.

Lay abstract: Congenital heart disease (CHD) refers to heart problems which are present from birth. These can affect quality and length of life. CHD can include holes within the heart and abnormal valves which hinder the ability of the heart to pump blood around the body effectively. Most CHD is treated by a surgeon operating on the heart to repair it. Before any heart operation the surgeon has to fully understand the heart problem to enable planning for the repair - this is vital if the patient’s condition is to improve. Before any surgical procedure, detailed pictures and videos of the heart are taken by ultrasound (echocardiography) or other specialist techniques. The images and videos from such scans are currently shown on a flat screen like a television screen. In recent years, it has become possible to obtain three-dimensional" pictures and videos but they are still presented on a flat screen. It would help surgeons understand the problem better if we could show moving pictures of the heart as truly three-dimensional pictures as if the heart were actually floating in front of our eyes. The system we are developing will project true three-dimensional pictures and videos which will allow surgeons to examine the heart from any aspect or view before an operation. Surgeons and others will be able to see and measure the heart better than they can at present. The money we are seeking would allow this to happen. Patients will benefit from the project in two major ways. Firstly, the results of the heart operation will improve because of better information being available to the surgeon. Secondly, many patients or parents still do not fully understand what is wrong with their or their child's heart and we will involve them fully in the evaluation of the system to improve this."

Research activity codes: 4.1 DISCOVERY AND PRECLINICAL TESTING OF MARKERS AND TECHNOLOGIES
Health categories: CARDIOVASCULAR