Why I Advocate Accessibility of Raw Sensor Signals?
By: Onur Dur
Tags: FDA, Healthcare, Quality Assurance, Wearables, Digital Health

Perspectives from wearable health device industry on data quality and FDA’s proposed real-world product performance surveillance process

Since its inception — over a decade now, the wearable devices industry has adopted the black-box approach which is typical to consumer electronics companies, for obvious reasons. First, this has enabled the vendors to provide services that are “proprietary”, therefore, “hard-to-debunk”. Plus, the product claims can be easily fueled with mass marketing tools to generate “high perceived value”. When it comes to health though, this generates a huge red flag and also an equally large barrier for adoption because [for all the good reasons] the health industry across-the-world is heavily regulated and, simply said, nobody wants to gamble when comes to their own health. On the flip side, “consumerization of healthcare” has been getting more and more popular across the general public, and also gaining support from healthcare professionals, who are now demanding services that enable individuals to take full control of their own health. Last couple years the government has also pledged allegiance to support the consumer-centric healthcare move by defining new regulatory pathways that aim granting consumers faster access to digital health services and products [1]. Though, the billion dollar question is yet to be solved: how to accelerate digital innovation in this new era of healthcare while properly establishing necessary quality control processes that ensure high quality information is translated to the end user.

Quality of health data generated by the wearable device is audited using the raw sensor signals.

Long story short, particularly for wearable health devices, I advocate that one answer to quality control in digital health is to aggregate raw sensor signals along with derived metrics, and open complete data access to all end users. Here is why:

“Aggregating raw signals to control quality of derived metrics will establish trust & accountability to the end-user, and enable real-world post-market auditing of wearable health data.”

Next couple paragraphs, my goal is to surface some of the current gaps in quality control of wearable device generated health information and to offer solutions for establishing quality standards, continuous data quality improvement and other added benefits for all stakeholders including consumers/patients, health professionals/researchers and regulators.

Before we start, it is important that I make a clarification about the term “raw data” collected by wearable health devices, as many misuse this by associating it with the “processed data”. For clarification, raw data refers to “raw sensor signals” (e.g. photoplethysmography signal), and the processed data refers to any health information (e.g. heart rate) derived by processing the raw sensor signals. Ok, now we are ready to talk about quality control in wearables through raw data aggregation.

Establishing quality standards for wearable device generated health information

Over last 3 years working with business and healthcare professionals, clinical researchers and scientist from academia and pharmaceutical industry, here are the top two common questions I was asked:

“When my app shows a value X for heart rate how can I believe this information?”

“How can the study data management team qualify the quality of information derived from health wearables?”

Essentially, these questions boil down to confirming the quality of derived information from the wearable devices. My short answer is “review your raw signal data”. If the collected raw sensor signals are high quality and comparable to those collected for validating the algorithms thumbs up, if not, thumbs down. When it comes down to real world use of wearables every application is different and every use-case has its own nuances. Often times, the quality of signal collected may be lower due to motion or other artifacts, particularly when devices are used outside of vendor’s recommended settings or tested conditions (age, perfusion, disease state etc.). Availability of raw sensor data for retrospective quality auditing generates the much needed and currently unmet need of trust, accountability and quality control of the derived information and insights translated to the end user. Therefore, traceability of the raw sensor signals allows quality control for the derived health information.

Illustration of high (left) and low (right) quality raw sensor signals and how they impact the quality of health information derived from them.

“… traceability of the raw sensor signals allows quality control for the derived health information”

This is also not a new concept for the medical device industry. To control and continuously improve quality of medical devices, each vendor is required to establish processes that ensure traceability of each hardware component including their model name, batch number, processes it went though until it is assembled into the final product. In terms of data traceability, most ECG devices do not just report events, they also record raw signals and make them available for retrospective review. Same as echocardiography, angiography, MRI, automated arterial pressure monitors. So, why not we incorporate the same system architecture for wearables? Current advancements in wireless technology has already enabled high bandwidth data transfer through BLE/BE, NFC etc. Therefore, technology-wise, there is absolutely no barrier to raw sensor data aggregation.

Raw sensor signals and derived metrics displayed on a typical patient monitor

Continuous quality improvement in wearable device generated health information

“In essence, raw health sensor data aggregation enables continuous value-harvesting from wearables through continuous quality improvement.”

What if one can re-process the aggregated signals with new and better algorithms in the future? What if we can generate more information and more accurate information; gain new and better insights from the same datasets had we re-processed them next year? Bingo, this is called a “continuous value-harvesting” on the aggregate data. The concept requires siloed the raw sensor signals and ability to re-process them with more advanced algorithms in the future. This way, virtually no data collected is wasted and the value of the health data increase over the time. In essence, raw health sensor data aggregation enables continuous value-harvesting from wearables through continuous quality improvement of algorithms. Continuous quality improvement is also not a new concept across industries. To mention couple examples: Toyota collects in-process and on-road data to improve its vehicles, Salesforce collect medidata from customers to improve its services.

Unleash full potential of wearable health devices:

To see the big picture, raw sensor signal aggregation has three main benefits for the stakeholders: 1) streamlining auditing for regulators, 2) boostering utility for healthcare professionals/researchers, and 3) democratization of health data generation for consumers/patients.

First and foremost, this new way of collecting health data removes vendor-specific algorithmic bias and monopoly of device companies over how to interpret personal health data. Of course, most companies will dislike this. Nevertheless, the black box approach does not to expose their secret sauce to the competition, or hardware and algorithmic flaws to the general public. Many motivated consumers demand more information about their health wearables, particularly about how the processed data is generated and whether it can be trusted. The level of transparency required is often beyond what’s written on vendor websites: “My algorithm is clinical-grade” or “See my pilot study, certs, my white paper”. I will debate even for FDA/CE approved wearable devices, which are tested and “cleared” for very particular indications, it is hard to assume that the quality of data collected will always be high without any retrospective auditing. Frankly, some [if not many] of the real-world use-cases of such wearable devices will be outside of what’s covered by the very specific use-case the devices is originally approved.

“…raw sensor signals aggregation may serve as a key element to streamline health information quality control and product performance auditing in FDA’s new regulatory framework.”

Is this a means to end for wearable health devices? Shall we never trust the health data collected from wearable health devices? Surely, the situation is not as worrisome, thanks to FDA who has already planted the post-market quality surveillance process as a key component of the new regulatory framework that will govern digital health products . Latest update was released last week. In nutshell, this new process aims to feed real-world device performance data back into the regulatory process, thereby continuously audit and improve the safety and efficiency of the regulated devices. A major missing link in this process is how to properly assess the product performance without any other context available, such as simultaneously collected health data from a reference device which will most likely be lacking. In real world settings, raw sensor data can be used for confirming the desired device behavior and thereby independently verifying product performance — in a device agnostic manner within the same device category. I believe that the raw sensor signal aggregation may serve as a key element to streamline health information quality control and product performance auditing in FDA’s new regulatory framework.

FDA’s proposed key components of a future Pre-Cert Program [1]

Clinical research organizations and pharmaceutical companies have already started embracing wearable health devices [2]. Recently several large-scale research initiatives involving parties across government, health institutions, private technology and pharmaceutical companies [3–5] have announced collecting orthogonal bio-psycho-social signals and biometrics derived from these signals in longitudinal studies to understand underlying risk factors leading to disease and hospitalizations. There is also a growing interest in the pharmaceutical industry to use mobile, wearable and connected health technologies to improve the operational efficiency during the product development cycle, and also to accelerate bench-to-bedside translational science. Simply said, raw data aggregation enables seeing the big picture. It will help investigators not only to expand beyond the “claimed-accurate” processed metrics, but also will open up possibilities to interpret the signals beyond the “level of thinking” that designed the study protocol for data collection. Let me explain this further. In traditional clinical studies, investigators will decide and lock down all the data collection and analysis procedures before the trial commences. With raw data aggregation, one can retrospectively decide how to process the signals and will not be limited to prospectively designed processed metrics for the data analysis. This way, as investigators learn more about the datasets and as their level of thinking elevates they can use this information to gain more insights from the data collected and to gain more value from the overall study. In other words, raw data aggregation enables developing new metrics [that are beyond heart rate or step counts], aka. new digital biomarkers. Digital biomarker development has been a key application of digital health in the biopharma and clinical research spaces for developing new endpoints that have not previously been possible to assess, or existing endpoints that can be measured in new and possibly better ways [6].

Finally, and most importantly, raw data aggregation democratizes understanding and interpreting health data for consumers/patients, fuels innovation by giving researchers the option to push the limits what is known today and challenge status quo. The obvious benefit is “why to get stuck” with the Vendor X’s solution/algorithm if one would like to use Vendor Y or rather develop my own algorithm. For many, having access to raw signals opens up the opportunity to educate individuals and organizations on how wearables health devices actually work, help understand and unleash the true strength and full potential of such devices, at the same time, expose their flaws and how to properly interpret and make proper sense of the derived information.

One common argument I hear often is how to store and let alone collect the raw data. More personal health data has been collected over last three years than last three decades combined by health organizations that are looking for ways to leverage big data to modernize medicine and reduce cost of healthcare delivery. It’s striking that the current healthcare systems are unable to ingest this large volume of personal health data collected from wearable and connected devices as the privatization of health data, inaccessibility of raw data and lack of an interoperable data infrastructure are the main culprits [7]. Unequivocally, the corporate monopoly over person generated data is unethical, and bound to change with the new wave of regulations and advances in information technology. Data generators’ ownership and support for the data generation process is the key to achieve the large-scale adoption of new healthcare delivery of the digital age. Therefore, there is an important unmet need to streamline health data collection, sharing and storage. An alliance between a “motivated” network of data generators, “well-supported” health device and service providers has yet to be defined to develop an interoperable scalable health data infrastructure. Ongoing clinical studies utilizing wearable and connected devices in multiple therapeutic areas, including atrial fibrillation, sleep apnea, heart failure, neurodegenerative diseases, should harness interoperable big data analysis platforms to demonstrate how data gathered with these devices translates into actionable insights that led to clinical decisions. The clinical research community, the biotech industry and the government agencies will be working collectively to demonstrate improved outcomes along with lower cost of healthcare delivery. I think streamlined “raw data aggregation and accessibility” will be a key enabler to to achieve this goal.

References:

[1] FDA Website, Digital Health Software Precertification (Pre-Cert) Program, www.fda.gov/MedicalDevices/DigitalHealth/DigitalHealthPreCertProgram/ucm567265.htm

[2] Dur O, Rhoades C, Ng MS, Elsayed R, van Mourik R, Majmudar MD, Design Rationale and Performance Evaluation of the Wavelet Health Wristband: Benchtop Validation of a Wrist-Worn Physiological Signal Recorder, JMIR Mhealth Uhealth 2018;6(10):e11040

[3] Boudreau, R.L., 2017, “The Next Best Thing in Cardiovascular Research: Highlights From the Basic Cardiovascular Sciences Scientific Sessions. Journal of the American Heart Association”, 6(9), p.e007245.

[4] Hudson, K.L. and Collins, F.S., 2017. “The 21st Century Cures Act — a view from the NIH”. New England Journal of Medicine, 376(2), pp.111–113.[5] Manos, D. ed., 2017. Rocking the Baseline: Verily, Duke, and Stanford Aim to Make Medicine More Predictive with a New Baseline Study.

[6] CTTI Recommendations, Developing Novel Endpoints Generated by Mobile Technology for Use in Clinical Trials, www.ctti-clinicaltrials.org/files/novelendpoints-recs.pdf

[7] Kish, L.J. and Topol, E.J., 2015. Unpatients — why patients should own their medical data. Nature biotechnology, 33(9), p.921.