# Presentation: 776898

## CDRH Review Practice for Over-the-Counter in Vitro Diagnostic Tests

- Center for Devices and Radiological Health (CDRH)
- Office of in Vitro Diagnostic Device Evaluation and Safety
- Arleen Pinkos
- Scientific Reviewer

- CDRH OTC Review Practice, BPAC, 051103

**Notes:**
I’ll be presenting a brief overview of CDRH review practices for OTC IVDs.
As a frame of reference......these products are.....

## Overview of Presentation

- A.  Background
- B.  Three Elements of OTC Review
- Accuracy and Reliability
- Labeling
- Risk vs. Benefit
- C. Conclusion

## OTC History

- 817 OTC products cleared for home use
- Diabetic Products
- Hormones
- Drugs of Abuse
- Urine Dipsticks
- Fecal Occult Blood
- Lipids, PT

**Notes:**
There are 817....
Diabetic products have the highest volumeThey are listed in decreasing order of the number that we’ve cleared
Of note is the absence of any ID devices

## OTC Infectious Disease Devices

**None cleared or approved**
**Devices reviewed for OTC-    Self diagnosis: **
- Group A Streptococcus, and Influenza A and B
**In professional hands: **
**Sensitivity** averaged **55-80%,** dependent on specimen matrix
**Specificity** averaged **80-90%**
**False negatives** –  risks are life threatening
**False positives** –  risk of masking serious underlying disease

**Notes:**
In addition to some other review issues, CDRH felt-  risks associated with both false negatives and false positives were too great.

## Indications for Use: What Drives the Review

- Need to know who, what, when, and why
- Special conditions for use that impact performance
- Contraindications
- Drives:
- Protocol requirements
- Questions raised
- Data required

**Notes:**
Before a review can begin, it is critical to know the who, what, where and whens of a device.
The Indications drive the review process......

## Three Review Elements

- 1.  Is the device **accurate** and **reliable** in the hands of lay-users?
- 2.  Is the device adequately **labeled**?
- 3.  Do **benefits** outweigh the **risks**?

**Notes:**
As I stated earlier, there are three main review elements, and we’ll go through each of them.... by CDRH....

## First Review Element

- Is the device accurate and reliable in the hands of the user?
- Can the test provide the right answer?
- Can the lay user get the right answer?
- Human Factors (HF)
- Risk Mitigation & Safeguards
- Stress Studies

**Notes:**
We’ll cover a number of subjects as we talk about this element.......

## Can the test provide the right answer?

**Analytical and Clinical Performance**
- From initial professional-use submission
- Analytical and Clinical Sensitivity and Specificity
- 	Accurate (true) result
- Quantitative -Bias, total error
- Qualitative- Clinical Sensitivity and Specificity, Predictive Values

**Notes:**
Typically – first thing-  the performance characteristics of a device under optimal conditions.
Because almost all of our OTC applications were previously cleared for professional use, this information is already available.
So- not going to spend.  But will run thru the primary parameters......

## Can the test provide theright answer?

- Other Important Parameters
- Cross-reactivity & Interference
- Reproducibility
- Reference intervals
- Environmental factors, stability, reportable range, calibration, linearity, lot to lot variability, etc.

**Notes:**
Here are some of the other parameters we look at:
Cross-Rx and interference are especially important with OTC, false pos and neg
Environmental conditions are also important with OTC devices, both in terms of storage req. and testing req.
Not unusual for consumers to leave a device in a hot car, or leave a kit out when it should be refrigerated.

## Can the lay user get the right answer?

- Consumer Study Goal
- Demonstrate comparable performance when used as intended
- Representative Lay Users
- Enrollment criteria, demographics
**“****Home”** environment
- Evaluate entire test system
- Consumers perform _all_ steps, unassisted
- Labeling – Representative labeling, or according to marketing plan.

**Notes:**
Once we know what a test is capable of doing under optimal conditions,
We want to see if lay users can achieve a comparable level of performance at home.
Usually- Consumer Study
To evaluate a test being used as intended , look at three ......
A home is not always a home-
Mall
isolated area - dr office or clinic

ENTIRE test system, from beginning to end 
Unassisted 
only PI. 
But the key is that the device is used according to how it will be used when its marketed.  So....
 educational information,              
training (e.g., video) 
help line 
important - consumers don’t get anything more 

## Consumer StudySample Considerations

- Clinical or spiked –
- Challenging and span expected range
- Prospectively collected if:
- establish clinical sensitivity and specificity
- demonstrate ability to collect the sample, or
- sample is not stable
- Biohazard
- Properly characterized- WHO, CDC, reference method, or to another device

**Notes:**
Samples used in the study- perhaps one of the areas we struggle with the most

Whether the samples are clinical samples or if they prepared in some way- 
Concentration- should both challenge the cutoff or the medical decision making point(s) of the assay.
And span the expected range 
Sometimes difficult- clinical samples

If prepared- 
- Both the matrix and what is being spiked into it- act the same as clinical 

Samples may need to be prospectively collected if.....

Also to be considered– don’t want to endanger participants by having them handle bio-hazardous materials

Last important consideration for samples is whether- properly characterized?

## Consumer StudyProtocol Considerations

- Comprehensive
- Enough participants and samples
- Correct Statistical Tools
- Masking and randomizing- avoid bias
- Written prior to study

## Human Factors (HF) 

- A discipline concerned with how people interact with the device
- Why are HF so important?
- Users- varying abilities and behaviors
- Impact on safety and effectiveness

**Notes:**
Another important area we consider is HF.

The little things users do or don’t do.
Why 
Because people have varying abilities, both physical an mentally
And we each have behavior patterns:
Everyone has-  like not reading the directions to their VCR
Particular to an individual-  being color blind

And those tendencies that we all have can impact S&E

## Human Factors

- “Errors” occur when users:
- Fail to perform necessary actions
- Add actions
- Choose the wrong action
- Perform actions poorly

**Notes:**
These “errors” or events occur when..........

Performing  poorly – not seeing well, or having poor technique

## Human Factors Who is responsible for “user” errors?

**Human Factors ****Who is responsible for “user” errors?**

- “The instructions clearly state that results are to be read from right to left.”
- “How was I to know they’d read it left to right?”

**Notes:**
Who’s responsible?
Sometimes we find that sponsors don’t accept the responsibility as is illustrated in this example.
But device manufacturers need to anticipate all “reasonable” errors.
...........................NEXT

## All user errors cannot be prevented

**All user errors cannot be prevented**
**but some can be avoided**

**Notes:**
Not all errors can be prevented...

We’ve seen- 
-diabetics have cut their test strips in half to save money.  No one- anticipated, but....
also 
Kit has 2 btls of reagent with two different color caps.

##  Risk Mitigation & Safeguards What do we expect regarding errors?

- Effective procedures to address
-  HF and other potential problems:
- Hazard Analysis: Identify potential errors and estimate their risk
- Mitigate Risks:  Eliminate or reduce risks to acceptable level
- Verify effectiveness of safeguards

**Notes:**
So what does CDRH expect from sponsors when it comes to potential errors of a device?
Not only HFs, but others – 
-battery loosing its chanrgep
-mechanical or electrical failure

We look for a systematic approach to these issues:
1. By conducting a thorough Haz Analysis sponsors can identify, and estimate the risk associated with that potential problem
2. Then to mitigate that risk by some mechanism to either eliminate or reduce risk- acceptable level
3. And after they have mitigated the risk, verify that the mechanism is effective.

## Risk Mitigation & Safeguards Examples 

- Built-in features to detect:
**Short sample **
**Wrong sample**
**Reagent integrity**
- Internal QC- often limited scope
- Lock-out mechanisms?
- Labeling- techniques, but when is it adequate?
- Goal- alert **before** results are compromised

**Notes:**
Some examples of what a sponsor might do to mitigate a risk are to build features into their device that detect: 
Short sample 
Wrong sample
Reagent integrity
Hardware failures
Environmental and storage conditions

Some have temperature strips that stay with the device, and show if it was shipped or stored improperly

May have an internal QC- typically monitor only one aspect of the device.
Example- immunochromatographic single-use tests –only volume

Lock out mechanisms to turn off the device or make it unusable if some parameter is not working properly or if QC wasn’t run, or was out of range

Labeling is often used as a way to mitigate the risks of a device.
Various techniques, bolding, highlighting, or blocking that are known to bring information to a users attention
Often difficult to know whether labeling will be an adequate mitigation until it goes on the market.
Before- short sample detection system –not work until almost half.

## Risk Mitigation & SafeguardsQuality Control

- Is External QC
- Available and practical?
- Adequate?
- Needed?
- Will consumers do it ?!!

**Notes:**
External QC, although beneficial and useful in professional settings,  not always a good fit.

Not always adequate - provide assurance that the result is correct.  
And even if it did monitor the entire analytical process, NOT correct sample collected or handled properly
False sense of security.

How critical is it to have an external check?
Great technological advances with devices, and...
Many argue that with the advent of single use, disposable devices, external QC is not the answer 
Can’t verify performance of the device the user is actually using to test themselves.

And even if you built the best external QC mouse-trap in town, if doesn’t mean that consumers would do it.

##  Stress Studies Demonstrating Effective Mitigation

- Need to verify effectiveness of safeguards
- What are stress studies?
- Challenges to the system in order to characterize performance under various conditions of use

**Notes:**
After a sponsor describes their mechanism for addressing a potential problem, how do we know its effective?

CDRH asks sponsors to provide evidence that it is effective.  
Often done with stress studies.....

What are.....................

For instance, if the directions say to wait 10 minutes, what happens when the user reads it early because they’re late for an appointment?

## Stress Studies How are they performed?

- Artificially simulate potential errors
- Concentrate on areas prone to errors or failures
- Technique
- Follow directions
- Deviations in environmental conditions
- Observe results

**Notes:**
Simulate deviations from expected perfect condition: 
-battery loosing its charge
-environmental conditions. (got delayed at the mall and kit was in the hot trunk for 3 hours)

Technique- shaking something instead of inverting 
subtle variations- not think about or realize

Examples..........

## Stress Study Examples

| Instructions | Type of Study |
| --- | --- |
| Procedure- store kit between 60-80 °F  | Store kit as directed.   Store the kit with intermittent freezing,
50 and 90 °F then run samples.   |
| Procedure- add 3 drops of sample.  | Perform the test by adding 2, 3, 4, and 5 drops of sample.  |

**Notes:**
Compare results to expected

If small deviations and big affect- may not suitable

Compare the results observed when deviations from instructions are simulated to
Results from when it was performed properly.
How significant is the difference?

Really looking at how much deviation from the instructions had to occur before you had.....
Have a clinically significant difference in the result?

## Second Review ElementIs the device adequately labeled?

- Are the step-by-step instructions simple, concise, and easy to follow?
**7****th**** grade reading level?**
- Do consumers know who should use the device and when?
- Clear instructions for interpretation and follow-up?
**Limitations**?
- Go to a doctor?
- What if it didn’t work?
- Repeat test?

**Notes:**
Are steps or reading results counter-intuitive
Immunochromatographic tests-  positive is the absence of a test line, line= negative

Concise- unnecessarily detailed and long- not read
Look for there to be as few words as possible to get the message across.
......
Limitations- 
Tests are not perfect”
Effects from meds or combinations
Cannot identify all possible interferences
- if inconsistent results- don’t assume correct

Great example of people not knowing the limitations of a device- hand held glucose meters.
Did you know it was dangerous to use a meter when you’re really sick?
If you are a critical care nurse or a diabetic, you might want to look at the limitations in the package insert...........

## LabelingHow do you know its adequate?

- Verified-Consumer Study
- Value is Limited
- “Trial effect”
- Will they read it?
- Questionnaire
- Completed by consumers after testing
- Must be challenging:
- 	Who, what, where, when, how & why?

**Notes:**
How do you know the directions are good enough?
Look to consumer study- but recognizing it has limits:
Studies typically evaluate a relatively small number of subjects.
Often we observe a “trial effect”.....when users are extra careful because they know they are either being watched or their results will be scrutinized
AND- WON’T READ

Ask challenging , not open ended!
NOT:
- have trouble performing?
- understand your test result? 

YES:
what would you do if..???  Got a particular test result, or 
how  affect interpretation IF LAST TIME?

## Third Review ElementRisk vs. Benefit Consideration

**Lay users are unique**:
**Unregulated environment**
**May lack medical and technical training**:
- evaluate other medical information
- collect and transport samples
- read and follow directions
**May or may not carry out follow-up actions**
**Highly ****variable**

**Notes:**
This brings us to the third and final element of our review.
As we consider R and B, we must also keep in mind that lay users are unique individuals......

Evaluate other... like results from other tests or from a physical exam.

Read and follow- language or comprehension barriers

One thing for certain- there is a broad spectrum of lay users in terms of their knowledge and skill level.
From those that are reading and functioning at a low level to...
-internet junkies reading everything or those....
-have jobs that require attention to details or needing fine motor skills.

A good test- one that was appropriate for use by all lay users.  

## Risk vs. Benefit

- Risk
- 1.  **What is the impact of a false-positive or false-negative result on the user and  public health e.g., no follow-up, adverse medical conditions, unnecessary treatment?**
- 2.   **What are the medical risks to the user and  public health if there is a delay in obtaining a professional examination and treatment, or if the device gives an equivocal result?**

**Notes:**
two questions perhaps capture 
Consider MEDICAL risks and benefits to both individuals and to the public.

## Risk vs. Benefit

- Benefit
**1.**  **What is the clinical benefit to the patient and public health in terms of  screening, diagnosing, or monitoring a particular disease or condition, or risk factor?**
** ****2. What are the medical benefits to the patient and public health in having the test available for home-use as opposed to having the test performed only by health care professionals?**

## CONCLUSIONThe Final Decision

- Is the device accurate and reliable in the hands of the lay user?
- Does the Labeling adequately convey all  information necessary to use the device safely and effectively?
- Do benefits outweigh risks?
- If yes, generally found suitable for home-use

**Notes:**
In the end, after reviewing all of the evidence, CDRH returns to the same three questions to make their decision:
Whether its accurate and reliable enough
 labeling is good enough, and
there are more benefits than risks????
If the answers to these three questions are yes, the device will generally be found suitable for home-use.