©2013 University of Illinois at Chicago Board of Trustees

Systematically Gathering Clinician 

Opinions: Linking User Needs to 

Health Care Innovation

Melissa Naiman, PhD, PMP, EMT-B 

Associate Director, Science & Technology, CADRE

Research Assistant Professor, School of Public Health

Research Program Manager, Advanced Cooling Therapy, LLC
©2013 University of Illinois at Chicago Board of Trustees 2 

Introduction to Q

 Q-methodology

 Mixed method technique

 Dependency analysis

 Multivariate analysis with no a priori 

variable definitions

 Measures “significances” between 

people 
©2013 University of Illinois at Chicago Board of Trustees 3 

Audience Poll

 #1: Are you familiar with factor analysis? 

 Yes

 No

 #2: Have you ever heard of Q-methodology? 

 Yes

 No
©2013 University of Illinois at Chicago Board of Trustees 4 

History of Q

 William Stephenson

 Technique of factor analysis. Nature, 1935: 136, 297. 

Stephenson Thomson
©2013 University of Illinois at Chicago Board of Trustees 5 

Multidimensional Continuum 

of Research Projects
©2013 University of Illinois at Chicago Board of Trustees 6 

Key Psychometric Distinctions

Q R 

Dependency analysis Interdependency analysis

Unit of Measure: Persons Unit of Measure: Tests

Measures Wholes Measures Characteristics

Population = Statements Population = People

Similarities Differences

No generality assumption Generality assumption

Variates may interact Variates do not interact

Requires forced choice Avoids forced choice
©2013 University of Illinois at Chicago Board of Trustees 7 

Using Q to Study Health Care Innovation

 Technology use in healthcare settings is a public health 

issue

 Technology Purchase ≠ Technology Use
©2013 University of Illinois at Chicago Board of Trustees 8 

Technology Adoption Variables

Organizational 

Variables

Individual

Variables

Attitudes Toward

Innovation

Trust Communication

Pathway

Ease 

Of Use

Usefulness

Compatibility

Function

Tasks
Project Rationale

Advocates for including clinician optechnology assessment:

Institute of Medicine1

American Medical 

Association2

Amer

of P

©2013 University of Illinois at Chicago Board of Trustees 

of P

1 Donaldson M, Sox H. Setting priorities for health technology assessment: A model process. Na

1992.

2 Jones RJ. JAMA. 1983, 250(3) 387-3883 Feussner JR, White LJ. Ann N Y Acad Sci. 1993 Dec 31;703:268-71.
©2013 University of Illinois at Chicago Board of Trustees 10

Research Questions

1. Can clinician opinions be used to identify radical 

innovations?

2. Can clinician opinions guide prioritization of radical 

innovations?

Radical Innovation = major departures 

from standard practice that may change 

workflow or professional roles. 
©2013 University of Illinois at Chicago Board of Trustees 11

Study Setting

 Advocate Christ Medical Center Emergency Department

 Situated within a 683-bed teaching hospital

 Level I Trauma Center designation
©2013 University of Illinois at Chicago Board of Trustees 12

Study Design

1. Concourse Development

2. Q-set/Condition of Instruction

3. Complete Q-sorts

4. Data Analysis

12
©2013 University of Illinois at Chicago Board of Trustees 13

Concourse Development

1. One-on-one Interviews

 Favorable technology characteristics

Focus Groups

Clinical challenges

Market Analysis

Identify products that embody 1 & 2

2.



3.



13
©2013 University of Illinois at Chicago Board of Trustees 14

Concourse To Q-set

14 

1. POC cardiac marker 

panel for 4 markers.

2. Ultrasound probes plug 

into Smartphones.

5. A portable head & neck 

CT scanner.
©2013 University of Illinois at Chicago Board of Trustees 15

Q-Set Excerpt

15 

1. POC cardiac marker 

panel for 4 markers.

2. Ultrasound probes plug 

into Smartphones.

5. A portable head & neck 

CT scanner.

4. A patient-controlled 

mobile health record.

6. Drug delivery through 

a patch.

3. POC test for sepsis 

risk. 7. POC metabolic panel.

8. A mobile app that 

assigns tasks.

9. A fully wireless 12-lead 

ECG. 
©2013 University of Illinois at Chicago Board of Trustees 16

Condition of Instruction

When thinking about technology and techniques to support 

improving care in the emergency department, which of the 

following do you feel would be most likely / most unlikely 

to improve the care you provide?
©2013 University of Illinois at Chicago Board of Trustees 17

9. A fully wireless 12-lead 

ECG. 

8. A mobile app that 

assigns tasks.

4. A patient-controlled 

mobile health record. 7. POC metabolic panel.

Initial Sort

Most Likely to 

Improve Care

2. Ultrasound probes plug 

into Smartphones.

5. A portable head & neck 

CT scanner.

6. Drug delivery through 

a patch.

3. POC test for sepsis 

risk.

Most Unlikely to 

Improve Care

1. POC cardiac marker 

panel for 4 markers.
©2013 University of Illinois at Chicago Board of Trustees 18

Most Unlikely to 

Improve Care

Most Likely to 

Improve Care

Final Sort

18 

1. POC cardiac 

marker panel for 

4 markers.

2. Ultrasound 

probes plug into 

Smartphones. 

5. A portable 

head & neck CT 

scanner. 

4. A patientcontrolled 

mobile health 

record.

3. POC test for 

sepsis risk.

7. POC 

metabolic panel.

8. A mobile app 

that assigns 

tasks. 

9. A fully wireless 

12-lead ECG.

6. Drug delivery 

through a patch.
©2013 University of Illinois at Chicago Board of Trustees 19

Correlation Matrix

Sorts 1 2 3 4 5 6 7 8 9 10

1 100 20 41 14 -11 24 18 -7 56 31

2 20 100 24 10 21 -10 33 45 37 15

3 41 24 100 -5 12 49 30 10 42 22

4 14 10 -5 100 -26 -4 17 22 13 -4

5 -11 21 12 -26 100 9 26 27 23 31

6 24 -10 49 -4 9 100 35 8 39 49

7 18 33 30 17 26 35 100 27 26 32

8 -7 45 10 22 27 8 27 100 27 29

9 56 37 42 13 23 39 26 27 100 35

10 31 15 22 -4 31 49 32 29 35 100

Person

Person

PQMethod v 2.32
©2013 University of Illinois at Chicago Board of Trustees 20

Data Collection

 Convenience sample

 40 participants

 13 MD, 24 RN, 3 undisclosed

 At least 50% FTE

 Recruitment strategy

 E-mail 

 In person

20

Emergency Department 

Currency
©2013 University of Illinois at Chicago Board of Trustees 21

Data Analysis

 Factor solution

 A priori conditions 

 Statistically convincing

 Inclusive

 Favor senior staff members

 Specifically, senior MDs

Model Significant 

Sorts

Senior 

Clinicians

Senior MDs Variance 

Explained

4 Component PCA 33 11 5 53

3 CentroidTheoretical 

Rotation

33 10 4 42

3 Centroid Varimax 

Rotation

30 11 4 42

3 Component PCA 29 10 3 46
©2013 University of Illinois at Chicago Board of Trustees 22

Factor Demographics

– Physicians and nurses aligned together

– Senior Staff (>10 yrs experience) aligned to 3 

of 4 factors

– Self-described innovation style in first half of 

adoption curve

Total MD RN Avg. Exp. >10 IN EA EM

1 11 3 8 10.6 5 2 3 5

2 4 3 1 4.5 0 0 2 2

3 8 1 6 8.6 3 1 0 5

4 10 4 7 13.3 4 0 3 7

Seniority Innovation Style

Factor

Count
©2013 University of Illinois at Chicago Board of Trustees 23

Individual Perspectives

 Innovation Profiles

23

– Speed Oriented (33%)

• Highest priority: fast access

– Holism Oriented (12%)

• Highest priority: patient experience

– Acuity Oriented (24%)

• Highest priority: sickest patients

– Information Oriented (30%)

• Highest priority: connectivity

SPEED (33%) HOLISM (12%) ACUITY (24%) INFO (30%)
©2013 University of Illinois at Chicago Board of Trustees 24

Factor 1: Most Likely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

A government-controlled health record. 3 2 -3 -1 

A patient-controlled mobile health record. 3 2 0 3 

POC cardiac marker panel. 3 -2 3 2 

POC test for stroke risk. 3 -1 2 0 

POC test for sepsis risk. 2 -3 1 2 

POC test for ectopic pregnancy risk. 2 -1 -1 1 

POC metabolic panel. 2 0 2 2 

POC lactate monitor. 2 -1 2 1 

POC test for H&H. 2 0 1 0 

Lab-based test to confirm stroke. 2 -1 3 -1 
©2013 University of Illinois at Chicago Board of Trustees 25

Factor 1: Most Unlikely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

A fully wireless 12-lead ECG. -3 -1 0 1 

A cream that delivers drugs through the skin. -3 1 -2 -1 

The ED doesn’t need new imaging technologies. -3 -3 -3 -3 

We don’t need POC tests. -3 0 -3 -3 

Vest that transmits 5-lead ECG & vital signs. -2 -1 -2 0 

Drug delivery through a patch. -2 -1 -2 -1 

A needleless injection system. -2 2 -1 0 

A portable full body CT scanner. -2 3 2 -2 

Pocket ultrasounds carried by all ED physicians. -2 2 0 -2 

Robots that locate and retrieve equipment. -2 -2 -3 -1 
©2013 University of Illinois at Chicago Board of Trustees 26

Factor 1 Interpretation

 Speed Oriented

 Fast access to blood levels

 Quantitative results

 Fast access to history
©2013 University of Illinois at Chicago Board of Trustees 27

Factor 2: Most Likely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

A portable full body CT scanner. -2 3 2 -2 

A portable head & neck CT scanner for the ED. 0 3 3 -3 

A mobile app that assigns tasks. 1 3 -1 3 

A mobile app that facilitates sharing charts. 1 3 0 3 

A needleless injection system. -2 2 -1 0 

Pocket ultrasounds carried by all ED physicians. -2 2 0 -2 

A portable x-ray machine dedicated to the ED. -1 2 0 -2 

Ultrasound probes plug into Smartphones. -1 2 3 3 

A patient-controlled mobile health record. 3 2 0 3 

A government-controlled health record. 3 2 -3 -1 
©2013 University of Illinois at Chicago Board of Trustees 28

Factor 2: Most Unlikely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

Lab-based test definitive ectopic pregnancy dx. 0 -3 -1 -1 

Lab-based test provides a definitive sepsis dx. 1 -3 1 1 

POC test for sepsis risk. 2 -3 1 2 

Robots that locate & retrieve shared equipment. -2 -2 -3 -1 

Video streaming of an EMS encounter. -1 -2 2 2 

A non-invasive sensor that measures H&H. -1 -2 -1 0 

A paper-based hand-off checklist. 0 -2 -2 -3 

POC cardiac marker test for 1 marker. 1 -2 1 0 

POC cardiac marker panel for 4 markers. 3 -2 3 2 
©2013 University of Illinois at Chicago Board of Trustees 29

Factor 2 Interpretation

 Holism Oriented

 Fast access to imaging

 Qualitative results

 Fast access to patient history and other clinicians

 Seek to decrease pain
©2013 University of Illinois at Chicago Board of Trustees 30

Factor 3: Most Likely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

POC cardiac marker panel for 4 markers. 3 -2 3 2 

Lab-based test to confirm stroke. 2 -1 3 -1 

Ultrasound probes plug into Smartphones. -1 2 3 3 

A portable head & neck CT scanner for the ED. 0 3 3 -3 

Video streaming of an EMS encounter. -1 -2 2 2 

POC lactate monitor. 2 -1 2 1 

POC test for stroke risk. 3 -1 2 0 

POC metabolic panel. 2 0 2 2 

Mobile app receives real-time ECGs from EMS. 1 1 2 2 

A portable full body CT scanner. -2 3 2 -2 
©2013 University of Illinois at Chicago Board of Trustees 31

Factor 3: Most Unlikely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

The ED doesn’t need new imaging technologies -3 -3 -3 -3 

Robots that locate and retrieve shared equipment -2 -2 -3 -1 

We don’t need POC tests -3 0 -3 -3 

A government-controlled health record. 3 2 -3 -1 

A paper-based hand-off checklist. 0 -2 -2 -3 

Vest that transmits 5-lead ECG & vital signs. -2 -1 -2 0 

Drug delivery through a patch. -2 -1 -2 -1 

A website that provides physician contact info 1 0 -2 0 

A cream that delivers drugs through the skin. -3 1 -2 -1 

Mobile app that supports video conferences -1 1 -2 2 
©2013 University of Illinois at Chicago Board of Trustees 32

Factor 3 Interpretation

 Acuity Oriented

 Faster diagnosis and monitoring for the “sickest”

 Improved monitoring of EMS activity
©2013 University of Illinois at Chicago Board of Trustees 33

Factor 4: Most Likely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

A mobile app that assigns tasks. 1 3 -1 3 

A mobile app that facilitates sharing charts. 1 3 0 3 

A patient-controlled mobile health record. 3 2 0 3 

Ultrasound probes plug into Smartphones. -1 2 3 3 

Mobile app that supports video conferences. -1 1 -2 2 

Video streaming of an EMS encounter. -1 -2 2 2 

Mobile app receives real-time ECGs from EMS. 1 1 2 2 

POC test for sepsis risk. 2 -3 1 2 

POC metabolic panel. 2 0 2 2 

POC cardiac marker panel for 4 markers. 3 -2 3 2 
©2013 University of Illinois at Chicago Board of Trustees 34

Factor 4: Most Unlikely to Improve Care

Factor 

Abbreviated Statements 1 2 3 4 

A paper-based hand-off checklist. 0 -2 -2 -3 

We don’t need POC tests. -3 0 -3 -3 

A portable head & neck CT scanner for the ED. 0 3 3 -3 

The ED doesn’t need new imaging technologies. -3 -3 -3 -3 

Blood purification system removes cytokines. 0 0 0 -2 

Blood purification system removes endotoxin. 0 1 -1 -2 

A portable x-ray machine dedicated to the ED. -1 2 0 -2 

A handheld x-ray machine. -1 0 1 -2 

A portable full body CT scanner. -2 3 2 -2 

Pocket ultrasounds carried by all ED physicians. -2 2 0 -2 
©2013 University of Illinois at Chicago Board of Trustees 35

Factor 4 Interpretation

 Information Oriented

 “Real-time” collaboration

 Faster diagnostics, even when slower tests are definitive 
©2013 University of Illinois at Chicago Board of Trustees 36

Departmental Perspective

 “Consensus” Innovations

 Positive 

 Ranked “Likely” (+2 or +3) to improve care by at least 2 Factors

 Ranked neutrally (-1 to +1) by remaining Factors

 Negative 

 Ranked “Unlikely” (-2 or -3) to improve care by at least 2 Factors

 Ranked neutrally (-1 to +1) by remaining Factors 
©2013 University of Illinois at Chicago Board of Trustees 37

Positive Consensus Technologies

Category Item # Product Inspiration Vendor

Health Communication 1 PerfectServe PerfectServe, Inc.

5 AirStrip CARDIOLOGY AirStrip Technologies

6 SurgiChart SurgiChart, LLC

8 MedXCom Giffen Solutions, Inc.

In vitro Diagnostics 14 Piccolo Xpress Abaxis

18 Lactate Scout + EKF Diagnostics

20 Cerebral Array I and II Randox Laboratories Ltd.

21 IschemiaCare Ischemia Care, LLC

Imaging 42 MobiUS SP1 MobiSante, Inc.

Factor 

Abbreviated Statements 1 2 3 4 

A mobile app that assigns tasks. 1 3 -1 3 

Mobile app receives real-time ECGs from EMS. 1 1 2 2 

Mobile app that facilitates sharing charts. 1 3 0 3 

A patient-controlled single health record 3 2 0 3 

POC metabolic panel. 2 0 2 2 

POC lactate monitor. 2 -1 2 1 

POC test for stroke risk & odds of tPA efficacy. 3 -1 2 0 

Lab-based test confirm stroke & tPA efficacy. 2 -1 3 -1 

Ultrasound probes plug into Smartphones -1 2 3 3 
©2013 University of Illinois at Chicago Board of Trustees 38

Negative Consensus Technologies

Category Item # Product Inspiration Vendor

Health Communication 3 Literature N/A

Non-invasive 

monitoring

26 “E-Bra” Prototype University of Arkansas

Treatment Options

33 Solid-in-oil nanodispersion

projects

Kyushu University

34 P.L.E.A.S.E. Professional Pantec Biosolutions

Prelude SkinPrep System Echo Therapeutics

Other 29 Automated TUG system Aethon, Inc.

Factor 

Abbreviated Statements 1 2 3 4 

A paper-based hand-off checklist. 0 -2 -2 -3 

We don’t need POC tests -3 0 -3 -3 

Vest that transmits 5-lead ECG & vital signs. -2 -1 -2 0 

Robots that locate and retrieve equipment. -2 -2 -3 -1 

A cream that delivers drugs through the skin. -3 1 -2 -1 

Drug delivery through a patch. -2 -1 -2 -1 

The ED doesn’t need new imaging technologies -3 -3 -3 -3 
©2013 University of Illinois at Chicago Board of Trustees 39

Applying the Results

 Departmental perspective

 Innovation strategy

 Individual profiles

 Change management (communication)

39
©2013 University of Illinois at Chicago Board of Trustees

Potential Applications in 

VA Women’s Health Research

Design 1: Prioritization/Strategy 

Design 2: Significance to Individuals

Design 3: Longitudinal 
©2013 University of Illinois at Chicago Board of Trustees 41

Sources

1. Yano EM, et al. (2011) “Using Research to Transform Care 

for Women Veterans: Advancing the Research Agenda and 

Enhancing Research–Clinical Partnerships”. Women's 

Health Issues 21-4S, S73–S83.

2. Bean-Mayberry B, et al. (2011) “Systematic Review of 

Women Veterans’ Health: Update on Successes and 

Gaps”. Women's Health Issues 21-4S, S84–S97.

3. Yano EM, et al. (2009) “Integration of Women Veterans 

into VA Quality Improvement Research Efforts: What 

Researchers Need to Know”. J Gen Intern Med 25(Suppl 

1):56–61.
©2013 University of Illinois at Chicago Board of Trustees 42

Benefit to VA Women’s Health Research

 Answer to the “Efficiency” Argument

 Q is multivariate analysis without a priori variable definitions

 Evaluating similarities, not differences

 No null hypothesis to reject

 Power is irrelevant

 Smaller sample sizes are acceptable (as in qualitative studies)
©2013 University of Illinois at Chicago Board of Trustees 43

Q Applications in VA Health Research Agenda

Excerpted from Table 4: E.M. Yano et al. / Women's Health Issues 21-4S (2011) S73–S83)

Main Topic Research Priorities

Access to care and

rural health

• Assess factors related to women veterans’ trust of VA and other providers and clinic 

environments. 

Primary care and 

prevention

• Evaluate VA comprehensive women’s primary care models (e.g., patient satisfaction, patient 

ratings of care)

Mental health • Examine structure and care models that support the patient-centered medical home.

• Understand similarities/differences between male and female Veterans with military sexual 

trauma, including barriers, needs and outcomes.

• Understand impact of mental health on sexual health and reproductive health over the lifetime.

• Determine barriers to caring for women who attempt/complete suicide.

• Identify risk factors for suicide among women Veterans.

• Evaluate variations in mental health care needs, use and outcomes of subgroups of women 

Veterans (e.g., racial–ethnic minorities).

Post deployment 

health 

• Evaluate functional status, quality of life, and resilience post-deployment, in addition to physical 

and mental health.

• Evaluate impacts of multiple deployments on women Veterans and their families.

• Develop combat exposure measure(s) that reflect women Veterans’ experiences.

Complex chronic

conditions/aging

and long-term care

• Understand the aging issues of women Veterans including needs, use, and preferences.

• Evaluate needs and care for disabled women Veterans.

• Determine reproductive health needs of women Veterans .

Reproductive health • Examine impacts of first experiences with reproductive health services (e.g., on perceptions of 

care, on later use).
©2013 University of Illinois at Chicago Board of Trustees 44

Design 1: Prioritization/Strategy

 Goal: Discover user profiles and group needs

 Similar to the Health Innovation study presented earlier

 Q-concourse 

 All possible interventions, structures, features, etc.

 Sources: literature, current practice, interviews

 Q-set

 Generic descriptions of interventions, structures, features, etc.

 Condition of Instruction

 Most needed/least needed, Most useful/least useful
©2013 University of Illinois at Chicago Board of Trustees 45

Design 1:

VA Women’s Health Research Priorities

Main Topic Research Priorities

Access to care and

rural health

Primary care and 

prevention

• Evaluate VA comprehensive women’s primary care models (e.g., patient satisfaction, patient 

ratings of care)

Mental health • Examine structure and care models that support the patient-centered medical home.

• Evaluate variations in mental health care needs, use and outcomes of subgroups of women 

Veterans (e.g., racial–ethnic minorities).

Post deployment 

health 

Complex chronic

conditions/aging

and long-term care

• Understand the aging issues of women Veterans including needs, use, and preferences.

• Evaluate needs and care for disabled women Veterans.

• Determine reproductive health needs of women Veterans .

Reproductive health
©2013 University of Illinois at Chicago Board of Trustees 46

Design 2: Significance to Individuals 

 Goal: Discover “ideal-types” within a population

 Use these to develop/tailor interventions

 Q-concourse 

 Experiential themes (derived from interviews, focus groups)

 Q-set

 Inductive design (mix and match across themes)

 Condition of Instruction

 Most like/most unlike me
©2013 University of Illinois at Chicago Board of Trustees 47

Design 2: 

VA Women’s Health Research Priorities

Main Topic Research Priorities

Access to care and

rural health

• Assess factors related to women veterans’ trust of VA and other providers and clinic 

environments. 

Primary care and 

prevention

Mental health • Understand similarities/differences between male and female Veterans with military sexual 

trauma, including barriers, needs and outcomes.

• Determine barriers to caring for women who attempt/complete suicide.

• Identify risk factors for suicide among women Veterans.

Post deployment 

health 

• Develop combat exposure measure(s) that reflect women Veterans’ experiences.

Complex chronic

conditions/aging

and long-term care

Reproductive health
©2013 University of Illinois at Chicago Board of Trustees 48

Design 3: Longitudinal

 Goal: Observe changes in perception/beliefs over time

 Combine with demographic data, analyze group and individual 

across time

 Q-concourse 

 Metrics of function (existing measures, preferably validated)

 Q-set

 Deductive design (to reflect population of metrics)

 Condition of Instruction

 Best Describes/Least Describes Me

 Data collection

 Repeat Q-sort over intervals
©2013 University of Illinois at Chicago Board of Trustees 49

Longitudinal Data Collection

2 

3 

1 

T1 T2 T3 
©2013 University of Illinois at Chicago Board of Trustees 50

Design 3:

VA Women’s Health Research Priorities

Main Topic Research Priorities

Access to care and

rural health

Primary care and 

prevention

Mental health • Understand impact of mental health on sexual health and reproductive health over 

the lifetime.

Post deployment 

health 

• Evaluate functional status, quality of life, and resilience post-deployment, in 

addition to physical and mental health.

• Evaluate impacts of multiple deployments on women Veterans and their families.

Complex chronic

conditions/aging

and long-term care

Reproductive health • Examine impacts of first experiences with reproductive health services (e.g., on 

perceptions of care, on later use).
©2013 University of Illinois at Chicago Board of Trustees 51

Audience Poll

 #3: Are you interested in using Q in the future?

 Yes (you had me at the Star Trek reference)

 Maybe 

 No
©2013 University of Illinois at Chicago Board of Trustees 52

Thank you!

 Contact information:

 Email: mnaima1@uic.edu

 Phone: 312-355-0078 

 Additional Reading

 Doing Q Methodological Research: Theory, Method & Interpretation 

by Simon Watts and Paul Stenner

 Q Methodology (Quantitative Applications in the Social Sciences) by 

Bruch McKeown and Dani Thomas