**Qualitative Methods and Rigorous Management Research:**

**(How) Are They Compatible?**

Brian S. Mittman, PhD

Senior Social Scientist

Center for the Study of Healthcare Provider Behavior

VA Greater Los Angeles Healthcare System (152)

16111 Plummer Street

Sepulveda, California 91343

> Tel: 818-895-9544
>
> Fax: 818-895-5838
>
> Email: <mittman@rand.org>\
> <brian.mittman@med.va.gov>

##### November 2001

White paper prepared for the Department of Veterans Affairs *Management
Research in VA Workshop*, sponsored by the HSR&D Management Decision and
Research Center

November 19-20, 2001

**Qualitative Methods and Rigorous Management Research: (How) Are They
Compatible?**

**Introduction**

The role, benefits and appropriate use of qualitative research methods
in the basic and applied social and clinical sciences have been
discussed extensively in the research literature. The field of health
services research, in particular, has benefited from several insightful,
comprehensive discussions of qualitative research methods and their
appropriate use.[^1] Proponents have convincingly argued that
qualitative methods contribute findings and insights that cannot be
derived from \"conventional\" or \"quantitative\" research methods, and
that research in the clinical, social and policy sciences requires
careful application of both types of approaches to properly study their
phenomena of interest

Although most discussions of qualitative research methods (including
qualitative data collection and analysis methods) are careful to avoid
over-simplifying or stereotyping, the dominant conception of qualitative
methods continues to equate these methods with research perspectives and
study goals often labeled \"non-traditional\" or \"unconventional.\"
Included in this broad category are exploratory or hypothesis-generating
(inductive) studies, interpretive research, historical research and
several other forms of knowledge creation distinct from deductive,
hypothesis-testing research conducted within the positivist tradition.
This view of qualitative methods is not without foundation: most authors
view inductive, interpretive and related applications of qualitative
methods as their strength and area of unique contribution, given their
superiority over quantitative methods in developing insights into
actors\' values, beliefs, understandings and interpretations of events
and other phenomena, or in explaining historical occurrences. Despite
these strengths, however, the contributions of qualitative methods to
deductive research are no less significant or unique, and are no less
important nor valuable in these realms than in their traditional fields
of usage. The contributions and role of qualitative methods in deductive
research are often overlooked, however, or fall short of their potential
when they are applied, due to shortcomings in their use. These
shortcomings (1) limit the value contributed by qualitative methods and
(2) undermine perceptions of their importance and applicability, further
contributing to a cycle of under-use and misuse.

This paper addresses issues in the use of qualitative research methods
in hypothesis-testing, deductive research (sometimes labeled
\"conventional\" empirical research). The paper aims to illustrate how
and why such research can benefit from increased application of
qualitative methods, and how these methods can be used in a manner
consistent with accepted standards of rigor and validity. The paper is a
motivated by the belief that researchers who are not trained in
qualitative methods (and are therefore accustomed to conducting
empirical research using quantitative methods alone) are less likely to
be interested in applying qualitative methods in inductive or
interpretive research, but can---and should---be interested in applying
qualitative methods to enhance the \"conventional\" forms of empirical
research they are already conducting. The paper is further motivated by
the observation that qualitative methods are too often applied
inappropriately when used as a complement to quantitative methods in
hypothesis-testing studies. The consequences of this misuse include
threats to the validity of the study and its findings, as well as
threats to the reputation or perception of qualitative methods as a
valuable set of tools and approaches for a diverse set of researchers
and research projects.

The paper does not attempt to provide an overview or tutorial in
qualitative methods (given the existence of numerous excellent books and
papers performing this role), nor does it discuss \"mainstream\"
applications of qualitative methods in exploratory, interpretive or
historical research. Instead, the paper has a far more modest aim,
discussing, illustrating and advocating for the appropriate application
of qualitative methods in hypothesis-testing research---including the
types of studies often conducted in management research and in its
reference disciplines in the social and behavioral sciences. A few key
examples of these methods are listed in an Appendix.

**Management research, qualitative methods and rigor**

Threats to the validity and value of qualitative methods within
management research arise throughout the entire process of their
application, including (1) study design and conceptualization, (2) data
collection instrument and protocol design and development, (3) data
collection, and (4) analysis and interpretation. Most of these threats
relate to need for rigor and for explicit, *a priori* goals, plans and
implementation of qualitative methods. When harnessed for inductive,
interpretive research, qualitative methods are typically used in a
flexible, emergent manner, in which *a priori* specification of
concepts, measures and data sources---and explicit limitations in the
domains addresses and questions asked---would act to impede discovery of
important phenomena and insights, thereby weakening achievement of the
research goals. In such applications, qualitative analysis methods rely
on formal, explicit techniques as well as less formal, implicit,
intuitive interpretation. In deductive research, however, over-reliance
on emergent, informal application of qualitative research methods and
implicit analysis methods typically weakens their value, often producing
inconsistent data and results with questionable validity.

*Study design and conceptualization*.

When applied to conventional, hypothesis-testing research, qualitative
methods must be used within a study design and framework incorporating
each of the key elements required in studies involving only quantitative
methods. These elements include careful reviews of relevant theoretical
and empirical research, derivation or development of formal hypotheses
within an explicit theoretical framework, and *a priori* specification
of variables and measures. Theory-based hypotheses specify expected
causal relationships, guide identification of the specific variables
deemed relevant and requiring measurement and guide analysis and
interpretation of data and results. Reviews of relevant literature also
provide guidance in conceptualization and measurement of key variables,
including identification of specific data sources, measurement tools
with known characteristics and guidance in use of these tools.

Failure to adequately specify hypotheses and relevant variables
typically results in unfocused measurement and analyses, including a
failure to identify and measure key variables and concepts, and a
failure to recognize or to properly interpret important findings. While
use of formal hypotheses is not consistent with the use of qualitative
methods in inductive or interpretive research, it is critical in the use
of these methods in deductive research, whether this research involves
quantitative data and analysis methods or qualitative methods.

*Data collection instrument/protocol development*

Data collection instruments and protocols in qualitative research are
often informal, flexible and subject to large variations in application.
While flexibility represents a strength in traditional qualitative
research, it produces unfocused data collection and variable data
quality when qualitative methods are applied in deductive research. For
example, interview guides specifying general topics of interest, using
broad, open-ended questions can be very effective in assessing interview
subjects\' assessment of important concept and issues and their beliefs
and values, but ineffective in ensuring that comparable measures of
identified variables are collected from a range of subjects (e.g.,
assessing organizational participants\' views or their ratings of
concepts or variables deemed important by the research team). In part,
the distinction here is between data collection approaches designed to
develop frameworks for understanding and describing the phenomena of
interest, versus applying *a priori* frameworks to collect pre-defined
data and test aspects of these frameworks. Similar problems result from
the use of observation guides or protocols lacking adequate specificity
and a firm foundation in *a priori* hypotheses and clearly-identified
variables: such protocols often produce inconsistent data by (1)
encouraging the observer to record events as they unfold and to record a
wide range of attributes of the situation under study (whether or not
they are deemed relevant to the hypotheses of interest), (2) limiting
the likelihood that the observer will note the significance of events
that do not occur, and (3) limiting the likelihood that the observer
will collect complete, consistent data required for direct comparisons
across observation samples.

Considerations of validity, intrusiveness or subject reactivity
(Hawthorne effects) and triangulation (to minimize bias) are also
too-often neglected in deductive applications of qualitative methods.
Distinctions between subjective and objective data and between formal
and informal organizational structures and processes are also frequently
neglected, threatening the validity of study conclusions.

Avoiding these problems requires careful design of data collection
plans, based on study goals and hypotheses, and involving use of
systematic tables or other methods for specifying key variables and
suitable, multiple measures. Depending on the importance of each
variable and the validity of available measures, two or more data
sources are typically needed in qualitative research. Data planning
tables listing concepts or variables, definitions and data sources are
effective in ensuring appropriate rigor; data collection instruments
(including document coding forms, survey questions and other data
specifications) can be developed directly from these tables.

### Rigor and validity are also enhanced through development and use of data collection instrument specifications and training protocols, including variable and measure definitions and instructions in instrument use. When used in management research, such protocols should include plans and instructions for approaching sites, making contacts, arranging interviews/visits, identifying and obtaining documents, following-up (to obtain documents and other post-visit/call information), managing informed consent and confidentiality, etc. Adequate pilot testing helps ensure the appropriateness of the data sources and measures, although data collection protocols must be flexible and allow for changes in data collection plans and strategies, when pilot testing fails to reveal valuable new data sources or validity problems with the sources in use.

Finally, study validity is further enhanced through development of data
analysis protocols and plans together with the actual instruments,
rather than after completion of data collection. Data planning tables
created to guide data collection activities can be used to develop data
reporting templates and specifications for translating raw data into
variables and preparing for analyses; management data are often reported
in a standardized "organizational profile" or other comparative format.
These profiles store raw data and summary variables from all data
sources, which are then converted to tables for analysis.

*Data collection and data management*

Use of qualitative methods, including interviews and observation, is
subject to wide variations and interviewer/observer bias and
interpretation. Steps to minimize these biases include adequate training
of data collection staff; comprehensive plans for data collection,
validation and storage; and frequent reviews of data quality and
interpretation. Data collection and management plans should include
immediate post-collection coding and review of data that are time- or
memory-sensitive (e.g., interviews and observation). While data validity
and completeness can be enhanced through tape recording of interviews,
other methods may be more cost-effective, including real-time survey
data entry and editing, use of paired interviewers, post-interview
debriefing, and other methods. Quality assurance methods should be
considered and operationalized for each instrument and data sources.
Problems such as incomplete, missing, unusable data should be identified
and resolved during the data collection phase, rather than after its
completion.

*Analysis and Interpretation*

Analysis of qualitative data should be guided by the pre-specified,
model-based hypotheses and detailed analysis plans developed at the
outset of the study. Unfortunately, while quantitative analysis methods
are well-established and accepted, methods for analysis of qualitative
data are subject to variability and lack of consensus. Analyses of
qualitative data are too-often informal, ad-hoc and emergent, with low
reliability and validity. These threats can be countered through the use
of formal table approaches, in which key variables relevant to each
hypotheses are listed in tables and manipulated in a blinded fashion,
using qualitative pattern-identification and non-parametric quantitative
techniques. The analysis tables summarizing and synthesizing information
from diverse sources in a standardized format may also serve as
reporting tools, in papers and reports.

Analysis and interpretation follows the study hypotheses and research
questions, but will often include detailed causal explanations and
exploratory questions and findings as well, taking advantage of
qualitative data\'s value in these areas. Combining the use of
qualitative methods for hypothesis-testing and interpretive, inductive
applications in this manner represents a powerful application of these
methods, using their strengths to enhance management studies and other
empirical research in important ways.

**Conclusion**

The origins and development of qualitative research methods and their
close association with inductive, interpretive and historical research
have led many researchers to associate these methods exclusively with
these forms of research, and to fail to recognize their value in
conventional deductive empirical research. Yet the rigorous application
of qualitative methods in deductive research requires many modifications
and adaptations. Qualitative methods, when properly applied, can
contribute significant value to management and organizational research
and research in health services more generally.

**Appendix.**

**Outline of key qualitative research methods**

**Mail, telephone surveys**

### Effective for brief, fact-oriented, simple short-answer questions

### Do not work well for exploratory, open-ended, inductive questions

### Facilitate consistency, objectivity, completeness

### Mail vs. telephone considerations include development vs. administration effort

**Telephone, in-person interviews**

### Required for elite interviews, opinion and long-answer questions

### Infeasible for high-volume data collection

### Reduce consistency, objectivity, completeness

### Telephone vs. in-person considerations include sensitivity of information, need for candor; need for stories; risk of bias, social desirability, cues; interview-interviewee relationship

**Collection and analysis of documents and other archival,
administrative data**

### May be more objective, consistent, complete, but incompleteness is more typical

### Interpretation and use require knowledge of creation conditions and factors

### Explanatory or supplementary interviews are generally needed

**Observation**

### Observation ranges from walk-throughs (to assess physical layout and facility characteristics), to attendance/observation at meetings, in routine work settings

### Observation is generally very expensive, often infeasible

### Observer's biases, organization's reactions to observers may affect validity of data, yet observation complements the biases, validity problems inherent in respondent reports

[^1]: Several excellent references are provided in the Workshop
    bibliography, including the special issue of *Health Services
    Research* devoted to \"Qualitative Methods in Health Services
    Research\" (Volume 23, No. 5, Part II, December 1999). Additional
    references are available upon request. The current draft of this
    white paper does not explicitly reference these sources, although it
    draws heavily upon many of them.