The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CACorrespondence: djohnson@operant-tech.com

Slide 1The Use andValidation ofPre-Intervention
Diagnostic Tools
in OBM

Douglas A. Johnson, Ph.D.
Heather M. McGee, Ph.D.
Western Michigan University

Slide 2Functional Analysis and Assessment
 Standard in many areas of ABA(Bailey & Burch, 2002; Iwata,Dorsey, Slifer, Bauman, & Richman,1982)o First assess maintaining
variableso Then identify targetedinterventions based on
results of assessment

Slide 3

Assessment in OBM
 Many choices
o ABC Analysis
o PIC/NIC Analysis
o Behavior Engineering Model(BEM)
o And others
 Most commonly seen in Journal of
Organizational Behavior
Management (JOBM) – Flagshipjournal of OBM
o Behavioral Systems Analysis(BSA)
o Performance Diagnostic
Checklist (PDC)

http://www.johnsonresearchlab.com

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 4Behavioral Systems Analysis
 Long history in OBM (Brethower,1972, 1982; Connellan, 1978;
Harshbarger & Maley, 1974;
Morasky, 1982)
 Subject of two special issues ofJOBM
 Consists of:o Series of maps, charts, andother tools
o Populated via observations,questionnaires and guided
interviewingo Aid in the pinpointing of
performance gaps incomplex environments andin guiding the subsequentchange efforts that result
from the identification of
those gaps

Slide 5

Performance Diagnostic Checklist Series of diagnostic questions
around four areas that affectperformance Based on study that investigatedthe process of question asking(Austin, 1996)
o Provided consultants andmanagers with a series ofperformance problems
o Asked them to talk aloud as
they tried to solve theperformance issueso Results showed mostsuccess occurred whenquestions were asked in fourareas
 Antecedents
 Equipment andprocesses Knowledge and skills
 Consequences

Performance Diagnostic Checklist involves aseries of diagnostic questions regarding

antecedents, equipment/processes,
knowledge/skills, and consequences

http://www.johnsonresearchlab.com

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan UniversityFebruary 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 6
• Provide the first comprehensive reviews ofBSA and PDC• Assess the degree towhich BSA and PDC havebeen validated via
well-controlled studies• Assess how effective each isin identifying and eliminating
performance gaps comparedto a control condition or otherbehaviorally based analysis
and improvement approaches

Slide 7
Reviewed all issuesof JOBM(vol. 1-31) throughthree levels of review

http://www.johnsonresearchlab.com

Purposes of Study Provide the first comprehensive
reviews of BSA and PDC exist
 Assess the degree to which BSA and
PDC have been validated via wellcontrolled studies
 Assess how effective each is in
identifying and eliminating
performance gaps compared to acontrol condition or otherbehaviorally based analysis andimprovement approaches

Method All issues of JOBM from 1977 thru
2011 (Volumes 1-31) reviewed Book reviews, announcements, andeditorials were excluded
 3 levels of review
o Two independent reviewers
recorded and scored eacharticle at each level ofreview
o Any classification
disagreements wereresolved by discussion andsubsequent independentrecoding

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 8• systems analysis First, searched for many terms• systems thinking
anywhere in the articles• processing system• receiving system• process maps
• relationship map• total performance system• performance matrix• super system• organizational scan• behavior engineering model• human performance system• Performance Diagnostic Checklist

Slide 9PDC or not?

Next, we reviewed theactual content of the article

http://www.johnsonresearchlab.com

First Level of Review
 Searched for the following terms:
o systems analysis
o systems thinking
o processing systemo receiving systemo process maps
o relationship map
o total performance systemo performance matrix
o super systemo organizational scano behavior engineering modelo human performance systemo Performance Diagnostic
Checklist

Second Level of Review
 All articles containing terms fromLevel 1 review were reviewed at
Level 2 by researchers with explicit
training and experience in both
Performance Management and BSA
 Careful examination of article
content to classify articles aso A systems article or not
o A PDC article or not Only articles that utilized either BSA
or PDC or included a substantial
discussion BSA or PDC wereselected at this level Brief mentions of BSA terms or PDC
was not sufficient

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan UniversityFebruary 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 10

Finally, we decided what typeof BSA or PDC article it wasSlide 11• Theoretical or Conceptual Articles
about BSA or PDC– Articles based purely at the level of discussion ortheory– Articles reporting hypothetical data
– Review articles (if they did not present previously
unpublished data sets)– Articles which were data-based but theintervention was not driven by a BSA tool

Slide 12• Minimal Empirical Data Involving BSA or thePDC– Articles presenting empirical evidence of use of
BSA or PDC, but utilized designs that prevent
establishment of cause and effect relations
regarding enhancements from the preintervention diagnostic tool

http://www.johnsonresearchlab.com

Third Level of Review
 All articles passing Level 2 review
were included in Level 3 review
 Two researchers classified articles
according to level of empiricalcomparison
 Any classification disagreements
were resolved by discussion andsubsequent independent recoding

Third Level of Review Classifications
 Theoretical or Conceptual Articles
about BSA or PDC
o Articles based purely at thelevel of discussion or theory
o Articles reporting
hypothetical datao Review articles (if they didnot present previously
unpublished data sets)o Articles which were databased but the intervention
was not driven by a BSA tool

Third Level of Review Classifications
 Minimal Empirical Data Involving
BSA or the PDC
o Articles presenting empiricalevidence of use of BSA orPDC, but utilized designs
that prevent establishmentof cause and effect relations
regarding enhancementsfrom the pre-intervention
diagnostic tool

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 13
• Experimental Evidence Involving BSA or thePDC Compared Against a Control Condition– Articles that directly compared effectiveness ofinterventions selected through use of BSA or PDC
to no active intervention (or no change toexisting interventions or initiatives)

Slide 14
• Experimental Evidence Involving BSA or the
PDC Compared Against an Alternative
Approach Condition– Articles that compared interventions selected
through use of BSA or PDC to another active
intervention selected by another type of analysis(or no prior analysis)

Slide 15

• Experimental Evidence Involving theComparison of One or More BSA Tools or theComparison of One or More PDC
Components– Articles that directly compared relative
effectiveness of one BSA tool/PDC component or
combinations of tools/components against
another BSA tool/PDC component or
combination of tools/components in improving
performance

http://www.johnsonresearchlab.com

Third Level of Review Classifications
 Experimental Evidence Involving
BSA or the PDC Compared Against aControl Conditiono Articles that directlycompared effectiveness ofinterventions selected
through use of BSA or PDC
to no active intervention (orno change to existing
interventions or initiatives)

Third Level of Review Classifications
 Experimental Evidence Involving
BSA or the PDC Compared Against
an Alternative Approach Conditiono Articles that comparedinterventions selected
through use of BSA or PDC
to another activeintervention selected byanother type of analysis (orno prior analysis)

Third Level of Review Classifications
 Experimental Evidence Involving theComparison of One or More BSATools or the Comparison of One orMore PDC Componentso Articles that directlycompared relative
effectiveness of one BSA
tool/PDC component or
combinations oftools/components against
another BSA tool/PDCcomponent or combinationof tools/components inimproving performance

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013California Association for Behavior Analysis, Garden Grove, CACorrespondence: djohnson@operant-tech.com

Slide 16

Slide 17

Slide 18Results First level of review
o 107 possible BSA articles
with IOA of 76%
o 12 possible PDC articles with
IOA of 100%
 Second level of review
o 31 BSA articles with IOA of92%
o 8 PDC articles with IOA of100%
 Third level of review
o Classification of articles into
an empirical comparison
category results in an IOA of94% for BSA and 100% forPDC

Category% of BSA Articles

Theoretical/conceptual

71%

Minimal empirical data

26%

BSA-driven intervention vs. nointervention control

3%

BSA-driven intervention vs.
alternative approach condition0%

Experiments comparing different
BSA tools

0%Category% of PDC Articles

Theoretical/conceptual

12.5%

Minimal empirical data

87.5%

PDC-driven intervention vs. nointervention control

0%

PDC-driven intervention vs.
alternative approach condition0%

Experiments comparing different
PDC components0%

http://www.johnsonresearchlab.com

Results of Level 3 Review – BSA
ArticlesResults of Level 3 Review – PDC
ArticlesThe Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CACorrespondence: djohnson@operant-tech.com

Slide 19

Discussion
 Use of a-priori or concurrent
assessments has increased overtime

Slide 20

Discussion
 However, pre-intervention
assessments are not explicitly
incorporated in the majority ofOBM articles
o From 2000 to 2011, lessthan 15% of published JOBM
articles made more than abrief mention of BSA or PDC
tools
o A cursory look through theremaining 85% of articlessuggests that it is not a caseof assessment tools besides
BSA and PDC being used
 Despite assertions regarding thepopularity of the PDC (Fante,Gravina, & Austin, 2007; Fante,Gravina, Betz, & Austin, 2010), theuse of this tool seems to be limited
and possibly in declineo During the last three yearsof this review (2009-2011),
no published articles used
the PDC

http://www.johnsonresearchlab.com

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 21
Despite the usage of thesetools, there is littleevidence to suggest

empirical validationof either approach

Slide 22

http://www.johnsonresearchlab.com

Discussion
 Results of current review indicatethat BSA and PDC are being used inOBM
 However, there is little evidence tosuggest empirical validation ofeither approacho 26% of the BSA studies and87.5% of the PDC studies
provided evidence that thetools were used to guide theselection of interventionso Only one BSA study and zero
PDC studies actuallyattempted to empiricallyvalidate the assessment
approach throughexperimental manipulationso The lack of empiricalvalidation may be acontributing factor to thestandard use of these toolsin published accounts oforganizational interventions
Possible Reasons for Lack of Empirical
Validation
 Environmental Complexityo Difficult to control
extraneous variables incomplex environments wellenough to determinefunctional relationships
o Difficult to recreate asufficiently complex
environment in the labo Both BSA and PDC require
investigator interaction withthe environment may leadto an unintentional alteringof the environment even
before an intervention isimplemented

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 23

Findingcomparison sitesof reasonable
complexity andsimilarity is difficult

Slide 24http://www.johnsonresearchlab.com

Possible Reasons for Lack of EmpiricalValidation
 Suitability of Comparison Sites orConditions
 No two sites or organizations arethe same
 Could potentially impact the results
if the investigator is attempting tocompare two different assessments
or different components of thesame assessment

Possible Reasons for Lack of EmpiricalValidation
 Cost and Timeo BSA can take several weeks
or even monthso Potential cost and timerequirements might
outweigh perceived valueo Delay to creatingorganizational change may
not be tolerated by
impatient clients unlessmeasurably superior results
can be shown or proof thata lack of detailed up-front
assessments may risk costlyerrors or interfere with
future assessment resultsand interventions
 Even more of anissue if client is not
persuaded by needfor scientific rigor

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 25

Suggestions for Future Research Experimental evidence involving
BSA or the PDC compared against acontrol condition
 Use two similar sites of the sameorganization or two differentorganizations under the same
parent company
 Compare the results of either BSA
or the PDC against a control
condition

Slide 26

Suggestions for Future Research Experimental evidence involving
BSA or the PDC compared against
an alternative approach condition
 Use two similar sites of the sameorganization, two different
organizations under the same
parent company, or different
individuals within same setting Determine whether different
assessments yield similarsuggestions for interventions
 Compare BSA assessment tools or
PDC checklist to otherorganizational analysis strategies to
see which strategies produce
superior results
 Compare BSA assessment tools
against the PDC checklist to assess:o differences in terms of thetypes of interventions thatare selectedo total time to select andimplement interventions
o differences in terms ofsuccess of interventions
driven by various analyses

http://www.johnsonresearchlab.com

The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 27

http://www.johnsonresearchlab.com

Suggestions for Future Research Experimental evidence involving thecomparison of one or more BSA
tools or the comparison of one ormore PDC components
o Compare visual
representations (maps) totext-based job-aids in termsof time to create and valuein pinpointing disconnects
o Compare number ofperformance levels assessedand success of interventionso Empirically analyze various
BSA tools to determine
which are essential foridentifying appropriate
intervention(s)o Empirically analyze
individual components ofspecific BSA tools todetermine which areessential for identifying
appropriate intervention(s)o Empirically analyze
individual components ofPDC checklist to determine
which are essential foridentifying appropriateintervention(s)The Use and Validation of Pre-Intervention Diagnostic Tools in OBM
DOUGLAS A. JOHNSON and Heather M. McGee
Western Michigan University
February 21st, 2013
California Association for Behavior Analysis, Garden Grove, CA
Correspondence: djohnson@operant-tech.com

Slide 28

Limitations
 Possible that search terms were notbroad enough and some articles
were missed Current review only included
articles from (JOBM)
 Possible that other publications
contain additional
 We specifically chose this journal
because it is the flagship journal ofOBM
 Possible the definition of BSA as amulti-level analysis or the use of
specific assessment tools used inthis review was too restrictive

Slide 29

Conclusions
 We still need to determine whether
OBM pre-intervention assessments
identify more disconnects and lead
to better interventions
 Scientific inquiries allow one's
respective field to better describe,
predict, understand, and control thephenomenon under study

Slide 30

http://www.johnsonresearchlab.com