Journal of Organizational Behavior Management, 34:104–121, 2014
Copyright © Taylor & Francis Group, LLC
ISSN: 0160-8061 print/1540-8604 online
DOI: 10.1080/01608061.2014.914009

The Use and Validation of Preintervention
Diagnostic Tools in Organizational
Behavior Management

Downloaded by [Western Michigan University] at 07:31 05 June 2014

DOUGLAS A. JOHNSON, SARAH E. CASELLA, HEATHER MCGEE,
and SIN CHIEN LEE
Western Michigan University, Kalamazoo, Michigan, USA

Functional analyses and assessments have become the accepted
gold standards in many applied areas of behavior analysis, including Organizational Behavior Management (OBM), in recent years.
Despite their acceptance, OBM data on such tools have been
largely absent. There are several assessment alternatives in OBM(e.g., ABC, PIC/NIC, Behavior Engineering Model), but two assessment approaches most common in the Journal of Organizational
Behavior Management are Behavioral Systems Analysis (BSA) and
the Performance Diagnostic Checklist (PDC). To date there havebeen no comprehensive reviews of BSA or the PDC from which one
might draw ideas for application, research, and the advancementof the field. The goals of this paper are to (a) provide a review of
the BSA and PDC literature within the Journal of Organizational
Behavior Management (JOBM), (b) discuss the implications of theresults of the review, and (c) provide suggestions for future researchutilizing BSA and the PDC.
KEYWORDS behavioral systems analysis (BSA), Performance
Diagnostic Checklist (PDC), preintervention analysis

During the past several decades, functional analysis and functional assessment have become the accepted standard in many applied areas of behavior
analysis (Bailey & Burch, 2002; Iwata, Dorsey, Slifer, Bauman, & Richman,1982). This paradigm shift has moved the field to first assess the variables
maintaining behavior or performance excesses/deficits and then use theAddress correspondence to Douglas A. Johnson, P.O. Box 20415, Kalamazoo, MI 49019,
USA. E-mail: djohnson@operant-tech.com
104

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

105

results of that assessment to determine the most appropriate intervention(s).
Organizational behavior management (OBM) is one such applied area inwhich authors have called for assessments being conducted either a priori or concurrently when designing performance improvement interventions(Austin, Carr, & Agnew, 1999).
As a method for deciding whether this call has been answered, we
reviewed all issues of the Journal of Organizational Behavior Management(JOBM) from 1977 through 2011 (book reviews were not included), Volumes
1–31, in order to determine the current state of the use of assessment
in OBM studies. We initially examined articles for at least one of thefollowing terms: assessment, systems analysis, systems thinking, processing
system, receiving system, process maps, relationship map, total performancesystem, performance matrix, super system, organizational scan, behavior
engineering model, human performance system, PIC/NIC, behavior engineering model, ABC analysis, or Performance Diagnostic Checklist. This list ofterms was first generated based on our knowledge and expertise in the field.
The list was expanded by conducting an informal review of OBM books
and articles to identify additional terms used when describing assessments
in OBM. Our review suggested that there are several assessment alternatives
in OBM, but two assessment approaches have been particularly visible (i.e.,
appear multiple times when search terms such as assessment or diagnostic are used) in JOBM, namely, behavioral systems analysis (BSA) and the
Performance Diagnostic Checklist (PDC). These two assessment approaches
are described in more detail next.

BSA
BSA is a methodology for improving performance that blends applied
behavior analysis (the application of the science of human behavior to theprediction and control of behaviors that are considered socially important;
Bailey & Burch, 2002) and general systems theory (an approach to understanding organizational systems by examining the relationships between
parts of the system, as well as the relationships between the system and theexternal environment; Ackoff & Emory, 1972; Bertalanffy, 1950, 1968). With
its focus on both the system and the person, BSA offers a framework forunderstanding how organizations work and how to make them work betterand has evolved through the work of many pioneers in the areas of behavior
analysis and performance improvement (Brethower, 1982; Daniels & Daniels,
2004; Gilbert, 1978; M. E. Malott, 2003; R. W. Malott, 1974; Rummler, 2004;
Rummler & Brache, 2013; Skinner, 1953).
BSA has a long history (Brethower, 1972, 1982; Connellan, 1978;
Harshbarger & Maley, 1974; Morasky, 1982) and is a very visible part ofOBM, as evidenced by the special issues dedicated to BSA in Volume 29 ofDownloaded by [Western Michigan University] at 07:31 05 June 2014

106

D. A. Johnson et al.

JOBM and other assorted articles. Over the years, the field of BSA has continued to grow and develop an elaborate and impressive series of maps, charts,and other tools intended to be populated via observations, questionnaires,and guided interviewing to aid in the pinpointing of performance gaps incomplex environments as well as in guiding the subsequent change efforts
that result from the identification of those gaps.
Behavioral systems analyses typically result in multilevel solutions
that may include performance management interventions, process design,
automation, policy changes, resource allocation, strategy development
and/or realignment, the development of incentive systems, organizational restructuring, performance-based training systems, and managing-themanager initiatives (Diener, McGee, & Miguel, 2009; McGee, 2007). The valueof BSA is that it considers identifying, planning, and managing the factors thatsignificantly impact individual and organizational performance, which could
impact the sustainability of performance change efforts. In fact, Redmon(1991) called for the explicit use of systems analysis in all published studies as a means of encouraging program adoption by organizations in which
OBM interventions are used.

THE PDCThe PDC was developed based on the results of Austin (1996), a study
that provided consultants and managers with a series of performance problems and asked them to talk aloud as they tried to solve the performanceissues. The results indicated that the participants who were most successful
in solving the performance problems asked questions related to the current state of antecedents, equipment and processes, knowledge and skills,
and consequences. Austin (2000) developed a series of diagnostic questions around these four areas, referred to as the PDC. The answers to thesequestions guide the selection of interventions aimed at improving performance. Multiple studies have asserted that the PDC is the most commonlyused assessment tool in OBM today (Fante, Gravina, & Austin, 2007; Fante,
Gravina, Betz, & Austin, 2010).

COMPARING BSA AND THE PDC
Both BSA and the PDC involve the use of tools to diagnose the functioning of an organization and pinpoint opportunities for improvement. Bothapproaches rely in part on informant assessment to infer functional relations,though proponents of both would recommend that direct observation anddata be used to confirm results of the informant assessment as much as
possible. Thus, there is a fair amount of overlap between these assessment

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

107

approaches. Both approaches attempt to pinpoint disconnects in work processes, identify necessary skills and resources for performing work, anddetermine the relation between performance and the environment.
There are several noteworthy differences as well. BSA tends to assign
a greater emphasis to processes and relations (Rummler & Brache, 2013),
whereas the PDC places a greater emphasis on environmental antecedents
and consequences (Austin, 2000). There are also formal differences between
these assessment approaches. Unlike the single checklist that mainly targets
the job or performer level of performance involved in PDC assessments,
BSA tends to involve multiple tools to diagnose goals, structure, and management at multiple levels (typically the organization, function, process, andjob or performer). Thus, BSA is more likely to provide a more thorough andmultifaceted picture of an organization, but such an assessment will likely
come at the price of a greater upfront time investment compared with the
PDC. Many of the BSA tools involve the creation of highly visual depictionsof the organization (such as flowchart tools). Representing structures andworkflow visually may lend itself to easier communication between the consultant and client or facilitate the consultant’s pinpointing of redundancies
or gaps. However, the many maps and diagrams may prove cumbersome
when contrasted with the PDC’s simple yes/no checklist format, confined toa single page. Nonetheless, such relative strengths and weaknesses of theseassessment tools remains an open empirical question.

STATEMENT OF THE PROBLEM
During our review of the BSA and PDC assessment approaches in JOBM,
a noticeable problem began to emerge. Namely, there appeared to be little
evidence to suggest empirical validation of either approach. To support thisobservation, we reviewed the BSA and PDC articles more carefully. We only
considered BSA and PDC articles that included a substantial discussion of
the tools/approaches (a description of how it was used in experimental
studies or comprising, at least in part, the topic of the article for theoreticalor conceptual articles). We also classified the articles according to the level of
empirical comparison. We utilized the following categories for classification:
theoretical or conceptual, minimal empirical data, comparison with control,and comparison with alternative.
The category of theoretical or conceptual was used for articles thatwere based purely at the level of discussion or theory. Articles reporting
hypothetical data were included in this category. Review articles were also
included in this category if they did not present previously unpublished data
sets. Discussion articles would clearly fit within this category, although someresearch articles could also be candidates for this category. Research articles
that presented data, even though the intervention was not driven by BSA,108

D. A. Johnson et al.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

were included in this category and indicated in Table 1 (no similar designations were necessary for the PDC articles). For example, an article that
included a discussion of BSA, but for which the interventions were not chosen based on the use of BSA tools or processes (or at least it was not madeclear in the Method section that this was the case), were included in this category. It is possible that the interventions were chosen based on a systems
analysis, but one cannot know this without a description of it. This category
was developed to capture any substantial articles regarding the diagnostic
tools of interest that may not have involved empirical data. Such a category
may prove useful in highlighting the degree to which the field simply discusses diagnostic tools compared to the degree to which the field actually
utilizes or examines such tools.
TABLE 1 Classification of Behavioral Systems Analysis Articles Published in Volumes 1–31 of
Journal of Organizational Behavior Management
Author(s) and year
Abernathy (2008)
Abernathy (2009)
Alavosius et al. (2009)
Austin et al. (1999)
Berglund & Ludwig (2009)
Brethower (2000)
Brethower & Wittkopp (1988)
Clayton et al. (1997)
Diener et al. (2009)
Frederiksen et al. (1985)
Gikalov et al. (1997)
Hayes et al. (2009)
Houmanfar et al. (2009)
Huberman & O’Brien (1999)
Hyten (2009)
Krapfl (1982)
Krapfl et al. (2009)
Kriesen (2011)
Langeland et al. (1998)
R. W. Malott & Garcia (1988)
Mawhinney (2009)
McSween & Matthews (2005)
Mihalic & Ludwig (2009)
Riley & Frederiksen (1984)
Sandaker (2009)
Sasson et al. (2006)b
Sasson & Austin (2002)
Smith & Chase (1990)
Sulzer-Azaroff et al. (1993)
Tosti & Herbst (2009)
Williams et al. (2002)ab

Classification
Theoretical or conceptual
Theoretical or conceptual
Theoretical or conceptuala
Theoretical or conceptual
Theoretical or conceptuala
Theoretical or conceptual
Minimal empirical data
Theoretical or conceptuala
Theoretical or conceptual
Minimal empirical data
Theoretical or conceptuala
Theoretical or conceptual
Theoretical or conceptual
Theoretical or conceptuala
Minimal empirical data
Theoretical or conceptual
Minimal empirical data
Minimal empirical data
Theoretical or conceptuala
Theoretical or conceptual
Theoretical or conceptual
Theoretical or conceptual
Minimal empirical data
Theoretical or conceptual
Theoretical or conceptual
Comparison with control; comparison
with alternative
Theoretical or conceptual
Theoretical or conceptual
Minimal empirical data
Minimal empirical data
Theoretical or conceptuala

Data-based article, but data presented did not involve behavioral systems analysis.
Utilized a 2 × 2 factorial design that allowed for multiple types of comparisons to be made.

109

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

The category of minimal empirical data was used for articles that
presented empirical evidence (i.e., actually used BSA or the PDC) but utilized designs that prevented the establishment of cause-and-effect relations
regarding enhancements to the interventions resulting from a particularpreintervention diagnostic tool. Examples would include case studies andAB designs.
The category of comparison with control was used for articles presenting empirical evidence that helps establish that interventions selected
through the use of BSA or the PDC are better than (or worse than) no active
intervention (or no change to existing interventions or initiatives).
The category of comparison with alternative was used for articles thatcompared interventions selected through the use of BSA or the PDC toanother active intervention selected by another type of analysis (or no prioranalysis).

SUMMARY OF JOBM ASSESSMENT LITERATURE
Our attempts to classify assessment tools in JOBM indicated that out of
31 BSA articles reviewed, approximately 71% of BSA articles were theoretical or conceptual in nature, 26% of BSA articles provided minimal empirical
data, 3% of BSA articles involved comparisons against a control, and only one
BSA article involved comparisons against an alternative (see Table 1). It is
important to note that the article by Sasson, Alvero, and Austin (2006) wasclassified as both a control and an alternative approach condition because it
utilized an experimental design that allowed for multiple types of comparisons. Specifically, a 2 × 2 design was used to compare two types of process
variables as well as compare intervention and no intervention conditions.
Results of the final review also indicated that out of eight PDC articles
reviewed, approximately 12.5% of PDC articles were theoretical or conceptual in nature, 87.5% of PDC articles provided minimal empirical data, andnone of the PDC articles involved comparisons with a control or comparisonswith an alternative (see Table 2).
TABLE 2 Classification of Performance Diagnostic Checklist Articles Published in Volumes
20–31 of Journal of Organizational Behavior Management
Author(s) and year
Amigo et al. (2008)
Doll et al. (2007)
Eikenhout & Austin (2005)
Gravina et al. (2008)
Pampino, Heering, et al. (2003)
Pampino, MacDonald, et al. (2003)
Rodriguez et al. (2005)
Weatherly & Malott (2008)

Classification
Minimal empirical data
Minimal empirical data
Minimal empirical data
Minimal empirical data
Minimal empirical data
Minimal empirical data
Minimal empirical data
Theoretical or conceptual

110

D. A. Johnson et al.

Downloaded by [Western Michigan University] at 07:31 05 June 2014IMPLICATIONS FOR ASSESSMENT IN OBM
The publication of the first functional analysis (Iwata et al., 1982) led to aparadigm shift in applied behavior analysis toward preintervention selection
assessments and analyses of the variables maintaining/suppressing behavior.
However, even in the year 2000, OBM researchers and practitioners werestill noting that assessments were not reported in the majority of published
OBM studies (Austin, 2000; Austin et al., 1999). Since that time, it would
appear that the use of a priori or concurrent assessments has increased,
as evidenced by the number of articles using two common approaches toorganizational assessment: BSA and the PDC. Despite their increasing prevalence, and despite calls for action by experts within the field, preintervention
assessments are not explicitly incorporated into the majority of OBM articles.
From 2000 to 2011, less than 15% of published JOBM articles made morethan a brief mention of BSA or PDC tools, despite the fact that this time
period included multiple special issues dedicated to these topics. A cursory
look through the remaining 85% of articles suggests that it is not the casethat other assessment tools are being utilized. Although it is not appropriatethat assessment tools be mentioned in every article, the data strongly suggest that the published use of these tools remains a rarity. Despite assertions
regarding the popularity of the PDC (Fante et al., 2007, 2010), the use of thistool seems to be limited and possibly in decline. During the last 3 years of
this review (2009–2011), no published articles used the PDC.
As we suggested earlier, there is little evidence to suggest empirical validation of either approach. In fact, although 26% of the BSA studies and 87.5%of the PDC studies provided evidence that the tools were used to guidethe selection of interventions, only one BSA study and zero PDC studies
actually attempted to empirically validate the assessment approach throughexperimental manipulations. Although a single BSA study nominally analyzed systems variables (Sasson et al., 2006), this study only looked at simple
variations of a process (deliver materials via manual or electronic means)
and may not well represent the lengthy processes seen in applied settingsor the typical in-depth analyses that result from examining such processes.
Thus, it could be argued that there are no true exemplars of BSA or PDC
experimentation in the existing literature. The lack of empirical validation
may be a contributing factor to the standard use of these tools in publishedaccounts of organizational interventions. There are many possible reasonsfor the lack of empirical validation of OBM assessment approaches.

Environmental Complexity
One potential reason for the lack of directly comparing BSA or the PDC toeither a control condition or an alternative assessment approach is that theseassessments occur within complex environments. In complex environments,

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

111

it is difficult to control extraneous variables well enough to determinefunctional relationships. Researchers could mitigate this by first conducting
research in the lab to determine whether functional relationships exist andthen following up with field studies to assess the generality of the findings.
However, recreating a sufficiently complex environment in the lab is difficult,if not impossible, especially for the purposes of testing a systems analysis.
Even if researchers could create a sufficiently complex environment, it isstill possible that an investigator–environment interaction would create anexperimental confound.
Whereas functional analyses used in other areas of behavior analysis relyon the direct observation of behaviors under a variety of tightly controlled
conditions, both BSA and the PDC (and many other assessment approaches)require investigator interaction with a complex environment throughout theassessment process. This may lead to an unintentional altering of the environment even before an intervention is implemented. For example, supposea researcher wanted to compare BSA against the PDC to determine whether
they yield different recommendations for interventions. Both approaches
utilize informant assessments. If the researcher conducted the assessments
within the same organization, the act of interviewing an organizational leader
during one assessment would have a strong potential to affect the responses
the same leader would give if interviewed again using the other assessment.
If this were to occur, it would contaminate the information gathered during
the alternative assessment. If the researcher tried to control for this by interviewing a different individual, even if that individual held a position at thesame level of the organizational hierarchy, there is no guarantee that the second individual would have the same knowledge of the organization as thefirst individual. This suggests the need to compare across sites or conditions
instead.

Suitability of Comparison Sites or Conditions
If significant confounds are likely in a study that compares two assessmentsin the same setting, another option would be to try to either create or findtwo very similar settings in which to conduct the assessments. As previously stated, recreating an environment that is complex enough to warrant asystemic analysis poses difficulties. Complex performance problems would
need to be created. Because the PDC and BSA both involve informant assessments, these environments would also need “employers” and “employees”
who possess information (both facts and opinions) about the performanceissues at hand. This might not be feasible or practical.
An alternative to creating the organization in a lab setting is to findtwo similar settings. Though this may still be difficult, it is at least feasible.
Possible options would be to look for either two similar sites (e.g., two different manufacturing plants) of the same organization or perhaps two different

112

D. A. Johnson et al.

organizations (e.g., two different franchises) under the same parent company (e.g., a fast food restaurant). Of course, no two sites or organizations
are the same, which could potentially impact the results if the investigatoris attempting to compare two different assessments or different componentsof the same assessment. However, this might be an acceptable approach if
one is comparing the results of one assessment type against a control condition, particularly if a multiple-baseline-across-settings design is used in whichmeasures are held constant across the sites.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Cost and Time
Two other potential reasons for the lack of empirical investigations of OBM
assessments are the cost and the time required to conduct these assessments.
This is of particular concern with BSA, which can take several weeks or evenmonths (Rummler, 2004). Although practitioners get paid to conduct theirassessments, the previously mentioned issues might preclude an experimental analysis of the validity of the assessment. For academic researchers, thepotential cost and time requirements to find or create the appropriate organizational environments and carry out the assessments might outweigh theperceived value of conducting the investigations. With both practitioners andacademic researchers, there is a benefit in that extended analyses are likely
to result in more complete information, allowing for a more sophisticatedand individually tailored intervention. However, for both types of researchers
there is a delay in creating organizational change. This may be a delay that
impatient clients, particularly those not persuaded by scientific rigor, will not
tolerate unless measurably superior results can be shown or it can be provedthat a lack of detailed upfront assessments may risk costly errors or interferewith future assessment results and interventions. On a related note, empirical evidence regarding the suboptimization principle could be collected.
This principle suggests that optimizing the effectiveness of one aspect of anorganization without consideration of other aspects of the organization may
result in a detrimental impact to the organization as a whole (M. E. Malott,2003). If such a principle could be empirically demonstrated, it could serve
as a justification to the client for the extra time and effort involved, particularly with potentially lengthy assessments such as those required by BSA.
Without such an empirical demonstration, it is possible that clients may prefer something like the PDC because it is a quick and simple checklist, often
based on a single interview. However, BSA tends to involve multiple tools
to diagnose organizational structure and workflow at multiple levels. This
often necessitates information being collected from many sources, potentially reducing the bias inherent in an interview with a single individual.
Thus, it is possible that BSA would provide a more objective picture of anorganization’s processes compared to the PDC. In other words, when usinga PDC checklist, one may risk overlooking the root cause of disconnects that

Preintervention Diagnostic Tools

113

may otherwise be noticeable if a BSA tool were used. Of course, this remains
pure speculation and reiterates the need for more empirical investigations.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Suggestions for Future Research
Based on our review of the literature, we would suggest that OBM has
yet to produce studies to validate common assessment procedures or determine optimal assessment approaches. The majority of the empirical evidence
currently available appears to comprise studies that use AB designs and onegroup pretest–posttest designs (classified as minimal empirical data in bothtables). Because these designs lack elements of replication or verification,such studies do not effectively establish cause-and-effect relations (Cooper,
Heron, & Heward, 2007; Shadish, Cook, & Campbell, 2002). The plurality ofanecdotes and case studies, although useful in describing how BSA and the
PDC are used in organizational settings, are not sufficient forms of empirical evidence for the use of preintervention diagnostic tools. OBM needs toextend its research agenda further to answer (a) whether preintervention
assessments enhance the selection of interventions by suggesting those thathave the greatest positive effect on the performance of the individual andthe organization and (b) whether that added benefit is worth the added cost.
Thus, we offer some suggestions for future research.
EXPERIMENTAL EVIDENCE INVOLVING BSA OR THE PDC COMPARED AGAINSTA CONTROL CONDITIONAs stated previously, two possible options would be to look for two similarsites (e.g., two different manufacturing plants) of the same organization or
perhaps two different organizations (e.g., two different franchises) under
the same parent company (e.g., a fast food restaurant) and compare theresults of either BSA or the PDC against a control condition (no assessment
conducted prior to choosing interventions). Critical performance measures
could be defined prior to baseline data collection using the same measuresacross sites. A multiple-baseline-across-settings design could then be used toassess whether interventions suggested by either BSA or the PDC impacted
those predefined measures over interventions chosen without the guidanceof an a priori assessment.
EXPERIMENTAL EVIDENCE INVOLVING BSA OR THE PDC COMPARED AGAINSTAN ALTERNATIVE APPROACH CONDITION

Although the conditions would not be perfect, the previously describedscenario for comparing BSA or the PDC against a control condition could
also be used to compare one assessment approach against an alternative
approach, though the design would need to be altered to a between-groups

Downloaded by [Western Michigan University] at 07:31 05 June 2014

114

D. A. Johnson et al.

or mixed between-groups and single-case multiple-baseline design. If thegoal of such a study is to simply assess whether the different approaches
yield similar suggestions for interventions, it might be possible to compare two sites, provided they are similar enough that one could assume
the same causal factors surrounding the performance issues (e.g., two manufacturing plants with the same physical layouts, processes, measures, andemployer/employee characteristics). Alternatively, researchers could instead
look for different individuals within the same setting, again provided thatthe researchers can demonstrate that each individual has similar knowledgeof organizational performance factors (e.g., two different plant managers or
supervisors from the same plant but possibly different shifts). Researchers
could conduct a study comparing BSA assessment tools or the PDC checklist to other organizational analysis strategies (e.g., PIC/NIC analysis, various
descriptive assessments, experimental analyses, intuitive guessing). There are
multiple measures by which these strategies can be compared: differences interms of the types of interventions that are selected, total time to select andimplement interventions, success of interventions driven by various analyses,
and so on.
The interventions identified by various assessment procedures could becompared in a number of ways, including stimuli typically targeted for intervention (antecedents or consequences), individuals targeted for intervention(management or subordinates), complexity of proposed intervention (single
component or package), longevity of proposed interventions (short-term or
long-term impact), performance aspect targeted for intervention (behavior
or result), level of performance targeted for intervention (e.g., performer,
process, function), and other relevant considerations. Reviews of the publication history of JOBM (Balcazar, Shupert, Daniels, Mawhinney, & Hopkins,
1989; Nolan, Jarema, & Austin, 1999; VanStelle, S. E., Vicars, S. M., Harr, V.,
Miguel, C. F., Koerber, J. L. et al., 2012) could provide the relevant intervention comparison criteria. If it were discovered that the type of proposedintervention tends to differ as the type of assessment differs, this would bea valuable contribution to the literature and would suggest that the selectionof a diagnostic tool may depend on the goals of the diagnostician.
EXPERIMENTAL EVIDENCE INVOLVING THE COMPARISON OF ONE OR MORE
BSA TOOLS OR THE COMPARISON OF ONE OR MORE PDC COMPONENTS
Several dismantling studies could be conducted to determine better methods for analysis, preferably methods that are more time efficient when theunneeded excess is trimmed away. For instance, one could examine whether
the mission statement needs to be considered and communicated (a standardpractice in BSA but not in PDC) and, if so, to how many levels. Also, manysystems tools rely on visual representations of organizational structures andprocesses (e.g., total performance system or process map). How such visualDownloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

115

representations compare to text-based job aids in terms of both time to create
these assessment tools and their value in pinpointing disconnects should be
investigated. Researchers could also investigate whether all possible levels(i.e., macrosystem, organization, process, task, behavior, management) ofanalysis are necessary to conduct an assessment for a successful intervention
compared to one or two levels of analysis. Researchers could also empirically analyze the various BSA tools (i.e., total performance system, process
map, relationship map) to determine which tools are essential for identifying
the appropriate intervention(s) and which tools are excess. The importanceof each of the individual components of the PDC checklist could also be
experimentally analyzed. For example, the PDC checklist contains a survey
item asking whether employees can see the effects of performance. Whether
this particular item (or any other component of the checklist) adds value tointervention selection could be examined through various dismantling strategies. It could also be examined whether the addition of any new items to
the checklist enhances intervention selection and implementation.
EXPERIMENTAL INVESTIGATION OF POSSIBLE INVESTIGATOR EFFECTS
As previously mentioned, BSA and the PDC both rely to some extent on
informant assessment. Informant assessments rely on interviews, surveys,
or other methods of gathering verbal reports of current performance, performance gaps, and potential causes of those gaps from key performancestakeholders. Depending on the performance(s) in question, these stakeholders could be leaders, managers, supervisors, customers, suppliers, and/or
employees. We have argued that comparing different assessment tools inthe same setting by interviewing the same individuals could be difficult if
the use of one assessment results in an investigator–environment interaction that affects the results provided on another assessment. However, thisinvestigator–environment interaction effect has not been experimentally validated. Researchers could experimentally investigate whether such an effectexists. Again, this would necessitate the identification of comparable sites inadvance (such as four separate franchises with similar policies, problems,and employees). Researchers could potentially utilize a multiple-baseline
design across settings (taking care to ensure that the target behavior[s] isthe same and the settings are similar): assess one site using BSA, a secondsite using the PDC, a third site using BSA and then the PDC, and a fourthsite using the PDC and then BSA. Sites that only use either the PDC or
BSA could serve as constant series control. This research idea could answer
many questions about the relative merits of BSA versus PDC and the potential contaminations/enhancements resulting from the cumulative effect ofmultiple assessment tools for a single site. Although this would obviously
be a difficult study, it could be completed as a series of studies to ensure
functional control.

116

D. A. Johnson et al.

Alternatively, a single site could be assessed by different independent
investigators. For example, two investigators could both use BSA or the
PDC to diagnose the organization’s functioning and propose interaction.
The degree to which these independent assessments match could be very
important in determining the reliability of various assessment approaches.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Concluding Comments
As with any critical review, there are some potential limitations of the present
discussion. It is possible that the search terms used were not broad enoughand as a result did not allow for certain assessment articles to be includedin the review process. Another limitation is that we only considered articles
from JOBM. It is possible that other publications outside of the reviewed
journal contain additional examples of BSA or PDC experiments. We specifically chose to review this journal because it is a flagship journal of the fieldwith more than 30 years of publications, and BSA and the PDC have beenshown to be an integral part of this field; no other publication possessed thatform of history with these tools. Lastly, it is possible that the definition of
BSA as a multilevel analysis or the use of specific assessment tools was too
restrictive.
In general, there remains a need to determine whether preintervention
diagnoses such as BSA and the PDC identify more disconnects and whether
they lead to better interventions. The fact that such a determination has neveroccurred is concerning given that members of the field often take great pridein the OBM approach being empirically driven, but the recommended assessment tools of the field lack empirical support. Overall, this is an area richfor confirmation through experimentation and refinement of tools and techniques, assuming that they actually do contribute to substantially improved
interventions. The mission of scientific inquiries is to allow one’s respective
field to better describe, predict, understand, and control the phenomenon
under study. The scientific process, although quite cumbersome and difficult, allows scientists to produce a number of socially important outcomes,
namely, empirical descriptions, predictive models, and an understanding of
causality. This last outcome, perhaps one of the most important outcomesproduced by science, needs to be supported by careful experimentation.
The results of this discussion suggest that there is an unfortunate disconnect between the scientific mission and the use of preintervention diagnostic
tools. Fortunately, acknowledging and understanding such a disconnect isthe first step toward remedying the problem. After all, as the science writer
Carl Sagan (1996) stated,
If we were not aware of our own limitations, though, if we were notseeking further data, if we were unwilling to perform controlled experiments, if we did not respect the evidence, we would have very little
leverage in our quest for the truth. (p. 263)

Preintervention Diagnostic Tools

117

ACKNOWLEDGMENT
The authors would like to thank Brittany Barnett and Michael Fantetti for
their help with data collection.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

REFERENCES
Abernathy, W. B. (2008). Implications and applications of a behavior systems
perspective. Journal of Organizational Behavior Management, 28, 123–138.
doi:10.1080/01608060802100980
Abernathy, W. B. (2009). Walden Two revisited: Optimizing behavioral systems.
Journal of Organizational Behavior Management, 29, 175–192. doi:10.1080/
01608060902874567
Ackoff, R. L., & Emory, F. E. (1972). On purposeful systems. Chicago, IL: AldineAtherton.
Alavosius, M., Getting, J., Dagen, J., Newsome, W., & Hopkins, B. (2009). Useof a cooperative to interlock contingencies and balance the commonwealth.
Journal of Organizational Behavior Management, 29, 193–211. doi:10.1080/
01608060902874575
Amigo, S., Smith, A., & Ludwig, T. (2008). Using task clarification, goal setting,and feedback to decrease table busing times in a franchise pizza restaurant.
Journal of Organizational Behavior Management, 28, 176–187. doi:10.1080/
01608060802251106
Austin, J. (1996). Organizational troubleshooting in expert management consultantsand experienced managers (Unpublished doctoral dissertation). Florida State
University, Tallahassee.
Austin, J. (2000). Performance analysis and performance diagnostics. In J. Austin &J. E. Carr (Eds.), Handbook of applied behavior analysis (pp. 321–349). Reno,
NV: Context Press.
Austin, J., Carr, J. E., & Agnew, J. L. (1999). The need for assessment of maintaining variables in OBM. Journal of Organizational Behavior Management, 19(2),59–87. doi:10.1300/J075v19n02_05
Bailey, J. S., & Burch, M. R. (2002). Research methods in applied behavior analysis.
Thousand Oaks, CA: Sage.
Balcazar, F. E., Shupert, M. K., Daniels, A. C., Mawhinney, T. C., & Hopkins, B. L.
(1989). An objective review and analysis of ten years of publication in the
Journal of Organizational Behavior Management. Journal of Organizational
Behavior Management, 10(1), 7–37. doi:10.1300/J075v10n01_02
Berglund, K. M., & Ludwig, T. D. (2009). Approaching error-free customer satisfaction through process change and feedback systems. Journal of Organizational
Behavior Management, 29, 19–46. doi:10.1080/01608060802660140
Bertalanffy, L. V. (1950). An outline of general system theory. British Journal for the
Philosophy of Science, 1(2), 134–165. doi:10.1093/bjps/I.2.134
Bertalanffy, L. V. (1968). General systems theory. New York, NY: George Braziller.
Brethower, D. M. (1972). Behavioral analysis in business and industry: A total
performance system. Kalamazoo, MI: Behaviordelia.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

118

D. A. Johnson et al.

Brethower, D. M. (1982). The total performance system. In R. M. O’Brien,
A. M. Dickinson, & M. P. Rosow (Eds.), Industrial behavior modification: A
management handbook (pp. 350–369). Elmsford, NY: Pergamon Press.
Brethower, D. M. (2000). A systemic view of enterprise. Journal of Organizational
Behavior Management, 20(3), 165–190. doi:10.1300/J075v20n03_06
Brethower, D. M., & Wittkopp, C. J. (1988). Performance engineering.
Journal of Organizational Behavior Management, 9(1), 83–104. doi:10.1300/
J075v09n01_07
Clayton, M. C., Mawhinney, T. C., Luke, D. E., & Cook, H. G. (1997).
Improving the management of overtime costs through decentralized controls.
Journal of Organizational Behavior Management, 17(2), 77–98. doi:10.1300/
J075v17n02_03
Connellan, T. K. (1978). How to improve human performance: Behaviorism inbusiness and industry. New York, NY: Harper & Row.
Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis.
Upper Saddle River, NJ: Pearson Education.
Daniels, A. C., & Daniels, J. E. (2004). Performance management: Changing behavior
that drives organizational effectiveness. Atlanta, GA: Performance Management
Publications.
Diener, L. H., McGee, H. M., & Miguel, C. F. (2009). An integrated approach for
conducting a behavioral systems analysis. Journal of Organizational Behavior
Management, 29, 108–135. doi:10.1080/01608060902874534
Doll, J., Livesey, J., McHaffie, E., & Ludwig, T. D. (2007). Keeping an uphill edge:
Managing cleaning behaviors at a ski shop. Journal of Organizational Behavior
Management, 27(3), 41–60 doi:10.1300/J075v27n03_04
Eikenhout, N., & Austin, J. (2005). Using goals, feedback, reinforcement, and aperformance matrix to improve customer service in a large department store.
Journal of Organizational Behavior Management, 24(3), 27–62. doi:10.1300/
J075v24n03_02
Fante, R., Gravina, N., & Austin, J. (2007). A brief pre-intervention analysis anddemonstration of the effects of a behavioral safety package on postural behaviors of pharmacy employees. Journal of Organizational Behavior Management,
27(2), 15–25. doi:10.1300/J075v27n02_02
Fante, R., Gravina, N., Betz, A., & Austin, J. (2010). Structural and treatment analyzes
of safe and at-risk behaviors and postures performed by pharmacy employees.
Journal of Organizational Behavior Management, 30, 325–338. doi:10.1080/
01608061.2010.520143
Frederiksen, L. W., Riley, A. W., & Myers, J. B. (1985). Matching technology and organizational structure. Journal of Organizational Behavior Management, 6(3),59–80. doi:10.1300/J075v06n03_05
Gikalov, A. A., Baer, D. M., & Hannah, G. T. (1997). The effects of work task
manipulation and scheduling on patient load, revenue, eyewear turnover,and utilization of staff and doctor time. Journal of Organizational Behavior
Management, 17(1), 3–35. doi:10.1300/J075v17n01_02
Gilbert, T. F. (1978). Human competence: Engineering worthy performance. New
York, NY: McGraw-Hill.
Gravina, N., VanWagner, M., & Austin, J. (2008). Increasing physical therapy
equipment preparation using task clarification, feedback and environmental

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

119

manipulations. Journal of Organizational Behavior Management, 28, 110–122.
doi:10.1080/01608060802100931
Harshbarger, D., & Maley, R. F. (1974). Behavior analysis and systems analysis: An
integrative approach to mental health programs. Kalamazoo, MI: Behaviordelia.
Hayes, L. J., Dubuque, E. M., Fryling, M. J., & Pritchard, J. K. (2009). A
behavioral systems analysis of behavior analysis as a scientific system.
Journal of Organizational Behavior Management, 29, 315–332. doi:10.1080/
01608060903092169
Houmanfar, R., Rodrigues, N. J., & Smith, G. S. (2009). Role of communication
networks in behavioral systems analysis. Journal of Organizational Behavior
Management, 29, 257–275. doi:10.1080/01608060903092102
Huberman, W. L., & O’Brien, R. M. (1999). Improving the therapist and patient
performance in chronic psychiatric group homes through goal-setting, feedback,and positive reinforcement. Journal of Organizational Behavior Management,
19(1), 13–36. doi:10.1300/J075v19n01_04
Hyten, C. (2009). Strengthening the focus on business results: The need for systems
approaches in organizational behavior management. Journal of Organizational
Behavior Management, 29, 87–107. doi:10.1080/01608060902874526
Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S.
(1982). Toward a functional analysis of self-injury. Analysis & Intervention in
Developmental Disabilities, 2(1), 3–20. doi:10.1016/0270-4684(82)90003-9
Krapfl, J. (1982). Behavior management in state mental health systems.
Journal of Organizational Behavior Management, 3(3), 91–105. doi:10.1300/
J075v03n03_10
Krapfl, J. E., Cooke, J., Sullivan, T., & Cogar, W. (2009). Iterative processes and reciprocal controlling relationships in a systemic intervention.
Journal of Organizational Behavior Management, 29, 136–154. doi:10.1080/
01608060902874542
Kriesen, G. L. (2011). Print still matters in an e-learning world, and training
companies need to properly manage it. Journal of Organizational Behavior
Management, 31, 179–195. doi:10.1080/01608061.2011.589721
Langeland, K. L., Johnson, C. M., & Mawhinney, T. C. (1998). Improving staff performance in a community mental health setting. Journal of Organizational
Behavior Management, 18(1), 21–43. doi:10.1300/J075v18n01_03
Malott, M. E. (2003). Paradox of organizational change: Engineering organizations
with behavioral systems analysis. Reno, NV: Context Press.
Malott, R. W. (1974). A behavioral systems approach to the design of human services.
In D. Harshbarger & R. F. Maley (Eds.), Behavior analysis and systems analysis:
An integrative approach to mental health programs (pp. 318–342). Kalamazoo,
MI: Behaviordelia.
Malott, R. W., & Garcia, M. E. (1988). A goal-directed model for the design of humanperformance system. Journal of Organizational Behavior Management, 9(1),
125–159. doi:10.1300/J075v09n01_09
Mawhinney, T. C. (2009). Identifying and extinguishing dysfunctional and deadlyorganizational practices. Journal of Organizational Behavior Management, 29,
231–256. doi:10.1080/01608060903092094

Downloaded by [Western Michigan University] at 07:31 05 June 2014

120

D. A. Johnson et al.

McGee, H. M. (2007). An introduction to behavioral systems analysis for OBMersand non-OBMers alike. OBM Network Newsletter, 21(2).
McSween, T., & Matthews, G. (2005). Should we teach the ABC’s? Journalof Organizational Behavior Management, 24(1), 135–143. doi:10.1300/
J075v24n01_08
Mihalic, M. T., & Ludwig, T. D. (2009). Behavioral system feedback measurement
failure: Sweeping quality under the rug. Journal of Organizational Behavior
Management, 29, 155–174. doi:10.1080/01608060902874559
Morasky, R. L. (1982). Behavioral systems. New York, NY: Praeger.
Nolan, T. V., Jarema, K. A., & Austin, J. (1999). An objective review of the Journal of
Organizational Behavior Management: 1987–1997. Journal of Organizational
Behavior Management, 19(3), 83–114. doi:10.1300/J075v19n03_09
Pampino, R. N., Jr., Heering, P. W., Wilder, D. A., Barton, C. G., & Burson, L. M.
(2003). The use of the Performance Diagnostic Checklist to guide intervention
selection in an independently owned coffee shop. Journal of Organizational
Behavior Management, 23(2), 5–19. doi:10.1300/J075v23n02_02
Pampino, R. N., Jr., MacDonald, J. E., Mullin, J. E., & Wilder, D. A. (2003). Weekly
feedback vs. daily feedback: An application in retail. Journal of Organizational
Behavior Management, 23(2), 21–43. doi:10.1300/J075v23n02_03
Redmon, W. K. (1991). Pinpointing the technological fault in applied behavior analysis. Journal of Applied Behavior Analysis, 14, 441–444. doi:10.1901/
jaba.1991.24-441
Riley, A. W., & Frederiksen, L. W. (1984). Organizational behavior management in
human service settings. Journal of Organizational Behavior Management, 5(3),3–16. doi:10.1300/J075v05n03_01
Rodriguez, M., Wilder, D. A., Therrien, K., Wine, B., Miranti, R., Daratany, K., . . .
Rodriguez, M. (2005). Use of the Performance Diagnostic Checklist to selectan intervention designed to increase the offering of promotional stamps at twosites of a restaurant franchise. Journal of Organizational Behavior Management,25(3), 17–35. doi:10.1300/J075v25n03_02
Rummler, G. A. (2004). Serious performance consulting according to Rummler. Silver
Spring, MD: International Society for Performance Improvement.
Rummler, G. A., & Brache, A. P. (2013). Improving performance: How to manage thewhite space on the organizational chart (3rd ed.). San Francisco, CA: JosseyBass.
Sagan, C. (1996). The demon-haunted world: Science as a candle in the dark. New
York, NY: Ballantine Books.
Sandaker, I. (2009). A selectionist perspective on systemic and behavioral change inorganizations. Journal of Organizational Behavior Management, 29, 276–293.
doi:10.1080/01608060903092128
Sasson, J. R., Alvero, A. M., & Austin, J. (2006). Effects of process and humanperformance improvement strategies. Journal of Organizational Behavior
Management, 26(3), 43–78. doi:10.1300/J075v26n03_02
Sasson, J. R., & Austin, J. (2002). Performer-level systems analysis. Journalof Organizational Behavior Management, 22(4), 27–58. doi:10.1300/J075v22
n04_03
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston, MA: Houghton
Mifflin.

Downloaded by [Western Michigan University] at 07:31 05 June 2014

Preintervention Diagnostic Tools

121

Skinner, B. F. (1953). Science and human behavior. New York, NY: Macmillan.
Smith, J. M., & Chase, P. N. (1990). Using the vantage analysis chart to solveorganization-wide problems. Journal of Organizational Behavior Management,
11(1), 127–148. doi:10.1300/J075v11n01_09
Sulzer-Azaroff, B., Pollack, M. J., & Fleming, R. K. (1993). Organizational
behavior management within structural and cultural constraints. Journal of
Organizational Behavior Management, 12(2), 117–137. doi:10.1300/J075v12
n02_10Tosti, D., & Herbst, S. A. (2009). Organizational performance and customer value.
Journal of Organizational Behavior Management, 29, 294–314. doi:10.1080/
01608060903092151
VanStelle, S. E., Vicars, S. M., Harr, V., Miguel, C. F., Koerber, J. L., Kazbour, R.,& Austin, J. (2012). The publication history of the Journal of Organizational
Behavior Management: An objective review and analysis: 1998–2009. Journalof Organizational Behavior Management, 32(2), 93–123. doi: 10.1080/
01608061.2012.675864
Weatherly, N. L., & Malott, R. W. (2008). An analysis of organizational behavior
management research in terms of the three-contingency model of performance
management. Journal of Organizational Behavior Management, 28, 260–285.
doi:10.1080/01608060802454643
Williams, W. L., Di Vittorio, T. D., & Hausherr, L. (2002). A description and extensionof a human services management model. Journal of Organizational Behavior
Management, 22(1), 47–71. doi:10.1300/J075v22n01_04