Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 1

•Performance feedback is among themost common interventions used for
performance in the field oforganizational behavior management(OBM)•Problematic as feedback’s definition•Which components are crucial for
successful implementation?

Slide 2

Slide 3

Both evaluationand information
are typically involved

Are both components

necessary?

•Most involve some component ofevaluation and information about past
performance•Unknown which of those components
are necessary for feedback to be
maximally effectiveIsolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 4

•Why objective feedback alone might
not be enough•Literature reviews: the effects offeedback are variable•Implementation of feedback is quitedifferent from study to study (broadterm!)•For example, Johnson, Dickinson, andHuitema examined computer-delivered
objective feedback (i.e., specificinformation about past performanceabsent evaluation such as praise orcriticism)•Purpose: documenting the effect that
performance feedback has on amonetary incentive system
Published in:Johnson, D. A., Dickinson, A. M., &Huitema, B. E. (2008). The Effects of
Objective Feedback on Performance
When Individuals Receive Fixed andIndividual Incentive Pay. Performance
Improvement Quarterly, 20(3/4), 53-74.

Slide 5

•Objective feedback in previous study•On-screen display indicating the total
number of checks completed correctly
and current rate of check completion

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 6

•Four experimental conditions:•Individual monetary incentive withoutfeedback•Individual monetary incentive with insession feedback•Fixed pay without feedback•Fixed pay with in-session feedback

Slide 7

•Findings of previous study•Significant improvement inperformance when incentives used,
relative to a fixed pay system•Addition of feedback, however,resulted in no significant performancegains for participants, regardless of thepay system that was implemented•One implication: numbers and data
themselves that are not important•Rather, it is what others say about
those numbers and data that is
important for altering performance•These results led the authors to
speculate that “some type of evaluative
component may be necessary for
feedback to enhance performance.” (p.
71)

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 8•Why objective feedback might not beneeded at all•Plausible that the evaluative
component alone influenced
performance•Objective feedback might be an
unnecessary superfluous component in
the studies examining objective
feedback combined with evaluation•Determine whether specificinformation about past performancemakes feedback more effective thanjust simple general appraisal of past
performance

Slide 9•Why objective feedback might beneeded•For feedback to be effective it should
detail specific information about
performance•Delivering reinforcement, it should be
both specific and sincere•Objective feedback may help withthese delivery guidelines bydemonstrating to the recipient ofreinforcement that the supervisor iscarefully paying attention to actual
performance•Not just blindly delivering insincere
praise or approval (such as a manager
saying “attaboy” to every employee
seen, regardless of actualaccomplishments)

On the other hand
objective feedback mightshow employees that thesupervisor isn’t justblindly delivering
insincere praise orapproval

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 10

Let’s see how the following measure up to one another:• Objective feedback alone• Evaluative feedback alone• Objective feedback with evaluation• No feedback•Lack of component analysis research•In order to disentangle the effects ofthe evaluative and objective
components of feedback, the current
study proposed to compare•Objective feedback alone,•Evaluative feedback alone,•Objective feedback with evaluation•No feedback conditions

Slide 11•105 undergraduate university students•One 45-minute pre-feedback session
and three 45-minute experimental
sessions•The experimental task is acomputerized data entry task

Slide 12•Pre-feedback session: obtain data forthe covariate•Gather information about
performance to be used in thesubsequent session’s feedback

All participants
worked the firstsession in theabsence of anyfeedback

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 13

•Performance labeled according to oneof four values: excellent, good, average,or poor
•Participants not told about theselabels or the criteria for these labels•These values were determined using
the standard deviation and average
performance of individuals who did notreceive incentives or feedback in aprevious study using the same
experimental task (Johnson, Dickinson,& Huitema, 2008)

Slide 14

•A 2 X 2 factorial design was used, withparticipants randomly assigned to one
of four experimental groups. Eachgroup had 25-27 participants assignedto it•Same general format as pre-feedback,except for the following differences
relating to assignment to condition

First pre-feedback,then three experimental sessions
25-27 participants per group

Slide 15Evaluative feedback only
• “You’ve been completing much morethan what the average person does!”• “We appreciate how hard you’ve beenworking at this task.”• “You’ve been processing checks at astandard level of performance.”• “Unfortunately, you should know thatyou’ve been completing a low numberof checks.”•Evaluative feedback only condition•Randomly read one of forty evaluative
statements about their previous
session’s performance•No objective details aboutperformance givenIsolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 16

Objective feedback only
• “During your previous session, youcorrectly completed _____ checks.”

Slide 17

Evaluative and objective feedback• “During your previous session, you correctly
completed _____ checks. That’s a really
impressive number of checks!”• “During your previous session, you correctly
completed _____ checks. That’s one of thebetter performances we’ve seen recently.”• “During your previous session, you correctly
completed _____ checks. That’s about what
the average person does.”• “During your previous session, you correctly
completed _____ checks. Unfortunately,
that’s considered a low number of checks.”•Objective feedback only condition•Participants read the following
sentence in a neutral tone: “During
your previous session, you correctly
completed _____ checks.”•Experimenters provided no evaluationof performance, taking care to not toconvene any body language that might
be interpreted as approving or critical(i.e., smiling, frowning, nodding ofhead, etc)

•Evaluation and objective feedback
condition•Participants randomly read one offorty evaluative statements about theirprevious session’s performance, include
objective details on performance•A variety of sentences used todecrease the possibility thatparticipants will perceive thestatements as rote and insincere•The sentence read depended in parton whether their previous performancewas categorized as “excellent”, “good”,“average”, or “poor” (ten random
sentences for each category).

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com•No feedback condition•No further instructions or information
about their performance provided

Slide 18

No feedback• “”

Slides19-22
No feedback

Evaluative feedback only

Initial performance: 670 checks
Final performances: 647 checks (loss of 23 checks)

Initial performance: 657 checks
Final performances: 742 checks (gain of 85 checks)

Objective feedback only

Objective feedback with evaluation

Initial performance: 662 checks
Final performances: 750 checks (gain of 88 checks)

Initial performance: 718 checks
Final performances: 893 checks (gain of 175 checks)•Raw results•No feedback•Initial performance: 670 checks•Final performances: 647 checks (lossof 23 checks)•Evaluation alone•Initial performance: 657 checks•Final performances: 742 checks (gainof 85 checks)•Objective information alone•Initial performance: 662 checks•Final performances: 750 checks (gainof 88 checks)•Objective information with evaluation•Initial performance: 718 checks•Final performances: 893 checks (gainof 175 checks)

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slides23-26•Visual inspection: Changes over time

No feedback
1000
950
900
850

800
750
700
650
600

12

3

4

Objective only
1000
950
900
850

800
750
700
650
600

1234

Evaluative only
1000
950
900
850

800
750
700
650
600

1234

Objective and evaluative
1000
950
900
850

800
750700
650
600

1234

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slides
27-28

Objective only

Evaluative only

1000

1000

950

950

900

900

850

850

800

800

750

750

700

700

650

650

600

600
1

234

1

2

Objective and evaluative

3434No feedback

1000

1000

950

950

900

900

850

850

800

800

750

750

700

700650650600600
123412

Objective+evaluative vs. no feedback
1000950
900
850
800
750700650
600

1Slide 29234•Analysis•Two-factor ANCOVA•Adjusted means•No feedback: 654 checks•Evaluation alone: 763 checks•Objective information alone: 766
checks•Objective information with evaluation:
850 checks•Evaluative and objective feedback
compared with:•Evaluative only (p = 0.029)•Objective only (p = 0.035)•No feedback (p = 0.000)•Evaluative feedback alone comparedwith:•Objective only (p = 0.999)•No feedback (p = 0.004)•Objective alone compared with
•No feedback (0.003)•Summary: Performance underevaluative and objective higher than allothers to a statistically significant
degree. Evaluative alone and objective
alone both statistically higher than nofeedback






Payoff
Intervention that requires less than
one minute of supervisor's time
17% increase in performance simplyby adding either evaluation or
objective information (delivered inperson)
30% improvement in performance if
you use both evaluation andobjective information compared tono feedback

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com

Slide 30

Objective feedback: Does

in-person delivery
make a difference?

•Conclusions•Conflict with previous study showing
no effect of objective only•Computer-delivered vs. persondelivered feedback (impliedevaluation?)•Employees have a history in which
their supervisors have evaluated theirperformance (at least typically). Given
that such an evaluation has, in the past,been correlated with contingent
consequences (at least to somedegree), even though the performancereward contingency for the newly
targeted performance may not be
explicitly explained or stated, thesupervisory feedback may “act much
like a more formal change in an
employees’ [sic] job description orcontract. The overall impact is ‘tonotify’ employees of the new or now to
be enforced contingencies operating in
the organization” (Prue & Fairbank,1981, p. 12). This notification, ofcourse, implies evaluation.

•Does evaluative feedback enhancemonetary incentives?
•Less specific feedback can be effective,
as long as it is evaluative (althoughmore specific works better)

Slide 31

How does evaluative feedback
impact monetary incentives?

Isolating the Critical Components of Effective Feedback on a Data Entry Task
DOUGLAS A. JOHNSON
Operant-Tech Consulting, Western Michigan University
May 29th, 2010
Association for Behavior Analysis International 36th Annual Convention, San Antonio, TX
Correspondence: djohnson@operant-tech.com•Normative influence?
•Punishment for being above or below(making everyone else look bad vs.
dropping the ball)•Social comparison

Slide 32

Slide 33

CONTACT
Doug Johnson, Ph.D.
Western Michigan University
Operant-Tech Consulting
E-MAIL: djohnson@operant-tech.com
WEBSITE: www.operant-tech.com