The 3rd International Symposium forEthics Crossroads (Nov.20th 2007)

Assessment and EvaluationofEthical Reasoning Skills
Kojiro Honda
Humanities and Social Science Program,
Kanazawa Institute of TechnologyIndex
1. Introduction
2. Assessment
3. Result

KIT 3rd International
Symposium (2007)

2

1. Introduction
In this Presentation• Our View of Assessment for Educational
Effect of Case Method• Our Practical Activities for Assessment
will be discussed.
KIT 3rd International
Symposium (2007)3

Case Method••

We use Case Method as core of ourcurriculum.
We provide 3 cases for students’ discussion
as follows.
1. Space Shuttle Challenger
2. Research Ethics in Our University
3. Gilbane GoldKIT 3rd International
Symposium (2007)

4

Case Method•

We expect that students can improve theircompetence such as

1. Analysis Skill
2. Explanation Skill
3. Question Skill
4. Skill to deal with Ethical Dilemma
KIT 3rd International
Symposium (2007)

5

Case Method••

We call these skills “Ethical ReasoningSkills”
Can they be taught?

1. Analysis Skill
2. Explanation Skill
3. Question Skill
4. Skill to deal with Ethical Dilemma
KIT 3rd International
Symposium (2007)

6

Theoretical BackgroundFive-Stage Skill-Acquisition Model(by Hubert L. Dreyfus)• Stage 1: Novice• Stage 2: Advanced Beginner• Stage 3: Competence• Stage 4: Proficiency• Stage 5: ExpertiseKIT 3rd International
Symposium (2007)

7• Stage 1: Novice
The beginner is given rules fordetermining actions (…) much like acomputer following a program.

KIT 3rd International
Symposium (2007)8• Stage 2: Advanced Beginner• As novice gains experience actuallycoping with real situations, he or she
begins to note (…) perspicuous
examples of meaningful additional
components of the situation.
KIT 3rd InternationalSymposium (2007)9• Stage 3: Competence• With increasing experience, the numberof features and aspects to be taken into
account becomes overwhelming. (…) theperformer learns to adopt a hierarchicalview of decision making.
KIT 3rd International
Symposium (2007)10

• Stage 4: Proficiency• As soon as the competent performer
stops reflecting on problematic situations
as a detached observer and stops
looking for principles to guide his or her
actions, the gripping, holistic experiencesfrom the competent stage become thebasis of the next advance in skill.
KIT 3rd InternationalSymposium (2007)11• Stage 5: Expertise• The proficient performer, immersedin the world of skillful activity, sees
what needs to be done but must
decide how to do it.
KIT 3rd International
Symposium (2007)

12

Theoretical Background• Case Method can’t give a real experience.
• But it can provide good pseudo-experience for
Dreyfus’s Stage1 or 2.
• The skill-acquisition doesn’t indicate moral
development (cf. Kohlberg’s theory) . We think
students can improve their skills by training.
KIT 3rd International
Symposium (2007)13

C (Check)• Assessing and estimating of students’ ability toresolve the ethical dilemma in the engineeringcontext
For what?
1 Measuring what degree the students share thevalues to
2 Assessing the educational effect against students
3 Assessing how the teachers keep their consistencyof moral standards or moral opinions to estimate
students ?
KIT 3rd International
Symposium (2007)

14

C (Check)So,• Our assessment is teacher-oriented

and• Our assessment is not concerned withstudents’ grades．

KIT 3rd InternationalSymposium (2007)15

2. Assessment

Achievement Aim• One of our achievement aim for students is“Can write short essay to prove your own
analysis skill about ethical dilemma in ascenario”

KIT 3rd InternationalSymposium (2007)17

AssessmentMethod• We gave the two scenarios at the beginning and theend of the semester.
Artificial Heart (at the beginning)
Trees(at the end)
• Students should write essay about these scenarioswith respect to ethical dilemma.
• We estimate their essays.
KIT 3rd International
Symposium (2007)

18

••

•

The Artificial Heart.
When Thomas Creighton, a 33-year old auto mechanic and divorced father of two rejected
the heart he had received from an accident victim, he was put on the heart/lung machine. Dr.
Jack Copeland and the transplant team at the University Medical Center in Tucson
immediately began searching for another donor heart. So on Wednesday, 6 March 1985 at
6:00 a.m. Kevin Chaing of Phoenix received a call from cardiac surgeon, Dr. Copeland.
When Copeland asked: is your heart ready to be implanted. Chaing thought the question
referred to implanting the artificial heart he had invented into a calf. He and Dr. Copeland
had done this experiment before and were planning to replicate it. When he learned that theintended recipient was a human being he balked. Wait a minute, that heart was designed fora calf it’s not ready for a human yet. Think about it and decide said Copeland, I’ll call you
back in ten minutes.
Questioned later Chaing said: I knelt and prayed. And when Dr. Copeland called him backhe answered: the pump is sterile and ready to go. Meanwhile the donor heart that Dr.
Copeland had requested from Utah had not yet arrived. By around noon the same day Mr.
Creighton’s physician decided that it would be dangerous to leave him on the heart/lungmachine any longer. And so Dr. Copeland implanted the artificial heart device in Mr.
Creighton. The artificial heart maintained Mr. Creighton’s circulation until the donor heart
arrived.
At 11:00 p.m. that night the device was turned off and Mr. Creighton was put back on theheart/lung machine. At 3:00 a.m. on Thursday, Dr. Copeland transplanted the second donor
heart. However, despite all efforts, Mr. Creighton died the following morning. The Phoenix
artificial heart had nothing to do with causing his death. Nevertheless, in deciding to use thePhoenix heart Kevin Chaing and Dr. Copeland had apparently violated FDA regulations byemploying a device that had not been approved for experimental use in humans. They
justified their actions by claiming that their use of the device was an emergency measure.
Chaing and Copeland had no intention of performing an experiment with Mr. Creighton.
They were trying to save his life. The only other option was to let him die. We had nothing tolose by using the heart said Dr. Copeland.
KIT 3rd International
Symposium (2007)

19•

•

•

•

•

Trees
Kevin Clearing is the engineering manager to the Verdant County Road Commission, whichhas primary responsibility for maintaining the safety of county roads. Verdant County’s
population has increased by 30 percent in the past ten years. This has resulted in increased
traffic flow on many secondary roads in the area. Forest Drive, still a two-lane road, has morethan doubled its traffic flow during this period. It is now one of the main arteries leading into
Verdant City, an industrial and commercial center of more than 60,000 people.
For each of the past seven years at least one person has suffered a fatal automobile
accident by crashing into trees closely aligned along a three-mile stretch of Forest Drive.
Many other accidents have also occurred, causing serious injuries, wrecked cars, anddamaged trees. Some of the trees are quite close to the pavement. Two lawsuits have been
filed against the road commission for not maintaining sufficient road safety along this threemile stretch. Both were dismissed because the drivers were going well in excess of the 45mph speed limit.
Other members of VCRC have been pressing Kevin Clearing to come up with a solution tothe traffic problem on Forest Drive. They are concerned about safety, as well as lawsuits that
may some day go against VCRC. Clearing now has a plan—widen the road. Unfortunately,
this will require cutting down about thirty healthy, long-standing trees along the road.
Clearing’s plan is accepted by VCRC and announced to the public. Immediately a citizen
environmental group forms and registers a protest. Pat Northington, spokesperson for thegroup, complains, “These accidents are the fault of careless drivers. Cutting down trees toprotect drivers from their own carelessness symbolizes the destruction of our naturalenvironment for the sake of human ‘progress.’ It’s time to turn things around. Sue the drivers
if they don’t drive sensibly. Let’s preserve the natural beauty and ecological integrity around
us while we can.”Many letters on both sides of the issue appear in the Verdant Press, the issue is heatedly
discussed on local TV, and Pat Northington presents VCRC with a petition to save the trees
signed by 150 local citizens.
KIT 3rd InternationalSymposium (2007)20

Assessment
Problems• How do we assess their skills Quantitatively?
• How do we assess their skills Objectively?

KIT 3rd International
Symposium (2007)21

Assessment
The Pittsburgh-Mines (P-M) Engineering Ethics
Assessment Rubric
By Larry J. Shuman et al.
•

The method for assessment of students’ ability toresolve ethical dilemma.

•

They identified five components with five levels ofachievement.

KIT 3rd International
Symposium (2007)22

Assessment
The Pittsburgh-Mines (P-M) Engineering Ethics
Assessment Rubric① Recognition of Dilemma② Information③ Analysis
Cf.) Larry J. Shuman, Mark F. Sindelar, Mary
④ Perspective
Besterfield-Sacre, Harvey Wolf and Rosa L.
Pinkus (2004) “Can Our Students Recognize and⑤ Resolution
Resolve Ethical Dilemmas?”, Proceeding of the2004 ASEE Annual Conference & Expositon

KIT 3rd International
Symposium (2007)23

Assessment•

We tried PM rubric, but We felt we shouldmake our own rubric.

•We needed easier and more speedy
assessment.

•We should have established our educational
policy by doing so.

KIT 3rd InternationalSymposium (2007)

24

KIT Simple Rubric Ver.1(See Material 1)

KIT 3rd International
Symposium (2007)

25

KIT Simple Rubric Ver.2

KIT 3rd International
Symposium (2007)

26

KIT Simple Rubric Ver.3

KIT 3rd International
Symposium (2007)

27

KIT Simple Rubric Ver.5

KIT 3rd International
Symposium (2007)

28

KIT Simple Rubric Ver.7

KIT 3rd International
Symposium (2007)

29

KIT Simple Rubric Ver.7(Continued)

KIT 3rd International
Symposium (2007)

30

KIT Simple Rubric Ver.7•

We have 6 components
A) Perception of ProblemB) Recognition of Facts
C) Recognition of Stakeholders
D) ResolutionE) Ethical ReasoningF) Perspective

KIT 3rd InternationalSymposium (2007)

31

KIT Simple Rubric Ver.7• We adopt 2 points scale (0 or 1)• Students are estimated by the total score.

• For easy rating．
• For easy checking the difference amongteachers.
KIT 3rd International
Symposium (2007)32

Inter-Rater Reliability•

We should estimate objectivity of ourassessment for1. Quantitative Assessment
2. Guarantee Our Consistency of Educational
Policy (Peer Estimation)

KIT 3rd International
Symposium (2007)33

Inter-Rater Reliability
5 Teachers did
1. Read and estimate same 10 students’essays
2. Estimate gaps of estimation among the 5
3. Reduce the gaps by discussion
4. Make a consistent policy of estimation
among the 5.
KIT 3rd International
Symposium (2007)34

Difference of Rating Result
among 5 Teachers(Artificial Heart)

KIT 3rd International
Symposium (2007)35

Difference of Rating Result
among 5 Teachers(Trees)

KIT 3rd International
Symposium (2007)36

Difference of Rating Result
among 5 Teachers(Ideal Result)

KIT 3rd International
Symposium (2007)37

Intra Crass Correlation (ICC)
Attached inSPSSKIT 3rd International
Symposium (2007)38

Inter-Rater Reliability
Artificial Heart
S1A

S2A

S3A

S4A

S5AS6A

S7A

S8A

S9A

S10A

T1

7456343346T2

7004320002T37357624335T4

7037530404T511007045020

KIT 3rd International
Symposium (2007)

ICC=０．８６１（ Cronbach ’s α）

39

Inter-Rater Reliability
Trees
S1B

S2B

S3B

S4B

S5B

S6B

S7B

S8B

S9B

S10B

T1

80776002411T2

405560000

7T35387732469T4

60978204510T585101212038711KIT 3rd International
Symposium (2007)

ICC=０．９６３ （ Cronbach ’s α）

40

3. ResultChange of Average Total Scores
during the Semester (1)
クラス別平均点の推移（標本数各１０）109876543210
学期始
KIT 3rd InternationalSymposium (2007)

学期終2007 Spring

3VA1
3VA2
3VD1
3VD2
3HM1
3HM2
3HB1
3HB2
3HP1
3HS142

Change of Average Total Scores
during the Semester (2)クラス 別平均点（標本数各１０）109876543210

KIT 3rd International
Symposium (2007)

3V
D23HM
13HM23H
B13H
B23H
P13H
S1

D13V

A23V

3V

A1

学期始
学期終

2007 Spring43

Change of Average Total Scores
during the Semester (3)総合平均点の推移（ 標本数１ ０ ０ ）8765Total

43210
学期始
KIT 3rd International
Symposium (2007)

学期終2007 Spring44

Change of Average Total Scores
during the Semester•

We can indicate 2 points

1. All classes increase their total score2. We have various growth rates of the scoreamong the classes

KIT 3rd International
Symposium (2007)45

ReviewWe did
1. Make our own simple rubric,
2. Make a method to share the educational
policy by checking ICC (Peer Estimation),
3. Make a method for quantitative assessmentof students’ ethical reasoning skills.
The change of average scores indicated thatStudents could increase their skills in thesemester
KIT 3rd International
Symposium (2007)

46

Challenge•

We have something to discuss as follows,

1. What does it mean, the difference of scoreamong the classes?
2. What does it mean, the difference of growthrate among the classes? Does it depend of
Teachers’ competence?
Etc.
KIT 3rd International
Symposium (2007)

47

Thank you for your attention!
E-mail:k-honda@neptune.kanazawa-it.ac.jp