The Role of Student Survey forAssessment

University Evaluation Forum 2012
7/23/2012
Reiko Yamada
Doshisha University

Changing Japanese Higher Education Policy
in a Knowledge-based Society◆ 2008 CCE Report “Toward the Construction for

Undergraduate Education”• Pressure for “Quality Assurance” under Globalization
to set common “Learning Outcome” of each institutions
for students• Exemplifying “Graduate Attributes” = reference guideline• Government emphasizes on Outcome Assessment

Construction of Undergraduate Education in Globalized Society
2Policy Change towards Reinforced Quality Assurance
The 2008 CCE Report made it clear that quality assurance would bestrengthened by reviewing the policies of diversified universities andthe flexible processes of approving the establishment or reorganizationof universities.
The report placed emphasis on the quality assurance of bachelor•’sdegrees, and made a proposal to implement comprehensive reform
covering “exit,” “content” and “entrance.” For “exit” in particular, it
put forward the idea of “bachelor’s competence” to specify what
abilities bachelor’s degrees awarded by Japanese universities
guaranteed.

Clarifying the “diploma policy” (exit), “curriculum policy”(content) and “admission policy” (entrance) is an essential task for
Doshisha University as well.
3New Council Reportin 2012
2012 CCE Report “In the unpredictable era, how

can university play the role of making student continue
to involve learning and active thinking”
Message= Qualitative Transformation ofUndergraduate Education
Most important message=
to increase learning hours of Japanese students
4Change between 2008 and 2012
Many faculties tended to be more teaching-centered(previously most of faculties were research-centered)
95% of universities present the syllabus
Almost universities introduce FYE
Other programs and pedagogies such asservice-learning, learning community, …
active learning method
But5

Criticism of Japanese HigherEducation
Strong criticism from the industrial world

and Japanese people toward the quality of college
students and undergraduate education
60% of respondents of the survey of newspaper
criticize that Japanese universities do not offer
education which can be dealt with globalization andknowledge based society
6Background toward Qualitative
Transformation of JHE
Average learning hours of students should be 8 hours(including class and homework) a dayHowever, average Japanese students learning hours are
4.6 hours a dayVery low compared with that of American
undergraduate students
Learning hours of students in the field of social
sciences are relatively low compared with that of
Natural Sciences, Health Sciences, and Arts.
7

Qualitative Transformation
Increase learning hours of students

Preparation and deep learningEffective teaching management
Systematization of curriculum (e.g. numbering)Systematic team management of teachingSubstantiality of syllabus

8What is Student Survey?

Two Kinds of Outcome Assessment•How do we assess learning outcome of students?
•Two methods : Combination of Direct & Indirect is general trende.g. Critical thinking test & student survey
Direct Assessment＝Direct Evidence＝Assess the Learning Outcome
Contents＝Test, Paper, Project, Portfolio, Graduation Exam, Rubric
Graduation Research Paper or Project, Standardized Test
Fields＝General Education, Major Education (discipline)
Indirect Assessment＝Indirect Evidence＝Assess the Learning Process＝Learning Behavior,Life Behavior, Self-Perception, Satisfaction=Process Assessment to the Outcome

10Contents＝Student Survey, Survey for Graduates
When＝Freshman, End of the First Year, Senior Year, After Graduation

Student Survey as a Process Assessment•Indirect assessment
・Process based assessment
What We Should Do for General Usage of Process Assessment?
Improve Validity and Reliability
What are new challenges?
Longitudinal Study to develop various kinds ofmethodology→To Check the Items →Develop Standard
Survey→to examine college impact (IEO model)→to support
the theory behind the study
Studies for the effectiveness of Surveys(Anaya, 1999; Carini, O_Day, and Kuh, 2002 )To combine both Direct and Indirect Assessment is Important11

Student Engagement from theJCIRP Data

Objectives and Characteristics of the Japanese
Cooperative Institutional Research Program (JCIRP)
To conduct surveys on the level of cognitive/noncognitive student growth, and to develop methods ofmeasuring the contribution of higher education
institutions to student growth.
Characteristics of the research program: Based on thetrends shown in college impact studies, whose results
have been gathered in the U.S. the program examines
the educational impact that universities have on students
and what outcomes they produce, referring to Astin’s
Input-Environment-Output (I-E-O) model; it also
verifies and develops college impact models.
To develop and carry out surveys comparable to the13student surveys conducted in the U.S.

Learning Hours outside Class in a Week
Information
Sciences
Arts
Home2.30
Economics
Education
MedicalSciences
Natural Science& EngineeringSocialSciences
Humanities

14

None

Less than
1 hour1 to 23 to 56 to 1011 to 15

16 to 20

More than 20

Class Attendance in a Week
None

Information
SciencesArts

Home
Economics
Education
MedicalSciences
Natural Science& EngineeringSocialSciences
Humanities
15

Less than
1 hour1 to 23 to 56 to 1011 to 15

16 to 20

More than 20

How Much Time Do Students Spend
Studying and Attending Classes?
 On the whole, students spend little time studying.

 There is a certain percentage of students in every discipline

who spend no time studying outside of class.
 Social sciences and humanities students spend fewer hours
studying than other students.
 Students attend more classes in the first semester.

Students tend to spend little time studying outside of class.
16

17

100

Total percentage of“greatly increased” and“increased”

90
80
70
60
50
40
30
20
100

18

How effective is university education?
Learning outcomes related to “bachelor’scompetence” have been steadily getting better.
異文化 地域社専門分異文化 リー 人間関
国民がプレゼコン
グローの人々 会が直分析や批判的文章表
野や学の人々 ダー 係を構
直面す
ンテー 数理的 ピュー バルな 外国語
一般的問題解に考え現の能と協力 面するな教養科の知に関す シップ 築するる問題
ション な能力 タの操 問題の の能力する能 問題の決能力る能力力
識る知識 の能力 能力の理解の能力
作能力 理解力
理解日本2005

75.9

60.7

87.1

56.8

47.5

26.3

55.1

22.2

30.4

49.3

46.3

44.3

21.4

76.5

49

32.9

日本2007

71.2

62.7

84

53.3

44.4

31.3

57.9

22.6

32.6

39.8

47.7

52.4

26.9

70.4

30.7

27.9

日本2010

71.7

68.4

83

55.8

48.9

38.2

64.3

29.9

41.1

45.4

49.8

55.8

28.9

67

3630

What Does the JCIRP Data Tell Us?
Learning outcomes have been improving slowly but surely.
What is behind this improvement?
International comparison is not so simple; self-assessment reflects

the characteristics of a people and culture.
Will comparison with an international standard assessment modelallow us to discover more about learning outcomes?
It is necessary that direct and indirect assessments complement
each other.

19

To What Extent Did Students in Different Majors
Acquire Knowledge and Skills?
Maximum
社会科
理工農
MedicalSocialpossibleSciences
人文系
医療系
最高点
F
Sciencesscores
学系
生系
Average
Average
Average
Average
平均
SD 平均
SD 平均
SD 平均
SD
Basic social skills
社会的基礎スキル18.39 2.9 18.04 3 17.89 3.1 18.04 2.9 25 6.27*
Classical liberal arts and skills
古典的教養とスキル
26.5 3.1 25.9 3.5 26.4 3.3 25.6 3.4 35 25.2*
Contemporary liberal arts and現代的教養とスキル
17.67 2.7 17.24 2.6 16.76 2.5 16.48 2.5 25 63.4*
skillsBasic academic achievement
基礎的学力
6.2 1.4 6.1 1.4 6.7 1.4 6.1 13 10 65.4*
Humanities

NaturalScience &EngineeringOverall, humanities students performed well.
Natural science and engineering students performed well in terms of basic academic

achievement.
Humanities and social sciences students were better at contemporary liberal arts and skills
than natural science, engineering and medical sciences students.
20

Panel Data Analysis of Students in Different
Divisions and Majors

Model of Possible Analysis
ＪＦＳ２００８First Year Students

ＡＢ

ＤＣ

ＪＣＳＳ２００９
Upper Grade Students
22

ＪＦＳ２００９First Year Students

ＣＡ

ＪＣＳＳ２０１０
Upper Grade Students

Data Set
Major%

Selectivity

JFS2008 Humanities
15.0 Lowest
3521 Social Sciences
23.1 Lower
Natural Science &(First-year
24.4EngineeringMiddle3482）
Medical Sciences
14.7 Higher
Education
16.5 Highest
Home Economics
3.9 TArts
2.5100T
JCSS2010 Humanities
23.4 Lowest
2875 Social Sciences系
22.1 Lower（Over 3rd21.1year 2142) Natural Science & Engineering Middle
Medical Sciences
10.8 Higher
Education
10.7 Highest
Home Economics
4.7 TArts
7.2100T

23

Type ofInstitution
13.3 National
26.4 Public%%Year%

Gender%20.6 1st
3.9 2nd3rd year

98.9 Man
1.0 Female

42.058.0

75.5 Other

0.1100100 T100

22.5 National
18.9 Public

24.3 1st6.9 2nd

12.0 Man
11.9 Female

43.0
57.0

13.1

68.8 3rd& 4th74.5

100

100 OtherT

1.610012.6Private
37.7 T
10.1
100

Private33.3 T
12.2100TTDegree of Satisfaction
1. Opportunity to Communicate with Faculty
2. Enrollment Number in a Class
3. Opportunity to Communicate with Other Students
4. Relevance of Learning Contents with Daily Life
5. Availability of Learning Contents for Future Career

JFS2008

JCSS2010

6. Quality of Teaching

JFS2008
JCSS2010

7. Sense of Student Community
8. Interaction with Other Students
9. Overall College Experience
10. Tolerance for Diversity
11. Student Support System
12. First-year Student Program
13. Liberal Education Program

JFS2008

JFS2008

JCSS2010

JCSS2010

14. Natural Science Program
15. Humanities Program

16. Social Science Program

24

JFS2008

JFS2008

JFS2008

JCSS2010

JCSS2010

JCSS2010

Institutional Environment (Curriculum & Program)College Environment

High school environment
Affective & Cognitive
Aspect in High SchoolDays
Learning Behavior
Learning Orientation
Motivation
Interest

Faculty
Environment
Encouragement
Pedagogy
Learning Program

Learning Quantity& Quality

Classmates
Friends
MentorMajor Environment
25

Affective /Cognitive
OutcomeThe JCIRP Study Has Clarified:
College Impact in Japanese Universities
1.Longitudinal study of students data cancontribute to grasp the student engagement byaccumulating large data
2.Panel data analysis shows the degree of student
satisfaction increased in the upper division (juniorand senior > freshman and sophomore and there isqualitative difference of the degree of student
satisfaction between majors.
The difference reflects the environment of the major
3. Process assessment of students learning is effective
and can be tied with direct assessment in Japanese
universities
26

Thank you
ryamada@mail.doshisha.ac.jp