国立大学法人 一橋大学 森有礼高等教育国際流動化機構

Working Paper SeriesMori Arinori Institute for Higher Education and Global Mobility

No.WP2021-03
The Effects of Large-Scale Online Classes on
Students' Course Evaluations: Evidence froma Japanese University under the COVID-19
Pandemic

Ryohei Kashima
2021年8月

Yohei Yamamoto

The E¤ects of Large-Scale Online Classes on Students’Course Evaluations: Evidence from a Japanese
University under the COVID-19 Pandemic
Ryohei Kashimay

Yohei Yamamotoz

Hitotsubashi University

Hitotsubashi University

August 16, 2021

Abstract
We investigate the e¤ects of an introduction of large-scale online classes under
the COVID-19 pandemic on students’ course evaluations using questionnaire data ina Japanese university from 2018-2020. We employ di¤erence-in-di¤erences analysis
with means-as-outcomes using means at the student and instructor levels, respectively.
Students’course evaluations improved for most question items when the raw questionnaire data are used. However, the improvements are weak or none if the instructor-levelmeans are used. These results suggest that improvements in course evaluations are lessattributed to teaching quality but more to patterns of students’class-taking.

JEL Classi…cation: A22, I21
Keywords: COVID-19; Teaching Quality; Means-as-Outcomes; Di¤erence-in-Di¤erences

We appreciate the useful comments from participants of the HIAS lunch seminar. All errors are ourown.
yGraduate School of Economics and Mori Arinori Institute for Higher Education and Global Mobility,
Hitotsubashi University, 2-1 Naka, Kunitachi, Tokyo, 186-8601 (rkashi1228@gmail.com).
z
Graduate School of Economics and Mori Arinori Institute for Higher Education and Global Mobility,
Hitotsubashi University, 2-1 Naka, Kunitachi, Tokyo, 186-8601 (yohei.yamamoto@r.hit-u.ac.jp).

1

Introduction

Universities across nations introduced large-scale online classes in the year of the COVID19 pandemic. There is a strong concern as to how they have a¤ected the students’course
evaluations and, ultimately, the quality of teaching.1 This study approaches such questions
using data of regularly conducted student course evaluations of an anonymous Japanese
university from 2018-2020. Using regular questionnaire data is advantageous in identifying
causal e¤ects, as the data are available from before the COVID-19. This helps overcome thenon-presence of a valid “control group” even when the entire university began implementing
online classes, because the data before COVID-19 can be regarded as a control group. As
such, we employ di¤erence-in-di¤erences (DID) analysis by taking the change from 2019-2020
as the treatment group and the most recent change from 2018-2019 as a control group.
Our goal is to disentangle multiple e¤ects that occurred under the COVID-19 pandemic
on students’course evaluations. In particular, we focus on the following two major e¤ects.
First, teaching online can a¤ect course evaluations through the quality of teaching. Somereasons for this include the following widely concerned facts: online classes may be considered
to discourage students in having discussions with the instructor and classmates, undermining
the strictness of examinations and evaluation fairness, making working in groups more di¢ cult, and creating a heavier homework burden.2 Second, large-scale online classes under the1

There is rapidly increasing literature that is concerned with the negative impacts of the COVID-19
countermeasures on university education. Orlov et al. (2021) showed that test scores declined after the
COVID-19 outbreak, and Aucejo et al. (2020) have raised the concern that many students had delayed
graduation and lost job opportunities. Agostinelli et al. (2020) point out that a temporary loss of educational opportunities could have a long-run negative e¤ect. Andrew et al. (2020) found that students spend a
shorter amount of time studying at home under COVID-19, and this is more pronounced for students from
low-income households. Engzell et al. (2020) warn that even students in countries with highly developed
information infrastructure lose educational opportunities. However, Bacher-Hicks et al. (2020) evaluatedrapid accumulations in online educational resources under the COVID-19 pandemic, and Ikeda and Yamaguchi (2021) propose using online learning services to compensate for the loss of educational opportunities
for Japanese high school students during the school closures.
2
More fundamentally, the e¤ects of online teaching were discussed even before the COVID-19 outbreak.
Coates et al. (2004) and Figlio et al. (2013) showed that when the same instructor gives the same classface to face and online, the students who took the online class had lower test scores. Alpert et al. (2016)
conducted the same class fully face to face, fully online, and a mixture of the two and found that students’performance in the online classes was signi…cantly lower than that in the face-to-face classes, and there wasno di¤erence between the mixed and face-to-face classes. Using non-experimental data but employing the
instrumental variable method, Bettinger et al. (2017) showed that online classes lower exam scores; hence,
they may lead to absence or withdrawal from school. Although the average e¤ect is negative, Bettinger et al.
(2017) found that the negative e¤ect of online classes disappears for students with a high prior college GPA,
and Figlio et al. (2013) found similar results for higher high school GPA students. Furthermore, Cacault etal. (2021) showed that taking classes through a livestreaming format had a positive e¤ect on students with a1

COVID-19 pandemic changed patterns of students’class-taking because physical restrictionsof students (such as time) and universities (such as classroom capacities) were lessened.
We use data of non-mandatory course evaluations. In addition to the DID analysis using
the raw questionnaire data, we estimate the model using two di¤erent types of means-asoutcomes to consider changes in patterns of students’class-taking that might have occurredin 2020. First, we use the student-level means as they adjust for the tendency that students
who tend to give higher evaluations take more courses and thus give more responses. Second,
we use the instructor-level means as they adjust for the tendency that students take morecourses and thus give more responses for highly rated instructors. These analyses are useful
to control for patterns of students’class-taking when teaching quality may also change.
Our …ndings are summarized as follows. First, results using the raw questionnaire data
and using the student-level means show improvements in students’ course evaluations in
all question items but one. However, the improvements become weak or none if we use
the instructor-level means. These results suggest that the overall improvements of the raw
questionnaire mean are less attributed to teaching quality but more to patterns of students’class-taking. In the remainder of this paper, we describe the data in section 2, explain ourempirical strategy in section 3, report our results in section 4, and conclude in section 5.
2

Data

We use repeated cross-sections of students’course evaluations from an anonymous Japanese
university for the spring and summer quarters of the academic years 2018-2020. The survey
is a web-based questionnaire that students can answer for all the courses they take toward
the course’s end. The response rates were 31.1% in 2018, 32.0% in 2019, and 30.1% in 2020,
which concern us regarding selection bias in estimating the population mean. However, since
we focus on the di¤erence between two years, the selection bias is expected to be o¤set if
the sources are unchanged, and the di¤erence estimator will not be much distorted.3
Table 1 shows the number of responses to the survey and summary statistics (mean,high GPA in high school. These results demonstrate the existence of heterogeneity in the online class e¤ect.
Other studies, such as those by Chen and Lin (2016), Heissel (2016), and Escueta et al. (2017), found thatonline classes are not perfect substitutes for face-to-face classes.
3
As an analogy, Perron and Yamamoto (2015) discuss the same issue in the context of a structural change
model. Even if endogeneity is present, as long as the magnitudes and the dates of structural changes aretaken into account, the simple ordinary least squares (OLS)-based estimator is consistent under certain
assumptions. Moreover, the OLS-based estimator is more precise than the instrumental variables (IV)-based
estimator in general. The tests for structural changes based on the OLS-based estimator have a higher power
than the tests based on the IV-based estimator.

2

standard deviation, skewness, and kurtosis) for each year by grade. The total number of responses for the 3 years is 26,583, with a tendency of fewer responses from students in upper
grades. We used seven question items commonly asked in the 3 years: frequency of student attendance (“Attendance”), time spent studying the subject outside the class (“Study
hours outside the class”), how clear the course objectives are (“Clearness of objectives”),
transparency of the grading criteria (“Clearness of evaluation”), how understandable theinstructor’s explanations are (“Clearness of explanation”), how enthusiastic the instructoris (“Enthusiasm of the instructor”), and how useful the student feels the course is (“Significance of the course”). For each question, the answer is selected from 1-5, and the higher
the score, the more positive the evaluation. 4 The mean of “Attendance”decreases, and its
standard deviation increases as students go to the upper grades. The mean and standard
deviation for “Study hours outside the class” decreases as students go to the upper grades.
For the other questions, the mean slightly increases, and the standard deviation decreases as
the students go to the upper grades. For “Attendance”, the number of students who rated
5 increased, and thus the skewness and the kurtosis are severe in 2020, although the sample
size is large enough to disregard these e¤ects in the following statistical analysis.
3

Empirical strategy

3.1

Di¤erence-in-di¤erences

To assess the e¤ect of large-scale online classes, it is useful to compare the change in the
averages of responses from 2019-2020 with that when they were not introduced but the sameotherwise. The former works as the treatment group, and the latter works as the control
group. Our particular choice of the latter is the sample that is as recent as possible, that is,the change from 2018-2019. Hence, we estimate the following quantity as the DID e¤ect:fE(Yi j2020; Zi )E(Yi j2019; Zi )gfE(Yi j2019; Zi )E(Yi j2018; Zi )g:(1)

To this end, we estimate the following regression model
Yi =

0+

1 D20i +0
2 D18i + Zi

+ ei ;(2)for i = 1; :::; N , where Yi is the raw questionnaire response, D18i is a dummy variable thattakes 1 if the response is in 2018 and 0 otherwise, D20i is a dummy variable that takes the4

For “Study hours outside of class”, since there are di¤erences in the classi…cation of response items
between school year 2018 and 2019-2020, we converted the items into a three-level evaluation classi…cationthat allows us to integrate the responses from the 3 years.

3value of 1 if the response is in 2020 and 0 otherwise, Zi is a vector of the control variables,and ei is the error term with mean zero. For the control variables Zi , we include dummy
variables for the major of student, grade of the student, courses, and instructors.
Since we used 2019 as the reference case in (2), the conditional expectation of Yi in termsof (2) in each year is described as follows:
E(Yi j2018; Zi ) =0+

E(Yi j2019; Zi ) =E(Yi j2020; Zi ) =

(3)

2;

(4)

0;

0+

(5)

1:

Hence, the causal e¤ect (1) can be estimated asfE(Yi j2020; Zi )= f( 0 +=1+

1)

E(Yi j2019; Zi )g

0g

f 0

fE(Yi j2019; Zi )E(Yi j2018; Zi )g;( 0 + ;2 )g;2.

Figure 1 illustrates how these quantities correspond to the DID model. If the change from2018-2019 occurred in 2020, the causal e¤ect was captured by 1 + 2 . If not, we may regard
1 as the causal e¤ect, assuming that no trend exists.

an F test for the null hypothesis of H0 :1 +2For the former, we can test using= 0 and the alternative hypotheses of

2 6= 0. For the latter, we can test for H0 :1 = 0 and H1 :

H1 :

1+

3.2

Di¤erence-in-di¤erences with means-as-outcomes

1 6= 0 by a t test.

Our response data have multiple attributes at higher levels: student, instructor, course, andyear. Besides estimating the DID model using the raw questionnaire data, we focus on somecategories by constructing means-as-outcomes prior to estimating the regression model (2).5
In particular, we use two types of means as follows. First, we take the means of the responsesof each student prior to estimating the DID model. We call this the student-level means.
Estimation using these can adjust the tendency that students who give higher evaluations
take more courses and thus give more responses in the COVID-19 year. Second, we takethe means of responses that each instructor receives before estimating the DID model. We
call this the instructor-level means. Estimation using these can adjust the tendency thatstudents take more courses and thus give more responses toward highly rated instructors in
5

See Bryk and Raudenbush (2002).

4the COVID-19 year. If this is the case, the mean of the raw questionnaire data may increase,even though the teaching quality remains the same, while analysis using the instructor-levelmeans keeps the mean unchanged.
Figure 2 illustrates our framework in a very simple example of changes in patterns ofstudents’class-taking. This is a case in which the students take more courses and thus give
more responses to highly rated instructors. Suppose that in 2019, a student gives a 4.0 to thecourse of instructor A, and another student gives a 2.0 to the course of instructor B. Then,the average score using the raw questionnaire data is 3.0 in 2019. Importantly, the averagescore is 3.0, even when the student-level and instructor-level means are used. Suppose now
that in 2020, patterns of students’class-taking change, and the number of students enrolled
in the courses of instructor A increase. Interestingly, even though the quality of courses
remains the same, the average score in 2020 increases to 3.3 if the raw questionnaire data areused. When the student-level means are used, the average is even higher (3.5). Interestingly,
the average score of the instructor-level means remains at 3.0.6 Therefore, the student-level
and instructor-level means are useful to consider such a tendency.
When we use the means-as-outcomes in (2), the control variables can di¤er. When we
use the student-level means, the dummy variables for the student’s major and the grade areincluded. When we use the instructor-level means, the dummy variables for the instructor’s
department are used, although these control variables do not a¤ect our main results.
4

Results

Table 2 shows the results of (2) using the raw questionnaire data. The coe¢ cient

1 , that is,(5) minus (4), represents the change in the conditional mean of the responses from 2019-2020.
The estimates are positive and highly signi…cant for all question items and indicate that thestudents’course evaluations improved from 2019-2020. To obtain a more appropriate causal
interpretation, we investigated the DID e¤ects 1 + 2 . The estimates are again positive andsigni…cant at the 1% level for all question items, but “Clearness of evaluation” is negative
but insigni…cant. This re‡ects students’anxiety about online evaluation methods comparedto face-to-face proctored exams. “Attendance”shows the largest increase, which may inducea change in patterns of class-taking. Finally, the DID e¤ects are smaller than 1 for all thequestion items. This suggests an overall tendency of improvement from 2018-2019. It is
important to ask whether this continues in 2020 to set an appropriate counterfactual in the6
We can also construct an example in which students who tend to give higher evaluations take morecourses, in which the student-level mean does not change.

5

DID analysis. Another choice is to assume no change as a counterfactual and to interpret
1 as a causal e¤ect. In either case, the …nal conclusions are qualitatively the same.

Table 3 shows the estimation results of (2) using the student-level means. The DID e¤ects
are larger compared with those in Table 2 for many question items, such as “Attendance”,“Study hours outside the class”, and “Signi…cance of the course”. These suggest the tendency
that students who take less courses have larger improvements in their evaluations in the
COVID-19 year. Moreover, the DID e¤ects are positive and signi…cant at the 1% level for
all question items but “Clearness of evaluation”. In this sense, we can say that the results
are overall similar to the previous analysis using the raw questionnaire data in Table 2.
The results using the instructor-level means are presented in Table 4. When we look at
the changes from 2019-2020 by 1 , “Attendance”, “Study hours outside the class”, “Clearnessof objectives”, and “Signi…cance of the course” are positive and signi…cant at the 1% level;
however, the values are much smaller than those using the raw questionnaire data and using
the student-level means. “Enthusiasm of the instructor”and “Clearness of explanation”arepositive, but the former is signi…cant only at the 10% level, and the latter is insigni…cant.
“Clearness of evaluation” turned negative, although it is insigni…cant. Interestingly, if welook at the DID e¤ects ( 1 +

2 ), all question items but “Attendance” are close to 0 andinsigni…cant, and “Clearness of evaluation”and “Enthusiasm of the instructor”are negative.
The last results go a long way toward our goal. If online classes improve teaching quality,
positive and signi…cant coe¢ cients would be observed in the instructor-level analysis too. In
particular, for question items such as “Clearness of objectives”, “Clearness of explanations”,“Enthusiasm of the instructor”, and “Signi…cance of the course” are closely related to it.
Hence, these results suggest that improvements in course evaluations are less attributed toteaching quality and more to patterns of students’class-taking.
5

Conclusion

Our results do not support the view that online classes improved teaching quality and thus
students’course evaluations under the COVID-19 pandemic. Rather, the results are moreconsistent with existing studies that are conservative about the e¤ect of online teaching(Cacault et al. 2021; Bettinger et al. 2017; Figlio et al. 2013). However, this does not
imply that online teaching itself never improves students’course evaluations and, ultimately,
teaching quality. We are sure that they do if online methods are ‡exibly and appropriately
used in the entire education system, hopefully even in the post-COVID-19 regime.

6References
Agostinelli, F., M. Doepke, G. Sorrenti, and F. Zilibotti (2020) “When the great equalizer
shuts down: schools, peers, and parents in pandemic times,”Working Paper 28264, National
Bureau of Economic Research.
Alpert, W. T., K. A. Couch, and O. R. Harmon (2016) “A randomized assessment of online
learning,”American Economic Review 106(5), 378–382.
Andrew, A., S. Cattan, M. Costa Dias, C. Farquharson, L. Kraftman, S. Krutikova, A.
Phimister, and A. Sevilla (2020) “Inequalities in children’s experiences of home learning
during the covid-19 lockdown in England,”Fiscal Studies 41(3), 653–683.
Aucejo, E. M., J. F. French, M. P. U. Araya, and B. Zafar (2020) “The impact of covid-19
on student experiences and expectations: Evidence from a survey,” Working Paper 27392,
National Bureau of Economic Research.
Bacher-Hicks, A., J. Goodman, and C. Mulhern (2020) “Inequality in household adaptation
to schooling shocks: Covid-induced online learning engagement in real time,”Working Paper
27555, National Bureau of Economic Research.
Bettinger, E.P., L. Fox, S. Loeb, and E.S. Taylor (2017) “Virtual classrooms: How online
college courses a¤ect student success,”American Economic Review 107 (9), 2855–2875.
Bryk, A.S. and S.W. Raudenbush (2002) “Hierarchical linear models: Applications and data
analysis methods,”Sage Publication, Inc., Thousand Oaks.
Cacault, M.P., C. Hidebrand, J. Laurent-Lucchetti, M. Pellizzari (2021) “Distance learningin higher education: Evidence from a randomized experiment,” forthcoming in Journal of
the European Economic Association.
Chen, J., and T.-F. Lin (2016) “. The bene…t of providing face-to-face lectures in online learning microeconomics courses: Evidence from a regression discontinuity design experiment,”
Economics Bulletin 36(4), 2094–2116.
Coates, D., B. R. Humphreys, J. Kane, and M. A. Vachris (2004) ““No signi…cant distance” between face-to-face and online instruction: evidence from principles of economics,”
Economics of Education Review 23(5), 533–546.
Engzell, P., A. Frey, and M. D. Verhagen (2020) “Learning inequality during the covid-19
pandemic,”SocArXiv Papers. (DOI: 10.31235/osf.io/ve4z7)
Escueta, M., V. Quan, A. J. Nickow, and P. Oreopoulos (2017) “Education technology: An
evidence-based review,”Working Paper 23744, National Bureau of Economic Research.

7

Figlio, D., M. Rush, and L. Yin (2013) “Is it live or is it internet? Experimental estimatesof the e¤ects of online instruction on student learning,” Journal of Labor Economics 31(4),763–784.
Heissel, J. (2016) “The relative bene…ts of live versus online delivery: Evidence from virtual
algebra I in North Carolina,”Economics of Education Review 53, 99–115.
Ikeda, M. and S. Yamaguchi (2021) “Online learning during school closure due to COVID19,”The Japanese Economic Review 72, 471-507.
Orlov, G., D. McKee, J. Berry, A. Boyle, T. DiCiccio, T. Ransom, A. Rees-Jones, and J.
Stoye (2021) “Learning during the covid-19 pandemic: It is not who you teach, but how youteach,”Economics Letters 202, 109812.
Perron, P., and Y. Yamamoto (2015) “Using OLS to estimate and test for structural changes
in models with endogenous regressors”, Journal of Applied Econometrics 30, 119-144, 2015.

8

Table 1. Summary statistics(1) From 2018 to 2020
Numb er ofAttendance

Study hours

Clearness ofClearness ofClearness ofEnthusiasm of

Signi…cance ofoutside the classob jectives

evaluation

explanation

the instructor

the course

4.77
4.64
4.55
4.32

1.70
1.55
1.41
1.47

3.89
4.02
3.98
4.06

3.84
3.99
3.93
4.04

3.88
4.05
4.04
4.15

4.14
4.25
4.25
4.33

4.00
4.09
4.07
4.19

0.65
0.80
0.89
1.00

0.70
0.67
0.60
0.62

0.88
0.83,
0.84
0.82

0.93
0.89
0.91
0.86

0.99
0.92
0.93
0.91

0.83
0.78
0.79
0.77

0.92
0.89
0.90
0.86

-3.55
-2.68
-2.34
-1.59

0.48
0.83
1.16
0.98

-0.92
-1.00
-1.09
-1.07

-0.75
-0.91
-1.00
-1.01

-0.89
-1.08
-1.08
-1.36

-1.09
-1.21
-1.24
-1.36

-1.01
-1.13
-1.14
-1.27

16.97
10.42
8.36
5.03

2.13
2.54
3.30
2.91

4.09
4.54
4.87
4.76

3.41
3.87
4.14
4.36

3.54
3.87
4.14
4.36

4.66
5.25
5.35
5.64

4.10
4.51
4.59
4.97

resp onses

mean
1st
12107
2nd
7812
3rd
4972
4th
1692
standard deviation
1st
12107
2nd
7812
3rd
4972
4th
1692
skewness
1st
12107
2nd
7812
3rd
4972
4th
1692
kurtosis
1st
12107
2nd
7812
3rd
4972
4th
1692

(2) 2020
Numb er ofAttendance

Study hours

Clearness ofClearness ofClearness ofEnthusiasm of

Signi…cance ofoutside the classob jectives

evaluation

explanation

the instructor

the course

4.92
4.90
4.84
4.75

1.95
1.91
1.76
1.73

4.06
4.21
4.12
4.20

3.86
4.07
3.97
4.06

3.99
4.17
4.15
4.20

4.22
4.33
4.30
4.36

4.17
4.25
4.20
4.27

0.39
0.44
0.58
0.70

0.67
0.67
0.66
0.65

0.77
0.74
0.80
0.74

0.91
0.89
0.93
0.89

0.92
0.87
0.90
0.93

0.79
0.76
0.79
0.78

0.81
0.80
0.84
0.86

-6.23
-5.67
-4.67
-3.53

0.06
0.11
0.32
0.34

-0.96
-1.15
-1.18
-1.27

-0.77
-1.02
-1.03
-1.22

-0.98
-1.30
-1.19
-1.55

-1.14
-1.40
-1.35
-1.61

-1.10
-1.32
-1.23
-1.55

49.25
40.83
27.24
16.57

2.26
2.20
2.22
2.25

4.72
5.63
5.27
6.32

3.51
4.13
4.06
4.90

3.94
5.16
4.38
5.63

5.00
6.11
5.65
6.76

4.70
5.61
4.92
-1.55

resp onses

mean
1st
3818
2nd
2830
3rd
1805
4th
522
standard deviation
1st
3818
2nd
2830
3rd
1805
4th
522
skewness
1st
3818
2nd
2830
3rd
1805
4th
522
kurtosis
1st
3818
2nd
2830
3rd
1805
4th
522

Table 1. Summary statistics (continued)(3) 2019
Numb er ofAttendance

Study hours

Clearness ofClearness ofClearness ofEnthusiasm of

Signi…cance ofoutside the classob jectives

evaluation

explanation

the instructor

the course

4.70
4.55
4.47
4.17

1.70
1.48
1.36
1.38

3.85
3.98
4.00
4.01

3.82
3.96
3.97
4.02

3.80
4.02
4.03
4.10

4.10
4.24
4.28
4.33

3.93
4.04
4.07
4.15

0.73
0.88
0.92
1.02

0.70
0.65
0.57
0.59

0.91
0.83
0.81
0.87

0.96
0.88
0.90
0.84

1.03
0.93
0.92
0.91

0.84
0.76
0.75
0.76

0.94
0.89
0.86
0.87

-3.03
-2.34
-2.03
-1.24

0.49
1.02
1.33
1.29

-0.89
-0.93
-1.05
-0.98

-0.72
-0.87
-1.05
-0.82

-0.83
-1.00
-1.05
-1.18

-1.00
-1.07
-1.18
-1.12

-0.98
-1.06
-1.05
-1.20

13.08
8.38
6.88
4.12

2.12
2.88
3.78
3.62

3.90
4.35
5.00
4.23

3.25
3.85
4.39
3.87

3.31
3.96
4.18
4.51

4.35
4.84
5.38
4.41

3.94
4.36
4.53
4.80

resp onses

mean
1st
3954
2nd
2798
3rd
1770
4th
599
standard deviation
1st
3954
2nd
2798
3rd
1770
4th
599
skewness
1st
3954
2nd
2798
3rd
1770
4th
599
kurtosis
1st
3954
2nd
2798
3rd
1770
4th
599

(4) 2018
Numb er ofAttendance

Studyhours

Clearnessof

Clearnessof

Clearnessof

Enthusiasm of

Signi…canceof

outside the classob jectives

evaluation

explanation

the instructor

thecourse

4.67
4.53
4.40
4.04

1.42
1.33
1.19
1.27

3.75
3.91
3.85
3.98

3.85
3.95
3.86
4.03

3.82
3.98
3.96
4.16

4.10
4.21
4.18
4.30

3.89
4.01
3.97
4.15

0.77
0.88
0.99
1.11

0.61
0.55
0.42
0.49

0.94
0.87
0.88
0.84

0.93
0.91
0.91
0.85

1.01
0.94
0.96
0.89

0.87
0.82
0.82
0.77

0.98
0.94
0.96
0.85

-2.87
-2.18
-1.89
-1.11

1.15
1.43
2.02
1.58

-0.79
-0.93
-1.06
-0.94

-0.76
-0.87
-0.93
-0.95

-0.83
-1.00
-1.03
-1.38

-1.09
-1.19
-1.19
-1.34

-0.88
-1.03
-1.11
-1.07

11.67
7.64
6.18
3.50

3.25
4.08
6.23
4.56

3.60
4.15
4.54
4.27

3.47
3.74
4.03
4.18

3.38
3.93
4.06
5.32

4.58
5.04
5.05
5.75

3.63
4.05
4.28
4.20

resp onses

mean
1st
4335
2nd
2184
3rd
1397
4th
571
standard deviation
1st
4335
2nd
2184
3rd
1397
4th
571
skewness
1st
4335
2nd
2184
3rd
1397
4th
571
kurtosis
1st
4335
2nd
2184
3rd
1397
4th
571

Table 2. Di¤erence-in-di¤erences estimation
using the raw questionnarie data
Attendance

Study hours

Clearness ofClearness ofClearness ofEnthusiasm of

Signi…cance ofoutside the classob jectives

evaluation

explanation

the instructor

the course

1

0.329***(0.014)0.299***(0.011)0.245***(0.016)

0.046***(0.017)0.207***(0.017)0.100***(0.015)0.234***(0.017)2-0.072***(0.016)-0.249***(0.012)-0.137***(0.017)-0.063***(0.019)-0.053***(0.019)-0.028*(0.016)-0.106***(0.018)1+ 2
F statp value

0.257***
105.250.000

0.050***
6.728
0.009

0.108***
14.9570.000-0.0170.326
0.568

0.154***
26.3120.000

0.072***
7.561
0.006

0.128***
18.8640.000

adj:R2
N

0.146
26583

0.315
26583

0.118
26583

0.097
26583

0.187
26583

0.128
26583

0.121
26583

Note: Standard errors are shown in parentheses. ***, **, and * indicate signi…cance at the 1%, the 5%,
and the 10% levels, respectively. “F stat ” shows the F statistic for H0 : 1 + 2 = 0
against H1 : 1 + 2 6= 0. The regression includes dummy variables for the major of student,
grade of the student, courses, and instructors, as control variables but their coe¢ cients aresuppressed.

Table 3. Di¤erence-in-di¤erences estimation
using the student-level means
Attendance

Study hours

Clearness ofClearness ofClearness ofEnthusiasm of

Signi…cance ofoutside the classob jectives

evaluation

explanation

the instructor

the course

1

0.377***(0.019)0.357***(0.015)0.195***(0.020)0.054***(0.021)0.171***(0.021)0.102***(0.018)0.215***(0.019)2-0.045***(0.018)-0.184***(0.014)-0.089***(0.019)-0.017***(0.020)-0.014***(0.020)-0.017*(0.018)-0.031***(0.019)1+ 2
F statp value

0.332***
114.290.000

0.173***
50.5220.000

0.106***
10.378
0.001

0.037
1.151
0.283

0.157***
27.6920.000

0.085***
7.520
0.006

0.184***
29.5250.000

adj:R2
N

0.122
7354

0.203
7354

0.044
7354

0.016
7354

0.039
7354

0.029
7354

0.036
7354

Note: Same as in Table 2 except that the control variables include the dummy variables for the
student’s major and grade.

Table 4. Di¤erence-in-di¤erences estimation
using the instructor-level means
Attendance

Study hours

Clearness ofClearness ofClearness ofEnthusiasm of

Signi…cance ofoutside the classob jectives

evaluation

explanation

the instructor

the course

10.270***(0.032)0.261***(0.034)0.137***(0.034)-0.001(0.038)0.088*(0.043)0.037(0.037)0.127***(0.038)2-0.062***(0.032)-0.196***(0.032)-0.114***(0.033)-0.056(0.036)-0.050(0.042)-0.042(0.035)-0.050(0.037)1+ 2
F statp value

0.208***
14.984
0.000

0.065
1.307
0.253

0.023
0.162
0.688

-0.057
0.787
0.375

0.038
0.271
0.603

-0.005
0.007
0.931

0.077
1.378
0.241

adj:R2
N

0.167
896

0.264
896

0.072
896

-0.001

0.033
896

0.020
896

0.054
896

896

Notes: Same as in Table 2 except that the control variables include the dummy variables for the
instructor’s department.

Figure 1. Di¤erence-in-di¤erences analysis

Figure 2. E¤ects of students’class-taking on average scores

2019

Average scores
instructor

student

questionnaire data

3.0

A

4

student-level

3.0

B

2

instructor-level

3.0

2020

Average scores

Questionnaire-level
instructor student
A

4

B

2

Student-level
instructor

student

A

4

B

3

4

questionnaire data

3.3

student-level

3.5

instructor-level

3.0

Instructor-level
instructor student
A

4B2