Educational disparities in STEM during COVID-induced distance learning and a potential strategy to address them

Domain: chemistry
Published: 2026-02-26
Editorial outcome: minor_revision

Abstract
The COVID-19 pandemic prompted widespread school closures and a swift transition to distance learning,raisingconcernsaboutconsequencesfIorstudentsuccess.Thisstudyexploresacademicout- comesinSTEM(Science,Technology,Engine ering,andMathematics)coursesduringCOVID-induced distance learning using student-course recEords at an U.S. public university. We particularly focus on students from underserved populations and who have below average academic outcomes. Employing an expected shortfall regression strategy, we examine outcomes in the bottom quintile and compare differences between students whose household income were in the lowest bracket or without a par- ent holding a four-year collegeIdegree and peers for whom this was not the case, in distance and in-person learning. We showTthat during distance learning, students in disadvantaged populations in the lowest quintile hadRaverage grade differences of 0.11 and 0.06 points, respectively. We also find that targeted instructional changes in an introductory physics course were associated with narrower achievementgaps.TAheseresultssuggestthatwhiledistancelearningposedchallengesforunderserved students,deliberatestrategiestoincreaseinteractionmaypotentiallysupportgreaterequityinSTEM education.

Introduction
The COVID-19 pandemic significantly disrupted the education system and led to mass school closures worldwide [1]. The majority of colleges and universities transitioned to online/distance learning—84% of all undergraduates in the US had some or all of their classes moved to online- only in Spring 2020 [2]. Distance learning con- tinued for several semesters following the onset of COVID-19. In Fall 2021, 61% of all under- graduate students were enrolled in at least one remote course while 28% were enrolled in remote courses exclusively [3]. Prior to the pandemic, thereweresomestudiesthatexploredthebenefits and limitations of online learning. For example, onecharacteristicofonlinelearningistheflexibil- ity to study at one’s own pace [4]. However, this suggeststhatstudentsuccessmaydependonhigh self-discipline and time management skills [5]. In addition,thedesignofacoursewillalsoneedtobe suitableforonlinelearning—thenatureandextent of peer and student-faculty interactions must be adapted to ensure student learning outcomes in online courses [6].

Student populations such as Black, Latin, Indigenous students, low-income (household income in the lowest bracket) or first-generation (parents did not complete a four-year college degree)arehistoricallyunderrepresentedinhigher education and face barriers to opportunity and high-quality instruction [7]. Promoting equity in higher education is a major priority in the United States [8], and with the increase of online learn- ing in higher education, equity in online learning should be explored. Investigating the effect of distance learning on disadvantaged students is crucialtoaddresstheextentofgapsineducation.

We focus primarily on students who are low- income (household income in the lowest bracket) or first-generation (parental education less than a four-year college degree), who face great chal- lenges in higher education [9]. Students who are low-income or first-generation disproportion- ately come from ethnic and racial minority back- grounds, have less academic preparation, tend to beolder,andfacemoreobligationsoutsidecollegIe. Even after taking their background, preparat ion, and enrollment characteristics into account, stu- dents who are low-income or first-geneLration still encounter greater risks of failure inChigher educa- tion [10–16]. They are less likely to participate in student experiences that proTmote academic suc- cess such as studying in groups, interacting with faculty and peers, and utilizing support services [15, 17–26].

Specifically, in distance learning, students whose household income is in the lowest bracket aremorelikelytofacepracticalchallengessuchas not having a quiet workplace, not knowing where to seek course assistance, and managing conflicts withhomeresponsibilities[27].Theyarealsomore likely to face internet connectivity issues, hard- ware and software issues that negatively impact the effectiveness of distance learning [8, 28]. On theotherhand,first-generationstudentsmayface challenges in distance learning due to staying in households where their parents lack the edu- cational experience to provide support. At the same time, they are more often expected to assist with sibling care at home [29]. Moreover, first- generation students tend to have lower access to digital tools and are less familiar with school cul- ture, knowledge and skills which in turn affect their academic performances [30].

When studying the academic gap, we are par- ticularly interested in gaps among students who have below average academic outcomes. Many studies have investigated strategies to improve the performance of low-performing postsecondary students [31–35]. Students who begin with poor performance tend to struggle throughout their academic journey [33], a problem which may be exacerbated by limited in-person instruction and interaction. Students at the lower end of aca- demic outcomes are at risk of being placed on academic probation, which can delay graduation [36]. First generation students and students from low-income families already face a graduation gap: undergraduates who are both low-income and first-geSneration have a six-year graduation rate atEfour-year institutions of 41% compared to 73% for undergraduates who are neither low- income nor first-generation [37]. We hypothesize that lower achieving students may be more nega- tively affected by distance learning than an aver- age student, further enlarging the achievement gapbetweenstudentswhoarelow-incomeorfirst- generation and their peers. While prior literature suggestthatdistancelearningmayhavenoimpact orevenimproveaveragestudentoutcomes[38–40], additional evidence suggests online learning may negativelyaffectlow-performingstudents[41,42].

In this study, we focus on STEM courses in higher education. Enhancing social mobility for students is important for universities, and STEM isoftenseenasavehicleforeconomicopportunity and social mobility [43]. STEM degrees have long been promoted to students as a potential path to high-paying careers [9, 44–47], and there is a robustandgrowingdemandforstudentsskilledin STEM disciplines in both STEM and non-STEM occupations[48].StudentspursuingSTEMmajors are more likely to have a job aligned with their field of study compared to those from non-STEM majors[46]andseesignificantwageadvantagesin the job market [45, 46]. However, students from underrepresented populations are less likely to study or complete STEM degrees. Once they are enrolled in a STEM program, retaining these stu- dents poses a great challenge due to factors such as insufficient pre-college preparation and lack of supportive social environments [49–52]. Students who are first-generation or whose parents have lowerincomesaremorelikelytoleavetheirSTEM program by changing majors or leaving college without completing a degree [49, 53]. The effect distance learning may have on the academic per- formance of students who are low-income and first-generation in STEM courses is of interest, as college GPA and first-semester GPA are impor- tant predictors of their STEM degree attainment [54–56] and of obtaining a STEM-related job [57].

Distance learning produces additional chal- lenges when it comes to STEM courses, as many STEM courses rely on specialized software and equipment which may not be readily accessi- ble outside of traditional classroom settings [58]. In distance learning, students in STEM courses report concern about losing hands-on experience in laboratories [59, 60] and struggle with engage- ment and motivation after the transition [27, 61– 63].TheintersectionofSTEM,barrierstosuccess in STEM faced by students who are low-income and first-generation, and challenges faced in a distancelearningsettingarethusofgreatinterest.

The primary objective of our study is to examine the association between COVID-induced learning modality (in particular, distance lear n- ing) and student academic outcomes and learn- ing gaps in STEM courses, particularlyLfocusing on underrepresented and lower perfCorming stu- dents.Toachievethisgoal,weobtainlongitudinal student administrative and eduTcational outcome data from the University of Michigan (referred to as the University hereafter) during the years from Fall 2016 to Fall 2022. Our dataset con- tains information about student course enroll- ment and their respective performances, as well as their demographic, socioeconomic and aca- demic backgrounds. Focusing on semester-long undergraduate courses, our data contains over 1.2 million student-course records, more than half of which are for STEM courses. The study has received approval from the University’s Institu- tional Review Board and the data are procured throughaMemorandumofUnderstanding(MOU) agreement with the University’s enrollment man- agement office to ensure strict protection of stu- dent privacy.

Before the onset of the COVID-19 pandemic, almostallclassesattheUniversityweretaughtin person. In March 2020, the University announced it would switch all classes to distance learning to limit the spread of COVID-19. It then resumed limited in-person classes in Fall 2020 and Winter 2021. However, the majority (75.1% and 89.2%, respectively) of classes were taught online dur- ing these terms. Later in Fall 2021, a significant portion (86.3%) of classes returned to in-person learning while some classes (13.7%) continued to be taught online or in a hybrid mode. The selection of learning modality is primarily at the discretion of individual instructors, in alignment with guidelines from the University and their respectiveacademicunits.Thesedifferentialtran- sitions in learning modality by courses thus allow us to compare academic achievements across dif- ferent student subgroups under different learning modalities, while controlling for the effect of the pandemic itself.

The resultSs indicate that COVID-induced dis- tancelearEningislinkedtolargeracademicgapsin STEM courses. Students whose household income were in the lowest bracket and first-generation students in the lowest quintile have grade point d ifferences of 0.114 and 0.064 under distance learning, respectively, compared to their peers. This difference is significant given the lower- quintile mean grade points of 2.48 overall, and 2.00, 2.02, and 1.69 for students who are low- income,first-generation,andbothlow-incomeand first-generation, respectively.

Moreover,atbelowa2.0cumulativeGPA,and for some colleges below a 2.0 semester GPA, stu- dents will be placed on academic probation with the risk of having enrollment withheld or being placed on academic recess for one year. In some majors, students who achieve below 2.0 grade points will fail to fulfill the minimum requirement for common required and core courses in their majors. Thus, a reduction of 0.114 or 0.064 in grade points for students within the lowest quin- tile can result in serious academic consequences and delays to graduation. In non-STEM courses, we see a smaller gap of around 0.05 grade points.

Given the greater disparities observed for lower-performing students, we then investigate potential instructional interventions which may reduce disparities in COVID-induced distance learning. Specifically, we consider an introductory physicscoursethat,ineachsemesterstartingfrom Fall 2019, offered additional sections as part of a recent university initiative, i.e., the Founda- tionalCourseInitiative(FCI),aimedatimproving learning experiences for students. Course sections offered as part of this programs provide students withgreateraccesstoinstructorsupportandmore interactions with their peers consistently imple- mentedduringthein-personanddistancelearning environments.Forexample,duringdistancelearn- ing, these sections continued to offer students greaterinteractionsthroughdiscussionsandgroup workfacilitatedbyonlineplatformssuchasZoom and Gather Town.

We find that this course initiative, which provide students with additional support and student-instructor and peer interactions, signifi- cantly reduces the observed achievement gaps in COVID-induced distance learning. These findings not only corroborate previous results in the lit- erature that increased interaction with peers and instructors is associated with greater success in online learning in general [6] and in the context of the pandemic [64, 65], they further demon- strate that enhanced interaction can effectively narrow the achievement gaps for students from disadvantaged backgrounds.

Results
2.1 Institutional background

We use student administrative and educational outcomedatafromalargepublicuniversityinthe US which enrolls around fifty thousand students everyyear,asof2022.TheUniversitytransitioned to distance learning inAthe middle of the Winter 2020 (WN20) semester. The majority of univer- sity courses were taught remotely in Fall 2020 and Winter 2021, and many returned to in per- son teaching in Fall 2021. A small percentage of courses were hybrid modality from Fall 2020 to Winter 2022. The proportion of STEM course sectionsineachlearningmodalitybysemesterare presented in Figure 1.

2.2 Empirical strategy

We estimate the association between distance learning and grade outcomes for students who are low-income and first-generation by contrast- ingthedifferenceineducationaloutcomesbetween in person and distance learning modality in stu- dents who are low-income and first-generation relative to that of students who are non-low- incomeandnon-first-generation.Thusourmethod focuses on the interaction between distance learn- ing and low-income or first-generation status. In additiontostudentswhoarelow-incomeandfirst- generation, we are interested in lower performing students within these categories. To estimate the association between distance learning and grade outcomesforthelowerperformingstudentsatthe lower quintile, we employ the expected shortfall (ES) regression which allows for this specifica- tion. ES regression allows us to study the average outcome for the students whose outcomes fall in the bottom quintile of the entire population. More details about ES regression can be found in Section 4.

Second,Swe estimate the outcomes of inter- ventionsEthat provide increased student atten- tion, student-peer and student-instructor interac- tion, particularly for students underrepresented in higher education and STEM, taking advantage of the university’s course initiatives implemented in a large introductory physics course. We imple- ment a three-way-interaction strategy within the expected shortfall regression, contrasting differ- ences between students who are low-income and their non-low-income peers and students who are first-generation and their non-first-generation peers enrolled within these special course sections versus in standard sections, in distance learning versus in person learning.

In our regressions, we control for other stu- dent, course, and term variables and separate the pandemic association with grade values from the distance learning association with grade values.

2.3 Data and definition of outcomes

Our data consist of semester data on student grades for undergraduate students at University between Fall 2016 and Fall 2022. Because classes transitioned to remote midway during the Winter 2020 semester at the onset of the pandemic, this semester is excluded from our analysis. Restrict- ing classes to undergraduate level semester-long classes that follow a typical A-E grading scale, our data comprise 1,256,632 observations. Limit- ing to STEM courses, our data comprise 675,463 observations. The outcome variable, grade value, is taken as the numerical equivalent of the letter grade given by an instructor to a student. Oth- erwise, the official letter grade received may be inSatisfactory/NotRecordedorPass/Failformat. A student’s low-income status, is described as low-income if a student reported their household gross income to be in the lowest income bracket (less than $25,000 a year) and non-low-income otherwise. A student’s first-generation status is described as first-generation if a student reported their parents’ maximum educational attainment level to be less than a Bachelor’s degree and non- first-generationotherwise.Inourdataset,4.88%of students are categorized as low-income and 13.0% are categorized as first-generation.

2.4 Association between distance learning and grade outcomes in STEM courses

Panel A of Table 1 presents estimates of the association between distance learning and grade outcomes(ingradepoints:0.0-4.0)forlow-income versusnon-low-incomestudentsinSTEMcourses. Our results suggest that there may exist differ- encesintheassociationbetweendistancelearning andeducationaloutcomesforlow-incomestuden ts in the bottom quintile of students in terms of grade values. Among the bottom quintilLe of low- income students and the bottom quiCntile of non- low-income students, low-income students show a significantgradedifferenceof–0.114(p<0.001;CI: –0.181,–0.047)points,onaverage,relativetotheir non-low-income peers during distance learning.

Panel B of Table 1 presents estimates of the association between distance learning and grade outcomes for first-generation versus non- first-generationstudentsinSTEMcourses.Similar to the first model, the table suggests that there exists differences in the association between dis- tance learning and educational outcomes for the bottom quintile of first-generation students. We found that the bottom quintile of first-generation students show a significant grade difference of -0.064 (p=0.003; CI: -0.106, -0.023) points, on average, relative to the bottom quintile of non- first-generationstudentsduringdistancelearning. Forbothfirst-generationandlow-incomestudents in the bottom quintile, there is no significant link between hybrid learning and grade outcomes as opposed to in person learning.

Theassociationbetweendistancelearningand gradeoutcomesforthebottomquintileofstudents in STEM courses can also be seen in Figure 2. We observe that the association between dis- tance learning and grade outcomes is significantly negative for both the bottom quintile of non-low- income and non-first-generation students and the bottomquintileoflow-incomeandfirst-generation students, but the estimate is much more negative for low-income and first-generation students.

Wedetectasmalldisparity(around-0.025)for low-income students but not for first-generation students in the mean results as shown in Sup- plementary Table 8. Results for all courses are displayed in Supplementary Table 9 and Supple- mentaryFigure4.ResultsforSTEMcoursesfrom a regression including interactions for both low- income and first-generation status are displayed in SupplemenStary Table 12 and Supplementary Figure 3.EThere is a significant negative associ- ation between distance learning and grade out- comes for the bottom quintile of non-low income and non-first generation, only low-income, only fi rst-generation, and both low-income and first-generation students. The additional association Ibetween distance learning and grade outcomes on only low-income, only first-generation, and both low-income and first-generation students is significantly negative. The results including inter- actions for both low-income and first-generation status for all courses are displayed in Supplemen- tary Table 13 and Supplementary Figure 5. We additionally considered the association between distance learning, hybrid learning, and grade out- comes by sex and by underrepresented minority status. The results are in Supplementary Table 1 in Supplementary Information A.1 and Supple- mentary Table 2 and Supplementary Figure 1 in Supplementary Information A.2, respectively. As COVID-induced distance learning is intertwined with factors such as at-home conditions, we con- sidered the association between distance learning and grade outcomes in courses that had distance learning after most courses returned to in-person on campus. The results are in Supplementary Tables 5-6 in Appendix A.4.

2.5 Association between distance learning and grade outcomes by subject category

InadditiontoacrossallSTEMcourses,wefurther consideredtheassociationbetweendistancelearn- ing and grade outcomes for courses in different subject categories. Specifically, we considered the following subject categories: Physical & Natural Sciences, Social Sciences, Engineering, Medicine & Health, and Humanities, Arts & Languages.

All Physical & Natural Sciences and Engineer- ing courses are considered STEM, while some courses in Medicine & Health and Social Sciences and almost no courses in Humanities, Arts & Languages are considered STEM. The association betweendistancelearning,low-incomeorfirstgen- erationstatus,andgradeoutcomesforthebottom quintile of students in these groups is presented in Figure 3. For Physical & Natural Sciences and Engineering, we observe similar results for low- income and first-generation students as in our main results for all STEM courses. The negative associationbetweendistancelearning,low-income or first-generation status, and grade outcomes is generally largest in Physical & Natural ScienceIs, Engineering,andMedicine&Healthcourses,c om- pared to the Social Sciences and HumEanities, Arts & Languages. Tables presentingLestimates of the association between distanceClearning and grade outcomes comparable to that in Table 1 are in Supplementary Table 1 T 0 in Supplementary Information B.

2.6 Association Abetween the COVID-19 pandemic and grade outcomes in STEM courses

In the previous sections, we considered the asso- ciation between distance learning and grade outcomes, controlling for associations with the COVID-19 pandemic. Specifically, we include in the regressions an indicator of post Winter 2020 semesters as well as its interaction with low- income and first-generation statuses to allow for differential associations between the pandemic and grade outcomes on different subgroups of students. The results are presented in Supple- mentary Figure 6 in Supplementary Information B. The figure shows clearly that there exists a negative association between the pandemic and gradeoutcomesforallcoursesandforthesubjects considered in Section 2.5 is presented in Supple- mentary Figure 7 in Supplementary Information B and Supplementary Figure 9 in Supplementary Information B, respectively.

2.7 Is an intervention associated with smaller disparity between groups?

In this section, we examine the link between an intervention that aims to increase student-peer interaction and student-instructor interaction on therelationSshipbetweendistancelearningandthe grade ouEtcomes of underrepresented students. We consider an introductory physics course that is offered in both Winter and Fall semesters, and e P nrollsonaverage582studentspersemester.This course is offered in multiple sections. Besides the standard sections, starting from Fall 2019, the course includes two sections of students as part of the Foundational Course Initiative (FCI). The FCI sections provide students with greater access to instructor support, more peer interaction by having small groups within a large class and frequent discussion and group/individual work instead of lectures.

FCI sections consist of around 200 students per term and are about a third of the total stu- dents in the course starting in Fall 2019. All sections(FCI,standard)offeredinpersoninstruc- tions before Winter 2020, and shifted to distance learning in Fall 2020 and Winter 2021. While FCI sections were in person the following semester (Fall 2021), standard sections remained distance and only shifted back to in person instructions in Winter 2022.

Our physics course data comprise 7,373 obser- vations, 1,212 of which are for FCI sections. The demographics of the FCI sections are approxi- matelythesameasthoseofthestandardsections. A brief summary of the students in different sections is presented in Table 2 in Section 4.

As all sections (FCI, standard) experienced both in person and distance learning, and the learning modality is mostly syncronized across all sections (except for one semester, Fall 2021), we can compare the relationship between dis- tance learning and grade outcomes based on the type of sections. We continue to use the instruc- tor inputted grade as our outcome as before.

In this section, we conduct expected shortfall regressions with three-way interaction terms to examine how the innovative course sections (i.e., FCI), moderate the association between distance learning and grade outcomes for low-income and first-generation students.

The difference in association between dis- tance learning and grade outcomes for the reg- ular sections and FCI sections is presented in Figure 4. For the bottom quintile of non-low- income and non-first-generation students, the association between distance learning and grade outcomes is not statistically significantly different between regular sections and FCI sections (effect size=0.133; p=0.179; CI: -0.061, 0.327 and effect size=0.099; p=0.333; CI: -0.102, 0.299, respec- tively). For first-generation students, the associa- tioneffectsizeincreasesfromstandardtoFCIand becomes significantly positive in the FCI sections (effect size=0.792; p=0.010; CI: 0.188, 1.396). For low-incomestudents,wealsoobserveaincreasein theeffectsizefortheassociationbetweendistan ce learning and grade outcomes from standard to FCIsectiontypes(effectsize=1.063;p=0.021;CI: 0.159, 1.968). In other words, under tChe interven- tion(i.e.,FCI),thenegativelinkbetweendistance learning and grade outcomes in first-generation and low-income students is reduced.

The intervention is not linked to a statisti- cally significant association between the COVID- 19 pandemic and grade outcomes. The results are presented in Supplementary Figure 8 in Supple- mentary Information B.

Methods
The study has received approval from the Uni- versity of Michigan’s The Health Sciences and Behavioral Sciences Institutional Review Board and the data are procured through a Memoran- dum of Understanding (MOU) agreement with the University’s enrollment management office to ensure strict protection of student privacy.

Data. Our data set consists of administrative data and educational outcomes for all students enrolled at the University from the mid-1990s through 2022. In our main sample of STEM courses for the years 2016 to 2022, we have a total of 675,463 data points for 1200 unique courses and 66,461 unique students. Of these, approximately 43% of the data points are after Winter 2020. Additionally, information about the introductory physics course sections were made available through the university’s collection of historical syllabi.

Measures and definitions. We define grade values, the outcome variable of interest, as the numerical equivalent of the letter grade given by an instructor to a student (A+/A=4.0, A- =3.7, B+=3.3, B=3.0, B-=2.7, C+=2.3, C=2.0, C-=1.7, D+=1.3, D=1.0, D-=0.7, E=0.0). The instructor inputted grade is the grade given by an instructor to a student; the official grade that appears on a student’s transcript is the trans- lated value of the instructor inputted grade as defined by the school or college grading policY. Official grades may take other forms other t han a letter grade. Due to COVID-19, official grades may be chosen to be masked by stLudents as Satisfactory/Not Recorded - COVICD.

Courses are categorized as STEM by their instructional area based on thTe 2023 U.S. Depart- ment of Homeland Security (DHS) STEM Desig- nated Degree Program List [103]. The list desig- nates all Engineering, Biological and Biomedical Sciences, Mathematics and Statistics, and Phys- ical Sciences (determined by the Department of Education’s Classification of Instructional Pro- grams) programs as STEM, and also includes some programs in related fields such as Natural Resources and Conservation, Education, Psychol- ogy, etc. For categorizing instructional areas into subject categories, Physical & Natural Sciences include life sciences, physics, chemistry, earth sci- ence, astronomy, and mathematics. Engineering includes all fields of engineering, as well as infor- mation and technology related subjects. Social Sciences include social and behavioral sciences such as sociology, economics law, anthropology, business, etc. Medicine and Health include areas inKinesiology,Pre-Medicine,Nursing,Pharmacy, Dentistry, and Public Health. Based on the DHS Designated Degree Program List, all Physical & Natural Sciences and Engineering divisions, some Social Sciences or Humanities divisions such as archaeology and psychology, and some Medicine and Health divisions such as kinesiology, phar- macology,immunology,andbiologicalsciencesare designated STEM.

For the student categories of interest, a stu- dent’s low-income status is defined as estimated gross family income less than $25,000 and first- generation status as parent’s maximum education level less than a Bachelor’s degree. Estimated gross family income and parent’s maximum edu- cation level are self-reported in a student’s appli- cation for college admission; students that do not report these are considered non-low-income or non-first-generation. A sensitivity analysis of our results wheSn excluding these observations where househoEldincomeorparent’seducationallevelare notreportedisinSupplementaryInformationA.3 Tables 3-4 and Supplementary Figure 2.

Of the unique students in our main data set, 4.88% are considered low-income and 13.0% are considered first-generation. Among low-income students, 52.0% are first-generation, and among first-generationstudents,19.6%arelow-income.A brief summary of the students in all considered courses and in the introductory physics course examined in Section 2.7 is presented in Table 2.

Intervention. The intervention we considered was FCI sections in a large introductory physics R course. We provide a brief description of how this program appears in the physics course. Starting in 2018, FCI partnered with the physics course over a three year pilot period. As a result of the partnership, starting from Fall 2019 onward, two sections of the physics course under study were transformed into a Studio Model , in which instructors used the same curriculum (e.g., same teaching materials) and students were held up to the same course requirements as those in the standard sections. The main difference, however, is that it created a radically different student experience. Each FCI section is split into several neighborhoods of students who spend the whole term together. Additionally, within a neighbor- hood, students work in small groups of three and cycle through groups throughout the term such that every student in a neighborhood will have workedwitheveryotherstudentbytheendofthe semester. Each neighborhood is directly overseen by a undergraduate learning assistant. A large portion of lecture time is dedicated group work within these groups which during distance learn- ing is done synchronously on Gather Town. The rest of the lecture is review or preview and indi- vidualworkperiods.Regularsectionsmeetforone hourlecturesfourtimesaweek,whileFCIsections meet twice a week for two hours each. Moreover, eachsectionemploysfiveadditionallearningassis- tants,agraduatestudentinstructor,andafaculty member, all part of the instruction team. In this way,theFCIsectionincreasesthestudents’inter- actions with one another and with instructors, and results in a larger instructor/assistant-to-student ratio. The inclusion of this type of section was maintained after the three year partnership with FCI ended.

Furthermore, one of the remaining sections of the physics is part of the Comprehensive Stud- iesProgram(CSP).CSPsectionsconsistofabout 20 students per term, while FCI sections have the same sizes as standard sections. We focus our discussiononFCIinsteadofCSP,becausethelat- ter is a small scale intervention representing no morethan5%ofthetotalstudentsenrolledint he physics course, while FCI is a larger-scale inter- vention with two sections of regular sizeLclasses. Moreover, FCI is a more recent innCovation with primarily in-class transformations I , while CSP is an existing program started inTthe 1980s which features additional outside resources for students part of the program such as advising and tutor- ing. However, we control for the existence of CSP sections in our regressions.

Empirical strategy. We are interested in con- trasting the difference in education outcomes between in-person and distance learning modal- ity in low-income and first-generation students relative to that in non-low-income and non-first- generationstudents.Thus,ourmethodfocuseson theinteractionbetweendistancelearningandlow- income or first-generation status. Ordinary least squares is a baseline approach for fitting linear models and focuses on inferring the conditional mean of the response given the predictors. How- ever, more aspects than the mean of the condi- tionaldistributionoftheresponsegivenpredictors are of interest. In particular, we are interested in lowerperformingstudentswithinthesecategories. Thus,weutilizeexpectedshortfall(ES)regression which allows for this specification.

The quantity of interest for ES regression is the expected shortfall, compared to the mean for ordinary least squares regression. If we consider a single variable, such as grades, the 0.20-ES refers totheaveragegradeforthesub-populationofstu- dents whose grades fall below the 0.20-quantile of the entire population, where the 0.20-quantile of the grade outcome refers to the highest grade of those who have the 20% lowest grades among the entire population. In other words, ES gives us the average outcome for the lowest-performing students. To extend this concept to a regression setting, we focus on the average outcomes for the students who fall below a certain threshold (at a specificquantilelevel)conditionalonagivensetof covariates. SpSecifically, ES regression allows us to estimate tEhe relationship between an explanatory variable (e.g. distance learning) and the average outcomeforthelowest-performingstudents,while controlling for other relevant factors/covariates.

We estimate the association between distance learning and educational outcomes for disadvan- tagedstudentswithafocusontheaverageperfor- mance of the bottom quintile using the following specification:

E[Y |Y ≤Q (Y |X ,W ),X ,W ]= ict ict τ ict ict ict ict ict α 0 +β 1 D i +β 2 Distance ct +β 3 Hybrid ct + β Post-WN20 +β D ·Distance + 4 ct 5 i ct β D ·Hybrid +β D ·Post-WN20 +γW 6 i ct 7 i t ict (1)

where Y is an outcome for student i in course ict c in term t, Q (Y |X ,W ) denotes the τ ict ict ict conditional quantile at quantile level τ, and E[Y |Y ≤ Q (Y |X ,W ),X ,W ] is ict ict τ ict ict ict ict ict the conditional ES at level τ, which can be inter- preted as the average outcome, given covariates X and W , for the subpopulation whose out- ict ict comes fall below the τ-quantile conditional on covariates X and W . Model (1) utilizes the ict ict conditionalES,targetingthelowerquintilecondi- tional on specified covariates rather than truncat- ingthestudentsatthebottomoftheperformance distribution. The advantage of ES regression over alternatives that also consider the entire condi- tional distribution such as quantile regression is that it can provide information about the aver- age in the tail of a distribution, in our case the bottom quintile, rather than only providing infor- mation about specific points along a distribution. More details about the use of quantiles in regres- sion analysis can be found in [104] and in depth discussion of ES and its mathematical properties can be found in [105] and [106].

The vector X includes main and interaction ict variables{D ,Distance ,Hybrid ,Post-WN20 , i ct ct ct D ·Distance , D ·Hybrid , D ·Post-WN20 }, i ct i ct i t where D is an indicator variable coded to 1 if i student i is in a disadvantaged group such as low-income or first-generation, Distance is an ct indicator variable coded to 1 if course c in term t is distance learning, Hybrid is an indicator vari- ct ablecodedto1ifcoursecintermtishybrid,and Post-WN20 is an indicator variable coded to 1 if t term t is after the Winter 2020 term. The covari- ates matrix W includes the following student, ict course, and term attributes: indicator variables for male, underrepresented minority status, part- timestudentstatus,overtimestatus(morethan18 credit hours in a term), the previous term cumu- lative GPA, indicators for first term attended, class standing (i.e., Sophomore), incomplete stIatus, unofficial withdrawal status, repeating c lass, winterterm,thelogscaledtotalenrollmentforthe course, and average class GPA excludiLng student i. C

We use this specification to measure the dis- parate association between diTstance learning and changes due to COVID-1 R 9 and the grade values of low-income and first-generation students, par- ticularly those in the A lower quintile of academic performance. The estimates are from ES regres- sions. The 95% confidence intervals are obtained following [106]. We consider τ = {0.20,1}, where τ = 1 is equivalent to an ordinary least squares regression. The ES regression estimator is asymp- totically normal [106] thus the p-value is deter- mined from a two-sided one-sample Z-test where the estimate is compared to zero and the test statisticiscalculatedastheESregressionestimate divided by the standard error (obtained following [106]).Theestimateofthedifferenceinassociation betweendistancelearningandgradevaluesforthe bottom (τ·100)% of D =1 students compared to the bottom (τ ·100)% of D = 0 students would be β , and the estimate of the difference in grade 5 valuesbetweenthebottom(τ·100)%ofD =1stu- dents and bottom (τ·100)% of D =0 students in distancelearningwouldbeβ +β .Weconsidered 1 5 additional analyses using τ ={0.1,0.15,0.25,0.5} and generally found significant results for τ ≤ 0.25.

We estimate how an intervention in the form of innovative course initiatives (i.e. FCI) relates to the association between distance learning and educational outcomes by utilizing three-way- interactionterms,inwhichwecontrastdifferences between disadvantaged and non-disadvantaged students within course sections that adopted an innovative instruction style versus those within standard course sections, in distance and in per- son learning. As the course we considered did not have any sections adopting hybrid learning modality and did not have any students with an unofficiaSl withdrawal, we excluded these two variableEs. We estimate the association between grade outcomes and distance learning and the pandemicondisadvantagedstudentsanditsinter- actionswithnon-standardcoursesectionsthrough a three-way-interaction strategy using the follow- ing specification:

E[Y |Y ≤Q (Y |X ,W ),X ,W ]= ist ist τ ist ist ist ist ist α +β D +β Distance +β Post-WN20 + 0 1 i 2 st 3 st β FCI +β CSP +β D ·Distance + i. C 4 st 5 st 6 i st β D ·Post-WN20 +β Distance ·FCI + 7 i t 8 st st β Distance ·CSP +β Post-WN20 ·FCI + 9 st st 10 st st β Post-WN20 ·CSP +β D ·FCI + 11 st st 12 i st β D ·CSP +β D ·Distance ·FCI + 13 i st 14 i st st β D ·Distance ·CSP + 15 i st st β D ·Post-WN20 ·FCI + 16 i st st β D ·Post-WN20 ·CSP +γW 17 i st st ist (2)

whereY isanoutcomeforstudentiinsections ist in term t, FCI is an indicator variable coded to st 1 if section s in term t is an FCI section, CSP st is an indicator variable coded to 1 if section s in term t is a CSP section, and W follows that in ist equation (1) excluding unofficial withdrawals and by section s rather than course c.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "I would like my report to remain anonymous\n-------------------------------\nReferee Report for the Manuscript: “Distance Learning Widens the Educational Gap for Underserved Students in STEM and\na Way to Close the Gap”\nThis manuscript investigates the impact of distance learning on academic outcomes in various STEM courses, focusing on\nunderserved and lower-performing students. Using data from the University of Michigan (Fall 2016 to Fall 2022) and using\nan expected shortfall regression technique, they show grade (small) disparities for low-income and first-generation students\nduring distance learning. Through an instructional intervention (FCI) in an introductory physics course, they showed a small\nreduction in these disparities.\nStrengths:\nThe manuscript employs a data analysis using a large dataset and rigorous statistical methods. It focuses on the challenges\nfaced by low-income and first-generation students. The identification of successful instructional interventions, such as the\nFoundational Course Initiative (FCI), provides practical recommendations for improving educational outcomes during\ndistance learning.\nPotential Weaknesses and Areas for Improvement:\nThe main concern is the small effect size (0.114 grade points for low-income students and 0.064 grade points for first-\ngeneration students), which raises questions about the practical significance of these results. These small differences (even\nif statistically significant) may not translate into meaningful impacts on students’ academic journeys or long-term outcomes.\nIt’s also possible that the small effect size is influenced by other factors, such as differing at-home conditions for low-income\nstudents during the pandemic, which may have exacerbated the disparities. This should be addresses in the manuscript.\nThe use of expected shortfall regression may be complex for some readers. In fact, the presentation of the statistical analysis\nwas overwhelming and distracted from the main point of the paper. Providing a clearer justification for choosing this method\nand a simpler explanation of the methodology is important.\nThe study attributes some effects to the pandemic but may not clearly separate these from the impacts of distance learning\nitself. A discussion distinguishing between the pandemic’s unique impacts and those specifically related to distance\nlearning is needed. The focus on the FCI in one physics course may not capture the full range of effective interventions.\nDiscussing the scalability and adaptability of the FCI or similar interventions across different contexts (and institutions)\nwould widen the applicability of these findings and recommendations.\nIt is also unclear if the impact of the FCI intervention continued after the pandemic (though Fall 2022 was presumably in-\nperson instruction.\n---------------------------\nThe studies’ focus on an important and timely issue, and the use of a large dataset with sophisticated statistical analysis,\naligns well with Nature Communications standards. However, the small effect sizes and potential confounding factors are\nsignificant concerns.\nI recommend publication in Nature Communications, contingent on addressing the following points:\nPractical Significance: Provide a deeper discussion on the practical significance of the observed effect sizes, potentially\nlinking these findings to long-term academic and career outcomes for students.\nConfounding Factors: Discuss the potential impact of differing at-home conditions during the pandemic on the observed\neffect sizes and consider any additional analyses to control for these factors.\nImpact Post-Pandemic: Investigate and report whether the impact of the FCI intervention continued after the pandemic.\nGeneralizability: Include a more detailed discussion on the generalizability of the findings, supplemented with comparisons\nto other institutions (at least a discussion of it).\nMethodological Clarity: Simplify the explanation of the expected shortfall regression method and provide a clear justification\nfor its use over more common alternatives. In its current form, this portion of the paper is inaccessible to most readers.", "reviewer_2": "Thank you for the opportunity to review the manuscript titled \" Distance learning widens the educational gap for underserved\nstudents in STEM and a way to close the gap.\" In this study, the authors explore the impact of distance learning on academic\noutcomes in STEM courses, focusing particularly on underserved and lower-performing students. They analyze a dataset\ncomprising seven hundred thousand student-course records from Fall 2016 to Fall 2022 at a major public university in the\nUnited States.\nThe topic addressed by the authors is both significant and timely, and I found the manuscript to be well-written. However,\nseveral aspects temper my enthusiasm:\n1. The main hypothesis appears inadequately justified. The literature review primarily discusses social class differences\nrather than variations in academic level, and the interaction between these variables lacks sufficient justification: \"When\nexamining the academic gap, our focus is on disparities among students with below-average academic outcomes. We\nhypothesize that lower-achieving students may be disproportionately affected by distance learning, exacerbating the\nachievement gap between low-income or first-generation students and their peers.\"\n2. The rationale for emphasizing STEM could benefit from further elaboration.\n3. The presentation of results at the end of the introduction is unclear: \"The results show that distance learning is associated\nwith wider academic disparities, i.e., 0.114 and 0.064 grade points (out of 4), in STEM courses between low-income and\nnon-low income students, and between first-generation and non-first-generation students, as measured by the average\ngrade points in the lowest quintile (lowerquintile mean = 2.46). In contrast, we do not detect statistically significant disparity\nat the mean, consistent with the results of [55] and [56] for low-income compared to non-low-income students. Additionally,\nwhen considering both STEM and non-STEM courses, we find a small gap of around 0.05 grade points between low-income\nand non-low-income, and between first-generation and non-first generation students, measured by the average grade points\nin the lowest quintile.\"\n4. The sequencing of presenting results before the methods may have contributed to my comprehension challenges, which I\nfound surprising.\nThese points highlight areas where clarity and justification could enhance the manuscript's impact and scholarly\ncontribution.\nI do not master the statistical analyses of this paper", "reviewer_3": "This paper investigates the impact of distance learning on academic outcomes in STEM courses, particularly among\nunderserved and lower-performing students. Using administrative records from a large US public university from Fall 2016 to\nFall 2022 and an expected shortfall regression strategy, the study reveals that distance learning exacerbates academic\ndisparities, with low-income and first-generation students in the bottom quintile experiencing a 0.11 and 0.06 grade point\ndrop, respectively, compared to their peers. The research also highlights the challenges of online education, especially in\nSTEM fields that require hands-on experience and specialized equipment. In addition, the study examines a promising\nintervention at the university to mitigate these disparities by way of increasing instructor-student and student-student. It finds\nthat enhanced support and interaction helped reduce achievement gaps during distance learning in the context of\nintroductory physics course. The study is claimed to provide large-scale evidence of the negative impact of distance learning\non disadvantaged students and demonstrate the effectiveness of specific interventions in reducing these gaps. The findings\nemphasize the need for continued efforts to support underserved students in online education environments to ensure\nequitable academic outcomes.\nOverall, this is a rigorously performed empirical study that adds to the literature on the long-term impacts of the COVID-19\npandemic on education and on online education in general. The statistical methods are appropriate and parsimonious for\nthe research questions and the results are scientifically properly interpreted. The writing is clear and flows well.\nVisualizations are also effectively used to convey key messages. In addition, the large-scale analyses use standard\nadministrative records that most higher education institutions maintain, so the study has a good potential to scale. All that\nsaid, I think the manuscript can benefit from a few important areas of improvement.\n1. The biggest issue is that distance learning during the COVID-19 pandemic is very different from distance learning under\nnormal conditions. Therefore, it is questionable to say findings in the context of COVID-19 would add much to the general\nscience of distance/online education. Other studies examining learning experience and outcomes during the pandemic\n(some cited by this study) only aim to estimate the impact of “COVID-19 pandemic” or “COVID-induced remote learning”, not\nthe impact of distance learning in general. While the authors attempt to differentiate the impacts of distance learning and\nCOVID-19 by incorporating two indicators in their regression models, it is clear from Figure 1 how intertwined these two\nimpacts are, at least at the studied university. In other words, most of the “distance learning” examined in this paper is\naccompanied by unusual factors like the disruptions in students’ everyday life, instructors’ lack of experience in teaching\nonline courses, and extremely lenient grading policies. Therefore, the findings would speak little about how typical distance\nlearning moderates learning outcomes and gaps across different student groups. I suggest that the authors reframe the goal\nof this study to something like “understand the long-term impacts of the COVID pandemic”, instead of focusing on distance\nlearning more broadly.\n2. The evaluated FCI sections encompassed pre-pandemic, emergency transition, and longer-term recovery stages, because “starting from Fall 2019 onward, two sections of the physics course under study were transformed into a studio\nmodel.” These stages are drastically different, as clarified in the first point, but the current analysis only estimates the\naverage effects of FCI across all these different contexts. It would therefore be beneficial to analyze the effects by time\nperiod, thereby generating fine-grained insights into the conditions under which the FCI intervention works.\n3. Because there is not much causal inference used in the analyses, the authors should be cautious about using causal\nlanguage such as “effects” throughout the manuscript. Alternatively, they may want to use more causal inference methods to\nstrengthen their causal claims about the impacts of distance learning (or COVID-19) and the effects of FCI interventions.\n4. As acknowledged in the limitation part, the university studied here is selective with a low percentage of students from\ndisadvantaged backgrounds. This sample limits the ability of this study to speak about “educational gap for underserved\nstudents in STEM”. Also, given that a decent body of existing research has examined the pandemic’s impacts on college-\nlevel performance gaps in under-resourced contexts like community colleges, I am not sure how much value added the\ncurrent analysis provides to research or practice. Maybe the most prominent contribution would be evaluating long-term\nimpacts of the pandemic as mentioned in the first point.\n5. It is a valuable practice to focus on students at the bottom of the performance distribution when examining educational\ngaps. However, it is necessary to provide more technical clarifications about the expected shortfall regression, especially\nabout the sample restriction. The current manuscript says that “0.20-ES refers to the average grade for the sub-population of\nstudents whose grades fall below the 0.20 quantile of the entire population.” This seems to suggest that the thresholding\nhappens at the student level. On the other hand, Equation (1) suggests that the thresholding may happen at the student-\ncourse-term level. It is hence important to know how the thresholding is done. In addition, because the grade distribution\nduring the emergency stages of COVID-19 should differ substantially from other periods, the thresholding might lead to very\nuneven representation of observations over different times periods and/or cohorts of students. This may essentially induce\nbias and harm the validity of the results. I suggest that the authors explicitly discuss, and if necessary, address this issue,\nwith more descriptives of the restricted sample and applications of alternative empirical strategies.\n6. I’m not super familiar with the typical format of papers published at Nature Communications, but it looks duplicative to\nhave “Approach” and “Empirical Strategy” subsections under Section 2 and an entire Section 3 of “Methods.” In most general\nscience papers I have read, Section 2 is only about (and typically titled) results, where methods are minimally described\nonly to facilitate readers’ understanding.\nOverall, I appreciate the authors’ effort to perform such a rigorous analysis, but the many limitations above make it necessary\nto substantially revise the current manuscript.", "reviewer_4": "The paper provides a thorough study to assess the impact of distance learning on STEM academic outcomes, particularly for\nunderserved and lower-performing students. Instead of using standard linear regression that focuses on the mean grade, the\npaper uses another technique, expected shortfall regression, which studies how the tau-th expected shortfall of grade\ndepends on different factors, such as distance learning, low-income and first-generation students. The 0.2th expected\nshortfall is defined to be the average grade for lower-performing students, especially those whose grades fall in the bottom\n20%. Therefore, the method could particularly assess the impact of distance learning on lower-performing students, a\nsubpopulation that might be more strongly affected, as well as the interactions with low-income and first-generation students.\nI found the paper well-written. The analysis procedure is clearly presented and reasonable. Findings from this paper also\nhighlight achievement disparities and the effectiveness of targeted interventions in reducing gaps during distance learning.\nThese could guide policymakers to develop informed strategies to support equitable educational outcomes. In addition, the\npaper introduces a new regression technique that is particularly appealing for addressing disparities to disadvantaged\ngroups, so the case study from the expected shortfall aspect is meaningful and impactful.\nI only have a few rather minor suggestions for the authors to consider:\n1. Page 10, left, it mentioned: “The covariate matrix W_{ict} includes the following…” However, on page 9, formula (1), the\ncoefficient of W_{ict} appears to be a scalar. This is confusing.\n2. Page 9 and 10, the paper explained that the tau-ES refers to the average grade for the sub-population of students whose\ngrades fall below the tau-quantile of the entire population. However, the definition in formula (1) indicates the subpopulation\nis defined to be Y<Q_tau(Y|X, W). Since the right-hand side is the conditional quantile given X and W, it is not the tau-th\nquantile of the “entire population,” which should be Q_tau(Y). The claim also appears on page 9, bottom right. Some\nclarification and further discussion are needed.\n3. Page 10, left, “The estimates are ES regressions” should be “The estimates are from ES regression.”"}, "author_response": "ResponsetoReviewers\nWethankthereviewersforreadingourmanuscriptcarefullyandprovidingmanydetailedandcon-structivecomments. Wehavemademajorrevisionstoourmanuscriptinresponse. Inthefollowing,we will provide responses addressing each reviewer’s comments and questions. We highlightedchangesmadetothemanuscriptinred.\nReviewer 1 Comments:\nThemainconcernisthesmalleffectsize(0.114gradepointsforlow-incomestudentsand0.064grade points for first-generation students), which raises significant questions about the prac-tical significance of these results. These small diffferences (even if statistically significant) maynot translate into meaningful impacts on students’ academic journeys or long-term outcomes. It’salsopossiblethatthesmalleffectsizeisinfluencedbyotherfactors,suchasdifferingat-homecon-ditions for low-income students during the pandemic, which may have exacerbated the disparities.Thisshouldbeaddressedinthemanuscript.\nThe use of expected shortfall regression may be complex for some readers. In fact, thepresentationofthestatisticalanalysiswasoverwhelminganddistractedfromthemainpointofthepaper. Providing a clearer justification for choosing this method and a simpler explanation of themethodologyisimportant.\nThestudyattributessomeeffectstothepandemicbutmaynotclearlyseparatethesefromtheimpactsofdistancelearningitself. Adiscussiondistinguishingbetweenthepandemic’suniqueimpacts and those specifically related to distance learning is needed. The focus on the FCI in onephysics course may not capture the full range of effective interventions. Discussing the scalabilityandadaptabilityoftheFCIorsimilarinterventionsacrossdifferentcontexts(andinstitutions)wouldwidentheapplicabilityofthesefindingsandrecommendations.\nItisalsouncleariftheimpactoftheFCIinterventioncontinuedafterthepandemic(thoughFall2022waspresumablyin-personinstruction).\n1. Practical Significance: Provide a deeper discussion on the practical significance of the ob-served effect sizes, potentially linking these findings to long-term academic and career out-comesforstudents.\nWe thank the reviewer for the helpful comment. In response to the reviewer’s comment, wehave provided more discussion of the potential consequences of the observed effect sizes inthe revised manuscript. Although the lower-quintile mean of grade points is 2.48, this is forSTEM courses overall. For low-income and first-generation students, who we are interestedin, the lower-quintile mean is around or below 2.0. This grade point is highly important,as for at least the two largest colleges at the university (Literature, Science, and Arts andEngineering), falling below a 2.0 cumulative GPA results in the student falling out of goodacademicstandingandputsthematriskofbeingplacedonleaveforayear. Generallymajorsalsorequirea2.0GPAformajorcourses,andsomemajorssuchasbiology,alsorequirea2.0GPAforsequenceswithinthemajor. Additionally,forindividualmajorcoursesinComputerScience and Mechanical Engineering in the college of engineering, a letter grade of below aC (below 2.0 grade points) means the course does not meet the major requirements. Thus,areductionof0.114or0.064ingradepointsforstudentswithinthelowestquintilecanresult in serious academic consequences and delays to graduation. We have added thefollowingdiscussionintheIntroductionoftherevisedmanuscript.\n“Low-income students and first-generation students in the lowest quintile havegrade point differences of 0.114 and 0.064 under distance learning, respectively,compared to their peers. This impact is significant given the lower-quintile meangradepointsof2.48overall,and2.00,2.02,and1.69forlow-income,first-generation,andbothlow-incomeandfirst-generationstudents, respectively. Moreover, atbe-lowa2.0cumulativeGPA,andforsomecollegesbelowa2.0semesterGPA,stu-dents will be placed on academic probation with the risk of having enrollmentwithheld or being placed on academic recess for one year. In some majors, stu-dentswhoachievebelow2.0gradepointswillfailtofulfilltheminimumrequire-mentforcommonrequiredandcorecoursesintheirmajors. Thus, areductionof0.114or0.064ingradepointsforstudentswithinthelowestquintilecanresultinserious academic consequences and delays to graduation. In non-STEM courses,weseeasmallergapofaround0.05gradepoints.”(page3)\nAdditionally, in other education papers we have seen similar effect sizes. Rodriguez-Planas[2022] reports a similar 5% decrease in GPA relative to the pre-pandemic mean and Wanget al. [2023] reports a 0.2 standard deviation (3.5 per 100 points) grade difference. If westandardize grade values, our analysis sees a 0.154 standard deviation (SD) difference forthe bottom quintile low-income and non-low-income students under distance learning and a0.143SDdifferenceforthebottomquintilefirst-generationandnon-first-generationstudentsunderdistancelearning.\n2. Confounding Factors: Discuss the potential impact of differing at-home conditions duringthepandemicontheobservedeffectsizesandconsideranyadditionalanalysestocontrolfor\nthesefactors.\nWe thank the reviewer for the insightful comment. We acknowledge that remote learningduring the pandemic may be exacerbated by circumstances related to the pandemic. As alimitation, we cannot completely distinguish the effect, especially carry-over effects such aspotential loss of knowledge in prerequisite courses. Thus, in our paper, we present our re-sults and contributions as related to the impact of distance learning during the COVID-19pandemic, and note in our discussion that more research is needed to study remote learningoutsideofthecontextofthepandemicforvulnerablepopulations. Weincludedthefollowinginthefinalparagraphofourdiscussion:\n“Ourstudyconsidersdistancelearninginthecontextofthepandemic, whichcanbe accompanied by and difficult to separate from factors such as disruptions instudents’ everyday life, differing at-home conditions, and carry-over effects suchaspotentiallossofknowledgeinpre-requisitecourses. Asalimitation,wecannotcompletely distinguish the impact of distance learning from these factors.” (page13)\nIn addition to acknowledging the limitations, we also investigate the potential impact of dif-feringat-homeconditionsduringthepandemic,andreporttheresultsinAppendixD.Tothisend,weconsidercoursesthatstillhadatleastoneremotesectionineitherFall2021,Winter2022, or Fall 2022, when students were required to return to campus. This means that thecourses were still remote for at least one semester after the majority of classes returned toin-personandstudentswerepresentoncampus,allowingustoconsiderthedistanceandlow-income/first-generation interaction without differing at-home conditions. We exclude Fall2020 and Winter 2021, which are the semesters where differing at-home conditions wouldoccur, and thus only have the remote courses in Fall 2021-Fall 2022 as comparison for dis-tancelearning.\nOur sample consists of 106,920 observations for 57 unique courses with 54 courses remotein Fall 2021, 15 in Winter 2022, and 10 in Fall 2022. We note that these courses are likelynot representative of the overall type of STEM courses and that the sample size is smalldue to the limited number of classes. The effect size for Low-income:Distance is -0.284*(-0.532, -0.036) and the effect size for First-generation:Distance is -0.130 (-0.0276, 0.016).For reference, when we consider all of the semesters (Fall 2016-Fall 2022, except Winter2020)forthesecourses,theeffectsizesare-0.129(-0.275,0.016)and-0.054(-0.147,0.039)for Low-income:Distance and First-generation:Distance, respectively. In other words, ex-cluding the semesters where at-home conditions differ, we observe greater effect sizes thanwhenweincludethesesemesters. Additionally,thedirectionandmagnitudeoftheestimatedeffects are largely consistent with that of all STEM courses (-0.114* and -0.064* for Low-income:DistanceandFirst-generation:Distance,respectively),albeitslightlylowerstatisticalsignificancepotentiallyduetosmallersamplesizes. ItshouldbenotedthataftertheFall2020and Winter 2021 semesters, students may be able to choose whether or not they will take adistancelearningcourseorsectionifbothin-personanddistancelearningmodalitiesareof-feredforacourse,andthemajorityofcoursesarein-person,sothemorenegativeeffectsizesmay be due to self-selection bias (e.g., less-motivated students choosing remote sections).TheresultsaresummarizedinTable1-2.\n3. Impact Post-Pandemic: Investigate and report whether the impact of the FCI interventioncontinuedafterthepandemic.\nWe thank the reviewer for the insightful comment. We want to point out that our goal is toassess the influence of FCI during distance learning, and thus, we focus on those semesterswith distance learning while using other semesters as comparison. FCI was only distancelearning during the semester Fall 2020 and Winter 2021. In Fall 2021, FCI sections re-turned to in-person learning while regular sections remained distance. Figure R-1 presentsthe difference between the lower quartile of low-income/first-generation and lower quartileof non-low-income/non-first-generation during different time periods. FCI helps during thepandemic (Fall 2020-Winter 2021), which may be due to lack of resources at home for low-incomeandfirst-generationstudents. Whenstudentsreturntocampus(Fall2021-Fall2022),the effectiveness of FCI reduces as there may be similar resources available on campus, orpossiblyduetoprolongedexposuretodistancelearning.\n4. Generalizability: Include a more detailed discussion of the findings, supplemented withcomparisonstootherinstitutions(atleastadiscussionofit).\nWethankthereviewerforthesuggestion. Inthediscussionsection,weprovidedmorecom-parisonstootherinstitutions. Weidentifiedotherinstitutions’courseinitiativesthataresim-ilar to the one we studied, in the sense that they place an emphasis on active learning andinteractionwithclassmates.\n“Initiatives to redesign courses, like FCI, have been explored at a number of uni-versities. TheseincludetheUniversityofPennsylvaniaStructured,Active,In-classlearning (SAIL), UNC Chapel Hill Associate of American Universities (AAU)STEM Project, Purdue University IMPACT Project, Cornell University ActiveLearningInitiative(ALI),andTRESTLE,amulti-institution(UniverityofKansas,IndianaUniversity,Queen’sUniversityatKingston,UniversityofCalifornia-Davis,University of Colorado-Boulder, University of Texas at San Antonio, UniversityofBritishColumbia)NSF-fundedprojecttopromoteimprovementofSTEMedu-cation at research universities [TRESTLE, 2017]. Many efforts focus on reform-ing science education, particularly introductory and required courses. One traitfrequently included in evidence-based course redesigns is the emphasis on activelearning and interaction with classmates, especially through the creation of smallgroups or teams, similar to the intervention in the physics course we examined.The popularity of this pedagogical method in courses is due to its strong posi-tive outcomes for students in terms of their performance in and perception of thecourse [Ballen et al., 2017, Donovon et al., 2018, Kim et al., 2005, MichaelsenandSweet,2008]. BrownUniversity’sAssociateofAmericanUniversities(AAU)undergraduateSTEMeducationinitiativeprojectaimedtoimproveretentioninin-troductoryandrequiredsciencecourses,andredesigned11coursesbyintegratingproblem solving sessions into recitations, encouraging engagement and collabo-ration with peers [Chasteen and Code, 2018]. These initiatives have also beenimplemented and studied during COVID-induced distance learning. As a part of\ntheActiveLearningInitiative(ALI)atCornellUniversity,instructorsimplementedteam-basedlearning(TBL)firstinanin-personlifesciencescourseandlaterbothadaptedthecoursewithTBLtoasmalleronlineplatformandutilizedTBLwhen\ntransitioningthein-personcoursetoonlineduringtheonsetofthepandemic[Ar-cila Herna´ndez et al., 2021]. They found that TBL enhanced the student experi-ence in the online course and students had high retention and achievement withonline teaching [Arcila Herna´ndez et al., 2021]. Another ALI at Cornell investi-gated student performance during the pandemic and found that teaching methodsthat promote active engagement, such as small group activities, can mitigate thenegativeeffectincurredinonlineclassesduringthepandemic[Orlovetal.,2021].”(page12-13)\n5. MethodologicalClarity: Simplifytheexplanationoftheexpectedshortfallregressionmethodandprovideaclearjustificationforitsuseovermorecommonalternatives. Initscurrentform,thisportionofthepaperisinaccessibletomostreaders.\nWe added a simpler and more thorough explanation of expected shortfall regression and ourjustificationforusingit. Thisislocatedpriortoourintroductionoftheformulaforthemodel.\n“Weareinterestedincontrastingthedifferenceineducationoutcomesbetweenin-personanddistancelearningmodalityinlow-incomeandfirst-generationstudentsrelative to that on non-low-income and non-first-generation students. Thus, ourmethod focuses on the interaction between distance learning and low-income orfirst-generation status. Ordinary least squares is a baseline approach for fittinglinearmodelsandfocusesoninferringtheconditionalmeanoftheresponsegiven\nthepredictors. However,moreaspectsthanthemeanoftheconditionaldistributionof the response given predictors are of interest. In particular, we are interestedin lower performing students within these categories. Thus, we utilize expectedshortfall(ES)regressionwhichallowsforthisspecification.\nThe quantity of interest for ES shortfall regression is the ES, compared to themean for ordinary least squares regression. If we consider a single variable, suchas grades, the 0.20-ES refers to the average grade for the sub-population of stu-dentswhosegradesfallbelowthe0.20-quantileoftheentirepopulation,wherethe0.20-quantile of the grade outcome refers to the highest grade of those who havethe 20% lowest grades among the entire population. In other words, ES gives usthe average outcome for the lowest-performing students. To extend this conceptto a regression setting, we focus on the average outcomes for the studentswho fall below a certain threshold based on certain factors. Specifically,ESregressionallows us to estimate the relationship between an explanatory variable (e.g. dis-tancelearning)andtheaverageoutcomeforthelowest-performingstudents,whilecontrollingforotherrelevantfactors/covariates.” (page10)\nReviewer 2 Comments:\n1. Themainhypothesisappearsinadequatelyjustified. Theliteraturereviewprimarilydiscussessocial class differences rather than variations in academic level, and the interaction between\nthesevariableslackssufficientjustification: “Whenexaminingtheacademicgap,ourfocusinon disparities among students with below-average academic outcomes. We hypothesize that\nlower-achievingstudentsmaybedisproportionatelyaffectedbydistancelearning,exacerbatingtheachievementgapbetweenlow-incomeorfirst-generationstudentsandtheirpeers.”\nThankyouforallowingtheopportunitytoelaborateonourmainhypothesis. WehaveaddedadditionalliteraturereviewintheIntroductiondiscussingthepotentialconsequencesoflower\nacademicperformance,andhowtheymayaffectlow-incomeandfirst-generationstudentsin\nparticular.\n“Many studies have investigated strategies to improve the performance of low-performingpostsecondarystudents[Deslauriersetal.,2012,Freemanetal.,2007,\nJensenandMoore,2008,2009,Moore,2005]. Studentswhobeginwithpoorper-formance tend to struggle throughout their academic journey [Jensen and Moore,\n2008], a problem which may be exacerbated by limited in-person instruction andinteraction. Students at the lower end of academic outcomes are at risk of being\nplaced on academic probation, which can delay graduation [Bowman and Jang,2022]. Low-income and first-generation students already face a graduation gap:\nundergraduateswhoarebothlow-incomeandfirst-generationhaveasix-yeargrad-uation rate at four-year institutions of 41% compared to 73% for undergraduates\nwhoareneitherlow-incomenorfirst-generation[Cahalanetal.,2018].” (page2)\n2. TherationaleforemphasizingSTEMcouldbenefitfromfurtherelaboration.\nWe thank the reviewer for the helpful comment. In response to the comment, we have in-cludedadditionalelaborationonourrationalforemphasizingSTEMintheIntroduction. The\nmain idea is that STEM has often been the focus of studies as it is seen as a way to enter\nlucrative careers, and there is constant and growing demand for those skilled in STEM dis-\nciplines. Due to gaps in enrollment and completion of STEM degrees for low-income and\nfirst-generation students, as well as additional challenges faced in STEM courses compared\ntonon-STEMcourses,weareinterestedinstudyingtheintersectionofSTEMacademicper-\nformance, low-income and first-generation status, and distance learning. We have included\nthefollowingsentencesintheIntroductionoftherevisedmanuscript.\n“Enhancing social mobility for students is important for universities, and STEM\nisoftenseenasavehicleforeconomicopportunityandsocialmobilityBroderick\nand Neufeldt [2019]. STEM degrees have long been promoted to students as a\npotentialpathtohigh-payingcareers...” (page2)\n“The effect distance learning may have on the academic performance of first-generation and low-income students in STEM courses is of interest, as college\nGPAandfirst-semesterGPAareimportantpredictors...ofobtainingaSTEM-related\njobXu[2013].” (page3)\n“TheintersectionofSTEM,barrierstosuccessinSTEMfacedbylow-incomeand\nfirst-generation students, and challenges faced in a distance learning setting are\nthusofgreatinterest.” (page3)\n3. Thepresentationofresultsattheendoftheintroductionisunclear: “Theresultsshowthatdis-tancelearningisassociatedwithwideracademicdisparities,i.e.,0.114and0.064gradepoints\n(outof4),inSTEMcoursesbetweenlow-incomeandnon-lowincomestudents,andbetween\nfirst-generation and non-first-generation students, as measured by the average grade points\nin the lowest quintile (lowerquintile mean = 2.46). In contrast, we do not detect statistically\nsignificant disparity at the mean, consistent with the results of [55] and [56] for low-income\ncomparedtonon-low-incomestudents. Additionally,whenconsideringbothSTEMandnon-\nSTEM courses, we find a small gap of around 0.05 grade points between low-income and\nnon-low-income, and between first-generation and non-first generation students, measured\nbytheaveragegradepointsinthelowestquintile.”\nIn response to the reviewer’s comment, we have reworded the presentation of results and\nomittedthemeanresultsintheintroductionsothemainresultsareeasiertounderstand. Note\nthat we also incorporated comments related to the practical significance of the effect size.\nThisisalsorelatedtoComment1asitjustifieswhyweareinterestedinstudentswithlower\nacademicperformance.\n“Low-income students and first-generation students in the lowest quintile havegrade point differences of 0.114 and 0.064 under distance learning, respectively,\ncompared to their peers. This impact is significant given the lower-quintile meangradepointsof2.48overall,and2.00,2.02,and1.69forlow-income,first-generation,\nandbothlow-incomeandfirst-generationstudents, respectively. Moreover, atbe-lowa2.0cumulativeGPA,andforsomecollegesbelowa2.0semesterGPA,stu-dents will be placed on academic probation with the risk of having enrollment\nwithheld or being placed on academic recess for one year. In some majors, stu-dentswhoachievebelow2.0gradepointswillfailtofulfilltheminimumrequire-mentforcommonrequiredandcorecoursesintheirmajors. Thus, areductionof\n0.114or0.064ingradepointsforstudentswithinthelowestquintilecanresultin\nserious academic consequences and delays to graduation. In non-STEM courses,\nweseeasmallergapofaround0.05gradepoints.”(page3)\n4. The sequencing of presenting results before the methods may have contributed to my com-prehension challenges, which I found surprising. These points highlight areas where clarity\nandjustificationcouldenhancethemanuscript’simpactandscholarlycontribution.\nWe thank the reviewer for the comment and we totally agree with the reviewer. How-ever, we could not change the order of the methods and results section due to Nature’s\nprovided template for submission at https://www.nature.com/nature/for-authors/\nformatting-guide.\n5. (Remarksoncodeavailability)Idonotmasterthestatisticalanalysesofthispaper.\nWethankthereviewerforpointingouttheconfusion. Thestatisticalanalysesofthismanuscript\narebasedonanewlydevelopedmethodinthestatisticalliterature–theexpectedshortfallre-gression–that target directly on the lower-tail of the response distribution. It can be thought\nof as a generalization of the quantile regression which targets a particular quantile of the re-sponsedistribution,andageneralizationofthelinearregressionthatfocusesonthemean.\nIn response to the reviewer’s comment, we have added more description to the functions in\nourcodesampleaswellasprovidedatoyexamplefollowingtheanalysisinthepaper. Addi-tionally,intheEmpiricalStrategysubsectionoftheMethodssection,weprovidedadditional\nbackground,motivation,anddescriptionofourstatisticalmethod. Wehopethatthestatistical\nanalysescannowbeunderstoodintherevisedmanuscript.\nReviewer 3 Comments:\n1. The biggest issue is that distance learning during the COVID-19 pandemic is very different\nfromdistancelearningundernormalconditions. Therefore,itisquestionabletosayfindings\nin the context of COVID-19 would add much to the general science of distance/online\neducation. Otherstudiesexamininglearningexperienceandoutcomesduringthepandemic\n(some cited by this study) only aim to estimate the impact of “COVID-19 pandemic”\nor “COVID-induced remote learning,” not the impact of distance learning in general.\nWhiletheauthorsattempttodifferentiatetheimpactsofdistancelearningandCOVID-19by\nincorporating two indicators in their regression models, it is clear from Figure 1 how inter-\ntwined these two impacts are, at least at the studied university. In other words, most of the\n“distancelearning”examinedinthispaperisaccompaniedbyunusualfactorslikethedisrup-\ntionsinstudents’everydaylife,instructors’lackofexperienceinteachingonlinecourses,and\nextremelylenientgradingpolicies. Therefore,thefindingswouldspeaklittleabouthowtyp-\nicaldistancelearningmoderateslearningoutcomesandgapsacrossdifferentstudentgroups.\nI suggest that the authors reframe the goal of this study to something like “understand\nthe long-term impacts of the COVID pandemic”, instead of focusing on distance learn-\ningmorebroadly.\nWethankthereviewerforpointingthisoutandweagreewiththereviewerthatitmaybehard\ntoseparatetheCOVID-19effectanddistancelearningeffect.\nWe acknowledge that remote learning during the pandemic is exacerbated by circumstances\nrelatedtothepandemic. Weattempttodistinguishtheeffectoffactorsthatshouldaffectstu-\ndentsuniformlysuchasinstructors’lackofexperienceinteachingonlinecoursesandlenient\ngrading policies by focusing on the interaction term in our model. Under lenient grading\npolicies, both low-income and non-low-income and first-generation and non-first generation\nstudents’ grades should increase or stay the same as before, however low-income and first-\ngenerationstudentsaremorenegativelyor“lesspositively”affectedduetosomefactors. As\na limitation, we cannot completely distinguish the effect, especially carry over effects such\nas potential loss of knowledge in prerequisite courses. Thus, in our paper, we present our\nresults and contributions as related to the impact of distance learning during the COVID-19\npandemic, and note in our discussion that more research is needed to study remote learning\noutsideofthecontextofthepandemicforvulnerablepopulations.\nFollowingthereviewer’ssuggestion,wefurtherreframethelanguageofourpapertoempha-\nsize that we are studying COVID-induced distance learning rather than distance learning in\ngeneralthroughoutthepaper(title,abstract,maintextandappendices),andincludedthelim-\nitation of COVID-induced distance learning in our final paragraph of the discussion, which\ndiscussesthe adoptionand studyof distancelearningoutside ofthe contextof thepandemic\nwithroomforfuturework.\n“Ourstudyconsidersdistancelearninginthecontextofthepandemic, whichcan\nbe accompanied by and difficult to separate from factors such as disruptions in\nstudents’ everyday life, differing at-home conditions, and carry-over effects such\naspotentiallossofknowledgeinpre-requisitecourses. Asalimitation,wecannot\ncompletelydistinguishtheimpactofdistancelearning.” (page13)\n2. The evaluated FCI sections encompassed pre-pandemic, emergency transition, and longer-\nterm recovery stages, because “starting from Fall 2019 onward, two sections of the physics\ncourseunderstudyweretransformedintoastudiomodel.” Thesestagesaredrasticallydiffer-\nent,asclarifiedinthefirstpoint,butthecurrentanalysisonlyestimatestheaverageeffectsof\nFCIacrossallthesedifferentcontexts. Itwouldthereforebebeneficialtoanalyzetheeffects\nby time period, thereby generating fine-grained insights into the conditions under which the\nFCIinterventionworks.\nWethankthereviewerfortheinsightfulcomment. Themaingoalofouranalysesistounder-\nstandwhethercourseinitiativesthatfocusonactivelearningandinteractionwithclassmates\n(seealsoresponsetoComment4ofReviewer1forathoroughdiscussionofsimilarinitiatives\nin other universities) can mitigate the gap in COVID-induced distance learning. Therefore,\nwereportedtheaverageeffectsofFCIinthemainmanuscriptsincethefocuswasnotonthe\nspecificmodalityofFCIinitiatives. Moreover,thelimitedsamplesizealsohindersourability\ntoanalyzethedifferentmodalitiesseparately.\nIn response to the reviewer’s comment, we now perform additional analyses to assess the\neffects of the course initiative in specific time periods. For more fine-grained insights, we\nconsiderthespecifictimeperiodswhereeitherFCIorregularsectionsaredistancelearning.\nWeinvestigatethedifferencebetweenlow-incomeandnon-low-incomestudentsinFall2020-\nWinter 2021, when all sections are distance and campus is closed, and in Fall 2021, where\nregularsectionsaredistancewhileFCIsectionsarein-personandallstudentsareoncampus.\nResults are in Figure R-1. During Fall 2020-Winter 2021, the difference between the bot-\ntom quintiles low-income and non-low income and first-generation and non-first-generation\nis positive for FCI sections, showing that there is an improvement for low-income and first-\ngenerationstudents. InFall2021,thedifferencebetweenthebottomquintileslow-incomeand\nnon-low income and first-generation and non-first-generation for FCI sections is above that\nof regular sections, but still negative for first-generation and non-first generation. FCI helps\nduring the pandemic (Fall 2020-Winter 2021), which may be explained by lack of resources\nathomeforlow-incomeandfirst-generationstudents. Whenstudentsreturntocampus(Fall\n2021-Fall2022),theeffectivenessofFCIreducesastheremaybesimilarresourcesavailable\non campus, or possibly due to prolonged exposure to distance learning. Additionally, there\nmaybeself-selectionbiaswhenstudentscanchoosebetweenadistancelearningorin-person\ncourseduringtheFall2021semester.\n3. Because there is not much causal inference used in the analyses, the authors should be cau-\ntiousaboutusingcausallanguagesuchas“effects”throughoutthemanuscript. Alternatively,\nthey may want to use more causal inference methods to strengthen their causal claims about\ntheimpactsofdistancelearning(orCOVID-19)andtheeffectsofFCIinterventions.\nInresponsetothereviewer’scomment,wehaveminimizedtheuseofcausallanguagesuchas\n“effect” and “affect” in the revised manuscript. We have also emphasized in the section and\nfigurestitlesthatwearefocusingonassociation(seeforinstancetitlesofSections2.4,2.4.1,\nand2.4.2,aswellastitlesforFigures2and3). Wekeeptermssuchasthe“maineffects”and\n“effectsize”sincetheyarealsocommonlyusedinregressionmodelsforassessingthemag-\nnitudeoftheassociation. Wenotethattheseshouldnotbeinterpretedascausalrelationship.\n4. As acknowledged in the limitation part, the university studied here is selective with a low\npercentage of students from disadvantaged backgrounds. This sample limits the ability of\nthis study to speak about “education gap for underserved students in STEM” Also, given\nthatadecentbodyofexistingresearchhasexaminedthepandemic’simpactsoncollege-level\nperformance gaps in under-resourced contexts like community colleges, I am not sure how\nmuch value added the current analysis provides to research or practice. Maybe the most\nprominentcontributionwouldbeevaluatinglong-termimpactsofthepandemicasmentioned\ninthefirstpoint.\nThe university studied is a large public four-year research institution that is comparable in\nterms of academics and student body to other selective universities such as Ivy Leagues,\nprivate universities such as Northwestern, and major public universities University of Cali-\nfornia schools or University of Illinois-Urbana/Champaign. Although the sample includes a\nlowerpercentageofstudentsfromdisadvantagedbackgrounds,thisisdueinparttotheself-\nreporting nature of the data, which may underrepresent the actual diversity reported by the\nuniversityandFAFSA.\nWhileexistingresearchonthepandemic’simpactinunder-resourcedcommunitycollegesis\ncertainlyvaluable,thisstudyfillsagapbyexaminingtheeffectsonadifferentsegmentofthe\nhigher education landscape. These selective institutions often serveas pipelines for students\ninto STEM postgraduate careers and further education. Understanding how remote learning\nduringthepandemicimpactedstudentsattheseuniversitiesiscrucial, asitprovidesinsights\nintothelong-termimplicationsforSTEMfieldsthatrelyheavilyontheseinstitutionsfortal-\nent. We have reframed our paper to highlight that we are studying distance learning in the\ncontextofthepandemic.\n5. It is a valuable practice to focus on students at the bottom of the performance distribution\nwhen examining educational gaps. However, it is necessary to provide more technical clari-\nficationsabouttheexpectedshortfallregression,especiallyaboutthesamplerestriction. The\ncurrent manuscript says that “0.20-ES refers to the average grade for the sub-population of\nstudents whose grades fall below the 0.20 quantile of the entire population.” This seems to\nsuggest that the thresholding happens at the student level. On the other hand, Equation (1)\nsuggests that the thresholding may happen at the student-course-term level. It is hence im-\nportant to know how the thresholding is done. In addition, because the grade distribution\nduringtheemergencystagesofCOVID-19shoulddiffersubstantiallyfromotherperiods,the\nthresholdingmightleadtoveryunevenrepresentationofobservationsoverdifferenttimespe-\nriodsand/orcohortsofstudents. Thismayessentiallyinducebiasandharmthevalidityofthe\nresults. Isuggestthattheauthorsexplicitlydiscuss,andifnecessary,addressthisissue,with\nmoredescriptivesoftherestrictedsampleandapplicationsofalternativeempiricalstrategies.\nWethankthereviewerforpointingouttheconfusion. Thestatisticalanalysesofthismanuscript\narebasedonanewlydevelopedmethodinthestatisticalliterature–theexpectedshortfallre-\ngression–that target directly on the lower-tail of the response distribution. It can be thought\nof as a generalization of the quantile regression which targets a particular quantile of the re-\nsponsedistribution.\nWe would like to clarify that the “thresholding” is not done at the student level. Model (1)\nutilizestheconditionalES,targetingtheconditional0.20-ES,conditionedoncovariates. For\nclarificationpurposes,wepresentanexamplefortheconditionalESwithonecovariate(sex).\nThe conditional ES at the 0.2 quantile level would be comparing the average of the bottom\n20%ofmaleandthebottom20%offemale. Thisisdifferentfromcomparingtofemaleand\nmale within the bottom 20% of the entire population. Thus, the “thresholding” will not be\nuneven over different time periods and cohorts of students and there should be no bias. The\nconceptissimilartothemorepopularlyusedquantileregressionwhereitlooksatthecondi-\ntionaldistributionataspecificquantilelevel. Differentfromquantileregression,ourstrategy,\ntheexpectedshortfallregression,lookattheaveragebelowaspecificquantilelevel,capturing\ntheaverageofthebottomtailoftheconditionaldistributionoftheresponsegivencovariates.\nWehopethatthiswillclearuptheconfusion.\nForeaseofunderstandingtheconceptoftheexpectedshortfall,wefirstpresenttheinterpre-\ntationforthemarginalexpectedshortfallinthedescriptionofthemanuscript,followedbyan\nextensiontotheregressionsetting. Intherevisedmanuscript,wehavemadesomeeditstothe\ndescriptionofthemethodasfollowsandwehopethatitisnowclear:\n“The quantity of interest for ES regression is the expected shortfall, compared to\nthe mean for ordinary least squares regression. If we consider a single variable,\nsuch as grades, the 0.20-ES refers to the average grade for the sub-population of\nstudentswhosegradesfallbelowthe0.20-quantileoftheentirepopulation,where\nthe 0.20-quantile of the grade outcome refers to the highest grade of those who\nhavethe20%lowestgradesamongtheentirepopulation. Inotherwords,ESgives\nus the average outcome for the lowest-performing students. To extend this con-\ncept to a regression setting, we focus on the average outcomes for the students\nwho fall below a certain threshold (at a specific quantile level) conditional on a\ngiven set of covariates. Specifically, ES regression allows us to estimate the rela-\ntionship between an explanatory variable (e.g. distance learning) and the average\noutcome for the lowest-performing students, while controlling for other relevant\nfactors/covariates.”(page10)\n“Model(1)utilizestheconditionalES,targetingthelowerquintileconditionalon\nspecifiedcovariatesratherthantruncatingthestudentsatthebottomoftheperfor-\nmancedistribution.”(page10)\n6. I’m not super familiar with the typical format of papers published at Nature Communica-\ntions,butitlooksduplicativetohave“Approach”and“EmpiricalStrategy”subsectionsunder\nSection 2 and an entire Section 3 of “Methods.” In most general science papers I have read,\nSection2isonlyabout(andtypicallytitled)results,wheremethodsareminimallydescribed\nonlytofacilitatereaders’understanding.\nWethankthereviewerforthecommentandweagreewiththereviewer. However, wecould\nnotchangetheorderofthemethodsandresultssectionduetoNature’sprovidedtemplatefor\nsubmissionathttps://www.nature.com/nature/for-authors/formatting-guide.\nAdditionally, we loosely followed the format of some papers in Nature Human Behavior\nand Nature Communications [Angrist et al., 2022, Lichand et al., 2022, Wu et al., 2022],\nwhichdescribethemethodsbrieflyintheapproachorresultssectionandhaveamoredetailed\nmethodssectionafterthediscussion.\nReviewer 4 Comments:\n1. Page 10, left, it mentioned: “The covariate matrix W includes the following...” However,\nict\nonpage9,formula(1),thecoefficientofW appearstobeascalar. Thisisconfusing.\nict\nThankyouforpointingthisout. WehavereplacedW withW toindicatethatitisamatrix.\nict\n2. Page 9 and 10, the paper explained that the tau-ES refers to the average grade for the sub-\npopulationofstudentswhosegradesfallbelowthetau-quantileoftheentirepopulation. How-\never,thedefinitioninformula(1)indicatesthesubpopulationisdefinedtobeY < Q (Y|X,W).\nτ\nSincetheright-handsideistheconditionalquantilegivenXandW,itisnotthetau-thquantile\nofthe“entirepopulation,”whichshouldbeQ (Y). Theclaimalsoappearsonpage9,bottom\nτ\nright. Someclarificationandfurtherdiscussionareneeded.\nWe thank the reviewer for the helpful comments. To introduce the concept of the ES, we\nhaveprovidedasimpleexampleonaunivariatevariablesuchasgrades, andthenextendthe\nideatotheregressionsetting. Inresponsetothereviewer’scomment,wehavenowrewritten\nour definition of the conditional ES such that it is consistent with the formula. Additionally,\nwehaveclarifiedthattheformulareferstoconditionalESandquantilewhereasourprovided\ninterpretations for reader understanding are for the marginal ES and quantile. We hope that\nthe revised manuscript is now clear. We attached a revised paragraph in the following and\nreferthereviewertopage10oftherevisedmanuscriptformodificationsinred.\n“The quantity of interest for ES regression is the expected shortfall, compared to\nthe mean for ordinary least squares regression. If we consider a single variable,\nsuch as grades, the 0.20-ES refers to the average grade for the sub-population of\nstudentswhosegradesfallbelowthe0.20-quantileoftheentirepopulation,where\nthe 0.20-quantile of the grade outcome refers to the highest grade of those who\nhavethe20%lowestgradesamongtheentirepopulation. Inotherwords,ESgives\nus the average outcome for the lowest-performing students. To extend this con-\ncept to a regression setting, we focus on the average outcomes for the students\nwho fall below a certain threshold (at a specific quantile level) conditional on a\ngiven set of covariates. Specifically, ES regression allows us to estimate the rela-\ntionship between an explanatory variable (e.g. distance learning) and the average\noutcome for the lowest-performing students, while controlling for other relevant\nfactors/covariates.”(page10)\n3. Page 10, left, “The estimates are ES regressions” should be “The estimates are from ES\nregression.”\nWe thank the reviewer for the suggestions. We have made the suggested edits in the revised\nmanuscript.\n4. (Remarksoncodeavailability)ProvideaREADMEfiletoexplainhowuserscanusethecode\nfor analysis. What are the meanings of each function, what are the main functions? Match\nthenotationsusedinthepaperwiththoseinthecode.\nInresponsetothereviewer’scomment, wehaveaddedadditionalexplanationstoeexplanationstoeachfunc-\ntioninthecode. Weadditionallychangedsomeofthefunctioninputnamestoalignwiththe\nnotationusedinthepaper.\n5. (Remarks on code availability) Provide a toy example (or a perturbation of the data, or a\nsubsample of the data, used in the paper) to demonstrate how to use the code to produce\nresultsinasimilarformatwiththosepresentedinthepaper.\nThanksfortheimportantsuggestions. Wehaveprovidedatoyexamplewithdataofthesame\nformat as the data used in the paper, and showed how we obtained the regression results,\nvalues used in the figures, and values used in the tables. The code are also appropriately\ndocumented."}, {"round": 2, "reviewer_comments": {"reviewer_1": "Revised Report: April 2025\nReport for revised manuscript entitled “Distance Learning Widens the Educational Gap for Underserved Students in STEM\nand a Way to Close the Gap.”\nThe authors have done an excellent job addressing the concerns I raised in my initial review. They now provide a clearer\ndiscussion of the practical significance of the observed effect sizes, particularly in the context of academic standing and\ngraduation thresholds. They also added additional analyses to assess confounding factors such as differing at-home\nconditions during the pandemic and strengthened their interpretation of distance learning effects relative to broader COVID-\nrelated disruptions.\nImportantly, they have clarified the post-pandemic impact of the FCI intervention and expanded the discussion on the\ngeneralizability of their findings by referencing similar initiatives at other institutions. I also appreciate the improved\nexplanation of the expected shortfall regression methodology, which now reads as far more accessible to a general scientific\naudience, though the methods section remains quite technical for a general audience.\nGiven these revisions, I believe the manuscript is now ready for publication.", "reviewer_2": "The authors have submitted a thoughtful and substantially improved revision of their manuscript. They have carefully\naddressed most of the reviewers’ concerns and clarified key elements of their contribution. The introduction now offers a\nstronger justification for the focus on STEM fields, particularly in relation to social mobility and structural barriers for\nunderserved students. The discussion of the practical implications of the GPA differences has been expanded and\ncontextualized with institutional policies.\nThe use of expected shortfall regression is better explained, and the addition of supplemental analyses focusing on post-\npandemic distance learning sections is a welcome effort to address concerns regarding confounding conditions. The authors\nalso respond to the generalizability issue by referencing comparable pedagogical initiatives in other universities, which\nhelps situate the FCI in a broader context.\nA few issues could still be slightly refined. While the distinction between distance learning and pandemic-related effects is\nacknowledged, it remains somewhat blurred in places. Similarly, causal language is still present in parts of the manuscript,\nalthough no identification strategy is employed. These could be easily adjusted in the final version.\nOverall, this is a well-executed, clearly written, and timely contribution that offers robust evidence on how remote instruction\nduring COVID-19 exacerbated educational inequalities, and how thoughtful pedagogical design can mitigate these effects.\nI support publication after minor revisions.", "reviewer_4": "The revision has satisfactorily addressed all my comments on the previous version. I have no further comments or\nsuggestions.\nThe functions in the R code have specifics regarding the inputs and outputs. The code file uses a toy example to produce\nresults in the paper. The functions are clearly written. However, I have difficulty installing one of the required packages with\none of the warnings as “ERROR: dependency 'osqp' is not available for package 'CVXR’”\nThe authors may provide some more specific instructions or alternative solutions in case other users encounter similar\nissues."}, "author_response": "ResponsetoReviewers\nWe thank the reviewers for reading our manuscript carefully and endorsing our manuscript for\npublication. Wehavemademinorrevisionstoourmanuscriptinresponse. Inthefollowing,wewill\nprovideresponsesaddressingeachreviewer’scomments.\nReviewer 1 Comments:\nTheauthorshavedoneanexcellentjobaddressingtheconcernsIraisedinmyinitialreview. They\nnow provide a clearer discussion of the practical significance of the observed effect sizes, partic-\nularly in the context of academic standing and graduation thresholds. They also added additional\nanalyses to assess confounding factors such as differing at-home conditions during the pandemic\nandstrengthenedtheirinterpretationofdistancelearningeffectsrelativetobroaderCOVID-related\ndisruptions. Importantly, they have clarified the post-pandemic impact of the FCI intervention and\nexpanded the discussion on the generalizability of their findings by referencing similar initiatives\natotherinstitutions. Ialsoappreciatetheimprovedexplanationoftheexpectedshortfallregression\nmethodology, which now reads as far more accessible to a general scientific audience, though the\nmethodssectionremainsquitetechnicalforageneralaudience. Giventheserevisions,Ibelievethe\nmanuscriptisnowreadyforpublication.\nThank you very much for your comments and providing us the opportunity to improve our\nmanuscriptwithyourpreviousreview.\nReviewer 2 Comments:\nThe authors have submitted a thoughtful and substantially improved revision of their manuscript.\nThey have carefully addressed most of the reviewers’ concerns and clarified key elements of their\ncontribution. The introduction now offers a stronger justification for the focus on STEM fields,\nparticularly in relation to social mobility and structural barriers for underserved students. The dis-\ncussionofthepracticalimplicationsoftheGPAdifferenceshasbeenexpandedandcontextualized\nwithinstitutionalpolicies.\nThe use of expected shortfall regression is better explained, and the addition of supplemental\nanalyses focusing on post-pandemic distance learning sections is a welcome effort to address con-\ncerns regarding confounding conditions. The authors also respond to the generalizability issue by\nreferencingcomparablepedagogicalinitiativesinotheruniversities, whichhelpssituatetheFCIin\nabroadercontext.\nAfewissuescouldstillbeslightlyrefined. Whilethedistinctionbetweendistancelearningand\npandemic-relatedeffectsisacknowledged,itremainssomewhatblurredinplaces. Similarly,causal\nlanguageisstillpresentinpartsofthemanuscript,althoughnoidentificationstrategyisemployed.\nThesecouldbeeasilyadjustedinthefinalversion.\nOverall, this is a well-executed, clearly written, and timely contribution that offers robust evi-\ndence on how remote instruction during COVID-19 exacerbated educational inequalities, and how\nthoughtfulpedagogicaldesigncanmitigatetheseeffects. Isupportpublicationafterminorrevisions.\nThankyouforyourcomments! Wemadeeditstoreducecausallanguage. Weacknowledgethat\nthe distinction between distance learning and pandemic-related effects may be blurred, and aim to\nrefertodistancelearningas”COVID-induceddistancelearning”throughoutourmanuscript.\nReviewer 4 Comments:\nTherevisionhassatisfactorilyaddressedallmycommentsonthepreviousversion. Ihavenofurther\ncommentsorsuggestions.\nThe functions in the R code have specifics regarding the inputs and outputs. The code file\nuses a toy example to produce results in the paper. The functions are clearly written. However,\nI have difficulty installing one of the required packages with one of the warnings as “ERROR:\ndependency ’osqp’ is not available for package ’CVXR’”. The authors may provide some more\nspecificinstructionsoralternativesolutionsincaseotherusersencountersimilarissues.\nThank you for your comments! We have investigated this issue, and have not encountered this\nwarningwheninstallingandloadingthelibraryCVXRonMacandWindowsRversion4.5.1. We\nhaveaddedanoteinthecodeexamplewhichversionofRthelibraryisconfirmedtoloadproperly\nunder."}]

Decision letter