https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 1/3

Deeply Flawed GMU Report on Online Education Asks 

Good Questions But Provides Misguided Analysis

Posted on January 27, 2019

Categories:Academics & Academia, Big Picture, Policy, Research

Tags:CBE, for-profits, GMU Report, Hoxby, IPEDS, Online Education

Phil Hill

Another year and another deeply flawed report about online education in US higher 

education, this time by Spiros Protopsaltis (associate professor and director of the 

Center for Education Policy and Evaluation at George Mason University, as well as 

former aide to Senate Democrats) and Sandy Baum (a fellow at the Urban Institute 

and professor emerita of economics at Skidmore College, as well as former advisor to 

Hillary Clinton's presidential campaign). As Inside Higher Ed described the report, 

titled "Does Online Education Live Up to Its Promise? A Look at the Evidence 

and Implications for Federal Policy":

Online education has not lived up to its potential, according to a new 

report, which said fully online course work contributes to socioeconomic 

and racial achievement gaps while failing to be more affordable than 

traditional courses.

The report aims to make a research-driven case discouraging federal 

policy makers from pulling back on consumer protections in the name of 

educational innovation.

In many ways this report takes a similar approach to the report by Caroline Hoxby 

from Stanford University, which was subsequently withdrawn, in asking important 

questions but providing flawed analysis to support conclusions. But unlike the 

previous report, the GMU one documents its sources well with 165 end notes, and 

for the most part this new report describes the underlying analysis accurately. 

Where the major problems arise is in conflating online education in general with the 
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 2/3

for- profit sector and in drawing conclusions that are not supported by the evidence.

The report is not easy to wade through, largely from its wide-ranging discussion of forprofits, online history, past federal policy, a snapshot of research on learning 

outcomes, and a discussion of current policy debates. Let's take the primary 

conclusions and discuss the analysis provided.

"Online education is the fastest-growing segment of higher 

education and its growth is overrepresented in the for-profit 

sector."

The report accurately describes the growth of online education, rising to point 

where one in three postsecondary students take at least one online course.

There is a disturbing tendency to describe this growth as "explosive" (mentioned 

five times in report) and an unexplained reliance in many cases on six year old data 

when new data exists. But the conclusion about growth is accurate.
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 3/3

The phrasing "overrepresented in the for-profit sector" and "concentration in the forprofit sector" in describing online education is very misleading, however. It is true that 

for-profit schools have a larger percentage of their students studying fully online, but 

the topic of the report is online education in general, and for-profits represent a rapidly 

shrinking minority of this case. Never mentioned in the report is the most salient point 

about for-profits - the sector is in major decline. As documented by IPEDS:

This decline seems relevant, even if you then look at fully-online programs (eLiterate analysis of IPEDS data).
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 4/3

Even in 2012, just two years after the for-profit peak, the for-profit sector accounted 

for less than 35% of fully online student enrollment, and as of Fall 2017 it was down 

to 21% with a clear trend. For- profits are rapidly becoming less and less relevant to 

the topic of online education, with no evidence to back up Protopsaltis claims that 

the for-profit sector is about to make a big comeback. It is high time that 

responsible analysts and scholars cease conflating online ed with for-profit 

schools, and the authors of this report should know better. If you want to study 

the for-profit sector, then describe it accurately and don't extrapolate beyond what the 

data supports.
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 5/3

"A wide range of audiences and stakeholders—including faculty and 

academic leaders, employers and the general public—are skeptical 

about the quality and value of online education, which they view as 

inferior to face-to-face education."

I find it strange to put this much emphasis on perceptions from an organization 

that purports to provide "timely, sound, evidence-based analysis", but 

perceptions are somewhat important to understand. The body of the report

describes a variety of research sources, but it is inaccurate to summarize that 

the wide range of stakeholders "view [online education] as inferior to face-toface education." Especially if you look at more recent data sources.

Consider the 2018 Inside Higher Ed / Gallup survey of faculty (starting page 32), 

where they found that faculty with actual experience teaching online have surprising 

high confidence in the quality potential of online education. For those who have 

taught online, the percentage that agree or strongly agree that “for-credit online 

courses can achieve student learning outcomes that are at least equivalent to those 

of in-person courses in the following context”, 39% for any institution, 52% at my 

institution, 54% in my department or discipline, and 58% in courses that I teach. Put 

simply, a majority of faculty who have experience teaching online think results can be 

at least equivalent to in-person.

Consider the 2019 Inside Higher Ed / Gallup survey of Chief Academic Officers, 

where fully 83% of them report plans to increase investment in online programs at 

their institution.

Consider the 2018 Northeastern University Survey on the Use and Value of 

Educational Credentials in Hiring, where they found that "Online credentials are 

now mainstream, with a solid majority (61%) of HR leaders believing that credentials 

earned online are of generally equal quality to those completed in-person, up from 

lower percentages in years past."

Yes, perception issues are important. But the report's conclusions are misleading and 
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 6/3

out of date

"Students in online education, and in particular underprepared and disadvantaged 

students, underperform and on average, experience poor outcomes. Gaps in 

educational attainment across socioeconomic groups are even larger in online than in 

traditional coursework."

This topic deserves its own report, and the GMU authors are right to point out that 

simply comparing online to face-to-face outcomes can obscure the important issue of 

underprepared and disadvantaged student experiences. On the surface, the 

conclusion about achievement gaps being "larger in online than in traditional 

coursework" is also accurate. But the more important question is not whether there is 

a problem, but rather how to minimize or reverse the achievement gap.

The report references several studies from the California Community College system, 

mostly from years ago, describing how students "were less likely to complete online 

courses and when they completed them, less likely to pass them". Yet the authors did 

not look at the trends within this system, as easily found in the most recent Distance 

Education report from the system, where the gap in performance overall for online 

versus tradition is closing rapidly.

.
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 7/3

More importantly, the achievement gains applied to all ethnic groups.

It does appear that the performance gaps within online education are not closing by 

ethnicity despite the broad improvements. That is a real question to consider. Rather 

than viewing a simplistic view that online = bad results, we should focus on how to 

maintain current improvements while figuring out how to do even better in providing 

equal opportunity.

"Online education has failed to improve affordability, frequently 

costs more, and does not produce a positive return on 

investment."

This conclusion is largely based on the NBER Hoxby report that was subsequently 

withdrawn, and for which I provided a detailed critique. I was not able to get a 

response from the report author. Beyond a gross mischaracterization of the source 

data, the Hoxby report made a fundamental flaw in its ROI analysis.

This view of online education - students choosing between non-selective 

face-to-face institutions or online institutions - takes a zero-sum approach, 
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 8/3

as if you have the same student population just choosing between 

institution types. This view ignores the large and growing number of 

working adults who can only attend college - often in degree- completion 

programs or masters level programs - because of an online option. Their 

real choice should be seen as online institution or not at all.

The GMU report relies on the withdrawn Hoxby report and does not even 

describe that it was withdrawn.

There is an excellent point made that pricing for students has largely not been lower 

for online education, but there are specific examples (UF Online, SNHU, WGU, to 

name a few) where they specifically provide much lower-priced offerings to students 

than comparable face-to-face programs. It would be interesting to study enrollment 

trends and student outcomes for lower-priced online programs compared to 

comparably-priced programs.

"Regular and substantive student- instructor interactivity is a key 

determinant of quality in online education; it leads to improved student 

satisfaction, learning, and outcomes."

"Online students desire greater student- instructor interaction and the 

online education community is also calling for a stronger focus on such 

interactivity to address a widely recognized shortcoming of current online 

offerings."

These last two points get to the primary purpose of the GMU report - current federal 

policy making efforts that include a re-evaluation of the Regular and Substantive 

Interaction (RSI) requirement for programs to be classified as online education and no 

correspondence courses.

The GMU report describes a large body of work documenting the importance of 

interaction to online student success, and the report accurately describes how "the 

online education community has also emphasized recently the importance of studentinstructor interaction for ensuring quality." This point is crucial - the vast majority of 

educators working in online education understand and accept the importance of 
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 9/3

interaction; there is not significant disagreement on the subject.

What the GMU report gets wrong is conflating actual quality interaction within 

courses with federal regulations. Much of the basis of the GMU analysis is a series of 

Office of Inspector General (OIG) reports calling out weak implementation of the RSI 

regulations. In the biggest case - a report on Western Governors University (WGU) 

and its competency-based model - this conflation is unwarranted, as I described in a 

detailed analysis of that action. There were two particular problems with the OIG 

findings in my view - the first is that the OIG defined their own terms due to the 

ambiguous nature of the RSI regulation.

The OIG used a binary role-based approach (you are an instructor or you 

are not) leading to conclusion that only course mentors and evaluators 

could be considered as instructors, however. The basis of this 

determination was an instructor must "provide instruction on course 

content" - clearly a content-dissemination view that rejects alternative 

pedagogies. And this interpretation that the OIG treats as unambiguous 

is not based on law, regulations, or commonly-accepted educational 

terminology. [snip]

This is why I call the audit methodology as hyper-literal. Somehow the 

OIG thinks they can determine - without any disagreement or ambiguity -

the "ordinary meaning of those terms" based on their own interpretations.

The second problem was that the OIG did not evaluate the actual courses or even 

address the issue of course quality.

Also note that the determination was entirely based on course design 

materials - think syllabus and course outlines. The OIG did not look at 

interactions arising during the course of actual course work, just whether 

there were pre-defined webinars, meetings, and student-instructor 

interactions. [snip]
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 10/3

These views essentially reject not just WGU's approach to CBE but also 

the broader movement of faculty from "sage on the stage to guide on the 

side". Instructors, from the OIG view, must provide instruction on course 

content and interactions must be pre-planned in the course design 

materials, at least for online courses.

The OIG did not look at student outcomes, applied its own hyper-literal translation of 

an ambiguous regulation, and did not look at the course interactions themselves -

just whether pre-planned course materials described future course interactions. 

Note, however, that despite the weakness of the OIG report, this does not mean 

that WGU is off the hook. Likewise, this report's over-reliance on the OIG reports 

mistakes regulation for actual interaction quality, but that does not mean that there is 

not an issue where many or most online courses could improve faculty-student 

interaction.

It is broadly understood that the RSI regulation is important but flawed. I agree with 

the GMU report that a simple elimination of the regulation would be a mistake. But it 

is overly simplistic and completely subjective for the GMU report to conclude that 

"unbundled faculty models that have difficulty complying should make changes to 

match the law instead of changing the law to match the needs of such models." That 

is a policy position and not based on "timely, sound, evidence-based analysis".

In Conclusion

This last point gets to the danger of this GMU report. It is a subjective set of policy 

recommendations disguised as extensively-documented evidence-based research. 

There is value in the questions asked, in much of the research documented in the 

footnotes, and in the clear policy position presented on regular and substantive 

interaction. But there is more harm than good from the report due to the 

mischaracterizations, selective data usage, and flawed analysis provided. Read it as 

a policy paper and not a research report.
https://mfeldstein.com/deeply-flawed-gmu-report-on-online-education/ 11/3

Paul Fain from Inside Higher Ed provided a valuable, pithy summary at the end of 

his article on the report.

The report’s co-authors and its critics agreed that further research is 

needed on the rapidly evolving field of online education, particularly as 

more high-quality colleges and universities ramp up their online offerings.