Commentary: Social Responsibility in College Admissions Requires a Reimagining of Standardized Testing

As college admissions becomes more competitive in the United States and globally, with more applicants competing for limited seats, many programs are transitioning away from standardized testing as an application requirement, in part due to the concern that testing can perpetuate inequities among an increasingly diverse student population. In this article, I argue that we can only address this concern by reimagining standardized testing from the ground up. Following a summary of the recent debate around testing at the University of California (UC), I discuss how my perspective aligns with that of Koljatic et al. (2021), who encourage the testing industry to accept more social responsibility. Building on themes from the focus article and other recent publications, I then propose that, to contribute to educational equity, we must work toward testing that is more transparent and openly accessible than ever before. Over the past 12 months, the longstanding alliance between the UC and its preferred admission test, the SAT, has itself been the subject of high-stakes examination. In the spring of 2020, roughly two months into a global pandemic and following scandals (Jaschik, 2019) and litigation (Public Counsel, 2020) from the previous year, the UC Board of Regents voted unanimously via Zoom call to suspend the SAT/ACT as an admission requirement (UC Office of the President, 2020a, 2020b). Their resolution left open the possibility of finding or creating an alternative test by 2025. However, in the event that an alternative test was not feasible in such a compressed timeframe, the UC would eliminate testing requirements entirely. This UC resolution addressed growing concerns around the disparate impact of admission testing on underrepresented students (e.g., Geiser, 2017), but it was also nearly orthogonal to the recommendations of the UC Academic Senate (Douglass, 2020). The Senate had organized the Standardized Testing Task Force, which concluded through a comprehensive study that the benefits of testing, including to underrepresented groups, outweighed the costs, even for high-stakes selection purposes (Systemwide Academic Senate, UC, 2020). In the fall of 2020, the Board convened what they called a feasibility study work group, of which I was a member, to evaluate options for a replacement test. We determined that, although an existing test (e.g., the Smarter Balanced system adopted in California) could support lower-stakes components of the admissions process like determining eligibility, it would not be feasible to develop a new test effectively by 2025, and any high-stakes testing for selection purposes was not recommended (UC Office of the President, 2021). I present this background information in part to highlight one of the more recent and consequential transitions away from admission testing, in terms of number of students involved (roughly 118,000 admitted to the UC in 2020 from over 170,000 applicants; UC Infocenter, 2021). The UC case also demonstrates a range of views regarding what our next steps should be, with arguments for (a) new and better tests (e.g., Cai, 2020), (b) repurposed existing end-of-year tests (e.g., Kurlaender, Reber, & Rothstein, 2020), and (c) no more testing (specifically, high-stakes, norm-referenced testing; e.g., Geiser, 2017; Geiser, 2020). The most recent committee review in the UC case takes (c) the view of no more testing one step further, by confirming a lack of confidence in the feasibility of any equitable high-stakes testing (UC Office of the President, 2021). Thus, in our final examination of exams, not only do current approaches to standardized testing not meet expectations, but they are not even approaching expectations, and, in the UC at least, the door seems closed to alternative solutions, as long as the stakes are high. In their focus article, Koljatic et al. (2021) briefly review the recent controversy surrounding standardized testing, with reference to the UC and national exams in Chile, and then they outline how the testing industry must accept more social responsibility with respect to the impact of commercial tests. I agree with the general message of the focus article and of other recent extensions of consequential validity (e.g., Sireci, 2021). The validity of the testing industry itself, and the measurement community more broadly, with respect to achieving our own intended purposes, requires that we be more proactive in test design and implementation in order to contribute to the reduction of educational inequities. If such a contribution is not within the limits of our test purpose, we should revisit the test purpose, which may involve redefining our constructs (Randall, 2021) and reconsidering the necessary extent of our test standardization (Sireci, 2020). How do we apply this general message to the specific context of admission testing? By way of analogy, Koljatic et al. refer to the shift from corporate deniability to responsibility exhibited by Nike, Inc. in the 1990s. The multinational footwear and apparel company faced intense backlash when it refused to address abusive working conditions in its overseas manufacturing. Nike argued that manufacturing was via subcontract and thus outside of their control, but the backlash eventually impacted sales such that Nike accepted responsibility and took steps to improve working conditions (see Spar & La Mure, 2003; Spar & Burns, 2002; and Nisen, 2013; as referenced in Koljatic et al., 2021). The focus article outlines how the testing industry must similarly “develop a consciousness and level of responsibility about how their products affect their customer base” and then “integrate educational improvement and gap reduction as part of their business model” (p. 3). At the risk of pushing this analogy beyond its reasonable limits, we should dig deeper into the comparison between the apparel and testing industries, acknowledging first that the social issues faced within each industry differ in complexity. Improving worker conditions is reasonably within the sphere of influence of an apparel company contracting for specific work. We can assume that others had done so, and the solution for a company like Nike was clear. On the other hand, reducing test score gaps is much less tractable, with research showing that a variety of factors contribute to differential academic achievement (Carter, 2018). This is not to say that tests themselves are not one of those factors, or that the testing industry is right to step back and observe. Instead, this is to highlight that solutions will require partnerships well beyond the measurement community, including K12 education and other public programs. Along these lines, we should also recognize that the admission testing industry is essentially a subcontractor to state and university systems, building instruments according to specification. I do not have direct experience in admission test development, but it seems that the industry has not yet integrated gap reduction into their business model because, not only was that not a product specification, but because it was intentionally excluded from the stated purposes of the test. UC admission testing principles, for example, include “fairness that follows from uniform comparisons across the whole applicant pool” (e.g., UC Office of the President, 2009, p. 9), in other words, fairness in norm-referencing made possible by standardization. So, whereas Nike appears to have initially chosen deniability for convenience and personal gain, the testing industry has assumed a neutral position, in part, because they were expected or even required to do so. To be clear, the focus article does not claim that the testing industry is solely responsible for performance gaps. They acknowledge from the start that “college admissions tests are not the cause of adverse impact on the tests” (p. 2; emphasis in original), and that to promote social responsibility we must involve multiple stakeholders. However, the article does seem to suggest that the testing industry is currently taking a defensive position on its own and for its own benefit, as the apparel industry apparently did, which need not be true. Regardless of the complexity of the problem, and regardless of motives and blame to be shared among the different stakeholders, the testing industry should take a more active role in working toward a solution. Demand for primarily single-use, limited-access, high-stakes testing products is changing. In the UC, the balance between “fairness that requires all applicants to be treated the same” and “fairness that requires everyone to be treated as individuals” (e.g., UC Office of the President, 2009, p. 4) is transitioning toward the latter, signaling a shift from highly standardized, normative score interpretations to less standardized, criterion-referenced ones (Sireci, 2020). The focus article suggests some next steps for how we can adapt. I review these below and then, considering the magnitude of the issue, propose that we need to go further. Koljatic et al. (2021) recommend two related next steps for reorienting toward social responsibility in admission testing. First, they recommend that we attend more closely to admission test content, to ensure it aligns with the knowledge and skills needed in college, as opposed to broad curricular coverage, for example, as specified in state standards. This recommendation seems to address a recent trend of states using admission tests in place of grade eleven federally mandated testing (Lorié, 2020). Second, they suggest we measure and account for opportunity to learn when interpreting admission test results. In response to their first recommendation, I refer back to the UC case, where alignment with curriculum and state standards has been identified as a key principle in admission testing. In addition to fairness, reliability, predictive power, and utility “that justifies its social and monetary costs,” the Board of Admissions and Relations with Schools articulates that “an admissions test should measure levels of mastery of content in UC-approved high-school preparatory coursework and provide information to students, parents, and educators, enabling them to identify academic strengths and weaknesses” (UC Office of the President, 2009, p. 18). Emphasis is placed on criterion-referenced tests of curriculum standards, where the rationale seems to be that standards (a) can be broadly aligned with what students would need to know and be able to do in college and (b) can produce test content that is more openly accessible to students than what might be covered in a strictly norm-referenced test. Criterion-referenced, curriculum-based tests can also better support course planning and remediation. Aligning an admission test to state standards may also have drawbacks. The focus article notes two (when serving multiple purposes, a test may struggle to meet any one well, and quality of instruction can depend heavily on socioeconomic status). The Standardized Testing Task Force mentioned above discussed the limitations of using Smarter Balanced for admissions (Systemwide Academic Senate, UC, 2020). This issue of alignment has also been addressed in terms of the construct being tested, achievement versus aptitude (e.g., Zwick, 2004). In response to the focus article's second recommendation regarding measuring opportunity to learn, the college admissions process is often a holistic one wherein admission professionals already attempt to account for opportunity to learn via evaluation of transcripts and courses available at an applicant's high school (the UC incorporates this information into what is called Eligibility in the Local Context; UC Office of the President, 2009). The testing industry could potentially support processes like these, but a commercialized measure, developed and maintained external to an admission program, may not be preferable to measures managed internally. While the focus article recommendations are worth consideration, more dramatic changes are needed to address the challenges we face. As I argue below, shifting our mission as an organization and our business model as an industry to include gap reduction will require that we reimagine standardized testing from the ground up. Standardized admission testing has been criticized for a variety of reasons, including its financial cost for test takers, reliance on narrow content, reliance on limited administrations, differential test preparation, bias in individual items, and marginal predictive power (see chapters in Zwick, 2004 for review and discussion). Tests are also viewed as inauthentic and inaccessible to diverse student populations (Arbuthnot, 2020; Randall, 2021; Trumbull & Nelson-Barber, 2019). Admission testing and other forms of standardized testing then contribute to educational inequities when they are applied, with some or all of these limitations in place, in high-stakes, norm-referenced contexts that can detract from the learning experience and differentially limit educational opportunities (Geiser, 2017). Though safeguards typically aim to protect against misuse, the benefits are still not seen as outweighing the costs. Critics and proponents of admission tests disagree on whether these issues are problematic or negligible, and whether they threaten effective test use (Zwick, 2019). However, there should be consensus on one key fact, that standardized testing is not having the positive impact we hope it will. After decades of use in the United States, educational inequities persist. Sireci (2020) recommends that we free ourselves from the constraints of standardization in the pursuit of more flexible and accurate measurement at the individual level as opposed to accurate comparisons across test takers. Extending Sireci's (2020;2021) bold vision, and drawing from culturally responsive pedagogy (Gay, 2018) and open pedagogy (Jhangiani & Biswas-Diener, 2017), I argue that the path to reducing adverse impacts in admission testing, and in educational measurement generally, will also require admission tests that integrate with K12 education and that are (a) openly accessible in their content, (b) crowd-sourced in their development and improvement, and (c) fully transparent in their results. By openly accessible content, I mean item banks that are freely available online. Of any principle we might pursue, open-access has the most potential for positive impact, as it removes the veil of secrecy from the testing process and encourages us toward tests that are worth taking for the learning experience alone (Rohrer & Pashler, 2010). The occasional release of sample items is insufficient. We may still be constrained by technology, but the goal must be to build and evaluate item banks so deep that they capture an entire domain or universe of items within a specific task, and can thus be open for practice and later summative use without security concerns. Administration, scoring, and scaling procedures would then be designed with constraints as needed, depending on the application. By crowd-sourced development, I mean that the public as a community will support the review of content so as to organically and dynamically improve test quality. Not only does this promise to be more transparent and efficient than review by selected groups, but, with the right training, it also empowers the public to contribute directly to assessing fairness, sensitivity, and accessibility. Furthermore, a more diverse population, potentially the entire target population, will have access to the test, which will facilitate the rapid development of content that is more representative of and engaging for historically marginalized and underrepresented groups. This community involvement need not replace or diminish expert review. It can supplement it. Finally, by transparency in results, I mean that deidentified test data will be readily available for analysis at the item level, so that the public need not rely on the test developer as an intermediary for evidence of test quality. Similar to crowd-sourced development, this too will empower the public to be more informed consumers of testing information. A given test need not address all of these principles, or any one in its most extreme form, to derive benefits. Instead, open-access, community engagement, and transparency in results should be goals that we work toward. As we do so, we can expect positive results to follow. These results should include more buy-in and support from testing users and consumers; more expansive, diverse, authentic, and culturally representative test content; and a decrease in differential test preparation and corresponding increase in performance for underrepresented and marginalized groups due to everyone having continuous and full access to the test. In closing, I return to Nike for another analogy. The company recently released a new shoe that can be put on and taken off hands-free, extending their lineup of more accessible footwear (Newcomb, 2021). This innovation is regarded as a major step forward, so to speak, in inclusive and individualized design (Patrick & Hollenbeck, 2021). However, concerns have been raised about accessibility in terms of high cost and limited availability (Weaver, 2021). We can compare to admission testing in a variety of ways, but this example highlights at the very least the need for a more comprehensive consideration of accessibility. Admission tests, like other large-scale assessments, have historically been inaccessible to students, by design, until the moment of administration. Integration with K12 assessment systems would provide significantly more access and richer data for admission decisions (e.g., Kurlaender et al., 2020), and testing innovations promise measurement that is more individualized and engaging (The Gordon Commission on the Future of Assessment in Education, 2013). Yet, despite these advances, our products will still be largely inaccessible outside controlled conditions, like inclusively designed shoes that can only be rented or worn on certain occasions and under supervision. Our vision should be to distribute full ownership of the product itself. In the interest of space, I will leave a discussion of feasibility and obstacles to such a vision, of which there are many, for a future paper. I recognize that the goals outlined above are ambitious, and that they would lead to measures that hardly resemble the standardized tests we are now familiar with. But the challenge we face as an industry and community is an existential one, as emphasized in the focus article, a challenge that demands disruptive innovation. The UC, like other colleges and universities, has sent a clear message that our current solutions are unacceptable, and the innovations that have been proposed (e.g., Systemwide Academic Senate, UC, 2020) do not appear to go far enough to address the persistent inequities in our education system. Let us explore how we can go further.