Query: Predictive Validity of Non-Cognitive Measures in College Admissions
Dear Editor,
I commend Paris, Beckowski, and Fiorot (2023) for exploring the predictive validity of motivational-developmental dimensions in test-optional college admissions. In my opinion, while aptitude tests are helpful, they only predict part of the variance in college outcomes, such as Grade Point Average (GPA), which measures a student's overall academic performance. I think the non-cognitive dimensions assessed in this study may indeed capture important, otherwise overlooked factors.
Social Desirability Bias in Non-Cognitive Measures
However, I believe the main challenge with using non-cognitive measures is the potential for social desirability bias. Students are highly motivated to present themselves in the best possible light in high-stakes contexts like college admissions. This pressure can lead them to craft responses that align with what admissions committees want to see rather than providing genuine reflections of their motivations or resilience. This response bias can significantly compromise the validity of these assessments.
Using an implicit test that forces examinees to choose an answer immediately without overthinking or incorporating impression management techniques like those used in Cattell’s 16 Personality Factor Questionnaire (16PF) could help overcome social desirability bias. The use of implicit measures, such as those developed in Project Implicit by Greenwald, Banaji, and Nosek, has shown promise in reducing response bias by assessing automatic associations rather than relying on deliberate self-reporting. This approach could provide a more accurate evaluation of applicants' non-cognitive traits. It would be more effective than using explicit tests like open-ended essay questions, which are prone to this type of bias. Establishing the predictive validity of non-cognitive measures is inherently challenging when the authenticity of responses is questioned. 
Selection Bias in Test-Optional Policies
Selection bias is also a concern. Since the study uses test-optional admissions policies, students who opted out of standardized tests might differ significantly from those who did not. This could introduce selection bias, affecting the generalizability of the findings. For instance, students choosing test-optional pathways may have different strengths or weaknesses than those who submitted standardized test scores, which could bias the study’s conclusions about the predictive validity of non-cognitive measures. Ultimately, this selection bias limits how much the findings can be applied to a broader population of college applicants.
Low Inter-Rater Reliability
Another issue is that these assessments are inherently subjective, as evidenced by the low inter-rater reliability reported in the study. To address this issue, standardizing scoring criteria or providing more comprehensive rater training could help improve consistency across different raters. In my view, as a fundamental principle of psychometrics, a test must be reliable before it can be considered valid. This limits the utility of non-cognitive measures in predicting academic performance.
Conclusion
Despite these challenges, this work opens essential avenues for expanding our understanding of student potential beyond standardized tests.
Call to Action
Cognitive tests used in admissions, such as aptitude tests, have limitations in predicting college GPA or measuring overall college success. Therefore, exploring and introducing alternative measures, such as non-cognitive assessments, is essential to capture variance in GPA not explained by cognitive measures. Incorporating well-researched methods that have shown better reliability is crucial to addressing the biases and weaknesses of non-cognitive measures.
I encourage admissions offices to pilot and evaluate implicit measures, such as those inspired by Project Implicit, alongside traditional tools. Collaboration between psychometricians, admissions professionals, and researchers can help refine these methods, ensuring they are valid and reliable. Additionally, further research into innovative strategies to address social desirability and improve the consistency of non-cognitive assessments is strongly recommended.


cite reference for this apa format
Subject: psychology
Intent: Metadata