# Trading off performance and human oversight in algorithmic policy: evidence from Danish college admissions

Magnus Lindgaard Nielsen, Jonas Skjold Raaschou-Pedersen, Emil Chrisander, David Dreyer Lassen, Julien Grenet, Anna Rogers, Andreas Bjerre-Nielsen

► To cite this version:

Magnus Lindgaard Nielsen, Jonas Skjold Raaschou-Pedersen, Emil Chrisander, David Dreyer Lassen, Julien Grenet, et al.. Trading off performance and human oversight in algorithmic policy: evidence from Danish college admissions. 2024. halshs-04859754

HAL Id: halshs-04859754

https://shs.hal.science/halshs-04859754v1

Preprint submitted on 30 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Trading off performance and human oversight in algorithmic policy: evidence from Danish college admissions*

Magnus Lindgaard Nielsen¹,²,*, Jonas Skjold Raaschou-Pedersen¹,², Emil Chrisander¹, David Dreyer Lassen¹, Julien Grenet³, Anna Rogers¹,⁴, and Andreas Bjerre-Nielsen¹,∗

¹University of Copenhagen

²Data Science Lab, Statistics Denmark

³CNRS and Paris School of Economics

⁴IT University of Copenhagen

*Corresponding authors, email addresses are respectively:, magnus.nielsen@sodas.ku.dk; abn@sodas.ku.dk

December 30, 2024

## Abstract

Student dropout is a significant concern for educational institutions due to its social and economic impact, driving the need for risk prediction systems to identify at-risk students before enrollment. We explore the accuracy of such systems in the context of higher education by predicting degree completion before admission, with potential applications for prioritizing admissions decisions. Using a large-scale dataset from Danish higher education admissions, we demonstrate that advanced sequential AI models offer more precise and fair predictions compared to current practices that rely on either high school grade point averages or human judgment. These models not only improve accuracy but also outperform simpler models, even when the simpler models use protected sociodemographic attributes. Importantly, our predictions reveal how certain student profiles are better matched with specific programs and fields, suggesting potential efficiency and welfare gains in public policy. We estimate that even the use of simple AI models to guide admissions decisions—particularly in response to a newly implemented nationwide policy reducing admissions by 10%—could yield significant economic benefits. However, this improvement would come at the cost of reduced human oversight and lower transparency. Our findings underscore both the potential and challenges of incorporating advanced AI into educational policy-making.

*We are grateful for comments that we have received in presentations at MD4SG/EAAMO, Copenhagen Education Network, Social Complexity Lab, DTU and CPH Tech Policy Committee as well as from Rene Kizilcec, Sune Lehmann, Mikkel H. Gandil and Tereza Blazkova. This project was funded by the Nation-Scale Social Networks grant from the Villum Foundation, a seed grant from Economic Policy Research Network and Independent Research Foundation Denmark grant 3099-00139B. All authors declare no competing interests. Views and conclusions expressed in the article are those of the authors and do not necessarily represent those of the Rectorate of University of Copenhagen.
# 1 Introduction

In the context of educational policy, data-driven insights are instrumental in addressing significant challenges, particularly in managing student dropout in traditional on-campus settings (see e.g., [1–3]) and online environments (see e.g., [4]). Student dropout carries significant costs at the individual, institutional, and societal levels, depending on educational and institutional arrangements [5, 6]. A key factor contributing to dropout is the potential mismatch between students and their chosen majors, which is exacerbated when students apply to specific fields without the possibility of changing later on [7, 8]. In Danish higher education, dropout rates of 25-30% have been a persistent concern for both colleges and universities, which are financially impacted due to funding mechanisms that reward timely graduation per student of 21-46 thousand USD, see Section SI 1.1 for details. This challenge is mirrored in admission systems globally, where admission officers prioritize applicants using a mix of assessment metrics such as GPA, test scores, and character assessments. These metrics aim to reflect merit and predict future life outcomes, albeit with varying degrees of success [9, 10]. One proposed solution is the use of early warning systems, as recent studies have found that the deployment of predictive methods to match students with courses and classrooms at their proficiency level can improve educational outcomes [11, 12]. Data-driven predictions could also improve admissions in the first place, bypassing the need for costly and sensitive post-admission data collection that raises privacy concerns [13]. More broadly, advances in machine learning (ML), particularly through large language models [14–16], offer novel data processing opportunities for public policy. In the fields of market and mechanism design, which focus on developing allocation systems like admissions, the potential of data-driven prioritization remains largely underexplored. However, the “black box” nature of such models also introduces new risks, emphasizing the need for increased oversight and regulation due to growing public concerns [17, 18].

Predicting dropout is challenging but essential, and it has been explored using various data sources and models [19]. In the context of admissions, however, research has struggled to identify predictive signals beyond prior grades or legally and ethically prohibited attributes [13, 20–22]. Transformer models, known for their ability to process high-granularity sequential information [14], present a promising breakthrough for more accurate predictions in individual life outcomes, including education, health, and labor [23–26]. However, their potential to improve decision-making or policies remains underexplored. Algorithmic models have been shown to reduce decision errors compared to humans, leading to positive net government revenue, thus being self-financing [27]. An essential consideration is the equity of policies and how they may discriminate against different sociodemographic groups. Predictive algorithms can be adjusted to reduce disparities in decision-making across demographic groups, potentially outperforming human decision-makers in terms of fairness [28, 29]. Despite this, current and new algorithmic policies have yet to be widely compared in terms of fairness, independent of decision thresholds. Moreover, in the context of admissions and allocation systems, existing studies have primarily focused on replicating human assessments of applicants or predicting academic performance in a limited number of programs [21, 22, 30–32]. Finally, these existing studies have not addressed the trade-offs related to compliance with new regulations, such as the EU AI Act or US Algorithmic Accountability Act, which emphasize transparency and human oversight.

We leverage a unique, nation-wide historical dataset on higher education admissions in Denmark, available to researchers through Statistics Denmark (see A.1 for details), to evaluate whether
prediction algorithms can improve admission decisions on a national scale. Our evaluation focuses on optimally ranking students based on available data by implementing a straightforward prediction policy, where model-generated risk scores create hypothetical rankings of students [1, 33]. When our predictive models utilize only pre-admission grades as inputs, they generate new, potentially non-linear transformations of grade data that capture more nuanced patterns beyond a simple point average. In centralized admission systems like Denmark's, these rankings directly determine which students are admitted to specific programs (Fig. 1A), enabling us to compare current admission policies with counterfactual algorithmic rankings for the admitted students. This strong connection to policy arises from the use of Deferred Acceptance, which admits students to their preferred study program among those where they rank above the admission cutoff [34]. Danish study programs primarily prioritize applicants based on high school GPA, but students can also apply for admission through a secondary quota where assessment is performed by humans [35, 36]. This dual approach (Fig. 1A) accommodates a diverse applicant pool and can improve graduation rates among those admitted through the secondary quota [35, 36]. By restricting our models to input features that are already accessible through the central registers, and focusing on predictions for future cohorts, we provide a credible evaluation of feasible admission policies based on data that is readily available before enrollment [6, 13, 37].
<figure><img src="image_2.png" /><figcaption>**Figure 1: Admission to higher education and sequence representation.** Panel A illustrates the general process for admission to higher education in Denmark, depicting two systems for ranking applicants: a mandatory GPA ranking, where all students are ranked by their high school GPA, and a merit-based human ranking, where students can opt-in voluntarily. Both rankings are used as inputs in a variant of the deferred acceptance mechanism [36]. Student receives a single enrollment offer from the highest-ranked institution where they qualify. Panel B displays the data observed after student enrollment, showing how information is stored in tabular form. Panel C shows the creation of 10 sequences based on these tabular representations, where the different colors indicate the source of the information in the tabular representation. These sequences begin with time-invariant events (e.g., socio-demographic information) followed by chronologically ordered events (e.g., grades). Each sequence encapsulates information regarding a specific aspect of events, represented by a column in the tabular representation, with the *i*th position in each sequence describing the *i*th event, regardless of its relevance.</figcaption></figure>
# 2 Results

We develop predictive models with study completion as the target variable, using various subsets of input data related to Danish students. These data subsets include raw grade transcripts from middle and high school, sociodemographic information about the students and their parents, application details, human assessments (when applicable), and records of study enrollment and completion. Our two primary models are based on distinct input sets: (1) the *academic* set, which includes only pre-admission grades and enrollment details, focusing solely on objective academic criteria while excluding protected attributes [38]; and (2) the *everything* set, which incorporates all available information, including sociodemographic data, application details, and human assessments (see Section A.5.2 for details).

We evaluate the performance of two deep learning architectures, the Transformer [14] and Long Short-Term Memory (LSTM) [39], comparing them against each other and against baseline models that are not optimized for sequential data, such as logistic regression and gradient-boosted trees (specifically XGBoost) [40]. Panels B and C of Figure 1 illustrate how student information is transformed into sequences for input into these advanced models.

## 2.1 Model performance

We begin by reviewing the performance of all estimated models using the AUC (Area Under the ROC Curve) score [41], as we are primarily interested in the rankings of students, which are used for seat allocation and can be compared to the currently implemented ranking methods (see Panel A of Figure 2). For the *academic* input set, the transformer model achieves the highest AUC score of 69.6% on our out-of-sample admission data, while the older LSTM architecture yields 67.4%. In comparison, ranking students based solely on their grade point averages (GPA) and study program indicators—included to encode idiosyncratic variation in graduation rates across study programs—results in AUC scores of up to 64.6%. Using aggregate grade information yields AUC scores of 68.5% and 68.3% for logistic regression and gradient-boosted trees, respectively. Thus, any ML algorithm using the *academic* input set to rank students outperforms the current methods of ranking by GPA or human assessment, with the Transformer showing a 1.1 percentage-point improvement in AUC compared to the second-best performing model, logistic regression (see standard errors in Table SI 2). In contrast, we do not observe a similar performance increase when using the LSTM architecture.

The transformer model based on the *academic* input set outperforms other models, regardless of input data type (see Figure 2.A), demonstrating its superior ability to extract signals with greater predictive power compared to conventional predictors like sociodemographic information in tabular models and other deep learning architectures using similar input data. We also observe that traditional ML methods improve when more data is included, while sequential models show no significant change in performance when protected group information is added (see Figure 2.A). We note that one of the core information sources in admissions is the *application* feature set, i.e., the programs applicants applied to and the order of their preferences. This feature set provides valuable insights into applicants’ ambitions and perceived skills, leading to the best-performing model when integrated. This highlights a fundamental tension between prediction quality and manipulability, a topic that is revisited in the Discussion section.
We also assess whether our graduation predictions reflect other aspects of student success. Specifically, we find a medium-sized correlation (0.34) between our baseline graduation predictions and student GPA, with correlations ranging from 0.35 to 0.42 across the different input sets and models (see Figure SI 2.A). Notably, the transformer model and logistic regression perform well in this regard. Similar correlation patterns are observed for students admitted based on human rankings, either as direct predictions or ranked within study programs, though these correlations tend to be lower overall (see Figures SI 2.B and SI 2.C). Another potential concern with large language models is the instability of their performance across different initializations [42]. However, we find that our performance measures remain stable, with standard errors across initializations at or below 0.3% (see Table SI 3).

**Counterfactual predictions and match quality.** A key question is whether our predictions accurately capture the match quality between students' backgrounds and their study programs. For instance, students with strong science backgrounds may be more suited to STEM programs than to the humanities. To investigate this, we use academic information to predict completion rates across different study programs. The correlations of these predictions across fields are shown in Fig. SI 1. Simpler models tend to generate more correlated (and thus less diverse) predictions across programs (Panel A). We decompose our predictions into three components (see methods in SI Section A.8): (i) program-specific completion rates that reflect, e.g., low predicted completion rates in natural science programs (Panel B); (ii) the individual student component, which captures, e.g., the high predicted completion rates for students from natural or social sciences backgrounds in other programs (Panel C); (iii) the residual component, which accounts for remaining factors, such as how students from social or natural sciences are less likely to complete service programs (Panel D). Our findings indicate that, in line with matching effects, students are generally better suited to programs within their chosen field. This is reflected in a 1% higher predicted completion rate for students in programs within the same field compared to those in different fields.

**Within program accuracy and contraction policy.** To evaluate the effectiveness of the different ranking methods, we simulate two counterfactual scenarios inspired by a recently enacted 10% reduction in student intake at Danish universities [43]. These scenarios involve increasing rejection rates across study programs while maintaining the relative sizes of the inputs. To simulate this, we divide the students into 10 deciles within each study program based on rankings generated by our models using the *academic* input set. We then calculate the mean program completion rate for each decile, as shown in Figure 2, separately for students admitted through GPA rankings (Panel B) and human rankings (Panel C). The 10% reduction policy corresponds to rejecting the lowest-ranked 10% of students in each study program, which allows us to evaluate the efficacy of current ranking methods (based on GPA or human assessment) against those produced by ML models. Our results show that both traditional and advanced ML models outperform current GPA and human-based rankings in predicting student success within study programs (Panel B and C in Figure 2). Specifically, when we apply a contraction by rejecting the lowest 10% (represented by bin 1), we find a significant improvement in graduation rates. For those admitted through GPA rankings, the difference is at least 9.3 pp (Figure 2.B), while for those admitted through human rankings, the difference is at least 7.9 pp compared to algorithmic rankings (see Table 1 for further details). Based on the bins in Figure 2, these find-
ings are robust for up to 20% of the admitted population, after which performance differences
between GPA, human assessment and ML are less clear.

The best-performing model is the transformer, followed by the LSTM, logistic regression, and gradient boosted trees. These findings indicate a trade-off between model complexity and performance, whether measured using AUC or contraction curves. Contraction curves for all input data types are shown in Figure SI 3.

<figure><img src="image_3.png" /><figcaption>**Figure 2: Predictive performance of models and admission criteria.** Panel A shows the model performance measured by AUC scores for different input types (in rows) and models (in columns). Panels B and C display the actual student completion rates as a function of predicted completion rates. The rates are binned by percent deciles, and performance is shown by model rank (solid lines) or observed admission rank (dashed lines). The performance scores are split by admission method: Panel B for GPA-based admissions and Panel C for human assessment. Students are perfectly ranked if completion rates start at 0% and abruptly jump to 100%. Horizontal dotted lines indicate the mean completion rate within each sample.</figcaption></figure>

If policymakers were to reduce student intake based solely on algorithmic risk scores, without considering specific study program admissions, we can evaluate overall risk by computing the mean predicted completion rates for all students rather than separately by program. This approach is reflected in our ungrouped contraction curves (see Figure SI 4). The analysis shows a similar ranking of models when using the *academic* input set, but with a notably lower graduation rate in the lowest bin, which highlights the unequal distribution of students likely to drop out across different study programs. In particular, we observe that the transformer model outperforms other models in the overall contraction curves compared to performance within individual study programs. This suggests that part of the Transformer's superior performance stems from its ability to more accurately capture the unequal distribution of dropout risks between study programs, compared to other models.
We also examine heterogeneity across academic fields (Fig. SI 5) to assess how well students admitted based on GPA rankings are screened across different disciplines. Due to small sample sizes and privacy concerns, human rankings are excluded from this analysis. We find that the effectiveness of algorithmic risk scores remains relatively consistent across fields. However, the screening ability of GPA varies considerably: in some fields, it performs on par with algorithmic scores, but in fields such as *Health and Welfare* or *Education*, GPA proves to be significantly less effective as a screening tool.

**Fairness.** A key aspect of regulating decision algorithms is assessing fairness by evaluating equal treatment across individuals based on sensitive attributes. We investigate three attributes that cover different dimensions of human life: socioeconomic status, nativity to country, and biological sex. To evaluate fairness, we assess whether our models and the admission criteria are equally accurate across these subgroups using the ABROCA (Absolute Between-ROC Area) measure [44], a merit-based fairness metric that is independent of arbitrary thresholds, offering a general assessment of fairness (see details in Section A.7) [45]. In our setting, the ABROCA also allows us to examine the fairness of current admission criteria. We display this comparison in Figure 3. Overall, our models and the admission criteria perform similarly in terms of fairness across these sensitive attributes. One exception is that the LSTM shows better fairness performance regarding sex and socioeconomic status, especially for students admitted through human assessment. In fact, the LSTM demonstrates higher fairness than the currently implemented systems across all dimensions (see standard errors in Figure SI 6). This suggests that it is possible to trade off some performance gains for improved fairness compared to the current policies. We observe no systematic difference in fairness according to the type of input data used. ABROCA scores across all input data types are shown in Figure SI 7, revealing that the LSTM generally has lower ABROCA scores, especially along the dimensions of sex and socioeconomic status, indicating better fairness in these areas.

Several other fairness metrics, such as sufficiency, independence, and separation (as defined in [46]) are commonly used in algorithmic audits. However, these metrics require a predicted score and thus cannot be calculated for rankings without further information about the underlying generating process. Therefore, we report these metrics for our prediction models only, as described in Section A.7 and displayed in Figure SI 8. For all models and sensitive attributes, we find that we cannot reject sufficiency, indicating that the models do not become more accurate when including the attribute. However, we do reject separation and independence for nearly all attributes and models, which means that we observe different error rates across groups (separation) and differing mean predicted scores across groups (independence), reflecting the inherent trade-offs between the fairness criteria [47, 48]. The direction of bias in the models is generally against historically disadvantaged groups, except in the case of sex, where we observe bias against males.
<figure><img src="image_4.png" /><figcaption>**Figure 3: Fairness of admission criteria and algorithms.** Measures of fairness for different models and inputs based on the ABROCA metric, weighted across institutions. Higher ABROCA values indicate more unequal performance, with 0 corresponding to no performance difference between groups. The fairness scores are divided by admission method: Panel A for GPA-based admissions and Panel B for human assessment. The columns display the fairness measure by sensitive attributes: whether a student is a Danish native (Native), sex (Female), and whether the student is above or below median socioeconomic status (SES).</figcaption></figure>

## 2.2 Explainability

Our models vary in complexity, with logistic regression being the least complex and gradient-boosted models representing medium complexity. Next, we showcase how one can generate local explanations of the predictions, even for the most complex model, the transformer. To achieve this, we apply a saliency technique borrowed from Natural Language Processing (described in Section A.6). This technique helps explain sequential models by highlighting which parts of the input data exert the greatest influence on the model's predictions.

Figure 4 illustrates how explainability analysis can offer insights into the transformer model. In Panel A, we present a portion of a saliency map for a sequence representing a student, providing a visualization of the saliency analysis. This map shows that the model places significant weight on some of the lower high school grades, and the prediction is influenced by all the contextual information about the events. Panel B through E analyze the same 100 students. Panels B and C analyze the transformer using the *academic* input set, while Panels D and E analyze the transformer using the *everything* input. Additionally, Panel C and E plot saliencies for reversed sequences with padding events removed, which allows for a clearer view of the saliency levels of the last non-padding events. For the *academic* input set in Panels B and C, saliency
increases as the sequence progresses, with the most salient event being the last one, which encodes enrollment. Similarly, in Panels D and E, we see the same pattern for the everything input sequences, though differences in the models' focus are apparent. In Panel D, there is a slight increase in saliency at the first event, which encodes the student's sex. Furthermore, the most salient events in these sequences show a more dispersed distribution towards the end, where multiple places of application are encoded. Overall, the saliency data suggest that the model 'pays attention' to specific events, in particular the later ones. Among these, the enrollment and application events seem to play the most important role in the model's decision-making process.

<figure><img src="image_5.png" /><figcaption>**Figure 4: Explanations of transformer model.** Panel A illustrates the application of saliency scores for local explanations, showing a segment of a sequence representing a student for the *academic* input set. Panels B and C display the event saliency across 100 sequences, both in order and reverse order, with padding events removed for the *academic* model. Panels D and E present the same analysis for the *everything* input set. Saliency scores are calculated as described in Section A.6.</figcaption></figure>

## 2.3 Public returns to algorithmic admissions

A key question for policymakers is how much value a specific public investment would generate. To assess this, we estimate the Marginal Value of Public Funds [49] for algorithmic admissions by comparing two counterfactual policies, both rejecting the 10% lowest-ranked students. The first policy uses the *academic* input data and is simulated from the gradient-boosted tree, which has the lowest performance among our models and hence yields a conservative estimate. The second policy serves as the baseline, reflecting the actual admission rankings in use. Detailed descriptions of both policies and our calculations are provided in Section A.9. Consistent with prior research, we assume that increasing the number of graduates produces positive social benefits, as education yields both monetary and non-monetary returns. We estimate that even the worst-performing algorithmic policy would increase government revenue by 85 million USD annually for each cohort (approximately 0.02% of Denmark's GDP [50]) compared to the baseline policy. This additional revenue would come from increased income and consumption taxes. Our
best estimate of the costs of implementing these policies includes 1 million USD in initial costs and 22 million USD in running costs, with 21 million USD of the running costs resulting from reduced screening performance when human judgment overrides the algorithmic decisions.

Given the uncertainty in the cost of implementing algorithmic admissions policies, we estimate the net government revenue under various cost scenarios, as shown in Fig. 5. We account for different delays in policy implementation, as well as variations in fixed and variable costs, discounting all values back to year 0. Our analysis shows that switching entirely to algorithmic admissions yields infinite Marginal Values of Public Funds even for initial and running costs significantly higher than our estimates. This result is driven by the scalability of algorithmic policies. Notably, when we focus on implementing the algorithmic ranking only to students admitted through human assessment (representing only 19% of the sample), we observe a more restrictive set of cost scenarios where policy implementation remains feasible (Fig. 8). These findings align with results from prior research [27], which also emphasizes that scalability is key to making such policies cost-effective.

Lastly, we assess the benefits of increased model complexity. While we observe a 1.1 percentage
point improvement in the AUC metric when using the transformer model compared to the
next-best performing model (logistic regression), it is important to weigh whether this leap
in model complexity—from an intrinsically interpretable model to a deep learning method—is
justified. The modest performance gain comes with increased challenges in meeting regulatory
compliance, a point we explore further in the Discussion section. However, the transformer
model generates 131 more graduates compared to logistic regression, translating to an estimated
additional 30 million USD in annual gains. This financial benefit could offset even substantial
cost increases associated with the added complexity and regulatory demands of using more
sophisticated models.

<figure><img src="image_6.png" /><figcaption>**Figure 5: Value of adopting prediction-based admission for different scenarios** The figure shows the net present value for various cost and development time scenarios. The two panels depict two different time horizons for implementing the prediction-based admission policies, corresponding to similar delays in revenue generation. We use the lowest revenue estimate if algorithmic screening were adopted, which is based on gradient-boosted trees. The dotted line indicates a revenue of 0, and the black marker indicates our estimated cost.</figcaption></figure>
# 3 Discussion

Predictive technologies and AI tools hold the potential to enhance college admission policies by improving accuracy, fairness, and transparency. However, realizing these benefits depends on the careful development and implementation of these tools to address all these objectives. In this discussion, we explore the implications of integrating ML models into public policy, focusing on the trade-offs between complexity and explainability, their roles in formalizing and diagnosing within admission policies, and the broader policy implications of adopting such technologies.

One fundamental aspect in deploying predictive models is the extent to which a model can be opaque, or “black-box”, a concern often tied to the accuracy/explainability trade-off [51]. This raises important questions about how actionable predictions are if we do not fully understand what drives them [6], and whether simpler mechanisms might outperform more complex predictive models [12]. In our analysis, we find that more complex models can indeed yield better performance and/or fairness metrics. One critical aspect of model complexity in AI deployment is actionability, which is often linked to the need for explainability. In our context, all risk scores generated by the models are capable of ranking applicants within the admissions system, making them equally actionable, regardless of the complexity of the model. However, transparent and explainable models are generally favored because they enhance public and expert trust, accountability, transparency, and ease of use and contestation [52–54].

**Regulatory compliance.** According to the recently enacted EU AI Act [55], AI systems used in college admissions would be automatically be classified as high-risk applications. This classification imposes strict compliance requirements “as regards risk management, the quality and relevance of data sets used, technical documentation and record-keeping, transparency and the provision of information to deployers, human oversight, and robustness, accuracy and cybersecurity” [55]. Although the detailed implementation of these regulations is still pending, we offer a discussion of how some of these requirements would apply to algorithmic solutions for student admissions.

Our results demonstrate that algorithmic admission policies outperform both GPA-based and human evaluations, with even the least effective algorithm showing significant long-term economic benefits (see Section A.9). A major factor contributing to this success is the availability of centralized registry data in Denmark, which covers the entire country and provides an exceptionally clean, complete, and well-documented dataset. This high-quality data ensures the reliability and accuracy of the predictions generated by the algorithms, which would help meet the stringent data quality requirements under the EU AI Act. However, these conditions may not be easily replicated in other countries or contexts where such comprehensive datasets are unavailable.

In terms of transparency, our second-best model, logistic regression, is inherently interpretable, offering clear insights into how predictions are made. By contrast, our best-performing model, the transformer, provides plausible explanations using current saliency-based methods. However, there are significant caveats to these explanations. Interpreting decisions made by deep learning models, such as transformers, remains a fundamentally unsolved challenge. State-of-the-art explainability methods can produce inconsistent results and may not accurately reflect the model's true decision-making process [56–58]. These inconsistencies are often addressed us-
ing ad-hoc heuristics, which can mislead stakeholders and end-users [59]. The lack of a widely accepted definition of a valid algorithmic explanation in policy further complicates the issue [60]. It remains unclear when a model is considered sufficiently transparent. Some advocate for inherently interpretable models, such as logistic regression, that offer transparency by design [54], while others argue that post-hoc explanations, which attempt to clarify how complex models reach their decisions, are sufficient [51]. Therefore, if transparency and the right to explanation are prerequisites in high-stake applications like college admissions, the deployment of deep learning models, such as transformers, hinges on the development of reliable, faithful explainability methods. Moreover, clear guidelines on what constitutes a valid explanation are necessary to ensure that these methods genuinely assist end-users.

While our algorithms generally outperform current policies, their performance varies across dif-
ferent fields of study. Assessing the robustness of these algorithms in terms of performance and
fairness for new cohorts, different study programs, and the broader student body—including
those who were not admitted—is essential due to the selective labels problem [29, 61]. Such
evaluations are critical before considering deployment. In Table SI 3, we assess the robust-
ness of the models to different initializations of the model parameters, showing that logistic
regression exhibits minimal variability in out-of-sample performance compared to more complex
models. Additionally, applying these models in other situations would require re-assessing their
performance to ensure they remain effective.

Even if a model is sufficiently interpretable for actionable human oversight, allocation problems
like student admissions present additional challenges. Overriding one prediction can affect the
rankings of other applicants, making decisions interdependent. This interdependence demands
careful consideration to ensure effective oversight and the ability to “override or disregard the
system” when necessary [62].

**Challenges and opportunities** Integrating predictive models into admission systems offers both opportunities and challenges. A key concern is the potential for feedback loops affecting student behavior. Unlike the transparent ranking by high school GPA, predictive models may require applicants to spend more time determining which programs are feasible, which could increase the risk of misinformed choices [63]. However, advancements in smart matching platforms can alleviate this concern by conveying which programs are likely to be feasible [64]. While we find that ranking students based on risk scores enhances accuracy, it also introduces the risk of manipulation, as applicants may try to game the system by adjusting their provided information. This trade-off mirrors broader concerns in market design, such as the strategy-proof Danish admission system, which intentionally excludes application information to prevent manipulation [36, 65]. Algorithmic rankings could undermine this policy, encouraging applicants to alter their course choices based on perceived effects on predicted scores, potentially compromising both fairness and accuracy [66].

The applications of predictive models and AI tools in admissions extend beyond simply ranking
applicants. A holistic evaluation of candidates could use these tools for various purposes, such as
replicating human rankings to enhance evaluation efficiency [30], or providing decision-makers
with summary information [6]. These algorithm-generated summaries could improve applicant
screening by compensating for human limitations in processing large amounts of information [67,
68]. In Denmark, the current secondary quota system tends to select high-performing students
through self-selection rather than an in-depth screening process [35]. However, integrating algo-
rithmic summaries into this process remains challenging because of human tendencies to ignore
or override algorithmic suggestions, which can sometimes result in suboptimal outcomes [69],
although there are instances where human judgment may outperform model predictions [29, 61,
69]. Moreover, algorithmic risk scores could be useful in both pre- and post-admission contexts.
Pre-admission, these tools can inform students about program compatibility, guide scholarship
distribution, and improve student tracking [11, 70]. Post-admission, predictive models could
support early warning systems aimed at improving student outcomes, such as identifying stu-
dents at risk of dropping out and providing timely interventions [71]. However, the effectiveness
of these systems, alongside issues of access and the potential societal implications for at-risk
students, is still being studied [6, 72].

An alternative use of algorithmic admission is to equalize access to education. Our findings suggest that ML-based admission policies do not exacerbate unfairness compared to current practices and, in some cases, may even reduce it, as observed with the LSTM model. Specifically, our models meet the sufficiency criterion, ensuring predictions are well-calibrated across different groups. However, due to inherent trade-offs in fairness measures, they violate the separation criterion [47, 48]. To further promote fairness and equalize opportunities for students from minority or low socioeconomic backgrounds, one approach would be to post-process the models to mitigate these disparities. This could involve formalizing affirmative action by adjusting the optimization process to prioritize predictive equality, effectively equalizing outcomes across groups [28].

4 Conclusion

This paper introduces a novel approach to algorithmic student admissions, which relies on repre-
senting sequential data and using deep learning architectures such as the transformer to predict
study completion. In our experiments, all machine-learning-based policies outperformed those
currently implemented in Denmark, when focusing on reducing student dropout and enhancing
academic performance. Our best-performing model is based on the transformer architecture,
which demonstrates superior performance by ranking students more effectively both within and
across study programs. However, logistic regression, while 1.1% AUC behind the Transformer,
still offers significant improvements over current methods and retains the advantage of being
more interpretable. Under a recently enacted policy that will reject the bottom 10% of appli-
cants in each program, our approach identifies subgroups with 16-23% lower completion rates
compared to the current policy. This suggests that a significant number of potential graduates
could be retained using our approach.

A critical consideration when applying predictive models in admissions is fairness. Our study demonstrates that the models we develop satisfy sufficiency [46] across a wide range of sensitive attributes. While some may advocate for models that satisfy equal opportunity, we use the more comprehensive fairness metric, Absolute Between ROC Area [44]. This metric allows us to compare our counterfactual data-driven rankings to current admissions practices without relying on arbitrary decision thresholds. Our results show that students admitted based on grade point averages or human assessment are ranked just as fairly by both our baseline models and the transformer. Additionally, the LSTM model ranks the students even more fairly, while also
delivering better performance when using algorithmic risk scores. This suggests that, in terms
of fairness and accuracy, algorithmic models can offer substantial improvements over traditional
admissions methods.

Given the high-risk nature of using machine learning in college admission policies, we conducted a preliminary assessment of its potential benefits and challenges with regard to regulatory compliance. While it remains difficult to provide faithful explanations for decisions made by the top-performing transformer-based model, we estimate that even intrinsically interpretable models, such as logistic regression, would significantly improve the matching between students and the programs they are offered. This improvement is economically significant, with an estimated increase of 85 million USD in yearly government revenue.

As outlined in our discussion, this study opens several avenues for further exploration in algorithmic public policy, both in admissions and other areas. One key question is whether the efficacy of machine learning can be extended to other policy-relevant domains. At a broader level, both transformers and LSTMs generate numerical embeddings of student academic trajectories, which may have value as covariates in causal machine learning models. Furthermore, there is a need to explore how fair or unfair current policies are in other domains compared to potential algorithmic alternatives. This invites further research into the broader impact and fairness of machine learning in public policy beyond education.

**Author contributions:** M.L.N. & A.B.-N. developed the study concept with input from the remaining authors. M.L.N., J.S.R.-P., & E.C. structured the data. M.L.N. & J.S.R.-P. performed the analysis under the supervision of A.B.-N. M.L.N. & A.B.-N. wrote the manuscript with input from the remaining authors.

References

[1] Himabindu Lakkaraju et al. "A Machine Learning Framework to Identify Students at Risk of Adverse Academic Outcomes". In: *Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*. KDD '15. New York, NY, USA: Association for Computing Machinery, Aug. 2015, pp. 1909-1918.

[2] Lovenoor Aulck et al. *Predicting Student Dropout in Higher Education*. arXiv:1606.06364 [cs, stat]. Mar. 2017.

[3] Dominik Glandorf et al. "Temporal and Between-Group Variability in College Dropout Prediction". In: *Proceedings of the 14th Learning Analytics and Knowledge Conference*. 2024, pp. 486-497.

[4] Jing Chen et al. "A systematic review for MOOC dropout prediction from the perspective of machine learning". In: *Interactive Learning Environments* 32.5 (May 2024). Publisher: Routledge eprint: https://doi.org/10.1080/10494820.2022.2124425, pp. 1642-1655.

[5] European Commission et al. *Dropout and completion in higher education in Europe : main report*. Publications Office, 2015.

[6] Lydia T. Liu et al. "Reimagining the machine learning life cycle to improve educational outcomes of students". In: *Proceedings of the National Academy of Sciences* 120.9 (Feb. 2023). Publisher: Proceedings of the National Academy of Sciences, e2204781120.
[7] Andreas Behr et al. "Dropping out of university: a literature review". en. In: *Review of Education* 8.2 (2020). eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/rev3.3202, pp. 614–652.

[8] Ralph Stinebrickner and Todd R. Stinebrickner. "A Major in Science? Initial Beliefs and Final Outcomes for College Major and Dropout". In: *The Review of Economic Studies* 81.1 (Jan. 2014), pp. 426–472.

[9] María Elena Oliveri. "Assessments Used in Higher Education Admissions". In: *Higher Education Admissions Practices: An International Perspective*. Ed. by María Elena Oliveri and Cathy Editors Wendler. Educational and Psychological Testing in a Global Context. Cambridge University Press, 2020, pp. 233–302.

[10] Lex Borghans et al. "What grades and achievement tests measure". eng. In: *Proceedings of the National Academy of Sciences of the United States of America* 113.47 (Nov. 2016), pp. 13354–13359.

[11] Peter Bergman, Elizabeth Kopko, and Julio Rodriguez. *A Seven-College Experiment Using Algorithms to Track Students: Impacts and Implications for Equity and Fairness*. en. SSRN Scholarly Paper. Rochester, NY, June 2021.

[12] Juan C. Perdomo et al. *Difficult Lessons on Social Prediction from Wisconsin Public Schools*. arXiv:2304.06205 [cs, econ, q-fin, stat]. Sept. 2023.

[13] Andreas Bjerre-Nielsen et al. "Task-specific information outperforms surveillance-style big data in predictive analytics". In: *Proceedings of the National Academy of Sciences* 118.14 (Apr. 2021). Publisher: Proceedings of the National Academy of Sciences, e2020258118.

[14] Ashish Vaswani et al. "Attention Is All You Need". In: *Advances in Neural Information Processing Systems* 2017-December (June 2017). arXiv: 1706.03762 Publisher: Neural information processing systems foundation, pp. 5999–6009.

[15] Jacob Devlin et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". In: *NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference* 1 (Oct. 2018). arXiv: 1810.04805 Publisher: Association for Computational Linguistics (ACL) ISBN: 9781950737130, pp. 4171–4186.

[16] OpenAI. *Introducing ChatGPT*. en-US. Mar. 2023.

[17] Michael Veale and Frederik Zuiderveen Borgesius. "Demystifying the Draft EU Artificial Intelligence Act — Analysing the good, the bad, and the unclear elements of the proposed approach". en. In: *Computer Law Review International* 22.4 (Aug. 2021). Publisher: Verlag Dr. Otto Schmidt, pp. 97–112.

[18] Jakob Mökander et al. "The US Algorithmic Accountability Act of 2022 vs. The EU Artificial Intelligence Act: what can they learn from each other?" en. In: *Minds and Machines* 32.4 (Dec. 2022), pp. 751–758.

[19] Eyman Alyahyan and Dilek Düştegör. "Predicting academic success in higher education: literature review and best practices". en. In: *International Journal of Educational Technology in Higher Education* 17.1 (Feb. 2020), p. 3.
[20] Clive R. Belfield and Peter M. Crosta. *Predicting Success in College: The Importance of Placement Tests and High School Transcripts*. en. Tech. rep. Publication Title: Community College Research Center, Columbia University ERIC Number: ED529827. Community College Research Center, Feb. 2012.

[21] Reem Al-Alawi, Gina Oliver, and Joe F. Donaldson. "Systematic review: Predictors of students' success in baccalaureate nursing programs". In: *Nurse Education in Practice* 48 (Oct. 2020), p. 102865.

[22] Kam Cheong Li, Billy Tak-Ming Wong, and Hon Tung Chan. "Predictive Analytics for University Student Admission: A Literature Review". en. In: *Blended Learning : Lessons Learned and Ways Forward*. Ed. by Chen Li et al. Cham: Springer Nature Switzerland, 2023, pp. 250-259.

[23] Youngduck Choi et al. "Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing". In: *Proceedings of the Seventh ACM Conference on Learning @ Scale*. L@S '20. New York, NY, USA: Association for Computing Machinery, Aug. 2020, pp. 341-344.

[24] Yikuan Li et al. "BEHRT: Transformer for Electronic Health Records". en. In: *Scientific Reports* 10.1 (Apr. 2020). Number: 1 Publisher: Nature Publishing Group, p. 7155.

[25] Keyon Vafa et al. "CAREER: Economic Prediction of Labor Sequence Data Under Distribution Shift". en. In: Oct. 2022.

[26] Germans Savcisens et al. "Using sequences of life-events to predict human lives". In: *Nature Computational Science* 4.1 (2024). Publisher: Nature Publishing Group US New York, pp. 43-56.

[27] Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. "The Unreasonable Effectiveness of Algorithms". en. In: *AEA Papers and Proceedings* 114 (May 2024), pp. 623-627.

[28] Moritz Hardt, Eric Price, and Nati Srebro. "Equality of opportunity in supervised learning". In: *Advances in neural information processing systems* 29 (2016).

[29] Jon Kleinberg et al. "Human Decisions and Machine Predictions". In: *The Quarterly Journal of Economics* 133.1 (Feb. 2018), pp. 237-293.

[30] Austin Waters and Risto Miikkulainen. "GRADE: Machine Learning Support for Graduate Admissions". en. In: *AI Magazine* 35.1 (Mar. 2014). Number: 1, pp. 64-64.

[31] Jennifer E. Timer and Marion I. Clauson. "The use of selective admissions tools to predict students' success in an advanced standing baccalaureate nursing program". In: *Nurse Education Today* 31.6 (Aug. 2011), pp. 601-606.

[32] Hansol Lee, René F. Kizilcec, and Thorsten Joachims. "Evaluating a Learned Admission-Prediction Model as a Replacement for Standardized Tests in College Admissions". In: *Proceedings of the Tenth ACM Conference on Learning @ Scale*. L@S '23. New York, NY, USA: Association for Computing Machinery, July 2023, pp. 195-203.

[33] Jon Kleinberg et al. "Prediction Policy Problems". en. In: *American Economic Review* 105.5 (May 2015), pp. 491-495.

[34] D. Gale and L. S. Shapley. "College Admissions and the Stability of Marriage". In: *The American Mathematical Monthly* 69.1 (Jan. 1962). Publisher: Taylor & Francis eprint: https://doi.org/10.1080/00029890.1962.11989827, pp. 9-15.
[35] Mikkel Gandil and Edwin Leuven. *College Admission as a Screening and Sorting Device*. en. SSRN Scholarly Paper. Rochester, NY, Sept. 2022.

[36] Andreas Bjerre-Nielsen and Emil Chrisander. *Voluntary Information Disclosure in Centralized Matching: Efficiency Gains and Strategic Properties*. arXiv:2206.15096 [econ, q-fin]. June 2022.

[37] Renzhe Yu, Hansol Lee, and René F Kizilcec. "Should college dropout prediction models include protected attributes?" In: *Proceedings of the eighth ACM conference on learning@ scale*. 2021, pp. 91–100.

[38] Uddannelses- og Forskningsministeriet. *Bekendtgørelse om adgang til universitetsuddannelser tilrettelagt på heltid*. Jan. 2022.

[39] Sepp Hochreiter and Jürgen Schmidhuber. "Long short-term memory". In: *Neural computation* 9.8 (1997). Publisher: MIT press, pp. 1735–1780.

[40] Tianqi Chen and Carlos Guestrin. "XGBoost: A Scalable Tree Boosting System". In: *Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*. KDD '16. New York, NY, USA: Association for Computing Machinery, Aug. 2016, pp. 785–794.

[41] Jin Huang and C.X. Ling. "Using AUC and accuracy in evaluating learning algorithms". In: *IEEE Transactions on Knowledge and Data Engineering* 17.3 (Mar. 2005). Conference Name: IEEE Transactions on Knowledge and Data Engineering, pp. 299–310.

[42] Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. *On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines*. en. arXiv:2006.04884 [cs, stat]. Mar. 2021.

[43] Uddannelses og Forskningsministeriet. *Udmøntning af sektordimensionering på universiteterne*. da. Apr. 2024.

[44] Josh Gardner, Christopher Brooks, and Ryan Baker. "Evaluating the Fairness of Predictive Student Models Through Slicing Analysis". en. In: *Proceedings of the 9th International Conference on Learning Analytics & Knowledge*. Tempe AZ USA: ACM, Mar. 2019, pp. 225–234.

[45] René F. Kizilcec and Hansol Lee. "Algorithmic fairness in education". en. In: *The Ethics of Artificial Intelligence in Education*. 1st ed. New York: Routledge, Aug. 2022, pp. 174–202.

[46] Solon Barocas, Moritz Hardt, and Arvind Narayanan. *Fairness and Machine Learning: Limitations and Opportunities*. MIT Press, 2023.

[47] Alexandra Chouldechova. "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments". In: *Big Data* 5.2 (June 2017). Publisher: Mary Ann Liebert, Inc., publishers, pp. 153–163.

[48] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. "Inherent Trade-Offs in the Fair Determination of Risk Scores". In: *8th Innovations in Theoretical Computer Science Conference (ITCS 2017)*. Ed. by Christos H. Papadimitriou. Vol. 67. Leibniz International Proceedings in Informatics (LIPIcs). ISSN: 1868-8969. Dagstuhl, Germany: Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 2017, 43:1–43:23.

[49] Nathaniel Hendren and Ben Sprung-Keyser. "A Unified Welfare Analysis of Government Policies". In: *The Quarterly Journal of Economics* 135.3 (Aug. 2020), pp. 1209–1318.
[50] International Monetary Fund. *Denmark: Country Profile*. 2024.

[51] Andrew Bell et al. "It's Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy". In: *Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency*. FAccT '22. New York, NY, USA: Association for Computing Machinery, June 2022, pp. 248-266.

[52] Michael Veale, Max Van Kleek, and Reuben Binns. "Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making". In: *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*. CHI '18. New York, NY, USA: Association for Computing Machinery, Apr. 2018, pp. 1-14.

[53] Daniel Antony Kolkman et al. "How to build models for government: criteria driving model acceptance in policymaking". en. In: *Policy Sciences* 49.4 (Dec. 2016), pp. 489-504.

[54] Cynthia Rudin. "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead". en. In: *Nature Machine Intelligence* 1.5 (May 2019). Publisher: Nature Publishing Group, pp. 206-215.

[55] European Parliament. *Artificial Intelligence Act*. en. Mar. 2024.

[56] Julius Adebayo et al. "Sanity Checks for Saliency Maps". In: *Advances in Neural Information Processing Systems*. Ed. by S. Bengio et al. Vol. 31. Curran Associates, Inc., 2018.

[57] Pepa Atanasova et al. "Faithfulness Tests for Natural Language Explanations". In: *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*. Ed. by Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki. Toronto, Canada: Association for Computational Linguistics, July 2023, pp. 283-294.

[58] Alon Jacovi and Yoav Goldberg. *Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?* en. arXiv:2004.03685 [cs]. Apr. 2020.

[59] Satyapriya Krishna et al. *The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective*. arXiv:2202.01602 [cs]. July 2024.

[60] Luca Nannini, Agathe Balayn, and Adam Leon Smith. "Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UK". In: *Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency*. FAccT '23. New York, NY, USA: Association for Computing Machinery, June 2023, pp. 1198-1212.

[61] Jens Ludwig and Sendhil Mullainathan. "Fragile Algorithms and Fallible Decision-Makers: Lessons from the Justice System". en. In: *Journal of Economic Perspectives* 35.4 (Nov. 2021), pp. 71-96.

[62] Lilian Edwards. "The EU AI Act: a summary of its significance and scope". In: *Artificial Intelligence (the EU AI Act)* 1 (2021).

[63] Emil Chrisander and Andreas Bjerre-Nielsen. *Why Do Students Lie and Should We Worry? An Analysis of Non-truthful Reporting*. arXiv:2302.13718 [econ, q-fin]. Mar. 2023.

[64] Felipe Arteaga et al. "Smart Matching Platforms and Heterogeneous Beliefs in Centralized School Choice*". In: *The Quarterly Journal of Economics* 137.3 (Aug. 2022), pp. 1791-1848.

[65] Alvin E. Roth. "The Economics of Matching: Stability and Incentives". en. In: *Mathematics of Operations Research* 7.4 (1982), pp. 617-628.
[66] Andreas Bjerre-Nielsen et al. *Playing the system: address manipulation and access to schools*. arXiv:2305.18949 [econ, q-fin]. May 2023.

[67] Daan Kolkman. "The usefulness of algorithmic models in policy making". In: *Government Information Quarterly* 37.3 (July 2020), p. 101488.

[68] Adam Harris and Maggie Yellen. "Decision-Making with Machine Prediction: Evidence from Predictive Maintenance in Trucking". en. In: (Jan. 2024).

[69] Victoria Angelova, Will S. Dobbie, and Crystal Yang. *Algorithmic Recommendations and Human Discretion*. Working Paper. Sept. 2023.

[70] Lovenoor Aulck, Dev Nambi, and Jevin West. "Increasing Enrollment by Optimizing Scholarship Allocations Using Machine Learning and Genetic Algorithms." In: *International Educational Data Mining Society* (2020). Publisher: ERIC.

[71] Robert Balfanz and Vaughan Byrnes. "Chapter 4 - Early Warning Indicators and Intervention Systems: State of the Field". In: *Handbook of Student Engagement Interventions*. Ed. by Jennifer A. Fredricks, Amy L. Reschly, and Sandra L. Christenson. Academic Press, Jan. 2019, pp. 45-55.

[72] Andrew D. Selbst et al. "Fairness and Abstraction in Sociotechnical Systems". en. In: *Proceedings of the Conference on Fairness, Accountability, and Transparency*. Atlanta GA USA: ACM, Jan. 2019, pp. 59-68.

[73] Atila Abdulkadiroğlu and Tayfun Sönmez. "School Choice: A Mechanism Design Approach". en. In: *American Economic Review* 93.3 (June 2003), pp. 729-747.

[74] Uddannelses- og Forskningsministeriet. *Tilskud til uddannelse*. da. Tekst. 2023.

[75] Ministry of Higher Education and Science. *Grading system*. en. Page. 2021.

[76] Wen Xiao, Ping Ji, and Juan Hu. "A survey on educational data mining methods used for predicting students' performance". en. In: *Engineering Reports* 4.5 (2022). eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/eng2.12482, e12482.

[77] Byung-Hak Kim, Ethan Vizitei, and Varun Ganapathi. *GritNet: Student Performance Prediction with Deep Learning*. arXiv:1804.07405 [cs, stat]. Apr. 2018.

[78] Shi Pu and Lee Becker. "Self-attention in Knowledge Tracing: Why It Works". en. In: *Artificial Intelligence in Education*. Ed. by Maria Mercedes Rodrigo et al. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2022, pp. 731-736.

[79] Fisnik Dalipi, Ali Shariq Imran, and Zenun Kastrati. "MOOC dropout prediction using machine learning techniques: Review and research challenges". In: *2018 IEEE Global Engineering Education Conference (EDUCON)*. ISSN: 2165-9567. Apr. 2018, pp. 1007-1014.

[80] Toan Q. Nguyen and Julian Salazar. *Transformers without Tears: Improving the Normalization of Self-Attention*. en. arXiv:1910.05895 [cs, stat]. Dec. 2019.

[81] Ilya Loshchilov and Frank Hutter. "Decoupled Weight Decay Regularization". In: *International Conference on Learning Representations*. 2018.

[82] William Falcon and The PyTorch Lightning team. *PyTorch Lightning*. Mar. 2019.

[83] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. *Learning Important Features Through Propagating Activation Differences*. en. arXiv:1704.02685 [cs]. Oct. 2019.
[84] Pepa Atanasova et al. "A Diagnostic Study of Explainability Techniques for Text Classification". In: *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*. 2020, pp. 3256-3274.

[85] Catherine Kung and Renzhe Yu. "Interpretable models do not compromise accuracy or fairness in predicting college success". In: *Proceedings of the seventh acm conference on learning@ scale*. 2020, pp. 413-416.

[86] Nathaniel Hendren and Ben Sprung-Keyser. *The Case for Using the MVPF in Empirical Welfare Analysis*. en. Tech. rep. w30029. Cambridge, MA: National Bureau of Economic Research, May 2022, w30029.

[87] George Psacharopoulos and Harry Anthony Patrinos. "Returns to investment in education: a decennial review of the global literature". In: *Education Economics* 26.5 (Sept. 2018). Publisher: Routledge eprint: https://doi.org/10.1080/09645292.2018.1484426, pp. 445-458.

[88] Michael Lovenheim and Jonathan Smith. "Chapter 4 - Returns to different postsecondary investments: Institution type, academic programs, and credentials". In: *Handbook of the Economics of Education*. Ed. by Eric A. Hanushek, Stephen Machin, and Ludger Woessmann. Vol. 6. Elsevier, Jan. 2023, pp. 187-318.

[89] Mie Dalskov. *Store samfundsøkonomiske gevinster af uddannelse*. Tech. rep. 2009.

[90] Philip Oreopoulos and Kjell G Salvanes. "Priceless: The Nonpecuniary Benefits of Schooling". en. In: *Journal of Economic Perspectives* 25.1 (Feb. 2011), pp. 159-184.

[91] Finansministeriet. *Økonomisk Analyse: Regneprincipper på beskæftigelses- og overførselsområdet*. da. Tech. rep. 2021.

[92] Finansministeriet. *Dokumentationsnotat om opgørelse af nettoafgiftsfaktoren*. da. Tech. rep. 2019.

[93] Finansministeriet. *Forslag til finanslov for finansåret 2017*. da. 2016.

[94] Jacob Gildenblat. *A python library for confidence intervals*. 2023.

[95] Eurostat. *International Standard Classification of Education (ISCED)*. en. 2023.
# A Material and Methods

In this section, we begin by describing the data sources being used in the study, the institutional setting of higher education applications in Denmark, and the specifics of our target and features. Next, we provide a more detailed explanation of the sequence creation method and introduce the models we selected. We then describe how we calculate saliency scores, define fairness metrics, how we examine match effects and, finally, how we estimate the Marginal Value of Public Goods.

## A.1 Data Sources and Ethical Considerations

We consider all students in higher education in Denmark for whom primary and secondary school grades are available between 2006 to 2017. All relevant data are sourced from Statistics Denmark registries, allowing us to link information on variables such as sex, immigration status, socioeconomic status, and more.

We applied for an Institutional Review Board (IRB) review but the study falls outside the purview of the IRB's jurisdiction. Since the IRB is primarily concerned with ensuring the ethical treatment of human participants in research, our study, which relies on secondary data analysis and involves no direct interaction with human subjects, does not require formal IRB oversight. All analyses are conducted at an aggregate level, with at least five observations for each measurement reported, to protect participants' privacy.

The study aims to assess how machine learning methods can help identify students at risk of dropping out and to compare the performance of these methods with the current admission system. In doing so, we seek to contribute to educational policy and practice by promoting student success and improving decision-making in higher education. To achieve this, we train our models on data from 2006 to 2016 and test the predictions on data from 2017. This approach serves two purposes: it mimics a policy scenario where only historical data are available and prevents data leakage that could occur due to correlated outcomes in a random train-test split [37].

## A.2 Institutional Setting

Higher education application and admission in Denmark is handled at the study program level, meaning applicants apply to a specific field of study at a particular institution (e.g., economics at the University of Copenhagen). All applications for higher education in Denmark are processed centrally by the Ministry of Higher Education and Science through a system called *The Coordinated Application (Den Koordinerede Tilmelding)*. Within this system, applicants rank the study programs they wish to attend, and they are offered enrollment at the highest-ranked study program for which they are eligible, based on the number of seats available and their ranking within each program.

There are two ranking systems for applicants, and study programs have the flexibility to choose how many students to admit using either system:

* GPA-based admission is considered the “default system” and is mandatory for all appli-
cants. In this system, referred to as *Quota 1*, applicants are ranked in descending order based on their high school GPA. A student's GPA is calculated from both exit exams and continuous assessments.

* Human-based screening is voluntary for students. In this system, referred to as *Quota 2*, students are ranked by humans, typically a faculty member from the institution they applied to. Rankings are legally required to be based on academic and objective criteria, but institutions have the autonomy to set specific criteria, which must be announced at least a year before admission [38]. These criteria vary across programs and may include factors such as *additional education, work experience, folk high school attendance, stays abroad, tests* (administered by the institution), and *motivated applications* [36].

Human-based admission was introduced to broaden access to higher education based on merit, allowing students without traditional GPAs—such as those from vocational schools—or those with GPAs below the cutoff to enroll [35]. Admission criteria for most study programs under this system typically include CVs, grades, and essays [35]. Students are matched with institutions using a Deferred Acceptance mechanism with Voluntary Information Disclosure, specifically the DAVID-Q mechanism as described by [36]. This corresponds to each institution having two separate rankings of students: based on GPA (mandatory) and one based on human assessment (voluntary), which are handled independently in the Deferred Acceptance mechanism [34]. Importantly, the mechanism is *strategy-proof* [73], meaning students have no incentive to misrepresent their preferences when ranking programs. Disclosing voluntary information, such as taking an admission test and performing poorly, does not negatively affect their chances of admission based on GPA. The only cost associated with applying through the human-based system is the additional effort required, such as writing a motivated application. This process is visually represented in Panel A of Figure 1.

We have access to student GPAs for the entire population, as well as the rankings of students by humans for the subset of individuals who applied for human-based ranking, both sourced from *The Coordinated Application*. This allows us to evaluate the GPA-based and human-based systems as if they were algorithms. However, it is important to note that we can only compute evaluation metrics that require rankings within individual study programs for human-based assessments, as we do not have access to an overall human ranking of all students.

## A.3 Target

In our analysis, we use degree completion at a study program, conditionally on being admitted, as our target outcome for two main reasons: (1) it aligns with one of the primary policy goals of the study programs themselves, as a portion of their funding depends on student graduation rates [74], and providing education is costly for society; and (2) in the institutional context of Denmark, where students do not pay for higher education and generally receive financial support from the government while studying, the incentives for voluntarily dropping out of a study program are reduced. We acknowledge that our target outcome may not fully capture the perspective of students who find value in their even without completing their education, and that the suitability of a target outcome depends on the stakeholders involved
## A.4 Features

The core features included in our models are grade transcripts for middle and high school, which provide detailed contextual information about the grades received, e.g., the course name, course level, type of examination, etc. In Denmark, grades are assigned on a 7-point scale, aligned with the ECTS scale, ranging from A to F [75]. Course level in Denmark indicates the number of years a course is undertaken and is categorized as C, B, and A, corresponding to workloads of 1, 2, and 3 years, respectively. For example, a student might receive a grade A in mathematics at level A in an oral exam. Each grade is also linked to a specific academic year and institution. In primary school, this consists only of an institutional identifier and grade level, while in high school, it includes additional information such as the type of high school (e.g., regular or technical) and study line, which students choose and which determines the minimum required workloads for different courses. In the previous example, the math grade could have been earned at institution 12345 while enrolled in a study line like *Math-A-Physics-A* at a regular high school. This results in a large number of unique combinations in the data. When focusing on courses, course levels, and examination types alone, we observe 649 unique combinations in our training data (see Table SI 10 in the Appendix).

We also incorporate information on student applications. In Denmark, students can rank up to eight study programs, and they are admitted to the highest-ranked study program for which they are eligible [35]. We observe the number of study programs a student applies to, whether the student opts into human assessment for each program, and the resulting human rankings if they opt in. However, we do *not* have access to the non-grade materials used for human assessments, such as essays or CVs. This results in a smaller feature set for our models compared to the information available to human evaluators.

## A.5 Models

A key challenge in educational data mining has been extracting meaningful signals about a student's academic potential from noisy data. Most existing studies rely on the common tabular format [76]. However, more recent research has explored the use of deep learning models in educational contexts, employing data representations and model architectures tailored to specific problems. For example, some studies have focused on predicting online course completion both before and during course enrollment [77], while others have investigated methods for measuring students' implicit knowledge [78].

### A.5.1 Sequence Creation

We rely on multiple data sources for each student, including sociodemographic information, grade transcripts, and enrollment records, as shown in Panel B of Figure 1. Each data source is preprocessed independently, breaking the task into simpler, manageable steps while allowing us to use tried-and-true preprocessing techniques. To prepare the data for modeling, we transform the tabular data into sequences of events, arranging them in chronological order. Each event is represented by categorical variables that describe different aspects of that event. To maintain consistency, all continuous numerical variables are converted into categorical variables. This is
done by first applying winsorization to the top and bottom 5% of values to mitigate the influence of outliers, and then converting the remaining values into percentiles.

For a given student, there exist C sequences, denoted as $f^c = \{v_1^c, v_2^c, \dots, v_n^c\}$, where n represents the number of events for that student. Each c in C corresponds to a specific aspect of events, with each aspect corresponding to a single column in the tabular representation. The value $v_i^j$ denotes the i-th value in the c-th sequence. We represent each value as a string, which means that, in practice, each sequence corresponds to a sentence, with each value being a word. The words at position i encode information about the same event i across all sentences. We then trim sentences by removing the initial words observed or pad sequences to a fixed length L, corresponding to the 95-th percentile of sequence lengths. Sociodemographic variables, when included, are never removed during trimming, ensuring their consistent inclusion across sequences. Padding is achieved using a special token [PAD].

As shown in Panel C of Figure 1, many sentences may lack a specific word $v_i^c$ associated with an event due to certain aspects being irrelevant. For example, if the i-th event is an enrollment, there will be no associated value in the grade aspect, as the grade aspect is irrelevant for enrollment events. In such cases, we replace these words in irrelevant aspects with a null token ([Null]). Finally, we prepend a [Null] token at the beginning of all sentences and generate a new sentence starting with a single classification token ([CLS]) followed by [Null] tokens until the sequence reaches the fixed length L. The [CLS] token is used to represent the full set of sentences and is utilized for making predictions in subsequent tasks, as described in [15]. Strings occurring fewer than 250 times in the training set are replaced with a special unknown token [UNK], to manage rare occurrences.

All the unique strings form the vocabulary, denoted as $V$. To represent these concepts numerically, we convert them into binary vectors using a one-hot encoding scheme. While the vocabulary size $|V|$ is large, we reduce the dimensionality of one-hot-encoded vectors by embedding them into a lower-dimensional space using a linear mapping, represented as $E_{v_i^c} = f(v_i^c) \in \mathbb{R}^H$, where H is the dimensionality of the embedding space. This linear mapping is optimized jointly with the other model parameters, except for $E([Null])$, which is specifically set to map to a vector of zeros.

We now have a vector of dimensions $L \times C \times D$, which we reduce to a vector of dimensions $L \times D$ by summing over the aspects for each event:

$$AGG_i = \sum_{c=1}^{C} E_{v_i^c}.$$

This $(L \times D)$-dimensional vector can then be used as input to any deep learning architecture that processes sequences of fixed dimensions, making our representation usable in many different contexts. This approach allows us to create sequences of contextualized events, where multiple columns provide context for each event. The aggregation method is visualized in Panel C of Figure 6.
<figure><img src="image_7.png" /><figcaption>**Figure 6: Sequential model architectures with aggregate embeddings** Panel A illustrates the pre-norm encoder-only transformer architecture, while Panel B presents the LSTM architecture. Panel C provides an example of input embedding and summation, corresponding to the red elements shown in Panels A and B. For each event, values are first embedded into vectors, and the aggregate embedding of each event is represented as a summation of these vectors. This process matches the input embedding and summation shown at the bottom of Panel A.</figcaption></figure>
### A.5.2 Data Input Variants

We develop six distinct models, both sequential and baseline, varying the input data used. We construct a baseline GPA input data set, consisting solely of students' GPA. Next, we construct an *academic* input dataset, designed to mirror the requirement of objective and academic criteria that apply to the human assessment protocol. We then extend this dataset in various ways: The *human assessment* input dataset incorporates human rankings, acknowledging the additional insights these rankings may provide. The *application* input dataset expands upon the academic input by including comprehensive application details, such as rankings of both enrolled and non-enrolled study programs. However, including application information introduces the risk of students strategically manipulating their application behaviour to gain an advantage. The *sociodemographic* model further extends the academic input by adding contextual information on the institutional environment (e.g., selection into specific primary and high schools), and sociodemographic information about students and their parents. While we expect this model to enhance performance, we also acknowledge potential fairness concerns. Lastly, the *everything* model encapsulates all elements from the preceding models, providing a multifaceted approach to predicting study program completion. The specific details of the input variables included in each variant of the sequential models are outlined in Table SI 4. In practice, we arrange all events chronologically and include all events occurring in the same year or prior to enrollment, while filtering out irrelevant event types (e.g., excluding application events and sociodemographic events from the *academic* models) and contextual details for each event (e.g., excluding the institutional ID associated with grades in primary and high school from the *academic* models). The variables used in the variants of the baseline model are presented in Table SI 6.

### A.5.3 Sequential Models

To predict an outcome from these sequences, we use deep learning models suited for classification tasks, with several options available in natural language processing. For our sequential models, we use an encoder-only transformer [15] and an LSTM (Long Short-Term Memory) model [39]. The attention mechanism in the transformer allows the model to capture long-range dependencies in the data [14], while the LSTMs have been widely used in the educational context [79] and are common in natural language processing, although transformers are increasingly favored. Given the widespread use of these architectures and their standard implementations, we do not provide a detailed description here. Instead, we refer readers to [14, 15, 80] for detailed discussions of the transformer architecture and to the PyTorch documentation for the LSTM. Both the transformer and LSTM modules are implemented in PyTorch. Visual illustrations of the transformer and LSTM architecture can be found in Panels A and B of Figure 6.

Model optimization is conducted using AdamW [81] with the hyperparameters $(\beta_1, \beta_2, \lambda) = (0.9, 0.999, 0.01)$, incorporating 100 warmup steps and a Cosine learning rate scheduler. We train using Distributed Data Parallel (DDP) in pytorch lightning [82]. All models are trained for 10 epochs on four NVIDIA A100 GPUs, minimizing binary cross-entropy loss. Early stopping is applied, monitoring validation loss with a patience of 3 epochs. We do not pretrain our models, focusing exclusively on the downstream task of predicting study program completion. For model selection, we reserve 5% of the training sample, stratified by year, as validation data.
**Transformers.** We follow the notation from [15], where *L* represents the number of layers, *H* the hidden size, and *A* the number of self-attention heads, while the feed-forward size is fixed at 4*H*. Our models are trained with *L* = 8, *H* = 512, and *A* = 8. Notably, scaling these hyperparameters up or down by a factor of two results in similar performance on the validation set, indicating robustness to hyperparameter variations. A dropout rate of 10% is applied to all non-embedding layers. For predictions, we extract the [CLS] token from the final layer and use a simple linear layer for classification. The GELU activation function is employed, and normalization is performed before the attention mechanism [80]. The models are trained with a learning rate of 5 · 10⁻⁴ and a batch size of 512 (equivalent to an effective batch size of 2048). Exploration within a learning rate range of 1 · 10⁻⁴ to 1 · 10⁻³ resulted in comparable performance on the validation set.

**LSTM.** We train LSTM models with a hidden size and input size of 128, using 2 layers and applying a dropout rate of 20%. Exploration across a range of hidden and input sizes (64 to 512), layers (1 to 3), and dropout rates (10% to 40%) yielded similar performance on the validation set. For predictions, we extract the last hidden state and pass it through a simple linear layer for classification. Training is performed with a learning rate of 5 · 10⁻⁴ and a batch size of 128 (equivalent to an effective batch size of 512). Exploration with higher learning rates (up to 1 · 10⁻³) a,d different batch sizes yielded comparable results, while the lower learning rates resulted in inferior performance.

### A.5.4 Baseline Models

For our baseline tabular models, we use logistic regression and gradient-boosted trees, specifically XGBoost [40]. Grade information is encoded by including the GPA and representing grades through the mean grade within each course. Additionally, we calculate and include the mean, standard deviation, and count of grades within three broad fields: *STEM*, *Languages*, and *Other*. Aggregate information is computed separately for primary and secondary schools (see summary statistics in Table SI 9). Missing values are handled differently across models: in logistic regression, we apply median imputation, while XGBoost handles missing values internally. In unknown values are encountered during one-hot encoding in the test year, a vector of zeros is used.

We select the best hyperparameters through 3-fold cross-validation coupled with a randomized grid search, selecting the hyperparameters that yield the highest average out-of-sample AUC score across the left-out folds. We evaluate 30 hyperparameter combinations for both gradient-boosted trees and logistic regression, drawing from the distributions specified in Table SI 7.

### A.6 Explainability

We use saliency scores to examine how the sequential models generate predictions. These scores are computed using the *InputXGrad* method [83] with l2-norm aggregation, as this approach has been shown to most closely mimic human annotation across various Natural Language Processing tasks [84].
The saliency score $S_j$ for each numeric input $x_j$ is calculated as the partial derivative of the neural network (NN(·)) with respect to the input, multiplied by the input:

$$S_j = x_j \cdot \frac{\partial \mathrm{NN}(x_j)}{\partial x_j}.$$

Since each input $v_i^c$ maps to a high-dimensional vector $E_{v_i^c} \in \mathbb{R}^H$, we compute $H$ saliency scores for each input. To aggregate these into a single saliency score for a given input, we use the $l_2$-norm [84]. Denoting $S_{jh}$ as the $h$-th element of $S_j$, the saliency for an input $v_i^c$ is given by:

$$\mathrm{Attr}_{ic} = \sqrt{\sum_{h=0}^{H} S_{jh}^2}.$$

We further define the saliency score of an event $i$ as the sum of the saliency scores for each aspect of the event:

$$\mathrm{Attr}_i = \sum_{c=1}^{C} \mathrm{Attr}_{ic}.$$

While an alternative approach would be to use local surrogate models with tabular input to explain predictions in a human-interpretable manner, we opted not to use this method. The performance improvements in our models are largely due to the inclusion of highly granular, which cannot be effectively captured or explained by local surrogate models based on tabular representations.

## A.7 Fairness

To assess the fairness of our models, we rely on three key concepts: independence, separation, and sufficiency, as described by [46]. These concepts are evaluated in terms of the sensitive attribute ($A$), the target variable ($Y$), and the model's score ($R$), respectively. However, a limitation of these fairness metrics is that they require a score for each observation, whereas the admissions system only provides rankings. To address this, we also use the Absolute Between-ROC Area (ABROCA) metric introduced by [44], which allows for a more direct comparison between our models and the current admissions system. These operationalizations of fairness have been applied in previous work on educational data mining [45, 85]. In this section, we briefly describe each of the fairness concepts and their normative underpinnings.

**Independence**, also referred to as demographic parity, requires that the model's score $R$ is independent of the sensitive attribute $A$. The normative assumption underlying this concept is that our model should predict the same graduation rates across various groups defined by the sensitive attribute. To test for demographic parity, we test the null hypothesis $H_0: P(\bar{Y}|A = a) = P(\bar{Y}|A = b)$, where $Y$ represents the target outcome (graduation).

**Separation**, also known as error rate parity or equalized odds [28], requires that $R$ is independent of $A$, conditional on the outcome $Y$. The normative assumption here is that the model should have the same true and false positive rates across the groups defined by the sensitive attribute, allowing distinctions between groups only when justified by the outcome. To test for
equal true and false positive error rates, we test the null hypotheses $H_0: P(\bar{Y}|Y = 1, A = a) = P(\bar{Y}|Y = 1, A = b)$ and $H_0: P(\bar{Y}|Y = 0, A = a) = P(\bar{Y}|Y = 0, A = b)$.

**Sufficiency** requires that $Y$ is independent of $A$ given $R$. The normative assumption is that our model should fully capture the information in the sensitive attribute. To test this, we group observations into five bins based on the quintiles of the scores, then test the null hypothesis $H_0: P(\bar{Y}|A = a, R = r) = P(\bar{Y}|A = b, R = r)$ for each bin $r$. If any test is rejected, the model fails to satisfy sufficiency.

To perform these hypothesis tests for independence, separation, and sufficiency, we use two-proportion z-tests at a 5% significance level. For sufficiency, given that it requires five tests for each sensitive attribute, we apply a Bonferroni correction to control the family-wise error rate at 5%, separately for each feature for each model. All other tests are performed without corrections.

**ABROCA** scores [44] measure the absolute difference in the area under the ROC curves for the baseline group and the sensitive group. It is defined as:

$$ABROCA = \int_{0}^{1} |ROC_{A=0}(t) - ROC_{A=1}(t)|dt,$$

where $t$ represents the threshold. Since ROC curves evaluate the trade-off between true and false positive rates across groups, this ABROCA metric is conceptually similar to separation and shares its normative foundation: it allows the model to distinguish between groups if such distinctions are justified by the outcome, making it a merit-based fairness metric [45]. Smaller ABROCA scores indicate less disparity in model performance between groups, and thus more equitable predictions, with the minimum score being 0. Given that we only observe rankings for each study program and admission system, we compute study program-specific ABROCA scores. To consolidate these into a single metric, we compute a weighted average of ABROCA scores across all study programs within each admission system, using the number of admitted students in each program as weights.

## A.8 Counterfactual predictions

In this section, we describe the calculations underlying Figure SI 1. In Panel A, we calculate the correlation between the predicted likelihood of completion at the study program of admission and the counterfactual likelihoods of completion for a group of study programs in a given field. Let $p_{i,l,m}$ denote the predicted likelihood of completion of student $i$ at their observed study program $l$ in field $m$, and let $p_{i,j,k}$ denote her counterfactual predicted completion likelihood at study program $j$ in field $k$. The data consists of pairs $(p_{i,l,m}, p_{i,j,k})$ for all $j \neq l$ and $k \neq m$, from which we calculate the correlation coefficients.

In panels B through E, we run the following regressions, respectively:

$$p_{i,j,k} = \alpha + \gamma_i + \epsilon_{i,j,k}, \qquad (1)$$

$$p_{i,j,k} = \alpha + \delta_j + \epsilon_{i,j,k}, \qquad (2)$$

$$p_{i,j,k} = \alpha + \gamma_i + \delta_j + \epsilon_{i,j,k}, \qquad (3)$$

$$p_{i,j,k} = \alpha + \beta \cdot \text{same\_field}_{i,j,k} + \gamma_i + \delta_j + \epsilon_{i,j,k}, \qquad (4)$$
where $p_{i,j,k}$ is student i's predicted likelihood of completion at study program j in field k, $\gamma_i$ and $\delta_j$ are individual and study program fixed effects, and same\_field$_{i,j,k}$ is an indicator for whether study program j is in the same field as the student's observed study program l. For each of these equations, we compute the residuals $\hat{e}_{i,j,k}$. A larger residual corresponds to a higher predicted likelihood of completion than that explained by the model's fixed effects. For example, in Equation (2), a positive residual means that the model predicts a higher likelihood of completion in the considered study program than expected based solely on the students' innate characteristics. The residuals are grouped by the observed field of admission l and counterfactual field of admission k, and the mean residuals are reported.

## A.9 Marginal Value of Public Funds Estimate

In this subsection, we explain how we estimate the Marginal Value of Public Funds (MVPF), a key metric in economic analysis used to assess the efficiency and effectiveness of public policies [49, 86]. Our analysis takes the recently enacted reduction in undergraduate capacity at Danish universities by 10.2% as a starting point[43]. We evaluate and compare two counterfactual scenarios: one in which student reductions are based on our predictive models, and the other using the current admissions assessment. The MVPF provides a quantitative measure of the additional value generated by allocating one more unit of public resources to a specific policy intervention or program. The MVPF is calculated as the ratio of public benefits to net government cost [86]:

$$MVPF = \frac{\text{Benefits}}{\text{Net Govt Cost}} = \frac{\Delta W}{\Delta E - \Delta C'} \quad (5)$$

where $\Delta W$ represents the benefits accrued by individuals affected by the policy, $\Delta E$ denotes the government's upfront expenditure, and $\Delta C$ is the long-run reduction in government costs.

We assume that students derive benefits from graduating, in line with the extensive literature on the returns to education, both in monetary terms [87–89] and non-monetary benefits [90], We assign students uniform weights, implying a welfare gain ($\Delta W > 0$) compared to the counterfactual policy, as fewer students fail to complete their education under our models.

We estimate the yearly long-run reduction in government costs ($\Delta C$) from the government revenue generated by the additional earnings of students who complete their degrees. For the return to higher education in Denmark, we rely on the findings of [89], who uses a propensity score matching method to calculate the returns for various educational levels. These returns reflect the difference between private income and government expenditure on education, compared to no higher education. In Denmark, returns to higher education vary by type, ranging from 2.4 million to 8.9 million DKK (2007 prices) per person. We conservatively use the lowest estimate of 2.4 million DKK. To estimate the increased government revenue from higher income, we calculate the rise in tax revenues from income and consumption taxes. We apply a marginal tax rate of 37.7%, which is the marginal tax rate for individuals receiving unemployment benefits, serving as a lower bound due to the progressive nature of the Danish tax system [91]. Additionally, we apply a consumption tax rate of 23%, as recommended by the Danish Ministry of Finance [92]. This results in a government revenue increase of 900,000 DKK (2007 prices) from income tax and 550,000 DKK (2007 prices) from consumption tax, for a total of 1.4 million DKK (2007 prices). Adjusted to 2016 prices using the consumer price index from Statistics Denmark's StatBank (Table PRIS8), this amounts to 1.6 million DKK or approxi-
<table><caption>**Table 1: Characteristics of contracted students across model and admission types.**
GPA, Human and Both refer to admission type. There are 3,470 contracted students admitted using GPA and 555 contracted admitted using human rankings, for a combined contraction of 4,025. All models use the *academic* input set.</caption><thead><tr><th></th><th>Transformer</th><th>Gradient boost.</th><th>Logistic reg.</th><th>LSTM</th><th>GPA / Human</th></tr></thead><tbody><tr><td><strong>Graduates</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>GPA</td><td>1464</td><td>1600</td><td>1585</td><td>1559</td><td>1923</td></tr><tr><td>Human</td><td>304</td><td>311</td><td>314</td><td>304</td><td>358</td></tr><tr><td>Both</td><td>1768</td><td>1911</td><td>1899</td><td>1863</td><td>2281</td></tr><tr><td><strong>Reduction in dropout</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>GPA</td><td>459</td><td>323</td><td>338</td><td>364</td><td>0</td></tr><tr><td>Human</td><td>54</td><td>47</td><td>44</td><td>54</td><td>0</td></tr><tr><td>Both</td><td>513</td><td>370</td><td>382</td><td>418</td><td>0</td></tr><tr><td><strong>Graduation rate</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>GPA</td><td>42.2</td><td>46.1</td><td>45.7</td><td>44.9</td><td>55.4</td></tr><tr><td>Human</td><td>54.8</td><td>56</td><td>56.6</td><td>54.8</td><td>64.5</td></tr><tr><td>Both</td><td>43.9</td><td>47.5</td><td>47.2</td><td>46.3</td><td>56.7</td></tr><tr><td><strong>%-point graduation rate difference</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>GPA</td><td>13.2</td><td>9.3</td><td>9.7</td><td>10.5</td><td>0</td></tr><tr><td>Human</td><td>9.7</td><td>8.5</td><td>7.9</td><td>9.7</td><td>0</td></tr><tr><td>Both</td><td>12.7</td><td>9.2</td><td>9.5</td><td>10.4</td><td>0</td></tr></tbody></table>

mately 230,000 USD (using a conversion rate of 1 USD to 7 DKK). For the model with the lowest performance, we observe an increase of 364 graduates (see Table 1).

Estimating the cost ($\Delta E$) associated with implementing and operating a prediction model for use in the centralized admission process is challenging. A possible benchmark comes from the cost of developing, implementing, and operating an algorithmic placement model in U.S. colleges, as reported in [11]. These costs ranged from 196,170 to 268,890 USD across six different colleges (2016 prices). The primary drivers of these costs were substantial fixed costs related to developing the algorithm and data entry. Given that our model builds on readily available data, we expect lower data entry costs. However, complying with regulatory requirements, such as ensuring the “right to explanation”, could increase costs. For instance, human oversight—needed to override the automated system at times—might worsen screening performance. Evidence from the Danish admissions process suggests that human screening is less effective, leading to poorer outcomes [35]. Although this may technically reduce government revenue rather than directly increasing costs, we account for this as a cost increase for simplicity. Given the substantial societal benefit of each graduating student, worsened screening performance could contribute to a significant portion of ongoing expenses. This largely depends on how regulatory compliance, particularly human oversight, is implemented. To maintain a conservative estimate, we project that developing and implementing a centralized admission algorithm will cost around 1 million USD, with an additional 1 million USD for annual operational costs. To account for the cost impact of worsened screening performance due to human oversight overriding predictions, we estimate that an 18% reduction in graduates would occur, based on 18% of judges overriding predictions in bail decisions [69], where we instead use the baseline human or GPA admission rule as the performance. This is a very uncertain estimate, and further work would be needed to
quantify these costs more precisely. This would lead to an additional cost increase of 13.4 million USD for GPA-based screening (232 additional graduates) and 1.8 million USD for human evaluation (44 additional graduates), totaling 15.2 million USD across both admission types, where we estimate the cost as the predicted increase of graduates multiplied by the 18% overriding probability multiplied by the revenue of 230,000 USD per student.

Due to the recurring nature of both the revenues and costs, we calculate the net present value (NPV) of the income and costs associated with this policy, as explained in the next paragraph. Given the uncertainty around the cost estimates, we illustrate the net government revenue across different cost scenarios. This is shown for all admitted students in Figure 5, for those admitted using GPA in Figure 7, and for those admitted via human evaluation in Figure 8. Our analysis reveals that net government revenue remains positive over a wide range of cost assumptions, leading to negative government costs and thus infinite Marginal Values of Public Funds.

<figure><img src="image_8.png" /><figcaption>**Figure 7:** Net government revenue for different cost scenarios for GPA admission.
The figure presents different cost and development time scenarios in a heatmap format. Each heatmap corresponds to a different starting year for the admission procedure, leading to similar delays in revenue generation. For revenue, we use the lowest revenue estimate under algorithmic screening, which corresponds to gradient-boosted trees. The dotted line represents 0 revenue, while the black marker indicates our estimated cost.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 8:** Net government revenue for different cost scenarios for human evaluation admission.
The figure presents different cost and development time scenarios in a heatmap format. Each heatmap corresponds to a different starting year for the admission procedure, leading to similar delays in revenue generation. For revenue, we use the lowest revenue estimate under algorithmic screening, which corresponds to logistic regression. The dotted line represents 0 revenue, while the black marker indicates our estimated cost.</figcaption></figure>
**Net present value calculation.** To analyze the net present value of a series of revenues (or costs if negative) from year 0 onward under varying conditions, we use a combination of finite and infinite geometric series to discount future payments to their present value. For a finite geometric series $S_n = ar^0 + ar^1 + ... + ar^n$, the present value is calculated as

$$S_n = \sum_{k=0}^{n} ar^k = a \frac{1 - r^{n+1}}{1 - r}$$

where $a$ is the yearly revenue (or cost), $r$ is the discount rate, and $n$ is the number of periods. For an infinite geometric series, $S = ar^0 + ar^1 + ar^2 + ...$, the present value is calculated as

$$S = \sum_{k=0}^{\infty} ar^{k-1} = \frac{a}{1-r}$$

where $a$ and $r$ are as defined above. It is important to note that these series begin with an undiscounted term, which is not the case for the periods 36-70 and 71+ years. This necessitates an additional discounting of the sum.

We use the official discount rates from the Danish Ministry of Finance, with the discount factor being $R_1 = 1 - 0.035$ for years 1 through 35, $R_2 = 1 - 0.025$ for years 36 through 70, and $R_3 = 1 - 0.015$ for year 71 and beyond.

For the first period (years 0-35), the present value is calculated using the finite geometric series formula:

$$PV_{0-35}(a) = a \frac{1 - (R_1)^{36}}{1 - R_1}.$$

For the second period (years 36-70), the present value is also calculated using a finite geometric series formula and then discounting it back to year 0:

$$PV_{36-70}(a) = a \frac{1 - (R_2)^{35}}{1 - R_2} \times R_2 \times R_1^{35}.$$

For the third period (years 71 and beyond), the present value is calculated using an infinite geometric series formula, discounted to the present value at year 0:

$$PV_{71+}(a) = \frac{a}{1 - R_3} \times R_3 \times (R_2)^{35} \times (R_1)^{35}.$$

The total present value is the sum of the present values of the three periods:

$$PV(a) = PV_{0-35}(a) + PV_{36-70}(a) + PV_{71+}(a).$$

In some scenarios, revenues may not start in year 0. To account for this, we exclude the revenue for the first $k$ years and calculate the present value for the first period as:

$$PV_{k-35}(a, k) = a \frac{1 - (R_1)^{36-k}}{1 - R_1} \times (R_1)^k,$$

where $k$ indicates the year in which revenues start. The total adjusted present value is:

$$PV^{\text{adjusted}}(a, k) = PV_{k-35}(a, k) + PV_{36-70}(a) + PV_{71+}(a)$$
We decompose revenues into fixed and variable (yearly) revenues (or costs). We assume that a fixed cost occurs in year 0, while variable costs start in year 0 as well. Variable revenues only start after k development years. The net government revenue is calculated as:

$$ \text{Net Govt Cost} = \text{PV}^{\text{adjusted}}(\text{Rev}, \text{DevTime}) - C_{\text{Fixed}} - \text{PV}(C_{\text{Var}}), $$

Where Rev is the revenue, CVar is the yearly variable cost, CFixed is the fixed costs, and DevTime is the development time (in years) before revenue starts.
# SI 1 Supplementary Information

## List of figures in supplementary information

<table><tr><td>SI 1 Correlation between counterfactual predictions and residuals from prediction compositions</td><td>38</td></tr><tr><td>SI 2 Correlation between predicted completion and first-year GPA</td><td>39</td></tr><tr><td>SI 3 Study program contraction curves across models and student populations</td><td>40</td></tr><tr><td>SI 4 Overall contraction curves across input sets and student populations</td><td>40</td></tr><tr><td>SI 5 Field-specific study program contraction curves</td><td>41</td></tr><tr><td>SI 6 Weighted standard error of ABROCA scores across models and current admission systems</td><td>42</td></tr><tr><td>SI 7 ABROCA scores across models and current admission systems</td><td>43</td></tr><tr><td>SI 8 Fairness results from z-tests for all models</td><td>44</td></tr></table>

## List of tables in supplementary information

<table><tr><td>SI 1 Rates for full-year equivalents and completion bonus</td><td>37</td></tr><tr><td>SI 2 AUC scores across models</td><td>37</td></tr><tr><td>SI 3 Mean AUC scores across different initializations</td><td>37</td></tr><tr><td>SI 4 Variables present in sequence model variants</td><td>45</td></tr><tr><td>SI 5 Model information for sequence models</td><td>45</td></tr><tr><td>SI 6 Variables present in baseline model variants</td><td>46</td></tr><tr><td>SI 7 Hyperparameter search space for baseline models</td><td>46</td></tr><tr><td>SI 8 Distribution of students across educational fields</td><td>47</td></tr><tr><td>SI 9 Distribution of course aggregate measures</td><td>47</td></tr><tr><td>SI 10 Number of unique values per categorical feature</td><td>48</td></tr><tr><td>SI 11 Summary statistics of sociodemographic and education information</td><td>48</td></tr></table>

## SI 1.1 Study Program Revenue Calculation

In this section, we briefly describe how we arrive at the estimate of 21,000 to 46,000 USD per bachelor's student. In Denmark, the majority of university funding is allocated through education taximeters, which provide funding based on the number of student full-year equivalents. A full-year equivalent represents the successful completion of exam activities corresponding to one year of standardized study time (60 ECTS credits). Therefore, a three-year bachelor's program corresponds to three rates. Additionally, universities receive a completion bonus for students who complete their studies within a specified timeframe, typically within one year beyond the standard duration for bachelor's degrees and three years for master's degrees. The taximeter rates and completion bonuses vary depending on the type of program, broadly categorized into humanities, mathematics, natural sciences, and others. The 2017 rates are outlined in Table SI 1 from [93].

These mechanisms lead to payouts ranging from 153,000 to 327,000 DKK per three-year bachelor student who completes their program on time. When converted at an exchange rate of 1 USD to 7 DKK, we arrive at an approximate range of 21,000 to 46,000 USD per student.
<table><caption>**Table SI 1: Rates for full-year equivalents and completion bonus.** Rates are given for our test year of 2017. All amounts are shown in DKK.</caption><thead><tr><th>Area</th><th>Rate per full-year equivalent</th><th>Bachelor completion bonus</th></tr></thead><tbody><tr><td>Humanities and others</td><td>44.000</td><td>21.000</td></tr><tr><td>Mathematics and others</td><td>63.200</td><td>34.100</td></tr><tr><td>Natural sciences and others</td><td>92.400</td><td>49.900</td></tr></tbody></table>

## SI 1.2 Additional Results

<table><caption>**Table SI 2: AUC scores across model types and input sets.** AUC scores are calculated on held-out data from the year 2017, with standard errors (in parentheses) calculated using [94]. The features included in each transformer are described in Table SI 4, while the features for logistic regression and XGBoost are detailed in Table SI 6.</caption><thead><tr><th>Model type</th><th>GPA baseline</th><th>Academic</th><th>Application</th><th>Human assessment</th><th>Socio-demographic</th><th>Everything</th></tr></thead><tbody><tr><td>LSTM</td><td>63.16 (0.27)</td><td>67.44 (0.26)</td><td>67.97 (0.26)</td><td>67.47 (0.26)</td><td>67.14 (0.27)</td><td>68.27 (0.26)</td></tr><tr><td>Gradient boost.</td><td>64.59 (0.27)</td><td>68.34 (0.26)</td><td>69.19 (0.26)</td><td>68.77 (0.26)</td><td>68.85 (0.26)</td><td>69.14 (0.26)</td></tr><tr><td>Logistic reg.</td><td>64.40 (0.27)</td><td>68.46 (0.26)</td><td>69.28 (0.26)</td><td>68.81 (0.26)</td><td>68.60 (0.26)</td><td>69.57 (0.26)</td></tr><tr><td>Transformer</td><td>63.76 (0.27)</td><td>69.64 (0.26)</td><td>70.46 (0.26)</td><td>69.79 (0.26)</td><td>68.89 (0.26)</td><td>70.29 (0.26)</td></tr></tbody></table>

<table><caption>**Table SI 3: Mean AUC scores across different initializations.** Mean AUC scores are calculated on held-out data from the year 2017 for each random seed, with standard errors reported in parentheses. The features included in each transformer are described in Table SI 4, while the features for logistic regression and XGBoost are detailed in Table SI 6.</caption><thead><tr><th>Model type</th><th>GPA baseline</th><th>Academic</th><th>Application</th><th>Human assessment</th><th>Socio-demographic</th><th>Everything</th></tr></thead><tbody><tr><td>LSTM</td><td>63.32 (0.006)</td><td>67.28 (0.081)</td><td>68.11 (0.226)</td><td>67.77 (0.136)</td><td>66.90 (0.095)</td><td>68.30 (0.123)</td></tr><tr><td>Gradient boost.</td><td>64.53 (0.046)</td><td>68.74 (0.183)</td><td>69.65 (0.158)</td><td>68.99 (0.184)</td><td>69.30 (0.283)</td><td>70.26 (0.250)</td></tr><tr><td>Logistic reg.</td><td>64.41 (0.006)</td><td>68.46 (0.007)</td><td>69.30 (0.004)</td><td>68.82 (0.012)</td><td>68.61 (0.008)</td><td>69.58 (0.004)</td></tr><tr><td>Transformer</td><td>63.73 (0.051)</td><td>69.39 (0.051)</td><td>70.69 (0.037)</td><td>69.89 (0.051)</td><td>69.05 (0.086)</td><td>70.37 (0.088)</td></tr></tbody></table>
<figure><img src="image_10.png" /><figcaption>**Figure SI 1: Correlation between counterfactual predictions and residuals from prediction decompositions.** For all heatmaps, the horizontal axis represents the actual field of admission and the vertical axis represents the counterfactual field of admission. Panel A presents correlations between actual predictions and counterfactual predictions. Panels B-E display residuals from the decomposition of predicted completion scores as a function of co-variates. In Panel B, we control for student ability. In Panel C, we control for counterfactual study program of admission. In Panel D, we control for both student ability and counterfactual study program of admission. In Panel E, we further control for whether the student attended a program in the same field as the counterfactual study program.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Figure SI 2: Correlations for predicted completion and first-year GPA.** This figure shows the correlations between the predicted likelihood of completing a program and the actual first-year GPA, computed for students with available numerical grades. Panel A shows the overall correlations for students admitted via GPA rankings. Panel B shows the overall correlations for students admitted via human rankings. In both panels, the correlation at the top is between high school GPA and first-year GPA. Panel C displays the within-study program rank correlations for students admitted through human rankings, with the top correlation representing human ranking. Predicted ranks are computed within each study program for each method, allowing us to compare the model rankings with human rankings, even when overall rankings are not observed.</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>**Figure SI 3: Study program contraction curves across models and student populations.** Students' completion probabilities within each study program are binned in ascending order, and mean completion rates are computed for each bin. The top row presents results for students admitted through GPA-based admissions, while the bottom row displays results for those admitted via human assessment. Perfect ranking would occur if completion rates started at 0% and then abruptly jumped to 100%. Dotted lines represent the mean completion rate for each sample.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Figure SI 4: Overall contraction curves across models and student populations.** Students' program completion probabilities are binned in ascending order, and mean completion rates are computed for each bin. The top row presents results for students admitted through GPA-based admissions, while the bottom row displays results for those admitted via human assessment. Each column corresponds to a different input set. Perfect ranking would occur if completion rates started at 0% and then abruptly jumped to 100%. Dotted lines represent the mean completion rate for each sample.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>**Figure SI 5: Field-specific study program contraction curves.** Students' completion probabilities within each study program are binned in ascending order, with mean completion rates computed for each bin. This analysis is conducted for students admitted through GPA-based admissions using the *academic* input set. Panels correspond to different fields, sorted from largest to smallest intake: (A) Business, administration and law, (B) Health and welfare, (C) Engineering, manufacturing and construction, (D) Arts and humanities, (E) Social sciences, journalism and information, (F) Natural sciences, mathematics and statistics, (G) Education (H) Information and communication technologies (ICTs), (I) Services, (J) Agriculture, forestry, fisheries, and veterinary.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>**Figure SI 6: Weighted standard error of ABROCA scores across models and current admission systems.** Fairness measures are calculated using the integral described in Section A.7. A weighted standard error of the Absolute Between-ROC Area (ABROCA) score, weighted by student intake across all institutions, is reported. Panel A presents the results for students admitted through GPA-based admission, while Panel B shows the results for those admitted through human assessment. The sensitive attributes are whether a student is first- or second-generation immigrant (*Native*), sex (*Female*), and whether the student is above or below median socioeconomic status (*High SES*).</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>**Figure SI 7: ABROCA scores across models and current admission systems.** Fairness measures are calculated using the integral described in Section A.7. An average Absolute Between-ROC Area (ABROCA) score weighted by student intake across all institutions is reported. Panel A presents the results for students admitted through GPA-based admission, while Panel B shows the results for those admitted through human assessment. The sensitive attributes are whether a student is native (*Native*, not a first- or second-generation immigrant), sex (*Female*), and whether the student is above or below median socioeconomic status (*SES*).</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>**Figure SI 8: Fairness results from z-tests for all models.** The fairness tests reported are the z-tests described in Section A.7. Panel A tests for binary independence, Panel B for equal false positive rates, Panel C for equal true positive rates, and Panel D for sufficiency. A significant positive (negative) test indicates a significantly lower mean in the reference (non-reference) group. The sensitive attributes are whether a student is a native of Danish origin (DO: Danish origin), sex (Fem: Female), and whether the student is above or below median socioeconomic status (SES: High SES)</figcaption></figure>
## SI 1.3 Transformer Variables

<table><thead><tr><th>Variable</th><th>Type</th><th>Baseline</th><th>Academic</th><th>Application</th><th>Human assessment</th><th>Socio-demographic</th><th>Everything</th></tr></thead><tbody><tr><td><strong>Temporal</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Relative year</td><td>Context</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td><strong>Courses</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Grade</td><td>Event</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course type</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course level</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Test type</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Type of education</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Study line</td><td>Context</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Primary or high school</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Institutional ID</td><td>Context</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td><strong>Higher education</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>GPA<sup>a</sup></td><td>Context</td><td>✓<sup>d</sup></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Place of enrollment, ID</td><td>Event</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Place of enrollment, ISCED broad group</td><td>Context</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Place(s) of application, ID</td><td>Event</td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>Place(s) of application, ISCED broad group</td><td>Context</td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>Place(s) of application, rank</td><td>Context</td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>Applied through Quota 2</td><td>Context</td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>Enrolled<sup>b</sup></td><td>Context</td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>Quota 2 ranking, decile</td><td>Context</td><td></td><td></td><td></td><td>✓</td><td></td><td>✓</td></tr><tr><td>GPA cut-off in previous year</td><td>Context</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td><strong>Demographic</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Age</td><td>Context</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Female</td><td>Event</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Danish origin</td><td>Event</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td><strong>Sociodemography of parents<sup>d</sup></strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Income</td><td>Event</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Wealth</td><td>Event</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Education, ISCED code</td><td>Event</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Education, months</td><td>Event</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr></tbody></table>

**Table SI 4: Variables present in sequence model variants.** The variables are grouped according to the type of information and further classified into either an *event* (i.e., creating a new event in sequence) or *context* (i.e., providing further context to an event). The ISCED broad groups of education follow the ISCED-F 2013 classification, and ISCED codes for education follow the ISCED-2011 classification [95]. Footnotes: a) GPA is calculated based on high school grades but is included in the higher education events as context, not as a course event (hence the placement). b) Due to the inclusion of full application information, a variable is introduced to encode the place of enrollment. This variable is unnecessary when only the place of enrollment is included in the sequence. c) Parental information is recorded for each parent. d) GPA is included as an event in the baseline model.

<table><thead><tr><th>Model</th><th>LSTM # Param.</th><th>Transformer # Param.</th><th>Sequence length</th><th># Channels</th><th>Vocab size</th></tr></thead><tbody><tr><td>Academic</td><td>0.5M</td><td>26.1M</td><td>78</td><td>13</td><td>1773</td></tr><tr><td>Human</td><td>0.5M</td><td>26.1M</td><td>78</td><td>14</td><td>1783</td></tr><tr><td>Application</td><td>0.5M</td><td>26.2M</td><td>80</td><td>16</td><td>2003</td></tr><tr><td>Sociodemography</td><td>0.8M</td><td>27.5M</td><td>88</td><td>17</td><td>4529</td></tr><tr><td>Everything</td><td>0.9M</td><td>27.7M</td><td>90</td><td>21</td><td>4778</td></tr><tr><td>Baseline</td><td>0.3M</td><td>25.5M</td><td>3</td><td>1</td><td>528</td></tr></tbody></table>

**Table SI 5: Model information for sequence models.** # Parameters include all trainable model parameters. # Channels includes the channel containing only the [CLS] token.
## SI 1.4 Baseline Models

<table><caption>**Table SI 6: Variables present in baseline model variants.** Ordinal variables are one-hot encoded in the logistic regression but are included as a single ordinal variable in XGBoost. Parental information is included for each parent. Study program, type of education, and institutional ID of courses are inferred by identifying the most commonly associated type and program for all secondary school grades received. The ISCED broad groups of education follow the ISCED-F 2013 classification and ISCED codes for education follow the ISCED-2011 classification [95].</caption><thead><tr><th>Variables</th><th>Type</th><th>Baseline</th><th>Academic</th><th>Application</th><th>Human assessment</th><th>Socio-demographic</th><th>Everything</th></tr></thead><tbody><tr><td><strong>Courses</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>GPA</td><td>Continuous</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course grade avg.</td><td>Continuous</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course field grade avg.</td><td>Continuous</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course field grade std.dev.</td><td>Continuous</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Course field number of grades</td><td>Continuous</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Type of education</td><td>Nominal</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Study line</td><td>Nominal</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Study program ID</td><td>Nominal</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td><strong>Higher education</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Place of enrollment, ID</td><td>Nominal</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Place of enrollment, ISCED broad group</td><td>Ordinal</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Place of enrollment, rank</td><td>Ordinal</td><td></td><td>✓</td><td></td><td></td><td>✓</td><td></td></tr><tr><td>Applied through Quota 2</td><td>Binary</td><td></td><td>✓</td><td></td><td></td><td></td><td>✓</td></tr><tr><td>Quota 2 ranking, decile</td><td>Ordinal</td><td></td><td></td><td></td><td>✓</td><td></td><td>✓</td></tr><tr><td>GPA cut-off in previous year</td><td>Continuous</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td><strong>Demographic</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Age</td><td>Continuous</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Female</td><td>Binary</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Danish origin</td><td>Binary</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td><strong>Sociodemography of parents*</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Income</td><td>Continuous</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Wealth</td><td>Continuous</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Nominal</td><td>Nominal</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Continuous</td><td>Continuous</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td></tr></tbody></table>

<table><caption>**Table SI 7: Hyperparameter search space for baseline models.** Distributions are specified as in `scipy.stats`.</caption><thead><tr><th>Parameter</th><th>Distribution</th></tr></thead><tbody><tr><td><strong>XGBoost</strong></td><td></td></tr><tr><td>learning_rate</td><td>Uniform(0.01, 0.99)</td></tr><tr><td>max_depth</td><td>RandInt(2, 13)</td></tr><tr><td>subsample</td><td>Uniform(0.01, 0.99)</td></tr><tr><td>colsample_bytree</td><td>Uniform(0.01, 0.99)</td></tr><tr><td>lambda</td><td>Uniform(1e-9, 100)</td></tr><tr><td>alpha</td><td>Uniform(1e-9, 100)</td></tr><tr><td>n_estimators</td><td>RandInt(50, 5001)</td></tr><tr><td><strong>Logistic regression</strong></td><td></td></tr><tr><td>C</td><td>LogUniform(1e-6, 1e6)</td></tr><tr><td>penalty</td><td>{None, 12, 11, elasticnet}</td></tr><tr><td>l1_ratio</td><td>Uniform(0.01, 0.99)</td></tr></tbody></table>
## SI 1.5 Summary Statistics

<table><caption>Table SI 8: Distribution of students across educational fields</caption><thead><tr><th rowspan="2">Educational field</th><th>2006-2016</th><th>2017</th></tr><tr><th>Count</th><th>Count</th></tr></thead><tbody><tr><td>Health and welfare</td><td>92,379</td><td>12,001</td></tr><tr><td>Business, administration and law</td><td>76,757</td><td>9,961</td></tr><tr><td>Arts and humanities</td><td>57,657</td><td>5,872</td></tr><tr><td>Social sciences, journalism and information</td><td>41,027</td><td>5,611</td></tr><tr><td>Engineering, manufacturing and construction</td><td>37,821</td><td>4,577</td></tr><tr><td>Natural sciences, mathematics and statistics</td><td>24,478</td><td>2,923</td></tr><tr><td>Education</td><td>22,860</td><td>2,662</td></tr><tr><td>Information and communication technologies (ICTs)</td><td>13,987</td><td>2,265</td></tr><tr><td>Services</td><td>6,147</td><td>1,036</td></tr><tr><td>Agriculture, forestry, fisheries and veterinary</td><td>3,538</td><td>480</td></tr><tr><td>Generic programmes and qualifications</td><td>151</td><td>16</td></tr></tbody><tfoot><tr><td>Total</td><td>376,802</td><td>47,404</td></tr></tfoot></table>

<table><caption>Table SI 9: Distribution of course aggregate measures. Means, standard deviations, and counts are calculated for each student separately.</caption><thead><tr><th rowspan="2"></th><th colspan="3">2006-2016</th><th colspan="3">2017</th></tr><tr><th>Count</th><th>Mean</th><th>S.D.</th><th>Count</th><th>Mean</th><th>S.D.</th></tr></thead><tbody><tr><td><strong>Means</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Primary school, languages</td><td>376,782</td><td>6.54</td><td>2.33</td><td>47,397</td><td>6.78</td><td>2.44</td></tr><tr><td>Primary school, other</td><td>376,793</td><td>7.26</td><td>2.14</td><td>47,403</td><td>7.48</td><td>2.31</td></tr><tr><td>Primary school, STEM</td><td>376,746</td><td>6.51</td><td>2.7</td><td>47,404</td><td>6.72</td><td>2.82</td></tr><tr><td>High school, languages</td><td>376,668</td><td>7.35</td><td>1.68</td><td>47,392</td><td>7.48</td><td>1.98</td></tr><tr><td>High school, other</td><td>362,136</td><td>8.09</td><td>2.31</td><td>46,914</td><td>7.83</td><td>2.31</td></tr><tr><td>High school, STEM</td><td>376,456</td><td>7.43</td><td>1.97</td><td>47,349</td><td>7.54</td><td>2.31</td></tr><tr><td><strong>Standard deviation</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Primary school, languages</td><td>376,767</td><td>2.25</td><td>0.8</td><td>47,393</td><td>2.28</td><td>0.82</td></tr><tr><td>Primary school, other</td><td>376,308</td><td>2.23</td><td>0.72</td><td>47,395</td><td>2.26</td><td>0.76</td></tr><tr><td>Primary school, STEM</td><td>370,302</td><td>2.19</td><td>0.93</td><td>46,335</td><td>2.2</td><td>0.98</td></tr><tr><td>High school, languages</td><td>376,641</td><td>2.01</td><td>0.5</td><td>47,386</td><td>2.09</td><td>0.53</td></tr><tr><td>High school, other</td><td>258,695</td><td>1.79</td><td>1.09</td><td>44,846</td><td>1.88</td><td>0.97</td></tr><tr><td>High school, STEM</td><td>376,392</td><td>1.77</td><td>0.59</td><td>47,345</td><td>1.83</td><td>0.58</td></tr><tr><td><strong>Counts</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Primary school, languages</td><td>376,782</td><td>7.51</td><td>2.41</td><td>47,397</td><td>7.23</td><td>2.06</td></tr><tr><td>Primary school, other</td><td>376,793</td><td>10.52</td><td>3.78</td><td>47,403</td><td>10.27</td><td>3.81</td></tr><tr><td>Primary school, STEM</td><td>376,746</td><td>7.17</td><td>3.41</td><td>47,404</td><td>6.75</td><td>3.15</td></tr><tr><td>High school, languages</td><td>376,668</td><td>18.56</td><td>6.2</td><td>47,392</td><td>18.77</td><td>5.79</td></tr><tr><td>High school, other</td><td>362,136</td><td>3.3</td><td>1.87</td><td>46,914</td><td>4.61</td><td>1.25</td></tr><tr><td>High school, STEM</td><td>376,456</td><td>11.35</td><td>3.1</td><td>47,349</td><td>11.87</td><td>2.77</td></tr></tbody></table>
<table><caption>Table SI 10: Number of unique values per categorical feature</caption><thead><tr><th></th><th>2006-2016</th><th>2017</th></tr></thead><tbody><tr><td><strong>Tertiary school information</strong></td><td></td><td></td></tr><tr><td>Study program</td><td>1,173</td><td>842</td></tr><tr><td>Education field</td><td>11</td><td>11</td></tr><tr><td><strong>Primary and secondary school information</strong></td><td></td><td></td></tr><tr><td>Types</td><td>9</td><td>9</td></tr><tr><td>High school specialization</td><td>2,234</td><td>815</td></tr><tr><td>Institutions</td><td>2,484</td><td>316</td></tr><tr><td><strong>Course information</strong></td><td></td><td></td></tr><tr><td>Course</td><td>139</td><td>96</td></tr><tr><td>Course with level</td><td>272</td><td>145</td></tr><tr><td>Course with level & test type</td><td>649</td><td>312</td></tr></tbody></table>

<table><caption>Table SI 11: Summary statistics of sociodemographic and education information</caption><thead><tr><th rowspan="2"></th><th colspan="3">2006-2016</th><th colspan="3">2017</th></tr><tr><th>Count</th><th>Mean</th><th>S.D.</th><th>Count</th><th>Mean</th><th>S.D.</th></tr></thead><tbody><tr><td><strong>Educational</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Graduated</td><td>376,802</td><td>0.69</td><td>0.46</td><td>47,404</td><td>0.68</td><td>0.47</td></tr><tr><td>GPA</td><td>376,136</td><td>7.08</td><td>2.28</td><td>47,345</td><td>7.22</td><td>2.33</td></tr><tr><td>Acceptance priority</td><td>376,802</td><td>1.47</td><td>4.26</td><td>47,404</td><td>1.30</td><td>0.82</td></tr><tr><td>Applied through quota 2</td><td>376,802</td><td>0.32</td><td>0.47</td><td>47,404</td><td>0.38</td><td>0.49</td></tr><tr><td>Accepted through quota 2</td><td>376,802</td><td>0.14</td><td>0.35</td><td>47,404</td><td>0.19</td><td>0.39</td></tr><tr><td>GPA cutoff previous year</td><td>342,125</td><td>4.22</td><td>2.06</td><td>44,822</td><td>4.66</td><td>2.20</td></tr><tr><td><strong>Demographic</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Age</td><td>371,717</td><td>20.35</td><td>1.88</td><td>46,760</td><td>20.99</td><td>2.40</td></tr><tr><td>Female</td><td>376,802</td><td>0.58</td><td>0.49</td><td>47,404</td><td>0.56</td><td>0.50</td></tr><tr><td>Danish origin</td><td>376,802</td><td>0.91</td><td>0.29</td><td>47,404</td><td>0.89</td><td>0.32</td></tr><tr><td>Income, mom</td><td>371,152</td><td>257,431</td><td>268,714</td><td>46,562</td><td>287,183</td><td>178,392</td></tr><tr><td>Income, dad</td><td>358,664</td><td>340,308</td><td>798,540</td><td>44,892</td><td>389,321</td><td>724,069</td></tr><tr><td>Wealth, mom</td><td>371,152</td><td>412,558</td><td>3,787,772</td><td>46,562</td><td>360,095</td><td>2,462,602</td></tr><tr><td>Wealth, dad</td><td>358,664</td><td>779,645</td><td>4,054,591</td><td>44,892</td><td>658,163</td><td>3,491,393</td></tr><tr><td>Education length, mom</td><td>372,049</td><td>174.28</td><td>31.16</td><td>46,697</td><td>174.27</td><td>31.92</td></tr><tr><td>Education length, dad</td><td>365,541</td><td>174.57</td><td>31.68</td><td>45,961</td><td>173.89</td><td>31.16</td></tr></tbody></table>