## 4 Case Study
### The Dutch Childcare Benefits Scandal
Many experienced debt, unemployment, evictions, mental health issues, and even family breakdowns due to the false fraud accusations and the subsequent financial burdens placed upon them. * Political and institutional response: The Dutch Government's admission that institutional racism of the Dutch Tax and Customs Administration was the root cause of the scandal marked a pivotal moment in acknowledging the serious issues within the ADM systems used. It highlighted the need for systemic change to prevent such injustices from occurring in the future. * Calls for reform: Organizations like Amnesty International have emphasized the scandal as a wake-up call, urging for an immediate ban on the use of discriminatory data of nationality and ethnicity in risk-scoring for law enforcement and public service delivery. They advocate for a framework that ensures ADM systems are transparent and accountable, and they safeguard human rights. The report \"Xenophobic Machines: Discrimination Through Unregulated Use of Algorithms in the Dutch Childcare Benefits Scandal\" [48] by Amnesty International meticulously examines the flawed application of algorithmic decision-making by the Dutch tax authorities, which precipitated a significant discrimination issue and led to the Dutch Childcare Benefits Scandal. The report discusses the growing deployment of algorithmic systems by governments for automating tasks, with a notable focus on fraud prevention. It particularly critiques the Dutch Childcare Benefits scheme, where the Dutch tax authorities implemented an algorithmic self-learning system, the \"risk classification model\", in 2013 to detect fraud. This system used various factors, including nationality, to assign risk scores to applications, unfairly flagging non-Dutch applicants for further scrutiny and accusations of fraud. Furthermore, the risk-detection algorithm scrutinized the social security documents submitted by parents for childcare allowances for any minor errors, leading to the automatic cessation of allowances and the demand for repayment of previously received aid, without applying the principle of proportionality. In addition, the responsible civil servants were given no information as to why the system had given the application a high-risk score, i.e., the ADM system was a black box. This process, particularly the emphasis on nationality as a significant factor for risk assessment, has raised concerns regarding discrimination and bias in the automated decision-making process. When assigned high-risk scores by the risk classification model, parents and caregivers were often required by government officials to submit additional documentation to validate their claims for benefits. However, attempts to query which aspects of their applications were flagged as inaccurate or to understand the nature of any missing documentation frequently led to a lack of response; the tax authorities systematically abstained from providing the necessary clarifications. Consequently, for an extended period, individuals implicated found it virtually impossible to contest their alleged misrepresentations. Moreover, acquiring detailed insights into the operation and foundation of the risk classification model proved to be a challenge not only for the affected families, but also for journalists, policymakers, regulatory entities, and members of civil society. It should also be noted that the Dutch Data Protection Authority fired the Dutch tax authorities EUR 2.75 million for unlawfully processing personal data, particularly dual nationality, and highlighted several GDPR violations due to the improper use of personal data over many years. The critical issues identified with the deployment and use of this ADM system can be summarized as follows:  
* Inefficiency in achieving the objectives of the deployment and use. * Discrimination-racial profiling, inadequate data source, and quality validation. * Black box systems, self-learning algorithms in the public sector, and lack of human supervision in the decision-making process.