AI Risk Assessment Tools: A judge’s best friend (?)Α 12-minute read
Eleftherios Chelioudakis 11

Independent Researcher, currently candidate at the Advanced M .Sc. Digital Humanities, Department of Computer Science, KU Leuvenchelioudakis.ele@protonmail.com

Abstract. In this short paper, the writer deals with the use of Artificial Intelligence Risk Assessment Tools (AI RATs) in the context of the criminal justicesystem. He describes the different stages of an AI RAT’s life cycle and the elements in each of these stages that require special attention. Finally, he concludesthat there is the need to carefully regulate, design, study, and implement the AI
RATs because the possibility of them to pose a significant danger to our democracy and freedom is particularly high.
Keywords: Risk Assessment Tools, Judiciary, Artificial Intelligence, AI Regulability, Inclusion, Non-discrimination, Fairness, Data quality.

1Introduction

Table 1. John G. Roberts, Jr., Chief Justice of the United States replies to the question of theRensselaer Polytechnic Institute President Shirley Ann Jackson during their conversation atRensselaer, New York on Tuesday, April 11, 2017 [1]- “Can you foresee a day when smart machines, driven with artificial intelligences,will assist with courtroom fact-finding or, more controversially even, judicial decision-making?”- “Well, it’s a day that’s here, and it's putting a significant strain on how the judiciary goes about doing things.”
As the quotes in the table above suggest, this short paper deals with the use of Artificial
Intelligence (AI) in the judicial system. More than 200 AI Risk Assessment Tools (AIRATs) are used around the world at different settings related to the criminal justicesystem [2,3]. AI RATs incorporate machine learning algorithms, which generate risk
models based on huge volumes of data. They are available at almost every juncture of
the criminal justice system, from pretrial risk assessment to prison rehabilitation programs, sentencing proceedings, and parole [4]. In simple terms, AI RATs analyze risks
factors and produce a risk assessment classification for an individual. This risk assessment classification will then potentially influence the judiciary in the decision-making
process. A decision-making process that AI RATs should not put “a significant strainon” contrary to what John G. Roberts, Jr., Chief Justice of the United States admits.

Famous examples of AI RATs are the “Level of Service Inventory – Revised (LSIR)” used in Australia [5], Canada [6], and in the U.S.A. [7] and the “Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) used in the U.S.A.
[8, 9,10]. Both tools aim to assist the judiciary with sentencing procedures and to identify offenders’ risk with regard to recidivism.
Moreover, predictive justice is becoming popular to other parts of the world as well,such as Argentina [11], while international organizations such as the Council of Europe
push for a common framework of standards for the use of AI in a court setting [12].
Finally, the European Commission has also expressed its support on investments forthe use of AI tools in the field of public administration, including justice [13].
However, justice is not simply just another part of the public administration. Instead,justice has a special position in our societies. It empowers individuals to protect themselves against inequities, and it encompasses core human rights, such as the right to afair trial, and the right to an effective remedy. Therefore, justice constitutes a core element of our democracy, and matters related to our democracy are matters that should
concern every citizen.
Thus, in this short paper, the significant body of literature that exists as regards AI
RATs in the judicial system will be used, in order to describe the life cycle of an AI
RAT and the elements that require attention during this life cycle. The research goal inthis 12-minute read is to answer the question: “What are the different stages o f an AI
RAT’s life cycle, and what are the elements in each of these stages that require special
attention?”.

2The first stage: The regulability of the use of AI RATs

The first stage of the AI RAT’s life cycle is related to the regulability of the AI RAT.
In the following paragraphs of this chapter two themes will be briefly addressed: First,the question as to who is to regulate AI RATs, and second, the question as to wha t
modalities can be used to regulate AI RATs.
The question as to who is to regulate AI RATs is a question of legitimacy, since it
refers to who has the rightful entitlement to regulate AI RATs. Such a question is particularly important because private parties are becoming more and more dominant at
national and international level and their influence on individuals and governments isrising. Legitimacy is a notion closely related to the notions of legality, moral justifiability, and authority, but it is not identical to them. It is a matter of conformity to rules(regulatory aspect), justification in relation to moral norms and values (morally normative aspect), and consent or representation of those involved or affected (social aspect)[14]. The AI RATs examined in this paper are used in the justice system, and such anenvironment is completely independent and protected from private interests. Moreover,
the justice system of a State is interrelated with this State’s independence against bothother States and private entities. Therefore, solely national governments can be perceived to have the rightful entitlement to regulate AI RATs used in their justice system.
Regarding the modalities that can be used to regulate AI RATs, there is the need tonot lose sight of the focus of this paper, namely AI RATs used in the judicial system.

3In general, regulation is a choice among different modalities. Specifically, law, socialnorms, the market, and the architecture of things which enable or disable certain typesof behavior can be used for regulating. Moreover, the regulator is the one to decide
which mix of modalities to use in order to achieve the purpose pursued [15]. However,AI RATs used in the criminal justice system assist the judiciary to make decisions
which can lead to pretrial detention and/or criminal sanctions. Pretrial detention, criminal sanctions and anything else related to their imposition shall, according to the principle of legality, be provided for by law. Thus, the use of AI RATs as assistance to thejudiciary’s decisions should be based on an appropriate legal basis which provides fo r
specific safeguards.

3The second stage: The phase of design and research

The second stage of the AI RAT’s life cycle is related to the architecture of the AI RAT
and the research on the legal and ethical risks that arise from this technology. In thefollowing paragraphs of this chapter, will be described the elements that require special
attention, firstly about the architecture and secondly about the research on the legal andethical risks that arise.
When it comes to the architecture of the AI RAT, the first element that needs special
attention is inclusion in the decision-making process: Decisions on the architecture of
the AI RAT should be open to public discussion from the very first stage of the architecture’s development. AI RATs are used in justice, and justice is a keystone for ourdemocracies. Therefore, discussions about an AI RAT’s design should not take placebehind closed doors. Instead it is important for public authorities, civil society organizations, academics, and citizens in general to be able to actively participate in the decision-making process about this architecture. Inclusion creates also a sense of accountability, since when the meetings are not secret their discussions are open to scrutiny.
Finally, inclusion creates transparency and boosts trust, since when you can actively
participate, pose questions, express opinions, and receive clarifications, you feel involved and have confidence in the decisions agreed upon.
The second element that requires attention lies in the development of AI RATs’ architecture: states that are willing to use AI RATs, will need to decide, whether they willdevelop these tools by themselves, or they will purchase them from a private vendor[16]. If the States decide to utilize a private vendor’s option, special attention should begiven to trade secrets and Intellectual Property Rights. Since justice is the symbol offairness, accountability, and transparency, the AI RAT used for its service should beenforcing these principles as well [17]. Therefore, the architecture of the AI RAT shallnot be protected by trade secrets because this would interfere with the open discussion
process, as described above. Another element that requires attention if States decide toproceed with the private vendor’s option, is the access to the personal data used for therisk assessment. Such personal data could be criminal records, racial origin, ethnicity,
gender, information on family and personal relations, etc. Thus, it is important to beclarified in the agreement between the State and the private vendor developing the AIRAT, that the vendor shall not be authorized to have access to this data. Instead only

the judiciary, the relevant public authorities, the defendants and their lawyers should be
able to access the defendants’ personal data used in the risk assessment.
Now, as regards the research on the legal and ethical risks that arise, there are various
concerns to be examined. First, attention needs to be paid to the classification of therisk categories of an AI RAT. Using terms like “high risk” to classify individuals’scores on AI RATs can influence the decision-making process of the judiciary and leadto biased choices [18]. Furthermore, the potential of an AI RAT to perpetuate bias might
exist in its source code as well, and this is the second concern that requires attention[19]. The quality of the data is of utmost importance for this assessment, because if theAI RAT model is trained on biased data, it will be biased as well. Combating AI RATs’
biases, which are based on gender, racial origin, ethnicity, perceived or actual religion,political opinions, sexual preferences and other protected grounds against discrimination should be at the center of this research, in order to correct such faults and biases inthe AI RAT.
The correction of such biases and faults is interrelated with the third concern that
requires attention, namely the intelligibility of the AI RAT. Even though, intelligibilit yis hard to achieve, especially with more accurate AI RATs, like neural nets, it is a matterthat should be addressed, since it is linked to issues on transparency and fairness [20].
In short, if the AΙ RAT concludes that an individual constitutes a high -risk, and thereasoning behind this conclusion cannot be explained, then consequen tly the judiciary’s
decision based on such conclusion is neither transparent, nor fair. Fairness is the fourthconcern that demands attention. Just like intelligibility, the relation of fairness with accuracy is problematic, since maximizing accuracy and fairness at the same time appears
to be hard to achieve [21].
This list of concerns is by no means exhaustive, and researchers should be open toother concerns as well, taking into consideration recent technological developments,
studies done in related fields, lessons learned from AI RAT’s pilot projects, and conclusions reached throughout all the stages of the AI RAT’s life cycle.

4

The third stage: Implementing AI RATs

This third stage of the AI RAT’s life cycle is related to its implementation. In this phase
the AI RAT will be used not on a test set, or on a pilot project like in stage 2, but insteadin a real-world context affecting the lives of real people. In the following paragraphs
the necessary elements that should accompany the implementation of AI RATs in thecriminal justice system will be described.
To begin with, adequate training of the judiciary is of crucial importance. The judges
need to be able to recognize the risks which arise from the use of AI RATs, and have abasic understanding of the technology behind such tools. They need to feel comfortablewith this technology, so that they will be able to understand its limits. Moreover, thejudges should be able to use the AI RATs as a diagnostic tool that can help them tounderstand the underlying drivers of a criminal activity and not as a predictive tool [22].
Finally, involvement of the judiciary in stage 1 and stage 2 of the AI RATs life cycle is5

significant as well. Only then they will be able to follow the AI RATs development andupdate their knowledge on this matter.
Secondly, during this stage scrutiny over the AI RATs should be obligatory. Here, I
do not refer to scrutiny over the judiciary’s decisions, but instead scrutiny over the AI
RATs classifications and predictions. Specifically, there should be regular and repeatedevaluation of the AI RATs validity [23]. A national supervisory authority with independent character, adequate financial resources and a staff of high kno wledge and expertise, is the ideal body to be given this supervisory task. Such authority should baseits assessments on both quantitative data, such as statistics on the efficiency of the AI
RATs, and qualitative data, such as conclusions drawn in a case-by-case basis.
Last but not least, civil society organizations that have at the center of their interest
criminal justice should be wide awake. They must act as watchdogs in order to reassurethat civil rights are adequately protected. Moreover, they shou ld put pressure on governments and supervisory authorities to properly fulfill their duties.

5Conclusion

So, is an AI RAT a judge’s best friend? By examining the different stages of the AI
RAT’s life cycle and describing the elements that require special at tention in each ofthese stages, one thing is clearly understood: there is the need to carefully regulate,design, study, and implement such a tool. If the risks that arise from the use of AI RATs
in the criminal justice system are not resolved, they will pose a significant danger toour democracy and freedom. Finally, the life cycle of an AI RAT shall not be perceived
as a process with a beginning and an end, but instead as a continuing cycle of problem
analysis, whereby findings from one stage are fed into another.

References
1. Rensselaer Polytechnic Institute, (12.04.2017). A Conversation with Chief Justice John G.
Roberts,
Jr.
(10:33-10:53).
Retrievedfromhttps://www.youtube.com/channel/UCdH4jKoTt9Lr18RxrRqv4gQ (accessed Oct. 2018).
2. Douglas, T., Pugh, J., Singh, I., Savulescu, J. and Fazel, S.. Risk assessment tools in criminal
justice and forensic psychiatry: The need for better data. European Psychiatry, 42, pp.134137 (2017).
3. Singh JP, Desmarais SL, Hurducas C, Arbach-Lucioni K, Condemarin C, Dean K, et al.
International perspectives on the practical application of violence risk assessment: a global
survey of 44 countries. Int J Forensic M ental Health 2014;13:193–206.
4. Kehl, Danielle, Priscilla Guo, and Samuel Kessler. 2017. Algorithms in the Criminal Justice
System: Assessing the Use of Risk Assessments in Sentencing. Responsive Communities
Initiative, Berkman Klein Center for Internet & Society, Harvard Law School.
5. Watkins I., The utility of level of service inventory – Revised (LSI-R) Assessments within
NSW correctional environments, Corrective Services NSW, Corporate Research, Evaluationand Statistics, Reseach Bulletin No 29, January 2011.

6. Kehl et al., Algorithms in the Criminal Justice System: Assessing the Use of Risk Assessments in Sentencing, supra note 4.
7. Sex Offender Sentencing in Washington State: Predicting Recidivism Based on the LSI-R,Washington State Institute for Public Policy (2006), http:// www.wsipp.wa.gov/ReportFile/935/Wsipp_Predicting-Recidivism-Based-on-the-LSI-R_Predicting-RecidivismBased-on-the-LSI-R.pdf. (accessed Oct. 2018).
8. California Department of Corrections and Rehabilitation, Correctional Offender M anagement Profiling for Alternative Sanctions (COM PAS) Fact Sheet, April 2009.
9. Practitioner’s Guide to COM PAS Core, Northpointe (M ar. 19, 2015), https://assets.documentcloud.org/documents/2840784/Practitioner-s-Guide-to-COM PAS-Core. Pdf (accessedOct. 2018).
10. Algorithms in the Criminal Justice System, Electronic Privacy Information Center (n.d.),https://epic.org/ algorithmic-transparency/crim-justice/ (accessed Oct. 2018).
11. Gustavo Corvalán Juan, Artificial Intelligence ,threats, challenges and opportunities Prometea, the first predictive artificial intelligence at the service of justice is Argentinian, Revue
Internationale de droit des données et du numérique, Vol.4 (2018).
12. Council of Europe, Parliamentary Assembly (PACE), Recommendation 2102 (2017) 1,
Technological convergence, artificial intelligence and human rights.
13. European Commission, Communication from the Commission to the European Parliament,
the European Council, the Council, the European Economic and Social Committee and theCommittee of the Regions on Artificial Intelligence for Europe, 25.04.2018.
14. Vedder A.H., Towards a defensible conceptualization of the legitimacy of NGOs, January
2007.
15. Lessig L., Code and Other Laws of Cyberspace, 1999.
16. Barabas C., Bavitz C., Budish R., Dinakar K., Dwork C., Gasser U., Hessekiel K., Ito J.,
Rivest R.L., Virza M ., and Zittrain J., An Open Letter to the M embers of the M assachusetts
Legislature Regarding the Adoption of Actuarial Risk Assessment Tools in the CriminalJustice System, November 2017.
17. Levine D.S., Can We Trust Voting M achines?, October 2012. (accessed Oct. 2018).
18. Barabas et al., An Open Letter to the M embers of the M assachusetts Legislature Regarding
the Adoption of Actuarial Risk Assessment Tools in the Criminal Justice System, supra note
16.
19. Chouldechova A., Fair prediction with Disparate Impact: A Study of Bias in Recidivism
Prediction Instruments, February 2017.
20. Caruana R., Elhadad E., Gehrke J, Koch P., Lou Y., Sturm M ., Intelligible M odels for
HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission, August 2015.
21. Berk R., Heidari H., Jabbari S., Kearns M ., Roth A., Fairness in Criminal Justice Risk Assessments: The State of the Art, M arch 2017.
22. Barabas C., Dinakar K., Ito J., Virza M ., Zittrain J., Interventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment, Conference on Fairness, Accountability, and Transparency , Proceedings of M achine Learning Research 81:1–15, 2018.
23. The Council of State Governments Justice Center, Risk and Needs Assessment and Race inthe Criminal Justice System, M ay 2016 (accessed Oct. 2018).