Conversational Artificial Intelligence and the Potential for Epistemic Injustice

In their article, Sedlakova and Trachsel (2023) propose a holistic, ethical, and epistemic analysis of conversational artificial intelligence (CAI) in psychotherapeutic settings. They mainly describe interesting challenges regarding the ambiguous therapeutic relationship with its introduction and recommend “conceptual analysis together with phenomenological insights into patients’ experiences” (Sedlakova and Trachsel 2023, 11). To contribute to this important reflection, our considerations aim to show that in order to provide a holistic framework of ethical and epistemological issues brought about by CAI in psychotherapy, an analysis of their potential to cause epistemic injustice cannot be left out of the picture. We map out issues in terms of testimonial and hermeneutical injustice that can emerge in connection with CAI and that deserve further attention. The concept of epistemic injustice was introduced by the feminist philosopher Miranda Fricker (2007) in her landmark monograph Epistemic Injustice: Power and the Ethics of Knowing. With this book, she highlighted specific forms of injustice at the intersection of ethics and epistemology. According to Fricker’s definition, epistemic injustice is a “wrong done (to) someone specifically in their capacity as a knower” (2007, 1). She identifies two such patterns of wrongdoing as “testimonial injustice” and “hermeneutical injustice. ” Testimonial injustice takes place at the communication level when certain prejudices and stereotypes cause a hearer to assign a deflated level of credibility to a speaker’s testimony. Hermeneutical injustice precedes communication and occurs when a gap in collective interpretative resources puts a person or group at a disadvantage when trying to make sense of their social experiences. This is either due to a lack of concepts to express one’s experience (e.g., experiencing postpartum depression before this very concept was collectively available) or to a misalignment between a person’s experience and the existing socially accepted concepts (Mason 2021; Pozzi 2023a). The original Frickerian framework has been refined and elaborated in the context of medicine and healthcare (Kidd and Carel 2017) and also of psychiatric practice (Kidd et al. 2022; Sanati and Kyratsous 2015), although its role in medical AI has received less attention as many discussions on epistemic injustice only focus on human conversations (Pozzi 2023b). We contend that the algorithms used for CAI in psychotherapy could create the perfect storm for epistemic injustice. Because of that, there are good reasons to pause before CAI tools are adopted too widely or permanently in psychotherapy. Sedlakova and Trachsel (2023) argue that CAI easily can get “epistemic supremacy in the conversation because it can provide data and analysis of a scale that humans would not be able to” (10). It is not hard to imagine how testimonial injustice can occur when persons seek help for their mental health by engaging in communication with CAI. If we prioritize CAI over human dialogue, persons may gradually lose confidence in themselves as epistemic agents because they can get the feeling of not being heard, especially when the chatbot fails to pick up on the user’s exact meaning. The limit of the scope of algorithms and their epistemic opacity can create a mismatch between the tracked data or the app’s prediction of their experience and their actual experience (Symons and Alvarado 2022). One cannot understand what goes on, even though one knows there is something concealed in an overall hybrid structure that is difficult to espy. Users can thus experience an unjust deflation of their credibility owing to their perceived inferiority to an automated data analytics system. This can be the case, particularly if a human professional entering the