<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_2.png" /></figure>

# What is There to Criticize About Voice, Speech and Face Recognition and How to Structure the Critique?

Murat Karaboga✉️, Frank Ebbers✉️, Greta Runge, and Michael Friedewald✉

Fraunhofer Institute for Systems and Innovation Research ISI, Karlsruhe, Germany
{murat.karaboga,frank.ebbers,greta.runge,
michael.friedewald}@isi.fraunhofer.de

**Abstract.** In view of a multitude of rapidly spreading applications based on voice, speech, and facial recognition technologies, there is a danger of criticism becoming fragmented and narrowed down to a few aspects. The workshop developed critiques of three application areas and collected initial suggestions on how these critiques could be categorized across multiple application areas.

**Keywords:** Facial Recognition · Speech Recognition · Voice Recognition · disease recognition · technology assessment

## 1 Introduction

Voice, speech and facial recognition technologies are becoming increasingly widespread in a broad range of applications. Voice and speech recognition, for example, are widely used in smart speakers and increasingly in call centers and for marketing. Facial recognition has become the subject of much public debate, mainly due to its use by law enforcement agencies, but it is also being used in many other areas – often in combination with voice and speech recognition: for example, to detect physical and mental illnesses, to recognize emotions, and (in China) to analyze the attention of schoolchildren. The corresponding media debate and NGO statements mostly focus on the use of facial recognition by police forces and place the danger of real-time mass surveillance at the center of criticism. However, other uses, such as disease and emotion recognition pose significant risks to individuals and society, too. Against the backdrop of the heated public and academic debate surrounding such recognition technologies, the workshop had two goals: on the one hand, to show the diversity of possible criticisms on the basis of the three fields of application. On the other hand, to try to develop criteria for categorizing these criticisms and to explore whether it would be possible to find categories that would allow for structuring the criticisms across all fields of application.

Methodologically, we built on our relevant findings from a comprehensive study on the technical, legal, and societal challenges posed by voice, speech, and facial recognition technologies, which we prepared on behalf of the Swiss Foundation for Technology
Assessment [1]. Thus, the initial part of the workshop served to introduce participants to the technological foundations and to provide entry points into the critique of voice, speech, and facial recognition technologies. Then, the participants were briefly introduced to the three application areas: 1) Smart speakers, 2) disease detection, and 3) police mass surveillance, each of which served as a starting point for discussion in the three breakout groups. These groups first collected what was critical about the respective application area and then developed a possible structuring of the points of criticism. Participants in the breakout sessions had different disciplinary backgrounds (law, economics and social science). At the end of the workshop, all groups reconvened and the results were consolidated. The following sections summarize the results from this process.

## 2 Technical Fundamentals of Voice, Speech, and Facial Recognition Technologies

Both face and voice recognition are biometric techniques, since they use attributes of the human body. They are typically used for either identification or verification of a person. In identification, an image or voice sample is compared with many others from a database (1:N), for example, to determine the name of a person. Verification, on the other hand, compares an image with a reference picture in a database. Such a 1:1 comparison is typically used for authentication, i.e. to determine whether a person is actually who s/he claims to be. In principle, identification is the more difficult task because the comparison of one image or audio sample with a large number of other samples is a much more complicated computing process. With regard to performance, the decisive factor is whether the voice or face patterns found were recognized correctly and how well they match the voice or face patterns of other persons. Errors occur when parts of, for example, the face are recognized incorrectly due to poor camera resolution or the algorithms are unable to recognize the face [1]. The latter is also due to the input data quality with which the algorithms were trained. Such data bias has regularly led to the false recognition of people of colour [2].

It is also important to distinguish between speech recognition on the one hand and voice or speaker recognition on the other. The former involves the extraction of content and meaning from a statement, e.g., to enable the control of information systems by means of speech input. The latter refers to the aforementioned biometric procedure for recognizing persons, e.g., for authentication on a computer system [1].

In addition, there are also techniques to recognize patterns in images and audio samples other than a person's identity. Pattern recognition is used, for example, in the areas of disease or emotion recognition.<fnref n="1" /> In both application areas, collected voice or facial expression patterns are compared with characteristics that are only assigned to emotions in one area and diseases in the other. By means of AI techniques, such as neural networks, anomalies in the speech pattern (spectrogram) or changes in facial

<footnote n="1">¹ Due to the workshop's focus on disease and emotion recognition, other application areas are not considered here. However, pattern recognition is the technical principle underlying many similar applications, such as the supposed recognition of sexual preferences based on facial recognition or the assessment of a person's dangerousness based on the analysis of their body movements.</footnote>
expressions, e.g. in facial paralysis or anatomical changes due to genetic defects, can be detected and assigned. Similarly, facial movements can be interpreted as to what emotion they represent. In addition to the challenge of correctly recognizing voice, speech, or facial patterns already described above, disease and emotion recognition adds the fundamental challenge of matching these patterns to diseases and emotions. For example, a facial feature may be recognized correctly in itself, but whether something like the correct recognition of drooping corners of the mouth actually provides information about facial paralysis, represents the emotion “sad”, or simply represents something else, such as an individual anatomical feature without any relation to disease or emotion, is scientifically disputed [3].

# 3 Criticism of Voice, Speech and Facial Recognition and Ways to Structure the Criticism

The discussions in the three breakout sessions all followed the same structure: first, group members were given a brief introduction to the respective application field. Then, possible points of criticism were collected. The workshop organizers had also prepared a collection of criticisms that would be used, if necessary, to trigger the discussion. The final step was a discussion to explore ways in which the points of criticism could be systematized supported by a virtual card sorting exercise.

## 3.1 Use Case: Smart Speaker

At the beginning of the session, the following written input was presented to the participants by the facilitator:

> *Smart speakers can be found in more and more households. Because they are intended to function as personal assistants, they are generally placed in living rooms and thus at the center of many people's private lives.*

> It turned out that out of the 5 participants, only one had set up a smart speaker in his home. The other attendees were reluctant, because of the privacy and security risks they attribute to the speakers and in particular its sensors. Due to this, a lively discussion started right away.

**Points of Criticism Mentioned.** The overarching theme of criticism was *transparency*. Nearly all other criticisms boiled down to the fact that smart speaker manufacturers would not transparently document or communicate what the speakers record, what data is processed and what other information they derive from their users. It was mentioned that the manufacturers aimed for “black boxing” the technology with its data flows, algorithms and profiling mainly for reasons of ease of use. This was reinforced by potentially long data persistence, participants said. Furthermore, they criticized a lack of information about whether and which third parties are involved in data processing. Accordingly, misuse of data by providers but also by third-party providers and intelligence services was assumed. This misuse was seen by one participant as a “prerequisite for manipulation.” This manipulation could be manifold and could be used to manipulate the results of an online search so that it leads to a phishing website, but could also be used for election manipulation or price discrimination, among other things.
Following this argument, it was argued that smart speakers would aim for a “data maximizing strategy” instead of data minimization. This means that smart speakers would collect as much data as possible by design, and only later it would be decided what kind of data was needed for the functionality of the device [4]. To counter this, participants suggested that manufacturers adopt a clear privacy by design strategy.

In addition, participants mentioned that most users may not be aware of the (potential) privacy and security risks they face from smart speaker use. The good usability (voice interaction with relatively good recognition performance) coupled with the anthropomorphism of the speakers were a “good and bad thing at the same time”.

In addition to the technological criticism, the participants also saw room for improvement in the regulation of personal use. In their opinion, there were still very unclear rules about the data protection rights of third parties. For example, they said, one could legally install a smart speaker on one's balcony or near an open window and thus record the voices of one's neighbors.

**Categorizing the Criticisms.** While the discussion about the points of criticism was lively, it turned out that categorizing them is no easy task. Many privacy and security risks were seen as interrelated, which is why a clear separation would not be possible. After a few rounds of virtual card sorting, a grouping into “technical,” “social,” and “legal” proved to be the most promising and workable (see Fig. 1).

<figure><img src="image_3.png" /><figcaption>Fig. 1. Categorization of the criticism of smart speakers</figcaption></figure>

## 3.2 Use Case: Detection of Medical Illnesses

The sessions started with an introduction to the application area with the following input:
> Recognition of physical and mental illnesses from recordings of the voice or face is currently still largely experimental, but the first applications are already coming onto the market. They should enable the (early) detection of various diseases or impairments, such as Parkinson's, Alzheimer's, autism or depression. In addition, applications could be used to support therapy.

The moderator then pointed out the difference between applications used by medical professionals and applications that patients can use themselves for self-diagnosis, e.g., using their smartphones.

**Discussion of the Criticism.** In the course of the workshop, the participants raised a number of points of criticism. These included dangers arising from the fact that collected data is not processed on the devices of patients or medical staff, but on remote servers. In this context, there was also criticism that the collected data could also be processed for purposes other than the original ones. In addition, fears were expressed that processing could provide too far-reaching conclusions about health that patients would not have known with usual procedures and that they might not even want to know. If the proliferation of self-diagnosis and self-therapy apps shifted more responsibility to patients, it could overburden the patients and lead to psychological stress. In this context, it was also discussed that potential cost savings from such new technologies could be used to redistribute costs or increase profits instead of improving healthcare, by stagnating the level of healthcare despite the efficiency gains, for example, because cost savings are made in personnel. In the course of this discussion, the danger of an expansion of social power balances was also raised. Accordingly, it is to be feared that the balance of power between data subjects (patients), companies and the state will continue to tilt toward the latter – assuming that such applications will be offered in the future primarily by private-sector companies under state supervision.

The difficulty of obtaining explicit consent from patients was mentioned as a fundamental challenge. Because some data also allowed conclusions to be drawn about other people (e.g., relatives), there was also a risk that the consent of these indirectly affected persons would regularly not be obtained.

It was also feared that medical professionals as well as patients might not be able to operate the recognition systems properly. This problem is particularly aggravated in the case of people who are even less able to operate such tools correctly, e.g. due to mental health problems or advanced age. Moreover, both operating errors and technical deficiencies can lead to false-positive or false-negative detections – with fatal consequences. False-negative results are particularly dangerous because they can lull a person into a false sense of security, even though the person needs to be medically examined and treated as soon as possible. This problem may persist in a mitigated form even if patients forgo more invasive but accurate diagnostic methods for convenience, relying instead on voice-, speech-, and face recognition-based applications.

**Categorizing the Criticism.** With regard to the structuring of the critique, a three-way division into technology, law and ethics was also suggested for this use case. More precisely, data security was also subsumed under the term technical issues. Aspects of data protection law were sorted under the heading of law, and social issues were added to the ethical questions (see Fig. 2).
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Categorization of the criticism of disease detection</figcaption></figure>

Issues of technical suitability were clearly assigned to the technical domain, but there were also issues that were considered to overlap with the legal domain. These include the use of training models for non-health purposes and processing on service provider servers. Challenges in the area of misinterpretation of analysis results were discussed as being at the intersection of technology, law, and ethics.

Issues of consent and intrusive processing, which is closely linked to the purposes of processing and transparent consent, were assigned to the legal area. Challenges regarding patient autonomy were assigned to the intersection between law and ethics.

Finally, the ethical aspects included individual overload, overburdening of disadvantaged persons, economic redistribution effects, and the widening imbalance of power between those affected and data processors and the state.

## 3.3 Use Case: Mass Surveillance

The following text was presented to the attendees at the beginning of the session as an introduction:

> Facial recognition by police agencies can be used both in real-time to search for missing persons or fugitives in public places, train stations, etc., and ex-post to match crime scene footage with police databases during a police investigation. Real-time recognition in particular is the focus of criticism.

During the discussion, the participants decided to categorize the criticisms according to the structure inspired by a data model, since the basis of almost every surveillance technology is data, from which models are created and on the basis of which decisions are made. Data models indicate, among other things, what information is available and
how it is related at the level of policy/law, data, model, and use (see Table 1). This approach has the advantage of systematizing the points of criticism at multiple levels and from the perspective of different stakeholders. The following section summarizes the criticisms discussed and the categories defined. Next, these categories are discussed individually.

<table><caption>**Table 1.** Categorization of the criticism of mass surveillance</caption><thead><tr><th colspan="4">Analysis Level</th></tr><tr><th>Policy; Legal level</th><th>Data Level</th><th>Model Level</th><th>Use Level</th></tr></thead><tbody><tr><td>"Unstable" legal ground for technology use</td><td>Bias in training data</td><td>Error rates vs. Identification rates</td><td>Criminality moving into unsurveyed places</td></tr><tr><td>Data handling issues/storage time/data ownership</td><td>Tendency to connect data sets</td><td>Racism and Transphobia encoded in the technology</td><td>Purpose of data use may change over time</td></tr><tr><td>Unlimited storage capabilities, no clear deletion periods</td><td>Data could be misused or used for other purposes</td><td>Strategies to deal with false positives</td><td>Due to the increased density of control, this can undermine trust in the state and the police</td></tr><tr><td>Function creep and sweeping interpretation of legal bases by law enforcement</td><td>Stigmatized or excluded on the basis of belonging to a particular social group</td><td></td><td>Fears could arise among the population regarding the misuse of data. Comprehensive data processing could undermine democratic values</td></tr><tr><td>Clandestine surveillance (no cooperation with citizens needed)</td><td></td><td></td><td>Permanent surveillance of everyday lives of large parts of the population</td></tr><tr><td>Securitization of society</td><td></td><td></td><td>General suspicion of the entire population<br>Over-reliance on recognition results<br>Unclear decision making (no transparency)</td></tr><tr><th colspan="4">Stakeholders</th></tr><tr><td>Administration; Policy makers</td><td></td><td>Industry; Research; Academia</td><td>Security agencies (Police, Private security companies)</td></tr></tbody></table>
**Policy; Legal Level.** On the first level, the participants saw possible risks in the use of voice, speech and facial recognition technologies arising from an unclear legal situation regarding the use of technology. Further questions arose with regard to the legal handling of data, the storage period or access rights to (personal) data. In particular, the Group considered the expansion and far-reaching interpretation of the legal basis by law enforcement authorities to be a risk. For example, it was possible to use raw biometric data to draw further conclusions about personal characteristics and attributes beyond the actual purpose of use. This access to further information from raw data (function creep) and possible misuse would have to be prevented by means of suitable and holistic legal regulations. At the level of the society, the participants also discussed the concept of “securitization of society” [5]. This involves the gradual penetration of society by security technologies with the aim of controlling societal development in as many areas as possible. The workshop participants saw a special duty to act here for players from administration and politics.

**Data Level.** At the level of data, the participants discussed the risks of bias and distortion of data sets (data bias). Accordingly, the use of technologies based on biased training data could have discriminatory effects. The group discussed that these biases could promote stigmatization or exclusion based on belonging to a particular social group. Once individuals are treated differently based on surveillance and biased data sets, categorization per se is discriminatory and potentially impacts their opportunities and possibilities in various social settings, the group argued.

**Model Level.** At the model level, the participants first clarified that if the biometric data was collected by sensors, it was necessary to first prepare it for model-based analysis. The false acceptance rate (FAR) indicates the average number of false acceptances of a biometric security application and thus evaluates the precision of a biometric system. At the model level, the breakout session participants discussed the relationship of the FAR to the Equal Error Rate (EER) and the FAR. The EER is considered the most informative measure of the performance of a biometric system and is determined by the equality of correct and incorrect detections. However, these rates are not theoretical or derivable, but have to be determined on the basis of practical testing [6]. If multiple individuals have the same biometric characteristics and are therefore accepted as matching each other, this is also considered a mismatch - the uniqueness of the captured biometric characteristic therefore limits the smallest achievable FAR [7]. These technical characteristics are closely related to the risks posed by the nature of the database. Therefore, according to the participants, strategies need to be developed for dealing with false positives. Furthermore, stakeholders from industry, and academic research in particular should address the risks and develop solutions for “fair” data models. In addition, it was discussed that the model borders are fluid - and data models successively evolve. This could become a risk if purposes and goals other than those originally known and intended are pursued in the data collection process. According to the group, it is practically impossible for the data subjects to judge whether the collected data are evaluated for biometric characteristics or not.

**Use Level.** At the level of the use of voice, speech, and facial recognition technologies in the context of mass surveillance, participants discussed societal risks. Among them
was the possibility that increased surveillance could lead to the relocation of criminal activities to less or non-surveillance locations. In addition, participants noted that the increased density of control by means of surveillance technologies can undermine trust in the state and the police (and democratic values). Mass surveillance via voice, speech and facial recognition technologies can give citizens the feeling that they are under general suspicion, which is an explicit vote of no confidence. Coupled with over-reliance on software results, it could also lead to non-transparent decisions. Finally, the group discussed how the permanent surveillance of everyday life of large parts of the population could lead to fear, self-censorship and conformist behavior among citizens (chilling effects).

# 4 Merging the Results

At the end of the workshop, the results of the breakout sessions were briefly summarized and a discussion on similarities regarding categorization began.

The discussion on smart speakers revealed a variety of criticisms of them.<fnref n="2" /> Most of the raised points criticize the lack of transparency of the algorithms and data that are recorded and processed. According to the participants, the anthropomorphism of the speakers' voice and the character of the everyday interaction lead users not to perceive surveillance as such. Specifically, the criticisms can be found in production/sale of speakers, usage, social impact, fairness, user autonomy, transparency, security and trust. Similarly, discussions on the topics of disease detection and mass surveillance fostered a wide range of criticisms. In the case of disease detection, these included issues such as fitness for purpose, false positives, processing on uncontrollable servers, challenges in obtaining consent, risk of user misunderstanding, individual overload, and shifting power relations. In the case of mass surveillance, there were indeed overlaps, e.g. with regard to the points of misidentification and misuse. However, new points of criticism were also mentioned, including the erosion of societal trust and thus the erosion of democratic values or challenges posed by function creep.

Before a consolidation of the results regarding the categorization of the critiques was possible, the workshop ended though. Nevertheless, it should be mentioned at this point that, in addition to the rather classic division into the three pillars of technology, law and ethics, it is particularly the model-centered approach of the third breakout session that sounds promising. It allows the inclusion of several dimensions in the X-axis as well as in the Y-axis, e.g. by also allowing an assignment of the stakeholders.

# 5 Conclusion

Whereas just a few years ago discussions about recognition technologies were clearly focused on the area of facial recognition and the scope of government surveillance

<footnote n="2">² As this was a workshop with limited time capacities, it was of course not possible to provide an exhaustive survey of all conceivable points of criticism, so these points should rather be understood as an exemplary selection. For a more exhaustive discussion, see [1].</footnote>
purposes, today we are already in a phase of massive proliferation of dozens of different applications into various areas of life.

Even though only a cursory summary of the results from the three breakout sessions was possible due to time constraints, the importance of this step for future work should nevertheless be emphasized here. The discussions in the workshop regarding the collection of criticisms of the three exemplary application areas of voice, speech, and face recognition technologies as well as the structuring attempts represent a valuable first step in keeping track of the critical aspects of these technologies in light of the increasingly confusing application landscape. In particular, the study of overlapping uses of data collected by voice, speech, and vision recognition technologies across different application domains and by different stakeholders is becoming increasingly important in light of this application diffusion. It would therefore be desirable if further research were carried out into developing evaluation matrices of such technologies, with which the various points of criticism can also be discussed and structured across different areas of application. Initial attempts are available, for example, in the form of the ethical matrix of Castelluccia and Métayer [8] or in the context of the publications around trustworthy AI [9].

## References

1. Karaboga, M., Frei, N., Ebbers, F., Rovelli, S., Friedewald, M., Runge, G.: Automatisierte Erkennung von Stimme, Sprache und Gesicht: Technische, rechtliche und gesellschaftliche Herausforderungen. vdf Hochschulverlag AG an der ETH Zürich (2022). https://doi.org/10.3218/4141-5

2. Lohr, S.: Facial Recognition Is Accurate, if You're a White Guy (2018). https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html

3. Stark, L., Hoey, J.: The Ethics of Emotion in AI Systems (2020). https://osf.io/9ad4u/. https://doi.org/10.31219/osf.io/9ad4u

4. Malkin, N., Deatrick, J., Tong, A., Wijesekera, P., Egelman, S., Wagner, D.: Privacy Attitudes of Smart Speaker Users. Proc. Priv. Enh. Technol. **2019** (2019). https://doi.org/10.2478/popets-2019-0068

5. Schuilenburg, M.: The Securitization of Society: Crime, Risk, and Social Order. New York University Press (2015). https://doi.org/10.18574/nyu/9781479854219.001.0001

6. Hornung, G.: Die digitale Identität: Rechtsprobleme von Chipkartenausweisen: digitaler Personalausweis, elektronische Gesundheitskarte, JobCard-Verfahren. Nomos (2005)

7. Strauss, S., Schaber, F., Peissl, W.: Biometrics: The body as universal ID? (ITA Dossier No.55en, April 2021) (2021). https://doi.org/10.1553/ita-doss-055en

8. Castelluccia, C., Métayer, D.L.: Analysing the Impacts of Facial Recognition Towards a Rigorous Methodology Position Paper. hal-02480647. 17 (2020)

9. Hallensleben, S., et al.: From principles to practice: an interdisciplinary framework to operationalise AI ethics (2020). https://doi.org/10.11586/2020013