# "Fire! Do not fire!": Investigating the effects of autonomous systems on agency and moral decision-making

Adriana Salatino, Arthur Prével, Emilie Caspar, Salvatore Lo Bue

► To cite this version:

Adriana Salatino, Arthur Prével, Emilie Caspar, Salvatore Lo Bue. "Fire! Do not fire!": Investigating the effects of autonomous systems on agency and moral decision-making. Acta Psychologica, 2025, 260, pp.105350. 10.1016/j.actpsy.2025.105350. hal-05449732

HAL Id: hal-05449732

https://lilloa.hal.science/hal-05449732v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# “Fire! Do not fire!”: Investigating the effects of autonomous systems on agency and moral decision-making

Adriana Salatino<sup>a, *,</sup> Arthur Prével<sup>b, *,</sup> Emilie Caspar<sup>c</sup>, Salvatore Lo Bue<sup>a</sup>

## ARTICLE INFO

### Keywords:

Human-autonomous systems interaction

Human performance

Moral decision-making

Agency

Responsibility

## ABSTRACT

Autonomous systems have pervaded many aspects of human activities. However, research suggests that the interaction with these machines may influence human decision-making processes. These effects raise ethical concerns in moral situations. We created an ad hoc setup to investigate the effects of system autonomy on moral decision-making and human agency in a trolley-like dilemma. In a battlefield simulation, 31 participants had to decide whether to initiate an attack depending on conflicting moral values. Our results suggest that human decision-making in morally challenging scenarios can be influenced by recommendations from autonomous systems. Interestingly, subjective judgement of responsibility decreased with higher levels of autonomy, suggesting that interaction with intelligent systems can influence both moral decision-making and sense of responsibility. Given the growing use of intelligent systems in sensitive domains, further research is essential to better understand these effects and their broader implications.

## 1. Introduction

In the last decades, the use of autonomous systems has become increasingly widespread in various fields of human activity. From driving (Ayoub, Zhou, Bao, & Yang, 2019; Chan, 2017) to aviation (Anderson, Fannin, & Nelson, 2018; Chialastri, 2012; Valdés, Comendador, Sanz, & Castán, 2018), medicine (Kawamoto, Houlihan, Balas, & Lobach, 2005; Sutton et al., 2020), and military defense and security (Mayer, 2015), there is almost no area where these technologies are not massively deployed. As these technologies become more widespread, the inevitable trend toward even more machine autonomy will lead to profound changes in the role played by humans during the execution of tasks. From workers in industry, agriculture, and transportation, to consumers in their daily lives, advances observed in technologies show that humans are rapidly moving from the position of direct tasks executors (with the help of mechanistic machines) to supervising tasks performed directly by intelligent machines with a high level of autonomy.

The main reason for the massive deployment of autonomous systems lies in the many benefits that these systems offer to users. Several laboratory experiments have shown that the introduction of some level of autonomy in tasks can have a significant impact on users' decision-making and performance. For example, studies have shown that

autonomous systems can help people recognize task-relevant cues and ignore irrelevant cues in the environment (Chavaillaz, Schwaninger, Michel, & Sauer, 2018; Goh, Wiegmann, & Madhavan, 2005), improve people's decisions in complex situations and reduce the number of errors they make (MacMillan, Deutsch, & Young, 1997; Rovira, McGarry, & Parasuraman, 2007; Sarter & Schroeder, 2001), or even shorten the time to make correct decisions (Chavaillaz et al., 2018). In addition, the introduction of autonomy has been shown to reduce users' mental workload and thus increase their ability to monitor multiple tasks simultaneously (e.g., Chen & Barnes, 2012; Wright, Chen, & Barnes, 2018). Laboratory experiments have also shown that the effects produced by the interaction with autonomous systems on human decision-making and performance depend on several factors (Mosier & Manzey, 2019; Parasuraman & Riley, 1997). In particular, autonomous systems operating at the decision stage of a task or with a high level of autonomy are generally associated with the largest benefits in terms of human performance (e.g., Endsley & Kaber, 1999; Manzey, Reichenbach, & Onnasch, 2012; Rovira et al., 2007). Based on these results, one could easily conclude that more autonomy is always better for the users, whether they are pilots flying on a plane, physicians analyzing the test results from a patient, or military drone operators deployed in a war zone.

<footnote><sup>a</sup> Department of Life Sciences, Royal Military Academy, Brussels, Belgium</footnote>

<footnote><sup>b</sup> University of Lille, CNRS, UMR 9193 – SCALab – Sciences Cognitives et Sciences Affectives, Lille, France</footnote>

<footnote><sup>c</sup> Moral & Social Brain Lab, Department of Experimental Psychology, Ghent University, Belgium</footnote>

<footnote>* Corresponding authors.</footnote>

<footnote>E-mail addresses: adriana.salatino@mil.be (A. Salatino), arthur.prevel@univ-lille.fr (A. Prével).</footnote>
However, the involvement of autonomous systems technologies into human activities has not been systematically associated with positive effects. Several studies have shown that their use can also have significant negative effects. Parasuraman, Molloy, and Singh (1993) provided a classic demonstration thereof. They reported low level of detection of automation failures with highly reliable systems, an effect they called automation complacency (also named automation overreliance). Other examples of negative effects include loss of situational awareness (Endsley, 2017), skill decay (Haslbeck & Hoermann, 2016; Volz & Dorneich, 2020), performance decrement in return-to-manual control (Endsley & Kris, 1995), or increased workload with too many autonomous systems to monitor (Wang, Lewis, Velagapudi, Scerri, & Sycara, 2009). Interestingly, the detrimental effects caused by the cooperation with autonomous systems seem to be directly related to the stage and level of autonomy of these systems. For example, Rovira et al. (2007) reported lower rates of correct decisions with highly reliable systems with high levels of autonomy (i.e., decision systems) compared to systems with lower levels of autonomy (i.e., information systems). Thus, while more autonomy seems to be clearly beneficial when the system's recommendations are correct, the negative effects also seem to be more pronounced for higher levels of autonomy in machines, which are, most of the time, imperfect.

In this context, another important aspect of human-autonomous systems interaction that has received increasing attention in recent years is the impact of autonomy on human agency (Berberian, 2019; Berberian, Sarrazin, Le Blaye, & Haggard, 2012; Coyle, Moore, Christensen, Fletcher, & Blackwell, 2012; Zanatto, Chattington, & Noyes, 2021). The sense of agency (SoA), defined as the *feeling of causing changes in the external world by controlling one's own voluntary actions* (Burin, Pyasik, Salatino, & Pia, 2017; Haggard, 2017; Jeannerod, 2003; Pyasik et al., 2019), is recognized as an important aspect of human consciousness. Because SoA enables us to perceive ourselves as causal agents, it is the basis for intentional behavior (Haggard & Tsakiris, 2009), and is closely related to moral responsibility (Caspar, Christensen, Cleeremans, & Haggard, 2016; Moretto, Walsh, & Haggard, 2011).

The recent interest in this topic has been triggered by the possibility offered by the “Intentional Binding” effect to implicitly measure the SoA. The Intentional Binding is a well-known phenomenon in which the perceived time between an action and its outcome is modulated by the intentionality of that action. Time appears compressed in situations where the person is active, while time appears stretched during involuntary action (caused, for example, by the Transcranial Magnetic Stimulation) (Haggard, Clark, & Kalogeras, 2002), or in situations in which the person is passive (e.g., Christensen, Di Costa, Beck, & Haggard, 2019; Imaizumi & Tanno, 2019; Malik & Obhi, 2019; see Moore & Obhi, 2012 for a review). Measuring the SoA by using this effect usually consists of asking subjects to estimate the time interval between an action they perform and the consequences of that action. In classical IB measurement tasks, participants have to estimate the time in milliseconds that elapsed between a keypress and an outcome following that keypress (Caspar, Cleeremans, & Haggard, 2015) or report the timing of an action and its outcome on a Libet clock (Haggard et al., 2002). Numerous studies have now shown that the time estimation between action and outcome is a valid implicit measure of SoA (e.g., Christensen et al., 2019; Haggard et al., 2002; Imaizumi & Tanno, 2019; Malik & Obhi, 2019; Moore & Obhi, 2012) and is preferable to a subjective explicit measurement of agency (or responsibility). Indeed, it is worth noting that explicit measures of SoA are subject to social desirability and other biases (e.g., Blackwood, Bentall, Simmons, Murray, & Howard, 2003; Wegner & Wheatley, 1999) and likely rely on higher-level conceptual judgements of agency that are influenced by social and contextual cues (Synofzik, Vosgerau, & Newen, 2008). There is evidence that these two systems operate via separate mechanisms (e.g., Saito, Takahata, Murai, & Takahashi, 2015) and that implicit and explicit measures of agency do not always align (Dewey & Knoblich,

2014).

In one of the first studies investigating the effects of autonomy on SoA by using the Intentional Binding paradigm (Berberian et al., 2012), participants took part in a flight simulation and were assisted in their task by different levels of automation. Berberian and colleagues' results showed a decrease in SoA with increasing levels of automation, suggesting that agency decreases with higher levels of automation. Further evidence using the same paradigm can be found in the study by Coyle et al. (2012), who investigated how assistance, in a machine-assisted point-and-click task, affects the user's SoA. Their results suggest that, up to a certain point, the computer could assist users while also allowing them to maintain a sense of control over their actions and outcomes, hence of their agency. More recently, Zanatto et al. (2021) have shown that automation has a similar negative effect on SoA and that mental workload may also play a role in reducing agency. Furthermore, providing information about the system's intentions prior to an action has been shown to increase the sense of agency in supervision tasks (Le Goff, Rey, Haggard, Oullier, & Berberian, 2018; see also Pagliari, Chambon, & Berberian, 2022), confirming that the sense of agency can be modulated by multiple factors. Taken together, these studies suggest that automation technology may affect the mechanism underlying human agency.

Hence, the evidence of negative effects on human decisions and performance, and the evidence of a decrease in the implicit and explicit SoA (Berberian et al., 2012; Coyle et al., 2012; Salatino, Prével, Caspar, & Bue, 2025; Vantrepotte, Berberian, Pagliari, & Chambon, 2022), that could result from the interaction with the autonomous systems, has serious implications for performance and safety. Engineers working on the development of new forms of autonomous technologies should be aware of these effects and take them into account. Furthermore, these results have important implications when these systems are used in sensitive or moral domains such as in medicine or in the military, in which decisions of life and death have to be made.

To date, however, very little is known about how the interaction with an autonomous machine affects SoA and the decisions made by someone engaged in a moral scenario, and how this is influenced by the level of autonomy of the system. Indeed, it is possible that interacting with autonomous systems to make moral decisions negatively affects the moral and ethical decision-making process and the resulting actions, particularly in tasks and domains of moral value such as in the military context (Christensen & Gomila, 2012; Cushman, 2013; Cushman, Kumar, & Railton, 2017). This is consistent with previous findings by Greene, Sommerville, Nystrom, Darley, and Cohen (2001), who showed, in a simulated moral dilemma, that participants were more likely to make moral judgements when they had to push a person off a bridge (a more personal, physically direct action) than when they had to pull a lever to achieve the same outcome. Although their study did not explicitly address the sense of agency, it is plausible that the psychological distance created by indirect actions (such as pulling a lever) reduces perceived personal agency and makes utilitarian choices more psychologically acceptable. This consideration could also extend to decisions related to autonomous systems, where the indirectness of an action mediated by a machine could similarly reduce perceived agency and influence moral judgement.

In recent years, research in the field of moral decision making and autonomy has mainly focused on the rules and/or algorithms that can be assigned to an autonomous system to perform ethical responses in moral situations (Arkin, Ulam, & Wagner, 2011; Jiang et al., 2021). Surprisingly, until recently, little attention has been paid to understanding how the ethical behavior of a human agent in moral decision-making situations can be influenced by its interaction with an autonomous system (Köbis, Bonnefon, & Rahwan, 2021). The available data suggest a mixed picture of the effects of autonomous systems in social and moral decision-making situations. Indeed, while some recent evidence suggests that interaction with automation could lead to the promotion of social behaviors (such as fairness and cooperation, see, e.g., de Melo,
Marsella, & Gratch, 2018, 2019), other studies have shown that it could also lead to unethical behaviors (Cohn, Gesche, & Maréchal, 2022; Leib, Köbis, Rilke, Hagens, & Irlenbusch, 2021). Concerning SoA, while some studies report that a moral context increases it (e.g., Moretto et al., 2011) and that a higher SoA is associated with higher prosocial decision-making (Caspar et al., 2022), it is not clear whether this is still true when decisions are made in collaboration with an autonomous intelligent machine.

Considering the lack of research on how autonomous systems impact the SoA and decision-making in a moral context, and how this can be modulated by the level of autonomy of the system, in the present study, we aimed to investigate how the level of system autonomy affects SoA and the moral decision-making. To this end, we developed a task in which participants (military cadets) played the role of drone operators on a simulated battlefield and had to decide whether or not to initiate an attack, based on the presence of enemies and the risk that allies might also be harmed. Participants were exposed to three types of trials representing three types of uncertainty (*Moral Decision-Making Trials, No Risk Trials, and No Enemy Trials*) with three different levels of system autonomy, including no system assistance, information assistance (i.e., the system gives processed-information about the presence of enemies and the risk for allies), and decision assistance (i.e., the system provides a recommendation on the best decision to make). In our study, SoA is measured both at the implicit level, using the Intentional Binding paradigm, and at the explicit level through a subjective assessment of responsibility (using an ad-hoc scale). We also measured performance by using reaction time, the proportion of trials in which participants chose to attack, and the proportion of *utilitarian* choices (i.e., a choice made on the option with the lowest expected allies losses compared to the other option).

The main aim of this research was to investigate how the interaction with autonomous systems can affect the SoA and the decisions made by people when facing moral choices, and how this is influenced by the level of autonomy of the system (i.e. the extent to which the decision is supported by the system). Central to this investigation is the degree of autonomy of the system, i.e. the extent to which the system influences or supports the user's decision. Following previous findings (Berberian et al., 2012; Coyle et al., 2012; Wen, Yamashita, & Asama, 2015; Zanatto et al., 2021), we hypothesized that the SoA decreases as the autonomy of the system increases, as indicated by a longer time estimation between action and outcome, and lower subjective judgements of responsibility at higher levels of system autonomy. As previous research suggests a close relationship between agency and moral responsibility (Caspar, Cleeremans, & Haggard, 2018), we also hypothesized that if the SoA would be affected by the autonomous system, as well as the sense of responsibility associated with SoA, the moral decision-making would also be affected, with the number of attacks increasing as system autonomy increases (Caspar et al., 2018; Chavaillaz et al., 2018; Goh et al., 2005). In addition, we expected shorter reaction times and more utilitarian choices with higher levels of system autonomy. With the present study, we hope to pave the way for new quantitative studies to understand how the interaction with autonomous systems affects agency and decision-making in a moral context. In turn, understanding these effects better could help in the development of safer and more efficient autonomous systems in the future.

# 2. Method

## 2.1. Participants

A total of 31 participants took part in the study ($M_{age} = 22$, $SD = 2.28$, Range = 19–36, 7 women, 24 men). The participants were cadets at the Royal Military Academy of Belgium and were recruited with the help of a Master student officer in the course of his thesis. The participants were in their third and fourth year of study, meaning they had notions of International Humanitarian Law, and thus about what is

legally allowed and forbidden in the conduct of armed conflicts. The study was performed in accordance with the principles expressed in the Declaration of Helsinki. This study was reviewed and approved by the local ethics review board at the Faculty of Psychology and Educational Sciences at Ghent University (approval number: 2022/047). Participants were informed about the general purpose and the duration of the experiment, and about their rights as participants in psychological research before giving their consent. Participation was voluntary and participants could withdraw their participation at any time without justification and without consequences. Written informed consent was obtained from each participant prior to the experiment.

## 2.2. Stimuli and procedure

All material can be found on the Open Science Framework (Salatino, Prével, & Lo Bue, 2023). The experiment was programmed and presented using MATLAB 2020b and the Psychophysics Toolbox extension. The experiment was run on a laptop computer (display resolution: 2560\*1600 pixels) and responses were collected via an AZERTY keyboard with left and right arrow keys serving as the response keys used in the experiment. The experiment was conducted in an experimental room in the department of Life Sciences at the Royal Military Academy. Participants sat on a chair in front of a table with the laptop screen being at approximately 40 cm away from the participants. Each trial of the task consisted of a 50 by 50 grid with dark grey cells (RGB = [064, 064, 064]) presented on a black screen (RGB = [000,000,000]; see Fig. 1, below). Participants were instructed that the grid represented a radar display used to inform on the position of allies and enemies. Out of the 2500 cells, 100 were colored in light grey (RGB = [192, 192, 192]) and each light grey cell represented the position of a group of 10 allies. The number of allied forces (i.e., light grey cells) was kept constant across trials, but their position varied randomly from trials to trials. In addition, in 66% of the trials one of the 2500 cells was colored in white (RGB = [255, 255, 255]) to represent the presence and the position of an enemy. Maximum one cell was colored in white during trials (i.e. a maximum of one enemy). When an enemy (i.e., a white cell) appeared in a trial, participants were asked to choose between attacking the enemy or not by pushing the left (attack) and right (no attack) arrows of the keyboard (i.e., Action 1- A1, if they decide to attack, and Action 0- A0, if they decide to not attack). Importantly, the participants were only required to make a decision - whether to fire or not - not to aim at the enemies, not to fire at them effectively and not even to be confronted with the consequences of their actions. Participants were instructed that not pushing the “attack” button whenever an enemy was present would result in the death of five allies because of the continued hostile activities of that enemy. Instead, pushing the “attack” button whenever an enemy was present would result in the death of the enemy, but with sometimes a risk of collateral damages considering the position of one to three groups of 10 allies on the 50 by 50 grid radar screen (see details in the next paragraph). Collateral damage was likely, but not sure, if the grey cells were separated from the enemy by less than five empty cells. Participants were also instructed that they would not risk collateral damage if the grey cells were separated from the enemy by five empty cells or more, and if no enemy was visible on the grid, participants were asked to not push on the “attack” button. These two situations served as control conditions to ensure that they attack in and do not attack, respectively, in not conflicting situations.

During the experiment, participants were tested with three different types of trials, representing three levels of Uncertainty: *No Risk* trials, *No Enemy* trials, and *Moral Decision-Making (Moral DM)* trials. In the *No Risk trials*, an enemy was shown on the radar display and allies were separated from five cells of more from the enemy, meaning there were no risk or collateral damage. In the *No Enemy trials*, no enemy was shown on the radar display. Finally, in *Moral DM trials* an enemy was shown on the radar screen and up to three groups of allies were on the first to the fifth range of cells next to the enemy. Here, participants received the
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Experimental setup. Each trial of the task consisted of a 50 × 50 grid of dark grey cells on a black screen (Left Panel), representing a radar display informing of the position of allies and enemies. Of the 2500 cells, 100 were colored light grey and represented the position of a group of 10 allies, which varied in position from trial to trial.</figcaption></figure>

In 66 % of the trials, one of the 2500 cells was colored white to represent an enemy and participants were asked to choose whether or not to attack the enemy by pressing the left (attack) or right (no attack) arrow key on the keyboard. The grid was displayed for 15,000 ms or until the participant pressed one of the two response keys. After the response, a black screen was displayed for a random duration (200, 500 or 800 ms) and a tone (200 ms). Participants were asked to indicate the duration of the interval between their choice and the tone on a horizontal scale from 0 ms to 1000 ms. Level 1 of system autonomy is shown in the right panel (top), where in addition to the basic visual assistance of Level 0, a scale has been displayed indicating the risk in terms of losses of allied forces in the event of an attack. In Level 2 (Lower Right Panel), in addition to the visual assistance of Level 0, participants were assisted by a decision support system that gave a yes/no recommendation for the best decision to make.

instruction that during those trials there was a risk of collateral damages if they decided to attack. More specifically, participants were instructed that allies found on the 1st range of cells next to enemy's position had a 0.50 probability of being killed if they decided to attack, while the probability of collateral damages decreased with the distance according to the following range – probability combinations: 2nd range – 0.40, 3rd range – 0.30, 4th range – 0.20, and 5th range – 0.10. The number of allies that could be killed during an attack varied from 10, 20, or 30 allies. Groups of allies were systematically positioned on the same range of cells and Moral DM trials consisted in 15 different number of allies by probabilities of collateral damages combinations (see Table 1).

<caption>**Table 1**</caption>

Probabilities of collateral damage. The table shows the expected losses in the event of an attack, calculated as the number of allies involved multiplied by the probability of the respective outcome. Red values indicate that the expected losses in case of an attack are higher than the sure losses in case of a no attack (5). Thus, the participants are expected to choose to not attack. Green values indicate that the expected losses in case of an attack are lower than the sure losses in case of a no attack. Thus, the participants are expected to choose to attack.

<table><thead><tr><th rowspan="2">Allies</th><th colspan="5">Probabilities of collateral damage</th></tr><tr><th>0.50</th><th>0.40</th><th>0.30</th><th>0.20</th><th>0.10</th></tr></thead><tbody><tr><td>30</td><td>15</td><td>12</td><td>9</td><td>6</td><td>3</td></tr><tr><td>20</td><td>10</td><td>8</td><td>6</td><td>4</td><td>2</td></tr><tr><td>10</td><td>5</td><td>4</td><td>3</td><td>2</td><td>1</td></tr></tbody></table>

We therefore expected those trials to be morally challenging for the participants. They were asked to choose between (1) pushing the attack button to neutralize the enemy, with the risk of killing allies, or (2) not pressing the attack button and allowing the enemy to kill five allies.

The experiment consisted of three blocks, with each block including a specific level of Autonomy (Level 0, Level 1, Level 2). In Level 0, the block with the lowest level of assistance from intelligent system, participants received basic visual assistance to determine the position of allies relative to the position of an enemy shown on the grid. During No Risk and Moral DM trials, by clicking with the help of the computer mouse on a white cell while the 50 by 50 grid was shown on the screen. Then, the cells on the 6th range next to the position of the enemy turned blue (RGB = [000, 000, 255]) until the participants made their decision. This visual information was designed to help the participants to detect how far allied forces were from the enemy's position and to compute the risk of collateral damages if they decided to press the attack button. However, allies found within that area were at risk for collateral damages, with the risk depending on the range.

Then, in Level 1 of Autonomy, in addition to the basic visual assistance found in Level 0, a 12 grade-scale was shown on the right part of the screen (see Fig. 1, right up) participants were instructed that this scale indicated the risk in terms of losses of allied forces if the attack button was pressed. The risk was computed based on the number of allies within the blue area (i.e., 10, 20, or 30) and the probability of collateral damages based on their position (i.e., 0.50, 0.40, 0.30, 0.20, or 0.10). The scale ranged from yellow (very low risk) to red (very high
risk).

Finally, in Level 2 of Autonomy, the visual assistance of Level 0 was still included, but in addition participants were assisted by a decision-support system that provided a yes/no recommendation for the best decision on each trial (see Fig. 1, bottom right). This recommendation was based on the choice associated with the lowest expected losses in terms of allied forces in each trial. If the expected losses were lower when the attack button was pressed, the ‘yes’ button was highlighted, while the ‘no’ button was highlighted if the expected losses were lower when the no attack button was pressed (except for the trial where the number of allies was 10 and the probability was 0.50, in which expected losses were equal for both choice). Participants were not informed on the computation on which the recommendations were determined. Overall, each block consisted of 15 No Risk trials, 15 No Enemy trials, and 15 Moral DM trials.

The experimental setup of this task is shown in Fig. 1 (left): (1) First a loading bar was presented for delay chosen randomly between 1000 ms and 2000 ms to signal a new trial to the participants. (2) This was followed by a black screen for 500 ms and then the presentation of the 50 by 50 grid, which was displayed for 15,000 ms or until the participant pressed one of the two response keys. (3) Participants were asked to confirm their choice by pressing the selected response key again, or they had the possibility to change their choice by pressing the other key. (4) Responses were followed by the presentation of black screen for a random duration of either 200, 500, or 800 ms, and a tone (frequency: 400 Hz) for 200 ms. The tone did not change with the task or the decision made. Finally, (5) participants were asked to report the duration of the interval between their confirmation choice and the tone on a horizontal scale ranging from 0 ms to 1000 ms. The trials were separated by an interval of 1000 ms.

## 2.3. Measurements and analysis

We used five dependent variables in this study: Decision, Utilitarian Choice, Response Time, Agency, and Subjective Responsibility. Decision (A1) was expressed by the proportion of trials on which participants decided to attack. Utilitarian Choice (UC) was expressed by the proportion of choices implying the lowest expected losses (in percentage). Response Time (RT) was the mean response time (in seconds) on each trial. SoA was measured by Intentional Binding (IB, in milliseconds). IB was computed by subtracting each interval estimate from the mean actual response-tone interval (500 ms) and averaged these scores for each Uncertainty X Autonomy condition. Each block of Autonomy ended with a subjective judgement of responsibility (SubjA), in which participants were asked to indicate how much they felt responsible of the decisions they made on a scale from -100 (not responsible at all) to 100 (entirely responsible).<fnref n="1" />

Statistical analyses were performed using JASP version 0.19.3. For the A1 decisions, we conducted an analysis using linear mixed models (LMMs), which are well suited to dealing with hierarchical data structures and do not assume homoscedasticity, making them appropriate for our data where many participants always choose not to attack in the *No Enemy* trials and always choose to attack on the *No Risk*. The model included the Uncertainty and Autonomy conditions as fixed effects and accounted for random intercepts for participants to capture individual variability. We specified the dependent variable as A1, with fixed effects for Uncertainty and Autonomy. Participants were included as a random grouping factor to account for individual variability. For the variables UC, RT, and IB, we performed separate repeated-measures ANOVAs using Uncertainty (*No Risk trials*, *No Enemy trials*, *Moral Decision-Making trials*) and Autonomy (*Level 0*, *Level 1*, *Level 2*) as within-subject factors. In addition, SubjA was compared by means of a repeated measures

ANOVA with Autonomy as within-subject factors. For each dependent variable, only data of participants within +/- 2.5 SDs were considered. The Greenhouse-Geisser correction was applied when sphericity was violated.

The primary focus of our analysis concerned the presence of a main effect of Uncertainty on 1) A1, 2) UC and 3) RT. We expected higher attack rates (A1) when an enemy was present (No Risk and Risk conditions) than when it was not present (No Enemy condition), and more Utilitarian Choices (UC) in the No Risk and No Enemy conditions compared to the Moral Decision-Making (Risk) condition. These effects might confirm participants' understanding of the task. We predicted longer response times (RT) in the Moral Decision-Making condition, reflecting increased cognitive and emotional processing due to the moral conflict. Additionally, following the findings of Moretto et al. (2011), we expected a main effect of Uncertainty with reduced intentional binding (IB), indicating an increase of SoA, during Moral Decision-Making trials in compared to the two control trials. We also expected a main effect of Autonomy on 1) UC, with an increased rate of UC with the level of autonomy of the task, 2) RT, congruent with the main effect of Uncertainty, 3) IB, with less IB, indicating a decrease of SoA, with increased level of autonomy in line with the conclusions of Berberian et al. (2012), and 4) SubjA, with lower SubjA with increased level of autonomy. The threshold selected for significance was *p* < .05 with a two-tailed approach. Raw data, scripts, and processed data can be found on the Open Science Framework (Salatino et al., 2023).

# 3. Results

## 3.1. Analyses on A1 decisions

The analysis on A1 decisions (i.e., the proportion of attacks) revealed a main effect of Uncertainty ($df = 2, F = 8230.50, p < .001$), but not of Autonomy ($p = .3$) on A1. Post-hoc comparisons with Bonferroni correction showed that participants were more likely to choose to attack during No Risk trials (Estimate: 99.28, $SE = 0.552$) in comparison with Moral DM trials (Estimate: 52.71, $SE = 2.56$) and No Enemy trials (Estimate: 0.72, $SE = 0.55$, (all $ps < 0.001$), see Fig. 2.

## 3.2. Analyses on Utilitarian Choice (UC)

The analysis on UC (i.e., the choices implying the lowest expected losses, Fig. 3, Panel A) revealed a significant effect of Uncertainty (F (1.18, 31.96) = 95.76, $p < .001$, $ηp^2 = 0.78$). Post hoc tests showed a significant difference between *Moral DM* trials (mean = 84.02, SEM = 1.09) and *No Risk* trials (mean = 99.30, SEM = 0.27), and between *Moral DM* and *No Enemy* trials (mean = 99.21, SEM = 0.50) (all $ps < 0.001$), with a reduced number of UC during *Moral DM* trials, but not between *No Risk* and *No Enemy* trials ($p = 1.000$). The analysis (Fig. 3, Panel B) also revealed a significant effect of Autonomy (F (1.76, 47.55) = 8.67, $p < .001$, $ηp^2 = 0.24$). Post hoc tests showed no significant difference between *Level 0* (mean = 94.10, SEM = 1.07) and *Level 1* (mean = 92.59, SEM = 1.17) ($p = .17$) and between *Level 0* and *Level 2* (mean = 95.84, SEM = 0.88) ($p = .089$). A significant difference was found in the proportion of UC between *Level 1* and *Level 2* ($p < .001$) with more UC on *Level 2*. Finally, the analysis revealed no significant interaction between Uncertainty and Autonomy (F (2.36, 63.80) = 2.48, $p = .08$, $ηp^2 = 0.08$).

## 3.3. Analyses on Response Time (RT)

The analysis on RT (Fig. 4, Panel A) revealed a significant effect of Uncertainty (F (1.73, 46.95) = 68.76, $p < .001$, $ηp^2 = 0.71$) with post hoc tests showing that all comparisons were significant (all $ps < 0.036$), with longer RT during *Moral DM* trials (mean = 4.62, SEM = 0.20) than during *No Risk* (mean = 2.27, SEM = 0.12) and *No Enemy* trials (mean = 2.82, SEM = 0.14). In addition, the analysis revealed no significant effect of Autonomy (F (1.68, 45.36) = 0.667, $p = .49$, $ηp^2 = 0.02$), but a

<footnote n="1">¹ This range is commonly used in human contingency assessment (for recent examples, see Prével, Krebs, Kukkonen, & Braem, 2021 or Vaghi et al., 2019).</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Proportion of A1 action (i.e. attacks performed). A significant increase of A1 choices during *No Risk* trials in compared to the *Moral Decision-Making* and *No Enemy* trials were found (all *p* < .001). \* = significant.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Proportion of UC (i.e. Utilitarian Choices). A significant difference in UC was found between *Moral Decision-Making* trials and *No Risk trials*, and between *Moral Decision-Making* and *No Enemy* trials (all *p* < .001), with a lower number of UC during *Moral Decision-Making* trials, but not between *No Risk* and *No Enemy* trials (Panel A). A significant difference in the proportion of UC was found between *Level 1* and *Level 2* (*p* < .001) with more UC on *Level 2*. \* = significant.</figcaption></figure>

significant interaction (Fig. 4, Panel B) between Uncertainty and Autonomy (F (3.43, 92.74) = 3.81, *p* = .009, ηp² = 0.12), with a simple main effect of Autonomy in *No Enemy* trials (*p* = .03), but not in other Uncertainty conditions (all *ps* > 0.37).

### 3.4. Analyses on Intentional Binding (IB)

The analysis on IB (Fig. 5) revealed a significant effect of Uncertainty (F (1.56, 46.87) = 8.12, *p* = .002, ηp² = 0.21) with post hoc tests showing a significant difference between *Moral DM* trials (mean = 118.85, SEM = 13.58) and *No Risk* trials (mean = 147.04, SEM = 13.99) and between *Moral DM* and *No Enemy* trials (mean = 145.53, SEM =
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Response time (in seconds). A significant increase in the RT was found during *Moral Decision-Making trials* compared to the *No Risk* and *No Enemy trials* (all *p* ≤ 0.036, Panel A). In addition, a significant (*p* = .009) interaction was found between *Uncertainty* and *Autonomy* (Panel B), with a simple main effect of *Autonomy* in *No Enemy trials* (*p* = .03). \* = significant.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Intentional binding (IB). A significant difference in the IB was found between *Moral Decision-Making trials* and *No Risk trials*, and between *Moral Decision-Making trials* and *No Enemy trials* (all *p* < .005), with shorter intervals reported in *No Risk* and *No Enemy trials* compared to the *Moral Decision-Making trials*. \* = significant.</figcaption></figure>

13.69) (all ps < 0.005), with shorter intervals reported in *No Risk* and *No Enemy trials* in comparison with *Moral Decision-Making trials*. No significant differences were found between *No Risk* and *No Enemy trials* (*p* = 1.000). Concerning *Autonomy*, the analysis revealed no significant effect (F (1.76, 52.82) = 0.67, *p* = .495, ηp² = 0.022) and no significant interaction between *Uncertainty* and *Autonomy* (F (3.52, 105.86) = 1.29, *p* = .27, ηp² = 0.04).

### 3.5. Analyses of subjective judgment of responsibility (SubjA)

The analysis on SubjA (i.e., how much participants felt responsible of the decisions made, Fig. 6) revealed a significant effect of *Autonomy* (F (1.61, 45.15) = 15.72, *p* < .001, ηp² = 0.36). Post hoc tests showed a significant difference between *Level 0* (mean = 84.13, SEM = 3.04) and *Level 1* (mean = 71.72, SEM = 4.60) (*p* = .006) and between *Level 0* and
<figure><img src="image_7.png" /><figcaption>Fig. 6. Subjective assessment of responsibility. A significant decrease in the subjective judgement of responsibility was found in *Level 0* compared to *Level 1* (*p* = .006) and in *Level 0* compared to *Level 2* (*p* < .001), with greater subjective judgement of responsibility in *Level 0* compared to *Level 1* and *Level 2*. \* = significant.</figcaption></figure>

*Level 2* (mean = 58.62, SEM = 69) (*p* < .001), with larger subjective responsibility rating during *Level 0* in both cases, but not between *Level 1* and *Level 2* (*p* = .065).

## 4. Discussion and future studies

In the present study, we aimed to investigate how the sense of agency and decision-making in morally and non-morally challenging scenario are influenced by the type of input received from an intelligent autonomous system. To this end, we programmed a task in which participants, in the role of drone operators, had to decide whether (or not) to initiate an attack on a simulated battlefield, during different trial types and with the support of an intelligent system at three levels of autonomy. Ultimately, the overall goal of this study was to investigate the mechanisms involved in human-AI interactions in the context of morally challenging situations and to better understand the determining factors in the effect of (inputs received from) autonomous systems on the decisions of human agents faced with moral choices.

Our results show that our experiment was sensitive enough to discriminate between moral and non-moral situations. Indeed, participants in our study had a 0.54 likelihood to initiate an attack in our moral situations, while they refrained from attacking when there were no enemies, and systematically attacked in situations where an enemy was present, but an attack posed no risk for allied troops. Furthermore, the moral situations were characterized by less utilitarian choices than our control condition, reflecting the uncertainty of the situation. Notably, the assistance of the autonomous system increased the number of utilitarian choices, demonstrating the influence of the machine on decision-making. Finally, the moral situations were characterized by a longer reaction time, indicating the participants' hesitation when they had to decide to fire or to not fire when allies' life were at stake.

Our results were partially consistent with our expectations, as the number of attacks increased when there was no risk and decreased when there was no enemy, confirming that our control trials were working properly as well. However, no effects related to the level of autonomy

were found regarding the proportion of attacks. One possibility is that human choices, when made in the context of moral dilemmas, are not influenced by the level of autonomy of a system, in contrast to choices made in a context without these dilemmas. However, this would be surprising, given that previous studies conducted in military scenarios (e.g., Chen & Joyner, 2009; Rovira et al., 2007) have found an influence of the level of autonomy on human performance and decision. Another interpretation for the lack of an effect of level of autonomy on the rate of A1 choices relies on the way the task was designed (and in particular the possibility for participants to calculate maybe too easily the expected losses for both alternatives in each choice), resulting in ceiling/floor effects in the control conditions and an A1 rate of around 50 % during *Moral Decision-Making* trials.

In our task, the decision between accepting a certain loss (no attack) and a probable loss (an attack with the potential for collateral damage) may at first appear analogous to classical risk-based decision-making scenarios, as described in the loss domain of Prospect Theory (Kahneman & Tversky, 2013). However, the framework of our paradigm extends beyond economic risk models and incorporates key aspects of moral reasoning. While the task does indeed involve loss trade-offs, the nature of the losses - in particular, harm to human life, including non-combatants such as allies and civilians- is fundamentally different from standard economic gambles. Previous studies in the field of moral psychology have shown that, even under probabilistic conditions, decisions involving harm to others engage distinct cognitive and affective processes than those captured by conventional risk calculations (Crockett, Siegel, Kurth-Nelson, Dayan, & Dolan, 2017; Cushman, 2013; Greene et al., 2001). The introduction of collateral damage considerations adds a moral dimension comparable to aversion to harm and perceived responsibility for negative outcomes, factors that are not adequately addressed by classical utility-based models. Nevertheless, we cannot exclude the possibility that individual differences in risk preferences may also contribute to decision-making in this context. Although this personal component was not analysed in the present study, it would be interesting to investigate in future research how
individual risk attitudes intersect with moral decision-making under uncertainty.

With respect to the Utilitarian Choice, i.e., the proportion of choices that lead to the least loss of allies, we expected an increase in UC in trials without moral conflict and an effect of autonomy leading to an increase in these choices at the highest level of Autonomy (as evidence of an effect of autonomous system on moral decision). Our results confirmed our expectations, with utilitarian choices increasing as a function of Uncertainty and Autonomy. Considering that the recommendations made with the highest level of system support were based on the lower expected losses computed, our results showed that our participants' moral choices were significantly changed by the input received from the system, suggesting that that human choices can be influenced by the recommendations received from a decision support system. The high rates of utilitarian choices in the control conditions (i.e., in the no risk and no enemy trials) indicate a global preference for maximizing the number of allies saved. This supports our interpretation that, in the absence of moral or contextual constraints, participants rely on utilitarian reasoning. Our results show that when scenarios become morally complex (as in the MDM trials), participants shift away from pure utility maximization, probably due to conflicts with other moral values such as responsibility or fairness (Cushman, 2013; Greene et al., 2001). Furthermore, we can speculate whether the reduction in UC arises because it is more difficult to compute the best decision (i.e., a problem of *task representation*) or because the decision itself becomes more morally difficult (i.e., a problem of *value conflict*). We believe that both mechanisms are plausible and may interact. Morally ambiguous situations can increase cognitive load, making it more difficult to represent and evaluate all relevant outcomes (Martin, Buon, & Cushman, 2021). At the same time, the emotional and ethical tension between competing values can make the decision itself more difficult, as shown by Greene et al. (Greene et al., 2001; Greene, Nystrom, Engell, Darley, & Cohen, 2004), who found that moral dilemmas elicit greater emotional engagement and slower decision-making. Thus, in our MDM trials, both cognitive and emotional factors likely contribute to the reduction in utilitarian choices.

In relation to the Response Time, we expected that participants would take a longer time to make a decision on trials with a moral conflict and Autonomy would shorten participants' response times. Our results confirmed our expectation, with RTs being longer in moral situations, suggesting that participants took longer to make a decision when a moral conflict was present, and that the highest level of system support shortened their response time in the No-Enemy trials. This last result is consistent with previous findings from laboratory experiments showing that autonomous systems can help users in detection tasks (e.g., Goh et al., 2005), although it is surprising that this effect was only found for the absence of target but not for the presence of target during trials without risk, which is somehow inconsistent with previous findings (e. g., Chavaillaz et al., 2018). A possible reason for this result could be that the target used in our task (i.e., a white square among grey squares on a black background) was too noticeable and thus too easy to recognize for the Level 1 and Level 2 functions to be useful to participants. Nevertheless, we cannot rule out the possibility that the longer RTs in moral situations reflect the computational complexity due to the task rather than the moral conflict. Indeed, in the moral trials, participants had to consider more elements and calculate more risks compared to the other conditions. Although risk calculation in uncertain conditions may increase cognitive load, moral conflict could also amplify this effect, so the two possibilities could interact. On the one hand, uncertain conditions require the integration of probabilistic outcomes with moral constraints, which strains working memory and consequently slows RT (see, e.g., the work of Martin et al., 2021). On the other hand, even when decisions are utilitarian, moral dilemmas can lead to conflicts between competing values (e.g., minimising harm vs. following deontological principles). Studies by Greene et al. (Greene et al., 2001, Greene et al., 2004) show that moral dilemmas activate emotional systems and prolong RTs when

individuals reconcile utilitarian outcomes with emotional intuitions (Greene et al., 2001). In summary, we argue that MDM trials are likely to present a double burden: resolving outcome uncertainty and managing ethical tensions, which is why RTs are slower in these trials.

Regarding SoA, given that a decrease in agency in human-machine interactions has been previously reported (Berberian et al., 2012; Salatino et al., 2025; Vantrepotte et al., 2022), we firstly expected that participants would show a decrease in the Intentional Binding and subjective sense of responsibility at higher levels of support, indicating a decrease in the SoA. Consistent with our hypothesis, our results showed a decrease in the SoA at the explicit level with higher levels of support. This result is consistent with our recent data (Salatino et al., 2025), which showed a decrease of agency at the explicit level during interaction with AI. It is also consistent with the recently described human tendency to attribute moral responsibility to non-human agents, which may lead to humans being willing to blame them (Furlough, Stokes, & Gillan, 2021; Kneer & Stuart, 2021; Liu & Du, 2022). Consistent with this is the concept of “diffusion of responsibility”, which suggests that the presence of others can reduce a person’s perceived responsibility for their actions (Bandura, 1999). This phenomenon has also been observed in interpersonal interactions, as shown, for example, by Caspar et al. (2018), where individuals who followed coercive instructions reported a diminished sense of responsibility than individuals who were free to choose their actions. Importantly, this dynamic also extends to human-machine interactions, as the distribution of responsibility can be influenced by the involvement of autonomous systems, leading to uncertainty or a redistribution of responsibility. Nevertheless, our results did not show a decrease in SoA at the implicit level, which is inconsistent with our expectations and previous studies (Berberian et al., 2012).

However, it has been suggested that transparency about the system's decisions may influence the sense of agency, as it may help to mitigate the loss of agency during interactions between humans and autonomous systems (Le Goff et al., 2018; Pagliari et al., 2022). Since in the present study the information about the system's decision was provided prior to the participants' actions, it is possible that this transparency helped to maintain the sense of agency.

Importantly, it should be emphasized that this discrepancy is not entirely unexpected, as previous studies have reported a dissociation between implicit and explicit measures of the sense of agency (SoA) (e. g., Moore & Obhi, 2012; Saito et al., 2015; Synofzik et al., 2008). It is assumed that explicit judgements of the sense of agency involve higher-order cognitive processes and are influenced by social and contextual cues (Synofzik et al., 2008). This indicates that intentional binding (an implicit measure) and explicit judgements of agency may rely on different underlying cognitive mechanisms (Dewey & Knoblich, 2014). Intentional binding is thought to be based on low-level sensorimotor predictions and relies on the brain's ability to predict the sensory consequences of one's actions through forward models (Haggard et al., 2002; Moore & Obhi, 2012). In contrast, explicit agency judgements involve higher-order cognitive processes, such as contextual interpretations, beliefs and causal inferences (Synofzik et al., 2008; Wegner, 2017). These judgements are often based on postdictive processes and are influenced by factors such as the consistency of outcomes with intentions, social context and prior expectations (Synofzik et al., 2008). Thus, while both Intentional Binding and explicit judgements of agency contribute to the experience of agency, the former is typically driven by predictive sensorimotor mechanisms and the latter by cognitive evaluation and inferential reasoning. As these two systems of measurement are separable (Saito et al., 2015), it is plausible that the dissociation between implicit and explicit measures observed in our study reflects this distinction.

In particular, in our experiment the three autonomy conditions may not have been different enough to yield a significant difference at the implicit level of agency, but only at the explicit level.

Still regarding SoA, consistent with previous findings (Moretto et al., 2011), we also expected an increase in SoA during Moral Decision-
Making trials. Consequently, we expected that moral decision making would also be affected. Because SoA appears to be closely related to moral responsibility (Caspar et al., 2016; Moretto et al., 2011) and this is reduced by the level of autonomy of the machine, we expected that a decrease in the sense of responsibility would lead to a change in the number of attacks at the highest level of system autonomy. However, contrary to our expectations, the results showed a significant decrease in the SoA in the *Moral Decision-Making* trials at the implicit level (i.e., in the IB). One possible explanation for these results is related to the human tendency to take more responsibility for positive than negative events, which appears to be a mechanism for enhancing self-esteem (Bradley, 1978; Greenberg, Pyszczynski, Burling, & Tibbs, 1992; Yoshie & Haggard, 2013). However, it has been pointed out that there is a tendency to overestimate one's agency, and that this bias is stronger when the outcome of an action is positive rather than neutral or negative (Haggard, 2017; Wegner & Wheatley, 1999). Because the risk of a potential hit to allies was constantly present in the Moral Decision-Making trials in our task, it is possible that participants in these trials had a reduced SoA and responsibility, and disengaged from the situation due to the risk of negative dramatic consequences of their actions. Alternatively, it is also possible that these results are related to the young adults' lower SoA, which has already been described by Caspar and colleagues (Caspar et al., 2018). However, these results should be taken with caution, considering a flaw in the preparation of the Matlab script used to run the experiment, and the possibility that the time intervals used for the time estimations (200, 500, and 800 ms) were not completely assigned equally across Uncertainty conditions (i.e., that each time interval are shown five times per Uncertainty condition). Indeed, the experiment was designed to generate for each three blocks (corresponding to the three levels of Autonomy) 15 presentations of each interval duration (45 intervals in total) presented in a random order across trials (45 trials/block). Albeit randomly presented across Uncertainty conditions, it is possible that the number of presentations of each interval duration was not perfectly the same across Uncertainty condition. Future research will therefore be necessary to determine whether the SoA decreases when human subjects interact with autonomous systems when making moral decisions compared to situations that do not pose a moral challenge, or whether this result is due to the methodological flaw observed in our experiment.

To summarize, our results show that human decisions in morally-challenging scenarios can be differentially influenced by the recommendations received from different level of autonomous systems, replicating and extending previous findings (e.g., Chen & Joyner, 2009; Rovira et al., 2007 for studies in the military domain). This was measured by the small but significant difference in the number of utilitarian choices between conditions and the decrement in response time in the No Enemy trials. Interestingly, this effect was completed by a decrement in the subjective measure of agency with higher levels of autonomy, which is consistent with previous research (Berberian et al., 2012; Vantrepotte et al., 2022). At the same time, however, our results show some inconsistencies with previous studies (Berberian et al., 2012; Moretto et al., 2011), with less agency measured in the moral decision-Making trials and no significant difference across Level of Autonomy at the implicit level of agency (IB). Thus, further experiments need to be conducted to determine whether the inconsistent results we found were related to the design of the experiment as we suggested above. In particular, in addition to the change need on the Intentional Binding measure, the scenario we used was closer to impersonal/neutral stimuli rather than a moral/emotional context. For example, the radar screen shown to participants was quite schematic and likely did not allow them to properly visualize the context of the choices they were making. In addition, they did not know the number of victims following their decision, and the victims were not clearly presented as individuals. Thus, it could be that the images we used did not have enough emotional content to enhance the SoA. Since the sense of agency could also be influenced by the actions' outcome, which is missing in the current version of the

task, future research should employ a less neutral task and content with more moral and emotional valence to address this issue. However, it is important to acknowledge that our results may be limited by the small sample size. Participants were specifically recruited from their third and fourth years of study to ensure that they had a foundational understanding of International Humanitarian Law, particularly in relation to the legal conduct of armed conflict. This recruitment criterion, combined with the logistical constraints imposed by the strict COVID-19 regulations at the time, resulted in a smaller sample size than would have been ideal for the study's objectives. Therefore, further research with larger samples is needed to confirm and extend our findings. Furthermore, although participants were asked to confirm their choice by pressing the selected response key again - or to change their choice by pressing the other key - only the final confirmed decision was recorded in the present study. Intermediate keystrokes (i.e. the initial versus final choice) were not included in the data set. Therefore, we cannot directly analyse hesitation or change of mind in the current data. As confirmation behavior could serve as a valuable indicator of hesitation in decision making, this possibility should be considered in future studies. To summarize, our study shows that the degree of moral conflict varied depending on factors such as the number of allies impacted by collateral damage and the likelihood of this damage occurring. Future research should systematically investigate how the level of moral conflict interacts with varying degrees of AI autonomy and how these factors influence decision-making processes and the sense of agency. This would provide deeper insights into the dynamics between humans and AI systems in morally complex situations.

## 5. Conclusion

Given the increasing presence of intelligent autonomous systems in our daily lives, it is crucial to conduct further research to better understand the implications in sensitive domains such as the military context to provide input for the successful design of innovative automated systems. Our findings suggest that the level of system autonomy influences participants' moral decision-making and that input received from an intelligent autonomous system influences SoA. By assessing the impact of human-machine interaction on moral decision-making in the military, with the present study we pave the way for further lines of research on the influence of autonomous systems on human moral behavior, considering the current lack of research on this issue.

## Declaration of the use of AI

The authors declare that they have not used any artificial intelligence (AI) tools or software for the work on the submitted manuscript.

## CRediT authorship contribution statement

**Adriana Salatino:** Investigation, Formal analysis, Data curation, Writing – review & editing, Writing – original draft. **Arthur Prével:** Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization, Writing – review & editing, Writing – original draft. **Emilie Caspar:** Supervision, Conceptualization, Writing – review & editing. **Salvatore Lo Bue:** Supervision, Methodology, Funding acquisition, Conceptualization, Writing – review & editing.

## Funding

This research was funded by Belgian Defense – Royal Higher Institute of Defense, grant number HFM 23-03.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence
the work reported in this paper.

## Data availability

The datasets generated and analysed during the current study are available in the Open Science Framework (Salatino, A., Prêvel, A., & Lo Bue, S. (2023, December 12). "Fire! Do Not Fire!": A pilot-new paradigm testing how autonomous systems affect agency and moral decision-making. Retrieved from osf.io/98ycf.

## References

Anderson, E., Fannin, T., & Nelson, B. (2018, September). Levels of aviation autonomy. In *In 2018 IEEE/AIAA 37th digital avionics systems conference (DASC) (pp. 1–8).* IEEE. https://doi.org/10.1109/DASC.2018.8569280.

Arkin, R. C., Ulam, P., & Wagner, A. R. (2011). Moral decision making in autonomous systems: Enforcement, moral emotions, dignity, trust, and deception. *Proceedings of the IEEE, 100*(3), 571–589. https://doi.org/10.1109/JPROC.2011.2173265

Ayoub, J., Zhou, F., Bao, S., & Yang, X. J. (2019, September). *From manual driving to automated driving: A review of 10 years of autisti*. In *Proceedings of the 11th international conference on automotive user interfaces and interactive vehicular applications* (pp. 70–90). https://doi.org/10.1145/3342197.3344529

Bandura, A. (1999). Moral disengagement in the perpetration of inhumanities. *Personality and Social Psychology Review, 3*(3), 193–209. https://doi.org/10.1207/s15327957pspr0303_3

Berberian, B. (2019). Man-machine teaming: A problem of agency. IFAC-PapersOnLine, 51(34), 118–123. https://doi.org/10.1016/j.ifacol.2019.01.049

Berberian, B., Sarrazin, J. C., Le Blaye, P., & Haggard, P. (2012). Automation technology and sense of control: A window on human agency. *PLoS One, 7*(3), Article e34075. https://doi.org/10.1371/journal.pone.0034075

Blackwood, N. J., Bentall, R. P., Simmons, A., Murray, R. M., & Howard, R. J. (2003). Self-responsibility and the self-serving bias: An fMRI investigation of causal attributions. *NeuroImage, 20*(2), 1076–1085. https://doi.org/10.1016/S1053-8119(03)00331-8

Bradley, G. W. (1978). Self-serving biases in the attribution process: A reexamination of the fact or fiction question. *Journal of Personality and Social Psychology, 36*(1), 56. https://doi.org/10.1037/0022-3514.36.1.56

Burin, D., Pyasik, M., Salatino, A., & Pia, L. (2017). That’s my hand! Therefore, that’s my willed action: How body ownership acts upon conscious awareness of willed actions. *Cognition, 166*, 164–173. https://doi.org/10.1016/j.cognition.2017.05.035

Caspar, E. A., Christensen, J. F., Cleeremans, A., & Haggard, P. (2016). Coercion changes the sense of agency in the human brain. *Current Biology, 26*(5), 585–592. https://doi.org/10.1016/j.cub.2015.12.067

Caspar, E. A., Cleeremans, A., & Haggard, P. (2015). The relationship between human agency and embodiment. *Consciousness and Cognition, 33*, 226–236. https://doi.org/10.1016/j.concog.2015.01.007

Caspar, E. A., Cleeremans, A., & Haggard, P. (2018). Only giving orders? An experimental study of the sense of agency when giving or receiving commands. *PLoS One, 13*(9), Article e0204027. https://doi.org/10.1371/journal.pone.0204027

Caspar, E. A., Joumpa, K., Arnaldo, I., Di Angelis, L., Gazzola, V., & Keysers, C. (2022). Commanding or being a simple intermediary: How does it affect moral behavior and related brain mechanisms? *ENeuro, 9*(5). https://doi.org/10.1523/ENEURO.0508-21.2022

Chan, C. Y. (2017). Advancements, prospects, and impacts of automated driving systems. *International Journal of Transportation Science and Technology, 6*(3), 208–216. https://doi.org/10.1016/j.ijtst.2017.07.008

Chavailaz, A., Schwaninger, A., Michel, S., & Sauer, J. (2018). Automation in visual inspection tasks: X-ray luggage screening supported by a system of direct, indirect or adaptable cueing with low and high system reliability. *Ergonomics, 61*(10), 1395–1408. https://doi.org/10.1080/00140139.2018.1481231

Chen, J. Y., & Barnes, M. J. (2012). Supervisory control of multiple robots: Effects of imperfect automation and individual differences. *Human Factors, 54*(2), 157–174. https://doi.org/10.1177/0018720811435843

Chen, J. Y., & Joyner, C. T. (2009). Concurrent performance of gunner's and robotics operator's tasks in a multitasking environment. *Military Psychology, 21*(1), 98–113. https://doi.org/10.1080/08995600802565785

Chialastri, A. (2012). *Automation in aviation*. INTECH Open Access Publisher.

Christensen, J. F., Di Costa, S., Beck, B., & Haggard, P. (2019). I just lost it! Fear and anger reduce the sense of agency: A study using intentional binding. *Experimental Brain Research, 237*, 1205–1212. https://doi.org/10.1007/s00221-018-5461-6

Christensen, J. F., & Gomila, A. (2012). Moral dilemmas in cognitive neuroscience of moral decision-making: A principled review. *Neuroscience & Biobehavioral Reviews, 36*(4), 1249–1264. https://doi.org/10.1016/j.neubiorev.2012.02.008

Cohn, A., Gesche, T., & Maréchal, M. A. (2022). Honesty in the digital age. *Management Science, 68*(2), 827–845. https://doi.org/10.1287/mnsc.2021.3985

Coyle, D., Moore, J., Kristensson, P. O., Fletcher, P., & Blackwell, A. (2012, May). I did that! *Measuring users' experience of agency in their own actions*. In *Proceedings of the SIGCHI conference on human factors in computing systems* (pp. 2025–2034). https://doi.org/10.1145/2207676.2208350

Crockett, M. J., Siegel, J. Z., Kurth-Nelson, Z., Dayan, P., & Dolan, R. J. (2017). Moral transgressions corrupt neural representations of value. *Nature Neuroscience, 20*(6), 879–885. https://doi.org/10.1038/nn.4557

Cushman, F. (2013). Action, outcome, and value: A dual-system framework for morality. *Personality and Social Psychology Review, 17*(3), 273–292. https://doi.org/10.1177/108868313495594

Cushman, F., Kumar, V., & Railton, P. (2017). Moral learning: Psychological and philosophical perspectives. *Cognition, 167*, 1–10. https://doi.org/10.1016/j.cognition.2017.06.008

de Melo, C. M., Marsella, S., & Gratch, J. (2018). Social decisions and fairness change when people's interests are represented by autonomous agents. *Autonomous Agents and Multi-Agent Systems, 32*, 163–187. https://doi.org/10.1007/s10458-017-9376-6

de Melo, C. M., Marsella, S., & Gratch, J. (2019). Human cooperation when acting through autonomous machines. *Proceedings of the National Academy of Sciences, 116*(9), 3482–3487. https://doi.org/10.1073/pnas.1817656116

Dewey, J. A., & Knoblich, G. (2014). Do implicit and explicit measures of the sense of agency measure the same thing? *PLoS One, 9*(10), Article e110118. https://doi.org/10.1371/journal.pone.0110118

Endsley, M. R. (2017). From here to autonomy: Lessons learned from human-automation research. *Human Factors, 59*(1), 5–27. https://doi.org/10.1177/0018720816681350

Endsley, M. R., & Kaber, D. B. (1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task. *Ergonomics, 42*(3), 462–492. https://doi.org/10.1080/001401399185595

Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. *Human Factors, 37*(2), 381–394. https://doi.org/10.1518/001872095779064555

Furlough, C., Stokes, T., & Gillan, D. J. (2021). Attributing blame to robots: I. The influence of robot autonomy. *Human Factors, 63*(4), 592–602. https://doi.org/10.1177/0018720819880641

Goh, J., Wiegmann, D. A., & Madhavan, P. (2005). Effects of automation failure in a luggage screening task: A comparison between direct and indirect cueing. In *Proceedings of the human factors and ergonomics society annual meeting (Vol. 49, no. 3, pp. 492–496).* Sage CA: Los Angeles, CA: SAGE publications. https://doi.org/10.1177/154193120504900

Greenberg, J., Pyszczynski, T., Burling, J., & Tibbs, K. (1992). Depression, self-focused attention, and the self-serving attributional bias. *Personality and Individual Differences, 13*(9), 959–965. https://doi.org/10.1016/0191-8869(92)90129-D

Greene, J. D., Nystrom, L. E., Engell, A. D., Darley, J. M., & Cohen, J. D. (2004). The neural bases of cognitive conflict and control in moral judgment. *Neuron, 44*(2), 389–400. https://doi.org/10.1016/j.neuron.2004.09.027

Greene, J. D., Sommerville, R. B., Nystrom, L. E., Darley, J. M., & Cohen, J. D. (2001). An fMRI investigation of emotional engagement in moral judgment. *Science, 293*(5537), 2105–2108. https://doi.org/10.1126/science.1062872

Haggard, P. (2017). Sense of agency in the human brain. *Nature Reviews Neuroscience, 18*(4), 196–207. https://doi.org/10.1038/nrn.2017.14

Haggard, P., Clark, S., & Kalogeras, J. (2002). Voluntary action and conscious awareness. *Nature Neuroscience, 5*(4), 382–385. https://doi.org/10.1038/nn827

Haggard, P., & Tsakiris, M. (2009). The experience of agency: Feelings, judgments, and responsibility. *Current Directions in Psychological Science, 18*(4), 242–246. https://doi.org/10.1111/j.1467-8721.2009.01644.x

Haslbeck, A., & Hoermann, H. J. (2016). Flying the needles: Flight deck automation erodes fine-motor flying skills among airline pilots. *Human Factors, 58*(4), 533–545. https://doi.org/10.1177/0018720816640394

Imaizumi, S., & Tanno, Y. (2019). Intentional binding coincides with explicit sense of agency. *Consciousness and Cognition, 67*, 1–15. https://doi.org/10.1016/j.concog.2018.11.005

Jeannerod, M. (2003). The mechanism of self-recognition in humans. *Behavioural Brain Research, 142*(1–2), 1–15. https://doi.org/10.1016/S0166-4328(02)00384-4

Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Liang, J., Dodge, J., ... Choi, Y. (2011). *Can machines learn morality? the delphi experiment.* *arXiv preprint arXiv: 1120.07574.* https://doi.org/10.48550/arXiv.2110.07574

Kahneman, D., & Tversky, A. (2013). *Prospect theory: An analysis of decision under risk.* In *handbook of the fundamentals of financial decision making: Part I* (pp. 99–127).

Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005). Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. *Bmj, 330*(7494), 765. https://doi.org/10.1136/bmj.33896.500764.8F

Kneer, M., & Stuart, M. T. (2021). *Playing the blame game with robots.* In * companion to the 2021 ACM/IEEE international conference on human-robot interaction* (pp. 407–411). https://doi.org/10.1145/3434074.3447202

Köbis, N., Bonnefon, J. F., & Rahwan, I. (2021). Bad machines corrupt good morals. *Nature Human Behaviour, 5*(6), 679–685. https://doi.org/10.1038/s41562-021-01128-2

Le Goff, K., Rey, A., Haggard, P., Oullier, O., & Berberian, B. (2018). Agency modulates interactions with automation technologies. *Ergonomics, 61*(9), 1282–1297. https://doi.org/10.1080/00140139.2018.1468493

Leib, M., Köbis, N. C., Rilke, R. M., Hagens, M., & Irlenbusch, B. (2021). The corruptive force of AI-generated advice. *arXiv preprint arXiv:2102.07536.* Doi: 10.48550/arXiv.2102.07536

Liu, P., & Du, Y. (2022). Blame attribution asymmetry in human-automation cooperation. *Risk Analysis, 42*(8), 1769–1783. https://doi.org/10.1111/risa.13674

MacMillan, J., Deutsch, S. E., & Young, M. J. (1997). A comparison of alternatives for automated decision support in a multi-task environment. *Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 41*(1), 190–194. Sage CA: Los Angeles, CA.

Malik, R. A., & Obhi, S. S. (2019). Social exclusion reduces the sense of agency: Evidence from intentional binding. *Consciousness and Cognition, 71*, 30–38. https://doi.org/10.1016/j.concog.2019.03.004

Manzey, D., Reichenbach, J., & Onnasch, L. (2012). Human performance consequences of automated decision aids: The impact of degree of automation and system
experience. *Journal of Cognitive Engineering and Decision Making*, 6(1), 57–87. https://doi.org/10.1177/1555343411433844

Martin, J. W., Buon, M., & Cushman, F. (2021). The effect of cognitive load on intent-based moral judgment. *Cognitive Science*, 45(4), Article e12965. https://doi.org/10.1111/cogs.12965

Mayer, M. (2015). The new killer drones: Understanding the strategic implications of next-generation unmanned combat aerial vehicles. *International Affairs*, 91(4), 765–780. https://doi.org/10.1111/1468-2346.12342

Moore, J. W., & Obhi, S. S. (2012). Intentional binding and the sense of agency: A review. *Consciousness and Cognition*, 21(1), 546–561. https://doi.org/10.1016/j.concog.2011.12.002

Moretto, G., Walsh, E., & Haggard, P. (2011). Experience of agency and sense of responsibility. *Consciousness and Cognition*, 20(4), 1847–1854. https://doi.org/10.1016/j.concog.2011.08.014

Mosier, K. L., & Manzey, D. (2019). Humans and automated decision aids: A match made in heaven?. *In Human performance in automated and autonomous systems* (pp. 19–42). CRC Press.

Pagliari, M., Chambon, V., & Berberian, B. (2022). What is new with artificial intelligence? Human-agent interactions through the lens of social agency. *Frontiers in Psychology*, 13, Article 954444. https://doi.org/10.3389/fpsyg.2022.954444

Parasuraman, R., Molloy, R., & Singh, I. L. (1993). Performance consequences of automation-induced “complacency”. *The International Journal of Aviation Psychology*, 3(1), 1–23. https://doi.org/10.1207/s15327108ijap03011

Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. *Human Factors*, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

Prével, A., Krebs, R. M., Kukkonen, N., & Braem, S. (2021). Selective reinforcement of conflict processing in the Stroop task. *PLoS One*, 16(7), Article e0255430. https://doi.org/10.1371/journal.pone.0255430

Pyasik, M., Salatino, A., Burin, D., Berti, A., Ricci, R., & Pia, L. (2019). Shared neurocognitive mechanisms of attenuating self-touch and illusory self-touch. *Social Cognitive and Affective Neuroscience*, 14(2), 119–127. https://doi.org/10.1093/scan/nsz002

Rovira, E., McGarry, K., & Parasuraman, R. (2007). Effects of imperfect automation on decision making in a simulated command and control task. *Human Factors*, 49(1), 76–87. https://doi.org/10.1518/001872007779598082

Saito, N., Takahata, K., Murai, T., & Takahashi, H. (2015). Discrepancy between explicit judgement of agency and implicit feeling of agency: Implications for sense of agency and its disorders. *Consciousness and Cognition*, 37, 1–7. https://doi.org/10.1016/j.concog.2015.07.011

Salatino, A., Prével, A., Caspar, E., & Bue, S. L. (2025). Influence of AI behavior on human moral decisions, agency, and responsibility. *Scientific Reports*, 15(1), 12329. https://doi.org/10.1038/s41598-025-95587-6

Salatino, A., Prével, A., & Lo Bue, S. (2023, December 12). “Fire! Do Not Fire!”: A pilot-new paradigm testing how autonomous systems affect agency and moral decision-making. Retrieved from osf.io/98ycf.

Sarter, N. B., & Schroeder, B. (2001). Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. *Human Factors*, 43(4), 573–583. https://doi.org/10.1518/001872001775870403

Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020). An overview of clinical decision support systems: Benefits, risks, and strategies for success. *npj Digital Medicine*, 3(1), 17. https://doi.org/10.1038/s41746-020-0221-y

Synofzik, M., Vosgerau, G., & Newen, A. (2008). Beyond the comparator model: A multifactorial two-step account of agency. *Consciousness and Cognition*, 17(1), 219–239. https://doi.org/10.1016/j.concog.2007.03.010

Vaghi, M. M., Cardinal, R. N., Apergis-Schoute, A. M., Fineberg, N. A., Sule, A., & Robbins, T. W. (2019). Action-outcome knowledge dissociates from behavior in obsessive-compulsive disorder following contingency degradation. *Biological Psychiatry: Cognitive Neuroscience and Neuroimaging*, 4(2), 200–209. https://doi.org/10.1016/j.bpsc.2018.09.014

Valdés, R. A., Comendador, V. F. G., Sanz, A. R., & Castán, J. P. (2018). Aviation 4.0: More safety through automation and digitization. In *aircraft technology. IntechOpen*.

Vantrepotte, Q., Berberian, B., Pagliari, M., & Chambon, V. (2022). Leveraging human agency to improve confidence and acceptability in human-machine interactions. *Cognition*, 222, Article 105020. https://doi.org/10.1016/j.cognition.2022.105020

Volz, K. M., & Dorneich, M. C. (2020). Evaluation of cognitive skill degradation in flight planning. *Journal of Cognitive Engineering and Decision Making*, 14(4), 263–287. https://doi.org/10.1177/1555343420962897

Wang, H., Lewis, M., Velagapudi, P., Scerri, P., & Sycara, K. (2009, March). *How search and its subtasks scale in N robots. In proceedings of the 4th ACM/IEEE international conference on human robot interaction* (pp. 141–148). https://doi.org/10.1145/1514095.1514122

Wegner, D. M. (2017). *The illusion of conscious will*. MIT press.

Wegner, D. M., & Wheatley, T. (1999). Apparent mental causation: Sources of the experience of will. *American Psychologist*, 54(7), 480. https://doi.org/10.1037/0003-066X.54.7.480

Wen, W., Yamashita, A., & Asama, H. (2015). The sense of agency during continuous action: Performance is more important than action-feedback association. *PLoS One*, 10(4), Article e0125226. https://doi.org/10.1371/journal.pone.0125226

Wright, J. L., Chen, J. Y., & Barnes, M. J. (2018). Human-automation interaction for multiple robot control: The effect of varying automation assistance and individual differences on operator performance. *Ergonomics*, 61(8), 1033–1045. https://doi.org/10.1080/00140139.2018.1441449

Yoshie, M., & Haggard, P. (2013). Negative emotional outcomes attenuate sense of agency over voluntary actions. *Current Biology*, 23(20), 2028–2032. https://doi.org/10.1016/j.cub.2013.08.034

Zanatto, D., Chattington, M., & Noyes, J. (2021). Human-machine sense of agency. *International Journal of Human-Computer Studies*, 156, Article 102716. https://doi.org/10.1016/j.ijhcs.2021.102716