# An Organizational Study into the Concept of Automation in a Safety Critical Socio-technical System

Paola Amaldi, Anthony Smoker

► To cite this version:

Paola Amaldi, Anthony Smoker. An Organizational Study into the Concept of Automation in a Safety Critical Socio-technical System. 3rd Human Work Interaction Design (HWID), Dec 2012, Copenhagen, Denmark. pp.183-197, 10.1007/978-3-642-41145-8_16. hal-01463387

HAL Id: hal-01463387

https://inria.hal.science/hal-01463387

Submitted on 9 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An organizational study into the concept of automation
in a safety critical socio-technical system

Paola Amaldi¹ and Anthony Smoker²

¹ School of Life and Medical Sciences, Department of Psychology; University of Hertfordshire Hatfield AL10 9AB England; ² NATS; Mailbox 50; Swanwick Centre; Southampton, SO31 7AY;

p.amaldi@herts.ac.uk; anthony.smoker@nats.co.uk

**Abstract.** Although automation has been introduced in all areas of public life, what seems to be missing is a reflection at the organizational or societal level about a *policy of automation*. By this we intend appropriate declarations made at the level of rationale, future plans and strategies to achieve intended goals and most importantly how those achievements will impact on various aspects of societal life, from legal responsibilities to moral and socio economic issues. In some public spheres these issues are becoming quite controversial because automation opens up possibilities of profound structural re-organization; however, we lack a discussion across and within different work domains to help us review methods or even methodological principles needed to gather and organize knowledge towards the construction of automation policies. This paper uses the UK service organization for Air Traffic Management Domain called NATS - National Air traffic Service, as a case study to illustrate an example of an organization currently undertaking critical self-reflection about automation policy or lack of such, along with the illustration of some unresolved deep concerns raised by the development, introduction, and continued use of automation.

**Keywords:** Policy of automation, organizational culture.

## 1 Introduction

Although automation has been introduced in all areas of public life, from production to tertiary sectors, what seems to be missing is a reflection at the organizational or societal level about a *policy of automation*. By this we intend a declaration of rationale, future plans and strategies to achieve intended goals expressed not longer at a single mission level such as gate-to-gate trajectory management, but how those achievements will impact on various aspects of societal life, from legal responsibilities to moral and socio economic issues. While in some public spheres these issues are becoming quite controversial because automation opens up possibilities of profound structural re-organization, we lack a discussion across and within different work domains that help us to review methods or even methodological principles needed to gather and organize knowledge to the construction of such policy. For example, cur-
rent concerns about the deep changes to be introduced in the British Public Health
sector would likely benefit from a more open discussion about the relationship be-
tween automation and higher level societal goals (see [1] Rozzi, Amaldi and Kirwan,
2010).

This paper uses NATS—National Air Traffic Service, the UK Agency for air traffic management as a case study to illustrate an example of an organization currently undertaking critical self-reflection about automation policy along with the illustration of some unresolved deep concerns raised by the development, introduction, and continued use of automation. Before discussing NATS-specific concerns we shall briefly review major issues within the cognitive ergonomic literature about the relationship between automated processes and human control.

## 1.1 Major pitfalls of the automation process

Information and computer technologies provide an increasing number of opportunities to develop new solutions to assist operators/professionals across many domains of practice in managing complex socio-technical systems. There are, however, a number of concerns highlighting complexities and paradoxes embedded in the automation process. Bainbridge [2] discussed the unexpected consequences of technology-driven automation, which often relies on human reliability to be safely operated. One of the “ironies” is that automated systems are often introduced on the ground that humans are less reliable than automation because of “intrinsic” limitations in their ability to monitor for unexpected, unsafe events in a stream of a routine flow of events.

Paradoxes linked to the introduction of expert systems have been extensively em-
phasized ([3, 4, 5, 6]). Inappropriate design choices might result in an increase of
operators' workload, in an excessive demand on working memory, in a difficulty to
co-operate with team members, and finally it might slow down the development of
expertise [7]. Automation is expected to assist operators in achieving the overall sys-
tem goals in a more cost-effective way. These expectations have, at times, relied on a
number of misbeliefs [8]. In fact, although it might lead to “de-skilling”, automation
does not decrease the requirements for expertise. Some have nevertheless been led to
believe that expert systems can replace the need for or even decrease the standard of
expert operators.

This claim does not consider that automation has to be constantly adapted to its op-
erational context to be effective [9]. This is because expert systems have a limited
scope with respect to the variety of objectives characterizing activities in complex
socio-technical systems. This variety reflects the ability of expert operators to identi-
fy ways to improve the system performance in routine situations. The role of opera-
tors when interacting with complex technical system has often been emphasized in
relation to their ability to manage exceptions. While this is true, it should not be ne-
glected the finding that operators systematically go beyond the prescribed practices to
enhance system's efficiency [10]. An old study involving observations of mainte-
nance operators, reported that almost one third of the times operators have been ob-
served making an informal use of available tools [11]. By informal it is meant that
the tool is not used for the purposes it was designed but rather with the intent of making the corrective action more effective.

This phenomenon has already been well documented and studied within the francophone ergonomic tradition quite long ago [11, 12, 13]. The informal use of tools and procedures reflect often a search for an improved efficiency, not just a solution to unexpected problems. Similarly, the spontaneous generation of linguistic code has been observed in different operational settings [12]. Notice that deviations from standard communication patterns have generated fatal misunderstandings such as in the case of air traffic controller / aircraft pilot radio communications [14]. In spite of these fatal accidents, deviations from standard use are often generated with the aim of achieving task goals. In this respect, automation should support operators in finding the best way to achieve the goals while limiting the negative consequences of possible misfits between tools adaptations and task constraints and goals [9]. As part of the integration process operators will engage in «finishing the design» of the tool with respect to its original intended use [15], i.e., improving the fitness between the tool and the complexities of the operational environment.

## 1.2 Automation and information processing stages

In an attempt to characterize automation with respect to models of information processing, Parasuraman, Sheridan & Wickens, [16], have used a simplified model of human decision making and problem solving. The model allows the classifying of technological innovations according to four stages:

- (i) information acquisition ;

- (ii) information analyses ;

- (iii) decision selection;

- (iv) action implementation.

The introduction of new technology might interact with cognitive processing in each of these stages with differing degrees of automation. The allocation of tasks to humans and machines depends then on the level of automation chosen. While the classification schema does offer a means to group technological innovations across different domains, guidelines for function allocation do not seem straightforward. Criteria for deciding how to do task sharing have to be based on an understanding of the impact of automation on the targeted communities of practitioners. Neglecting, for example, the crucial role of cooperation or adaptation processes like "finishing the design" does not seem a very promising start for deciding on task allocation. Further, such a simplified model of human cognition and of human-computer interaction might mislead designers and engineers to believe that a fairly simple algorithm can generate the desired answer to a very complex and still unsettled issue (see also [17]). Some examples discussed below will illustrate the pitfalls of such naïve assumption.

For example, automation concerning the (i) acquisition and (ii) information analysis stages involves the organization of incoming sensory data. A stated purpose is to decrease attentional demands of operators by highlighting or cueing relevant information while leaving the rest un-cued but still accessible. Yet, this apparently simple solution neglects considering a number of issues. The assumption here is that human
information processing capacity is limited and thus the number of items that can be
processes at anytime cannot exceed that capacity. While this is not wrong, this state-
ment neglects considering that there is no an obvious way of “measuring” that capaci-
ty as it is subject to people’s expertise, organization of labor and the development of
new working practices. In addition filtering “relevant” information raises the issue of
“context sensitivity” [18]. What needs to be noticeable depends on the situation,
which includes other related data, the “history” of the process, the intentions and
expectations of the observers, [19].

A higher level of automation within the information analysis stage implies the
temporarily or permanently hiding of certain information. For example in Air Traffic
Control (ATC), certain electronic displays of future traffic problems “hide” or “re-
veal” information according to the role of operator within the team. Or, the available
data might be automatically organized in terms of problems to be dealt with in a given
priority order. Notice that information filtering, problem formulation and priority
assignment, all involve anticipating how the system under control is going to evolve.
Automation of some anticipatory functions is then involved in the design of predictor
displays introduced in both the flight deck and ATC to assist operators to project
future courses of flight.

Automation interacting with the third stage of decision making leads to the selec-
tion of a course or several courses of actions. Automation here might assist operators
in calculating the best option(s) given the constraints of the current situation. For
example the Flight Management System (FMS) in the cockpit can, more effectively
than pilots, calculate the most cost-effective trajectory in terms of gas consumption
and timing.

In ATC, decision aids assist controllers by offering solutions to traffic problems
and in this respect several systems have been proposed and evaluated (e.g., ERATO,
HIPS, URET, IFACTS) [20]. At this level of automation a range of alternatives are
proposed, leaving operators responsible for making the final choice. A more ad-
vanced automation would give very little or no choice to operators as to what solution
to implement. This implies automating the process of evaluating costs and benefits
associated with each alternative. The problem is that the criteria used in the automat-
ed evaluation process are not likely to include all of the factors included by human
decision-makers. In fact there will always be a number of conditions where the auto-
mated solution would need to be adjusted to reflect local contingencies. Therefore it
seems crucial that a high degree of automation at this stage of decision-making leaves
open the possibility of deciding whether or not to implement the course of actions.
For example a number of studies on the onboard warning called Traffic Collision and
Avoidance System (TCAS) have shown that pilots do not always comply with the
advice provided by the automation ([21] Amaldi, under review; [22]) unless they can
verify its compatibility with other conditions. Notice that improving an understand-
ing of the criteria underlying the solution proposed facilitates a complying behavior
([23, 24]).

Automation intervenes in the last stage of decision making through the implemen-
tation of the course of actions. For example digital data link will allow air traffic
controllers to uplink a pre-edited clearance into the plane's FMS. Notice that the
clearance could be a computer-selected option to an automatically identified traffic
problems. Current proposals to uplink ACAS advisories to FMS are another example
automaton taking over the decision making and implementing.

1.3 What is automation for?

What is then automation? It is a transformation of a world state accomplished by
an electro mechanical device with or without human intervention. Our previous dis-
cussion aims at classifying the levels of the involvement and mode of human-
automation interaction with respect to controlling and decision-making.

What is automation for then? It is the means by which we (i) extend our cognitive skills; (ii) aim at increasing the resilience of the operational system (by introducing for example, back up sub-components); (iii) aim at increasing productivity by enabling the system with new tools that increase the throughput.

What can be automated? There have been cases, in the history of R&D in Air Traf-
fic Management when (over) ambitious automation projects have been withdrawn.
Lack of mature technology or too many contingencies that made it impossible to
proceduralise operational practices. One apparent rational was that whatever could be
made faster and more reliable through the use of automated device, it should.

For a few decades now, ATM operations have been the object of R&D efforts to
make them faster and more reliable through the introduction of automation. There was
a lack of consideration of the wider impact of those innovations. For example, in-
creasing traffic throughput en route was not connected with the need to increase air-
port capacity. Increasing airport capacity, however, has been the target of serious
environmental concerns.

Starting on the assumption that the complexities of current system require automated
aids, the coupling of computers to air traffic management has surely resulted in an
increase of safety and productivity. Nowadays targeting individual human limitations
with respect to system control is not longer a viable strategy for expanding current
business. The main challenge seems to have shifted from designing interfaces usable
or trustable (although these are still serious concerns) to mapping out the added com-
plexities and the profound consequences of the technological innovation process. For
example, recent debates by environmentalists have challenged that Civilian Air Traf-
fic is an important contributor to CO2 emission. To what extent, then the design of
new technology should be planned to address issues of atmospheric pollution? The
main point to be raised is that the human-computer interaction unit of analysis has to
be embedded in a larger context to target limitations and contradictions of the entire
system, rather than marginalizing the human as the ‘limiting factors’ to system devel-
opment.

2 Organizational culture in NATS

This is a study about NATS organizational culture. In particular what are the existing
views and expectations held by the middle layer management, toward the increasing
dependency on digitized information processing systems? By doing this investigation we wish to articulate the model of cultural analysis suggested by [25] and a number of researchers working in the area of organizational cultural analyses (e.g., [26]). Culture, safety and safety-culture, have been treated as 'components' of systems and as such discussed as either the source or the cause of behavior. Further isolating components prompts researchers to treat them as measurable and manipulable to control on their 'effects' on system behavior. We are basically seeking to advance our understanding of the main claims of what a cultural analysis is NOT: (a) culture as causal attitude as the engine that pushes processes; (b) culture as engineered organization, proposing a set of indicators that verify the cultural recommendations have worked. Rather, we support the view that culture is to be understood as in a dialectic relation with practice, one cannot be constituted as an object of study without the other. Their relationship is not one of cause-effect but rather one of mutual dependency. In other words, we can't study safety culture without inquiring into those regular patterns that characterize organizational behavior. The safety culture literature, on the other hand, hardly makes reference to the following features of organizations: (i) power, (ii) group interest, (iii) conflict or (iv) inequality.

These features (organisational) are feeding the following cultural schema and interpretative mechanisms:

- Normative heterogeneity

- Competitive and conflicting interests

- Inequalities in power and authority

What is missing from a number of account on safety culture is a focus on process that produce systemic meanings where isolated factors like understaffing, excessive workload, lack of effective communication are seen as a constituent part of a general pattern ... Within the interpretative schema of a cultural analysis, these factors are not caused by a 'wrong culture' but they are a constitutive part of it, give and take meaning from it. They become 'dysfunctional' only when clashing with public images or other competing interests. Manipulating these factors as though they are independent from the historical-cultural context that did not simply *produce* them, but from which they derive their intelligibility and at the same time 'feed in' a more general patterns, is not very promising. Normalizing deviance, informational secrecy, credibility gap, are examples of mechanisms that constitute cultural schema.

Is NATS enacting a moment of critical self-reflection to unsettle what is taken for granted and make space for innovative practice?

## 2.1 NATS main concerns with automation

NATS is currently reviewing their position and their implicit assumptions with respect to automation. At this stage NATS is seeking views in the face of unexpected side effects linked to increasing complexities from all parties involved in the design, implementation and use of the existing or planned automated systems. Such process of critical self-reflection aims at enhancing its resilience in the face of increasing complexities linked to ongoing technological innovation. The notion of organizational resilience has become popular in the area of organizational risk management [27]. The more an organization builds its own resilience, the more is capable of adaptively
responding to hazardous events. We extend the notion to situations where the plan-
ning of far reaching changes cannot indeed account for in advance for all of its major
contingencies. Given the increased complexity introduced by more and more power-
ful technology, NATS attempts to move from a rather patchwork to a more holistic
approach seems worth reporting. In the following we report a two-stage data collec-
tion aiming to document NATS main concerns, suggestions, recommendations to the
problem of lack of automation policy.

3 Method

The main objective of the data collection was to elicit subject matter experts a wide as
possible range of issues deemed to be associated with past practices of automation
development and introduction into air traffic control management. Data collection
occurred in three stages.

First in December 2011 we devised a survey and distributed it mainly to NATS of-
ficials. Three main themes were suggested as guidelines but then participants were
encouraged to raise any other issue and think about in terms of

a) What the problem was—the problem statement

b) Why was it a problem

c) What needs to be done about it

The guidelines for reflection were centered around three main themes:

i) What is NATS scope and vision for automation;

ii) What is the role and responsibility of the human;

iii) What are the skills that need to be developed and maintained.

Second, we organized a Workshop in April 2012 attended by approximately 70 peo-
ple partly from NATS and partly from a number of disparate industries. Participants
were asked to participate in 4 activities designed to encourage constructive and crea-
tive thinking about automation and how it should be developed, deployed and utilised.
For each of these activities the following provides a brief definition of their aim and
the opening question used to initiate the discussion

Activity 1: *Reversal*. Aim: elicit recommendations for improvements through iden-
tifying weaknesses. This is done by asking the opposite of the question you want to be
answered, and then by reversing the results as appropriate.

Opening question: *What things can we do to make the introduction of Automation less likely to succeed and less likely to be safe? How could we make it worse?*
**Activity 2: Reframing Matrix<fnref n="1" />** Aim: Looking at problems from different perspectives. The proposed perspectives were Pilot; ATCO; SRG and NATS although groups were free to select their own perspectives.

Opening question: How should we prepare for the introduction of Automation in NATS?

**Activity 3:Brainstorming:** Aim: Participants were asked to elicit solution to the following:

Opening question: What actions should we take to ensure Automation is introduced safely and avoid the problems we have discussed?

**Activity 4: Force Field Analysis<fnref n="2" />:** Aim: Understanding the pressures for and against change. Participants were asked to select ideas from their brainstorms for force-field analysis.

The stated objectives of the 4 activities were:

* To validate the draft 'Use of automation in NATS operations' position paper by exposing it to expert scrutiny

* To identify enablers and blockers to the delivery of an effective policy on automation within NATS

* To identify problems with introducing automation within NATS and create potential solutions to these problems

* To engage experts (from around NATS, the UK and the world from Aviation, Regulation, Medicine, MOD, Academia etc.) to form a community of expert resource

* To identify the risk landscape regarding automation and thus provide a metric against which safe introduction of automation can be assessed

Finally a third workshop was organized in July 2012 and attended by 25 participants, approximately. A well-known domain expert was invited as well to review and comment on main challenges/issues of automation. Participants had to rank the priority of the fourteen problem statements that were reviewed in Workshop 1. Further they were asked to write statements about potential negative outcomes of automation, along with their mitigation. Last they were presented with two definitions about automation and asked to comment on them. All comments have been transcribed.

<footnote n="1">¹ Adapted from http://www.odi.org.uk/resources/details.asp?id=5221&title=reframing-matrix</footnote>

<footnote n="2">² Adapted from http://www.mindtools.com/pages/article/newTED_06.htm</footnote>
# 4 Findings

## Stage One: Generating problem statements

In reply to the survey, NATS has compiled 14 statements that we grouped into 6 groups.

Group Statement 1. Lack of definition/vision: There is not an agreed set of definitions about the scope of automation, i.e., to what extent is mostly technology- or problem-driven. This results in confusion and lack of clarity in planning and communication. Different people have different expectations and different requirements regarding what automation will deliver. Similarly there is not a single agreed vision for automation. The scope of automation needs to be defined and agreed. Automation affects every aspect of the business – it determines how people are selected and trained; how many people remain in the system and NATS' capacity to generate income. There is no clear definition of the future levels of automation that NATS should be planning for.

Group Statement 2. Responsibility and role allocation. No single clear picture of how automation will affect MOPS<fnref n="3" /> – in particular the responsibility of the operational staff for the decision making process. The literature previously reviewed suggests how automation can interact with the problem solving and decision making process. The introduction of new automated technology will affect the role of the human. It is vital that human strengths and vulnerabilities are accounted for in the design and attribution of roles. Also, it is vital that the resulting role is one that can be trained for.

Assumptions about role allocation are being made at the moment and are affecting how NATS plans and implements projects but these assumptions are not being made explicit.

The allocation of responsibility between the machine and the human needs to be defined clearly and explicitly over time and at each key milestone of system operation.

Group Statement 3. The introduction of automation will be neither as safe nor as effective as it could be. Automation could be used to remove risks from the current operation – unless this is done in a focused way (aimed at specific known risks) the full benefit will not be realized and, in fact, automation may add risks. Automation needs to be focused upon removing key risks from the operation and exploiting the different strengths of the human and the machine.

The operational effectiveness of our systems relies heavily upon the close relationship between the human and the machine – if this is not optimized then maximum effectiveness will not be realized. Automation needs to be focused upon achieving the most effective balance between human & machine. The cost/benefit balance of automation needs to be managed.

Group Statement 4. There is no clarity on how the relationship between the human and machine will change due to technical failure (or cyber-attack). Current assumptions

<footnote n="3">³ Minimum Operational Performance Standard</footnote>
tions regarding the capacity of the human to revert to manual operations are likely to prove incorrect after a short while of automated operations. If this is the case, we might not have a mode to revert to.

It is difficult to place limits on the extent to which automated systems should be implemented in order to ensure that they ultimately remain under human control. In general, the greater the level of automation the further the human is removed from the control loop and therefore the harder it is for them to recover control. The skills that the human will need to exercise in order to effectively participate in Human-Automation interaction need to be identified and the impacts of automation anticipated.

Group Statement 5. Aspects of human behavior indicative of their ability to effectively use future automated systems are not receiving the emphasis required. We do not yet know the number of people and the types of skills/capabilities we will need to provide for the future ATM changes. These have been planned over the next few years and will require current controllers to significantly adapt their ways of working. The extent to which they will need to be helped to do this will depend on NATS ability to effectively assess their automation “competence”.

Group Statement 6. There isn't yet an agreed and validated methodology for assuring the performance of the automated system (cooperative performance of human and automated technical system). Co-ordinative/co-operative requirements are neglected. As automation levels increase, the complexity of the emergent system interactions will also increase. Traditional methods of analysis and validation are unlikely to provide sufficient assurance that the system will be stable. It will be necessary to set and measure demanding performance standards for the total system.

NATS is planning for levels of automation that have not yet matured into operational systems. They might not mature. Automation of human-centered socio-technical systems has far reaching consequences that can be framed only at an organizational/societal level (see [28]).

## Stage Two: Automation Workshop.

There was considerable overlap between the statements generated in the two stages, so analysis of respective contents and overlap is ongoing. At the moment we have compiled a table including the frequency at which a number of activities have been suggested in order to cope with present and future challenges of automation.

<table><caption>Table 1. Most commonly elicited activities to cope with automation challenges</caption><thead><tr><td>Proposed Activity</td><td>Votes</td></tr></thead><tbody><tr><td>Define the future role of the controller</td><td>15</td></tr><tr><td>New aviation system model</td><td>13</td></tr><tr><td>Improve training strategy and delivery and design for automation</td><td>12</td></tr></tbody></table>
<table><tr><td>Proper R&D Phase</td><td>11</td></tr><tr><td>Produce NATS led industry strategy</td><td>10</td></tr><tr><td>Training adequate and appropriate</td><td>9</td></tr><tr><td>Understand system needs, requirements and levels of automation</td><td>9</td></tr><tr><td>Recruit and employ people with the right skills for the new operation</td><td>8</td></tr></table>

## Stage Three

About 25 participants were asked to rate the importance and priority for solving 14 claims used in Stage 1. Those have been grouped into 4 main themes: (i) agreed upon vision of automation; (ii) role played by human over time; (iii) allocation of responsibility between human and artificial agents over time; (iv) supervision and leadership in team working. While they were all considered 'priority', only the lack of an agreed vision was judged an issue to be addressed immediately. Next, participants were asked to make suggestions as to how to mitigate on the consequences of some of the points raised in Stage II concerning potential negative outcomes. Ten major themes were summarized along with the most commonly cited mitigation strategies.

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>
<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

# 5 Discussion and Conclusion

We looked at NATS current critical reflection upon automation and its consequences as a case study of engineering resilience against unexpected and undesirable effects of automation. Given the initial stage of the work, our conclusion takes the form of a working hypothesis to be further confirmed. Given the increasing complexities of socio technical systems, traditional HCI and human-automation interaction issues cannot be handled outside a general framework of automation policy. This includes a set of goals, values, costs and strategies to cope with uncertainties and unintended effects of automation. First, NATS needs to address more explicitly what the long term and scope of automation is going to be. This goes beyond a piecemeal approach where automation innovation would be technology and task driven. The
latter means that two main rationales for introducing automation i.e., availability of
the technology and a focus on a specific (set of) task are inadequate. In fact the as-
sumption that interventions on subcomponents of the system do not need to consider
the long-term effects on the operations in their ensemble, seems under scrutiny. How-
ever as we stated, as automation level increases, the complexity of the emergent sys-
tem interactions will also increase and traditional methods of analysis and validation
are unlikely to provide an overall assurance of system stability.

Second given a better specification of NATS requirements and expectations about automation, what will be the range of roles that humans are expected to engage with? The history of automation both in the cockpit and on the ground has shown that humans act as ‘mediators’ [29] between the automation and the environmental contingencies and operational complexities. Typically operators have to monitor for unexpected interactions among apparently unconnected subcomponents. Further they need to reconcile the need for standardization (like the European Sky) with the need to locally tailor tools and procedures.

Third, what sort of competencies will be required; how they will affect the selec-
tion process; the training and the maintenance of them has to be informed and guided
by a policy of automation. This goes beyond the specific characteristics of the system
in use. For example, a recent study by [30] has documented that in the case of a Na-
tional Service Provider, a vision of automation has affected decisions about changes
on a specific interface and thus about competencies required in an interim phase.

Fourth, safety has reached level that can be hardly improved through the develop-
ment of further safety nets alone. Rather the roles played by the various agents across
the organizational levels of control have to be openly discussed, identified and de-
fined.

Designing for human capabilities as modeled by certain computational/cognitive theories of mind, might be misleading because these theories are not sufficiently concerned with how the meaning of symbols and symbol manipulations is grounded in the goals, constraints and possibilities of the task domain ([31], Dowell and Long, 1998, p. 132). Rather the aim should be designing for automation that is fit for purpose, where ‘purpose’ is defined by the joint human- technical system. Focusing on requirements of either one misses the fundamentally interactive nature of human work design.

References

1. Rozzi, S., Amaldi, P. & Kirwan, B. (2010) *IT Innovation and its organizational condition in safety critical domains*. In System Safety 2010, 5th IET International Conference Manchester.

2. Bainbridge, L. (1986). Ironies of automation. In L.P. Goodstein, H.B. Andersen, & S.E. Olsen (Eds) *Tasks, errors and mental models*. London: Taylor & Francis.

3. Billings, C.E. (1991). *Human-centered aircraft automation : A concept and guidelines (NASA Tech. Memorandum 103885).* Springfield, VA : National Technical Information Service.

4. Norman, D.A. (1988) *The psychology of everyday things*. New York: Basic Books
5. Woods, D.D., Cooks, R.I., Billings, C. (1995). The impact of technology on physician cognition and performance. *Journal of Clinical Monitoring*, 11, 5-8

6. Sarter, N. D., & Woods, D.D. (1997) Teamplay with a powerful and independent agent. In *Human Factors*, **39**(4), pp. 553-569.

7. Woods, D.D., Sarter, N. & Billings, C (1997). Automation surprises. In G. Salvendy (ed.) Handbook of human factors and ergonomics. N.Y: Wiley Interscience Publication.

8. Mosier, K.L., & Skitka, L.J. (1996). Human decision makers and automated decision aids: Made for each other? In R. Parasuraman & M. Mouloua (eds.) *Automation and human performance. Theory and applications*. Mahwah, N.J.: LEA

9. Vicente, K. (1999). *Cognitive work analysis*. Mahwah, N.J.: Lawrence Erlbaum Associates

10. Wright, P. & McCarthy, J. (2003) *Analysis of procedure following as concerned work*. In E. Hollnagel (ed.) Handbook of cognitive task design. LEA: Mahwah, N.J.

11. Leport, B. (1970) Les utilisations des outils et la fiabilité de l'organisation. Rapport interne à la CCE Roneó, Paris.

12. Cuny, X (1979). Different levels of analyzing process control tasks. In *Ergonomics* **22**, pp. 425-526.

13. De Keyser, V. (1991). Works analysis in French language ergonomics: origins and current research trends. In *Ergonomics*, **34**(6), 653-669.

14. Mell, J. (1993). Emergency calls: messages out of the «blue», In *Le Traspondeur*. **11**, INGENAC Toulouse, France.

15. Rasmussen, J. (1986) *Information processing and human-machine interaction. An approach to cognitive engineering*. Elsevier Science.

16. Parasuraman, R., Sheridan, T. B., Wickens, C. D. (2000). A Model for Types and Levels of Human Interaction with Automation. *IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans*, Vol. 30, N0 3, pp. 286-297.

17. Dekker, S.W.A. and Woods, D. D. (2002). MABA-MABA or Abracadabra? Progress on human-automation co-ordination. In *Cognition Technology & Work* **4**: pp. 240-244.

18. Woods, D.D., Patterson, E.S., and Roth, E.M. (2002). Can we ever escape from data overload? A cognitive systems diagnosis. In *Cognition, Technology, and Work*, **4**(1): 22-36.

19. Burns, C.M., Mumaw, R. J., Roth, E.M., & Vicente, K.J., (2000). There Is More to Monitoring a Nuclear Power Plant Than Meets the Eye. In: *Journal of Human Factors and Ergonomics*, Vol. 42.

20. Mendoza, M. (1999) Current state of ATC Conflict Resolution. Eurocontrol: EEC Note 12/99 available at http://www.eurocontrol.fr/public/reports/eecnotes/1999/12.pdf

21. Amaldi, P. (under review); The integration of alert devices into socio-technical systems. The case of an airborne alert device (ACAS).

22. Garfield, D. & Baldwin, T. (2004). European ACAS Operational Monitoring 2002 Report. Eurocontrol EEC Report No. 393. Available at http://www.eurocontrol.int/eec/publications/eecreports/2004/393.pdf

23. Lees, M.N. & Lee, J.D. (2007). The influence of distraction and driving context on driver response to imperfect collision warning systems. In *Ergonomics*, **50** (8), pp. 1264-1286

24. Pritchett, A. R. & Hansman, R.J. (1997). Pilot non-conformance to alerting system commands during closely spaced parallel approaches. In *Digital Avionics System Conference. 16th DASC. AIAA/IEE*

25. Silbey, S. (2009). Taming Prometheus: Talk about safety culture. *Annual Review of Sociology*, **35**, pp. 341-369.

26. Bergström, J., Dekker, S., Nyce, J.M. & Amer-Wählin, I. (2012). The social process of escalation: a promising focus for crisis management research. In *BMC Health Services Research*, **12**(161). Available at http://www.biomedcentral.com/1472-6963/12/161, last accessed October 2012

27. Hollnagel, E., Paries, J. Woods, D.D., & Wreathal, J. (2011). Resilience Engineering in practice: A guidebook. Farnham: Ashgate
28 Rasmussen J, Svendung I. (2000) Proactive risk management in a dynamic society. Rad-dningsverket, Sweden: Swedish Rescue Services Agency.

29 Downer, J (2009). When failure is an option: Redundancy, reliability and regulation in complex technical systems. LSE, Discussion paper 53.

30 Amaldi, P., & Rozzi, S. (2012) Inter-organizational safety debate. In *International Journal of Socio-Technology and knowledge Development*, **4**(1) pp 30-47.

31 Dowell, J., and Long, J. (1998). Conception of the cognitive engineering design problem. In *Ergonomics*, **41**(2), pg. 126-139