# A Norm Compliance Approach for Open and Goal-Directed Intelligent Systems

Authors: Patrizia Ribino; Carmelo Lodato
Journal: Complexity
 (2019)

---

## Abstract

The increasing development of autonomous intelligent systems, such as smart vehicles, smart homes, and social robots, poses new challenges to face. Among them, ensuring that such systems behave lawfully is one of the crucial topics to be addressed for improving their employment in real contexts of daily life. In this work, we present an approach for norm compliance in the context of open and goal-directed intelligent systems working in dynamic normative environments where goals, services, and norms may change. Such an approach complements a goal-directed system modifying its goals and the way to achieve them for taking norms into accounts, thus influencing the practical reasoning process that goal-oriented systems implement for figuring out what to do. The conformity to norms is established at the goal level rather than at the action level. The effect of a norm that acts at the goal level spreads out at the lower level of actions, thus also improving system flexibility. Recovery mechanisms are provided to face exceptional situations that could be caused by normative changes. A case study in the field of the business organizations is presented for demonstrating the strengths of the proposed solution.


---

## Body

## 1. Introduction

Intelligent systems are increasingly used with some degree of autonomy. Thus, their behaviour is not entirely defined by the designer, but it is the result of cognitive capabilities. Modern intelligent systems can determine by themselves the behaviour to adopt for achieving their objectives. In doing so, they could be involved in behaviours that in real-world life are regulated by more or less stringent norms that could produce different effects. For example, smart cars have to obey the city traffic laws. Intelligent information systems have to comply with data protection laws to manage private information. Social robots have to respect social norms during interaction with humans. Smart workflow management systems have to respect business rules to perform business processes and so on.A growing issue in the field of artificial intelligence is how to ensure that the behaviour of intelligent systems complies with the normative environment they should operate so that these systems can be well-accepted and efficiently employed in real contexts of the everyday life [1].In conventional approaches, norms are fully specified at design-time, and the system is designed in such a way that its behaviour does not violate such rules. These approaches, based on hard-coded static norms, are not practical solutions because all the possible situations that a system has to manage should be established at design-time. Otherwise, system redesigning is necessary [2]. Most advanced approaches use model-checking techniques for verifying the system behaviour off-line. Such methods result in impracticable or limited when the autonomy of the system increases making its behaviours not entirely predicted. Moreover, because normative environments in which systems operate are increasingly dynamic, to avoid the shutdown of the system and its reconfiguration, norm compliance has to be guaranteed at run-time.In this paper, we present an approach for ensuring system run-time compliance with a dynamic set of norms in the context of open and goal-directed systems. The conformity with norms is guaranteed at a higher level of abstraction (i.e., the goal level). Such an approach complements a goal-directed system modifying its goals and the way to achieve them for taking norms into account, thus influencing the practical reasoning process [3] that the goal-oriented systems implement for figuring out what to do. In our approach, we also faced the problem to manage normative changes during system execution that may produce system incoherence with existing norms providing recovery mechanisms. In this work, we refined some theoretical foundations that have been preliminarily presented in [4], and we introduced new ones for defining the algorithms that implement the proposed approach. A widely known case study in the context of business organizations [5] is also presented for illustrating the behaviour of a goal-oriented system that implements our algorithms.The rest of the paper is organized as follows. Sections2 and 3 present some grounding literature and the theoretical background of the paper, respectively. In Section 4, a general overview of the approach is presented. Sections 5 and 6 present the key concepts and the algorithms for normative reasoning. Section 7 illustrates a case study in the context of business organizations. Finally, in Section 8 conclusions are drawn.


## 2. Related Works

Norms like obligations, permissions, and prohibitions have been implemented in automatic systems to specify (un)desired or (un)lawful behaviours. Normative systems [6] are commonly defined as systems that specify every possible transition, whether or not that transition is considered to be legal or not. They determine which actions or which states should be achieved or avoided [7–9].Much work has been done about normative frameworks in the field of Electronic Institutions or Virtual Organizations where norms have found a natural implementation. Only to cite a few, Alechina et al. [10] present a programming framework for developing normative organizations based on N-2APL, a BDI-based agent programming language supporting normative concepts such as obligations, prohibitions, and sanctions. In such a work, the interaction between agents and the environment is regulated by a “normative exogenous organization” which is defined using a set of conditional norms. A norm-aware deliberation approach is also proposed. It allows agents to determine the set of plans (adopted for satisfying a goal) of highest priority which does not violate higher priority prohibitions. In [11], Kollingbaum and Norman proposed the Normative Agent Architecture (NoA). It supports the implementation of norm-governed practical reasoning agents. NoA agents are motivated by norms to act. In the NoA language, all the effects of a plan are declared in a plan specification. These effects are considered by agents for reasoning about plan selection and execution. Moreover, the norms governing the behaviour of an NoA agent refer to actions that are obligatory, permitted, forbidden, or states of affairs that are obligatory, permitted, or forbidden. The NoA language enables an agent to be programmed in terms of plans and norms. Normative statements formulated in the NoA language express obligations, permissions, and prohibitions of an agent. In [12], the problem of regulating the operation of open multiagent systems in which multiple interrelated activities take place is addressed, thus involving the distributed management of norms. Authors propose normative structures as a means to observe and manage the evolution of normative positions in each activity and their propagation in distributed activities. In [13], authors treated a computer supported cooperative work (CSCW) system as an organization in a society to use the abstraction of an organizational model. Hence, they propose a logical predication-based organizational model with an organizational state machine (OSM) to describe norms in CSCW systems. The organizational model allows describing the behavioural rules of roles, and the OSM allows for checking the logical conflict among the rules.Some other works have been conducted for addressing norm change and norm consistency providing agents with mechanisms for enacting behaviour modification. Typically, new plans/actions have been created to comply with new norms [14–16]. In [17], Jiang et al. propose a normative structure, namedNorm Nets (NNs) for modeling sets of interrelated regulations. NNs are aimed at verifying whether executions of business processes complied with process regulations. Authors define a norm as a tuple of elements that specify the type of deontic operator, the pair role-action (the target) to which the deontic modality is assigned, a deadline of norm validity, and a precondition that determines when the target is initiated. A formal method for checking norm compliance by using Colored Petri Nets is proposed. In [18, 19], authors propose a means for automatically detecting and solving conflict and inconsistency in norm-regulated Virtual Organization and Electronic Institution.This work is developed in the context of the open and goal-directed systems that are able to work in dynamic normative environments and to organize their behaviour according to environmental changes.The approach we propose is founded on a norm compliance algorithm, starting from the knowledge about norms, goals, and state of the world and acting at the goal level, which allows the system to plan the right behaviour in conformity with the normative context. Such an algorithm also allows the system to address exceptional situations that may occur, mainly when several norms act simultaneously. Three possible cases we considered in detail: inconsistent norms that contain a logical contradiction, the presence of an antinomy, namely, a conflict between two norms that are mutually exclusive, and, finally, norms that are incompatible with system requirements, in other words, that make goals not satisfiable.


## 3. Background

This section is organized in two parts. The first one introduces norms and normative reasoning. The second one gives some details about antinomies in the legal theory.


### 3.1. Normative Reasoning

Norms are everywhere in our daily life. We are obligated to follow the traffic regulations during driving. We have to respect contractual restrictions at work. We try to follow etiquette to be well-accepted in society, and so on. These examples spread from hard to soft constraints. Generally speaking, norms can be seen as normative rules used for governing conducts, procedures, or state of affairs within a particular sphere of knowledge. The normative reasoning is the reasoning conforming to or based on norms. It is formally studied through the deontic logic [20], which is the study of the logical relationships among propositions asserting that specific actions or state of affairs are obligatory, forbidden, or permitted. From the application point of view, deontic logic is the logic that deals with actual as well as the ideal behaviour of systems [21]. Differently from the classic logic, when working with norms the main issue to be considered is that norms are neither true nor false. For overcoming this issue, a common approach is to consider the deontic logic as a logic of normative propositions. The underlying concept is that, even if norms are neither true nor false, someone may state that something ought to be done (according to norms): the statementMary ought to pay the taxes is, then, true or false description of a normative situation [22].Standard deontic logic [23, 24] is the most studied system of deontic logic and one of the first deontic logics axiomatically specified. It is obtained from the modal logic and employs three modal operators O, P, and F, where OA means thatit is obligatory that A, PA means thatit is permitted that A and FA thatit is forbidden that A. Let O be primitive, the operators P and F can be defined by the equivalences PA≡¬O¬A and FA≡O¬A, wherePA means that it is not obligatory that not A andFA means thatit is obligatory that not A. In this context, A designates a proposition that asserts that an act of the sort A is done. Thus,OA is read as “it is obligatory that the situation described by the descriptive sentence A is realized”.


### 3.2. Antinomy in Legal Theory

In legal theory, an antinomy is defined as an incompatibility relation between two norms belonging to the same legal system. If we refer to the classic logic, the incompatibility between two propositions is determined by the impossibility that they are both true. In legal theory, the concept of incompatibility is founded on the deontic logic, which refers to the obligatory of the prescriptive assertions.The deontic square of oppositions [24] shows all the possible relations between two norms in terms of obligations, prohibitions, permissions, and negative permissions (see Figure 1). The most common types of oppositions between the four modalities are as follows:(1) Incompatibility between a norm that prescribesPand a norm that prohibitsP. Only one of them may be in effect at any time.Figure 1
The deontic square of oppositions.(2) Incompatibility between a norm that prescribes P and a norm that allows not P. An action is either obligatory or not. Omissible corresponds to the absence of an obligation (may ¬(X)≡¬ must(X)) and vice versa.(3) Incompatibility between a norm that prohibits P and a norm that allows P. An action is either forbidden or allowed. Permission therefore corresponds to the absence of a prohibition (may(X)≡¬ (must not(X))). For solving antinomy, three criteria are adopted in legal theory. Thelegis posterior that states the younger law overrides the older law. Thelegis specialis establishes that a law governing a specific subject matter (lex specialis) overrides a law which only governs general matters (lex generalis). Finally, thelex superior derogat legi inferiori principle states that higher law overrides the lower law, because a legal system is commonly based on a power hierarchy. The proposed approach takes inspiration from this theory for addressing conflicting situations in the context of open and goal-directed intelligent systems.


## 3.1. Normative Reasoning

Norms are everywhere in our daily life. We are obligated to follow the traffic regulations during driving. We have to respect contractual restrictions at work. We try to follow etiquette to be well-accepted in society, and so on. These examples spread from hard to soft constraints. Generally speaking, norms can be seen as normative rules used for governing conducts, procedures, or state of affairs within a particular sphere of knowledge. The normative reasoning is the reasoning conforming to or based on norms. It is formally studied through the deontic logic [20], which is the study of the logical relationships among propositions asserting that specific actions or state of affairs are obligatory, forbidden, or permitted. From the application point of view, deontic logic is the logic that deals with actual as well as the ideal behaviour of systems [21]. Differently from the classic logic, when working with norms the main issue to be considered is that norms are neither true nor false. For overcoming this issue, a common approach is to consider the deontic logic as a logic of normative propositions. The underlying concept is that, even if norms are neither true nor false, someone may state that something ought to be done (according to norms): the statementMary ought to pay the taxes is, then, true or false description of a normative situation [22].Standard deontic logic [23, 24] is the most studied system of deontic logic and one of the first deontic logics axiomatically specified. It is obtained from the modal logic and employs three modal operators O, P, and F, where OA means thatit is obligatory that A, PA means thatit is permitted that A and FA thatit is forbidden that A. Let O be primitive, the operators P and F can be defined by the equivalences PA≡¬O¬A and FA≡O¬A, wherePA means that it is not obligatory that not A andFA means thatit is obligatory that not A. In this context, A designates a proposition that asserts that an act of the sort A is done. Thus,OA is read as “it is obligatory that the situation described by the descriptive sentence A is realized”.


## 3.2. Antinomy in Legal Theory

In legal theory, an antinomy is defined as an incompatibility relation between two norms belonging to the same legal system. If we refer to the classic logic, the incompatibility between two propositions is determined by the impossibility that they are both true. In legal theory, the concept of incompatibility is founded on the deontic logic, which refers to the obligatory of the prescriptive assertions.The deontic square of oppositions [24] shows all the possible relations between two norms in terms of obligations, prohibitions, permissions, and negative permissions (see Figure 1). The most common types of oppositions between the four modalities are as follows:(1) Incompatibility between a norm that prescribesPand a norm that prohibitsP. Only one of them may be in effect at any time.Figure 1
The deontic square of oppositions.(2) Incompatibility between a norm that prescribes P and a norm that allows not P. An action is either obligatory or not. Omissible corresponds to the absence of an obligation (may ¬(X)≡¬ must(X)) and vice versa.(3) Incompatibility between a norm that prohibits P and a norm that allows P. An action is either forbidden or allowed. Permission therefore corresponds to the absence of a prohibition (may(X)≡¬ (must not(X))). For solving antinomy, three criteria are adopted in legal theory. Thelegis posterior that states the younger law overrides the older law. Thelegis specialis establishes that a law governing a specific subject matter (lex specialis) overrides a law which only governs general matters (lex generalis). Finally, thelex superior derogat legi inferiori principle states that higher law overrides the lower law, because a legal system is commonly based on a power hierarchy. The proposed approach takes inspiration from this theory for addressing conflicting situations in the context of open and goal-directed intelligent systems.


## 4. Overview of the Proposed Approach

As previously said, the proposed work is developed in the context of open and goal-directed intelligent systems that work in dynamic normative environments. In this kind of systems, goals can be seen as motivators that provide them with the reason for doing something. Moreover, the open systems we considered may evolve at run-time because(i) new services could be made available for satisfying existing goals and(ii) new goals may be required to the system.Hence, working in dynamic normative environments with systems that may evolve their behaviour requires new methods able to ensure norm compliance also for norms and system behaviours that are not defined at design-time. For addressing such issues, our approach is based on a triplet of elements, namely, state of the world, goal, and norm. Astate of the world represents the particular conditions of the system and the context in which it works in a specific time. AGoal expresses the desired state of the world the system wants to achieve when certain conditions are verified. Finally,Norms regulate the state of the world using obligations, permissions, and prohibitions. In particular, we considered two different cases. In the first case, obligation and prohibition norms modify the desired state of the world according to the admissible state expressed by obligations or prohibitions. Permission may or may not change the desired state of the world. In the second case, norms are considered as promoters or inhibitors of the system in pursuing goals. Practically, a permission relaxes the constraints expressed by the conditions under which a goal has to be satisfied. Conversely, a prohibition nullifies the commitment with a goal under the circumstances defined by the prohibition norm. Finally, an obligation introduces further conditions under which a goal has to be reached. In this vision, the effect of the norms is that they may increase the possibility for the system to pursue a goal (permissions). Conversely, they may inhibit system intentions to pursue a goal (prohibitions). Finally, norms may force the system to pursue a goal (obligations). The main feature of the proposed approach is to obtain a norm compliance behaviour by exploiting normative reasoning applied to the state of affairs a system wants to achieve. Thus, norms regulate the system at goal level providing some advantages and overcoming some limitations of conventional approaches.For illustrating typical problems, we provide some examples (some norms used in the following examples are inspired to existing real norms). Let us suppose an intelligent system able to plan and organize a personal agenda of a user. Let us suppose that such a system can satisfy the user goalG1:I want to go to country A. We also assume the user has already obtained the entry visa for the country A that is a prerequisite for achieving G1. Let us also suppose that, at the moment of the goal commitment, the system knows only two ways for satisfying the goal G1: by booking a flight or a train. Figure 2 shows a goal model with services the system may use for producing a postsituation which satisfies G1. We have also depicted two levels of abstractions to empathize the difference between goals and services.Figure 2
Services that could be chosen for satisfying the goal.Let us suppose that a normN1 states that “It is prohibited that a person visits the country_A if he has visited a country_B”. Let us assume that the user has already visited thecountry_B. Thus, the presence of such norm does not permit the user to go tocountry_A. For complying with such a normative requirement, the system does not have to plan the user trip. The proposed approach allows the system to revise its current commitment to that goal. Hence, rather than disabling all the possible ways the system can follow to satisfy the goal, the norm applied at the goal level does not allow the system to pursue the goal because it inhibits its intentions.Let us assume, instead, that the personal agenda is committed to fulfilling two user goals,go to country_A andgo to country_B. For complying with N1, the personal agenda has to be able to plan the trip for country_A firstly and then the trip for country_B. Our approach allows the system to reason about the effect of the norm on its desired state of the world. Because choosing firstly to pursue the goalgo to country_B leads the system in an unlawful state of the world, the system will plan the two goals opportunely.Let us suppose now that a new service “rent a car” is available at run-time for satisfying the goal“go to (country)” (see Figure 3). The run-time introduction of this new element does not involve any change in the system configuration.Figure 3
The availability of the new serviceRent a car gives the system a new mean for satisfying the goalGo to (country).We do not need to modify anything to adapt the behaviour of the system to manage this new situation because the norm defined at the goal level spreads to the service level. The approach we propose allows maintaining norm compliance although the service level is changed.Conversely, let us suppose that a normN2 states that“In country A, it is prohibited to enter foreign cars”. Let us suppose that available rent car services do not have cars produced in thecountry A. This kind of norm does not prohibit to pursue the goal G1, but it has effects on it. In our approach, such kind of norm introduces constraints to the final state of the world the system wants to reach. Thus, the system will choose thebook a flight orbook a train service in order to achieve G1.Finally, let us suppose that a normN3 “It is permitted that a person goes to country_A if he is a citizen of a member state of the organization X” is at run-time introduced in the system. We also assume that the user has not an entry visa for the country A, yet. The single effect of N3 on the system (i.e., without considering the presence of N1 and N2) is to relax the conditions under which the goal has to be satisfied. The system could plan to fulfill the goalgo to country_A also without its prerequisite which is satisfied (i.e., without an entry visa for Country A). On the contrary, the simultaneous presence of N1 and N3 creates a joint effect on the system because they affect the same goal“go to country_A”. In particular, the injection of N3 could cause a system deadlock. Indeed, if the conditions of N1 and N3 are simultaneously valid for the user, an antinomy is generated, and the system does not know how to behave. The compliance with N3 causes to be uncompliant with N1. It is a classic example of a conflict generated when two norms are contradictory. Our normative reasoning approach implements recovery criteria for addressing such situations. In particular, in this situation N3 is a norm defined by superior institutional power. Thus it should prevail on the second one.Such examples show only some of the situations that an open system should manage working in a dynamic normative environment. Several other anomalous situations determining system deadlock can occur in implementing norm compliance. In this work, we address the following ones: (1) The first isThe introduction into the system of an inconsistent norm. It means that the norm is self-contradictory because it contains a logical contradiction, namely, the conjunction of a statement S and its denied, not S. (2) The second isThe presence of an antinomy. An antinomy designates a conflict of two norms that are mutually exclusive or that oppose one another. (3) The third isThe run-time injection of norms incompatible with a system goal. This means that pursuing that goal always violates the prescribed norms.In our approach, we manage these situations in dynamically changing environments, where conflicting situations among norms may change according to the particular execution context. For addressing this concern, we introduce some new definitions about conflicts and inconsistencies that are based on a representation of the execution context. The norm compliance approach is defined through algorithms that are based on these definitions. In the following section, the theoretical foundations of the approach are introduced.


## 5. Theoretical Foundations

This section formally introduces the theoretical foundations of the proposed approach. Firstly, the definitions of the state of the world, goals, and norms are introduced. Then, formal definitions about norm compliance and anomalous situations are presented. It is worth noting that such definitions imply a formalization of asphere of knowledge within the system operates. In particular, to establish a relationship between the formal specification of goals and state of the world and the formal representation of the normative framework it is necessary to refer to the same semantic layer. This requirement can be satisfied by adopting a knowledge formalization based on proper domain ontology (it is out of the scope of the paper how to build the ontology. The reader may refer to the several approaches proposed in the literature).


### 5.1. State of the World, Goals and Norms

Thestate of the world represents a set of declarative information about events occurring within the environment and relations among events at a specific time. An event can be defined as the occurrence of some fact that can be perceived by or be communicated to the intelligent system. Events can be used to represent any information that can characterize the situation of an interacting user as well as a set of circumstances in which the intelligent system operates at a specific time.Definition 1 (state of the world).
LetD be the set of concepts defining a domain. Let L be a first-order logic defined on D with ⊤ a tautology and ⊥ a logical contradiction, where an atomic formula p(t1,t2…,tn)∈L is represented by a predicate applied to a tuple of terms (t1,t2…,tn)∈D and the predicate is a property of or relation between such terms that can be true or false. Astate of the world in a given time t (Wt) is a subset of atomic formulae whose values are true at the time t:(1)Wt=p1t1,t2,…,th,…,pnt1,t2,…,tmDefinition1 is based on the close world hypothesis [25] that assumes all facts that are not in the state of the world are considered false. Resuming the previous example, a possible state of the world at a given timet could be represented as(2)Wt=hasentry_visa,country_A,bookedflight,AR302It means that, at a given timet, the user has an entry visa for country_A and has received the confirmation of the flight AR302 reservation.Definition 2 (goal).
LetD, L, and p(t1,t2…,tn)∈L be as previously introduced in Definition 1. Let tc∈L and fs∈L be formulae that may be composed of atomic formulae by means of logic connectives AND(∧), OR (∨), and NOT (¬). AGoal is a pair tc,fs where tc (trigger condition) is a condition to evaluate over a state of the world Wt when the goal may be actively pursued and fs (final state) is a condition to evaluate over a state of the world Wt+Δt when it is eventually addressed: (i)
a goal is active iftc(Wt)∧¬fs(Wt)=true(ii)
a goal is addressed iffs(Wt+Δt)=trueAGoal describes a desired state of affairs the actor wants to achieve. It is represented by a couple of elements named trigger condition and final state. The final state represents the desired state of affairs. A goal is activated when some conditions occur (i.e., the trigger conditions) and it is satisfied when a new state of the world contains the final state.The previous goalG1=go_to(country_A) (according to common goal-oriented methodologies, we give a name to a goal for referring to it [26]) could be represented as follows:(3)G1=hasentry_visa,country_A,visitedcountry_AIt means thatg will be actively pursued after receiving theentry_visa forcountry A. The goal is considered addressed in a new state of the world where the user is incountry A.Definition 3 (norm).
LetD, L, and p(t1,t2…,tn)∈L be as previously introduced in Definition 1. Let ϕ∈L and ρ∈L be formulae composed of atomic formula by means of logic connectives AND(∧), OR (∨), and NOT (¬). Moreover, let Dop={permission,obligation, prohibition} be the set of deontic operators. ANorm is defined by the elements of the following tuple:(4)n=r,g,ρ,ϕ,dscope,where scope identifies the field of reference of the norm; r∈R is theRole the norm refers to; g∈G is theGoal the norm refers to; ρ∈L is a formula expressing the set of actions and/or state of affairs that the norm disciplines; ϕ∈L is a logic condition (to evaluate over a state of the world Wt) under which the norm is applicable; d∈Dop is the deontic operator applied to ρ that the norm prescribes to the couple (r,g)∈R×G: (5)dρ=ρifd=obligation¬ρ,ifd=prohibitionρ∨¬ρifd=permissionAn obligation imposes to obtain the state of affairsρ. A prohibition defines ρ as a not-acceptable state of affairs. Finally, permissions do not have a restrictive role. We also introduce the concept of scope for relating a norm to a specific context. The scope also allows establishing some hierarchy among norms.The previous norm stating that “It is prohibited that a person visits the country_A if he has visited a country_B”  could be represented as follows: (6)n1=tourist,g1,visitedcountry_A,visitedcountry_B,prohibitionsafetyHereafter, we assume that given a normn=r,g,ρ,ϕ,d; the actor that is pursuing the goal g plays the role r.Definition 4 (applicable norm).
LetWt be a state of the world in a given timet. A norm n=r,g,ρ,ϕ,d is applicable at timet if(7)ϕWt=true∨ϕ=⊤Definition4 establishes when in a given context a norm is workable. When the statement representing the applicability condition results in a tautology, the norm is applicable in each state of the world the system is.Definition 5 (active norm).
LetWt be a state of the world in a given timet and let g=tc,fs be a not addressed goal. A norm n=r,g,ρ,ϕ,d is active at timet if(8)nisapplicableattimet∧tcWt=trueDefinition5 establishes when a given norm may produce an effect on the system. In particular, it occurs when the norm is applicable and the goal is active (namely, the system intends to pursue that goal). For the previous example, n1 is applicable for a user that has already visited country B, but it may produce an effect only if the user has anentry_visa forcountry A that triggers the system to try to pursue the goal.


### 5.2. Norm Compliance

The definition of normative compliance is based on the concept of inadmissible state of the world defined as follows.Definition 6 (inadmissible state of the world).
A state of the world at a given timet(9)Wt=p1t1,t2,…,th,…,pnt1,t2,…,tmis anInadmissible State of the World if ∃n=r,g,ρ,ϕ,d∣CondA∧CondB are verified where(10)CondA:ϕWt-Δt∧¬ρWt-Δt=truewhend=prohibitionCondB:p1t1,t2,…,th∧⋯∧pnt1,t2,…,tm∧dρ=⊥The first condition allows ensuring the nonretroactive effect of a prohibition norm. It disciplines the case where the state of affair regulated by a prohibition norm has occurred before the applicability of the norm. Therefore, ifn is a prohibition and ρ=true before the norm came into force, the state of the world cannot be considered inadmissible. The second condition verifies if Wt is contrasting with the deontic constraint the norm prescribes.It is worth noting that because ofρ may refer to a state of affairs then it might coincide with the desired state of the world (i.e., ρ=fs). This means that the norm disciplines directly to the goal fulfillment. Conversely, when ρ≠fs, the norm constrains the way to reach the final state of the world by pursuing the goal. In the following definition, we differentiated these two cases.Definition 7 (norm compliance).
Let us consider a normn=r,g,ρ,ϕ,d and a goal g=tc,fs. Let us consider a state of the world Wt in a given time t in which n is active and let Wt+Δt be the state of the world in which fs is true. Pursuing the goal g is compliant with the norm n if W is an admissible state of the world where(11)W≡Wt+Δtfs=ρ⋃k=tt+ΔtWkfs≠ρThe first case allows ensuring that the final state of the world achieved by pursuing a goal does not contain any violations of the normative constraints. The second case allows establishing that the system moved along a path which satisfies the norm passing through various states of the worlds appropriately. Definition7 is strictly correlated to practical reasoning of goal-oriented systems.Practical reasoning is reasoning directed towards actions; it is the process of figuring out what to do [3]. It consists of two activities:deliberation, deciding what goals to achieve, andmeans-ends reasoning, determining how to meet these goals. The central aspect of goal deliberation is “How can the system deliberate on its goals to decide which ones shall be pursued?” [27]. A goal-oriented system sees some of its goals merely as possible options. Goal deliberation has the task to decide which goals a system actively pursues, which ones it delays, and which ones it abandons. Conversely, means-ends reasoning are aimed at providing operationalization of goals. It is the process of deciding how to achieve a goal using the available means (e.g., actions, services, and resources). In our approach, a mean describes a particular trajectory in terms of the state of the world the system may intentionally use to address a given result. Thus the system knows its effect on the state of the world. The definition we introduce about norm compliance directly influences the process of goal deliberation. The first condition of Definition 7 has a direct impact on the choice of goals that can be pursued. A system can deliberate to pursue a goal based on run-time conditions by envisaging the normative effects of the goal. Conversely, means-ends reasoning is a process that allows choosing the appropriate ways to fulfill a deliberated goal. The second condition of norm compliance is implicitly related to this process. A system can determine the way to reach a goal by envisaging the normative effects of available means that it can choose.


### 5.3. Anomalous Situations

This section formalizes the types of exceptional situations that can occur working with normative propositions.Definition 8 (inconsistent norm).
A normn=r,g,ρ,ϕ,d[scope] is inconsistent if ϕ=⊥.A normn=r,g,ρ,ϕ,d[scope] is inconsistent when it contains a logical contradiction. This means that its applicability condition ϕ contains the conjunction of a statement S and its denied not S. For example, let us suppose a norm N:It is prohibited that a person enters in building site if he is unauthorized, and if he is without protection and he is authorized. This norm contains the conjunction of two contradictory statements (i.e., he is an unauthorized user and he is an authorized user). In this case, the norm will always be not applicable without effect on the system behaviour. Such a situation could occur during the definition of norms with a complex condition, or a writing error could determine it. In the previous example, the corrected norm could be as follows:It is prohibited that a person enters in building site if he is unauthorized or if he is without protection and he is authorized.Definition 9 (incompatibility).
Letg=tc,fs be a not addressed goal. A norm n=r,g,ρ,ϕ,d isincompatible with  g if(12)iϕ=⊤iifs∧dρ=⊥An incompatibility exists between a norm and a goal when pursuing the goal always violates the prescribed norm.Let us consider a normn=r,g,ρ,ϕ,d and a goal g=tc,fs; the following incompatibility cases may arise.Case A. fs=ρ, d=Prohibition and ϕ=⊤; then n modifies the final state of g as follows:(13)fs′=fs∧¬fsIn this case,n prohibits the state of affairs fs directly; as a consequence it forbids to pursue the goal g in any way. In some cases, this type of norm could be used for inhibiting some system behaviours, but a run-time injection of a norm whose applicability condition is erroneously written in such a way to determine a tautology may cause an undesired system deadlock. For example, let us consider the following normIt is prohibited to go to country A. It directly constrains the achievement of the goalgo to country A.Case B. (fs≠ρ)∧(fs∩ρ≠⌀), d=Prohibition and ϕ=⊤; then n modifies the final state of g as follows:(14)fs′=fs∧¬ρIn this case,n indirectly constraints the achievement of the goal because n forbids to be in a state that is necessary for the accomplishment of the goal. For example, let us consider a simple example to remain in the context of the previous case. The normIt is prohibited to be in motion highlights the impossibility to pursue the goalgo to country A in compliance with the norm.Definition 10 (deontic contradiction).
LetWt be a state of the world at time t and let n1=r1,g1,ρ1,ϕ1,d1 and n2=r2,g2,ρ2,ϕ2,d2 be two norms where r1=r2, g1=g2, ρ1=ρ2. Norms n1 and n2 are deonticallycontradictory if(15)iϕ1Wt∧ϕ2Wt=trueiid1≠d2As previously said, an antinomy designates a conflict of two norms that are mutually exclusive or that oppose one another. Some norms can generate an antinomy under certain circumstances. In autonomous systems, such cases are not predetermined. Thus, systems have to be able to evaluate each particular situation at run-time. In the following, we discuss some possible scenarios that could occur during system execution that are related to the possible kinds of antinomy presented in Section3.2.Letn1 and n2 be two norms, n1=r1,g1,ρ1,ϕ1,d1 and n2=r2,g2,ρ2,ϕ2,d2, where r1=r2, g1=g2=g, and g=tc,fs:Case C. If d1=Prohibition, d2=Obligation and ρ1=ρ2=ρ, the joint effect of n1 and n2 modifies g as follows:(16)g′=tc,fs∧¬ρ∧ρifϕ1∧ϕ2tc,fs∧¬ρifϕ1∧¬ϕ2tc,fs∧ρif¬ϕ1∧ϕ2In the first case, the system is in a conflict situation because of the contemporaneous applicability ofn1 and n2. Thus, there is no way to be compliant with both norms; the system adopts the recovery criteria. In the other cases, the final state is constrained to be compliant with the applicable norm.Case D. If d1=Obligation, d2=Permission, ρ1=ρ, and ρ2=¬ρ, the joint effect of n1 and n2 modifies g as follows:(17)g′=tc,fs∧ρ∨fs∧¬ρ∧ρifϕ1∧ϕ2tc,fs∧ρifϕ1∧¬ϕ2tc,fs∧¬ρ∨fs∧ρif¬ϕ1∧ϕ2In the first case, the system avoids a conflict situation pursuingg in such a way that the final state of the world includes the state of affair expressed by ρ. In other cases, n1 and n2 do not create a conflict.Case E. If d1=Prohibition, d2=Permission, and ρ1=ρ2=ρ, the joint effect of n1 and n2 modifies the final state of g as follows:(18)g′=tc,fs∧¬ρ∧ρ∨fs∧¬ρifϕ1∧ϕ2tc,fs∧¬ρifϕ1∧¬ϕ2tc,fs∧ρ∨fs∧¬ρif¬ϕ1∧ϕ2In the first case, to be compliant with norms the system fulfillsg in such a way to avoid the state of affair expressed by ρ. In the other cases, n1 and n2 do not create a conflict.In particular, whenϕ1=true and ϕ2=true and fs=ρ the previous cases can be represented as follows.Case F. If d1=Prohibition, d2=Obligation, fs=ρ1, and ρ1=ρ2, the joint effect of n1 and n2 leads the final state of g in a contradictory state of the world:(19)fs′=¬fs∧fsIn this case, there is no way to be compliant with both norms; the system adopts the recovery criteria.Case  G. If d1=Obligation, d2=Permission, fs=ρ1, and ρ2=¬ρ1, the joint effect of n1 and n2 modifies the final state of g as follows:(20)fs′=fs∧¬fs∨fsThis case does not have a real effect on the system. In any case, the system pursuesg.Case H. If d1=Prohibition, d2=Permission, fs=ρ1, and ρ1=ρ2, the joint effect of n1 and n2 modifies the final state of g as follows:(21)fs′=fs∧¬fsAlso, in this case, there is no way to be compliant with both norms. Indeed, the system should not pursueg for not violating any norms.It is worth noting that a norm islogically contradictory when the contradiction concerns the logical conditions (ϕ∈L) under which the norm is applicable. On the contrary, we talk aboutdeontically contradictory when the contradiction concerns the semantic meaning of the deontic operator (d∈Dop) the norms apply to.


## 5.1. State of the World, Goals and Norms

Thestate of the world represents a set of declarative information about events occurring within the environment and relations among events at a specific time. An event can be defined as the occurrence of some fact that can be perceived by or be communicated to the intelligent system. Events can be used to represent any information that can characterize the situation of an interacting user as well as a set of circumstances in which the intelligent system operates at a specific time.Definition 1 (state of the world).
LetD be the set of concepts defining a domain. Let L be a first-order logic defined on D with ⊤ a tautology and ⊥ a logical contradiction, where an atomic formula p(t1,t2…,tn)∈L is represented by a predicate applied to a tuple of terms (t1,t2…,tn)∈D and the predicate is a property of or relation between such terms that can be true or false. Astate of the world in a given time t (Wt) is a subset of atomic formulae whose values are true at the time t:(1)Wt=p1t1,t2,…,th,…,pnt1,t2,…,tmDefinition1 is based on the close world hypothesis [25] that assumes all facts that are not in the state of the world are considered false. Resuming the previous example, a possible state of the world at a given timet could be represented as(2)Wt=hasentry_visa,country_A,bookedflight,AR302It means that, at a given timet, the user has an entry visa for country_A and has received the confirmation of the flight AR302 reservation.Definition 2 (goal).
LetD, L, and p(t1,t2…,tn)∈L be as previously introduced in Definition 1. Let tc∈L and fs∈L be formulae that may be composed of atomic formulae by means of logic connectives AND(∧), OR (∨), and NOT (¬). AGoal is a pair tc,fs where tc (trigger condition) is a condition to evaluate over a state of the world Wt when the goal may be actively pursued and fs (final state) is a condition to evaluate over a state of the world Wt+Δt when it is eventually addressed: (i)
a goal is active iftc(Wt)∧¬fs(Wt)=true(ii)
a goal is addressed iffs(Wt+Δt)=trueAGoal describes a desired state of affairs the actor wants to achieve. It is represented by a couple of elements named trigger condition and final state. The final state represents the desired state of affairs. A goal is activated when some conditions occur (i.e., the trigger conditions) and it is satisfied when a new state of the world contains the final state.The previous goalG1=go_to(country_A) (according to common goal-oriented methodologies, we give a name to a goal for referring to it [26]) could be represented as follows:(3)G1=hasentry_visa,country_A,visitedcountry_AIt means thatg will be actively pursued after receiving theentry_visa forcountry A. The goal is considered addressed in a new state of the world where the user is incountry A.Definition 3 (norm).
LetD, L, and p(t1,t2…,tn)∈L be as previously introduced in Definition 1. Let ϕ∈L and ρ∈L be formulae composed of atomic formula by means of logic connectives AND(∧), OR (∨), and NOT (¬). Moreover, let Dop={permission,obligation, prohibition} be the set of deontic operators. ANorm is defined by the elements of the following tuple:(4)n=r,g,ρ,ϕ,dscope,where scope identifies the field of reference of the norm; r∈R is theRole the norm refers to; g∈G is theGoal the norm refers to; ρ∈L is a formula expressing the set of actions and/or state of affairs that the norm disciplines; ϕ∈L is a logic condition (to evaluate over a state of the world Wt) under which the norm is applicable; d∈Dop is the deontic operator applied to ρ that the norm prescribes to the couple (r,g)∈R×G: (5)dρ=ρifd=obligation¬ρ,ifd=prohibitionρ∨¬ρifd=permissionAn obligation imposes to obtain the state of affairsρ. A prohibition defines ρ as a not-acceptable state of affairs. Finally, permissions do not have a restrictive role. We also introduce the concept of scope for relating a norm to a specific context. The scope also allows establishing some hierarchy among norms.The previous norm stating that “It is prohibited that a person visits the country_A if he has visited a country_B”  could be represented as follows: (6)n1=tourist,g1,visitedcountry_A,visitedcountry_B,prohibitionsafetyHereafter, we assume that given a normn=r,g,ρ,ϕ,d; the actor that is pursuing the goal g plays the role r.Definition 4 (applicable norm).
LetWt be a state of the world in a given timet. A norm n=r,g,ρ,ϕ,d is applicable at timet if(7)ϕWt=true∨ϕ=⊤Definition4 establishes when in a given context a norm is workable. When the statement representing the applicability condition results in a tautology, the norm is applicable in each state of the world the system is.Definition 5 (active norm).
LetWt be a state of the world in a given timet and let g=tc,fs be a not addressed goal. A norm n=r,g,ρ,ϕ,d is active at timet if(8)nisapplicableattimet∧tcWt=trueDefinition5 establishes when a given norm may produce an effect on the system. In particular, it occurs when the norm is applicable and the goal is active (namely, the system intends to pursue that goal). For the previous example, n1 is applicable for a user that has already visited country B, but it may produce an effect only if the user has anentry_visa forcountry A that triggers the system to try to pursue the goal.


## 5.2. Norm Compliance

The definition of normative compliance is based on the concept of inadmissible state of the world defined as follows.Definition 6 (inadmissible state of the world).
A state of the world at a given timet(9)Wt=p1t1,t2,…,th,…,pnt1,t2,…,tmis anInadmissible State of the World if ∃n=r,g,ρ,ϕ,d∣CondA∧CondB are verified where(10)CondA:ϕWt-Δt∧¬ρWt-Δt=truewhend=prohibitionCondB:p1t1,t2,…,th∧⋯∧pnt1,t2,…,tm∧dρ=⊥The first condition allows ensuring the nonretroactive effect of a prohibition norm. It disciplines the case where the state of affair regulated by a prohibition norm has occurred before the applicability of the norm. Therefore, ifn is a prohibition and ρ=true before the norm came into force, the state of the world cannot be considered inadmissible. The second condition verifies if Wt is contrasting with the deontic constraint the norm prescribes.It is worth noting that because ofρ may refer to a state of affairs then it might coincide with the desired state of the world (i.e., ρ=fs). This means that the norm disciplines directly to the goal fulfillment. Conversely, when ρ≠fs, the norm constrains the way to reach the final state of the world by pursuing the goal. In the following definition, we differentiated these two cases.Definition 7 (norm compliance).
Let us consider a normn=r,g,ρ,ϕ,d and a goal g=tc,fs. Let us consider a state of the world Wt in a given time t in which n is active and let Wt+Δt be the state of the world in which fs is true. Pursuing the goal g is compliant with the norm n if W is an admissible state of the world where(11)W≡Wt+Δtfs=ρ⋃k=tt+ΔtWkfs≠ρThe first case allows ensuring that the final state of the world achieved by pursuing a goal does not contain any violations of the normative constraints. The second case allows establishing that the system moved along a path which satisfies the norm passing through various states of the worlds appropriately. Definition7 is strictly correlated to practical reasoning of goal-oriented systems.Practical reasoning is reasoning directed towards actions; it is the process of figuring out what to do [3]. It consists of two activities:deliberation, deciding what goals to achieve, andmeans-ends reasoning, determining how to meet these goals. The central aspect of goal deliberation is “How can the system deliberate on its goals to decide which ones shall be pursued?” [27]. A goal-oriented system sees some of its goals merely as possible options. Goal deliberation has the task to decide which goals a system actively pursues, which ones it delays, and which ones it abandons. Conversely, means-ends reasoning are aimed at providing operationalization of goals. It is the process of deciding how to achieve a goal using the available means (e.g., actions, services, and resources). In our approach, a mean describes a particular trajectory in terms of the state of the world the system may intentionally use to address a given result. Thus the system knows its effect on the state of the world. The definition we introduce about norm compliance directly influences the process of goal deliberation. The first condition of Definition 7 has a direct impact on the choice of goals that can be pursued. A system can deliberate to pursue a goal based on run-time conditions by envisaging the normative effects of the goal. Conversely, means-ends reasoning is a process that allows choosing the appropriate ways to fulfill a deliberated goal. The second condition of norm compliance is implicitly related to this process. A system can determine the way to reach a goal by envisaging the normative effects of available means that it can choose.


## 5.3. Anomalous Situations

This section formalizes the types of exceptional situations that can occur working with normative propositions.Definition 8 (inconsistent norm).
A normn=r,g,ρ,ϕ,d[scope] is inconsistent if ϕ=⊥.A normn=r,g,ρ,ϕ,d[scope] is inconsistent when it contains a logical contradiction. This means that its applicability condition ϕ contains the conjunction of a statement S and its denied not S. For example, let us suppose a norm N:It is prohibited that a person enters in building site if he is unauthorized, and if he is without protection and he is authorized. This norm contains the conjunction of two contradictory statements (i.e., he is an unauthorized user and he is an authorized user). In this case, the norm will always be not applicable without effect on the system behaviour. Such a situation could occur during the definition of norms with a complex condition, or a writing error could determine it. In the previous example, the corrected norm could be as follows:It is prohibited that a person enters in building site if he is unauthorized or if he is without protection and he is authorized.Definition 9 (incompatibility).
Letg=tc,fs be a not addressed goal. A norm n=r,g,ρ,ϕ,d isincompatible with  g if(12)iϕ=⊤iifs∧dρ=⊥An incompatibility exists between a norm and a goal when pursuing the goal always violates the prescribed norm.Let us consider a normn=r,g,ρ,ϕ,d and a goal g=tc,fs; the following incompatibility cases may arise.Case A. fs=ρ, d=Prohibition and ϕ=⊤; then n modifies the final state of g as follows:(13)fs′=fs∧¬fsIn this case,n prohibits the state of affairs fs directly; as a consequence it forbids to pursue the goal g in any way. In some cases, this type of norm could be used for inhibiting some system behaviours, but a run-time injection of a norm whose applicability condition is erroneously written in such a way to determine a tautology may cause an undesired system deadlock. For example, let us consider the following normIt is prohibited to go to country A. It directly constrains the achievement of the goalgo to country A.Case B. (fs≠ρ)∧(fs∩ρ≠⌀), d=Prohibition and ϕ=⊤; then n modifies the final state of g as follows:(14)fs′=fs∧¬ρIn this case,n indirectly constraints the achievement of the goal because n forbids to be in a state that is necessary for the accomplishment of the goal. For example, let us consider a simple example to remain in the context of the previous case. The normIt is prohibited to be in motion highlights the impossibility to pursue the goalgo to country A in compliance with the norm.Definition 10 (deontic contradiction).
LetWt be a state of the world at time t and let n1=r1,g1,ρ1,ϕ1,d1 and n2=r2,g2,ρ2,ϕ2,d2 be two norms where r1=r2, g1=g2, ρ1=ρ2. Norms n1 and n2 are deonticallycontradictory if(15)iϕ1Wt∧ϕ2Wt=trueiid1≠d2As previously said, an antinomy designates a conflict of two norms that are mutually exclusive or that oppose one another. Some norms can generate an antinomy under certain circumstances. In autonomous systems, such cases are not predetermined. Thus, systems have to be able to evaluate each particular situation at run-time. In the following, we discuss some possible scenarios that could occur during system execution that are related to the possible kinds of antinomy presented in Section3.2.Letn1 and n2 be two norms, n1=r1,g1,ρ1,ϕ1,d1 and n2=r2,g2,ρ2,ϕ2,d2, where r1=r2, g1=g2=g, and g=tc,fs:Case C. If d1=Prohibition, d2=Obligation and ρ1=ρ2=ρ, the joint effect of n1 and n2 modifies g as follows:(16)g′=tc,fs∧¬ρ∧ρifϕ1∧ϕ2tc,fs∧¬ρifϕ1∧¬ϕ2tc,fs∧ρif¬ϕ1∧ϕ2In the first case, the system is in a conflict situation because of the contemporaneous applicability ofn1 and n2. Thus, there is no way to be compliant with both norms; the system adopts the recovery criteria. In the other cases, the final state is constrained to be compliant with the applicable norm.Case D. If d1=Obligation, d2=Permission, ρ1=ρ, and ρ2=¬ρ, the joint effect of n1 and n2 modifies g as follows:(17)g′=tc,fs∧ρ∨fs∧¬ρ∧ρifϕ1∧ϕ2tc,fs∧ρifϕ1∧¬ϕ2tc,fs∧¬ρ∨fs∧ρif¬ϕ1∧ϕ2In the first case, the system avoids a conflict situation pursuingg in such a way that the final state of the world includes the state of affair expressed by ρ. In other cases, n1 and n2 do not create a conflict.Case E. If d1=Prohibition, d2=Permission, and ρ1=ρ2=ρ, the joint effect of n1 and n2 modifies the final state of g as follows:(18)g′=tc,fs∧¬ρ∧ρ∨fs∧¬ρifϕ1∧ϕ2tc,fs∧¬ρifϕ1∧¬ϕ2tc,fs∧ρ∨fs∧¬ρif¬ϕ1∧ϕ2In the first case, to be compliant with norms the system fulfillsg in such a way to avoid the state of affair expressed by ρ. In the other cases, n1 and n2 do not create a conflict.In particular, whenϕ1=true and ϕ2=true and fs=ρ the previous cases can be represented as follows.Case F. If d1=Prohibition, d2=Obligation, fs=ρ1, and ρ1=ρ2, the joint effect of n1 and n2 leads the final state of g in a contradictory state of the world:(19)fs′=¬fs∧fsIn this case, there is no way to be compliant with both norms; the system adopts the recovery criteria.Case  G. If d1=Obligation, d2=Permission, fs=ρ1, and ρ2=¬ρ1, the joint effect of n1 and n2 modifies the final state of g as follows:(20)fs′=fs∧¬fs∨fsThis case does not have a real effect on the system. In any case, the system pursuesg.Case H. If d1=Prohibition, d2=Permission, fs=ρ1, and ρ1=ρ2, the joint effect of n1 and n2 modifies the final state of g as follows:(21)fs′=fs∧¬fsAlso, in this case, there is no way to be compliant with both norms. Indeed, the system should not pursueg for not violating any norms.It is worth noting that a norm islogically contradictory when the contradiction concerns the logical conditions (ϕ∈L) under which the norm is applicable. On the contrary, we talk aboutdeontically contradictory when the contradiction concerns the semantic meaning of the deontic operator (d∈Dop) the norms apply to.


## 6. Algorithms for Norm Compliance

In this section, the algorithms that implement norm compliance for open and goal-directed systems are presented. For space concerns, they are placed at the end of the paper.Algorithm1 implements the normative compliance. It ensures that the system behaves in conformity with the normative environment it is operating. The triple of elements it works is astate of the worldWt, aset of goalG the system has to satisfy and finally aset of normsN the system has to obey to comply with the normative environment. Both N, Wt, and G may change during system execution. The state of the world may change due to some events that can occur or some actions that can be performed in the environment. The set of norms may change due to the introduction of new normative requirements that can come into force or due to the deletion of existing ones. The updating of the existing norms is regarded as new norms. For the scope of the paper, in the algorithm, we highlight the case of the introduction of new norms that is the most interesting case that can generate anomalous situations. Finally, the set of goals may change for satisfying new user requirements.Algorithm 1:Norm compliance.
Data: Wt, G, N while system is running do Ninjectedt←inject_new_norms ①Nout←Check_Norm_Contradiction(Ninjectedt) N←concat(Nout,N) ②foreachgi∈Gdo tci,fsi←gi if¬fsi(Wt)then Ni←{n∈N:n=r,gi,ρ,ϕ,d∧ϕ(Wt)=true} forj←1tosize(Ni)do r,gi,ρj,ϕj,dj[scope]←nj iffsi=ρjthenadd(nj,List1) else ifdj=Obl∨dj=Prohibthenadd(nj,List2) Ⓐ ifcard{Ni}=0thengi′←tci,fsi Ⓑ ifcard{Ni}=1then ifcard{List1}=1then r,gi,ρ,ϕ,d←n if(d=Perm∨d=Obl)thentci′=OR_composition(tci,ϕ) elsetci′=AND_composition(tci,¬ϕ) gi′←tci′,fsi ifcard{List2}=1then iftci(Wt)=true)then fsi′←Const_Final_State(n,fsi) gi′←tci,fsi′ Ⓒ ifcard{Ni}>1then ifcard{List1}≥2then List1←Chek_Ant(List1,type1)Case F List1←Chek_Ant(List1,type3)Case H ifcard{List2}≥2thenList2←Chek_Ant(List2,type1)Case C (ϕOR,ϕAND)←Comp_Norm_Cond(List1) tci′←OR_composition(tci,ϕOR) tci′←AND_composition(tci′,ϕAND) fsi′←fsi; h←1 whileh≤size(List2)do r,gi,ρh,ϕh,dh[scope]←nh cont←false; k←h+1 whilek≤size(List2)∧(cont=false)do r,gi,ρk,ϕk,dk[scope]←nk; k←k+1 ifρh=ρkthen if(dh=Obl∧dk=Perm)∨(dk=Obl∧dh=Perm)then fsi′=fsi′∧ρhCase D if(dh=Proh∧dk=Perm)∨(dk=Proh∧dh=Perm) thenfsi′=fsi′∧¬ρhCase E cont←true List2←remove(n,ρ=ρh,List2) ifcont=falsethen fsi′←Constr_Final_State(nh,fsi′); h←h+1 gi′←tci′,fsi′ iftci′(Wt)=truethen pursue(gi′)Step ① makes a preliminary check on new norms (if any) that are dynamically introduced in the system to avoid the presence of anomalies according to Definitions8 and 9 (see Algorithm 2). Such step ensures that norms are consistent and there is no incompatibility with the goal they refer. Step ② is the core of the normative reasoning. The system is in the state of the world (Wt). The norms may have effects on the system goals only if goals are not addressed yet. Thus, the set of applicable norms is filtered (i.e., ϕ(Wt)=true). For each goal gi the system has to satisfy, the set of associated norms (Ni) is initially processed to separate norms where fs=ρ by norms with fs≠ρ because they produce a different effect on the system behaviour. As previously said, when fs≠ρ norms act as constraints on the final state of the world, in particular, the compliance has to be ensured for obligations and prohibitions. It is worth noting that when fs≠ρ, permissions are not taken into consideration. Permissions do not play a direct role in norm compliance because they cannot be violated. Conversely, when fs=ρ, norms act as constraints on system goal fulfillment. In particular, norms are promoters or inhibitors of the system in pursuing goals. In this case, permissions, obligations, and prohibitions have to be considered. As previously said, prohibitions do not allow the system to pursue a goal; obligations further constrain the conditions under which a goal can be pursued, and permissions conversely relax such conditions. Then, three different situations can occur.Algorithm 2:Check_Norm_Contradiction.
Data: a list of norms N Result: a list Nout of consistent norms Nout←⌀ ①fori←1tolength(N)do r,g,ρ,ϕi,d[scope]←ni tc,fs←g if(ϕ≠⊥)then if(fs∧d(ρ)≠⊥)∨(ϕi≠⊤)then addni to Nout else iffs=ρthen alert(ni) (see Case A) else revise(ni) (see Case B) else revise(ni)The most simple one (stepⒶ) is that there are no norms for gi. In such a case, there are no restrictions and the system can pursue the goal gi. The second situation (step Ⓑ) is a basic case in which the set of norms contains only one norm. In such a case, there are two possible occurrences:(i)
Step: fs=ρ then if the norm is a permission/obligation and it is applicable, the system can fulfill that goal even if the original tc(Wt)=false; if the norm is a prohibition, it further constraints the goal activation and the system cannot pursue that goal as long as the norm is applicable (i.e., ϕ(Wt)=true).(ii)
Step: fs≠ρ then a new constrained final state is determined (see Algorithm 3) and the system tries to achieve such constrained final state.Algorithm 3:Constrain_Final_State.
Data: an applicable norm n, a final state fs Result: a constrained final state fs′ r,g,ρ,ϕ,d←n switch d do casePermission fs′←fs∧(ρ∧¬ρ) caseProhibition fs′←fs∧¬ρ caseObligation fs′←fs∧ρThe last situation (stepⒸ) is the general case in which norms are more than one, and they can have different deontic operators. In this case, Algorithm 1 allows modifying goals, making them norm compliant according to a set of norms. Because norms are more than one, the presence of antinomy is possible. Thus, for each set of norms, antinomies are checked (see Algorithm 4). If fs≠ρ the recovery is applied only for Case C. Conversely, if fs=ρ the recovery is applied for Case F and Case H. Hence, Algorithm 1 firstly works on norms where fs=ρ that modify goal’s trigger condition. Indeed, by encapsulating the condition expressed by the norms into the goal they refer, it is possible to modify the activation of that goal thus making it compliant with the norms.Algorithm 4:Check_Antinomy.
Data: a list of applicable norms N, Type of Antinomy Result: a list Nout of consistent norms switch T y p e do caseType1 dA=Prohibition dB=Obligation caseType2 dA=Obligation dB=Permission caseType3 dA=Prohibition dB=Permission ①Nout←priority_order(N) Mconflicts←⌀ ②fori←1tolength(Nout)do r,g,ρ,ϕi,di←ni Mconflicts[i][i]←1 forj←i+1tolength(Nout)do ifri=rj∧ρi=ρj∧di≠dj∧((di=dA∧dj=dB)∨(di=dB∧dj=dA))then Mconflicts[i][j]←1 else Mconflicts[i][j]←0 Mconflicts[j][i]←Mconflicts[i][j] Nout′←Nout ③fori←length(Nout′)to1do ncurrent←Nout′[i] forj←i-1to2do ifMconflicts[i][j]=1then delete(Nout,ncurrent) break;Such composition (see Algorithm5) takes into consideration different types of norms, and it accordingly modifies the activation of a goal. Secondly, Algorithm 1 works on the set of norms where fs≠ρ that modify goal’s final state. In particular, if such set of norms contains at least two norms, Algorithm 1 checks if Case D or Case E exists and in such case it accordingly modifies the resulting final state to comply with such norms, and it removes from the list all norms that refer to the analysed case. Otherwise, it constrains the final state of the goal with the current norm according to Algorithm 3. In the following, we detail the algorithms that are invoked by the Algorithm 1.Algorithm 5:Compose_Norm_Condition.
Data: a list of norms NormList Result: a couple (ϕmergedOR,ϕmergedAND) Listϕ_OR←⌀ Listϕ_AND←⌀ // Identification of norm types for j ← 1 to s i z e ( N o r m L i s t ) do r,g,ρ,ϕ,d←NormList[j] switchddo caseObligation break caseProhibition add¬ϕ to Listϕ_AND casePermission addϕ to Listϕ_OR // Permissions give alternatives (OR) if S i z e ( L i s t ϕ _ O R ) ≠ 0 then ϕmergedOR←Listϕ_OR[1] forh←2toSize(Listϕ_OR)do ϕmergedOR←OR_composition(ϕmergedOR,Listϕ_OR[h]) // Prohibition are mandatory (AND) if S i z e ( L i s t ϕ _ A N D ) ≠ 0 then ϕmergedAND←Listϕ_AND[1] forh←2toSize(Listϕ_AND)do ϕmergedAND←AND_composition(ϕmergedAND,Listϕ_AND[h])


### 6.1. Check Norm Contradiction Algorithm

Check Norm Contradiction Algorithm (see Algorithm2) allows detecting incompatibility and inconsistent norms. In particular, Algorithm 2, for each norm, firstly checks the applicability condition to verify that it does not contain a logical contradiction (i.e., ϕ≠⊥). If it is the case, the norm has to be revised. This control is useful for avoiding the useless work of the system in evaluating some norms that will always be inapplicable. The second check of the Algorithm 2 (i.e., (fs∧d(ρ)≠⊥)∧(ϕ≠⊤)) allows avoiding norms that are always in contradiction with the goal they refer to. Therefore, norms prevent the fulfillment of a goal in any state of the world. During this control, two different cases are highlighted (i.e., fs=ρ and fs≠ρ). If fs=ρ, typically the norm is a prohibition for pursuing a goal (Case A). Because this norm could be deliberately introduced in the system to block a particular functioning, a simple alert is notified. If fs≠ρ and fs∩ρ≠⌀, this means that the norm affects only a portion of the final state of the world of the goal (Case B). Because in this case, the system would try to reach a goal which effectively cannot be reached creating a system overflow, a norm revision is mandatory.


### 6.2. Constrain Final State Algorithm

Algorithm3 allows modifying a final state fs according to the constraints expressed by a norm n. In particular, if norm n is a permission the constrained final state can or cannot include the state of affair the norm disciplines (i.e., fs′=fs∧(ρ∧¬ρ)). If norm n is a prohibition the constrained final state can not include the state of the affair the norm disciplines (i.e., fs′=fs∧¬ρ) and the contrary in the case of an obligation norm (i.e., fs′=fs∧ρ ).


### 6.3. Check Antinomy Algorithm

When there is more than an applicable norm in the system, it is necessary to check for deontological contradictions among norms and remove them (if any). In our context, antinomy could be generated by different norms relating to the same goal. For instance, if there is an applicable normn1 that prohibits to pursue a goal g1 and another applicable norm n2 that obliges to pursue the same goal g1, then n1 and n2 generate an antinomy. According to Section 5.3, Algorithm 4 manages three types of antinomy. We adopt thelegis posterior andlegis superior criteria coming from the legal theory. By adopting thelegis posterior criterion, the most recent norm takes precedence. Conversely, by adopting thelegis superior, the norm imposed by the prominent institutional power takes precedence. As we previously said, norms for different scopes could be defined. Norms for user well-being, for emergency management, and for ordinary situations could come in force. Some scopes are more crucial compared to other ones; a weight is applied for determining the relevance of a norm. We assign a priority to each norm taking into account the scope’s weight and the publication date of the norm in the system, as follows.Definition 11 (priority).
Let a normn=r,g,ρ,ϕ,d[scope]. Let wscope be the weight of the scope and let wtime be the weight of the time. The priority of n is (22)Pn=wscope+wtimeFor removing conflicts, Algorithm4 generates a conflicts matrix Mconflicts where Mconflictsi[j]=1 indicates a conflict between ith and jth norm (step ②). Such a matrix is built starting from an ordered list of norms according to their priority (step ①). Hence, norms with the lowest priority are removed (step ③).


### 6.4. Compose Norm Condition Algorithm

Compose Norm Condition Algorithm, taking in input a list of norms related to a single goalg where for each norm ρ=fs, allows merging in a unique activation condition, the conditions expressed by each norm. The algorithm takes into consideration different types of deontic operators. In the case of permissions, the resulting merged condition is an OR chain obtained by the single norm conditions, which relaxes the activation condition of the goal the norms refer to. Conversely, in the case of prohibitions, the resulting merged condition is an AND chain obtained by negating the conditions of the norms, which constrains the activation condition of the goal the norms refer to.In the following, the approach is evaluated using the well-known case study in the field of the business rules.


## 6.1. Check Norm Contradiction Algorithm

Check Norm Contradiction Algorithm (see Algorithm2) allows detecting incompatibility and inconsistent norms. In particular, Algorithm 2, for each norm, firstly checks the applicability condition to verify that it does not contain a logical contradiction (i.e., ϕ≠⊥). If it is the case, the norm has to be revised. This control is useful for avoiding the useless work of the system in evaluating some norms that will always be inapplicable. The second check of the Algorithm 2 (i.e., (fs∧d(ρ)≠⊥)∧(ϕ≠⊤)) allows avoiding norms that are always in contradiction with the goal they refer to. Therefore, norms prevent the fulfillment of a goal in any state of the world. During this control, two different cases are highlighted (i.e., fs=ρ and fs≠ρ). If fs=ρ, typically the norm is a prohibition for pursuing a goal (Case A). Because this norm could be deliberately introduced in the system to block a particular functioning, a simple alert is notified. If fs≠ρ and fs∩ρ≠⌀, this means that the norm affects only a portion of the final state of the world of the goal (Case B). Because in this case, the system would try to reach a goal which effectively cannot be reached creating a system overflow, a norm revision is mandatory.


## 6.2. Constrain Final State Algorithm

Algorithm3 allows modifying a final state fs according to the constraints expressed by a norm n. In particular, if norm n is a permission the constrained final state can or cannot include the state of affair the norm disciplines (i.e., fs′=fs∧(ρ∧¬ρ)). If norm n is a prohibition the constrained final state can not include the state of the affair the norm disciplines (i.e., fs′=fs∧¬ρ) and the contrary in the case of an obligation norm (i.e., fs′=fs∧ρ ).


## 6.3. Check Antinomy Algorithm

When there is more than an applicable norm in the system, it is necessary to check for deontological contradictions among norms and remove them (if any). In our context, antinomy could be generated by different norms relating to the same goal. For instance, if there is an applicable normn1 that prohibits to pursue a goal g1 and another applicable norm n2 that obliges to pursue the same goal g1, then n1 and n2 generate an antinomy. According to Section 5.3, Algorithm 4 manages three types of antinomy. We adopt thelegis posterior andlegis superior criteria coming from the legal theory. By adopting thelegis posterior criterion, the most recent norm takes precedence. Conversely, by adopting thelegis superior, the norm imposed by the prominent institutional power takes precedence. As we previously said, norms for different scopes could be defined. Norms for user well-being, for emergency management, and for ordinary situations could come in force. Some scopes are more crucial compared to other ones; a weight is applied for determining the relevance of a norm. We assign a priority to each norm taking into account the scope’s weight and the publication date of the norm in the system, as follows.Definition 11 (priority).
Let a normn=r,g,ρ,ϕ,d[scope]. Let wscope be the weight of the scope and let wtime be the weight of the time. The priority of n is (22)Pn=wscope+wtimeFor removing conflicts, Algorithm4 generates a conflicts matrix Mconflicts where Mconflictsi[j]=1 indicates a conflict between ith and jth norm (step ②). Such a matrix is built starting from an ordered list of norms according to their priority (step ①). Hence, norms with the lowest priority are removed (step ③).


## 6.4. Compose Norm Condition Algorithm

Compose Norm Condition Algorithm, taking in input a list of norms related to a single goalg where for each norm ρ=fs, allows merging in a unique activation condition, the conditions expressed by each norm. The algorithm takes into consideration different types of deontic operators. In the case of permissions, the resulting merged condition is an OR chain obtained by the single norm conditions, which relaxes the activation condition of the goal the norms refer to. Conversely, in the case of prohibitions, the resulting merged condition is an AND chain obtained by negating the conditions of the norms, which constrains the activation condition of the goal the norms refer to.In the following, the approach is evaluated using the well-known case study in the field of the business rules.


## 7. Eurent Case Study

The Eurent scenario is a widely known case study adopted in the field of business processes for demonstrating the capabilities of the proposed solution. It was initially developed by Model System, Ltd. Briefly, the case study presents a car rental company with branches in several countries which provides common rental services. It also owns information about cars, branches, employees, and so on. Particular attention is paid to the information about the customers that allows establishing if they are good or bad clients. Each branch of the company owns a predetermined number of cars that are available for rental. A customer may rent a car through an advanced reservation or a walk-in rental. In an advanced reservation, the rental period and the car group are specified at the time of reservation. Conversely, walk-in rentals (i.e., immediate reservations) are also accepted if cars are available. When cars are returned, the renting branch has to ensure that it is returned at the end of the rental period. Moreover, a customer can have several reservations but only one car rented at the time. The company keeps information about customers, their rentals, and bad experiences such as late return, problems with payments, and damage to cars. The Eurent company adopts several rules for governing their business.Some rules are related to the driver. For example, each customer must have a valid driver’s license for driving the car and (s)he has to be insured. Some rules are defined for rental reservation acceptance. For example, if the customer requesting the rental has been blacklisted, the rental must be refused. A customer may have multiple future reservations but may have only one car at any time. In the case of advance reservations, if the requested model is not available, a car in the same group as the request model has to be assigned. If there are several available cars of the model requested, the one with the lowest mileage has to be assigned and so on. A small portion of a conceptual schema of the Eurent case study is shown in Figure4. It defines the domain of knowledge used in the following. The proposed approach has been implemented in a simulated intelligent system able to manage car reservations. Figure 5 shows a portion of the system goal model. We adopt the following template for exemplifying a goal goal(tc,fs)[Id], where tc is the trigger condition of the goal, fs is the final state to be reached, and Id is an identification code for the goal. In the following, a small subset of Eurent goals is as follows:g0:
goal(start(customer,rental_agreement), or([state(rental_agreement,closed), state(rental_agreement,open), state(rental_agreement, cancelled)])g1:
goal(requested(customer,rental_agreement),done(make_reservation))g2:
goal(cancel_requested(customer,rental_agreement),done(cancel_reservation))Figure 4
A portion of Eurent domain.Figure 5
A portion of Eurent goal model.The first one means that when the system is committed to managing reservation, the system tries to reach a state of the world where one of three possible final states is verified. The choice depends on the current state of the world the system is. In the following, it is shown a small subset of norms related to ag2 where ρ=fs (i.e., N1, N2, N3) and ρ≠fs (i.e., N4, N5):N1:
norm(role(_),g2, done(cancel_reservation),is_blacklisted(customer,true)),type(obligation))N2:
norm(role(customer),g2, done(cancel_reservation),state(car,assigned),type(permission))N3:
norm(role(customer),g2, done(cancel_reservation),state(car,in_rent)),type(prohibition))N4:
norm(role(_),g2, activity(cancel(record)),is_blacklisted(customer,true),type( prohibition))N5:
norm(role(_),g2, activity(signal(authority)),and([owns(customer,driver_license),validity(driver_license,false)]),type(obligation))In particular,N1 states if a customer is placed on a blacklist then all his bookings must be cancelled. N2 states that it is permitted to cancel a reservation (without penalties) if a car has been only assigned to the customer. In this case, the customer is charged with no cost. Conversely complying to N3, it is prohibited to cancel a reservation (without penalties) if the car is in rent (the state of the car is changed fromassigned toin rent the day of the pick-up). In this case, one day-rental will be charged. N4 differently states that it is prohibited to cancel the record related to a blacklisted customer. This norm avoids making erroneous cancellations from the organization database of blacklisted customers. Finally, N5 imposes the obligation on the rental companies to alert the authority if a customer has no valid driver license. Figure 6 highlights some available services the system may use for satisfying its goals. For example, according to Eurent case study, two services allow the system for satisfying g2, although they produce a different state of the world. In particular, the serviceCancel Booking cancels only the customer’s reservation. Conversely, the serviceDelete Booking deletes not only the reservation but also all the data relating to the customer. Moreover, in Figure 6 we also emphasizeAuthority Notification Service that will be used by the system for informing the authority about unlawful behaviours of the customers.Figure 6
A subset of available services for Eurent case study.Each simulated scenario presents(i) an initial state of the world (W0) that determines the applicability of some norms and the activation of some goals,(ii) a set of norms,(iii) the steps of the normative reasoning the system performs, and, finally,(iv) a brief analysis of the behaviour of the system. For the sake of simplicity, we isolated only the reasoning about goal g2.Scenario 1.
INITIAL CONDITIONS:W0=cancel_requested(customer,rental_agreement),tc(W0)=trueAPPLICABLE NORMS: No applicable normsNORMATIVE REASONING:[system] Trying to achieveg2[system] Found available service: cancel_booking_service[system] EXECUTING STEP A: NO NORMS forg2[system] …invoking cancel_booking_service[monitor] The project is correctly terminated!Scenario1 shows the most simple situation, where no norm constrains the system. Thus, no change occurs in the normal behaviour of the system. The system pursues the triggered goal (g2) by executing the initially selected service.Scenario 2.
INITIAL CONDITIONS:W0=is_blacklisted(customer,true),tc(W0)=falseAPPLICABLE NORMS:N1= norm(role(customer),g2, done(cancel_reservation),is_blacklisted(customer,true)),type(obligation))NORMATIVE REASONING:[system monitor] InjectedN1[system] SKIP STEP A[system] EXECUTING STEP Bi: ONE NORM (Permission or Obligation) where RHO=FS forg2[system] Trying to achieveg2[system] Found available service: cancel_booking_service[system] …invoking cancel_booking_service[monitor] The project is correctly terminated!Without normative constraints, the system pursues a goal when its trigger condition is verified. By introducing a norm that obligates to obtain some state of affairs, the system is forced to pursue that goal when the condition of the obligation is verified although the trigger condition is not verified yet.Scenario 3.
INITIAL CONDITIONS:W0=state(car,in_rent),tc(W0)= falseAPPLICABLE NORMS:N3= norm(role(customer),g2, done(cancel reservation),state(car,in_rent)),type(prohibition))NORMATIVE REASONING:[system monitor] InjectedN3[system] SKIP STEP A[system] EXECUTING STEP Bi: ONE NORM (Prohibition) where RHO=FS forg2[system] SKIP STEP A[system] EXECUTING STEP Bi: ONE NORM (Prohibition) where RHO=FS forg2…When the trigger condition of a goal is false, the presence of a norm that prohibits pursuing that goal has no effects on the system behaviour. According to Definitions4 and 5, such norm is applicable but not active. The system waits for any change in its state of the world.Scenario 4.
INITIAL CONDITIONS:W0=state(car,in_rent), cancel_requested(customer,rental_agreement),tc(W0)= trueAPPLICABLE NORMS:N3= norm(role(customer),g2, done(cancel reservation),state(car,in_rent)),type(prohibition))NORMATIVE REASONING:[system] Trying to achieveg2[system] Found available service: cancel_booking_service[monitor] InjectedN3[system] SKIP STEP A[system] EXECUTING STEP Bi: ONE NORM (Prohibition) where RHO=FS forg2[system] REPLANNING……In this case, the presence of an active norm that prohibits pursuing a goal (g2) nullifies the commitment of the system with that goal, thus causing a system replanning. As a result, after the replanning, the goal (g2) is deleted by the set of the committed goals.Scenario 5.
INITIAL CONDITIONS:W0=is_blacklisted(customer,true), cancel_requested(customer,rental_agreement),tc(W0)= trueAPPLICABLE NORMS:N4= norm(role(_),g2, activity(cancel( record)),
is_blacklisted(customer,true),type(prohibition))NORMATIVE REASONING:[system] Trying to achieveg2[system] Found available service: delete_booking_service[monitor] InjectedN4[system] SKIP STEP A[system] SKIP STEP Bi[system] EXECUTING STEP Bii: ONE NORM (any type) where RHO/==FS forg2[system] REPLANNING…[system] Trying to achieveg2′
g2′=goal(condition(cancel_requested(customer,rental_agreement)),condition(and([done(cancel_reservation), neg(cancel( record))])),[system] Found available service: cancel_booking_service[system] EXECUTING STEP A, NO NORMS forg2′[system] …invoking cancel_booking_service[monitor] The project is correctly terminated!To pursue the goalg2, the system can invoke two services, Cancel or Delete Booking. As we previously said, the first one allows cancelling only the reservation. The second one allows also deleting the customer details. Thus, the system initially chooses the Delete Booking to pursue g2. At run-time, a new norm that prohibits cancelling customer details if the customer is blacklisted is injected. Thus, to be compliant with the new norm, the system replans its goals. The effect of such norm is a change in the final state of the goal g2, as can be seen after the replanning. Hence, to pursue g2′, the system chooses the service Cancel Booking, and because in the current state of the world the trigger condition of such goal is verified, the system executes the step A because after replanning no other norms have been introduced. As a result, the system has achieved the desired state of affair compliant with the normative in force.Scenario 6.
INITIAL CONDITIONS:W0=is_blacklisted(customer,true), owns(customer,driver_license), validity(driver_license,false)   tc(W0)= falseAPPLICABLE NORMS:N1:norm(role(_),g2, done(cancel_reservation), is_blacklisted(customer,true)),type(obligation))N4:norm(role(_),g2, activity(cancel(record)), is_blacklisted(customer,true),type(prohibition))N5:norm(role(_),g2, activity(signal(authority)), and([owns(customer,driver_license),validity(driver license,false)]),type(obligation))NORMATIVE REASONING:[monitor] InjectedN1,N4,N5[system] SKIP STEP A[system] SKIP STEP Bi[system] SKIP STEP Bii[system] EXECUTING STEP C: ONE OR MORE NORMS (permission or obligation) where RHO==FS, NORMS (anytype) where RHO/==FS forg2[system] REPLANNING…[system] Trying to achieveg2′g2′=goal(condition(is_blacklisted(customer,true)), condition(and([and([done(cancel_reservation),signal(authority)]), neg(cancel(record))]))[system] Found available service: cancel_booking_service, authority_notification_service[system] EXECUTING STEP A, NO NORMS forg2′[system] …invoking authority_notification_service[system] …invoking cancel_booking_service[monitor] The project is correctly terminated!In this scenario the run-time injection of three norms applicable simultaneously is simulated. The effect of the first oneN1 is to force the system to cancel reservations made by blacklisted customers although nobody has demanded the system to commit with g2 (indeed tc(Wt)=false). For obtaining such an effect, the trigger condition of g2 is changed. Conversely, the other two norms (N4 and N5) act on the final state of g2 to lead the system in an admissible state of the world. As can be seen, after the replanning the system tries to achieve the new goal g2′ by selecting two services that allow the system to reach the new constrained final state. It is worth noting that our approach does not change the practical reasoning of common goal-oriented systems. Our approach modifies the elements on which the system performs the practical reasoning, thus leading the system to make appropriate choices, both about what goals to deliberate and what means to choose for achieving the deliberated goal.Scenario 7.
INITIAL CONDITIONS:W0=is_blacklisted(customer,true),is_memberof(eurent,true), cancel_requested(customer,rental_agreement)tc(W0)=trueAPPLICABLE NORMS:N4:norm(role(_),g2, activity(cancel(record)), is_blacklisted(customer,true),type(prohibition))N6:norm(role(_),g2, activity(cancel(record)), is_memberof(eurent,true),type(obligation))NORMATIVE REASONING WITHOUT ANTINOMY DETECTION:[monitor] InjectedN6[monitor] InjectedN4[system] SKIP STEP A[system] SKIP STEP Bi[system] SKIP STEP Bii[system] EXECUTING STEP C: TWO OR MORE NORMS (any Type) with RHO/==FS forg2[system] REPLANNING…[system] Trying to achieveg2′g2′=goal(condition(cancel_requested(customer,rental_agreement)),condition(and([and([done(cancel_reservation),cancel(record)]), neg(cancel(record))]))[system] ERROR solutions not found …NORMATIVE REASONING WITH ANTINOMY DETECTION:[monitor] InjectedN6[monitor] InjectedN4[system] SKIP STEP A[system] SKIP STEP Bi[system] SKIP STEP Bii[system] EXECUTING STEP C:TWO OR MORE NORMS (any Type) where RHO/==FS forg2[system] ANTINOMY DETECTED … REMOVEDN6[system] REPLANNING…[system] Trying to achieveg2′g2′=goal(condition(cancel_requested(customer,rental agreement)),condition(and([done(cancel_reservation),neg(cancel( record))])),[system] Found available service: cancel_booking_service[system] EXECUTING STEP A, NO NORMS forg2′[system] …invoking cancel_booking_service[monitor] The project is correctly terminated!In this scenario, we introduced a new normN6 that is not defined in the original Eurent case study. Its purpose is to simulate the presence of an anomalous situation. Such a norm is contrary with respect to N4.N 6: norm(role(_),g2, activity(cancel(record)),is_memberof(eurent,true),type(obligation))In the first situation without antinomy detection, the system is not able to find an appropriate solution to maintain compliance with both norms. Conversely, in the second case, the system detects the antinomy and removes it. In doing so, the system deletesN6 because it is before N4 and pursues the new goal constrained only by N4. In such simulated system, we assume without loss of generality that each norm refers to the same scope and the time to come into force is trivially considered to be the one in which the norm was injected into the system.The scenarios we presented show some possible situations that can occur in a dynamic normative context, highlighting the strengths of the proposed approach that we can summarize as follows:(i)
Working with a dynamic set of norms: norms may change for several reasons. For example, for employing the system in different working environments regulated by different normative corpus, for managing unexpected situations, and so on. In the proposed approach, the ability of the system to work with a dynamic set of norms is realized by decoupling the norms from the execution level. Norms act to a higher level of abstraction (i.e., goal level) without any modification to the action level. Scenario 5 shows how the introduction of a new norm acts directly on the goal by modifying it to reach the state of the world the norm prescribes. No change occurs at the action level.(ii)
Ensuring norm compliance at run-time: working with a dynamic set of norms requires that the system has to be able to verify the compliance with new injected norms during its execution. Scenarios 2 and 4 provide two examples of run-time compliance. Precisely, we can see the different effects on the system of an obligation and a prohibition. To act according to an obligation means to do what the obligation prescribes. Conversely, behaving compliant with a prohibition means not perform what the norm forbids. Scenario 2 shows how the system changes its behaviour as a consequence of the introduction of an obligation during the execution, by deliberating that goal whose fulfillment leads the system to be compliant to the norm. Conversely, Scenario 4 shows how the system changes its behaviour nullifying that already deliberated goal whose achievement would lead to a violation of the prohibition. Finally, Scenario 5 shows how the system reacts to a prohibition that regulates not the final state of the world, but the way the system has to reach the goal. Indeed, although the system has already planned the behaviour to execute for satisfying the current goal, the introduction of the norm leads the system to adapt to this new situation replans its behaviour choosing a new service to comply with the new norm.(iii)
Handling norm conflict: working with a dynamic set of norms may cause anomalous situations as aforementioned. Our approach provides mechanisms to avoid system deadlocks. An example is provided by Scenario 7 that shows how the system can detect an antinomy, remove it, and replan its behaviour taking into consideration the most recent norm.(iv)
Improving system flexibility: system flexibility is commonly viewed as the ability of the system to adapt to changes. The proposed approach improves flexibility endowing the system with the ability to adjust its behaviour for ensuring norm compliance also for norms that have not been defined at design-time. Precisely, the system may pursue a goal not initially deliberated that is become mandatory by the introduction of a norm, or it may inhibit the fulfillment of prohibited goals. In other cases, the system changes its behaviour to comply with new norms by choosing different services to pursue its goals.


## 8. Conclusions

The increasing employment of artificial systems that perform autonomously relevant tasks in several contexts raises the problem to provide them with the normative reasoning to ensure that they behave lawfully.In this paper, we presented a normative reasoning approach for addressing norm compliance in the context of open and goal-directed systems. They are systems conceived for also satisfying not designed user requirements. They may evolve by increasing the objectives they can fulfill and by discovering new ways of achieving them. The approach presented in this work ensures run-time compliance with a dynamic set of norms that may change during system execution. The reasoning algorithm starting from the knowledge about norms, goals, and the state of the world allows the system to make decisions in conformity with the normative context, thus ensuring norm compliance at a higher level of abstraction (i.e., goal level) compared to the actions level. Thus, the effect of a norm acting at a goal level spreads out at the lower level of actions. Hence, a norm at goal level avoids defining normative constraints for each possible actions that a system could perform for reaching an objective and it makes the system free to choose such actions for reaching not constrained goals.The flexibility of the approach also lies in the possibility of changing the set of norms dynamically without system redesigning. This feature poses a new issue to be addressed related to the possibility of introducing conflicting norms. The proposed algorithm taking into consideration the simultaneous presence of multiple norms related to the same goal, thus determining their joined effect by checking and solving norm conflicts and inconsistencies during system execution.We are working for enlarging the normative reasoning by introducing alethic operators (such as necessity and possibility) to consider also definitional norms that are directed to regulate the elements of the knowledge domain rather than the behavioural aspect. Currently, we are also moving in the context of the social robots where the conformity of their behaviour according to social norms is considered extremely important to be accepted in daily human life.



---

*Source: 7895875-2019-04-08.xml*