# ‘If Only I Would Have Done that…’: A Controlled Adaptive Network Model for Learning by Counterfactual Thinking

Raj Bhalwankar, Jan Treur

► To cite this version:

Raj Bhalwankar, Jan Treur. ‘If Only I Would Have Done that…’: A Controlled Adaptive Network Model for Learning by Counterfactual Thinking. 17th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2021, Hersonissos, Crete, Greece. pp.3-16, 10.1007/978-3-030-79150-6_1 . hal-03287691

HAL Id: hal-03287691

https://inria.hal.science/hal-03287691

Submitted on 15 Jul 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
‘If Only I Would Have Done That...’:
A Controlled Adaptive Network Model
for Learning by Counterfactual Thinking

Raj Bhalwankar, Jan Treur

Social AI Group, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands

**Abstract** In this paper counterfactual thinking is addressed based on literature mainly from Neuroscience and Psychology. A detailed literature review was conducted in identifying processes, neural correlates and theories related to counterfactual thinking from different disciplines. A familiar scenario with respect to counterfactual thinking was identified. Based on the literature, an adaptive self-modeling network model was designed. This model captures the complex process of counterfactual thinking and the learning and control involved.

**Keywords:** counterfactual thinking, adaptive network model, mental model, learning, control.

# 1 Introduction

Human beings have a great ability to think and infer how a current situation (especially goal failure) could have turned out differently given a set of alternative actions or decisions they could have chosen from [4]. This process of deconstructing the current reality to imagine (a) new one(s) is called Counterfactual Thinking [17]. Such a type of thinking is important as in the first place it helps in making sense of the past, in planning actions, in making emotional & social judgements. Not less important, in the second place it plays an important functional role to guide adaptive behavior and learning [22] for the own benefit for the future. Such learning is a form of learning from mistakes, which involves the notion of regret which arises from comparing the alternative realities. This type of learning helps to generate new courses of actions which, after the failure experience, are believed to be more successful when similar situations occur in future. Various parts of the brain have been implicated to play a role in counterfactual thinking.

Yet also many questions about counterfactual thinking still have no full answers. How does the process of counterfactual thinking actually work in day-to-day life? What is its role in mental health, learning and decision-making? How does it affect our emotional health and how does it update our beliefs or perceptions? How can counterfactual thinking prove useful in AI applications like reinforcement learning?

Thus, the present study is meant to contribute some answers to these questions by providing a Neuroscience-inspired controlled adaptive network model that is able to
simulate processes of counterfactual thinking, including the learning effects of it and control over it. Computational modeling plays a important role here in making sense of behavioral and neurological data. Computational models represent different ‘algo- rithmic hypotheses’ about how behavior is generated [24]. Such simulations involve running the model with specific parameter settings to generate ‘fake’ behavioral data. These simulated data can then be analyzed in much the same way as one would analyze real data, to make precise, falsifiable predictions about qualitative and quantitative pat- terns in the data. These simulations contribute to theory building by making theoretical predictions more precise and testable [24].

Mental models are essential for construction of knowledge and play a crucial role in
learning, retrieving and problem solving. Van Hoeck et al. [22] proposed that counter-
factuals depend upon the mental models of alternative possibilities in form of mental
simulations, which suggests that these previous modeling approaches to mental model
development can be used to study counterfactual thinking. In the present study, a net-
work-oriented modeling approach was utilized to study the process of counterfactual
thinking based on literature and neuro-scientific evidence. Network-Oriented Modeling
is a useful method to represent the complex real-world processes concerning human
beings and has proven to be able to address adaptivity and control that play an important
role in counterfactual thinking.

The paper begins in Section 2 with a brief literature overview of the existing state of
research related to counterfactual thinking. Then, after a brief introduction of the mod-
eling approach used in Section 3, in Section 4 the design of the developed controlled
adaptive network model with its various parts is discussed. Simulations of some exam-
ple scenarios are discussed in Section 5; here it is shown that the model generates pat-
terns as expected from the empirical literature. In Section 6, correctness of the imple-
mented model against its conceptual design specifications is verified by analysis of sta-
tionary points. Section 7 addresses discussion and conclusions.

## 2 Literature Review

As stated earlier, counterfactual thinking can be helpful in learning from past mistakes
and in developing more promising intentions for the future [12-13, 15]. Mental models
of imagined past events or future outcomes that have not yet occurred support *counter-
factual thinking* [4, 11]. Norm Theory proposed by Kahneman and Miller [11] provides
a theoretical basis to describe the rationale for counterfactual thoughts. According to
them, counterfactual thinking is driven by simulations of previously encoded exemplars
and they emphasize the role of counterfactual thinking in reframing such scenarios—
generating alternative possibilities that change the norms (and expectations) used to
interpret a situation [22]. The theory suggests that the counterfactual alternatives cre-
ated depend on the ease of imagining different outcomes. The norms involve a pairwise
comparison between a cognitive standard and an experiential outcome. A discrepancy
that is created by such a comparison elicits an affective response which is influenced
by the magnitude and direction of the difference.
Rational Imagination Theory proposed by Byrne [5] says that the counterfactual im-
agination is rational and it depends on three assumptions: (1) humans are capable of
rational thought; (2) they make inferences by thinking about possibilities; and (3) their
counterfactual thoughts rely on thinking about possibilities, just as rational thoughts do.
Byrne [6] proposed a set of cognitive directives that guide these possibilities when peo-
ple imagine alternatives. The theory states that individuals' ability to entertain multiple
parallel models corresponding to alternative possibilities suggests that counterfactual
thought is engaged to search the space of possible alternatives.

According to Byrne [7], an algorithm to specify the mental representations and cog-
nitive processes that create counterfactuals takes as input the relevant facts of the actual
event and produces as output a counterfactual alternative. The intervening processes
change aspects of the mental representation of the facts to create a second mental rep-
resentation, the counterfactual alternative. According to [8], the dynamic nature of
memory reconstruction allows to mentally modify aspects of autobiographical memory
when simulating on retrieval, leading to counterfactual thinking. Computational mech-
anisms underlying counterfactual thinking maintain and update two representations, the
imagined alternative and the known or presupposed reality.

The neural representations of counterfactual inference that are implicated in the neu-
ral systems for constructing mental models of the past and future, involve prefrontal
and medial temporal lobe structures of the brain [10, 21]. A functional perspective on
counterfactual thinking views it as a useful, and essential component of behavior regu-
lation. It considers counterfactual thoughts closely connected to goal cognitions where
counterfactual thinking is activated usually by goal-failure [9]. It suggests that at its
root counterfactual thinking is a regulatory loop-governing behavior which operates
through a negative feedback model. This model operates by preserving homeostasis by
correcting behavior when a discrepancy is detected between the current state and an
ideal reference state for example goal-progress. In the theory of core affect [14], affec-
tive experiences function as indicators of a discrepancy between current and an ideal
state, thus affect often mediates behavior change. Also, once the discrepancy reduces
between the current state and the reference state, corrective activity is terminated. In a
review on counterfactual thinking [9], it was mentioned that cognitive experiments in-
dicated counterfactual thinking influencing behavior by either of two routes: (1) a con-
tent-specific pathway where specific informational events affect behavioral intentions
which then influence the behavior itself and (2) a content neutral pathway which has
indirect effects by affect, mindsets and motivations.

The structured event complex theory proposed by Barbey et al. [1], state that coun-
terfactual thinking engages a network of regions within prefrontal cortex (PFC) that
represent alternative goals, behavioral intentions, mindsets, motivations, and self-infer-
ences that enable behavioral change and adaptation [22]. They also stated that counter-
factual thought depends on mental models of alternative possibilities that are repre-
sented in the form of structured event complexes (SEC). SEC is a goal-oriented set
of events that is structured in sequence and represents event features (like agents, ob-
jects, actions, mental states and background settings), social norms of behavior, ethical
and moral rules and temporal event boundaries.
Building upon these frameworks, Van Hoeck et al. [22] proposed that counterfactual thinking depends upon the co-ordination of multiple information processing systems that involve three neural networks: (1) the mental simulation network, (2) the cognitive control network, (3) the reward network. Thus, they proposed three stages of processing in counterfactual thinking: Activation, Inference and Adaptation; see Fig. 1.

<figure><img src="image_2.png" /><figcaption>**Fig 1.** Schematic overview of the stages in counterfactual thinking [22]</figcaption></figure>

**Activation.** Counterfactual thoughts are triggered automatically in response to real world experiences, especially negative emotions triggered by violations of expectations and motivations (e.g., an unsuccessful application for a university), implicit/explicit goal failure (e.g., failing an exam), or close calls (e.g., missing the train by few seconds). As mentioned earlier, they proposed that counterfactuals depend upon the mental models of alternative possibilities in form of mental simulations. Such simulations provide the foundation for constructing and evaluating mental models of reality and of imagined alternative possibilities. Van Hoeck et al. [22] state that counterfactuals activate areas of the medial prefrontal cortex which are related to conflict detection.

**Counterfactual Inference.** Counterfactual inference adheres to the ‘nearest possible world’ constraint. Thus, a counterfactual must closely model one’s own experience of the real state of the world which helps in setting the constraint to specific situational features and prior knowledge of the situation. The alternatives suggested from such a constraint deviates only marginally from the reality, thus is probable. This also separates counterfactual thinking from less constrained forms of imagination or fantasy. For example, the chosen counterfactuals often are around factors like: (1) factor that played the strongest role (2) most deviation from expectation, or (3) is mostly under participant’s own control. Apart from being influenced by meaning and relevance of specific events, counterfactuals are influenced by a specific individual perspective where implicit belief of attainability or self-efficacy plays an important role. The counterfactual outcome value plays an important role and impacts how an individual perceives the factual, as well as the experienced outcome and its relative value. Such evaluations of how a counterfactual could have been better lead to emotional and social reactions.

**Learning and Adaptation.** The information inferred from counterfactual state of affairs will then be incorporated in the representation of the current state of affairs leading to re-evaluation and updating of prior beliefs and action-values. This leads to behavioral and affective modifications that last for the future. Counterfactual thinking provides an opportunity to improve performance for the future and elicits behavioral motivations to pursue the counterfactual outcome. This regulates one’s perception of control and preparedness, boosting persistence and performance [22].
## 3 The Modeling Approach for Controlled Adaptive Networks

Temporal-causal network models as addressed in [19, 20] can be represented by a conceptual representation and by a numerical representation. A conceptual representation involves representing in a declarative manner states and connections between them that represent the causal impacts of states on each other. The states are assumed to have activation levels that vary over time. Causal relations have weights. Furthermore, when more than one causal relation affects a state, some way to aggregate multiple causal impacts on a state is used. This aggregation is indicated by some combination function from a library. For the timing of the state dynamics, a speed factor is used, so that no synchronous processing is required. The notions for connectivity (connection weights $\omega_{X,Y}$), aggregation (combination functions $c_Y(\cdot)$) and timing (speed factors $\eta_Y$) are the network characteristics that define a conceptual representation of a temporal-causal network model; they are summarized in Table 1, first four rows.

<table><caption>**Table 1.** Conceptual and numerical representation of a temporal-causal network [19, 20].</caption><thead><tr><th>Concepts</th><th>Notation</th><th>Explanation</th></tr></thead><tbody><tr><td>States and connections</td><td>X, Y, X→Y</td><td>Describes the nodes and links of a network structure</td></tr><tr><td>Connection weight</td><td>ω<sub>X,Y</sub></td><td>The connection weight ω<sub>X,Y</sub> represents the strength of the causal impact of state X on state Y with X→Y</td></tr><tr><td>Aggregating multiple causal impacts</td><td>c<sub>f</sub>(..)</td><td>For each state Y a combination function c<sub>Y</sub>(..) is chosen to combine the causal impacts of other states on state Y</td></tr><tr><td>Timing of the causal effect</td><td>η<sub>Y</sub></td><td>For each state Y a speed factor η<sub>Y</sub> ≥ 0 is used to represent how fast a state is changing upon causal impact</td></tr><tr><th>Concepts</th><th>Numerical representation</th><th>Explanation</th></tr><tr><td>State values over time t</td><td>Y(t)</td><td>At each time point t each state Y in the model has a real number value</td></tr><tr><td>Single causal impact</td><td>impact<sub>X,Y</sub>(t) = ω<sub>X,Y</sub>X(t)</td><td>At t state X with a connection to state Y has an impact ω<sub>X,Y</sub>X(t) on Y, using weight ω<sub>X,Y</sub></td></tr><tr><td>Aggregating multiple causal impacts</td><td>aggl impact<sub>Y</sub>(t) = c<sub>Y</sub>(impact<sub>X<sub>1,1</sub></sub>(t),..., impact<sub>X<sub>k,k</sub></sub>(t)) = c<sub>f</sub>(ω<sub>X<sub>1,1</sub></sub>X<sub>1</sub>(t), ..., ω<sub>X<sub>k,k</sub></sub>X<sub>k</sub>(t))</td><td>The aggregated impact of X<sub>1</sub> on Y at t, is determined by applying combination function c<sub>f</sub>(..) on impact<sub>X<sub>k,k</sub></sub>(t).</td></tr><tr><td>Timing of the causal effect per state</td><td>Y(t+Δt) = Y(t) + η<sub>Y</sub> [aggl impact<sub>Y</sub>(t) - Y(t)] Δt = Y(t) + η<sub>Y</sub>[c<sub>Y</sub>(ω<sub>X<sub>1,1</sub></sub>X<sub>1</sub>(t), ..., ω<sub>X<sub>k,k</sub></sub>X<sub>k</sub>(t)) - Y(t)]Δt</td><td>The causal impact on Y is exerted over time gradually, using speed factor η<sub>Y</sub></td></tr></tbody></table>

Combination functions can be selected from an available combination function library provided by the dedicated software environment that has been developed. For each state Y one or more basic combination functions $c_f(\cdot)$, $j = 1, \dots, m$ can be selected by indicating weights $\gamma_{j,Y}$ (real numbers) which makes that within the software environment a weighted average of these functions $c_f(\cdot)$, $j = 1, \dots, m$, from the library is used as combination function $c_Y(\cdot)$ for state Y; these basic combination functions $c_f(\cdot)$ have combination function parameters $\pi_{i,j,Y}$. Currently there are more than 40 combination functions in the library. New combination functions can be added easily and the library has also facilities to apply function composition to define new functions by composing any number of functions from the library. In the model presented here, for the states, the two basic combination functions shown in Table 2 were used.
<table><caption>**Table 2** The basic combination functions from the library used in the presented model</caption><thead><tr><th></th><th>Notation</th><th>Formula</th><th>Parameters</th></tr></thead><tbody><tr><td>Advanced lo-gistic sum</td><td>a<br>logistic<sub>σ,τ</sub>(V<sub>1</sub>, ..., V<sub>k</sub>)</td><td>[<sup>1</sup>&frasl;<sub>1+e<sup>-σ(V<sub>1</sub>+...+V<sub>k</sub>-τ)</sup></sub> - <sup>1</sup>&frasl;<sub>1+e<sup>-στ</sup></sub>](1+e<sup>-στ</sup>)</td><td>Steepness σ&gt;0<br>Threshold τ</td></tr><tr><td>Stepmod</td><td>stepmod<sub>p,δ</sub>(V<sub>1</sub>, ..., V<sub>k</sub>)</td><td>0 if t mod ρ &lt; δ, else 1</td><td>Repetition ρ<br>Duration δ</td></tr></tbody></table>

Note that ‘network characteristics’ and ‘network states’ are two distinct concepts for a network. Self-modeling is a way to relate these concepts to each other in an interesting and useful way. A self-model is making the implicit network characteristics (such as connection weights or excitability thresholds) explicit by adding states to the network representing these characteristics. Thus, the network gets an internal self-model of part of the network structure; this can be used to obtain an *adaptive network*; see [19]. In this way, multiple self-modeling levels can be created where network characteristics from one level relate to states at a next level. This can be used to design *second-order* or *higher-order adaptive networks*; see, for example, [19]. More specifically, adding a self-model for a temporal-causal network is done in the way that for some of the states *Y* of the base network and some of its related network structure characteristics for connectivity, aggregation and timing (i.e., some from *ω*<sub>X,Y</sub>, *γ*<sub>i,Y</sub>, *π*<sub>i,j,Y</sub>, *η*<sub>Y</sub>), additional network states *W*<sub>X,Y</sub>, *C*<sub>i,Y</sub>, *P*<sub>i,j,Y</sub>, *H*<sub>Y</sub> (self-model states) are introduced:

**Connectivity self-model**

Self-model states $W_{X,Y}$ are added representing connection weights $\omega_{X,Y}$

**Aggregation self-model**

Self-model states $C_{j,Y}$ are added representing combination function weights $\gamma_{i,Y}$ and/or self-model states $P_{i,j,Y}$ representing combination function parameters $\pi_{i,j,Y}$

**Timing self-model**

Self-model states $H_Y$ are added representing speed factors $\eta_Y$

The notations $W_{X,Y}$, $C_{i,Y}$, $P_{i,j,Y}$, $H_Y$ for the self-model states indicate the referencing relation with respect to the characteristics $\omega_{X,Y}$, $\gamma_{i,Y}$, $\pi_{i,j,Y}$, $\eta_Y$: here $W$ refers to $\omega$, $C$ refers to $\gamma$, $P$ refers to $\pi$, and $H$ refers to $\eta$, respectively. For the processing, these self-model states define the dynamics of state $Y$ in a canonical manner according to equations in Table 2, bottom row, whereby $\omega_{X,Y}$, $\gamma_{i,Y}$, $\pi_{i,j,Y}$, $\eta_Y$ are replaced by the state values of $W_{X,Y}$, $C_{i,Y}$, $P_{i,j,Y}$, $H_Y$ at time $t$, respectively.

As the outcome of the addition of a self-model to a temporal-causal network is also a temporal-causal network model itself, as has been shown in [19], Ch 10, this construction can easily be applied iteratively to obtain multiple levels of self-models. Therefore second-order adaptation as, for example, plays an important role to control adaptive processes, can easily be modelled as well. This also has been applied here for the control of the processes in counterfactual thinking.
4 A Controlled Adaptive Network Model for Counterfactual
Thinking

To explain the introduced network model, the following scenario is used.

**Scenario:** Jimmy believes he can do an internship & study at the same time (Belief). He fails an exam (goal failure), this created an unpleasant situation for him (Feeling).

**Activation Process:** Initially this evokes a mental simulation of the entire event (simulating factual state of affairs) and previous memories are triggered. Activation spreads leading to recall of memories in similar situation from past (Search Space). He Focuses on the most relevant memories (Nearness); for example: (1) Recalls time he joined private tuitions, (2) Time he studied with friends/study group, (3) Spent extra time on weekends.

**Inference Process:** Jimmy mentally simulates based on the most relevant memories (near-est), alternatives to the situation and makes inferences and evaluations on them.

**Learning & Adaptation Process:** Once the counterfactuals are inferred, the present situation is re-evaluated by incorporating them in the present situation. This leads to changes in beliefs, as well as action-values (learning). Jimmy's behavior changes, he joins private tuitions and studies more effectively for next exams.

<figure><img src="image_3.png" /><figcaption>**Fig 2.** Connectivity of the controlled adaptive network model for counterfactual thinking.</figcaption></figure>

The lower base (pink) plane contains the base network where the different colored outlined sections are just added for reasons of presentation to make a clear visual distinction between current states, counterfactuals, future states (C-states, BS-states, AS-states, WS-States, FS-states) involved in the three stages. These colored outlines are not part of the network specification as such. The (blue) plane above the base plane represents the first-order self-model (WBS-states, ES-states, CFA-states, RCF-states, N-states). The upper (purple) plane above the first-order self-model plane represents the second-order self-model for the control (CS-states). These states at the different
levels are explained in more detail in Tables 3, 4, and 5. In the network model introduced here, the states about the current situation are represented in the red outlined section of the base plane. As mentioned in the scenario, the current situation leads to unpleasant feeling (FS1) which leads to re-evaluation and then to updating the beliefs. This update of beliefs is modelled by ES-states and WBS-states in the first-order self-model. The NS-states allow to only focus on the counterfactuals which have small deviations from reality and then choose the best among them (within the orange outlined area in the base plane).

<table><caption>**Table 3.** Description of abbreviations used in the model</caption><tr><td>C</td><td>Context States</td></tr><tr><td>BS</td><td>Belief States</td></tr><tr><td>AS</td><td>Action States</td></tr><tr><td>WS</td><td>World States</td></tr><tr><td>FS</td><td>Feeling States</td></tr><tr><td>ES</td><td>Evaluation States</td></tr><tr><td>WBS</td><td>Belief weight representation states for connections from C-states to BS-States</td></tr><tr><td>NS</td><td>Nearness Indication States</td></tr><tr><td>CFA</td><td>Counterfactual Activation states</td></tr><tr><td>RCF</td><td>Representation states for active Counterfactuals</td></tr><tr><td>CS</td><td>Control States for the three stages Activation, Inference, and Learning and Adaptation</td></tr></table>

<table><caption>**Table 4.** Overview of the Base States</caption><thead><tr><th>Base States</th><th>Explanation</th></tr></thead><tbody><tr><td>X<sub>1</sub></td><td>Context State 1</td></tr><tr><td>X<sub>2</sub></td><td>Belief State 1: 'I can do internship and studies at the same time'</td></tr><tr><td>X<sub>3</sub></td><td>Action State 1 for taking up the internship, not studying enough</td></tr><tr><td>X<sub>4</sub></td><td>World state 1 for Failing the Exam</td></tr><tr><td>X<sub>5</sub></td><td>Feeling state 1: unpleasant</td></tr><tr><td>X<sub>6</sub></td><td>Context State 2</td></tr><tr><td>X<sub>7</sub></td><td>Belief State 2 for Counterfactual thinking: 'Taking private tuitions helps to study well' so 'If I would have taken up private tuitions to prepare, as I had done in the past'</td></tr><tr><td>X<sub>8</sub></td><td>Action State 2 for Counterfactual thinking: 'Taking up private tuitions and studying effectively with help from a tutor'</td></tr><tr><td>X<sub>9</sub></td><td>World state 2 for BS2: 'Passing the Exam'</td></tr><tr><td>X<sub>10</sub></td><td>Feeling state 2: 'Feeling pleasant after passing'</td></tr><tr><td>X<sub>11</sub></td><td>Belief State 3 for Counterfactual thinking: 'one must work extra on the weekends to pass the exam' so 'If I would have studied extra on the weekends as well'</td></tr><tr><td>X<sub>12</sub></td><td>Action State 3 for Counterfactual thinking: 'Studying extra time on weekends as done in the past'</td></tr><tr><td>X<sub>13</sub></td><td>World state 3 for BS3: 'Passing the Exam'</td></tr><tr><td>X<sub>14</sub></td><td>Feeling state 3: 'Feeling pleasant after passing'</td></tr><tr><td>X<sub>15</sub></td><td>Belief State 4 for Counterfactual thinking: 'Studying with friends help to study effectively' so 'If I would have studied with my friends'</td></tr><tr><td>X<sub>16</sub></td><td>Action State 4 for Counterfactual thinking: 'studying with friends, learning effectively'</td></tr><tr><td>X<sub>17</sub></td><td>World state 4 for BS4: 'Passing the Exam'</td></tr><tr><td>X<sub>18</sub></td><td>Feeling state 4: 'Feeling pleasant after passing'</td></tr><tr><td>X<sub>19</sub></td><td>Context State 3 (Future context)</td></tr><tr><td>X<sub>20</sub></td><td>Belief State 5: 'I can will have to study harder and study more on the weekends'</td></tr><tr><td>X<sub>21</sub></td><td>Action State 5 for studying more on the weekends</td></tr><tr><td>X<sub>22</sub></td><td>World state 5 for passing the Exam</td></tr><tr><td>X<sub>23</sub></td><td>Feeling State 5: pleasant</td></tr><tr><td>X<sub>24</sub></td><td>Context State 4 (Future context)</td></tr><tr><td>X<sub>25</sub></td><td>Belief State 6: 'I can join tuitions, it will help me to study as in the past'</td></tr></tbody></table>
<table><tr><td>X<sub>26</sub></td><td>AS6</td><td>Action State 6 for joining private tuitions</td></tr><tr><td>X<sub>27</sub></td><td>WS6</td><td>World state 6 for Passing the Exam</td></tr><tr><td>X<sub>28</sub></td><td>FS6</td><td>Feeling state 6: pleasant</td></tr><tr><td>X<sub>29</sub></td><td>C5</td><td>Context state 5 (Future context)</td></tr><tr><td>X<sub>30</sub></td><td>BS7</td><td>Belief state 7: 'I Should take help from friends, study with them'</td></tr><tr><td>X<sub>31</sub></td><td>AS7</td><td>Action state 7 for taking help from friends</td></tr><tr><td>X<sub>32</sub></td><td>WS7</td><td>World state 7 for passing the Exam</td></tr><tr><td>X<sub>33</sub></td><td>FS7</td><td>Feeling state 7: pleasant</td></tr></table>

These choices make use of the ES-states ES2 to ES4 for counterfactuals which have links from the active counterfactual representation RCF-states. Based on the (persistent) evaluation states, the learning takes place: the belief weight representation WBS1 is suppressed by evaluation state ES1, and evaluation states ES2 to ES4 make that the belief weight representations WBS5 to WBS7 will become activated as soon as a relevant context occurs. Note that, in addition to the second-order CS-states for control, the first-order CFA-states and WBS-states play a crucial role in control as well. These CFA-states and WBS-states are controlled by the CS-states and in turn they themselves control the related BS-states: BS2 to BS4 by CFA1 to CFA3 and BS5 to BS7 by WBS5 to WBS7. Through this overall control, the processes involved in counterfactual thinking will take place in a structured manner according to the three stages Activation, Inference, and Learning and Adaptation as found from the literature in Section 2.

<table><caption>**Table 5** Overview of the First-Order and Second-Order Self-Model States</caption><thead><tr><th>Self-Model States</th><th>Explanation</th></tr></thead><tbody><tr><td>X<sub>34</sub></td><td>ES1 Evaluation state 1 for BS1 &amp; FS1</td></tr><tr><td>X<sub>35</sub></td><td>ES2 Evaluation state 2 for BS2 &amp; FS2 via RCF1 &amp; RCF2</td></tr><tr><td>X<sub>36</sub></td><td>ES3 Evaluation state 3 for BS3 &amp; FS3 via RCF3 &amp; RCF4</td></tr><tr><td>X<sub>37</sub></td><td>ES4 Evaluation state 4 for BS4 &amp; FS4 via RCF5 &amp; RCF6</td></tr><tr><td>X<sub>38</sub></td><td>WBS1 Belief weight representation state for the connection from C1 to BS1</td></tr><tr><td>X<sub>39</sub></td><td>WBS2 Belief weight representation state for the connection from C3 to BS5</td></tr><tr><td>X<sub>40</sub></td><td>WBS3 Belief weight representation state for the connection from C4 to BS6</td></tr><tr><td>X<sub>41</sub></td><td>WBS4 Belief weight representation state for the connection from C5 to BS7</td></tr><tr><td>X<sub>42</sub></td><td>NS1 Nearness Indication State for BS2</td></tr><tr><td>X<sub>43</sub></td><td>NS2 Nearness Indication State for BS3</td></tr><tr><td>X<sub>44</sub></td><td>NS3 Nearness Indication State for BS4</td></tr><tr><td>X<sub>45</sub></td><td>CFA1 Counterfactual Activation State 1 for BS2</td></tr><tr><td>X<sub>46</sub></td><td>CFA2 Counterfactual Activation State 2 for BS3</td></tr><tr><td>X<sub>47</sub></td><td>CFA3 Counterfactual Activation State 3 for BS4</td></tr><tr><td>X<sub>48</sub></td><td>RCF1 Counterfactual Representation State 1 for BS2</td></tr><tr><td>X<sub>49</sub></td><td>RCF2 Counterfactual Representation State 2 for FS2</td></tr><tr><td>X<sub>50</sub></td><td>RCF3 Counterfactual Representation State 3 for BS3</td></tr><tr><td>X<sub>51</sub></td><td>RCF4 Counterfactual Representation State 4 for FS3</td></tr><tr><td>X<sub>52</sub></td><td>RCF5 Counterfactual Representation State 5 for BS4</td></tr><tr><td>X<sub>53</sub></td><td>RCF6 Counterfactual Representation State 6 for FS4</td></tr><tr><td>X<sub>54</sub></td><td>CS1 Control State 1 for stage 1: via activation of WBS1 to BS1</td></tr><tr><td>X<sub>55</sub></td><td>CS2 Control State 2 for stage 2: via activation of CFA2, CFA3, CFA4 to BS2, BS3, BS4</td></tr><tr><td>X<sub>56</sub></td><td>CS3 Control State 3 for stage 3: via activation of WB5, WB6, WB7 to BS5, BS6, BS7</td></tr></tbody></table>
# 5 Simulation Results

The computational network model was simulated using a dedicated software environment implemented in MatLab described in [19], Ch 9. For an example simulation, see Figs 5 to 7 which all display one and the same simulation but just for the sake of presentation are displayed in parts for the overall processes according to the three stages found in Section 2. For the simulation $\Delta t = 0.5$ was chosen, the total time 100. The context states are considered external factors and use the **stepmod** function to let them occur at some time. The speed factor for the context states C1 and C2 was set at 0 so that they always are there, whereas for C3 to C5 it was set at 2, and by setting appropriate values for the **stepmod** function's parameters they occur at time 60. For all other states (BS-, AS-, WS-, FS-states) the speed factor was set 0.5. The connection weights between the states and the other characteristics of the network model and the initial values are shown in the Appendix; see the Linked Data at https://www.researchgate.net/publication/348324898. All states in the model have initial value 0 except C1, C2 which all have it as 1, and the NS-states (1 to 3) which have it as specific values depending on the considered variant of the scenario.

In the first stage shown in Fig. 3, due to evaluation state ES1, via WBS1 the initial belief state BS1 is suppressed and only BS5 and states that follow it go up representing that an appropriate counterfactual was chosen based on the outcome of the first stage.

The second stage is shown in Fig. 4. The NS-states are set in such a way that only one of the future states (within the green outline in the base plane) goes for value 1. A typical pattern is that first initial context state and belief state trigger a chain of events leading to activation of ES1. This leads to the activation of different counterfactuals based on their nearness and to their evaluation via evaluation states ES2 to ES4.

<figure><img src="image_4.png" /><figcaption>**Fig 3.** Present Situation States representing the impact of ES-1, WBS-1 and CS-1 on initial belief state BS-1.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>**Fig 4.** Counterfactual thinking leading to evaluation states, nearness states and updating of beliefs represented by WBS2 going up to 1.</figcaption></figure>

For the third stage shown in Fig. 5, the evaluation states ES2 to ES4 use their links to the CS3 state which controls the base-level BS5 to BS7 states through WBS5 to WBS7 states in order to make better future choices than in the past.

<figure><img src="image_6.png" /><figcaption>**Fig 5.** Future states showing the impact from WBS-states and CS-states.</figcaption></figure>
# 6 Verification of the Model by Analysis of Stationary Points

To verify whether the implemented network model behaves as expected from its conceptual specification, a number of stationary points were analyzed for a simulation example. As a stationary point for a state Y is a point where $dY(t)/dt = 0$, from the equation in Table 2, bottom row, the following general criterion for it can be derived: $\eta_Y = 0$ or

$$c_Y(\omega_{X_1,Y}X_1(t), \dots, \omega_{X_k,Y}X_k(t)) = Y(t) \quad (2)$$

where $X_1$ to $X_k$ are the states from which Y gets its incoming connections. It has been verified that the aggregated impact $\text{aggimpact}_{X_i}(t)$ defined by the left hand side of (2) matches the state value for some stationary points observed in the example simulation. The results are shown in Table 5. The model generated correct values as there were no serious deviations for the stationary points as can be seen from the table: the maximal deviation was 0.001.

<table><caption>**Table 5.** Analysis of stationary points of the model</caption><thead><tr><th>State X<sub>i</sub></th><th>X<sub>38</sub>-WBS1</th><th>X<sub>2</sub>-BS1</th><th>X<sub>3</sub>-AS1</th><th>X<sub>4</sub>-WS1</th><th>X<sub>5</sub>-FS1</th><th>X<sub>54</sub>-CF1</th><th>X<sub>52</sub>-RCF5</th></tr></thead><tbody><tr><th>Time point t</th><td>8.48</td><td>10.21</td><td>11.62</td><td>13.02</td><td>14.42</td><td>12.41</td><td>27.29</td></tr><tr><th>X<sub>i</sub>(t)</th><td>0.984526</td><td>0.98714</td><td>0.987113</td><td>0.986998</td><td>0.986879</td><td>0.910897</td><td>0.964102</td></tr><tr><th>aggimpact<sub>X<sub>i</sub></sub>(t)</th><td>0.983823</td><td>0.986161</td><td>0.98606</td><td>0.986017</td><td>0.985946</td><td>0.909804</td><td>0.963555</td></tr><tr><th>Deviation</th><td><strong>-0.0007</strong></td><td><strong>-0.00098</strong></td><td><strong>-0.00105</strong></td><td><strong>-0.00098</strong></td><td><strong>-0.00093</strong></td><td><strong>-0.00109</strong></td><td><strong>-0.00055</strong></td></tr></tbody></table>

# 7 Discussion

In the presented paper, counterfactual thinking was studied based on empirical literature from Neuroscience and Psychology. First, a detailed literature review was conducted in identifying processes, neural correlates and theories related to counterfactual thinking from different disciplines. Especially, the dynamic, adaptive and control aspects of counterfactual thinking were given the attention they deserve, as they are important but often neglected as soon as formalization or computational modeling of counterfactual thinking is addressed. A realistic scenario with respect to counterfactual thinking processes was identified. Based on the literature, a self-modeling temporal-causal network model was designed. This model captures the process of counterfactual thinking, including the dynamics, learning and control involved. Counterfactual thinking has been studied from various perspectives. But, as far as the authors know, a formalized computational model for it from a neuroscientific perspective including dynamics, adaptation and control of the thoughts and learning was never proposed.

By the implemented adaptive network model, the process of counterfactual thinking was simulated and shown to work as expected from the literature. For the model also some parameter tuning experiments have been performed which produced appropriate parameters as was expected from the literature, with a Root Mean Square Error RMSE < 0.08. Further mathematical analysis was conducted that has shown that the implemented model generated correct values compared to the model's design with maximal deviation of 0.001.
The network-oriented modeling approach used here makes it easy to integrate dif-
ferent theories and findings about a phenomenon into a more complex whole. It helps
to understand the interactions within the processes. For the current focus, the present
study contributes to theory building in understanding the processes of counterfactual
thinking from a human-like modeling perspective, thereby taking into account adapta-
tion and control for these processes. This goes much further than what is found in log-
ical approaches to counterfactual reasoning that often abstract most of the dynamic,
adaptive and control aspects involved in realistic counterfactual thinking away; e.g.,
[16].

The development of mental models for other types of learning processes has recently been addressed for the case of how mental models for operating a device are formed and how certain learning processes at the primary school take place [2-3, 20].

The notion of counterfactual thinking can also be a useful source of inspiration for (not necessarily human-like) AI applications. Counterfactual thinking has already been used as a source of inspiration to refine and optimize well-known Q-learning approaches to reinforcement learning to let agents in a multi-agent setting improve their competitive abilities. A study done by Wang et al [23], showed that counterfactual thinking can make the agents obtain more accumulative rewards from the environments with fair information in comparison to their opponents.

Further research detailing the selection of the features and evaluation process of
counterfactuals can build more accurate human-like or more useful non-human-like
models. Also further validation of the network model against empirical data or other
types of empirical information is one of the next steps to be undertaken; the process of
model building is an iterative one and always ends with an invitation to make a next
iteration.

References

1. Barbey, A.K., Krueger, F., Grafman, J.: Structured event complexes in the medial prefrontal cortex support counterfactual representations for future planning. Philosophical Transactions of the Royal Society B: Biological Sciences **364**(1521), 1291-1300 (2009).

2. Bhalwankar, R., Treur, J.: Modeling the Development of Internal Mental Models by an Adaptive Network Model. In: Proc. of the 11th Annual International Conference on Brain-Inspired Cognitive Architectures for AI, BICA*AI'20. Procedia Computer Science, Elsevier. (2021).

3. Bhalwankar, R., Treur, J.: A Second-Order Adaptive Network Model for Learner-Controlled Mental Model Learning Processes. In: Benito R.M., Cherifi C., Cherifi H., Moro E., Rocha L.M., Sales-Pardo M. (eds), Proc. of the 9th International Conference on Complex Networks and their Applications. Studies in Computational Intelligence, vol 944, pp. 245-259. Springer Nature Switzerland AG (2021).

4. Byrne, R.M.J.: Mental models and counterfactual thoughts about what might have been. Trends in cognitive sciences **6**(10), 426-431 (2002).

5. Byrne, R.M.J.: The Rational Imagination: How People Create Alternatives to Reality. MIT Press (2005).

6. Byrne, R.M.J.: Precis of ‘The Rational Imagination: How People Create Alternatives to Reality’. Behavioral and Brain Sciences **30**(5–6): 439–453 (2007).
7. Byrne, R.M.J.: Counterfactual thought. Annual review of psychology 67, 135-157 (2016).

8. De Brigard, F., Hanna, E., St Jacques, P. L., Schacter, D.L.: How thinking about what could have been affects how we feel about what was. Cognition and Emotion 33, 646-659 (2019).

9. Epstude, K., Roese, N.J.: The functional theory of counterfactual thinking. Personality and social psychology review 12(2), 168-192 (2008).

10. Fortin, N.J., Agster, K.L., Eichenbaum, H.B.: Critical role of the hippocampus in memory for sequences of events. Nature neuroscience 5(5), 458-462 (2002).

11. Kahneman, D., Miller, D.T.: Norm theory: Comparing reality to its alternatives. Psychological review 93(2), 136 (1986).

12. Markman, K.D., Gavanski, I., Sherman, S.J., McMullen, M.N.: The mental simulation of better and worse possible worlds. J. of Exp. Social Psychology 29(1), 87-109 (1993).

13. Roese, N.J.: The functional basis of counterfactual thinking. Journal of personality and Social Psychology 66(5), 805 (1994).

14. Russell, J.A.: Core affect and the psychological construction of emotion. Psychological review 110(1), 145 (2003).

15. Sanna, L.J., Schwarz, N., Small, E.M.: Accessibility experiences and the hindsight bias: I knew it all along versus it could never have happened. Memory & Cognition 30(8), 1288-1296 (2002).

16. Starr, W.B.: Conditional and Counterfactual Logic. In: Knauff, M., Spohn, W. (eds.). The Handbook of Rationality. Cambridge, MA: MIT Press (2020).

17. Timberlake, B.: The effects of counterfactual comparison on learning and reasoning (Doctoral dissertation, University of Trento) (2019).

18. Treur, J.: Network-Oriented Modeling: Addressing Complexity of Cognitive, Affective and Social Interactions. Springer Nature, Cham, Switzerland (2016).

19. Treur, J.: Network-Oriented Modeling for Adaptive Networks: Designing Higher-Order Adaptive Biological, Mental and Social Network Models. Springer Nature, Cham, Switzerland (2020).

20. Treur, J.: An Adaptive Network Model Covering Metacognition to Control Adaptation for Multiple Mental Models. Cognitive Systems Research 67, 18-27 (2021).

21. Tulving, E., Markowitsch, H.J.: Episodic and declarative memory: role of the hippocampus. Hippocampus 8(3), 198-204 (1998).

22. Van Hoeck, N., Watson, P.D., Barbey, A.K.: Cognitive neuroscience of human counterfactual reasoning. Frontiers in human neuroscience 9, 420 (2015).

23. Wang, Y., Wan, Y., Zhang, C., Bai, L., Cui, L., Yu, P.: Competitive Multi-Agent Deep Reinforcement Learning with Counterfactual Thinking. In 2019 IEEE International Conference on Data Mining (ICDM) (pp. 1366-1371). IEEE (2019).

24. Wilson, R.C., Collins, A.G.: Ten simple rules for the computational modeling of behavioral data. Elife 8, e49547 (2019).