# On Cognitive Searching Optimization in Semi-Markov Jump Decision Using Multistep Transition and Mental Rehearsal

Authors: Bingxuan Ren; Tangwen Yin; Shan Fu
Journal: Complexity
 (2021)

---

## Abstract

Cognitive searching optimization is a subconscious mental phenomenon in decision making. Aroused by exploiting accessible human action, alleviating inefficient decision and shrinking searching space remain challenges for optimizing the solution space. Multiple decision estimation and the jumpy decision transition interval are two of the cross-impact factors resulting in variation of decision paths. To optimize the searching process of decision solution space, we propose a semi-Markov jump cognitive decision method in which a searching contraction index bridges correlation from the time dimension and depth dimension. With the change state and transition interval, the semi-Markov property can obtain the action by limiting the decision solution to the specified range. From the decision depth, bootstrap re-sampling utilizes mental rehearsal iteration to update the transition probability. In addition, dynamical decision boundary by the interaction process limits the admissible decisions. Through the flight simulation, we show that proposed index and reward vary with the transition decision steps and mental rehearsal frequencies. In conclusion, this decision-making method integrates the multistep transition and mental rehearsal on semi-Markov jump decision process, opening a route to the multiple dimension optimization of cognitive interaction.


---

## Body

## 1. Introduction

The human-computer cognitive interaction (HC2I) process can be embodied to analyze human factors, interactive performance, and decision uncertainty. In terms of decision making, the decision solution space constructed by the estimation and searching for the solution path is under effect with the uncertainty of human decision [1]. In the HC2I process, the chronologically ordered decision path based on human experience is composed of each decision action step which is uniquely determined under the estimation of the future decision path. From a prior perspective, due to the influence of decision jumpy intervals and the multiple estimation of decision paths, there are infinite possibilities while deciding the decision path from its solution space. It is necessary to reduce the impact of the exploration of solutions on the efficiency of decision making. In the optimization of the cognitive searching, human’s high-level control hierarchy makes preparations for upcoming decision before people realize it [2]. When exploring the decision solution space, searching contraction optimization is used to show that people have subconsciously eliminated some decision paths that would not actually be made.In order to analyze decision behaviors, human performance modeling (HPM) has been researched in the last few decades [3]. HPM tends to demonstrate the interactive relations through designing different inner structures. It evolves from the broad symbolism cybernetic approaches [4] to the new stage of computational rational modeling [5], involving human cognitive behavior at various decision hierarchies. Similar to the non-homogeneous sequential model, decisions are continuously generated in the chronological order. Through depicting the potentially possible distribution caused by differentiation structures, HPM essentially contracts and prunes the immense decision sequence formed by rehearsal.Similar to the human-like behavior, multistep decision is mutually influenced during the periods rather than the instant moment. The Markov property, strengthening the correlation in decision path, constricts that the selection of action elements is only relevant to the decision adopted at the previous moment. Based on the Markov decision principle, the policy iteration method calculates the one-step reward value by introducing the state of decision object to computation [6]. However, existing physical obstruction makes humans unable to access state parameters without sensor measurement in the interaction environment, which causes unavoidable deviation. To cover this shortage, the partially observable Markov decision method uses interactive object state as an uncertain estimation of observable state set, which also is the main difference from the observable state Markov decision method [7]. Another completely unobservable method named hidden Markov decision analyzes human cognitive behaviors through the state of the interactive object (such as machine). In the aforementioned methods, there exist similar deficiencies on depicting the transition state interval and calculating reward value in a locality. Therefore, the variant intervals and reward value combined by history decision steps and future decision steps are crucial.On the other hand, adjacent action which lacks mutual influence is a shortcoming for Markov property [8]. It leads to being short of tightness interference in human-like decisions. Different from the single-state inference in Markov process, the semi-Markov decision focuses on the cross-correlation of transition interval, even though the transition state is not ergodic and innumerable. Its state is jumpy and changeable accompanying with the decision process.In this paper, a semi-Markov jump decision method is proposed to optimize the human cognitive searching decision path through the multistep transition part and mental rehearsal part in a specified airplane pilot interaction scenario. We define a searching contraction index to represent the coverage degree. The coverage degree refers to a ratio between decision behaviors chosen subconscious and all accessible decision behaviors. For a more general situation, the semi-Markov decision process overcomes the restriction by adopting the time-varying transition rate. Thus, the sojourn time between each mode can be of any non-exponential distribution. Besides, the human making decision is different from a fixed step decision controller. The time interval in sequential decision is not a constant and is arbitrary. In addition, it cannot be modeled by noise like exponential distribution which obeys the Markov transition law. We consider the semi-Markov process and human-centered reinforcement Q-learning to realize the estimated decision solution. Depending on the state of inconsistent transition interval, the composition decision step accomplishes making decisions. As the core of decision making, the dynamical transition probability motivates state transition and action adopted. The bootstrap re-sampling frequency can abstract the mental rehearsal process by re-screening the transition probability. Finally, decision boundary influenced by the interactive object constricts the final human decision. Figure1 briefly shows above compounding relation. To summarize, this paper puts forward the following four contributions:(i)
A semi-Markov jump cognitive decision method is proposed to evaluate the dynamical cognitive interaction process. Our method integrates the semi-Markov decision transition interval, the multiple decision path estimation, and the changeable decision solution space for jump state.(ii)
The transition interval and sojourn time, which are of vital importance characteristics, have been preferably reflected in our method. By adding mental rehearsal property, our method addresses the reduction of infinite-dimensional decision solution space and forward advances the dimension deduction to a smaller range.(iii)
An introduced index named searching contraction can efficiently reflect cognitive computation ability of human while exploring the decision solution space.(iv)
Our method incorporates the relation in decision time and depth, conforming to the human being’s logic of deciding and the property of transition state jumpy property.Figure 1
Semi-Markov jump decision in one kind ofHC2I process. The searching contraction ratio decreases with the decision time increasing. The ratio is calculated by different decision possibilities. The state is jumpy and changeable with time. Parts A and B show the mental rehearsal and multistep transition happening at one specified decision state. We use the vertical axes to represent mental rehearsal. It constructs the decision dimension together with the multistep decision.The rest of the paper is organized as follows. Section2 briefly describes the related works about multistep transition, mental rehearsal, and dynamical searching dimension in decision. Section 3 and Section 4 emphasize the specific problem and illustrate how our decision method is built for decision making. Section 5 and Section 6 detail the experiments and the integral analysis of this model, including its shortcomings, and the future directions in this area. Section 7 summarizes this paper.


## 2. Related Work

### 2.1. Multistep Transition

The multistep transition happening in the decision making has been developed with many methods [9], such as reinforcement learning [10], utility selection theory [11], and networked control system [12]. Compared to the continuously accumulated and improved process, the common decision framework is to obtain an optimal decision strategy via the feedback effect [13] and evaluate the potential outcome values caused by events when decision is formulated by the feedback loop design [14]. Focusing on cognitive analysis, Yanco and Drury [15] modified a taxonomy of multiagent systems and treated the human-computer interactions as a process of two heterogeneous agent interactions. Moratz et al. [16] experimented with a comparison test between human-robot and human-human to illustrate the difference in spatial features. The comparative trials implicitly revealed that the complexity of cognitive space plays a prominent role in the interaction process. From the view of multiple timescales, such as cognition and decision, Purcell and Kiani [17] designed a hierarchy of multistep transition decision on processes to disambiguate the detrimental factors such as flawed information. All of the above works mainly focused on the differences between human and robot as the autonomous agent. They considered human as uncertain and non-monotonous agent by adding the stochastic-dynamic transition interval which accords with exponential distribution. Wu et al. [18] added the transition time restriction into the semi-Markov model while its finite system state was limited into the noninteractive modal. To the best of our knowledge, the non-exponential distribution of transition interval which can be used into cognitive decision making has received limited effort so far. Decision interval of Markov jumping provides a more general way to describe the multistep transition for cognitive decision.


### 2.2. Mental Rehearsal

Mental rehearsal, also known as mental simulation, is one of the cognitive strategies [19]. This strategy takes future action-practice without outer observed physical performance. In the typical task, it is regarded as one of the efficient methods to improve the decision performance of the psychomotor and sport. For example, Miranda et al. [20] used mental rehearsal to decrease depressive predictive certainty, which showed the gains in making optimistic predictions. Ignacio et al. [21] proposed that different health disciplines can utilize mental rehearsal strategy as a part of clinical training. Su et al. [22] designed incremental deep convolutional neural network process to demonstrate the human-like learning behavior. Moreover, researchers analyzed its different effects on the user’s learning decision in the theory of working memory [23]. As a computational model, Oberauer and Lewandowsky [24] designed a time-based resource-sharing theory to derive unambiguous predictions about the effect of rehearsal on memory, which is beneficial for differentiating between varying forms of mental practice. Besides, mental rehearsal can be analyzed by parameterized formation. To demonstrate the advantage of rehearsal, Mazher et al. [25] found that rehearsal was beneficial for memorized long-term learning by discriminating the learning decision states using electroencephalography.


### 2.3. Dynamical Decision Dimension

Exploration-exploitation related to the dynamical decision dimension is a crucial aspect, especially for searching the feasible solutions [26]. The dimensional optimization method connects with the decision-making property such as the non-Markovian property [27], which is used to describe the cross-influence between different decision states. To reduce the dimension of decision searching, Engel et al. [28] considered the stochastic jumpy interval in human cognitive decision behaviors and handled it with a linearity weighted logic according to monotonically increased time [29]. To get the global optimal solution, the brute-force calculation method is used. But it is easily trapped into the plight to search the space in finite polynomial time, especially under the non-convex issues. Some proposed optimal algorithms such as best proximity points [30] and particle swarm optimization [31] were applied to evade the non-convex difficulties. However, there still exists an enormous gap between human physical simulation and computational simulation like emotions [32]. The state caused by human action is discretely jumpy rather than the inflexible inference from a fixed step to another. In addition, the uncertain human factors enlarge the difficulty covering decision solution space of all accessible scenes [33].


## 2.1. Multistep Transition

The multistep transition happening in the decision making has been developed with many methods [9], such as reinforcement learning [10], utility selection theory [11], and networked control system [12]. Compared to the continuously accumulated and improved process, the common decision framework is to obtain an optimal decision strategy via the feedback effect [13] and evaluate the potential outcome values caused by events when decision is formulated by the feedback loop design [14]. Focusing on cognitive analysis, Yanco and Drury [15] modified a taxonomy of multiagent systems and treated the human-computer interactions as a process of two heterogeneous agent interactions. Moratz et al. [16] experimented with a comparison test between human-robot and human-human to illustrate the difference in spatial features. The comparative trials implicitly revealed that the complexity of cognitive space plays a prominent role in the interaction process. From the view of multiple timescales, such as cognition and decision, Purcell and Kiani [17] designed a hierarchy of multistep transition decision on processes to disambiguate the detrimental factors such as flawed information. All of the above works mainly focused on the differences between human and robot as the autonomous agent. They considered human as uncertain and non-monotonous agent by adding the stochastic-dynamic transition interval which accords with exponential distribution. Wu et al. [18] added the transition time restriction into the semi-Markov model while its finite system state was limited into the noninteractive modal. To the best of our knowledge, the non-exponential distribution of transition interval which can be used into cognitive decision making has received limited effort so far. Decision interval of Markov jumping provides a more general way to describe the multistep transition for cognitive decision.


## 2.2. Mental Rehearsal

Mental rehearsal, also known as mental simulation, is one of the cognitive strategies [19]. This strategy takes future action-practice without outer observed physical performance. In the typical task, it is regarded as one of the efficient methods to improve the decision performance of the psychomotor and sport. For example, Miranda et al. [20] used mental rehearsal to decrease depressive predictive certainty, which showed the gains in making optimistic predictions. Ignacio et al. [21] proposed that different health disciplines can utilize mental rehearsal strategy as a part of clinical training. Su et al. [22] designed incremental deep convolutional neural network process to demonstrate the human-like learning behavior. Moreover, researchers analyzed its different effects on the user’s learning decision in the theory of working memory [23]. As a computational model, Oberauer and Lewandowsky [24] designed a time-based resource-sharing theory to derive unambiguous predictions about the effect of rehearsal on memory, which is beneficial for differentiating between varying forms of mental practice. Besides, mental rehearsal can be analyzed by parameterized formation. To demonstrate the advantage of rehearsal, Mazher et al. [25] found that rehearsal was beneficial for memorized long-term learning by discriminating the learning decision states using electroencephalography.


## 2.3. Dynamical Decision Dimension

Exploration-exploitation related to the dynamical decision dimension is a crucial aspect, especially for searching the feasible solutions [26]. The dimensional optimization method connects with the decision-making property such as the non-Markovian property [27], which is used to describe the cross-influence between different decision states. To reduce the dimension of decision searching, Engel et al. [28] considered the stochastic jumpy interval in human cognitive decision behaviors and handled it with a linearity weighted logic according to monotonically increased time [29]. To get the global optimal solution, the brute-force calculation method is used. But it is easily trapped into the plight to search the space in finite polynomial time, especially under the non-convex issues. Some proposed optimal algorithms such as best proximity points [30] and particle swarm optimization [31] were applied to evade the non-convex difficulties. However, there still exists an enormous gap between human physical simulation and computational simulation like emotions [32]. The state caused by human action is discretely jumpy rather than the inflexible inference from a fixed step to another. In addition, the uncertain human factors enlarge the difficulty covering decision solution space of all accessible scenes [33].


## 3. Problem Formulation

TheHC2I process is able to give people insight into and observe the state information from the interactive machine object. According to the state obtained by observation and the state of the historical decision path, people make new decision in limited period. Although related work provides state of the art in terms of multistep decision, mental rehearsal, and dynamical decision dimension, existing methods cannot optimize the cognitive decision searching from dimension of time and depth. The current methods implicitly contain a flaw that the process of searching decision solution space depends on the partial history information. Limited to the single sample estimation, evaluation of decision also leads to losing the unbiasedness of decision. Additionally, cognitive searching optimization is involved with the human cognitive properties which have not been fully used in historical research, such as jumpy decision interval and multiple decision path estimation. Therefore, the problem in our work is to optimize the decision reward ℛ, generate efficient decision path b, increase the searching contraction ratio μ, and stabilize the decision solution space scope under the two human cognitive characteristics. We design a semi-Markov jump decision method using the hierarchical transition probability optimization from the different step lengths of multistep transition and mental rehearsal frequency. First, we design the semi-Markov process and reinforcement Q-learning to form the multistep transition on the basis of history fragment strategies and subjective estimation about future’s expectation feedback. Then, we build the mental rehearsal optimization based on bootstrap re-sampling, which plays an essential role for human’s subconscious simulation and optimizes transition probability. Besides, the decision space boundary ensures that decision admissible is developed.


## 4. Semi-Markov Jump Decision Method

In this section, we show the semi-Markov jump decision method for cognitive searching optimization. As shown in Figure2, the interior of method can be divided into two hierarchies. The first hierarchy indicates the targeted decision inference block. During the decision process, this block limits the interaction target domain and illustrates the maximum-minimum reward for the decision process. The second hierarchy determines searching contraction block by estimating transition distribution. Human decision memory is not amnesic instantly once action completed while considering non-Markov property [34]. It implicitly indicates that decision making does not rely on a point but a fragment. Here we use block of semi-Markov process and human-centered reinforcement Q-learning decision maker to estimate the decision state on the limited length fragment, which is capable of jumpy transition interval belonging to non-exponential distribution. The bootstrap re-sampling controller block is designed to explore the optimal transition probability for the human mental rehearsal. Depending on flight phase, decision space boundary block limits decision to admissible scope for decision inference and flight dynamics. Besides, airplane flight simulating block receives the observable airplane state information from space N∗, whereas it handles executable action parameters from the block of decision inference target.Figure 2
The architecture of semi-Markov jump decision method. The multistep transition and mental rehearsal are represented separately by the decision makerKΔ and bootstrap re-sampling controller KΛ in the bottom block. It receives the probability tj and calculates b and Q for searching contraction block. The airplane model block receives different control parameters from human decision making and outputs the observable state stl=2. The decision space boundary calculates the admissible control scope for the final decision judgment.


### 4.1. Targeted Decision Inference on Receding Horizon

The whole cognitive interaction is defined in theHC2I space N. Interior of decision method is defined in decision solution space N∗. Also, we define the inner bootstrap space as re-sampling with replaceable space N†, which is a subset of space N∗. Similar to a sliding surface forcing the system state in semi-Markov jump system [35], the targeted decision inference is addressed on receding horizon. In space N, the aimless interaction decision is excluded in the scope of paper. We assume that the HC2I process has preassigned target set Θ where its elements relate to machine (computer) state at each decision π. The constrained loss function is given below, which is similar to a filter using energy comprehensive index to get the optimal decision trail Π under the specified target.(1)∏π∗∈Πargmaxθ1argminθ2ℱπ|Eτ∼pθϕτℛtotalπ,where(2a)pθϕπ=μs0∏t=1T−1pϕat+1πθat,st,(2b)pϕat+1=Paj+1|a1,a2,…,aj=Paj+1|aj−i,aj−i−1,…,aj−1,aj,i≥1,j=t,(2c)πθat,st=supπ=x∗|ϵ∗x∗,Fn=θ^fn∗−θfn,where ℛ represents the synthesis reward function decided by decision process; it minimizes the energy consumed by interaction of cognitive and gets a sequence maximizing the computer performance while the decision π follows distribution of trajectory density function parameterized by both transition probability and stochastic error factor. pϕat+1 states that the HC2I process of is capable of semi-Markov factor θ1. Also, both the temporal fragment factor and jumpy factor make contribution and intervention in process. πθat,st states that mental rehearsal factor θ1 contributes to the decision. We need to get the optimal policy under the minimum condition θ2 to ascertain the maximum θ1. θ1 denotes the maximum value πθat,st process. θ2 relates to the minimum of smaller worst cost function ℛ. At each time step t, the agent is in state st∈S and must choose an action at∈A, transitioning it to a new state st+1Pst+1|st,at and yielding a reward Rst,at. A policy π:S×A⟶0,1 is defined as a probability distribution over state-action pairs, where πat|st represents the density of selecting action at in state st. Upon consequent interactions with the environment, the agent collects a trajectory τ of state-action pairs. The goal is to determine an optimal policy π∗ by this loss function.Besides, the constrained loss function satisfies two implicitly postulated conditions. The first condition indicates that the step number of decision is limited. It shows that the cognitive interaction exists in a terminal state. Also, the solution space is bounded by the environment tasks. The second assumption explains that the computer or machine state pattern is similar during this trail process, and it assures that the process can be properly classified into several stages.


### 4.2. Semi-Markov Process and Human-Centered Q-Learning Decision Maker for Multistep Transition

Decision makerKΔ is composed of a hybrid semi-Markov process with the forward human-centered Q-learning estimation. To determine Q parameter in ℬkQsi,ai for equation (19), we consider composite decision maker KΔ by the semi-Markov process and human-centered Q-learning. The former takes jumpy property and sojourn time of decision interval into consideration by the semi-Markov process, while the latter calculates future predictive estimation in N†. Figure 3 shows the sketch of this part.Figure 3
Semi-Markov process and human-centered Q-learning decision makerKΔ for synthesis consideration. The figure shows how history consideration decision and future estimation decision make effect together on the time series names as receding horizon. Interval time Δ is jumpy transition between accord with the non-exponential distribution.The sampled discrete state trail (specified valuek) of state-action pair is Strail≜S0,S1,…,St,…,Sn. Subscript n is the number of state elements in trail. It is hidden left part of b∗ from start of decision process. The action ai is chosen from the action set. We define A as A≜a1,a2,…,am, where m is the number of admissible control elements in decision πt. O=o0,o1,o2,…,ov is observable state variable from process M^t and its subscript v is dimension for stl when l refers to M^t. Based on the Markov property PXt=y|Xr,0≤r≤s=PXt=y|Xs and jumpy transition probability PXt+δt=x|Xt=y=αy,xδt+oδt,y≠x, the following functions state the semi-Markov relation between adjacent states si. Here, Xt is the stochastic process, αx,y denotes the jumpy transfer rate from state x to adjacent state y, δt is the decision interval following the non-exponential distribution, and oδt denotes a high-order stochastic variable which is small.(3a)pst|st−1,at−1,…,s0,a0=pst|st−1,at−1,…,st−step,at−step,(3b)pst|st−1,at−1=αst,st−1δt+oΔt,(3c)∑S∑st,atpst|∪st−1,at−1,…,st−Sstep,at−Sstep=1.In spaceN†, the quadruple S,A,O,T is the composite of elements in observable semi-Markov discrete process Mt. T=0,∞ and t is discrete Lebesgue additive on the measure μ. The state transition is denoted as operation F:O×S⟶AS′π; then, the observable state ot is from rigorous time homogeneous continuous Markov process M^t whose tuple form is S^,A^,T and F^:S^×A^⟶S^′π^. According to m, dimension of action set at is dynamically changeable under the updating transition probability p. For M^t process, the element of action set A^ is the same as A and it receives the decision ai from Mt. When a new decision is determined by Mt, at will be transmitted to M^t+1 after the correction v⋅. v⋅ is normal distribution represented by the transition error. Here we assume that M^t does not exhibit parameter drift such as time delay factor. We write the actual action set form A and A^ as(4a)At=Ap⋅|t|st=a1,a2,…,am′,m′∈1,m,(4b)A^t=vAt=a^t1,a^t2,…,a^tm′.The accumulated stateS and indicator state S^ are calculated from state in the Mt and M^t. Therein, semi-Markov process Mt follows the non-exponential distribution sojourn time Δt but M^t is continuous without sojourn time. We estimate it by the interval information entropy. Here wat−1 is the weight coefficient of action, and superscript r′ is the observable state dimension in M^t.(5a)St=diagst,os^t,st=∑1mwAt−1at−1m′,(5b)S^t=s^t=s^t1,s^t2,…,s^tr′.Also, the semi-Markov processMt within human and regular Markov process M^t within machine (computer) happen synchronously, while the former is discrete and the latter is continuous. Therefore, the intervention relations between Mt and M^t can be represented as follows. Here ξ,ζ is the error variable that follows normal distribution. ΔMt^,ΔMt separately indicate the continuous and discrete interval period from different processes.(6a)dS=dS^+ξ+OS^ΔMt^,(6b)dA=dA^+ζ+OA^ΔMt.For regular Markov processMt^, the expectation performance EMt^ is obtained at decision πt. According to Doeblin lemma [36], let P^ be a transition probability matrix: ∀i∈N†,Pij0≥ε when j0∈Sandε>0, there exists only stationary probability vector πj0≥ε in P, and for all initial distribution μ, μPn−πv≤21−εn,n≥0. This lemma presents that the amnesic initial of distribution exists in Mt^. On the other hand, the expectation performance EMt is an accumulated reward about state St∈S. It is a continuous additive from the initial of history consideration step to the current decision time, including jumpy transition interval and decision action time that follows normal distribution. Let Rt−ξ=t−τNtξ,Rt+ξ=τNt+1ξ−t where Ntξ=supn:τnξ≤t is the number of jumps for function ξ up to a time t. Then, probability kernel function of the semi-Markov process is(7)PxRt−≥r,Rt+≥s,Xt∈S=∑k=0∞PxXtn∈A,τn≤t−r,τn+1≥t+s=n⟶∞∫0t−r∫SPx1τ1>t+2−s1Ux,ds1×dx1,where Ux,0,t×S=∑k=0∞Pxτn≤t,Xτn∈S. Px is a measure of intensity of random point field for fixed x. Assuming that NB×S=∑n=0∞IB×Sτn,Sn represent the number of discontinuity pairs belonging to a set B×SB∈ℬR+,S∈ℬS, we have expectation EMt.(8)ExNB×S=∑n=0∞ExIB×Sτn,Xn=∑n=0∞Pxτn∈B,Xτn∈S=Ux,B×S,EMt=Ex=tnNtξ,Rt−ξ,α=∫minmaxe−δtPx=tnRt−≥r,Rt+≥s,Xt∈S=EMt−Sstep+∫t−Sstept1HEst|π,where π is history decision determined and H≡supiRi=supi−αii. In the multistep decision process, humans anticipate fuzzy assessment before making the decision [37]. Next, we use human-centered reinforcement Q-learning to estimate accumulated feedback and the maximized performance as the transition probability in short future period Qstep.b|b=π1,π2,…,πn denotes the optimization limited trail for future perdition decision sequences, and subscript n=Qstep is predicted future decision step length. Given a future estimation process X:xk⊂N†, which is right continuous part of b, we write performance index as Qn=∑k=0k=nVxk,ak. Bellman optimal theory [38] illustrates that optimal decision sequence can be divided into several blocks staying in optimal state space. It makes sure the sufficiency for division of b. According to Bellman optimal theory, we derive Qn to Qn∗=minu0Vx0,u0+Qn−1∗ [39].(9)VTπx=Eπ1Tr1+T−1T1T−1∑t=2Trt|x0=x=∑a∈Aπx,a∑x′∈Xϵx⟶x′a1TRx⟶x′a+T−1TEπ1T−1∑t=1T−1rt|x0=x′=∑a∈Aπx,a∑x′∈Xϵx⟶x′a1TRx⟶x′a+T−1TVT−1πx′,where R=1S^t, ϵ is the exploration ratio, and r is the reward. To calculate the value function Vxk,ak, we have the following derivations. First, we consider(10)VTπx=Eπ1T∑t=1Trt|x0=x=Eπ1Tr1+1T∑t=2Trt|x0=x.Furthermore, we letRx⟶x′a to denote r1 and(11)VT−1πx′=Eπ1T−1∑t=1T−1rt|x0=x′.Through the law of total probability expansion, the operationEπ⋅ is substituted by the following expression:(12)∑a∈Aπx,a∑x′∈Xϵx⟶x′a.Then, we can get the function equality equation (9). S^ is an indicator function format.(13)1S^t=1,forS^⊂Θ∗,0,forS^⊂Θ∗.For prediction processX, we assume that discrete time sequence is equidistance Tj−Ti=const. According to the Bellman equation and formula of total probability, we have the recurrence accumulated feedback. Also, this predicted reward will be used to determine optimal transition probability on space N† in the next section.(14)QTπx,a=EMt+∑x′∈Xϵx⟶x′a1TRx⟶x′a+T−1TVT−1πx′.


### 4.3. Bootstrap Re-Sampling Controller for Mental Rehearsal

Bootstrapping was introduced as a flexible method to estimate the sampling distribution of an independent observation function [40]. It takes distribution Fn from sample data to substitute the global whole data, R∗x∗,Fn=θ^Fn∗−θ^Fn, and is useful for estimating of uncertainty in subspace identification. Figure 4 shows a segment that describes bootstrap re-sampling training to search the optimal transition probability where the bootstrap re-sampling controller KΛ is used to determine the transition possibility.Figure 4
Bootstrap re-sampling controllerKΛ for transition probability through different mental rehearsal frequencies. The set of curves denotes the machine state composed of different mental rehearsal frequencies. Correspondingly, the left part is the human decision between future estimation step length Q.We haveN†≜b1∪b2∪⋯∪bk where the subscript k stands for the different sample index. To explore the theoretically infinite solution decision space N∗, we assume that its probability distribution function is in accordance with fx,θ. For each N†, it stands for a brevity decision sequence according to Q-learning estimation introduced in the next section. We use the limited re-sampling characteristic f^ to represent the global big sampling range.(15)f<st,at>,b=f^<st,at>∗,b∗.Herest,at is the state from observed aspect and decision in space N; therein, st≜diagstl=1,stl=2. The subscript t is the discrete step index in decision sequence. The superscript l stands for different observed states, such as human state and machine state.We assume that transition probability discrete distributionPr dominates the pair st,at transferring in decision sequence. To calculate Prt⋅|st−1,at−1,…,at−S≜p^t1,pt2,…,ptrT, we first set initial probability Pr0=p01,p02,…,p0rT. The capital subscript S stands for step length of history consideration decision. The superscript r is the account of decision category in that different kinds of decisions are independently identically distributed. Next, each probability element is calculated by(16)ptj=gPr0Prts=Pr0∑1ptj∑ptj.Here1pji is an indicator function.(17)1ptj=1,ifj∈1,r,0,ifj∈1,r.Furthermore, the searched optimal transition probability groupP will be determined for searching contraction Ω. In bootstrapped sample k for future estimation, reward function r, which is assumed monotonically increasing as performance improves, is used to compare the accumulated reward.(18)ℬkQ<st,at>∗=maxat−S,…,atrst−S,at−S∗+⋯+γt−1rst−1,at−1∗+γtQst,at∗k,where γ is damping factor and Q is a Q-learning estimation from time i. After given the bootstrap sample estimation value, we derive the transition probability distribution and get the decision at+1 from the optimal decision trail b∗ for the next decision step. Algorithm 1 shows the comprehensive block of searching contraction Ω.(19)PQkst=b^∗πi=1|b^∗=argmaxb∗ℬkQst,at∗.Algorithm1: Searching contraction in space N†.
(1)
Initializeα,ϵ,v,δt, and t∈T,a∈A,s∈S(2)
InitializeT,S,A,S^,Sstep,Qstep(3)
repeat(4)
Calculate history rewardEMt(5)
Initialize the boots frequencyk(6)
repeat(7)
Calculate predicted rewardQTπx,a(8)
untilQstep and b=boots(9)
Calculatetj(10)
CalculateℬkQst,at∗(11)
until Temporal fragment decision completed at t(12)
CalculatePQkst(13)
Obtain the possible decisionπ∗


### 4.4. Decision Space Boundary for Admissible Decision

The decision space boundary blockΔ in Figure 2 limits the accessible decision action set and relies on the airplane flight dynamic state variables. Based on the observed state stl=2, admissible decision action at is the subset of action control scope at=t1,…,tl′T. Meanwhile, allowable state space S=s^t is a hypercube field minV1,maxV1×⋯×minVr′,maxVr′, which will inversely limit the decision action generated from the searching contraction method. This self-triggered policy caused by interaction object contributes to jumpy updating state and executing action by relying on the latest sampled state information [41]. In the sequel, we consider the flight longitudinal dynamic system V model as follows:(20)V˙1=1mTcosV4−DV4,V3,δe−GsinV2V˙2=1mvTsinV4+LV4,V3−GcosV2V˙3=MV4,V3,δeIyV˙4=V3−V˙2.V˙1,V˙2,V˙3,V˙4 are the various rates of airspeed, vertical speed, attack angle, and pitch angular velocity. M⋅,Iy is the rotational inertia, T is the power of airplane, and δe is the engine mounting angle. We assume that airspeed is approximately equal to tangential velocity. To get the boundary block, the Hamilton–Jacobi function is needed to be solved as follows:(21)−∂Sat,t∂t=min0,H∗at,∂Sat,t∂at.ThereinH∗=sup∇V∗TVa,stl=2, and the terminal boundary condition is set as S=minmaxV1−V1,V1−minV1,…,maxVr′−Vr′,Vr′−minVr′. Then, we can obtain the effective action u∗ which stays in the scope of decision space boundary.(22)u∗=argsupat∈atHat,S,stl=2.At each decision time updated, the bounded observable stateSt,t will be generated and compared with the boundary S. Only until the decision π∗∈u∗, π can be transmitted into airplane simulating block and then the interaction can be completed. Below, we provide Algorithm 2 for the whole decision inference on the receding horizon.Algorithm2: Targeted decision inference on receding horizon.
(1)
Initial target setΘ(2)
repeat(3)
Calculate the allowable state spaceS(4)
repeat(5)
searching contractionΩ block(6)
ifπ∈u∗=argsupt∈atHt,S,stl=2then(7)
output decisionπ(8)
end if(9)
untilπ(10)
until target set is completed


## 4.1. Targeted Decision Inference on Receding Horizon

The whole cognitive interaction is defined in theHC2I space N. Interior of decision method is defined in decision solution space N∗. Also, we define the inner bootstrap space as re-sampling with replaceable space N†, which is a subset of space N∗. Similar to a sliding surface forcing the system state in semi-Markov jump system [35], the targeted decision inference is addressed on receding horizon. In space N, the aimless interaction decision is excluded in the scope of paper. We assume that the HC2I process has preassigned target set Θ where its elements relate to machine (computer) state at each decision π. The constrained loss function is given below, which is similar to a filter using energy comprehensive index to get the optimal decision trail Π under the specified target.(1)∏π∗∈Πargmaxθ1argminθ2ℱπ|Eτ∼pθϕτℛtotalπ,where(2a)pθϕπ=μs0∏t=1T−1pϕat+1πθat,st,(2b)pϕat+1=Paj+1|a1,a2,…,aj=Paj+1|aj−i,aj−i−1,…,aj−1,aj,i≥1,j=t,(2c)πθat,st=supπ=x∗|ϵ∗x∗,Fn=θ^fn∗−θfn,where ℛ represents the synthesis reward function decided by decision process; it minimizes the energy consumed by interaction of cognitive and gets a sequence maximizing the computer performance while the decision π follows distribution of trajectory density function parameterized by both transition probability and stochastic error factor. pϕat+1 states that the HC2I process of is capable of semi-Markov factor θ1. Also, both the temporal fragment factor and jumpy factor make contribution and intervention in process. πθat,st states that mental rehearsal factor θ1 contributes to the decision. We need to get the optimal policy under the minimum condition θ2 to ascertain the maximum θ1. θ1 denotes the maximum value πθat,st process. θ2 relates to the minimum of smaller worst cost function ℛ. At each time step t, the agent is in state st∈S and must choose an action at∈A, transitioning it to a new state st+1Pst+1|st,at and yielding a reward Rst,at. A policy π:S×A⟶0,1 is defined as a probability distribution over state-action pairs, where πat|st represents the density of selecting action at in state st. Upon consequent interactions with the environment, the agent collects a trajectory τ of state-action pairs. The goal is to determine an optimal policy π∗ by this loss function.Besides, the constrained loss function satisfies two implicitly postulated conditions. The first condition indicates that the step number of decision is limited. It shows that the cognitive interaction exists in a terminal state. Also, the solution space is bounded by the environment tasks. The second assumption explains that the computer or machine state pattern is similar during this trail process, and it assures that the process can be properly classified into several stages.


## 4.2. Semi-Markov Process and Human-Centered Q-Learning Decision Maker for Multistep Transition

Decision makerKΔ is composed of a hybrid semi-Markov process with the forward human-centered Q-learning estimation. To determine Q parameter in ℬkQsi,ai for equation (19), we consider composite decision maker KΔ by the semi-Markov process and human-centered Q-learning. The former takes jumpy property and sojourn time of decision interval into consideration by the semi-Markov process, while the latter calculates future predictive estimation in N†. Figure 3 shows the sketch of this part.Figure 3
Semi-Markov process and human-centered Q-learning decision makerKΔ for synthesis consideration. The figure shows how history consideration decision and future estimation decision make effect together on the time series names as receding horizon. Interval time Δ is jumpy transition between accord with the non-exponential distribution.The sampled discrete state trail (specified valuek) of state-action pair is Strail≜S0,S1,…,St,…,Sn. Subscript n is the number of state elements in trail. It is hidden left part of b∗ from start of decision process. The action ai is chosen from the action set. We define A as A≜a1,a2,…,am, where m is the number of admissible control elements in decision πt. O=o0,o1,o2,…,ov is observable state variable from process M^t and its subscript v is dimension for stl when l refers to M^t. Based on the Markov property PXt=y|Xr,0≤r≤s=PXt=y|Xs and jumpy transition probability PXt+δt=x|Xt=y=αy,xδt+oδt,y≠x, the following functions state the semi-Markov relation between adjacent states si. Here, Xt is the stochastic process, αx,y denotes the jumpy transfer rate from state x to adjacent state y, δt is the decision interval following the non-exponential distribution, and oδt denotes a high-order stochastic variable which is small.(3a)pst|st−1,at−1,…,s0,a0=pst|st−1,at−1,…,st−step,at−step,(3b)pst|st−1,at−1=αst,st−1δt+oΔt,(3c)∑S∑st,atpst|∪st−1,at−1,…,st−Sstep,at−Sstep=1.In spaceN†, the quadruple S,A,O,T is the composite of elements in observable semi-Markov discrete process Mt. T=0,∞ and t is discrete Lebesgue additive on the measure μ. The state transition is denoted as operation F:O×S⟶AS′π; then, the observable state ot is from rigorous time homogeneous continuous Markov process M^t whose tuple form is S^,A^,T and F^:S^×A^⟶S^′π^. According to m, dimension of action set at is dynamically changeable under the updating transition probability p. For M^t process, the element of action set A^ is the same as A and it receives the decision ai from Mt. When a new decision is determined by Mt, at will be transmitted to M^t+1 after the correction v⋅. v⋅ is normal distribution represented by the transition error. Here we assume that M^t does not exhibit parameter drift such as time delay factor. We write the actual action set form A and A^ as(4a)At=Ap⋅|t|st=a1,a2,…,am′,m′∈1,m,(4b)A^t=vAt=a^t1,a^t2,…,a^tm′.The accumulated stateS and indicator state S^ are calculated from state in the Mt and M^t. Therein, semi-Markov process Mt follows the non-exponential distribution sojourn time Δt but M^t is continuous without sojourn time. We estimate it by the interval information entropy. Here wat−1 is the weight coefficient of action, and superscript r′ is the observable state dimension in M^t.(5a)St=diagst,os^t,st=∑1mwAt−1at−1m′,(5b)S^t=s^t=s^t1,s^t2,…,s^tr′.Also, the semi-Markov processMt within human and regular Markov process M^t within machine (computer) happen synchronously, while the former is discrete and the latter is continuous. Therefore, the intervention relations between Mt and M^t can be represented as follows. Here ξ,ζ is the error variable that follows normal distribution. ΔMt^,ΔMt separately indicate the continuous and discrete interval period from different processes.(6a)dS=dS^+ξ+OS^ΔMt^,(6b)dA=dA^+ζ+OA^ΔMt.For regular Markov processMt^, the expectation performance EMt^ is obtained at decision πt. According to Doeblin lemma [36], let P^ be a transition probability matrix: ∀i∈N†,Pij0≥ε when j0∈Sandε>0, there exists only stationary probability vector πj0≥ε in P, and for all initial distribution μ, μPn−πv≤21−εn,n≥0. This lemma presents that the amnesic initial of distribution exists in Mt^. On the other hand, the expectation performance EMt is an accumulated reward about state St∈S. It is a continuous additive from the initial of history consideration step to the current decision time, including jumpy transition interval and decision action time that follows normal distribution. Let Rt−ξ=t−τNtξ,Rt+ξ=τNt+1ξ−t where Ntξ=supn:τnξ≤t is the number of jumps for function ξ up to a time t. Then, probability kernel function of the semi-Markov process is(7)PxRt−≥r,Rt+≥s,Xt∈S=∑k=0∞PxXtn∈A,τn≤t−r,τn+1≥t+s=n⟶∞∫0t−r∫SPx1τ1>t+2−s1Ux,ds1×dx1,where Ux,0,t×S=∑k=0∞Pxτn≤t,Xτn∈S. Px is a measure of intensity of random point field for fixed x. Assuming that NB×S=∑n=0∞IB×Sτn,Sn represent the number of discontinuity pairs belonging to a set B×SB∈ℬR+,S∈ℬS, we have expectation EMt.(8)ExNB×S=∑n=0∞ExIB×Sτn,Xn=∑n=0∞Pxτn∈B,Xτn∈S=Ux,B×S,EMt=Ex=tnNtξ,Rt−ξ,α=∫minmaxe−δtPx=tnRt−≥r,Rt+≥s,Xt∈S=EMt−Sstep+∫t−Sstept1HEst|π,where π is history decision determined and H≡supiRi=supi−αii. In the multistep decision process, humans anticipate fuzzy assessment before making the decision [37]. Next, we use human-centered reinforcement Q-learning to estimate accumulated feedback and the maximized performance as the transition probability in short future period Qstep.b|b=π1,π2,…,πn denotes the optimization limited trail for future perdition decision sequences, and subscript n=Qstep is predicted future decision step length. Given a future estimation process X:xk⊂N†, which is right continuous part of b, we write performance index as Qn=∑k=0k=nVxk,ak. Bellman optimal theory [38] illustrates that optimal decision sequence can be divided into several blocks staying in optimal state space. It makes sure the sufficiency for division of b. According to Bellman optimal theory, we derive Qn to Qn∗=minu0Vx0,u0+Qn−1∗ [39].(9)VTπx=Eπ1Tr1+T−1T1T−1∑t=2Trt|x0=x=∑a∈Aπx,a∑x′∈Xϵx⟶x′a1TRx⟶x′a+T−1TEπ1T−1∑t=1T−1rt|x0=x′=∑a∈Aπx,a∑x′∈Xϵx⟶x′a1TRx⟶x′a+T−1TVT−1πx′,where R=1S^t, ϵ is the exploration ratio, and r is the reward. To calculate the value function Vxk,ak, we have the following derivations. First, we consider(10)VTπx=Eπ1T∑t=1Trt|x0=x=Eπ1Tr1+1T∑t=2Trt|x0=x.Furthermore, we letRx⟶x′a to denote r1 and(11)VT−1πx′=Eπ1T−1∑t=1T−1rt|x0=x′.Through the law of total probability expansion, the operationEπ⋅ is substituted by the following expression:(12)∑a∈Aπx,a∑x′∈Xϵx⟶x′a.Then, we can get the function equality equation (9). S^ is an indicator function format.(13)1S^t=1,forS^⊂Θ∗,0,forS^⊂Θ∗.For prediction processX, we assume that discrete time sequence is equidistance Tj−Ti=const. According to the Bellman equation and formula of total probability, we have the recurrence accumulated feedback. Also, this predicted reward will be used to determine optimal transition probability on space N† in the next section.(14)QTπx,a=EMt+∑x′∈Xϵx⟶x′a1TRx⟶x′a+T−1TVT−1πx′.


## 4.3. Bootstrap Re-Sampling Controller for Mental Rehearsal

Bootstrapping was introduced as a flexible method to estimate the sampling distribution of an independent observation function [40]. It takes distribution Fn from sample data to substitute the global whole data, R∗x∗,Fn=θ^Fn∗−θ^Fn, and is useful for estimating of uncertainty in subspace identification. Figure 4 shows a segment that describes bootstrap re-sampling training to search the optimal transition probability where the bootstrap re-sampling controller KΛ is used to determine the transition possibility.Figure 4
Bootstrap re-sampling controllerKΛ for transition probability through different mental rehearsal frequencies. The set of curves denotes the machine state composed of different mental rehearsal frequencies. Correspondingly, the left part is the human decision between future estimation step length Q.We haveN†≜b1∪b2∪⋯∪bk where the subscript k stands for the different sample index. To explore the theoretically infinite solution decision space N∗, we assume that its probability distribution function is in accordance with fx,θ. For each N†, it stands for a brevity decision sequence according to Q-learning estimation introduced in the next section. We use the limited re-sampling characteristic f^ to represent the global big sampling range.(15)f<st,at>,b=f^<st,at>∗,b∗.Herest,at is the state from observed aspect and decision in space N; therein, st≜diagstl=1,stl=2. The subscript t is the discrete step index in decision sequence. The superscript l stands for different observed states, such as human state and machine state.We assume that transition probability discrete distributionPr dominates the pair st,at transferring in decision sequence. To calculate Prt⋅|st−1,at−1,…,at−S≜p^t1,pt2,…,ptrT, we first set initial probability Pr0=p01,p02,…,p0rT. The capital subscript S stands for step length of history consideration decision. The superscript r is the account of decision category in that different kinds of decisions are independently identically distributed. Next, each probability element is calculated by(16)ptj=gPr0Prts=Pr0∑1ptj∑ptj.Here1pji is an indicator function.(17)1ptj=1,ifj∈1,r,0,ifj∈1,r.Furthermore, the searched optimal transition probability groupP will be determined for searching contraction Ω. In bootstrapped sample k for future estimation, reward function r, which is assumed monotonically increasing as performance improves, is used to compare the accumulated reward.(18)ℬkQ<st,at>∗=maxat−S,…,atrst−S,at−S∗+⋯+γt−1rst−1,at−1∗+γtQst,at∗k,where γ is damping factor and Q is a Q-learning estimation from time i. After given the bootstrap sample estimation value, we derive the transition probability distribution and get the decision at+1 from the optimal decision trail b∗ for the next decision step. Algorithm 1 shows the comprehensive block of searching contraction Ω.(19)PQkst=b^∗πi=1|b^∗=argmaxb∗ℬkQst,at∗.Algorithm1: Searching contraction in space N†.
(1)
Initializeα,ϵ,v,δt, and t∈T,a∈A,s∈S(2)
InitializeT,S,A,S^,Sstep,Qstep(3)
repeat(4)
Calculate history rewardEMt(5)
Initialize the boots frequencyk(6)
repeat(7)
Calculate predicted rewardQTπx,a(8)
untilQstep and b=boots(9)
Calculatetj(10)
CalculateℬkQst,at∗(11)
until Temporal fragment decision completed at t(12)
CalculatePQkst(13)
Obtain the possible decisionπ∗


## 4.4. Decision Space Boundary for Admissible Decision

The decision space boundary blockΔ in Figure 2 limits the accessible decision action set and relies on the airplane flight dynamic state variables. Based on the observed state stl=2, admissible decision action at is the subset of action control scope at=t1,…,tl′T. Meanwhile, allowable state space S=s^t is a hypercube field minV1,maxV1×⋯×minVr′,maxVr′, which will inversely limit the decision action generated from the searching contraction method. This self-triggered policy caused by interaction object contributes to jumpy updating state and executing action by relying on the latest sampled state information [41]. In the sequel, we consider the flight longitudinal dynamic system V model as follows:(20)V˙1=1mTcosV4−DV4,V3,δe−GsinV2V˙2=1mvTsinV4+LV4,V3−GcosV2V˙3=MV4,V3,δeIyV˙4=V3−V˙2.V˙1,V˙2,V˙3,V˙4 are the various rates of airspeed, vertical speed, attack angle, and pitch angular velocity. M⋅,Iy is the rotational inertia, T is the power of airplane, and δe is the engine mounting angle. We assume that airspeed is approximately equal to tangential velocity. To get the boundary block, the Hamilton–Jacobi function is needed to be solved as follows:(21)−∂Sat,t∂t=min0,H∗at,∂Sat,t∂at.ThereinH∗=sup∇V∗TVa,stl=2, and the terminal boundary condition is set as S=minmaxV1−V1,V1−minV1,…,maxVr′−Vr′,Vr′−minVr′. Then, we can obtain the effective action u∗ which stays in the scope of decision space boundary.(22)u∗=argsupat∈atHat,S,stl=2.At each decision time updated, the bounded observable stateSt,t will be generated and compared with the boundary S. Only until the decision π∗∈u∗, π can be transmitted into airplane simulating block and then the interaction can be completed. Below, we provide Algorithm 2 for the whole decision inference on the receding horizon.Algorithm2: Targeted decision inference on receding horizon.
(1)
Initial target setΘ(2)
repeat(3)
Calculate the allowable state spaceS(4)
repeat(5)
searching contractionΩ block(6)
ifπ∈u∗=argsupt∈atHt,S,stl=2then(7)
output decisionπ(8)
end if(9)
untilπ(10)
until target set is completed


## 5. Experiment and Results

In this section, we present an experimental case and its results for airplane manipulating scenario. A typical task is to manually control the aircraft to descend altitude by the pilot. Some extra tasks are set on the designated altitude. Those particular subtasks require pilots to execute special operations. We apply our method into this experiment case. Results show that our method reflects the cognitive searching optimization from searching contraction index.


### 5.1. Experiment Setting

Table1 shows the flight altitude descending stages from 11000 ft to 2000 ft in experiment. Stages 2, 4, and 6 require the pilot to complete the specific tasks on specified altitude scope, and stages 1, 3, and 5 are the normal descending procedures involving basic flight joystick and throttle control [42]. Table 2 lists the multiresource channels involved in the experiment. We calculate the situation channels occupied with equipment to assess the workload taken by action. Table 3 lists the correlation between control rules and related equipment. Table 4 points out that delay in decision process separately represents state interval time delay and action time delay. We use the Poisson distribution and normal distribution to represent those two types of delay. The Poisson distribution is a non-exponentiation distribution satisfying the semi-Markov property in the context of continuous time domain. μ and σ stand for the mean value and variance value, respectively [44, 45].Table 1
Flight stages and tasks.
Flight stageFlight height (ft)Maneuvering apparatusSpecific task111000–7000Normal descending27000–6000SwitchDe-icing switch36000–3500Normal descending43500–3300CommunicationTower communication53300–3000Normal descending63000–2000FlapsAdjust flapsTable 2
Multiresource workload channels and weights [43].
ChannelVisualAuditoryBalancingHandFootAnalysisWeight0.20.20.10.20.10.2Table 3
Control rule, maneuvering apparatus, and channel occupation.
LocationControl rulesruManeuvering apparatusChannel occupationRule 1Pitch controlElevator, throttle1 0 1 1 0 1Rule 2Vertical-speed controlElevator1 0 1 1 0 1Rule 3Height controlElevator1 0 1 0 0 1Rule 4Configuration controlFlaps1 0 1 0 0 1Rule 5Dynamical controlThrottle1 0 1 0 0 1Rule 6Information obtainObserve, scan, microphone1 1 0 0 0 1Rule 7Button controlSwitch1 0 1 0 0 1Rule 8Idle controlKeep1 1 1 0 0 0Table 4
Transition interval and action time delay.
SeriesDistributionParameterState transition intervalPoisson distributionμ=3Action time delayNormal distributionμ=1.02,…,,σ2=0.025Action accuracy errorNormal distributionμ=1,σ2=0.05Action transition errorNormal distributionμ=1,σ2=0.05In experiments, we use the control rule to substitute the action in traditional decision action set. Each control rule corresponds to specified control equipment, which is chosen depending on the interactive processMt^. The number of actions can be one or more depending on observed states. The parameter mean in normal distribution of action time delay relates to the different equipment. Here we adopt parameters from the NASA timeline analysis report [46].


### 5.2. Simulation Results

In order to assess the pilot’s cognitive decision ability, flight performance, human accumulated workload, and the number of manipulations are three indexes measuring our multistep decision method. Trends of dynamical dimension inN† show the searching contraction results from infinity to limited. We use inner batch frequency to simulate the mental rehearsal in which the pilot chooses the suitable action from anticipating. Here inner batch frequency refers to rehearse times. By varying SMDP-step (semi-Markov decision property) and Q-step tuple, we simulate the time scale of history state influence and future prediction impact. Table 5 lists the main experiment results, and it shows the detailed contrast data in different combinations of parameters. Figure 5 shows flight height performance. Figure 6 depicts the workload accumulated speed increased with working time under different parameter configurations. Figure 7 shows the value of searching contraction ratio.Table 5
Different decision method parameters in simulation.
Batch frequencyMethodSMDP-stepQ-stepTime (seconds)Accumulated workloadDecision stepsBatch = 1MDP (S1Q1)11256.48223.9969Q-learning (S1Q3)13238.14205.8165SMDP (S3Q1)31246.45182.8765S3Q434211.52191.9357S4Q343264.69243.8273S8Q383213.46202.1261S3Q838218.93208.3258Batch = 5MDP (S1Q1)11222.58230.6649Q-learning (S1Q3)13222.11224.8052SMDP (S3Q1)31218.79181.3443S3Q434243.79265.4359S4Q343242.76215.0752S8Q383218.75230.7846S3Q838229.04230.6251Batch = 10MDP (S1Q1)11216.87225.4647Q-learning (S1Q3)13257.72218.6751SMDP (S3Q1)31234.12244.9052S3Q434256.71258.6763S4Q343238.40215.0350S8Q383230.78220.3553S3Q838268.97264.8664Batch = 20MDP (S1Q1)11264.65247.4354Q-learning (S1Q3)13287.11289.0963SMDP (S3Q1)31262.07236.3753S3Q434266.08231.4153S4Q343258.91239.2355S8Q383253.59228.4555S3Q838236.54228.8957Figure 5
Airplane descending performance. The vertical axis represents the corresponding airplane height/ft, and the horizontal axis represents the horizontal flight distance/ft. (a) Batch = 1. (b) Batch = 5. (c) Batch = 10. (d) Batch = 20.
(a)(b)(c)(d)Figure 6
Value of continuously accumulated cost increased with working time. The horizontal axis represents the variable working time, and the vertical axis represents the reward value. (a) Batch = 1. (b) Batch = 5. (c) Batch = 10. (d) Batch = 20.
(a)(b)(c)(d)Figure 7
Dimension variation percentage for searching contraction inHC2I process. The horizontal axis represents the serial number of decision steps, and the vertical axis represents the percentage of dimension contraction. In order to facilitate the comparison of values, we intercept the decision step number 50 as the maximum number of steps in the figure. (a) Batch = 1. (b) Batch = 5. (c) Batch = 10. (d) Batch = 20.
(a)(b)(c)(d)In Table5, time of flight task, human accumulated workload, and steps of manipulation are three indexes to build the overall evaluation assessing the decision methods. The accumulated workload caused increases slightly when the parameter inner batch frequency is boosted. This result is consistent with the fact that human mental workload increased with cognitive time pressure [47]. The bigger batch frequency is, the more the time pressure is. It is worth noting that our experiment setting batch = 20 is an extreme situation, exceeding the ordinary [48]. Regarding the human cognitive, their capability enlarges as batch frequency increases. The compound step tuple is another critical parameter. By setting history consideration steps (SMDP-step) and future estimation steps (Q-step), we compose the different multistep decision methods. For example, when the SMDP-step and Q-step are equal to 1, the method is essentially a Markov decision process (MDP).Figure5 intuitively reflects airplane flight height effect. On the whole, descending trajectories show the optimal stationary distribution at batch = 5, where curve differences are less. The differences between different trajectories, decided by steps tuple, are significantly increased. This result illustrates that searching contraction is relevant to human mind rehearsing action. The bigger the rehearsing frequency, the more the difference caused by different multistep decision methods. From the view of trajectory smoothness, the descending trend is similar to flight stage 1. But the accumulated effect caused by different multistep decision methods starts to appear from stage 2. In Figure 5(a), fluctuating range of S8Q3 is flatter compared with others. In Figure 5(b), fluctuating range of S3Q8 is flatter compared with others. In Figure 5(c), fluctuating range of MDP is flatter compared with others. In Figure 5(d), fluctuating range of S8Q3 is flatter compared with others. Corresponding to the rough scope of higher local value in each subfigure, flight stages 2, 4, and 6 which cover more control tasks show the hysteresis effect in descending trend curves. Also, it can be found that all multistep decision methods in experiments can converge airplane state to the target position.Figure6 shows a composite reward calculated by airplane flight performance and human cognitive workload performance. Reward value is more uniformly distributed when batch frequency is bigger. When human makes decisions after the repeated estimations, reward caused by manipulation tends to be similar. But higher repeated estimation times bring higher reward value, meaning that excessive anticipation leads to excessive cognitive workload. The reward accumulated speed increases with the batch frequency based on the slope of curves. Aside from Figure 6(d), the green line (S8Q3) and blue line (S4Q3) state the superior result from the horizontal axis (time, less is better) and vertical axis (reward value, less is better). The gray line (SMDP) at batch = 5 takes the best effect, which means that the multiple estimations also take effects on future estimation. The deepskyblue line (MDP) at batch = 10 takes the best effect, while the yellow line (S3Q4), green line (S8Q3), and the blue line (S4Q3) present proximate effect.Figure7 shows dimension variation percentage of searching contraction ratio. We calculate the ratio μ by the accumulated searching result of history decision space and predetermined searching scope at the parameter tuple (Sstep, Qstep, batch or mental rehearsal frequency). The value of μ is smaller, and the searching contraction ratio is higher. Equation (23) calculates μ, where ru refers to the number of rules and 1π is an indicator function.(23)μ=∑1πboots⋅Sstepru⋅Qstepru1/γ×100%.In this way, the initial infinity searching dimension is related to the history step influence, future step estimation, and bootstrapping frequency. The probability determined by the mental rehearsal and multistep transition, which is influenced by the dimension of decision, reflects the contraction effect of the search dimension. As shown in the results, cognitive searching optimization process shows a downward trend overall. When the batch frequency increases, the stability of the searching dimension contraction ratio gradually improves. Also, the ratio is distinguished according to different types of combined decision steps. For example, when future estimation stepQstep equals 1, such as MDP (S1Q1) and SMDP (S3Q1), the longitudinal change amplitude of dimension percentage changes intuitively from big to small within the batch frequency but independent from other methods.


### 5.3. Cost and Performance Analysis

Figure8 shows the changes in various indicators and three primary parameters (two types of decision steps, rehearsal frequency) in our decision-making method. Under different batch parameters, Figure 8(a) shows the variation of statistical standard deviation for each cluster’s flight descent curve as the mission progresses. When the batch number is larger, the standard deviation firstly climbs up and then declines. Figure 8(b) shows that the average accumulated workload increases with the batch frequency. It proves that the more the decisions people anticipate, the greater the workload caused by mental rehearsal. An inflection point exists in time index when batch = 5 shows that suitable mental rehearsal can decrease work time. Figure 8(c) shows that manipulation step drops down with batch increases, but its decline trend slows down. When the batch frequency is less (e.g., batch = 1), the manipulation step is more larger. The negative correlation can illustrate that non-optimal strategy step leads to generate more decision steps to revise the former. Figures 8(d) and 8(e) show the influence of different types of decision step. There is a peak in the histogram group under all of the different indicators showing that the appropriate number of decision steps can reduce the corresponding indicator’s performance. Instead, the inappropriate number of decision step will increase the index value. Figure 9 analyzes the reward value presented in Figure 6. The combined reward index defined by flight performance and workload shows that batch 5 is the peak of these data, which shows that although the workload will increase, the overall value of batches 10 and 20 will decrease under the influence of the mission. Therefore, considering the three types of index data and reward values, the batch frequency between 1 and 5 is more appropriate.Figure 8
Analysis of flight altitude decline performance and reward index value. To compare the effects of different batches (mental rehearsal parameters), we calculate the overall mean values of flight height performance and reward indexes (time, accumulated workload, and decision manipulation steps) under different multistep transfer parameters. To compare the effect of multistep transition effect (the number of steps in historical consideration and the number of steps in future estimation) on the reward index, we calculate the overall mean value under different batch parameters. (a) Standard deviation of airplane performance for different batch frequencies. (b) Average cost index for different batch frequencies. (c) Average steps of manipulation for different batch frequencies. (d) Average steps of manipulation for different batch frequencies. (e) Average steps of manipulation for different batch frequencies.
(a)(b)(c)(d)(e)Figure 9
Analysis value of continuously accumulated reward. According to different batch parameters, we use the evolution curve of standard deviation to describe the overall fluctuation degree of each cluster curve. At the same time, a relatively stable standard deviation was selected between working time [150, 200], and a curve was fitted to reflect the change of the fluctuation degree of reward with the increase of the batch. (a) Standard deviation. (b) Fitting function.
(a)(b)


### 5.4. Transition Probability and Searching Contraction Ratio Analysis

Figure10 shows transition probability distribution varying in different types of methods. Transition probability, which is calculated from inner simulated estimation, reflects the dynamical selection from rules. The changing trend is more consistent with normal human decision-making behavior because inferring transition probability lies at the core of human sequence knowledge [49]. It demonstrates that cognitive interaction behavior constantly attempts to infer the time-varying matrix of transition probabilities when it receives the outer observed machine states. Therefore, dynamical transition probabilities are ensured by the bootstrap re-sampling controller in searching contraction method.Figure 10
Transition probability evolving process for different control rules in (a) S4Q3, batch = 5, and (b) S4Q3, batch = 10. The horizontal axis represents the serial number of decision step, and the vertical axis represents the value of transition probability.
(a)(b)Additionally, Figure10 shows transition probability with regard to control rules in Table 3 and parameters in Table 5. Transition probability is dynamically changeable during the flight manipulation stage. The transition probability value of pitch control rule (Rule 1) is higher than that of other rules on average. The transition probability of vertical-speed control rule (Rule 2) is less focused than the height control rule (Rule 3). Configuration control rule (Rule 4) is not used until the flight altitude attains the allowable range. At different flight stages, rule transition probability is verified by the specific tasks and its corresponding control rules. On the other hand, transition probability is prominently influenced by step tuple. The overall fluctuation of transition probabilities varies strengthening accompanied by the increase of estimated part in the decision step tuple. Fluctuation of probability variation in MDP and Q-learning methods is less than others.On the other hand, searching contraction change happens in the continuously multistep decisionHC2I process. It denotes the damping of decision admissible exploitation dimension. The solution space in which human chooses strategy rules contracts within the receding horizon controller KΛ, after completing the inner inference by K∇. Research about the brain prefontal also demonstrated this point that the existing high-level control area reprocesses the upcoming decision before it finally enters awareness [2]. Therefore, under the influence of interactive environment, the degree of searching contraction is a critical factor in cognitive decision.We compare the searching contraction ratio from different aspects in Figure11. First, stability of the dynamical dimension goes down as the batch frequency gradually goes up. Figure 11(a) compares the search contraction ratio in terms of the composition of multistep transfer steps and the frequency of mental previews. By calculating the average change of the overall contraction ratio under the specified batch parameters, the figure shows that under higher batch frequency situation, the contraction ratio fluctuation decreases when the contraction percentage increases. The intersection between the batch parameters 1–5 will provide better overall performance. The S4Q3 example in Figure 11(b) emphasizes that as the batch value increases, the amplitude of the contraction percentage fluctuation will decrease, but when the number of batches is greater than 5, the standard deviation of the fluctuation amplitude tends to be flat. It demonstrates that the increase in batch frequency at this time has little effect on the increase of the fluctuation amplitude. Figures 11(c) and 11(d) compare the number’s increase of history consideration steps Sstep and future estimated step Qstep groups. The result in this figure states that during the HC2I process, the contraction ratio caused by the history consideration steps and future estimated step in the experiment decreases and increases, respectively, with the number of steps increasing. But, they both indicate that too many steps will suppress the improvement of contraction ratio (e.g., the number of steps equals 8). And with the increase in the batch frequency, increment in the contraction ratio will gradually slow down. This result shows the fact that searching contraction is difficult for human brain under the situation of the excessive rehearsal numbers, such as batch = 20. The above data analysis explains the feasibility of searching contraction decision method proposed in this paper.Figure 11
Sample comparison from four aspects. (a) Mean value comparison in different batch frequencies. (b) S4Q3 sample comparison in different batch frequencies. (c) History transition steps influence on searching contraction ratio. (d) Future estimation steps influence on searching contraction ratio. Fitted variation curves of standard deviation and mean value are also plotted. In (c) and (d), two variation trends of search contraction are plotted based on the different parameter values of the decision step number.
(a)(b)(c)(d)


## 5.1. Experiment Setting

Table1 shows the flight altitude descending stages from 11000 ft to 2000 ft in experiment. Stages 2, 4, and 6 require the pilot to complete the specific tasks on specified altitude scope, and stages 1, 3, and 5 are the normal descending procedures involving basic flight joystick and throttle control [42]. Table 2 lists the multiresource channels involved in the experiment. We calculate the situation channels occupied with equipment to assess the workload taken by action. Table 3 lists the correlation between control rules and related equipment. Table 4 points out that delay in decision process separately represents state interval time delay and action time delay. We use the Poisson distribution and normal distribution to represent those two types of delay. The Poisson distribution is a non-exponentiation distribution satisfying the semi-Markov property in the context of continuous time domain. μ and σ stand for the mean value and variance value, respectively [44, 45].Table 1
Flight stages and tasks.
Flight stageFlight height (ft)Maneuvering apparatusSpecific task111000–7000Normal descending27000–6000SwitchDe-icing switch36000–3500Normal descending43500–3300CommunicationTower communication53300–3000Normal descending63000–2000FlapsAdjust flapsTable 2
Multiresource workload channels and weights [43].
ChannelVisualAuditoryBalancingHandFootAnalysisWeight0.20.20.10.20.10.2Table 3
Control rule, maneuvering apparatus, and channel occupation.
LocationControl rulesruManeuvering apparatusChannel occupationRule 1Pitch controlElevator, throttle1 0 1 1 0 1Rule 2Vertical-speed controlElevator1 0 1 1 0 1Rule 3Height controlElevator1 0 1 0 0 1Rule 4Configuration controlFlaps1 0 1 0 0 1Rule 5Dynamical controlThrottle1 0 1 0 0 1Rule 6Information obtainObserve, scan, microphone1 1 0 0 0 1Rule 7Button controlSwitch1 0 1 0 0 1Rule 8Idle controlKeep1 1 1 0 0 0Table 4
Transition interval and action time delay.
SeriesDistributionParameterState transition intervalPoisson distributionμ=3Action time delayNormal distributionμ=1.02,…,,σ2=0.025Action accuracy errorNormal distributionμ=1,σ2=0.05Action transition errorNormal distributionμ=1,σ2=0.05In experiments, we use the control rule to substitute the action in traditional decision action set. Each control rule corresponds to specified control equipment, which is chosen depending on the interactive processMt^. The number of actions can be one or more depending on observed states. The parameter mean in normal distribution of action time delay relates to the different equipment. Here we adopt parameters from the NASA timeline analysis report [46].


## 5.2. Simulation Results

In order to assess the pilot’s cognitive decision ability, flight performance, human accumulated workload, and the number of manipulations are three indexes measuring our multistep decision method. Trends of dynamical dimension inN† show the searching contraction results from infinity to limited. We use inner batch frequency to simulate the mental rehearsal in which the pilot chooses the suitable action from anticipating. Here inner batch frequency refers to rehearse times. By varying SMDP-step (semi-Markov decision property) and Q-step tuple, we simulate the time scale of history state influence and future prediction impact. Table 5 lists the main experiment results, and it shows the detailed contrast data in different combinations of parameters. Figure 5 shows flight height performance. Figure 6 depicts the workload accumulated speed increased with working time under different parameter configurations. Figure 7 shows the value of searching contraction ratio.Table 5
Different decision method parameters in simulation.
Batch frequencyMethodSMDP-stepQ-stepTime (seconds)Accumulated workloadDecision stepsBatch = 1MDP (S1Q1)11256.48223.9969Q-learning (S1Q3)13238.14205.8165SMDP (S3Q1)31246.45182.8765S3Q434211.52191.9357S4Q343264.69243.8273S8Q383213.46202.1261S3Q838218.93208.3258Batch = 5MDP (S1Q1)11222.58230.6649Q-learning (S1Q3)13222.11224.8052SMDP (S3Q1)31218.79181.3443S3Q434243.79265.4359S4Q343242.76215.0752S8Q383218.75230.7846S3Q838229.04230.6251Batch = 10MDP (S1Q1)11216.87225.4647Q-learning (S1Q3)13257.72218.6751SMDP (S3Q1)31234.12244.9052S3Q434256.71258.6763S4Q343238.40215.0350S8Q383230.78220.3553S3Q838268.97264.8664Batch = 20MDP (S1Q1)11264.65247.4354Q-learning (S1Q3)13287.11289.0963SMDP (S3Q1)31262.07236.3753S3Q434266.08231.4153S4Q343258.91239.2355S8Q383253.59228.4555S3Q838236.54228.8957Figure 5
Airplane descending performance. The vertical axis represents the corresponding airplane height/ft, and the horizontal axis represents the horizontal flight distance/ft. (a) Batch = 1. (b) Batch = 5. (c) Batch = 10. (d) Batch = 20.
(a)(b)(c)(d)Figure 6
Value of continuously accumulated cost increased with working time. The horizontal axis represents the variable working time, and the vertical axis represents the reward value. (a) Batch = 1. (b) Batch = 5. (c) Batch = 10. (d) Batch = 20.
(a)(b)(c)(d)Figure 7
Dimension variation percentage for searching contraction inHC2I process. The horizontal axis represents the serial number of decision steps, and the vertical axis represents the percentage of dimension contraction. In order to facilitate the comparison of values, we intercept the decision step number 50 as the maximum number of steps in the figure. (a) Batch = 1. (b) Batch = 5. (c) Batch = 10. (d) Batch = 20.
(a)(b)(c)(d)In Table5, time of flight task, human accumulated workload, and steps of manipulation are three indexes to build the overall evaluation assessing the decision methods. The accumulated workload caused increases slightly when the parameter inner batch frequency is boosted. This result is consistent with the fact that human mental workload increased with cognitive time pressure [47]. The bigger batch frequency is, the more the time pressure is. It is worth noting that our experiment setting batch = 20 is an extreme situation, exceeding the ordinary [48]. Regarding the human cognitive, their capability enlarges as batch frequency increases. The compound step tuple is another critical parameter. By setting history consideration steps (SMDP-step) and future estimation steps (Q-step), we compose the different multistep decision methods. For example, when the SMDP-step and Q-step are equal to 1, the method is essentially a Markov decision process (MDP).Figure5 intuitively reflects airplane flight height effect. On the whole, descending trajectories show the optimal stationary distribution at batch = 5, where curve differences are less. The differences between different trajectories, decided by steps tuple, are significantly increased. This result illustrates that searching contraction is relevant to human mind rehearsing action. The bigger the rehearsing frequency, the more the difference caused by different multistep decision methods. From the view of trajectory smoothness, the descending trend is similar to flight stage 1. But the accumulated effect caused by different multistep decision methods starts to appear from stage 2. In Figure 5(a), fluctuating range of S8Q3 is flatter compared with others. In Figure 5(b), fluctuating range of S3Q8 is flatter compared with others. In Figure 5(c), fluctuating range of MDP is flatter compared with others. In Figure 5(d), fluctuating range of S8Q3 is flatter compared with others. Corresponding to the rough scope of higher local value in each subfigure, flight stages 2, 4, and 6 which cover more control tasks show the hysteresis effect in descending trend curves. Also, it can be found that all multistep decision methods in experiments can converge airplane state to the target position.Figure6 shows a composite reward calculated by airplane flight performance and human cognitive workload performance. Reward value is more uniformly distributed when batch frequency is bigger. When human makes decisions after the repeated estimations, reward caused by manipulation tends to be similar. But higher repeated estimation times bring higher reward value, meaning that excessive anticipation leads to excessive cognitive workload. The reward accumulated speed increases with the batch frequency based on the slope of curves. Aside from Figure 6(d), the green line (S8Q3) and blue line (S4Q3) state the superior result from the horizontal axis (time, less is better) and vertical axis (reward value, less is better). The gray line (SMDP) at batch = 5 takes the best effect, which means that the multiple estimations also take effects on future estimation. The deepskyblue line (MDP) at batch = 10 takes the best effect, while the yellow line (S3Q4), green line (S8Q3), and the blue line (S4Q3) present proximate effect.Figure7 shows dimension variation percentage of searching contraction ratio. We calculate the ratio μ by the accumulated searching result of history decision space and predetermined searching scope at the parameter tuple (Sstep, Qstep, batch or mental rehearsal frequency). The value of μ is smaller, and the searching contraction ratio is higher. Equation (23) calculates μ, where ru refers to the number of rules and 1π is an indicator function.(23)μ=∑1πboots⋅Sstepru⋅Qstepru1/γ×100%.In this way, the initial infinity searching dimension is related to the history step influence, future step estimation, and bootstrapping frequency. The probability determined by the mental rehearsal and multistep transition, which is influenced by the dimension of decision, reflects the contraction effect of the search dimension. As shown in the results, cognitive searching optimization process shows a downward trend overall. When the batch frequency increases, the stability of the searching dimension contraction ratio gradually improves. Also, the ratio is distinguished according to different types of combined decision steps. For example, when future estimation stepQstep equals 1, such as MDP (S1Q1) and SMDP (S3Q1), the longitudinal change amplitude of dimension percentage changes intuitively from big to small within the batch frequency but independent from other methods.


## 5.3. Cost and Performance Analysis

Figure8 shows the changes in various indicators and three primary parameters (two types of decision steps, rehearsal frequency) in our decision-making method. Under different batch parameters, Figure 8(a) shows the variation of statistical standard deviation for each cluster’s flight descent curve as the mission progresses. When the batch number is larger, the standard deviation firstly climbs up and then declines. Figure 8(b) shows that the average accumulated workload increases with the batch frequency. It proves that the more the decisions people anticipate, the greater the workload caused by mental rehearsal. An inflection point exists in time index when batch = 5 shows that suitable mental rehearsal can decrease work time. Figure 8(c) shows that manipulation step drops down with batch increases, but its decline trend slows down. When the batch frequency is less (e.g., batch = 1), the manipulation step is more larger. The negative correlation can illustrate that non-optimal strategy step leads to generate more decision steps to revise the former. Figures 8(d) and 8(e) show the influence of different types of decision step. There is a peak in the histogram group under all of the different indicators showing that the appropriate number of decision steps can reduce the corresponding indicator’s performance. Instead, the inappropriate number of decision step will increase the index value. Figure 9 analyzes the reward value presented in Figure 6. The combined reward index defined by flight performance and workload shows that batch 5 is the peak of these data, which shows that although the workload will increase, the overall value of batches 10 and 20 will decrease under the influence of the mission. Therefore, considering the three types of index data and reward values, the batch frequency between 1 and 5 is more appropriate.Figure 8
Analysis of flight altitude decline performance and reward index value. To compare the effects of different batches (mental rehearsal parameters), we calculate the overall mean values of flight height performance and reward indexes (time, accumulated workload, and decision manipulation steps) under different multistep transfer parameters. To compare the effect of multistep transition effect (the number of steps in historical consideration and the number of steps in future estimation) on the reward index, we calculate the overall mean value under different batch parameters. (a) Standard deviation of airplane performance for different batch frequencies. (b) Average cost index for different batch frequencies. (c) Average steps of manipulation for different batch frequencies. (d) Average steps of manipulation for different batch frequencies. (e) Average steps of manipulation for different batch frequencies.
(a)(b)(c)(d)(e)Figure 9
Analysis value of continuously accumulated reward. According to different batch parameters, we use the evolution curve of standard deviation to describe the overall fluctuation degree of each cluster curve. At the same time, a relatively stable standard deviation was selected between working time [150, 200], and a curve was fitted to reflect the change of the fluctuation degree of reward with the increase of the batch. (a) Standard deviation. (b) Fitting function.
(a)(b)


## 5.4. Transition Probability and Searching Contraction Ratio Analysis

Figure10 shows transition probability distribution varying in different types of methods. Transition probability, which is calculated from inner simulated estimation, reflects the dynamical selection from rules. The changing trend is more consistent with normal human decision-making behavior because inferring transition probability lies at the core of human sequence knowledge [49]. It demonstrates that cognitive interaction behavior constantly attempts to infer the time-varying matrix of transition probabilities when it receives the outer observed machine states. Therefore, dynamical transition probabilities are ensured by the bootstrap re-sampling controller in searching contraction method.Figure 10
Transition probability evolving process for different control rules in (a) S4Q3, batch = 5, and (b) S4Q3, batch = 10. The horizontal axis represents the serial number of decision step, and the vertical axis represents the value of transition probability.
(a)(b)Additionally, Figure10 shows transition probability with regard to control rules in Table 3 and parameters in Table 5. Transition probability is dynamically changeable during the flight manipulation stage. The transition probability value of pitch control rule (Rule 1) is higher than that of other rules on average. The transition probability of vertical-speed control rule (Rule 2) is less focused than the height control rule (Rule 3). Configuration control rule (Rule 4) is not used until the flight altitude attains the allowable range. At different flight stages, rule transition probability is verified by the specific tasks and its corresponding control rules. On the other hand, transition probability is prominently influenced by step tuple. The overall fluctuation of transition probabilities varies strengthening accompanied by the increase of estimated part in the decision step tuple. Fluctuation of probability variation in MDP and Q-learning methods is less than others.On the other hand, searching contraction change happens in the continuously multistep decisionHC2I process. It denotes the damping of decision admissible exploitation dimension. The solution space in which human chooses strategy rules contracts within the receding horizon controller KΛ, after completing the inner inference by K∇. Research about the brain prefontal also demonstrated this point that the existing high-level control area reprocesses the upcoming decision before it finally enters awareness [2]. Therefore, under the influence of interactive environment, the degree of searching contraction is a critical factor in cognitive decision.We compare the searching contraction ratio from different aspects in Figure11. First, stability of the dynamical dimension goes down as the batch frequency gradually goes up. Figure 11(a) compares the search contraction ratio in terms of the composition of multistep transfer steps and the frequency of mental previews. By calculating the average change of the overall contraction ratio under the specified batch parameters, the figure shows that under higher batch frequency situation, the contraction ratio fluctuation decreases when the contraction percentage increases. The intersection between the batch parameters 1–5 will provide better overall performance. The S4Q3 example in Figure 11(b) emphasizes that as the batch value increases, the amplitude of the contraction percentage fluctuation will decrease, but when the number of batches is greater than 5, the standard deviation of the fluctuation amplitude tends to be flat. It demonstrates that the increase in batch frequency at this time has little effect on the increase of the fluctuation amplitude. Figures 11(c) and 11(d) compare the number’s increase of history consideration steps Sstep and future estimated step Qstep groups. The result in this figure states that during the HC2I process, the contraction ratio caused by the history consideration steps and future estimated step in the experiment decreases and increases, respectively, with the number of steps increasing. But, they both indicate that too many steps will suppress the improvement of contraction ratio (e.g., the number of steps equals 8). And with the increase in the batch frequency, increment in the contraction ratio will gradually slow down. This result shows the fact that searching contraction is difficult for human brain under the situation of the excessive rehearsal numbers, such as batch = 20. The above data analysis explains the feasibility of searching contraction decision method proposed in this paper.Figure 11
Sample comparison from four aspects. (a) Mean value comparison in different batch frequencies. (b) S4Q3 sample comparison in different batch frequencies. (c) History transition steps influence on searching contraction ratio. (d) Future estimation steps influence on searching contraction ratio. Fitted variation curves of standard deviation and mean value are also plotted. In (c) and (d), two variation trends of search contraction are plotted based on the different parameter values of the decision step number.
(a)(b)(c)(d)


## 6. Discussion

Our experimental results illustrate that cognitive search contraction is a subconscious phenomenon that commonly occurs in the decision-making process. Therein, the associated multistep transition and mental rehearsal are two crucial factors. The multistep transition factor, which combines the influence of cumulative fragments and the jumpy transition interval, is the basis of interactive decision making. Under the credible admissible decision-making boundary, the mental rehearsal that covers the parallel execution of the decision fragments will screen the decision again until obtaining the optimal decision-making behavior.The analysis of experimental results shows that different combinations of multistep transition step values and different rehearsal frequencies will affect the search contraction ratio and decision-making rewards. On the one hand, the number of historically considering decision steps will reduce the workload of brain (by increasing the searching contraction ratio). In contrast, the increase in the number of future estimated steps will increase the workload of brain (by decreasing the contraction ratio of searching). At the same time, too much rehearsal frequency makes the contraction efficiency of decision search in decision solution space decrease. This is consistent with the research result that cognitive channel is limited when human completes multitask [50].Meanwhile, the increase in the rehearsal frequency will reduce the fluctuation of search contraction index. It shows that the if a person can get more information or experience before decision, the bias of result will lower. On the other hand, the decision reward of all different multistep transition step types presents a trend of increasing first and then decreasing. This reflects that the number of decision-making steps are not positively correlated with the decision reward. Moderate composition of multistep transition step can bring the optimal decision reward (interval time, workload, and number of decision steps). It is the same as the rehearsal frequency.Here we define the time complexity as the sum of the worst-case running time for each operation (e.g., multiplication, division, and addition) required to process an output. The growth rate is then obtained by making the parameters of the worst-case complexity tend to infinity. Memory complexity is estimated as the number of 32 bit registers needed during the learning process to store variables. And also, only the required worst-case memory space is considered during the process phase. From the following complexity formulations, we can find that the time complexity grows quadratically withN and linearly with T, and its memory complexity of the algorithm grows linearly with T and N. When T is very large, the memory complexity will not exceed the resources available for the training process, avoiding overflow from internal system memory to disk storage.(i)
Time complexity: for a decision sequence with lengthT and ergodic Markov state N, the time complexity is composed of decision iteration process. We can get the time complexity as(24)OT=T∗Sstep+boots∗N∗logmN∗Qstep≈T∗N∗logmN.(iii)
Memory complexity: the memory complexity relates to the decision space. In this paper, the decision space is composed of decision horizon space and bootstrap sampling space.(25)OS=T∗boots∗NS+NA+NOQstep≈T∗NQstep.Research studies about decision tree considered the different algorithms to optimize the decision searching tree, such as the UCB1, UCT, and other non-greedy methods. However, the main index searching contraction ratio designed in our paper is not similar to those studies, where they verify their efficiency through the true/false ratio. On the other hand, the method in our paper used the Hamilton function to limit the admissible decision action while the other existing studies analyzed the decision state as a discrete classification problem. In Table5, when parameters Bboots, SSstep, and QQstep are different, our proposed method can be transformed into other existing methods, such as the standard Markov decision method (B1S1Q1), standard semi-Markov decision method (B1S3Q1), and standard Q-learning method (B1S1Q3). In Table 6, we add the comparison at the index searching contraction from its mean value and variation, and the index time is also listed. From the table, it can be shown that the stability and time efficiency of our proposed method are better than those of the previous studies.Table 6
Comparison of different methods.
RandMethodsIndex: searching contractionIndex: timeMean valueVariation1MDP0.6050.0250256.482SMDP0.5460.0091246.453Q-learning0.6450.0182238.144B5, S3Q40.7650.0015243.795B5, S3Q80.7940.0005229.046B10, S3Q10.5490.0031234.12


## 7. Conclusion

In this paper, we propose a semi-Markov jump decision method to optimize the decision path and a searching contraction index to indicate cognitive searching optimization. The main difference of our work is using jumpy decision interval and multiple path estimation as deterministic features. Under the specific interaction target in decision boundary, this modification leads to optimizing cognitive interaction decision from the time dimension and depth dimension. Semi-Markov jump transition decision outperforms the traditional Markov method by strengthening the correlation from the dimension of decision time interval. The mental rehearsal improves the searching depth of decision solution space. The decision boundary filters out the infeasible human decision by the estimated admissible action boundary. Furthermore, numerical simulation shows the characteristic of searching contraction, and our decision method can be applied to evaluate a class of multiple element types’ decision path. The reduction in searching contraction ratio proves that proper transition step length and mental rehearsal frequency can reduce and stabilize the searching space and reward of decision path in theHC2I process. The future work will address the decision switch relation happening in the semi-Markov cognitive decision. To investigate the human fatigue influence on control accuracy and stationary, we will research the jumpy switch control according to the limited human behaviour rule. And the arbitrary number of historical decision steps in the decision-making is also deserved to be explored.



---

*Source: 3343494-2021-10-06.xml*