# Risk-Sensitive Multiagent Decision-Theoretic Planning Based on MDP and  One-Switch Utility Functions

Authors: Wei Zeng; Hongtao Zhou; Mingshan You
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

In high stakes situations decision-makers are often risk-averse and decision-making processes often take place in group settings. This paper studies multiagent decision-theoretic planning under Markov decision processes (MDPs) framework with considering the change of agent’s risk attitude as his wealth level varies. Based on one-switch utility function that describes agent’s risk attitude change with his wealth level, we give the additive and multiplicative aggregation models of group utility and adopt maximizing expected group utility as planning objective. When the wealth level approaches infinity, the characteristics of optimal policy are analyzed for the additive and multiplicative aggregation model, respectively. Then a backward-induction method is proposed to divide the wealth level interval from negative infinity to initial wealth level into subintervals and determine the optimal policy in states and subintervals. The proposed method is illustrated by numerical examples and the influences of agent’s risk aversion parameters and weights on group decision-making are also analyzed.


---

## Body

## 1. Introduction

Decision-theoretic planning is to compute optimal policy that is formed by courses of action to maximize expected reward with considering actions that have uncertain outcomes [1]. In high stakes situations with the possibility of high wins and losses, such as emergency and crisis response, business and investment decision, military battle, and lottery, decision-makers are often risk-averse. In the risk-sensitive decisions, exponential utility function is one of the typical utility functions to model decision-maker’s risk aversion and maximizing expected utility is the most commonly used rule instead of maximizing expected reward. However, the risk attitude of a decision-maker modeled by exponential utility function is independent of his wealth level and does not change as his wealth level varies, while in reality personal risk attitude often changes with his wealth level [2–4]. Bell proposes a kind of utility function, named one-switch utility function, to model an agent who is always risk-averse but becomes risk neutral as his wealth increases [2]. Bell and Fishburn take further studies on the characteristics of a form of one-switch utility function that is a combination of linear utility function and exponential utility function [5]. Liu and Koenig give a form of one-switch utility function with considering agent’s risk aversion parameter γ
(
0
<
γ
<
1
); that is, u
(
w
)
=
w
-
D
γ
w, where w denotes the wealth level and D is a parameter to adjust tradeoff between risk neutrality and risk aversion. This form of one-switch utility function not only describes the change of agent’s risk attitude, but also presents the degree of agent’s risk aversion by the quantitative risk aversion parameter γ [6–8].For decision-theoretic planning problems, Markov decision processes (MDPs) framework is adopted broadly as an underlying model. Howard and Matheson in their seminal paper introduce risk-sensitive MDPs based on maximizing the expected exponential utility [9]. In the follow-up related studies structural properties of optimal solution and algorithms to compute optimal policy are investigated based on exponential utility function [10–12]. If an agent is risk-sensitive, it is necessary to consider the possible change of agent’s risk attitude when his wealth level varies and further influence on the decisions in the next stage. Liu and Koenig study Markov decision processes with considering agent’s risk-sensitive attitude modeled by one-switch utility function and propose an exact backward-induction algorithm to compute optimal policy [8].In reality decision-making processes often take place in group settings due to a single decision-maker’s limited decision-making ability. For the group decision-making problem, group utility is usually got by aggregating personal utilities and then group decisions are made based on the group utility. The aggregation methods include additive value model and multiplicative value rule. Other methods such as multiobjective linear programming [13], fuzzy sets method [14, 15], and interactive approach [16, 17] are used to aggregate individual decision information including attribute weights and attribute values into group decisions. Besides, some researches on group decision-making problem take time into consideration. Xu investigates multistage multiattribute group decision-making problems in which the weight information on a collection of attributes and the decision information on a finite set of alternatives with respect to the attributes are collected at different stages [18].This paper focuses on decision-theoretic planning problem in which sequential decisions are made by a group of risk-sensitive members. Considering agent’s risk-sensitive attitude and wealth level, this paper studies the risk-sensitive multiagent decision-theoretic planning problem based on one-switch utility function and MDP framework. Two group utility functions based, respectively, on additive value model and multiplicative value model of one-switch utility functions are given. Backward-induction algorithms for these two kinds of group utility functions to compute optimal policy of risk-sensitive group decision-making under MDP framework are proposed.The rest of this paper is organized as follows. One-switch utility function and risk-sensitive MDP model augmented with wealth level are introduced in Section2. In Section 3, additive and multiplicative aggregation model of one-switch utility functions are given. We analyze the characteristics of optimal policy when the wealth level approaches negative infinity for additive and multiplicative aggregation model, respectively, in Section 3. In Section 4, detailed backward-induction algorithms are proposed to solve the multiagent decision-theoretic planning problem based, respectively, on additive and multiplicative aggregation model. Numerical examples are used to illustrate the proposed method and analyze the influences of agent’s risk aversion parameters and weights on group decision-making in Section 5. Finally, a conclusion of this paper and suggested topics for future research are presented in Section 6.


## 2. Risk-Sensitive MDP Model Augmented with Wealth Level

### 2.1.  One-Switch Utility Function

One-switch utility function is a kind of utility function to describe the change of agent’s risk attitude as his wealth level varies. In detail, there exists a wealth levelw; when the agent’s wealth level is below w, the agent is risk-averse, but when his wealth level increases and becomes higher than w, the agent becomes risk neutral. For agent i, one-switch utility function given by Liu and Koenig is shown as follows [6–8]:
(1)
u
i
(
w
)
=
w
-
D
i
γ
i
w
=
u
l
+
D
·
u
i
,
e
(
w
)
,
where w is wealth level, γ
i is agent i’s risk aversion parameter, and 0
<
γ
i
<
1. D
i is a constant that provides an adjustable tradeoff between risk neutrality (linear term) and risk aversion (exponential term). u
l
=
w is a linear utility function. u
i
,
e
(
w
)
=
-
γ
i
w is agent i’s exponential utility function.


### 2.2. Risk-Sensitive MDP Model Augmented with Wealth Level

In the paper goal directed Markov decision problem (GDMDP) is adopted as underlying model of decision-theoretic planning problem [8]. GDMDP is a kind of MDP with a finite set of goal states. When an agent reaches a goal state, he stops acting and receives no more rewards thereafter. One-switch utility function is used to describe the agent’s risk-sensitive attitude and maximizing expected utility is adopted as planning objective instead of maximizing expected reward. As wealth level is included in the one-switch utility function, it is necessary to consider the wealth level as a component of the system state of GDMDP.Formally, a GDMDP consists of a finite set of statesS with wealth levels W, so the augmented state set of GDMDP is denoted by (
S
,
W
). Goal state set is (
G
,
W
), where G
⊆
W. Nongoal state set is (
S
′
,
W
), where S
′
=
S
∖
G.The agent’s action set isA
s. The agent chooses an action a
∈
A
s to execute in its current state s
∈
S.The agent’s execution of actiona in state s results in a finite reward r
(
s
,
a
,
s
′
) and a transition to successor state s
′
∈
S with probability P
(
s
′
∣
s
,
a
). In the paper only cost is considered and assumed reward is strictly negative, r
(
s
,
a
,
s
′
)
<
0.We also uses
t and a
t to denote the state and action at time step t  (
t
=
0,1
,
2
,
…
). r
t
=
r
(
s
t
,
a
t
,
s
t
+
1
) is used to denote the reward for executing action a
t. After the agent reaches a goal state, r
t
=
0.w
t
=
w
0
+
∑
i
=
0
i
=
t
-
1
r
i is the agent’s wealth level at time step t, where the initial wealth level is denoted by w
0.For the MDP model augmented with wealth level, the optimal policy maps every combination of a states
∈
S
′ and wealth level w to an action a
∈
A
s that an agent in state s with wealth level w should execute to maximize expected utility.For agenti’s exponential utility function u
i
,
e
(
w
) and all policies π, we define the value v
i
,
e
π
(
s
,
w
)
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
i
,
e
(
w
t
)
] as the expected exponential utility of agent i with initial state s and initial wealth level w that follows policy π.The optimal valuev
i
,
e
*
(
s
,
w
)
=
max
⁡
π
⁡
v
i
,
e
π
(
s
,
w
) is defined as the highest possible expected exponential utility of agent i with initial state s and initial wealth level w. Assume v
i
,
e
*
(
s
,
w
) is finite for all state s
∈
S and wealth levels w.An optimal policyπ
i
,
e
*
(
s
,
w
) is defined as a policy with v
i
,
e
π
i
,
e
*
(
s
,
w
)
=
v
i
,
e
*
(
s
,
w
) for all state s
∈
S and wealth levels w.Similarly, for linear utility functionu
l
(
w
), we use v
l
π
(
s
,
w
), v
l
*
(
s
,
w
), and π
l
*
(
s
,
w
) to denote expected utility, optimal value, and optimal policy, respectively.It is worth noting that differently from Liu and Koenig [8], the paper focuses on the decision-making in group setting. So the value function in MDP will be replaced by group utility function which is the aggregation of personal one-switch utilities and the planning objective is to maximize the expected group utility.


## 2.1.  One-Switch Utility Function

One-switch utility function is a kind of utility function to describe the change of agent’s risk attitude as his wealth level varies. In detail, there exists a wealth levelw; when the agent’s wealth level is below w, the agent is risk-averse, but when his wealth level increases and becomes higher than w, the agent becomes risk neutral. For agent i, one-switch utility function given by Liu and Koenig is shown as follows [6–8]:
(1)
u
i
(
w
)
=
w
-
D
i
γ
i
w
=
u
l
+
D
·
u
i
,
e
(
w
)
,
where w is wealth level, γ
i is agent i’s risk aversion parameter, and 0
<
γ
i
<
1. D
i is a constant that provides an adjustable tradeoff between risk neutrality (linear term) and risk aversion (exponential term). u
l
=
w is a linear utility function. u
i
,
e
(
w
)
=
-
γ
i
w is agent i’s exponential utility function.


## 2.2. Risk-Sensitive MDP Model Augmented with Wealth Level

In the paper goal directed Markov decision problem (GDMDP) is adopted as underlying model of decision-theoretic planning problem [8]. GDMDP is a kind of MDP with a finite set of goal states. When an agent reaches a goal state, he stops acting and receives no more rewards thereafter. One-switch utility function is used to describe the agent’s risk-sensitive attitude and maximizing expected utility is adopted as planning objective instead of maximizing expected reward. As wealth level is included in the one-switch utility function, it is necessary to consider the wealth level as a component of the system state of GDMDP.Formally, a GDMDP consists of a finite set of statesS with wealth levels W, so the augmented state set of GDMDP is denoted by (
S
,
W
). Goal state set is (
G
,
W
), where G
⊆
W. Nongoal state set is (
S
′
,
W
), where S
′
=
S
∖
G.The agent’s action set isA
s. The agent chooses an action a
∈
A
s to execute in its current state s
∈
S.The agent’s execution of actiona in state s results in a finite reward r
(
s
,
a
,
s
′
) and a transition to successor state s
′
∈
S with probability P
(
s
′
∣
s
,
a
). In the paper only cost is considered and assumed reward is strictly negative, r
(
s
,
a
,
s
′
)
<
0.We also uses
t and a
t to denote the state and action at time step t  (
t
=
0,1
,
2
,
…
). r
t
=
r
(
s
t
,
a
t
,
s
t
+
1
) is used to denote the reward for executing action a
t. After the agent reaches a goal state, r
t
=
0.w
t
=
w
0
+
∑
i
=
0
i
=
t
-
1
r
i is the agent’s wealth level at time step t, where the initial wealth level is denoted by w
0.For the MDP model augmented with wealth level, the optimal policy maps every combination of a states
∈
S
′ and wealth level w to an action a
∈
A
s that an agent in state s with wealth level w should execute to maximize expected utility.For agenti’s exponential utility function u
i
,
e
(
w
) and all policies π, we define the value v
i
,
e
π
(
s
,
w
)
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
i
,
e
(
w
t
)
] as the expected exponential utility of agent i with initial state s and initial wealth level w that follows policy π.The optimal valuev
i
,
e
*
(
s
,
w
)
=
max
⁡
π
⁡
v
i
,
e
π
(
s
,
w
) is defined as the highest possible expected exponential utility of agent i with initial state s and initial wealth level w. Assume v
i
,
e
*
(
s
,
w
) is finite for all state s
∈
S and wealth levels w.An optimal policyπ
i
,
e
*
(
s
,
w
) is defined as a policy with v
i
,
e
π
i
,
e
*
(
s
,
w
)
=
v
i
,
e
*
(
s
,
w
) for all state s
∈
S and wealth levels w.Similarly, for linear utility functionu
l
(
w
), we use v
l
π
(
s
,
w
), v
l
*
(
s
,
w
), and π
l
*
(
s
,
w
) to denote expected utility, optimal value, and optimal policy, respectively.It is worth noting that differently from Liu and Koenig [8], the paper focuses on the decision-making in group setting. So the value function in MDP will be replaced by group utility function which is the aggregation of personal one-switch utilities and the planning objective is to maximize the expected group utility.


## 3. Utility Aggregation Model of One-Switch Utility Functions

Group utility is the aggregation of personal utilities. The common methods include additive value model and multiplicative value model. In the following sections we will discuss additive and multiplicative value model for the aggregation of personal one-switch utility functions, respectively.


### 3.1. Additive Aggregation Model of One-Switch Utility Functions

In general, additive aggregation model of group utility is defined as follows:(2)
u
(
w
)
=
∑
i
=
1
n
k
i
u
i
(
w
)
,
where k
i is the weight of agent i’s utility, 0
<
k
i
<
1, and ∑
i
=
1
n
k
i
=
1
.Thus the additive aggregation model of one-switch utility functions is defined as follows:(3)
u
(
w
)
=
∑
i
=
1
n
k
i
u
i
(
w
)
=
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
-
⋯
-
k
n
D
n
γ
n
w
.
According to MDP, the expected group utility for all policies π and state (
s
,
w
) is presented as follows:
(4)
v
π
(
s
,
w
)
=
u
(
w
)
=
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
-
⋯
-
k
n
D
n
γ
n
w
∀
s
∈
G
,
∀
w
v
π
(
s
,
w
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
π
(
s
,
w
)
)
·
v
π
(
s
′
,
w
+
r
(
s
,
π
(
s
,
w
)
,
s
′
)
)
∀
s
∈
S
′
,
∀
w
.
sssssssssssss
sssss
Then, the optimal value v
*
(
s
,
w
) is presented as follows:
(5)
v
*
(
s
,
w
)
=
u
(
w
)
=
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
-
⋯
-
k
n
D
n
γ
n
w
∀
s
∈
G
,
∀
w
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
v
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
∀
s
∈
S
′
,
∀
w
.
Next, we will derive the relationship between the expected group utility and the expected personal linear and exponential utility for the additive aggregation model.For all policiesπ, the expected group utility of the additive aggregation model of one-switch utility functions is presented as follows:
(6)
v
π
(
s
,
w
)
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
l
(
w
t
)
+
k
1
D
1
u
1
,
e
(
w
t
)
s
s
s
s
s
s
s
s
s
s
s
s
s
s
+
k
2
D
2
u
2
,
e
(
w
t
)
s
s
s
s
s
s
s
s
s
s
s
s
s
s
+
⋯
+
k
n
D
n
u
n
,
e
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
l
(
w
t
)
]
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
1
D
1
u
1
,
e
(
w
t
)
]
+
⋯
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
n
D
n
u
n
,
e
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
w
+
∑
i
=
0
t
-
1
r
i
]
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
1
D
1
γ
1
w
+
∑
i
=
0
t
-
1
r
i
]
+
⋯
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
n
D
n
γ
n
w
+
∑
i
=
0
t
-
1
r
i
]
=
w
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
∑
i
=
0
t
-
1
r
i
]
+
k
1
D
1
γ
1
w
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
γ
1
∑
i
=
0
t
-
1
r
i
]
+
⋯
+
k
n
D
n
γ
n
w
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
γ
n
∑
i
=
0
t
-
1
r
i
]
=
w
+
v
l
π
(
s
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
(
s
)
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
(
s
)
,
where v
l
π
(
s
) and v
i
,
e
π
(
s
) satisfy the following policy-evaluation equations, respectively, [8, 19, 20]:
(7)
v
l
π
(
s
)
=
0
s
∈
G
v
l
π
(
s
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
π
(
s
)
)
[
r
(
s
,
π
(
s
)
,
s
’
)
+
v
l
π
(
s
′
)
]
s
∈
S
′
v
i
,
e
π
(
s
)
=
-
1
s
∈
G
v
i
,
e
π
(
s
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
π
(
s
)
)
[
γ
i
r
(
s
,
π
(
s
)
,
s
′
)
·
v
i
,
e
π
(
s
′
)
]
s
∈
S
′
.
From the above policy-evaluation equation (6), we obtain the relationship between v
π
(
s
,
w
) and expected linear utility v
l
π
(
s
) and expected personal exponential utility v
i
,
e
π
(
s
) for the additive aggregation model, where v
l
π
(
s
) and v
i
,
e
π
(
s
) are independent of the wealth level w.


### 3.2. Multiplicative Aggregation Model of One-Switch Utility Functions

In the paper we adopt the following multiplicative aggregation model of group utility:(8)
u
(
w
)
=
∑
i
=
1
n
k
i
u
i
(
w
)
+
∑
i
=
1
,
j
>
i
n
k
i
j
u
i
(
w
)
u
j
(
w
)
+
⋯
+
k
123
⋯
n
u
1
(
w
)
u
2
(
w
)
⋯
u
n
(
w
)
,
where k
1
,
k
2
,
…
,
k
n
,
k
12
,
k
23
,
…
,
k
123
⋯
n are constants and 0
<
k
i
<
1.For simplicity, in the paper we only consider the casen
=
2. For n
>
2, the derivation of multiplicative aggregation model is similar.Forn
=
2, multiplicative aggregation model of one-switch utility functions is simplified as follows:
(9)
u
(
w
)
=
k
1
u
1
(
w
)
+
k
2
u
2
(
w
)
+
k
12
u
1
(
w
)
u
2
(
w
)
=
k
12
w
2
-
k
12
D
1
w
γ
1
w
-
k
12
D
2
w
γ
2
w
+
k
12
D
1
D
2
(
γ
1
γ
2
)
w
+
k
1
w
+
k
2
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
.
According to MDP, the optimal value v
*
(
s
,
w
) is presented as follows:
(10)
v
*
(
s
,
w
)
=
u
(
w
)
∀
s
∈
G
,
∀
w
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
sssssssssss
·
v
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
∀
s
∈
S
′
,
∀
w
.
For the multiplicative aggregation model of one-switch utility functions and all policies π, the expected group utility is
(11)
v
π
(
s
,
w
)
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
1
u
1
(
w
)
+
k
2
u
2
(
w
)
+
k
12
u
1
(
w
)
u
2
(
w
)
]
=
k
1
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
1
(
w
)
]
+
k
2
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
2
(
w
)
]
+
k
12
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
1
(
w
)
]
·
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
2
(
w
)
]
.
According to the fact that lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
i
(
w
)
]
=
w
+
v
l
π
(
s
)
+
D
1
γ
1
w
v
i
,
e
π
(
s
) [8], we have
(12)
v
π
(
s
,
w
)
=
k
1
(
w
+
v
l
π
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
(
s
)
)
+
k
2
(
w
+
v
l
π
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
(
s
)
)
+
k
12
(
w
+
v
l
π
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
(
s
)
)
×
(
w
+
v
l
π
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
(
s
)
)
.
From the above policy-evaluation equation (12), we obtain the relationship between expected group utility v
π
(
s
,
w
) and expected linear utility v
l
π
(
s
) and expected personal exponential utility v
i
,
e
π
(
s
) for the multiplicative aggregation model.


## 3.1. Additive Aggregation Model of One-Switch Utility Functions

In general, additive aggregation model of group utility is defined as follows:(2)
u
(
w
)
=
∑
i
=
1
n
k
i
u
i
(
w
)
,
where k
i is the weight of agent i’s utility, 0
<
k
i
<
1, and ∑
i
=
1
n
k
i
=
1
.Thus the additive aggregation model of one-switch utility functions is defined as follows:(3)
u
(
w
)
=
∑
i
=
1
n
k
i
u
i
(
w
)
=
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
-
⋯
-
k
n
D
n
γ
n
w
.
According to MDP, the expected group utility for all policies π and state (
s
,
w
) is presented as follows:
(4)
v
π
(
s
,
w
)
=
u
(
w
)
=
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
-
⋯
-
k
n
D
n
γ
n
w
∀
s
∈
G
,
∀
w
v
π
(
s
,
w
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
π
(
s
,
w
)
)
·
v
π
(
s
′
,
w
+
r
(
s
,
π
(
s
,
w
)
,
s
′
)
)
∀
s
∈
S
′
,
∀
w
.
sssssssssssss
sssss
Then, the optimal value v
*
(
s
,
w
) is presented as follows:
(5)
v
*
(
s
,
w
)
=
u
(
w
)
=
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
-
⋯
-
k
n
D
n
γ
n
w
∀
s
∈
G
,
∀
w
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
v
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
∀
s
∈
S
′
,
∀
w
.
Next, we will derive the relationship between the expected group utility and the expected personal linear and exponential utility for the additive aggregation model.For all policiesπ, the expected group utility of the additive aggregation model of one-switch utility functions is presented as follows:
(6)
v
π
(
s
,
w
)
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
l
(
w
t
)
+
k
1
D
1
u
1
,
e
(
w
t
)
s
s
s
s
s
s
s
s
s
s
s
s
s
s
+
k
2
D
2
u
2
,
e
(
w
t
)
s
s
s
s
s
s
s
s
s
s
s
s
s
s
+
⋯
+
k
n
D
n
u
n
,
e
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
l
(
w
t
)
]
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
1
D
1
u
1
,
e
(
w
t
)
]
+
⋯
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
n
D
n
u
n
,
e
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
w
+
∑
i
=
0
t
-
1
r
i
]
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
1
D
1
γ
1
w
+
∑
i
=
0
t
-
1
r
i
]
+
⋯
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
n
D
n
γ
n
w
+
∑
i
=
0
t
-
1
r
i
]
=
w
+
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
∑
i
=
0
t
-
1
r
i
]
+
k
1
D
1
γ
1
w
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
γ
1
∑
i
=
0
t
-
1
r
i
]
+
⋯
+
k
n
D
n
γ
n
w
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
γ
n
∑
i
=
0
t
-
1
r
i
]
=
w
+
v
l
π
(
s
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
(
s
)
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
(
s
)
,
where v
l
π
(
s
) and v
i
,
e
π
(
s
) satisfy the following policy-evaluation equations, respectively, [8, 19, 20]:
(7)
v
l
π
(
s
)
=
0
s
∈
G
v
l
π
(
s
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
π
(
s
)
)
[
r
(
s
,
π
(
s
)
,
s
’
)
+
v
l
π
(
s
′
)
]
s
∈
S
′
v
i
,
e
π
(
s
)
=
-
1
s
∈
G
v
i
,
e
π
(
s
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
π
(
s
)
)
[
γ
i
r
(
s
,
π
(
s
)
,
s
′
)
·
v
i
,
e
π
(
s
′
)
]
s
∈
S
′
.
From the above policy-evaluation equation (6), we obtain the relationship between v
π
(
s
,
w
) and expected linear utility v
l
π
(
s
) and expected personal exponential utility v
i
,
e
π
(
s
) for the additive aggregation model, where v
l
π
(
s
) and v
i
,
e
π
(
s
) are independent of the wealth level w.


## 3.2. Multiplicative Aggregation Model of One-Switch Utility Functions

In the paper we adopt the following multiplicative aggregation model of group utility:(8)
u
(
w
)
=
∑
i
=
1
n
k
i
u
i
(
w
)
+
∑
i
=
1
,
j
>
i
n
k
i
j
u
i
(
w
)
u
j
(
w
)
+
⋯
+
k
123
⋯
n
u
1
(
w
)
u
2
(
w
)
⋯
u
n
(
w
)
,
where k
1
,
k
2
,
…
,
k
n
,
k
12
,
k
23
,
…
,
k
123
⋯
n are constants and 0
<
k
i
<
1.For simplicity, in the paper we only consider the casen
=
2. For n
>
2, the derivation of multiplicative aggregation model is similar.Forn
=
2, multiplicative aggregation model of one-switch utility functions is simplified as follows:
(9)
u
(
w
)
=
k
1
u
1
(
w
)
+
k
2
u
2
(
w
)
+
k
12
u
1
(
w
)
u
2
(
w
)
=
k
12
w
2
-
k
12
D
1
w
γ
1
w
-
k
12
D
2
w
γ
2
w
+
k
12
D
1
D
2
(
γ
1
γ
2
)
w
+
k
1
w
+
k
2
w
-
k
1
D
1
γ
1
w
-
k
2
D
2
γ
2
w
.
According to MDP, the optimal value v
*
(
s
,
w
) is presented as follows:
(10)
v
*
(
s
,
w
)
=
u
(
w
)
∀
s
∈
G
,
∀
w
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
sssssssssss
·
v
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
∀
s
∈
S
′
,
∀
w
.
For the multiplicative aggregation model of one-switch utility functions and all policies π, the expected group utility is
(11)
v
π
(
s
,
w
)
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
(
w
t
)
]
=
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
k
1
u
1
(
w
)
+
k
2
u
2
(
w
)
+
k
12
u
1
(
w
)
u
2
(
w
)
]
=
k
1
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
1
(
w
)
]
+
k
2
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
2
(
w
)
]
+
k
12
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
1
(
w
)
]
·
lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
2
(
w
)
]
.
According to the fact that lim
⁡
t
→
∞
⁡
E
s
,
w
π
[
u
i
(
w
)
]
=
w
+
v
l
π
(
s
)
+
D
1
γ
1
w
v
i
,
e
π
(
s
) [8], we have
(12)
v
π
(
s
,
w
)
=
k
1
(
w
+
v
l
π
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
(
s
)
)
+
k
2
(
w
+
v
l
π
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
(
s
)
)
+
k
12
(
w
+
v
l
π
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
(
s
)
)
×
(
w
+
v
l
π
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
(
s
)
)
.
From the above policy-evaluation equation (12), we obtain the relationship between expected group utility v
π
(
s
,
w
) and expected linear utility v
l
π
(
s
) and expected personal exponential utility v
i
,
e
π
(
s
) for the multiplicative aggregation model.


## 4. Preparation for Backward-Induction Method

To solve the optimal policy of the additive and multiplicative aggregation model of one-switch utility functions, backward-induction method is adopted. In the paper the value range of wealth level is a continuous interval(
-
∞
,
w
0
]; we first compute the optimal policy when wealth level w
→
-
∞ that is represented by π
-
∞
*. Then increase the wealth level until π
-
∞
* is no longer an optimal policy and get a wealth level threshold. Increase further the wealth level and get the next wealth level threshold similarly. The backward-induction method ends when the wealth level is larger than initial wealth level w
0. Thus the continuous wealth level interval is divided into subintervals by the thresholds. We use (
(
s
,
(
w
i
,
w
i
+
1
]
)
,
a
) to denote action a executed in state s and wealth level interval (
w
i
,
w
i
+
1
] (w
i denotes a wealth level threshold and i
=
0,1
,
2
,
…). In this section we will analyze the characteristics of optimal policy when the wealth level w approaches negative infinity for additive and multiplicative aggregation model, respectively.Lemma 1.
For additive aggregation model of one-switch utility functions, if agenti is the most risk-averse one, that is, γ
i
<
γ
j, for any j
≠
i, i
,
j
=
1,2
,
…
,
n, then
(13)
lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
i
-
w
=
k
i
D
i
v
i
,
e
*
(
s
)
f
o
r
a
l
l
s
t
a
t
e
s
s
∈
S
.Proof.
For all optimal policiesπ
*, we have
(14)
lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
i
-
w
=
lim
⁡
w
→
-
∞
⁡
v
π
*
(
s
,
w
)
γ
i
-
w
=
lim
⁡
w
→
-
∞
[
w
γ
i
-
w
+
v
l
π
*
(
s
)
γ
i
-
w
+
k
1
D
1
(
γ
1
γ
i
)
w
v
1
,
e
π
*
(
s
)
s
s
s
s
s
s
s
s
s
s
s
s
+
⋯
+
k
i
D
i
v
i
,
e
π
*
(
s
)
+
⋯
+
k
n
D
n
(
γ
n
γ
i
)
w
v
n
,
e
π
*
(
s
)
]
.
As γ
j
/
γ
i
>
1, j
≠
i, we can derive lim
⁡
w
→
-
∞
⁡
k
j
D
j
(
γ
j
/
γ
i
)
w
v
j
,
e
π
*
(
s
)
=
0.
Thuslim
⁡
w
→
-
∞
v
*
(
s
,
w
)
γ
i
-
w
=
k
i
D
i
v
i
,
e
π
*
(
s
)
≤
k
i
D
i
v
i
,
e
*
(
s
).
On the other hand, for all optimal policiesπ
i
,
e
*, according to the fact that v
*
(
s
,
w
)
≥
v
π
(
s
,
w
) for all policies π, we have
(15)
lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
i
-
w
≥
lim
⁡
w
→
-
∞
⁡
v
π
i
,
e
*
(
s
,
w
)
γ
i
-
w
=
lim
⁡
w
→
-
∞
[
D
n
(
γ
n
γ
i
)
w
v
n
,
e
π
i
,
e
*
(
s
)
w
γ
i
-
w
+
v
l
π
i
,
e
*
(
s
)
γ
i
-
w
n
n
+
k
1
D
1
(
γ
1
γ
i
)
w
v
1
,
e
π
i
,
e
*
(
s
)
n
n
+
⋯
+
k
i
D
i
v
i
,
e
π
i
,
e
*
(
s
)
n
n
+
⋯
+
k
n
D
n
(
γ
n
γ
i
)
w
v
n
,
e
π
i
,
e
*
(
s
)
]
=
k
i
D
i
v
i
,
e
*
(
s
)
.
Therefore, the lemma holds.Lemma1 implies that the optimal policy for the additive aggregation model of one-switch utility functions is the same as the most risk-averse agent’s optimal policy for the exponential utility function as the wealth level w
→
-
∞.Lemma 2.
For multiplicative aggregation model of one-switch utility functions,lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
3
-
w
=
k
12
D
1
D
2
v
3
,
e
*
(
s
) for all states s
∈
S and γ
3
=
γ
1
γ
2, where v
3
,
e
*
(
s
,
w
) is the highest expected exponential utility with risk aversion parameter γ
3.Proof.
For all optimal policiesπ
*, we have
(16)
lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
3
-
w
=
lim
⁡
w
→
-
∞
⁡
v
π
*
(
s
,
w
)
γ
3
-
w
=
lim
⁡
w
→
-
∞
γ
3
-
w
[
k
1
(
w
+
v
l
π
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
*
(
s
)
)
n
n
n
n
+
k
2
(
w
+
v
l
π
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
*
(
s
)
)
n
n
n
n
+
k
12
(
w
+
v
l
π
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
*
(
s
)
)
n
n
n
n
×
(
w
+
v
l
π
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
*
(
s
)
)
]
.
As γ
i
/
γ
3
>
1, i
=
1,2,
Thenlim
⁡
w
→
-
∞
⁡
k
i
D
i
(
γ
i
/
γ
3
)
w
v
i
,
e
π
*
(
s
)
=
0, lim
⁡
w
→
-
∞
⁡
k
i
D
i
w
(
γ
i
/
γ
3
)
w
v
i
,
e
π
*
(
s
)
=
0.
Therefore,(17)
lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
3
-
w
=
k
12
D
1
D
2
v
1
,
e
π
*
(
s
)
v
2
,
e
π
*
(
s
)
=
k
12
D
1
D
2
lim
⁡
w
→
-
∞
E
s
,
w
π
*
[
γ
1
∑
i
=
0
t
-
1
r
i
]
·
lim
⁡
w
→
-
∞
⁡
E
s
,
w
π
[
γ
2
∑
i
=
0
t
-
1
r
i
]
=
k
12
D
1
D
2
lim
⁡
w
→
-
∞
E
s
,
w
π
*
[
(
γ
1
γ
2
)
∑
i
=
0
t
-
1
r
i
]
≤
k
12
D
1
D
2
v
3
,
e
*
(
s
)
.
On the other hand, for all optimal policies π
3
,
e
*, according to the fact that v
*
(
s
,
w
)
≥
v
π
(
s
,
w
) for all policies π, we have
(18)
lim
⁡
w
→
-
∞
⁡
v
*
(
s
,
w
)
γ
3
-
w
≥
lim
⁡
w
→
-
∞
⁡
v
π
3
,
e
*
(
s
,
w
)
γ
3
-
w
=
lim
⁡
w
→
-
∞
γ
3
-
w
[
k
1
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
3
,
e
*
(
s
)
)
nnn
+
k
2
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
3
,
e
*
(
s
)
)
nnn
+
k
12
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
3
,
e
*
(
s
)
)
nnn
×
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
3
,
e
*
(
s
)
)
]
=
k
12
D
1
D
2
v
1
,
e
π
3
,
e
*
(
s
)
v
2
,
e
π
3
,
e
*
(
s
)
=
k
12
D
1
D
2
v
3
,
e
*
(
s
)
.
Therefore, the lemma holds.Lemma2 implies that the optimal policy for the multiplicative aggregation of one-switch utility functions is the same as the optimal policy for a virtual agent’s exponential utility function as the wealth level w
→
-
∞. The virtual agent’s risk aversion parameter is the product of every agent’s risk aversion parameter in the group.


## 5. Division of Wealth Level Interval and Backward-Induction Method

The above section gives the optimal policy as the wealth level approaches negative infinity for additive and multiplicative aggregation model of one-switch utility functions, respectively. The next step is to divide the wealth level interval and determine the wealth level thresholds and optimal policies in the intervals by using backward-induction method. In this section we will discuss the backward-induction method in the cases of additive and multiplicative aggregation model.


### 5.1. The Case of Additive Aggregation Model

For the additive aggregation model of one-switch utility functions, we first give the following theorem to prove the existence of a wealth level thresholdw
_ and then give the backward-induction algorithm.Theorem 3.
For all optimal policiesπ
*, there exists a wealth level threshold w
_ such that it holds for all states s
∈
S
′ and all wealth levels w
≤
w
_ that
(19)
v
l
*
(
s
,
w
)
=
v
l
π
*
(
s
)
,
v
i
,
e
*
(
s
,
w
)
=
v
i
,
e
π
*
(
s
)
,
v
*
(
s
,
w
)
=
v
π
*
(
s
,
w
)
,
i
=
1,2
,
…
n
.
Please see Appendix A for the proof of Theorem 3.Theorem3 shows the existence of the wealth level threshold w
_. Next we will show how to determine the wealth level threshold w
_.After gettingπ
-
∞
* when wealth level w
→
-
∞, assume π
-
∞
* is the optimal policy for the wealth level interval (
-
∞
,
w
_
]; then for wealth level w in the interval (
w
_
,
min
⁡
a
∈
A
s
⁡
(
w
_
-
r
(
s
,
a
,
s
′
)
)
⁡
], where r
(
s
,
a
,
s
′
) is the reward got by executing one step action, π
-
∞
* is no longer the optimal policy, and assume π
′ is the optimal policy; according to (5); for all nongoal states, we have
(20)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
’
∈
S
P
(
s
′
∣
s
,
a
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
.
As w
+
r
(
s
,
a
,
s
′
)
<
w
_,
(21)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
+
k
1
D
1
γ
1
w
+
r
(
s
,
a
,
s
′
)
v
1
,
e
π
-
∞
*
(
s
)
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
+
⋯
+
k
n
D
n
γ
n
w
+
r
(
s
,
a
,
s
′
)
v
n
,
e
π
-
∞
*
(
s
)
)
=
max
⁡
a
∈
A
s
[
w
+
v
l
π
-
∞
*
(
s
,
a
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
)
ssssssssss
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
-
∞
*
(
s
,
a
)
]
,
where
(22)
v
l
*
(
s
,
a
)
=
∑
s
’
∈
S
P
(
s
′
∣
s
,
a
)
[
r
(
s
,
a
,
s
′
)
+
v
l
*
(
s
′
)
]
,
v
i
,
e
*
(
s
,
a
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
[
γ
i
r
(
s
,
a
,
s
′
)
v
i
,
e
*
(
s
′
)
]
.
For s
∈
S
′
∖
G, a
j
∈
A
s
∖
π
-
∞
*
(
s
), because the optimal policy is π
′, not π
-
∞
* now under assumption, we have
(23)
∑
s
’
∈
S
P
(
s
′
∣
s
,
π
-
∞
*
(
s
)
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
π
-
∞
*
(
s
)
,
s
′
)
)
≥
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
j
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
j
,
s
′
)
)
or, equivalently,
(24)
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
≥
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
j
)
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
-
∞
*
(
s
,
a
j
)
.
We can get a wealth level threshold w
_
π
-
∞
*
,
s
,
a
j in equality case of the above weak inequality.From the algorithm above, we can get the wealth level thresholdw
_:
(25)
w
_
=
min
⁡
s
∈
s
′
⁡
min
⁡
a
j
∈
A
s
∖
π
-
∞
*
(
s
)
⁡
min
⁡
⁡
(
w
0
,
w
_
π
-
∞
*
,
s
,
a
j
)
.
After getting w
_, the next step is to divide further the wealth level interval (
w
_
,
w
0
] into subintervals and solve the optimal policy for each subinterval similarly to the above algorithm. The main procedure of the backward-induction algorithm for group decision-making in the case of additive aggregation model is listed as follows.Step 1.
By maximizing the expected exponential utility, get the optimal policyπ
i
,
e
* of the most risk-averse agent when w
→
-
∞.Step 2.
According to (6), we have
(26)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
i
,
e
*
(
s
)
{
w
+
v
l
π
i
,
e
*
(
s
,
a
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
i
,
e
*
(
s
,
a
)
n
n
n
n
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
i
,
e
*
(
s
,
a
)
}
,
where A
i
,
e
*
(
s
)
=
arg
⁡
max
⁡
a
∈
A
s
⁡
∑
s
∈
S
P
(
s
′
∣
s
,
a
)
[
γ
i
r
(
s
,
a
,
s
’
)
·
v
i
,
e
*
(
s
′
)
]; get the optimal value v
*
(
s
,
w
) and the values of v
1
,
e
*
(
s
)
,
v
2
,
e
*
(
s
)
,
…
,
v
n
,
e
*
(
s
)
,
v
l
*
(
s
) for all states s
∈
s
′ when w
→
-
∞.Step 3.
For all statess
∈
s
′, a
∈
A
s
∖
π
-
∞
*
(
s
), get the values of v
l
π
-
∞
*
(
s
,
a
)
,
v
1
,
e
π
-
∞
*
(
s
,
a
)
,
…
,
v
n
,
e
π
-
∞
*
(
s
,
a
) by (7); then get w
_
π
-
∞
*
,
s
,
a
j by Expression (24).Step 4.
Calculate the wealth level thresholdw
_ according to (25).Step 5.
For the wealth level interval(
w
_
,
w
0
], increase further the wealth level according to the reward got by executing one step action, and determine the wealth level threshold and optimal policy similarly to the above steps.Step 6.
If, for alls
∈
s
′, the wealth level w is larger than w
0, then end the algorithm.


### 5.2. The Case of Multiplicative Aggregation Model

For the multiplicative aggregation model of one-switch utility functions, we also have the following theorem that shows the existence of a wealth level thresholdw
_.Theorem 4.
For all optimal policiesπ
*, there exists a wealth threshold w
_. For any state s
∈
S
′, wealth level w
≤
w
_,
(27)
v
l
*
(
s
,
w
)
=
v
l
π
*
(
s
)
,
v
i
,
e
*
(
s
,
w
)
=
v
i
,
e
π
*
(
s
)
,
s
s
s
s
s
s
s
s
s
s
v
*
(
s
,
w
)
=
v
π
*
(
s
,
w
)
,
i
=
1,2
.
Please see Appendix B for the proof of Theorem 4.Similarly to additive aggregation model, we determine the wealth level thresholdw
_ for the multiplicative aggregation model. Assume that π
-
∞
* is the optimal policy for the wealth level interval (
-
∞
,
w
_
]; then for the wealth level w in the interval (
w
_
,
min
⁡
a
∈
A
s
⁡
(
w
_
-
r
(
s
,
a
,
s
′
)
)
⁡
], π
-
∞
* is no longer the optimal policy, and assuming π
′ is the optimal policy, according to (10), for all nongoal states, we have
(28)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
.
As w
+
r
(
s
,
a
,
s
′
)
<
w
_,
(29)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
[
k
1
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nn
+
D
1
γ
1
w
+
r
(
s
,
a
,
s
′
)
v
1
,
e
π
-
∞
*
(
s
)
)
nn
+
k
2
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nnnn
+
D
2
γ
2
w
+
r
(
s
,
a
,
s
′
)
v
2
,
e
π
-
∞
*
(
s
)
)
nn
+
k
12
(
D
1
γ
1
w
+
r
(
s
,
a
,
s
’
)
v
1
,
e
π
-
∞
*
(
s
)
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nnnn
+
D
1
γ
1
w
+
r
(
s
,
a
,
s
′
)
v
1
,
e
π
-
∞
*
(
s
)
)
nn
×
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nnnnnn
+
D
2
γ
2
w
+
r
(
s
,
a
,
s
′
)
v
2
,
e
π
-
∞
*
(
s
)
)
]
=
max
⁡
a
∈
A
s
[
k
1
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
)
)
n
+
k
2
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
)
)
n
+
k
12
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
)
)
n
×
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
)
)
]
.
For w, because the optimal policy is π
′, not π
-
∞
* now under assumption, we have
(30)
∑
s
’
∈
S
′
P
(
s
′
∣
s
,
π
-
∞
*
(
s
)
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
π
-
∞
*
(
s
)
,
s
′
)
)
≤
∑
s
′
∈
S
′
P
(
s
′
∣
s
,
a
j
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
j
,
s
′
)
)
or, equivalently,
(31)
k
1
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
+
k
2
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
+
k
12
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
×
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
≤
k
1
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
j
)
)
+
k
2
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
a
j
)
)
+
k
12
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
j
)
)
×
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
a
j
)
)
.
We can get a wealth level point w
_
π
-
∞
*
,
s
,
a
j in equality case of the above weak inequality.Then, we can get the wealth level thresholdw
_ according to (25).After gettingw
_, the next step is to divide further the interval (
w
_
,
w
0
] into subintervals and compute the optimal policy for each subinterval. The main procedure of the backward-induction algorithm for group decision-making in the case of multiplicative aggregation model is listed as follows.Step 1.
By maximizing the expected exponential utility, get the optimal policyπ
3
,
e
* of the visual agent when w
→
-
∞.Step 2.
According to (12), we have
(32)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
3
,
e
*
(
s
)
{
k
1
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
3
,
e
*
(
s
)
)
nnn
+
k
2
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
3
,
e
*
(
s
)
)
nnn
+
k
12
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
3
,
e
*
(
s
)
)
nnn
×
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
3
,
e
*
(
s
)
)
}
,
where A
3
,
e
*
(
s
)
=
arg
max
⁡
a
∈
A
s
∑
s
∈
S
P
(
s
′
∣
s
,
a
)
[
γ
3
r
(
s
,
a
,
s
′
)
·
v
3
,
e
*
(
s
′
)
]; get the optimal value v
*
(
s
,
w
) and the values of v
1
,
e
*
(
s
), v
2
,
e
*
(
s
), v
l
*
(
s
) for all states s
∈
s
′ when w
→
-
∞.Step 3.
For all statess
∈
s
′, a
∈
A
s
∖
π
-
∞
*
(
s
), get the values of v
l
π
-
∞
*
(
s
,
a
), v
1
,
e
π
-
∞
*
(
s
,
a
), v
2
,
e
π
-
∞
*
(
s
,
a
) by (7); then get w
_
π
-
∞
*
,
s
,
a
j by Expression (31).Step 4.
Calculate the wealth level thresholdw
_ according to (25).Step 5.
For the wealth level interval(
w
_
,
w
0
], increase further the wealth level according to the reward got by executing one step action and determine the wealth level threshold and optimal policy similarly to the above steps.Step 6.
If, for alls
∈
s
′, the wealth level w is larger than w
0, then end the algorithm.


## 5.1. The Case of Additive Aggregation Model

For the additive aggregation model of one-switch utility functions, we first give the following theorem to prove the existence of a wealth level thresholdw
_ and then give the backward-induction algorithm.Theorem 3.
For all optimal policiesπ
*, there exists a wealth level threshold w
_ such that it holds for all states s
∈
S
′ and all wealth levels w
≤
w
_ that
(19)
v
l
*
(
s
,
w
)
=
v
l
π
*
(
s
)
,
v
i
,
e
*
(
s
,
w
)
=
v
i
,
e
π
*
(
s
)
,
v
*
(
s
,
w
)
=
v
π
*
(
s
,
w
)
,
i
=
1,2
,
…
n
.
Please see Appendix A for the proof of Theorem 3.Theorem3 shows the existence of the wealth level threshold w
_. Next we will show how to determine the wealth level threshold w
_.After gettingπ
-
∞
* when wealth level w
→
-
∞, assume π
-
∞
* is the optimal policy for the wealth level interval (
-
∞
,
w
_
]; then for wealth level w in the interval (
w
_
,
min
⁡
a
∈
A
s
⁡
(
w
_
-
r
(
s
,
a
,
s
′
)
)
⁡
], where r
(
s
,
a
,
s
′
) is the reward got by executing one step action, π
-
∞
* is no longer the optimal policy, and assume π
′ is the optimal policy; according to (5); for all nongoal states, we have
(20)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
’
∈
S
P
(
s
′
∣
s
,
a
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
.
As w
+
r
(
s
,
a
,
s
′
)
<
w
_,
(21)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
+
k
1
D
1
γ
1
w
+
r
(
s
,
a
,
s
′
)
v
1
,
e
π
-
∞
*
(
s
)
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
+
⋯
+
k
n
D
n
γ
n
w
+
r
(
s
,
a
,
s
′
)
v
n
,
e
π
-
∞
*
(
s
)
)
=
max
⁡
a
∈
A
s
[
w
+
v
l
π
-
∞
*
(
s
,
a
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
)
ssssssssss
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
-
∞
*
(
s
,
a
)
]
,
where
(22)
v
l
*
(
s
,
a
)
=
∑
s
’
∈
S
P
(
s
′
∣
s
,
a
)
[
r
(
s
,
a
,
s
′
)
+
v
l
*
(
s
′
)
]
,
v
i
,
e
*
(
s
,
a
)
=
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
[
γ
i
r
(
s
,
a
,
s
′
)
v
i
,
e
*
(
s
′
)
]
.
For s
∈
S
′
∖
G, a
j
∈
A
s
∖
π
-
∞
*
(
s
), because the optimal policy is π
′, not π
-
∞
* now under assumption, we have
(23)
∑
s
’
∈
S
P
(
s
′
∣
s
,
π
-
∞
*
(
s
)
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
π
-
∞
*
(
s
)
,
s
′
)
)
≥
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
j
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
j
,
s
′
)
)
or, equivalently,
(24)
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
≥
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
j
)
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
-
∞
*
(
s
,
a
j
)
.
We can get a wealth level threshold w
_
π
-
∞
*
,
s
,
a
j in equality case of the above weak inequality.From the algorithm above, we can get the wealth level thresholdw
_:
(25)
w
_
=
min
⁡
s
∈
s
′
⁡
min
⁡
a
j
∈
A
s
∖
π
-
∞
*
(
s
)
⁡
min
⁡
⁡
(
w
0
,
w
_
π
-
∞
*
,
s
,
a
j
)
.
After getting w
_, the next step is to divide further the wealth level interval (
w
_
,
w
0
] into subintervals and solve the optimal policy for each subinterval similarly to the above algorithm. The main procedure of the backward-induction algorithm for group decision-making in the case of additive aggregation model is listed as follows.Step 1.
By maximizing the expected exponential utility, get the optimal policyπ
i
,
e
* of the most risk-averse agent when w
→
-
∞.Step 2.
According to (6), we have
(26)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
i
,
e
*
(
s
)
{
w
+
v
l
π
i
,
e
*
(
s
,
a
)
+
k
1
D
1
γ
1
w
v
1
,
e
π
i
,
e
*
(
s
,
a
)
n
n
n
n
+
⋯
+
k
n
D
n
γ
n
w
v
n
,
e
π
i
,
e
*
(
s
,
a
)
}
,
where A
i
,
e
*
(
s
)
=
arg
⁡
max
⁡
a
∈
A
s
⁡
∑
s
∈
S
P
(
s
′
∣
s
,
a
)
[
γ
i
r
(
s
,
a
,
s
’
)
·
v
i
,
e
*
(
s
′
)
]; get the optimal value v
*
(
s
,
w
) and the values of v
1
,
e
*
(
s
)
,
v
2
,
e
*
(
s
)
,
…
,
v
n
,
e
*
(
s
)
,
v
l
*
(
s
) for all states s
∈
s
′ when w
→
-
∞.Step 3.
For all statess
∈
s
′, a
∈
A
s
∖
π
-
∞
*
(
s
), get the values of v
l
π
-
∞
*
(
s
,
a
)
,
v
1
,
e
π
-
∞
*
(
s
,
a
)
,
…
,
v
n
,
e
π
-
∞
*
(
s
,
a
) by (7); then get w
_
π
-
∞
*
,
s
,
a
j by Expression (24).Step 4.
Calculate the wealth level thresholdw
_ according to (25).Step 5.
For the wealth level interval(
w
_
,
w
0
], increase further the wealth level according to the reward got by executing one step action, and determine the wealth level threshold and optimal policy similarly to the above steps.Step 6.
If, for alls
∈
s
′, the wealth level w is larger than w
0, then end the algorithm.


## 5.2. The Case of Multiplicative Aggregation Model

For the multiplicative aggregation model of one-switch utility functions, we also have the following theorem that shows the existence of a wealth level thresholdw
_.Theorem 4.
For all optimal policiesπ
*, there exists a wealth threshold w
_. For any state s
∈
S
′, wealth level w
≤
w
_,
(27)
v
l
*
(
s
,
w
)
=
v
l
π
*
(
s
)
,
v
i
,
e
*
(
s
,
w
)
=
v
i
,
e
π
*
(
s
)
,
s
s
s
s
s
s
s
s
s
s
v
*
(
s
,
w
)
=
v
π
*
(
s
,
w
)
,
i
=
1,2
.
Please see Appendix B for the proof of Theorem 4.Similarly to additive aggregation model, we determine the wealth level thresholdw
_ for the multiplicative aggregation model. Assume that π
-
∞
* is the optimal policy for the wealth level interval (
-
∞
,
w
_
]; then for the wealth level w in the interval (
w
_
,
min
⁡
a
∈
A
s
⁡
(
w
_
-
r
(
s
,
a
,
s
′
)
)
⁡
], π
-
∞
* is no longer the optimal policy, and assuming π
′ is the optimal policy, according to (10), for all nongoal states, we have
(28)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
,
s
′
)
)
.
As w
+
r
(
s
,
a
,
s
′
)
<
w
_,
(29)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
s
⁡
∑
s
′
∈
S
P
(
s
′
∣
s
,
a
)
·
[
k
1
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nn
+
D
1
γ
1
w
+
r
(
s
,
a
,
s
′
)
v
1
,
e
π
-
∞
*
(
s
)
)
nn
+
k
2
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nnnn
+
D
2
γ
2
w
+
r
(
s
,
a
,
s
′
)
v
2
,
e
π
-
∞
*
(
s
)
)
nn
+
k
12
(
D
1
γ
1
w
+
r
(
s
,
a
,
s
’
)
v
1
,
e
π
-
∞
*
(
s
)
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nnnn
+
D
1
γ
1
w
+
r
(
s
,
a
,
s
′
)
v
1
,
e
π
-
∞
*
(
s
)
)
nn
×
(
w
+
r
(
s
,
a
,
s
′
)
+
v
l
π
-
∞
*
(
s
)
nnnnnn
+
D
2
γ
2
w
+
r
(
s
,
a
,
s
′
)
v
2
,
e
π
-
∞
*
(
s
)
)
]
=
max
⁡
a
∈
A
s
[
k
1
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
)
)
n
+
k
2
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
)
)
n
+
k
12
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
)
)
n
×
(
w
+
v
l
π
-
∞
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
)
)
]
.
For w, because the optimal policy is π
′, not π
-
∞
* now under assumption, we have
(30)
∑
s
’
∈
S
′
P
(
s
′
∣
s
,
π
-
∞
*
(
s
)
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
π
-
∞
*
(
s
)
,
s
′
)
)
≤
∑
s
′
∈
S
′
P
(
s
′
∣
s
,
a
j
)
·
v
π
-
∞
*
(
s
′
,
w
+
r
(
s
,
a
j
,
s
′
)
)
or, equivalently,
(31)
k
1
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
+
k
2
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
+
k
12
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
×
(
w
+
v
l
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
π
-
∞
*
(
s
)
)
)
≤
k
1
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
j
)
)
+
k
2
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
a
j
)
)
+
k
12
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
1
γ
1
w
v
1
,
e
π
-
∞
*
(
s
,
a
j
)
)
×
(
w
+
v
l
π
-
∞
*
(
s
,
a
j
)
+
D
2
γ
2
w
v
2
,
e
π
-
∞
*
(
s
,
a
j
)
)
.
We can get a wealth level point w
_
π
-
∞
*
,
s
,
a
j in equality case of the above weak inequality.Then, we can get the wealth level thresholdw
_ according to (25).After gettingw
_, the next step is to divide further the interval (
w
_
,
w
0
] into subintervals and compute the optimal policy for each subinterval. The main procedure of the backward-induction algorithm for group decision-making in the case of multiplicative aggregation model is listed as follows.Step 1.
By maximizing the expected exponential utility, get the optimal policyπ
3
,
e
* of the visual agent when w
→
-
∞.Step 2.
According to (12), we have
(32)
v
*
(
s
,
w
)
=
max
⁡
a
∈
A
3
,
e
*
(
s
)
{
k
1
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
3
,
e
*
(
s
)
)
nnn
+
k
2
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
3
,
e
*
(
s
)
)
nnn
+
k
12
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
1
γ
1
w
v
1
,
e
π
3
,
e
*
(
s
)
)
nnn
×
(
w
+
v
l
π
3
,
e
*
(
s
)
+
D
2
γ
2
w
v
2
,
e
π
3
,
e
*
(
s
)
)
}
,
where A
3
,
e
*
(
s
)
=
arg
max
⁡
a
∈
A
s
∑
s
∈
S
P
(
s
′
∣
s
,
a
)
[
γ
3
r
(
s
,
a
,
s
′
)
·
v
3
,
e
*
(
s
′
)
]; get the optimal value v
*
(
s
,
w
) and the values of v
1
,
e
*
(
s
), v
2
,
e
*
(
s
), v
l
*
(
s
) for all states s
∈
s
′ when w
→
-
∞.Step 3.
For all statess
∈
s
′, a
∈
A
s
∖
π
-
∞
*
(
s
), get the values of v
l
π
-
∞
*
(
s
,
a
), v
1
,
e
π
-
∞
*
(
s
,
a
), v
2
,
e
π
-
∞
*
(
s
,
a
) by (7); then get w
_
π
-
∞
*
,
s
,
a
j by Expression (31).Step 4.
Calculate the wealth level thresholdw
_ according to (25).Step 5.
For the wealth level interval(
w
_
,
w
0
], increase further the wealth level according to the reward got by executing one step action and determine the wealth level threshold and optimal policy similarly to the above steps.Step 6.
If, for alls
∈
s
′, the wealth level w is larger than w
0, then end the algorithm.


## 6. Numerical Examples

Consider a simple GDMDP model. There are two agents named Agent1 and Agent2 with risk aversion parameters γ
1 and γ
2, respectively. The state set of the GDMDP model includes initial state s
0 and goal state g. Agent’s action set is {
a
1
,
a
2
}. In state s
0 executing action a
i  (
i
=
1,2
) results in a finite reward r
i and a transition to goal state g with possibility p
i. When agent reaches the goal state g it stops acting and receives no more rewards thereafter. Figure 1 shows the transitions of system states. Agents need to make an optimal policy together to reach the goal state.Figure 1
System state transitions.Without loss of generality, the GDMDP model’s parameters are assumed as follows:p
1
=
0.4, p
2
=
0.005, r
1
=
-
15, and r
2
=
-
40; the one-switch utility functions of two agents are defined as u
1
(
w
)
=
w
-
0.952
w and u
2
(
w
)
=
w
-
0.93
w with γ
1
=
0.952 and γ
2
=
0.93, respectively. Initial wealth level of each agent is set 0.First, consider the situation that each agent makes decisions alone. The agent’s optimal policy and the wealth level threshold are solved by utilizing the method proposed by Liu and Koenig [8]. The results are shown as follows:
(33)
Agent
1
:
π
*
(
s
0
,
w
)
=
{
a
1
w
∈
(
-
106.5
,
0
]
a
2
w
∈
(
-
∞
,
-
106.5
]
Agent
2
:
π
*
(
s
0
,
w
)
=
{
a
1
w
∈
(
-
6.9
,
0
]
a
2
w
∈
(
-
∞
,
-
6.9
]
.
Next, we consider the group decision-making based on additive and multiplicative aggregation model of one-switch utility functions. In the case of additive aggregation model, assume each agent has equal weight; that is, k
1
=
k
2
=
0.5; then the optimal policy and the wealth level threshold based on the proposed method in the paper are solved as follows:
(34)
π
*
(
s
0
,
w
)
=
{
a
1
w
∈
(
-
16.4
,
0
]
a
2
w
∈
(
-
∞
,
-
16.4
]
.
The above result shows that, in the wealth level interval (−106.5, −16.4), if Agent1 makes decisions alone, the optimal policy is taking action a
1 in state s
0 but a
2 if Agent2 makes decisions alone. If they make decisions together, then action a
2 is taken.Now we consider how the wealth level thresholdw
_ of group decision-making changes as the weights of agents vary. In detail, k
1 changes from 0.05 to 0.95; meanwhile k
2
=
1
-
k
1 changes from 0.95 to 0.05. The change of the wealth level threshold of group decision-making is shown in Figure 2.Figure 2
The change of the wealth level threshold of group decision-making with different risk aversion parameters of agents.Figure2 shows that the values of the wealth level threshold w
_ of group decision-making are near to the wealth level threshold of Agent2 who is more risk-averse even if k
2 is small and k
1
=
k
2
-
1 is large. This means that the influence of weights on group decision-making is not obvious if the risk aversion parameters of agents are different, while the risk aversion parameters play an important role in this situation.Consider the situation that two agents have similar risk attitude; that is, their risk aversion parameters are similar; for example, their one-switch utility functions areu
1
(
w
)
=
w
-
0.952
w and u
2
(
w
)
=
w
-
0.95
w, respectively. When each agent makes decisions alone their optimal policies and the wealth level thresholds are solved as follows:
(35)
Agent
1
:
π
*
(
s
0
,
w
)
=
{
a
1
w
∈
(
-
106.5,0
]
a
2
w
∈
(
-
∞
,
-
106.5
]
Agent
2
:
π
*
(
s
0
,
w
)
=
{
a
1
w
∈
(
-
71.5,0
]
a
2
w
∈
(
-
∞
,
-
71.5
]
.
Change the values of weights in the same way as Figure 2; the result is shown in Figure 3. Difference from Figures 2 and 3 shows that the values of the wealth level threshold w
_ of group decision-making are near to the wealth level threshold of Agent1 when k
2 is small and k
1
=
k
2
-
1 is large. So if the difference between the risk aversion parameters of agents is not obvious, the weights of agents will play a critical role.Figure 3
The change of the wealth level threshold of group decision-making with similar risk aversion parameters of agents.Finally, we consider group decision-making based on the multiplicative aggregation model and especially focus on the influence of product term of group utility, that is,k
12
u
1
(
w
)
u
2
(
w
), on the group decision-making. Given the same one-switch utility functions of agents in Figure 3 and assuming two sets of k
1 and k
2 value, in detail, k
1
=
0.85, k
2
=
0.1 and k
1
=
0.1, k
2
=
0.85. If the value of k
12 is changed from −0.001 to −0.05, we get two curved lines of wealth level threshold of group decision-making in Figure 4.Figure 4
The change of the wealth level threshold of group decision-making based on the multiplicative aggregation model.The two curved lines gradually approach each other to a point when the absolute value ofk
12 increases from 0.001 to 0.05. If compared with the result of additive aggregation model, we can find that the point is very close to the wealth level threshold of additive aggregation model with k
1
=
k
2
=
0.5. This is because when the absolute value of k
12 is small the absolute values of k
1
u
1
(
w
) and k
2
u
2
(
w
) in group utility are larger than the absolute value of k
12
u
1
(
w
)
u
2
(
w
), so k
12
u
1
(
w
)
u
2
(
w
) has little influence on group decision-making. When the absolute value of k
12 increases enough k
12
u
1
(
w
)
u
2
(
w
) will mainly influence the group decision-making; furthermore, k
12
u
1
(
w
)
u
2
(
w
) has same influence on Agent1 and Agent2; therefore the wealth level threshold of multiplicative aggregation model will approach the threshold of additive aggregation model with k
1= k
2
=
0.5. This implies that the multiplicative aggregation model avoids group decision-making being dominated by the weights of individuals completely.


## 7. Conclusion and Future Works

This paper has put an effort on how to extend a single agent’s risk-sensitive decision-theoretic planning under the MDP framework to the multiagent problem. Based on one-switch utility function that is used to describe agent’s risk-sensitive attitude, the additive and multiplicative aggregation models of group utility have been proposed in this paper. According to the characteristics of group utility, a backward-induction method has been presented to divide the wealth level interval and compute the optimal policy. The paper has also offered numerical examples and discussed how the weights and risk aversion parameters influence the group decision-making. From numerical examples we can observe that, for the additive aggregation model, if the risk aversion parameters of agents are different, the risk aversion parameters will have an obvious influence on the group decision-making, while the weights of agents will play a critical role if the risk aversion parameters are similar. For the multiplicative aggregation model, group decision-making will not be dominated by the weights of individuals completely. The product term of group utility will also influence the group decision-making.In the future we intend to further study multiattribute group decision-making under the MDP framework with one-switch utility function. Based on the work of Tsetlin and Winkler [21], we will further study how to extend our method to the group decision-making problem.



---

*Source: 697895-2014-07-07.xml*