# Estimation Markov Decision Process of Multimodal Trip Chain between Integrated Transportation Hubs in Urban Agglomeration Based on Generalized Cost

Authors: Min Yue; Shu-Hong Ma; Wei Zhou; Xi-Fang Chen
Journal: Journal of Advanced Transportation
 (2022)

---

## Abstract

An efficient multimodal transportation network is crucial to the development of urban agglomeration. Rapid transfer of travelers between integrated transportation hubs is essential for long-distance multimodal trip chains. The Markov Decision Process framework was estimated to explore the optimal transfer trip chain of different income groups, given that the states are considered nodes between hubs, the reward functions are calculated by using the generalized travel cost between states after travelers make action decision, and the actions between states contain bus, subway, taxi, and walk. The optimal trip chain can be obtained through a value iteration algorithm. In the case study, multimodal transfer trip chains of different types between Beijing Capital International Airport and Beijingxi Station in Beijing-Tianjin-Hebei urban agglomeration were constructed by MDP to compare the optimal trip chains of various groups. The findings of this study are as follows: (1) long-distance travelers always prefer to choose the unimodal fewer transfers trip chain between hubs; (2) long-distance travelers are more likely to choose the trip chain with more transfers more than long waiting time; (3) individual income difference affects the generalized cost of trip chains and also influences the optimal choice of trip chain through the MDP framework. One potential application of this study is to complement the research on the transfer behavior of multimodal trip chains in long-distance travel, which can be used to help management alleviate the excessive pressure of passenger flow between integrated hubs due to the sudden colossal travel flow during severe weather days or holidays.


---

## Body

## 1. Introduction

The United States predicted that 70% of the world’s population would live in cities by 2050 [1]. The urbanization processes of China have accelerated since 1978, and the proportion of urban proportion has also continued to rise [2]. As a model of global urbanization, China’s speed of urbanizing is reflected in the rapid gathering of the urban population. People’s aggregation usually tends to form urban agglomeration, composed of the core megacity, and at least 2∼3 large cities provide a close connection between economy and transportation for the coordinated development between regions. In the formation process of urban agglomeration, transportation construction is a crucial investment section. The multimodal comprehensive transportation network connects core cities and surrounding cities to ensure the rapid flow of people, information, and material in urban agglomeration to realize the coordinated development of multiple cities. Compared with urban transportation, more intercity travel modes and integrated hubs have been added to the urban agglomeration multimodal transportation comprehensive network.Travelers across the urban agglomeration always choose multiple travel modes and transfer through more than two integrated hubs from pointi of city A to point j of city B. The comprehensive transportation network provides passengers with more quick and convenient travels, but it also makes passengers’ choices more complex and changeable, and more factors influence their choices. The multimodal transportation network’s characteristics of the urban agglomeration were different from those of urban transportation networks, such as dynamic, uncertain, complex, and confluent. Travelers are mainly affected by distance, time, and cost when choosing the unimodal [3–5]. It is worth noting that they would care about other factors such as transfer convenience, waiting time, walking time, location patterns, and the purpose of travel when they have the multimodal long-distance trip [6–8]. The previous research discussed the trip chains that have mainly emphasized the multimodal transportation in urban transportation [9–14]. A limited number of studies have addressed the long-distance trip chain formed by urban travel and intercity travel through large integrated transport hubs in urban agglomeration.The dynamic multimodal transportation network in urban agglomeration refers to the uncertainty of the traveler’s trip chain. Compared with a single transportation network, the reliability of the multimodal transportation network is poorer. Different natural disasters influence one or more transportation networks, or the abnormal increase of passenger flow caused by holidays or events, the efficiency, and reliability of multimodal transportation networks in urban agglomeration decrease. However, with the recent growth driven by the government and transportation department, there are more forms of intermodal transport in the existing multimodal trip chain supply system, such as air-railway combination, highway-railway combination, and air-highway combination. It is still not enough to solve the increasingly changeable and complex trip chain of multimodal travelers in the urban agglomeration. It also affects improving the efficiency of extremely limited multimodal transportation.Most previous studies explore factors that affect travelers’ behavior and transfer habits in urban multimodal transportation networks, including travel cost, time, walking distance, and accessibility of hubs [12, 15–17]. However, there is a lack of knowledge about intercity or urban agglomeration matters. Understanding the impact of different travel times, travel costs, and transfer times is essential for understanding the traveler’s decision on their trip chains and improving the cooperativeness of multimodal transportation. This paper uses Markov Decision Process (MDP) to estimate travelers’ trip chain decision-making in the multimodal transportation network of urban agglomeration. Markov Chain is a random process with Markov property in discrete exponential set and space state in probability theory and mathematical statistics. Markov Chain suitable for the continuous exponential set is called MDP. MDP framework is a machine learning algorithm based on Markov Chain theory, which can analyze the system control problems related to random probability. The study accounted for the vacant taxi’s long-term profit over the full working period by formulating the MDP; it defined the state as the node vacant taxi located, and action as the link taxi taken out of the node. This MDP problem is solved by value iteration to find an optimal routing policy [18]. Another study about driving behavior also applied the MDP framework to obtain the Personally Revealed Choices through a value iteration algorithm. This study defined states as the surrounding states of the vehicle, and actions as driver’s behavior including acceleration, deceleration, and maintaining speed decisions, and the individual’s reward function are estimated by MNL model. The Personally Revealed Choices that maximize the expected sum of rewards for individual drivers were found by MDP to understand driver’s behavior decision based on the surrounding states of vehicles [19]. It has been shown that the application of the MDP framework can solve complex system control problems consisting of random elements such as states, actions, and transition probability. It is essential to decompose the complex system problem to construct the corresponding MDP framework, for example, to define the states and actions, and to determine the reward function and transition probability.Therefore, this study builds upon previous studies and constructs the multimodal transportation network to extract travelers’ decision-making when trip chains include different modes and cross several integrated hubs. This study benefits from the availability of multisource transportation and economic data, allowing this study to explore the multimodal trip chain travelers’ choice behavior from different income groups’ perspectives. The study aims to find the optimal trip chains for different income groups among the multiple types of trip chains available. By treating the optimal action value as the realization of an optimal policy in an MDP framework, it is possible to define states over as transportation nodes (i.e., bus station, subway station, HSR station, airport, etc.). The framework derives a state’s value for different actions they take (i.e., bus, railway, taxi, and walk), quantified in terms of accumulated discounted rewards. The state transition probability depends on the authors’ previous survey of the passengers’ transfer behavior decisions during their multimodal trip chains under abnormal events [20]. The reward function is determined by the generalized travel cost, which considers both the travel time and the travel cost when they leave from state Si to state Si+1. The generalized travel cost is calculated by weighting the travel time value and the travel cost sum. A more accurate calculation method is used to obtain the travel time of the trip chain depending on whether or not to change the station and the difference in walking distance during the interchange. The weights of the travel time value and the travel cost are mediated according to different levels of income groups in an attempt to analyze the impact of changes in the reward function of varying income groups on the value of nodes through the MDP framework, to understand the advantages and disadvantages of different trip chains, and to choose the suitable transfer trip chain for different income groups. MDP is a reasonable approach for providing a framework for modeling decisions to multimodal trip chain travelers between two integrated transportation hubs of urban agglomeration. The methods applied in this paper have the potential to form a foundation for multimodal trip chain transfer passengers evacuation in different scenarios (i.e., holiday peak traffic, unexpected events traffic aggregation) and can provide suggestions for the resolution of passenger aggregation brought about by unusual events on the operation of multimodal transportation systems in urban agglomerations.The outline of this paper is as follows. Section2 provides a review of the literature. Section 3 introduces the problem and definitions of the multimodal trip chain model based on the MDP framework. Section 4 presents the computational experiments of the Beijing-Tianjin-Hebei urban agglomeration. Finally, Section 5 concludes the study and discusses potential directions for future work.


## 2. Literature Review

Previous works provided early focus on the trip chain. Primerano defined the trip chain as linking secondary activities to a primary activity through travel that is made when an individual leaves home to when they return home [21–23]. The trip chain in early studies was usually defined as home-to-home loops. Strathman and Dueker listed seven types of trip chains and divided them into simple and complex trip chains [24]. Therefore, the trip chain with two trips (such as from home to work and work to home) is called a simple trip chain, and a trip chain with more than two trips (add trips from work to a restaurant or shopping mall) is called a complex trip chain [25]. In some studies, stop frequency was utilized to measure the complexity of the trip chain [26].Most studies of the relationship between travel behavior and trip chains found that trip chaining precedes mode choice, and travelers’ included activity locations in trip chains are generally referred to as influence mode choice of travel [9, 10, 27]. This pattern is often reflected in the commuter’s trip chain. For example, commuters often add shopping, picking up children, or dinner after work and then make their travel mode choices after these activities have been determined. Of course, the choice of mode in the trip chain is influenced by its activities and by travelers’ gender, age, household income level, work or nonwork, and numbers of children [22, 26, 28–30].Research on the complex trip chain has mainly focused on individuals’ travel behavior, especially comparing private cars and public transport [10]. A prominent finding illustrates the following pattern: the more complex the trip chain is, the more the travelers depend on private car use, or the less likely travelers to choose public transport [31]. Findings in these research are elementary to understand that travelers tend to choose the greater flexibility and convenience mode (such as auto for long-distance, walk/bike for a short distance) when they have trips with multiple destinations [32, 33]. An exciting finding also shows that the higher the minimum density at destinations, the lower the odds of a complex trip chain and auto mode choice [34]. In Daisy’s study, similar findings proposed that the complex trip chain could be a significant barrier to shifting from drive-along to public transport [35].For the urban multimodal travelers, previous studies have analyzed the transfer perception and traveler’s intention of multimodal trip chain, especially the traveler’s attention to the transfer times, waiting time, and walking time. Some research suggested that moving auto users to the public transportation network requires reducing barriers to transfers, such as long initial and final walking time and long waiting time [36, 37]. Other research has investigated the perception of transfers and analyzed the travelers’ perceptions of transfers from different perspectives [16, 38, 39]. Most show that waiting time is more penalized than walking time, and travelers perceived out-of-vehicle time as more demanding than in-vehicle times [15, 40].The simple and complex trip chain mentioned above is the chain that starts or ends with a home. Another type of trip chain starts fromi in city A to j in city B, this kind of trip chain always concludes urban and intercity traffic, and it is also closely related to integrated transportation hubs. It is called multimodal transportation or integrated transportation in previous studies; the Madrid Regional Transport Authority in 1985 defined multimodal interchange as “An area whose purpose is to minimize the inevitable sensation of having to change from one mode of transportation to another and efficiently using the inevitable waiting time.” [41] The multimodal transportation hub plays a vital role in society, and it can benefit the government and different stakeholders [42]. It is regarded as a symbol of “urban identity” and “urban mobility.” A critical linkage in the multimodal travel chain proposed that the more smooth the transportation hub is, the more probability travelers choose multimodal travel [43, 44]. Their research found that the transfer penalty is strongly related to the internal corridor structure of the hub.Studies on the long-distance trip chain are not as detailed as urban complex trip chains. Most of them focused on the factors that influence travelers’ choices, such as travel time, travel cost, and transportation service. In the research of multimodal transportation behavior, scholars tended to analyze the competition relationship and coupling relationship of several modes (i.e., HSR and airway, train, and bus) [45, 46]. A MIMIC model was constructed to explore the causal relationship between the socioeconomic attributes of passengers and transfer characteristics (such as transfer comfort, transfer convenience, and transfer economy) in Ma’s research [20]. It is found that gender, occupation, and departure time greatly influence the choice of travel chain.A research by D’Este proposed a model using Markov chains to model trip chaining behavior to extend the utility of the traditional four-step travel demand models [47]. A sequence of system states was calculated to represent an individual’s likelihood of participating in an activity at a particular segment of a trip chain. Other studies used Markov chain or MDP in transportation problems to predict the travel time or distribution from simulation data. Past studies have defined the trip chain and divided the trip chain from home into the simple trip chain and complex trip chain. However, there is no definition of multimodal trip chain in urban agglomeration or intercity. Most of the focus and resources are pooled towards exploring the factors that influence travelers’ mode choice of long-distance.


## 3. Problem Statement and Definitions

### 3.1. Multimodal Trip Chain in Urban Agglomeration

For the multimodal transportation network in the urban agglomeration, which is generally composed of railway, highway, aviation, and urban transport, the efficient, coordinated operation is inseparable from the connectivity provided by various transportation hubs. The combination of urban public transportation and intercity transportation is the main object of research on multimodal travel of urban agglomeration. Compared with multimodal travel in urban, the similarity is that the transfer time between different modes needs to be considered. The difference is that the waiting time between urban public transport and intercity transport needs to be considered in multimodal travel of urban agglomeration, especially for the transfer that happened in the railway hubs, highway passenger station, or airway hubs. As a result, the multimodal travel chain in urban agglomeration is more complex than urban.


### 3.2. MDP Framework in Multimodal Trip Chain

#### 3.2.1. States

In a sequential process, if the stateSi+1 at time i + 1 only depends on the state Si at time i and has nothing to do with any state before time i, then the state Si at time i is considered a Markov property. To the travelers in a multimodal transportation network, the state at node i + 1 only depends on the travel decision made at node i and has nothing to do with the trip before node i. Therefore, it can be seen as a multimodal trip chain with Markov property. The whole process of the traveler’s trip chain from the origin to the destination can be regarded as a Markov process. The origin, destination, and nodes between the original destination can be considered the state Si.


#### 3.2.2. Rewards (Penalties)

Markov Process only considers the transition probability between states, but it cannot integrate the rewards associated with the state transition. Markov Reward Process (MRP) is derived to better solve the dynamic process decision. The MRP is a tuple consisting of <S, P, R, γ>, and R represents the rewards (penalties) function, defined as the reward expectation that the state Si will achieve at the next state Si+1.In this study, the generalized travel cost of leaving statei to the next state i + 1 is considered as the reward function. The generalized travel cost is obtained by weighting travel time and travel cost. In the multimodal trip chain, since the travel time of choosing different trip chains is not only related to the travel time of each segment of the chain but also related to the transfer time and waiting time between two distinct segments, the destination of rewards (penalties) consisted of the transfer time (RSac), waiting time (RSaw), and transport time (RSat) from Si to Si+1The waiting time can be determined according to the departure frequency of the public traffic line. De Cea and Fernandez [48] give formula (1) of waiting time as a deviation factor related to the time reliability of a vehicle operating in a traffic network, and fk represents the departure frequency.(1)RSaw=1/fk2+σ22/fk.The travel time from stateSi to state Si+1 is one of four situations (see Table 1).Table 1
Four types of travel time calculation formula in the multimodal trip chain.
StateSi to state Si+1Formulation1There is no modal transfer, or there is a modal transfer without waiting or transferring; it only has transport timeRSa=RSat2There is a modal transfer with outbound, transport time, and transfer time compose travel timeRSa=RSac+RSat3There is a line transfer without outbound (e.g., in-station transfer), transport time and waiting time compose travel timeRSa=RSaw+RSat4There is a modal transfer, transport time, transfer time, and waiting time compose travel timeRSa=RSac+RSaw+RSat


#### 3.2.3. Harvest

In the MRP, the sum of all rewards with attenuation starting from stateS0 until the end state is called harvest (G), the mathematical expression of harvest is as formula (2), γ is an attenuation factor, to make the model mathematically tractable, and the discount factor is restricted to 0 < γ < 1. Harvest can reflect every state’s importance in the state sequence. However, there may have been a few sequences.(2)Gi=Ri+1+γRi+2+…=∑k=0∞γkRi+k+1.In the multimodal transportation network in the urban agglomeration, there may be several trip chains from original to destination. The total reward or punishment of each trip chain can be calculated according to Table1.


### 3.3. States’ Value and the Action-Value Function

#### 3.3.1. States’ Value

Harvest can reflect every state’s importance in the state sequence. However, there may have a few sequences, and the same stateSi may appear in several different state sequences. The value function realizes the mapping from the state to the value. To solve the inconvenience of harvesting in describing the importance of a state in different state sequences, the concept of “value” is introduced more accurately. Value is the expectation of the state’s harvest in the Markov reward process, and it can be calculated according to formulas (3) and (4). Formula (4) shows that the value of a state is composed of the rewards leaving the state and the value of the subsequent states according to the probability distribution sum in a certain attenuation ratio.(3)vi=ΕGi|Si=i,(4)vi=Ri+γ∑i+1∈SPSiSi+1vi+1.When selecting individual behavior is involved in the Markov Reward Process, it is necessary to introduce the Markov Decision Process. MDP is a tuple composed of <S, A, P, R, γ>, in which S is a finite set of states. A is a finite set of behaviors, including all the choices an individual may make in the decision. P is the set’s behavior-based state transition probability matrix, and R is the state and behavior-based reward function.


#### 3.3.2. The Action-Value Function

In MDP, an individual has the right to select an action from the behavior set according to his understanding of the current state, while the dynamics of the environment determine the subsequent state of an individual after selecting a specific action. The policy is represented by the letterπ, defined as an individual choosing an action from a set of behaviors in a given state.(5)πa|i=PAi=a|Si=i.For the same MDP, different strategies will produce other Markov processes or MRP, and then there will be additional state value functions. The previously defined value functions should be improved to formula (6), and the Policy-based value function in MDP represents the expected harvest following current policy π starting with the state Si.(6)vπi=EGi|Si=i.The value of different actions selected by individuals in the same state is called the action-value function based on policyπ, representing the expected harvest by performing a specific action a (a ∈ A) on the current state Si when following the policy π. The value of different actions is represented by qπi,a.(7)qπi,a=EGi|Si=i,At=a.a2+b2According to Bellman (1954), we can obtain two Bellman equations formulas (8) and (9). It can be seen that the action is a bridge between two adjacent states in MDP.(8)vπi=ERi+1+γvπi+1|Si=i,(9)qπi,a=ERi+1+γqπi+1,a′|Sn=i,Ai=a,Ai+1=a′.The value of an action is related to both the state value before this action and the subsequent state value after this action. The formula can be expressed as(10)qπi,a=RSia+γ∑Si+1∈SPSiSi+1a∑a′∈Aπa′|Si+1qπi+1,a′.Similarly, the value of stateSi can be expressed by the value of all possible actions under this state:(11)vπi=∑a∈Aπa|sqπi,a.In searching for the optimal policy, a better policy is usually determined by comparing two policies, which need to be defined. The optimal state value function (v∗s) is the optimal state value produced under all policies; the optimal action-value function (q∗s,a) is the optimal action-value under all policies.


## 3.1. Multimodal Trip Chain in Urban Agglomeration

For the multimodal transportation network in the urban agglomeration, which is generally composed of railway, highway, aviation, and urban transport, the efficient, coordinated operation is inseparable from the connectivity provided by various transportation hubs. The combination of urban public transportation and intercity transportation is the main object of research on multimodal travel of urban agglomeration. Compared with multimodal travel in urban, the similarity is that the transfer time between different modes needs to be considered. The difference is that the waiting time between urban public transport and intercity transport needs to be considered in multimodal travel of urban agglomeration, especially for the transfer that happened in the railway hubs, highway passenger station, or airway hubs. As a result, the multimodal travel chain in urban agglomeration is more complex than urban.


## 3.2. MDP Framework in Multimodal Trip Chain

### 3.2.1. States

In a sequential process, if the stateSi+1 at time i + 1 only depends on the state Si at time i and has nothing to do with any state before time i, then the state Si at time i is considered a Markov property. To the travelers in a multimodal transportation network, the state at node i + 1 only depends on the travel decision made at node i and has nothing to do with the trip before node i. Therefore, it can be seen as a multimodal trip chain with Markov property. The whole process of the traveler’s trip chain from the origin to the destination can be regarded as a Markov process. The origin, destination, and nodes between the original destination can be considered the state Si.


### 3.2.2. Rewards (Penalties)

Markov Process only considers the transition probability between states, but it cannot integrate the rewards associated with the state transition. Markov Reward Process (MRP) is derived to better solve the dynamic process decision. The MRP is a tuple consisting of <S, P, R, γ>, and R represents the rewards (penalties) function, defined as the reward expectation that the state Si will achieve at the next state Si+1.In this study, the generalized travel cost of leaving statei to the next state i + 1 is considered as the reward function. The generalized travel cost is obtained by weighting travel time and travel cost. In the multimodal trip chain, since the travel time of choosing different trip chains is not only related to the travel time of each segment of the chain but also related to the transfer time and waiting time between two distinct segments, the destination of rewards (penalties) consisted of the transfer time (RSac), waiting time (RSaw), and transport time (RSat) from Si to Si+1The waiting time can be determined according to the departure frequency of the public traffic line. De Cea and Fernandez [48] give formula (1) of waiting time as a deviation factor related to the time reliability of a vehicle operating in a traffic network, and fk represents the departure frequency.(1)RSaw=1/fk2+σ22/fk.The travel time from stateSi to state Si+1 is one of four situations (see Table 1).Table 1
Four types of travel time calculation formula in the multimodal trip chain.
StateSi to state Si+1Formulation1There is no modal transfer, or there is a modal transfer without waiting or transferring; it only has transport timeRSa=RSat2There is a modal transfer with outbound, transport time, and transfer time compose travel timeRSa=RSac+RSat3There is a line transfer without outbound (e.g., in-station transfer), transport time and waiting time compose travel timeRSa=RSaw+RSat4There is a modal transfer, transport time, transfer time, and waiting time compose travel timeRSa=RSac+RSaw+RSat


### 3.2.3. Harvest

In the MRP, the sum of all rewards with attenuation starting from stateS0 until the end state is called harvest (G), the mathematical expression of harvest is as formula (2), γ is an attenuation factor, to make the model mathematically tractable, and the discount factor is restricted to 0 < γ < 1. Harvest can reflect every state’s importance in the state sequence. However, there may have been a few sequences.(2)Gi=Ri+1+γRi+2+…=∑k=0∞γkRi+k+1.In the multimodal transportation network in the urban agglomeration, there may be several trip chains from original to destination. The total reward or punishment of each trip chain can be calculated according to Table1.


## 3.2.1. States

In a sequential process, if the stateSi+1 at time i + 1 only depends on the state Si at time i and has nothing to do with any state before time i, then the state Si at time i is considered a Markov property. To the travelers in a multimodal transportation network, the state at node i + 1 only depends on the travel decision made at node i and has nothing to do with the trip before node i. Therefore, it can be seen as a multimodal trip chain with Markov property. The whole process of the traveler’s trip chain from the origin to the destination can be regarded as a Markov process. The origin, destination, and nodes between the original destination can be considered the state Si.


## 3.2.2. Rewards (Penalties)

Markov Process only considers the transition probability between states, but it cannot integrate the rewards associated with the state transition. Markov Reward Process (MRP) is derived to better solve the dynamic process decision. The MRP is a tuple consisting of <S, P, R, γ>, and R represents the rewards (penalties) function, defined as the reward expectation that the state Si will achieve at the next state Si+1.In this study, the generalized travel cost of leaving statei to the next state i + 1 is considered as the reward function. The generalized travel cost is obtained by weighting travel time and travel cost. In the multimodal trip chain, since the travel time of choosing different trip chains is not only related to the travel time of each segment of the chain but also related to the transfer time and waiting time between two distinct segments, the destination of rewards (penalties) consisted of the transfer time (RSac), waiting time (RSaw), and transport time (RSat) from Si to Si+1The waiting time can be determined according to the departure frequency of the public traffic line. De Cea and Fernandez [48] give formula (1) of waiting time as a deviation factor related to the time reliability of a vehicle operating in a traffic network, and fk represents the departure frequency.(1)RSaw=1/fk2+σ22/fk.The travel time from stateSi to state Si+1 is one of four situations (see Table 1).Table 1
Four types of travel time calculation formula in the multimodal trip chain.
StateSi to state Si+1Formulation1There is no modal transfer, or there is a modal transfer without waiting or transferring; it only has transport timeRSa=RSat2There is a modal transfer with outbound, transport time, and transfer time compose travel timeRSa=RSac+RSat3There is a line transfer without outbound (e.g., in-station transfer), transport time and waiting time compose travel timeRSa=RSaw+RSat4There is a modal transfer, transport time, transfer time, and waiting time compose travel timeRSa=RSac+RSaw+RSat


## 3.2.3. Harvest

In the MRP, the sum of all rewards with attenuation starting from stateS0 until the end state is called harvest (G), the mathematical expression of harvest is as formula (2), γ is an attenuation factor, to make the model mathematically tractable, and the discount factor is restricted to 0 < γ < 1. Harvest can reflect every state’s importance in the state sequence. However, there may have been a few sequences.(2)Gi=Ri+1+γRi+2+…=∑k=0∞γkRi+k+1.In the multimodal transportation network in the urban agglomeration, there may be several trip chains from original to destination. The total reward or punishment of each trip chain can be calculated according to Table1.


## 3.3. States’ Value and the Action-Value Function

### 3.3.1. States’ Value

Harvest can reflect every state’s importance in the state sequence. However, there may have a few sequences, and the same stateSi may appear in several different state sequences. The value function realizes the mapping from the state to the value. To solve the inconvenience of harvesting in describing the importance of a state in different state sequences, the concept of “value” is introduced more accurately. Value is the expectation of the state’s harvest in the Markov reward process, and it can be calculated according to formulas (3) and (4). Formula (4) shows that the value of a state is composed of the rewards leaving the state and the value of the subsequent states according to the probability distribution sum in a certain attenuation ratio.(3)vi=ΕGi|Si=i,(4)vi=Ri+γ∑i+1∈SPSiSi+1vi+1.When selecting individual behavior is involved in the Markov Reward Process, it is necessary to introduce the Markov Decision Process. MDP is a tuple composed of <S, A, P, R, γ>, in which S is a finite set of states. A is a finite set of behaviors, including all the choices an individual may make in the decision. P is the set’s behavior-based state transition probability matrix, and R is the state and behavior-based reward function.


### 3.3.2. The Action-Value Function

In MDP, an individual has the right to select an action from the behavior set according to his understanding of the current state, while the dynamics of the environment determine the subsequent state of an individual after selecting a specific action. The policy is represented by the letterπ, defined as an individual choosing an action from a set of behaviors in a given state.(5)πa|i=PAi=a|Si=i.For the same MDP, different strategies will produce other Markov processes or MRP, and then there will be additional state value functions. The previously defined value functions should be improved to formula (6), and the Policy-based value function in MDP represents the expected harvest following current policy π starting with the state Si.(6)vπi=EGi|Si=i.The value of different actions selected by individuals in the same state is called the action-value function based on policyπ, representing the expected harvest by performing a specific action a (a ∈ A) on the current state Si when following the policy π. The value of different actions is represented by qπi,a.(7)qπi,a=EGi|Si=i,At=a.a2+b2According to Bellman (1954), we can obtain two Bellman equations formulas (8) and (9). It can be seen that the action is a bridge between two adjacent states in MDP.(8)vπi=ERi+1+γvπi+1|Si=i,(9)qπi,a=ERi+1+γqπi+1,a′|Sn=i,Ai=a,Ai+1=a′.The value of an action is related to both the state value before this action and the subsequent state value after this action. The formula can be expressed as(10)qπi,a=RSia+γ∑Si+1∈SPSiSi+1a∑a′∈Aπa′|Si+1qπi+1,a′.Similarly, the value of stateSi can be expressed by the value of all possible actions under this state:(11)vπi=∑a∈Aπa|sqπi,a.In searching for the optimal policy, a better policy is usually determined by comparing two policies, which need to be defined. The optimal state value function (v∗s) is the optimal state value produced under all policies; the optimal action-value function (q∗s,a) is the optimal action-value under all policies.


## 3.3.1. States’ Value

Harvest can reflect every state’s importance in the state sequence. However, there may have a few sequences, and the same stateSi may appear in several different state sequences. The value function realizes the mapping from the state to the value. To solve the inconvenience of harvesting in describing the importance of a state in different state sequences, the concept of “value” is introduced more accurately. Value is the expectation of the state’s harvest in the Markov reward process, and it can be calculated according to formulas (3) and (4). Formula (4) shows that the value of a state is composed of the rewards leaving the state and the value of the subsequent states according to the probability distribution sum in a certain attenuation ratio.(3)vi=ΕGi|Si=i,(4)vi=Ri+γ∑i+1∈SPSiSi+1vi+1.When selecting individual behavior is involved in the Markov Reward Process, it is necessary to introduce the Markov Decision Process. MDP is a tuple composed of <S, A, P, R, γ>, in which S is a finite set of states. A is a finite set of behaviors, including all the choices an individual may make in the decision. P is the set’s behavior-based state transition probability matrix, and R is the state and behavior-based reward function.


## 3.3.2. The Action-Value Function

In MDP, an individual has the right to select an action from the behavior set according to his understanding of the current state, while the dynamics of the environment determine the subsequent state of an individual after selecting a specific action. The policy is represented by the letterπ, defined as an individual choosing an action from a set of behaviors in a given state.(5)πa|i=PAi=a|Si=i.For the same MDP, different strategies will produce other Markov processes or MRP, and then there will be additional state value functions. The previously defined value functions should be improved to formula (6), and the Policy-based value function in MDP represents the expected harvest following current policy π starting with the state Si.(6)vπi=EGi|Si=i.The value of different actions selected by individuals in the same state is called the action-value function based on policyπ, representing the expected harvest by performing a specific action a (a ∈ A) on the current state Si when following the policy π. The value of different actions is represented by qπi,a.(7)qπi,a=EGi|Si=i,At=a.a2+b2According to Bellman (1954), we can obtain two Bellman equations formulas (8) and (9). It can be seen that the action is a bridge between two adjacent states in MDP.(8)vπi=ERi+1+γvπi+1|Si=i,(9)qπi,a=ERi+1+γqπi+1,a′|Sn=i,Ai=a,Ai+1=a′.The value of an action is related to both the state value before this action and the subsequent state value after this action. The formula can be expressed as(10)qπi,a=RSia+γ∑Si+1∈SPSiSi+1a∑a′∈Aπa′|Si+1qπi+1,a′.Similarly, the value of stateSi can be expressed by the value of all possible actions under this state:(11)vπi=∑a∈Aπa|sqπi,a.In searching for the optimal policy, a better policy is usually determined by comparing two policies, which need to be defined. The optimal state value function (v∗s) is the optimal state value produced under all policies; the optimal action-value function (q∗s,a) is the optimal action-value under all policies.


## 4. Analysis

### 4.1. The Multimodal Network and Transportation Data

Beijing-Tianjin-Hebei urban agglomeration includes a core megacity (Beijing) and two large cities (Tianjin, Shijiazhuang) with a permanent resident population of 110 million, an essential component in this specified area. It is one of the crucial urban agglomerations in China. It plays a vital role in defusing Beijing’s noncapital functions, strengthening the linkage between Beijing and Tianjin, and enhancing the comprehensive ability of Hebei province. The 2022 Winter Olympics will be held in Beijing and Zhangjiakou in Hebei province. It will bring a surge in traffic flow, resulting in more pressure and bringing risk to the transportation network of the Beijing-Tianjin-Hebei urban agglomeration.Take the transportation corridor from Beijing Capital International Airport to Beijingxi Railway Station as an example; the more multimodal trip chain travelers through the above two hubs, the more pressure there is in the corridor between the two hubs, and the more likely to generate passengers gathering in the two hubs. The traditional way to evacuate the passengers in transportation hubs always ignores the traveler’s final destination. It is not helpful to the transportation efficiency of urban agglomeration. Considering the travel characteristics of passengers in the trip chain of urban agglomerations, travel time, travel cost, and transfer times are taken as the critical factors for their transfer between hubs. The study on evacuation schemes under abnormal passenger flow is combined with the different needs of different travel groups, which is more effective in solving the blocked operation of interhub passages in urban agglomerations. In this case, the multimodal transportation network between Beijing Capital International Airport (BCIA) and Beijingxi Railway Station (BJXRS) has been constructed in Figure1.Figure 1
The multimodal trip chains between BCIA and BJXRS.Trip chains shown in Figure1 refer to the recommended scheme of AutoNavi Map in different periods, including the main modes of transportation, such as subway, bus, taxi (including online car-hailing), and airport bus. There are nine trip chains according to Figure 1 from BCIA to BJXRS.As shown in Table2, the nine trip chains were divided into four categories in the study according to the transfer times and mode of transfer. Considering that long-distance multimodal trip chain passengers with luggage in urban agglomeration always consider travel time, travel cost, and comfort, multimodal transfers may bring more walking distance and waiting time. Therefore, the type of travel mode and the number of transfers are used to determine the category of the trip chain. The trip chain with more than one transfer will be called “more transfers,” and the trip chain with no transfer or only one transfer will be called “fewer transfers.” Combined with whether passengers switch travel modes in the trip chain, the following four categories of trip chains (unimodal more transfer; multimodal more transfers; unimodal fewer transfers; unimodal more transfers) are formed.Table 2
Four types of trip chains between BCIA and BJXRS.
Trip chainRouteModesTransfer timesTypeNo.10---2 (S) ---3---4---713Unimodal more transfersNo.20---2 (B) ---3---4---723Multimodal more transfersNo.30---1 (B) ---5---722Multimodal more transfersNo.40---1 (S) ---721Multimodal fewer transfersNo.50---6 (B) ---721Multimodal fewer transfersNo.60---7 (BUS)10Unimodal fewer transfersNo.70---7 (TAXI)10Unimodal fewer transfersNo.80---2 (S) ---721Multimodal fewer transfersNo.90---2 (B) ---711Unimodal fewer transfersTable3 shows each segment’s travel information of trip chain No. 1 to No. 9, rows 3∼11 in Table 3. 1 represents nine trip chains, and the vertical I, II, III, and IV represent the first, second, third, and fourth trip segments of this trip chain. For example, line 2, row 3, “0 ⟶ 2 (B)” indicates that the first trip segment of the trip chain No. 1 is from the origin node (BICA) to the node “2” (Dongzhimen), and “(S)” represents that travelers choose subway in this trip segment. Line 3, row 3 indicates the type of travel time calculation for this trip segment, referring to Table 1, which calculates the travel time contained with waiting and walking time. Line 4, row 3 indicates the travel time calculation result for this trip segment, and line 5, row 3 shows the travel cost. The total travel time and cost of each multimodal trip chain are summed to obtain the results in rows 14∼15. For example, line 3, row 14 indicates that the total travel time of trip chain No. 1 is 1.46 hours, and line 3, row 15 indicates that the total travel cost of trip chain No. 1 is CNY30. The transportation information was collected from the official websites of AutoNavi Map, Capital Airport Group Corporation, Beijing Metro, Beijing Public Transportation Company.Table 3
Travel information of every segment of each trip chains.
Trip chainNo. 1No. 2No. 3No. 4No. 5No. 6No. 7No. 8No. 9IRoute0 ⟶ 2(S)0 ⟶ 2(B)0 ⟶ 1(B)0 ⟶ 1(B)0 ⟶ 60 ⟶ 70 ⟶ 70 ⟶ 2(S)0 ⟶ 2(B)Type(1)(3)(3)(3)(3)(3)(3)(3)(3)(3)Time/hr0.771.152.452.451.481.921.220.771.15Cost/CNY2525252530301002525IIRoute2(S) ⟶ 32(B) ⟶ 31(B) ⟶ 51(B) ⟶ 76 ⟶ 72(S) ⟶ 72(B) ⟶ 7Type(1)(4)(4)(4)(4)(4)(4)(4)Time/hr0.410.410.830.710.460.790.79Cost/CNY2234444IIIRoute3 ⟶ 43 ⟶ 45 ⟶ 7Type(1)(4)(4)(2)Time/hr0.070.070.15Cost/CNY110IVRoute4 ⟶ 74 ⟶ 7Type(1)(4)(4)Time/hr0.210.21Cost/CNY22TotalTime/hr1.461.843.433.161.941.921.221.561.94Cost/CNY3030282934301002929(1): rewards function type according to Table1.


### 4.2. Comparison of Harvest

The multimodal trip chain of MDP conducted in this study regards the generalized cost fromSi to Si+1 as rewards. The generalized cost better describes the travel cost of different income groups who care about both travel time and travel cost, and it can be obtained by weighting the sum of the travel time after value calculation and cost in the same segment of the trip, and the weight of time and cost can be adjusted to represent different income groups. Travel time is usually converted into a time cost dimensionalized with expense by drawing into a time value. Formula (12) shows the way of valuing travel time.(12)T=1.5Y22×8×12t.T-valued time (CNY/h); t—travel time (h); Y—annual income of traveler (CNY).In formula (12), “Y” is the annual income of urban residents, “22” is the average number of working days per month, “8” is the average number of working hours per day, multiplied by 12 to get the result that is the number of working hours per year for urban residents, and “t” represents the travel time in travelers’ each trip segments of their trip chains. According to the survey data of the National Bureau of Statistics of China in 2020, the per capita disposable income of Chinese residents in 2020 is 32,189 yuan, which is divided into five equal income groups. The low-income, low-middle-income, middle-income, upper-middle-income, and high-income groups’ per capita disposable income are 7869 yuan, 16,443 yuan, 26249 yuan, 41,172 yuan, and 80,294 yuan, respectively. Formula (13) was adopted to calculate the generalized travel cost of each income group.(13)Ri=μ1Ti+μ2ci,c—travel cost (yuan); μ1 + μ2 = 1; 0 ≤ μ1 ≤ 1; 0 ≤ μ2 ≤ 1.In formula (13), “R(i)” is the reward function of each trip segment of the trip chain; μ1 and μ2 are the weighting factors for the valued time and travel cost, respectively; “Ti” is the valued time of the trip segment of the trip chain, and “ci” is the travel cost of the trip segment of trip chain.It can be seen that the time value coefficients of different income groups are different, and it is mainly related to the average annual income of other groups. Travelers are divided into seven groups in this study to explore the various groups’ trip chain choices deeply. In addition to the five groups of varying incomes (see Table4), groups a and c were set up to represent those who only consider travel costs and travel time. For the group that only cares about economic expenditure, the generalized cost of trip chain formula consists of travel cost only, and for the group who cares two, the procedure is determined by adjusting the weight of valued time and cost (it is shown in Table 4 rows 5∼6; lines 4∼8).Table 4
The time value coefficient of different income groups in China.
The national per capitaab1b2b3b4b5cCostsLow- incomeLow-middle- incomeMiddle-incomeUpper- middle- incomeHigh- incomeTimeIncome (yuan)32189—786916443262494117280294—Coefficient22.86—5.5911.6818.6429.2457.0322.86The generalized costμ100.10.30.50.70.91.0μ21.00.90.70.50.30.10Formulac0.1 T + 0.9c0.3 T + 0.7c0.5 T + 0.5c0.7 T + 0.3c0.9 T + 0.1cTAccording to formula (2) in Section 3.2, the harvest of different trip chains can be calculated, and the results are shown in Table 5. Results of harvest shown in Table 5 illustrate every trip chain’s total rewards in the different traveler groups. Comparison of calculation results on harvest shows that trip chain No. 3 is the least cost choice for people who only care about cost when transferring from BCIA to BJXRS, and trip chain No. 9 is the least for both low-income and low-middle-income groups. Trip chain No. 1 is the minimum generalized cost to the middle-income and upper-middle-income groups. Trip chain No. 7 is the minimum generalized cost to the high-income group and passengers who only care about travel time.Table 5
The harvest of different trip chains of seven groups of travelers.
Trip chainNo. 1No. 2No. 3No. 4No. 5No. 6No. 7No. 8No. 90–2(S)-3-4-70–2(B)-3-4-70–1(B)-5-70–1(S)-70–6(B)-70–7(B)0–7(T)0–2(S)-70–2(B)-7TypeUni-more(1)Multi -more(2)Multi-moreMult-fewer(3)Multi-fewerUni-fewer(4)Uni-fewerMulti-fewerUni-fewera3030282934301002929b127.8228.0327.1227.8731.6828.0790.6826.9721.78b232.1233.4537.2237.1737.4033.7394.2731.5727.50b328.6132.1545.9743.9535.0832.8961.3729.0429.58b438.8846.6678.6173.3849.9148.3054.9740.6346.61b577.9497.44178.85165.09102.97101.5572.6282.97101.87c87.6110.4205.8189.6116.4115.273.293.6116.4(1): unimodal more transfer; (2): multimodal more transfers; (3): unimodal fewer transfers; (4): unimodal more transfers.Figure2 compares the generalized travel cost of each trip chain in different groups more intuitively. Trip chain No. 7, which belongs to unimodal fewer transfer type, is the maximum generalized cost chain in groups a, b1, b2, b3. Still, it changes to the better or best choice in groups b4, b5, and group c. Travelers who take taxis from BCIA to BJXRS may get the least generalized cost when they have more annual income. In contrast to this phenomenon, trip chains No. 3 and No. 4 show the less generalized cost in group a, b1, b2, b3, but they transfer to be the most generalized cost in group b5 and group c.Figure 2
Comparison of different trip chains of seven groups of travelers.From line 2 in Table5, it is readily observed that trip chains Nos. 3, 4, 8, and 9 belong to the lower generalized cost one, but none of them is the least generalized choice in group b1∼b5 and group c except trip chain 9. Differences in their types might cause this result. Based on Table 2, trip chains 3, 4, and 8 are the multimodal trip chain, but trip chain 9 is the unimodal trip chain. It can be explained that travelers prefer to choose the trip chain that concluded with fewer modes in their trip.An exciting finding is that trip chain 1 (type (1)) is the least generalized cost in theb3 and b4 groups. However, those trip chains with fewer transfers (type (3) and (4)) are not the least. It can be found that there is a long period of waiting time in the first segment in trip chains No. 3 and No. 4 by checking the information in Table 3. The average waiting time of that segment is 1.5 hours, and the time value coefficient is higher between the middle and upper-middle-income groups. It means that more attention should be taken to the out-of-vehicle time in the trip chains of urban agglomeration than before.Figure3 shows the comparison of the harvest for four pairs of trip chains considering differences in traveler income. Figure 3(a) shows the comparison between unimodal and multimodal trip chains with more transfers; Figure 3(b) shows the comparison between the two multimodal trip chains with more and fewer transfers, respectively; Figure 3(c) shows the comparison of harvest for the two unimodal trip chains without transfer; Figure 3(d) shows the comparison between unimodal and multimodal trip chains with fewer transfers. Looking first at part a of Figure 3, the generalized cost of unimodal trip chain (No. 1) is gradually lower than the multimodal one (No. 2) for middle, upper-middle, high-income groups, and travelers who do not care about travel costs. Therefore, this change becomes more and more evident as income increases. Comparatively similar is the comparison between trip chains No. 3 and No. 4. Like Figure 3(b), the fewer transfers type of trip chain has lower generalized cost than the more transfers type when the income increases. However, the difference is that the change in the gap between the two trip chains is relatively tiny. The apparent discrepancy between the same type of trip chain appears on trip chains No. 6 and No. 7. As shown in Figure 3(c), the generalized cost of unimodal fewer transfers type of trip chain presents a sharp difference with the change of income of different groups. The generalized cost of taking an airport bus is less than a taxi to the middle and below middle-income groups and those who do not care about travel time.Figure 3
Comparison of the harvest for similar types of trip chains.
(a)(b)(c)(d)On the contrary, taking a taxi is the lower generalized cost trip chain for travelers with more annual income or who do not care about travel costs. The equivalence point of trip chains No. 6 and No. 7 shows the gap between upper-middle and high-income groups. Over 60% of travelers may choose the airport bus to have the unimodal fewer transfer trip between two hubs in urban agglomeration. Similar but critically different to the previous comparison that appears on trip chains No. 8 and No. 9 (Figure3(d)), the exciting turning point occurs for the middle-income group. The only difference is whether travelers choose the subway or not in the first segment, and it can be seen that the low-middle-income and low-income groups need to select the traditional bus to ensure the lower generalized cost. Thus, travelers who do not care about travel costs or have more income always choose the subway for the first trip to provide the lower generalized cost. It can also be seen that the growth of income has a positive association with the subway mode preference.


### 4.3. States’ Value and the Action-Value Function

Table6 presents the calculation results of states’ value of MDP by Python. As shown in Figure 1, eight nodes occur for the multimodal transportation network, while travelers need to travel from S0 to S7. This study depends on data from the survey conducted by the authors in May 2020, which focuses on the transfer decisions of multimodal trip chain passengers between integrated hubs of Beijing-Tianjin-Hebei urban agglomeration. The results of this survey show that the transition probability of transfer passengers when leaving the integrated hub is bus (34.8%), subway (53.7%), and taxi (11.5%). In this part of the analysis, the generalized cost is considered as penalties (reward function), so when solving for the state’s value, the generalized travel cost was taken as negative values as input. Since the formula for calculating the generalized cost function is different for each income group, the corresponding state’s values were calculated separately for groups a to c (see Table 6).Table 6
States’ value of the different trip chains of different groups.
vSiab1b2b3b4b5c0−6375.00−5847.15−5151.00−5018.40−5931.30−10715.10−4488.561−1020.00−1020.00−1348.95−2198.10−4011.15−9394.20−4138.802−510.00−517.65−742.20−1229.10−2292.45−5416.20−2390.013−255.00−239.70−242.25−293.25−441.15−940.95−408.054−510.00−489.60−545.70−754.80−1249.50−2799.90−1224.1550.000.000.000.000.000.000.006−1020.00−984.30−1124.55−1603.95−2708.10−6122.55−2681.4870.000.000.000.000.000.000.00As shown in Figure1, travelers will make several choices from state Si to the next state Si+1. Different mode choices and routes are included in their various actions. It can be considered the action-value function for the traveler’s decision in Si. It represents the expectation of reward function of the traveler when they take action in Si. The Bellman equation can derive formula (14) of the action-value function. The optimal action-value q∗s,a of 14 segments starting from node S0 and arriving at the destination S7 through different travel modes can be calculated (see Table 7). One of our analysis interests is comparing different types of trip chains on the MDP framework of the multimodal transportation networks. For this purpose, the results of 9 trip chains’ total action-value were calculated, as shown in Table 8.(14)q∗s,a=RSa+γ∑Sn∈SPSnSn+1av∗Sn+1.Table 7
The action-value of 14 segments in 9 trip chain.
Segmentab1b2b3b4b5cq1−229.00−227.87−295.87−474.95−859.88−2007.09−883.77q2−3.00−3.16−5.01−9.24−17.89−42.90−18.97q30.00−0.08−0.53−1.40−3.07−7.70−3.43q4−30.00−27.83−26.19−28.79−39.29−78.96−33.83q5−4.00−3.86−4.41−6.29−10.62−24.01−10.52q6−127.00−126.46−168.64−265.50−481.75−1125.26−495.60q7−129.50−121.88−123.96−151.45−229.57−491.72−213.40q8−511.00−490.54−546.65−755.95−1251.23−2803.59−1225.75q9−2.00−1.92−2.14−2.96−4.90−10.98−4.80q10−4.00−4.00−5.29−8.62−15.73−36.84−16.23q11−30.00−28.07−27.73−32.89−48.30−101.55−43.89q12−127.00−126.67−169.97−269.04−489.53−1144.77−504.29q13−4.00−4.04−5.57−9.36−17.37−40.95−18.06q14−100.00−90.68−74.27−61.37−54.97−72.62−27.89Table 8
Total of the action value in 9 trip chains of different groups.
Trip chainNo. 1No. 2No. 3No. 4No. 5No. 6No. 7No. 8No. 90–2(S)-3-4-70–2(B)-3-4-70–1(B)-5-70–1(S)-70–6(B)-70–7(B)0–7(T)0–2(S)-70–2(B)-7TypeUni-more(1)Multi-more(2)Multi-moreMulti-fewer(3)Multi- fewerUni-fewer(4)Uni-fewerMulti- fewerUni-fewera−769.5−769.5−232−233−136−30−100−131−131b1−740.8−741.01−231.12−231.87−130.11−28.07−90.68−130.5−130.71b2−841.38−842.71−301.41−301.16−143.05−27.73−74.27−174.21−175.54b3−1175.85−1179.39−485.59−483.57−195.48−32.89−61.37−274.86−278.4b4−1967.45−1975.23−880.84−875.61−320.72−48.3−54.97−499.12−506.9b5−4431.55−4451.06−2057.69−2043.93−715.23−101.55−72.62−1166.21−1185.71c−1939.55−1948.24−906.17−900−312.5−43.89−27.89−513.66−522.35Results in Table8 and Figure 4 show that, for high-income groups and travelers who do not care about travel cost, trip chain No. 7, with the least transfer and unimodal travel, has become the best choice for choosing the minimum generalized cost and the optimal total action-value. More attention is taken to the groups of low, middle, upper-middle-income, and travelers who only care about travel costs. The unimodal trip chains almost show advantages in the comparison. The unimodal fewer transfers rip chain No. 6 was the optimal total action-values choice. Also, the unimodal more transfers trip chain No. 1 shows better than others in some groups (middle and upper-middle-income). The possible explanation is that transfers of trip chain No. 1 happen in the same station, with shorter walking and waiting time.Figure 4
Comparison of the total of the optimal action-value.Another finding was found in this study. There is no trip chain of multimodal type that presents an advantage in comparing total action-value. It may be explained that travelers with long-distance trip chains are less likely to transfer between different modes than different routes of the same model.


## 4.1. The Multimodal Network and Transportation Data

Beijing-Tianjin-Hebei urban agglomeration includes a core megacity (Beijing) and two large cities (Tianjin, Shijiazhuang) with a permanent resident population of 110 million, an essential component in this specified area. It is one of the crucial urban agglomerations in China. It plays a vital role in defusing Beijing’s noncapital functions, strengthening the linkage between Beijing and Tianjin, and enhancing the comprehensive ability of Hebei province. The 2022 Winter Olympics will be held in Beijing and Zhangjiakou in Hebei province. It will bring a surge in traffic flow, resulting in more pressure and bringing risk to the transportation network of the Beijing-Tianjin-Hebei urban agglomeration.Take the transportation corridor from Beijing Capital International Airport to Beijingxi Railway Station as an example; the more multimodal trip chain travelers through the above two hubs, the more pressure there is in the corridor between the two hubs, and the more likely to generate passengers gathering in the two hubs. The traditional way to evacuate the passengers in transportation hubs always ignores the traveler’s final destination. It is not helpful to the transportation efficiency of urban agglomeration. Considering the travel characteristics of passengers in the trip chain of urban agglomerations, travel time, travel cost, and transfer times are taken as the critical factors for their transfer between hubs. The study on evacuation schemes under abnormal passenger flow is combined with the different needs of different travel groups, which is more effective in solving the blocked operation of interhub passages in urban agglomerations. In this case, the multimodal transportation network between Beijing Capital International Airport (BCIA) and Beijingxi Railway Station (BJXRS) has been constructed in Figure1.Figure 1
The multimodal trip chains between BCIA and BJXRS.Trip chains shown in Figure1 refer to the recommended scheme of AutoNavi Map in different periods, including the main modes of transportation, such as subway, bus, taxi (including online car-hailing), and airport bus. There are nine trip chains according to Figure 1 from BCIA to BJXRS.As shown in Table2, the nine trip chains were divided into four categories in the study according to the transfer times and mode of transfer. Considering that long-distance multimodal trip chain passengers with luggage in urban agglomeration always consider travel time, travel cost, and comfort, multimodal transfers may bring more walking distance and waiting time. Therefore, the type of travel mode and the number of transfers are used to determine the category of the trip chain. The trip chain with more than one transfer will be called “more transfers,” and the trip chain with no transfer or only one transfer will be called “fewer transfers.” Combined with whether passengers switch travel modes in the trip chain, the following four categories of trip chains (unimodal more transfer; multimodal more transfers; unimodal fewer transfers; unimodal more transfers) are formed.Table 2
Four types of trip chains between BCIA and BJXRS.
Trip chainRouteModesTransfer timesTypeNo.10---2 (S) ---3---4---713Unimodal more transfersNo.20---2 (B) ---3---4---723Multimodal more transfersNo.30---1 (B) ---5---722Multimodal more transfersNo.40---1 (S) ---721Multimodal fewer transfersNo.50---6 (B) ---721Multimodal fewer transfersNo.60---7 (BUS)10Unimodal fewer transfersNo.70---7 (TAXI)10Unimodal fewer transfersNo.80---2 (S) ---721Multimodal fewer transfersNo.90---2 (B) ---711Unimodal fewer transfersTable3 shows each segment’s travel information of trip chain No. 1 to No. 9, rows 3∼11 in Table 3. 1 represents nine trip chains, and the vertical I, II, III, and IV represent the first, second, third, and fourth trip segments of this trip chain. For example, line 2, row 3, “0 ⟶ 2 (B)” indicates that the first trip segment of the trip chain No. 1 is from the origin node (BICA) to the node “2” (Dongzhimen), and “(S)” represents that travelers choose subway in this trip segment. Line 3, row 3 indicates the type of travel time calculation for this trip segment, referring to Table 1, which calculates the travel time contained with waiting and walking time. Line 4, row 3 indicates the travel time calculation result for this trip segment, and line 5, row 3 shows the travel cost. The total travel time and cost of each multimodal trip chain are summed to obtain the results in rows 14∼15. For example, line 3, row 14 indicates that the total travel time of trip chain No. 1 is 1.46 hours, and line 3, row 15 indicates that the total travel cost of trip chain No. 1 is CNY30. The transportation information was collected from the official websites of AutoNavi Map, Capital Airport Group Corporation, Beijing Metro, Beijing Public Transportation Company.Table 3
Travel information of every segment of each trip chains.
Trip chainNo. 1No. 2No. 3No. 4No. 5No. 6No. 7No. 8No. 9IRoute0 ⟶ 2(S)0 ⟶ 2(B)0 ⟶ 1(B)0 ⟶ 1(B)0 ⟶ 60 ⟶ 70 ⟶ 70 ⟶ 2(S)0 ⟶ 2(B)Type(1)(3)(3)(3)(3)(3)(3)(3)(3)(3)Time/hr0.771.152.452.451.481.921.220.771.15Cost/CNY2525252530301002525IIRoute2(S) ⟶ 32(B) ⟶ 31(B) ⟶ 51(B) ⟶ 76 ⟶ 72(S) ⟶ 72(B) ⟶ 7Type(1)(4)(4)(4)(4)(4)(4)(4)Time/hr0.410.410.830.710.460.790.79Cost/CNY2234444IIIRoute3 ⟶ 43 ⟶ 45 ⟶ 7Type(1)(4)(4)(2)Time/hr0.070.070.15Cost/CNY110IVRoute4 ⟶ 74 ⟶ 7Type(1)(4)(4)Time/hr0.210.21Cost/CNY22TotalTime/hr1.461.843.433.161.941.921.221.561.94Cost/CNY3030282934301002929(1): rewards function type according to Table1.


## 4.2. Comparison of Harvest

The multimodal trip chain of MDP conducted in this study regards the generalized cost fromSi to Si+1 as rewards. The generalized cost better describes the travel cost of different income groups who care about both travel time and travel cost, and it can be obtained by weighting the sum of the travel time after value calculation and cost in the same segment of the trip, and the weight of time and cost can be adjusted to represent different income groups. Travel time is usually converted into a time cost dimensionalized with expense by drawing into a time value. Formula (12) shows the way of valuing travel time.(12)T=1.5Y22×8×12t.T-valued time (CNY/h); t—travel time (h); Y—annual income of traveler (CNY).In formula (12), “Y” is the annual income of urban residents, “22” is the average number of working days per month, “8” is the average number of working hours per day, multiplied by 12 to get the result that is the number of working hours per year for urban residents, and “t” represents the travel time in travelers’ each trip segments of their trip chains. According to the survey data of the National Bureau of Statistics of China in 2020, the per capita disposable income of Chinese residents in 2020 is 32,189 yuan, which is divided into five equal income groups. The low-income, low-middle-income, middle-income, upper-middle-income, and high-income groups’ per capita disposable income are 7869 yuan, 16,443 yuan, 26249 yuan, 41,172 yuan, and 80,294 yuan, respectively. Formula (13) was adopted to calculate the generalized travel cost of each income group.(13)Ri=μ1Ti+μ2ci,c—travel cost (yuan); μ1 + μ2 = 1; 0 ≤ μ1 ≤ 1; 0 ≤ μ2 ≤ 1.In formula (13), “R(i)” is the reward function of each trip segment of the trip chain; μ1 and μ2 are the weighting factors for the valued time and travel cost, respectively; “Ti” is the valued time of the trip segment of the trip chain, and “ci” is the travel cost of the trip segment of trip chain.It can be seen that the time value coefficients of different income groups are different, and it is mainly related to the average annual income of other groups. Travelers are divided into seven groups in this study to explore the various groups’ trip chain choices deeply. In addition to the five groups of varying incomes (see Table4), groups a and c were set up to represent those who only consider travel costs and travel time. For the group that only cares about economic expenditure, the generalized cost of trip chain formula consists of travel cost only, and for the group who cares two, the procedure is determined by adjusting the weight of valued time and cost (it is shown in Table 4 rows 5∼6; lines 4∼8).Table 4
The time value coefficient of different income groups in China.
The national per capitaab1b2b3b4b5cCostsLow- incomeLow-middle- incomeMiddle-incomeUpper- middle- incomeHigh- incomeTimeIncome (yuan)32189—786916443262494117280294—Coefficient22.86—5.5911.6818.6429.2457.0322.86The generalized costμ100.10.30.50.70.91.0μ21.00.90.70.50.30.10Formulac0.1 T + 0.9c0.3 T + 0.7c0.5 T + 0.5c0.7 T + 0.3c0.9 T + 0.1cTAccording to formula (2) in Section 3.2, the harvest of different trip chains can be calculated, and the results are shown in Table 5. Results of harvest shown in Table 5 illustrate every trip chain’s total rewards in the different traveler groups. Comparison of calculation results on harvest shows that trip chain No. 3 is the least cost choice for people who only care about cost when transferring from BCIA to BJXRS, and trip chain No. 9 is the least for both low-income and low-middle-income groups. Trip chain No. 1 is the minimum generalized cost to the middle-income and upper-middle-income groups. Trip chain No. 7 is the minimum generalized cost to the high-income group and passengers who only care about travel time.Table 5
The harvest of different trip chains of seven groups of travelers.
Trip chainNo. 1No. 2No. 3No. 4No. 5No. 6No. 7No. 8No. 90–2(S)-3-4-70–2(B)-3-4-70–1(B)-5-70–1(S)-70–6(B)-70–7(B)0–7(T)0–2(S)-70–2(B)-7TypeUni-more(1)Multi -more(2)Multi-moreMult-fewer(3)Multi-fewerUni-fewer(4)Uni-fewerMulti-fewerUni-fewera3030282934301002929b127.8228.0327.1227.8731.6828.0790.6826.9721.78b232.1233.4537.2237.1737.4033.7394.2731.5727.50b328.6132.1545.9743.9535.0832.8961.3729.0429.58b438.8846.6678.6173.3849.9148.3054.9740.6346.61b577.9497.44178.85165.09102.97101.5572.6282.97101.87c87.6110.4205.8189.6116.4115.273.293.6116.4(1): unimodal more transfer; (2): multimodal more transfers; (3): unimodal fewer transfers; (4): unimodal more transfers.Figure2 compares the generalized travel cost of each trip chain in different groups more intuitively. Trip chain No. 7, which belongs to unimodal fewer transfer type, is the maximum generalized cost chain in groups a, b1, b2, b3. Still, it changes to the better or best choice in groups b4, b5, and group c. Travelers who take taxis from BCIA to BJXRS may get the least generalized cost when they have more annual income. In contrast to this phenomenon, trip chains No. 3 and No. 4 show the less generalized cost in group a, b1, b2, b3, but they transfer to be the most generalized cost in group b5 and group c.Figure 2
Comparison of different trip chains of seven groups of travelers.From line 2 in Table5, it is readily observed that trip chains Nos. 3, 4, 8, and 9 belong to the lower generalized cost one, but none of them is the least generalized choice in group b1∼b5 and group c except trip chain 9. Differences in their types might cause this result. Based on Table 2, trip chains 3, 4, and 8 are the multimodal trip chain, but trip chain 9 is the unimodal trip chain. It can be explained that travelers prefer to choose the trip chain that concluded with fewer modes in their trip.An exciting finding is that trip chain 1 (type (1)) is the least generalized cost in theb3 and b4 groups. However, those trip chains with fewer transfers (type (3) and (4)) are not the least. It can be found that there is a long period of waiting time in the first segment in trip chains No. 3 and No. 4 by checking the information in Table 3. The average waiting time of that segment is 1.5 hours, and the time value coefficient is higher between the middle and upper-middle-income groups. It means that more attention should be taken to the out-of-vehicle time in the trip chains of urban agglomeration than before.Figure3 shows the comparison of the harvest for four pairs of trip chains considering differences in traveler income. Figure 3(a) shows the comparison between unimodal and multimodal trip chains with more transfers; Figure 3(b) shows the comparison between the two multimodal trip chains with more and fewer transfers, respectively; Figure 3(c) shows the comparison of harvest for the two unimodal trip chains without transfer; Figure 3(d) shows the comparison between unimodal and multimodal trip chains with fewer transfers. Looking first at part a of Figure 3, the generalized cost of unimodal trip chain (No. 1) is gradually lower than the multimodal one (No. 2) for middle, upper-middle, high-income groups, and travelers who do not care about travel costs. Therefore, this change becomes more and more evident as income increases. Comparatively similar is the comparison between trip chains No. 3 and No. 4. Like Figure 3(b), the fewer transfers type of trip chain has lower generalized cost than the more transfers type when the income increases. However, the difference is that the change in the gap between the two trip chains is relatively tiny. The apparent discrepancy between the same type of trip chain appears on trip chains No. 6 and No. 7. As shown in Figure 3(c), the generalized cost of unimodal fewer transfers type of trip chain presents a sharp difference with the change of income of different groups. The generalized cost of taking an airport bus is less than a taxi to the middle and below middle-income groups and those who do not care about travel time.Figure 3
Comparison of the harvest for similar types of trip chains.
(a)(b)(c)(d)On the contrary, taking a taxi is the lower generalized cost trip chain for travelers with more annual income or who do not care about travel costs. The equivalence point of trip chains No. 6 and No. 7 shows the gap between upper-middle and high-income groups. Over 60% of travelers may choose the airport bus to have the unimodal fewer transfer trip between two hubs in urban agglomeration. Similar but critically different to the previous comparison that appears on trip chains No. 8 and No. 9 (Figure3(d)), the exciting turning point occurs for the middle-income group. The only difference is whether travelers choose the subway or not in the first segment, and it can be seen that the low-middle-income and low-income groups need to select the traditional bus to ensure the lower generalized cost. Thus, travelers who do not care about travel costs or have more income always choose the subway for the first trip to provide the lower generalized cost. It can also be seen that the growth of income has a positive association with the subway mode preference.


## 4.3. States’ Value and the Action-Value Function

Table6 presents the calculation results of states’ value of MDP by Python. As shown in Figure 1, eight nodes occur for the multimodal transportation network, while travelers need to travel from S0 to S7. This study depends on data from the survey conducted by the authors in May 2020, which focuses on the transfer decisions of multimodal trip chain passengers between integrated hubs of Beijing-Tianjin-Hebei urban agglomeration. The results of this survey show that the transition probability of transfer passengers when leaving the integrated hub is bus (34.8%), subway (53.7%), and taxi (11.5%). In this part of the analysis, the generalized cost is considered as penalties (reward function), so when solving for the state’s value, the generalized travel cost was taken as negative values as input. Since the formula for calculating the generalized cost function is different for each income group, the corresponding state’s values were calculated separately for groups a to c (see Table 6).Table 6
States’ value of the different trip chains of different groups.
vSiab1b2b3b4b5c0−6375.00−5847.15−5151.00−5018.40−5931.30−10715.10−4488.561−1020.00−1020.00−1348.95−2198.10−4011.15−9394.20−4138.802−510.00−517.65−742.20−1229.10−2292.45−5416.20−2390.013−255.00−239.70−242.25−293.25−441.15−940.95−408.054−510.00−489.60−545.70−754.80−1249.50−2799.90−1224.1550.000.000.000.000.000.000.006−1020.00−984.30−1124.55−1603.95−2708.10−6122.55−2681.4870.000.000.000.000.000.000.00As shown in Figure1, travelers will make several choices from state Si to the next state Si+1. Different mode choices and routes are included in their various actions. It can be considered the action-value function for the traveler’s decision in Si. It represents the expectation of reward function of the traveler when they take action in Si. The Bellman equation can derive formula (14) of the action-value function. The optimal action-value q∗s,a of 14 segments starting from node S0 and arriving at the destination S7 through different travel modes can be calculated (see Table 7). One of our analysis interests is comparing different types of trip chains on the MDP framework of the multimodal transportation networks. For this purpose, the results of 9 trip chains’ total action-value were calculated, as shown in Table 8.(14)q∗s,a=RSa+γ∑Sn∈SPSnSn+1av∗Sn+1.Table 7
The action-value of 14 segments in 9 trip chain.
Segmentab1b2b3b4b5cq1−229.00−227.87−295.87−474.95−859.88−2007.09−883.77q2−3.00−3.16−5.01−9.24−17.89−42.90−18.97q30.00−0.08−0.53−1.40−3.07−7.70−3.43q4−30.00−27.83−26.19−28.79−39.29−78.96−33.83q5−4.00−3.86−4.41−6.29−10.62−24.01−10.52q6−127.00−126.46−168.64−265.50−481.75−1125.26−495.60q7−129.50−121.88−123.96−151.45−229.57−491.72−213.40q8−511.00−490.54−546.65−755.95−1251.23−2803.59−1225.75q9−2.00−1.92−2.14−2.96−4.90−10.98−4.80q10−4.00−4.00−5.29−8.62−15.73−36.84−16.23q11−30.00−28.07−27.73−32.89−48.30−101.55−43.89q12−127.00−126.67−169.97−269.04−489.53−1144.77−504.29q13−4.00−4.04−5.57−9.36−17.37−40.95−18.06q14−100.00−90.68−74.27−61.37−54.97−72.62−27.89Table 8
Total of the action value in 9 trip chains of different groups.
Trip chainNo. 1No. 2No. 3No. 4No. 5No. 6No. 7No. 8No. 90–2(S)-3-4-70–2(B)-3-4-70–1(B)-5-70–1(S)-70–6(B)-70–7(B)0–7(T)0–2(S)-70–2(B)-7TypeUni-more(1)Multi-more(2)Multi-moreMulti-fewer(3)Multi- fewerUni-fewer(4)Uni-fewerMulti- fewerUni-fewera−769.5−769.5−232−233−136−30−100−131−131b1−740.8−741.01−231.12−231.87−130.11−28.07−90.68−130.5−130.71b2−841.38−842.71−301.41−301.16−143.05−27.73−74.27−174.21−175.54b3−1175.85−1179.39−485.59−483.57−195.48−32.89−61.37−274.86−278.4b4−1967.45−1975.23−880.84−875.61−320.72−48.3−54.97−499.12−506.9b5−4431.55−4451.06−2057.69−2043.93−715.23−101.55−72.62−1166.21−1185.71c−1939.55−1948.24−906.17−900−312.5−43.89−27.89−513.66−522.35Results in Table8 and Figure 4 show that, for high-income groups and travelers who do not care about travel cost, trip chain No. 7, with the least transfer and unimodal travel, has become the best choice for choosing the minimum generalized cost and the optimal total action-value. More attention is taken to the groups of low, middle, upper-middle-income, and travelers who only care about travel costs. The unimodal trip chains almost show advantages in the comparison. The unimodal fewer transfers rip chain No. 6 was the optimal total action-values choice. Also, the unimodal more transfers trip chain No. 1 shows better than others in some groups (middle and upper-middle-income). The possible explanation is that transfers of trip chain No. 1 happen in the same station, with shorter walking and waiting time.Figure 4
Comparison of the total of the optimal action-value.Another finding was found in this study. There is no trip chain of multimodal type that presents an advantage in comparing total action-value. It may be explained that travelers with long-distance trip chains are less likely to transfer between different modes than different routes of the same model.


## 5. Conclusion

In this paper, the multimodal trip chain between hubs in urban agglomeration is regarded as an MDP problem, aiming to analyze trip chains of different types in various traveler groups. To accurately analyze the travel time of the trip chain, four types of calculation were used to obtain travelers’ time according to multiple patterns of transfer, and trip chains are divided into four categories depending on the transfer times and travel modes interchange. Then, travelers were divided into seven groups to compare the generalized cost of different trip chains, making the comparison more meticulous. Starting from the multimodal transportation network’s dynamic and random characteristics, we conducted the Markov Decision Progress to research an individual’s optimal choice on the network by comparing traveler’s total action-value of the trip chain.It should be noted that almost all existing results for the analysis in this study show that the unimodal trip chain, including fewer transfers or more transfers, always becomes the best choice of a particular group. And the comparison of the minimum generalized cost and the optimal total action-value in MDP shows other factors that have essential impaction on the traveler’s decision-making, such as waiting time, walking time, and transfer times, whether transfer happens in the same station. New findings in this study complement previous studies on influencing factors of trip chain choice (including on-vehicle time and travel cost).According to the above analysis and discussion, several conclusions can be drawn from this study. First, for the travelers who have long-distance trips with transfers, the waiting time plays a decisive role in the travel behavior. Compared with the long waiting time, people may prefer to transfer more times. Second, individual income difference and personal characteristics of travel affect their choices. It shows a significant difference in comparing same type trip chains, such as unimodal fewer transfers type and multimodal more transfers type of trip chain. The transition occurs in the middle and upper-middle-income groups when travelers hesitate between taxis and buses from one integrated transportation hub to another in the urban agglomeration. However, the middle-income group changes when people choose between subway and buses in the multimodal more transfers type of trip chain. Finally, travelers are more likely to choose unimodal than multimodal trip chains, even if the unimodal trip chain involves more transfer times, consistent with previous research [15].The findings of this study suggest the great importance of travel management for long-distance multimodal trip chains in urban agglomeration. The essential difference of multimodal trip chain between urban agglomeration and urban is that the former does not start or end at home, and travelers always determine the activities before mode choice of trip chain, especially the mode choice of a long trip in their trip chain. Another difference is that a long-distance trip chain traveler will not stay a long time in a hub or node, and it may delay the following trip. Transportation efficiency is the most important factor for travelers with long-distance trip chains. Based on the results of this study, the unimodal fewer transfers type of trip chain is the better choice to enhance the capacity of transportation between two hubs. Therefore, to solve the traffic pressure rises during lousy weather or abnormal situations, the shuttle bus and subway from one comprehensive hub to another should be increased, along with more taxi or online car-hailing. Appropriate discounts for shuttle buses can attract middle and low-middle-income travelers for a different group of travelers. The pressure of subway transportation can be reduced, and the sharing of multiple modes of transport will be more balanced.This research chooses a part of the long-distance multimodal trip chain in the urban agglomeration, and more attention is focused on the trip chain between two integrated hubs in their long-distance trip, and it chooses two important hubs in Beijing-Tianjin-Hebei urban agglomeration as a case study. In the future, the intercity trip can be added to the long trip chain to compare the different types of trip chains in urban agglomeration. In summary, this study provides a method of comparing the other multimodal trip chains of urban agglomeration multimodal travelers and considers more factors to influence long-distance travelers’ behavior.



---

*Source: 5027133-2022-02-18.xml*