City Metro Network Expansion with Reinforcement Learning

1 INTRODUCTION . City metro network plays an important role in public transportation system . With the development of city , new transportation demands have led to the expansion of metro network . The last few years have witnessed tremendous expansion of metro network ( Sun et al. , 2018 ) . On the other hand , the expansion of the metro network in turn has a profound impact on the city . The expanded lines may change the mobility trend of city population . Most previous research focuses on the design of metro network from scratch ( Gutiérrez-Jarpa et al. , 2018 ; Laporte & Pascoal , 2015 ) . However , in the subsequent construction process , the dynamic of the city has been different from that in the initial stage . The original scheme may not be suitable for the current situation . Therefore , it is more reasonable to gradually design new lines to expand the metro network according to current city dynamic . Usually , transportation planning objectives are mobility-based , such as maximizing OD trips . As the society progressed , sustainability has increasingly become the demand of city development . The sustainability prompts governments to re-recognize the role of the transport system , and thereby influences their transport policy ( Manaugh et al. , 2015 ) . One conception of sustainability , social equity , which can be measured by the distributable benefit accessibility ( Behbahani et al. , 2019 ) , has been acknowledged important . There have been several real transportation plans considering social equity ( Arsenio et al. , 2016 ) . Metro network , an important transportation system , has a great influence on social equity . Therefore , in this work , we consider both OD trips and social equity to expand city metro network . However , the problem becomes difficult when the city becomes large . First , it is difficult to formulate the problem efficiently ( Laporte & Mesa , 2015 ) . Existing studies formulate the problem as nonlinear integer programming , and call for an exponential number of subtour elimination constraints to ensure the rationality of expanded metro lines ( Gutiérrez-Jarpa et al. , 2013 ; Wei et al. , 2019 ) , which hinders solving the problem efficiently . Second , the huge solution space makes it difficult to find a good solution effectively . Exact methods are inapplicable for integer programming problems with large solution space ( Farahani et al. , 2013 ) . One common method in existing studies is to limit the search space . Previous work ( Gutiérrez-Jarpa et al. , 2013 ; Wei et al. , 2019 ; Laporte & Pascoal , 2015 ; Gutiérrez-Jarpa et al. , 2018 ) predefines corridors based on expert knowledge , and only consider to design metro lines in these corridors . Their results depend on expert guidance , and it is possible that the best solution is left out . Therefore , we call for a method which does not require expert knowledge . Carefully handcrafted heuristics , embedded with problem-specific knowledge , are usually efficient for large search space ( Dufourd et al. , 1996 ) . However , the factors considered in the expansion of metro line vary with different cities and stages . Once the objectives of metro expansion change , heuristic methods need to be revised . Rather than handcrafting different heuristics for different objectives , a general method is necessary . In this paper , we propose a method without expert knowledge to solve the city metro network expansion problem using RL . We formulate the metro line expansion as a process of sequential station selection , an MDP , and design feasibility rules based on the selected station sequence to ensure the reasonable connection patterns of the metro line . This formulation efficiently characterize the expansion of the metro line , without heavy constraints like existing studies ( Wei et al. , 2019 ) . Following this formulation , we propose an actor critic model ( Konda & Tsitsiklis , 2000 ) to generate the next metro line . The actor is based on the seq2seq model ( Sutskever et al. , 2014 ) . In the actor , an encoder characterizes the timely metro station information in the expansion process . After encoding , an RNN decoder is used to characterize the sequence information of the selected stations . Moreover , we employ an attention layer to produce the probability distribution over feasible candidate stations . Only requiring the reward calculation , we train the model with the critic reducing training variance , in order to find the high-priority metro line following feasibility rules . The reward function takes the final output station sequence as input only , which is friendly to objective changing . Therefore , our model is general for different objectives . Without expert knowledge , the learning procedure drives the policy to keep track of the better solution during the search and to search for better solutions . Its natural exploration mechanism determines that RL is suitable for large scale solution space . Based on real city-scale human mobility information of 24,770,715 mobile phone users obtained from a citywide 3G cellular network , we expand the current metro network in Xi ’ an , China . The results demonstrate the effectiveness of our method . Our contributions are as follows : 1 . We formulate the expansion of a metro line as a process of sequential stations selection , a Markov decision process ( MDP ) . We design feasibility rules based on the selected station sequence to ensure the reasonable connection patterns of metro line , which is more efficient to formulate the problem than integer programming models . 2 . We firstly propose a RL based method to solve the city metro network expansion problem . Without expert knowledge , our general method can be easily extended to the metro expansion considering multi-factors . 3 . We incorporate social equity concerns into metro network expansion . Compared with the realistically planned lines , the results show the rationality of considering social equity in transportation planning . 4 . We use real city-scale human mobility information to expand a metro network . The experimental results demonstrate the effectiveness of our method . 2 RELATED WORK . 2.1 METRO NETWORK DESIGN . Metro network plays an important role in public transportation system . For metro network design , Laporte & Pascoal ( 2015 ) predefine a set of corridors , and propose path based algorithms . Considering the connection with existing lines , Wei et al . ( 2019 ) expand the metro network in predefined corridors . However , these methods depend on predefined corridors by planner , which may be interfered by human judgment . Gutiérrez-Jarpa et al . ( 2018 ) propose a greedy generation heuristic to select a set of corridors with higher passenger traffic from a set of candidate corridors . Enumerating all candidate corridors is impossible , which may leave out some good solutions . The corridors may need frequent adjustments , once the objectives change . In addition , these studies adopt non-linear integer programming models to formulate the metro network design problem , and call for exponential number of constraints to ensure the rationality of expanded metro lines , which makes it difficult to solve the problem efficiently . To cope with these problems above , we propose a general method that is suitable for diverse objectives and does not require prior knowledge to expand city metro network . 2.2 REINFORCEMENT LEARNING . The strength of RL lies in its powerful decision-making ability . RL has made great progress in complicated tasks like playing Atari games ( Mnih et al. , 2013 ) , robot training ( Yang et al. , 2017 ) and combinatorial optimization ( Bello et al. , 2016 ; Nazari et al. , 2018 ) . RL is proved efficient in problems with high dimensional search space , such as Go ( Silver et al. , 2016 ) and StarCraft ( Vinyals et al. , 2017 ) . Meanwhile , classical combinatorial optimization problems like minimum cut ( Li et al. , 2018b ) , traveling salesman problem ( Bello et al. , 2016 ; Kool & Welling , 2018 ) and vehicle routing problem ( Nazari et al. , 2018 ) are solved by RL . Li et al . ( 2018a ) points out that RL can find important factors which may be ignored by human . We believe that RL has the ablity to solve the metro expansion problem . 3 PROBLEM FORMULATION . In this paper , we design the next metro line to expand the current metro network in the target city . The metro line is determined by stations and line routing , and is allowed to connect with existing lines to form transfer stations . For a target city , we divide it into n× n grids in a two dimensional space { gi } n 2−1 i=0 . Each grid gi is a square with a width of d , and its center is a candidate station si . We define the expansion of metro network on an undirected graph G = ( S , E ) , where S = { s0 , s1 , ... , sn2−1 } contains all candidate stations and E = { ( si , sj ) |si , sj ∈ S } contains all direct links between stations in existing lines . Each grid gi is associated with a compound index of developmentDi , and any two candidate stations si and sj are associated with the two-way symmetrical OD trips odi , j ( = odj , i ) between them . We present the expanded metro line as an ordered station sequence Z = ( z1 , z2 , . . . , zT ) , zi ∈ S , and it should satisfy the following constraints . • The consecutive stations must follow the minimum-maximum distance rules . • The line routing should ensure reasonable connection patterns of stations , avoiding subtour and meandering line . • The number of the stations is limited by N . • The budget is limited by B . Satisfied by the new line Z , we denote the newly added OD trips as Rod ( Z ) and the social equity indicator as Rac ( Z ) , which aims to maximize the total benefits of the society . More details about these two are in Appendix A . The objective of the new metro line is to maximize the weighted sum of OD trips and social equity ω ( Z ) = α1 ×Rod ( Z ) + α2 ×Rac ( Z ) ( 1 ) where α1 and α2 are the weights of added OD trips and social equity , and α1 + α2 = 1 . 4 METHOD . 4.1 RL FORMULATION . We focus on the city metro network expansion problem . We denote the station selected at step t as zt , and the selected station sequence until step t as Zt = { z1 , z2 , . . . , zt } . Given the graph G , we aim to learn a parameterized policy P ( Z|G ) to maximize ω. P ( Z|G ) = T∏ t=1 P ( zt|G , Zt−1 ) , ( 2 ) and Mt =M ( Zt ) ( 3 ) Equation ( 3 ) characterizes the update of feasibility rules . We use a binary vector Mt to indicate whether each station can be selected in next step . According to the rules , feasible stations are set to 1 , oherwise 0 ( see Section 4.2 for more details ) . Important RL elements are listed as follows . Environment . The environment correspond to the city in which the metro is expanded . The RL agent will build the station in order , completing the metro network expansion . State . For step t , the selected station sequence Zt is the current state . After the agent chooses the next station zt+1 , the state will be updated with Zt+1 . Action . We define the station zt which the agent chooses to build next step as the action . Once the action is decided , the current state will be updated . Reward . Since RL are set to maximize the reward , we define the value of objective ω as reward . The reward function is calculated when the metro expansion is done .

Summary: This paper introduces a method for solving the problem of network expansion, in particular, considers the city metro network and its expansion within the new metro line. Since this problem was previously represented as the non-linear integer problem, having an exponential number of constraints or requiring expert knowledge, authors had an initial motivation to appeal to learnable algorithms from the reinforcement learning paradigm. Authors proposed to consider the problem as Markov Decision Process, so that the environment is represented as the city grid, with constructed metro network, before some timestamp conditioned by additional transportation constraints, while each new state in the expansion process of construction one separate metro line is the state of the metro network as graph with new station added to the metro line, considered as the agent’s action. The custom reward function was represented based on metrics like OD trips used in the traditional approaches and the first time used in this problem specific social equity score. The permissibility constraints of action, i.e. adding a new station, were incorporated into policy function, while the attention mechanism was used here as in other seq2seq models, to query the most suitable station among candidates. Authors use the actor-critic approach, widely exploited in reinforcement learning problems, and with all the above-mentioned modification reached the baseline results on the real-world data, obtained from Xi’an city metro network, Shaanxi province, China.