City Metro Network Expansion with Reinforcement Learning

Abstract: This paper presents a method to solve the city metro network expansion problem using reinforcement learning (RL). In this method, we formulate the metro expansion as a process of sequential station selection, and design feasibility rules based on the selected station sequence to ensure the reasonable connection patterns of metro line. Following this formulation, we train an actor critic model to design the next metro line. The actor is a seq2seq network with attention mechanism to generate the parameterized policy which is the probability distribution over feasible stations. The critic is used to estimate the expected reward, which is determined by the output station sequences generated by the actor during training, in order to reduce the training variance. The learning procedure only requires the reward calculation, thus our general method can be extended to multi-factor cases easily. Considering origin-destination (OD) trips and social equity, we expand the current metro network in Xi'an, China, based on the real mobility information of 24,770,715 mobile phone users in the whole city. The results demonstrate the effectiveness of our method. 

Reviews and author responses:

This paper introduces a method for solving the problem of network expansion, in particular, considers the city metro network and its expansion within the new metro line. Since this problem was previously represented as the non-linear integer problem, having an exponential number of constraints or requiring expert knowledge, authors had an initial motivation to appeal to learnable algorithms from the reinforcement learning paradigm. Authors proposed to consider the problem as Markov Decision Process, so that the environment is represented as the city grid, with constructed metro network, before some timestamp conditioned by additional transportation constraints, while each new state in the expansion process of construction one separate metro line is the state of the metro network as graph with new station added to the metro line, considered as the agent’s action. The custom reward function was represented based on metrics like OD trips used in the traditional approaches and the first time used in this problem specific social equity score. The permissibility constraints of action, i.e. adding a new station, were incorporated into policy function, while the attention mechanism was used here as in other seq2seq models, to query the most suitable station among candidates. Authors use the actor-critic approach, widely exploited in reinforcement learning problems, and with all the above-mentioned modification reached the baseline results on the real-world data, obtained from Xi’an city metro network, Shaanxi province, China.

Main arguments 

Apart from the interesting approach, this paper should be rejected due to several reasons: (1) the specific application field, considered in this paper, have to be generalized to more general cases, in order to be valuable for Machine Learning community, (2) from RL algorithms perspective, the novelty of proposed method is questionable due to lack of literature review and advanced approach to deep reinforcement learning, (3) requirement of explainable AI is essential for deployment and in spite of the quite sufficient explanations of the algorithm’s works, this paper does not well justify its superiority over the traditional methods either by theory or practice, due to experiments suffering from the lack of variability and missing the main point of improvement, which leads to generally insufficient contribution (4) the paper is unpolished and lacks specific formulations from the existing literature on related subjects.
 
In detail

The paper does a great job of justifying neither the novelty of its method nor the guarantees of increased performance or efficiency compared to existing methods. Although the proposed method discusses the specific problem and there could be a lack of existing sufficiently good methods in this specific topic, the reinforcement learning paradigm is utilized by authors in a very superficial and general manner. 

Firstly, the paper excludes any comparison to similar problems and already existing methods of RL used by similar fields, which can be expressed as planning with constraints (for example, path planning in mobile robot navigation), thus will be more valuable to ML community than specific application of graph expansion problem, limited to line-by-line addition. 

Secondly, except general notice of combinatorial problems solved by RL algorithms, the literature review lacks mentioning graph representation methods or any seq2seq algorithms (which efficiently uses attention), which should serve as an initial guess for the well-performing model in this particular problem. Regarding the RL approach, there are plenty of already solved planning problems, papers comparing RL algorithms and their performances on grid-based, discrete environment problems in order to mention here and ideally, comparing to proposed one within the same specific problem formulation. Because of this reason, there is no justification of concrete architecture construction and choice of actor-critic model for learning this policy in this specific example and so there is no intuition, why this model will work better than existing methods. Besides, this leads to the doubtful novelty of the proposed algorithm, although used first time in this specific field, but composed totally from the same components and sometimes in the same combination as in other works related to RL methods in planning.

Moreover, the explanation of the methods’ architecture and basic components’ influence on the system is insufficient but might serve as an initial explanation of how this algorithm works. However, the number of experiments and their variations leaves much to be desired to justify its practical advantages and its utilization purposes as explainable AI. (1) there is no ablation study conducted to justify the choice of the proposed model over existing reinforcement learning methods, like actor-critic methods and their several variations with baseline, including modifications like attention model, graph representation, encoder and decoder architecture. The mentioning of these methods in the literature review is enough to answer this issue. (2) Using only one, hardly tuned baseline method on one dataset (without any reference to existing benchmark results on it) does not prove the efficiency and good performance of the model. Additional experiments can on different environments structure (city grids, metro network) of complexity of the environments (the width of the cell in the grid) to measure the performance of the algorithm based on hyperparameters (3) Real-world data brings more application-based matter into the subject, but requires more thorough investigation on the optimality of the solutions, proposed by RL method. This means that the comparison with the groundtruth metro network expansion (built after October 15) cannot be used as it is without using the expert evaluation. As a solution and habit in the machine learning literature, without the expert knowledge on the dataset, authors may propose a comparison of the real-world solution with the algorithm’s results, based on the optimized metrics (OD trips score and social equity) included into reward function. 
 
Finally, the proposed paper is imprecise in several ways. In terms of RL formulation and usage of RL terms (commonly introduced in literature), it reveals inappropriate usage of MDP terms and insufficient description policy-gradient approach in actor-critic training algorithm. While not appealing to this issue, authors missed the opportunity to formulate this problem as discrete MDP with high-dimensional action space and sparse rewards, thus had less opportunity to research different model-based reinforcement learning problems in discrete environments to conduct better model selection and experiments. Besides, the paper lacks details in some parts, like the usage of 3G cellular mobile data, which was not mentioned in this work, although it is emphasized as an essential part if not a significant contribution of this research.  
 

 
Questions to answer
Introduction
	What is the reason by constructing the city metro networks line by line, not iteratively adding stations to the existing network on the grid? This would make your problem closer to general grid-based planning problems. Do traditional methods expand metro networks only line-by-line and can this be there a limitation?
	How does a “good solution” is represented in the literature with traditional approaches? What are the general measures and constraints in practice?
	Is incorporating social equity concern your contribution? Then why does the maximization of OD trips is not enough (there is no mention of the preferability of social equity metric based on the results of experiments)?
	The effectiveness of the method due to experimental results is still questionable.
Related work
	If the main concern of the paper is still the limitation of other methods which use expert knowledge then it is better to state the usage of additional data (development index) in social equity metric as a reward design part in Appendix to justify this reward engineering
	“We believe that RL has the ability to solve the metro expansion problem” is the statement, which should be substituted by extensive literature review on RL methods used for planning with constraints or specifically graph-based expansion methods. 
Problem formulation
	Is the network graph undirected or directed (imprecise inclusion of “direct links”)? How does this the OD trips are measured (no information here or in Appendix A).
	What is the reason behind 4 constraints provided in problem formulation? Are they used or defined in the literature review papers?
 
Method
	RL and deep learning methods included in the literature review should be the reference of choosing one or another component of the algorithm, including the formula for policy and critic update.
	It is essential to use notation based on RL formulation, for example, use r(s_t,a_t) = 0 if s_t  is not terminal state and r(s_t,a_t) = w(Z|G) in order to state the sparsity of the problem. S_t here is Z_t sequence of chosen stations during the episode (line expansion process) and so on.
	Use precise formulation, for example, the proposed P(Z|G) probability distribution is the trajectory distribution and not policy function, which could be denoted as π(z_t | S,G,M(Z_t)) 
	How does the choice of filter design is justified, based on similar works in the transportation field?
	What is the reason by choosing the 1-dimensional CNN layer to represent candidate stations and what is the concrete input information? There is no formal specification of the input data, regarding the existing graph structure, only a short mention of it.
	Is the attention mechanism architecture, used here, similar to other models used in seq2seq problems? (Answer: Yes) What is the motivation to use this concrete architecture? 
	Why does the permissibility mapping using filter is used in policy function? Are there reasons to use it in policy function, rather than include in reward design? For example, the commonly used approach is to penalize unfeasible actions (stations) with a very low reward, during the episode learning?
	Training: 
	How does the sparsity of reward influence the performance? Is there a way to better design reward, so that there will not be a necessity to update actor and critic only after the end of the episode (termination step)?
	What is the reason for learning critic as V(Z) = w(Z|G)? Is there a need for baseline if the state and the sequence of actions have bijective mapping, meaning that one sequence of stations can generate the unique state of the environment – line expansion, and vice versa? The intuition of the baseline is to measure the value of the state as the average among all possible actions, which lead to this state.
	What are the batches B used in the actor-critic training procedure?
	How do we generalize the environment training? Do we need to retrain the reinforcement learning algorithm for each different initial metro city network configuration, or it is generalizable to other grids, using the same weights?
Experiments
	Please, include the training time of the RL algorithm and its inference time, and compare it to the performance of the baseline algorithm, which should be one of the most essential contributions, due to the high time-complexity of traditional methods.
	How does the choice of corridor width influence the performance of the baseline method? 
	Is there the baseline performance on the same city metro network (Xi’an city metro) mentioned in other literature, to directly compare with? This is necessary to fill the gap in justification of proposed results optimality (for the baseline case).
	How does the partial similarity of the 2 times line expansion by RL method and 6 real-world lines of the city metro network can justify the optimality of the proposed method? Can you provide the measurement based on OD trips and social equity? Can you provide a truly optimal solution based on the grid granularity, initial network graph, and constraints, to compare with as an optimal solution?


Review of “City Metro Network Expansion with Reinforcement Learning”

In this work, they investigate the use of RL (actor-critic) for planning the expansion of a metro subway network in a City. They formulate the problem of expanding the metro system as sequence of actions, hence a seq2seq+attn type architecture is used for the actor. The reward calculation is based on a combination of origin-to-destination and social equity metrics. They apply their method to plan the expansion of the current metro lines in a large city in China where they can leverage the mobility information information gathered from ~ 25M mobile phones in the city.

I think this work has great potential, as they identify a data-driven approach that can have a high impact (i.e. design subway lines affecting 25M+ people in a real city). That being said, there are some issues that I feel needs to be addressed before the work can be published at a high quality conference, so I want to help the authors improve the work by highlighting important points that will make the work better:

The related work section on metro network design is only a paragraph. I think both the related work and experiments sections are missing many substantial contributions, as there is vast literature of work from Operations Research area about solving such problems through constraint optimization. Even a simple google scholar search brings many examples [1]. This is before discussing about existing ML and Genetic Algorithm approaches [2], or with Monte Carlo Tree Search [3].

Without discussing existing work and offering detailed comparisons and experiments, this paper essentially just shows that RL can be applied to such problems, but the reader wouldn't know whether it is the best tool, or simply if RL is the hammer that is used to treat every problem like a nail. The only baseline the paper compared against is another paper published in 2019 which IMO is not satisfactory.

On a related note, there are also a few projects doing similar network optimizations with slime mold (Physarum) and different variations using it for shortest path finding in mazes and all sorts of interesting problems [4].

I'm reminded of a nice work called “Neural Combinatorial Optimization with Reinforcement Learning” [5] that proposed the use of neural nets to solve TSP problem, but ultimately needed to put in the work to compare with traditional approaches. I'm including the reference here so the authors can learn from that paper's experiences to help improve the work.

Regarding the dataset: One of the most impressive points is that the work utilized a giant dataset of ~ 25M mobile phones. For an important dataset like this that is central to an impactful application of ML, would be nice to have a discussion (even in the Appendix) to describe how the data is collected, and what regulations / user privacy issues the research team might have to overcome, as these types of issues are becoming very important to the wider research community. Would also like to see a discussion about whether the large amount of data points can be reduced to a simpler 2D density map and achieve similar performance? Would there be any plans to release an anonymized version of the dataset for demonstration purpose?

[1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=transportation+network+operations+research+constraint+optimization&btnG=

[2] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=transportation+network+design+machine+learning&btnG=

[3] i.e. Link Prediction with Monte Carlo Tree Search (https://paperswithcode.com/paper/m-walk-learning-to-walk-over-graphs-using)

[4] https://www.researchgate.net/publication/324791496_Physarum-Inspired_Solutions_to_Network_Optimization_Problems

[5] https://openreview.net/forum?id=rJY3vK9eg

In this paper the authors train a seq2seq model through reinforcement learning to iteratively expand a city metro network. The authors show that different objectives can be satisfied with this approach, such as the accessibility to different areas (something the authors call social equity indicator) or maximising origin-destination trips. 

The paper is interesting but could use a more extensive comparison to alternative approaches or ablated  version of the same approach. For example, what if the approach would only take into account the last metro station instead of the complete previous sequence? Would it work less well? Additionally, the baseline method the approach is compared against is not explained in enough detail. In addition to RL methods, method such as genetic algorithm have shown great promise in producing layouts, such as for wind turbines (e.g. Grady et al. “Placement of wind turbines using genetic algorithms”). I wonder if such an approach would work equally well for designing metro lines and if RL is really the best technique here (which it might be but I’m not convinced yet). Because of the mentioned shortcomings, I believe the paper should be improved before publication. 

Additionally, the paper would benefit from a careful spell and grammar check. I found multiple typos, especially in the introduction.  

Meta-review: The paper explores the use of RL (actor-critic) for planning the expansion of a metro subway network in a City. The reviewers felt that novelty was limited and there was not enough motivation on what is special about this application, and what lessons can be learned from this exercise.  

Decision: reject