# Application Method of Environmental Protection Building Elements Based on Artificial Intelligence Technology in the Field of Urban Planning and Design

Authors: Lihong Yan
Journal: Advances in Multimedia
 (2022)

---

## Abstract

In order to improve the efficiency of urban planning and design and reduce the construction waste caused by inefficient design schemes, this paper proposes an application method of environmental protection building elements based on artificial intelligence technology in the field of urban planning and design. This model enables agents to learn how to interact with the environment with appropriate strategies, so as to generate the road network design scheme required by users. In order to better control the generation of the result scheme, this paper designs the rules and feedback of the agent. Among them, rules restrict the behavior of agents to avoid design schemes that do not meet the road design specifications. Feedback determines the values followed by agents, that is, the direction of agent strategy optimization. The experimental results show that after 4×105 iterations of the training process, the model is derived and used to generate 10000 plot road network schemes, the mean and variance of key indicators are calculated, and the scores of 153 plots with similar area and shape and the same land function in the same indicators from the real case base are calculated. From the performance of these indicators, it can be seen that the performance of the road network generated by the model is similar to that of the real road network in terms of traffic performance.Conclusion. the model completely retains the structural geographic information inside the plot, and the generated results can be directly applied to mainstream urban design software through data format conversion.


---

## Body

## 1. Introduction

Urban center is the core area of urban function and structure and the “market center” of urban and regional socioeconomic operation. Since the 1960s, in terms of the layout of big cities, China’s cities have gradually changed from a closed single center urban layout to an open multicenter urban layout [1]. The multicenter urban layout promotes the development of urban center planning and construction. In addition, the urban center not only provides a space carrier for urban expansion and industrial adjustment, but also the starting point of urban vitality. At the same time, as a typical gathering place in the city, the public space of the urban center also constitutes a catalyst for urban vitality. Therefore, the planning and construction of urban centers plays a very important role in urban planning [2].Due to historical reasons, with the rapid urbanization process in China, more universal problems have been exposed, forming a “Chinese style urban crisis,” such as traffic congestion, the lack of public service facilities, and the loss of urban style caused by the mutual cloning of foreign modern buildings. Nowadays, in some developing countries, such as Beijing, Shanghai and Guangzhou, Mumbai and New Delhi in India, and Bangkok in Thailand, all these cities are developing rapidly, and the urban space continues to expand, attracting countless eyes around the world. But these cities have one common feature: rapidly expanding urban space, crowded traffic, bad air, and mountains of domestic garbage.With the increasing attention to resources and environment, and with the proposal and deepening of the slogan of building a resource-saving and environment-friendly “two oriented society” in China, green and environmental friendly urban centers are gradually coming into our eyes [3]. The urban center planning and design in urban construction pays more attention to the harmony between the artificial building environment and the surrounding natural ecological environment, the integration of man and nature, and the development of a reasonable urban scale, so as to achieve the optimization of social, economic, and environmental benefits.


## 2. Literature Review

Tirkolaee and others integrate ecological construction into the process of urban planning according to the perspective of management system and governance mode, take low energy consumption and low emission as the goal of urban development, and establish an analytical framework to evaluate costs and benefits through objective analysis and evaluation of emission reduction policies, so as to make the city move towards a sustainable development mode [4]. Ambole and others proposed to coordinate various issues of urban strategic development in different departments through new tools. Taking Pompeii ancient city as an example, through the linkage and coordination between well-structured planning departments, improves the supervision and feedback in the process of urban planning and finally achieves the purpose of urban planning of complex cities and resource optimization of urban construction [5]. Liu and others focused on the unequal relationship between power and urban planning participants, drew lessons from Habermas’ concept of rational communication and negotiation, proposed the role of public discourse in planning, realized that the stakeholders in planning and design played their respective roles in the planning process, and realized the purpose of communication planning with the planning idea as a consensus [6]. Singh and others proposed to apply big data to urban master planning and proposed to build an urban planning compilation system supported by big data based on the whole planning process [7]. Das and others put forward the idea of collaborative planning by analyzing the urban problems in the process of urbanization and integrated public consultation into planning [8]. Cho and others, respectively, studied the planning system and proposed the idea of “planning integration” based on business collaboration [9]. Matheson and others based on the deficiencies of current urban planning in China, through summarizing the relevant research of collaborative urban planning, put forward a theoretical system of collaborative planning from the ideological level to the technical level and to the value orientation [10]. In order to improve the integration and scalability of various urban planning platforms and improve the interoperability between platforms, Santhanalakshmi and others used the idea of service-oriented software architecture and web services and designed the integration of digital urban planning platform based on service-oriented architecture [11].For the first time, this paper attempts to model the generation of urban road network as a reinforcement learning task in which agents interact with the environment and integrates a series of cutting-edge deep reinforcement learning research results. It includes generalized dominance estimation, priority experience playback, long-term and short-term memory networks, near end strategy optimization, exploration utilization equilibrium, random network distillation, and multihead feedback. This paper presents a reinforcement learning generation algorithm for parcel road network based on strategy optimization. In addition, we designed an evaluation system considering traffic performance and morphological characteristics as the main component of the agent feedback signal in the algorithm by consulting the experts in the planning field of the partners of the team and combining the realizability of the algorithm design.


## 3. Research Methods

### 3.1. Problem Description

Block is the basic unit of urban planning, design, and research. The block is usually composed of several blocks, which have the same or related land use attributes [12]. The theory and design method of land network planning are important parts of urban master planning. The road network of the plot refers to the road network inside the plot boundary. The International Association of Traffic Engineers divides urban roads into five categories according to their functions in the road network: expressways and expressways, trunk roads, secondary trunk roads, distribution roads, and community roads, as shown in Table 1. The collecting and distributing road plays a converging role in connecting all blocks in the urban road network. It is a road with a living service function, which mainly helps solve the traffic problems in local areas of the city. From a functional point of view, it is the most consistent with the internal road network of the plot. Therefore, in this paper, the expressways, expressways, and trunk roads in the urban built roads are taken as the boundary roads of the plot; the collecting and distributing road is regarded as a real plot road. From the perspective of layout, there are four common forms of modern road network: checkerboard, ring, freestyle, and hybrid. In the traditional urban local plot design, planners comprehensively consider the land use function and geographical environment of the plot and consider a variety of possible three-dimensional forms to match the setting and adjustment of the total development volume and the distribution of indicators among multiple blocks. This generation work from indicators to forms is manually completed by designers and planners. The workload is huge, and the whole workflow needs to be revisited when indicators need to be adjusted. On the one hand, it increases the cost of urban planning and design; on the other hand, it affects the enthusiasm of planners for design deliberation and ultimately affects the design quality.Table 1
Classification standards of urban roads recommended by the International Association of Traffic Engineers.
ClassificationHighways and expresswaysTrunk roadSecondary trunk roadCollecting and distributing roadCommunity roadFunctionMotor vehicle traffic flowHigh volume traffic flow, long-distance traffic flow between clusters and inbound and outbound traffic flow in the urban areaUrban traffic flow, medium distance traffic flow between clusters and inbound and outbound trafficTrunk road system and community road distribution, access traffic and inter community trafficTraffic flow in and out of the community and within the plotRoad mileage rate (%)5~1010~205 ~ 1060~80ContinuityContinuityContinuityContinuityIt does not need to be continuous and should not cross the trunk lineContinuous between neighbors onlyGeneral length (m)Great changes8047~402334828~160933219~8047≤3219General spacing (m)64371069~3219805~1069≤805No regulationVehicle kilometers (%)40~6525~405~1010~30Road front openingDo not speakLimitLimitSecurity or specialized restrictionsSecurity restrictionsMinimum intersection spacing (m)1609~32194022019191Speed limit45~6535~4530~3525~3520~30RemarksIncrease traffic capacity and provide expresswayUrban road skeletonRestricted transit traffic feesThe main data sources and data formats involved in this study can be summarized in Table2.Table 2
Data source and format.
Data sourceSource data formatAlgorithm data formatResult data formatOSM datasetShapefile3darrayShapefile/JSONAfter quantifying, grading and weighting the road network shape, layout index, road network connectivity, road network nonlinear coefficient, and road network load balance, a comprehensive road network scheme evaluation system is established, as shown in Figure1 below.Figure 1
Evaluation index system of urban road network generation scheme.


### 3.2. Modeling of Land Network Generation

In order to retain structured information, we need to establish a spatial data structure. There are two commonly used methods of spatial representation in geographic information system (GIS): vector based method and grid based method. The former uses basic geometric elements (such as points, lines, and polygons) to represent spatial objects in Cartesian coordinates or some geographical projection coordinates (such as WGS84), while the latter quantifies the spatial region as tiny discrete grid pixels. Although the two spatial representation methods can be converted to each other, the spatial representation data structure used in problem modeling has a significant impact on subsequent research [13]. This influence is mainly reflected in the action space of agents.Figure2(a) shows an example of agent action space based on vector representation. There are two variables that determine the agent’s transition from state s1 to state s2: deflection angle α, and moving distance l. therefore, the action of the agent at time t can be expressed as a binary at=αt,lt. Figure 2(b) shows an example of agent action space based on grid representation. Since space has been quantified as tiny grid units, the action of the agent only needs to decide which direction of the adjacent grid to move in the next moment [14]. At this time, the action of the agent at time t can be represented by a discrete value, for example, αt=1 represents up, αt=2 represents right, αt=3 represents down, and αt=4 represents left.Figure 2
Comparison of two action space representations. (a) Motion space based on vector representation. (b) Action space based on grid representation.
(a)(b)Finally, this paper uses grid based spatial representation. Vectorized line and surface geometric elements are divided into grid cells by using vertical and horizontal lines. For each cell, calculate the geographic information component with the largest proportion, and then take this component as the geographic attribute of the grid. The final formation of the grid can be seen that there is no significant change in the road shape of the land after the grid and before the grid. The data structure behind is a matrix. The value of each element in the matrix represents the geographical form of the grid where it is located. Such a two-dimensional matrix can also be regarded as a single channel gray value image, which has many properties of the image, such as local features and translation invariance. Therefore, the full convolution neural network FCN can be used to extract the characteristics of the grid matrix. In addition, unlike image data, geospatial structured information is contained in the elements of grid matrix.


### 3.3. Basic Concepts of Reinforcement Learning

In the reinforcement learning task, the interaction process between the agent and the environment can be regarded as a perception action learning cycle structure, as shown in Figure3. This involves three basic concepts: state, action, and feedback. State refers to the external environment information of the agent; action refers to the behavior that the agent decides to take according to the current state; feedback refers to the feedback of the environment to the action of the agent. The feedback can be positive (also known as reward at this time) or negative (also known as punishment at this time).Figure 3
RL agent environment interaction model.The agent decides the actionat to be taken at the next moment based on the current state st and the feedback rt of the environment to the previous action at−1. Under the action of at, the agent transfers from state st to the next state st+1, and such a sequential decision-making process can be formally expressed as a finite markov decision process (FMDP). A specific FMDP can be defined by one-step dynamics of its state, action, and environment [15]. Given any state and action s and a and the probability of each possible next state s′ and reward r, the available state transition probability is as follows:
(1)ps′,rs,a=Prst+1=s′,rt+1=rst=s′,at=a.This means that the state transition probability completely describes the dynamic characteristics of the environment, that is, the probability of each possible value ofst and rt depends only on the previous state st−1 and the previous action at−1 and is completely independent of the earlier states and actions. This property is called Markov property. Although reinforcement learning tasks usually do not require agents to be strictly Markov in practical applications, the learning process of tasks with this characteristic can often converge more stably.In reinforcement learning, the learning goal of the agent is to maximize the expected return, which is recorded asGt, which is defined as the total return weighted by the discount coefficient received in the future, that is calculated as follows:
(2)Gt=rt+1+γrt+2+γ2rt+3+⋯=,where 0≤γ≤1 is the discount coefficient. The discount coefficient affects the emphasis tendency of agent behavior decision-making: When γ is close to 0, the agent is more concerned about how to maximize the current income; at this time, the agent is easy to show “shortsightedness”; when γ is close to 1, the agent is more concerned about how to maximize future benefits. At this time, the agent shows “foresight”.In practice, different behaviors of agents will bring different rewards. The behavior of an agent can be defined as a mapping of the probability distribution from a state to an action, which is called a policy, usually represented by the notationπ, and is defined as follows:
(3)πas=pAt=aSt=s.The goal of reinforcement learning is to find the optimal strategyπ∗ of the agent so that the agent can get the highest expected return under this strategy, as shown as follows:
(4)π∗=argmaxπEGπ.Expected return, also known as value, is a function of state (or state and action binary), which is used to estimate the return of the current agent in a given state (at this time, the strategy is also determined). It is defined as follows:(5)vπs=EπGtSt=s=Eπ∑k=0∞γkRt+k+1St=s.The functionvπ at this time is called the state value function of the strategy π, which represents the value of the current state of the agent.Given the state and action binary, similarly, the value of actiona taken by the agent in state s under strategy π can be recorded as qπs,a, which is defined as follows:
(6)qπs,a=EπGtSt=s,At=a=Eπ∑k=0∞γkRt+k+1St=s,At=a.The functionqπ at this time is called the action value function of strategy π, which represents the value of action a selected by the agent in the current state.Combining Equations (2) and (5), we can get the recurrence relationship of the value function of the following equation:
(7)vπs=EπGtSt=s=EπRt+1+γGt+1St=s=∑aπas∑s′∑rps′,rs,ar+γEπGt+1St+1=s′=∑aπas∑s′∑rps′,rs,ar+γvπs′.Equation (7) is called the Bellman equation of MDP, which is the basis of various methods such as calculation, approximation, and learning.


### 3.4. Generation Model and Network Structure of Urban road Network

The urban road network generation model proposed in this paper is based on the reinforcement learning framework of “actor evaluator.” The core of the actuator is composed of a strategy network. Its input is the state featurest extracted from the input state Gt by the feedforward feature network, and its output is the action at taken by the agent at the current time. The environment determines the feedback rt of the environment to the agent and the state Gt+1 of the agent at the next time according to the dynamic characteristics pst+1,rtst,at of FMDP. The core of the evaluator is composed of a value network, which is used to estimate the dominance function At, thereby reducing the training variance of the actuator [16]. The experience cache is a queue, which stores historical experience samples of the interaction between the agent and the environment, and takes the sample fragments at continuous time llstm as the unit. When the environment detects that the current state of the agent triggers the termination condition of FMDP (i.e., moving to the plot boundary), it means the end of a complete FMDP round (i.e., the generation of a road). The experience τ=s1,a1,r1,s2,a2,r2,⋯,sT,aT,rT of this round will be stored in the experience cache, and the advantage function of each moment will be calculated in the experience cache.Figure4 shows the internal network structure of feature network, strategy network, and value network. The feature network is composed of FCN composed of six CNN layers, and there is a BN layer between each CNN layer for batch normalization. The function of FCN is to take the input G as three channel image data for feature extraction. There are a reshape operation layer and a squeeze operation layer at both ends of the FCN, which are mainly used to process the data dimension, down sample the input data into a size of 64×64, and convert the output 512 channel features into a one-dimensional vector with a length of 512. The feature state processed by the feature network is recorded as s.Figure 4
Schematic diagram of network structure of urban land network generation based on strategy optimization.The strategy network and the value network share an LSTM network as a feedforward network. The function of LSTM is to learn the time correlation between the front and rear states in the experience segment. The time stepllstm of the LSTM is set to 16. In this step, the input of the LSTM is the state characteristic sτ of the τ-th sample in the experience segment and the implicit state e of the LSTM output in the previous step. Its output is as follows:
(8)hτ,cτ=LSTMsτ,hτ−1,cτ−1,where cτ is the cellstate of LSTM in step τ. Both h0 and c0 are initialized with all zero tensors.Both the policy network and the value network are composed of three layers of FC. The input of the policy network and the value network is the hidden stateh of the LSTM output, and the output of the policy network is the possibility of each action selection. After passing through the softmax layer, it is normalized to the discrete action probability distribution πθ=pnull,pup,pright,pdown,pleft, and the actions selected from πθ enter the mask layer [17]. The mask layer is used to implement feasibility rules that is shown in the following equation:
(9)patst,G0=softmaxReLUFCh⊗Fst,G0.Actions that do not conform toF will be output as 0 after passing through the mask layer, that is, stay in place. Since this action will not produce any feedback, and its advantage function is A=0, such an experience segment in the learning process will not help the strategy update towards the gradient of the objective function, so the agent will learn to avoid decisions that do not conform to F. The output of the value network is the value function estimation v^st of the current state, which is used for the calculation of the advantage function in the next stage.What distinguishes reinforcement learning from supervised learning and unsupervised learning is that the learning samples are generated by the interaction between agents and the environment, rather than the training data provided in advance [18]. In the process of strategy optimization, the training samples generated by the interaction between the agent and the environment will be closer and closer to the learning goal. Such a process of data enhancement is also the origin of the word “reinforcement.” Therefore, the first thing to introduce is the generation algorithm of learning samples (i.e., roads), as shown in Algorithm 1.Algorithm 1: Urban parcel road generation algorithm. Road generation algorithm of urban land.
Input Initial environmentG0, initial state s0Output Round sampleτ=s0,a0,r0,v0,π0,s1,a1,r1,v1,π1,⋯,sT,aT,rT,vT,πT1 Initialization round length t = 02 Initialize sample queueτ, status queue s, feedback queue r, value queue v, policy queue π3 Initialize the current states=s0, feedback r=0, whether to terminate done=False4 fort=1,2,⋯,Tmax do5  ifdone==True then6   break7  else8   Feature network extracts current state featuresf=FCNs,G09   Update LSTM featuresht,ct=LSTMs,ht−1,ct−1.10   Policy network output current policy distributionπ=FCπht11   Softmax layer selects the current actiona=softmaxπ12   Mask layer for feasibility rule testa=maska13   Value network outputs current status valuev=FCVht14   Adds to s, r to r, v to v15   Interact with the environment and update s,r,done=env(a)16  end if17 end for18 returnτThe samples of the interaction between the agent and the environment are not directly used for model training. In order to break the correlation between the samples, the samples are stored in the experience pool, and the advantage function and cumulative return are calculated in the experience pool. In order to improve the parallel operation efficiency of LSTM, the samples stored in the experience pool are unified into sample segments with lengthllstm, and the segments with length less than llstm are zeroed. The data structure of the experience pool is a queue. When the queue capacity reaches the maximum, samples are deleted from the end of the queue [19]. The storage algorithm of experience pool is shown in Algorithm 2.Algorithm 2: Experience pool storage algorithm.
Input Round sampleτ=s0,a0,r0,v0,π0,s1,a1,r1,v1,π1,⋯,sT,aT,rT,vT,πTTask Store the round samples into the experience pool and calculate the advantage function1 Obtain the length of round samplesT, super parameters γ and λ, the current capacity of experience pool length, and the maximum capacity of experience pool length_max2 Initialization TD errorδ=0, advantage function A=0, cumulative return G=03 fort=T−1,T−2,⋯,0 do4δt=rt+1+γνt+1−vt5At=δt+γλAt+16Gt=rt+γGi+17 endfor8 Obtain experience samplese=s0,a0,A0,G0,π0,s1,a1,A1,G1,π1,⋯,sT,aT,AT,GT,πT9 Calculate the number of sample segmentsN=llstmT10 Slicee according to N to get sample segment e0,e1,⋯,eN−1, and zero the sample segment with length less than llstm11 iflength+N>length_maxthen12  Delete the last length+N-length_max sample segments from the end of the experience pool queue13 endif14 returnDuring training, training samples are collected from the experience pool according to the priority distribution based on the dominance function. In order to speed up the calculation, batch size samples are collected at a time for parallel training. The optimization algorithm used in training is Adam algorithm.The feedback functions used in this paper can be summarized in Table3. Among them, the feedback head weight and feedback term weight are the optimal numerical combination obtained through many experiments [20].Table 3
Feedback structure of urban land network generation problems.
Feedback itemFeedback sparsityFeedback item weightFeedback type (head)Feedback head weightLand base incomerbaseDenser0.05Road network densityrdSparse1.5Traffic performance1Traffic accessibilityrtaSparse0.02Intermediary centralityrbcSparse1.5Deflection anglerturnDenser1Bending degreerbendDenser0.99Intersection anglercrossDenser0.9Morphological character1Balanced distribution of entrances and exitsrgateSparse0.1Distribution balance of sub plotsrblockSparse0.002Rnd prediction errorrrndSparse1Internal incentives0.005


## 3.1. Problem Description

Block is the basic unit of urban planning, design, and research. The block is usually composed of several blocks, which have the same or related land use attributes [12]. The theory and design method of land network planning are important parts of urban master planning. The road network of the plot refers to the road network inside the plot boundary. The International Association of Traffic Engineers divides urban roads into five categories according to their functions in the road network: expressways and expressways, trunk roads, secondary trunk roads, distribution roads, and community roads, as shown in Table 1. The collecting and distributing road plays a converging role in connecting all blocks in the urban road network. It is a road with a living service function, which mainly helps solve the traffic problems in local areas of the city. From a functional point of view, it is the most consistent with the internal road network of the plot. Therefore, in this paper, the expressways, expressways, and trunk roads in the urban built roads are taken as the boundary roads of the plot; the collecting and distributing road is regarded as a real plot road. From the perspective of layout, there are four common forms of modern road network: checkerboard, ring, freestyle, and hybrid. In the traditional urban local plot design, planners comprehensively consider the land use function and geographical environment of the plot and consider a variety of possible three-dimensional forms to match the setting and adjustment of the total development volume and the distribution of indicators among multiple blocks. This generation work from indicators to forms is manually completed by designers and planners. The workload is huge, and the whole workflow needs to be revisited when indicators need to be adjusted. On the one hand, it increases the cost of urban planning and design; on the other hand, it affects the enthusiasm of planners for design deliberation and ultimately affects the design quality.Table 1
Classification standards of urban roads recommended by the International Association of Traffic Engineers.
ClassificationHighways and expresswaysTrunk roadSecondary trunk roadCollecting and distributing roadCommunity roadFunctionMotor vehicle traffic flowHigh volume traffic flow, long-distance traffic flow between clusters and inbound and outbound traffic flow in the urban areaUrban traffic flow, medium distance traffic flow between clusters and inbound and outbound trafficTrunk road system and community road distribution, access traffic and inter community trafficTraffic flow in and out of the community and within the plotRoad mileage rate (%)5~1010~205 ~ 1060~80ContinuityContinuityContinuityContinuityIt does not need to be continuous and should not cross the trunk lineContinuous between neighbors onlyGeneral length (m)Great changes8047~402334828~160933219~8047≤3219General spacing (m)64371069~3219805~1069≤805No regulationVehicle kilometers (%)40~6525~405~1010~30Road front openingDo not speakLimitLimitSecurity or specialized restrictionsSecurity restrictionsMinimum intersection spacing (m)1609~32194022019191Speed limit45~6535~4530~3525~3520~30RemarksIncrease traffic capacity and provide expresswayUrban road skeletonRestricted transit traffic feesThe main data sources and data formats involved in this study can be summarized in Table2.Table 2
Data source and format.
Data sourceSource data formatAlgorithm data formatResult data formatOSM datasetShapefile3darrayShapefile/JSONAfter quantifying, grading and weighting the road network shape, layout index, road network connectivity, road network nonlinear coefficient, and road network load balance, a comprehensive road network scheme evaluation system is established, as shown in Figure1 below.Figure 1
Evaluation index system of urban road network generation scheme.


## 3.2. Modeling of Land Network Generation

In order to retain structured information, we need to establish a spatial data structure. There are two commonly used methods of spatial representation in geographic information system (GIS): vector based method and grid based method. The former uses basic geometric elements (such as points, lines, and polygons) to represent spatial objects in Cartesian coordinates or some geographical projection coordinates (such as WGS84), while the latter quantifies the spatial region as tiny discrete grid pixels. Although the two spatial representation methods can be converted to each other, the spatial representation data structure used in problem modeling has a significant impact on subsequent research [13]. This influence is mainly reflected in the action space of agents.Figure2(a) shows an example of agent action space based on vector representation. There are two variables that determine the agent’s transition from state s1 to state s2: deflection angle α, and moving distance l. therefore, the action of the agent at time t can be expressed as a binary at=αt,lt. Figure 2(b) shows an example of agent action space based on grid representation. Since space has been quantified as tiny grid units, the action of the agent only needs to decide which direction of the adjacent grid to move in the next moment [14]. At this time, the action of the agent at time t can be represented by a discrete value, for example, αt=1 represents up, αt=2 represents right, αt=3 represents down, and αt=4 represents left.Figure 2
Comparison of two action space representations. (a) Motion space based on vector representation. (b) Action space based on grid representation.
(a)(b)Finally, this paper uses grid based spatial representation. Vectorized line and surface geometric elements are divided into grid cells by using vertical and horizontal lines. For each cell, calculate the geographic information component with the largest proportion, and then take this component as the geographic attribute of the grid. The final formation of the grid can be seen that there is no significant change in the road shape of the land after the grid and before the grid. The data structure behind is a matrix. The value of each element in the matrix represents the geographical form of the grid where it is located. Such a two-dimensional matrix can also be regarded as a single channel gray value image, which has many properties of the image, such as local features and translation invariance. Therefore, the full convolution neural network FCN can be used to extract the characteristics of the grid matrix. In addition, unlike image data, geospatial structured information is contained in the elements of grid matrix.


## 3.3. Basic Concepts of Reinforcement Learning

In the reinforcement learning task, the interaction process between the agent and the environment can be regarded as a perception action learning cycle structure, as shown in Figure3. This involves three basic concepts: state, action, and feedback. State refers to the external environment information of the agent; action refers to the behavior that the agent decides to take according to the current state; feedback refers to the feedback of the environment to the action of the agent. The feedback can be positive (also known as reward at this time) or negative (also known as punishment at this time).Figure 3
RL agent environment interaction model.The agent decides the actionat to be taken at the next moment based on the current state st and the feedback rt of the environment to the previous action at−1. Under the action of at, the agent transfers from state st to the next state st+1, and such a sequential decision-making process can be formally expressed as a finite markov decision process (FMDP). A specific FMDP can be defined by one-step dynamics of its state, action, and environment [15]. Given any state and action s and a and the probability of each possible next state s′ and reward r, the available state transition probability is as follows:
(1)ps′,rs,a=Prst+1=s′,rt+1=rst=s′,at=a.This means that the state transition probability completely describes the dynamic characteristics of the environment, that is, the probability of each possible value ofst and rt depends only on the previous state st−1 and the previous action at−1 and is completely independent of the earlier states and actions. This property is called Markov property. Although reinforcement learning tasks usually do not require agents to be strictly Markov in practical applications, the learning process of tasks with this characteristic can often converge more stably.In reinforcement learning, the learning goal of the agent is to maximize the expected return, which is recorded asGt, which is defined as the total return weighted by the discount coefficient received in the future, that is calculated as follows:
(2)Gt=rt+1+γrt+2+γ2rt+3+⋯=,where 0≤γ≤1 is the discount coefficient. The discount coefficient affects the emphasis tendency of agent behavior decision-making: When γ is close to 0, the agent is more concerned about how to maximize the current income; at this time, the agent is easy to show “shortsightedness”; when γ is close to 1, the agent is more concerned about how to maximize future benefits. At this time, the agent shows “foresight”.In practice, different behaviors of agents will bring different rewards. The behavior of an agent can be defined as a mapping of the probability distribution from a state to an action, which is called a policy, usually represented by the notationπ, and is defined as follows:
(3)πas=pAt=aSt=s.The goal of reinforcement learning is to find the optimal strategyπ∗ of the agent so that the agent can get the highest expected return under this strategy, as shown as follows:
(4)π∗=argmaxπEGπ.Expected return, also known as value, is a function of state (or state and action binary), which is used to estimate the return of the current agent in a given state (at this time, the strategy is also determined). It is defined as follows:(5)vπs=EπGtSt=s=Eπ∑k=0∞γkRt+k+1St=s.The functionvπ at this time is called the state value function of the strategy π, which represents the value of the current state of the agent.Given the state and action binary, similarly, the value of actiona taken by the agent in state s under strategy π can be recorded as qπs,a, which is defined as follows:
(6)qπs,a=EπGtSt=s,At=a=Eπ∑k=0∞γkRt+k+1St=s,At=a.The functionqπ at this time is called the action value function of strategy π, which represents the value of action a selected by the agent in the current state.Combining Equations (2) and (5), we can get the recurrence relationship of the value function of the following equation:
(7)vπs=EπGtSt=s=EπRt+1+γGt+1St=s=∑aπas∑s′∑rps′,rs,ar+γEπGt+1St+1=s′=∑aπas∑s′∑rps′,rs,ar+γvπs′.Equation (7) is called the Bellman equation of MDP, which is the basis of various methods such as calculation, approximation, and learning.


## 3.4. Generation Model and Network Structure of Urban road Network

The urban road network generation model proposed in this paper is based on the reinforcement learning framework of “actor evaluator.” The core of the actuator is composed of a strategy network. Its input is the state featurest extracted from the input state Gt by the feedforward feature network, and its output is the action at taken by the agent at the current time. The environment determines the feedback rt of the environment to the agent and the state Gt+1 of the agent at the next time according to the dynamic characteristics pst+1,rtst,at of FMDP. The core of the evaluator is composed of a value network, which is used to estimate the dominance function At, thereby reducing the training variance of the actuator [16]. The experience cache is a queue, which stores historical experience samples of the interaction between the agent and the environment, and takes the sample fragments at continuous time llstm as the unit. When the environment detects that the current state of the agent triggers the termination condition of FMDP (i.e., moving to the plot boundary), it means the end of a complete FMDP round (i.e., the generation of a road). The experience τ=s1,a1,r1,s2,a2,r2,⋯,sT,aT,rT of this round will be stored in the experience cache, and the advantage function of each moment will be calculated in the experience cache.Figure4 shows the internal network structure of feature network, strategy network, and value network. The feature network is composed of FCN composed of six CNN layers, and there is a BN layer between each CNN layer for batch normalization. The function of FCN is to take the input G as three channel image data for feature extraction. There are a reshape operation layer and a squeeze operation layer at both ends of the FCN, which are mainly used to process the data dimension, down sample the input data into a size of 64×64, and convert the output 512 channel features into a one-dimensional vector with a length of 512. The feature state processed by the feature network is recorded as s.Figure 4
Schematic diagram of network structure of urban land network generation based on strategy optimization.The strategy network and the value network share an LSTM network as a feedforward network. The function of LSTM is to learn the time correlation between the front and rear states in the experience segment. The time stepllstm of the LSTM is set to 16. In this step, the input of the LSTM is the state characteristic sτ of the τ-th sample in the experience segment and the implicit state e of the LSTM output in the previous step. Its output is as follows:
(8)hτ,cτ=LSTMsτ,hτ−1,cτ−1,where cτ is the cellstate of LSTM in step τ. Both h0 and c0 are initialized with all zero tensors.Both the policy network and the value network are composed of three layers of FC. The input of the policy network and the value network is the hidden stateh of the LSTM output, and the output of the policy network is the possibility of each action selection. After passing through the softmax layer, it is normalized to the discrete action probability distribution πθ=pnull,pup,pright,pdown,pleft, and the actions selected from πθ enter the mask layer [17]. The mask layer is used to implement feasibility rules that is shown in the following equation:
(9)patst,G0=softmaxReLUFCh⊗Fst,G0.Actions that do not conform toF will be output as 0 after passing through the mask layer, that is, stay in place. Since this action will not produce any feedback, and its advantage function is A=0, such an experience segment in the learning process will not help the strategy update towards the gradient of the objective function, so the agent will learn to avoid decisions that do not conform to F. The output of the value network is the value function estimation v^st of the current state, which is used for the calculation of the advantage function in the next stage.What distinguishes reinforcement learning from supervised learning and unsupervised learning is that the learning samples are generated by the interaction between agents and the environment, rather than the training data provided in advance [18]. In the process of strategy optimization, the training samples generated by the interaction between the agent and the environment will be closer and closer to the learning goal. Such a process of data enhancement is also the origin of the word “reinforcement.” Therefore, the first thing to introduce is the generation algorithm of learning samples (i.e., roads), as shown in Algorithm 1.Algorithm 1: Urban parcel road generation algorithm. Road generation algorithm of urban land.
Input Initial environmentG0, initial state s0Output Round sampleτ=s0,a0,r0,v0,π0,s1,a1,r1,v1,π1,⋯,sT,aT,rT,vT,πT1 Initialization round length t = 02 Initialize sample queueτ, status queue s, feedback queue r, value queue v, policy queue π3 Initialize the current states=s0, feedback r=0, whether to terminate done=False4 fort=1,2,⋯,Tmax do5  ifdone==True then6   break7  else8   Feature network extracts current state featuresf=FCNs,G09   Update LSTM featuresht,ct=LSTMs,ht−1,ct−1.10   Policy network output current policy distributionπ=FCπht11   Softmax layer selects the current actiona=softmaxπ12   Mask layer for feasibility rule testa=maska13   Value network outputs current status valuev=FCVht14   Adds to s, r to r, v to v15   Interact with the environment and update s,r,done=env(a)16  end if17 end for18 returnτThe samples of the interaction between the agent and the environment are not directly used for model training. In order to break the correlation between the samples, the samples are stored in the experience pool, and the advantage function and cumulative return are calculated in the experience pool. In order to improve the parallel operation efficiency of LSTM, the samples stored in the experience pool are unified into sample segments with lengthllstm, and the segments with length less than llstm are zeroed. The data structure of the experience pool is a queue. When the queue capacity reaches the maximum, samples are deleted from the end of the queue [19]. The storage algorithm of experience pool is shown in Algorithm 2.Algorithm 2: Experience pool storage algorithm.
Input Round sampleτ=s0,a0,r0,v0,π0,s1,a1,r1,v1,π1,⋯,sT,aT,rT,vT,πTTask Store the round samples into the experience pool and calculate the advantage function1 Obtain the length of round samplesT, super parameters γ and λ, the current capacity of experience pool length, and the maximum capacity of experience pool length_max2 Initialization TD errorδ=0, advantage function A=0, cumulative return G=03 fort=T−1,T−2,⋯,0 do4δt=rt+1+γνt+1−vt5At=δt+γλAt+16Gt=rt+γGi+17 endfor8 Obtain experience samplese=s0,a0,A0,G0,π0,s1,a1,A1,G1,π1,⋯,sT,aT,AT,GT,πT9 Calculate the number of sample segmentsN=llstmT10 Slicee according to N to get sample segment e0,e1,⋯,eN−1, and zero the sample segment with length less than llstm11 iflength+N>length_maxthen12  Delete the last length+N-length_max sample segments from the end of the experience pool queue13 endif14 returnDuring training, training samples are collected from the experience pool according to the priority distribution based on the dominance function. In order to speed up the calculation, batch size samples are collected at a time for parallel training. The optimization algorithm used in training is Adam algorithm.The feedback functions used in this paper can be summarized in Table3. Among them, the feedback head weight and feedback term weight are the optimal numerical combination obtained through many experiments [20].Table 3
Feedback structure of urban land network generation problems.
Feedback itemFeedback sparsityFeedback item weightFeedback type (head)Feedback head weightLand base incomerbaseDenser0.05Road network densityrdSparse1.5Traffic performance1Traffic accessibilityrtaSparse0.02Intermediary centralityrbcSparse1.5Deflection anglerturnDenser1Bending degreerbendDenser0.99Intersection anglercrossDenser0.9Morphological character1Balanced distribution of entrances and exitsrgateSparse0.1Distribution balance of sub plotsrblockSparse0.002Rnd prediction errorrrndSparse1Internal incentives0.005


## 4. Result Analysis

### 4.1. Experimental Setup

The software and hardware environment of the experiment in this paper is shown in Table4. The experimental code in this paper is composed of python3.7 and a series of related libraries. The GPU operation part code is written by PyTorch and runs on the server with Linux operating system, Inter(R)E5-2620v4CPU and Nvidia2080TiGPU. In the initial stage of the experiment, we used a single GPU to train the model, and it took a total of 4×105 iterations to converge the model, which took 12 hours. In order to speed up the model training process, we used the GPU cluster of the laboratory, called 10 Nvidia2080TiGPU, and used PyTorch’s distributed data parallel (DDP) mechanism to train the model distributed [21].Table 4
Experimental software and hardware environment.
CPUGPUOperating systemPrograming languageRunning timeIntel(R)E5-2620v4@2.10GHzNvidia2080tiLinuxPython/PyTorch12 h/order GPU1.5 h/DDPDDP distributed training has the following advantages: (1) It eases the Gil limitation. In DDP mode,n processes will be started, and each process loads a model on a graphics card. The parameters of these models are the same in value. (2) Ring reduce acceleration. Because the communication cost increases linearly with the number of GPU between distributed machines in the system. In contrast, the communication cost of ring reduce algorithm is constant, independent of the number of GPU in the system, and completely determined by the slowest connection between GPU in the system. The GPU in the DDP ring are arranged in a logical ring. Each GPU should have a left neighbor and a right neighbor; it will only send data to its right neighbor and receive data from its left neighbor. During model training, each process communicates with other processes through a method called ring reduce to exchange their own gradients, so as to obtain the gradients of all processes [22]. (3) Each process updates its parameters with the average gradient. Because the initial parameters and update gradient of each process are consistent, the updated parameters are also exactly the same.After DDP acceleration, the time of model training to convergence is reduced from 12 hours to 1.5 hours.A summary of all the super parameters mentioned above and involved in the experiment is shown in Table5.Table 5
Experimental super parameters.
Super parameter nameSuper parameter valueBatch sizeB128Number of sampling rounds per stepE20Number of policy iterations per stepI32LSTM lengthllstm200Experience pool capacitylmax100000GAE estimation stepT200GAEγ0.99GAEλ0.95PPO cutoff factorεppo0.2Value loss itemLV weight c10.25Entropy gainLS weight c20.01 → 0.001Learning rate Flr5e-5 → 1e-6Adamβ10.9Adamβ20.999x⟶y means that the value of the super parameter decays smoothly from xto y with the training process.We randomly selected a plot of land in a city, including residential land blocks such as Park A, Park B, and Park C. During training, the deflection angle feedbackrturn is set to rturn zero, the road network density target is set to Dtarget=0.075, the bending degree feedback weight is rbend=0.95, and the entrance initialization over parameter is set to ρ=0.2. All results were presented using ArcGis, a planning domain software [23]. The results generated by this method basically conform to the design specifications of urban road network. In terms of morphology, it can generate not only grid road network similar to the real road network, but also branch road network and other forms of road network. It has certain practical value and can assist planners to complete the design of urban road network more efficiently [24].From the changes of various feedback signals during the experiment, we can know that with the progress of training, the agent converges towards the goal of our design on all feedback terms. This shows that agents can transform the value system we designed into behavioral strategies through the algorithm model proposed in this paper. After 4×105 iterations in the training process, the model is derived and used to generate 10000 road network schemes. According to the road network generation scheme evaluation system in Figure 1, the mean and variance of the key indicators of these schemes are calculated, respectively, and the scores of 153 internal road networks of plots with similar area and shape and the same land function in the real case base on the same indicators are calculated. The comparison results are shown in Figures 5–7. From the performance of these indicators, it can be seen that the road network generated by the model has little difference in traffic performance from the real road network. In terms of morphological characteristics, because the real road network is more diverse, the variance of subplot area distribution and entrance and exit distance distribution is larger than the result generated by the model. Generally speaking, the road network generated by this model conforms to the basic norms of urban design, has great similarity with the real road network, and has certain application value [25].Figure 5
Performance comparison of real cases and generated results on the average value of generated results.Figure 6
Comparison of the performance of real cases and generated results in the variance of generated results.Figure 7
Performance comparison of real cases and generated results on the average value of real cases.


## 4.1. Experimental Setup

The software and hardware environment of the experiment in this paper is shown in Table4. The experimental code in this paper is composed of python3.7 and a series of related libraries. The GPU operation part code is written by PyTorch and runs on the server with Linux operating system, Inter(R)E5-2620v4CPU and Nvidia2080TiGPU. In the initial stage of the experiment, we used a single GPU to train the model, and it took a total of 4×105 iterations to converge the model, which took 12 hours. In order to speed up the model training process, we used the GPU cluster of the laboratory, called 10 Nvidia2080TiGPU, and used PyTorch’s distributed data parallel (DDP) mechanism to train the model distributed [21].Table 4
Experimental software and hardware environment.
CPUGPUOperating systemPrograming languageRunning timeIntel(R)E5-2620v4@2.10GHzNvidia2080tiLinuxPython/PyTorch12 h/order GPU1.5 h/DDPDDP distributed training has the following advantages: (1) It eases the Gil limitation. In DDP mode,n processes will be started, and each process loads a model on a graphics card. The parameters of these models are the same in value. (2) Ring reduce acceleration. Because the communication cost increases linearly with the number of GPU between distributed machines in the system. In contrast, the communication cost of ring reduce algorithm is constant, independent of the number of GPU in the system, and completely determined by the slowest connection between GPU in the system. The GPU in the DDP ring are arranged in a logical ring. Each GPU should have a left neighbor and a right neighbor; it will only send data to its right neighbor and receive data from its left neighbor. During model training, each process communicates with other processes through a method called ring reduce to exchange their own gradients, so as to obtain the gradients of all processes [22]. (3) Each process updates its parameters with the average gradient. Because the initial parameters and update gradient of each process are consistent, the updated parameters are also exactly the same.After DDP acceleration, the time of model training to convergence is reduced from 12 hours to 1.5 hours.A summary of all the super parameters mentioned above and involved in the experiment is shown in Table5.Table 5
Experimental super parameters.
Super parameter nameSuper parameter valueBatch sizeB128Number of sampling rounds per stepE20Number of policy iterations per stepI32LSTM lengthllstm200Experience pool capacitylmax100000GAE estimation stepT200GAEγ0.99GAEλ0.95PPO cutoff factorεppo0.2Value loss itemLV weight c10.25Entropy gainLS weight c20.01 → 0.001Learning rate Flr5e-5 → 1e-6Adamβ10.9Adamβ20.999x⟶y means that the value of the super parameter decays smoothly from xto y with the training process.We randomly selected a plot of land in a city, including residential land blocks such as Park A, Park B, and Park C. During training, the deflection angle feedbackrturn is set to rturn zero, the road network density target is set to Dtarget=0.075, the bending degree feedback weight is rbend=0.95, and the entrance initialization over parameter is set to ρ=0.2. All results were presented using ArcGis, a planning domain software [23]. The results generated by this method basically conform to the design specifications of urban road network. In terms of morphology, it can generate not only grid road network similar to the real road network, but also branch road network and other forms of road network. It has certain practical value and can assist planners to complete the design of urban road network more efficiently [24].From the changes of various feedback signals during the experiment, we can know that with the progress of training, the agent converges towards the goal of our design on all feedback terms. This shows that agents can transform the value system we designed into behavioral strategies through the algorithm model proposed in this paper. After 4×105 iterations in the training process, the model is derived and used to generate 10000 road network schemes. According to the road network generation scheme evaluation system in Figure 1, the mean and variance of the key indicators of these schemes are calculated, respectively, and the scores of 153 internal road networks of plots with similar area and shape and the same land function in the real case base on the same indicators are calculated. The comparison results are shown in Figures 5–7. From the performance of these indicators, it can be seen that the road network generated by the model has little difference in traffic performance from the real road network. In terms of morphological characteristics, because the real road network is more diverse, the variance of subplot area distribution and entrance and exit distance distribution is larger than the result generated by the model. Generally speaking, the road network generated by this model conforms to the basic norms of urban design, has great similarity with the real road network, and has certain application value [25].Figure 5
Performance comparison of real cases and generated results on the average value of generated results.Figure 6
Comparison of the performance of real cases and generated results in the variance of generated results.Figure 7
Performance comparison of real cases and generated results on the average value of real cases.


## 5. Conclusion

The combination of urban design and artificial intelligence is an inevitable trend in the process of urbanization in China from quantity growth to quality improvement. The research of this paper focuses on the important basic work in intelligent city design, that is, the generation of urban land network. In the traditional design method, this part of the work is mainly completed manually by the design planners, and the workload is huge. In order to speed up the whole process of urban design and improve the efficiency of urban design, researchers try to apply artificial intelligence technology to the generation of road network, in order to make this work large-scale, automatic, and intelligent. The existing research has shortcomings in practical application. The schemes generated based on computer modeling have large convergence, many repeated schemes, and insufficient diversity; the road generated based on evolutionary algorithm lacks internal logic, and the generated scheme is chaotic; the method based on image learning is a black box tool, which cannot introduce expert prior knowledge, and has insufficient explicability and expansibility; the method based on interactive software does not solve the problem of manual dependence, and cannot significantly improve the efficiency of urban design. The research goal of this paper is to assist planners to design urban plots more efficiently and try to introduce behavioral intelligence based on deep reinforcement learning into the generation of urban plot road network for the first time, in order to combine the advantages of image learning and interactive design methods. This paper expects to realize the generation of urban parcel road network with good interpretability and interactivity, which can not only automatically generate road network schemes with sufficient diversity, but also retain the structural information between geographical elements.



---

*Source: 8994088-2022-09-26.xml*