# Method for Effectively Utilizing Node Energy of WSN for Coal Mine Robot

Authors: Xiliang Ma; Ruiqing Mao
Journal: Journal of Robotics
 (2018)

---

## Abstract

As a special application scenario, the data collected by wireless sensor networks of coal mine robot is from vital and dangerous environment. Therefore, the nodes need to work as long as possible. In order to efficiently utilize the node energy of wireless sensor network, this paper proposes a self-organizing routing method for wireless sensor networks based onQ-learning. The method takes many factors into account, such as the hop number, distance, residual energy, and node communication loss and energy. Each node of the wireless sensor networks is mapped into an Agent. Periodic training is carried out to optimize the route choice. Each Agent chooses the optimal path for data transmission according to the calculatedQ evaluation value. Simulation results show that the self-organizing sensor networks usingQ-learning can balance the energy consumption of the nodes and prolong the lifetime of the networks.


---

## Body

## 1. Introduction

Coal mine environment is often threatened by toxic gases and high temperature, so coal mine robots [1] often replace human to enter the pit to carry on the detection or the rescue tasks. Coal mine robots need to sense the states of themselves, as well as the environment of the underground. So a variety of sensors need to be deployed outside coal mine robots, such as the sensors to detect temperature, humidity, wind speed, wind direction, wind pressure, light intensity, coal dust and toxic gases, distance, the speed of the robot’s left and right tracks, and pitch angle. There are several disadvantages with wired sensors. First, cables for power and signal connections are required, and wiring is a piece of tedious work. Second, temporary data collection cannot be performed and the system expansion is inconvenient., Third, the robot shells need to provide threading holes and location holes, which will reduce the stiffness and strength of the robot shells and reduce the explosion-proof performance of the coal mine robots. Vibration or collision during robots’ motion affects the connection and joint quality of the cable. This will affect the reliability of the data detection and even cause coal mine robots not to work properly in the underground. Wireless sensor network technologies can make up for these deficiencies.Different from the general wireless sensor networks (WSNs) [2], a wireless sensor network deployed on a coal mine robot consists of a number of small sensor nodes. The distance between the nodes is relatively small. As coordinators and data collectors, these nodes transmit the data to the sink nodes and monitor the environment and the state parameters of the robot in real time. Each node has different data types and must meet the requirements of explosion-proof or intrinsic safety. So the nodes are usually powered by batteries. The data is collected by the nodes if from vital and dangerous environments. Therefore, the nodes need to work as long as possible. As a special application scenario, how to efficiently utilize the node energy of coal mine robot WSN is a difficult problem. Node energy consumption is mostly generated by sensor communications. In order to prolong the lifetime of the network, this paper designs a routing mechanism based on reinforcement learning and applies it to wireless body area networks. This paper also studies the energy efficiency, which further improves the energy efficiency and prolongs the network life cycle.


## 2. Wireless Sensor Networks for Coal Mine Robot

### 2.1. A Wireless Sensor Network Model for Coal Mine Robot

The coal mine robot wireless sensor network has a single layer structure, which can be represented by a directed graphGV,A. The graph is shown in Figure 1, where V represents the sensor node set, V1 is a sink node, and the wireless sensor network is wired to the coal mine robot control system by the sink node. Each node is fixed relative to the coal mine robot. The nodes transmit messages to the sink node in a direct or multihop manner.A represents a set of directed arcs, each of which represents the connection of two nodes in V that can communicate directly. The label beside an edge in the graph represents the corresponding path weight wij. The path with a smaller weight means that the channel has a better quality. The nodes on this path have more residual energy and the hop consumes less energy.Figure 1
Model of wireless sensor network for coal mine robots.


### 2.2. Node Energy Model

The energy consumption of wireless sensor network nodes is divided into 3 parts: perception, communication, and data processing. In this paper, we mainly consider the energy consumption of data communication, that is, the energy consumed by sending and receiving data. This paper refers to the energy model in LEACH [3], as shown in Figure 2. The energy consumption of the sending node includes two parts: the processing circuit and the power amplifier. The energy consumption of the receiving node is only used to process the circuit [4]. Suppose the distance between the receiving point and the sending point is d and the threshold of distance is d0. When d<d0, the free-space path loss model is adopted. If d≥d0, multipath fading model [5] is adopted.Figure 2
Energy consumption model of wireless sensor nodes in coal mine robot.In Figure2, it can be seen that the energy consumption of nodes comes mainly from the process of transmitting and receiving data, which is closely related to the transmission distance. In Figure 2, the meaning of each symbol is as follows: Eelec is node energy consumption for sending and receiving 1-bit data. εamp is the power coefficient of the received data; n is the path decay index, whose typical value is 2 or 4. d is communication distance between two nodes; k is the number of bytes sent; EPA is the amount of energy consumed by processing received data.In short distance free-space communication, the transmission power is directly proportional to the transmission distance. And the transmission power can be simply expressed asE=λd, in which λ is a fixed parameter. The energy consumption will be four times as much as the original energy when the distance is doubled. Therefore, when the transmission distance is large, the energy consumption will be greatly reduced by adding forwarding nodes. As shown in Figure 2, a power amplifier involves the path decay index n. When the communication distance between nodes d<d0, the path attenuation index n is set to 2 and the energy consumed between two nodes is proportional to the square of their distance. When d≥d0,n is set to 4. The communication distance between nodes is large. The energy consumed by sending k-bit data to a node at intervals ofN is shown in (1)ETxd,k=kEelec+kεampd2,d<d0kEelec+kεampd4,d≥d0.The energy consumed by receivingk-bit data is shown in (2)ERxk=kEelec+kEPA.In WSN, the energy consumption of nodes is very sensitive to the number of hops [6, 7]. Therefore, we define the energy function of the number of hops as follows: Eh=ehopi,s/H, where hopi,s represents the hop count of the next hop from Agent j to sink. The hop count from Agent i to sink is H. Assuming that Agent i selects Agent j as the next hop, the energy consumption for transmitting k-bit packets between Agent i and Agent j is(3)Ei,j=2kEelec+kεampdθehopi,s/Hθ∈2,4.


## 2.1. A Wireless Sensor Network Model for Coal Mine Robot

The coal mine robot wireless sensor network has a single layer structure, which can be represented by a directed graphGV,A. The graph is shown in Figure 1, where V represents the sensor node set, V1 is a sink node, and the wireless sensor network is wired to the coal mine robot control system by the sink node. Each node is fixed relative to the coal mine robot. The nodes transmit messages to the sink node in a direct or multihop manner.A represents a set of directed arcs, each of which represents the connection of two nodes in V that can communicate directly. The label beside an edge in the graph represents the corresponding path weight wij. The path with a smaller weight means that the channel has a better quality. The nodes on this path have more residual energy and the hop consumes less energy.Figure 1
Model of wireless sensor network for coal mine robots.


## 2.2. Node Energy Model

The energy consumption of wireless sensor network nodes is divided into 3 parts: perception, communication, and data processing. In this paper, we mainly consider the energy consumption of data communication, that is, the energy consumed by sending and receiving data. This paper refers to the energy model in LEACH [3], as shown in Figure 2. The energy consumption of the sending node includes two parts: the processing circuit and the power amplifier. The energy consumption of the receiving node is only used to process the circuit [4]. Suppose the distance between the receiving point and the sending point is d and the threshold of distance is d0. When d<d0, the free-space path loss model is adopted. If d≥d0, multipath fading model [5] is adopted.Figure 2
Energy consumption model of wireless sensor nodes in coal mine robot.In Figure2, it can be seen that the energy consumption of nodes comes mainly from the process of transmitting and receiving data, which is closely related to the transmission distance. In Figure 2, the meaning of each symbol is as follows: Eelec is node energy consumption for sending and receiving 1-bit data. εamp is the power coefficient of the received data; n is the path decay index, whose typical value is 2 or 4. d is communication distance between two nodes; k is the number of bytes sent; EPA is the amount of energy consumed by processing received data.In short distance free-space communication, the transmission power is directly proportional to the transmission distance. And the transmission power can be simply expressed asE=λd, in which λ is a fixed parameter. The energy consumption will be four times as much as the original energy when the distance is doubled. Therefore, when the transmission distance is large, the energy consumption will be greatly reduced by adding forwarding nodes. As shown in Figure 2, a power amplifier involves the path decay index n. When the communication distance between nodes d<d0, the path attenuation index n is set to 2 and the energy consumed between two nodes is proportional to the square of their distance. When d≥d0,n is set to 4. The communication distance between nodes is large. The energy consumed by sending k-bit data to a node at intervals ofN is shown in (1)ETxd,k=kEelec+kεampd2,d<d0kEelec+kεampd4,d≥d0.The energy consumed by receivingk-bit data is shown in (2)ERxk=kEelec+kEPA.In WSN, the energy consumption of nodes is very sensitive to the number of hops [6, 7]. Therefore, we define the energy function of the number of hops as follows: Eh=ehopi,s/H, where hopi,s represents the hop count of the next hop from Agent j to sink. The hop count from Agent i to sink is H. Assuming that Agent i selects Agent j as the next hop, the energy consumption for transmitting k-bit packets between Agent i and Agent j is(3)Ei,j=2kEelec+kεampdθehopi,s/Hθ∈2,4.


## 3. Energy Saving Methods

In order to prolong the lifetime of wireless sensor networks for coal mine robots, it is necessary to balance the energy consumption of all nodes. At the same time, the nodes with less remaining energy should be used as few as possible. In this paper, theQ-learning algorithm of reinforcement learning is applied to the wireless sensor networks of coal mine robots. This algorithm takes into account the distance between nodes, hops, communication energy consumption, and the residual energy consumption of nodes.


### 3.1. Agent Reinforcement Learning

Broadly speaking, an intelligent Agent senses the environment and performs an action, reinforcement learning, which is derived from animal learning and stochastic approximation [8]. It is an unsupervised machine learning technique. It can use the uncertain environment reward to find optimal behavior sequence and realize on-line learning in dynamic environment. Agents are independent, cooperative, and self-learning. They do not require human control. Agent nodes perceive the change of the environment and get the maximum reward value and get an optimal strategy for corresponding decision actions. The principle of Agent’s enhanced learning is shown in Figure 3. As can be seen from the diagram, Agent needs to interact with the environment. First, Agent senses information from complex environments. Then, Agent processes information, improves the performance, chooses a behavior, and makes group behavior choices. According to its individual and group behavior choices, Agent makes a decision, selects an action, and influences the environment.Figure 3
Basic principles of Agent’s reinforcement learning.


### 3.2.Q-Learning Algorithm

Q-learning [9] is the most widely used algorithm in reinforcement learning algorithms. It is an unsupervised learning method whose input is a feedback from constantly changing and complex environment.Q-learning can be modeled by Markov decision process (MDP) [10]. MDP can be defined as a four-tuple S,A,P,R, where S represents the set of states, A represents a set of actions, P represents the state transition function of the environment, and R is the reward function of the environment.In MDP, the state transition functionP and the reward functionR are only related to the current state and actions and not related to the previous states and actions. The purpose of Agent’s reinforcement learning is to learn a strategy [11] π:S→A. Agent takes an action at according to the current status St; that is, πSt=at. By following an arbitrary policy, the cumulative value Vπ(St) obtained from any initial state St is(4)VπSt=rt+γrt+1+γ2rt+2+⋯=rt+γVπSt+1=∑i=0∞γirt+i.In formula (4), r represents the return value, and γ represents the discount factor which reflects the relative proportion of the delay return to the immediate return. The goal of Agent is to learn a strategy to make Vπ(st) maximum; this strategy is called an optimal policy and is represented by π∗:(5)π∗=argπ⁡max⁡VπSt,∀s.Q-learning is a model independent reinforcement learning algorithm. It learns the optimal strategy directly.Q-learning neither needs a prior knowledge model about the state transfer functionP and the return functionRnor needs to learn these models in the course of learning. According to the characteristics of the coal mine robots’ wireless sensor networks, a model-freeQ-learning algorithm is proposed in this paper. The algorithm is simple, fast, and easy to use.The single stepQ value updating formula of theQ-learning algorithm is(6)Qt+1st,αt=1-αQtst,αt+αrt+1+γmax⁡Qtst+1,α′-Qts,αt.The contents of theQ table are constantly updated through formula (6) in the learning process [12]. In the formula, Qt(st,αt) is the function of status-action pair (st,αt) at moment t; α and γ are learning rates and discount factors, respectively. st is the state of the environment at momentt. rt+1 is the return value of the state given by the environment at moment t+1, max⁡Qtst+1,α′ is the maximumQ value of the environment state st+1 at the moment t+1, and α′ is any action taken by Agent in the environment state. TheQ-learning algorithm converges to an optimal solution by iteratively searching the state space.


### 3.3. QLSORP Algorithm

In this paper, a routing algorithm for wireless sensor networks based onQ-learning is designed. The algorithm is applied to the routing protocol of wireless sensor networks for coal mine robots. The routing protocol is called QLSORP (Q-Learning-based Self-Organization Routing Protocol).Q-learning is applied to the routing algorithms in wireless sensor networks.The process of finding the optimal path in wireless sensor networks is equivalent to a Markov process. In each step, an instantaneous reward value is generated. The prerequisite for implementing the routing algorithm is that the return valueR, the statuss, and the actiona must be determined. The return value is the ratio of the residual energy of Agent nodes to the energy consumed by communication. With Rj representing the current remaining energy of the node, according to formula (3), the return value of the Agent node j can be calculated as follows:(7)Vπj=RjEi,j=Rj2kEelec+kεampdθehopi,s/Hθ∈2,4.In the formula, the return value takes into account the remaining energy in the network and the energy consumption required for node communication. Then, a key point is found to balance the energy loss of nodes. State changes with time. Action refers to the process in which Agent nodes choose a path as the optimal one. Considering the communication energy consumption and the remaining energy, the Agent nodes on the routing path continuously send learning packets to the neighbor nodes to obtain the desired return value. The nodes will select the path with the highestQ value as the optimal one. Thus, the whole energy of the network is balanced and the network life cycle is prolonged under the premise of guaranteeing the efficiency of data transmission. The algorithm is described as follows.Step 1.
The sink node sends a learning evaluation message to its neighboring nodes in the same cycle. The wireless sensor network is initialized, and all sensor nodes are started. Each sensor node records the number of its neighboring nodes and energy consumption to its neighboring nodes. At the same time, the energy consumption threshold of each sensor node is set and the return value of the node is set to 0.Step 2.
Define a setD to store the information of the Agent nodes that have been learned during this cycle to prevent infinite loops in the path creation process. The source node periodically sends learning information to its neighboring nodes and determines whether the neighboring nodes exist or not in the setD; at the same time, the return value to each neighboring node is calculated. The neighboring node with a high return value is selected as the next hop routing node.Step 3.
Repeat Step2. The neighboring node of the selected node is calculated and then probes the route of the next hop. The sink nodes are found successively.Step 4.
When the return value of other nodes is received, the sink node updates theQ value table by formula (6) according to theQ-learning mechanism in the reinforcement learning algorithm. The path is selected according to theQ value in the table. When the energy of the sensor nodes in the selected path is lower than the set threshold or the sensor node no longer has any effect, a message is sent to the source node along the opposite direction of the selected path. Then the source sensor node gives up choosing this path. Instead, the path with the second largestQ value will be chosen to transmit the information.Step 5.
The source node selects the path with largeQ value to transmit the information stably to the sink node. The residual energy information of each sensor node is updated simultaneously.Step 6.
The sink node periodically sends learning message. According to the message, the source node probes the path, selects the path, and sends the message to the sink node. The value ofQ varies with the return value of the node. TheQ table is updated and stored in the sink node.Figure4 is the flow chart of the QLSORP algorithm.Figure 4
Flow chart of the algorithm.


## 3.1. Agent Reinforcement Learning

Broadly speaking, an intelligent Agent senses the environment and performs an action, reinforcement learning, which is derived from animal learning and stochastic approximation [8]. It is an unsupervised machine learning technique. It can use the uncertain environment reward to find optimal behavior sequence and realize on-line learning in dynamic environment. Agents are independent, cooperative, and self-learning. They do not require human control. Agent nodes perceive the change of the environment and get the maximum reward value and get an optimal strategy for corresponding decision actions. The principle of Agent’s enhanced learning is shown in Figure 3. As can be seen from the diagram, Agent needs to interact with the environment. First, Agent senses information from complex environments. Then, Agent processes information, improves the performance, chooses a behavior, and makes group behavior choices. According to its individual and group behavior choices, Agent makes a decision, selects an action, and influences the environment.Figure 3
Basic principles of Agent’s reinforcement learning.


## 3.2.Q-Learning Algorithm

Q-learning [9] is the most widely used algorithm in reinforcement learning algorithms. It is an unsupervised learning method whose input is a feedback from constantly changing and complex environment.Q-learning can be modeled by Markov decision process (MDP) [10]. MDP can be defined as a four-tuple S,A,P,R, where S represents the set of states, A represents a set of actions, P represents the state transition function of the environment, and R is the reward function of the environment.In MDP, the state transition functionP and the reward functionR are only related to the current state and actions and not related to the previous states and actions. The purpose of Agent’s reinforcement learning is to learn a strategy [11] π:S→A. Agent takes an action at according to the current status St; that is, πSt=at. By following an arbitrary policy, the cumulative value Vπ(St) obtained from any initial state St is(4)VπSt=rt+γrt+1+γ2rt+2+⋯=rt+γVπSt+1=∑i=0∞γirt+i.In formula (4), r represents the return value, and γ represents the discount factor which reflects the relative proportion of the delay return to the immediate return. The goal of Agent is to learn a strategy to make Vπ(st) maximum; this strategy is called an optimal policy and is represented by π∗:(5)π∗=argπ⁡max⁡VπSt,∀s.Q-learning is a model independent reinforcement learning algorithm. It learns the optimal strategy directly.Q-learning neither needs a prior knowledge model about the state transfer functionP and the return functionRnor needs to learn these models in the course of learning. According to the characteristics of the coal mine robots’ wireless sensor networks, a model-freeQ-learning algorithm is proposed in this paper. The algorithm is simple, fast, and easy to use.The single stepQ value updating formula of theQ-learning algorithm is(6)Qt+1st,αt=1-αQtst,αt+αrt+1+γmax⁡Qtst+1,α′-Qts,αt.The contents of theQ table are constantly updated through formula (6) in the learning process [12]. In the formula, Qt(st,αt) is the function of status-action pair (st,αt) at moment t; α and γ are learning rates and discount factors, respectively. st is the state of the environment at momentt. rt+1 is the return value of the state given by the environment at moment t+1, max⁡Qtst+1,α′ is the maximumQ value of the environment state st+1 at the moment t+1, and α′ is any action taken by Agent in the environment state. TheQ-learning algorithm converges to an optimal solution by iteratively searching the state space.


## 3.3. QLSORP Algorithm

In this paper, a routing algorithm for wireless sensor networks based onQ-learning is designed. The algorithm is applied to the routing protocol of wireless sensor networks for coal mine robots. The routing protocol is called QLSORP (Q-Learning-based Self-Organization Routing Protocol).Q-learning is applied to the routing algorithms in wireless sensor networks.The process of finding the optimal path in wireless sensor networks is equivalent to a Markov process. In each step, an instantaneous reward value is generated. The prerequisite for implementing the routing algorithm is that the return valueR, the statuss, and the actiona must be determined. The return value is the ratio of the residual energy of Agent nodes to the energy consumed by communication. With Rj representing the current remaining energy of the node, according to formula (3), the return value of the Agent node j can be calculated as follows:(7)Vπj=RjEi,j=Rj2kEelec+kεampdθehopi,s/Hθ∈2,4.In the formula, the return value takes into account the remaining energy in the network and the energy consumption required for node communication. Then, a key point is found to balance the energy loss of nodes. State changes with time. Action refers to the process in which Agent nodes choose a path as the optimal one. Considering the communication energy consumption and the remaining energy, the Agent nodes on the routing path continuously send learning packets to the neighbor nodes to obtain the desired return value. The nodes will select the path with the highestQ value as the optimal one. Thus, the whole energy of the network is balanced and the network life cycle is prolonged under the premise of guaranteeing the efficiency of data transmission. The algorithm is described as follows.Step 1.
The sink node sends a learning evaluation message to its neighboring nodes in the same cycle. The wireless sensor network is initialized, and all sensor nodes are started. Each sensor node records the number of its neighboring nodes and energy consumption to its neighboring nodes. At the same time, the energy consumption threshold of each sensor node is set and the return value of the node is set to 0.Step 2.
Define a setD to store the information of the Agent nodes that have been learned during this cycle to prevent infinite loops in the path creation process. The source node periodically sends learning information to its neighboring nodes and determines whether the neighboring nodes exist or not in the setD; at the same time, the return value to each neighboring node is calculated. The neighboring node with a high return value is selected as the next hop routing node.Step 3.
Repeat Step2. The neighboring node of the selected node is calculated and then probes the route of the next hop. The sink nodes are found successively.Step 4.
When the return value of other nodes is received, the sink node updates theQ value table by formula (6) according to theQ-learning mechanism in the reinforcement learning algorithm. The path is selected according to theQ value in the table. When the energy of the sensor nodes in the selected path is lower than the set threshold or the sensor node no longer has any effect, a message is sent to the source node along the opposite direction of the selected path. Then the source sensor node gives up choosing this path. Instead, the path with the second largestQ value will be chosen to transmit the information.Step 5.
The source node selects the path with largeQ value to transmit the information stably to the sink node. The residual energy information of each sensor node is updated simultaneously.Step 6.
The sink node periodically sends learning message. According to the message, the source node probes the path, selects the path, and sends the message to the sink node. The value ofQ varies with the return value of the node. TheQ table is updated and stored in the sink node.Figure4 is the flow chart of the QLSORP algorithm.Figure 4
Flow chart of the algorithm.


## 4. Simulation Experiment Analysis

### 4.1. Experiment General Situation

Network simulation is a basic method for wireless sensor network research. The wireless sensor network of coal mine robot mainly involves temperature, humidity, wind speed, wind direction, light intensity, coal dust, oxygen, gas, distance sensor, track speed, and pitching angle of robot. Communication between wireless sensor nodes and between nodes and sink nodes constitutes a wireless sensor network. The topology is shown in Figure5. Simulation analysis is carried out to verify the efficiency of routing algorithm based onQ-learning for wireless sensor networks. It is assumed that the transmission radius of the sensor node is 20–60 cm. The transmitted packet size is 10 K bits. The initial energy of the nodes is 5 J. The nodes send data to the sink node in a single hop or multihop. The sink node is wired to the control system of the coal mine robot, whose energy is always constant. The movement of the nodes is not considered because the wireless sensor network is fixed after the arrangement of the sensor and the robot body. The QLSORP algorithm was developed and implemented in the environment of MATLAB 2016; other simulation parameters are set as follows: the size of the network area is 120cm∗200cm, Eelec is 50 nJ/bit, EPA is 100 pJ/bit/cm2, and d0 is 100 cm.Figure 5
Node topology of wireless sensor network for coal mine robot.


### 4.2. Experimental Results and Analysis

From formula (6), it can be seen that in theQ-learning based self-organizing routing method for wireless sensor networks (QLSOP), the two parameters γ and α remain unknown and need to be preset. In order to reduce the computational complexity of the routing algorithm, γ=1 is used to reduce the delay. α is desirable for any number of 0-1, such as 0.9, 0.6, and 0.3. The simulations are conducted to verify the influence of parameter α on the lifetime of wireless sensor networks, the choice of node paths, the routing energy loss, and the residual energy of nodes.Figure6 shows the routing loss of the source nodes at different distances to sink nodes. It is obvious that the energy loss of the nodes routing to the sink nodes is proportional to the distance between them. The energy consumed by routing increases as the distance between the nodes and the sink nodes increases. α is the learning rate. Its size has a certain influence on the loss of routing energy. The smaller the learning rate is, the more the loss of routing is.Figure 6
Routing energy loss.The number of nodes also affects the routing energy consumption of the source node routing to the sink node. The residual energy of the node is obtained by simulation, as shown in Figure7. When the number of nodes in wireless sensor networks increases, the residual energy in the nodes becomes larger. The higher the learning factor, the more the residual energy the nodes have.Figure 7
Residual energy of node.α is the learning rate, its value affects convergence ofQ-learning. The effects of different α onQ value are shown in Figure 8. With the increasing of α, the value ofQ in the fixed path decreases gradually. Formula (6) shows that when α value is close to 0, the value ofQ depends on the preserved Agent node’sQ value. When α value is close to 1,Q value depends on the maximumQ value of the Agent’s neighboring node and the return value from the neighboring node.Figure 8
The impact of alpha values onQ.The lifetime of the wireless sensor network for the coal mine robot is shown in Figure9. Figure 9(a) is a 3D surface for node number, communication path distance, and network lifetime in wireless sensor networks. The shorter the communication path, the less the energy consumed by the nodes and the longer the network lifetime. In addition, with the increasing number of nodes in wireless sensor networks, the number of alternative routes through which the source node sends data to the sink node increases. By continuously updating the return value, the optimal path is selected and the consumption of the routing energy is reduced, which results in a longer network lifetime. Figure 9(b) shows the comparison between the DSR algorithm and the QLSORP algorithm proposed in this paper. The curve at the bottom of the graph represents the network lifetime of the DSR under the same simulation conditions. Its performance is lower than that of QLSORP. The reason is that the QLSORP algorithm balances the energy between the nodes of wireless sensor networks. From Figures 9(a) and 9(b) we can see that when the number of nodes is near 18, with the increase of α, the lifetime of WSNs also increases accordingly. It is noteworthy that the optimal path is invariable, regardless of the value of α.Figure 9
Network lifetime.
(a)
Network lifetime under node number and path distance
(b)
Comparison of network lifetime between QLSORP and DSR


## 4.1. Experiment General Situation

Network simulation is a basic method for wireless sensor network research. The wireless sensor network of coal mine robot mainly involves temperature, humidity, wind speed, wind direction, light intensity, coal dust, oxygen, gas, distance sensor, track speed, and pitching angle of robot. Communication between wireless sensor nodes and between nodes and sink nodes constitutes a wireless sensor network. The topology is shown in Figure5. Simulation analysis is carried out to verify the efficiency of routing algorithm based onQ-learning for wireless sensor networks. It is assumed that the transmission radius of the sensor node is 20–60 cm. The transmitted packet size is 10 K bits. The initial energy of the nodes is 5 J. The nodes send data to the sink node in a single hop or multihop. The sink node is wired to the control system of the coal mine robot, whose energy is always constant. The movement of the nodes is not considered because the wireless sensor network is fixed after the arrangement of the sensor and the robot body. The QLSORP algorithm was developed and implemented in the environment of MATLAB 2016; other simulation parameters are set as follows: the size of the network area is 120cm∗200cm, Eelec is 50 nJ/bit, EPA is 100 pJ/bit/cm2, and d0 is 100 cm.Figure 5
Node topology of wireless sensor network for coal mine robot.


## 4.2. Experimental Results and Analysis

From formula (6), it can be seen that in theQ-learning based self-organizing routing method for wireless sensor networks (QLSOP), the two parameters γ and α remain unknown and need to be preset. In order to reduce the computational complexity of the routing algorithm, γ=1 is used to reduce the delay. α is desirable for any number of 0-1, such as 0.9, 0.6, and 0.3. The simulations are conducted to verify the influence of parameter α on the lifetime of wireless sensor networks, the choice of node paths, the routing energy loss, and the residual energy of nodes.Figure6 shows the routing loss of the source nodes at different distances to sink nodes. It is obvious that the energy loss of the nodes routing to the sink nodes is proportional to the distance between them. The energy consumed by routing increases as the distance between the nodes and the sink nodes increases. α is the learning rate. Its size has a certain influence on the loss of routing energy. The smaller the learning rate is, the more the loss of routing is.Figure 6
Routing energy loss.The number of nodes also affects the routing energy consumption of the source node routing to the sink node. The residual energy of the node is obtained by simulation, as shown in Figure7. When the number of nodes in wireless sensor networks increases, the residual energy in the nodes becomes larger. The higher the learning factor, the more the residual energy the nodes have.Figure 7
Residual energy of node.α is the learning rate, its value affects convergence ofQ-learning. The effects of different α onQ value are shown in Figure 8. With the increasing of α, the value ofQ in the fixed path decreases gradually. Formula (6) shows that when α value is close to 0, the value ofQ depends on the preserved Agent node’sQ value. When α value is close to 1,Q value depends on the maximumQ value of the Agent’s neighboring node and the return value from the neighboring node.Figure 8
The impact of alpha values onQ.The lifetime of the wireless sensor network for the coal mine robot is shown in Figure9. Figure 9(a) is a 3D surface for node number, communication path distance, and network lifetime in wireless sensor networks. The shorter the communication path, the less the energy consumed by the nodes and the longer the network lifetime. In addition, with the increasing number of nodes in wireless sensor networks, the number of alternative routes through which the source node sends data to the sink node increases. By continuously updating the return value, the optimal path is selected and the consumption of the routing energy is reduced, which results in a longer network lifetime. Figure 9(b) shows the comparison between the DSR algorithm and the QLSORP algorithm proposed in this paper. The curve at the bottom of the graph represents the network lifetime of the DSR under the same simulation conditions. Its performance is lower than that of QLSORP. The reason is that the QLSORP algorithm balances the energy between the nodes of wireless sensor networks. From Figures 9(a) and 9(b) we can see that when the number of nodes is near 18, with the increase of α, the lifetime of WSNs also increases accordingly. It is noteworthy that the optimal path is invariable, regardless of the value of α.Figure 9
Network lifetime.
(a)
Network lifetime under node number and path distance
(b)
Comparison of network lifetime between QLSORP and DSR


## 5. Conclusions

This paper proposes a QLSORP routing algorithm for wireless sensor network in coal mine robot based onQ-learning. The wireless sensor network nodes are regarded as intelligent Agent nodes. A self-organizing routing method based onQ-learning for wireless sensor networks is designed. It takes into account the hop number, distance, residual energy, communication loss, energy, and so on. By calculating the return value of neighboring nodes,Q value of the path is updated constantly. The updatedQ value table is kept in the sink node. The sink node searches the optimal path by comparingQ values and selects the path with the largestQ value for data transmission. When the node in the path is accidentally dead or the node energy is exhausted, the path with the second largestQ value is selected for data transmission. This balances the energy between nodes. Compared with the DSR algorithm, the QLSORP algorithm can significantly reduce the energy consumption of the network. The algorithm can prevent the premature death of some nodes in the network. The coal mine robot wireless sensor network node energy can be utilized effectively. The lifetime of wireless sensor networks is prolonged.This paper mainly studies the effectiveness of the routing algorithm based onQ-learning to balance the node energy of the WSN and improve the lifetime of the whole network. The robustness of WSN has not been considered, such as the impact of environmental noise interference on WSN, or the coverage hole problem of WSN [13] caused by the failure of some nodes or the exhaustion of energy. These problems all affect the practical effect of WSN. The next work will further study the topology self-cure algorithm [14] for node failure or energy exhaustion and improve the robustness to environmental noise [15] and other disturbances.



---

*Source: 1487908-2018-01-11.xml*