# A Multiagent Reinforcement Learning Solution for Geometric Configuration Optimization in Passive Location Systems

Authors: Shengxiang Li; Haisi Li; Ke Ke; Ou Li; Guangyi Liu; Siyuan Ding; Yijie Bai
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Passive location systems receive electromagnetic waves at one or multiple base stations to locate the transmitters, which are widely used in security fields. However, the geometric configurations of stations can greatly affect the positioning precision. In the literature, the geometry of the passive location system is mainly designed based on empirical models. These empirical models, being hard to track the sophisticated electromagnetic environment in the real world, result in suboptimal geometric configurations and low positioning precision. In order to master the characteristics of complicated electromagnetic environments to improve positioning performance, this paper proposes a novel geometry optimization method based on multiagent reinforcement learning. In the proposed method, agents learn to optimize the geometry cooperatively by factorizing team value function into agentwise value functions. To facilitate cooperation and deal with data transmission challenges, a constraint is imposed on the data sent from the central station to vice stations to ensure conciseness and effectiveness of communications. According to the empirical results under direct position determination systems, agents can find better geometric configurations than the existing methods in complicated electromagnetic environments.


---

## Body

## 1. Introduction

Passive location techniques are used for various scenarios, such as telecommunication pseudobase station discovery, and aviation interference investigation. Traditional passive location algorithms [1] mainly include angle of arrival (AOA) [2], difference of time arrival (TDOA) [3], and frequency difference of arrival (FOA) [4]. These algorithms locate transmitters after estimating the signal parameter called the two-step positioning methods. The direct position determination (DPD) [5, 6] uses observations from all the stations to locate the transmitter without estimating the signal parameters, which outperforms two-step methods in low signal-to-noise ratio (SNR) scenarios [7].The geometric configurations of stations can significantly affect the positioning precision [8], both in two-step and DPD positioning algorithms. In the literature, some existing studies tried to obtain general principles in geometric configurations from massive experiments [9, 10]. And, only some rough conclusions have been drawn. For instance, all stations should not line up, or stations should form a triangle to surround the transmitter. There also exist several studies that have employed heuristic methods, such as genetic algorithm (GA) [11] or particle swarm optimization (PSO) [12], to search the optimal geometry. These methods are based on empirical models in which signals are assumed to propagate ideally. However, in the real world, an electromagnetic environment changes abruptly with the positions of stations due to various factors, such as signal frequency, interference, attenuation, multipath, obstacles, and noises. These factors can hardly be described fully in empirical models, leading to suboptimal geometric configurations and low positioning precision. Therefore, it is vital to adjust geometric configuration to fit the sophisticated electromagnetic environment, so as to improve precision in a passive positioning task. This problem is regarded as a sequential decision-making problem in a real-world complex electromagnetic environment, rather than an optimal geometry searching problem based on empirical models.Reinforcement learning (RL) is a viable and elegant approach to yield an optimal policy for sequential decision-making problems [13]. The tricky electromagnetic environment can be tracked by RL in a trial-and-error paradigm. The nonlinear and parameterized deep neural network (DNN), providing the compact and powerful representation of experiences, can adapt to the complicated electromagnetic spatial distribution accurately. Therefore, this paper addresses the problem of finding optimal geometric configuration in the passive location system through deep reinforcement learning (DRL) [14].Under the framework of DRL, a station is used as a mobile agent. The terms’ station and agent are hereafter used interchangeably which can receive signals and decide where to go. Agents need to optimize the geometric configuration collaboratively to improve the positioning precision, and they can share information via communication channels to facilitate the collaboration. However, the communication traffic matters when the number of agents increases, especially in adverse communication conditions.This paper proposes an efficient multiagent reinforcement learning algorithm to optimize the geometric configuration for passive location systems. To this end, each station is regarded as a mobile agent with all agents having a collective objective of finding an optimal geometry to improve the positioning precision. To facilitate the collaborations among agents, they are trained based on value function decomposition, which can solve the credit assignment problem among agents implicitly. For a vice station, it needs to obtain information from other stations to improve the evaluation of the situation and promote the quality of decisions on where to go. Meanwhile, it is necessary to reduce the communication traffic due to transmission and processing challenges. A mutual information objective function then is employed to constrain the messages sent to vice stations to ensure the expressiveness and conciseness. The proposed method is evaluated on simulated DPD positioning tasks in a complicated electromagnetic environment. The results demonstrated that the agents can find better geometric configurations than existing methods.


## 2. Background

This section introduces the relevant background on passive locations (concretely, DPD) and MARL.


### 2.1. Passive Location with DPD

ConsiderH transmitters and L stations intercepting the transmitted signal, as shown in Figure 1. Each station is equipped with an antenna array consisting of M elements. The hth transmitter’s position is denoted by ph=xh,yh⊤. The complex envelopes of the signals observed by the ℓth station are given by the following equation [5]:(1)zℓt=∑h=1Hbℓhαℓphvht−τℓph−th0+nℓt,where 0≤t≤T, zℓt is a complex time dependent M×1 observation vector, and bℓ is an unknown complex scalar representing the channel attenuation between the hth transmitter and the ℓth station. Moreover, αℓph is the ℓth array response to the signal transmitted from position ph, and vht−τℓph−th0 is the hth signal waveform transmitted at time th0 and delayed by τℓph. The vector nℓt represents noise, interference, and multipath effects on the signals.Figure 1
The topology of a passive location system.For brevity, we useαℓh and τℓh instead of αℓph and τℓph. The observed signal can be partitioned into K sections with length T/K≫maxτℓ. Taking the Fourier transform of each section, we obtain(2)zℓj,k=∑h=1Hbℓhαℓqvhj,ke−iωjτℓh+th0+nℓj,k=∑h=1Hα¯ℓj,ph,bℓhs¯hj,k+nℓj,k,(3)=Aℓjv¯j,k+nℓj,k,where j=1,…,J is the index of Fourier coefficients and k=1,…,K is the time section index, i=−1.In (2), we have(4)α¯ℓj,ph,bℓh≜bℓhαℓhe−iωjτℓh,s¯hj,k≜vhj,ke−iωjth0.Then, the vectorα¯ℓj,ph,bℓh concludes all information about the transmitter’s position. Furthermore, the phase shift caused by the transmit time th0 is cancelled out when s¯hj,k is used by the DPD method.In (3), the received signal is presented in matrix notation with(5)Aℓj≜α¯ℓj,p1,bℓ1,…,α¯ℓj,pH,bℓH,v¯j,k≜s¯1j,k,…,s¯Hj,k.Since the vectorv¯j,k is the same at all stations, the observed vectors of all stations can be concatenated together as(6)zj,k=Ajv¯j,k+nj,k,where(7)zj,k≜z1⊤j,k,…,zL⊤j,k⊤,Aj≜A1⊤j,…,AL⊤j⊤,nj,k≜n1⊤j,k,…,nL⊤j,k⊤.Assume thehth column of Aj is denoted by α¯j,ph,bh, corresponding to the hth emitter, and can be factored as(8)α¯j,ph,bh=ΓhjHbh,where Γhj≜diagα1h⊤e−iωjτ1h,…,αLh⊤e−iωjτLh is a diagonal matrix whose elements are the response of the arrays at all stations, bh≜b1h,…,bLh⊤, H≜I⊗1M, IL stands for the identity matrix of sizes L×L, 1M stands for M×1 column vector of ones, and ⊗ stands for the Kronecker product.The additive noise vectornj,k is assumed to be a realization of a circularly complex Gaussian process with zero mean. The second-order moments is given by(9)Enj,knHi,m=Wδk,mδj,i,Enj,kn⊤i,m=0.The covariance matrixW represents the thermal noise as well as interference. In the case of spatially white noise, W is a block diagonal matrix given by(10)W=diagσ12,…,σL2⊗1M.Assume that signals and noise are uncorrelated so that(11)limK⟶∞1K∑k=1Kv¯j,knHj,k=0.A matrix is defined to construct the DPD estimator as follows:(12)Uj≜ZvzHjZvv−1j,where ZsrHj≜1/K∑k=1Kv¯j,kzHj,k and Zvv−1j≜1/K∑k=1Kv¯j,kv¯Hj,k. The matrix Zvvj becomes diagonal for large K if the signals are uncorrelated. The hth column of Uj and its ℓth subvector are denoted by uhj and uℓhj, respectively. The DPD estimator for general noise covariance is presented as(13)ph^=argmaxphuhHW¯h−1/2PA¯hW¯h−1/2uh,where uh≜uh⊤1,…,uh⊤J, A¯h≜W¯h−1/2ΓhH, Γh≜Γh⊤1,…,Γh⊤J⊤,W¯h≜Gh,h−1⊗W,Gh,h≜diagRvv1h,h,…,RvvJh,h,PA¯h≜A¯hA¯h+, and A¯h+ is the pseudoinverse of A¯h.In the case of partially white noise with a spectral density matrix defined in (10), the DPD estimator becomes(14)ph^=argmaxph∑ℓ=1Lσℓ−2∑j=1JeiωjτℓphαℓhHuℓhj2.According to [5], the Cramér–Rao lower bound (CLRB) on the covariance of any unbiased estimator of the position vector with no model errors is(15)CRLB=12KReΩHΩ−1,where Ω≜PA¯h⊥D˜hI2⊗bh, D˜h≜W¯h−1/2∂Γh/∂xh∂Γh/∂yhH, and PA¯h⊥≜=I−PA¯h.The CRLB obtained from (15), determined by received signals z and locations of stations p, is utilized as the reward function. The CRLB plays a major role in developing the passive location agents through MARL.


### 2.2. Multiagent Reinforcement Learning

In reinforcement learning, an agent interacts with the environment for a given goal. At timet, it observes state st∈S with S denoting the state space, takes action at∈A with A representing the action space, receives reward rt∈ℝ, and moves to the new state st∈S. The agent aims to learn a policy that maximizes the long-term reward. The action-value function, which starts from state s, takes action a, and follows policy π, is denoted by Qπs,a [13]:(16)Qπs,a=E∑t=0∞γtrt|st=s,at=a,where γ∈0,1 is the discount factor that determines the importance of future rewards.In the multiagent reinforcement learning (MARL) [15, 16], agents (robots, UAVs, sensors, etc.) interact with a shared environment to complete the given tasks. Basically, agents are the learnable units that want to learn policies in order to maximize the long-term reward through interactions with the environment. Most MARL problems are classified as NP-hard problems [17] for the sophisticated environments and the combinatorial nature of the problem.In a cooperative MARL problem, agents must jointly optimize an accumulative scalar team reward over time. The centralized RL approach can be employed to solve the cooperation problem, i.e., all state observations are merged together and the problem is reduced to a single agent problem with a combinatorial action space. Whereas, according to Peter [18], the naive centralized RL methods fail to find the global optimum even if it is possible to solve the problems with such huge state and action spaces. The challenge lies in the fact that some of the agents may become lazy and unable to learn and cooperate as they should. This may cause the whole system to face a failure. They addressed these problems by training individual agents with a value decomposition network (VDN) architecture. The agents learn to decompose the team value function into agentwise value functions as follows:(17)Qtotρ,a=∑iQiρi,ai;ωi,where ρ and a represent the observation-action history and joint action, respectively, and ωi is the value function parameters of agent i. VDN aims to learn an optimal linear value decomposition from the team reward signal, by backpropagating the total Q gradient through deep neural networks representing the summation of individual value functions. The VDN solves the credit assignment among agents implicitly without any specific reward for individual agents. Rashid [19] regarded the cooperative MARL problem as the VDN does, but added a constraint on the objective:(18)∂Qtot∂Qi≥0,∀i,which makes the weights of the mixing network positive and ensures monotonic improvement.


## 2.1. Passive Location with DPD

ConsiderH transmitters and L stations intercepting the transmitted signal, as shown in Figure 1. Each station is equipped with an antenna array consisting of M elements. The hth transmitter’s position is denoted by ph=xh,yh⊤. The complex envelopes of the signals observed by the ℓth station are given by the following equation [5]:(1)zℓt=∑h=1Hbℓhαℓphvht−τℓph−th0+nℓt,where 0≤t≤T, zℓt is a complex time dependent M×1 observation vector, and bℓ is an unknown complex scalar representing the channel attenuation between the hth transmitter and the ℓth station. Moreover, αℓph is the ℓth array response to the signal transmitted from position ph, and vht−τℓph−th0 is the hth signal waveform transmitted at time th0 and delayed by τℓph. The vector nℓt represents noise, interference, and multipath effects on the signals.Figure 1
The topology of a passive location system.For brevity, we useαℓh and τℓh instead of αℓph and τℓph. The observed signal can be partitioned into K sections with length T/K≫maxτℓ. Taking the Fourier transform of each section, we obtain(2)zℓj,k=∑h=1Hbℓhαℓqvhj,ke−iωjτℓh+th0+nℓj,k=∑h=1Hα¯ℓj,ph,bℓhs¯hj,k+nℓj,k,(3)=Aℓjv¯j,k+nℓj,k,where j=1,…,J is the index of Fourier coefficients and k=1,…,K is the time section index, i=−1.In (2), we have(4)α¯ℓj,ph,bℓh≜bℓhαℓhe−iωjτℓh,s¯hj,k≜vhj,ke−iωjth0.Then, the vectorα¯ℓj,ph,bℓh concludes all information about the transmitter’s position. Furthermore, the phase shift caused by the transmit time th0 is cancelled out when s¯hj,k is used by the DPD method.In (3), the received signal is presented in matrix notation with(5)Aℓj≜α¯ℓj,p1,bℓ1,…,α¯ℓj,pH,bℓH,v¯j,k≜s¯1j,k,…,s¯Hj,k.Since the vectorv¯j,k is the same at all stations, the observed vectors of all stations can be concatenated together as(6)zj,k=Ajv¯j,k+nj,k,where(7)zj,k≜z1⊤j,k,…,zL⊤j,k⊤,Aj≜A1⊤j,…,AL⊤j⊤,nj,k≜n1⊤j,k,…,nL⊤j,k⊤.Assume thehth column of Aj is denoted by α¯j,ph,bh, corresponding to the hth emitter, and can be factored as(8)α¯j,ph,bh=ΓhjHbh,where Γhj≜diagα1h⊤e−iωjτ1h,…,αLh⊤e−iωjτLh is a diagonal matrix whose elements are the response of the arrays at all stations, bh≜b1h,…,bLh⊤, H≜I⊗1M, IL stands for the identity matrix of sizes L×L, 1M stands for M×1 column vector of ones, and ⊗ stands for the Kronecker product.The additive noise vectornj,k is assumed to be a realization of a circularly complex Gaussian process with zero mean. The second-order moments is given by(9)Enj,knHi,m=Wδk,mδj,i,Enj,kn⊤i,m=0.The covariance matrixW represents the thermal noise as well as interference. In the case of spatially white noise, W is a block diagonal matrix given by(10)W=diagσ12,…,σL2⊗1M.Assume that signals and noise are uncorrelated so that(11)limK⟶∞1K∑k=1Kv¯j,knHj,k=0.A matrix is defined to construct the DPD estimator as follows:(12)Uj≜ZvzHjZvv−1j,where ZsrHj≜1/K∑k=1Kv¯j,kzHj,k and Zvv−1j≜1/K∑k=1Kv¯j,kv¯Hj,k. The matrix Zvvj becomes diagonal for large K if the signals are uncorrelated. The hth column of Uj and its ℓth subvector are denoted by uhj and uℓhj, respectively. The DPD estimator for general noise covariance is presented as(13)ph^=argmaxphuhHW¯h−1/2PA¯hW¯h−1/2uh,where uh≜uh⊤1,…,uh⊤J, A¯h≜W¯h−1/2ΓhH, Γh≜Γh⊤1,…,Γh⊤J⊤,W¯h≜Gh,h−1⊗W,Gh,h≜diagRvv1h,h,…,RvvJh,h,PA¯h≜A¯hA¯h+, and A¯h+ is the pseudoinverse of A¯h.In the case of partially white noise with a spectral density matrix defined in (10), the DPD estimator becomes(14)ph^=argmaxph∑ℓ=1Lσℓ−2∑j=1JeiωjτℓphαℓhHuℓhj2.According to [5], the Cramér–Rao lower bound (CLRB) on the covariance of any unbiased estimator of the position vector with no model errors is(15)CRLB=12KReΩHΩ−1,where Ω≜PA¯h⊥D˜hI2⊗bh, D˜h≜W¯h−1/2∂Γh/∂xh∂Γh/∂yhH, and PA¯h⊥≜=I−PA¯h.The CRLB obtained from (15), determined by received signals z and locations of stations p, is utilized as the reward function. The CRLB plays a major role in developing the passive location agents through MARL.


## 2.2. Multiagent Reinforcement Learning

In reinforcement learning, an agent interacts with the environment for a given goal. At timet, it observes state st∈S with S denoting the state space, takes action at∈A with A representing the action space, receives reward rt∈ℝ, and moves to the new state st∈S. The agent aims to learn a policy that maximizes the long-term reward. The action-value function, which starts from state s, takes action a, and follows policy π, is denoted by Qπs,a [13]:(16)Qπs,a=E∑t=0∞γtrt|st=s,at=a,where γ∈0,1 is the discount factor that determines the importance of future rewards.In the multiagent reinforcement learning (MARL) [15, 16], agents (robots, UAVs, sensors, etc.) interact with a shared environment to complete the given tasks. Basically, agents are the learnable units that want to learn policies in order to maximize the long-term reward through interactions with the environment. Most MARL problems are classified as NP-hard problems [17] for the sophisticated environments and the combinatorial nature of the problem.In a cooperative MARL problem, agents must jointly optimize an accumulative scalar team reward over time. The centralized RL approach can be employed to solve the cooperation problem, i.e., all state observations are merged together and the problem is reduced to a single agent problem with a combinatorial action space. Whereas, according to Peter [18], the naive centralized RL methods fail to find the global optimum even if it is possible to solve the problems with such huge state and action spaces. The challenge lies in the fact that some of the agents may become lazy and unable to learn and cooperate as they should. This may cause the whole system to face a failure. They addressed these problems by training individual agents with a value decomposition network (VDN) architecture. The agents learn to decompose the team value function into agentwise value functions as follows:(17)Qtotρ,a=∑iQiρi,ai;ωi,where ρ and a represent the observation-action history and joint action, respectively, and ωi is the value function parameters of agent i. VDN aims to learn an optimal linear value decomposition from the team reward signal, by backpropagating the total Q gradient through deep neural networks representing the summation of individual value functions. The VDN solves the credit assignment among agents implicitly without any specific reward for individual agents. Rashid [19] regarded the cooperative MARL problem as the VDN does, but added a constraint on the objective:(18)∂Qtot∂Qi≥0,∀i,which makes the weights of the mixing network positive and ensures monotonic improvement.


## 3. MARL-Based Geometry Optimization

This section proposes a MARL-based geometric configuration optimization method for passive location systems.


### 3.1. Model Framework

In this paper, a DPD location system is considered withL mobile stations (e.g., UAVs equipped with positioning equipment), i.e., L DPD agents. Each agent transfers the intercepted signals to a central processing agent where the emitter’s position is estimated. Agents have no knowledge of the emitter and the electromagnetic environment. Due to the influence of multipath and noises, the signals received by different agents vary. To adapt to the complicated electromagnetic spatial distribution accurately, a MARL-based method, with positioning error being the reward function, is considered. The key elements in the MARL scheme are defined as follows:(i)
States. At each time step t, agent i intercepts signals, oti, emitted by the transmitter. The total messages it receives from other agents are denoted by miin. The state of agent i is denoted by sti=oti,miin,pti∈Si, where pti is the position of agent i at time t, defined in Section 3.1. Then, the global state is represented as s=s1,…,sL∈S, where S=∏i=1LSi.(ii)
Actions. Actions represent the decisions regarding where to receive signals at next step. Let ai=φi,di∈Ai denote the action of agent i, where φi and di represent its moving direction and distance, respectively. And, the joint action of all agents is denoted as a=a1,…,aL∈A, A=∏i=1LAi.(iii)
Rewards. This paper aims to develop agents that can properly adjust the geometry automatically to improve the positioning precision. To this end, we evaluate agents’ behavior by positioning errors. Two types of positioning errors are considered:(a)
CRLB is an effective index for evaluating the precision of a passive location system. Let the background position of the transmitter bep⋆=x⋆,y⋆. Then, the CRLB is a function of state s and the background position p⋆, i.e., CRLBp⋆,s, according to (15).(b)
The statistic error is a popular class of position errors, such as the mean error (ME) and the mean square error (MSE).Among the errors listed above, only CRLB can assess the geometry without estimating the target position, which reduces considerable amounts of time and computing in training. Therefore, CRLB is used as the reward function in training the DPD agents.


### 3.2. Learn to Optimize the Geometry

This section presents an efficient multiagent actor-critic algorithm for geometric configuration optimization in passive location tasks. The overall architecture of the proposed method is illustrated in Figure2. It is developed based on two main considerations: (i) factorizing the global value function into individual value functions with local observations for better collaboration and (ii) utilizing information constraints to facilitate communications and optimize the messages to tackle the transmission challenges.Figure 2
Schematics of the proposed method. The global value function is decomposed into the agentwise value functions.


#### 3.2.1. Value Decomposition

As shown in Figure2, the global value function is factorized into linear combination of individual value functions as follows:(19)Qtotρ,a;ω=∑iQiρi,ai;ωi,where ρ=ρ1,…,ρN. And, ρi=s1i,a1i,…,sti,ati is the history of local observations, actions, and messages received. Local value functions are parameterized by ωV=ω1⊤,…,ωN⊤⊤. The policy of each station maps the history of observations and actions to the next action: πθiρi,ai. The joint policy for the location system is denoted by πθρ,a=∏iπθiρi,ai. Both actor and critic of each agent utilize the gated recurrent unit (GRU) [20] to process the input of observation history. GRU is a special kind of recurrent neural network that has the ability to capture the long range connections of states. The mixing network and individual value functions are trained in an end-to-end manner by minimizing the TD loss as follows:(20)ℒTDωV=r+γQtotρ′,a′;ωV−Qtotρ,a;ωV2.


#### 3.2.2. Information Constraint

For the central station, it must collect observations from all the stations to estimate the transmitter’s position. Nevertheless, a vice stationj just needs the data that can help to make better decisions. Therefore, the central station must learn how to send messages as short as possible but enough for vice stations to act better. A natural solution is to add information constraints. In practice, to improve the effectiveness of messages sent to vice stations, it is necessary to maximize the mutual information of messages and station’s actions. Let c be the index of the central station; then, mutual information is defined by(21)Imc,av=∫pmc,avlogpmc,avpmcpavdmcdav,where mc represents the message sent from central station to a vice station and mjin=mcj≠c and av is the joint action of all vice stations.However, if this is the only objective, agents could always ensure a maximally informative representation by taking the identity encoding of raw data (mc=ρ), which contradicts the transmission reduction goal. To increase the conciseness of messages, the complexity of the messages is limited by the constraint Imc,ρ≤H0. It is then possible to learn an encoding mc, which is maximally expressive about av in addition to being maximally compressive about ρ. With Lagrange multiplier β, the information bottleneck (IB) is defined as follows:(22)ℒIB=−Imc,av;ωIB+βImc,ρ;ωIB,where ωIB represents the parameters of the encoder and the decoder network.The value networks are then trained together with the encoder and the decoder by minimizing an overall objective:(23)ℒω=ℒTDωV+λℒIBωIB,where ω consists of ωV and ωIB and λ is the weight that trades off between these two subobjectives.The policy gradient [21] of station i is defined as(24)gi=∇πθiρi,aiAρi,ai,where(25)Aρi,ai=Qρi,ai−∑ai,′∈Aiπρi,ai,′Qρi,ai,′.The policy of stationi is optimized through the gradient ascend:(26)θi←θi+ηgi=θi+η∇πθiρi,aiAρi,ai,where θi refers to the parameters of station i’s policy and η>0 shows the step size. The details of the training process are shown in Algorithm 1.Algorithm 1:Geometric configuration optimization with multiagent reinforcement learning.
(1)
Initialize the DPD passive location system with target transmitter emitting signals, specify the number of stationsL and the central station c;(2)
Initialize neural network parametersω, θ(3)
Initialize the iteration countert⟵0.(4)
repeat(5)
fori=1:L,i≠cdo(6)
Intercept the signalsoti;(7)
Send the statesti,ati to the central station;(8)
end for(9)
The central station intercepts signalsotc and send mc to vice stations;(10)
Update the parameters of value networks:ω←ω+η∇ωℒω;(11)
for alli  do(12)
Update the parameters of policy network:θi←θi+η∇πθiρi,aiAρi,ai;(13)
end for(14)
Update the countert←t+1;(15)
until the task is completed or reaching the maximum of counter


## 3.1. Model Framework

In this paper, a DPD location system is considered withL mobile stations (e.g., UAVs equipped with positioning equipment), i.e., L DPD agents. Each agent transfers the intercepted signals to a central processing agent where the emitter’s position is estimated. Agents have no knowledge of the emitter and the electromagnetic environment. Due to the influence of multipath and noises, the signals received by different agents vary. To adapt to the complicated electromagnetic spatial distribution accurately, a MARL-based method, with positioning error being the reward function, is considered. The key elements in the MARL scheme are defined as follows:(i)
States. At each time step t, agent i intercepts signals, oti, emitted by the transmitter. The total messages it receives from other agents are denoted by miin. The state of agent i is denoted by sti=oti,miin,pti∈Si, where pti is the position of agent i at time t, defined in Section 3.1. Then, the global state is represented as s=s1,…,sL∈S, where S=∏i=1LSi.(ii)
Actions. Actions represent the decisions regarding where to receive signals at next step. Let ai=φi,di∈Ai denote the action of agent i, where φi and di represent its moving direction and distance, respectively. And, the joint action of all agents is denoted as a=a1,…,aL∈A, A=∏i=1LAi.(iii)
Rewards. This paper aims to develop agents that can properly adjust the geometry automatically to improve the positioning precision. To this end, we evaluate agents’ behavior by positioning errors. Two types of positioning errors are considered:(a)
CRLB is an effective index for evaluating the precision of a passive location system. Let the background position of the transmitter bep⋆=x⋆,y⋆. Then, the CRLB is a function of state s and the background position p⋆, i.e., CRLBp⋆,s, according to (15).(b)
The statistic error is a popular class of position errors, such as the mean error (ME) and the mean square error (MSE).Among the errors listed above, only CRLB can assess the geometry without estimating the target position, which reduces considerable amounts of time and computing in training. Therefore, CRLB is used as the reward function in training the DPD agents.


## 3.2. Learn to Optimize the Geometry

This section presents an efficient multiagent actor-critic algorithm for geometric configuration optimization in passive location tasks. The overall architecture of the proposed method is illustrated in Figure2. It is developed based on two main considerations: (i) factorizing the global value function into individual value functions with local observations for better collaboration and (ii) utilizing information constraints to facilitate communications and optimize the messages to tackle the transmission challenges.Figure 2
Schematics of the proposed method. The global value function is decomposed into the agentwise value functions.


### 3.2.1. Value Decomposition

As shown in Figure2, the global value function is factorized into linear combination of individual value functions as follows:(19)Qtotρ,a;ω=∑iQiρi,ai;ωi,where ρ=ρ1,…,ρN. And, ρi=s1i,a1i,…,sti,ati is the history of local observations, actions, and messages received. Local value functions are parameterized by ωV=ω1⊤,…,ωN⊤⊤. The policy of each station maps the history of observations and actions to the next action: πθiρi,ai. The joint policy for the location system is denoted by πθρ,a=∏iπθiρi,ai. Both actor and critic of each agent utilize the gated recurrent unit (GRU) [20] to process the input of observation history. GRU is a special kind of recurrent neural network that has the ability to capture the long range connections of states. The mixing network and individual value functions are trained in an end-to-end manner by minimizing the TD loss as follows:(20)ℒTDωV=r+γQtotρ′,a′;ωV−Qtotρ,a;ωV2.


### 3.2.2. Information Constraint

For the central station, it must collect observations from all the stations to estimate the transmitter’s position. Nevertheless, a vice stationj just needs the data that can help to make better decisions. Therefore, the central station must learn how to send messages as short as possible but enough for vice stations to act better. A natural solution is to add information constraints. In practice, to improve the effectiveness of messages sent to vice stations, it is necessary to maximize the mutual information of messages and station’s actions. Let c be the index of the central station; then, mutual information is defined by(21)Imc,av=∫pmc,avlogpmc,avpmcpavdmcdav,where mc represents the message sent from central station to a vice station and mjin=mcj≠c and av is the joint action of all vice stations.However, if this is the only objective, agents could always ensure a maximally informative representation by taking the identity encoding of raw data (mc=ρ), which contradicts the transmission reduction goal. To increase the conciseness of messages, the complexity of the messages is limited by the constraint Imc,ρ≤H0. It is then possible to learn an encoding mc, which is maximally expressive about av in addition to being maximally compressive about ρ. With Lagrange multiplier β, the information bottleneck (IB) is defined as follows:(22)ℒIB=−Imc,av;ωIB+βImc,ρ;ωIB,where ωIB represents the parameters of the encoder and the decoder network.The value networks are then trained together with the encoder and the decoder by minimizing an overall objective:(23)ℒω=ℒTDωV+λℒIBωIB,where ω consists of ωV and ωIB and λ is the weight that trades off between these two subobjectives.The policy gradient [21] of station i is defined as(24)gi=∇πθiρi,aiAρi,ai,where(25)Aρi,ai=Qρi,ai−∑ai,′∈Aiπρi,ai,′Qρi,ai,′.The policy of stationi is optimized through the gradient ascend:(26)θi←θi+ηgi=θi+η∇πθiρi,aiAρi,ai,where θi refers to the parameters of station i’s policy and η>0 shows the step size. The details of the training process are shown in Algorithm 1.Algorithm 1:Geometric configuration optimization with multiagent reinforcement learning.
(1)
Initialize the DPD passive location system with target transmitter emitting signals, specify the number of stationsL and the central station c;(2)
Initialize neural network parametersω, θ(3)
Initialize the iteration countert⟵0.(4)
repeat(5)
fori=1:L,i≠cdo(6)
Intercept the signalsoti;(7)
Send the statesti,ati to the central station;(8)
end for(9)
The central station intercepts signalsotc and send mc to vice stations;(10)
Update the parameters of value networks:ω←ω+η∇ωℒω;(11)
for alli  do(12)
Update the parameters of policy network:θi←θi+η∇πθiρi,aiAρi,ai;(13)
end for(14)
Update the countert←t+1;(15)
until the task is completed or reaching the maximum of counter


## 3.2.1. Value Decomposition

As shown in Figure2, the global value function is factorized into linear combination of individual value functions as follows:(19)Qtotρ,a;ω=∑iQiρi,ai;ωi,where ρ=ρ1,…,ρN. And, ρi=s1i,a1i,…,sti,ati is the history of local observations, actions, and messages received. Local value functions are parameterized by ωV=ω1⊤,…,ωN⊤⊤. The policy of each station maps the history of observations and actions to the next action: πθiρi,ai. The joint policy for the location system is denoted by πθρ,a=∏iπθiρi,ai. Both actor and critic of each agent utilize the gated recurrent unit (GRU) [20] to process the input of observation history. GRU is a special kind of recurrent neural network that has the ability to capture the long range connections of states. The mixing network and individual value functions are trained in an end-to-end manner by minimizing the TD loss as follows:(20)ℒTDωV=r+γQtotρ′,a′;ωV−Qtotρ,a;ωV2.


## 3.2.2. Information Constraint

For the central station, it must collect observations from all the stations to estimate the transmitter’s position. Nevertheless, a vice stationj just needs the data that can help to make better decisions. Therefore, the central station must learn how to send messages as short as possible but enough for vice stations to act better. A natural solution is to add information constraints. In practice, to improve the effectiveness of messages sent to vice stations, it is necessary to maximize the mutual information of messages and station’s actions. Let c be the index of the central station; then, mutual information is defined by(21)Imc,av=∫pmc,avlogpmc,avpmcpavdmcdav,where mc represents the message sent from central station to a vice station and mjin=mcj≠c and av is the joint action of all vice stations.However, if this is the only objective, agents could always ensure a maximally informative representation by taking the identity encoding of raw data (mc=ρ), which contradicts the transmission reduction goal. To increase the conciseness of messages, the complexity of the messages is limited by the constraint Imc,ρ≤H0. It is then possible to learn an encoding mc, which is maximally expressive about av in addition to being maximally compressive about ρ. With Lagrange multiplier β, the information bottleneck (IB) is defined as follows:(22)ℒIB=−Imc,av;ωIB+βImc,ρ;ωIB,where ωIB represents the parameters of the encoder and the decoder network.The value networks are then trained together with the encoder and the decoder by minimizing an overall objective:(23)ℒω=ℒTDωV+λℒIBωIB,where ω consists of ωV and ωIB and λ is the weight that trades off between these two subobjectives.The policy gradient [21] of station i is defined as(24)gi=∇πθiρi,aiAρi,ai,where(25)Aρi,ai=Qρi,ai−∑ai,′∈Aiπρi,ai,′Qρi,ai,′.The policy of stationi is optimized through the gradient ascend:(26)θi←θi+ηgi=θi+η∇πθiρi,aiAρi,ai,where θi refers to the parameters of station i’s policy and η>0 shows the step size. The details of the training process are shown in Algorithm 1.Algorithm 1:Geometric configuration optimization with multiagent reinforcement learning.
(1)
Initialize the DPD passive location system with target transmitter emitting signals, specify the number of stationsL and the central station c;(2)
Initialize neural network parametersω, θ(3)
Initialize the iteration countert⟵0.(4)
repeat(5)
fori=1:L,i≠cdo(6)
Intercept the signalsoti;(7)
Send the statesti,ati to the central station;(8)
end for(9)
The central station intercepts signalsotc and send mc to vice stations;(10)
Update the parameters of value networks:ω←ω+η∇ωℒω;(11)
for alli  do(12)
Update the parameters of policy network:θi←θi+η∇πθiρi,aiAρi,ai;(13)
end for(14)
Update the countert←t+1;(15)
until the task is completed or reaching the maximum of counter


## 4. Experiments

In this section, we develop a simulated electromagnetic environment for passive location tasks, based on which the agents are evaluated.


### 4.1. Environment

In the experiment, the simulator’s geographical coverage is 10 km×10 km, as shown in Figure 3. The transmitter is located in the center of the map and is equipped with an isotopically radiating antenna. The signal model, defined by (1), is employed with some modifications. The channel attenuation is a function of the receiver’s position p: bℓp∝λs/4πd, which follows the free space path loss. The noise and interference, as well as the multipath effect, are all compassed in the noise nℓ, which is modeled by the spatially white noise in (10). There are some low regions, highlighted in green in Figure 3, where the noises are stronger than other areas. It should be noted that due to these low SNR regions, the contours of SNR turn into irregular concentric rings. Furthermore, in the real world, it is also impossible to approach too close to the transmitter; therefore, there is a forbidden 1 km area around the transmitter in the simulator.Figure 3
The passive location task environment. There are one transmitter and three mobile stations. Noises become strong in the low SNR regions. Agents are randomly distributed near the edge of the square at the beginning of a task.


### 4.2. Setup

Consider one central station andL−1 vice stations with the task of cooperatively optimizing the geometric configuration in an area consisting of free propagation regions, low SNR regions, and forbidden regions. At each time step t, stations observe the environment to obtain the state st and make decisions about moving in direction φ on distance d, e.g., at. While moving, stations shut off the positioning and communication devices until arriving the next positions. If the time step t reaches the maximum of tmax, the location task ends. Figure 4 demonstrates the process of executing a passive location task in training and execution mode in different branches. With geometry formed by stations at each time step t, the reward is given by the theoretic error bound, CRLB:(27)rtst,at=−CRLBz,p,p⋆,where z is the received signals and p=p1,…,pL refers to the positions of all L stations. Also, the root mean square error (RMSE) is calculated to describe the positioning error more intuitively:(28)RMSEp,p⋆=1Nest∑k=1Nestp^⋆k−p⋆2,where p^⋆k is the kth estimation of p⋆ and Nest denotes the estimation times for each geometric configuration.Figure 4
The flowchart of finding optimal geometry in based on MARL algorithm.


### 4.3. Results and Analysis

The agents are trained in the passive positioning task mentioned above by setting the maximum time step to 100. For the sake of comparison, a basic version of the proposed method is also evaluated. In that version, the central station sends nothing except for the reward (naive DPD agents).The top segment of Figure5 shows the learning curve in terms of the averaged reward for DPD agents with communications versus naive DPD agents. DPD agents with communications converge to a much higher return than the naive DPD agents, which indicates that, with messages sent by the central station, vice stations are able to estimate the value function more accurately. In other words, communications are essential to geometry optimization in DPD location tasks. The bottom segment of Figure 5 illustrates the information bottleneck loss ℒIB against the training epochs. ℒIB declines quickly through training. The proposed agents can achieve a higher position precision with lighter communication overhead.Figure 5
Learning Curves. Top : the averaged return (CRLB) of the naive and communication version of the agents. Bottom: the information bottleneck lossℒIB curve over five random seeds.To show the learned decomposition of value functions, Figure6 demonstrates the error curve, normalized value functions, and the agents’ situations when learned agents perform a certain DPD positioning task. According to the top segment of Figure 6, both CRLB and RMSE decline with more steps taken by agents. Furthermore, the RMSE of positioning converges to the CRLB with respect to optimization steps. It means that agents can find geometric configurations where estimation error becomes closer and closer to the CRLB, which is the best achievable output for passive location systems. The middle and bottom segments of Figure 6 show that when the agents are in the low SNR area, their value functions decrease and the positioning error increases, which is consistent with our experiences.Figure 6
Learned agents executing the passive location task. Top : the CRLB and RMSE decline with more steps taken by agents. Middle and bottom: the agentwise factorized value functions change when agents are in different situations.Figure7 demonstrates the final geometric configuration found by the proposed agents as well as that optimized by the GA [1]. According to the geometry yielded by the GA, there is a station in the low SNR region, which is a suboptimal geometry. In other words, the GA optimizes the geometry on the empirical model, which cannot identify the low SNR regions in the simulator. By contrast, the trained agents can avoid low SNR regions and find the optimal geometry successfully.Figure 7
Optimized geometric configuration compared to the existing methods. Red : geometry found by our trained agents. Blue: geometry found by PSO, with one station in the low SNR region.


## 4.1. Environment

In the experiment, the simulator’s geographical coverage is 10 km×10 km, as shown in Figure 3. The transmitter is located in the center of the map and is equipped with an isotopically radiating antenna. The signal model, defined by (1), is employed with some modifications. The channel attenuation is a function of the receiver’s position p: bℓp∝λs/4πd, which follows the free space path loss. The noise and interference, as well as the multipath effect, are all compassed in the noise nℓ, which is modeled by the spatially white noise in (10). There are some low regions, highlighted in green in Figure 3, where the noises are stronger than other areas. It should be noted that due to these low SNR regions, the contours of SNR turn into irregular concentric rings. Furthermore, in the real world, it is also impossible to approach too close to the transmitter; therefore, there is a forbidden 1 km area around the transmitter in the simulator.Figure 3
The passive location task environment. There are one transmitter and three mobile stations. Noises become strong in the low SNR regions. Agents are randomly distributed near the edge of the square at the beginning of a task.


## 4.2. Setup

Consider one central station andL−1 vice stations with the task of cooperatively optimizing the geometric configuration in an area consisting of free propagation regions, low SNR regions, and forbidden regions. At each time step t, stations observe the environment to obtain the state st and make decisions about moving in direction φ on distance d, e.g., at. While moving, stations shut off the positioning and communication devices until arriving the next positions. If the time step t reaches the maximum of tmax, the location task ends. Figure 4 demonstrates the process of executing a passive location task in training and execution mode in different branches. With geometry formed by stations at each time step t, the reward is given by the theoretic error bound, CRLB:(27)rtst,at=−CRLBz,p,p⋆,where z is the received signals and p=p1,…,pL refers to the positions of all L stations. Also, the root mean square error (RMSE) is calculated to describe the positioning error more intuitively:(28)RMSEp,p⋆=1Nest∑k=1Nestp^⋆k−p⋆2,where p^⋆k is the kth estimation of p⋆ and Nest denotes the estimation times for each geometric configuration.Figure 4
The flowchart of finding optimal geometry in based on MARL algorithm.


## 4.3. Results and Analysis

The agents are trained in the passive positioning task mentioned above by setting the maximum time step to 100. For the sake of comparison, a basic version of the proposed method is also evaluated. In that version, the central station sends nothing except for the reward (naive DPD agents).The top segment of Figure5 shows the learning curve in terms of the averaged reward for DPD agents with communications versus naive DPD agents. DPD agents with communications converge to a much higher return than the naive DPD agents, which indicates that, with messages sent by the central station, vice stations are able to estimate the value function more accurately. In other words, communications are essential to geometry optimization in DPD location tasks. The bottom segment of Figure 5 illustrates the information bottleneck loss ℒIB against the training epochs. ℒIB declines quickly through training. The proposed agents can achieve a higher position precision with lighter communication overhead.Figure 5
Learning Curves. Top : the averaged return (CRLB) of the naive and communication version of the agents. Bottom: the information bottleneck lossℒIB curve over five random seeds.To show the learned decomposition of value functions, Figure6 demonstrates the error curve, normalized value functions, and the agents’ situations when learned agents perform a certain DPD positioning task. According to the top segment of Figure 6, both CRLB and RMSE decline with more steps taken by agents. Furthermore, the RMSE of positioning converges to the CRLB with respect to optimization steps. It means that agents can find geometric configurations where estimation error becomes closer and closer to the CRLB, which is the best achievable output for passive location systems. The middle and bottom segments of Figure 6 show that when the agents are in the low SNR area, their value functions decrease and the positioning error increases, which is consistent with our experiences.Figure 6
Learned agents executing the passive location task. Top : the CRLB and RMSE decline with more steps taken by agents. Middle and bottom: the agentwise factorized value functions change when agents are in different situations.Figure7 demonstrates the final geometric configuration found by the proposed agents as well as that optimized by the GA [1]. According to the geometry yielded by the GA, there is a station in the low SNR region, which is a suboptimal geometry. In other words, the GA optimizes the geometry on the empirical model, which cannot identify the low SNR regions in the simulator. By contrast, the trained agents can avoid low SNR regions and find the optimal geometry successfully.Figure 7
Optimized geometric configuration compared to the existing methods. Red : geometry found by our trained agents. Blue: geometry found by PSO, with one station in the low SNR region.


## 5. Conclusions

This paper analyzed the geometry optimization problem of passive location systems in a complex electromagnetic environment and proposed a MARL method to address it in a try-and-error fashion. In the method, by factorizing the global value function into the agentwise value functions, agents can learn to optimize the geometric configuration cooperatively. Moreover, by adding the mutual information constraints, the communication traffic from the central station to vice stations can be greatly reduced while effectiveness is ensured. A simulator with a sophisticated electromagnetic environment for passive location task is also developed, the results on which showed that the agents could find better geometric configurations than existing methods.This paper should be seen as a first attempt at learning geometric configuration optimization through MARL in a passive location task. Although DPD is used in the proposed method, it can be replaced by any other passive location algorithm (e.g., TDOA or AOA) to enhance the algorithm flexibility in various location scenarios.



---

*Source: 6691254-2021-06-02.xml*