Reward-based Autonomous Online Learning Framework for Resilient Cooperative Target Monitoring using a Swarm of Robots
Authors: Shubhankar Gupta
Saksham Sharma
Suresh Sundaram
Venue: Accepted by TMLR ()

Abstract: This paper addresses the problem of decentralized cooperative monitoring of an agile target using a swarm of robots undergoing dynamic sensor failures. Each robot is equipped with a proprioceptive sensor suite for the estimation of its own pose and an exteroceptive sensor suite for target detection and position estimation with a limited field of view. Further, the robots use broadcast-based communication modules with a limited communication radius and bandwidth. The uncertainty in the system and the environment can lead to intermittent communication link drops, target visual loss, and large biases in the sensors’ estimation output due to temporary or permanent failures. Robotic swarms often operate without leaders, supervisors, or landmarks, i.e., without the availability of ground truth regarding pose information. In such scenarios, each robot is required to exhibit autonomous learning by taking charge of its own learning process while making the most out of available information. In this regard, a novel Autonomous Online Learning (AOL) framework has been proposed, in which a decentralized online learning mechanism driven by reward-like signals, is intertwined with an implicit adaptive consensus-based, two-layered, weighted information fusion process that utilizes the robots’ observations and their shared information, thereby ensuring resilience in the robotic swarm. In order to study the effect of loss or reward design in the local and social learning layers, three AOL variants are presented. A novel perturbation-greedy reward design is introduced in the learning layers of two variants, leading to exploration-exploitation in their information fusion's weights' space. Convergence analysis of the weights is carried out, showing that the weights converge under reasonable assumptions. Simulation results show that the AOL variant using the perturbation-greedy reward in its local learning layer performs the best, doing $182.2\%$ to $652\%$ and $94.7\%$ to $150.4\%$ better than the baselines in terms of detection score and closeness score per robot, respectively, as the total number of robots is increased from $5$ to $30$. Further, AOL's Sim2Real implementation has been validated using a ROS-Gazebo setup.

Published in Transactions on Machine Learning Research (12/2024)
Reward-based Autonomous Online Learning Framework for Resilient Cooperative Target Monitoring using a Swarm of Robots
Shubhankar Gupta
skubhankorg@Visc..cc
Artificial Intelligence and Robotics Lab Deportment of Aerospace Engineering Indian Institute of Science, Bengakuru, India
Saksham Sharma
saksham. Mikshammppoo@@maal.com
Artificial Intelligence and Robolics Lab Department of Aerospace Engineering Indian Institute of Science, Bengakuru, India
Suresh Sundaram
ussuresh
Artificial Intelligence and Robotics Lab
Department of Acrospace Engineering Indian Institute of Science, Bengaluru, India
Reviewed on OpenReview: https: // openreview. net/ fortmr id= PamaWLgKKee
Abstract
This paper addresses the problem of decentralized cooperative monitoring of an agile target using swarm of robots undergoing dynamic sensor failures. Fach robot is equipped with a proprioceptive sensor suite for the estimation of its own pose and an exteroceptive sensor suite for target detection and position estimation with a limited field of view. Further, the robots use broadcast-based communication modules with a limited communication radius and bandwidth. The uncertainty in the system and the environment can lead to intermittent communication link drops, target visual loss, and large biases in the sensors' estimation out- put due to temporary permanent failures. Robotic swarms often operate without leaders, supervisors, or landmarks, i.e., without the availability of ground truth regarding pose infor- mation. In such scenarios, cach robot is required to cxhibit autonomous lcarning by taking charge of its own learning process while making the most out of available informat ion. In this regard, a novel Autonomous Online Learning (AOL) framework has been proposed, in which a decentralized online learning mechanism driven by reward-like signals, intertwined with an implicit adaptive consensus-based two-layered, weighted information fusion process that utilizes the robots' observations and their shared information, thereby ensuring resilience in the robotic swarm. In order to study the effect of loss reward design in the local and social learning layers, three AOL variants are presented. A novel pnrturbbttionggreedd reward design is introduced in the learning layers of two variants, leading to exploration-exxpootaall in their information fusion's weights' space. Convergence analysis of the weights carried out, showing that the weights converge under reasonable assumptions. Simulation results show that the AOL variant using the perturbation greedy reward in its local learning layer performs the best, doing 182.2% to 652% and 94.7% to 150.4% better than the baselines in terms of detection score and closeness score per robot, respectively, as the total nunber of robots is increased from 5 to 30. Further, AOL's Sim2Real implementation has been validated using ROS- Gazebo setup.

---

Published in Transactions on Machine Learning Research (12/2024)
1 Introduction
Target search, detection, tracking, and monitoring are crucial elements in variety of high-impact real-world applications like search and rescue (Scherer el al. 2015), firefighting (Harikunar el al.) 2018 Zhang et al. 2022), convoy protection (Spry et al. 2005), traffic monitoring (Khan et al. 2020), surveillance, etc., where the use of a multi-robot system can prove to be advantageous in terms of robustness and tracking perfor- mance, especially in unsafe and uncertain environments (Mohiuddin et al. 2020). With the advancement in sensor and communication technologies, making them miniaturized, inexpensive, and reliable, there has been an increased interest in the use of robotic swarms for target search and tracking (Senanayake et al. 2016 Robotic swarms mostly operate under the philosophy of Swarm Intelligence (SI) (Beni & Wang, 1993 Beni 2004). which is characterized by decentralized local sensing and control, local communication, and the emergence of self-organizing global behaviors (Sharkey 2007), along with three main advantages scalability, robustness, and adaptability (Bayindir & Sahin 2007), which are highly desirable in robotic swarms.
The problem of cooperative target scarch and tracking has been studied widely in the literature. There are many target-tracking works that solve the multi-robot problem settings involving an agile target being chased by a swarm of robots. However, most of these works tackle this problem mainly from control perspective. Morcover, such works assume accurate localization as well as target position estimation (Senanayake et al. 2016), i.e,, they do not consider temporaty or permanent failures in the proprioceptive (IMU, INS, odometry, etc.) and exteroceptive (camera, LiDAR, RaDAR, etc. - algorithmic/seen /sensor uncertainty in detection and relative pose sensing) sensors onboard the robots. Yao et al. (2007) propose a stable and decentralized control strategy based on artificial potentials and sliding mode control to capture a moving target using a robotic swarm in a specific formation, where the artificial potentials lake care of both tracking and formation tasks, and the sliding mode control ensures that the robots follow the required motion. |Wang & Gu (2011) consider the problem of cooperative target tracking using robot swarm with limited communication range, and propose distributed Kalman flter-based estimation scheme with implicit consensus for the target's 's position estimation and a distributed flocking algorithm for motion control. Blazovics et al. (0022) propose a simple rule-based distributed algorithm for target tracking and surrounding using a swarm of homogeneous robots based on the concept of basis behaviors, where the robots are aware of the target's 's position at all times.
To achieve dynamic obstacle avoidance and target tracking using a swarm of robots, RRdian et al. (013 use a distributed Kalman filter to estimate the velocity and position of the unknown dynamic convex obstacles and stochastic target, and a potential field approach to enable target tracking and obstacle avoidance among the robots. Kwa et al. (2020) present a fully decentralized swarming strategy offering tunable exploration- exploitation multi-agent dynamics, which is achieved by combining adaptive inter-agent repulsion and an adjustable network PSO-based strategy, thereby resulting in the optimal collective performance of the swarm corresponding to a specific k-nearest neighbor graph connectivity.
Unlike many works in the target search and tracking literature, this paper's main focus is on the target's position estimation part of the overall cooperative target search and tracking task, rather than the control part this paper considers temporary or permanent sensor failures that cause inaccurate localization and communication
target position estimation among the robots in the swarm. The robots use broadcast-basse modules with a limited communication radius and bandwidth, exteroceptive sensors for target detection and position estimation with limited field-of-vie,, and proprioceptive sensors for their own position estimation. The robot swarm has to operate under adverse situations involving temporary or permanent failures in the sensors resulting in large biases in their estimates, intermittent communication link drops, and target visual loss. Robot swarms often operate without leaders, supervisors, or landmarks, i.e., without the availability of ground truth regarding pose information; typically, the robots use the ground truth to identify which sensors or neighboring robots are undergoing failures. Without the availability of a landmark or a leader, the robots are required lo collaborate among thersselves. In this regard, a decent online learning framework called the Autonomous Online Learning (AOL) framework has been proposed in this paper, which is used to drive an adaptive and fault-tolerant information fusion process among robots. The AOL franework takes its philosophical inspiration from the concepts of 'independent learning' (Hockings et al., 2018) and "self-directed learning' (Garrison] 1997) in educational psychology in which the learner, devoid of
2

---

Published in Transactions on Machine Learning Research (12/2024)
any supervisor teacher, takes charge of its own learning process while making the most out of the available resources and information.
In the AOL framework, an autonomous online learning mechanism is intertwined with an implicit adaptive consensus- two-layered (local and social), weighted information fusion process, which is driven by the robots' observations and their shared information. This enables each robot to figure out which neighboring robot's information about the target's position can be trusted at a given time instant, thus bringing in aspect of resilience to the robotic swarm the robots undergoing sensor failures are less likely to be trusted by themselves well their neighbors for their sensor information, thereby improving their target's position estimates by giving more weight to the information shared by robots with functional sensors. The learning process in the AOL framework is inspired by that of the exponentially weighted online learning forecaster (Cesa- Bianchi & Lugosi 2006), which is a centralized online learning algorithm driven by a loss function calculated using ground truth. But unlike the exponentially weighted online learning forecaster, the AOL framework involves decentralized online learning which is driven by reward-like signals, exhibiting exploration-exxloitaliin
behaviors in the online learning process. Since the main focus of this paper is not the control part of the target search and tracking task, a simplified target search and tracking control strategy has been used, which can be replaced by any other advanced control strategy. For instance, the control strategies proposed by Radian et al. (013)) and Kwa et al. (2020) can be used along with the AOL framework to achieve better swarm formation control behavior while ensuring resilience in the robotic swarm.
To understand the effect of loss/reward designs in the local and social learning layers, three different AOL algorithms are presented, namely AOL-C, AOL-1P, and AOL-2P. novel perturbation- greedy reward design is introduced in the learning process of two AOL variants, leading to exploration-exploitatioo in their infor- mation fusion's s weights' space. Among the AOL variants, AOL-C involves comparative loss function in its local learning layer, AOL-1P involves the perturbation -reddygggyyyyy reward function in its local learning layer, and AOL-2P involves the perturbation-greedy reward functions in both its local and social learning layers.
Theoretical analysis of the three AOL variants carried out, showing that the weights converge under reasonable assumptions. The performance of the three AOL variants is then evaluated a simulated envi- ronment with adverse conditions involving sensor failures. The three AOL variants are compared against two decentralized fusion methods - Averaging- Congiinsu-nnssuusssss Fusion (ACF) and Galman-Consensiie Fusion (KCF). Averaging consensus and Kalman filter-based approaches have been used in various works (Katragadda el al.
2017 Zhang & Li 2019 Azam et al. 2020) for the purpose of multi-estimatee fusion. Hence, a comparison of the proposed AOL algorithms is provided with ACF and KCF fusion methods as baselines. However, sensor failures inducing sudden biases in the estimates may or may not increase their covariance, and therefore, covariance-based methods may not perform satisfactorily or may even fail. Simulation results show that the best-performing variant, AOL-IP, performs 182.2% to 652% and 94.7% 150.4% better than the baselines in terms of cumulative average detection score per robot and cumulative average closeness score per robot, respectively, the total number of robots is increased from 5 to 30. Simulation results further reveal that using the target detection confidence as reward signal for the update of weights in the social learning layer, as in AOL-1P, does better than using a perturbation-greedy reward-based learning strategy as in AOL-2P. However, using a perturbation-green learning layer, as in AOL-1P, does better than using a comparative Euclidean distance- lissance-based loss function as in AOL-C. Further, the top two performing AOL variants' (AOL-1P and AOL-C) Sim2Rcal aspects are evaluated in Gazebo using ROS1.
reward- based learning strategy for the update of weights in the local
The rest of this paper is organized follows: section 2 presents the problem formulation, and Section 3 presents the AOL framework, along with its three variants. Section 4 presents a theoretical analysis of the convergence of weights involved in the AOL, variants. Section 5 presents AOL's performance evaluation, comparing the three variants with two baselines. Finally, section concludes this paper. Further, a table of nomenclature is provided in the appendix (A.1), along with a brief discussion on the generalizability of the AOL framework (A.2) and the pseudo-code for all three AOL variants (A.3).
3

---

Published in Transactions on Machine Learning Research (12/2024)
Target
Robots
FOV
Comm. link
3
Comm. range
Figure 1: Cooperative Target Monitoring using Robotic Swarm
2 Problem Formulation
The problem of Decentralized Cooperative Target Monitoring considered in this paper involves a swarm of robots equipped with broadeast-based communication modules with limited range and bandwidth, propri- oceptive sensors (e.g., IMU, INS, optical encoders, etc.), and exteroceptive sensors (e.g., camera, LiDAR, RaDAR, etc.) with limited field-of-iiw (FOV), as shown in Fig. [ The goal of the robots is to detect, track, and stay as close as possible to the target while maintaining some distance from it, thus ensuring a high degree of detection and monitoring of the target at all times. However, the sensors onboard the robots may undergo temporary or permanent failures, along with intermittent larget visual loss and communication link drops due system and envirommental uncertainty, thereby jeopardizing the target monitoring goal of the robotic swarm. Therefore, the robots are required to collaborate over the communication network to help each other detect and monitor the target successfully.
let the total no. of robots be N. With AT as the sampling period (seconds), each robot i, Vi E [N], follows discrete-time 3-DOF kinematic model stated below:
Robot Kinematic Model:
cos Φ1,2 sin D..5
sinn Φi.i cos Φr,
X4+1; Xt,i + AT
Vt,
(1a)
01411 òτ,i + ATw
(1b)
where Nt,; E IR2 is the th robot's 2-D position vector (in m), Vr,i E R2 is the tt robot's body-axis velocity vector (m/s), dr, E IR is the tth robot's heading angle (radians), and We,s E IR is the ith robot's 's yaw rate (rad/s) at discreto-time t, respectively. Here, the body-axis velocity Ve,i and yaw rate Wy,s are bounded control inputs for the 7th robot.
Target Kinematic Model: the target's kinematics also follow the model stated as equation ( but its dynamics is unknown. The target's position vector X,, E I22 (in m). heading angle Φt I (radians), body-axis velocity Vr E R² (m/s), and yaw rate Ü.. E R (rud/s), respectively, can be represented by replacing with B (Bogey) in the set of equations ① Similarly, Vt.8 and Wr,B are the bounded control inputs to the target at time t, which are unknown to the robots since its dynamics is unknown.
Communication Model: let Reomm. be the range of communication, and let Pia be the communication link drop probability. The topology of the communication network between robots represented by a uni-directional dynamic graph G., whose adjacency matrix A, satisfies the following, Vi,j E [N] such that # j,
AAlks = {
[1|x,, Xt, & Reomm.) A (U (0, 1) 2 Pt))
(2)
: otherwise
4

---

Published in Transactions on Machine Learning Research (12/2024)
Longitudinal
Body-axis
FOV
RFoO
ROO
Robot
(a)
(b)
Figure 2: (a) Exteroception's field-of-viww (FOV). (b) Target Search strategy inspired by food foraging pattern used by Oxyrrhis Marina.
where [Alyy is the thth element of A,, (0, 10 is uniform random variable. For i i= [A.]i 0. The above equation implies that even if two robots are in the communication range of each other, the communication link between them may still drop with a probability of Pis-
The neighbor set of the tth robot at time t, denoted by Qa,i can be defined as follows, Vi E [N]:
2 [A.l IAj E [N]}
(3)
Further, define nt = 10a, i.e. ned the number of communicating neighbors of the ith robot at time t. Morcover, due to limited bandwidth, there is limit the number of neighbors the 8th robot can have at time iee, Ne, < Ri, where hardware. In case more than robots appear in the communication range of the ith robot, ny robots are chosen randomly out of them to be considered as communication neighbors, such that nz, S my holds.
21. N] based on the limitations posed by the communication
Proprioceptive Sensor Model: the proprioceptive sensor suite onboard the 7th robot is responsible for the estimation of its 2-D position and yaw angle, where the estimates are denoted as x R2 and d. R, respectively, and are modeled as follows:
x b, = X1,i + Mi,
(4a)
3 oe,i + Pi.
(4b)
where nh, E IR2 and uf, E RR represent. bounded arbitrary noise in the 7th robot's proprioceptive sensor suite's estimates, Vi [N].
Exteroceptive Sensor Model: the exteroceptive sensor suite of the tt robot is responsible for the detec- tion and relative position estimntion of the larget, in terms of the target detection confidence di, [0, 1] and the target's relative position estimate AxE; E R.?. The exteroceptive sensor suite is further characterized by a limited field-of-view (FOV), with detection range Reov and an angle-of-view 8 shown in Fig. hh. Let Pet be the probability of target visual loss. Tf the target lies outside the FOV region, de, = 0. On the other hand, if the target lies inside the FOV region, the detection confidence de, follows the model as stated below:
1 bo T
: r To 1o < S REOV ;r > RFOV) V (U:(0,1) < pa)
de,
(1 bo) exp (3 -
(5)
0
where := ||x,B x,, U,( 1) e [0 a uniform random variable, Vi E [N], and by E [0,1) and To E (0, Rrov) model parameters. The parameters by and To be tuned based the characteristics
5

---

Published in Transactions on Machine Learning Research (12/2024)
of the exteroception system to be installed in the robots. Note that the above model implies linear decay of the detection confidence till r ro, after which the decay is exponential, leading to near zero detection confidence value at r = Reov. In real-world usage, similar behavior is observed in camera and LiDAR- exteroception (Singh & Davis 2017: Najibi et al. 2019). Further, even the target lies in the FOV region of the robot, there may a visual loss of the larget with probability Put-
The relative target position estimation (Ax,,) model is stated as follows:
A L Xt, X1,i + Vt,i
(6)
where U4 IR2 represent bounded arbitrary noise in the jth robot's exteroception's relative target position's estimate, Vi e [N].
Thus, the combination of the estimates from proprioception and exteroception yields the combined sensor estimate of the larget's position x.D as follows:
LB = x + A
(7)
Control Law: robots use hybrid switching control law, which switches between the target chase mode and the target search mode based on whether they or their neighbors have detected the target.
For both the modes, the th robot's translational control law consists of two terms:
Ve, R Av.,;
(8)
where vf, is the 7th robot's velocity reference command signal responsible for either chasing or searching the target and AV,, is the 8t robot's velocity correction control signal responsible for avoiding collisions with other robots (more details in the supplementary document). Note that any other advanced target search and tracking control law can be used here. But for the sake of simplicity, the following search and tracking control strategy has been used in this paper.
Target Chase Mode: This mode gets activated whenever the 7th robot has detected the target (dr, >> 0) or one of its neighbors has detected the target (de, > 0, j E 94 In this mode, the 7th robot considers its estimate of the target's position K the reference position for the reference velocity control law. The yaw control of the robot makes sure that the robot's longitudinal body axis points towards the target's estimated position 8 A more detailed description, along with the control algorithm, is included in the supplementary document.
This mode gets activated whenever the hth robot has not detected the target (de, = 0) and either none of its neighbors have detected the target (de, 0, Vjj E Pe,i) or it has no neighbors (nr, = 0). In this mode, the 5th robot executes search pattern inspired by the food foraging pattern used by Oxyrrhis
Target Search Mode:
Marina Lowe et al. 2011 Harikumar et al. 2018). as shown in Fig. 2b. The robot first chooses random direction to move towards. With its longitudinal body axis aligned with that direction, it moves in that direction using its longitudinal velocity control while doing a growing sinusoidal maneuver (green curve in Fig. p) using its lateral velocity control to cover more area as it moves. After T3 discrete-time steps, the robot randomly chooses a new dircction and repeats the process.
3 Reward-based Autonomous Online Learning for Cooperative Target Monitoring
The Reward-based Autonomous Online Learning (AOL) framework proposed in this paper is a decentralized online learning framework designed for cooperative target monitoring using swarm of robots in non- stationary environment. Its ability to learn in real-time without the need for any ground truth allows the robots to exhibit high target monitoring success in scenarios where the sensors onboard the robots may be undergoing temporary or permanent failures, even when landmarks, supervisors, or leaders are unavailable. A systems diagram highlighting where the AOL module fits in each the robots is shown in Fig.
Within the AOL framework, three variants of AOL algorithms are proposed, namely AOL-C, AOL-1P, and AOL-2P, where AOL-C involves comparative loss function in one of its learning layers, AOL-1P involves
6

---

Published in Transactions on Machine Learning Research (12/2024)
robot' targel dataction conficence. target relative posttion estimate
Kinematics/Dynamiiis
Exterocappivv sensor (Gamerail (AA/iKaaar))
target confidence, larget position combined sensor
Lim Target visual loss. probability:
Control Law
robot's yaw angle estmates
Target Search and Chase Control reference velocity command
Propriaceptive (MU//NS/epica encoders)
postion
Collision Avoidance Control velocity correction contrel
social extimate
Communication Module limited comm. range link drop probabillly: Ph.
AOL
rebot's target detection confidence
Collision Avoidance Module
Tobot's target detection
confidancee
Communication
Reception
target posilion intermediate estimate, social waight
target detection confidence, target position intermediate estimates, social weights from neighbaring robots
Communication Broadcast
Figure 3: Systems diagram for each robot. in the swarm; AOL enables each robot to figure out which neighboring robot's information about the target's ' position can be trusted at given time instant, thus bringing in an aspect of resilience to the robotic swarm, especially under sensor failures.
novel perturbation- reward function in one of its learning layers, and AOL-2P involves the perturbation- greedy reward functions in both of its learning layers. Three different AOL variants are considered in order to study: a) the effect of using comparative loss compared to the perturbation-greet reward in the local learning layer (AOL-C versus AOL-1P), and b) the effect of using the perturbation-greet reward in the social learning layer (AOT.-1P versus AOT-2P).
All three variants of AOL algorithus involve four phases: communication phase sandwiched between estimation phases, and learning phase. The discrete-time variable 1,2, .T, where T is the discrete-time horizon.
The first estimation phase involves weighted fusion of the information available locally to the tt robot, i.e., its combined sensor estimate of the target position *,B and its previous-time final estimate of the target position 8 to form the intermediate estimate x of the target position, as follows:
Local estimation phase:
If the target is detected by ith robot's exteroception, i.e., de, > 0, then E ai(t 1) + a, 1)) 1.B
(9)
otherwise, if the target is undetected by 7th robot's exteroception, i.e., de, = O, then
8/. = x
(10)
Here, the local weight ar(t 1) is calculated as follows:
ailt 1) à,(t - ā,(t - 1)
(11)
+ at(t -1)
where the weights át 1) and (t(t - 1) are initialized as (; = 6)( 1. These weights are updated
the local learning layer of the learning phase. Note that the weight a, 1) is indicative of how much weight should be given to the combined sensor estimate in the local estimate fusion process at time i relative to the

---

Published in Transactions on Machine Learning Research (12/2024)
robot's previous-tine final estimate of the target position. Further, let á,(0) a²(0) á,(-1) df(-1) (for the AOL variants with perturbations, called AOI.-IP and AOT-2P).
Communication phase: The ith robot broadcasts the information {t, de, 8/, x wzs - 1)} and receives the information dt. 2 & Wz - 1)} from its communcctting noighbors j E 9 Here, the weight (t - 1) is initialized as w; = 1, and is updated in the social learning layer of the learning phase.
Social estimation phase: The second estimation phase involves weighted fusion of the information available socially to the robots, i.e., the intermediate estimates K given by its communicating neighbors ar and its own intermediate estimate x/.pp : to form its final estimate &i of the target's position, follows:
Σ
wiz(t 1)8
(12)
AjEAa,
where Aes:= = 22, U {i), and the social weights Way(t - 1) are calculated as follows:
0ū Exjees, (y
wiy(t (tj(t-1) 1) - -
(13)
where the weight (1:(( - 1) is updated in the jth robot's social learning layer of the learning phase, and the weights (t- 1 1) are obtained from its communicating neighbors j E 94, during the communication phase. The weight (t - 1) is initialized as (=)(0) = 1, Vi E [N]. Note that the weight wig (t - 1) is indicative of how much weight should be given to the jth robot's 's intermediate estimate in the social estimate fusion process at time relative to other robots in the set A4,5- Further, let th = (a((-1) = 1 (for the AOL variants with perturbations, called AOL-IP and AOL-2P).
Learning phase: The online learning of the tth robot is primarily driven by the objective of maximizing its target detection confidence de, directly or indirectly, which acts as a reward-like signal guiding mul- tiplicative exponential update process for the weights used in both the estimation phases. The learning phase for all three variants involves a periodic reset of the weights (s,,,, d((1), and @a(() to deal with the system uncertainty and non-stationary environment; the periodic reset is required to eliminate any biases accumulated in the learnod weights. In practice, the multiplicative exponential weight update process allows for higher learning rate that can cover the loss in performance due to periodic reset of the learned weights.
Three variants of the AOL learning phase are proposed, resulting in three different AOL algorithms, which are described as follows:
1) AOL-C: the weight, ()-(t-1) is updated via the multiplicative expooentiall strategy using := as the loss function, as follows:
: (1 -1-66 de.;)
(i(()) = ta (1- - 1) exp (-nw dr,)))
(14)
where Yhu is learning rate parameter. With the above update strategy, note that a decrease in the detection confidence of the ith robot leads to decrease in the weight (f). Equation (14 constitutes as the social learning layer.
The weights 1) and 11(1-1) are updated via the multiplicative exponential strategy using the following loss functions
Jo
i0; = : (1 - xx, 1r::11/ ID/ 1} zel = min || - x " 11 [D/
(15a)
(15b)
where ri, == arg jax,, Wiy(! - 1), and way(1 - 1) is calculated as per equation 13 normalization parameter, and rt, th robot is one whose intermediate estimate is considered to be the most accurate among the estimates of robots in the set A = 2g U (i) based the weights wiy(t 1). Thus,
Here, D, is a
8

---

Published in Transactions on Machine Learning Research (12/2024)
the weights âz(t- 1) and 6t 1) are updated as follows:
dx(t) 6; 1) exp (n,
(16a)
(t(t) dt - exp (-111
(16b)
where Ta 0 and The 0 are the learning rate parameters. Equations 16 constitute the local learning layer. Comparative Loss Function: given by equations (15) the loss functions ff and if are measure of
T. how close the estimates x,,n and 8i-1.B are to the socially best intermediate estimate 8, respectively, Therefore, an increase in these loss functions leads to a decrease in the weights (t) and (t), respectively. This implies an increase in the local weight (t(t) the estimate x S: is more close to the socially best intermediate estimate compared to - and vice versa.
Further, the learning phase also involves periodic reset - the weights á,(t), dt(t), and (t, are reset to 1 after every Tp discrete time steps.
AOL-C is summarized as algorithm (check appendix A.3
2) AOL-1P: update strategy for the weight wt 1) is given by equation 14 which the same in the previous variant, AOL-C.
The weights 6,(t- - 1) and at - 1) are updated via the multiplicative exponential strategy using novel perturbation-greeddy reward function definition described as follows:
T. = cal (t 1acfff--jdddap (t) + cp - de,i) + ea20,(t 11)d
(17a)
r == - -ca1 (t - (t) + 2 de,;) + 6a2(1 ar(t - 1))dt,
(17b)
where Aa; - 1) == az - at - 2), (t) == de,- 1 ep and e'p are the perturbation signals, and Cal > 0 and 6a2 > are learning rate parameters. With the above-described reward function definitions, the weights (t (t(1-1) and (: - 1) are updated as follows:
â, (t) = 6,(t - 1) exp (rf,)
(18a)
dt(t) át(t - 1) exp (rf.)
(18b)
where equations
188 constitule the local learning layer.
The learning phase involves a periodic reset the weights (t),), af(t), and (t(()) are reset 1 after every Tp discrete time steps, along with the perturbation signal Ep either taking the value ep Pmag R Ep 0 with equal probability, whereas e/ Pmag Ep- Otherwise, ep = =4-=0 0 at all other times.
Perturbation--reedl Reward Function: given by equations 17 , the reward functions "h and rf consist of three terms a difference-bused correction term (Aa,(t- (T)d, an inertia term (a:(t-1)dk or (I-a((t-))d..)) and perturbation term (e, or ea (I-d,)). The difference- based correction term acts the greedy part of the reward function, whereas the role of the inertia term is to resist abrupt changes in the reward due to the greediness of the difference-based correction term. The role of the perturbation term is to apply a perturbation periodically, thereby bringing in exploratory behaviors in the online learning process. After a perturbation is applied when the periodic reset is hit, the weight ag(t 1) may increase (ta,(t-a) - 1) > 0) or decrease (ta,(t-1) - 1) < 0), possibly leading to a change in the detection performance as well (td(()) > 0 (t, < 0). Therefore, a positive difference-based correction term (Aa,(t-1)Ad(()) > 0) leads to an increase in rū; and a decrease in rf., thereby increasing (;)()) and decreasing d'((), and vice versa. Note that the inertia term (a,(t- 1)d,; (1 (1-oa a, 1))d;.) is directly proportional to detection confidence (dr,);; a higher detection confidence promotes a larger inertia. Since perturbations should be avoided if the detection confidence is already high, note that the magnitude of the perturbation term (cp - de,)) or e1,(1 - -r..)) decreases as the detection confidence de,; increases.
AOL-1P is summarized as algorithm 2 (check appendix A.3
3) AOL-2P: the weights á,(t 1) and at(t 1) are updated using the same strategy as in the previous variant, AOT--PP, given by equations 17 and
9

---

Published in Transactions on Machine Learning Research (12/2024)
The weight (t:((t11) is updated via the multiplicative exponential strategy using novel perturbation-gredly reward function definition described as follows:
TT. Cwi CwI (AAu(t-1)Ad((t)) + c1 - -d,s) Cuz 1)d4,
(19)
where Aw;(t - 1) == w:i(t - w;,(t - 2), (t) == d,,; de ey is the perturbation signal, and Ow1 > O and 6w2 > 0 are learning rate parameters. The terms Aw ())d,(t) and wa((t - 1)d,, can be considered as difference-based correction term and the inertia term, respectively, whereas ext (1 - d,,a) is the perturbation term that only active when the periodic reset is hit. With the above-described reward function definition, the weights tha 1) are updated as follows:
(ta(t) ri - 1) exp (r)).)
(20)
where equation 20 constitutes the social learning layer.
The learning phase involves a periodic reset the weights a,(t), df((), and (t((t) are reset to 1 after every Tp discrete time steps, along with the perturbation signal egt taking the value ex Unif. (m.eg)) where Ponag E Ryo and Unif. Pmag) is a uniform random variable; this ensures that the ex: values for different i E [N] are likely to be different at the periodic reset. Otherwise, ep" = at all other times.
19 the reward function ry consists of three terms a difference-basod correction term (Awr,(t Ad (T), an inertia term (wu(t-1)d, and a perturbation term (e (1 d1,)). The difference-based correction term acts as the greedy part of the reward function, whereas the role of the inertia term is to resist. abrupt changes in the reward due to the greediness of the difference-basee
cetturbation-greedee Reward Function: given by equation
correction term. The role of the perturbation term is to apply perturbation periodically, thereby bringing in exploratory behaviors in the online lcarning process. After a perturbation is applied when the periodic reset is hit, the weight wa(t - 1) may increase (Awa(t 1 0) or decrease
- 1) < 0), possibly leading to a change in the detection performance as well (Ad,(t) > 0 or (t) < 0). Therefore, a positive sifference-baseer correction term (Awa(t - ()) > 0) leads to an increase in ri thereby increasing (t), and vice versa. Note that the inertia term (ta(t-1)d, 1)d,,) is directly proportional to detection confidence (dr,); higher detection confidence promotes a larger inertia. Since perturbations should be avoided if the detection confidence is already high, note that the magnitude of the perturbation term (e% (1 de,;)) decreases the detection confidence di, increases.
AOL-2P is summarized as algorithm 3 (check appendix A.3
4
Convergence Analysis of Weights
Note that at time t = 0, the weights a,(0) = 1, d((0) = 1, and (a)(0) 1. Without the loss of generality, analysis of the three AOL variants is carried out without considering a periodic reset; this is done to gain insights into their convergence behavior just after a periodic reset is hit and just before the next periodic reset is about to hit. Note that each time instant when the periodic reset hits can be considered as t = O from an analysis perspective. Therefore, the analysis can be regarded as valid for each time interval between two successive periodic reset hits. In practice, given a periodic reset time Tp, a higher learning rate (sct by learning rale parameters) would lead to quick convergence of the weights before the next periodie reset hits. If the learning rate is low, the weights may not converge their respective convergence values by the time the periodic reset hits. However, their convergence behavior would remain the same, as shown in the theoretical analysis in this section.
In order to analyze the effect of perturbations in the rewards of AOL-1P and AOL-2P, for ease in theo- retical analysis, it assumed that their reward functions undergo perturbations only t t= 1 due to the perturbation- cetturbation-greddy reward design. Note that in practice, the perturbations occur just after the periodic reset after every Tp discrete-time steps.
10

---

Published in Transactions on Machine Learning Research (12/2024)
4.1 AOL-C
Consider the loss function 47, == (1 - de,), and note that 1h E [0, I], since dr,, E [0, Thus, using equation [14,, the weight (t) can be written as: (t) = exp( La where the cumulative loss Li; := E 5,51 and ths (0) = 1, Vi E [N].
Without the loss of generality, define ni == arg minjeA.; Lijj" such that the ii. th robot is a unique robot that incurs the least cumulative loss among the robots in the set As, Lt U {i} at time t, where Qg is the neighbor-set of the 8th robot at time L.
Assumption 1: lime A and lim Ni, exist uniquely, such that li Ar, = Aoott and lime st, is Vi E [M]
Remark Assumption implies that the neighborhood configuration (in terms of the set Maa)) and the performance configuration (in terms of detection confidence dz,)) get fixed as + co. That is, for the 7th robot, Vi E [N], at t oo, there exists a unique robot foc,i that incurs the least cumulative loss DM, among the robots j E A
Note that the cumulative loss satisfies 0 S Lt I t (due to loss function's definition), Vj E Ar, and Lis.. < <Lig (due to jł,3 's definition); Vj E Ar, \ (15,1-
Assumption 2: LN3 - LWi. 2 e > 0, such that B E (0, and 0 O<EEL 1.
Remark 2: Assumption 2 implies that the lower bound on the difference between the cumulative loss incurred by the jth robot and that of the Jr, th robot grows sub-linearly (0 < 3 < 1) or lincarly (B 1) with the discrete-time 4, such that the magnitude and the rate of growth are finite but can be arbitrarily small (0 < & 1 and 0 < 3 & 1). Note that assumption 2 generalizes over many practical scenarios. For instance, consider the scenario where the jł, robot is fixed over time, i.e., Ji.; = jō.; and satisfies [", 4.... 2 0,
ne
for 2 1, i.., the jū,; th robot incurs the least loss at any time t. This implies that LM - - 2 et > 0, where 0 < << 1, which a special case under assumption 2 when , = 1. In practice, assumption 2 would be satisfied for any such scenario where the best (jž () robot stays fixed for some finite time duration and may change intermittently over time. In the simulation studies (section 5), such intermittent changes result from cemporary/permanee sensor failures, intermittent communication ion link loss, and target visual loss.
Theorem 1. Under assumptions and 2, Vi E [N], AOL-C algorithm's 's weights wiy(t) satisfy the following:
lim way (1) 0, 1-000
Vj Aoo,t \ (j
(21)
and
lim Wis (1)
(22)
1000
where i is the index of the robot that incurs the least cumulative loss among the robots in the set Aso,s as ( + co.
Proof. Since (t) = expl The
expp-mmo.L LY,), from equation 13 note that (ty(t) can be ro-written as follows:
exp( no (on
(ty(t) ExpEA. exp( (- Li
(23)
Multiply both numerator and denominator by exp ( (-Tho ) to get:
exp(-nnu mpe 1Y
Wiy (t) 1+ 1 ExyEA...VET))) expl ma M (L
(24)
L
Note that the cumulative loss satisfies 0 5 Lij < t (due to loss function's definition), Vj E Ar, Using this condition along with assumption 2, Vj E Ars (5 we get
t 2 D
L w Li
2 ct >
(25)
11

---

Published in Transactions on Machine Learning Research (12/2024)
where 3 E (0. 1] and 0 O<<< << 1. Using equation 25 in equation
(4
Vj E As,s \ (its), we get
nut)
expl exp(-nuuct()
+ EnyEA exp(- 1 c1)) S wis(1) S
(26)
1 1+E Enyemublitn exp(-1mut)
and for j j= j8 we get
Exyes. Mit expl )B) Way, (t)
(27)
1+ ExyEA. exp(-nuut)
Taking lim (+) on both sides in equations 26 and 27 under assumption 1, gives the desired result as equations 21 and 22
Considering the loss functions given by equations 15a) and 15p), note that E. E [0, 1] and dg E [0, 1]. Therefore, the convergence results for the weights at(t) can be derived by carrying out an analysis similar to that of the weights (1) under an assumption similar to assumption 2, as shown above.
4.2 AOL-1P
Since the updale stralegy for the weights ()((() for AOL-1P is the same as that of AOL-C, therefore, the convergence proof for AOL-1P's weights wxy(t) (t) is given by theorem !
Consider the weights (t), Vi E [N], defined by equation 11 and the update strategy given by equations
17 and 18 Define R2; == Σx= ri and Rg. = El E.s where ri and rf, are given by equations
I7a) and p), respectively. Note that (0) = 1 and (0) = 1. Further, a,(t) E [0, and de, E [], Vi E [M]. hhreffore, for t 2 1, the difference-based correction term satisfies Aa;(t (t) E [-1,
where Aa,(t 1) = (t - 1) - at(t 2) 2) and (d,(t) = d,, - d--1.-
Assumption 3: For 2 1, the difference- basedd correction term La, (t,(t-1)ddd ()) satisfies the following: either E Aa,(s ()) 2 è
or E.m1 Aay(s 1))d((s) S eel < 0, where 0 < < 1, E (0, Aa,(t - 1) = (t 1) (t - 2), and (d,(t) de, d
Remark 3: Assumption 3 implics that the cumulative sum of the difference-based terms over time is cither lower bounded by a positive term which is linear (3 = 1) or sub-linear (3 E (0, 1)) in time, or upper bounded by a negative term which is linear (3 = 1) or sub-linear (B E 10) in time, such that the magnitude and the rate of growth of these bounds are finite but can be arbitrarily small (0 < c & and 0 <<> 3 < 1). Note that assumption 3 generalizes over many practical scenarios. For instance, consider the scenario in which either Aos(t (l) 2 O or (l, (1) I -e 0, for t 2 1, implying either E.= Aas(s- (s) 2 et > 0 or Σ (ai(s (s) S -et < 0, which is a special case under assumption 3 when B = 1. This case corresponds to the scenario in which an increase in a;(t) always causes either an increase or decrease in de,s for 2 1. Considering equation (9), this further implies that either the estimate S the estimate K|-1,B is more accurately estimating the target position x,,8 for 2 1. In practice, assumption 3 would be satisfied for any such scenario where either & or x/ stays more accurate than the other for some finite time duration and but the accuracy ranking among these may change intermittently over time. In the simulation studies (section 5 such intermittent changes result from temporary/ tettraryypeemmanne ppermanent sensor failures, intermittent communication link loss, and target visual loss.
Theorem 2. For t 2 1, under assumption 3, given ea < 2ee. (check equations AOL-IP algorithm's 's weights ax(t) satisfy the following;
17 ), Vi E [N],
lim ax(t) = 1, if Σ Day(s - -s 1)dd((s) 2 et 0 t0000
(28)
and
lim az(1)= 00 if Σ Aaz(s 1)Ad((s) S -ct3 <0 t-000
(29)
where 0 <6<1, 8 (0, Aoi( -1) 1) as 1) as(I -2), and Adi(1) de,s de-1.i-
12

---

Published in Transactions on Machine Learning Research (12/2024)
Remark 4: Since 0 < 1, the condition Ca2t'-8 2cca1 further yields ca2t!-8 < 20a1 This can be satisfied either by choosing 6a2 = 0, or by choosing a time-varying 6a2 such that ea2(() xx t for e > 0, where 1 1- 3 - S O. In practice since the AOL-1P algorithm involves a periodic reset with a period of T, discrete-ting stops, this condition implies eazTp < 2eal. This condition can be satisfied when ea2 is chosen to be much smaller than cal or 622 2 O.
Proof. Note that (4) E [0, 1] and de, E [0, 1J. Thus, Aa - 1) E [-1, and (d)() E [-1, Further, a,(t - 1)dd, 1]. Using equations [11b and 18 the weights (t), V E [V], can be written as follows
exp (
as exp (R + exp (Ri + exp (R RTy,))
(30)
0 or ep = & cg = Pmag: and for > 1, ep e = (assumed for ease in analysis, without the loss of generality), vi E [N]. Thus, Re. Ri, can be written as
Since the perturbations occur just after = 0, therefore at = 1, either ep = Pmag c 0
Rq Ry, Pmmag d1,)) 2e Σ Dai(s 1))d((s) + ea2 Σ(1 [1
2ar(s 1))da,
(31)
Note that (1- 2c.(( - 1))dk, E [-1,1] and Ao,(1 1)Ad((1) E [-1,1] For / 2 1, using assumption 3, we get the following two cases (0 E < 1, _ (0,
1) Case A: Ei=1 (ai(s (s) 2 at >0
Since (t,(t- - (t) E [1-1, note that t 2 Ei (oa(s1)/dd( a. Further, (1 - (t 1))de, E [-1,1]. Therefore, using equation (31,, the weight a:(() satisfies the following
1
S aa((t) S
(32)
1 + exp P#mma (1 - d1.s) - 2 + eazt)
+ exp P+ d1.i) 2eatt ca2t)
Taking (-m on equation 32 given ex2t < 2ecal, yields the desired result as equation 28
2) Case B: Ei (s 1)Ad S ct <
Since (t-1t))d e [-1, 1], note that -t S Ei= []., Therefore, using equation (31), the weight (t) satisfies the following
-et.. Further, (t- E
1 + exp (# (1 - d2.)) + 2eat + eazt) S (a((t) S
(33)
1 exp (+ (1 d1.;) 2cc Ca2t)
Taking (-) on equation 33 given eazt! 1 < 2cca1, yiclds the desired result as equation 29
4.3 AOL-2P
Since the update strategy for the weights dn(!) and (!) for AOL-2P is the same as that of AOL- therefore, the convergence proof for AOL-2P's weights (t) is given by theorem 2
Consider the weights w. Vi E [N], defined by equation 113 and the update strategy given by equations 19h and (20h. Define RY, == D ri, where ris is given by equation (19h. Note that (0) 1. Further, (t) E I and de E 10, Vi [N]. Therefore, for t 2 1, the difference- based correction term satisfies (wa(t- (t) 1-,, where Awa(t 1) wa(t 1) wi(t - 2) and Ad, de, d
Without the loss of generality, define of = arg maxjeAA, E'=1 Awyy(s (s),;(s), such that T. is unique among j E Ar, this implies that a cunulative change in the vi,," robot's social weight causes the most positively or least negatively aligned cumulative change in its confidence score compared to other robots including the 7th robot and its neighbors at time t. Therefore, as per the weight update rule given by equations 19 and (20). q th robot is a unique robot among the robots in the Ar, whose intermediate estimate is given more weight while performing the weighted fusion as given by equation 12
13

---

Published in Transactions on Machine Learning Research (12/2024)
Assumption 4: lime-+2o A and lim E. exist uniquely, such that lim A Azo,s and lime-roo H7.3 To Vi [N].
Remark 5: Assumption 4 implies that the neighborhood configuration (in terms of the set Ae,)) and the performance configuration (in terms of the difference- based correction term (Amr (t - ())d get fixed as 1 20. That is, for the 7th robot, Vi [N], at t + co, there exists unique robot Loo that exhibits the greatest cumulative sum of the siference-bbsed correction term (i.e., E.=1 Awyy(s 1s)d among the robots E Acc.
Note that the cumulative sum of lifference-based correction terms satisfies -t = E.= Awiy(s-1)Ad I since Aw;(t (t) E [-1,1]. Further, due to L.. 's definition, the cumulative sum of difference-baset correction terms satisfies Ee=1 Away - 1)) E.=1 Aw 9i. (s-1)
1)Adg;, (s), Aq E Ar, \ (g
Assumption 5: Ex=1 Amar oi, (s 1) 1) (8) - Amgg(s - 1)ddg(s) 2 el > 0, such that B E (0, 1] and 0 < < 1, Vy E Ae, Ae, (
Remark 6: Assumption 5 implies that the difference between the cumulative sum of sifference-based correc- tion terms corresponding to l and q (4q E At, \ (a),)); i.c., the difference Σi= Awgr, .i, (s- (s-1)Adg; (s)- A - (s), has lower bound that varies linearly (B 1) or sub-linearly (0 < B 1) with the discrete-time i, such that the magnitude and the rate of growth of this lower bound is finite but can be arbitrarily small (0 < e < 1 and 0 < B < 1). Note that assumption 5 generalizes over many practical scenarios. For instance, consider the scenario where the L. th robot is fixed over time, i.e., L. = %. and satisfies Awa.... (t - 11)AAd (t) - A - 1) (t) 2 e <> 0 for t 2 1, Vq E At,s \ (95. i.e. the % robot's change in its social weight shows the most positively aligned or the least negatively aligned change in its detection confidence score compared to other robots including the 7th robot and its neighbors for time 2 1. This implies E.= AWlr, 9i. (s 1)Adg;, (s) Ei=1 AW (s 1) 2 et, which is special case under assumption when = 1. Considering cquation 122 this further implics that the estimate 1
, is more accurately estimating the target position x,, compared to other estimates in the weighted fusion, for 1 1 1. In practice, assumption 5 would be satisfied for any such scenario where the best (g), th) robot stays fixed for some finite time duration and may change intermittently over time. In the simulation studies (section 5). such intermittent changes result from temporary haaeey/peermnn /permanent sensor failures, intermittent communication link loss, and target visual loss.
Theorem 3. Under assumptions 4 and 5, given ew2t -B < cew1 (check equation algorithm's 's weights Wiy() (1) satisfy the following:
19 19), Vi E [M, AOL-2P
lim wiy(t) = =0, Vjj E Aoc, \ (g 1000
(34)
and
lim Wio (1) -1 14000
(35)
where 0 < e 1, 8 (0, 1], and goo, is the index of the robot that exhibits the greatest cumalative sum of the difference-based correction term among the robots j E A
Remark 7: Since 0 < c < 1, the condition euzt! < cow1 further yiclds Cuzt! < CwI- This can be satisfied either by choosing 6w2 = O, or by choosing a lime-varying ew2 such that ew2(l) xx t- for e >>> 0, where 1 - B - c S 0. In practice, since the AOL-2P algorithm involves a periodic reset with a period of T, discrete-time stops, this condition implics T, 1-B < Cwi- This condition can be satisfied when ow2 is chosen to be much smaller than ew1 or 6w2 O.
Proof. Consider wiy( given by equation equations 19 and 20 to get
13 ; given wa(()) = 1, Vi E [N], use the updale rule given by
(t)
exp (R
(i)(1) = 1 E Ar.i) Eger,, (gg(1)
1 E Ac.i)
(36)
Egese, exp (
14

---

Published in Transactions on Machine Learning Research (12/2024)
where R4, Ee=1 12 and 1(:) is the indicator function; 1(j E A4,3) if the condition j Ar, is satisfied,
otherwise 1(j E A k.l 0. Dividing both numerator and denominator in equation (36) by exp (Fi,nt.)) we
get
exp (R RV E Lg exp (Ri, R.g.
exp (Rt Ri.o5,
(1)(t) Ar.))
((,))s,) 1 EgEAr, (o7,)) exp (RY,
(37)
R L.7.
Using the reward definition of ri, as given by equation 19 therefore at 1, eg Unáf (0, Pmag) and for t > 1, ey of generality), Vi E [ R,, can be written as
since the perturbations occur just after 0, (assumed for ease in analysis, without the loss
Ry e d1.;) + Call Σ
Awsils - ()) + cm2 Σ wiils - 1)d
(38)
5=1
where Awa(t- - 1) = Wrr(t (t- - 1) - wis 2) and (d; = de,s -1-1.3- Using equation 188
Vg E A we get
ep d2,g) - Ey whz (1 d2.0),)
R1g
Ri
twwl E11= AWgg (8 1) - -E.=1 Awar, (8- 11)11)ddg,. (8)
(39)
+ew2 ===1
Wgyo( 1)da,9 97. - 1)dear..))
Since (8) E [0, 1] and de.g E [0, 1], note that Awgy(t 1) E 1[-1, 4q Ae.s. Using this along with
assumption 5, we get
< at I
Au 9t (s 1)Adu; (5)
Σ
Aw 1)dd (s) St
(40)
==1
where 0 < < <<< 1 and B E (0, 1]. Further, note that Wgy(t i 1)de, E (0. Therefore,
(w 8 1)da.9 Wo, (8 1)da.tt.)) S
(41)
Using equations (40) and in equation 39
4a E Asoit \ (9sc,ih we get
vi (1 - d1.9ta.)) - Ewit - Ewit
Ep (1 d1 - Ep
< [RY, - Rtoo
(42)
wer x. (1 d1 twiel + ewzt
2e (1 d2.g)
Ep
Given ewat!-P cow1, taking lim (+) equation
42 Vq Abous fg implics
RR
0 as 1 x
(43)
Taking (+) both sides of equation 37 ( we get
under assumption 4, and using equation
43
Vj E Aso.i
lim Wiy =
(44)
100
and for j == i: get.
lim Wint. (t)
(45)
100
Equations 44 and 45 complete the proof.
15

---

Published in Transactions on Machine Learning Research (12/2024)
AOL ver.2 Time = 40.0 sec.
20
77
25
10
76
x17
25
74 -105
-10
2G
-20
X7
X27 xil
X33
X26
22221 229
-30
X22
X2930 XIO
X23
-40
3o4 x22
xLg 2f
-50
x11 x16
-60
-70
-40
-20
20
40
60
21 (m)
Figure 4: Screenshot of the AOL-1P simulation; orange triangle target, blue x robots with functional proprioception, red x robots with faulty proprioception, green dash-dot lines functional exteroception, red dash-dot lines faulty exteroception, purple-yellow links communication links, the flow of information is from purple towards yellow.
5 Performance Evaluation
The proposed AOL framework is evaluated using simulation setup involving N = 5, 10, 20,20,22 30 robots executing the cooperative target monitoring task discussed in the problem formulation. The communication range Reomm. and the commmicctiin link drop probability Pia are set to be 30 m and 0.1, respectively, with the limit on the number of communication neighbors as n4 = 3. The exteroceptive sensor model's parameters are set as Reov = 15 m, 0Fov = 160 degrees, with the target visual loss probability as pw = 0.1. The parameters for the detection confidence model are set as r, = 10 m, and b, 0.1.
The simulation results averaged over 100 simulation runs. Each run involves a time horizon of T = 600 discrete time steps, with a sampling period of AT 0.1 sec. The robots follow the control law described in the problem formulation while trying to maintain a safe distance of 8 m from the target (more details in the supplementary document). The target randomly changes its velocity and yaw rate after every 5 seconds. The robots and the target always stay insido a square region of side length 100 m by overriding their control laws to gel away from the region boundary. At the starl of each simulation run, the robots are always spawned near the center of the square region, whereas the target spawned randomly but sufficiently near to the robots so that at least one of the robots is likely to detect it at the start of the run. This is done since the main focus of this paper is not the target search but target detection and monitoring.
In the considered adverse scenario for the simulation, 50% of the total robots are chosen randomly at times 0, 10, 20, 30, and 50 seconds, that exhibit temporary failures in their exteroceptive sensors. Further, 50% of the total robots are chosen randomly at time 10 sec., that exhibit permanent failures in their proprioceptive sensors. The noise H², and "f: in the proprioceptive sensors (equations (4)) that are functional is assumed to be Gaussian with a mean of 0.01m and 0.02 deg,, respectively, with a covariance of 0.01m² and 0.0lrad.2, respectively. The proprioceptive sensors that fail have noise terms that are still Gaussian but with large bias (mean) of 5m and 10 deg., respectively, with either a covariance of 0.01m² or 5m² with equal probability and 0.01rad.? 2 or 5rad.? with equal probability, respectively, since the bias and the covariance may be uncorrelated. Similarly, the noise ug in the exteroceptive sensors (equation ( that functional is
16

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm
Features
Remarks
performs poorly when sensor failures do not lead to increase in estimation covariance performs poorly under adverse scenarios considerable no. of robots undergoing failures
KCF
Not adaptive; covariance-based
ACF
Not adaptive; equal weights
2nd best AOL variant; performs significantly better than AOL-2P, KCF, ACF in
adaptive; comparative loss + local weights; detection confidence as a reward - social weights
AOL-C
adverse scenarios
the best AOL variant; performs significantly better than AOI-2P, KCF, ACF in adverse scenarios
adaptive; perturbation greeody reward - local weights; AOL-1P
detection confidence reward + social weights
the worst AOL variant; performs better than KCF, ACF in adverse scenarios
adaptive; perturbation--reedly reward - local weights; porturbation-greelt reward + social weights
AOL-2P
Table Comparison of the three AOL variants with Kalman-Consensuu Fusion (KCF) and Averaging- Consensus Fusion (ACF)
Avg. Cumal. Detection Scoce per robot after 80 sec.
Avg. Oamad. Dotection Soore robut after 60 sec.
ADL-C
250
ADL-C
250
ADL LP
Cuaml Detection Store
AOL
200
Cumai. Datectio
200
150
1.50
50
0
1: AOL-C, AOL-IP, 3: AOL-2P, 4: No comon, S: ACIE KCF
1: AOL-C, 2: AOL-1P, AOL-2P, 4: No commi, 5: ACE; 6: KCF
Avg. Oamul. Classress per rohot
Avg. Canal. Closeness Scote vobot o/ter
ADM--C
ADL-C
AOL-2P
ADL-1P
Comnl Cloemeee Sooee
400
X0L-2P
A01-1P
Nu Cunnn.
Nu Counts
Closmos
AUS
300
NCV
XOP
KCP
200
3 4 5
2 3
1: AOI-C, AOL-IP, AOL-2P, commn
KCF
AOL-C, AOL-IP, AOL-2P, No comnm., ACF, &. KCF
(a)
(b)
Figure 5: (a) Comparison results; 100 sim. runs for each method, N = 5. (b) Comparison results; 100 sim. runs for each method, N 10.
assumed be Ganssian with mean 0.01m with a covariance of 0.01m². For the exteroccptive sensors that fail, the noise term is still Gaussian but with large bias (mean) of 10m with a covariance of 0.01m² 5m2 with equal probability since the fault may may not lead to an increase in the covariance.
The algorithms are evaluated based on two types of scores 1) cunulative average detection score per robot: + レ ET, dr,s, and 2) cumulative average closeness score per robot: 1 AN ET (f,,, where (E == min 8
|= }.. Note that the robots try to maintain a distance of 8 m from the target. The cumulative average detection score per robot is measure of the target detection performance of the robots
17

---

Published in Transactions on Machine Learning Research (12/2024)
in the swarm; higher score means that larger mumber of robots are able to detect and monitor the target for a longer duration. As the number of robots in the swarm increases, crowding occurs, making it harder for all the robots to be able to detect and monitor the target- the robots closest to the target are likely to block the robots behind them in the swarm. Therefore, in such scenarios, the cumulative average closeness score per robot becomes a better criterion to judge the swarm's performance, since is a measure of how close the robots are to the target. Note that in scenarios where the target can be fully observable to the swarm (i.e., all the robots can view the target simultaneously), the cumulative average detection score and the cumulative average closeness score play equivalent role in judging the swarm's performance. Whereas their roles differ in scenarios where the target is partially observable to the swarm (ie., not all robots can view the target simultaneously).
For all three AOL variants, a coaulation-based parametric study is carried out to find a suitable set of parameters that result desirable performance. Based on the parametric study, the parameters for AOL-C are set as D. 15 m, In 15, Tw = 15, and Ta 0.01; that of AOL-1P are set as Tp = 15, T/ 15, eal 10, 6a2- = 0.1, and Pmag 0.1; that of AOT-2P are set as T = 15, ewl = 1, 6w2 0.01, eat 20, 6a2 5, and Pmag = 0.1. Further, the three variants of AOL are compared with two baselines (check Table D
Average-Consensun Consensus Fusion (ACF; equations ( 10) and 12 with equal weights), and Ciiman-Consonsuss
Fusion (KCF)- more details in the supplementary document.
A snapshot of the simulation of the AOL-1P in adverse conditions is shown in Fig. Simulation videos are submitted as a supplementary file.
Figures 5/. 5p. Ga, and 6 show the comparison results for N = 5, N = 10, N = 20, and N = 30, respectively. For cach method, 100 simulations are run, and the resulting cumulative scores are collected as points, which are then used to generate their corresponding box plots. Note that on each box, the central mark indicates the median, and the bottom and top edges of the box indicate the 25th and 75 percentiles, respectively. For N= 5, N = 10, N = 20, and N = 30, the best variant AOL-1P performs 182.2%, 329.7%, 79, 463%, and 652% better in terms of cumulative detection score and 94.7%, 138.6%, 167.3% and 150.4% better in terms of cumulative closeness score than the best baseline ACF, respectively. Further, for N = 5, N 10, N = 20, and N = 30, AOL-1P performs 3.62%, 9.41%, 8.86%, and 1.1% better in terms of cumulative detection score and 8.37%, 3.42%, 4.5%, and 0.0% better in terms of cumulative closeness score than the second best AOL variant, AOL-C. Among the baselines, ACF performs betler than KCF for all the cases in terms of both types of scores.
The three AOL variants perform significantly better compared to KCF; since KCF is a covarianco- method, it is unable to handle adverse conditions involving temporary or permanent sensor failures inducing large biases that may or may not increase the covariance. Since the weights are adapted through an online learning process in the AOL variants, their performance is better than ACF as well; unlike the AOL variants, ACF gives equal weight to all the input estimates being fused in the weighted estimate fusion process. The no-communication case performs better than KCF and comparably to ACF; unlike the AOL variants, KCF and ACF do not have a mechanism to adaptively filter out the corrupt information coming from the robots undergoing sensor failures, due to which the corrupt information propagates throughout the communication network without any inhibition, thereby corrupting even those robots having functional sensors.
Using the dotection confidence de,; as a reward signal does better than having a certurbation-greed reward- based learning strategy for the update of weights va((() in the social learning layer, which is quite evident by the improved performance of AOL-1P and AOL-C compared to the AOL-2P. However, using a perturbation- greedy reward-based learning update strategy for the update of weights (t) and (t(t) in the local lcarning layer, as in AOL-1P, does better than using a comparative Euclidean distance based loss function as in AOL- C. Moreover, for AOL-1P, when N is increased from 5 to 10, the cumulative detection score and the closeness score increase by 1.02% and 7.3%, which then decrease by 20.2% and 11.0% from N = 10 to N = 20, and then decrease further by 22.4% and 18.8% from N = 20 to N 30, respectively. Similar trends are observed for other algorithms as well. These observations are justified since the robots, while trying to avoid collisions, effectively push cach other, therefore, making it harder for all the robots to be able to detect and stay close to the target when crowding occurs as the nunber of robots N is further increased to much larger values.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Avg. Canal. Dotection Sture per robus aber 60 sec.
Avg. Cumrl. Detrcsion Soine per robus aftr 00 sec.
AGE.C
150
ADL-C
Desectio Score
150
AGL-2P
AGL-11
Cunsa Dovecton Soors
ADL-1P
ATH-2P
No Coosa
100
100
50
000
5
2
AAOT-C, A02-IP. AOT-2T;
RCF
AOT-C. AOL-2T, No
XCT
Aug. Cunra. Clowrress Scorn ger mhor whter All sec.
Aug. Cumml. Chowmeo Stow per rohor after
MDL
AGE-11
ADL-2P
Cuuaal Clocues Stoce
300
AOL,
J0L-11
300
Nu Cherm
No Dierm
Cloories
ACE
KCP
200
100
3
AOL.C A02-1P. AOL-P.
ACP,
1: AOL-C. AOL-1P. AOL-2P, No & NCF
(b)
Figure 6: (a) Comparison results; 100 sim. runs for each method, N = 20. (b) Comparison results; 100 sim. runs for each method, N = 30.
Further, Sim2Real aspects of the top two performing AOL variants, AOL-1P and AOL-C, are validated using ROS-Gazebo sctup (simulation video submitted as supplementary file; check the supplementary document)..
6 Conclusion
This paper proposes novel Aulonomous Online Learning (AOL) framework for the decentralized monitoring of an agile target using a swarm of robots undergoing sensor failures, communcattin link drops, and target visual loss and operating without the assistance of a supervisor or a landmark, i.c., without the availability of ground truth regarding pose information. the AOL framework, a decentralized online learning mecha- nism driven by reward-like signals (based on detection confidence) is intertwined with an implicit adaptive consensus-basedd two-layered, weighted information fusion process, thereby allowing the robotic swarm to exhibit improved robustness and adaptability. Within the AOL framework, three variants are proposed in order to study the effect of using different loss reward function designs in the learning phase. A novel perturbation greedy reward design is introduced in the learning process of two AOL variants, leading to exploration -ip--explittttion in their information fusion's weights' space. For the three AOL variants, conver- gence analysis of the weights involved in their weighted information fusion process shows that the weights converge under reasonable assumptions. Moreover, the AOL algorithms involve analytic expressions making them computationally inexpensive and therefore, ideal for use in robotic swarms. Simulation results show that among the three variants, AOL-1P performs the best in terms of detection and closeness scores, owing to its use of the cetturbbaaiinggreddd reward for learning the weights that belong to the local estimation phase, and detection confidence-basee reward for learning the weights that belong to the social estimation phase. AOL-1P performs 182.2% to 652% and 94.7% to 150.4% better than the baselines in terms of cumulative average detection score per robot and cumulative average closeness score per robot, respectively, as the total number of robots is increasod from 5 to 30. The Sim2Rcal aspects of the top two performing AOL variants are evaluated using a ROS- Coz-Ga setup. The current variants of AOL are synchronous; the asynchronous variant. is left as future work.
19

---

Published in Transactions on Machine Learning Research (12/2024)
Broader Impact Statement
The reward-based autonomous online learning framework proposed in this paper, called the AOL frame- work, is an adaptive information fusion framework that can be applied to various reward-based multi-agent applications, where the agents are required be smart about which neighbors should be trusted when for particular type of information, to increase their rewards as well help others in the communication net- work. The problem considered in this paper; i.e., target search, tracking, and monitoring, can be used for many applications, like search and rescue, firefighting, disaster relief, convoy protection, wildlife monitoring, surveillance, etc. Moreover, the AOL framework can be useful in the scarch and detection of socictal thrcats on the web by the use of multiple collaborative bots the weights involved in the information ion fusion indicate which neighboring bot is detecting an increased threat level, thus helping in pinpointing the source of the activity in the overall network. Further, AOL can also used in e-commerce applications, such as trading bots collaborating to increase profits, recommendation bots collaborating improve customer reviews, etc.
Acknowledgments
The authors would like to thank the Kotak IISc AI-ML Centre (KIAC) for their support.
References
Md Ali Azam, Shawon Dey, Hans D Mittelmann, and Shankarachary Ragi. Average consensus- consensus-based data fu- sion in networked sensor systems for target tracking. In 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0964-0969. IEEE, 2020.
Levent Bayindir and Erol Şahin. A review of studies in swarm robotics. Turkish Journal of Electrical Engineering and Computer Sciences, 15(2):115-147, 2007.
Gerardo Beni. From swarm intelligence to swarm robotics. In International Workshop on Swarm Robotics, pp. 1-9. Springer, 2004.
Gerardo Beni and Jing Wang. Swarm intelligence in cellular robotic systems. In Robots and biological systems: towards new bionics?, pp. 703-712. Springer, 1993.
Laszlo Blazovics, Kristof Csorba, Bertalan Forstner, and Charaf Hassan. Target tracking and surround- ing with swarm robots. In 2012 IEEE 19th International Conference and Workshops Engincering of Computer-Based Systems, pp. 135 141. IEEE, 2012.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
D Randy Garrison. Self-directed learning: Toward a comprehensive model. Adult education quarterly, 48(1): 18 33, 1997.
K Harikumar, J Sent hilaath, and Suresh Sundaram. Multi-uav oxyrrhis marina-inspired search and dynamic formation control for forest firefighting. IEEE Transactions on Automation Science and Engineering, 16 (2):863 873, 2018.
Christine Hockings, Liz Thomas, Jim Otlaway, and Rob Jones. Independent learning what we do when you're not there. Teaching in Higher Education, 23(2):145 161, 2018.
Sandeep Katragadda, Carlo S Regazzoni, and Andrea Cavallaro. Average consensus- based asynchronous tracking. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEFF, 2017.
pp.
Navid Ali Khan, NZ Jhanjhi, Sarfraz Nawaz Brohi, Raja Sher Afgun Usmani, and Anand Nayyar. Smart traffic monitoring system using unmanned aerial vehicles (uavs). Computer Communications, 157:434-443, 2020.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Hian Lee Kwa, Jabez Leong Kit, and Roland Bouffanais. Optimal swarm strategy for dynamic target search and tracking. In Proceedings of the 19th International Conference on Aulonomous Agents and MulliAgent Systems, pp. 672 680, 2020.
Chris D Lowe, Patrick J Keeling, Laura E Martin, Claudio H Slamovits, Phillip C Watts, and David JS Montagnes. Who is oxyrrhis marina? morphological and phylogenetic studies on an unusual dinoflagellate. Journal of Plankton Research, 33(4):555 567, 2011.
Abdullah Mohiuddin, Taha Tarek, Yahya Zweiri, and Dongming Gan. A survey of single and multi-uav acrial manipulation. Unmanned Systems, 8(02):119-147, 2020.
Mahyar Najibi, Bharat Singh, and Larry S Davis. Autofocus: Efficient multi-scale inference. In Proceedings of the IEEE/CVE international conference computer vision, pp. 9745-977777 2019.
Ghazaleh Radian, Yahya Kheiri Ghazijahani, Ahmadreza Saadatkhah, and Vahid Majd. Dynamic ob- stacle avoidance and target tracking for swarm of robots using distributed kalman filter. In The 3rd International Conference on Control, Instrumentation, and Automation, pp. 255 259. IEEE, 2013.
Jürgen Scherer, Saeed Yahyanejad, Samira Hayal, Evsen Yanmaz, Torsten Andre, Asif Khan, Vladimir Vukadinovic, Christian Bettstetter, Hermann Hellwagner, and Bernhard Rinner. An autonomous multi- system for search and rescue. In Proceedings of the First Workshop Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use, pp. 33-38, 2015.
Madhubhashi Scnanayake, Ilankaikone Senthooran, Jan Carlo Barca, Hoam Chung, Joarder Kamruzzaman, and Manzur Murshed. Search and tracking algorithins for swarmns of robots: A survey, Robotics and Autonomous Systems, 75:422-4344 434, 2016.
Amanda JC Sharkey. Swarm robotics and minimalism. Connection Science, 19(3):245-260, 2007.
B Singh and LS Davis. An analysis of scale invariance in object detection snip. arxiv. arXio preprint ar110:1711.08180, 2017.
Stephen Spry, Anouck Girard, and Karl Hedrick. Convoy protection using multiple unmanned aerial vehicles: organization and coordination. In Proceedings of the 2005, American Control Conference, 2005., pp. 3529- IEEE, 2005.
Zongyao Wang and Dongbing Gu. Cooperative target. tracking control of multiple robots. IEEE Transactions on Industrial Electronics, 59(8):3232 3240, 2011.
Jingyi Yao, Raúl Ordóñez, and Veysel Gazi. Swarm tracking using artificial potentials and sliding mode control. Journal of Dynamic Systems, Mensurement, and Control, 494477777777 2007.
Qiming Zhang and Wenling Li. Consensus-based kalman filter for multi-target tracking with integrated measurements. In 2019 Chinese Automation Congress (CAC), pp. 5628- IEEE, 2019.
Shuo Zhang, Jiantao Yao, Ruochao Wang, Zisheng Liu, Chenhao Ma, Yingbin Wang, and Yongsheng Zhao Design of intelligent fire-fihhtig robot based on multi-sensor fusion and experimental study on fire scene patrol. Robotics and Autonomous Systems, 154:104122, 2022.
21

---

Published in Transactions on Machine Learning Research (12/2024)
A
Appendix
A.1 Nomenclature
|-1
Cardinality operator (cardinality of a set)
2-norm Euclidean norm
N
Total number robots in the swarm
AT
Sampling period (seconds)
The jth robot's 2-D position vector (in m) at time
Vi,i
The th robot's body-axis velocity vector (m/s) at time
Ø1,i
The hh robot's heading angle (radians) at time t The hh robot's yaw rate (rad/s) at time The target's 2-D position vector (in m) at time t
Wt.i
X1,B
Vt,B
The target's body-axis velocity vector (m/s) at time
Φt,B
The target's heading angle (radians) at time t The target's yaw rate (rad/s) at time t
Dt.B
Reommm.
The range of communicatinn
Pid
The communication link drop probability
G
A uni-directional dynamic graph representing communication ion network topology among the robots
A:
Adjacency matrix of the graph Gt
10,
A uniform random variable, Uny 1) E [0, 1], for i,j E [N]
The neighbor set of the 7th robot as per the graph GG, at time L
The number of commuicating neighbors of the 4th robot at time t, nt,a == [445)
ni.5
m
Limit on the number of neighbors the ith robot can have, Lee, ne, S my. where E (1,2,...,N)
The th robot's propriocoptive sensor suitc's estimate of its 2-D position
The 4th robot's proprioccptive sensor suite's estimate of its yaw angle
Bounded arbitrary noise in the ith robot's proprioceptive sensor suite's estimate of its 2-D position
Bounded arbitrary noise in the ith robot's proprioceptive sensor suite's estimate of its yaw angle
22

---

Published in Transactions on Machine Learning Research (12/2024)
The 1th robot's exteroceptive sensor suitc's target detection confidence, de, [0,1] 1]
A
The 1d robot's exteroceptive sensor suite's estimate of the target's relative position
Roov
Detection range of the exteroceptive sensor suite
Fov
Angle-of-viers of the exteroceptive sensor suite
Pol
The probability of larget visual loss
Bounded arbitrary noise in the jth robot's exteroception's estimate of the target's relative position
Vta
The combined sensor estimate of the target's position; combines the estimates from proprioception and exteroception
The jt robot's velocity reference command signal
Via
AV
The 1h robot's velocity correction control signal Discrete-time horizon
T
The tt robot's intermediate estimate of the target's position The th robot's final estimate of the target's position Weights involved in the local estimation phase
*i.B
as(()
Ar,i
Ar Pr.i U (i)
(ty(t)
Weights involved in the social estimation phase
(+())
Weights updated in the local learning layer
(a(())
Weights updated in the social learning layer
Loss function involved weights update in the social learning layer AOL-C and AOL-1P, 4y == (1 -d,)
De,
LE Σi 15:
The
Learning rate parameter for the social learning layer in AOL-C and AOL-1P
Learning rate parameter for the local learning layer in AOL-C
Perturbation-grealy reward functions involved weights update in the local learning layer of AOL-1P and AOL-2P
R Re!
R = Ei re and Rd Ei= 1
23

---

Published in Transactions on Machine Learning Research (12/2024)
fal
Learning rate parameter associated with the difference-based correction term in f2 and r2
ea2
Learning rate parameter associated with the inertia term in ri and ri
Ep. S
Perturbation signals associated with the perturbation terms in rū, and ri
(t
Aa;(t 1) 1) at, 1) at
(1,
Ad;(1) d1.5 d
hh f == min (12;;1 &, 11/Do, 13, 12 == min
1/ 8, I1/ 1}
Do
A normalization parameter
ri
T7, arg max Wij(t- (t-()) 1)
Tta
Perturbation-greet reward functions involved in weights update in the social learning layer of AOL-2P
Ry = Ei=1" r
ewl
Learning rate parameter associated with the difference- based correction term in ri.
Ew2
Learning rate parameter associated with the inertia term in ri,
ep"
Perturbation signal associated with the perturbation term in r2,
Awa(t - 1)
Awy(t - 1) == wait - 1) wa(t 2)
fi.s
H, arg minje 4M
Asc,i
lim4 Ae.5 Axo,
lime-soo Fú.4 Jo
iis
Gi, = arg maxjeNe E.m Awjy(s 1) (8)
lim gi.5 = Toc.i
24

---

Published in Transactions on Machine Learning Research (12/2024)
Communication Link
Agent 5
Reward, Info: actiir
Agent 4
action
Reward, Info.
Environment
Agent 3
1TA
Agent 2
Agent
Figure 7: The AOL framework can be useful in a multi-agent problem setting where the agents have a shared environment, both a common objective and an information space, and can fuse information from their communicating neighbors to increase their reward.
A.2
Generalizability of AOL framework
The idea behind the proposed AOL framework can be implemented any such multi-agent problem setting application where:
The agents have a shared enviromment, common objective, and a common information space ppreicccionnniiiiiiiiin space).
An agent's performance is governed by reward-like signal.
How the agents fuse information from their neighbors (i.c., which neighbor to listen to) influences their performance or reward accurale or quality information leads to beller reward.
The proposed AOL framework can be viewed reward-driven, online learning-based, adaptive information fusion framework that can be used by agents in multi-agent problem setting (Fig. 7 to improve their reward/performent perffrmacc.ce Therefore, the AOL framework is useful in problems where the wwwrd/performande of each agent is dependent on the quality fatiit//ccraacyy of the information that the agent has the dependence can be direct (the fused information is directly responsible for reward generation) or indirect (the fused. information used make a decision which then responsible for reward generation).
The problem of cooperative target monitoring considered in this paper involves an indirect dependence of the reward (target detection confidence) on the fused information (target position estimates). The reward-lii signal, called target detection confidence score (dr,), is produced by each robot's exteroceptive sensor suite, The communicating robots utilize the AOL framework to determine what information should be given more weight in the weighted information fusion that improves their cumulative larget detection score (cumulative reward). Therefore, the AOL framework (along with its theoretical analysis) works for any particular set of tasks, robots, or robot dynamics as long as the above-mentioned general conditions are satisfied. Examples inchude monitoring and controlling the spread of fire (target) in forest, wildlife monitoring, search and rescue, etc.
25

---

Published in Transactions on Machine Learning Research (12/2024)
Similar problems where the AOL framework be used are searching and detecting societal threats the web by using multiple collaborative bots, trading bots collaborating to increase profits, recommendation bots collaborating to improve customer reviews, etc.
A.3 Pseudo-code
Algorithm AOL-C (for the tth robot, E [M]
Choose: D, > Ta 0, The > 0, T, 2 (integer-valued)
Initialize: (.) = (0) = 1, wa( = 1, &. = %
Input: L, dr,i, a; 1), a/(t 1), du(( 1), & R
Output: (t),), (t), (t),), K
at - 1) d,(t-1)+d((t-1)
(1-1-))
2: dr, > then
L at - 1)x, B + (1 as(t - 1))
4: else
5:
1 = 8
end if
Broadcast {1, de,i: &/ 8 1.B (1- 1)} and receive (1ijdes, R/i0: 88- 10 1)} from the com-
municating neighboring robots j E P4
8: wir(l- - = Exeem, (1-(
(1)(1-1)
8 EvJear, Wig(t- 1)8 where Ar,; = S4 U [i}
10: ri, = arg max Wig(1 1)
Iri. 11: fr = =minfllk?.p R, 11/ 13; iy minn | x, :: Dl: 1}
1.-
12: Periodic resel: set the weights 0;(l 1), 6/(1 1), and ( to after every Tp discrete time steps
13: (t(t) = a,(t - 1) exp (-natt)) (((t) = dt(t - -1) ex)epp(-naal
14: (t) @n(tt 1) exp (- d,,))
26

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm 2 AP (for the ht robot, i e [N])
Choose: D, > 0, 1/o > 0, the > 0, T 2 1 (inleger-valued), eat > 0, 6a2 2 0, Pmnag
Initialize: á,( = (0) = á,(-1) = d'(-1) = (0) = 1, 8 x
Input: t, de,, á; - 1), 6((t- (1) at - 1), x & d 1,5 at(t - 2)
Output: (n))), (!), (1), 8i
(,(t-1) 1: (t - 1) [
if de, then
x, at 1)82,8 aslt 1))8/11.B
clse
* 8
6: end if
from the com- 7: Broadcast (t, i, dr &1.B. 81,, (t 1)} and receive {t.j. de.. & 8 (t (1-1)) municating neighboring robots j E Ω4,
Wty(t- 16--1 1) E (t-1)
(t-(t
* ExjeA, wt 1)x where A4, S4,4441 U
10:
ailt 1) - at 1 2); (t, = dr,i - d:-1;
11: Periodic resel: set the weights à; - 1), 0f(1- 1), and wiall - 1) to after every Tp discrete time steps
12: Periodic perturbation: after every Tp discrete time steps, ep either takes the value ep Pmag or ep 0 with equal probability, whereas ej Pmog epi otherwise, Ep = eg = 0 at all other times
13: rfe Aa, (t) + ep(1 d1,)) ezza; 1)de,i
14: ri, = -eaa Aa, - (1) - e de,)) + ea2(1 at - 1))dk,
15: dg át 1) exp (rf,); (1) 0t 1) exp (1())
16: (t) ():(1 1) exp Tho (1 d,,))
27

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm 3 AOI.2P (for the 7th robot, i e [N]
Choose: Do > 0, Ta > 0, > 0, T 2 1 integer-valuedyy eai > 0, 6a2 2 0, Pmag > 0, ew1 > 0, ew2 2 0
Initialize: d.(0) (0( = â.(-)) = d-(-1) = w;((0) w=(-1) = 1, 8 2
Input: t, dr,, á;(t - 1), df(t-1), aa - 1), x, x de-1,i ag(t - 2), wi (t 2)
Output: (t),, (t), (t), K
1: at 1) = (,(1-1)+d((t-]
(1-1-1)
if d, then
x at 1) 1)x (1 as 1))
else
x = 8
6: end if
7: Broadcast i, de &/B. 81- (1- 1)} and receive di,i, 8i &/-1B (15((-1)
from the com-
municating neighboring robots j E Q4
wij(1- (1-1)) Evyene (1-1)
(1)(1-1)
K,B EvjeAn, Wiy(1 1) where Ar, = 04, (1}
10: Ao; -
- -1)(1 whi(t - 2); Ad((t) = de,i - de-1,
11: Awa(t- - wn((t
12: Periodic reset: set the weights (t 1), f and wnit - 1) to after every T, discrete time steps
13: Periodic perturbation: after every Tp discrete time steps, ep either takes the value Ep = Pmag or ep with equal probability, whereas eg Pmag epi otherwise, ep = es 0 at all other times
14: ri, Aa, (t) + ep(1 dr,i) eaza; 1)de,
15: ri, = - (t - (1) + (1(1 de,)) + ea2(1 - as,(t - 1))de,.
16: (t) āj(t 1) exp (rf,): (t(t) át 1) exp (r21)
17: Periodic perturbation: after every тр discrete time steps, e/ Unif. Pmag); otherwise, Cx 0 at all other times; Unif. (0, pmag) is uniform random variable
18: Th. c.. Aw;; 1) dd + ex de,)) + ew 1)d,,
19: (t) w. - exp (r?)))
28


Reviews:
[{"review_id": "nTdmRy2QGb", "reviewer": "Reviewer_aS9w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: (1) the paper addresses a problem which seems important for applications; (2) the paper contains a theoretical analysis\n\nWeaknesses: (1) it is not clear to me how to check some of the assumptions in practical scenarios; (2) I am not sure how the proposed algorithm compares with state-of-the-art method for similar problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of monitoring a target with a swarm of robots. Two robots can communicate if they are close enough to each others, but the communication may fail with some probability. The problem is cooperative in nature. Three algorithms are proposed, based on Autonomous Online Learning (AOL). At a high level, the three algorithms have in common to involve a local estimation phase, a communication phase, a social estimation phase and then a learning phase. They differ in how the learning is implemented.  A convergence analysis as well as experimental results are provided, including a comparison with two existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Page 9, Section 4: Could you please clarify what you mean by \"Without the loss of generality\"? How would the assumptions and the results change with a periodic reset?\n\n2. Page 13, Assumption 5: Below the assumption, you provide an example (\"For instance ...\"). But this example requires the i-th robot to be fixed over time. Does this mean in Theorem 3 that all the robots would need to be fixed over time? This seems overly restrictive.", "broader_impact_concerns": "A discussiong on the potential ethical implications (e.g., real-world tracking by robots) could be discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s9X4qZNTQy", "reviewer": "Reviewer_MuFk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I think the approaches used to aggregate estimates are pretty reasonable. I appreciate that there is theoretical analysis of the convergence of the weights. I'm not sure how important the perturbations are to final performance - this sort of idea is pretty common in reinforcement learning style literature (see many works on optimizing under uncertainty). However the technicals of the paper seem good.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses a target monitoring task. Each robot has a narrow FOV, and a certain range at which it can communicate with nearby robots. These communications all have some probability $p$ of failing even if in range. Each robot follows 2 modes, object searching and object tracking.\n\nBased on this setting, an online learning algorithm is proposed to optimize tracking. The reward used is the detection confidence \n$d$, modeled after camera / LiDAR and increasing as distance approaches 0. All the approaches involve each robot maintaining a local estimate of the target's location, broadcasting that local estimate to nearby robots, and aggregating those estimates into a final estimate of position.\n\nMuch weight is placed upon the perturbation-greedy reward, which introduces some random noise into the reward estimate to encourage more exploration in target finding, scaled according to the models current confidence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No changes", "broader_impact_concerns": "The described task does feel very applicable to surveillance / military use cases. Although this is discussed in the introduction.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FDMHmEJFxl", "reviewer": "Reviewer_SbGK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The literature review seems complete and does a good job at introducing the field of cooperative target tracking\n- The proposed algorithm is simple yet provides encouraging results in a challenging setting (with a high probability of link or sensor failure)\n\nWeaknesses:\n\n- The overal clarity of the paper is quite low, and it is difficult to quickly find what the contribution is, and what are the differences between AOL-C, AOL-1P and AOL-2P. It would help to have a table that compares a selection of related approaches with the 3 proposed algorithms (algorithm name in the rows, \"features\" in the columns such as \"perturbation-greedy reward function\").\n- The paper seems on the edge of the scope of this journal, with its \"learning\" being the tuning of weights for an averaging operation, and the paper being very applied (it precisely focuses on one particular task for one particular set of robots, and even describes how the robots move, instead of focusing on the general problem of winding what neighbors to listen to to maximize some coverage or tracking objective).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers a setting in which a swarm of robots has to follow and monitor a moving target. The robots have sensors that may experience failure (or occlusion), and they have local communication with limited bandwidth. The paper introduces a way for the robots to tune how they combine information received from their neighbors, by changing weights given to their neighbors according to some reward-based formula.\n\nThe proposed algorithm is analyzed theoretically, and numerical experiments show that it outperforms several baselines (albeit with high variance).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given the wide scope of the TMLR journal, I believe that clarity is of utmost importance, and that any work has to be precisely positioned with regards to related work, and the general field it considers. As such, my requested changes focus on clarity.\n\n- Add some concise (e.g. a table) comparison of the proposed algorithms and some selected related work. This will also help the readers understand the novelty of the paper and the importance of the reward-based equations.\n- One or all of the proposed methods should also have a concise pseudocode representation somewhere, for ease of reproducibility.\n- Figure 3 is too big and has too much white space in it. It should be truncated, and the room feed by this operation used to show a nice system diagram of the proposed method, with a depiction of the target to track (what is known about it, what sensors produce that information, what the failure rate is), and some of the robots (with information about what sensors they have, what they observe about the target, how they communicate, what they communicate). This will be a repetition of the contents of the paper, but will greatly help give an overview of the proposed method. The cherry on the cake would be to highlight in that systems diagram the parts that are novel in this paper (for instance the learning equations), and have, in the caption, a one-line motivation of why that contribution really makes the whole system better.", "broader_impact_concerns": "The broader impact of this paper is not discussed in the paper. There does not seem to be a critical need for it, but for this particular paper, very applied and focusing on a sensitive topic (target tracking), I think that a broader impact statement would be welcome. How could this research be used for fugitive or political dissident tracking? Could the target be something else than a moving physical thing (for instance, would the equations still work to track some activity on social networks?)", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: SbGK, after the rebuttal: "The authors increased the clarity of the paper and added a comparison against state-of-the-art baselines. The challenges tackled by the paper are now clear enough, and the contribution meaningful and generalizable."

SbGK, before the rebuttal: "The paper seems on the edge of the scope of this journal, with its "learning" being the tuning of weights for an averaging operation, and the paper being very applied (it precisely focuses on one particular task for one particular set of robots, and even describes how the robots move"

MuFk: "I appreciate that there is theoretical analysis of the convergence of the weights. I'm not sure how important the perturbations are to final performance"
Decision: Accept (poster)