Fleet Supervisor Allocation: A Submodular Maximization Approach
Authors: Oguzhan Akcin
Ahmet Ege Tanriverdi
Kaan Kale
Sandeep P. Chinchali
Venue: CoRL 2024 (2024)
Keywords: Imitation Learning
Submodular Maximization
Fleet Learning

Abstract: In real-world scenarios, the data collected by robots in diverse and unpredictable environments is crucial for enhancing their perception and decision-making models. This data is predominantly collected under human supervision, particularly through imitation learning (IL), where robots learn complex tasks by observing human supervisors. However, the deployment of multiple robots and supervisors to accelerate the learning process often leads to data redundancy and inefficiencies, especially as the scale of robot fleets increases. Moreover, the reliance on teleoperation for supervision introduces additional challenges due to potential network connectivity issues. 
To address these issues in data collection, we introduce an Adaptive Submodular Allocation policy, ASA, designed for efficient human supervision allocation within multi-robot systems under uncertain connectivity conditions. Our approach reduces data redundancy by balancing the informativeness and diversity of data collection, and is capable of accommodating connectivity variances. We evaluate the effectiveness of ASA in simulations with 100 robots across four different environments and various network settings, including a real-world teleoperation scenario over a 5G network. We train and test our policy, ASA, and state-of-the-art policies utilizing NVIDIA's Isaac Gym. Our results show that ASA enhances the return on human effort by up to $3.37\times$, outperforming current baselines in all simulated scenarios and providing robustness against connectivity disruptions.

Fleet Supervisor Allocation: A Submodular Maximization Approach
Oguzhan Akcin, Ahmet Ege Tanriverdi, Kaan Kale and Sandeep P. Chinchali The University of Texas Austin (oguzhanakcin, aet2776, kaankale, sandespe)@utexas.co
Abstract: In real-world scenarios, the data collected by robots in diverse and un- predictable environments is crucial for enhancing their perception and decision- making models. This data predominantly collected under human supervision, particularly through imitation learning (IL), where robots learn complex tasks by observing human supervisors. However, the deployment of multiple robots and supervisors to accelerate the learning process often leads to data redundancy and inefficiencies, especially as the scale of robot fleets increases. Moreover, the re- liance on teleoperation for supervision introduces additional challenges due to potential network connectivity issues. To address these issues in data collection, we introduce Adaptive Submodular Allocation policy, ASA, designed for ef- ficient human supervision allocation within multi-robot systems under uncertain connectivity conditions. Our approach reduces data redundancy by balancing the informativeness and diversity of data collection, and is capable of accommodat- ing connectivity variances. We evaluate the effectiveness of ASA in simulations with 100 robots across four different environments and various network settings, including soal-world teleoperation scenario over 5G network. We train and test our policy, ASA, and state-of-the- policies utilizing NVIDIA's Isaac Gym. Our results show that ASA enhances the return human effort by up 3.37x, out- performing current baselines all simulated scenarios and providing robustness against connectivity disruptions.
Keywords: Imitation Learning, Submodular Maximization, Fleet Learning
Introduction
Today, diverse industries deploy robotic fleets for tasks ranging from autonomous driving [1, 2] to healthcare [3] and package delivery [4]. These robots are often deployed with policies trained on dataset that is primarily based on simulations, along with a small amount of data collected through real-world interactions. While effective within their training contexts, these models often fail to adapt new or evolving real-world scenarios [5], making data collection critical for the success of the roboties applications [6, 7].
A popular approach to collecting such data is through human supervision, where humans directly guide the robots to perform the tasks. These data are then used to train the robots via Imitation Learning (IL), where the robots learn to perform tasks by imitating the human demonstrations [8]. Imitation Learning (II.) has been effective many roboties applications, ranging from autonomous driving [9] to robotic manipulation [10, 11]. However, the breadth of scenarios necessary for ef- fective emphasizes the need for continual data collection [12], commonly done with numerous robots in parallel. Usually, the number of humans is less than that of deployed robots. For in- stance, a recent autonomous delivery company, Starship Technologies, operates 1700 autonomous robots while teleoperating only 1% of this robotic fleet [13, 14]. The scarcity of human supervisors necessitates the selection of informative robots for supervision [15. 16].
Human supervision is often provided through real-time teleoperation over a network, especially when supervising fleets of robots distributed across the globe. For example, various companies, cluding Cruise, utilize human supervisors located in their control centers to teleoperate autonomous
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

H
Cloud
A
D
D
D
++1 T
π
C2
Chrotot
Retrain C
1
2
3
N Noobot
Figure 1: Supervisor Allocation Problem. At each time step t, the human supervisors with policy TH are allocated to the robots with policy zं based the allocation policy TA. Each robot operates in a different state st and the human supervisors allocated to the robots based the uncertainty of the robots and the similarity between robots. Additionally, by the connection random variable, C₁ of each robot i. At the end of each time step t, the data collected by human supervisors D is added to the dataset D' to create updated dataset D which is then used to train the robot policy ng
connections the robots may not be stable, which is indicated
vehicles deployed across the world [17, 18). However, these networks might be susceptible to con- nection failures [19], and is important to be robust against network uncertainties. Combining these challenges with selecting informative robots, we formulate the Supervisor Allocation Problem ( FFi.. 1), which involves managing the limited human resources to maximize data diversity and quality under uncertain network connectivity. Our problem is an extension of the Interactive Fleet Learning (IFL) setting introduced by [15]. We extend the IFL setting account for network elements that play an important role in real- world teleoperation scenarios [20].
We then introduce novel human supervisor allocation policy called Adaptive Submodular Allo- cation (ASA). ASA distributes human supervisory capacity across a fleet, ensuring a balance of data informativeness and diversity minimize redundancy in data collection. Our allocation pol- icy shown to be robust against network instabilities and is able to adapt to the dynamic nature of data collection, which we demonstrate through extensive simulations in diverse network environ- ments, including real-world 5G scenarios. We show that ASA improves the Return on Human Effort (RoHB) [15], which is human supervision efficacy metric, by up 3.37x compared to existing benchmarks.
2 Related Work
Data collection is a critical problem in robotics and machine learning that is essential for continually improving the performance of robots [21-25]. closely related to active learning [26-30], where the goal is to select the most informative samples to label. Although the goal of data collection similar to active learning, the focus on collecting data samples that are the most informative for training the models. In our case, however, the aim is to select the robots that provide the most informative data for human supervisors.
IL is a popular approach in robotic learning, where robots learn policies from human demonstrations [3 1-43.. Despite potential, the reliance on purely offline data introduces several challenges, such as distribution shifts [35]. which occur when robots encounter states that were not previously expe- rienced by humans. These issues can be alleviated through online data collection methods, such as Dataset Aggregation (DAgger) [35] and various forms of interactive IL [36, 37]. Most interactive IL methods rely on human supervision to decide when to intervene the robot's learning process. This presents scalability challenges, especially when applied to extensive robot networks [38] or during prolonged learning phases [39]. Robot-initiated interactive strategies like EnsembleDAgger [40] and ThriffyyDAge [41] have been proposed to mitigate these constraints, enabling robots to request.

---

human input under specific conditions. However, these methods are designed for single-robot task allocation scenarios and do not consider multi-robot scenarios. Closest to our work, Fleet-DAgger [15) has been proposed to address the supervisor allocation problem in a multi-robot scenaaii.. How- ever, Fleet-DAgger does not consider operational constraints that might limit the allocation of human supervisors, such as network connectivity and the potential redundancy from employing multiple human supervisors in similar environments. Our work, on the other hand, focuses on learning an allocation policy that is adaptable to the operational constraints while minimizing the redundancy in the data collection, which is crucial for the system's scalability [27].
One popular approach to mitigate redundancy in data collection is using submodular maximization. Submodularity refers to the property of the marginal gain of adding an item to small set being higher than adding the same item to large set. As submodularity is a common trend in data collection, it has been widely used in machine learning tasks, such as sensor placement [42], active learning [27, 30, 433, and summarization [44]. Submodular maximization has also been extended to stochastic settings [45, 46], where the goal is to select a subset of items to maximize the expected value of a submodular function. Despite its wide use in machine learning, stochastic submodular maximization has not been used in the context of IL and multi-robot data collection scenarios. Our work is the first to use stochastic submodular maximization in the context of human supervision and multi-robot scenarios to address the supervisor allocation problem.
3 Problem Formulation
Consider a geo-distributed system of Nyubol robots, I = (1,.... ,Nroool) Each robot i operates in parallel within an independent Markov Decision Process (MDP) with a different initial state. However, all robots operate within the same state and action spaces S and A, respectively, Each robot observes the state of the environment of E S at time / and selects an action at E A based on a policy T S + A. The robots share the same policy गे that has been trained using the collective data D' accumulated up time step t. We define the collection of states and actions for all robots as s² = =si_..... ) E Natce and a' = (af....,.o, dN E A Natee These robots can be supervised by Nimman human supervisors with oracle policy TH S - Ad, respectively. In addition to the robot action space A, the human action space A₁ includes a reset action, which can return the robot to safe state. Only robots supervised by humans operate with human policy лh while the rest of the robots operate with the robot policy T...
In each time step I, Nuuman human supervisors can be assigned to the robots for assistance. However, the connections to the robots are unreliable, with C² [Ci,-.. Naame } E {0, 1) J denoting independent random variables associated with the connection reliability of the robots. C E {0, 1} indicates whether successful connection with robot i can be established (C)- = 1) or not (C²- = 0) at time t. Under this uncertain connectivity, we are interested in finding an allocation policy æa : {0, 1 x S.Nana x Nonw x -> 2 that selects robots to be supervised X C I based on connection reliability C³, collection of states s' and actions a'. Our setting extends the Interactive Fleet Learning setting [15] by incorporating imperfect time- vime-narying network connectivity.
Supervisor Allocation and Connectivity:
Data Collection and Policy Retraining: Upon allocation, human supervisors contribute data only from successful connections, forming the human supervision data D This new data is integrated into the current dataset and the robot policy o is retrained using a retraining function g:
D D' D'UD' 1-1 = =-kk- D(+1).
D/r {(sh, mH(s)) : X and C. =1,,
(1)
(2)
m
Our objective is to develop an allocation policy TA that maximizes the expected Return on Human Effort (RoHE) over the connectivity C". RoHE metric was introduced along with Inter- active Fleet Learning setup [15] to set benchmark in Fleet Learning settings. It is a ratio of the total reward obtained by the fleet to the total number of human actions. Formally, the objective is to maximize the expected RoHE over the connection probabilities:
Objective:
N Naunnan
a() Nrobot 1 1+ +I (TA s' a 1)
max EC ES)
(3)
3

---

Uncertainty
0.0
0.6
1.5
2,1 2.4
Figure 2: Submodular maximization balances uncertainty and diversity. This figure illustrates toy example of our alloca- tion problem in 2D state space. The blue contours indicate the uncertainty levels, with darker shades representing higher uncer- tainty. Purple crosses (traditional acore-based allocations) and yellow plus signs (our submodularity- the positions of selected robots. Unlike score-based methods that often select highly uncertain but potentially overlapping states, approach strategically picks fectively balancing trade-off' between high uncertainty coverage, thereby reducing data redundancy and enhancing the training data's & representativennss.
allocations) mark
diverse set of states, ef-
Scora Based
Submodularity Basad
Here, C= (C....., CT) represents the set of all connection random variables from time 0 to time A - the reward function, |- denotes the Frobenius norm,
horizon T. The function and Ωi refers to set of all allocation policies. Intuitively, RoHE measures the overall performance of the robotic fleet, normalized by the total number of human interventions.
Stochastic Submodular Maximization Approach
We now present our novel policy, adaptive submodular allocation (ASA), for the problem outlined in Eq. 3. First, we define the stochastic submodular maximization problem, which represents the value of robot supervision, and then we define the greedy algorithm, which is used to pick the robots to supervise.
4.1 Submodular Maximization Problem
To address the optimization problem presented in Eq. 3, we use stochastic submodular maximiza- tion. Stochastic submodular maximization is particularly suited to our scenario because it leverages the diminishing returns property that naturally reflects the decrease in the marginal gain of super- vising additional robots. Furthermore, the method inherently discourages the selection of similar robots, thereby avoiding the assignment of humans to robots that offer overlapping information, which decreases the return on human effort. Finally, stochastic submodular maximization accounts for the inherent network uncertainty in our problem by acknowledging the non-deterministic con- nections to the robots. This is crucial for developing a robust solution across different connectivity patterns.
We first define a submodular objective function fcr N R that quantifies the value of supervising selected set of robots X, considering the allocation reliability outcomes for these robots C We define our objective function based on the facility location problem, a classic example of submodular maximization objective [42], as follows:
for (X)
Σ max M GET JEX
Here, X is the set of robots selected for human supervision, and CJ indicates whether the connection the robot is successful or not. Mj., represents the value of supervision of the robot on the robot i at time I, and we consider two factors: the informativeness of the robot i and the similarity between the robot and the robot Additionally, other factors. such as prioritizing the robots that have violated the safety constraints or those in critical states. With all factors combined, we define the value of supervision Mf,, as:
formulation is modular and be extended to include
My. (j. (1) IC)(1).
(S)
Here, St defines the similarity between the robots and at time t, and 2/' (i) is the informative-
ness of the robot i, while (i) is an indicator of whether the robot i violates the safety constraints or is in critical state. Our definition of Mf,, is modular, and each factor can be defined based on
specific requirements. For example, the similarity function S can be defined as the cosine similar- ity, the Fuclidean distance, or any other similarity metric. The informativeness of the robot L/ (1)

---

can be defined as the entropy of the robot's 's policy or the uncertainty of the robot's state, while the constraint function (2) can be defined based on the safety constraints or the critical states for the robots, similar to previous works [15, 47]. For the exact definitions of similarity, informativeness, and constraint functions, please refer to the Appendix. With the objective function define, we pose the following maximization problem to optimize our allocation policy:
max
F(X),
(6)
X
subject to: |X| S Nuuman
where the goal to identify the subset of robots X that maximizes the expected value of for, denoted as F(X)= = Ee [ff (X)] over all possible connection outcomes C*. The constraint |.X' v Nuuman ensures that the number of selected robots limited by the number of available human
supervisors Nhuman-
4.2 Adaptive Submodular Allocation (ASA) Policy
Now, we can present our allocation policy ASA based on a greedy algorithm given Algorithm 1. Starting from empty solution set X (line 1), ASA iteratively selects the robot with the highest marginal gain based the estimated expected value function F (line 3), which estimates the ex- pected value of the submodular objective function f. Then, ASA computes the expected marginal gain of selecting the robot x: (line 4), and if the expected marginal gain is below certain threshold, the algorithm stops the selection process (line 5). This threshold ensures that the algorithm avoids using unnecessary human effort by stopping when the marginal gain of selecting
additional robot is low. Otherwise, the chosen robot x² is added to the solution set X (line 8). Finally, if there is observation whether the connection to the robot was successful not, the estimated expected value function F is updated (line 9).
The estimated expected value function F is an important factor in our allocation policy and can be defined as:
F(X) Ex(()) xe) II PE.
Here, the function fe denotes the value supervising the set of robots X for specific realization of the connection random variable C Pg represents the estimated connection probability of the realization ç, whereas Pe, denotes the estimated connection probability for the robot within the realization ç. Initially, PG., values are set to the actual initial connection probabilities Pe, + P(s. CP) for all robots i. Then, if an observation on the success of supervisor allocation is made, the connection probability estimate Pe, is updated for the selected robot Pc. + P(g: = C.).
Another important factor in the allocation policy is the marginal threshold parameter. which determines the trade-off between the cost of additional supervision and the incremental benefit derived from including an additional robot in the supervision set. In practice, this threshold can be set to 0 all human supervisors want be allo- cated to maximize the amount of data col- lected. Then, the threshold can be gradu- ally increased to optimize the efficiency of human supervision.
Algorithm 1 ASA Policy
Input: Estimated expected value submodular objec- tive function F. set of all robots Output: robots selected for supervision X
1: Initialize X +- 0
2: for k = 1 to Nhuman do
3: x +- argmax (x))
4:
Compute A + F(X
(x*}) ( F(X))
5: if A < threshold then
break 7: end if
Based on the availability of the obser-
T-Xur if ASA then
vations of the connection probabilities. 9:
we define two variants of our policy:
10:
Observe whether connection to robot xn successful and update F
non-Adaptive Submodular Allocation (n- ASA) and Adaptive Submodular Alloca- tion (ASA). In n-ASA, we are not able to observe the connection probabilities, and thus, the allocations are done beforehand. eratively based on the success of the allocations, and the connection probability estimates Åę are
11: end if
12: end for
In ASA, on the other hand, the robots are selected it-
5

---

updated. To visualize the differences between n-ASA and ASA, consider the following: in both cases, robot 1 is selected for supervision in the first iteration. While selecting the robot supervise, n-ASA considers both possibilities (successful and unsuccessful connection to robot 1) and selects the second robot that maximizes the expected marginal gain over both cases. However, in ASA, after selecting robot 1 for supervision, we observe whether the connection was successful or not and select the next robot that maximizes the expected marginal gain based on the observation. We then update the connection probability estimate of the robot I with its actual probability: Pe, + P(( C1).
When the marginal threshold parameter is set to zero and the estimated expected value function F is equal to the actual expected value of supervision F, ASA is equivalent to the greedy algorithm for submodular maximization, which is proven approximate the optimal solution for the submodular maximization problem in Eq. 6 with a factor of 1 1/c [45]. Additionally, n-ASA approximates the optimal adaptive policy with a factor of (1 1/e)² [45]. To compute selected robots faster, use the lazy greedy algorithm [46]; this has the same time complexity as Algorithm but has a better empirical performance.
5 Experiments
We consider a fleet of Norht = 100 robots that can be supervised by Nnuman 5 human supervisors. The human supervisors are implemented as reinforcement learning agents using the Proximal Policy Optimization (PPO) algorithm [48]. We utilize the behavior cloning algorithm to initialize the robot policies based on an offline dataset of 5000 state-action pairs and use our allocation policy to collect data from the robots and update the models. In all of our experiments, when the robots violate the constraints, we perform a hard reset to bring the robots back to a safe state. We set the hard reset time tn 5 timesteps, the minimum intervention time fr 5 timesteps, and the fleet operation time to = 10, 000 timesteps. We average the results over 3 random seeds for each task and network configuration. We have chosen these parameters align with the settings used in the environments of the benchmark algorithms [15] for a fair comparison.
Environments: We consider four different environments in our experiments. First is Humanoid, where the robots focus on bipedal locomotion. Second is ANYmal, which involves quadruped locomotion using the ANYmal robot. Third is Allegro Hand, focusing dexterous manipulation tasks. The last environment is Ball Balance and it requires the robots to balance a ball on a plate.
We use five different network configurations based on various connec- tivity probabilities. Always allows robots to always be supervised by human supervisors. Mixed- Scarce, some robots have a high connection probability, while others have low probability. Ookla features varying connection probabilities according to cellular network performance metrics [49]. 5G uses real-world connectivity data, including latency and throughput, collected from a university robotics lab (further details in the Appendix). Finally, Changing-Scared starts with the same con- nection probabilities Mixed-Scaree but evolves over time. It is important to note that while the first four configurations maintain stationary network connectivity, the Changing- Scarce configura- tion introduces dynamic network connectivity that changes over time.
Network Configurations:
Metrics: We evaluate the performance of the allocation policies based on the following metries: (1) Return on Human Effort (RoHE), which is given in Eq. 3 and (2) the cumulative number of successfully completed tasks by the entire fleet, which we will refer to as cumulative success. To simplify, RoHE measures the fleet performance per human intervention, while cumulative success only considers the total successful task completion without considering the number of interventions. For example, simply allocating all human supervisors would improve cumulative success but de- crease RoHE An ideal allocation policy should balance the two, as an ideal system would require a high total success while using humans as efficiently as possible.
Allocation Policies: We compare our proposed Adaptive Submodular Allocation (ASA) and non- Adaptive Submodular Allocation (n-ASA) policies with several baselines. The ASA policy uses the observed connection information to update the network connection probabilities, while the n- ASA policy does not use this information. The Random baseline simply selects robots for su- pervision randomly. Fleet-EnsemblelAggee bined with constraint-based prioritization [40]. Fheet-ThriftyDDAeee and goal-oriented prioritization, adapting ThriftyDAgger for fleet settings goal-oriented environ- ments [41]. DAgerr (FD) prioritizes robots violating constraints and selects those with the
(FE) uses variance for uncertainty estimation, com- (FT) integrates uncertainty

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
Always
12.
10
2.5
2.0
1.0
15
5.0
0.5
10
Mixed-Scan -carcee
2.5
1.50
1.25
ROHE
1.00
0.75
1.0
0.50
5
15
2.5
Ookla
ROHE
1.4
1.79
1.50
1.29
5G
ROME
1.00
0.75
0.30
02
0.25
0.0
0.00
Changing Scarre
12
3.0
0.8
ROME
0.6
0.4
2000 4020 5000 80co 30000 Steps (t)
2000 4000 4D0D RO00 10000 Staps (t)
2000 4000 ROOD B000 10000 Staps (t)
2000 4000 60D0 2000 20000 Steps (t)
n-FD Figure Our ASA and n-ASA policies outperform other benchmarks across all environments and network combinations. Here, each row represents different network configuration, and each column corre- sponds to different environment. Compared other policies, ASA and n-ASA are affected less by changes in the network configurations because of their stochastic submodular maximization-bel based policies that incor- porate network uncertainties. The submodular maximization objective improves the performance when there are no network uncertaintics (row 1) due to ability to cover diverse and informative scenarios. Additionally, ASA outperforms n-ASA in the Changing-Scand Scarce network configuration (row 5) due to its adaptive nature to network connectivity changes.
ASA (Ours)
n-ASA (Ours)
FD
Random
highest uncertainty and risk of failure [15]. Lastly, D-Fleet-DAgger (n-FD) adapts Fleet-DAgger to network uncertainties by filtering out robots with low connection probabilities.
How do ASA and n-ASA perform under different network configurations?
We evaluate the performance our policies, ASA and n-ASA, under different network configura- tions for each environment. The RoHE metric for each time step has been shown in Fig. 3, and cumulative success values at the final time step have been presented in Table In both metrics,
see that our ASA and A-ASA policies able to outperform other benchmarks. This is
7

---

ALLOCATION POLICY
ALLEGRO HAND CUMULATIVE
ANYMAL
BALL BALANCE CUMULLATIVE
HUMANOID
CUMULATIVE
CUMULATIVE
ROHE
SUCCESS
ROHE
SUCCESS
ROHE
SUCCESS
ROHE
SUCCESS
RANDOM
387.94
92.66
2.90
1448.73
FT
6.49
2119.53
1.77
956,53
FE
5.80
1774.47
1.44
113.47
4.46
1133,20
0.44
130.27
2251.06
1.77
171.60
4.28
1175.2
0.76
266.
N-FD
9.02
3723.80
1.07
195.40
3.05
1035.53
0.87
329.07
N-ASA (OURS)
9.75
3729.50
2.31
232.47
5.10
1167.27
1.61
408.07
ASA (OURS)
11.29
4217.70
2.30
234.27
8.15
1636.53
2.93
478.6
Table 1: Average RoHE and cumulative success of different allocation policies. As shown by the RoHE
and cumulative success values averaged over all network configurations, our ASA and n-ASA policies outper- form other baselines in all environments (columns). This shows the adaptabilityy of our methods to different environments and network configurations in terms of efficient use of human effort and high task completion.
because our policies can incorporate network uncertainty information into their allocation algorithm through stochastic submodularity. On the other hand, other benchmark policies, except n-FD, not designed to incorporate such network uncertainty. The n-FD policy is able to incorporate net- work uncertainties and outperforms our policies in the 5G network for the Allegro Hand task, but fails to generalize to other network configurations and tasks. We can see that our allocation policy outperforms other baselines in ferms of the RoHE metric by up 1.25x, 1.31 1.83 and 3.37x in Allegro Hand, ANYmal, Ball Balance, and Humanoid environments, respectively.
How do ASA and n-ASA compare when the network connectivity is stable?
To test whether our RoHE gains are only due to adaptability to different network configurations, we have also simulated network where all robots are always reachable. In row 1 of Fig. 3, we can see that our ASA and n-ASA policies still outperform other allocation benchmarks due to their ability to diversify the selected robots to cover more states. As we have shown in the 2D toy example in Fig. 2, rather than only focusing on the states with high uncertainty, ASA and n-ASA consider the whole state space to collect combined data that is more informative.
How do ASA and n-ASA compare to benchmarks with time-varying connection probabilities?
To test the adaptability of our allocation policies to network connectivities varying over time, we have run experiments the Sranging-Scaree network scenario. We can see in row 5 of Fig. 3 that the ASA policy consistently outperforms n-ASA, n-FD, and FD, due to its ability to dynamically update the network connection estimates based on the observations each robot is successful or not. This adaptability is particularly important in scenarios where network conditions are unpredictable and can change over time, such as mobile roboties applications.
whether the connection to
Limitations: Our work has several limitations. First, it uses only real-world data collected from 5G network without hardware robotics experiments and is tested only in NVIDIA's Isaac Gym en- vironment like previous benchmarks. Additionally, it assumes that network connectivity and robot states and policies are independent across robots, while in real-world scenarios, robots might share the same network or physical location, meaning their policies might affect each other.
Conclusion and Future Work
We present two novel supervisor allocation policies, ASA and n-ASA, for assigning human super- visors to robotic fleet for data collection. These policies, based on stochastic submodular maxi- mization, offer a modular approach for incorporating diverse allocation objectives, informativeness metrics, and retraining methods. Our allocation policies beat current benchmarks in RoHE and cu- mulative success metrics across most environments and network configurations. These performance gains are thanks to their stochastic submodular maximization objective, which incorporates network connectivity while balancing the diversity and informativeness of selected robots. Finally, we collect real-world 5G network data from field dedicated to teleoperated robots and show the applicability of our allocation policy real-world scenarios as well.
In a future project, we plan to extend our work to include hardware robotics experiments and other simulation environments, including teleoperation over 5G network, which would be possible in an application such as autonomous driving. We also plan to investigate the impact of different imitation learning methods to test the generalizability of our allocation policies.
8

---

Acknowledgements
This work was supported by the National Science Foundation under grant no. 2148186 and 2133481 as well as federal agencies and industry partners specified in the Resilient & Intelligent NextG Systems (RINGS) program. This article solely reflects the opinions and conclusions of its authors and does not represent the views of any sponsor.
References
[1] Kara Carlson. You now ride in driverless car in austin, gm-owned cruise expands rideshare services, Dec 2022. URL
ctt.storyooo//o///////oaooooo austnncce--uusiin-e austin-d
ss ////esccnolllog///22////////i//e
[2] Anthony James. Waymo expands us public operations with deployments in los angeles and
austin, Mar 2024. URL http::://wwwwwwwtooooooooooom
http://bbbbbotaaaa///aaaeaeaaaar i wit.sss-wwttd-deleememe.tt... n a-loosangeless and anddaustin..hm.
[3] Robert Valner, Houman Masnavi, Igor Rybalskii, Rauno Pölluäär, Erik Köiv, Alvo Aabloo, Karl Kruusamäe, and Arun Singh. Scalable and heterogenous mobile robot fleet-based task automation in crowded hospital environments-a field test. Frontiers Robotics and AI, 9, 08
2022. doi: 00.3889/frobb
[4] Joseph Quinlivan. How amazon deploys collaborative robots in its operations to benefit em- ployees and customers, Jun 2023. URL https: //ww: about wwwpw/ww///c./tt...t...t..... ope rations/ /how- ama zon depl oys robots- in- iitnn-its-ee cetess-pppaaiiinon facilitia 8.
[5] Cosmin Paduraru, Daniel Jaymin Mankowitz, Gabriel Dulac-Arnold, Jerry Li, Nir Levine, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning 110:2419 -106241922468, 2021. URL https://apt. semant semanticschen wws.hhlicclar..o.c org Co:pusID::338653
[6] Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020. URL https: //ppp/a..../ semant ott.:/ooo....ooooooooooooooa:::.
org/
[7] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learn- ing. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings Machine Learning Research, pages 991-1002. PMLR, 08-11 Nov 2022. URL https: pttttp::/ppocceeinngg mlr. pres 2vv64/jjang22 html.
[8] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018. ISSN 1935-8253. 16::0011561/30000053. URL http://dxx doi www.orrg//... 1561 1et.115561200000000 /2300000053.
[9] Eli Bronstein, Mark Palatucci, Dominik Notz, Brandyn Allen White, Alex Kuefler, Yiren Lu, Supratik Paul, Payam Nikdel, Paul Mougin, Hongge Chen, Justin Fu, Austin Abrams, Punit Shah, Evan Racah, Benjamin Frenkel, Shimon Whiteson, and Drago Anguelov. Hier- archical model-based imitation learning for planning in autonomous driving. 2022 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pages 8652-8659, 2022.
URL httpt:ttt///mm //api. semant www.chisscss..cc
[10] Bin Fang, Shi-Dong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362 369, 2019. URL https:
/ //ppi/app..///a semart wwwemattisccccaarrr.o orrg/oorp
USID:20273342
[11] Zipeng Fu, Tony Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile ma- nipulation with low-cost whole-body teleoperation. ArXiv, abs/2401.02117, 2024. URI. https: / api. semanticscholats org/ wttp/oorpssD::225555777777

---

[12] Carlos Celemin, Rodrigo Dretz-Dattrr, Eugenio Chisari, Giovanni Franzese, Leandro de Souza Rosa, Ravi Prakash, Zlatan Ajanović, Marta Ferraz, Abhinav Valada, and Jens Kober. Interactive imitation learning in robotics: A survey. Foundations and Trends@) in Robotics, 10(1-2):1-197, 2022. ISSN 1935-8253. do 1511/33000000000007. URL. http:// http:///xx.oo...oo. org/10. 1561 /2300000022.
[13] Kevin Jost. Starship goes deep on college food deliveries, Dec 2022. URL https://innst ceautoonooosvveiicle.
deep on conn-oolleee-ooo food-d
eliveries/.
[14] Mark DiPietm. Starship technologies paving way for robotics in food delivery, Oct 2022. URL http:////.loogg.......ee//aaaaaa
tex.=FFouneeet000022222222222200 20per$20day.
[15] Ryan Hoque, Lawrence Yunliang Chen, Satvik Sharma, Karthik Dharmarajan, Brijen Thanan- jeyan, Pieter Abbeel, and Ken Goldberg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Karen Liu, Dana Kulic, and Jeff Ichnowski, edi- tors, Proceedings of The 6th Conference Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 368-380. PMLR, 14-18 Dec 2023. URL. https: 7/ //ppooeeedingss ml prwss http////0////uu///
httl.
[16) Shivin Dass, Karl Pertsch, Hejia Zhang, Youngwoon Lee, Joseph J. Lim, and Stefanos Niko- laidis. Pato: Policy assisted telcoperation for scalable robot data collection, 2023.
[17] View All Posts by Mario Herger. First incuumeeeeeddeeeeeeettiiiion intervention by cruise teleoperator, Febru- ary httpa///eeeoo////saaaaaaaaaaaaaa
[18) View All Posts by Mario Herger. Rew.oee-ooottolllaaiiiiiniaaeeee February ot
phanton auto,
phantom
/ //remooteeccee wwwpotteeconttolllle---iii---iii
[19] Abhinav Dahiya. Route Planning and Operator Allocation in Robot Fleets. PhD thesis, Uni-
versity of Waterloo, 2023.
[20] Adriana Noguera Cundar, Reza Fotouhi, Zachary Ochitwa, and Haron Obaid. Quantifying the effects of network latency for teleoperated robot. Sensors, 23:8438, 10 2023. doi: s23208438.
[21] Sandeep Chinchali, E. Pergament, M. Nakanoya, E. Cidon, E. Zhang, D. Bharadia, M. Pavone, and S. Katti. Harvestnet: Mining valuable training data from high-volume robot sensory streams. In 2020 International Symposium on Experimental Roboties (ISER), Valetta, Malta, 2020.
[22] Sandeep Chinchali, E. Pergament, M. Nakanoya, E. Cidon, E. Zhang, D. Bharadia, M. Pavone, and S. Katti. Sampling training data for continual learning between robots and the cloud. In 2020 International Symposium on Experimental Robotics (ISER), Valetta, Malta, 2020.
[23] Yuchong Geng, Dongyue Zhang, Po-han Li, Oguzhan Akein, Ao Tang, and Sandeep P Chin- chali. Decentralized sharing and valuation of fleet robotic data. In 5th Annual Conference on Robot Learning, Blue Sky Submission Track, 2021.
[24] Oguzhan Akein, Po-han Li, Shubhankar Agarwal, and Sandeep P. Chinchali. Decentralized data collection for robotic fleet learning: A game-theoretic approach. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 978-988. PMLR, 14-18 html.
Dec 2023. URL httpp:://toocceeeeeeeeeeeeeeee
ngs ml wttggssml.....sss//..... wtwppp/w////aaa//ppppppp
[25] Crystal Chao, Maya Cakmak, and Andrea L. Thomaz. Transparent active learning for robots. In 2010 5th ACM/IEEE International Conference on Human- Interaction (HRI), pages 317-324, 2010. 100.1100//LL2200..55444
[26] Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madiisnn, 2009.
[27] Oguzhan Akein, Orhan Unuvar, Onat Ure, and Sandeep Chinchali. Fleet active learning: A submodular maximization approach. In Jie Tan, Mare Toussaint, and Kourosh Darvish,
10

---

editors, Proceedings of The 7th Conference on Robot Learning. volume 229 of Proceedings of Machine Learning Research, pages 1378-1399, PMLR, 06-09 Nov 2023. URL https: 7/ 7/proceed: ngs ml press / /229/2222//// //ccciakiiii n23a .html.
[28] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with im- age data. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1192 PMLR, 06-11 Aug 2017. URL http:t/t////ooo v70/ga1172 htt.gall7..h..........
ngs mll prrssss////
[29] David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. Active learning with statistical models. JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH, 4:129-145, 1996.
[30] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d' A'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL hitps http:tts:://rcceeeeeeeeeeeeeeeee http://tccceeee.eccooo neur file//illlle /tt.e/ee//52330222222222222
cpss.ccpppeerffiie foarr_fllee www.ffeeeeppppe 5ed90-Paper. pdf..
[31] Faraz. Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the (wenty-Seventh International Joint Conference on Artificial Intelligence, IICAI-18, pages 1950049577 International Joint Conferences on Artificial Intelligence Organi- zation, 2018. doi: 1o::100.4993//jcaa..200 8/6877 ai. 20.2016867.
URL https: pestpss//dollo //tp///oiii/i/ org/ 249.2.......//.
[32] Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limita- tions of behavior cloning for autonomous driving. 2019 IEEE/CVF International Conference
on Computer Vision (ICCV), pages 9328-9337, 2019. URL htt.t:/t//// /api. semantea remantion iscc holar.
[33] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 4572-4580, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 97815
[34] Brenna Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems. 57:469-483 05 2009. doi: 10. (0.1016/j.robo..208.. 10.024.
[35] Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 627-635, Fort Lauderdale, FL. USA, 11-13 Apr 2011. PMLR. URL https::/proceed: wtw..mmr....ss.e.mmmmmmmmmm wwwpppresssvvv /tassss/s///// ross html.
[36] Sonia Chernova and Manuela Veloso. Interactive policy learning through confidence-based autonomy. J. Artif. Int. Res., 34(1):1-25, jan 2009. ISSN 1076-9757.
[37] Snchal Jauhri, Carlos Celemin, and Jens Kober. Interactive imitation learning in state-space. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 682-692. PMLR, 16-18 Nov 2021. URL https:/ http::///occeigeeeee mlr. wttpp//15s////e//ppppppppp jauhr i2la.htma html.
[38] Shih-V Yi Chien, Yi-Ling Lin, Pei-Ju Lee, Shuguang Han, Michael Lewis, and Katia Sycara. Attention allocation for human multi-robot control: Cognitive analysis based on behavior data and hidden states. International Journal of Human Computer Studies, 117:30 44, 2018. ISSN
1071-5819. dorhttps://doioor//1..00 wttp:...c..............o
2018 2000003.00000000000 005. URL https: / //ww.. sc 00009993399XX Cognitive
iencedi reett.diret..c cwm/ sclssciiiinn htt.c////.aaaeceeeeereee//////// Assistants.
[39] Robin R. Murphy and Erika Rogers. Cooperative Assistance for Remote Robot Supervision.
Presence: Teleoperators and Virtual Environments, 5(2):224-240, 08 1996. 10. 1162prrssssee 1996.5.2.224. URL https: doi http///oo.ogg/////////oo///////t 1tl.:11222ppp /62/pree...... 5..6..2..2....
[40] Kunal Menda, Katherine Rose Caapbela and Mykel J. Kochenderfer. Ensembledag- ger: A bayesian approach to safe imitation learning. In 2019 IEEE/RSJ International
11

---

Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pages 5041-5048. IEEE, 2019. 2019. 8968287.
201. URL
https: //t:::///o.... www.o/or//.11.... 11..0.1TTO0100999000000 TROS40897
[41] Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Alcksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 598-608. PMLR, 08-11 Nov 2022. URL stttps://en pttpps:/proceeeiing mlr press/ v1 64 2066/4/uue2222 html.
[42] Andreas Krause and Daniel Golovin. Submodular function maximization. In Tractability, 2014.
[43] Baharan Mirzasoleiman, Amin Karbasi, Rik Sarkar, and Andreas Krause. Distributed sub- modular maximization. Journal of Machine Learning Research, 17(235):1-44, 2016. URL
htttp://mmie wwtpt//ppppppp.ppppeeeeeeeeeeeet 7 /mi www.rraaasaaaaaaa.a
html.
[44] Hui Lin and Jeff Bilmes. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies Volume 1, HLT 11, pages 510- 110-520, USA, 2011, Association for Computational Linguistics. ISBN 9781 9781992432879.
[45] Arash Asadpour, Hamid Nazerzadeh, and Amin Saberi. Stochastic submodular maximization. In Christos Papadimitriou and Shuzhong Zhang, editors, Internet and Network Economics, pages 477-489, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg.
[46] Daniel Golovin and Andreas Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal of Artificial Intelligence Research, 42, 03 2010. doi: 10.110..13/jaal 613/rair.32775
[47] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Re- covery rl: Safe reinforcement learning with learned recovery zones, 2021. URL https: //arxiv. www. abs/ 15920.
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[49] Ookla. Internet speed dataset. https::///// //t: kaggle, le. com homm/ddaaeet//dchaa e looklal
dav 2022. [Online; accessed 15-February-2024)
[50] A. M. Frieze. cost function property for plant location problems. Math. Program,, 7(1): 245-248. dec 1974. ISSN 0025-5610. dot:100.1077/F01555522 URL https://doil or g/10. 100100007/0 (C00/BF0158500 0158551..
[51] Aditya Narayanan, Pranav Kasibhatla, Minkyu Choi, Po han Li, Ruihan Zhao, and Sandeep Chinchali. Peernet: An end-to-end profiling tool for real-time networked robotic systems, www.ii.orgrbb////00000 06078.
2024. URL httpp////aaaii.oog//ba//////////
12

---

A Appendix
Code Availability: The code and related materials can be found in the following code repository:
https://gittii com/ www//eoo//uueeeeeeeeeee UUtuutttutttttt in- www.ssswwawaaeeeeeeeeeeeeeeeeeee Sww.maaab///eee---eeeereeee Allooattio....oiiioooo. git
The organization of the appendix follows:
Subsection A.I describes the details the submodular maximization function we have used our experiments.
2. Subsection A.2 describes the details of the theoretical bounds of our allocation algorithm.
3. Subsection A.3 presents the environment setups and exact parameters we have used for our simula- tions.
4. Subscction A.4 provides the details for the network configurations used our experiments.
5. Subsection A.5 presents all the simulation results for all tasks and network configurations as well as additional metrics providing an insight into why methods outperform other benchmarks.
Subsection provides ablation studies over hyperparameters and simulation parameters.
7. Subsection A.7 presents the complexity analysis of our allocation policies.
Subsection presents our 5G network data collection setup.
A.1
Submodular Maximization Parameters
This section provides the exact parameters and functions used in the submodular maximization method given in Section 4.
A.1.1 Informativeness Function
The informativeness function Lt' measures the informativeness of each rohot i in the flect at time f. To measure the informativeness of the robots, the uncertainty of the robot policy πR the environment, robot i violating the constraints K. Then, the informativeness function can be defined as follows:
weighted combination of functions. The first function measures
second function measures risk of
(1 = audhine (i)
ou Miie
(8)
Here, ou a hyperparameter that controls the weight of the uncertainty the overall informativeness measure, Mane (f) and Miste (i) are the uncertainty and risk functions the robot at time f, respectively. When the robot is taking discrete actions, the uncertainty function (1) is defined as the entropy of the robot policy Th. and when the robot is taking continuous actions, the uncertainty function is defined as the ensemble variance of the robot policy Tk [40]. The risk function 24 (i) is defined the likelihood of the robot exiting the constraint space K [15).
both uncertainty and risk functions, if value of the function the robot below certain threshold, we set it to zero. We define this threshold parameter as Ueltee and thres for uncertainty and risk functions, respectively. We present specific Letlees and fi parametens for the experiments in Table 2.
A.1.2 Similarity Function
The similarity function St measures the similarity between two robots and j in the fleet at time t. In our experiments. we utilize both the similarity between the states and the similarity between the actions taken by the robots. More formally, the similarity function is defined follows:
of
of
(i = as || |5]]
+
(1 - as)
(9)
of ||
where and sf are the states of the robots and j, respectively; of and a are the actions taken by the robots i and j. respectively, at time t. as is a hyperparameter that controls the weight of the state similarity in the overall similarity measure.
A.1.3 Constraint Violation Function
The constraint violation function (C)(1) measures whether robot violates the constraints at time f. In our experiments, we used the constraint violation as an indicator function that returns if the constraint is violated and 0 otherwise. The ox is parameter that controls the relative importance of the constraint violation in the
13

---

overall objective function. In our experiments, we set 10000 to prioritize the robots with constraint violations. The constraint violation function is defined follows:
= 11 axc if
(1)
(10)
otherwise.
Here, the K refers to the safe states that the robot can operate without any human intervention. As the constraint function causes the system to prioritize the robots violating the constraints, in initial time steps, we set the ax: 10000 to ensure that the robots violating the constraints not prioritized. This is because, in the initial steps. robots explore the environment, the constraint violations. control the length of this period which the constraint- viioiaing robots are not prioritized by the fw parameter.
collecting more informative data more critical than
A.2 Theoretical Bounds
Here, we provide the definitions and necessary theorems to show the optimality bound of our proposed ASA and n-ASA policies. We first define submodular and monotone functions and then provide the necessary theorems show the optimality bounds of proposed policies.
Definition (Submodular Function). set function f 2 + is submodular for all A C B V and
V B, we have:
f(Au f(A) 2
Definition (Monotone Function), set function f 2 TR monotone for A C V, we have:
f(A) S (B)
Definition 3 (Facility Location Problem). The facility location problem is combinatorial optimization prob- lem where the goal is to select a subset of facilities from a set of candidate locations minimize the cost of serving demand points. The objective function of the facility location problem defined as follows:
f(A)
2 min My. JEA
where is the set selected facilities, My is the cost of serving demand point from facility and is the number of demand points.
Corollary 1 (Facility Location Problem is submodular for non-negative Mog.) The facility location problem submodular and monotone if the matrix Mij non-negative. Please refer [50] for the detailed proof.
Exactly following that corollary, we can show that the facility location problem presented in Equation 4 submodular and monotone, as cost matrix Me non negative and C is binary random variable.
Definition 4 (Greedy Policy). The greedy policy is a sequential optimization method that selects the element with the maximum marginal gain each step. The greedy policy that selects & elements is defined in Algorithm
Algorithm Greedy Policy
1: A +-0 0
2: for 1 k do
3: I; + www.aa ((AU(z)) ((A)
4: A Au
5: end for
6: return A
Theorem (Adaptive Greedy Policy Optimality Bound), Consider the adaptive greedy policy selecting ele- ments with maximun marginal value, conditioned on the realized value of previously chosen elements. The approximation ratio the adaptive greedy policy with respect to the optimal adaptive policy is 1. For the detailed proof of this theorem, please refer to [45].
In work, 8). our policy becomes adaptive. Additionally, when the and F' functions match, and the marginal threshold parameter is set Theorem can be applied to show the optimality bound of our proposed policies.
able observe whether the connectivity robots is established not (Alg. 1. line
14

---

Theorem 2 (Non-Adaptive Greedy Policy Optimality Bound) ). Consider the non-udaptive greedy policy that, each step. chooses the element with the maximum marginal increase in value. The appreximation ratio this policy with respect to the optimal udaptive policy is at least (1 - 2)² For the detailed proof of this theorem, please refer to [45].
Unlike ASA policy, n-ASA policy is not able to observe whether connection to robot i has been established or not, so it is non-adaptive. Under the same conditions as the ASA policy, when the F and F functions match, and the marginal threshold parameter is set to O, Theorem 2 can be applied to show the optimality bound of our proposed n-ASA policy.
A.3 Experimental Setups and Parameters
As stated previously, we run simulations using four different environments: mal Allegro Hand, Hu- manoid, and Ball Balance. Each environment has its own defined tasks, success criteria, and constraint viola- tions. For the ANYmal robot, constraint violation occurs when there is excessive force on the robot's knees, indicating that the robot has fallen on knees, or when no force is exerted on the bottom its toes, indicating that the robot has fallen on its torso. For the Ball Balance environment, a constraint violation occurs when the ball is no longer on the plate. In the Allegro Hand environment, constraint violation happens when the cube is no longer in the robot's 's hand. For the Humanoid environment, constraint violation occurs when the robot's s position below the termination height, indicating that the Humanoid has fallen down.
The definition of success is specific to each task. For instance, in locomotion tasks, success is achieved if the robot does not violate constraints and reaches a reward amount that exceeds a predefined reward threshold. For goal-specific tasks such as Ball Balance and Allegro Hand, success corresponds to reaching the goal state without violating constraints, For Ball Balance, goal state may be one where the ball on the plate is moving within radius smaller than the plate's radius, indicating that the robot successfully managed to control and balance the ball. For Allegro Hand, the goal state may be defined as holding the cube stable after rotating so that the red surface faces up. That how single success corresponds to different achievements depending on the specific tasks assigned to each robot.
For all experiments, the key parameters are fixed and do not depend on the allocation policies: Nuuman = 5, Notoa = 100. T' = 10, 000 time steps, is 5 time steps and (y = 5 time steps. The hyperparameters that vary depending on the task, along with the values that yielded the best performances, Table 2. [S] and |A| are the dimensionalities of the state and action spaces, respectively, 24dres and 24 are the uncertainty and risk threshold values below which the uncertainty and risk are treated as zero, tw is the period during which constraints are not prioritized, allowing the robot policies to be improved by selecting informative robots rather than resetting failing robots in the first tw time steps, threshold is the marginal increase threshold below which the robots not prioritized, as is the parameter which controls the weight of the state similarity in the overall similarity measure, and cry is the parameter that controls the weight of the uncertainty in the overall informativeness measure.
provided below in
Task
|S|
|A|
thtres
thtree risk
fw
threshold
0g
AllegroHand
88
21
0.53
0.12
1250
0.04
0.37
0.53
AnyMAL
48
12
0.19
0.49
1000
0.69
0.72
0.05
BallBalance
24
3
0.47
0.21
1750
0.51
0.98
0.46
Humanoid
108
21
0.18
0.20
2500
0.23
0.50
0.10
Table 2: Simulation environment hyperparameters for each task.
A.4 Network Configurations
Here, we explain the details of the network configurations used in our experiments. We have used four different network configurations to evaluate the adaptability of the allocation algorithms in different network conditions. Additionally, we show the connection probabilities in each network configuration in Figure 4. The network configurations are as follows:
Always: Always is a simple network configuration where the probability of connection to all the robots is set to 1. In this network configuration. our supervisor allocation problem is equivalent to the Interactive Fleet Learning (IFL) problem presented in [15], where the supervisor can connect all the robots at all times.
15

---

Mixed-Scarce: Mixed-Scarce is network configuration where the probability of connection to robots can be set to two different values. In this network configuration, we first divided the robots into two groups with ratios of 0.7 and 0.3. We then set the probability of connection to the robots in the first group to 0.9 and the probability of connection to the robots in the second group to 0.1. This network configuration is used to evaluate the adaptability of allocation algorithms when the connectivity to the robots is heterogeneous. Idcally, the resources to the robots with higher connectivity maximize the performance
supervisor should allocate of the fleet.
Ookla: Ookla is network configuration where the probability of connection to the robots is hased the Ookla cellular network performance data [49]. This dataset includes the download speed, upload speed, and latency of the cellular network in different locations. We use the download speed as the metric to determine the probability of connection to the robots. We first divided the data collection points into grid of 10 x 10 cells. We then calculated the average download speed of the data collection points in each cell. After that, we log-normalize the average download speed of each cell be in the range of [0.5, 1]. We have set the lower bound to 0.5 to ensure that the robots in the cell with the lowest download speed have a non-zero probability of connection. We then sct the probability of connection to the robots in cach cell to be the normalized average download speed of the cell. This network configuration is used to evaluate the adaptability of allocation algo- the robots is based on real-world cellular network performance data, which is
rithms when the connectivity heterogeneous and has a more complex structure than the Mixed-Scare network configuration.
5G: SG a network configuration where the probability of connection to the robots is set based on the real- world 5G network performance data. Please refer to Section A.8 for more details on the data collection process. The collected data was divided into 100 groups, with average latency and throughput calculated for each group and normalized to value between 0.015 and 1. A lower bound of 0.015 ensures non-zero connection probability for robots with the lowest throughput and highest latency. Robots in groups with throughput below 0.4 and latency above 0.6 were assigned normalized value of 0.015. The connection probability for each group corresponds to the normalized average throughput and latency. This configuration evaluates the adaptability of allocation algorithms to realistic, heterogeneous connectivity based on real-world 5G network performance, which is more complex than other network configurations.
Langing-Scareee Changing- Scarce is network configuration where the probability of connection to robots first set as in the Mixed-Scarce network configuration. Then, over time, the probability connection to the robots in the first group is decreased from 0.9 to 0,1, and the probability of connection to the robots in the second group is increased from 0.1 to 0.9 linearly. This network configuration is used to evaluate the adaptability of allocation algorithms when the connectivity the robots changes over time. Ideally, the supervisor should adapt the allocation of humaan supervisors the robots based the changes in connectivity maximize the performance of the fleet.
A.5 Numerical Results and Additional Metrics
this section, we present the numerical values for all allocation policies and for all tasks under each network configuration to demonstrate that our method outperforms the baseline algorithms in all simulated scenarios, providing a novel approach to the supervisor allocation problem. We also present the percentage performance differences between our methods (ASA and n-ASA) and other methods. We present all numerical results recorded in the final timestep (t 10, 000) in Table and the percentage differences in Figure 5.
In addition the presented metrics in the main paper, we also provide the following metries evaluate the performance of our method against the haselines: (1) cumulative human actions, which measures the total number of time-steps humans supervised the robots, (2) cumulative idle time, which is the total duration that robots remain in the constraint violating states while awaiting hard resets, (3) cumulative hard resets, which records the total number of hard resets performed by the human supervisors, (4) cumulative reward, which is the total reward accumulated by the all robots the fleet, and (5) cumulative success, which is the total number of tasks successfully completed by the entire flect. We present the these metrics in Figures 6, 7, 8, 9, and 10 respectively.
In Figures 6. 7 and 8, we can observe that the ASA and n-ASA methods use fewer human actions and are able to reduce idle time and hard resets compared to the other benchmarks. This is due to the fact that our method prioritizes the robots with higher connectivity to reset and supervise, wasting less human resources on robots with low connectivity, unlike the other method possibly trying to reset the robots with low connectivity. Additionally, in Figures 9 and 10, we can see that our ASA and n-ASA methods are among the top top-performing methods in terms of cumulative reward and cumulative success in all network configurations and tasks, proving that our method is much more efficient in terms of human resource utilization.
16

---

Always
Mixed-Scarce
Ookla
1.0
1.0
1.0
Connection Probability
0.6
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
Region
Region
Region
5G
Changing-Scarce
1.0
Connection
Probablity
0.6
0.6
0.2
0.2
0.0
0.0
Region
Region
figure shows the connection probabilities of the robots in the fleet for each network configuration. For easier visualization, we have grouped 100 robots into 10 groups of 10 robots each and presented the average connection probability for each group. As go from the Always network the Changing- Scarce network, the connection probabilities of the robots get more heterogeneous. This heterogeneity the connection probabilities is crucial for evaluating adaptability allocation algorithms in different network configurations.
Figure 4: Connection probabilities for each network configuration.
A.6 Ablation Studies
In this section, we conduct further experiments using our adaptive submodular allocation, ASA, policy to ex- plore the following: (1) the sensitivity of the system the ratio of the number of robots Nrabx to the number of humans Naaman (Figure 11), (2) the impact of varying the minimum intervention time tr (Figure 12), and (3) the impact changing the hard reset time tR (Figure 13). Each experiment averaged over three different random seeds, and the shaded regions correspond to one standard deviation. We plotted four different metrics: (1) cumulative success, (2) RoHIE, (3) cumulative hard resets, and (4) cumulative idle time. Cumulative hard resets represent the total number of hard resets performed by human supervisors when the robots violate straints. Cumulative idle time the total time, time steps, that robots remain idle while waiting hard reset.
Number of Human Supervisors: We tested the ASA policy to evaluate its sensitivity to different numbers of human supervisors (Fig. 11). Keeping the number of robots constant, we simulated scenarios with 1, 5, 10, 25, and 50 human supervisors. In all simulated tasks, the number of human supervisors increases, idle time decreases because more human are telcoperated and reset. However, despite cumulative success values rising with more supervisors, RoHE values tend to decrease. This happens because allocating more humans doesn't 't always lead to higher return on human effort. The most informative and important robots are already being selected, so adding more supervisors doesn't necessarily result in significant marginal gain. Therefore, low number of human supervisors insufficient as robots remain idle for long periods and violatc constraints more frequently, while a large number of supervisors creates a surplus and decreases efficiency.
available, resulting in shorter idle periods before robots
Minimum Intervention Time: While keeping other hyperparameters fixed, we varied the minimum interven- tion time and ran our policy. We observed that when the minimum intervention time long, such as 100 or 500 time steps. the robot fleet performance significantly decreases. This is because human supervisors spend lot of time teleoperating single robot, which results in lower RoHE and cumulative success values, and substantial increase idle time. Conversely, when minimum intervention time very short, as time step. performance improves in terms of both RoHE and cumulative success for most tasks. This is because each human supervisor spends less time single robot and can attend to more robots within 10,000 time steps, thus euhancing overall flect performance the minimum intervention time decreases.
17

---

NETWORK
ALLOCATION POLICY
ALLEGROHAND CUMULATIVE
ANYMAL
BALLBALANCE
HUMANOED
ROHE
SUCCESS
ROHE
CUMULATIVE
SUCCESS
ROHE
CUMULATIVE
SUCCESS
ROHE
CUMULATIVE
SUCCESS
10.48
649.7
160.0
3.61
1796.0
0.01
11.10
2751.
287
1437.0
FE
10.03
2021.67
2.81
127.33
6.82
1185.0
1.04
296.67
ALWAYS
189
424
1348
189
N-ASA (OURS)
16.30
5094.67
3.76
246.33
10.67
1802.67
505 33
(OURS)
14.77
3064
3.41
241
10.87
785.07
2.28
530.
302.0
0.65
2.64
0.67
17130
198
985.0
FE
3.69
1616.33
0.89
110.33
3.54
1327.67
0.09
15.0
MIXED-SCARCE
FD
1.94
2031.0
1.29
180
3.32
1253,33
0.33
139.33
2370.66
1.07
0.37
N-ASA (OURS)
8,15
3850.67
2.46
244.33
5.85
1580.67
1.95
499.67
(OURS)
8.16
3817.33
2:29
235.33
7.27
1621.0
1.83
491.0
RANDOM
8.45
576.33
1.97
164.67
3.57
1782.33
0.01
1.67
583.33
FE
8,34
2072.33
2:21
126.0
7.45
1301.33
0:70
235.33
OOKLA
FD
1063
29866 33
2.76
136.07
6.45
1443:31
1:47
392 33
N-FD
6191.33
0.55
275.33
200
1491.67
517.33
N.ASAYOURS)
4694.33
3.20
1791.67
2.25
506.67
(OURS)
1575
4936.0
5:11
225.67
11.41
1741.33
2.13
ST0.33
RANDOM
2.47
228.67
0.36
47.33
2.81
1401.33
0.00
0,00
2131.67
3.98
1789.67
1.07
137.0
3.01
1315.0
0.37
169.33
5G
FD
5.10
2213.0
140
200
2.75
1209.0
0.79
319.33
N-PD
12.01
6909
0.52
259.33
2.86
1431
0.96
479.67
(OURS)
3705.33
7.45
1758.33
ASA (OURS)
1.43
365K00
1.63
239,33
6.39
1703.33
1.51
470.0
RANDOM
1.9
183
0.08
12
1.89
945
0
1418.66
270.66
1.48
CHANGING- SAARGE
FD
1.66
672.66
0.27
102.33
0.98
470.66
0.07
35.33
N-PD
1.39
695.66
0.23
115.56
0.45
219
0.07
39
A:ASA(OURS)
2.65
1302.64
0.52
185.34
1.51
661.33
0.19
86.33
(OURS)
3612.33
1328.33
1.01
Table 3: Numerical results for all network and task simulations. We present RoHE and cumulative success values the final timestep (t 10, 000) for 4 different environments under 4 different network configurations.
Hard Reset Time:
Finally, the ASA policy with different hard reset times. We observed that the hard reset time increases, the fleet performance decreases. This is because takes longer for human supervisors to reset the robots, resulting in fewer hard resets within 10,000 time steps. Consequently, the idle time increases, reducing the overall performance of the robot fleet.
A.7
Complexity Analysis and Optimality Bounds for Allocation Algorithms
Here, present the complexity analysis and optimality bounds for our allocation algorithms.
A.7.1
Complexity Analysis
now explain the complexity of our allocation algorithms terms the number of robots Nesbor and the number of humans Nuumm in the system and function evaluations. As we have discussed in Section 4, our allocation algorithm is based on a greedy algorithm that selects the robots based on the stochastic submodular maximization objective. It is well-known result that the number of function evaluations for the greedy algo- rithm N.noonnNumma As both of our algorithms are based on the greedy algorithm, the computational and time complexities of our algorithms, ASA and n-ASA, are both (NNrno Nuuman))
A.8 Real World 5G Network Data
In addition to the simulated network connectivity data, we also evaluate our allocation policies using real- world 5G network connectivity data collected via setup detailed in [51]. This setup includes two hardware components: mobile edge device and a local server. The edge device, which can be robot, mobile phone, or a computer, acts as the connection client. The local server functions the cloud. In our scenario, we consider the edge device to be the robot and the local server to be the cloud or the server from which human supervisors connect to the robots. The local server sends packets to the edge device, and the edge device responds with packets to confirm receipt. During this process, the local server calculates latency and throughput, saving this data local file. This continues for predetermined data collection period of 24 hours. Two key aspects of this setup are: (1) the edge device connected to a 5G cellular network, specifically 5G cellular provided by AT&T, and (2) it is mobile. This allows us to collect data wnywhere whether moving or stationary. for any desired period. To obtain data that realistically simulates human teleoperation connectivity, we collected data in a building where actual teleoperation and robotic tasks conducted. After collecting the data, divided and
18

---

Always
Mixed Scarce
Ookla
Allegro Hand
+20.96%
+39.32%
+12.99%
Ball Balance
+37.24%
+105.56%
+52.98%
Humanoid
+18.22%
426.09%
153.20%
ANYmal
+20.43%
103.24%
+16.26%
(%)
(%)
Changing Scarce
Average
Allegro Hand
-36.27%
+660.44%
+59.59%
Ball Balance
+147.26%
+154.18%
+99 +99.44%
Humanoid
157.02%
13122 71%
1373.45%
ANYmal
+15.88%
+58.10%
+42.77 78%
150
250
1000
300
200
300
RoHE (%)
ROHE
Always
Mixed Scarce
Ookla
Allegro Hand
Ball Balance
Humanoid
ANYmal
-12.83%
10
20
30
40
50
50
100
150
-20
-15
-10
-5
0
& Cumulative Success (%)
Cumulative Success (%)
A Cumulative Success (%)
5G
Changing Scarce
Average
Allegro Hand
38.34%%
+154,86%
+42.13%
Balance
+22.70%
+40.51%
+17.24%
Humanoid
.2.0%%
40
20
20
200 400
GOD 800
50 100 150 200
5 Cumulative Success (%)
Cumulative Success (%)
Cumulative Success (%)
Figure 5: Performance gains of ASA and n-ASA policies ronments and network combinations. The top two show the percentage difference in RoHE, while the bottom two rows depict the pereentage difference in cumulative success compared the best baseline methods. Notably, as highlighted in the bottom right figure, our policies outperform the bascline methods, particularly when averaged across different network combinations, demonstrating the robustness and effectiveness of our approach.
baseline methods across different envi-
clustered into 100 different groups. This division helps correlate the data with our fleet lcarning simulation environment, which has 100 robots in different locations. For each group, calculated the average latency and throughput. We normalized the average values between and such that groups with high latency and low throughput values have a connection probability closer to 0, and groups with low latency and high throughput values have a connection probability closer to I. Now that we have 100 different connection probability values, we randomly assigned them to 100 simulated robots. We illustrate the data collection setup in Figure 14. Additionally, present the average throughput and latency for each group Figure 15.
19

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
50000
15000
40000
Always
Cumulative
00000
4000
15000
20000
3000
2000
5000
10000
Mixed-Scare carce
50000
50000
50000
Cumulative
30000
30000
30000
20000
20000
20000
20000
00000
50000
50000
50000
50000
60000
40000
60000
40000
Ookla
Cumulative
00000
20000
20000
20000
10000
10000
10000
10000
50000
50000
50000
50000
5G
00000
30000
30000
30000
20000
20000
20000
20000
Changing-Scane Scarce
50000
50000
60000
40000
60000
40000
Cumulative
00000
20000
20000
20000
30000
10000
10000
10000
2520 5000 7500 10000
2500 5030 7500 10000
2500 5000 750D 10000
2500
5000 2500 10000 Time Steps (t)
Time Sieess (t)
Time Steps (t)
Time Steps (t)
ASA (Ours)
n-ASA (Ours)
FD
Random
FE
FT
n-FD
Figure 6: Cumulative human actions for different network configurations in different euvironments.
This figure shows cumulative human actions. which is the sum of the total number of teleoperation/supervviioo attempts the human supervisors, for different network configurations different environments. The x-axis represents the time steps, and the y-axis represents the cumulative human actions.
20

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
350000
140000
Always
352000
330200
120000
100000
25000
200000
15000
150000
200000
200000
40000
50000
20000
5000
Mixed-Scarce
630000
160000
700000
8000000
500000
620000
Cumulative
500930
400000
400000
300000
300000
300000
200000
200000
200000
100000
140000
800000
20000
120000
20000
Ookla
Cumulative
300000
100000
25000
B0000
300000
20000
00000
15000
200000
100000
10000
200000
150000
Cumulative
500000
150000
600000
125000
5G
400000
400000
100000
300000
100000
200000
50000
100000
25000
Changing-Scaii Scarce
800000
800000
500000
500000
Cumulative
500000
400000
300000
200000
400000
400000
200000
2500 5000 7500 10000
2500 5000
7500 10000
2500 5000 7500 12000 2500 5000 7500 10000
Time Steps
Time Steps
Time Steps
Time Steps
ASA (Ours)
n-ASA (Ours)
FD
Random
FE
+
FT
n-FD
Figure 7: Cumulative idle time for different network configurations in different environments. The plots show the cumulative idle time, which the of the total duration that robots remain in the constraint- violating states while awaiting hard resets, different environments and network configurations. The x-axis represents the time steps, and the y- y-axis represents the cumulative idle time.
21

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
14030
17300
12000
19280
Always
Cumulative
12580
33400
12500
1000
7100
1500
5000
5000
1000
2500
500
12000
12000
12000
Mixed-Scarce carce
10000
10000
10000
2500
Cumulative
8000
8000
5000
6000
6000
6000
4000
4000
2000
16000
12000
37500
Ookla
Cumulative
1,0000
10200
13000
32500
2500
2000
6000
8000
6000
10000
7500
1000
5000
2000
2500
12000
14000
2500
5G
Cumulative
10000
8000
2300
8000
15000
1000
10000
4000
5000
Changing-Scant Scarce
7000
2500
Cumulative Hard Resets
4000
4000
3000
2000
2000
1000
1000
2002 AD00 S000 M000100000
2000 4000 naDo nonc11000 10000
2000 4000 6000 RD0D1000D Time Steps (t)
2000 4000 6000 000012000 Time Steps (t)
Time Steps
Time Steps (t)
ASA (Ours)
-/-= n-ASA (Ours)
FD
Random
FE
+
FT
n-FD
Figure 8: Cumulative hard resets for different network configurations in different environments. The plots show the total number of hard resets performed the human supervisors in different environments and network configurations. The x-axis represents the time steps, and the y-axis represents the cumulative hard resets.
22

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
Cumulative Reward
20000
Always
10000
5000
Srixed-Scarce
Cumulative Reward
20000
500000
100000
13900
0.75
300000
0.0
5000
0.00
Cumulative Reward
25000
20000
Ookla
1.0
10000
3020
25000
Cumulative Reward
700000
000000
20000
5G
500000
15000
400000
300000
10000
5.
200000
100000
0.0
Changing- Saancee
3.50
500000
400000
2.00
Cumulative
300000
8.50
0.25
2000 4000 R00d 100010000 Time Steps
2000 4000 6000 R300LD000 Time Steps
2000 4000 1020 50C010D0D
2000 4000 6000 000012000 Time Steps (t)
Time Steps (t)
ASA (Ours)
n-ASA (Ours)
FD
Random
FE +
FT
n-FD
Figure 9: Cumulative reward for different network configurations in different environments. The plots show the cumulative reward. which is the sum of the total reward accumulated by all robots in the fleet, for different network configurations different environments. The x-axis represents the time steps, and the y-axis represents the cumulative reward.
23

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
1000
1790
250
1590
Always
1250
400
2000
300
100
500
1000
250
100
Mixcd-SSarce
2000
1.50
400
200
1000
300
190
2000
1500
1000
100
50
500
8000
3000
5000
1900
Ookla
1290
Cumulative
300
150
1000
2000
500
100
100
50
8000
1750
5.0
5000
400
200
5G
300
Cumulative
1000
100
1.0
Changing- -carrce
432
1200
892
30000
1000
302
2500
230
2000
200
1500
150
200
2000 4000 K0n0 maco 10000
7000 4000 6000 1000 10000
2000 4000 6000 100010000 10000
2000 40D0 GOOD 8000 1000 Time Steps
Time Steps (t)
Time Steps
Time Steps (t)
n-FD Figure 10: Cumulative success for different network configurations in different environments. The plots
ASA (Ours)
- n-ASA (Ours)
FD
Random
FE
FT
show the cumulative success, which is the of the total number of successful tasks completed by all robots the fieet, for different network configurations in different environments. The x-axis represents the time steps, and the y-axis represents the cumulative success.
24

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
10000
700
1750
230
8000
1100
Cumulative Success
1250
200
100
750
50
0010
10000
3100
32000
1000
Cumulative
20003
1000
8003
4000
3000
1800
2003
2000
1/1000
175000
Cumulative
110000
150000
600000
123000
121005
100000
50000
100009
100000
40000
75,000
75000
100080
20000
50000
20000
25000
20
5.0
ROHE
2.0
1.0
3.5
8.5
1.0
8.5
0.0
2500 Steps
2500
Steps (t)
Steps (t)
Steps
Number of Human Supervisors Nnumans
10
25
50
Figure 11: Simulation results with varying numbers of human supervisors. The figure shows the simula- tion results of the ASA policy for each task under the Always network configuration, with the number of robots Notho fixed 100, while the number of human supervisors Nhuma varies.
25

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
250
Cumulative Success
510
1250
400
1000
240
100
Cumulative
10000
5000
1000
5000
21000
6200
4000
1500
3000
4200
2000
2000
1000
509
000000
000000
500030
Cumulative
#00000
300000
400050
203003
100000
12
3.0
1.8
ROHE
1.0
1.0
0.
0.0
2500 5000 7500 10000 Time Steps (t)
2500 5000 7500 10000
2500 5000 7500 10000
2500 5000 7500 10000 Time Steps (t)
Time Steps (t)
Time Steps (t)
Minimum Intervention Time tr
20
50
100
500
Figure 12:
Simulation results with varying minimum intervention times. The figure shows simulation
results of the ASA policy for each task under the Always network contiguration over = 10, 000 time steps, with the number of robots Nav fixed at 100, the number of human supervisors Nauman fixed at 5, and the minimum intervention time fr varying.
26

---

Allegro Hand
Ball Balance
Humanoid
ANYmal
5000
1/50
250
Cumulative Success
260
110
2000
750
500
1000
10000
Cumulative
40000
40009
20000
30000
53000
20000
10000
10009
3.0
Cumulative
800000
100030
400000
0.4
200020
200000
100010
17.5
15.0
10
ROME
1.0
2500 Steps
2500 Steps (t)
2500 Steps
Steps (t)
Hard Reset Time
100
Figure 13: Simulation results with varying hard reset times. The figure shows the simulation results of the ASA policy for each task under the Always network configuration, with the number of robots Naabor fixed 100, the number human supervisors Nhaman fixed at and the hard reset time i varying.
27

---

Data Collection Field
Connection Setup
5G Network
Human Supervisor
Robot
5G
Figure 14: Data collection setup network. collected 5G network connectivity data from real- world robotics laboratory floor. An example of the floor plan is shown the left figure. In this floor plan, the red devices represent the locations of the robots the laboratory floor. Our data collection setup is shown in the right figure. For each location, established connection between human supervisor using 5G-enabled smartphone and rohot server through 5G base station and 5G modem. We have collected various network parameters including throughput, latency, and signal strength each location. We then processed this data to obtain the network connectivity information for the rohots in our experiments.
Throughput
Latency
Figure 15:
5G network performance metrics. This figure shows the key performance metrics the 5G network data collected from the real world. Here, on the left, we show the average throughput for each group of rohots. The throughput is normalized between () and 1, where 0 represents low throughput and represents high throughput. On the right, we show the average latency for each group of robots, where the latency is normalized between and 1, where represents low latency, and represents high latency. We then these metrics determine the probability of connection the robots our experiments.
28


Reviews:
[{"review_id": "1Pt0cc1Bbi", "reviewer": "Reviewer_hzFg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper extends the interactive fleet learning problem formulation proposed by Hoque et al. to model uncertainty over network connectivity between robots and their human supervisors. The paper presents a new algorithm for human-to-robot supervisor allocation called Adaptive Submodular Allocation (ASA) based on the theory of submodular maximization. Simulation results suggest ASA outperforms existing baselines in terms of cumulative successes and return on human effort in a range of network configurations.", "summary_of_recommendation": "Interesting extension of work on Robot Fleet Supervision.  The paper addresses an important and timely challenge for training and managing real-world robot fleets. To my knowledge, it is the first to study multi-robot interactive imitation learning with variable network connectivity. The proposed approach is intuitive and the use of submodularity is interesting.   the manuscript is written clearly. The empirical results are convincing, with evaluation over different realistic network configurations and a demonstrated advantage over existing state-of-the-art techniques.   However, to be accepted it is important to address some weaknesses.  Weak Accept:    Strong Accept if questions are addressed in rebuttal.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "Please respond to the points raised in the main review.  Is it possible to perform additional experiments?\n\nIn addition to the clarifications requested in the main review, why does N-ASA often outperform ASA in Table 1? Should ASA not be strictly better than N-ASA, given its additional observability?\n\nAlso, please expand the Limitations section with more details, including those above...", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qL9EBc0JiQ", "reviewer": "Reviewer_rf9j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work uses submodular maximization to solve the supervisor allocation problem under network connectivity constraints.", "summary_of_recommendation": "The work is incremental and has no robot experiments. Its formulation limits its use in mobile robotics, where connectivity concerns would be most relevant.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "Would it be possible to highlight the differences between ASA and n-ASA in Algorithm 1?\n\nPlease provide additional discussion of [46]. How is Algorithm 1 derived from existing work and how are the bounds derived at the end of section 4.2?\n\nPlease discuss the marginal threshold parameter $\\Delta$ and how might that be chosen in practice.\n\nI would like to see more discussion of Eq. 5 to explain the $U(i)$ and $S(j,i)$ functions and how they were selected for the experiments. \nSpecifically, Figure 4 shows that ASA and n-ASA outperform all other baselines when connectivity is stable, which is an interesting result. Can you provide more numerical detail on why these methods outperform the baselines?\n\nThe assumption that robots' connectivity probabilities $C$ are known a priori and are constant limits the applicability of this approach to mobile robotics. Please explain how this affects the relative performance of n-ASA vs. ASA. You may want to add this analysis to expand the Limitations section.\n\nA few other corrections:\n- Ensure that the acronym ASA is defined prior to its first use in the abstract, and capitalized in text. \n- Table 1 typo: N-ASSA and ASSA should be N-ASA and ASA", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TakzT3mRo7", "reviewer": "Reviewer_ZFVB", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper considers the problem of human supervision allocation in multi-robot settings with uncertain connectivity. The proposed method is based on the design of a submodular allocation function and its greedy optimization. Simulations are used to investigate the impact of connectivity quality across different domains and against different baselines.", "summary_of_recommendation": "This paper has potential but I believe this work does not get to the core of the problem.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Could you please elaborate on the notion of \"hard reset time\" which was set to 5 time steps? How could a smaller / larger value impact an experiment?\n2. For each intervention, how is the duration determined during an experiment (in addition to having a minimum time of 5 steps)? Why is the duration not a part of the problem statement / output of the allocation algorithm?\n3. During an experiment, what happens when the robot is not being supervised? Does it keep following its current policy? Is that deterministic or random? How is the data collected in those steps used?\n4. Did you consider integrating the temporal aspect in the allocation algorithm? If so, what difficulties did you encounter?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Addressing a timely and challenging problem in the field. 
- The paper is relatively well-written. 
- The proposed idea is interesting and seems to be technically sound, somehow supported by evaluations. 

Weaknesses:
- The extendability of problem fomulation needs to be better investigated. 
- More detailed analysis of the results would be beneficial. 
- The evaluation can be improved by comparing against more diverse types of baselines. 
- The contribution over Fleet-Dagger needs to be more clearly described.
Decision: Accept