Fleet Supervisor Allocation: A Submodular Maximization Approach

Authors: oguzhan  akcin
ahmet ege tanriverdi
kaan  kale
sandeep p chinchali
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:07:18

Abstract: In real-world scenarios, the data collected by robots in diverse and unpredictable environments is crucial for enhancing their perception and decision-making models. This data is predominantly collected under human supervision, particularly through imitation learning (IL), where robots learn complex tasks by observing human supervisors. However, the deployment of multiple robots and supervisors to accelerate the learning process often leads to data redundancy and inefficiencies, especially as the scale of robot fleets increases. Moreover, the reliance on teleoperation for supervision introduces additional challenges due to potential network connectivity issues. 
To address these issues in data collection, we introduce an Adaptive Submodular Allocation policy, ASA, designed for efficient human supervision allocation within multi-robot systems under uncertain connectivity conditions. Our approach reduces data redundancy by balancing the informativeness and diversity of data collection, and is capable of accommodating connectivity variances. We evaluate the effectiveness of ASA in simulations with 100 robots across four different environments and various network settings, including a real-world teleoperation scenario over a 5G network. We train and test our policy, ASA, and state-of-the-art policies utilizing NVIDIA's Isaac Gym. Our results show that ASA enhances the return on human effort by up to $3.37\times$, outperforming current baselines in all simulated scenarios and providing robustness against connectivity disruptions.

Main paper: 
 introduction: 
 Today, diverse industries deploy robotic fleets for tasks ranging from autonomous driving [1, 2] to healthcare [3] and package delivery [4] . These robots are often deployed with policies trained on a dataset that is primarily based on simulations, along with a small amount of data collected through real-world interactions. While effective within their training contexts, these models often fail to adapt to new or evolving real-world scenarios [5] , making data collection critical for the success of the robotics applications [6, 7] .A popular approach to collecting such data is through human supervision, where humans directly guide the robots to perform the tasks. These data are then used to train the robots via Imitation Learning (IL), where the robots learn to perform tasks by imitating the human demonstrations [8] . Imitation Learning (IL) has been effective in many robotics applications, ranging from autonomous driving [9] to robotic manipulation [10, 11] . However, the breadth of scenarios necessary for effective IL emphasizes the need for continual data collection [12] , commonly done with numerous robots in parallel. Usually, the number of humans is less than that of deployed robots. For instance, a recent autonomous delivery company, Starship Technologies, operates 1700 autonomous robots while teleoperating only 1% of this robotic fleet [13, 14] . The scarcity of human supervisors necessitates the selection of informative robots for supervision [15, 16] .Human supervision is often provided through real-time teleoperation over a network, especially when supervising fleets of robots distributed across the globe. For example, various companies, including Cruise, utilize human supervisors located in their control centers to teleoperate autonomous 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. At each time step t, the human supervisors with policy πH are allocated to the robots with policy π t R based on the allocation policy πA. Each robot i operates in a different state s t i , and the human supervisors are allocated to the robots based on the uncertainty of the robots and the similarity between the robots. Additionally, the connections to the robots may not be stable, which is indicated by the connection random variable, C t i , of each robot i. At the end of each time step t, the data collected by human supervisors D t H is added to the dataset D t to create an updated dataset D t+1 , which is then used to train the robot policy π t+1 R .vehicles deployed across the world [17, 18] . However, these networks might be susceptible to connection failures [19] , and it is important to be robust against network uncertainties. Combining these challenges with selecting informative robots, we formulate the Supervisor Allocation Problem ( Fig. 1 ), which involves managing the limited human resources to maximize data diversity and quality under uncertain network connectivity. Our problem is an extension of the Interactive Fleet Learning (IFL) setting introduced by [15] . We extend the IFL setting to account for network elements that play an important role in real-world teleoperation scenarios [20] .We then introduce a novel human supervisor allocation policy called Adaptive Submodular Allocation (ASA). ASA distributes human supervisory capacity across a fleet, ensuring a balance of data informativeness and diversity to minimize redundancy in data collection. Our allocation policy is shown to be robust against network instabilities and is able to adapt to the dynamic nature of data collection, which we demonstrate through extensive simulations in diverse network environments, including real-world 5G scenarios. We show that ASA improves the Return on Human Effort (RoHE) [15] , which is a human supervision efficacy metric, by up to 3.37× compared to existing benchmarks. 
related work: 
 Data collection is a critical problem in robotics and machine learning that is essential for continually improving the performance of robots [21] [22] [23] [24] [25] . It is closely related to active learning [26] [27] [28] [29] [30] , where the goal is to select the most informative samples to label. Although the goal of data collection is similar to active learning, the focus is on collecting data samples that are the most informative for training the models. In our case, however, the aim is to select the robots that provide the most informative data for human supervisors.IL is a popular approach in robotic learning, where robots learn policies from human demonstrations [31] [32] [33] [34] . Despite its potential, the reliance on purely offline data introduces several challenges, such as distribution shifts [35] , which occur when robots encounter states that were not previously experienced by humans. These issues can be alleviated through online data collection methods, such as Dataset Aggregation (DAgger) [35] and various forms of interactive IL [36, 37] . Most interactive IL methods rely on human supervision to decide when to intervene in the robot's learning process. This presents scalability challenges, especially when applied to extensive robot networks [38] or during prolonged learning phases [39] . Robot-initiated interactive IL strategies like EnsembleDAgger [40] and ThriftyDAgger [41] have been proposed to mitigate these constraints, enabling robots to request human input under specific conditions. However, these methods are designed for single-robot task allocation scenarios and do not consider multi-robot scenarios. Closest to our work, Fleet-DAgger [15] has been proposed to address the supervisor allocation problem in a multi-robot scenario. However, Fleet-DAgger does not consider operational constraints that might limit the allocation of human supervisors, such as network connectivity and the potential redundancy from employing multiple human supervisors in similar environments. Our work, on the other hand, focuses on learning an allocation policy that is adaptable to the operational constraints while minimizing the redundancy in the data collection, which is crucial for the system's scalability [27] .One popular approach to mitigate redundancy in data collection is using submodular maximization. Submodularity refers to the property of the marginal gain of adding an item to a small set being higher than adding the same item to a large set. As submodularity is a common trend in data collection, it has been widely used in machine learning tasks, such as sensor placement [42] , active learning [27, 30, 43] , and summarization [44] . Submodular maximization has also been extended to stochastic settings [45, 46] , where the goal is to select a subset of items to maximize the expected value of a submodular function. Despite its wide use in machine learning, stochastic submodular maximization has not been used in the context of IL and multi-robot data collection scenarios. Our work is the first to use stochastic submodular maximization in the context of human supervision and multi-robot scenarios to address the supervisor allocation problem. 
problem formulation: 
 Consider a geo-distributed system of N robot robots, I = {1, • • • , N robot }. Each robot i operates in parallel within an independent Markov Decision Process (MDP) with a different initial state. However, all robots operate within the same state and action spaces S and A, respectively. Each robot i observes the state of the environment s t i ∈ S at time t and selects an action a t i ∈ A based on a policy π t R : S → A. The robots share the same policy π t R that has been trained using the collective data D t accumulated up to time step t. We define the collection of states and actions for all robots as s t = (s t 1 , . . . , s t Nrobot ) ∈ S Nrobot and a t = (a t 1 , . . . , a t Nrobot ) ∈ A Nrobot . These robots can be supervised by N human human supervisors with an oracle policy π H : S → A H , respectively. In addition to the robot action space A, the human action space A H includes a reset action, which can return the robot to a safe state. Only robots supervised by humans operate with human policy π H while the rest of the robots operate with the robot policy π t R . Supervisor Allocation and Connectivity: In each time step t, N human human supervisors can be assigned to the robots for assistance. However, the connections to the robots are unreliable, withC t = {C t 1 , • • • , C t Nrobot } ∈ {0, 1}Nrobot denoting independent random variables associated with the connection reliability of the robots. C t i ∈ {0, 1} indicates whether a successful connection with robot i can be established (C t i = 1) or not (C t i = 0) at time t. Under this uncertain connectivity, we are interested in finding an allocation policy π A : {0, 1} Nrobot × S Nrobot × A Nrobot × I → 2 I that selects robots to be supervised X ⊆ I based on connection reliability C t , collection of states s t and actions a t . Our setting extends the Interactive Fleet Learning setting [15] by incorporating imperfect time-varying network connectivity.Data Collection and Policy Retraining: Upon allocation, human supervisors contribute data only from successful connections, forming the human supervision data D t H . This new data is integrated into the current dataset and the robot policy π t R is retrained using a retraining function g:EQUATIONEQUATIONObjective: Our objective is to develop an allocation policy π A that maximizes the expected Return on Human Effort (RoHE) over the connectivity C t . RoHE metric was introduced along with Interactive Fleet Learning setup [15] to set a benchmark in Fleet Learning settings. It is a ratio of the total reward obtained by the fleet to the total number of human actions. Formally, the objective is to maximize the expected RoHE over the connection probabilities: EQUATION 
a stochastic submodular maximization approach: 
 We now present our novel policy, adaptive submodular allocation (ASA), for the problem outlined in Eq. 3. First, we define the stochastic submodular maximization problem, which represents the value of robot supervision, and then we define the greedy algorithm, which is used to pick the robots to supervise.To address the optimization problem presented in Eq. 3, we use stochastic submodular maximization. Stochastic submodular maximization is particularly suited to our scenario because it leverages the diminishing returns property that naturally reflects the decrease in the marginal gain of supervising additional robots. Furthermore, the method inherently discourages the selection of similar robots, thereby avoiding the assignment of humans to robots that offer overlapping information, which decreases the return on human effort. Finally, stochastic submodular maximization accounts for the inherent network uncertainty in our problem by acknowledging the non-deterministic connections to the robots. This is crucial for developing a robust solution across different connectivity patterns.We first define a submodular objective function f C t : 2 Nrobot → R that quantifies the value of supervising a selected set of robots X, considering the allocation reliability outcomes for these robots C t . We define our objective function based on the facility location problem, a classic example of a submodular maximization objective [42] , as follows:EQUATIONHere, X is the set of robots selected for human supervision, and C t j indicates whether the connection to the robot j is successful or not. M t j,i represents the value of supervision of the robot j on the robot i at time t, and we consider two factors: the informativeness of the robot i and the similarity between the robot j and the robot i. Additionally, our formulation is modular and can be extended to include other factors, such as prioritizing the robots that have violated the safety constraints or those in critical states. With all factors combined, we define the value of supervision M t j,i as:EQUATIONHere, S t (j, i) defines the similarity between the robots i and j at time t, and U t (i) is the informativeness of the robot i, while K t (i) is an indicator of whether the robot i violates the safety constraints or is in a critical state. Our definition of M t j,i is modular, and each factor can be defined based on specific requirements. For example, the similarity function S t can be defined as the cosine similarity, the Euclidean distance, or any other similarity metric. The informativeness of the robot U t (i) can be defined as the entropy of the robot's policy or the uncertainty of the robot's state, while the constraint function K t (i) can be defined based on the safety constraints or the critical states for the robots, similar to previous works [15, 47] . For the exact definitions of similarity, informativeness, and constraint functions, please refer to the Appendix. With the objective function defined, we pose the following maximization problem to optimize our allocation policy:EQUATIONsubject to:|X| ≤ N human ,where the goal is to identify the subset of robots X that maximizes the expected value off C t , denoted as F (X) = E C t [f C t (X)] over all possible connection outcomes C t . The constraint |X| ≤ N human ensures that the number of selected robots is limited by the number of available human supervisors N human .Now, we can present our allocation policy ASA based on a greedy algorithm given in Algorithm 1.Starting from an empty solution set X (line 1), ASA iteratively selects the robot with the highest marginal gain based on the estimated expected value function F (line 3), which estimates the expected value of the submodular objective function f . Then, ASA computes the expected marginal gain of selecting the robot x * (line 4), and if the expected marginal gain is below a certain threshold, the algorithm stops the selection process (line 5). This threshold ensures that the algorithm avoids using unnecessary human effort by stopping when the marginal gain of selecting an additional robot is low. Otherwise, the chosen robot x * is added to the solution set X (line 8). Finally, if there is an observation on whether the connection to the robot was successful or not, the estimated expected value function F is updated (line 9).The estimated expected value function F is an important factor in our allocation policy and can be defined as:EQUATIONHere, the function f ξ denotes the value of supervising the set of robots X for a specific realization ξ of the connection random variable C t . pξ represents the estimated connection probability of the realization ξ, whereas pξi denotes the estimated connection probability for the robot i within the realization ξ. Initially, pξi values are set to the actual initial connection probabilities pξi ← P (ξ i = C 0 i ) for all robots i. Then, if an observation on the success of supervisor allocation is made, the connection probability estimate pξi is updated for the selected robot i: pξi ← P (ξ i = C t i ).Algorithm 1 ASA Policy Input: Estimated expected value of submodular objective function F , set of all robots I Output: robots selected for supervision X1: Initialize X ← ∅ 2: for k = 1 to N human do 3: x * ← argmax x∈I\X F (X ∪ {x}) 4: Compute ∆ ← F (X ∪ {x * }) -F (X)) 5: if ∆ < threshold then 6: break 7: end if 8: X ← X ∪ x * 9:if ASA then 10:Observe whether connection to robot x * is successful and update F 11:end if 12: end for Another important factor in the allocation policy is the marginal threshold parameter, which determines the trade-off between the cost of additional supervision and the incremental benefit derived from including an additional robot in the supervision set. In practice, this threshold can be set to 0 if all human supervisors want to be allocated to maximize the amount of data collected. Then, the threshold can be gradually increased to optimize the efficiency of human supervision.Based on the availability of the observations of the connection probabilities, we define two variants of our policy: non-Adaptive Submodular Allocation (n-ASA) and Adaptive Submodular Allocation (ASA). In n-ASA, we are not able to observe the connection probabilities, and thus, the allocations are done beforehand. In ASA, on the other hand, the robots are selected iteratively based on the success of the allocations, and the connection probability estimates pξ are updated. To visualize the differences between n-ASA and ASA, consider the following: in both cases, robot 1 is selected for supervision in the first iteration. While selecting the robot to supervise, n-ASA considers both possibilities (successful and unsuccessful connection to robot 1) and selects the second robot that maximizes the expected marginal gain over both cases. However, in ASA, after selecting robot 1 for supervision, we observe whether the connection was successful or not and select the next robot that maximizes the expected marginal gain based on the observation. We then update the connection probability estimate of the robot 1 with its actual probability: pξ1 ← P (ξ 1 = C t 1 ). When the marginal threshold parameter is set to zero and the estimated expected value function F is equal to the actual expected value of supervision F , ASA is equivalent to the greedy algorithm for submodular maximization, which is proven to approximate the optimal solution for the submodular maximization problem in Eq. 6 with a factor of 1 -1/e [45] . Additionally, n-ASA approximates the optimal adaptive policy with a factor of (1 -1/e) 2 [45] . To compute selected robots faster, we use the lazy greedy algorithm [46] ; this has the same time complexity as Algorithm 1 but has a better empirical performance. 
experiments: 
 We consider a fleet of N robot = 100 robots that can be supervised by N human = 5 human supervisors. The human supervisors are implemented as reinforcement learning agents using the Proximal Policy Optimization (PPO) algorithm [48] . We utilize the behavior cloning algorithm to initialize the robot policies based on an offline dataset of 5000 state-action pairs and use our allocation policy to collect data from the robots and update the models. In all of our experiments, when the robots violate the constraints, we perform a hard reset to bring the robots back to a safe state. We set the hard reset time to t R = 5 timesteps, the minimum intervention time to t T = 5 timesteps, and the fleet operation time to T = 10, 000 timesteps. We average the results over 3 random seeds for each task and network configuration. We have chosen these parameters to align with the settings used in the environments of the benchmark algorithms [15] for a fair comparison.Environments: We consider four different environments in our experiments. First is Humanoid, where the robots focus on bipedal locomotion. Second is ANYmal, which involves quadruped locomotion using the ANYmal robot. Third is Allegro Hand, focusing on dexterous manipulation tasks. The last environment is Ball Balance and it requires the robots to balance a ball on a plate.We use five different network configurations based on various connectivity probabilities. Always allows robots to always be supervised by human supervisors. In Mixed-Scarce, some robots have a high connection probability, while others have a low probability. Ookla features varying connection probabilities according to cellular network performance metrics [49] . 5G uses real-world connectivity data, including latency and throughput, collected from a university robotics lab (further details in the Appendix). Finally, Changing-Scarce starts with the same connection probabilities as Mixed-Scarce but evolves over time. It is important to note that while the first four configurations maintain stationary network connectivity, the Changing-Scarce configuration introduces dynamic network connectivity that changes over time.We evaluate the performance of the allocation policies based on the following metrics:(1) Return on Human Effort (RoHE), which is given in Eq. 3 and (2) the cumulative number of successfully completed tasks by the entire fleet, which we will refer to as cumulative success. To simplify, RoHE measures the fleet performance per human intervention, while cumulative success only considers the total successful task completion without considering the number of interventions. For example, simply allocating all human supervisors would improve cumulative success but decrease RoHE. An ideal allocation policy should balance the two, as an ideal system would require a high total success while using humans as efficiently as possible.Allocation Policies: We compare our proposed Adaptive Submodular Allocation (ASA) and non-Adaptive Submodular Allocation (n-ASA) policies with several baselines. The ASA policy uses the observed connection information to update the network connection probabilities, while the n-ASA policy does not use this information. The Random baseline simply selects robots for supervision randomly. Fleet-EnsembleDAgger (FE) uses variance for uncertainty estimation, combined with constraint-based prioritization [40] . Fleet-ThriftyDAgger (FT) integrates uncertainty and goal-oriented prioritization, adapting ThriftyDAgger for fleet settings in goal-oriented environments [41] . Fleet-DAgger (FD) prioritizes robots violating constraints and selects those with the highest uncertainty and risk of failure [15] . Lastly, n-Fleet-DAgger (n-FD) adapts Fleet-DAgger to network uncertainties by filtering out robots with low connection probabilities.How do ASA and n-ASA perform under different network configurations?We evaluate the performance of our policies, ASA and n-ASA, under different network configurations for each environment. The RoHE metric for each time step has been shown in Fig. 3 How do ASA and n-ASA compare when the network connectivity is stable?To test whether our RoHE gains are only due to adaptability to different network configurations, we have also simulated a network where all robots are always reachable. In row 1 of Fig. 3 , we can see that our ASA and n-ASA policies still outperform other allocation benchmarks due to their ability to diversify the selected robots to cover more states. As we have shown in the 2D toy example in Fig. 2 , rather than only focusing on the states with high uncertainty, ASA and n-ASA consider the whole state space to collect combined data that is more informative.How do ASA and n-ASA compare to benchmarks with time-varying connection probabilities?To test the adaptability of our allocation policies to network connectivities varying over time, we have run experiments on the Changing-Scarce network scenario. We can see in row 5 of Fig. 3 that the ASA policy consistently outperforms n-ASA, n-FD, and FD, due to its ability to dynamically update the network connection estimates based on the observations on whether the connection to each robot is successful or not. This adaptability is particularly important in scenarios where network conditions are unpredictable and can change over time, such as mobile robotics applications.Limitations: Our work has several limitations. First, it uses only real-world data collected from 5G network without hardware robotics experiments and is tested only in NVIDIA's Isaac Gym environment like previous benchmarks. Additionally, it assumes that network connectivity and robot states and policies are independent across robots, while in real-world scenarios, robots might share the same network or physical location, meaning their policies might affect each other. 
conclusion and future work: 
 We present two novel supervisor allocation policies, ASA and n-ASA, for assigning human supervisors to a robotic fleet for data collection. These policies, based on stochastic submodular maximization, offer a modular approach for incorporating diverse allocation objectives, informativeness metrics, and retraining methods. Our allocation policies beat current benchmarks in RoHE and cumulative success metrics across most environments and network configurations. These performance gains are thanks to their stochastic submodular maximization objective, which incorporates network connectivity while balancing the diversity and informativeness of selected robots. Finally, we collect real-world 5G network data from a field dedicated to teleoperated robots and show the applicability of our allocation policy in real-world scenarios as well.In a future project, we plan to extend our work to include hardware robotics experiments and other simulation environments, including teleoperation over a 5G network, which would be possible in an application such as autonomous driving. We also plan to investigate the impact of different imitation learning methods to test the generalizability of our allocation policies. 
 
 Appendix: Code Availability: The code and related materials can be found in the following code repository:https://github.com/UTAustin-SwarmLab/Fleet-Supervisor-Allocation.gitThe organization of the appendix is as follows:1. Subsection A.1 describes the details of the submodular maximization function we have used in our experiments.2. Subsection A.2 describes the details of the theoretical bounds of our allocation algorithm.3. Subsection A.3 presents the environment setups and exact parameters we have used for our simulations.4. Subsection A.4 provides the exact details for the network configurations used in our experiments.5. Subsection A.5 presents all the simulation results for all tasks and network configurations as well as additional metrics providing an insight into why our methods outperform other benchmarks.6. Subsection A.6 provides ablation studies over the hyperparameters and simulation parameters.7. Subsection A.7 presents the complexity analysis of our allocation policies.8. Subsection A.8 presents our 5G network data collection setup.This section provides the exact parameters and functions used in the submodular maximization method given in Section 4.The informativeness function U t measures the informativeness of each robot i in the fleet at time t. To measure the informativeness of the robots, we use a weighted combination of two functions. The first function measures the uncertainty of the robot policy π t R in the environment, and the second function measures the risk of the robot i violating the constraints K. Then, the informativeness function can be defined as follows:EQUATIONHere, αU is a hyperparameter that controls the weight of the uncertainty in the overall informativeness measure, and U t unc (i) and U t risk (i) are the uncertainty and risk functions of the robot i at time t, respectively. When the robot is taking discrete actions, the uncertainty function U t unc (i) is defined as the entropy of the robot policy π t R , and when the robot is taking continuous actions, the uncertainty function is defined as the ensemble variance of the robot policy π t R [40] . The risk function U t risk (i) is defined as the likelihood of the robot i exiting the constraint space K [15] .For both uncertainty and risk functions, if the value of the function for the robot i is below a certain threshold, we set it to zero. We define this threshold parameter as U thres unc and U thres risk for uncertainty and risk functions, respectively. We present specific U thres unc and U thres risk parameters for the experiments in Table 2 .The similarity function S t measures the similarity between two robots i and j in the fleet at time t. In our experiments, we utilize both the similarity between the states and the similarity between the actions taken by the robots. More formally, the similarity function is defined as follows:EQUATIONwhere s t i and s t j are the states of the robots i and j, respectively; a t i and a t j are the actions taken by the robots i and j, respectively, at time t. αS is a hyperparameter that controls the weight of the state similarity in the overall similarity measure.The constraint violation function K t (i) measures whether robot i violates the constraints at time t. In our experiments, we used the constraint violation as an indicator function that returns αK if the constraint is violated and 0 otherwise. The αK is a parameter that controls the relative importance of the constraint violation in the overall objective function. In our experiments, we set αK = 10000 to prioritize the robots with constraint violations. The constraint violation function is defined as follows:EQUATION)Here, the K refers to the safe states that the robot can operate without any human intervention. As the constraint function causes the system to prioritize the robots violating the constraints, in initial time steps, we set the αK = -10000 to ensure that the robots violating the constraints are not prioritized. This is because, in the initial steps, the robots explore the environment, and collecting more informative data is more critical than the constraint violations. We control the length of this period in which the constraint-violating robots are not prioritized by the tW parameter.Here, we provide the definitions and necessary theorems to show the optimality bound of our proposed ASA and n-ASA policies. We first define submodular and monotone functions and then provide the necessary theorems to show the optimality bounds of our proposed policies.Definition 1 (Submodular Function). A set function f : 2 V → R is submodular for all A ⊆ B ⊆ V and e ∈ V \ B, we have:f (A ∪ {e}) -f (A) ≥ f (B ∪ {e}) -f (B).Definition 2 (Monotone Function). A set function f : 2 V → R is monotone if for all A ⊆ B ⊆ V , we have:f (A) ≤ f (B).Definition 3 (Facility Location Problem). The facility location problem is a combinatorial optimization problem where the goal is to select a subset of facilities from a set of candidate locations to minimize the cost of serving the demand points. The objective function of the facility location problem is defined as follows:f (A) = n i=1 min j∈A Mij,where A is the set of selected facilities, Mij is the cost of serving demand point i from facility j, and n is the number of demand points.Corollary 1 (Facility Location Problem is submodular for non-negative Mij.). The facility location problem is submodular and monotone if the cost matrix Mij is non-negative. Please refer to [50] for the detailed proof.Exactly following that corollary, we can show that the facility location problem presented in Equation 4is submodular and monotone, as the cost matrix Mij is non-negative and C is a binary random variable.Definition 4 (Greedy Policy). The greedy policy is a sequential optimization method that selects the element with the maximum marginal gain at each step. The greedy policy that selects k elements is defined in Algorithm 2.1: A ← ∅ 2: for i = 1 to k do 3:x i ← argmax x∈V \A f (A ∪ {x}) -f (A) 4:A ← A ∪ {x i } 5: end for 6: return A Theorem 1 (Adaptive Greedy Policy Optimality Bound). Consider the adaptive greedy policy selecting elements with the maximum marginal value, conditioned on the realized value of the previously chosen elements. The approximation ratio of the adaptive greedy policy with respect to the optimal adaptive policy is 1 -1 e . For the detailed proof of this theorem, please refer to [45] .In our work, once we are able to observe whether the connectivity to robots is established or not (Alg. 1, line 8), our policy becomes adaptive. Additionally, when the F and F functions match, and the marginal threshold parameter is set to 0, Theorem 1 can be applied to show the optimality bound of our proposed policies.Theorem 2 (Non-Adaptive Greedy Policy Optimality Bound ). Consider the non-adaptive greedy policy that, at each step, chooses the element with the maximum marginal increase in value. The approximation ratio of this policy with respect to the optimal adaptive policy is at least (1 -1 e ) 2 . For the detailed proof of this theorem, please refer to [45] .Unlike ASA policy, our n-ASA policy is not able to observe whether a connection to robot i has been established or not, so it is non-adaptive. Under the same conditions as the ASA policy, when the F and F functions match, and the marginal threshold parameter is set to 0, Theorem 2 can be applied to show the optimality bound of our proposed n-ASA policy.As stated previously, we run simulations using four different environments: ANYmal, Allegro Hand, Humanoid, and Ball Balance. Each environment has its own defined tasks, success criteria, and constraint violations. For the ANYmal robot, a constraint violation occurs when there is excessive force on the robot's knees, indicating that the robot has fallen on its knees, or when no force is exerted on the bottom of its toes, indicating that the robot has fallen on its torso. For the Ball Balance environment, a constraint violation occurs when the ball is no longer on the plate. In the Allegro Hand environment, a constraint violation happens when the cube is no longer in the robot's hand. For the Humanoid environment, a constraint violation occurs when the robot's position is below the termination height, indicating that the Humanoid has fallen down.The definition of success is specific to each task. For instance, in locomotion tasks, success is achieved if the robot does not violate constraints and reaches a reward amount that exceeds a predefined reward threshold. For goal-specific tasks such as Ball Balance and Allegro Hand, success corresponds to reaching the goal state without violating constraints. For Ball Balance, a goal state may be one where the ball on the plate is moving within a radius smaller than the plate's radius, indicating that the robot successfully managed to control and balance the ball. For Allegro Hand, the goal state may be defined as holding the cube stable after rotating it so that the red surface faces up. That is how a single success corresponds to different achievements depending on the specific tasks assigned to each robot.For all experiments, the key parameters are fixed and do not depend on the allocation policies: Nhuman = 5, Nrobot = 100, T = 10, 000 time steps, tR = 5 time steps and tT = 5 time steps. The hyperparameters that vary depending on the task, along with the values that yielded the best performances, are provided below in Table 2 . |S| and |A| are the dimensionalities of the state and action spaces, respectively, U thres unc and U thres risk are the uncertainty and risk threshold values below which the uncertainty and risk are treated as zero, tW is the period during which constraints are not prioritized, allowing the robot policies to be improved by selecting informative robots rather than resetting failing robots in the first tW time steps, threshold is the marginal increase threshold below which the robots are not prioritized, αS is the parameter which controls the weight of the state similarity in the overall similarity measure, and αU is the parameter that controls the weight of the uncertainty in the overall informativeness measure. Here, we explain the details of the network configurations used in our experiments. We have used four different network configurations to evaluate the adaptability of the allocation algorithms in different network conditions. Additionally, we show the connection probabilities in each network configuration in Figure 4 . The network configurations are as follows:Always: Always is a simple network configuration where the probability of connection to all the robots is set to 1. In this network configuration, our supervisor allocation problem is equivalent to the Interactive Fleet Learning (IFL) problem presented in [15] , where the supervisor can connect to all the robots at all times.Mixed-Scarce: Mixed-Scarce is a network configuration where the probability of connection to robots can be set to two different values. In this network configuration, we first divided the robots into two groups with ratios of 0.7 and 0.3. We then set the probability of connection to the robots in the first group to 0.9 and the probability of connection to the robots in the second group to 0.1. This network configuration is used to evaluate the adaptability of allocation algorithms when the connectivity to the robots is heterogeneous. Ideally, the supervisor should allocate more resources to the robots with higher connectivity to maximize the performance of the fleet.Ookla: Ookla is a network configuration where the probability of connection to the robots is set based on the Ookla cellular network performance data [49] . This dataset includes the download speed, upload speed, and latency of the cellular network in different locations. We use the download speed as the metric to determine the probability of connection to the robots. We first divided the data collection points into a grid of 10 × 10 cells. We then calculated the average download speed of the data collection points in each cell. After that, we log-normalize the average download speed of each cell to be in the range of [0.5, 1]. We have set the lower bound to 0.5 to ensure that the robots in the cell with the lowest download speed have a non-zero probability of connection. We then set the probability of connection to the robots in each cell to be the normalized average download speed of the cell. This network configuration is used to evaluate the adaptability of allocation algorithms when the connectivity to the robots is based on real-world cellular network performance data, which is heterogeneous and has a more complex structure than the Mixed-Scarce network configuration.5G: 5G is a network configuration where the probability of connection to the robots is set based on the realworld 5G network performance data. Please refer to Section A.8 for more details on the data collection process. The collected data was divided into 100 groups, with average latency and throughput calculated for each group and normalized to a value between 0.015 and 1. A lower bound of 0.015 ensures a non-zero connection probability for robots with the lowest throughput and highest latency. Robots in groups with throughput below 0.4 and latency above 0.6 were assigned a normalized value of 0.015. The connection probability for each group corresponds to the normalized average throughput and latency. This configuration evaluates the adaptability of allocation algorithms to realistic, heterogeneous connectivity based on real-world 5G network performance, which is more complex than other network configurations.Changing-Scarce: Changing-Scarce is a network configuration where the probability of connection to robots is first set as in the Mixed-Scarce network configuration. Then, over time, the probability of connection to the robots in the first group is decreased from 0.9 to 0.1, and the probability of connection to the robots in the second group is increased from 0.1 to 0.9 linearly. This network configuration is used to evaluate the adaptability of allocation algorithms when the connectivity to the robots changes over time. Ideally, the supervisor should adapt the allocation of human supervisors to the robots based on the changes in connectivity to maximize the performance of the fleet.In this section, we present the numerical values for all allocation policies and for all tasks under each network configuration to demonstrate that our method outperforms the baseline algorithms in all simulated scenarios, providing a novel approach to the supervisor allocation problem. We also present the percentage performance differences between our methods (ASA and n-ASA) and other methods. We present all numerical results recorded in the final timestep (t = 10, 000) in Table 3 and the percentage differences in Figure 5 .In addition to the presented metrics in the main paper, we also provide the following metrics to evaluate the performance of our method against the baselines: (1) cumulative human actions, which measures the total number of time-steps humans supervised the robots, (2) cumulative idle time, which is the total duration that robots remain in the constraint violating states while awaiting hard resets, (3) cumulative hard resets, which records the total number of hard resets performed by the human supervisors, (4) cumulative reward, which is the total reward accumulated by the all robots in the fleet, and (5) cumulative success, which is the total number of tasks successfully completed by the entire fleet. We present the these metrics in Figures 6, 7, 8, 9, and 10 respectively.In Figures 6, 7 and 8, we can observe that the ASA and n-ASA methods use fewer human actions and are able to reduce idle time and hard resets compared to the other benchmarks. This is due to the fact that our method prioritizes the robots with higher connectivity to reset and supervise, wasting less human resources on robots with low connectivity, unlike the other method possibly trying to reset the robots with low connectivity. Additionally, in Figures 9 and 10 , we can see that our ASA and n-ASA methods are among the top-performing methods in terms of cumulative reward and cumulative success in all network configurations and tasks, proving that our method is much more efficient in terms of human resource utilization. In this section, we conduct further experiments using our adaptive submodular allocation, ASA, policy to explore the following: (1) the sensitivity of the system to the ratio of the number of robots Nrobot to the number of humans Nhuman (Figure 11 ), (2) the impact of varying the minimum intervention time tT (Figure 12 ), and(3) the impact of changing the hard reset time tR (Figure 13 ). Each experiment is averaged over three different random seeds, and the shaded regions correspond to one standard deviation. We plotted four different metrics:(1) cumulative success, (2) RoHE, (3) cumulative hard resets, and (4) cumulative idle time. Cumulative hard resets represent the total number of hard resets performed by human supervisors when the robots violate constraints. Cumulative idle time is the total time, in time steps, that robots remain idle while waiting for a hard reset.We tested the ASA policy to evaluate its sensitivity to different numbers of human supervisors (Fig. 11 ). Keeping the number of robots constant, we simulated scenarios with 1, 5, 10, 25, and 50 human supervisors. In all simulated tasks, as the number of human supervisors increases, idle time decreases because more human resources are available, resulting in shorter idle periods before robots are teleoperated and reset. However, despite the cumulative success values rising with more supervisors, the RoHE values tend to decrease. This happens because allocating more humans doesn't always lead to a higher return on human effort. The most informative and important robots are already being selected, so adding more supervisors doesn't necessarily result in a significant marginal gain. Therefore, a low number of human supervisors is insufficient as robots remain idle for long periods and violate constraints more frequently, while a large number of supervisors creates a surplus and decreases efficiency.Minimum Intervention Time: While keeping other hyperparameters fixed, we varied the minimum intervention time and ran our policy. We observed that when the minimum intervention time is long, such as 100 or 500 time steps, the robot fleet performance significantly decreases. This is because human supervisors spend a lot of time teleoperating a single robot, which results in lower RoHE and cumulative success values, and a substantial increase in idle time. Conversely, when the minimum intervention time is very short, such as 1 time step, performance improves in terms of both RoHE and cumulative success for most tasks. This is because each human supervisor spends less time on a single robot and can attend to more robots within 10,000 time steps, thus enhancing overall fleet performance as the minimum intervention time decreases. Hard Reset Time: Finally, we ran the ASA policy with different hard reset times. We observed that as the hard reset time increases, the fleet performance decreases. This is because it takes longer for human supervisors to reset the robots, resulting in fewer hard resets within 10,000 time steps. Consequently, the idle time increases, reducing the overall performance of the robot fleet.Here, we present the complexity analysis and optimality bounds for our allocation algorithms.We now explain the complexity of our allocation algorithms in terms of the number of robots Nrobot and the number of humans Nhuman in the system and function evaluations. As we have discussed in Section 4, our allocation algorithm is based on a greedy algorithm that selects the robots based on the stochastic submodular maximization objective. It is a well-known result that the number of function evaluations for the greedy algorithm is O(NrobotNhuman). As both of our algorithms are based on the greedy algorithm, the computational and time complexities of our algorithms, ASA and n-ASA, are both O(NrobotNhuman).In addition to the simulated network connectivity data, we also evaluate our allocation policies using realworld 5G network connectivity data collected via setup detailed in [51] . This setup includes two hardware components: a mobile edge device and a local server. The edge device, which can be a robot, a mobile phone, or a computer, acts as the connection client. The local server functions as the cloud. In our scenario, we consider the edge device to be the robot and the local server to be the cloud or the server from which human supervisors connect to the robots. The local server sends packets to the edge device, and the edge device responds with packets to confirm receipt. During this process, the local server calculates latency and throughput, saving this data to a local file. This continues for a predetermined data collection period of 24 hours. Two key aspects of this setup are: (1) the edge device is connected to a 5G cellular network, specifically 5G cellular provided by AT&T, and (2) it is mobile. This allows us to collect data anywhere, whether moving or stationary, for any desired period. To obtain data that realistically simulates human teleoperation connectivity, we collected data in a building where actual teleoperation and robotic tasks are conducted. After collecting the data, we divided and clustered it into 100 different groups. This division helps correlate the data with our fleet learning simulation environment, which has 100 robots in different locations. For each group, we calculated the average latency and throughput. We normalized the average values between 0 and 1, such that groups with high latency and low throughput values have a connection probability closer to 0, and groups with low latency and high throughput values have a connection probability closer to 1. Now that we have 100 different connection probability values, we randomly assigned them to 100 simulated robots. We illustrate the data collection setup in Figure 14 . Additionally, we present the average throughput and latency for each group in Figure 15 .  


Decision: Strengths:
- Addressing a timely and challenging problem in the field. 
- The paper is relatively well-written. 
- The proposed idea is interesting and seems to be technically sound, somehow supported by evaluations. 

Weaknesses:
- The extendability of problem fomulation needs to be better investigated. 
- More detailed analysis of the results would be beneficial. 
- The evaluation can be improved by comparing against more diverse types of baselines. 
- The contribution over Fleet-Dagger needs to be more clearly described.

Reviews: review: title: title: Interesting extension of work on Robot Fleet Supervision.  Weak Accept:    Strong Accept if questions are addressed in rebuttal.
paper_summary: paper_summary: This paper extends the interactive fleet learning problem formulation proposed by Hoque et al. to model uncertainty over network connectivity between robots and their human supervisors. The paper presents a new algorithm for human-to-robot supervisor allocation called Adaptive Submodular Allocation (ASA) based on the theory of submodular maximization. Simulation results suggest ASA outperforms existing baselines in terms of cumulative successes and return on human effort in a range of network configurations.
main_review: main_review: The paper addresses an important and timely challenge for training and managing real-world robot fleets. To my knowledge, it is the first to study multi-robot interactive imitation learning with variable network connectivity. The proposed approach is intuitive and the use of submodularity is interesting.   the manuscript is written clearly. The empirical results are convincing, with evaluation over different realistic network configurations and a demonstrated advantage over existing state-of-the-art techniques. 

The detailed Appendix is very helpful!

However, to be accepted it is important to address some weaknesses.  

First, the evaluation is limited. All the simulation results are limited to NVIDIA Isaac Gym environments. Figure 3, Figure 4, and Table 1 all show results from the same set of experiments rather than different ones. As such, it is unclear whether the authors’ findings extend beyond Isaac Gym to other domains or physical robots.  Also, As the authors note, while there are simulation experiments with real-world datasets, there is no evaluation with physical hardware.   The paper would be greatly improved with simulation in another environment and ideally on real hardware although the latter is not required.

Second, the  problem formulation is very similar to that of the Fleet-DAgger work it extends, the only addition being connection probabilities. The contributions and new novel aspects with respect to the prior work should be explicitly mentioned in the text. Also, the connection probabilities appear to be assigned statically to each individual robot; should they not vary based on the current location of each robot or otherwise depend on the state? Some clarification is needed here. Moreover, the value of supervision provided in Equation 5 is said to allow modular definitions for each component, but the text does not specify how these are defined for the ASA policy. Without clarification, these components look to be similar to the baselines: U(i) and C(i) appear to be the uncertainty and constraint violation metrics from the Fleet-DAgger work, and it is unclear if the proposed S(i) similarity metrics such as Euclidean distance can scale to image-based policies.  The pape would be improved if these can be addressed in the rebuttal.

Third, the previous Fleet-DAgger baselines appear to be evaluated without modification in the variable network connectivity setting of this work. These baselines should not be expected to perform as well since they have no knowledge of the robot connectivity. A more appropriate choice of baseline would be a heuristic that adapts the Fleet-DAgger algorithms to this setting: for instance, one that thresholds on the connection probabilities before allocating supervisors with Fleet-DAgger. The paper could be improved with additional experiment, especially since the performance benefit of ASA over Fleet-DAgger is small for the fully connected network in Figure 3.
questions: questions: Please respond to the points raised in the main review.  Is it possible to perform additional experiments?

In addition to the clarifications requested in the main review, why does N-ASA often outperform ASA in Table 1? Should ASA not be strictly better than N-ASA, given its additional observability?

Also, please expand the Limitations section with more details, including those above...
limitations: limitations: 2
review_summary: review_summary: Interesting extension of work on Robot Fleet Supervision.  The paper addresses an important and timely challenge for training and managing real-world robot fleets. To my knowledge, it is the first to study multi-robot interactive imitation learning with variable network connectivity. The proposed approach is intuitive and the use of submodularity is interesting.   the manuscript is written clearly. The empirical results are convincing, with evaluation over different realistic network configurations and a demonstrated advantage over existing state-of-the-art techniques.   However, to be accepted it is important to address some weaknesses.  Weak Accept:    Strong Accept if questions are addressed in rebuttal.
title: title: Submodular Maximization for the Supervisor Allocation Problem
paper_summary: paper_summary: This work uses submodular maximization to solve the supervisor allocation problem under network connectivity constraints.
main_review: main_review: This work builds on the formulation and algorithm of the FleetDAgger solution to the robot supervisor allocation problem. 
The supervisor allocation problem is extended to include variable connectivity between supervisors and robots, with each robot having a fixed probability of a failed connection. The authors then introduce a modular objective for the value of supervision for each robot. 
This formulation is the most interesting and original portion of the paper, and it captures the similarity across robots, and safety and informativeness of each robot. The use of the greedy solution in Algorithm 1 follows directly from the use of the modular objective in the maximization problem. The only difference from standard submodular maximizaiton is the marginal threshold parameter. 

The experiments are thorough and interesting. Real 5G data was collected from robots in the lab to improve the realism of the simulated experiments. Also, the authors implement several Fleet versions of multiple state-of-the-art baselines, and show their algorithm's improvement over these baselines. The experimental results are all in simulation, but are very thorough and use multiple simulated robotic domains. There is a lack of discussion about why the adaptive ASA approach vs. n-ASA doesn't make a big difference. My opinion is that it has to do with the formulation of the connectivity problem having constant probability of dropped connections.
questions: questions: Would it be possible to highlight the differences between ASA and n-ASA in Algorithm 1?

Please provide additional discussion of [46]. How is Algorithm 1 derived from existing work and how are the bounds derived at the end of section 4.2?

Please discuss the marginal threshold parameter $\Delta$ and how might that be chosen in practice.

I would like to see more discussion of Eq. 5 to explain the $U(i)$ and $S(j,i)$ functions and how they were selected for the experiments. 
Specifically, Figure 4 shows that ASA and n-ASA outperform all other baselines when connectivity is stable, which is an interesting result. Can you provide more numerical detail on why these methods outperform the baselines?

The assumption that robots' connectivity probabilities $C$ are known a priori and are constant limits the applicability of this approach to mobile robotics. Please explain how this affects the relative performance of n-ASA vs. ASA. You may want to add this analysis to expand the Limitations section.

A few other corrections:
- Ensure that the acronym ASA is defined prior to its first use in the abstract, and capitalized in text. 
- Table 1 typo: N-ASSA and ASSA should be N-ASA and ASA
limitations: limitations: 2
review_summary: review_summary: The work is incremental and has no robot experiments. Its formulation limits its use in mobile robotics, where connectivity concerns would be most relevant.
title: title: Interesting research problem, preliminary results
paper_summary: paper_summary: This paper considers the problem of human supervision allocation in multi-robot settings with uncertain connectivity. The proposed method is based on the design of a submodular allocation function and its greedy optimization. Simulations are used to investigate the impact of connectivity quality across different domains and against different baselines.
main_review: main_review: This paper considers an interesting and important problem. The current definition, however, is missing the temporal aspect. The problem is reduced to taking a snapshot of the current state, and running a static allocation algorithm. While it is true that greedily optimizing a submodular function offers theoretical guarantees, these only apply to the static version of the problem only. No reasoning is performed over what might happen during subsequent steps, which limits the novelty of the work. However, please note that I am not suggesting that this is not a valid method (the experimental results suggest that it is, and as a practitioner this is the first thing I would have tried), but as a reader of a conference paper I am left wondering if something better could be done be planning over multiple allocation steps. 

Regarding the temporal aspects, I believe the binary connectivity model could be further generalized to account, for example, the need for humans to spend non-negligible time for context switching, or unexpected loss of connectivity while in the middle of a task (it is not clear if the current setting connectivity might drop during an intervention). In summary, I would classify this as an experimental paper with some interesting preliminary results. 

I believe there are a few points requiring further clarifications (see my questions for rebuttal). A very interesting (in my opinion) scenario - self-driving cars - is missing from the experimental evaluation. It would have been nice to have it.

Minor comments:

1. Consider using R to denote the set of robots and 2^R to denote all the possible subsets (instead of the current X, which does not seem to be used consistently (see the paragraph before Eq. (4)).
questions: questions: 1. Could you please elaborate on the notion of "hard reset time" which was set to 5 time steps? How could a smaller / larger value impact an experiment?
2. For each intervention, how is the duration determined during an experiment (in addition to having a minimum time of 5 steps)? Why is the duration not a part of the problem statement / output of the allocation algorithm?
3. During an experiment, what happens when the robot is not being supervised? Does it keep following its current policy? Is that deterministic or random? How is the data collected in those steps used?
4. Did you consider integrating the temporal aspect in the allocation algorithm? If so, what difficulties did you encounter?
limitations: limitations: 2
review_summary: review_summary: This paper has potential but I believe this work does not get to the core of the problem.