Decentralized Data Collection for Robotic Fleet Learning: A Game-Theoretic Approach
Authors: Oguzhan Akcin
Po-han Li
Shubhankar Agarwal
Sandeep P. Chinchali
Venue: CoRL 2022 Poster (2022)

Abstract: Fleets of networked autonomous vehicles (AVs) collect terabytes of sensory data, which is often transmitted to central servers (the ``cloud'') for training machine learning (ML) models. Ideally, these fleets should upload all their data, especially from rare operating contexts, in order to train robust ML models. However, this is infeasible due to prohibitive network bandwidth and data labeling costs. Instead, we propose a cooperative data sampling strategy where geo-distributed AVs collaborate to collect a diverse ML training dataset in the cloud. Since the AVs have a shared objective but minimal information about each other's local data distribution and perception model, we can naturally cast cooperative data collection as an $N$-player mathematical game. We show that our cooperative sampling strategy uses minimal information to converge to a centralized oracle policy with complete information about all AVs. Moreover, we theoretically characterize the performance benefits of our game-theoretic strategy compared to greedy sampling. Finally, we experimentally demonstrate that our method outperforms standard benchmarks by up to $21.9\%$ on 4 perception datasets, including for autonomous driving in adverse weather conditions. Crucially, our experimental results on real-world datasets closely align with our theoretical guarantees.

Decentralized Data Collection for Robotic Fleet Learning: A Game-Theoretis Approach
Oguzhan Akcin', Po-han Li", Shubhankar Agarwal and Sandeep P. Chinchali
Abstract:
Fleets of networked autonomous vehicles (AVs) collect terabytes of sensory data, which is often transmitted to central servers (the "cloud") for train- ing machine learning (ML) models. Ideally, these fleets should upload all their data, especially from rare operating contexts, in order to train robust ML models. However, this is infeasible due prohibitive network bandwidth and data label- ing costs. Instead, we propose a cooperative data sampling strategy where geo- distributed AVs collaborate collect a diverse ML training dataset in the cloud. Since the AVs have shared objective but minimal information about each other's s local data distribution and perception model, we can naturally cast cooperative data collection an N-player mathematical game. We show that our coopera- tive sampling strategy uses minimal information to converge to a centralized or- acle policy with complete information about all AVs. Moreover, we theoretically characterize the performance benefits our game-theoretic strategy compared to greedy sampling. Finally, experimentally demonstrate that our method outper- forms standard benchmarks by up to 21.9% on 4 perception datasets, including for autonomous driving in adverse weather conditions. Crucially, experimen- tal results on real- world datasets closely align with our theoretical guarantees.
Introduction
Envision fleet of autonomous vehicles (AVs) that observes heterogencous street scenery, weather conditions, and rural/urban traffic patterns. To train robust ML models for perception or trajectory prediction, these AVs should share as much diverse fleet data possible in the cloud, while balanc- ing network bandwidth, data storage, and labeling costs.? ? Given these constraints, we argue that AVs must coordinate how to sample rare, -ifor--distribution (OoD) data with common examples based on their unique local data distributions. For example, if only a few AVs operate in heavy snow, they should specialize in sending snowy images to the cloud, while others should send data from more common scenarios like sunny weather. Since the AVs have shared target data distribution (objective) but limited information on each other's local data distribution and potentially private ML models, our key contribution is to cast data collection as a N-Player mathematical game.
In our comne-theoretie formulation (Fig. 1), the AVs exchange minimal information to choose data sampling strategy (what limited subset of data-points to upload). Importantly, we prove that an AV fleet will quickly converge to Nash equilibrium (i.e., fixed point where each robot does not change its sampling strategy) [3, 4] with bounded communication. Morever, our practical formula- tion accounts for perceptual uncertainty from imperfect computer vision models and heterogenous local data distributions. As such, to the best of our knowledge, we are the first to cast data sampling from networked robots as mathematical game. In summary, key contributions are:
1.
We provide novel formulation for distributed data collection the robots attempt to minimize a common convex objective function that incentivizes them to reach a balanced target data distribution in the cloud. We prove that our strategy converges to centralized oracle policy and, under mild assumptions, converges in a single iteration.
potential game [5] since
+1 Department of Electrical and Computer Engineering (ECE), The University of Texas at Austin, Austin, Sutrospppeeuteess....mm @utexas edu
TX oguzzannakii, pohaaannaaiiii....nll...... i @@utezas edu, {somi.
2 A single AV can measure over 20-30 Gigabytes (GB) per second video and LiDAR data [1] while typical SG wireless network only provides 10 Gbps of bandwidth for multiple users [2].
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Cloud
Action
ai
Retrain
rr+1
a
0
Device Dataset
Cloud Dataset
xi
x
Network
+
Action
a
a +
+
D
D
0
an
min L(D D. Diarget)
x11
Xn t
Figure I: Game-Theoretic Data Collection: Each step cooperative algorithu is numbered in blue. First, each AV observes a sequence images xē each round of data collection (step 1). Then, classifies each image x with a local vision model with parameters 8 (step 2). Then, it samples a limited set of Neache images according to its action policy of, which governs what distribution of data-points to upload. Crucially, the action of is chosen cooperatively with other AVs using distributed optimization problem (step 3). Next, each AV transmits its local cache of data-points the cloud (step-) The current cloud dataset, D updated with the new uploaded data-points of (step 5). The combined cloud dataset, 2 can be used to periodically re- re-train new model parameters 0-+1 (step 6). which are then downloaded by the AVs (step 7). All AVs share a goal of minimizing the distance between the collected cloud dataset 9 (green) and the target Purget.
2. We provide theoretical performance bounds characterizing the benefits of our game-theoretic approach compared to greedy, individual behavior.
3. We show that our proposed strategy outperforms competing benchmarks by 21.9% on 4 datasets, including the challenging Berkeley DeepDrive autonomous driving dataset [6].
Related Work: Data collection from networked robots is related to cloud robotics [7-15] and active learning [16-20]. In such prior works, robots either send all their data to the cloud or select samples individually without coordination. In contrast, exploit the fact that networked AVs can coordinate how to sample rare data to achieve a better outcome (i.e., balanced data distribution).
Federated learning (FL) [21-29] enables fleet of mobile devices to train ML models on local private data and only share anonymized gradient updates with the cloud. However, our work is fundamentally different, and even complementary, to standard FL. First, FL makes the restrictive assumption that each robot has perfectly labeled local data, which is infeasible for AVs that observe rare, OoD images. Instead, we address practical scenario where robots run local inference with only an imperfect vision model that guides data collection. Moreover, FL does not statistically sample data but trains on all of it locally, while our approach only uploads limited set of images to reduce network and data labeling costs. Finally, we assume robots only receive ground-truth labels for the uploaded data in the cloud, which is required for training rare classes.
Our setting non-cooperative game since the robots do not explicitly form coalitions and act with minimal information about each other [5, 30-34J. Specifically, our setting is potential game since each robot attempts to maximize shared concave objective function (the common potential function) that rewards progress towards a balanced target data distribution in the cloud. As detailed in Sec. 2 and Appendix 5.1, changes in the common potential function directly translate to changes in each robot's policy towards Nash Equilibrium. While concave games have been applied to problems such as wireless network resource allocation [35], ours game-theoretic formulation for distributed data collection from a fleet of robots.
first work to contribute
2 Problem Formulation
We now formulate practical scenario, shown in Fig. 1, where distributed robots collect data to train robust ML model in the cloud. Our goal is to select appropriate action for each robot, specif- ically the data-points should upload, so that the overall collected cloud dataset closely matches a given target, such an equal distribution over all classes. Fig. 2 intuitively depicts data sampling.

---

Tareet Distriouiion
Why Cooperate? Consider a toy example with only 2 classes and 2 rohots. The axes represent the number of data-points for each class. Our goal is to reach the target distribution (blue cross) where each class has 120 data-points, represented by (120, 120). The robots start at (0,0) with no data-points in the cloud. The possible combinations that can be uploaded from robots 1 and 2 are shown as the shaded feasi- ble action spaces (yellow and purple). This shaded region is determined by the robot's local data distribution and vision model accuracy (Def. 2). GREEDY (black) individually calculates the projection of the tar- get distribution onto each robot's feasible action space, but the sum of actions may not be optimal, leading to a high error (red). However, OR- ACLE accounts for the two robots' action spaces and thus minimizes the error between the target dataset and the sum of actions (grey).
Figure 2:
Robot Action Space
Num. Images Class 2
Robot Action Space
Greedy Action
300
Oracle Action
Error
20
20
26
20
62 20
100
120
Num. Images Class 1
Our general formulation applies to any robot constrained by network, storage, or labeling costs, ranging from Mars Rovers constrained by the Deep Space Network (< 5 Mbps) [36] or AV fleets.
Robot Perception Model: For simple exposition, we first consider a general computer vision classification task with Nelass classes. The dataset used for training the model is stored in the cloud. Each period of data collection, such as a day, is denoted by round and data is uploaded to the cloud at the end of a round r. The cumulative dataset stored in the cloud at the end of round r is denoted by He, whose size is given by No: 1 Ny, denotes the number of class j data- points in the dataset He. Therefore, the distribution of classes in the dataset O. is denoted by P (N -... NNyyetmn Each robot has a perception model, such as a deep neural network (DNN), where local inference is denoted by y f(x; 0()) Here, 0f) is a model with parameters 0 at round r, is the predicted label for input x, and y is the corresponding ground-truth label.
Importantly, the models can be imperfect each model has a confusion matrix, C7 E R N (Eq. 4) that captures the probability of predicting class 8j for an image with true class yj, denoted by p(frlr)). In practice, one of the Natass classes can represent an "unknown" category while the rest of the Nelass - classes can represent a mixture of rare and well-understood concepts. Further details on the confusion matrix are provided in Appendix 5.2. Finally, while we use (likely imperfect) classification model to sample images, the uploaded data can be used to train models for diverse tasks such as object detection, semantic segmentation etc.
Robot Fleet: We consider a fleet of Neobot robots, where each robot i collects a data-point xf at time f from its local environment (i.e., camera image or LiDAR scan). The distribution of true classes observed by robot in round is denoted by pT()) E R.Nitee which sums to over the Nelass classes. From this distribution, a robot observes a large dataset of images on round r denoted by 2.' of size |2:1- = NY. However, limit network bandwidth and data laheling costs, each robot i can only upload Neache < Ny images to the cloud at the end of round r, which it stores in an on- board cache within the round. The size of Neache can be flexibly set by roboticist based data upload and labeling budgets. The class predictions, fj. are generated by running local inference of the classification model, 8i = fx/; 0,), for the collected data-points x/. Finally, (f) denotes the distribution of predicted classes observed by robot i.
Assumption 1. The number of data-pont points collected by a robot on any round Nj, is significantly greater than the size of the local robot cuche, N >>> Neuche-
This is a valid assumption since each robot will collect much more data compared to the amount can economically upload. Our formulation is extremely general each robot can have different (or the same) model parameters 0f and observe different distribution (r()) of the Nelass classes.
Robot Statistical Sampling Action: At each round r. each robot i takes an action which deter- mines how many data-points of each class to send to the cloud. We define each robot i's action at round r as df Nyy .,...,,,,,, Nywan i.c. the number of data-points of each class j. Our key technical innovation will be to illustrate how to cooperatively select an optimal action. Importantly, since each robot i has imperfect perception model with confusion matrix Cj, there is uncertainty in the effect of taking any action a/- As such, our natural next step is to define the set of feasible actions any robot can upload given its local data distribution and perceptual uncertainty.
3

---

Definition 1 (Feasible data matrices). A feasible data matrix. P. E R Netae, of robot in round r is the probability matrix defined as:
P. PPi,. C......PNNaaa
where PL, C1 [[C,)e,][)] - p(yl;)) E Noaus Vj= 1...ll,,,Nlass Nelass- We use * as element-wise multiplica-
tion of vectors, I1- as the L1 norm, and the second subscript j to denote the j-th column of a matrix. We assume Pf has linearly independent columns, so there exists left inverse. In other words, assume the mapping from action to feasible action (Defs. 2, 4) is one-to-one. This assumption is justified in the Appendix due to space limils.
Definition 2 (Feasible spaces of robots). A feasible space, Hi, of robot i in round is the set of feasible data-points the robot can send to the cloud:
-v= 1 di S Neache of E BRW"iun
Hद is the convex hull of all columns of PÊ and 0. To simplify notation, =P= represents a feasible action vi,, which is obtained by multiplying an intended action at by the feasible data matrix P..
af
Intuitively, the feasible space (see Fig. 2) represents the expected number of datapoints uploaded per class but not the exact number due to perceptual uncertainty. Each robot uploads Neache data-points sampled from action df, which is pooled in the cloud. We assume we only get ground-truth labels y in the cloud, since the limited cache of images can be scalably annotated by human. Then, re-train a new perception model on the new dataset 2c+1. Each robot then downloads the new model parameters 0 along with the new confusion matrix and latest cloud dataset distribution, Poy+1- Our formulation is general - models and confusion matrices do not have to be updated every round r and we can, for example, simply re-train a model after M rounds of data collection.
Collective Goal: Achieving Target Data Distribution Often, we want to achieve balanced dataset in the cloud with ample representation of rare events in order to train a robust ML model. As such, the shared goal of all the robots is to achieve any user-specified target dataset Paage which defines the number of data-point, of each class the robots want to collect in the cloud. The fleet's s goal is to choose actions df; Vi I,...,,Noahi distance metric, denoted by PPung)) penalizing the difference between the current cloud dataset Pog and the target dataset P Our general framework can handle any strictly convex distance metric, such as the L2 norm or the Kullback-Leibler (KL. Divergence [37]. Since all robots BeeegpPausee Phungge) ), which is concave potential
round r to collectively reduce strictly convex
have a common goal maximize the negative loss -L (p function, our setting is a potential game with concave rewards (see Sec. 5.1).
We now provide formal optimization problem for distributed data collection. To provide key insight, we first describe a centralized "oracle" solution that has perfect information about all robots i, namely their confusion matrix C. and statistics of their data distribution (v). Then, we formalize a greedy, individualized approach and our interactive game- theoretic approach that matches the oracle policy's ' performance.
Centralized Oracle Action Policy:
An oracle action policy, denoted by ORACLE, has access to all robots' data distributions and con- fusion matrices C, The oracle calculates each robot's 's action of by solving the convex optimization problem in Eq. 1. The constraint (Eq. Ic) ensures that the actions af do not exceed the cache limit Neache- Eq. 1d shows the update the cloud dataset for round r+ based on the actions of taken in the feasible space, a/ by each robot for round which we now detail.
update the cloud dataset Pgg+i by merging the current cloud dataset por and each robots' uploaded dataset af. However, each robot's action imperfect it might think it is uploading class but due to perceptual uncertainty it might actually upload another class j'. Specifically, the robot's transmitted dataset of calculated from the predicted class labels 8; and not the true class labels y,, which are not available on- on-robot. However, we can use predicted class probabilities (f)
A key subtlety
to estimate true class probabilities (y;)) by: pf(r;) = I Nahes (f(()) pf (f) Note that each robot only receives confusion matrix C/ from the cloud which consists of conditional probabilities
br)) and not pf)))))) Therefore, we still need to figure out a way to calculate pf ())))))) Due
to space limits, we present the Bayesian update of pf ())))) in the Appendix 5.3.

---

PROBLEM 1: ORACLE OPTIMIZATION
PROBLEM 2: GREEDY OPTIMIZATION
min (P 11PPann Gr..aiiar
(1a)
min L(pg
(2a)
subject to: of 2 0; Vi I,..., Mobot
(1b)
subject to: of 20 1s - Neache
(2b)
1 of S Meuchei V =i=b 11=1... I,.., Nobot
(2c)
(Ic)
pg = po: + (P/d)
(2d)
Mudul
Dopui Por Σ (Pfa.)
(1d)
Greedy Action Policy: greedy action policy, referred to as GREEDY, will not have any infor- mation about other robots' local data distribution, confusion matrix, or observed datasets. Thus, the best the robot can do is to attempt to minimize the loss function (P PPaggrr by only optimiz- ing its own action of individually, as shown in Eq. 2. The optimization program 2 is very similar to that of the ORACLE policy (Eq. 1), with the only difference being that the decision variables reduced to one. Since the ORACLE (Eq. 1) and GREEDY (Eq. 2) policy optimization programs have convex objective with linear constraints, they guaranteed converge to optimal solution.
3 Cooperative Algorithm for Data Collection
We propose an INTERACTIVE algorithm for generating actions for each robot, which only requires interaction between the robots and cloud coordination. Rather than the cloud calculating actions for each robot in one-shot, as shown in the ORACIE optimization program (1), each robot calculates its actions individually using shared information from other robots. Importantly, each robot only shares its feasible action without divulging its confusion matrix or local data distribution to others.
Alg. 1 describes our INTERACTIVE policy, which runs for each round The inputs (line 1), which are visible to each robot, are the target dataset Paaaz and the current cloud dataset pør. We initialize each robot's action af in lines 2 using the GREEDY policy (Eq. 2) because the robots have not yet communicated any information about each others' tentative actions. In lines 5 10, we calculate optimal actions for each robot using the INTERACTIVE message passing algorithm.
We start by sharing each robot's product of feasible data matrix and initial action (line 3) with all other robots (line 5). Then, we iterate over each robot (lines 7 10) and calculate its best action of using the optimization program Eq. 3 while considering the other robots' actions fixed (line 8). The optimization program Eq. is similar to that of the ORACLE policy (Eq. 1): the difference lies in the calculation of the cloud dataset at round r+1 1 in Eq. 3d and having one decision variable.
In line 9, each robot shares its product of the feasible data matrix and the optimal action calculated using Eq. 3 with the others. This repeats until our system reaches a Nash equilibrium (i.e. a fixed point, where robot would change its action). Finally, after convergence, we upload data from each robot sampled according to its final calculated action of (line 13). Since the INTERACTIVE optimization program 3 is convex, it converges to an optimal solution (see Thm. 1).
Input: Target, Cloud Dataset Prrenn Po! for i= I.......NNutuu NNrsuu do
PROBLEM 3: INTERACTIVE OPTIMIZATION
Initialize af using GREEDY actions Eq. end
Share af with all robots. while Not Converged do
min of
L(perti .Pange)
(3a)
subject to: of 20
(3b)
for imri= I...... Niob....Nobbot.. do
Get action df using opt. program Eq. 3 Share actions af with all robots.
1" S Nexche
(3c)
Noser
end 11 end
P Par:
(P(
(P)
1:
for I...., Nisbur do
(3d)
13
Upload caches determined by actions of
14
end
Algorithm 1: INTERACTIVE Algorithm
5

---

Theoretical Analysis: We first show that the while loop (lines 6- 11) in our proposed Alg. 1 will eventually converge. Moreover, we provide easily-obtaing conditions for when it converges in one iteration, which minimizes inter-robot communication. Crucially, we also show that our interactive policy matches the omniscient oracle policy. All proofs are in the Appendix 5.6 5.9.
Theorem (Convergence). The while loop (lines 6 11) in Alg. / will eventually converge.
Next, we show one of the main technical contributions of this paper, which states that our proposed INTERACTIVE algorithm will reach the same optimal solution as the ORACLE upon termination.
Theorem 2 (INTERACTIVE converges ORACLE). The while loop in Alg. I lines 6 11 is guaran- teed to return an action (denoted by dmm,)) that is equal to the ORACLE action denoted by d's
Next, we provide practical conditions for when our proposed INTERACTIVE ACTIVE action policy will con- verge in one iteration of message passing, which bounds inter-robot communication.
Theorem 3 (Bounded Communication), When the total number of uploaded data-points is smaller than the difference between the size of target dataset Parger and the current cloud dataset Ve namely (Burget
D²)= > Nrobot x Neache- the while loop in Alg. lines 6 11 terminates in one iteration.
The condition in Thm. holds for all rounds except for the last round that reaches the target distri- bution, upon which data collection terminates. All our theory assumes that all actions df E RR can realize any feasible real-valued vector. However, in reality, action vector since we can only upload discrete set of images, which becomes an integer program- ming problem. However, for real-world datasets with thousands of images, we can just round the continuous solution to get very close approximation to the (generally intractable) integer case.
will only have integer-valued
4 Experiments and Conclusion
We now compare our Alg. with benchmark methods on four diverse datasets. The first two datasets of MNIST [38] and CIFAR-10 [39] serve as proof-of-concepts for the domains of handwritten digit and common object classification. Then, we use the Adverse-Weather dataset [40], which contains tens of thousands of images to train self-driving vehicles to classify rain, fog, snow, sleet, overcast, sunny, and cloudy driving conditions. To show the generality of our theory, we then extend to the state-of-te the-hhterttt Berkeley Deep Drive (DeepDrive) dataset [6], which has 100K images of various weather conditions and road scenarios for self diiviii cars.
Comparison Metric: To compare all methods, we use the 12-norm (the optimization objective) be- tween the target Pauge and the current cloud dataset pør. For statistical confidence, all experiments are repeated for more than 10 times with different random seeds that capture uncertainty in sampling from the confusion matrix C and observing different distributions of local data per robot. Further experiment parameters are detailed in the Appendix 5.4. We compare the following methods:
1. GREEDY solves the optimization program in Eq. 2 individually per robot by minimizing the L2-norm between the target and cloud distribution without information about other robots.
2. OR solves the optimization program in Eq. 1. It perfectly knows all incoming class data distributions py) and confusion matrices C, for all robots and thus calculates the optimal action for each robot in one common optimization problem (Eq. 1).
UNIFORM is a deterministic policy which assigns the same probabilities all classes for each Netuss It represents a simple heuristic for equally sampling all classes. (derived in Lemma 6) is the lower bound of the objective function of the ORA- CLE policy for a given target dataset, Pauugen current cloud dataset, Pig. and local data distribution
4. robot, i.e. a;- Nens
(()).. It represents how well sample in the absence of perceptual uncertainty.
5. INTERACTIVE runs our Alg. 1. It is not an Oracle policy, as it only shares the action taken by other robots and not the actual class data distribution, p/ (y), nor the confusion matrix.
Results: Our experimental results (Fig. 3) demonstrate that our proposed INTERACTIVE policy performs as well as the ORACLE, as proved in Thm. Additionally, we demonstrate that our IN- TER ACTIVE policy is much better than the GREEDY and UNIFORM policies on all datasets. Finally, we show that no action policy can perform better than the derived LOWER- LOWER-BOUND action policy.

---

Initial
Greedy
Interactive (Ours)
Oracle
Uniform
Lower Bound
Wilcoson 2,029e-00
Dataset Distance
Occurrence Frequency
1.59
2.255
2023
MNIST
Accuracy
0.75
2.203
2.39
1825
L.
1009
2375
Round
Classes
1686
Gready Interatttes Drade onftrm
Policy
Norm Dataset Distance
2329
Wilcomn p 1.053-03
Occurrence Frequency
0339
CIFAR10
4.225
0.30
Accuracy
3.78
4
Round
2.00
Classes
1405
Bready Interaciive Biade Undhorn
Pollcy
Wilcoson p 4.8836-04
1. Norm Dataset Distance
Adversarial Weather
Occurrence Frequency
Accuracy
6000
Round
Classes
190aa
Greedy
Inferactive
Uxiforn
Wilcoon 1,053e
Norm Datazet Distance
0.07
Occurrence Frequency
DeepDrive
2.22
0.00
Accuracy
2.22
0.0
2.42
Round
Classes
Policy
Figure 3: Our game-theoretic INTERACTIVE policy outperforms benchmarks and converges to the OR- ACLE. Each row is a different dataset. Column 1: As expected by our theory. INTERACTIVE minimizes the L2-norm distance (optimization objective, y-axis) better than GREEDY and matches the omniscient ORACLE. Colamn Clearly, INTERACTIVE achieves much more balanced distribution classes (target distribution uniform) than benchmarks. Column 3: Since INTERACTIVE achieves a more balanced datasct, this experimen- tally translates to higher DNN accuracy (statistically significant) on a held-out test datasct.
Does cooperation minimize distance to the target data distribution?
Our optimization objective is to minimize the L2-norm distance between the cloud dataset and tar- get data distribution, which we plot in the first column. Clearly, our INTERACTIVE policy sig- nificantly outperforms the GREEDY and UNIFORM policies on all datasets. Specifically, we beat the GREEDY policy by 23.6% 44.8%, 40.3%, 30..%% 38.7% in L2-norm distance on the MNIST, CIPAR-10, Siverse-Weather and DeepDrive datasets respectively. These benefits arise because the INTER- ACTIVE policy allows robots to coordinate the rare classes they upload, but GREEDY policy might lead to uncoordinated uploading of redundant data. Moreover, our INTERACTIVE policy performs nearly identically as the ORACLE method, with small deviations due to imperfect vision models and randomness in local data distributions between trials. This is natural since proved

---

that the INTERACTIVE action policy reaches the same optimal value in expectation as the ORACLE policy in Thm. 2. Finally, we observe that no action policy outperforms our LOWER-BOUNN policy derived in Lemma 6.
Does cooperation achieve more balanced datasets?
In column two, we see that the initial data distribution among robots (gray) is highly imbalanced since they operate in diverse contexts, However, see that our INTERACTIVE policy (green) achieves a much more balanced dataset distribution compared to GREEDY (orange), which is natural since the convex objective minimizes the distance uniform distribution.
What the final accuracy of trained models?
In column 3, we show the final accuracy of re-training DNN classification models on the datasets accrued by each method in the cloud. Importantly, our proposed INTERACTIVE action policy leads to better accuracy gains than the GREEDY and UNIFORM action policies. We beat the GREEDY policy by 1.4%, 1.7%, 21.9%, 12.4% in accuracy on the MNIST, CIFAR-10, Adverse-Weather, and DeepDrive datasets respectively. This is because the INTERACTIVE action policy makes sure we collect classes lacking in the current cloud dataset, thus preventing class-imbalance issues in model training. While our theory only addresses convex distances between dataset distributions (column 1 and 2), we show strong experimental results for re-training non-convex DNN classifiers. The INTERACTIVE and ORACLE algorithms lead to slightly different final accuracies since they can potentially upload different set of images and there is not a closed form relationship between the number of images and accuracy of non-convex DNN. As detailed in the Appendix, INTERACTIV achieves very close to state-of-the-art accuracy for each dataset with only a limited set of uploaded datapoints. DNN architectures are also detailed in the Appendix. Collectively, these results closely align with our theory and show strong experimental benefits on real-world data.
Our work assumes each robot can interact, which does not scale for extremely large fleets. Moreover, we assume that we sample images according to classification model, even though we can train models for other tasks on the uploaded images. In future work, we aim to extend our theoretical guarantees for sub-clusters of communicating robots and cluster a continuous data distribution based on similar embeddings that serve as virtual "classes". Such an ability to generalize beyond discrete classes may enable our algorithm scale to learning data-driven control policies.
Limitations:
Conclusion: This paper presents a sheoreticlly-groonded,, cooperative data sampling policy for networked robotic fleets, which converges an oracle policy upon termination. Additionally, converges in a single iteration under a mild practical assumption, which allows communication efficiency on real-world AV datasets. Our approach is a first step towards an increasingly timely problem as today's AV fleets measure terabytes of heterogenous data in diverse operating contexts [1]. In future work, we plan to develop policies that approximate the oracle solution when only a subset of robots can form coalitions and certify their resilience to adversarial node failures.
Acknowledgements
This material based upon work supported in part by Cisco Systems, Inc. under MRA MAG00000005. This material is also based upon work supported by the National Science Foun- dation under grant no. 2148186 and is supported in part by funds from federal agency and industry partners as specified in the Resilient & Intelligent Nex1G Systems (RINGS) program.
8

---

References
[1] Data the new oil in the future of automated driving. https://newaroom.cc int.:/arrrroo..oo......eeee.eee. com/e di ritorials/krzanie whwthhh-fuuuee-eeeeeeeeeee-eeeeh htt.:uee-.--ttttttaaataadiitteee LoDU2b0 2016. [Online; accessed 10-June-2021J.
[2] 10gbps 5g data speeds qualcomm and keysight achieve industry-first milestone. https: //datacenternews..... asia/storyy1ogbpe http:///0000poo---taaa-aeeeeeeee -od---uu---d---eeeeeeeeee--
t-
2021. [Online; accessed 14-June-2022].
[3] Nash Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48-49, 1950.
[4] Nash. Non-cooperative games. Annals of mathematics, pages 286-295, 1951.
[5] J. B. Rosen. Existence and uniqueness of equilibrium points for concave n-person games. Econometrica, 33(3):520-534, 1965. ISSN 00129682, 14680262. URL http::tttt//tt jsto
r. ceattaalee199117 1911749..
[6] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. Proccedings the IEEE/CVF conference computer vision and pattern recognition, pages 2636-2645, 2020.
[7] Cloud robotics and automation. http: httt:://godbbrrg berkeley edu/ cadd/clouu-robotics.. 2019. [Online; accessed 02-J9n-20199)
[8] B. Kehoe, S. Patil, P. Abbeel, and Goldberg. A survey of research on cloud robotics and automation. IEER Trans. Automation Science and Engineering, 12(2):398 -09, 2015.
[9] Kuffner. Cloud-enabled robots in: leee-ras international conference on humanoid robots. Piscataway, NJ: IEEE. 2010.
[10] S. Chinchali, E. Pergament, M. Nakanoya, E. Cidon, E. Zhang, D. Bharadia, M. Pavone, and
S. Katti. Sampling training data for continual learning between robots and the cloud. In 2020 International Symposium Experimental Robotics (ISER), Valetta, Malta, 2020.
[11] K. Goldberg and Kehoe. Cloud robotics and automation: survey of related work. EECS Department, University of California, Berkeley, Tech. Rep. TLBB/ECC--201341 2013.
[12] S. Chinchali, A. Sharma, J. Harrison, Elhafsi, Kang, Pergament, Cidon, S. Katti, and M. Pavone. Network offloading policies for cloud robotics: a learning- based approach. Autonomous Robots, 45(7):997-1012, 2021. doi: 10. 7/doi.org/
URL https:
[13] A. K. Tanwani, N. Mor, D. Kubiatowicz, E. Gonzalez, and Goldberg. fog robotics approach to deep robot learning: Application to object recognition and grasp planning in sur- face decluttering. 2019 International Conference on Roboties and Automation (ICRA), pages 4559-4566, 2019.
[14] Y. Geng, D. Zhang, P.-h. Li, O. Akcin, A. Tang, and S. Chinchali. Decentralized sharing and valuation of fleet robotic data. In 5th Annual Conference on Robot Learning, Blue Sky Submission Track, 2021.
[15) V.C. Pujol and S. Dustdar. Fog Sabbtics--uddesstaande the research challenges. IEEE Internet Computing, 25(05):10-17, sep 2021. ISSN 1941-0131. doi: 10.1 10.1100//C..222213300000
[16] D. A. Cohn, Z. Ghahramani, and M. Jordan. Active learning with statistical models. JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH, 4:129-145, 1996.
[17] Y. Gal, Islam, and Ghahramani. Deep Bayesian active learning with image data. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1183-1192, PMLR, 06-11 Aug 2017. URL http::htttccceeeee http://proceedingss mlr wtw..mm.rrrsso/o//... wwwpss/ooogaalllaaa html.

---

[18] B. Settles. Active learning literature survey. Technical report, University of Wisconsin-
Madison Department of Computer Sciences, 2009.
[19] S. Tong. Active learning: theory and applications, volume 1. Stanford University USA, 2001.
[20] K. Yang. J. Ren, Y. Zhu, and W. Zhang. Active learning for wireless iot intrusion detection.
IEEE Wireless Communications, 25(6): 25, 2018, Ntt1101100/MW22211..18000
[21] Federated learning:
Collaborative machine learning without centralized training data. https: //ai. . googleblog.....ooooooooooooooooo com
wew.eennnnoollaolooooooooooooo .hmml, 2017.
[Online; accessed 15-June-2021).
[22] J. Konečný. H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. CoRR, abs/1610.05492, 2016. http: http///axvvvvo// www.ooog/////oa...o ab1//610..11...///.//.////////1/ 00t.s/592........... 05492.
[23] Y. Xianjia, J. P. Queralta, J. Heikkonen, and T. Westerlund. Federated learning in robotic and
autonomous systems. Procedia Computer Science, 191:135-142, 2021. ISSN 1877-0509. doi: https://doi.org/100 p016/ 2021.07.01..1 URL hattps://wwe sciencedirect. com/so The 18th International Conference on Mobile Systems and Pervasive Computing (MobiSPC), The 16th International Conference Future Networks and Communications (FNC), The 11th International Conference on Sustainable En- engy Information Technology.
sonce/articn sancc//artilleep./SStteet
[24] E. Rizk, S. Vlaski, and A. H. Sayed. Optimal importance sampling for federated learning. In ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3095-3099, 2021. (01110..10011AAPP9977220000
[25] A. Armacki, D. Bajovic, D. Jakovetic, and S. Kar. Personalized federated learning via convex clustering. arXiv preprint arXiv:2202.00718. 2022.
[26] H. Xing. O. Simeone, and S. Bi. Decentralized federated learning via sgd over wireless d2d net- works. In 2020 IEEE 21 International Workshop Signal Processing Advances in Wireless Communications (SPAWC), pages 1-5, 2020, doi: 100T10000//AAA555..22202 222.99134332..
[27] B. Gu, A. Xu, Z. Huo, C. Deng, and H. Huang. Privacy-pree preserving asynchronous vertical federated learning algorithms for multiparty collaborative learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2021. 110110111177NNNS522211.00000 38.
[28] S. Niknam, H. S. Dhillon, and J. H. Reed. Federated learning for wireless communications: Motivation, opportunities, and challenges. IEEE Communications Magazine, 58(6):46-51 51, 2020. doi: 10. (009///CMM.O0..1000000
[29] S. Caldas, J. Konečny, H. B. McMahan, and A. Talwalkar, Expanding the reach of federated learning by reducing client resource requirements, 2018.
[30] W. Kim and K. Lee. The existence of nash equilibrium in n-person games with c-concavity. Computers & Mathematics with Applications, 1219)1219--1228, 2002. ISSN (889-12211 doi: https://doi.org/10 http://..oorroo..oooooooosss//ea 1221 m/ m/science/artiiaieeee httfa//c//rrrceee/////////ee2220
URL htttps://wwe http://www.ceeccceecc.cmo www.:ssciennddreetccomm
[31] Wang, D. Wang, and W. Jianhua. The Theory of Games. Oxford mathematical monographs. Tsinghua University Press, 1988. ISBN 9780198535607. URL https://books: htt:::///oooooooolll.eeee
google.c
om htt/bbooss2.d-JSTAAAAA..
[32] T. Driessen. Cooperative Games, Solutions and Applications. Theory and Decision Library C. Springer Netherlands, 2013. ISBN 9789401577878.. URL htt.httts://obote google. com.
[33] H. W. Stuart. Cooperative Games and Business Strategy, pages 189-211. Springer US, Boston, MA, 2001. ISBN 978-0-906-47568-9 11211..100700330--75566 URL. https///s: org /0/ 1007 011//0000000----- 0000000000077-.66666666-6
10

---

[34] S. Tijs. Introduction to game theory. Springer, 2003.
[35] Z. Han, D. Niyato, W. Saad, T. Başar, and A. Hjørungnes. Game theory in wireless and communication networks: theory, models, and applications. Cambridge university press, 2012.
[36] Mars reconnaissance orbiter: Communications with earth. https: /tte://marr /mars.aaaaaaaa nasa. nasa.gov/nro/ mi missiinn/on communi miision/communicca cations/ 2020. [Online; accessed 18-Oct.-2020).
[37] S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1))79-86, 1951.
[38] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://mw..ccu.c//////////maaa
2010.
[39] A. Krizhevsky. Learning multiple layers features from tiny images. Technical report, 2009.
[40] Adverse weather dataset. attp://sar-Da htt.:b/nntttttteeeeeeeeeeeeeeere
[41] S. Durand. Analysis of Best Response Dynamics in Potential Games. Theses Université Grenoble Alpes, Dec. 2018. URL https://ee/ wrwchheess-uueersss..tttteteeeee


Reviews:
[{"review_id": "akh8L_rMUv-", "reviewer": "Reviewer_YAM6", "summary": "This paper studies distributed data collection by autonomous robots, in which a large number of robots aim to collaboratively collect the data such that the joint dataset matches some target distribution. The proposed algorithm cast this problem as a potential game, and hence propose an algorithm which does not require centralised access to all information yet achieves comparable performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach of casting the problem of decentralized data collection as a game is interesting.\n- The approach taken for the proposed algorithm, i.e., firstly giving a centralised algorithm with perfect information and then relaxing it to derive a practical algorithm with less required information, is reasonable.\n- The empirical results are promising, and indeed shows that the performance of the proposed method can match that of the ideal algorithm with oracle access.\n\nWeaknesses:\n- I'm not entirely sure about the practical benefit of the proposed algorithm compared with the oracle algorithm with perfect information. If I understand correctly, the oracle algorithm can be achieved by letting every agent share their information with the central coordinator, and the central coordinator can then do all the calculation; the proposed INTERACTIVE algorithm does not require sending all information to the central coordinator yet it still requires the robots to share information with each other. I wonder is it possible to give some justification as to whether the proposed algorithm requires less computation than the oracle algorithm?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper takes a nicely grounded approach to solve an interesting problem from a game-theoretical perspective.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see \"Strengths And Weaknesses\" for details.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "QiQTxRBMIc6", "reviewer": "Reviewer_scJ6", "summary": "This paper poses the following question: in a setting with a large number of autonomous vehicles that operate under communication bandwidth constraints, what data should each vehicle send to a central server (e.g., for labeling and training a machine learning model to be used by the vehicles for tasks such as image classification)? If there were no bandwidth constraints and no constraints on labeling costs on the central server, each vehicle would send all its data. In a practical setting this is not possible. The challenge is to have the vehicles send a diverse set of data such that central server sees as 'balanced' a data distribution as possible. The paper casts this distributed data collection problem as a potential game and shows that the (distributed) strategy converges to a centralized 'oracle' policy. The paper also gives theoretical performance bounds and compares with other (albeit simpler) strategies for the problem on an autonomous driving dataset. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has the following strengths:\n1. The formulation is novel (at least to this reviewers knowledge) in this setting (autonomous vehicles uploading data to the cloud over a bandwidth constrained channel). \n2. The proof of convergence of the INTERACTIVE algorithm (and the convergence to ORACLE).\n3. Experimentally, INTERACTIVE achieves a more balanced distribution of classes. \n\nFood for thought:\n1. The paper says that the algorithm converges under a mild practical assumption. This is true in the setting explored by the authors, but the real problem is that the setting itself is vastly oversimplified due to the assumption that all vehicles may communicate with each other. The authors acknowledge this in the limitation section. \n2. I would have liked to see a discussion of a more general set of tasks than image classification. What if the goal of data collection was to improve the control policy on each vehicle instead of simply having better image classification tomorrow than was available today? \n3. It is not clear to me that an agreed upon fixed round at which updates are sent is needed. Would things change (possibly improve) if some vehicles decided to send updates at lower frequencies? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this is a nice piece of work with interesting ideas. It is well situated in the body of work that asks questions about how autonomous vehicles should use the cloud efficiently (i.e. while respecting constraints e.g., on bandwidth and labeling costs). If one takes a broad view of 'robotics' (which I do), then this work is certainly relevant to the field of robotics. I am not enough of an expert to know whether the ideas in the paper have been developed in other contexts and are simply novel to robotics people or are more fundamentally new which is really the only reason I marked the paper as a weak accept instead of a strong accept. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. I would encourage the authors to replace swarms with fleets (or some other word). To often in robotics, people simply say swarm when they mean a large number. A swarm is more than just a large number of things. The morning rush hour traffic in a major city has a very large number of vehicles, but they do not constitute a swarm in the way that a large number of bees migrating to a new location or a large number of tourists suddenly descending upon the Louvre. \n\n2. Is there a contradiction between convex objective function (line 37) and concave objective function (line 59)? \n\n3. What happens if the number of classes is unknown (section 2) ? \n\n"}}, {"review_id": "CD9s4dJtMt", "reviewer": "Reviewer_hwcw", "summary": "The paper is written in the context of multiple-agents that collect data in the field. They consider the problem where the data collected by these agents need to be aggregated in a central location in order to train models on the data. As bandwidth might be limited the agents cannot transfer all the data. Hence the authors propose an algorithm that sub-samples the data to only send a fraction of the entire dataset over the network. They claim their proposed methods outperforms baselines while maintaining a target distribution required at the central location.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) The proposed approach is quite general and has some convergence guarantees theoretically.\n\nWeaknesses\n1) The experimental section of the paper is not quite related with robotics. The datasets collected by multiple robots in the field will likely not have IID distribution. The datasets considered in this evlatuation section of this work have IID distribution. Any proposed method to tackle this problem should consider experiments where the distributions of each agent might be quite different from each other but have strong correlation between the datapoints on one robot. For example, robot 1 would see picture of rain for quite some time while robot 2 might be in an area where it is sunny.\n\n2) How is the model initialized in the data collection process? It seems the agent needs access to a model with non-trivial accuracy in order to choose data samples correctly.\n\n3) How does accuracy change if there is a drift because of errors made by an inaccurate model?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation is mostly due to the experimental section not considering scenarios more relevant to robotics setup.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "NA", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "8OwowmSfNgb", "reviewer": "Reviewer_2cKZ", "summary": "This paper presents cooperative data sampling policy for networked robotic fleets, which converges to an oracle policy upon termination. The motivation is the fact that a lot of AV problems would collect a lot of data, and we cannot quite upload all the data to a common database, so we would want to somehow learn from all the data without actually transferring it to the cloud. The authors cast this problem as a co-operative game, and under some linearity conditions derive an approach that performs well. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The problem being studied is interesting and real, especially at scale from an AV company point of view.\n\n2. I liked the experiments in the paper -- they start from simple, but intuitive CIFAR and MNIST experiments, but scale it up to BDD-level experiments, which is interesting It would be great to see this scaled up even further, since this strategy really matters when scaling.\n\nWeaknesses:\n\n1. Limitations section is not present, which is compulsory, though I hope that authors will revise the paper to add it.\n\nUnfortunately, I am not an expert in this area, and have only very limited knowledge of the related works. The idea of framing into a co-operative game is interesting, but I don't know if that's studied already or even in general, how is distributed data collection posed. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am unfortunately not very knowledgeable in this area, and would have to rely on other reviewers to assess the novelty and systems impact of this work. That said, I did read through the paper and checked the experiments, and briefly the theorems, and they do make sense to me. That's why I am going with a weak accept, with a confidence of 2.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)