Decentralized Learning in Online Queuing Systems

1 Introduction . Inefficient decisions in repeated games can stem from both strategic and learning considerations . First , strategic agents selfishly maximize their own individual reward at others ’ expense . The price of anarchy [ Koutsoupias and Papadimitriou , 1999 ] measures this inefficiency as the social welfare ratio between the best possible situation and the worst Nash equilibrium . Although reaching the best collective outcome might be illusory for selfish agents , considering the worst Nash equilibrium might be too pessimistic . In games with external factors , more complex interactions intervene and might lead the agents to the best equilibrium . Instead , the price of stability [ Schulz and Moses , 2003 ] measures the inefficiency by the social welfare ratio between the best possible situation and the best Nash equilibrium . Second , the agents also have to learn their environment , by repeatedly experimenting different outcomes . Learning equilibria in repeated games is at the core of many problems in computer science and economics [ Fudenberg et al. , 1998 , Cesa-Bianchi and Lugosi , 2006 ] . The interaction between multiple agents can indeed interfere in the learning process , potentially converging to no or bad equilibria . It is yet known that in repeated games , if all agents follow no internal regret strategies , their actions converge in average to the set of correlated equilibria [ Hart and Mas-Colell , 2000 , Blum and Monsour , 2007 , Perchet , 2014 ] . ∗Equal contributions 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Many related results are known in the classical repeated games [ see e.g. , Cesa-Bianchi and Lugosi , 2006 , Roughgarden , 2010 ] , where a single game is repeated over independent rounds ( but the agents strategies might evolve and depend on the history ) . Motivated by packet routing in computer networks , Gaitonde and Tardos [ 2020a ] introduced a repeated game with a carryover feature : the outcome of a round does not only depend on the actions of the agents , but also on the previous rounds . They consider heterogeneous queues sending packets to servers . If several queues simultaneously send packets to the same server , only the oldest packet is treated by the server . Because of this carryover effect , little is known about this type of game . In a first paper , Gaitonde and Tardos [ 2020a ] proved that if queues follow suitable no-regret strategies , a ratio of 2 between server and arrival rates leads to stability of the system , meaning that the number of packets accumulated by each queue remains bounded . However , the assumption of regret minimization sort of reflects a myopic behavior and is not adapted to games with carryover . Gaitonde and Tardos [ 2020b ] subsequently consider a patient game , where queues instead minimize their asymptotic number of accumulated packets . A ratio only larger than ee−1 then guarantees the stability of the system , while a smaller ratio leads to inefficient Nash equilibria . As a consequence , going below the ee−1 factor requires some level of cooperation between the queues . This result actually holds with perfect knowledge of the problem parameters and it remained even unknown whether decentralized learning strategies can be stable with a ratio below 2 . We first argue that decentralized queues need some level of cooperation to ensure stability with a ratio of rates below 2 . Policy regret can indeed be seen as a patient alternative to the regret notion . Yet even minimizing the policy regret might lead to instability when this ratio is below 2 . An explicit decentralized cooperative algorithm called ADEQUA ( A DEcentralized QUeuing Algorithm ) is thus proposed . It is the first decentralized learning algorithm guaranteeing stability when this ratio is only larger than 1 . ADEQUA does not require communication between the queues , but uses synchronisation between them to accurately estimate the problem parameters and avoid interference when sending packets . Our main result is given by Theorem 1 below , whose formal version , Theorem 5 in Section 4 , also provides bounds on the number of accumulated packets . Theorem 1 ( Theorem 5 , informal ) . If the ratio between server rates and arrival rates is larger than 1 and all queues follow ADEQUA , the system is strongly stable . The remaining of the paper is organised as follows . The model and existing results are recalled in Section 2 . Section 3 argues that cooperation is required to guarantee stability of learning strategies when the ratio of rates is below 2 . ADEQUA is then presented in Section 4 , along with insights for the proof of Theorem 1 . Section 5 finally compares the behavior of ADEQUA with no-regret strategies on toy examples and empirically confirms the different known theoretical results . 1.1 Additional related work . Queuing theory includes applications in diverse areas such as computer science , engineering , operation research [ Shortle et al. , 2018 ] . Borodin et al . [ 1996 ] for example use the stability theorem of Pemantle and Rosenthal [ 1999 ] , which was also used by Gaitonde and Tardos [ 2020b ] , to study the problem of packet routing through a network . Our setting is the single-hop particular instance of throughput maximization in wireless networks . Motivated by resource allocation in multihop radio problem , packets can be sent through more general routing paths in the original problem . Tassiulas and Ephremides [ 1990 ] proposed a first stable centralized algorithm , when the service rates are known a priori . Stable decentralized algorithms were later introduced in specific cases [ Neely et al. , 2008 , Jiang and Walrand , 2009 , Shah and Shin , 2012 ] , when the rewards Xk ( t ) are observed before deciding which server to send the packet . The main challenge then is coordination and queues aim at avoiding collisions with each other . The proposed algorithms are thus not adapted to our setting , where both coordination between queues and learning the service rates are required . We refer the reader to [ Georgiadis et al. , 2006 ] for an extended survey on resource allocation in wireless networks . Krishnasamy et al . [ 2016 ] first considered online learning for such queuing systems model , in the simple case of a single queue . It is a particular instance of stochastic multi-armed bandits , a celebrated online learning model , where the agent repeatedly takes an action within a finite set and observes its associated reward . This model becomes intricate when considering multiple queues , as they interfere when choosing the same server . It is then related to the multiplayer bandits problem which considers multiple players simultaneously pulling arms . When several of them pull the same arm , a collision occurs and they receive no reward [ Anandkumar et al. , 2010 ] . The collision model is here different as one of the players still gets a reward . It is thus even more closely related to competing bandits [ Liu et al. , 2020a , b ] , where arms have preferences over the players and only the most preferred player pulling the arm actually gets the reward . Arm preferences are here not fixed and instead depend on the packets ’ ages . While collisions can be used as communication tools between players in multiplayer bandits [ Bistritz and Leshem , 2018 , Boursier and Perchet , 2019 , Mehrabian et al. , 2020 , Wang et al. , 2020 ] , this becomes harder with an asymmetric collision model as in competing bandits . However , some level of communication remains possible [ Sankararaman et al. , 2020 , Basu et al. , 2021 ] . In queuing systems , collisions are not only asymmetric , but depend on the age of the sent packets , making such solutions unsuited . While multiplayer bandits literature considers cooperative players , Boursier and Perchet [ 2020 ] showed that cooperative algorithms could be made robust to selfish players . On the other hand , competing bandits consider strategic players and arms as the goal is to reach a bipartite stable matching between them . Despite being cooperative , ADEQUA also has strategic considerations as the queues ’ strategy converges to a correlated equilibrium of the patient game described in Section 2 . An additional difficulty here appears as queues are asynchronous : they are not active at each round , but only when having packets left . This is different from the classical notion of asynchronicity [ Bonnefoi et al. , 2017 ] , where players are active at each round with some fixed probability . Most strategies in multiplayer bandits rely on synchronisation between the players [ Boursier and Perchet , 2019 ] to avoid collisions . While such a level of synchronisation is not possible here , some lower level of synchronisation is still used to avoid collisions between queues . 2 Queuing Model . We consider a queuing system composed of N queues and K servers , associated with vectors of arrival and service rates λ , µ , where at each time step t = 1 , 2 , . . . , the following happens : • each queue i ∈ [ N ] receives a new packet with probability λi ∈ [ 0 , 1 ] , that is marked with the timestamp of its arrival time . If the queue currently has packet ( s ) on hold , it sends one of them to a chosen server j based on its past observations . • Each server j ∈ [ K ] attempts to clear the oldest packet it has received , breaking ties uniformly at random . It succeeds with probability µj ∈ [ 0 , 1 ] and otherwise sends it back to its original queue , as well as all other unprocessed packets . At each time step , a queue only observes whether or not the packet sent ( if any ) is cleared by the server . We note Qit the number of packets in queue i at time t. Given a packet-sending dynamics , the system is stable if , for each i in [ N ] , Qit/t converges to 0 almost surely . It is strongly stable , if for any r , t ≥ 0 and i ∈ [ N ] , E [ ( Qit ) r ] ≤ Cr , where Cr is an arbitrarily large constant , depending on r but not t. Without ambiguity , we also say the policy or the queues are ( strongly ) stable . Naturally , a strongly stable system is also stable [ Gaitonde and Tardos , 2020a ] . In the following , x ( i ) will denote the i-th order statistics of a vector x , i.e. , λ ( 1 ) ≥ λ ( 2 ) ≥ . . . ≥ λ ( N ) and µ ( 1 ) ≥ . . . ≥ µ ( K ) . Without loss of generality , we assume K ≥ N ( otherwise , we simply add fictitious servers with 0 service rate ) . The key quantity of a system is its slack , defined as the largest real number η such that : k∑ i=1 µ ( i ) ≥ η k∑ i=1 λ ( i ) , ∀ k ≤ N. We also denote by P ( [ K ] ) the set of probability distributions on [ K ] and by ∆ the margin of the system defined by ∆ : = min k∈ [ N ] 1 k k∑ i=1 ( µ ( i ) − λ ( i ) ) . ( 1 ) Notice that the alternative system where λ̃i = λi + ∆ and µ̃k = µk has a slack 1 . In that sense , ∆ is the largest margin between service and arrival rates that all queues can individually get in the system . Note that if η > 1 , then ∆ > 0 . We now recall existing results for this problem , summarized in Figure 1 below . Theorem 2 ( Marshall et al . 1979 ) . For any instance , there exists a strongly stable centralized policy if and only if η > 1 . Theorem 3 ( Gaitonde and Tardos 2020a , informal ) . If η > 2 , queues following appropriate no regret strategies are strongly stable . For each N > 0 , there exists a system and a dynamic s.t . η > 2 − o ( 1/N ) , all queues follow appropriate no-regret strategies , but they are not strongly stable . In the above theorem , an appropriate no regret strategy is a strategy such that there exists a partitioning of the time into successive windows , for which the incurred regret is o ( w ) with high probability on any window of length w. This for example includes the EXP3.P.1 algorithm [ Auer et al. , 2002 ] where the k-th window has length 2k . The patient queuing game G = ( [ N ] , ( ci ) ni=1 , µ , λ ) is defined as follows . The strategy space for each queue is P ( [ K ] ) . Let p−i ∈ ( P ( [ K ] ) ) N−1 denote the vector of fixed distributions for all queues over servers , except for queue i . The cost function for queue i is defined as : ci ( pi , p−i ) = lim t→+∞ T it /t , where T it is the age of the oldest packet in queue i at time t. Bounding T i t is equivalent to boundingQ i t. Theorem 4 ( Gaitonde and Tardos 2020b , informal ) . If η > ee−1 , any Nash equilibrium of the patient game G is stable .

Summary: The paper proposed a distributed scheduling algorithm for N queues and K servers, with heterogeneous arrival rates for the queues and service rates for the servers. The rates are unknown to the players (each player is a queue). Each queue to which server (if any) to send a packet. A server that receives multiple packets attempts to serve the oldest one and ignores the others (e.g., sends them back). In the proposed algorithm, players explore the arrival and service rates on the fly. To exploit, players use their local estimated rates to compute a dominant mapping between queues and servers based. Then players use Birkhoff von Neumann decomposition to obtain a doubly stochastic matrix of the random strategies, which gives the dominant mapping on average. It is proved that the proposed algorithm is strongly stable for when the slack eta is greater than one ("sum of service rate is larger than the sum of arrival rates").