Decentralized Learning in Online Queuing Systems

1 Introduction . Inefficient decisions in repeated games can stem from both strategic and learning considerations . First , strategic agents selfishly maximize their own individual reward at others ’ expense . The price of anarchy [ Koutsoupias and Papadimitriou , 1999 ] measures this inefficiency as the social welfare ratio between the best possible situation and the worst Nash equilibrium . Although reaching the best collective outcome might be illusory for selfish agents , considering the worst Nash equilibrium might be too pessimistic . In games with external factors , more complex interactions intervene and might lead the agents to the best equilibrium . Instead , the price of stability [ Schulz and Moses , 2003 ] measures the inefficiency by the social welfare ratio between the best possible situation and the best Nash equilibrium . Second , the agents also have to learn their environment , by repeatedly experimenting different outcomes . Learning equilibria in repeated games is at the core of many problems in computer science and economics [ Fudenberg et al. , 1998 , Cesa-Bianchi and Lugosi , 2006 ] . The interaction between multiple agents can indeed interfere in the learning process , potentially converging to no or bad equilibria . It is yet known that in repeated games , if all agents follow no internal regret strategies , their actions converge in average to the set of correlated equilibria [ Hart and Mas-Colell , 2000 , Blum and Monsour , 2007 , Perchet , 2014 ] . ∗Equal contributions 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Many related results are known in the classical repeated games [ see e.g. , Cesa-Bianchi and Lugosi , 2006 , Roughgarden , 2010 ] , where a single game is repeated over independent rounds ( but the agents strategies might evolve and depend on the history ) . Motivated by packet routing in computer networks , Gaitonde and Tardos [ 2020a ] introduced a repeated game with a carryover feature : the outcome of a round does not only depend on the actions of the agents , but also on the previous rounds . They consider heterogeneous queues sending packets to servers . If several queues simultaneously send packets to the same server , only the oldest packet is treated by the server . Because of this carryover effect , little is known about this type of game . In a first paper , Gaitonde and Tardos [ 2020a ] proved that if queues follow suitable no-regret strategies , a ratio of 2 between server and arrival rates leads to stability of the system , meaning that the number of packets accumulated by each queue remains bounded . However , the assumption of regret minimization sort of reflects a myopic behavior and is not adapted to games with carryover . Gaitonde and Tardos [ 2020b ] subsequently consider a patient game , where queues instead minimize their asymptotic number of accumulated packets . A ratio only larger than ee−1 then guarantees the stability of the system , while a smaller ratio leads to inefficient Nash equilibria . As a consequence , going below the ee−1 factor requires some level of cooperation between the queues . This result actually holds with perfect knowledge of the problem parameters and it remained even unknown whether decentralized learning strategies can be stable with a ratio below 2 . We first argue that decentralized queues need some level of cooperation to ensure stability with a ratio of rates below 2 . Policy regret can indeed be seen as a patient alternative to the regret notion . Yet even minimizing the policy regret might lead to instability when this ratio is below 2 . An explicit decentralized cooperative algorithm called ADEQUA ( A DEcentralized QUeuing Algorithm ) is thus proposed . It is the first decentralized learning algorithm guaranteeing stability when this ratio is only larger than 1 . ADEQUA does not require communication between the queues , but uses synchronisation between them to accurately estimate the problem parameters and avoid interference when sending packets . Our main result is given by Theorem 1 below , whose formal version , Theorem 5 in Section 4 , also provides bounds on the number of accumulated packets . Theorem 1 ( Theorem 5 , informal ) . If the ratio between server rates and arrival rates is larger than 1 and all queues follow ADEQUA , the system is strongly stable . The remaining of the paper is organised as follows . The model and existing results are recalled in Section 2 . Section 3 argues that cooperation is required to guarantee stability of learning strategies when the ratio of rates is below 2 . ADEQUA is then presented in Section 4 , along with insights for the proof of Theorem 1 . Section 5 finally compares the behavior of ADEQUA with no-regret strategies on toy examples and empirically confirms the different known theoretical results . 1.1 Additional related work . Queuing theory includes applications in diverse areas such as computer science , engineering , operation research [ Shortle et al. , 2018 ] . Borodin et al . [ 1996 ] for example use the stability theorem of Pemantle and Rosenthal [ 1999 ] , which was also used by Gaitonde and Tardos [ 2020b ] , to study the problem of packet routing through a network . Our setting is the single-hop particular instance of throughput maximization in wireless networks . Motivated by resource allocation in multihop radio problem , packets can be sent through more general routing paths in the original problem . Tassiulas and Ephremides [ 1990 ] proposed a first stable centralized algorithm , when the service rates are known a priori . Stable decentralized algorithms were later introduced in specific cases [ Neely et al. , 2008 , Jiang and Walrand , 2009 , Shah and Shin , 2012 ] , when the rewards Xk ( t ) are observed before deciding which server to send the packet . The main challenge then is coordination and queues aim at avoiding collisions with each other . The proposed algorithms are thus not adapted to our setting , where both coordination between queues and learning the service rates are required . We refer the reader to [ Georgiadis et al. , 2006 ] for an extended survey on resource allocation in wireless networks . Krishnasamy et al . [ 2016 ] first considered online learning for such queuing systems model , in the simple case of a single queue . It is a particular instance of stochastic multi-armed bandits , a celebrated online learning model , where the agent repeatedly takes an action within a finite set and observes its associated reward . This model becomes intricate when considering multiple queues , as they interfere when choosing the same server . It is then related to the multiplayer bandits problem which considers multiple players simultaneously pulling arms . When several of them pull the same arm , a collision occurs and they receive no reward [ Anandkumar et al. , 2010 ] . The collision model is here different as one of the players still gets a reward . It is thus even more closely related to competing bandits [ Liu et al. , 2020a , b ] , where arms have preferences over the players and only the most preferred player pulling the arm actually gets the reward . Arm preferences are here not fixed and instead depend on the packets ’ ages . While collisions can be used as communication tools between players in multiplayer bandits [ Bistritz and Leshem , 2018 , Boursier and Perchet , 2019 , Mehrabian et al. , 2020 , Wang et al. , 2020 ] , this becomes harder with an asymmetric collision model as in competing bandits . However , some level of communication remains possible [ Sankararaman et al. , 2020 , Basu et al. , 2021 ] . In queuing systems , collisions are not only asymmetric , but depend on the age of the sent packets , making such solutions unsuited . While multiplayer bandits literature considers cooperative players , Boursier and Perchet [ 2020 ] showed that cooperative algorithms could be made robust to selfish players . On the other hand , competing bandits consider strategic players and arms as the goal is to reach a bipartite stable matching between them . Despite being cooperative , ADEQUA also has strategic considerations as the queues ’ strategy converges to a correlated equilibrium of the patient game described in Section 2 . An additional difficulty here appears as queues are asynchronous : they are not active at each round , but only when having packets left . This is different from the classical notion of asynchronicity [ Bonnefoi et al. , 2017 ] , where players are active at each round with some fixed probability . Most strategies in multiplayer bandits rely on synchronisation between the players [ Boursier and Perchet , 2019 ] to avoid collisions . While such a level of synchronisation is not possible here , some lower level of synchronisation is still used to avoid collisions between queues . 2 Queuing Model . We consider a queuing system composed of N queues and K servers , associated with vectors of arrival and service rates λ , µ , where at each time step t = 1 , 2 , . . . , the following happens : • each queue i ∈ [ N ] receives a new packet with probability λi ∈ [ 0 , 1 ] , that is marked with the timestamp of its arrival time . If the queue currently has packet ( s ) on hold , it sends one of them to a chosen server j based on its past observations . • Each server j ∈ [ K ] attempts to clear the oldest packet it has received , breaking ties uniformly at random . It succeeds with probability µj ∈ [ 0 , 1 ] and otherwise sends it back to its original queue , as well as all other unprocessed packets . At each time step , a queue only observes whether or not the packet sent ( if any ) is cleared by the server . We note Qit the number of packets in queue i at time t. Given a packet-sending dynamics , the system is stable if , for each i in [ N ] , Qit/t converges to 0 almost surely . It is strongly stable , if for any r , t ≥ 0 and i ∈ [ N ] , E [ ( Qit ) r ] ≤ Cr , where Cr is an arbitrarily large constant , depending on r but not t. Without ambiguity , we also say the policy or the queues are ( strongly ) stable . Naturally , a strongly stable system is also stable [ Gaitonde and Tardos , 2020a ] . In the following , x ( i ) will denote the i-th order statistics of a vector x , i.e. , λ ( 1 ) ≥ λ ( 2 ) ≥ . . . ≥ λ ( N ) and µ ( 1 ) ≥ . . . ≥ µ ( K ) . Without loss of generality , we assume K ≥ N ( otherwise , we simply add fictitious servers with 0 service rate ) . The key quantity of a system is its slack , defined as the largest real number η such that : k∑ i=1 µ ( i ) ≥ η k∑ i=1 λ ( i ) , ∀ k ≤ N. We also denote by P ( [ K ] ) the set of probability distributions on [ K ] and by ∆ the margin of the system defined by ∆ : = min k∈ [ N ] 1 k k∑ i=1 ( µ ( i ) − λ ( i ) ) . ( 1 ) Notice that the alternative system where λ̃i = λi + ∆ and µ̃k = µk has a slack 1 . In that sense , ∆ is the largest margin between service and arrival rates that all queues can individually get in the system . Note that if η > 1 , then ∆ > 0 . We now recall existing results for this problem , summarized in Figure 1 below . Theorem 2 ( Marshall et al . 1979 ) . For any instance , there exists a strongly stable centralized policy if and only if η > 1 . Theorem 3 ( Gaitonde and Tardos 2020a , informal ) . If η > 2 , queues following appropriate no regret strategies are strongly stable . For each N > 0 , there exists a system and a dynamic s.t . η > 2 − o ( 1/N ) , all queues follow appropriate no-regret strategies , but they are not strongly stable . In the above theorem , an appropriate no regret strategy is a strategy such that there exists a partitioning of the time into successive windows , for which the incurred regret is o ( w ) with high probability on any window of length w. This for example includes the EXP3.P.1 algorithm [ Auer et al. , 2002 ] where the k-th window has length 2k . The patient queuing game G = ( [ N ] , ( ci ) ni=1 , µ , λ ) is defined as follows . The strategy space for each queue is P ( [ K ] ) . Let p−i ∈ ( P ( [ K ] ) ) N−1 denote the vector of fixed distributions for all queues over servers , except for queue i . The cost function for queue i is defined as : ci ( pi , p−i ) = lim t→+∞ T it /t , where T it is the age of the oldest packet in queue i at time t. Bounding T i t is equivalent to boundingQ i t. Theorem 4 ( Gaitonde and Tardos 2020b , informal ) . If η > ee−1 , any Nash equilibrium of the patient game G is stable .

Summary: This paper studies a decentralized learning problem for queueing systems. In my opinion, the paper is difficult to follow. I could not understand the problem setup and the paper’s contributions. Also, the article has a strong economics component which I am not familiar with.