Optimal Routing to Parallel Servers With Unknown Utilities—Multi-Armed Bandit With Queues

We consider the optimal routing problem in a discrete-time system with a job dispatcher connected to <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> parallel servers. At every time slot, the job dispatcher sends the incoming jobs to a server for execution, with each server having a queue that stores the jobs. The arrival process of incoming jobs, and the service processes of the servers are stochastic with unknown and possibly heterogeneous rates. Each server <inline-formula> <tex-math notation="LaTeX">$s_{m}$ </tex-math></inline-formula> is associated with an underlying utility <inline-formula> <tex-math notation="LaTeX">$v_{m}$ </tex-math></inline-formula> that is initially unknown. Whenever server <inline-formula> <tex-math notation="LaTeX">$s_{m}$ </tex-math></inline-formula> completes a job, a utility of <inline-formula> <tex-math notation="LaTeX">$v_{m}$ </tex-math></inline-formula> is obtained and a noisy observation of <inline-formula> <tex-math notation="LaTeX">$v_{m}$ </tex-math></inline-formula> is received. The goal is to design a policy that makes routing decisions to maximize the total utility obtained by the end of a finite time horizon <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula>. The performance of policies is measured in terms of regret, which is the additive difference between the expected total utility obtained by the policy and the supremum of the expected total utility over all the policies. The optimal routing problem can be interpreted as a problem of multi-armed bandit with queues where each server is viewed as an arm and the completion of a job is viewed as a pull of an arm. The key distinction between the optimal routing problem and traditional multi-armed bandit problems is in the queueing dynamics at the server, which arises due to the stochastic nature of the arrival and service processes. Our results combine techniques from control of stochastic queueing systems and stochastic multi-armed bandits to provide insights to the design and analysis of policies for the optimal routing problem. We first present analytical bounds that link the regret to the utilization and queue length of servers. Next, we start by assuming that the ordering of the underlying utilities is known and introduce the Priority-<inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> routing policy which makes priority-based routing decisions that send the incoming jobs to the server of the highest underlying utility with queue length no larger than a threshold <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula>. We prove that Priority-<inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> achieves <inline-formula> <tex-math notation="LaTeX">$O(\log T)$ </tex-math></inline-formula>-regret with an appropriately chosen <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula>. Next, removing the assumption of known utility ordering, we propose the Upper-Confidence Priority-<inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> policy, which essentially combines the Priority-<inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> policy with the ordering based on the upper-confidence bounds of the underlying utilities, and establish that the Upper-Confidence Priority-<inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> policy achieves an instance-dependent <inline-formula> <tex-math notation="LaTeX">$O(\log ^{3} T)$ </tex-math></inline-formula>-regret. Finally, we extend our results to the a generalized version of the optimal routing problem with multiple job dispatchers in a bipartite network. Our theoretical results are also validated by simulations.