$\texttt{LeadCache}$: Regret-Optimal Caching in Networks

1 Introduction . We consider an online structured learning problem , called Bipartite Caching , that lies at the core of many large-scale internet services , including Content Distribution Networks ( CDN ) and Cloud Computing . Formally , a set I of n users is connected to a set J of m caches via a bipartite network G ( I ⊍J , E ) . Each cache is connected to at most d users , and each user is connected to at most ∆ caches ( see Figure 1 ( b ) ) . There is a catalog consisting of N unique files , and each of the m caches can host at most C files at a time ( in practice , C ≪ N ) . The system evolves in discrete time slots . Each of the n users may request any file from the catalog at each time slot . The file requests could be dictated by an adversary . Given the storage capacity constraints , an online caching policy decides the files to be cached on different caches at each slot before the requests for that slot arrive . The objective is to maximize the total number of hits by the unknown incoming requests by coordinating the caching decisions among multiple caches in an online fashion . The Bipartite Caching problem is a strict generalization of the online k-sets problem that predicts a set of k items at each round so that the predicted set includes the item chosen by the adversary [ Koolen et al. , 2010 , Cohen and Hazan , 2015 ] . However , unlike the k-sets problem , which predicts a single subset at a time , in this problem , we are interested in sequentially predicting multiple subsets , each corresponding to one of the caches . The interaction among the caches through the non-linear reward function makes this problem challenging . ∗Work done at the Indian Institute of Technology Madras as a part of the first author ’ s Master ’ s thesis . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The Bipartite Caching problem is a simplified abstraction of the more general Network Caching problem central to the commercial CDNs , such as Akamai [ Nygren et al. , 2010 ] , Amazon Web Services ( AWS ) , and Microsoft Azure [ Paschos et al. , 2020 ] . In the Network Caching problem , one is given an arbitrary graph G ( V , E ) , a set of users I ⊆ V , and a set of caches J ⊆ V . A user can retrieve a file from a cache only if the cache hosts the requested file . If the ith user retrieves the requested file from the jth cache , the user receives a reward of rij ≥ 0 for that slot . If the requested file is not hosted in any of the caches reachable to the user , the user receives zero rewards for that slot . The goal of a network caching policy is to dynamically place files on the caches so that cumulative reward obtained by all users is maximized . The Network Caching problem reduces to the Bipartite Caching problem when the rewards are restricted to the set { 0,1 } . It will be clear from the sequel that the algorithms presented in this paper can be extended to the general Network Caching problem as well . 1.1 Problem Formulation . Denote the file requested by the ith user by the one-hot encoded N -dimensional vector xit . In other words , xitf = 1 if the i th user requests file f ∈ [ N ] at time slot t , or xitf = 0 otherwise . Since a user may request at most one file per time slot , we have : ∑Nf=1 x i tf ≤ 1 , ∀i ∈ I , ∀t . An online caching policy prefetches files on the caches at every time slot based on past requests . Unlike classical caching policies , such as LRU , LFU , FIFO , Marker , that fetch a file immediately upon a cache-miss , we do not enforce this constraint in the problem statement . The set of files placed on the jth cache at time t is represented by the N -dimensional incidence vector yjt ∈ { 0,1 } N . In other words , yjtf = 1 if the jth cache hosts file f ∈ [ N ] at time t , or yjtf = 0 otherwise . Due to cache capacity constraints , the following inequality must be satisfied at each time slot t : ∑Nf=1 y j tf ≤ C , ∀j ∈ J . The set of all admissible caching configurations , denoted by Y ⊆ { 0,1 } Nm , is dictated by the cache capacity constraints . In principle , the caching policy is allowed to replace all elements of the caches at every slot , incurring a potentially huge downloading cost over an interval . However , in Section 4 , we show that the total number of files fetched to the caches under the proposed LeadCache policy remains almost surely finite under very mild assumptions on the file request process . The ith user receives a cache hit at time slot t if and only if any of the caches connected to the ith user hosts the file requested by the user at slot t. In the case of a cache hit , the user obtains a unit reward . On the other hand , in the case of a cache miss , the user receives zero rewards for that slot . Hence , for a given aggregate request vector from all users xt = ( xit , i ∈ I ) and the aggregate cache configuration vector of all caches yt = ( y j t , j ∈ J ) , the total reward q ( xt , yt ) obtained by the users at time t may be expressed as follows : q ( xt , yt ) ≡∑ i∈I xit ⋅min { 1N×1 , ( ∑ j∈∂+ ( i ) yjt ) } , ( 1 ) where a ⋅b denotes the inner-product of the vectors a and b , 1N×1 denotes the N -dimensional all-one column vector , the set ∂+ ( i ) denotes the set of all caches connected to the ith user , and the “ min '' operator is applied component wise . The total reward Q ( T ) accrued in a time-horizon of length T is obtained by summing the slot-wise rewards , i.e. , Q ( T ) = ∑Tt=1 q ( xt , yt ) . Following the standard practice in the online learning literature , we measure the performance of any online policy π using the notion of ( static ) regret Rπ ( T ) , defined as the maximum difference in the cumulative rewards obtained by the optimal fixed caching-configuration in hindsight and that of the online policy π , i.e. , Rπ ( T ) ( def . ) = sup { xt } Tt=1 ( T ∑ t=1 q ( xt , y ∗ ) − T ∑ t=1 q ( xt , y π t ) ) , ( 2 ) where y∗ is the best static cache-configuration in hindsight for the file request sequence { xt } Tt=1 , i.e. , y∗ = arg maxy∈Y ∑ T t=1 q ( xt , y ) . We assume that the file request sequence is generated by an oblivious adversary , i.e. , the entire request sequence { xt } t≥1 is fixed a priori . Note that the problem is non-convex , as we seek binary cache allocations . With an eye towards efficient implementation , later we will also consider the problem of designing efficient policies that guarantee a sub-linear α-regret for a suitable value of α < 1 [ Garber , 2021 , Kakade et al. , 2009 , Fujita et al. , 2013 ] . 2 Background and Related Work . Online Linear Optimization ( OLO ) is a canonical online learning problem that can be formulated as a repeated game played between a learner ( also known as the forecaster ) and an adversary [ CesaBianchi and Lugosi , 2006 ] . In this model , at every time slot t , the policy selects an action yt from a feasible set Y ⊆ Rd . After that , the adversary reveals a reward vector xt from a set X ⊆ Rd . The adversary is assumed to be oblivious , i.e. , the sequence of reward vectors is fixed before the game begins . With the above choices , the policy receives a scalar reward q ( xt , yt ) ∶= ⟨xt , yt⟩ at slot t. A classic objective in this setting is to design a policy with a small regret . Follow the Perturbed Leader ( FTPL ) , is a well-known online policy for the OLO problem [ Hannan , 1957 ] . At time slot t , the FTPL policy adds a random noise vector γt to the cumulative reward vector Xt = ∑t−1τ=1 xτ , and then selects the best action against this perturbed reward , i.e. , yt ∶= arg maxy∈Y⟨Xt + γt , y⟩ . See Abernethy et al . [ 2016 ] for a unifying treatment of the FTPL policies through the lens of stochastic smoothing . A large number of papers on caching assume some stochastic model for the file request sequence , e.g. , Independent Reference Model ( IRM ) and Shot Noise Model ( SNM ) [ Traverso et al. , 2013 ] . Classic page replacement algorithms , such as MIN , LRU , LFU , and FIFO , are designed to minimize the competitive ratio with adversarial requests [ Borodin and El-Yaniv , 2005 , Van Roy , 2007 , Lee et al. , 1999 , Dan and Towsley , 1990 ] . These algorithms , being non-prefetching in nature , replace a page on demand upon a cache-miss . However , since the competitive ratio metric is multiplicative in nature , there can be a large gap between the hit ratio of a competitively optimal policy and the optimal offline policy . To design better algorithms , the caching problem has recently been investigated through the lens of regret minimization with prefetching policies that learn from the past request sequence [ Vitter and Krishnan , 1996 , Krishnan and Vitter , 1998 ] . Daniely and Mansour [ 2019 ] considered the problem of minimizing the regret plus the switching cost for a single cache . The authors proposed a variant of the celebrated exponential weight algorithm [ Littlestone and Warmuth , 1994 , Freund and Schapire , 1997 ] that ensures the minimum competitive ratio and a small but sub-optimal regret . The Bipartite Caching model was first proposed in a pioneering paper by Shanmugam et al . [ 2013 ] , where they considered a stochastic version of the problem with known file popularities . Paschos et al . [ 2019 ] proposed an Online Gradient Ascent ( OGA ) -based Bipartite Caching policy that allows caching a fraction of the Maximum Distance Separable ( MDS ) -coded files . Closely related to this paper is the recent work by Bhattacharjee et al . [ 2020 ] , where the authors designed a regret-optimal single-cache policy and a Bipartite Caching policy for fountain-coded files . However , the fundamental problem of designing a regret-optimal uncoded caching policy for the Bipartite Caching problem was left as an open problem . Why standard approaches fail : A straightforward way to formulate the Bipartite Caching problem is to pose it as an instance of the classic Prediction with Expert Advice problem [ CesaBianchi and Lugosi , 2006 ] , where each of the possible ( N C ) m cache configurations is treated as experts . However , this approach is computationally infeasible due to the massive number of resulting experts . Furthermore , the expected FTPL policy for convex losses in Hazan [ 2019 ] and its sampled version in Hazan and Minasyan [ 2020 ] are both computationally intensive . In view of the above challenges , we now present our main technical contributions .

Summary: The paper studies the Bipartite Caching problem, from an online learning perspective. A bipartite graph describes how n users are connected to m caches, each of which supports C files. At each time t, N files are distributed across m caches (same file can be placed in multiple caches), and each user requests one file. Every cache-hit (a user that requests a file in one of its caches) counts as a +1 towards the total reward. The goal is to maximize the reward (or minimize the regret, which is the difference with respect to the reward of an optimal policy).  The file demands are assumed to be chosen by an "oblivious" adversary, which means that the file demands for the entire time horizon T are chosen by an adversary, but a priori (i.e., before seeing the cache allocations).  The paper proposes a cache allocation algorithm called LeadCache. The algorithm is based on the "follow the perturbed leader" policy. In essence, at time t, we aggregate all previous user-file requests until time t-1, add Gaussian noise to them, and then perform the file allocation to caches as if the resulting perturbed vector was the current file demand. The paper builds on previous works that study the problem of Online Linear Optimization and provide regret bounds for "follow the perturbed leader" algorithms. However, in order to apply these ideas in the bipartite caching context, several new ideas need to be introduced. In particular, the file allocation for time t is found by first solving a linear relaxation of the combinatorial optimization problem, and then using an interesting rounding algorithm, for which a performance guarantee can be proven.  The paper provides a careful bound on the regret achieved by LeadCache and shows that it is at most a O(n^(3/8)) factor away from a minimax lower bound. Furthermore, the paper analyzes the number of file fetches needed by LeadCache (i.e., how many times a cache needs to "download" a new file) and shows that, interestingly, the file fetches stop after a finite amount of time.