Title: Online Weighted Paging with Unknown Weights
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Learning

Online Weighted Paging

Online Algorithms

Competitive Ratio

Regret

Theory
Abstract: Online paging is a fundamental problem in the field of online algorithms, in which one maintains a cache of $k$ slots as requests for fetching pages arrive online. 
In the weighted variant of this problem, each page has its own fetching cost; a substantial line of work on this problem culminated in an (optimal) $O(\log k)$-competitive randomized algorithm, due to Bansal, Buchbinder and Naor (FOCS'07).

Existing work for weighted paging assumes that page weights are known in advance, which is not always the case in practice.
For example, in multi-level caching architectures, the expected cost of fetching a memory block is a function of its probability of being in a mid-level cache rather than the main memory.
This complex property cannot be predicted in advance; over time, however, one may glean information about page weights through sampling their fetching cost multiple times.

We present the first algorithm for online weighted paging that does not know page weights in advance, but rather learns from weight samples.
In terms of techniques, this requires providing (integral) samples to a fractional solver, requiring a delicate interface between this solver and the randomized rounding scheme; we believe that our work can inspire online algorithms to other problems that involve cost sampling.

Online Weighted Paging with Unknown Weights

Abstract

Online paging is a fundamental problem in the field of online algorithms, in which one maintains a cache of 𝑘 slots as requests for fetching pages arrive online. In the weighted variant of this problem, each page has its own fetching cost; a substantial line of work on this problem culminated in an (optimal) 𝑂 ( log 𝑘 ) -competitive randomized algorithm, due to Bansal, Buchbinder and Naor (FOCS'07).

Existing work for weighted paging assumes that page weights are known in advance, which is not always the case in practice. For example, in multi-level caching architectures, the expected cost of fetching a memory block is a function of its probability of being in a mid-level cache rather than the main memory. This complex property cannot be predicted in advance; over time, however, one may glean information about page weights through sampling their fetching cost multiple times. We present the first algorithm for online weighted paging that does not know page weights in advance, but rather learns from weight samples. In terms of techniques, this requires providing (integral) samples to a fractional solver, requiring a delicate interface between this solver and the randomized rounding scheme; we believe that our work can inspire online algorithms to other problems that involve cost sampling.

1 Introduction

Online weighted paging. In the online weighted paging problem, or OWP, one is given a cache of 𝑘 slots, and requests for pages arrive online. Upon each requested page, the algorithm must ensure that the page is in the cache, possibly evicting existing pages in the process. Each page 𝑝 also has a weight 𝑤 𝑝 , which represents the cost of fetching the page into the cache; the goal of the algorithm is to minimize the total cost of fetching pages. Assuming that the page weights are known, this problem admits an 𝑂 ( log 𝑘 ) -competitive randomized online algorithm, due to Bansal, Buchbinder, and Naor [2010, 2012]; This is optimal, as there exists an Ω ( log 𝑘 ) -competitiveness lower bound for randomized algorithms due to Fiat et al. [1991] (that holds even for the unweighted case).

However, all previous work on paging assumes that the page weights are known in advance . This assumption is not always justified; for example, the following scenario, reminiscent of real-world architectures, naturally gives rise to unknown page weights. Consider a multi-core architecture, in which data can be stored in one of the following: a local 'L1' cache, unique to each core; a global 'L2' cache, shared between the cores; and the (large but slow) main memory. As a specific core requests memory blocks, managing its L1 cache can be seen as an OWP instance. Suppose the costs

Noam Touitou

of fetching a block from the main memory and from the L2 cache are 1 and 𝜖 ≪ 1 , respectively. Then, when a core demands a memory block, the expected cost of fetching this block (i.e., its weight) is a convex combination of 1 and 𝜖 , weighted by the probability that the block is in the L2 cache; this probability can be interpreted as the demand for this block by the various cores. When managing the L1 cache of a core, we would prefer to evict blocks with low expected fetching cost, as they are more likely to be available in the L2 cache. But, this expected cost is a complicated property of the computation run by the cores, and estimating it in advance is infeasible; however, when a block is fetched in the above example, we observe a stochastic cost of either 1 or 𝜖 . As we sample a given block multiple times, we can gain insight into its weight.

Multi-armed bandit. The above example, in which we learn about various options through sampling, is reminiscent of the multi-armed bandit problem, or MAB. In the cost-minimization version of this problem, one is given 𝑛 options (or arms), each with its own cost in [ 0 , 1 ] . At each time step, the algorithm must choose an option and pay the corresponding cost; when choosing an option 𝑝 , rather than learning its cost 𝑤 𝑝 , the algorithm is only revealed a sample from some distribution whose expectation is 𝑤 𝑝 . In this problem, the goal is to minimize the regret , which is the difference between the algorithm's total cost and the optimal cost (which is to always choose the cheapest option). Over 𝑇 time steps, the best known regret bound for this problem is ˜ 𝑂 ( √ 𝑛𝑇 ) , achieved through multiple techniques. (See, e.g., Slivkins et al. [2019], Lattimore and Szepesvári [2020]).

1.1 Our Results

We make the first consideration of OWP where page weights are not known in advance, and show that the optimal competitive ratio of 𝑂 ( log 𝑘 ) can still be obtained. Specifically, we present the problem of OWP-UW (Online Weighted Paging with Unknown Weights), that combines OWP with bandit-like feedback. In OWP-UW, every page 𝑝 has an arbitrary distribution, whose expectation is its weight 0 < 𝑤 𝑝 ≤ 1 . Upon fetching a page, the algorithm observes a random, independent sample from the distribution of the page. We present the following theorem for OWP-UW.

Theorem 1.1. There exists a randomized algorithm ON for OWP-UW such that, for every input 𝑄 ,

√

where ON ( 𝑄 ) is the cost of ON on 𝑄 , OPT ( 𝑄 ) is the cost of the optimal solution to 𝑄 , and the expectation is taken over both the randomness in ON and the samples from the distributions of pages.

Note that the bound in Theorem 1.1 combines a competitive ratio of 𝑂 ( log 𝑘 ) with a regret (i.e., additive) term of ˜ 𝑂 ( √ 𝑛𝑇 ) . To motivate this type of bound, we observe that OWP-UW does not admit sublinear regret without a competitive ratio. Consider the lower bound of Ω ( log 𝑘 ) for the competitive ratio of paging; stated simply, one of 𝑘 + 1 pages of weight 1 is requested at random. Over a sequence of 𝑇 requests, the expected cost of any online algorithm is Ω ( 𝑇 / 𝑘 ) ; meanwhile, the expected cost of the optimal solution is at most 𝑂 ( 𝑇 /( 𝑘 log 𝑘 )) . (The optimal solution would be to wait for a maximal phase of requests containing at most 𝑘 pages, whose expected length is Θ ( 𝑘 log 𝑘 ) , then change state at constant cost.) Without a competitive ratio term, the difference between the online and offline solutions is Ω ( 𝑇 / 𝑘 ) , i.e., linear regret. We note that this kind of bound appears in several previous works such as Basu et al. [2019], Foussoul et al. [2023]. As OWP-UW generalizes both standard OWP and MAB, both the competitive ratio and regret terms are asymptotically tight: a competitiveness lower bound of Ω ( log 𝑘 ) is known for randomized algorithms for online (weighted) paging [Fiat et al., 1991], and a regret lower bound of ˜ Ω ( √ 𝑛𝑇 ) is known for MAB [Lattimore and Szepesvári, 2020].

1.2 Our Techniques

Interface between fractional solution and rounding scheme. Randomized online algorithms are often built of the following components:

A deterministic, 𝛼 -competitive online algorithm for a fractional relaxation of the problem.

An online randomized rounding scheme that encapsulates any online fractional algorithm, and has expected cost 𝛽 times the fractional cost.

Combining these components yields an 𝛼𝛽 -competitive randomized online (integral) algorithm.

For our problem, it is easy to see where this common scheme fails. The fractional algorithm cannot be competitive without sampling pages; but, pages are sampled by the rounding scheme! Thus, the competitiveness of the fractional algorithm is not independent of the randomized rounding, which must provide samples. One could think of addressing this by feeding any samples obtained by the rounding procedure into the fractional algorithm. However, as the rounding is randomized, this would result in a non-deterministic fractional algorithm. As described later in the paper, this is problematic: the rounding scheme demands a globally accepted fractional solution against which probabilities of cache states are balanced.

Instead, we outline a sampling interface between the fractional solver and the rounding scheme. Once the total fractional eviction of a page reaches an integer, the fractional algorithm will pop a sample of the page from a designated sampling queue, and process that sample. On the other side of the interface, the rounding scheme fills the sampling queue and ensures that when the fractional algorithm demands a sample, the queue will be non-empty with probability 1 .

Optimistic fractional algorithm, pessimistic rounding scheme. When learning from samples, one must balance the exploration of unfamiliar options and the exploitation of familiar options that are known to be good. A well-known paradigm for achieving this balance in multi-armed bandit problems is optimism under uncertainty . Using this paradigm to minimize total cost, one maintains a lower confidence bound (LCB) for the cost of an option, which holds with high probability, and tightens upon receiving samples; then, the option with the lowest LCB is chosen. As a result, one of the following two cases holds: either the option was good (high exploitation); or, the option was bad, which means that the LCB was not tight, and henceforth sampling greatly improves it (high exploration).

Our fractional algorithm for weighted paging employs this method. It optimistically assumes that the price of moving a page is cheap, i.e., is equal to some lower confidence bound (LCB) for that page. It then uses multiplicative updates to allocate servers according to these LCB costs. The optimism under uncertainty paradigm then implies that the fractional algorithm learns the weights over time.

However, the rounding scheme behaves very differently. Unlike the fractional algorithm, the (randomized) rounding scheme is not allowed to use samples to update the confidence bounds; otherwise, our fractional solution would behave non-deterministically. Instead, the rounding scheme takes a pessimistic view: it uses an upper confidence bound (UCB) as the cost of a page, thus assuming that the page is expensive. Such pessimistic approaches are common in scenarios where obtaining additional samples is not possible (e.g., offline reinforcement learning [Levine et al., 2020]), but rarely appear as a component of an online algorithm as we suggest in this paper.

1.3 Related Work

The online paging problem is a fundamental problem in the field of online algorithms. In the unweighted setting, the optimal competitive ratio for a deterministic algorithm is 𝑘 , due to Sleator and Tarjan [1985]. Allowing randomization improves the best possible competitive ratio to Θ ( log 𝑘 ) [Fiat et al., 1991]. As part of a line of work on weighted paging and its variants (e.g., Young [1994], Manasse et al. [1990], Albers [2003], Irani [2002], Fiat and Mendel [2000], Bansal et al. [2008], Irani [1997]), the best competitive ratios for weighted paging were settled, and were seen to match the unweighted setting: 𝑘 -competitiveness for deterministic algorithms, due to Chrobak et al. [1991]; and Θ ( log 𝑘 ) -competitiveness for randomized algorithms, due to Bansal et al. [2012].

Online (weighted) paging is a special case of the 𝑘 -server problem, in which 𝑘 servers exist in a general metric space, and must be moved to address requests on various points in this space; the cache slots in (weighted) paging can be seen as servers, moving in a (weighted) uniform metric space. The Θ ( 𝑘 ) bound on optimal competitiveness in the deterministic for paging also extends to general 𝑘 -server [Manasse et al., 1990, Koutsoupias and Papadimitriou, 1995]. However, allowing randomization, a recent breakthrough result by Bubeck et al. [2023] was a lower bound of Ω ( log 2 𝑘 ) -competitiveness for 𝑘 -server, diverging from the 𝑂 ( log 𝑘 ) -competitiveness possible for paging.

Multi-Armed Bandit (MAB) is one of the most fundamental problems in online sequential decision making, often used to describe a trade-off between exploration and exploitation. It was extensively studied in the past few decades, giving rise to several algorithmic approaches that guarantee optimal regret. The most popular methods include Optimism Under Uncertainty (e.g., the UCB algorithm [Lai and Robbins, 1985, Auer et al., 2002a]), Action Elimination [Even-Dar et al., 2006], Thompson

Sampling [Thompson, 1933, Agrawal and Goyal, 2012] and Exponential Weights (e.g., the EXP3 algorithm [Auer et al., 2002b]). For a comprehensive review of the MAB literature, see Slivkins et al. [2019], Lattimore and Szepesvári [2020].

2 Preliminaries

In OWP-UW, we are given a memory cache of 𝑘 slots. A sequence of 𝑇 page requests then arrives in an online fashion; we denote the set of requested pages by 𝑃 , define 𝑛 : = | 𝑃 | , and assume that 𝑛 > 𝑘 . Each page 𝑝 has a corresponding weight 0 < 𝑤 𝑝 ≤ 1 ; the weights are not known to the algorithm. Moreover, every page 𝑝 has a distribution D 𝑝 supported in ( 0 , 1 ] , such that E 𝑥 ∼D 𝑝 [ 𝑥 ] = 𝑤 𝑝 .

The online scenario proceeds in 𝑇 rounds. 3 In each round 𝑡 ∈ { 1 , 2 , . . . , 𝑇 } :

A page 𝑝 𝑡 ∈ 𝑃 is requested.

If the requested page is already in the cache, then it is immediately served.

Otherwise, we experience a cache miss, and we must fetch 𝑝 𝑡 into the cache; if the cache is full, the algorithm must evict some page from the cache to make room for 𝑝 𝑡 .

Upon evicting any page 𝑝 from the cache, the algorithm receives an independent sample from D 𝑝 .

The algorithm incurs cost when evicting pages from the cache: when evicting a page 𝑝 , the algorithm incurs a cost of 𝑤 𝑝 4 . Our goal is to minimize the algorithm's total cost of evicting pages, denoted by ON, and we measure our performance by comparison to the total cost of the optimal algorithm, denoted by OPT. We say that our algorithm is 𝛼 -competitive with R regret if E [ ON ] ≤ 𝛼 · OPT +R .

3 Algorithmic Framework and Analysis Overview

We present an overview of the concepts and algorithmic components we use to address OWP-UW. We would like to follow the paradigm of solving a fractional problem online, and then randomly rounding the resulting solution; however, as discussed in the introduction, employing this paradigm for OWP-UW requires a well-defined interface between the fractional solver and the rounding procedure. Thus, we present a fractional version of OWP-UW that captures this interface.

Fractional OWP-UW. In fractional OWP-UW, one is allowed to move fractions of servers, and a request for a page is satisfied if the total server fraction at that point sums to 1 . More formally, for every page 𝑝 ∈ 𝑃 we maintain an amount 𝑦 𝑝 ∈ [ 0 , 1 ] which is the fraction of 𝑝 missing from the cache; we call 𝑦 𝑝 the fractional anti-server at 𝑝 . (The term anti-server comes from the related 𝑘 -server problem.) The feasibility constraints are:

At any point in the algorithm, it holds that ˝ 𝑝 ∈ 𝑃 𝑦 𝑝 ≥ 𝑛 -𝑘 . (I.e., the total number of pages in the cache is at most 𝑘 .)

After a page 𝑝 is requested, it holds that 𝑦 𝑝 = 0 . (I.e., there exists a total server fraction of 1 at 𝑝 .)

Evicting an 𝜖 server fraction from 𝑝 (i.e., increasing 𝑦 𝑝 by 𝜖 ) costs 𝜖 · 𝑤 𝑝 .

Sampling. The fractional algorithm must receive samples of pages over time in order to learn about their weights. An algorithm for fractional OWP-UW receives a sample of a page 𝑝 whenever the total fraction of 𝑝 evicted by the algorithm reaches an integer . In particular, the algorithm obtains the first sample of 𝑝 (corresponding to 0 eviction) when 𝑝 is first requested in the online input.

Algorithmic components. We present the fractional algorithm and randomized rounding scheme.

Fractional algorithm. In Section 4, we present an algorithm ONF for fractional OWP-UW. Fixing the random samples from the pages' weight distributions, the fractional algorithm ONF is deterministic. For every page 𝑝 ∈ 𝑃 , the fractional algorithm maintains an upper confidence bound UCB 𝑝 and a lower confidence bound LCB 𝑝 . These confidence bounds depend on the samples provided for that page; we define the good event E to be the event that at every time and for every page 𝑝 ∈ 𝑃 , it holds that LCB 𝑝 ≤ 𝑤 𝑝 ≤ UCB 𝑝 . We later show that E happens with high probability, and analyze the complementary event separately 5 . Thus, we henceforth focus on the good event.

The following lemma bounds the cost of ONF subject to the good event. In fact, it states a stronger bound, that applies also when the cost of evicting page 𝑝 is the upper confidence bound UCB 𝑝 ≥ 𝑤 𝑝 .

Lemma 3.1. Fixing any input 𝑄 for fractional OWP-UW , and assuming the good event, it holds that

√

where ONF is the cost of the algorithm on the input where the cost of evicting a page 𝑝 is UCB 𝑝 ≥ 𝑤 𝑝 .

Randomized rounding. In Section 5 we present the randomized algorithm ON for (integral) OWP-UW. It maintains a probability distribution over integral cache states by holding an instance of ONF, to which it feeds the online input. For the online input to constitute a valid fractional input, the randomized algorithm ensures that samples are provided to ONF when required. In addition, the randomized algorithm makes use of ONF's exploration of page weights; specifically, it uses the UCBs calculated by ONF.

Lemma 3.2. Fixing any input 𝑄 for (integral) OWP-UW , assuming the good event E , it holds that

where ONF ( 𝑄 ) is the cost of the algorithm on 𝑄 such that the cost of evicting a page 𝑝 is UCB 𝑝 ≥ 𝑤 𝑝 .

Figure 1 provides a step-by-step visualization of the interface between the fractional algorithm and the rounding scheme over the handling of a page request.

4 Algorithm for Fractional OWP-UW

We now describe our algorithm for the fractional relaxation of OWP-UW, proving Lemma 3.1. Our fractional algorithm, presented in Algorithm 1 below, uses samples provided by the rounding scheme to learn the weights. A new sample for page 𝑝 is provided and processed whenever the sum of fractional movements (in absolute value) 𝑚 𝑝 hits a natural number. (At this point the number of samples 𝑛 𝑝 is incremented.) The algorithm calculates non-increasing UCBs and non-decreasing LCBs that will be specified later in Section C and guarantee with high probability, for every page 𝑝 ∈ 𝑃 and time step 𝑡 ∈ [ 1 , 𝑇 ] , LCB 𝑝 ≤ 𝑤 𝑝 ≤ UCB 𝑝 .

At each time step 𝑡 , upon a new page request 𝑝 𝑡 , the algorithm updates its feasible fractional cache solution { 𝑦 𝑝 } 𝑝 ∈ 𝑃 . The fractions are computed using optimistic estimates of the weights, i.e., the LCBs, in order to induce exploration and allow the true weights to be learned over time. After serving page 𝑝 𝑡 (that is, setting 𝑦 𝑝 𝑡 = 0 ), the algorithm continuously increases the anti-servers of all the other pages in the cache until feasibility is reached (that is, until ˝ 𝑝 ∈ 𝑃 𝑦 𝑝 = 𝑛 -𝑘 ). The fraction 𝑦 𝑝 for some page 𝑝 in the cache is increased proportionally to 𝑦 𝑝 + 𝜂 LCB 𝑝 , which is our adaption of the algorithmic approach of Bansal et al. [2010] to the unknown-weights scenario. Finally, to fulfil its end in the interface, the fractional algorithm passes its feasible fractional solution to the rounding scheme together with pessimistic estimates of the weights, i.e, the UCBs.

4.1 Analysis

In this analysis section, our goal is to bound the amount ONF with respect to the UCBs and LCBs calculated by the algorithm; i.e., to prove Lemma 4.1. Lemma C.1 and Lemma C.2 from Appendix C then make the choice of confidence bounds concrete, such that combining it with Lemma 4.1 yields the final bound for the fractional algorithm, i.e., Lemma 3.1.

This figure visualizes the running of the algorithm over a request in the input. (a) shows the state prior to the arrival of the request. The integral algorithm maintains an instance of the fractional algorithm ONF, as well as a distribution over integral cache states that upholds some properties w.r.t. ONF (specifically, the consistency property and the subset property); note that these properties are a function of both the anti-server state GLYPH<8> 𝑦 𝑝 GLYPH<9> 𝑝 ∈ 𝑃 and the upper-confidence bounds GLYPH<8> UCB 𝑝 GLYPH<9> 𝑝 ∈ 𝑃 in ONF. The integral algorithm also maintains a set of page samples, to be demanded by ONF at a later time. In (b), a page is requested (in red); thus, the fractional algorithm must fetch it into the cache, i.e., set its anti-server to 0 . To maintain feasibility, the fractional algorithm will increase the anti-server at other pages in which some server fraction exists (in green). These changes in anti-server are also fed into the integral algorithm, which modifies its distribution to maintain consistency and the subset property w.r.t. ONF. In (c), ONF reaches integral total eviction of a page 𝑝 , and demands a sample from the integral algorithm. (We show that such a sample always exists when demanded.) The bound UCB 𝑝 is updated in ONF, and is then fed to the integral algorithm to maintain the desired properties. After this sample, continuous increasing of anti-server continues until feasibility is reached in (d). Then, the integral algorithm ensures that a sample exists for the requested page 𝑝 𝑡 , sampling the page if needed. (As 𝑝 𝑡 now exists in 𝜇 with probability 1 , sampling is done through evicting and re-fetching 𝑝 𝑡 .)

where (a) ONF is the cost of the algorithm on the input such that the cost of evicting a page 𝑝 is UCB 𝑝 ≥ 𝑤 𝑝 , and (b) UCB 𝑝,𝑖 , LCB 𝑝,𝑖 are the values of UCB 𝑝 and LCB 𝑝 calculated by the procedure UPDATECONFBOUNDS (found in Appendix C) immediately after processing the 𝑖 'th sample of 𝑝 , and (c) LCB 𝑝 is the value after the last sample of page 𝑝 was processed.

Proof sketch. We prove the lemma using a potential analysis. In Bansal et al. [2010], a potential function was introduced that encodes the discrepancy between the state of the optimal solution and the state of the algorithm. In our case, we require an additional term which can be viewed as a fractional exploration budget. This budget is 'recharged' upon receiving a sample; the cost of this recharging goes into a regret term. □

This figure visualizes a single step in REBALANCESUBSETS. Subfigure (a) shows the distribution of anti-cache states prior to this step; specifically, the 𝑥 axis is the probability measure, and the 𝑦 -axis is the number of pages of class 𝑖 and above in the anti-cache, i.e., 𝑚 : = | 𝑆 ∩ 𝑃 ≥ 𝑖 | . The red line is 𝑌 𝑖 , which through consistency, is the expectation of 𝑚 ; the blue dotted lines are thus the allowed values for 𝑚 , which are | ⌈ 𝑌 𝑖 ⌉ , ⌊ 𝑌 𝑖 ⌋| . The total striped area in the figure is the imbalance measure, formally defined in Definition B.2. Subfigure (b) shows a single rebalancing step; we choose the imbalanced anti-cache 𝑆 that maximizes | 𝑚 -𝑌 𝑖 | ; in our case, 𝑚 > 𝑌 𝑖 , and thus we match its measure with an identical measure of anti-cache states that are below the upper blue line, i.e., can receive a page without increasing imbalance. Then, a page of class 𝑖 is handed from 𝑆 to every matched state 𝑆 ′ ; note that every matched state might get a different page from 𝑆 , but some such page in 𝑆 \ 𝑆 ′ is proven to exist. Finally, Subfigure (c) shows the state after the page transfer; note the decrease in imbalance that results. The REBALANCESUBSETS procedure performs such steps until there is no imbalance; then, the procedure would advance to class 𝑖 -1 .

5 Randomized Rounding

This section describes a randomized algorithm for (integral) OWP-UW, which uses Algorithm 1 for fractional OWP-UW to maintain a probability distribution over valid integral cache states, while obtaining and providing page weight samples to Algorithm 1. The method in which the randomized algorithm encapsulates and tracks the fractional solution is inspired by Bansal et al. [2012], which maintains a balanced property over weight classes of pages. However, as the weights are unknown in our case, the classes are instead defined using the probabilistic bounds maintained by the fractional solution (i.e., the UCBs). But, these bounds are dynamic, and change over the course of the algorithm; the imbalance caused by these discrete changes increases exponentially during rebalancing, and thus requires a more robust rebalancing procedure.

Following the notation in the previous sections, we identify each cache state with the set of pages not in the cache. Observing the state of the randomized algorithm at some point in time, let 𝜇 ( 𝑆 ) be the probability that 𝑆 ⊆ 𝑃 is the set of pages missing from the cache, also called the anti-cache . For the algorithm to be a valid algorithm for OWP-UW, the cache can never contain more than 𝑘 pages; this is formalized in the following property.

Definition 5.1 (valid distribution) . A probability distribution 𝜇 is valid, if for any set 𝑆 ⊆ 𝑃 with 𝜇 ( 𝑆 ) > 0 it holds that | 𝑆 | ≥ 𝑛 -𝑘 .

Instead of maintaining the distribution's validity, we will maintain a stronger property that implies validity. This property is the balanced property, involving the UCBs calculated by the fractional algorithm.

For every page 𝑝 , we define the 𝑖 'th UCB class to be 𝑃 𝑖 : = GLYPH<8> 𝑝 ∈ 𝑃 : 6 𝑖 ≤ UCB 𝑝 < 6 𝑖 + 1 GLYPH<9> . (Note that UCB 𝑝 ∈ ( 0 , 1 ] .) We also define 𝑃 ≥ 𝑗 : = -𝑖 ≥ 𝑗 𝑃 𝑗 , the set of all pages that their UCB is at least 6 𝑗 .

Let { 𝑦 𝑝 } 𝑝 ∈ 𝑃 be the fractional solution. The balanced property requires that, for every set 𝑆 such that 𝜇 ( 𝑆 ) > 0 and every index 𝑗 , the number of pages in 𝑆 of class at least 𝑗 is the same as in the fractional solution, up to rounding. Formally, we define the balanced property as follows.

Definition 5.2 (balanced distribution) . A probability distribution 𝜇 has the balanced subsets property with respect to 𝑦 , if for any set 𝑆 ⊆ 𝑃 with 𝜇 ( 𝑆 ) > 0 , the following holds for all 𝑗 :

Algorithm 3: Rebalancing procedure for randomized algorithm

Choosing the minimum UCB class in Definition 5.2, and noting that ˝ 𝑝 ∈ 𝑃 𝑦 𝑝 ≥ 𝑛 -𝑘 through feasibility, immediately yields the following remark.

Remark 5.3 . Every balanced probability distribution is also a valid distribution.

To follow the fractional solution, we also demand that the distribution 𝜇 is consistent with the fractional solution, meaning, the marginal probability in 𝜇 that any page 𝑝 is missing from the cache must be equal to 𝑦 𝑝 .

Definition 5.4 (consistent distribution) . A probability distribution 𝜇 on subsets 𝑆 ⊆ 𝑃 is consistent with respect to a fractional solution GLYPH<8> 𝑦 𝑝 GLYPH<9> 𝑝 ∈ 𝑃 if for every page 𝑝 it holds that ˝ 𝑆 ⊆ 𝑃 | 𝑝 ∈ 𝑆 𝜇 ( 𝑆 ) = 𝑦 𝑝 .

In the following we describe the online maintenance of the distribution 𝜇 , that yields a distribution satisfying all of the above.

Algorithm overview. The randomized algorithm for OWP-UW is given in Algorithm 2. The algorithm encapsulates an instance of Algorithm 1 for fractional OWP-UW, called ONF. Upon a new page request 𝑝 𝑡 at round 𝑡 , the algorithm forwards this requests to ONF. As ONF makes changes to its fractional solution, the algorithm modifies its probability distribution accordingly to remain consistent and balanced (and hence also valid).

Upon any (infinitesimally small) change to a fractional variable, the algorithm first changes its distribution to maintain consistency: when the fractional algorithm ONF increases the variable 𝑦 𝑝 ′ of any page 𝑝 ′ by an 𝜖 -measure, the algorithm identifies an 𝜖 -measure of cache states 𝑆 ⊆ 𝑃 in which there is no anti-server at 𝑝 ′ and adds anti-server at 𝑝 ′ . The case in which the fractional algorithm decreases a variable is analogous.

However, this procedure may invalidate the balanced property. Specifically, for some class 𝑗 , letting 𝑌 𝑗 be the total anti-server fraction of pages of at least class 𝑗 in ONF, there might now be states with GLYPH<6> 𝑌 𝑗 GLYPH<7> + 1 or GLYPH<4> 𝑌 𝑗 GLYPH<5> -1 such pages in the anti-cache. Thus, the algorithm makes a call to REBALANCESUBSETS, which restores the balanced property class-by-class, in a descending order. For every class 𝑗 , the procedure repeatedly identifies a violating state 𝑆 where the number of pages of class ≥ 𝑗 in the anti-cache is not in GLYPH<8> GLYPH<6> 𝑌 𝑗 GLYPH<7> , GLYPH<4> 𝑌 𝑗 GLYPH<5> GLYPH<9> ; suppose it identifies such a state with more than GLYPH<6> 𝑌 𝑗 GLYPH<7> such pages (the case of less than GLYPH<4> 𝑌 𝑗 GLYPH<5> pages is analogous). The procedure seeks to move a page of class 𝑗 from this state to another state in a way that does not increase the 'imbalance' in class 𝑗 . Thus, the procedure identifies a matching measure of anti-cache states that contain at most GLYPH<6> 𝑌 𝑗 GLYPH<7> -1 such pages, and moves a page of class 𝑗 from 𝑆 to 𝑆 ′ , for every 𝑆 ′ in the matched measure; a visualization of the procedure is given in Figure 2. (The existence of this matching measure, as well as a page to move, are shown in the analysis.) In particular, note that the probability of every page being in the anti-cache remains the same, and thus REBALANCESUBSETS does not impact consistency.

Regarding samples, the algorithm can maintain a sample 𝑠 𝑝 for every page 𝑝 . A sample for 𝑝 is obtained upon a request for 𝑝 after 𝑝 is fetched with probability 1 into the cache, if no such sample already exists (i.e., 𝑠 𝑝 = NULL). Whenever ONF requests a sample for a page 𝑝 , the randomized algorithm provides the sample 𝑠 𝑝 , and sets the variable 𝑠 𝑝 to be NULL (we show that 𝑠 𝑝 is never NULL when ONF samples 𝑝 ). A fine point is the sampling of a new page in Line 11 of Algorithm 1; this happens after Line 17 of Algorithm 2.

6 Conclusions

In this paper, we presented the first algorithm for online weighted paging in which page weights are not known in advance, but are instead sampled stochastically. In this model, we were able to recreate the best possible bounds for the classic online problem, with an added regret term typical to the multi-armed bandit setting. This unknown-costs relaxation makes sense because the problem has recurring costs; that is, the cost of evicting a page 𝑝 can be incurred multiple times across the lifetime of the algorithm, and thus benefits from sampling.

We believe this paper can inspire future work on this problem. For example, revisiting the motivating case of managing a core-local L1 cache, the popularity of a page among the cores can vary over time; this would correspond to the problem of non-stationary bandits (see, e.g., Auer et al. [2019b,a], Chen et al. [2019]), and it would be interesting to apply techniques from this domain to OWP-UW.

Finally, we hope that the techniques outlined in this paper could be extended to additional such problems. Specifically, we believe that the paradigm of using optimistic confidence bounds in lieu of actual costs could be used to adapt classical online algorithms to the unknown-costs setting. In addition, the interface between the fractional solver and rounding scheme could be used to mediate integral samples to an online fractional solver, which is a common component in many online algorithms.

Acknowledgments

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science Foundation, the Yandex Initiative for Machine Learning at Tel Aviv University and a grant from the Tel Aviv University Center for AI and Data Science (TAD).

References

Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

Albers. Online algorithms: a survey. Math. Program. , 97(1-2):3-26, 2003. doi: 10.1007/ s10107-003-0436-0. URL https://doi.org/10.1007/s10107-003-0436-0 .

Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002a.

Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002b.

Auer, Y. Chen, P. Gajane, C. Lee, H. Luo, R. Ortner, and C. Wei. Achieving optimal dynamic regret for non-stationary bandits without prior information. In A. Beygelzimer and D. Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 159-163. PMLR, 2019a. URL http://proceedings.mlr.press/v99/auer19b.html .

Auer, P. Gajane, and R. Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In A. Beygelzimer and D. Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 138-158. PMLR, 2019b. URL http://proceedings.mlr.press/ v99/auer19a.html .

Bansal, N. Buchbinder, and J. Naor. Randomized competitive algorithms for generalized caching. In C. Dwork, editor, Proceedings of the 40th Annual ACM Symposium on Theory of Computing, Victoria, British Columbia, Canada, May 17-20, 2008 , pages 235-244. ACM, 2008. doi: 10.1145/ 1374376.1374412. URL https://doi.org/10.1145/1374376.1374412 .

Bansal, N. Buchbinder, and J. S. Naor. A simple analysis for randomized online weighted paging. Unpublished Manuscript , 2010.

Bansal, N. Buchbinder, and J. Naor. A primal-dual randomized algorithm for weighted paging. J. ACM , 59(4):19:1-19:24, 2012. doi: 10.1145/2339123.2339126. URL https://doi.org/10. 1145/2339123.2339126 .

Basu, R. Sen, S. Sanghavi, and S. Shakkottai. Blocking bandits. Advances in Neural Information Processing Systems , 32, 2019.

Bubeck, C. Coester, and Y. Rabani. The randomized k-server conjecture is false! In B. Saha and R. A. Servedio, editors, Proceedings of the 55th Annual ACM Symposium on Theory of Computing, STOC 2023, Orlando, FL, USA, June 20-23, 2023 , pages 581-594. ACM, 2023. doi: 10.1145/3564246.3585132. URL https://doi.org/10.1145/3564246.3585132 .

Chen, C. Lee, H. Luo, and C. Wei. A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free. In A. Beygelzimer and D. Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 696-726. PMLR, 2019. URL http://proceedings.mlr.press/ v99/chen19b.html .

Chrobak, H. J. Karloff, T. H. Payne, and S. Vishwanathan. New results on server problems. SIAM J. Discret. Math. , 4(2):172-181, 1991. doi: 10.1137/0404017. URL https://doi.org/ 10.1137/0404017 .

Even-Dar, S. Mannor, Y. Mansour, and S. Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Fiat and M. Mendel. Better algorithms for unfair metrical task systems and applications. In F. F. Yao and E. M. Luks, editors, Proceedings of the Thirty-Second Annual ACM Symposium on Theory of Computing, May 21-23, 2000, Portland, OR, USA , pages 725-734. ACM, 2000. doi: 10.1145/335305.335408. URL https://doi.org/10.1145/335305.335408 .

Fiat, R. M. Karp, M. Luby, L. A. McGeoch, D. D. Sleator, and N. E. Young. Competitive paging algorithms. J. Algorithms , 12(4):685-699, 1991. doi: 10.1016/0196-6774(91)90041-V. URL https://doi.org/10.1016/0196-6774(91)90041-V .

Foussoul, V. Goyal, O. Papadigenopoulos, and A. Zeevi. Last switch dependent bandits with monotone payoff functions. In International Conference on Machine Learning , pages 10265-10284. PMLR, 2023.

Irani. Page replacement with multi-size pages and applications to web caching. In F. T. Leighton and P. W. Shor, editors, Proceedings of the Twenty-Ninth Annual ACM Symposium on the Theory of Computing, El Paso, Texas, USA, May 4-6, 1997 , pages 701-710. ACM, 1997. doi: 10.1145/ 258533.258666. URL https://doi.org/10.1145/258533.258666 .

Irani. Randomized weighted caching with two page weights. Algorithmica , 32(4):624-640, 2002. doi: 10.1007/s00453-001-0095-6. URL https://doi.org/10.1007/s00453-001-0095-6 .

Koutsoupias and C. H. Papadimitriou. On the k-server conjecture. J. ACM , 42(5):971-983, 1995. doi: 10.1145/210118.210128. URL https://doi.org/10.1145/210118.210128 .

L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

S. Manasse, L. A. McGeoch, and D. D. Sleator. Competitive algorithms for server problems. J. Algorithms , 11(2):208-230, 1990. doi: 10.1016/0196-6774(90)90003-W. URL https://doi. org/10.1016/0196-6774(90)90003-W .

D. Sleator and R. E. Tarjan. Amortized efficiency of list update and paging rules. Commun. ACM , 28(2):202-208, 1985. doi: 10.1145/2786.2793. URL https://doi.org/10.1145/2786.2793 .

Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

E. Young. The k-server dual and loose competitiveness for paging. Algorithmica , 11(6):525-541, 1994. doi: 10.1007/BF01189992. URL https://doi.org/10.1007/BF01189992 .

A Analysis of the Fractional Algorithm

In this analysis section, our goal is to bound the amount ONF with respect to the UCBs and LCBs calculated by the algorithm; i.e., to prove Lemma 4.1. Lemma C.1 and Lemma C.2 from Appendix C then make the choice of confidence bounds concrete, such that combining it with Lemma 4.1 yields the final bound for the fractional algorithm, i.e., Lemma 3.1.

Proof of Lemma 4.1. For the sake of this lemma, we assume without loss of generality that the optimal solution is lazy; that is, it only evicts a (single) page in order to fetch the currently-requested page. (It is easy to see that any solution can be converted into a lazy solution of lesser or equal cost.) In the following we present a potential analysis to prove that ONF ≤ 2 log ( 1 + 𝑘 ) · OPT + U 𝑇 , where U 𝑇 is a regret term summed over 𝑇 time steps that will be defined later. To that end, we show that the following equation holds for every round 𝑡 ,

where Δ 𝑋 𝑡 is the change in 𝑋 in time 𝑡 and Φ is a potential function that we define next.

Let 𝐶 ∗ 𝑡 denote the set of pages in the offline (optimal) cache at time 𝑡 . The potential function we chose is an adaptation of the potential function used by Bansal et al. [2010], but modified to encode the uncertainty cost for not knowing the true weights. We define it as follows.

where UCB 𝑝 , LCB 𝑝 are the confidence bounds in time step 𝑡 , 𝑛 𝑝 is the number of samples of page 𝑝 collected until that point, and 𝑚 𝑝 is the total fractional movement of that page until that point. Note that 𝑛 𝑝 -𝑚 𝑝 ∈ [ 0 , 1 ] . We now show that Equation 1 holds in the three different cases in which the costs of the potential or the regret change. Before moving forward, we define the regret term, denoted U , as follows,

We note that each time the algorithm gets a new sample, 𝑛 𝑝 is incremented and U increases.

Case 1 - the optimal algorithm moves. Note that a change in the cache of the optimal solution does not affect ONF or U , and thus Δ ONF = Δ U = 0 . However, Δ OPT might be non-zero, as the optimal solution incurs a moving cost; in addition, ΔΦ might be non-zero, as the change in the optimal cache might affect the summands in the first term of the potential function.

Thus, proving Equation (1) for this case reduces to proving ΔΦ 𝑡 ≤ 2 log ( 1 + 1 / 𝜂 ) Δ OPT. Assume the optimal solution moves; as we assume that the optimal solution is lazy, it must be that the requested page 𝑝 𝑡 is not in 𝐶 ∗ 𝑡 -1 , and that the optimal solution fetches it into the cache, possibly evicting a (single) page from its cache.

First, consider the case in which there exists an empty slot in the cache, and no evictions take place when fetching 𝑝 𝑡 . In this case, Δ OPT = 0 ; moreover, as 𝐶 ∗ 𝑡 -1 ⊆ 𝐶 ∗ 𝑡 , it holds that ΔΦ ≤ 0 . Thus, Equation (1) holds.

Otherwise, at time 𝑡 , page 𝑝 𝑡 is fetched and another page 𝑝 is evicted. Thus, Δ OPT = 𝑤 𝑝 . For the potential change, 𝑝 starts to contribute to Φ . In the worst case, 𝑦 𝑝 = 0 and then Φ is increased by at most 2 LCB 𝑝 log ( 1 + 1 / 𝜂 ) ≤ 2 𝑤 𝑝 log ( 1 + 1 / 𝜂 ) = 2 log ( 1 + 1 / 𝜂 ) Δ OPT , as desired.

Case 2 - the fractional algorithm moves. In this case, only the potential and the cost of the fractional algorithm change, i.e., Δ OPT 𝑡 = Δ U 𝑡 = 0 . Thus, Equation (1) reduces to proving Δ ONF 𝑡 + ΔΦ 𝑡 ≤ 0 in this case.

There are two types of movement made by the algorithm: the immediate fetching of the requested page 𝑝 𝑡 , and the continuous eviction of other pages from the cache until feasibility is reached (i.e., there are 𝑛 -𝑘 anti-servers). First, consider the fetching of 𝑝 𝑡 into the cache; as we charge for

evictions, this action does not incur cost ( Δ ONF = 0 ). In addition, through OPT's feasibility, it holds that 𝑝 𝑡 ∈ 𝐶 ∗ 𝑡 , and thus changing 𝑦 𝑝 𝑡 does not affect the potential function ( ΔΦ = 0 ); thus, Equation (1) holds for this sub-case.

Next, consider the continuous eviction of pages. Suppose the fractional algorithm evicts 𝑑 𝑦 page units, where 𝑑 𝑦 is infinitesimally small. Let 𝑆 = { 𝑝 𝑡 } ∪ GLYPH<8> 𝑝 : 𝑦 𝑝 < 1 GLYPH<9> . Since the fractional algorithm increases 𝑦 𝑝 proportionally to 𝑦 𝑝 + 𝜂 LCB 𝑝 for each 𝑝 ∈ 𝑆 \ { 𝑝 𝑡 } , it holds that, 𝑑𝑦 𝑝 = 1 𝑁 · 𝑦 𝑝 + 𝜂 LCB 𝑝 𝑑𝑦, for 𝑁 : = ˝ 𝑝 ∈ 𝑆 \{ 𝑝 𝑡 } 𝑦 𝑝 + 𝜂 LCB 𝑝 . Thus, Δ ONF can be bounded as follows.

where the final equality stems from having 𝑑𝑦 𝑝 = 0 for every 𝑝 ∉ 𝑆 \ { 𝑝 𝑡 } . We now bound the term ( 𝑎 ) using similar arguments to those presented in Bansal et al. [2010]. We present them below for completeness.

where the first inequality implied since 𝑦 𝑝 = 1 for 𝑝 ∉ 𝑆 , which yields that

and the second inequality implied as ( 𝑥 -1 ) 𝜂 ≤ 𝑥 -𝑘 for any 𝑥 ≥ 𝑘 + 1 and using that | 𝑆 | ≥ 𝑘 + 1 . Next, we upper bound ΔΦ 𝑡 . The rate of change in the potential with respect to 𝑦 𝑝 is

The non-trivial part in the above calculation is that 𝑑 ( UCB 𝑝 -LCB 𝑝 ) · ( 𝑛 𝑝 -𝑚 𝑝 ) 𝑑𝑦 𝑝 = -GLYPH<0> UCB 𝑝 -LCB 𝑝 GLYPH<1> . This is true since 𝑛 𝑝 is uncorrelated with the change in 𝑦 𝑝 , however, 𝑚 𝑝 is increasing with with respect to 𝑦 𝑝 in 1 -linear ratio. Using the above we get that

Thus, Δ ONF 𝑡 + ΔΦ 𝑡 ≤ 0 , as required.

Case 3 - the LCBs are updated (a new sample is processed). Suppose the algorithm samples a page 𝑝 for the 𝑖 'th time, and updates UCB 𝑝 , LCB 𝑝 accordingly. Note that this sample does not incur any cost for the fractional algorithm or optimal solution, and thus Δ ONF = Δ OPT = 0 . Thus, proving Equation (1) reduces to proving ΔΦ ≤ Δ U for this case. Recalling the definition of U , it holds that

Denote by 𝑛 𝑝 , 𝑚 𝑝 the variables of that name prior to sampling 𝑝 ; note that 𝑚 𝑝 remains the same after sampling, while 𝑛 𝑝 is incremented to 𝑛 ′ 𝑝 : = 𝑛 𝑝 + 1 . Moreover, note that sampling always occurs when 𝑚 𝑝 = 𝑛 𝑝 . Thus, the change in potential function is bounded as follows.

where the second inequality is due to the LCBs being monotone non-decreasing. □

B Proofs from Section 5

B.1 Analysis and Proof of Lemma 3.2

In this subsection, we analyze Algorithm 2 and prove Lemma 3.2. Throughout this subsection, we assume the good event E ; that is, the UCBs and LCBs generated by ONF throughout the algorithm are valid upper and lower bounds for the weights of pages.

We start by proving that the algorithm is able to provide a new sample whenever the fractional algorithm requires one.

Proposition B.1. Algorithm 2 provides page weight samples whenever demanded by ONF .

Proof. We must prove that whenever a sample of page 𝑝 is requested by ONF, it holds that the variable 𝑠 𝑝 in Algorithm 2 is not NULL. Note that Algorithm 2 samples 𝑠 𝑝 at the end of a request for 𝑝 . Now, note that:

The first sample of 𝑝 is requested after the first request for 𝑝 is handled by ONF, and thus after 𝑝 is sampled.

Between two subsequent requests for samples of 𝑝 by ONF, the eviction fraction 𝑚 𝑝 increased by 1 . But, this cannot happen without a request for 𝑝 in the interim; this request ensures that 𝑠 𝑝 ≠ NULL, and thus the second request is satisfied.

Combining both cases, all sample requests by ONF are satisfied by Algorithm 2. □

Next, we focus on proving that the distribution maintained by the algorithm is consistent and balanced (and hence also valid). To prove this, we first formalize and prove the guarantee provided by the REBALANCESUBSETS procedure. To this end, we define an amount quantifying the degree to which a class is imbalanced in a given distribution.

Definition B.2. Let 𝜇 be a distribution over cache states, let GLYPH<8> 𝑦 𝑝 GLYPH<9> 𝑝 ∈ 𝑃 be the current fractional solution, and let 𝑗 be some class. We define the imbalance of class 𝑗 in 𝜇 to be

Definition B.2 quantifies the degree to which a given class is not balanced in a given distribution; one can see that if the class is balanced, this amount would be 0 .

Lemma B.3. Suppose REBALANCESUBSETS is called, and let 𝜇, 𝜇 ′ be the distributions before and after the call to REBALANCESUBSETS , respectively; let 𝑗 max be the maximum non-balanced class in 𝜇 .

Suppose that (a) 𝜇 is consistent, and (b) there exists 𝜖 > 0 such that the imbalance of any class 𝑗 ≤ 𝑗 max in 𝜇 is at most 𝜖 . Then, it holds that 𝜇 ′ is both consistent and balanced. Moreover, the total cost of REBALANCESUBSETS is at most 12 𝜖 · 6 𝑗 max .

Proof. The running of REBALANCESUBSETS consists of iterations of the For loop in Line 5; we number these iterations according to the class considered in the iteration (e.g., "Iteration 𝑖 " considers class 𝑖 ). We make a claim about the state of the anti-cache distribution after each iteration, and prove this claim inductively; applying this claim to the final iteration implies the lemma. Specifically, where 𝑗 max as defined in the lemma statement, for every 𝑖 ≤ 𝑗 max consider the distribution immediately before Iteration 𝑖 , denoted 𝜇 𝑖 . We claim that (a) the 𝜇 𝑖 is consistent, (b) all classes greater than 𝑖 are balanced in 𝜇 𝑖 , and (c) classes at most 𝑖 have imbalance at most 𝜖 · 3 𝑗 max -𝑖 in 𝜇 𝑖 . Where 𝑗 min is the minimum class, note that 𝜇 ′ = 𝜇 𝑗 min -1 , and that this claim implies that 𝜇 ′ is consistent and balanced. (Note that the claim implies that class 𝑗 min is balanced, which also implies that all classes smaller than 𝑗 min are balanced.).

We prove this claim by descending induction on 𝑖 . The base case, in which 𝑖 = 𝑗 max , is simply a restatement of the assumptions made in the lemma, and thus holds. Now, assume that the claim holds for any class 𝑖 ≤ 𝑗 max ; we now prove it for class 𝑖 -1 .

Consistency. First, as we've assumed that 𝜇 𝑖 is consistent, note that Iteration 𝑖 does not change the marginal probability of a given page 𝑝 being in the anti-cache, as pages are only moved between identical anti-cache measures. Thus, the anti-cache distribution remains consistent at any step during Iteration 𝑖 ; in particular, 𝜇 𝑖 -1 is consistent.

Existence of destination measure. Next, observe that every changes in Iteration 𝑖 consists of identifying a measure of a violating anti-cache, and matching this measure to an identical measure of anti-cache states to which pages can be moved. To show that the procedure is legal, we claim that this measure always exists. Consider such a change that identifies violating anti-cache 𝑆 , and let ˆ 𝜇 be the distribution at that point. Assume that 𝑆 is an 'upwards' violation, i.e., 𝑚 ≥ ⌈ 𝑌 𝑖 ⌉ + 1 , where 𝑚 : = | 𝑆 ∩ 𝑃 ≥ 𝑖 | ; the case of a 'downwards' violation is analogous. Note that consistency implies that E 𝑆 ′ ∼ ˆ 𝜇 | 𝑆 ′ ∩ 𝑃 ≥ 𝑖 | = 𝑌 𝑖 . Also note that 𝑆 was chosen to maximize | 𝑚 -𝑌 𝑖 | , i.e., the distance from the expectation. Thus, there exists a measure of at least ˆ 𝜇 ( 𝑆 ) of anti-caches 𝑆 ′ such that | 𝑆 ′ ∩ 𝑃 ≥ 𝑖 | < 𝑌 𝑖 (and thus | 𝑆 ′ ∩ 𝑃 ≥ 𝑖 | ≤ ⌈ 𝑌 𝑖 ⌉ -1 , as required).

Existence of page to move. After matching the aforementioned 𝑆 to some 𝑆 ′ , we want to identify some page 𝑝 ∈ 𝑃 𝑖 such that 𝑝 ∈ 𝑆 \ 𝑆 ′ , so we can move it from the measure of 𝑆 to the measure of 𝑆 ′ . Indeed, from the choice of 𝑆 and 𝑆 ′ , it holds that | 𝑆 ∩ 𝑃 ≥ 𝑖 | ≥ ⌈ 𝑌 𝑖 ⌉ + 1 ≥ | 𝑆 ′ ∩ 𝑃 ≥ 𝑖 | + 2 . But, from the induction hypothesis for Iteration 𝑖 , class 𝑖 + 1 was balanced in 𝜇 𝑖 , and thus remains balanced at every step during Iteration 𝑖 (as this iteration never moves pages of classes 𝑖 + 1 and above). This implies that | 𝑆 ∩ 𝑃 ≥ 𝑖 + 1 | ≤ ⌈ 𝑌 𝑖 + 1 ⌉ ≤ | 𝑆 ′ ∩ 𝑃 ≥ 𝑖 + 1 | + 1 . We can thus conclude that there exists 𝑝 ∈ 𝑃 𝑖 ∩ ( 𝑆 \ 𝑆 ′ ) as required.

Balanced property. Next, we prove that in 𝜇 𝑖 -1 after Iteration 𝑖 , class 𝑖 is balanced, and the imbalance of any class 𝑗 < 𝑖 is at most 𝜖 · 3 𝑗 max -( 𝑖 -1 ) . Consider any step in Iteration 𝑖 , where a measure 𝑥 of a violating state is identified; then, a page is moved from a measure 𝑥 to another measure 𝑥 . The induction hypothesis for Iteration 𝑖 implies that the imbalance of class 𝑖 at 𝜇 𝑖 is at most 𝜖 · 3 𝑗 max -𝑖 . Note that:

This step decreases the imbalance of class 𝑖 by at least 𝑥 , as it decreases the imbalance in the violating state, but does not increase imbalance in the matched measure.

This step can increase the imbalance of a class 𝑗 < 𝑖 by at most 2 𝑥 , in the worst case in which moving the page increased imbalance in both measures of 𝑥 .

As a result, we can conclude that for 𝜇 𝑖 -1 , at the end of iteration 𝑖 , class 𝑖 is balanced, while the imbalance of every class 𝑗 < 𝑖 increased by at most 2 · 𝜖 · 3 𝑗 max -𝑖 . Combining this with the hypothesis for iteration 𝑖 , the imbalance of every class 𝑗 at 𝜇 𝑖 -1 is at most 𝜖 · 3 · 3 𝑗 max -𝑖 = 𝜖 · 3 𝑗 max -( 𝑖 -1 ) , as required.

This concludes the inductive proof of the claim.

Cost analysis. As mentioned before, every step in Iteration 𝑖 reduces imbalance at class 𝑖 by (at least) 𝑥 , where 𝑥 is the measure of the chosen violating anti-cache state. The cost of this step is the cost of evicting a single page in 𝑃 𝑖 from a measure 𝑥 ; as we assume that the weight of a page is at most its UCB, this cost is at most 𝑥 · 6 𝑖 + 1 . The inductive claim above states that the imbalance of class 𝑖 at the beginning of Iteration 𝑖 is at most 𝜖 · 3 𝑗 max -𝑖 ; thus, the total cost of Iteration 𝑖 is at most 𝜖 · 3 𝑗 max -𝑖 · 6 𝑖 + 1 = 𝜖 · 6 𝑗 max + 1 / 2 𝑗 max -𝑖 . Summing over iterations, the total cost of REBALANCESUBSETS

is at most:

We can now prove that the distribution is consistent and balanced.

Lemma B.4. The distribution maintained by Algorithm 2 is both consistent and balanced.

Proof. First, we prove that the distribution is consistent. Indeed, note that consistency is explicitly maintained in Lines 9 and 12, and that Lemma B.3 implies that the subsequent calls to REBALANCESUBSETS does not affect this consistency.

As the distribution is consistent at any point in time, Lemma B.3 also implies that it is balanced immediately after every all to REBALANCESUBSETS; as the handling of every request ends with such a call, the distribution is always balanced after every request. □

At this point, we've shown that Algorithm 2 is legal: it maintains a valid distribution (through Lemma B.4 and Remark 5.3), and it provides samples to ONF when required (Proposition B.1); thus, it is a valid randomized algorithm for OWP-UW. It remains to bound the expected cost of Algorithm 2, thus proving Lemma 3.2; recall that this bound is in terms of ONF, the cost of the fractional algorithm in terms of its UCBs rather than actual page weights.

Proof of Lemma 3.2. We consider the eviction costs incurred by Algorithm 2, and bound their costs individually.

First, consider the eviction cost due to maintaining consistency Line 9 (note that Line 12 only fetches pages, and incurs no cost). An increase of 𝜖 in 𝑦 𝑝 ′ causes an eviction of 𝑝 ′ with 𝜖 probability; the expected cost of 𝜖 · 𝑤 𝑝 ′ can be charged to the eviction cost of 𝜖 · UCB 𝑝 ′ incurred in ONF, and thus the overall cost due to this line is at most ONF.

Next, consider the cost due to eviction during sampling (Line 17). Observe a page 𝑝 ∈ 𝑃 that is evicted in this way; the cost of this eviction is 𝑤 𝑝 . For the first and second samples of 𝑝 , we note that 𝑤 𝑝 ≤ 1 ; summing over 𝑝 ∈ 𝑃 , the overall cost of those evictions is at most 2 𝑛 . For subsequent samples of 𝑝 , note that for 𝑖 > 2 , the 𝑖 'th sample of 𝑝 is taken when 𝑚 𝑝 ∈ ( 𝑖 -2 , 𝑖 -1 ] . Thus, we can charge this sample to the fractional eviction that increased 𝑚 𝑝 from 𝑖 -3 to 𝑖 -2 , which costs UCB 𝑝 . Thus, the overall cost of this sampling is at most ONF + 2 𝑛 .

It remains to bound the cost of the REBALANCESUBSETS procedure. First, consider the cost of REBALANCESUBSETS due to sampling (Lines 16 and 20). Consider the state prior to such a call; some page 𝑝 has just been sampled, possibly decreasing UCB 𝑝 and decreasing the class of page 𝑝 , which could break the balanced property. Specifically, let 𝑖, 𝑖 ′ be the old and new classes of 𝑝 , where 𝑖 ′ < 𝑖 . Then, imbalance could be created only in classes 𝑗 ∈ { 𝑖 ′ + 1 , · · · , 𝑖 } . In such class 𝑗 , both 𝑌 𝑗 could decrease, and GLYPH<12> GLYPH<12> 𝑆 ∩ 𝑃 ≥ 𝑗 GLYPH<12> GLYPH<12> could decrease for any anti-cache state 𝑆 . However, as only one page changed class, one can note that the total imbalance in any such class 𝑗 is at most 1 . Thus, Lemma B.3 guarantees that the total cost of REBALANCESUBSETS is at most 12 · 6 𝑖 ≤ 12 · UCB 𝑝 . Using the same argument as for the cost of sampling, the total cost of such calls is at most 12 ONF + 24 𝑛 .

Now, consider a call to REBALANCESUBSETS in Line 10; A page 𝑝 ′ was evicted for fraction 𝜖 in ONF, and an 𝜖 measure of 𝑝 ′ was evicted in the distribution. Let 𝑗 be the class of 𝑝 ′ ; there could only be imbalance in classes at most 𝑗 . For any such 𝑖 ≤ 𝑗 , 𝑌 𝑖 increased by 𝜖 , and | 𝑆 ∩ 𝑃 ≥ 𝑖 | increased by 1 in at most 𝜖 measure of states 𝑆 . In addition, let 𝑌 -𝑖 , 𝑌 + 𝑖 : = 𝑌 -𝑖 be the old and new values of 𝑌 𝑖 . Consider the imbalance in class 𝑖 :

If GLYPH<4> 𝑌 + 𝑖 GLYPH<5> = GLYPH<4> 𝑌 -𝑖 GLYPH<5> + 1 , then the imbalance of class 𝑖 can increase due to states 𝑆 where | 𝑆 ∩ 𝑃 ≥ 𝑖 | = GLYPH<4> 𝑌 -𝑖 GLYPH<5> becoming unbalanced. But, due to consistency, the fact that 𝑌 -𝑖 ≥ GLYPH<6> 𝑌 -𝑖 GLYPH<7> -𝜖 implies that the measure of such pages is at most 𝜖 ; thus, the imbalance grows by at most 𝜖 .

The adding of page 𝑝 ′ to an 𝜖 -measure of pages can add an imbalance of at most 𝜖 .

□

Overall, the imbalance in classes at most 𝑗 prior to calling REBALANCESUBSETS is at most 2 𝜖 . Through Lemma B.3, the cost of REBALANCESUBSETS is thus at most 24 𝜖 · 6 𝑗 , which is at most 24 𝜖 · UCB 𝑝 ′ . But, the increase in ONF due to the fractional eviction is at least 𝜖 · UCB 𝑝 ′ ; thus, the overall cost of REBALANCESUBSETS called in Line 10 is at most 24 · ONF.

Finally, consider a call to REBALANCESUBSETS in Line 13, called upon a decrease in 𝑦 𝑝 ′ of 𝜖 for some page 𝑝 ′ . Using an identical argument to the case of eviction, we can bound the cost of this call by 24 times the 'fetching cost' of 𝜖 · UCB 𝑝 ′ . Now, note that the fractional fetching of a page exceeds the fractional eviction by at most 1 ; thus, the total cost of such calls is at most 24 ONF + 24 𝑛 .

Summing all costs, the total expected cost of the algorithm is at most 62 ONF + 50 𝑛 . □

B.2 Proof of Theorem 1.1

We now combine the ingredients in this paper to prove the main competitiveness bound for Algorithm 2.

Proof of Theorem 1.1. First, assume the good event E . Combining Lemma 3.2, Lemma 3.1 and Lemma C.2, it holds that

Next, assume that E does not occur. Upper bound the cost of the algorithm by 𝑂 ( 𝑛 ) per request, as in the worst case, the algorithm replaces the entire cache and samples each page in 𝑃 once. Thus, an upper bound for the cost of the algorithm is 𝑂 ( 𝑛𝑇 ) . But, through Lemma C.1, the probability of E not occurring is at most 1 𝑛𝑇 . Thus, we can bound the expected cost of the algorithm as follows.

C Choosing Confidence Bounds and Bounding Regret

In this section we define the UCBs and LCBs, prove that they hold with high probability and bound the regret term.

Let 𝑤 𝑖 𝑝 be the 𝑖 -th sample of page 𝑝 . For the initial confidence bounds, we sample each page once and set LCB 𝑝, 1 = 1 2 𝑛 2 𝑇 · 𝑤 1 𝑝 , UCB 𝑝, 1 = 1 . Once we have 𝑖 > 1 samples of page 𝑝 , we define the confidence bounds as follows. Let ¯ 𝑤 𝑝,𝑖 : = 1 𝑖 ˝ 𝑖 𝑗 = 1 𝑤 𝑗 𝑝 be the average observed weight, and

𝜖 𝑝,𝑖 = √︃ log ( 4 𝑛 3 𝑇 3 ) 2 𝑖 be the confidence radius. Then, we set LCB 𝑝,𝑖 = max { LCB 𝑝,𝑖 -1 , ¯ 𝑤 𝑝,𝑖 -𝜖 𝑝,𝑖 } and UCB 𝑝,𝑖 = min { UCB 𝑝,𝑖 -1 , ¯ 𝑤 𝑝,𝑖 + 𝜖 𝑝,𝑖 } . The following procedure updates the confidence bounds online.

Weshow that the confidence bounds indeed bound the true weights with high probability (Lemma C.1), and then bound the regret term (Lemma C.2).

Lemma C.1. Let 𝑛 𝑝 be the final number of samples collected for page 𝑝 . With probability at least 1 -1 𝑛𝑇 , the following properties hold.

Proof. By definition, the LCBs are monotonically non-decreasing and the UCBs are monotonically non-increasing. Moreover,

Algorithm 4: Optimistic-Pessimistic Weights Estimation Procedure

1 Function UPDATECONFBOUNDS( 𝑝, e 𝑤 𝑝 ) // update confidence bounds for 𝑝 upon new sample ˜ 𝑤 𝑝 . 2 if this is the first sample of 𝑝 (i.e., 𝑛 𝑝 = 1 ) then 3 define ¯ 𝑤 𝑝 ← e 𝑤 𝑝 . 4 define LCB 𝑝 ← 1 2 𝑛 2 𝑇 · ¯ 𝑤 𝑝 . 5 define UCB 𝑝 ← 1 . 6 else 7 update mean estimation ¯ 𝑤 𝑝 ← ( 𝑛 𝑝 -1 ) ¯ 𝑤 𝑝 + e 𝑤 𝑝 𝑛 𝑝 . 8 define 𝜖 𝑝 ← √︂ log ( 4 𝑛 3 𝑇 3 ) 2 𝑛 𝑝 . 9 set LCB 𝑝 ← max GLYPH<8> LCB 𝑝 , ¯ 𝑤 𝑝 -𝜖 𝑝 GLYPH<9> . 10 set UCB 𝑝 ← min GLYPH<8> UCB 𝑝 , ¯ 𝑤 𝑝 + 𝜖 𝑝 GLYPH<9> .

Thus, it remains to show that 1 2 𝑛 2 𝑇 · 𝑤 1 𝑝 ≤ 𝑤 𝑝 and that | 𝑤 𝑝 -¯ 𝑤 𝑝,𝑖 | ≤ 𝜖 𝑝,𝑖 for every page 𝑝 and 𝑖 ∈ [ 1 , 𝑇 ] with probability 1 -1 𝑛𝑇 . For the first event, by Markov inequality and a union bound over all the pages 𝑝 ∈ 𝑃 , we have

For the second event, by Hoeffding inequality and a union bound over all the pages 𝑝 ∈ 𝑃 , all the time steps 𝑡 ∈ [ 1 , 𝑇 ] and all the possible number of samples for each page 𝑖 ∈ [ 1 , 𝑛𝑇 ] , we have

The proof is now finished by taking a union bound over these two events.

□

Define U : = ˝ 𝑝 ∈ 𝑃 ˝ 𝑛 𝑝 𝑖 = 1 GLYPH<0> UCB 𝑝,𝑖 -LCB 𝑝,𝑖 GLYPH<1> + 2 log ( 1 + 1 / 𝜂 ) ˝ 𝑝 ∈ 𝑃 LCB 𝑝 , the regret term used in Lemma 4.1.

Lemma C.2. Under the good event of Lemma C.1, it holds that

Proof. The following holds under the good event.

where the last inequality holds by Cauchy-Schwarz and our choice of 𝜂 = 1 / 𝑘 . Lastly, the stated upper bound follows since 𝑘 ≤ 𝑛 and 𝑛 ≤ √ 𝑛𝑇 . □

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide an algorithm for the problem of online weighted paging with unknown weights, and analyse it. See Section 1.1 for a summary of our results, that will be proven throughout the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: This is a theory paper that present an algorithm for the online weighted paging where the weight are unknown and prove the stated bounds. We do not believe there are additional limitations to discuss.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated in the statements of the theorem and related lemmas. A proof is provided for each theoretical claim.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: No additional information is required to obtain the results in this paper, which are theoretical. A full pseudocode for the algorithm appears in the paper and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not contain experiments at all.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not contain experiments at all.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: There are no experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: There are no experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the code of ethics and concluded the paper conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper describes an algorithm for the online weighted paging problem, where the weights are unknown, and analyse it. The paper is theoretical. Thus, we do not predict any societal impact due to this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is theoretical. No data or models are associated with it.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper is theoretical. No existing assets are associated with it.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper is theoretical. No new assets are released in it.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper is theoretical. It does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper is theoretical. It does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a generalization of the classical online paging problem. In classical online paging, one needs to maintain a cache of k slots as requests for fetching pages arrive online. If the requested pages are in the cache, there is no charged cost; otherwise, w_p cost will be charged for page p. The goal is to maintain a cache of k slots such that the total fetching cost is minimized. This paper extends this classical setting to the case where the weight of all pages is not known prior to algorithms. Instead, the algorithm needs to sample a value from an unknown distribution, and such a value will be an estimated value of the actual weight. The goal is still to minimize the total fetching cost.\n\nThe main contribution of this work is an online algorithm that achieves alg$\\leq O(\\log k)$ opt + $O(\\sqrt{nT})$, where k is the size of the cache, n is the number of pages and T is the number requests. The main technique is fractional solution+rounding. To bridge these two phases, the authors design an interface, which aims to learn an estimation of weights for each page.\n\nSTRENGTHS:\n1. The studied problem is well-motivated and it should be interested in the ML community. I expect that the studied problem will have a positive impact on practice.\n\n2. I like the interface idea to bridge the fractional solution and rounding. The main purpose of the interface is to learn the weights. Although all these ideas are not novel, it is interesting to see that a simple combination works well.\n\n3. The paper is well-structured. I appreciate that the authors give a very clear statement for the algorithmic framework in Section 3. This is very helpful for readers.\n\nWEAKNESSES:\nThere is no lower bound in the paper. The regret bound might be far from tight, but this is expected in the first paper.\n\nQUESTIONS:\nCould you comment on how tight the regret bound O(\\sqrt{nT}) is? This seems to be a large number.\n\nLIMITATIONS:\nThis is a theoretical paper, there is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is the first to model and study to the online weighted paging problem with unknown weights. In this model, the weights $w_p$ of pages are initially unknown, and the eviction cost is drawn from an unknown distribution with an expectation equal to $w_p$. This study extends previous research on online weighted paging with unknown weights by integrating concepts from online learning.\n\nThe authors present an algorithm that achieves the performance bound as follows:\n$$\nE[ALG]\\ \\leq O(\\log k) OPT + \\tilde{O}(\\sqrt{nT}).\n$$\nRemark that $O(\\log k) is the competitive ratio in the known-weight setting. The proposed solution comprises two main components: a fractional algorithm and a rounding framework. The fractional algorithm is designed to approximate the optimal cost, while the rounding framework ensures consistency with the fractional solution through a rebalancing routine. The methodology for the fractional algorithm and the rounding and rebalancing framework are from [Bansal et al. 2010] and [Bansal et al. 2012]. Additionally, the weight estimation utilizes a classical Upper Confidence Bound (UCB) method from the multi-armed bandit problem in online learning. It is a non-trivial combination of the two techniques.\n\nSTRENGTHS:\nThis paper presents their motivation for studying the new model clearly, using the example of multi-level cache structure as a compelling rationale. The proposed model is clean, providing a robust foundation that is likely to inspire future research.\n\nWEAKNESSES:\nThe presentation in section 5 is not clear and may only be reader-friendly to those familiar with it [Bansal et al. 2012]. We may need a high-level idea and some introductions, like anti-cache, before presenting the pseudocode.\n\nQUESTIONS:\nIn section 5, the author mentions that they need a more robust rebalancing procedure. What is the detailed difference between it and the previous approach? Can you give some more explanations?\n\nIt seems that the algorithm runs really slow because of the continuous rebalancing step. Can you remark on the running time of the algorithm?\n\nLIMITATIONS:\nThe author does not address the limitations of the proposed algorithm. The efficiency of the algorithm is not considered. Given that the algorithm maintains an exponential number of subsets $S \\subset P$ and may adjust some $\\mu(S)$ each time an $\\epsilon$ fraction of $\\mu(S)$ changes, the running time may be quite large.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of online weighted paging with eviction costs drawn from unknown page-dependent distributions. As evictions occur, the authors demonstrate how to learn an effective eviction strategy online using previous cost samples, framing this as a multi-armed bandit problem. They first approach a fractional relaxation of the problem, then apply a rounding technique to derive a randomized algorithm for the original problem. The resulting algorithm incurs an expected cost of $O(\\log k) \\cdot \\text{OPT} + O(\\sqrt{nT})$, where $k$ is the cache size, $n$ is the number of pages, and $T$ is the number of time steps.\n\nSTRENGTHS:\n* The paper is well-written and well-organized.\n* The setting is both well-motivated and theoretically interesting, as it combines techniques from competitive analysis and regret analysis.\n* The main result is strong and might inspire future work in the field of online algorithm design.\n\nWEAKNESSES:\nI do not see any major weaknesses in this paper, but there are a few minor ones:\n* Although the setting is different, the paper is closely related to the framework of learning-augmented algorithms. I suggest that the authors mention this connection in the related work section.\n* A very similar setting has been studied in \"On Preemption and Learning in Stochastic Scheduling\" (ICML 2023) for the non-clairvoyant scheduling problem, which also couples the analyses of the competitive ratio and regret. Maybe this should be cited as a related work.\n* While the main contribution is theoretical, it would be nice to conduct a few experiments to demonstrate how the proposed algorithms compare to other benchmark algorithms where the weights are known\n\nQUESTIONS:\nSince the algorithm only chooses which of the $k$ pages in the cache to evict, would it be possible to improve the regret term in Theorem 1.1 and have instead a term of the form $O(\\sqrt{f(k,n) T})$, with $k \\leq f(k,n) < n$? If not, can you prove that the bound $O(\\sqrt{nT})$ is the best possible for any values of $n$ and $k$?\n\nLIMITATIONS:\nThe assumptions of the theorems are clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a the online paging problem where the weights to retrieve a items are independent random variables with potentially different distributions. The paper introduces an algorithm whose expected performance differ from the optimal one by a multiplicative factor (logarithmic in the size of the cache) and an additive term (sublinear in the number of the requests).\n\nSTRENGTHS:\nIf correct, the paper makes a very interesting advance on an important theoretical topic. The authors position well the paper in the existing literature.\n\nWEAKNESSES:\nI found this paper the most difficult to evaluate. I feel NeurIPS may not be the most suited venue for such work, both for its theoretical focus (it seems more a STOC-paper) and the 9-page limit which forced the authors to compress much their reasoning. Probably for this reason, as well as some disorganization in the presentation, I was not able to figure out how the algorithm really works as well as to follow some more technical arguments. I provide some specific comments below\n\nDoubts\n- about algorithm 1, what is really behind \"continually increase\" at line 4, should we assume that that y_p and m_p are updated according to differential equations? This seems to be supported also by later statements like at line 276 \"upon any (infinitesimally small) change to a fractional variable\"\n- I was not able to understand the reasoning about why the problem may not exhibit sublinear regret without a competitive ratio (point b at page 2) \n- in section 1.2, the paper argues that the solution cannot be built in the standard way by combining a solution to the fractional problem and a rounding scheme. I was not able to follow the reasoning in lines 86-94. Even more, because the proposed solution combines a solution to the fractional problem and a rounding scheme\n- in footnote 2, \"note that k \\le \\sqrt{nT}\", I did not get if this follows from the previous reasoning or it is an implicit assumption that T is large enough.\n- algorithm 2, what does it mean line 9 \"add p' to the anti-cache in an \\epsilon-measure of states without p'\"? If find this kind of expressions too vague.\n- lines 254-255, there is probably an error, how can be UCB_p between 6^i and 6^{i+1} if, as observed in the same line UCB_p is between 0 and 1?\n\nMotivation\n- I find the motivation in terms of a cache inserted in the a hierarchy of caches quite weak. The content stored at caches of higher level would be determined by the caching decisions at lower levels. It would the not lead to the random weights considered in this paper.\n\nConfused presentation\n- the fact that costs may be considered to be paid upon eviction rather than upon fetching is mentioned in footnote 2 at page 4, but it is already implicitly used at page 2 when explaining why the problem cannot admit logarithmic competitive ratio without regret\n- the paper starts talking about moving fraction and servers at line 166 before referring to the parallel with the k-server problem\n- the (huge) caption of figure 1 refers to the \"consistency property\" and the \"subset property\". The first one is never really formally defined. The second one is only introduced two page later.\n- in the caption of figure 2, there is a reference to \"class i\" but this is only introduced at page 8. Also note that there are two classes i P_i and P_{\\ge i}\n\n\n\nMinor\n- the problem is sometimes referred to as OWP-UW, others as UW-OWP\n- line 95: \"the and rounding scheme\"\n- footnote 1: \"known in advanced\"\n- in the figures the lables of the different subfigures (e.g. a, b, c, d in figure 1 are missing)\n\nUpdate after rebuttal\n- the authors have addressed some of my technical questions and I have increased the score.\n\nQUESTIONS:\nSee questions above.\n\nLIMITATIONS:\nThe computational complexity of the algorithm is not discussed. This is important, in particular if, as I think, the algorithm relies on solving some differential equations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthor consider caching with stochastic weights. In particular,\nthe cost incurred by the algorithm at eviction of a page p\nis drawn independently from a fixed distribution D_p\nwhich is different for every page and is not known to the algorithm in advance.\nAuthors propose an algorithm with guarantees which combine both\ncompetitive ratio and regret. Authors claim that neither regret nor competitive\nratio are possible on their own.\nTheir algorithm is a modification of the classical algorithm\nby Bansal et al. for weighted paging, incorporating upper and lower confidence\nbounds on the average weight of the page.\nThe more challenging part seems to be randomized rounding, where they\nhave to deal with the changes of the upper confidence bounds and such\nelement was not present in the original paper of Bansal et al.\n\nSTRENGTHS:\n* Problem seems interesting\n* Result is non-trivial from technical point of view\n\nWEAKNESSES:\n* I find the framing of their result in the context of hierarchical caching\nand CPU caches very unfortunate for several reasons:\n\n(1) hierarchical caching is already studied (see work of Bansal, Buchbinder, Naor SICOMP'12). Authors do not seem to be aware of this.\n\n(2) It is completely unclear why should there be an underlying distribution\nof the cost of loading each page in a hierarchical (or multi-level) cache:\nIt is enough that the algorithm operating on different levels of the cache\nare mis-aligned in some way to make weights of the pages\nadversarial instead of stochastic.\n\n(3) I cannot imagine implementing anything resembling their approach\nin CPU caches.\n\n* Provided justification of the optimality of the presented results\nis very informal. It is just a few lines.\nIn particular, authors argue about the necessity of the regret term using using\nlower bounds for bandits. However, their input is stochastic which\nallows much better regret bounds when the mean weights of the pages\ndiffer significantly. On the other hand, if the weights of the pages are\nsimilar, it is easy to achieve good competitive ratio.\n\n* Statements of their results are not completely clear: E.g., what is Q in\nTheorem 1.1? Theorem says it is an \"input\":\nIs it composed of the request sequence and the distribution of\npage weights, or does it contain also the realization of the page weights?\nWhat is OPT(Q) then? I do not see these things explained even in preliminaries.\n\nQUESTIONS:\n* Please comment on the last two weaknesses mentioned above.\n\nLIMITATIONS:\nAuthors do not discuss possible difficulties in implementing their algorithm\nin CPU caches although they use such use case as a main motivation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}