SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Online shared caching: For the online shared caching problem, the admissible set  $\Delta$  is given by Eq. (5). Clearly, incidence vectors of all  $k$ -sets, containing  $k$ 's and  $N - k$  zeros, belong to  $\Delta$ . Furthermore, given any vector in  $\mathbf{y} \in \Delta$ , a randomized integral allocation can be sampled in linear time using Madow's sampling scheme, described in Algorithm 3, which yields the vector  $\mathbf{y}$  in expectation.
Algorithm 3 MADOW-SAMPLE  $(p)$
1: Input: A universe  $[N]$  of size  $N$ , the cardinality of the sampled set  $k$ , and a marginal inclusion probability vector  $p \in \Delta_k^N$ .  
2: Output: A random  $k$ -set  $S$  with  $|S| = k$  such that,  $\mathbb{P}(i \in S) = p_i$ ,  $\forall i \in [N]$ .  
3: Define  $\Pi_0 = 0$ , and  $\Pi_i = \Pi_{i-1} + p_i$ ,  $\forall 1 \leq i \leq N$ .  
4: Sample a uniformly distributed random variable  $U$  from the interval  $[0,1]$ .  
5:  $S \gets \emptyset$   
6: for each  $i \gets 0$  to  $k-1$  do  
7: Select the element  $j$  if  $\Pi_{j-1} \leq U + i < \Pi_j$ .  
8: end for each  
9: return  $S$
2. Online job scheduling: In the online job scheduling problem, the admissible action set  $\Delta$  is given by the  $N$ -simplex. Given any point on the  $N$ -simplex, it is trivial to randomly sample a coordinate using a single uniform random variable.
3. Online matching: In the online matching problem, the admissible action set  $\Delta$  is given by the Birkhoff polytope (13). Given any point  $y$  in the Birkhoff polytope, it can be efficiently decomposed as a convex combination of a small number of matchings using the Birkhoff-von-Neumann (BvN) decomposition algorithm [Valls et al., 2021]. Using the decomposition coefficients, a matching can be randomly sampled that exactly matches the point  $y$  in expectation.
C.8 Non-Convexity of the optimal offline benchmark for the online job scheduling problem
Consider the job scheduling problem in the reward maximization setting as described in Example B.1. Clearly, the  $\alpha$ -fairness metric accumulated by the online policy is concave in the regime  $0 \leq \alpha \leq 1$ . However, the following proposition shows that the offline static benchmark for this problem is non-convex for the  $\alpha$ -fair utility function in the regime  $0 < \alpha < 1$ .
Proposition 1. The offline optimal  $\alpha$ -fair reward function for the job scheduling problem is non-convex for  $0 < \alpha < 1$ .
Proof. Let the probability distribution  $\pmb{y}^*$  be an optimal static offline allocation vector for the reward sequence  $\{\pmb{x}_t\}_{t\geq 1}$ . Also, let  $R_{T}(i)\equiv \sum_{t}x_{i}(t)$  be the cumulative reward observed by the  $i^{\mathrm{th}}$  machine,  $1\leq i\leq m$ . Then the optimal allocation  $\pmb{y}^*$  maximizes the following objective:
$$
(1 - \alpha) ^ {- 1} \sum_ {i} \left(y _ {i} R _ {T} (i)\right) ^ {1 - \alpha}. \tag {28}
$$
s.t. the constraints  $\sum_{i}y_{i} = 1, y\geq 0$ . Since  $0 < \alpha < 1$ , the objective function is strongly concave and the optimal solution  $\pmb{y}^{*}$  is unique. Using the standard Hölder's inequality with the conjugate norms  $p = 1 / 1 - \alpha, q = 1 / \alpha$ , we can upper bound the objective (28) as
$$
\begin{array}{l} (1 - \alpha) ^ {- 1} \sum_ {i = 1} ^ {m} \left(y _ {i} R _ {T} (i)\right) ^ {1 - \alpha} \leq (1 - \alpha) ^ {- 1} \left(\sum_ {i} y _ {i}\right) ^ {1 - \alpha} \left(\sum_ {i} R _ {T} (i) ^ {\frac {1 - \alpha}{\alpha}}\right) ^ {\alpha} \\ = \left(1 - \alpha\right) ^ {- 1} \left(\sum_ {i} R _ {T} (i) ^ {\frac {1 - \alpha}{\alpha}}\right) ^ {\alpha}, \tag {29} \\ \end{array}
$$
where we have used the constraint  $\sum_{i}y_{i} = 1$  in the last equality. The upper-bound is achieved by the following distribution
$$
y _ {i} ^ {*} = \frac {R _ {i} ^ {\frac {1 - \alpha}{\alpha}}}{\sum_ {i} R _ {i} ^ {\frac {1 - \alpha}{\alpha}}}, \forall i \in [ m ]. \tag {30}
$$
Hence, the optimal offline reward (28) is given by
$$
\mathcal {R} ^ {*} = (1 - \alpha) ^ {- 1} \left(\sum_ {i} R _ {T} (i) ^ {\frac {1 - \alpha}{\alpha}}\right) ^ {\alpha}. \tag {31}
$$
To show that  $\mathcal{R}^*$  is non-convex in the cumulative reward vector  $R_{T}$  in the regime  $0 < \alpha < 1$ , consider the case of two users. Letting  $x = R_{T}(1), y = R_{T}(2)$  and ignoring the positive pre-factor, the function under consideration is given as follows:
$$
f (x, y) = \left(x ^ {\frac {1 - \alpha}{\alpha}} + y ^ {\frac {1 - \alpha}{\alpha}}\right) ^ {\alpha}. \tag {32}
$$
Recall that a function is convex if and only if the determinants of all leading principal minors of its Hessian matrix are non-negative. A straightforward computation yields the following expression for the determinant of the Hessian of the function  $f(x,y)$ :
$$
\det  \left(\nabla^ {2} f (x, y)\right) = (2 \alpha - 1) \frac {\left(\alpha - 1\right) ^ {2} x ^ {1 / \alpha - 1} y ^ {1 / \alpha - 1} \left(x ^ {1 / \alpha - 1} + y ^ {1 / \alpha - 1}\right) ^ {2 \alpha}}{\left(y x ^ {1 / \alpha} + x y ^ {1 / \alpha}\right) ^ {2}}.
$$
The determinant becomes strictly negative in the regime  $0 < \alpha < \frac{1}{2}$ . This shows that the function (32) is non-convex for  $0 < \alpha < \frac{1}{2}$ . On the other hand, we have
$$
\frac {\partial^ {2} f (x , y)}{\partial x ^ {2}} = (\alpha - 1) \frac {y x ^ {1 / \alpha - 2} (x ^ {1 / \alpha - 1} + y ^ {1 / \alpha - 1}) ^ {\alpha} (\alpha^ {2} y x ^ {1 / \alpha} + 2 \alpha x y ^ {1 / \alpha} - x y ^ {1 / \alpha})}{\alpha (y x ^ {1 / \alpha} + x y ^ {1 / \alpha}) ^ {2}}.
$$
In particular, at the point  $(1,1)$  the above second partial derivative evaluates to be
$$
\frac {\partial^ {2} f (x , y)}{\partial x ^ {2}} | _ {(x, y) = (1, 1)} = - \frac {2 ^ {\alpha - 2} (1 - \alpha)}{\alpha} (\alpha^ {2} + 2 \alpha - 1),
$$
which is strictly negative for  $\sqrt{2} - 1 < \alpha < 1$ . Taking the above two results together, we conclude that the offline optimal reward function (32) is non-convex for  $0 < \alpha < 1$ .
C.9 Proof of Theorem 2 (lower bounding the approximation factor)
Consider the following instance of the online shared caching problem with  $m = 2$  users and a cache of unit capacity. Assume that the library's size ( $N$ ) is sufficiently large. Let the total length of the request sequence be  $T$  rounds and let  $\eta \in [0,1]$  be some constant fraction to be fixed later. Consider two different file request sequences:
- Instance 1: For the first  $\eta T$  rounds, user 1 always requests file 1 and user 2 always requests file 2. For the next  $(1 - \eta)T$  rounds, user 1 always requests file 2 and user 2 requests a file chosen uniformly at random from the library.
- Instance 2: For the first  $\eta T$  rounds, as in Instance 1, user 1 always requests file 1 and user 2 always requests file 2. However, for the next  $(1 - \eta)T$  rounds, user 1 requests a random file chosen uniformly at random from the library and user 2 always requests file 1.
We now lower bound the approximation factor achievable by any online policy for the above two request sequences.
Optimal static offline policy: It is easy to see that the optimal offline strategy is to cache file 2 for all  $T$  rounds. Hence, the total  $\alpha$ -fairness objective accrued by the static offline policy is
$$
\text {O f f l i n e} \alpha \text {- f a i r n e s s} = (1 - \alpha) ^ {- 1} T ^ {1 - \alpha} \left[ \eta^ {1 - \alpha} + (1 - \eta) ^ {1 - \alpha} \right].
$$
Clearly, the same fairness objective is achieved for the second instance by a static policy that always caches file 1.
Figure 7: Comparison between the upper and lower bounds of the approximation ratio
Online Policy: Suppose that during the first  $\eta T$  rounds, the online policy caches file 1 for  $\gamma$  fraction of the time and file 2 for  $1 - \gamma$  fraction of the time. Since the policy is online, the fraction  $\gamma$  remains the same for both instances. Clearly, for the next  $(1 - \eta)T$  time slots, an optimal online policy caches file 2 for instance 1 and file 1 for instance 2. Taking the worst of the two instances, any online policy achieves the following fairness objective:
$$
\text {O n l i n e} \alpha \text {- f a i r n e s s} = (1 - \alpha) ^ {- 1} T ^ {1 - \alpha} \min  (g (\gamma), g (1 - \gamma)),
$$
where  $g(\gamma) \equiv \left(\gamma \eta + (1 - \eta)\right)^{1 - \alpha} + \left((1 - \gamma)\eta\right)^{1 - \alpha}$ . From simple calculus, it follows that the function  $g(\gamma)$  is non-increasing for  $0 \leq \eta \leq 1/2, 0 \leq \alpha < 1$ . This implies that the online  $\alpha$ -fairness is maximized when  $\gamma^{*} = \frac{1}{2}$ . Hence, the  $\alpha$ -fairness metric achieved by any online policy for the worst of the above two instances is upper bounded by:
$$
\text {O n l i n e} \alpha \text {- f a i r n e s s} \leq (1 - \eta / 2) ^ {1 - \alpha} + (\eta / 2) ^ {1 - \alpha}.
$$
Hence, the achievable approximation ratio  $c_{\alpha}$  for any online policy with a sublinear  $c_{\alpha}$ -regret is lower bounded as follows:
$$
\text {A p p r o x .} \geq \frac {\text {O f f l i n e} \alpha \text {- f a i r n e s s}}{\text {O n l i n e} \alpha \text {- f a i r n e s s}} = \frac {\eta^ {1 - \alpha} + (1 - \eta) ^ {1 - \alpha}}{(1 - \eta / 2) ^ {1 - \alpha} + (\eta / 2) ^ {1 - \alpha}}.
$$
Taking the maximum of the RHS with respect to the parameter  $\eta$  yields the desired result.
D Additional Numerical Experiments
In this section, we provide a more detailed description of how the CDN dataset was preprocessed and additional numerical experiments on fair scheduling.
D.1 Preprocessing the CDN dataset
We give additional details regarding preprocessing the CDN dataset for the experiments here. As explained in section 4 we set  $T = 400$  rounds, number of users  $m = 4$ , cache size  $C = 10$  and, library size  $N = 50$ . For preprocessing, first, we sort the data according to the timestamp and discard all the files with an ID greater than  $N$ , and then only consider the first  $m \times T = 1600$  requests among the remaining requests. We then relabel the files from 1 to  $N$  in descending order of the number of requests such that among these 1600 requests, file 1 is the most requested file, followed by the second file, and so on. We then assign the requests sequentially to users in decreasing order of priority from users 1-4, i.e., we first try to assign a request to user 1, failing which we try to assign it to user 2, and then to users 3 and 4. We can only assign a request to a user if the total number of requests assigned to that user is less than or equal to  $T = 400$  and the following conditions are met for users 1-3: (1)
user 1: file ID between 1-6; (2) user 2: file ID between 1-9; (3) user 3: file ID between 1-12. User 4 can be assigned any file. Thus, we allocate files to users in decreasing order of their popularity, such that user 1 tends to request common files, while user 4 tends to request less common files. This skews the file request distribution, making a fair cache allocation for all users difficult.
D.2 Fair Scheduling
Dataset: The datasets are generated by simulating a wireless channel. For the first dataset (WLAN), the simulator generates channel gains and shadowing values for each user, computes the received power and noise for each user, adds Rayleigh fading to the signal-to-noise ratios (SNRs), and then normalizes the SNRs by the maximum value to generate rewards. We take the horizon  $T = 5000$  and number of users  $m = 5$ . The second dataset (MCS) is generated using Modulation and Coding Scheme table which defines the data rates for each user depending on the radio link quality. Here, the number of users is  $m = 2$  and the horizon is at  $T = 2000$ . Although regret guarantees for  $\alpha > 1$  become vacuous, to clearly demonstrate the fairness characteristics of algorithms, we run simulations for  $\alpha \in [0, 2]$ .
(a) Reward Rates: GA
(b) Reward Rates: OFA
(c) Jain's Index
Figure 8: Scheduling (WLAN)
(b) Reward Rates: OFA
Figure 9: Scheduling (WLAN)
Comparison: We compare our policy with the Gradient Algorithm (GA) described by Stolyar [2005b]. The Gradient Algorithm is a widely used in fairly scheduling transmission of multiple users via a time-varying wireless channel. Stolyar considers a model where a finite set of queues are served by a switch. The switch has a finite number of switch states  $M$ . The sequence of states at each time steps form an irreducible Markov chain. At each state  $m$ , a scheduling decision  $k \in K(m)$  is made, where  $K(m)$  is a finite set of decisions. Each scheduling decision has a corresponding vector of service rates given by  $\mu^m(k)$ . Let  $V$  be the set of all long-term average service rate vectors. The goal is to have a scheduling policy that gives an average service rate solving the maximization problem
$$
\max  _ {u \in V} H (u), \tag {33}
$$
where  $H(\cdot)$  is a strictly concave smooth utility function. The Gradient Algorithm makes a scheduling decision
$$
k (t) \in \operatorname * {a r g m a x} _ {k} \langle \nabla H (X (t)), \mu^ {m} (k) \rangle ,
$$
where  $X(t)$  is updated as
$$
X (t + 1) = (1 - \beta) X (t) + \beta \mu^ {m} (k),
$$
for some initial value of  $X(0)$  and a fixed parameter  $\beta > 0$ . The Gradient Algorithm converges to an optimal  $u^{*}$  that maximizes (33) as  $\beta \to 0$ .
Figures 8a, 8b, 9a and 9b compares the average, minimum and maximum reward rates of both the policies as  $\alpha$  increases. The Jain's Index of both the policies is shown in Figures 8c and 9c. It can be clearly observed that for  $\alpha \in [0,1)$ , the OFA policy performs better in terms of fairness, but for larger  $\alpha$ 's, the GA outperforms the OFA policy by a small margin. Recall that Stolyar only provides theoretical guarantees for a stochastic setting whereas we give a stronger regret bound in an adversarial setting at the expense of slight performance degradation in practice.