SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the case when we have access to two models, we need to design a good caching and model multiplexing strategy jointly. We can compute the optimal caching and model multiplexing policy as
$\mathcal{L}^{\star}, \pi^{\star} = \arg \min_{\mathcal{L}, \pi} \mathrm{cost}(\mathcal{L}, \pi)$ , which gives the following solution:
$$
\begin{array}{l} \pi^ {\star} (q) = \mathbb {1} \left(c _ {s} ^ {\star} (q) \leq c _ {l} ^ {\star} (q)\right), \\ \mathcal {L} ^ {\star} = \underset {\mathcal {L}: | \mathcal {L} | \leq L} {\arg \min } \sum_ {q \in \mathcal {Q}} P (q) \mathbb {1} (q \notin \mathcal {L}) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right). \\ \end{array}
$$
Such optimal strategies are straightforward:  $\pi^{\star}$  always assigns the query to the model with a smaller cost, and  $\mathcal{L}^{\star}$  saves the  $L$  queries with the largest  $P(q)\cdot \min (c_s^\star (q),c_l^\star (q))$ .
For the model multiplexing algorithm, we consider two baselines: (a) one always uses large model  $\pi_l(q)\equiv 0$ ; (b) one always uses the small model  $\pi_s(q)\equiv 0$ . This is related to the LLM cascade idea in the concurrent work of Chen et al. (2023b). We provide more discussion in Appendix A, and present comparisons between baselines and  $\pi^{\star}$  in Appendix D.
4.2 Finite sample setting: Offline learning
We now consider the finite sample case. Let  $\mathcal{D}_N = \{(q_1, c_{s,1}, c_{l,1}), \dots, (q_N, c_{s,N}, c_{l,N})\}$ , where  $c_{s,n}$  is a sample from random variable  $C_s(q_n)$ , the observed cost for processing query  $q_n$  with the small model in round  $n$ . And  $c_{l,n}$  is a sample from random variable  $C_l(q_n)$ , the observed cost for processing query  $q_n$  with the large model in round  $n$ . We consider estimating  $P, c_s^{\star}, c_t^{\star}$  with some oracles  $\hat{P} = \mathrm{DenEstOracle}(q_1, \dots, q_N)$ ,  $\hat{c}_s(q), \hat{c}_t(q) = \mathrm{RegressionOracle}(\mathcal{D}_N)$ . We focus on the tabular case for theoretical analysis, where we set  $\hat{P}, \hat{c}_s(q)$  and  $\hat{c}_l(q)$  to be the plug-in estimator:
$$
\begin{array}{l} \hat {P} (q) = \frac {\sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q)}{N}, \hat {c} _ {l} (q) = \left\{ \begin{array}{l l} \frac {\sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q) c _ {l , i}}{\sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q)}, & \text {i f} \sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q) > 0 \\ B _ {1}, & \text {i f} \sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q) = 0, \end{array} \right. \\ \hat {c} _ {s} (q) = \left\{ \begin{array}{l l} \frac {\sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q) c _ {s , i}}{\sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q)}, & \text {i f} \sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q) > 0 \\ B _ {1}, & \text {i f} \sum_ {i = 1} ^ {N} \mathbb {1} (q _ {i} = q) = 0. \end{array} \right. \\ \end{array}
$$
Similar to the case of caching without model multiplexing, for a long-tailed distribution  $P(q)$ , the estimation of  $c_s^\star (q)$ ,  $c_l^\star (q)$  can be bad for the queries that are visited less. To select the maximum  $L$  elements from the plug-in estimator, we introduce pessimism to the estimate of  $\hat{c}_l$  and  $\hat{c}_s$ . This leads to the following design of caching and model multiplexer  $\hat{L}$  and  $\hat{\pi}$ :
$$
\hat {\pi} (q) = \mathbb {1} (\hat {c} _ {s} (q) \leq \hat {c} _ {l} (q)),
$$
$$
\hat {\mathcal {L}} = \underset {\mathcal {L}: | \mathcal {L} | \leq L} {\arg \min} \sum_ {q \in \mathcal {Q}} \mathbb {1} (q \notin \mathcal {L}) \hat {P} (q) \max \left(B _ {1}, \min (\hat {c} _ {s} (q), \hat {c} _ {l} (q)) - (B _ {2} - B _ {1}) \sqrt {\frac {\log (8 | \mathcal {Q} | / \delta)}{2 \sum_ {n = 1} ^ {N} \mathbb {1} (q _ {n} = q)}}\right).
$$
We now show the cost for the caching and model multiplexer obtained from the empirical estimate is close to the optimal cost. The proof is deferred to Appendix G.
Theorem 3. Assume that  $N \geq \frac{8B_2|\mathcal{Q}|\log(4L / \delta)}{B_1}$  and take  $\delta = 1 / N$ . We have
$$
\mathbb {E} [ \mathrm {c o s t} (\hat {\mathcal {L}}, \hat {\pi}) - \mathrm {c o s t} (\mathcal {L} ^ {\star}, \pi^ {\star}) ] \leq C L (B _ {2} - B _ {1}) \cdot \sqrt {\frac {B _ {2} | \mathcal {Q} | \log (8 | \mathcal {Q} | N)}{B _ {1} N}}.
$$
4.3 Finite sample setting: Online learning
We turn to the online case. We first propose a meta-algorithm in Algorithm 2. We provide a theoretical analysis of the meta-algorithm for the tabular case, with DenEstOracle  $\hat{P}_t(q) = \frac{\sum_{i=1}^t \mathbb{1}(q_i = q)}{t}$ , and the RegressionOracle defined as follows:
$$
\hat{c}_{l,t}(q) = \left\{ \begin{array}{ll}B_{1}, & \text{if}\sum_{i = 1}^{t}\mathbb{1}\big(s_{i} = 0,q_{i} = q\big) = 0\\ \max \Big(B_{1},\frac{\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 0,q_{i} = q)c_{l,i}}{\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 0,q_{i} = q)} -(B_{2} - B_{1})\sqrt{\frac{\log(8T|\mathcal{Q}| / \delta)}{2\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 0,q_{i} = q)}}\Big),\text{otherwise}, \end{array} \right.
$$
$$
\hat{c}_{s,t}(q) = \left\{ \begin{array}{ll}B_{1}, & \text{if}\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 1,q_{i} = q) = 0,\\ \max \left(B_{1},\frac{\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 1,q_{i} = q)c_{s,i}}{\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 1,q_{i} = q)} -(B_{2} - B_{1})\sqrt{\frac{\log(8T|\mathcal{Q}| / \delta)}{2\sum_{i = 1}^{t}\mathbb{1}(s_{i} = 1,q_{i} = q)}}\right),\text{otherwise}. \end{array} \right.
$$
We provide the following theorem on the regret of the overall algorithm.
Algorithm 2 Joint Design of Caching and Model multiplexing
1: Initialize the set of cache  $\mathcal{L}_1 = \{\}$  past observations  $\mathcal{H}_1 = \{\}$ ,  $\hat{c}_{l,0}(q) = B_1$ ,  $\hat{c}_{s,0}(q) = B_1$ , model multiplexing policy  $\pi_0(q) = 1, \forall q \in \mathcal{Q}$ .  
2: For iteration  $t = 1, 2, \dots, T$   
3: Receive query  $q_t$ .  
4: Update the density estimation  $\hat{P}_t = \text{DenEstOracle}(q_1, \dots, q_t)$ .  
5: If  $q_t \in \mathcal{L}_t$ : output the cached result, set  $\hat{c}_{s,t} = \hat{c}_{s,t-1}, \hat{c}_{l,t} = \hat{c}_{l,t-1}, \pi_t = \pi_{t-1}$ , update the past observation  $\mathcal{H}_t = \mathcal{H}_{t-1} \bigcup (q_t, \times, \times)$ , and continue.  
6: Select the models according to  $s_t = \pi_t(q_t)$ .  
7: Update the past observation  $\mathcal{H}_t = \mathcal{H}_{t-1} \bigcup (q_t, s_t, c_t)$ .  
8: Update  $\hat{c}_{l,t}, \hat{c}_{s,t} = \text{RegressionOracle}(\mathcal{H}_t)$ . Set  $\pi_{t+1}(q) = \mathbb{1}(\hat{c}_{s,t}(q) < \hat{c}_{l,t}(q))$ .  
9: If  $|\mathcal{L}_t| < L$ : let  $\mathcal{L}_{t+1}$  be the union of  $\mathcal{L}_t$  and  $q_t$ .  
10: Else if  $\hat{P}_t(q_t) \cdot \min(\hat{c}_{s,t}(q_t), \hat{c}_{l,t}(q_t)) > \min_{q \in \mathcal{L}_t} \hat{P}_t(q) \cdot \min(\hat{c}_{s,t}(q), \hat{c}_{l,t}(q))$ : replace the minimizer element in the cache  $\mathcal{L}_t$  on the RHS with  $q_t$  to get  $\mathcal{L}_{t+1}$ .
Theorem 4. Substituting the oracles in Algorithm 2 with the oracles above and  $\delta = 1 / T$ , we have
$$
\mathrm {R e g r e t} _ {\mathsf {s e l}} (T) \leq \frac {C L (B _ {2} - B _ {1}) B _ {2} | \mathcal {Q} | L \log^ {2} (T | \mathcal {Q} |)}{B _ {1}} \cdot \sqrt {T}.
$$
The proof is deferred to Appendix H. Compared with the lower bound in Theorem 2, we see that the dependency on  $T$  is tight. The pessimism plays two different roles here: on the one hand, it encourages the exploration for model multiplexing to choose the ones with more uncertainty in the cost; on the other hand, it encourages the exploitation to be conservative about which query to save into the cache.
For the model multiplexer to work well, one needs to have a small yet accurate model multiplexer. In the case when the model multiplexer is not accurate, the small model always comes with a much smaller cost, and we are allowed to regenerate the responses and make corrections for the output, one may combine LEC with cascade (Chen et al., 2023b) to achieve better performance.
We conduct both simulations and real-world experiments with our proposed methods. The code is available at https://github.com/Ying1123/llm-caching-multiplexing.
5.1 Simulations for algorithm analysis
We conduct synthetic online and offline experiments for joint optimization of caching and model switching. In Figure 2, we plot the cumulative cost and regret in online learning for LFU and LEC caching algorithms. For LFU, we consider model switchers which always select the small or large models as the baselines. We consider 20 distinct prompts and set the cache size to be 10. We set the frequency distribution as power distribution with  $\alpha = 0.9$ . The ground truth cost for each query processed by both models is set as a sample from  $100X + 1$ , where  $X$  is a random variable generated from a Bernoulli distribution with the parameter 0.5. We repeat the simulation 100 times and plot the mean and standard deviation in the figure. Our simulation suggests that LEC with model switcher greatly improves the two baselines by a factor of  $50\times$  when the cost ratio is 100. We include additional results on the synthetic datasets for both online and offline settings with different  $\alpha$  values, cost ratios, and switcher accuracy in Appendix I.1.
5.2 Experiments on real datasets
We evaluate our algorithms on two tasks: next-token prediction on the Lambada (Paperno et al., 2016) dataset and chat assistant on the OpenAssistant (Köpf et al., 2023) dataset.
For the next-token prediction task, we run the offline algorithm with two models: OPT-1.3B and OPT-13B (Zhang et al., 2022) and use FLOPs as the cost. The target performance metric is the number of correct tokens predicted, where we get the ground-truth token from the Lambada dataset.
Figure 2: Comparisons between LFU with either small or large model switching and LEC with model switcher. Both the  $x$ -axis and  $y$ -axis are logarithmic scales. The shaded regime represents the standard deviation calculated from the repeated experiments.
For a given query, an algorithm can choose to run the small model or the large model. If the small model is chosen but its result is wrong, the large model must be run and it will incur an additional penalty. We fine-tune a BERT base model with 2000 samples as the model switcher by predicting whether the small model can give the correct result and achieve  $80.2\%$  accuracy. We work with 100 unseen distinct prompts in the offline setting with total queries 10000 and cache size 40. We compare our offline caching and switcher algorithms against LFU, large-model-only, and cascade (which always calls the small model first). As shown in Table 1, LEC is better than LFU in all cases. Combining LEC and switcher brings up to  $4.3 \times$  cost reduction compared to the baseline “LFU + Large.” However, as the predictor accuracy is limited, the model switcher may not be as good as the cascade algorithm in some cases. We leave the training of a better switcher as future work.
On the chat assistant task, we run the online algorithm with two models: FastChat-T5-3B and Vicuna-13B (Chiang et al., 2023), and use the inference latency as the cost. The quality of response is evaluated by GPT4 evaluation (Liu et al., 2023). We say a response is satisfying if the score is larger than 6 out of 10, and unsatisfying otherwise. If the response from the small model is unsatisfying, we will call the large model again and incur an additional cost in latency. The ratio between the average latency of the large model and the small model is 1.85. We work with 100 distinct prompts in the online setting with total queries 10000 and cache size 40. After a sufficient number of online learning steps, the switcher learns the accurate costs of two models on this finite prompts set, so "LEC + switcher" outperforms other algorithms in all cases on Table 2 with up to  $1.8 \times$  latency reduction compared to "LFU + large" baseline.
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	80%	3.49	3.81	2.60	3.44	1.50	2.00
0.8	80%	10.81	11.80	8.06	10.36	4.11	4.76
0.2	100%	3.49	3.81	1.91	3.44	1.50	0.99
0.8	100%	10.81	11.80	5.90	10.36	4.11	2.50
Table 1: Evaluation of offline algorithms on the Lambada dataset with OPT-1.3B and OPT-13B, 100 distinct prompts, total query size 10000 and cache size 40.  $\alpha$  is the parameter of the power distribution of the prompts. The table lists cumulative costs  $(10^{3})$  for different algorithms.
We provide more experiments in Appendix I, where we evaluate both FLOPs and latency for both offline and online setting on both synthetic and real dataset, with varying cache size, query size and distinct prompts.
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	9.31	13.88	7.24	8.74	8.82	5.93
0.5	20.04	29.88	15.11	18.68	16.90	11.87
0.8	28.24	42.12	21.14	26.07	20.31	15.49
Table 2: Evaluation of online algorithms on the OpenAssistant dataset with FastChat-T5-3B and Vicuna-13B, 100 distinct prompts, total query size 10000 and cache size 40.  $\alpha$  is the parameter of the power distribution of the prompts. The table lists cumulative costs  $(10^{3})$  for different algorithms.
We have studied the joint optimization of caching and model multiplexing and proposed an optimal algorithm for the tabular case. There are a variety of further work that can be pursued in this vein, including:
- Designing the optimal caching and model multiplexing algorithm when there is a query queue, such that the query arrives at a random interval rather than a fixed interval. A more complicated serving pattern also needs to take batching strategies into consideration.
- Understanding the scaling law of the predictors. We hope to use a small yet accurate model for prediction to reduce overhead introduced by the predictor. It is important to understand the trade-off between prediction accuracy, model size, and training data size.
- Designing optimal caching algorithm when the responses generated in each round have diverse qualities.
Banghua and Jiantao are partially supported by NSF IIS-1901252, CIF-1909499 and CIF-2211209. Michael I. Jordan is partially supported by NSF IIS-1901252. Ying Sheng and Clark Barrett are supported in part by NSF-2110397 and the Stanford Center for Automated Reasoning.
M. Arlitt, L. Cherkasova, J. Dilley, R. Friedrich, and T. Jin. Evaluating content management techniques for web proxy caches. ACM SIGMETRICS Performance Evaluation Review, 27(4): 3-11, 2000.
H. Bahn. Web cache management based on the expected cost of web objects. Information and Software Technology, 47(9):609-621, 2005.
A. Bakhtiarnia, Q. Zhang, and A. Iosifidis. Single-layer vision transformers for more accurate early exits with less overhead, 2022.
H. Bast, B. Buchhold, E. Haussmann, et al. Semantic search on text and knowledge bases. Foundations and Trends® in Information Retrieval, 10(2-3):119-271, 2016.
E. Beeching, Y. Belkada, K. Rasul, L. Tunstall, L. von Werra, N. Rajani, and N. Lambert. StackLLaMA: An RL fine-tuned LLaMA model for Stack Exchange question and answering, 2023. URL https://huggingface.co/blog/stackllama.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent
O. Sharir, B. Peleg, and Y. Shoham. The cost of training NLP models: A concise overview. arXiv preprint arXiv:2004.08900, 2020.
J. Shuja, K. Bilal, W. Alasmary, H. Sinky, and E. Alanazi. Applying machine learning techniques for caching in next-generation edge networks: A comprehensive survey. Journal of Network and Computer Applications, 181:103005, 2021.
A. J. Smith. Cache memories. ACM Computing Surveys (CSUR), 14(3):473-530, 1982.
W. Stallings and G. K. Paul. Operating Systems: Internals and Design Principles, volume 9. Pearson New York, 2012.
J. Wang. A survey of web caching schemes for the internet. ACM SIGCOMM Computer Communication Review, 29(5):36-46, 1999.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a.
T. Wei, J. Qi, and S. He. A flexible multi-task model for BERT serving, 2022b.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, H. Peng, J. Li, J. Wu, Z. Liu, P. Xie, C. Xiong, J. Pei, P. S. Yu, and L. Sun. A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT, 2023.
D. M. Ziegler, N. Stiannon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
A Discussions on the Choice of Output, Model and Cost
The proposed framework is flexible in the choice of outputs, models and costs. Below we discuss several possible choices and combinations of output, models and costs that are most practically relevant.
Per-token Output and Per-sentence Output. We have two design choices of the desired output in each round, namely per-token output and per-sentence output.
For per-token output, we aim at generating one token at each round as a response of the queries. In this case, we only cache the next token for a given query and estimate the cost for generating next token. We also have the flexibility of choosing different models to generate each token in each round.
For per-sentence output, we aim at generating a complete response at each round. In this case, we cache the whole responses for a given query, and estimate the cost for generating the whole responses. This may introduce more variance in the cost due to the variation and randomness in the length of the generated responses.
Choices of Costs The cost can be chosen as FLOPS, latency of the model, the price for API calls, user satisfaction of the results, or a combination of all the four factors.
Model multiplexing A common choice of model ensembles is a pair of small and large models. The cost for small model  $C_s(q)$  can be written as  $C_s(q) = C_{s,0}(q) + Y(q)C_{s,1}(q)$ . Here  $Y(q)$  is a binary random variable, indicating whether the small model outputs satisfying results  $(Y(q) = 0)$  or not  $(Y(q) = 1)$ . In the case when the small model outputs a satisfying response, the incurred cost is  $C_{s,0}(q)$ . In the case when the small model outputs a bad response, the incurred cost is  $C_{s,0}(q) + C_{s,1}(q)$ . We discuss two possible choices of  $Y(q)$ ,  $C_{s,0}(q)$  and  $C_{s,1}(q)$  based on two different evaluation pipeline as below.
- One-time evaluation pipeline. For the one-time evaluation pipeline, we can only call one of the models once and the generated content cannot be changed. In this case,  $C_{s,0}(q)$  can be set as the cost for running the small model to generate responses,  $Y(q)$  is set to be 1 if the user is not satisfied with the response, and  $C_{s,1}(q)$  is the incurred cost for unsatisfactory of the user. One can similarly set the same cost for the large model.
- Correction-based evaluation pipeline. For correction-based evaluation, we may re-generate the content with a different model if it is unsatisfying, and get an extra cost for fixing the content. Such evaluation can be easily combined with LLM Cascade (Frantar et al., 2023) or the idea from Big Little Transformer Decoder (Kim et al., 2023) and Spectulative sampling (Chen et al., 2023a). For example, after running the small model, we run the large model once to infer all the log probabilities of the small model output in parallel, and reject its output if the log probabilities are low. If the small model output is rejected, we will set  $Y(q) = 1$  and run large model to re-generate the responses. In this case,  $C_{s,0}(q)$  is the cost of running the small model for generating responses, and running the large model once for checking the probability. And  $C_{s,1}(q)$  is the cost of running the large model to generate the response.
We also remark here that in the special case when the cost for the small model is much smaller than that of the large model under the correction-based evaluation pipeline, the cascade selector which always runs the small model first may give better performance than the model multiplexer if the accuracy of the model multiplexer is low, since running small model does not introduce too much cost compared to running large model. In this situation, the cascade selector can also be combined with LEC caching to further improve the performance.
On the other hand, we may also choose among models with similar size but different expertise, including coding, summarization and chat etc. In this case, we also expect to see different qualities and cost of responses for specific queries.
B Generalization to Variable Size Cache
For the variable-size caching problem, assume that the cache size of  $q$  is a deterministic scalar, denoted as  $S(q)$ . In the population case we design the cache as follows:
$$
\mathcal {L} ^ {\star} = \underset {\mathcal {L}: \sum_ {q \in \mathcal {L}} S (q) \leq L} {\arg \min } \sum_ {q \in \mathcal {Q}} P (q) \mathbb {1} (q \notin \mathcal {L}) \min  \left(c _ {s} ^ {\star} (q), c _ {l} ^ {\star} (q)\right).
$$
In the case when all  $S, P, c_s^\star, c_l^\star$  are known, one may solve the above constrained optimization problem for the optimal caching. When  $S(q) \ll L$ , a good cache replacement algorithm is GDSF itself, which replaces the query with the smallest expected cost per-size  $P(q) \min (c_s^\star (q), c_l^\star (q)) / S(q)$  rather than expected cost per-query.
A more practical setting is the case when the cache size for each query  $S(q)$  is a random variable. Due to the randomness in the generation procedure, we expect to see responses of different lengths even when we use the same model to process the same query. In each round, we will have a generated response with size  $s(q)$  that is sampled from the random variable  $S(q_{t})$ . We conjecture that the optimal cache replacement algorithm is to replace the query with the smallest expected cost per-size  $P(q) \min (c_s^{\star}(q), c_l^{\star}(q)) / s(q)$  as well, where  $s(q)$  is the size of the cached queries and responses.
C Generalization to Multiplexing of Multiple Models
The proposed algorithm can be generalized to model multiplexing with multiple models. Assume that we have  $K$  models, and each model has a random cost function  $C_k(q)$  with expectation  $c_k^\star (q)$ . In this case, the optimal population algorithm is
$$
\pi^{\star}(q) = \operatorname *{arg  min}_{k\in [K]}c_{k}^{\star}(q),
$$
$$
\mathcal {L} ^ {\star} = \underset {\mathcal {L}: | \mathcal {L} | \leq L} {\arg \min } \sum_ {q \in \mathcal {Q}} P (q) \mathbb {1} (q \notin \mathcal {L}) \underset {k \in [ K ]} {\min } c _ {k} ^ {\star} (q).
$$
And the finite sample algorithm is natural to follow. In practice, one may train a neural network with  $K$  dimensional output to predict the cost for each of the models.
D Differences Between the Optimal Policy and the Baseline
Consider the population setting in Section 3, where we optimize caching without model multiplexing. We show via a simple example below that without considering the cost for individual query, LFU can be highly sub-optimal compared to the optimal caching strategy in the population. The ratio
Proposition 1. For any fixed cost function  $c_{l}^{*}$ , one can design some distribution of queries  $P$  such that for any  $\epsilon > 0$ ,