SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \sum_ {t = t _ {k} (q) + 1} ^ {t _ {k + 1} (q)} \mathbb {1} \left(q \notin \mathcal {L} _ {t _ {k + 1} (q)}, \pi_ {t _ {k + 1}} (q) = \pi\right) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q , \pi)}}\right) \mid E \right] \\ \leq \sum_ {k = 0} ^ {T} \mathbb {E} \left[ g _ {k} (q) \cdot \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}, \pi_ {t _ {k + 1}} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q , \pi)}}\right) | E \right] \\ \leq \frac {B _ {2} | \mathcal {Q} | \log (4 T L / \delta)}{B _ {1}} \cdot \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}, \pi_ {t _ {k + 1}} (q) = \pi) \cdot \min \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q , \pi)}}\right) \mid E \right] \\ \end{array}
$$
We know that  $T_{t_{k+1}(q)+1}(q, \pi) = T_{t_{k+1}(q)}(q, \pi) + 1 = T_{t_k(q)+1}(q, \pi) + 1$  if  $q \notin \mathcal{L}_{t_{k+1}(q)}$  and  $\pi_{t_{k+1}}(q) = \pi$  since the query  $q$  missing the cache will be sent to one of the models, and only the one selected will observe the cost. Thus overall, we know that we have either  $T_{t_{k+1}(q)+1}(q, \pi) = T_{t_k(q)+1}(q, \pi) + 1$ , or  $\mathbb{1}(q \notin \mathcal{L}_{t_{k+1}(q)}, \pi_{t_{k+1}}(q) = \pi) \cdot \min\left(1, \sqrt{\frac{1}{T_{t_k(q)+1}(q, \pi)}}\right) = 0$  and  $T_{t_{k+1}(q)+1}(q, \pi) = T_{t_k(q)+1}(q, \pi)$ . Thus overall, we have
$$
\sum_ {k = 0} ^ {T} \mathbb {E} \left[ \mathbb {1} (q \notin \mathcal {L} _ {t _ {k + 1} (q)}, \pi_ {t _ {k + 1}} (q) = \pi) \cdot \min  \left(1, \sqrt {\frac {1}{T _ {t _ {k} (q) + 1} (q , \pi)}}\right) | E \right] \leq \sum_ {k = 1} ^ {T} \frac {1}{\sqrt {k}} \leq C \sqrt {T}.
$$
Thus we know that the second difference satisfies
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \operatorname {c o s t} \left(\mathcal {L} _ {t}, \pi^ {\star}\right) - \operatorname {c o s t} \left(\mathcal {L} ^ {\star}, \pi^ {\star}\right) \mathbb {I} \left(E ^ {t}\right) \right] \leq C \cdot \left(B _ {2} T \delta + L B _ {2} \sqrt {2 T \log (8 / \delta)} \right. \\ \left. + \frac {L (B _ {2} - B _ {1}) B _ {2} | \mathcal {Q} | L \log^ {2} (T | \mathcal {Q} | / \delta)}{B _ {1}} \cdot \sqrt {T}\right). \\ \end{array}
$$
Overall by taking  $\delta = 1 / T$ , we know that
$$
\mathrm {R e g r e t} (T) \leq \frac {C L (B _ {2} - B _ {1}) B _ {2} | \mathcal {Q} | L \log^ {2} (T | \mathcal {Q} |)}{B _ {1}} \cdot \sqrt {T}.
$$
I.1 Synthetic Datasets
We conduct synthetic online and offline experiments for joint optimization of caching and model multiplexing. We use i.i.d. Bernoulli distributions for two models because we want to mimic the model ensemble use case and give a large penalty to the wrong output. In Figure 2, we plot the cumulative cost and regret in online learning for LFU and LEC caching algorithms. We present more data points in Table 3 and Table 4, under the same setting of 10000 requests with 20 different queries and cache size 10. Similar to the real dataset setting, we compare all combinations of caching strategy choices and model multiplexer choices. We consider the frequency distribution as power distribution with  $\alpha = 0.5$  and 0.8. The ground truth cost for each query processed by both models is set as a sample from  $r \cdot X + 1$ , where  $r$  is called as cost ratio and  $X$  is a random variable generated from a Bernoulli distribution with the parameter 0.5. We consider the model multiplexer accuracy with 0.8 and 1. We repeat the simulation 1000 times and take the mean. Consistent with Figure 2, our simulation suggests that LEC with a perfect model multiplexer significantly improves the baselines when the cost ratio is large. Simulation 1000 times cannot remove all randomness so we can observe some fluctuations. Theoretically, the columns of choosing model 1 and the columns of choosing model 2 should behave similarly.
α	cost ratio	selector accuracy	LFU+ model 1	LFU+ model 2	LFU+ selector	LEC+ model 1	LEC+ model 2	LEC+ selector
0.5	1.5	0.8	5.25	5.24	4.09	4.40	4.36	3.59
0.8	1.5	0.8	7.61	7.60	5.93	5.73	5.68	4.85
0.5	1.5	1	5.25	5.24	3.31	4.40	4.36	2.74
0.8	1.5	1	7.61	7.60	4.81	5.73	5.68	3.68
0.5	100	0.8	148.05	147.38	103.43	29.93	26.83	39.32
0.8	100	0.8	214.93	213.88	150.31	43.77	39.02	49.88
0.5	100	1	148.05	147.38	73.94	29.93	26.83	3.12
0.8	100	1	214.93	213.88	107.63	43.77	39.02	4.19
Table 3: offline synthetic dataset
α	cost ratio	LFU+ model 1	LFU+ model 2	LFU+ selector	LEC+ model 1	LEC+ model 2	LEC+ selector
0.5	1.5	5.35	5.34	4.34	4.60	4.59	3.75
0.8	1.5	7.79	7.78	6.32	6.01	5.98	5.09
0.5	100	150.93	150.37	76.80	31.88	28.65	4.85
0.8	100	220.19	219.49	112.26	46.44	41.45	6.31
Table 4: online synthetic dataset
I.2 Real Datasets
In this section, we provide additional experiments on real-world dataset. We run both offline and online algorithms on Lambada dataset and OpenAssistant dataset, with OPT-1.3B vs OPT-13B or FastChat-T5-3B vs Vicuna-13B. We mainly consider three settings:
- In Table 5-12, we consider distinct prompt size 100, total query size 10000, cache size 40, same as the experiments in the main text.
- In Table 13-20, we consider distinct prompt size 1000, total query size 2000, cache size 100.
- In Table 21-28, we consider distinct prompt size 1000, total query size 2000, cache size 0, same as the experiments in the main text.
Our proposed algorithm mostly gives dominant results over other baselines. In Table 21-28, the cache size is set to be 0 so there is no difference between LFU and LEC.
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	3.77	4.11	2.17	3.71	1.83	1.26
0.5	8.11	8.85	4.54	7.93	3.51	2.29
0.8	11.43	12.47	6.35	10.91	4.63	2.86
Table 5: FLOPs for online lambda dataset, opt-1.3b vs opt-13b
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	349.44	425.05	224.33	349.47	241.89	170.02
0.5	751.78	916.58	469.74	751.77	462.34	321.46
0.8	1059.95	1290.26	656.61	1060.01	596.92	397.01
Table 6: Latency for online lambda dataset, opt-1.3b vs opt-13b
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	1.66	1.68	0.90	1.12	0.67	0.48
0.5	3.56	3.61	1.88	2.20	1.22	0.87
0.8	5.01	5.09	2.64	2.75	1.49	1.04
Table 7: FLOPs for online oasst dataset, fastchat-t5 vs vicuna
Table 8: Latency for online oasst dataset, fastchat-t5 vs vicuna
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	3.49	3.81	2.60	3.44	1.50	2.00
0.5	0.8	7.71	8.43	5.76	7.54	3.09	3.94
0.8	0.8	10.81	11.80	8.06	10.36	4.11	4.76
0.2	1	3.49	3.81	1.91	3.44	1.50	0.99
0.5	1	7.71	8.43	4.22	7.54	3.09	1.97
0.8	1	10.81	11.80	5.90	10.36	4.11	2.50
Table 9: FLOPs for offline lambda dataset, opt-1.3b vs opt-13b
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	324.02	394.22	261.89	324.02	207.71	214.03
0.5	0.8	714.87	872.60	578.91	714.87	417.01	442.05
0.8	0.8	1002.26	1220.93	810.45	1002.25	538.56	549.06
0.2	1	324.02	394.22	197.00	324.02	207.71	141.55
0.5	1	714.87	872.60	435.74	714.87	417.01	287.72
0.8	1	1002.26	1220.93	609.95	1002.25	538.56	358.33
Table 10: Latency for offline lambda dataset, opt-1.3b vs opt-13b
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	1.54	1.55	1.09	1.01	0.56	0.66
0.5	0.8	3.39	3.42	2.41	2.08	1.10	1.31
0.8	0.8	4.74	4.82	3.38	2.61	1.36	1.62
0.2	1	1.54	1.55	0.79	1.01	0.56	0.38
0.5	1	3.39	3.42	1.75	2.08	1.10	0.76
0.8	1	4.74	4.82	2.45	2.61	1.36	0.93
Table 11: FLOPs for offline oasst dataset, fastchat-t5 vs vicuna
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	8.64	12.86	8.06	8.03	7.76	6.73
0.5	0.8	19.07	28.42	17.81	17.66	15.63	14.14
0.8	0.8	26.70	39.91	24.98	24.53	19.01	18.12
0.2	1	8.64	12.86	6.28	8.03	7.76	4.86
0.5	1	19.07	28.42	13.86	17.66	15.63	10.43
0.8	1	26.70	39.91	19.44	24.53	19.01	13.80
Table 12: Latency for offline oasst dataset, fastchat-t5 vs vicuna
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	1.81	1.97	1.62	1.80	1.72	1.52
0.5	3.14	3.42	2.59	3.12	3.01	2.41
0.8	3.67	3.99	2.95	3.64	3.57	2.76
Table 13: FLOPs for online lambda dataset, opt-1.3b vs opt-13b
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.17	0.21	0.17	0.17	0.19	0.17
0.5	0.30	0.36	0.27	0.30	0.33	0.26
0.8	0.35	0.42	0.31	0.35	0.39	0.30
Table 14: Latency for online lambda dataset, opt-1.3b vs opt-13b
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	1.03	1.16	0.96	0.88	0.91	0.85
0.5	1.78	2.01	1.52	1.45	1.47	1.27
0.8	2.08	2.34	1.73	1.64	1.69	1.43
Table 15: FLOPs for online oasst dataset, fastchat-t5 vs vicuna
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	4.60	7.12	5.86	4.53	6.54	5.73
0.5	7.98	12.33	9.33	7.86	11.27	9.08
0.8	9.31	14.38	10.64	9.19	13.12	10.35
Table 16: Latency for online oasst dataset, fastchat-t5 vs vicuna
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	1.40	1.52	1.04	1.38	1.06	0.91
0.5	0.8	2.62	2.85	1.95	2.59	2.15	1.69
0.8	0.8	3.09	3.37	2.30	3.05	2.62	1.98
0.2	1	1.40	1.52	0.76	1.38	1.06	0.57
0.5	1	2.62	2.85	1.43	2.59	2.15	1.13
0.8	1	3.09	3.37	1.68	3.05	2.62	1.35
Table 17: FLOPs for offline lambda dataset, opt-1.3b vs opt-13b
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	0.13	0.16	0.11	0.13	0.12	0.10
0.5	0.8	0.25	0.30	0.20	0.25	0.24	0.18
0.8	0.8	0.29	0.36	0.24	0.29	0.29	0.21
0.2	1	0.13	0.16	0.08	0.13	0.12	0.07
0.5	1	0.25	0.30	0.15	0.25	0.24	0.13
0.8	1	0.29	0.36	0.18	0.29	0.29	0.15
Table 18: Latency for offline lambda dataset, opt-1.3b vs opt-13b
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	0.79	0.90	0.61	0.51	0.41	0.35
0.5	0.8	1.48	1.67	1.13	0.97	0.84	0.68
0.8	0.8	1.75	1.97	1.34	1.13	0.99	0.79
0.2	1	0.79	0.90	0.45	0.51	0.41	0.23
0.5	1	1.48	1.67	0.84	0.97	0.84	0.46
0.8	1	1.75	1.97	0.99	1.13	0.99	0.54
Table 19: FLOPs for offline oasst dataset, fastchat-t5 vs vicuna
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	3.55	5.49	3.42	3.41	4.43	3.15
0.5	0.8	6.65	10.28	6.41	6.45	8.50	5.90
0.8	0.8	7.86	12.14	7.57	7.61	10.06	6.95
0.2	1	3.55	5.49	2.69	3.41	4.43	2.40
0.5	1	6.65	10.28	5.04	6.45	8.50	4.59
0.8	1	7.86	12.14	5.95	7.61	10.06	5.44
Table 20: Latency for offline oasst dataset, fastchat-t5 vs vicuna
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	4.13	4.49	2.88	4.13	4.49	2.88
0.5	4.13	4.49	3.12	4.13	4.49	3.12
0.8	4.13	4.49	3.21	4.13	4.49	3.21
Table 21: FLOPs for online lambda dataset, opt-1.3b vs opt-13b
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.39	0.48	0.31	0.39	0.48	0.31
0.5	0.39	0.48	0.33	0.39	0.48	0.33
0.8	0.39	0.48	0.34	0.39	0.48	0.34
Table 22: Latency for online lambda dataset, opt-1.3b vs opt-13b
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	2.50	2.66	1.72	2.50	2.66	1.72
0.5	2.36	2.63	1.84	2.36	2.63	1.84
0.8	2.34	2.64	1.88	2.34	2.64	1.88
Table 23: FLOPs for online oasst dataset, fastchat-t5 vs vicuna
α	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	10.45	16.08	10.37	10.45	16.08	10.37
0.5	10.48	16.18	11.25	10.48	16.18	11.25
0.8	10.48	16.19	11.54	10.48	16.19	11.54
Table 24: Latency for online oasst dataset, fastchat-t5 vs vicuna
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	4.13	4.49	3.07	4.13	4.49	3.07
0.5	0.8	4.13	4.49	3.07	4.13	4.49	3.07
0.8	0.8	4.13	4.49	3.07	4.13	4.49	3.07
0.2	1	4.13	4.49	2.24	4.13	4.49	2.24
0.5	1	4.13	4.49	2.25	4.13	4.49	2.25
0.8	1	4.13	4.49	2.25	4.13	4.49	2.25
Table 25: FLOPs for offline lambda dataset, opt-1.3b vs opt-13b
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	0.39	0.48	0.32	0.39	0.48	0.32
0.5	0.8	0.39	0.48	0.32	0.39	0.48	0.32
0.8	0.8	0.39	0.48	0.32	0.39	0.48	0.32
0.2	1	0.39	0.48	0.24	0.39	0.48	0.24
0.5	1	0.39	0.48	0.24	0.39	0.48	0.24
0.8	1	0.39	0.48	0.24	0.39	0.48	0.24
Table 26: Latency for offline lambda dataset, opt-1.3b vs opt-13b
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	2.50	2.66	1.84	2.50	2.66	1.84
0.5	0.8	2.36	2.63	1.79	2.36	2.63	1.79
0.8	0.8	2.34	2.64	1.79	2.34	2.64	1.79
0.2	1	2.50	2.66	1.35	2.50	2.66	1.35
0.5	1	2.36	2.63	1.32	2.36	2.63	1.32
0.8	1	2.34	2.64	1.32	2.34	2.64	1.32
Table 27: FLOPs for offline oasst dataset, fastchat-t5 vs vicuna
α	selector accuracy	LFU+ large	LFU+ cascade	LFU+ selector	LEC+ large	LEC+ cascade	LEC+ selector
0.2	0.8	10.45	16.08	10.05	10.45	16.08	10.05
0.5	0.8	10.48	16.18	10.09	10.48	16.18	10.09
0.8	0.8	10.48	16.19	10.10	10.48	16.19	10.10
0.2	1	10.45	16.08	7.91	10.45	16.08	7.91
0.5	1	10.48	16.18	7.94	10.48	16.18	7.94
0.8	1	10.48	16.19	7.94	10.48	16.19	7.94
Table 28: Latency for offline oasst dataset, fastchat-t5 vs vicuna