# To Cache or Not to Cache
## 4 Evaluation
### Setup
#### 4.1.4 Metrics  
As previously mentioned, the cache simulators collect information about read, writes, cache hits and misses, and total writes incurred. **FOMO** aims to not only improve upon its underlying cache replacement algorithms, but also be comparatively better than its non-datapath cache algorithm peers and admission policy peers. For the evaluation, metrics include the mean cache hit rate and the mean write rate; greater write rates indicate lower flash cache device lifetime. To evaluate cache performance, the normalized results are compared across workloads and cache sizes to understand the performance of **FOMO**. Normalized hit rates and write rates are computed for a given workload-cache size combination with respect to the best performing algorithm. This normalization method better shows the general performance of an algorithm among its peers. Furthermore, it provides a fair method of comparison, as a total average hit rate comparison along would hide  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **\# Traces** & **Details** \\ \hline \multirow{3}{*}{FIU [21; 22]} & \multirow{3}{*}{10} & End user/developer home directories; Web server for faculty/staff/students; Apache webserver for research projects; Web interface for the mail server; Online course management system \\ \cline{1-1} \cline{2-3} MSR [21; 23] & & User home directories; Hardware monitoring; Source control; Web staging; Test web server \\ \hline CloudVPS [24] & 18 & VMs for cloud provider \\ \hline \multirow{2}{*}{CloudCache [24]} & \multirow{2}{*}{6} & Online course management website; Web server for a CS department user webpages \\ \cline{1-1} \cline{2-3} CloudPhysics [25] & & \\ \cline{1-1} \cline{2-3}  & & VMware VM block traces. Due to the size of many of the traces within this set, only the first day of each trace was used for tests. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Sources and descriptions for the five storage datasets used in this paper.  
how well (or poorly) an algorithm was performing compared to others. When comparing write rates, the best performer has the lowest write rate, as this translates to the fewest writes/updates, which is one of the goals of non-datapath caching algorithms. In order to prevent small differences in metrics leading to large normalized differences, the results where the performance used for normalizing was less than 10% were not included. Caches for which the best performing algorithm produces extremely small hit-rates are unlikely to be of practical use in any case.  
Each measurable aspect of **FOMO** is given its own section. First, we discuss how **FOMO** is able to limit writes to the cache in a manner similar to the state-of-the-art non-datapath caching policies. Second, we discuss **FOMO**'s hit rate performance compared to other non-datapath caching algorithms. Third, we compare the **FOMO** and TinyLFU admission policies using different cache replacement algorithms. Fourth, we analyzed how **FOMO**'s Miss-History is able to detect patterns of reuse within the workload, using a particular workload for a case study. Finally, we discuss the _adversarial workloads_ of CloudCache and why **FOMO** and the other non-datapath caching algorithms see performance degradation compared to datapath cache algorithms.