# SMARTEN--A Sample-Based Approach towards Privacy-Friendly Data Refinement
## 5 Evaluation
### Performance Measurement  
To determine the performance overhead caused by SMARTEN, we have implemented three different data preparation rules. Each of these rules reflects one of the three transformation operators, namely a filter operator, a map operator, and a reduce operator: The filter operator eliminates all data items with a certain attribute value (i.e., a selection). The map operator removes a certain attribute (i.e., a projection). The reduce operator groups the data items by a categorical attribute and calculates the mean value of an attribute for each group (i.e., an aggregation). These rules are then applied to base data. A relational database (_SQLite DB 3.37.2_, see [[https://www.sqlite.org/](https://www.sqlite.org/)]([https://www.sqlite.org/](https://www.sqlite.org/)), accessed on 31 July 2022) is used to store the base data. We implemented the processing engine which is required in the Preparation Zone using _Python 3.10.3_ (see [[https://www.python.org/](https://www.python.org/)]([https://www.python.org/](https://www.python.org/)), accessed on 31 July 2022) and _Pandas 1.4.1_ (see [[https://pandas.pydata.org/](https://pandas.pydata.org/)]([https://pandas.pydata.org/](https://pandas.pydata.org/)), accessed on 31 July 2022). All base data are loaded into a _DataFrame_, where the transformations are applied. The database as well as the processing engine are deployed on a computer with an _Intel Core i7-1165G7_ and 32 GB DDR4-3200 main memory. For the evaluation, we used synthetic base data. On that account, we generated stepwise from 500 to 5000 k data items. In each step, we have increased the amount of data tenfold. This represents a maximum data volume of about 12 GB. In our evaluation, we measured both the data throughput, i.e., how much data per second is processed, and the main memory peak usage. For each operator and each base data size, we performed the measurements ten times. After each run, we reset the database to exclude distortions due to warm caches. In Figure 7, the median of the runs are shown--we use the medians to exclude side effects caused by background processes. For comparison purposes, we have also implemented the three operators as SQL commands which can be executed directly by the SQLite DB. We use this as our baseline. This represents an optimum. In this baseline, the initial retrieval of the data is eliminated and the available index structures of the database can be used to support the processing. Therefore, the closer the processing engine used in SMARTEN approximates this baseline, the better the performance has to be assessed. Figure 7: _Cont._It can be noticed that the selection data throughput and the projection data throughput hardly show any differences (see Figure 7a,c). Even though the throughput of SMARTEN is lower than the one of the baseline, both show an identical progression. That is, the throughput of SMARTEN is in \(\mathcal{O}(baseline)\). For the aggregation, the two curves are also somewhat similar (see Figure 7e). Here, however, the difference between the two curves is significantly bigger than in the case of the selection and the projection. This is due to the fact that, especially in the case of grouping, the database can make use of index structures concerning the categorical attribute. It is noticeable that the performance of SMARTEN drops for the biggest set of base data for in all three operators. This is due to the Pandas-based implementation of the processing engine. Pandas loads a DataFrame completely into main memory. Since in our evaluation the database and the processing engine share the available main memory, there is almost no free main memory left for this base data volume.