## 4 Experiment
### Accuracy  
The primary feature of Mobibench is to measure the performance of file operation and the performance of database operation. The accuracy of performance measurement should be guaranteed. We extracted the performance of file I/O using Mobibench, IOzone, and AndroBench. Figure 6 illustrates the result.  
File size and I/O size in the experiment were set to 512 MB and 4 KB, respectively. Mobibench and IOzone show similar throughput and IOPS on sequential and random operations, respectively. Random write performances observed in Mobibench and IOzone differ only by \(2\%\).  
Figure 6: I/O on internal eMMC. (**a**) Synchronous write; (**b**) Direct read. Filesize: 512 MB; I/O-size: 4 KB [Samsung Galaxy S4, Android 4.2.2 (JB), /data partition, EXT4].  
Mobibench and IOzone support various file open modes. We compared the two benchmarks using six different I/O modes: buffered read/write, mmap read/write, synchronous write, and direct write. File size and I/O size were set to 512 MB and 4 KB, respectively. We measured the performances of sequential and random I/O on all six modes. The performance results from Mobibench are similar to the results from IOzone (Figure 7). However, sequential read performances of Mobibench in buffered and mmap mode are about 20\(\%\) to 160\(\%\) higher than those of IOzone. It is because IOzone calls fsync() at the end of the run.  
Next, we measured the performance of SQLite operation using Mobibench with various smartphone models and Android versions. We used TRUNCATE journal mode for SQLite to measure the performance. The presented results are averages of 10,000 runs. Figure 8 shows the insertion and update performances of SQLite. We have tested a total of twelve smartphone models. We used two different versions of Android (Jelly Bean and Ice Cream Sandwich) on Samsung Galaxy S3. Most of Jelly Bean loaded devices exhibited better performance than Ice Cream Sandwich loaded devices. There are two reasons for the performance difference. The first reason is the hardware difference. Jelly Bean loaded  
Figure 7: Mobibench _vs_. IOzone, I/O on internal eMMC. (**a**) Buffered write; (**b**) Buffered read; (**c**) mmap write; (**d**) mmap read; (**e**) Synchronous write; (**f**) Direct write. Filesize: 512 MB; I/O-size: 4 KB [Samsung Galaxy S4, Android 4.2.2 (JB), /data partition, EXT4].  
smartphone models are equipped with faster CPU and faster eMMC (NAND storage for smartphone). The second reason is the software optimization. We loaded the Jelly Bean and Ice Ceram Sandwich on the same smartphone device, Galaxy S3 and examined the performance. Jelly Bean loaded GS3 shows approximately 150\(\%\) better performance than Ice Ceram Sandwich loaded GS3 (63 insertions/sec _vs_. 40 insertions/sec).  
On the other hand, two Jelly Bean loaded smartphones, Nexus 7(Ne7\({}_{j}\)), and Galaxy Nexus(GNe\({}_{j}\)) show 20 insertions/sec and 10 insertions/sec, respectively. These performance numbers are much lower than the performance number of Ice Ceram Sandwich loaded Galaxy S3(GS3\({}_{i}\)), 40 insertions/sec. Mobibench analysis brings us two important findings. First, Android software stack does evolve from SQLite performance' perspective. Each upgrade in Android is known only for its new features, new UI/UX, and security enhancements compared to its predecessor but not much is being said on the performance optimization effort. We found that Linux kernel and SQLite are becoming more efficient as they proceed and SQLite exhibits 1.5\(\times\) performance gain compared to its predecessor. Second, hardware inefficiencies can be offset by software optimization. On the same token, smartphone vendors need to use better hardware, e.g., CPU, DRAM, and eMMC, to overcome the inefficiencies in software.