# An Optimization Methodology for Adapting Legacy SGX Applications to Use Switchless Calls
## 4 Approach
### Building an Optimization Strategy  
Our key insight to address this problem is that an application developer can selectively determine whether utilize switchless calls for ECALL and OCALL. Rather than naively adopting switchless calls to entire ECALLs and OCALLs, our approach selectively applies switchless calls by examining the performance benefit for each ECALL and OCALL. To achieve this, we explore factors that affect the efficiency of switchless SGX when adopting it to legacy SGX applications. For simplicity, we assume that a caller thread and a worker thread are allocated per core. A switchless call is efficient for an ECALL or OCALL when it satisfies the below condition. For a unit of synchronous execution, the gain of using switchless call is defined as:  
\[\text{Gain of using switchless} =\frac{\text{The CPU time saved during ECALL or OCALL}}{\text{The pended CPU time due to worker threads}} \tag{1}\] \[=\frac{N_{et}*T_{et}}{\text{The pended CPU time due to worker threads}}\] \[>1\]  
where \(N_{et}\) is the number of enclave transitions occurred and \(T_{et}\) is an enclave transition time. However, calculating the pended CPU time due to worker threads is not trivial. Therefore, we examine which factors of ECALL and OCALL affect the pended CPU time. Based on the analysis, we learn that two major factors are related to pending caller threads: a frequency of ECALL/OCALL and a completion time of ECALL/OCALL. Frequency of ECALL/OCALL: The pended CPU time raised by the worker thread is affected how ECALL/OCALL is frequently called. For convenience, we will explain the case of ECALL. We first take a look at the implementation of switchless calls to understand the execution flow. The execution flow of switchless worker threads is divided into three steps:  
1. If there are sleeping workers, an SGX application wakes them up when an ECALL is invoked. 2. Once a worker thread is scheduled, it looks up the request queue and processes an enclave function. 3. If the request queue is empty, it retries until max_retries (default = 20,000) and falls asleep. As the Figure 2 shows, the workflow of worker threads depends on the emptiness of the request queue. If an ECALL is rarely invoked, the corresponding worker thread accesses the request queue until max_retries (Step 3) and waking up worker threads (Step 1) frequently happens, which wastes the CPU time. Therefore, pended CPU time due to worker threads is proportional to the frequency of enclave transition. Completion time: The other factor that affects the pending time is the completion time of ECALL/OCALL. We bring the execution flow of the SGX-Box thread with switchless calls as an example for better understanding. Figure 3 demonstrates workflows of the SGX-Box caller thread and the corresponding ECALL conducted by the switchless worker thread. When an ECALL (ecall_start_tls_process in Figure 3) is invoked by an SGX-Box thread, it updates the request queue. Then, worker threads execute the corresponding ECALL function once the worker is scheduled. While executing the ECALL, context switches might occur if the completion time of the ECALL is long enough to make the worker thread scheduling out. It leads to pending of the SGX-Box caller thread because it works synchronously, which means that it cannot perform the remaining tasks until the worker thread completes the execution of the ECALL. Therefore, a completion time of ECALL/OCALL affects the number of context switching, which also involves the pending time due to the switchless worker thread. Figure 3: An example of Switchless ECALL in SGX-Box application. Context switches while executing ECALL function introduce pending of the SGX-Box caller thread. Figure 2: Execution flow of switchless worker threads depending on the queue’s emptiness. In summary, we learned that  
\[\text{Pended CPU time due to worker threads}\propto T_{w},\quad\propto\frac{1}{N_{et}} \tag{2}\]  
where \(T_{w}\) is a completion time of \(\text{ECALL}/\text{OCALL}\).