# Revisiting the Design of Parallel Stream Joins on Trusted Execution Environments
## 6 Towards More Efficient Parallel Stream Join on Multicore Processors with TEEs  
In the baseline scenario of a standard stream join code ported directly to the sgx environment without any consideration for architectural differences, the performance overhead imposed by the SGX environment will be modeled as depicted in Equations (1) and (2). The secure enclave will have to allocate memory for both input streams as well as the output join. The size of the input streams is known to the developer beforehand, whilst the output join's size can only be determined after the computation is complete. Hence, the maximum memory required is allocated. We assume the size of the hashtable key values and auxiliary variables to be negligible in comparison.  
\[\#EPC_{S}\approx\frac{(N_{R}\times|R|+N_{S}\times|S|+N_{R}\times N_{S}\times| J|-Mem_{Encl}}{4\times 10^{3}} \tag{1}\]  
\[T_{\mathsf{M}}^{Encl}\approx L_{ECall}+L_{OCall}+\#EPC_{S}\times(L_{EPC}+C_{ Intl}+C_{Shut})+C \tag{2}\]  
During enclave initialization and shutdown (\(C\)), all EPC pages must be processed and deallocated respectively. Hence, the more memory is available to the enclave, the higher the initialization and shutdown overheads are expected to be. Note that \(Mem_{Encl}\) refers to the effective amount of memory available to the enclave as opposed to the total amount of memory accounted for. For instance, in a 128 Mib Enclave, \(Mem_{Encl}\) is 93 Mib.  
With the introduction of SGX2 [53], the instruction set of SGX was extended to support dynamic memory allocation and deallocation which allows us to free the memory occupied by the input tuples as they are being processed into the hashtable. SGX2 also brings an enhancement to the previous TCS by allowing their allocation at runtime. This is enabled by the new capability of changing page permissions post-enclave-initiation. However, as elaborated on in [53], concurrency-related complexity in SGX2 is greatly increased over SGX1.  
In contrast to the single-threaded baseline expressed in Equations (1) and (2), we consider multiple threads accessing the same enclave. Note that the threads are not dynamically spawned inside the enclave, but rather defined at the untrusted code level and access the enclave independently in parallel. In this context, the ECalls and OCalls performed will evoke a series of EPC page swaps to process the parameters associated;  
Figure 7: Effect of Partition Matrix Shape.  
the overhead of which will accumulate linearly in the worst-case scenario, as captured by Equation (3).  
\[T_{\mathbf{\kappa}}^{Encl}\approx(\frac{N_{R}}{M_{R}}+1)\times(\frac{N_{S}}{M_{S}}+1) \times L_{ECall}+C \tag{3}\]  
Finally, we consider a runtime environment whereby multiple secure enclaves are initialised. Ideally, the enclaves can be locally initialised on the server provided it can sufficiently scale-up. Or alternatively, an MPI (scale-out) configuration of servers can be established to provide the resources required. In such configuration, provided proper code design, the EPC size is no longer a limitation and only a single call into the respective enclaves need to be made, effectively reducing all hardware-induced overheads to the bare possible minimum expressed in Equation (4).  
\[T_{\mathbf{\kappa}}^{Encl}=L_{ECall}+C \tag{4}\]