# Parallelism Strategies for Big Data Delayed Transfer Entropy Evaluation
## 3 Material and Methods
### Transfer Entropy
```
1:for experiment in experiment_list_file do
2: experiment_signal \(\leftarrow\) load_signal_from_disk (experiment)
3:for each channel in experiment_signal do
4: calculate embedding(channel)
5:endfor
6:for each two-channel permutation in experiment_signal do
7: calculate DTE(channel1, channel2, channel2_embedding)
8:endfor
9:for\(i=0\) to num_surrogates do
10: surrogate \(\leftarrow\) generate_surrogate (experiment_signal)
11:for each two-channel permutation in surrogate do
12: calculate DTE(channel1, channel2, channel2_embedding)
13:endfor
14:endfor
15:endfor
```  
**Algorithm 2** Execute DTE with surrogate  
Our surrogate algorithm (Algorithm 1 uses FFTW library [84] to archive the best performance during FFT and inverse fast Fourier transform (IFFT) routines using arbitrary-size vector transforms. Embedding was calculated to find the target variable's past, which can be found in the first local minimum of auto-mutual information or first zero crossing auto-correlation measures [85]. The previous existent code comes from several years of development, is arbitrarily named as code A and is shown as a flowchart in Figure 2a. code A is a Python [86] script which uses libraries such as numpy [87], IPython [64] and lpslib (developed in-house by LPS laboratory). It tackled performance issues by using Data Parallelism, made possible by an IPython _ipyparallel_ library. On code a, for each experiment, signals are read on the host node, the embedding code is executed on the host node, embedding pushes corresponding channel data to all computing nodes, and each computing node processes one part of the data and results are gathered on the host node. Note that embedding is executed once for each channel. After embedding, DTE is executed on the host node for each two channel permutation; it pushes channel data to all computing nodes, each node processes one part of the data and results are gathered on the host node. The decision to try another parallelism strategy comes from analyzing CPU load during code a execution and observing node processors' sub-utilization. This observation was done by executing _hotp_[88] program on a computing node and checking that system load average was significantly smaller than the number of processors. By reading Linux _proc_ manual [88], DTE calculation was not in the run queue or waiting for disk I/O to fully load (meaning the load average is equal to the number of processors) each node. Further investigation by adding extensive logging facilities to code a confirmed that the low load average was caused by network communication bottleneck. The code was refactored to use task-based parallelism, aiming to mitigate communication bottleneck. The idea is to load signal data from local storage instead of network transfer from host node. Refactored code was named as code b and is shown in Figure 2b. Before executing code b, every signal data must be copied to each computing node. The main difference from code a is that every experiment is read once, every embedding task is executed, and, finally, one task is created for each two-channel permutation for every experiment. Tasks are asynchronously executed and scheduled by IPython. Figure 2: Data and Task Parallelism algorithm flowcharts. Note that each dashed line means data transfer to every cluster node and every dotted line means data gathering and synchronization to host node. Objects of this study are highlighted in blue. Both code a and code b were executed with a different number of surrogates (1, 5, 10 and 20) to compare performance between them, except 20 surrogates for code a due to excessively long execution time (estimated in more than 6000 min by extrapolating results from code a with a smaller number of surrogates).