## 5 IO Data Management and Data Sharing
### Distributed SQL Databases with RPC
#### 5.1.1. Replicated Data Bases and Tables
## 7 Data-Path Parallelism on GPU

Besides control-path parallelism, considered in the previous sections, in this section additional data-path parallelism is considered, performing general-purpose computation on graphics processor units (GPGPU) and the widely available WebGL/OpenGL API. Any computation that can be divided into independent data partitions, like matrix computations, can be processed by multiple GPU threads in parallel. However, GPU transpiled algorithms are not generally faster than the strict sequential code processed by the CPU. Most numerical frameworks (like Tensorflow) that support different CPU/GPU back-end packages either use CPU or GPU computing only. The overhead of GPU transpilation and the low sequential speed of the GPU processing elements can lead to speeding up below 1. GPU transpilation and code execution produce a significant overhead that is only justified if the real computation time (and, hence, the computational complexity) is sufficiently large compared to the overhead, including memory transfer and data code transformations. In this work, and in the PScilab framework, dynamic CPU/GPU switching based on a priori and profiling information chooses the best back-end at run-time, according to complexity measures. The dynamic switching (multiplexing) of computations between CPU and GPU, based on profiling, is eased by the fact that the GPU instruction code is derived from JavaScript kernel functions that are mostly equal to the Vanilla JS functions, except that the kernel function only processes a partition of the entire data. GPU access when performing a computational task is typically limited to one process at a time. If there are multiple worker processes that access the GPU concurrently, then a serialization of the GPU tasks occurs without any additional speeding-up. ### Matrix Operations

Example 9 shows matrix multiplication with a complexity of O(\(N^{3}\)) and \(N\) as the number of rows and columns of the matrix, respectively, using Vanilla JS and GPU kernel functions (compiled by the gpu.js package with WebGL/OpenGL [27]). Note that O(\(N^{3}\)) \(\rightarrow\) O(\(N^{2}\)) is only achieved if there are \(N\) parallel threads or processes. \[ \begin{split}\hat{a},\hat{b},\hat{c}:\mathrm{R}^{N\times N}\\ c_{ij}=\sum\limits_{k=1}^{N}a_{ik}b_{kj}\end{split}  \tag{1}\]

const gpu = new GPU.GPU({}) var N = 1024 var multiplyMatrixGPU = gpu.createKernel(function(a, b) { var sum = 0; var x = this.thread.x % this.constants.N, y = Math.floor(this.thread.x / this.constants.N); for (var i = 0; i < this.constants.N; i++) { sum += a[y*this.constants.N+i] * b[i*this.constants.N+x];return sum; }).setConstants((N:N)).setOutput([N*N]); //usingflatarray functionmultiplyMatrixJS(a, b) { var c = matrix(N,N); for(var y=0;y<N;y++) { for(var x=0;x<M;x++) {  var sum = 0; for(var i = 0; i < N; i++) {  sum += a[y*N+i] * b[i*N+x]; }  c[y*N+x]=sum  } } return c } var a = matrix(N,N), b=matrix(N,N) var c1 = multiplyMatrixGPU(a,b), c2 = multiplyMatrixJS(a,b) Example 9. Matrix multiplication a \(\times\) b, by Vanilla JS and GPU kernel functions (multi-threaded) assuming linear and compact Float32 TypedArray data. Both functions will return a new destination matrix c.

With respect to ML, matrix operation with a complexity of O(\(N^{3}\)) occurs in convolutional neural network models. The convolution operation itself (with a kernel \(K\)) has a complexity of O(\(N^{2}\)), but the next (pooling) artificial neuron layer can pose a complexity of O(\(N^{3}\)) if there are \(N\) neurons in the succeeding layer. A neural network is a pipelined chain of functions. If neurons are arranged in layers, each layer \(L_{1}\)(\(n_{1}\),..., \(n_{i}\)) depends on the computation of the previous layer \(L_{i-1}\). This data dependency limits parallelism on a data-path level significantly.