## 5 IO Data Management and Data Sharing
### Distributed SQL Databases with RPC
#### 5.1.1. Replicated Data Bases and Tables
The parallelism degree can be maximal \(j\) (neurons), considering the parallelism of the summation unit of neuron rather than maximal \(j\times k\), with \(k\) as the number of neurons of the previous layer (and assuming a fully connected layer interconnects). \[ \begin{array}{c}\hat{a},\hat{c}:\mathbb{R}^{N\times N}\\ \stackrel{{\rightarrow}}{{l}},\stackrel{{ \rightarrow}}{{m}}:\mathbb{R}^{N}\\ \hat{K}:\mathbb{R}^{2r+1\times 2r+1}\\ \hat{c}=\text{conv}(a,K)\\ c_{ij}=\sum\limits_{n=-r}^{r}\sum\limits_{m=-r}^{r}a_{i+n,j+m}\cdot K_{n,m}\\ \stackrel{{\rightarrow}}{{l}}=\text{sigmoid}\left(\overrightarrow{ \text{sum}}(\hat{c})\right)\\ \text{sum}_{i}(c)=\sum\limits_{n=1}^{N}\sum\limits_{m=1}^{N}w_{i,n,m}\cdot c_{ n,m}\\ \stackrel{{\rightarrow}}{{m}}=\text{sigmoid}\left(\overrightarrow{ \text{sum}}\left(\stackrel{{\rightarrow}}{{l}}\right)\right)\\ \text{sum}_{i}(l)=\sum\limits_{n=1}^{N}w_{i,n}\cdot l_{n}\\ \text{sigmoid}(x)=\frac{1}{1+e^{-x}}\end{array}  \tag{2}\]

The program code for the light version of a CNN is shown in Example 10.

const gpu = new GPU.GPU({}) //Pipe-linedcomposedGPUkernel const convMatrix = gpu.createKernel(function(src)const kSize = 2*this.constants.kernelRadius+1; let sum=0; const tx = this.thread.x, ty = this.thread.y; for(let i = -this.constants.kernelRadius; i <= this.constants.kernelRadius; i++) { const x = tx + i; if (x < 0 || x >= this.constants.width) continue; for(let j = -this.constants.kernelRadius;j <= this.constants.kernelRadius;j++) { const y = ty + j; if (y < 0 || y >= this.constants.height) continue; const kernelOffset = (j+this.constants.kernelRadius)*kSize+ i+this.constants.kernelRadius; const weight = this.constants.kernel[kernelOffset]; const pixel = src[y][x]; sum += (pixel * weight); } } return sum }).setPipeline(true).setOutput([N,N]).setConstants({width:N, height:N, kernel:kernels.boxBlur,kernelRadius:1}); const layer1 = gpu.createKernel(function(src) { let sum=0; for(let i=0;i<this.constants.width;i++) { for(let j=0;j<this.constants.height;j++) { sum += (1/this.constants.height*src[j][i]); } } return 1/(1+Math.exp(-sum)); }).setPipeline(true).setOutput([N]).setConstants({width:N, height:N }); const layer2 = gpu.createKernel(function(src) { let sum=0; for(let i=0;i<this.constants.height;i++) { sum += (1/this.constants.height*src[i]); } return 1/(1+Math.exp(-sum)); }).setPipeline(true).setOutput([N]).setConstants({height:N }); const cnn = gpu.combineKernels(convMatrix,layer1,layer2,function (a) { return layer2(layer1(convMatrix(a))) }) var input = GPU.input(newFloat32 Array(..),[N,N]); var output = cnn(input).toArray(); ```
Example 10. The source code for the CNN light GPU test with the gpu.js library. ### Evaluation

The following analysis shows some selected experiments for a single-matrix multiplication and a pipelined computation that is typical for an ML task with CNN models performing matrix convolution (O(N\({}^{2}\))) and fully connected perceptron layers, containing summation and a functional application (sigmoid function) to all elements of the input matrix (vector) (up to an O(N\({}^{3}\)) complexity). The figures show the computation times for CPU and GPU processing, depending on the matrix size \(N\). On the right side, the tables show the achieved CPU/GPU speeding-up. Using generic integrated GPUs (Intel), a speeding-up of up to 5 could be achieved depending on the data size, CPU, JS VM, and GPU, as shown in Figures 7 and 8. The matrix data were stored in flat and compact binary Float32 arrays. The top figure analysis shows the results achieved in a WorkBook worker, and the bottom figure analysis shows the results achieved with WorkShell workers. Below a threshold of about \(N\) = 300, the speeding-up is below 1 when using GPU processing. The reasons are the CPU-GPU memory transfer and GPU warm-up times becoming dominant for small-loop iterations. The processing time of Vanilla JS follows the expected O(N\({}^{3}\)) complexity, except in the web browser, where is a strong degradation around \(N\) = 1000 that was also observed in different browsers.