## 5 IO Data Management and Data Sharing
### Distributed SQL Databases with RPC
#### 5.1.1. Replicated Data Bases and Tables
``` for(vari=0;i<P;i++) workers[i]= newWorker('ws:shellhost...',i);Shellworker remotely newWorker('ws:proxyhost...',i);orPoolProxy newWorker(i);//orWeb/Shellworkerlocally for(vari=0;i<P;i++)awaitworkers[i].ready(); functionloadAndProcessData(OPTIONS) { vardb=DB.sql(OPTIONS) for(i=1;i<ROWS;i++) data.push(awaitdb.select(INPUTDATATABLE,'*','rowid='+i); vardata=data.map(function(row){returnSQL2 DATA(row)} data=data.map(function(row){returnML.scale(row,scale)} varparts=ML.split(data,TRAINPART,TESTPART) this.dataTrain=parts[0];this.dataTest=parts[1]; send({RESULT}) } functioncreateModel(OPTIONS){ this.model=ML.learner(ML.ML.ANN,{OPTIONS}) send({RESULT}) } functiontrainModel(OPTIONS){ ML.train(this.model,this.dataTrain,{OPTIONS}) send({RESULT}) } functiontestModel(OPTIONS){ ML.predict(this.model,this.dataTest,{OPTIONS}) send({RESULT}) } //Foursequentialphases for(vari=0;i<P;i++) workers[i].eval(loadAndProcessData,OPTIONS);//asynccall for(vari=0;i<P;i++) results[i]=awaitworkers[i].receive(),checkResult(results[i]) for(vari=0;i<P;i++) workers[i].eval(createModel,OPTIONS);//asynccall for(vari=0;i<P;i++) results[i]=awaitworkers[i].receive(),checkResult(results[i]) for(vari=0;i<P;i++) workers[i]. workers[i].evalf(testModel,OPTIONS); // async call for(var i=0;i<P;i++) results[i]=await workers[i].receive(), checkResult(results[i]) ```
Example 12. A typical ML multi-model training program, with hybrid parallel-distributed worker clusters (shortened form). The problem size in this case study is dynamically growing. Each new worker trains a new independent model, i.e., the number of workers _PN_ is equal to the parallel training of _PN_ models from the same input training data. For the sake of comparability, it is assumed that models trained in parallel differ only in their parameter state (i.e., due to randomness in the training process) but they are structurally equal, with the same computational complexity. The training is incremental and selective. Due to the stochastic initialization of the model parameters and, optionally, stochastic gradient descent (SGD) training (selecting single data instances randomly), the different models develop differently and demonstrate different training progress, as given by the loss function. During parallel training, the best models are selected for further training iterations. Additionally, different model network configurations (static model parameters) can be evaluated, i.e., performing a parallel hyperparameter space exploration. The problem used is a typical image classification problem (here, with 4 different class labels). A standard convolutional neuronal network (CNN) was used. The input data consist of a large set of small-segment images (64 \(\times\) 64 pixels) taken from 4 K underwater images; the output is the prediction of a scene class from a set of classes (three different scene classes). The CNN software framework that was used was derived from the ConvNet.js framework [29]. The CNN model had the following configuration and the trainer was an _adadelta_ algorithm:
``` layers:[ {type:'input',out_sx:64,out_sy:64,out_depth:3}, {type:'conv',sx:5,filters:8,stride:1,pad:2,activation:'relu'}, {type:'pool',sx:2,stride:2}, {type:'conv',sx:5,filters:16,stride:1,pad:2,activation:'relu'}, {type:'pool',sx:3,stride:3}, {type:'softmax',num_classes:4} ] trainer: {method: 'adadelta', 12_decay:0.002, batch_size:10} ```
Different worker cluster architectures were investigated, as shown in Figure 13. The entire image data set consists of about 1900 64 \(\times\) 64 \(\times\) 3 RGB image volumes (8 bits/element). The training and test data were split 1:1, randomly, i.e., about 900 training and 900 test images. The loading from local and remote SQL data bases, the pre-processing times, and the ML-CNN training times for one epoch (iteration) using shell workers are shown in Figure 14.  
The entire ML task is divided into four phases (the first three are considered in this experiment):  
1. Loading of the input data. 2.