## 3 Techniques for CNN-Based BUS Classification Framework
### Training and Evaluation  
For the experiment, we used NVIDIA(NVIDIA Corporation, Santa Clara, CA, USA) Tesla P100 and Tesla T4 over the cloud to train and test the convolution networks. Although the GPUs were different, they did not affect the results of CNN classification. However, they affected the training and inference speed of the network. TensorFlow2 (v2.8.0, Google Inc., Mountain View, CA, USA) and Python 3.9 (v3.9.0, Python Software Foundation, Beaverton, OR, USA) were used for implementing the backbone and the proposed networks. Individual CNN backbones were trained for 30 epochs with a batch size of 8. The Adam optimizer and sigmoid function were used to optimize and predict binary results,  
Figure 5: Architectures of CNN-based deep learning networks. (**a**) Traditional classification network with the B-mode image; (**b**) Proposed ensemble network with four parametric images. respectively. Because we had only two classes, benign or malignant, the binary cross-entropy (BCE) was used as the loss function. \[BCE(y,\,\hat{y})=-ylog(\hat{y})-(1-y)\log(1-\hat{y}), \tag{2}\]  
where \(y,\,\hat{y}\) represents the correct category of input image and estimated result from the binary classifier, which is the output of the sigmoid function, respectively. The learning rates of the backbones varied according to the network model owing to their different structures. Training started with initialization of networks randomly. In each fold, we train 30 epochs then validate once. During the training process, the best validation accuracy weights of each fold were saved for later use in inference and voting procedures. Algorithm 2 presents the training procedure of our networks. ```
1:Input 5 sets of multiple parametric images \(\mathbb{S}_{0}\), \(\mathbb{S}_{1},\ldots,\mathbb{S}_{5}\)
2:for i = 1 to 4
3: Validation set = Si
4: Training set = \(\mathbb{S}_{0}\), \(\mathbb{S}_{1},\ldots,\mathbb{S}_{1\text{-}1}\), \(\mathbb{S}_{1\text{+}1}\), \(\ldots,\mathbb{S}_{4}\)
5: Training 30 epochs using training set
6: Validate using validation set
7: If validation accuracy is better than before, save model
```  
**Algorithm 2 Training Procedure**  
After training each backbone network using parametric images, all weights were determined using the Bayesian optimization method [43] shown in Algorithm 3. ```
1:Input correct answer \(\mathbf{y}\), estimated result of training set \(\hat{y}\), number of iterations N
2:Output weight vector of classifier \(\boldsymbol{w}_{best}\)
3: Randomly choose initial weights \(\boldsymbol{w}_{0}\)
4: for i = 1 to N
5: Get \(\boldsymbol{w}_{i}\) by optimizing acquisition function over gaussian process: \(w_{i}=argmax_{w}u(x|\mathcal{D}_{1:i-1})\)
6: Calculate accuracy of weighted voting result \(\mathbf{a}\) using \(\boldsymbol{y},\,\,\,\hat{y},\,\,\,\boldsymbol{w}_{i}\). 7: if accuracy of \(\boldsymbol{w}_{i}>\) accuracy of \(\boldsymbol{w}_{best}\) then \(\boldsymbol{w}_{best}=w_{i}\)
8: Augment the observation \(\mathcal{D}_{1:i}=\{\mathcal{D}_{1:i-1},(\mathbf{x}_{i},\,\mathbf{a})\}\) and update gaussian process
```  
**Algorithm 3 Bayesian Optimization for Weights of Each Domain**  
We iterated this process 160 times to find the optimal weights for each backbone network, and the final prediction result was calculated using uniform and weighted soft voting in the decision layer. The formulas of two voting methods are as follows:  
\[\hat{y}_{uniform}=\frac{\sum_{i=0}^{N}\hat{y}_{i}}{N}, \tag{3}\]  
\[\hat{y}_{weighted}=\frac{\sum_{i=0}^{N}w_{i}\cdot\hat{y}_{i}}{N}, \tag{4}\]  
where \(\hat{y}_{i}\) denotes estimated result of each parametric image, \(w_{i}\) denotes determined weights for estimated result of each parametric images, and N represents number of classifiers. The performance metrics chosen for this study were accuracy, recall, precision, F1-score, and area under the curve (AUC). These metrics were calculated using true positive (TP), true negative (TN), false positive (FP), and false negative (FN) as variables.