# Segmentation of Intracranial Hemorrhage Using Semi-Supervised Multi-Task Attention-Based U-Net
## 5 Methods
### Training Procedure and Loss Functions  
While [PERSON] et al. [34] used both joint and alternating training strategies, after experimentation, we found that both strategies (especially when dealing with a very limited and volatile dataset) led to unstable behavior, causing the loss to explode early on in training. We instead proposed a curriculum learning strategy, which decayed with probability as training progressed. This mechanism regulated alternating learning, which is explained below. Originally, [PERSON]'s alternating learning strategy optimized alternately the supervised portion of the network and the unsupervised portion of the network. In each run, the unsupervised training randomly picked data points from the larger dataset. For example, training the network with a labeled dataset of 100 points and an unlabeled dataset of 1000 points would proceed with first training the model's supervised portion with the 100 labeled points, then training the unsupervised portion with 100 randomly selected samples from the 1000 unlabeled data points. Through the experimental study, we found this method could lead to gradient explosions. We hypothesized that if the supervised portion of the network were inaccurate, the attention mechanism would identify the wrong portions of the image for the autoencoder to reconstruct. This pushed the weights of the autoencoder to change in the wrong direction, resulting in divergence. Figure 2: Semi-supervised attention network with one shared encoder (green) and two decoders. The first one (blue) is the same decoder shown in Figure 1, and the second one (red) is the same decoder except without residual connections (effectively making it an autoencoder when paired with the encoder). The color of an arrow signifies the gradient flow between the supervised and unsupervised portions of the model. The loss functions \(L_{s}\) and \(L_{u}\) correspond to the supervised and unsupervised loss, respectively. The training procedure, gradient flow, and loss functions are detailed in Section 5.3. To solve the encountered problem, we proposed to use an inverse sigmoid curriculum learning strategy, which is outlined next. Rather than alternating equally between training the supervised and unsupervised portion of the network, we decided which portion to train based on a Bernoulli random variable with parameter \(p\), where \(p=f(x)\) and \(x\) is the current epoch. Let \(f(x)=\max\left(\frac{k}{k+\exp(\frac{1}{k})},0.5\right)\), where \(k\) is a preset hyperparameter. This allowed the algorithm at the start of the training to optimize mainly the supervised portion of the model, then eventually converge to [PERSON]'s alternating training algorithm. This version of the inverse-sigmoid learning curriculum was first introduced by [PERSON] et al. [46] when training forecasting systems. We found in our experimental study that this curriculum learning strategy produced better results, which will be shown later, in comparison to other training methods. The loss functions \(L_{s}\) and \(L_{u}\), shown in Figure 2, are the Jaccard loss and Weighted Mean Squared Error (WMSE), respectively. Formally, the \(L_{u}\) loss is defined in Equation (1), which weighs the MSE of the foreground and background reconstruction by the size of the segmentation as follows:  
\[L_{u}=\frac{\sum\hat{y}_{b}^{(i)}}{N}\text{MSE}[x\odot\hat{y}_{b},\hat{y}_{b}] +\frac{\sum\hat{y}_{f}^{(i)}}{N}\text{MSE}[x\odot\hat{y}_{f},\hat{y}_{f}] \tag{1}\]  
where \(\hat{y}\) and \(\hat{y}\) are the predictions of the reconstruction and segmentation paths, respectively, for the background (\(b\)) and foreground (\(f\)); \(N\) is the number of voxels in an input image \(x\); and \(\odot\) is the Hadamard product [34]. To show the effectiveness of our semi-supervised model, we trained and tested our model with N-fold cross-validation, where the training set contained 20%, 50%, and 80% of the labeled data, corresponding to the 5-fold, 2-fold, and 5-fold cross-validation schemes, respectively.