# Shadow Estimation for Ultrasound Images Using Auto-Encoding Structures and Synthetic Shadows
## 3 Materials and Methods
### Use of Pixel-Level Labels and Extension to Semi-Supervised Learning  
Pixel-level labels for US shadows are expensive as mentioned above, but if available, they can contribute to the improvement of the estimation performance. We assume that we have some pixel-level labeled data which labels are binary; whether each pixel is in shadows or not. Ideally, we expect labels that express semi-transparency of shadows but the correct intensities of shadows are unknown even for experts. Hence, we introduce a method to effectively utilize the binary labels for the proposed shadow estimation framework. Let \(l\in\{0,1\}^{H\times W}\) be a pixel-level binary label that represents where shadows exist. For each element in \(l\), 0 and 1 correspond to a shadowed pixel and a shadow-free pixel, respectively. Recall that the proposed auto-encoding model is based on the idea: each input US image is considered to be generated by an element-wise product of the ideal shadow-free image and the shadow image. If the ideal shadow-free image \(x^{*}\) for an input \(x\) is available, we can calculate the intensity of the labeled shadow by  
\[l_{hw}^{*}:= \begin{cases}\frac{\chi_{hw}}{x_{hw}^{*}}&(l_{hw}=0)\\ 1&(l_{hw}=1)\end{cases} . \tag{8}\]  
However, \(x^{*}\) is actually unknown. Here, we estimate \(x^{*}\) as a mean brightness over the shadow-free area that is written as  
\[\hat{x}^{*}:=\frac{1}{\sum_{i,j}M_{ij}}\sum_{i=1}^{H}\sum_{j=1}^{W}M_{ij}x_{ij}, \tag{9}\]  
where \(M\in\{0,1\}^{H\times W}\) is a mask that represents the region without shadows. The mask \(M\) is given as  
\[M_{hw}:= \begin{cases}1&(l_{hw}=1\text{ and }x_{hw}>T)\\ 0&(\text{otherwise})\end{cases} , \tag{10}\]  
where \(T\in[0,1]\) is a given threshold to ignore almost completely black areas that have no shadows. In US images, liquids are shown in black. By thresholding, we can reject such areas and estimate \(x^{*}\) more precisely. Besides, for simplicity and stability of the training,we assume that the intensity of each labeled shadow is constant. Hence, Equation (8) is rewritten and the resulting label with the estimated intensity is  
\[l^{\prime}_{hw}:= \begin{cases}\frac{1}{\hat{x}^{*}}\cdot\frac{\sum_{z\in \mathcal{S}(x_{hw})}z}{|\mathcal{S}(x_{hw})|}&(l_{hw}=0)\\ 1&(l_{hw}=1)\end{cases} , \tag{11}\]  
where \(\mathcal{S}(x_{hw})\) is a subset of \(\{x_{ij}\}(i=1,\dots,H,\ j=1,\dots,W)\) that consists of \(x_{ij}\) inside the shadow that contains \(x_{ij}\). Calculation of \(l^{\prime}\) is summarized in Algorithm 1. ```
0: A US image \(x\in[0,1]^{H\times W}\), a pixel-level label of shadows \(l\in\{0,1\}^{H\times W}\), and a threshold \(T\). 0: Semi-transparent label \(l^{\prime}\in[0,1]^{H\times W}\)
1:\(M\gets l\odot\mathbf{1}[x>T]\)
2:\(x^{*}\leftarrow\frac{1}{\sum_{ij}M_{ij}}\sum_{i=1}^{H}\sum_{j=1}^{W}M_{ij}x_ {ij}\)
3:\(l^{\prime}\gets l\)
4:for each labeled shadow \(l_{c}\) in \(l\) (i.e., each connected component \(l_{c}\) in \(l\) with a value \(0\)) do
5:for each coordinate \((i,j)\) that corresponds to \(l_{c}\)do
6:\(l^{\prime}_{ij}\gets l^{\prime}_{ij}+\frac{x_{ij}}{\hat{x}^{\prime}|l_{c}|}\)
7:endfor
8:endfor
```  
**Algorithm 1** Estimation of shadow intensities using a pixel-level binary label. Using the estimated semi-transparent label \(l^{\prime}\), we can construct a loss function based on Equation (5) as follows;  
\[L_{\text{label}}(l^{\prime},s):=\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}(s_{hw }-l^{\prime}_{hw})^{2}. \tag{12}\]  
In contrast to Equation (5), we do not need the masking term anymore because the label \(l^{\prime}\) tells not only shadowed areas but also shadow-free areas. Besides, the regularization term given by Equation (6) is neither needed. The resulting loss function for labeled data is given as  
\[L_{\text{labeled}}(x,l^{\prime},\hat{x},s):=\lambda_{\text{recon}}L_{\text{ recon}}(x,\hat{x})+\lambda_{\text{label}}L_{\text{label}}(l^{\prime},\hat{s}), \tag{13}\]  
where \(\lambda_{\text{recon}},\lambda_{\text{label}}>0\) are hyperparameters. Figure 1c illustrates the training procedure with labeled data.