<figure><img src="image_1.png" /></figure>

HAL
open science

# Relay-UNet: Reduce Semantic Gap for Glomerular Image Segmentation

Zhen Cao, Chuanfeng Ma, Qian Wang, Hongwei Zhang

► To cite this version:

Zhen Cao, Chuanfeng Ma, Qian Wang, Hongwei Zhang. Relay-UNet: Reduce Semantic Gap for Glomerular Image Segmentation. 5th International Conference on Intelligence Science (ICIS), Oct 2022, Xi'an, China. pp.378-385, 10.1007/978-3-031-14903-0_40. http://dx.doi.org/10.1007/978-3-031-14903-0_40

HAL Id: hal-04666413

https://hal.science/hal-04666413v1

Submitted on 1 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Relay-UNet: Reduce Semantic Gap for Glomerular Image Segmentation

Zhen Cao, Chuanfeng Ma, Qian Wang, and Hongwei Zhang

School of Artificial Intelligence
Xidian University, Xi'an, Shaanxi, P.R. China
icaozhen@163.com

**Abstract.** With the increasing number of patients with chronic kidney disease (CDK), the workload of pathologists is heavy and the diagnostic efficiency is low. Therefore, the use of computer technology to assist the diagnosis of nephropathy becomes the trend of future development with a great application space. In recent years, UNet network has been widely used in medical image segmentation, and many improved algorithms have appeared. Most of existing methods focus on adding new modules or incorporating other design concepts on the basis of UNet, however, the structure of UNet network has not been fully analyzed. This paper points out two problems existing in UNet network: insufficient feature extraction in encoder interferes with the accuracy of image segmentation, and the fixed mode of skip connection in each layer leads to information redundancy in the network. To solve these problems, we improve the encoder and feature fusion method of UNet and named the new network as Relay-UNet. Experiment results show that under the condition of almost no change in consumption time, the Dice coefficient of Relay UNet on the glomerular dataset reaches 97.5%, and the training process is more stable than that of UNet.

**Keywords:** Glomerular image segmentation, Feature fusion, Skip connection

## 1 Introduction

CDK is a general term for a group of kidney diseases, also known as chronic nephrotic syndrome, which refers to the structural changes and functional disorders of chronic kidney caused by various reasons. According to the survey report of the world's authoritative medical journal the LANCET, by 2017, the number of patients with chronic kidney disease in the world had reached 697.5 million[1]. In the process of diagnosis and treatment of CDK patients, renal biopsy plays a very important role. Pathologists can judge the type and development stage of the disease by observing the morphological changes of glomeruli in tissue sections. However, in the traditional diagnostic methods, pathologists need to look for glomeruli in tissues and compare their morphological changes through long-term naked eye observation under a microscope. High-intensity and repetitive work will greatly reduce the diagnostic efficiency of pathologists. Therefore, if computer technology can be used to
automatically and accurately mark the position of glomerulus, accurately segment its
outline and visually present it, the workload of pathologists will be greatly reduced.

At present, the common glomerular segmentation methods can be divided into
edge-based segmentation algorithm, region-based segmentation algorithm and deep
learning-based segmentation algorithm[2]. Edge-based segmentation algorithm uses
the edge information of glomerulus in pathological slices to segment. If the gray value
of a certain part of the slice image has obvious step change, it can be determined as
the edge of glomerulus. The biggest disadvantage of this method is its weak anti-noise
ability, especially in the case of high noise frequency, which may lead to false
detection results. The region based segmentation algorithm divides the image into
several different regions according to the similarity. However, after the kidney section
is stained, the attribute values such as color and gray scale are similar among different
tissue areas, so it is necessary to use the prior information of glomerular shape and
position to achieve accurate segmentation.

In recent years, the deep learning model represented by UNet network has become a research hotspot, and achieved good results in various medical image segmentation tasks. These deep learning models make use of the powerful computing power of modern computers, and continuously extract and learn the feature information of images through convolution operations, so as to complete the target segmentation. After UNet was put forward, many researchers improved its structure and put forward new models, but most of the improvement methods were to add new modules or incorporate other design concepts on the basis of UNet. For example, LinkNet added residual operation in the encoder of the original UNet[3], and Attention-UNet added attention mechanism in the jumping connection process of the original UNet[4]. Although these methods can improve the segmentation performance of UNet, they do not fully consider some problems existing in the network itself. Considering that the setting of UNet network depth and feature fusion mode is not optimal, UNet++ embeds U-shaped structures with various depths into the network and designs a more flexible hopping connection mode[5]. However, this design also makes the network structure too complicated, and a large number of parameters increase the training time of the network.

To sum up, the main contributions of this paper are as follows:

**The problems of UNet are pointed out and verified:** Firstly, because of the small number of convolution layers in the encoder, the semantic difference between the two parts of feature fusion between codecs is large, which interferes with the accuracy of image segmentation; Secondly, the fixed mode that each layer of the network structure is connected by hopping leads to information redundancy in the network.

**Relay-UNet is proposed to improve UNet:** First, add a feature extraction module to the encoder to narrow the semantic gap with the decoder; Secondly, by adding a weighting factor in the process of feature image splicing, the constraint of the network on the jump connection can be slowed down, so that the network can decide which layer to fuse features on.

**The performance of Relay-UNet is better than UNet:** on the glomerular data set, the Dice coefficient of Relay-UNet reaches 97.5%, which is 1.5% higher than UNet.
## 2 Related work

Because this paper is to improve UNet and increase its accuracy in glomerular segmentation. Therefore, in the following paragraphs, we briefly review some existing work in the field of glomerular segmentation and the basic structure of UNet.

### 2.1 Glomerular image segmentation

**Glomerular image segmentation based on traditional methods.** Jiaxin Ma et al. proposed using Canny edge detection algorithm to extract the discontinuous edge of glomerulus[6]. After the noise is removed by labeling, the genetic algorithm is used to search the best repair segment, and these edges are connected together to form a complete glomerular contour. Jun Zhang et al. proposed a glomerular extraction method based on wavelet transform and watershed algorithm[7]. Firstly, the LOG filter is applied to the low-resolution image, which corresponds to the low-frequency sub-band after wavelet transform, so that rough edge information can be obtained. After eliminating noise, genetic algorithm is used to search the best fitting curve.

**Glomerular image segmentation based on deep learning method.** In 2019, Robin Liu et al. made use of the accurate positioning ability of Faster R-CNN and the powerful segmentation ability of U-Net[8], and realized the localization of glomerulus from the whole slice image, so that the segmentation only focused on the most relevant areas in the image. In 2020, Gloria Bueno et al. compared the pixel-level segmentation of UNet and SegNet[9]. In 2021, Yuan Liu et al. proposed a semantic segmentation method FEU-Net for pathological sections, which improved the accuracy of glomerular segmentation and achieved end-to-end segmentation[10]. FEU-Net uses EfficientNet after transfer learning as the encoder and UNet as the decoder.

### 2.2 UNet structure

UNet is a full convolution segmentation network, which is mainly used for semantic segmentation of biomedical images. It inherits the idea of FCN, puts forward a completely symmetrical U-shaped encoding-decoding structure, and realizes the semantic fusion of different levels by using jump connection.

In UNet structure, if the two convolution operations of the same layer are regarded as a whole, which is called block and sorted in turn, then block1 and block9 are spliced, and so on. It can be found that the original image is only convolved twice in block1, and the semantic features of the image are not fully extracted. As the last part of the network, block9 contains rich semantic features after a large number of convolution operations. The original UNet structure directly splices the output feature mapping of block1 to block9, which may affect the accuracy of image segmentation due to the differences between the two parts and too much semantic information. The splicing of other layers also has the same problem.

In addition, the codec in each layer of UNet structure is connected by default, and four feature fusion is performed accordingly. However, the semantic information of
the feature map output by the adjacent layers of the encoder is similar. Frequent feature fusion leads to information redundancy in the network and increases the risk of over fitting.

To sum up, there may be two problems in the feature fusion part of UNet structure: insufficient feature extraction in the coding part affects the segmentation result; The fixed mode of splicing each layer will make the information redundant in the network.

In order to verify the two questions, a control experiment was designed in this paper. Randomly delete some hopping connections in UNet network, and form five groups of different skip connection structures. **Table 1** shows the grouping.

<table><caption>**Table 1** Experimental grouping table</caption><thead><tr><td>Group</td><td>Skip connection structure</td></tr></thead><tbody><tr><td>L1</td><td>Delete all skip connections.</td></tr><tr><td>L2</td><td>Keep the skip connection of the first layer</td></tr><tr><td>L3</td><td>Keep the skip connection between the second layer and the third layer.</td></tr><tr><td>L4</td><td>Keep the skip connections of the second, third and fourth layers.</td></tr><tr><td>L5</td><td>Keep all layers of skip connections.</td></tr></tbody></table>

In addition to the number of layers including skip connections, all other parameters are fixed, and the network performance of these five groups of different network structures on the same data set is observed. **Fig. 1** shows the training results of each group of networks, from which it can be seen that: (1) the performance of network structure segmentation with all hopping connections is not optimal, and the performance of the network may be better after some hopping connections are removed, such as L4 and L5; (2) The addition of the first layer hop connection may cause the decline of network performance, such as L1 and L2. Therefore, the rationality of the two questions mentioned above can be verified.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Segmentation results of different groups</figcaption></figure>
# 3 Methodology

## 3.1 Relay-UNet structure

To solve the above two problems, this paper improves the encoder and feature fusion method of UNet, and calls the new network Relay-UNet. Fig. 2 shows the concrete structure of the improved network. The pink background part is the original UNet network structure, and the blue background part is the newly added or improved structure.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Relay-UNet structure</figcaption></figure>

## 3.2 Improved method

Relay-UNet firstly improves UNet coding structure by adding four relay blocks, namely relay-block1 to relay-block4. As can be seen from Fig. 2 when UNet performs downsampling and feature graph dimension splicing, Relay-UNet divides into two branches. The first is down sampling, and continue the encoding operation; The other one is not directly spliced into the decoder of the same layer, but enters the relay-block to continue convolution operations for many times, and its feature map is spliced into the decoding structure only after deeper semantic information is extracted. But at the same time, the addition of relay-block will also bring new problems. On the one hand, it is not certain that setting several convolutions in relay-block will make the network performance the best. If the number of convolutions is insufficient, the semantic level difference between the two sides of the jump connection is still too big to achieve the expected effect; If the convolution times are too many, the final output will lose low-level features, which will also decrease. On the other hand, the addition of relay-block makes the network structure more complicated, which may lead to over-fitting in network training and reduce its performance. Therefore, learn from Resnet's idea[11], add the residual structure, and add the output of relay-block to the result of the last convolution layer in front of it, as
shown by the black arrow in the Fig. 2. In this way, the fitting of relay-block becomes the learning residual, which reduces the risk of over-fitting, and also enables the network itself to decide how to use the convolution operation in relay-block.

Another improvement of Relay-UNet is to improve the splicing method of feature graphs in jump connection. The original UNet directly splices the two feature maps. The Relay-UNet multiplies the two spliced parts by a weight factor $\alpha_i$ and $\beta_i$, respectively, so that $\alpha_i + \beta_i = 1$. The weight factor acts on each pixel of each channel of the feature map to reduce the overall value of the feature map, and then the dimension is spliced. Let $F \in R^{H \times W \times C}$ be the feature tensor to be spliced, where $H$, $W$, and $C$ are the height, width, and channel dimensions of the feature tensor, then the formula for multiplying the ith channel of the tensor by the weight $\alpha_i$ is:

$$ 
\begin{aligned}
\mathbf{F}_i \otimes \alpha_i &= \begin{pmatrix} F_i^{1,1} & F_i^{1,2} & \cdots & F_i^{1,W} \\ F_i^{2,1} & F_i^{2,2} & \cdots & F_i^{2,W} \\ \vdots & \vdots & \ddots & \vdots \\ F_i^{H,1} & F_i^{H,2} & \cdots & F_i^{H,W} \end{pmatrix} \otimes \begin{pmatrix} \alpha_i & \alpha_i & \cdots & \alpha_i \\ \alpha_i & \alpha_i & \cdots & \alpha_i \\ \vdots & \vdots & \ddots & \vdots \\ \alpha_i & \alpha_i & \cdots & \alpha_i \end{pmatrix} \\
&= \begin{pmatrix} F_i^{1,1} \cdot \alpha_i & F_i^{1,2} \cdot \alpha_i & \cdots & F_i^{1,W} \cdot \alpha_i \\ F_i^{2,1} \cdot \alpha_i & F_i^{2,2} \cdot \alpha_i & \cdots & F_i^{2,W} \cdot \alpha_i \\ \vdots & \vdots & \ddots & \vdots \\ F_i^{H,1} \cdot \alpha_i & F_i^{H,2} \cdot \alpha_i & \cdots & F_i^{H,W} \cdot \alpha_i \end{pmatrix}
\end{aligned} 
\quad (1) $$

Fig. 3 shows this process. When the features from the encoder are not needed in the decoder, the smaller $\alpha_i$ is, the larger $\beta_i$ is; When the features from the encoder are needed in the decoder, the larger $\alpha_i$ is, the smaller $\beta_i$ is, which realizes the effect of feature dilution by weighting.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Improved feature fusion method</figcaption></figure>

# 4 Experiment and result analysis

## 4.1 Dataset

The data used in this experiment are 1250 kidney biopsy specimens with different staining from Shanxi Provincial People's hospital. After obtaining the slice samples,
the biopsy microscope slides were digitized into WSIS using a digital slide scanner. 1000 glomerular images with good staining and structure were cut out from renal tissue sections as the data set of this experiment. The data is enhanced by cutting and rotating, expanded to 2500 pieces, and divided into training set and test set according to the ratio of 8:2.

## 4.2 Evaluating Metrics

In order to evaluate the network performance, this paper uses two evaluation indexes, namely IOU(intersection over union) and Dice (similarity coefficient).

**IOU** is the overlapping area between predicted segmentation and labels divided by the joint area between predicted segmentation and labels. The range of this indicator is 0-1, where 0 means no overlapping and 1 means complete overlapping segmentation; **Dice** coefficient is defined as twice the intersection divided by the sum of pixels, also known as F1 score. Similar to IOU, they all range from 0 to 1, where 1 indicates the maximum similarity between prediction and reality;

## 4.3 Training process

**Fig. 4** shows the changes of Dice evaluation index in the process of UNet and Relay-UNet network training, respectively.

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Change of Dice coefficient during network training.</figcaption></figure>

It can be seen from the **Fig. 4** that after the Relay-UNet network converges, Dice reaches 0.975, which is greatly improved compared with UNet's 0.961, and it can be found that the training process is more stable. Therefore, the improved method in this paper has a good effect on improving the UNet segmentation effect.
## 5 Conclusion

In this paper, a glomerular image segmentation algorithm based on improved U-Net is proposed, named Relay-UNet. A feature extraction module relay-block is added to the encoder to reduce the semantic information, and the constraint of the network on the hop connection is slowed down by adding a weight factor in the process of splicing feature graphs. Experiments show that the proposed method has better segmentation performance than the traditional UNet, and its accuracy on glomerular data sets is improved by about 1.5%.

## 6 Acknowledgements

This work was supported in part by the China Postdoctoral Science Foundation under Grant 2019M663637, in part by the Natural Science Basic Research Program of Shaanxi under Program 2021JQ-201, and in part by the National Natural Science Foundation of China under Grant 62104176.

## References

1. Rule A D, Bergstrahl E J, Melton L J, et al. Kidney stones and the risk for chronic kidney disease[J]. Clinical Journal of the American Society of Nephrology, 2009, 4(4): 804-811.

2. Yuheng S, Hao Y. Image segmentation algorithms overview[J]. arXiv preprint arXiv:1707.02051, 2017.

3. Zhou L, Zhang C, Wu M. D-LinkNet: LinkNet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018: 182-186.

4. Oktay O, Schlemper J, Folgoc L L, et al. Attention u-net: Learning where to look for the pancreas[J]. arXiv preprint arXiv:1804.03999, 2018.

5. Zhou Z, Siddiquee M, Tajbakhsh N, et al. UNet++: A Nested U-Net Architecture for Medical Image Segmentation[J]. 2018.

6. Ma J, Zhang J, Hu J. Glomerulus extraction by using genetic algorithm for edge patching[C]//2009 IEEE Congress on Evolutionary Computation. IEEE, 2009: 2474-2479

7. Zhang J, Fan J. Medical image segmentation based on wavelet transformation and watershed algorithm[C]//2006 IEEE International Conference on Information Acquisition. IEEE, 2006: 484-488.

8. Liu R, Wang L, He J, et al. Towards Staining Independent Segmentation of Glomerulus from Histopathological Images of Kidney[J]. bioRxiv, 2019: 821181.

9. Bueno G, Fernandez-Carrobles M M, Gonzalez-Lopez L, et al. Glomerulosclerosis identification in whole slide images using semantic segmentation[J]. Computer methods and programs in biomedicine, 2020, 184: 105273.

10. Liu Y, Wang J. FEU-Net: Glomeruli region segmentation network based on pseudo-labelling and channel attention mechanisms[C]//International Conference on Image Processing and Intelligent Control (IPIC 2021). SPIE, 2021, 11928: 41-51.

11. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.