Title: STACKED U-NETS: A NO-FRILLS APPROACH TO NATURAL IMAGE SEGMENTATION

Abstract: Many imaging tasks require global information about all pixels in an image. Conventional bottom-up classification networks globalize information by decreasing resolution; features are pooled and down-sampled into a single output. But for semantic segmentation and object detection tasks, a network must provide higherresolution pixel-level outputs. To globalize information while preserving resolution, many researchers propose the inclusion of sophisticated auxiliary blocks, but these come at the cost of a considerable increase in network size and computational cost. This paper proposes stacked u-nets (SUNets), which iteratively combine features from different resolution scales while maintaining resolution. SUNets leverage the information globalization power of u-nets in a deeper network architectures that is capable of handling the complexity of natural images. SUNets perform extremely well on semantic segmentation tasks using a small number of parameters.
Under review as a conference paper at ICLR 2019 inputs to deconvolutional layers come from relatively shallow layers that do not effectively encode semantic information. 2. Image classification networks are parameter heavy (44.5M parameters for ResNet-101), and segmentation methods built on top of these classification networks are often even more burdensome. For example, on top of the resnet-101 architecture, PSPNet (Zhao et al., 2017) uses 22M additional parameters for context aggregation, while the ASPP and Cascade versions of the Deeplab network utilize 14.5M (Chen et al., 2016a) and 40M (Chen et al., 2017) additional parameters, respectively.
A popular and simple approach to segmentation is u-nets, which perform a chain of convolutional/downsampling operations, followed by a chain of deconvolutional/upsampling layers that see information from both low-and high-resolution scales. These u-net architectures are state-of-the art for medical image segmentation (Ronneberger et al., 2015) , but they do not perform well when confronted with the complex color profiles, lighting effects, and occlusions present in natural images.
We expand the power of u-nets by stacking u-net blocks into deep architectures. This addresses the two challenges discussed above: As data passes through multiple u-net blocks, high-resolution features are mixed with low-resolution context information and processed through many layers to produce informative high-resolution features. Furthermore, stacked U-net models require fewer feature maps per layer than conventional architectures, and thus achieve higher performance with far fewer parameters. Our smallest model exceeds the performance of ResNet-101 on the PASCAL VOC 2012 semantic segmentation task by 4.5% mIoU, while having ∼ 7× fewer parameters.

1. INTRODUCTION:

Semantic segmentation methods decompose an image into groups of pixels, each representing a common object class. While the output of a segmentation contains object labels assigned at the local (pixel) level, each label depends on global information about the image, such as textures, colors, and object boundaries that may span large regions. Simple image classification algorithms consolidate global information by successively pooling features until the final output is a single label containing information from the entire image. In contrast, segmentation methods must output a full-resolution map of labels. Thus, a successful segmentation method must address this key question: how can we learn long-distance contextual information while at the same time retaining high spatial resolution at the output for identifying small objects and sharp boundaries?

For natural image processing, most research has answered this question using one of two approaches. One approach is to use very few pooling layers, thus maintaining resolution (although methods may still require a small number of deconvolution layers (Lin et al., 2017a; Fu et al., 2017; Jégou et al., 2017; Ghiasi & Fowlkes, 2016) ). Large fields of view are achieved using dilated convolutions, which span large regions. By maintaining resolution at each layer, this approach preserves substantial amounts of signal about smaller and less salient objects. However, this is achieved at the cost of computationally expensive and memory exhaustive training/inference. The second related approach is to produce auxiliary context aggregation blocks (Krähenbühl & Koltun, 2011; Chen et al., 2016a; Chandra & Kokkinos, 2016; Chandra et al., 2017; Zheng et al., 2015; Yu & Koltun, 2015; Zhao et al., 2017) that contain features at different distance scales, and then merge these blocks to produce a final segmentation. This category includes many well-known techniques such as dense CRF (Krähenbühl & Koltun, 2011) (conditional random fields) and spatial pyramid pooling (Chen et al., 2016a) .

These approaches suffer from the following challenges:

1. Deconvolutional (i.e., encoder-decoder) architectures perform significant nonlinear computation at low resolutions, but do very little processing of high-resolution features. During the convolution/encoding stage, pooling layers can move information over large distances, but information about small objects is often lost. During the deconvolution/decoding stage, high-and lowresolution features are merged to produce upsampled feature maps. However, the high-resolution 2.1 USE OF PRE-TRAINED NETS Many of the networks described above make extensive use of image classification networks that were pre-trained for other purposes. Although parameter heavy, much fundamental work on segmentation (FCN (Long et al., 2015) , dilated nets (Yu & Koltun, 2015) , u-nets (Ronneberger et al., 2015) and CRF (Zheng et al., 2015) ) was built on VGG. These architectures share common origins in that they were designed for the ImageNet competition and features are processed bottom-up. This prototype works well when the network has to identify only a single object without exact pixel localization. However, when extended to localization tasks such as segmentation and object detection, it is not clear whether the complete potential of these networks has been properly tapped. Recent work on object detection (Singh & Davis, 2017 ) also echoes a similar concern.

3. U-NETS REVISITED:

The original u-net architecture was introduced by Ronneberger et al. (2015) , and produced almost perfect segmentation of cells in biomedical images using very little training data. The structure of u-nets makes it possible to capture context information at multiple scales and propagate them to the higher resolution layers. These higher order features have enabled u-nets to outperform previous deep models on various tasks including semantic segmentation (Milletari et al., 2016) , depth-fusion (Riegler et al., 2017) , image translation (Isola et al., 2017) and human-pose estimation (Newell et al., 2016) . Moreover, driven by the initial success of u-nets, many recent works on semantic segmentation (Lin et al., 2017a; Ghiasi & Fowlkes, 2016; Fu et al., 2017) and object detection (Shrivastava et al., 2016; Lin et al., 2017b) also propose an encoder-decoder deep architecture.

The u-net architecture evenly distributes its capacity among the encoder and decoder modules. Moreover, the complete network can be trained in an end-to-end setting. In contrast, the more recent architectures reviewed in Section 2 do not equally distribute the processing of top-down and bottomup features. Since these architectures are built on top of pre-trained feature extractors (Simonyan & Zisserman, 2014; He et al., 2016a) , the decoder modules are trained separately and sometimes in multiple stages. To overcome these drawbacks, Jégou et al. (2017) proposed an equivalent u-net based on the Densenet (Huang et al., 2017) architecture. However, DenseNet is memory intensive, and adding additional decoder layers leads to a further increase in memory usage. Given these drawbacks, the effectiveness of these architectures on different datasets and applications is unclear.

The goal of this paper is to realize the benefits of u-nets (small size, easy trainability, high performance) for complex natural image segmentation problems. Specifically, we propose a new architecture composed of multiple stacks of u-nets. The network executes repeated processing of both top-down as well as bottom-up features and captures long-distance spatial information at multiple resolutions. The network is trained end-to-end on image classification tasks and can be seamlessly applied to semantic segmentation without any auxiliary modules on top.

Our stacked u-net (SUNet) architecture shares some similarity with other related stacked encoderdecoder structures. Fu et al. (2017) use multiple stacks (upto 3) of de-convolutional networks on top of a powerful encoder (DenseNet) while Newell et al. (2016) apply multiple stacks of u-net modules for human-pose estimation. However, the processing of features inside each u-net module in Newell et al. (2016) differs from ours. Newell et al. (2016) replace each convolutional block with a residual module, utilizes nearest-neighbor upsampling for deconvolution and operate at fix resolution. In contrast, SUNets retain the basic u-net structure from Ronneberger et al. (2015) , operates without any intermediate supervision and processes features by progressively downsampling.

3.1. U-NET MODULE IMPLEMENTATION:

Figure 1 illustrates the design of the u-net module employed in our stacked architecture. Each module is composed of 10 pre-activated convolutional blocks each preceded by a batch-normalization and a ReLU non-linearity. The pooling/unpooling operation, handled by the strided convolutional/deconvolutional layers, facilitates information exchange between the lower and the higher resolution features. A skip connection branches off at the output of the first encoder block, E1. Following this, the E2 and D2 blocks capture long distance context information using lower resolution feature maps and merge the information back with the high resolution features from E1 at the output of D2. Every layer (except for the bottleneck layers) uses 3 × 3 kernels, and outputs a 

1 x 1 3 x 3 3 x 3 3 x 3 3 x 3 3 x 3 3 x 3 3 x 3 3 x 3 1 x 1 M ⇥ 64 2 FoV 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " x t Y t q k / N A Q j j / d f 8 8 n u 0 5 b 7 + f g 0 = " > A A A B 5 3 i c b V B N T 8 J A E J 3 6 i f i F e v S y k Z h 4 I i 0 X 9 U b 0 4 h E S K y T Q k O 0 y h Z X t t t n d m p C G X + D F g x q v / i V v / h s X 6 E H B l 0 z y 8 t 5 M Z u a F q e D a u O 6 3 s 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 4 f d J I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v Z 3 7 7 C Z X m i b w 3 k x S D m A 4 l j z i j x k o t r 1 + p u j V 3 D r J K v I J U o U C z X / n q D R K W x S g N E 1 T r r u e m J s i p M p w J n J Z 7 m c a U s j E d Y t d S S W P U Q T 4 / d E r O r T I g U a J s S U P m 6 u + J n M Z a T + L Q d s b U j P S y N x P / 8 7 q Z i a 6 C n M s 0 M y j Z Y l G U C W I S M v u a D L h C Z s T E E s o U t 7 c S N q K K M m O z K d s Q v O W X V 4 l f r 1 3 X 3 F a 9 2 r g p 0 i j B K Z z B B X h w C Q 2 4 g y b 4 w A D h G V 7 h z X l 0 X p x 3 5 2 P R u u Y U M y f w B 8 7 n D + h J j I Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x t Y t q k / N A Q j j / d f 8 8 n u 0 5 b 7 + f g 0 = " > A A A B 5 3 i c b V B N T 8 J A E J 3 6 i f i F e v S y k Z h 4 I i 0 X 9 U b 0 4 h E S K y T Q k O 0 y h Z X t t t n d m p C G X + D F g x q v / i V v / h s X 6 E H B l 0 z y 8 t 5 M Z u a F q e D a u O 6 3 s 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 4 f d J I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v Z 3 7 7 C Z X m i b w 3 k x S D m A 4 l j z i j x k o t r 1 + p u j V 3 D r J K v I J U o U C z X / n q D R K W x S g N E 1 T r r u e m J s i p M p w J n J Z 7 m c a U s j E d Y t d S S W P U Q T 4 / d E r O r T I g U a J s S U P m 6 u + J n M Z a T + L Q d s b U j P S y N x P / 8 7 q Z i a 6 C n M s 0 M y j Z Y l G U C W I S M v u a D L h C Z s T E E s o U t 7 c S N q K K M m O z K d s Q v O W X V 4 l f r 1 3 X 3 F a 9 2 r g p 0 i j B K Z z B B X h w C Q 2 4 g y b 4 w A D h G V 7 h z X

l 0 X p x 3 5 2 P R u u Y U M y f w B 8 7 n D + h J j I Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x t Y t q k / N A Q j j / d f 8 8 n u 0 5 b 7 + f g 0

= " > A A A B 5 3 i c b V B N T 8 J A E J 3 6 i f i F e v S y k Z h 4 I i 0 X 9 U b 0 4 h E S K y T Q k O 0 y h Z X t t t n d m p C G X + D F g x q v / i V v / h s X 6 E H B l 0 z y 8 t 5 M Z u a F q e D a u O 6 3 s 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 4 f d J I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v Z 3 7 7 C Z X m i b w 3 k x S D m A 4 l j z i j x k o t r 1 + p u j V 3 D r J K v I J U o U C z X / n q D R K W x S g N E 1 T r r u e m J s i p M p w J n J Z 7 m c a U s j E d Y t d S S W P U Q T 4 / d E r O r T I g U a J s S U P m 6 u + J n M Z a T + L Q d s b U j P S y N x P / 8 7 q Z i a 6 C n M s 0 M y j Z Y l G U C W I S M v u a D L h C Z s T E E s o U t 7 c S N q K K M m O z K d s Q v O W X V 4 l f r 1 3 X 3 F a 9 2 r g p 0 i j B K Z z B B X h w C Q 2 4 g y b 4 w A D h G V 7 h z X l 0 X p x 3 5 2 P R u u Y U M y f w B 8 7 n D + h J j I Y = < / l a t e x i t > 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " t 8 a F 3 V U l 9 Y v E k j 0 l n f v 7 1 J J e p + U = " > A A A B 5 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I z G x I n d Y q B 3 R x h I S T 0 j g Q v a W O V j Z 2 7 v s 7 p k Q w i + w s V B j 6 1 + y 8 9 + 4 w B U K v m S S l / d m M j M v T A X X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x 6 0 E m m G P o s E Y l q h 1 S j 4 B J 9 w 4 3 A d q q Q x q H A V j i 6 n f m t J 1 S a J / L e j F M M Y j q Q P O K M G i s 1 L 3 r l i l t 1 5 y C r x M t J B X I 0 e u W v b j 9 h W Y z S M E G 1 7 n h u a o I J V Y Y z g d N S N 9 O Y U j a i A + x Y K m m M O p j M D 5 2 S M 6 v 0 S Z Q o W 9 K Q u f p 7 Y k J j r c d x a D t j a o Z 6 2 Z u J / 3 m d z E R X w Y T L N D M o 2 W J R l A l i E j L 7 m v S 5 Q m b E 2 B L K F L e 3 E j

a k i j J j s y n Z E L z l l 1 e J X 6 t e V 9 1 m r V K / y d M o w g m c w j l 4 c A l 1 u I M G + M A A 4 R l e 4 c 1 5 d F 6 c d + d j 0 V p w 8 p l j + A P n 8 w f r T 4 y I < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " t 8 a F 3 V U l 9 Y v E k j 0 l n f v 7 1 J J e p + U = " > A

A A B 5 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I z G x I n d Y q B 3 R x h I S T 0 j g Q v a W O V j Z 2 7 v s 7 p k Q w i + w s V B j 6 1 + y 8 9 + 4 w B U K v m S S l / d m M j M v T A X X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x 6 0 E m m G P o s E Y l q h 1 S j 4 B J 9 w 4 3 A d q q Q x q H A V j i 6 n f m t J 1 S a J / L e j F M M Y j q Q P O K M G i s 1 L 3 r l i l t 1 5 y C r x M t J B X I 0 e u W v b j 9 h W Y z S M E G 1 7 n h u a o I J V Y Y z g d N S N 9 O Y U j a i A + x Y K m m M O p j M D 5 2 S M 6 v 0 S Z Q o W 9 K Q u f p 7 Y k J j r c d x a D t j a o Z 6 2 Z u J / 3 m d z E R X w Y T L N D M o 2 W J R l A l i E j L 7 m v S 5 Q m b E 2 B L K F L e 3 E j

a k i j J j s y n Z E L z l l 1 e J X 6 t e V 9 1 m r V K / y d M o w g m c w j l 4 c A l 1 u I M G + M A A 4 R l e 4 c 1 5 d F 6 c d + d j 0 V p w 8 p l j + A P n 8 w f r T 4 y I < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " t 8 a F 3 V U l 9 Y v E k j 0 l n f v 7 1 J J e p + U = " > A

A A B 5 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I z G x I n d Y q B 3 R x h I S T 0 j g Q v a W O V j Z 2 7 v s 7 p k Q w i + w s V B j 6 1 + y 8 9 + 4 w B U K v m S S l / d m M j M v T A X X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x 6 0 E m m G P o s E Y l q h 1 S j 4 B J 9 w 4 3 A d q q Q x q H A V j i 6 n f m t J 1 S a J / L e j F M M Y j q Q P O K M G i s 1 L 3 r l i l t 1 5 y C r x M t J B X I 0 e u W v b j 9 h W Y z S M E G 1 7 n h u a o I J V Y Y z g d N S N 9 O Y U j a i A + x Y K m m M O p j M D 5 2 S M 6 v 0 S Z Q o W 9 K Q u f p 7 Y k J j r c d x a D t j a o Z 6 2 Z u J / 3 m d z E R X w Y T L N D M o 2 W J R l A l i E j L 7 m v S 5 Q m b E 2 B L K F L e 3 E j

a k i j J j s y n Z E L z l l 1 e J X 6 t e V 9 1 m r V K / y d M o w g m c w j l 4 c A l 1 u I M G + M A A 4 R l e 4 c 1 5 d F 6 c d + d j 0 V p w 8 p l j + A P n 8 w f r T 4 y I < / l a t e x i t >

7:

< l a t e x i t s h a 1 _ b a s e 6 4 = " 1 H w h e p 5 3

I + B R B 1 9 l b R 3 A U W z H 8 O 0 = " > A A A B 5 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I z G x I n c 0 a E e 0 s Y T E E x K 4 k L 1 l D l b 2 9 i 6 7 e y a E 8 A t s L N T Y + p f s / D c u c I W C L 5 n k 5 b 2 Z z M w L U 8 G 1 c d 1 v p 7 C x u b W 9 U 9 w t 7 e 0 f H B 6 V j 0 8 e d J I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v 5 3 7 7 C Z X m i b w 3 k x S D m A 4 l j z i j x k q t e r 9 c c a v u A m S d e D m p Q I 5 m v / z V G y Q s i 1 E a J q j W X c 9 N T T C l y n A m c F b q Z R p T y s Z 0 i F 1 L J Y 1 R B 9 P F o T N y Y Z U B i R J l S x q y U H 9 P T G m s 9 S Q O b W d M z U i v e n P x P 6 + b m e g q m H K Z Z g Y l W y 6 K M k F M Q u Z f k w F X y I y Y W E K Z 4 v Z W w k Z U U W Z s N i U b g r f 6 8 j r x a 9 X r q t u q V R o 3 e R p F O I N z u A Q P 6 t C A O 2 i C D w w Q n u E V 3 p x H 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w D x W 4 y M < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 H w h e p 5 3 I + B R B 1 9 l b R 3 A U W z H 8 O 0 = " > A A A B 5 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I z G x I n c 0 a E e 0 s Y T E E x K 4 k L 1 l D l b 2 9 i 6 7 e y a E 8 A t s L N T Y + p f s / D c u c I W C L 5 n k 5 b 2 Z z M w L U 8 G 1 c d 1 v p 7 C x u b W 9 U 9 w t 7 e 0 f H B 6 V j 0 8 e d J I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v 5 3 7 7 C Z X m i b w 3 k x S D m A 4 l j z i j x k q t e r 9 c c a v u A m S d e D m p Q I 5 m v / z V G y Q s i 1 E a J q j W X c 9 N T T C l y n A m c F b q Z R p T y s Z 0 i F 1 L J Y 1 R B 9 P F o T N y Y Z U B i R J l S x q y U H 9 P T G m s 9 S Q O b W d M z U i v e n P x P 6 + b m e g q m H K Z Z g Y l W y 6 K M k F M Q u Z f k w F X y I y Y W E K Z 4 v Z W w k Z U U W Z s N i U b g r f 6 8 j r x a 9 X r q t u q V R o 3 e R p F O I N z u A Q P 6 t C A O 2 i C D w w Q n u E V 3 p x H 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w D x W 4 y M < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 H w h e p 5 3 I + B R B 1 9 l b R 3 A U W z H 8 O 0 = " > A A A B 5 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I z G x I n c 0 a E e 0 s Y T E E x K 4 k L 1 l D l b 2 9 i 6 7 e y a E 8 A t s L N T Y + p f s / D c u c I W C L 5 n k 5 b 2 Z z M w L U 8 G 1 c d 1 v p 7 C x u b W 9 U 9 w t 7 e 0 f H B 6 V j 0 8 e d J I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v 5 3 7 7 C Z X m i b w 3 k x S D m A 4 l j z i j x k q t e r 9 c c a v u A m S d e D m p Q I 5 m v / z V G y Q s i 1 E a J q j W X c 9 N T T C l y n A m c F b q Z R p T y s Z 0 i F 1 L J Y 1 R B 9 P F o T N y Y Z U B i R J l S x q y U H 9 P T G m s 9 S Q O b W d M z U i v e n P x P 6 + b m e g q m H K Z Z g Y l W y 6 K M k F M Q u Z f k w F X y I y Y W E K Z 4 v Z W w k Z U U W Z s N i U b g r f 6 8 j r x a 9 X r q t u q V R o 3 e R p F O I N z u A Q P 6 t C A O 2 i C D w w Q n u E V 3 p x H 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w D x W 4 y M < / l a t e x i t > 11 < l a t e x i t s h a 1 _ b a s e 6 4 = " N Q C P 9 J 6 J a e + C 4 H Q w t Q L O + Y O y f 3 Y = " > A A A B 6 H i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m L i i b R c 1 B v R i 0 c 0 V k i g I d t l C x u 2 2 2 Z 3 a k I a / o E X D 2 q 8 + p O 8 + W 9 c o A c F X z L J y 3 s z m Z k X p l I Y d N 1 v p 7 S 2 v r G 5 V d 6 u 7 O z u 7 R 9 U D 4 8 e T Z J p x n 2 W y E R 3 Q m q 4 F I r 7 K F D y T q o 5 j U P J 2 + H 4 Z u a 3 n 7 g 2 I l E P O E l 5 E N O h E p F g F K 1 0 7 3 n 9 a s 2 t u 3 O Q V e I V p A Y F W v 3 q V 2 + Q s C z m C p m k x n Q 9 N 8 U g p x o F k 3 x a 6 W W G p 5 S N 6 Z B 3 L V U 0 5 i b I 5 5 d O y Z l V B i R K t C 2 F Z K 7 + n s h p b M w k D m 1 n T H F k l r 2 Z + J / X z T C 6 D H K h 0 g y 5 Y o t F U S Y J J m T 2 N h k I z R n K i S W U a W F v J W x E N W V o w 6 n Y E L z l l 1 e J 3 6 h f 1 d 2 7 R q 1 5 X a R R h h M 4 h X P w 4 A K a c A s t 8 I F B B M / w C m / O 2 H l x 3 p 2 P R W v J K W a O 4 Q + c z x 9 X f 4 z B < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N Q C P 9 J 6 J a e + C 4 H Q w t Q L O + Y O y f 3 Y = " > A A A B 6 H i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m L i i b R c 1 B v R i 0 c 0 V k i g I d t l C x u 2 2 2 Z 3 a k I a / o E X D 2 q 8 + p O 8 + W 9 c o A c F X z L J y 3 s z m Z k X p l I Y d N 1 v p 7 S 2 v r G 5 V d 6 u 7 O z u 7 R 9 U D 4 8 e T Z J p x n 2 W y E R 3 Q m q 4 F I r 7 K F D y T q o 5 j U P J 2 + H 4 Z u a 3 n 7 g 2 I l E P O E l 5 E N O h E p F g F K 1 0 7 3 n 9 a s 2 t u 3 O Q V e I V p A Y F W v 3 q V 2 + Q s C z m C p m k x n Q 9 N 8 U g p x o F k 3 x a 6 W W G p 5 S N 6 Z B 3 L V U 0 5 i b I 5 5 d O y Z l V B i R K t C 2 F Z K 7 + n s h p b M w k D m 1 n T H F k l r 2 Z + J / X z T C 6 D H K h 0 g y 5 Y o t F U S Y J J m T 2 N h k I z R n K i S W U a W F v J W x E N W V o w 6 n Y E L z l l 1 e J 3 6 h f 1 d 2 7 R q 1 5 X a R R h h M 4 h X P w 4 A K a c A s t 8 I F B B M / w C m / O 2 H l x 3 p 2 P R W v J K W a O 4 Q + c z x 9 X f 4 z B < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N Q C P 9 J 6 J a e + C 4 H Q w t Q L O + Y O y f 3 Y = " > A A A B 6 H i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m L i i b R c 1 B v R i 0 c 0 V k i g I d t l C x u 2 2 2 Z 3 a k I a / o E X D 2 q 8 + p O 8 + W 9 c o A c F X z L J y 3 s z m Z k X p l I Y d N 1 v p 7 S 2 v r G 5 V d 6 u 7 O z u 7 R 9 U D 4 8 e T Z J p x n 2 W y E R 3 Q m q 4 F I r 7 K F D y T q o 5 j U P J 2 + H 4 Z u a 3 n 7 g 2 I l E P O E l 5 E N O h E p F g F K 1 0 7 3 n 9 a s 2 t u 3 O Q V e I V p A Y F W v 3 q V 2 + Q s C z m C p m k x n Q 9 N 8 U g p x o F k 3 x a 6 W W G p 5 S N 6 Z B 3 L V U 0 5 i b I 5 5 d O y Z l V B i R K t C 2 F Z K 7 + n s h p b M w k D m 1 n T H F k l r 2 Z + J / X z T C 6 D H K h 0 g y 5 Y o t F U S Y J J m T 2 N h k I z R n K i S W U a W F v J W x E N W V o w 6 n Y E L z l l 1 e J 3 6 h f 1 d 2 7 R q 1 5 X a R R h h M 4 h X P w 4 A K a c A s t 8 I F B B M / w C m / O 2 H l x 3 p 2 P R W v J K W a O 4 Q + c z x 9 X f 4 z B < / l a t e x i t > 19 < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 Q B R O O c 3 s 3 J a Y X c i l t o O h B g Z 3 / E = " > A A A B 6 H i c b V B N T 8 J A E J 3 6 i f i F e v S y k Z h 4 I i 0 X 5 U b 0 4 h G N F R J o y H a Z w o b t t t n d m p C G f + D F g x q v / i R v / h s X 6 E H B l 0 z y 8 t 5 M Z u a F q e D a u O 6 3 s 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 4 f d Z I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v Z n 7 7 C Z X m i X w w k x S D m A 4 l j z i j x k r 3 X q N f q b o 1 d w 6 y S r y C V K F A q 1 / 5 6 g 0 S l s U o D R N U 6 6 7 n p i b I q T K c C Z y W e 5 n G l L I x H W L X U k l j 1 E E + v 3 R K z q 0 y I F G i b E l D 5 u r v i Z z G W k / i 0 H b G 1 I z 0 s j c T / / O 6 m Y m u g p z L N D M o 2 W J R l A l i E j J 7 m w y 4 Q m b E x B L K F L e 3 E j a i i j J j w y n b E L z l l 1 e J X 6 8 1 a u 5 d v d q 8 L t I o w S m c w Q V 4 c A l N u I U W + M A g g m d 4 h T d n 7 L w 4 7 8 7 H o n X N K W Z O 4 A + c z x 9 j l 4 z J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 Q B R O O c 3 s 3 J a Y X c i l t o O h B g Z 3 / E = " > A A A B 6 H i c b V B N T 8 J A E J 3 6 i f i F e v S y k Z h 4 I i 0 X 5 U b 0 4 h G N F R J o y H a Z w o b t t t n d m p C G f + D F g x q v / i R v / h s X 6 E H B l 0 z y 8 t 5 M Z u a F q e D a u O 6 3 s 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 4 f d Z I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v Z n 7 7 C Z X m i X w w k x S D m A 4 l j z i j x k r 3 X q N f q b o 1 d w 6 y S r y C V K F A q 1 / 5 6 g 0 S l s U o D R N U 6 6 7 n p i b I q T K c C Z y W e 5 n G l L I x H W L X U k l j 1 E E + v 3 R K z q 0 y I F G i b E l D 5 u r v i Z z G W k / i 0 H b G 1 I z 0 s j c T / / O 6 m Y m u g p z L N D M o 2 W J R l A l i E j J 7 m w y 4 Q m b E x B L K F L e 3 E j a i i j J j w y n b E L z l l 1 e J X 6 8 1 a u 5 d v d q 8 L t I o w S m c w Q V 4 c A l N u I U W + M A g g m d 4 h T d n 7 L w 4 7 8 7 H o n X N K W Z O 4 A + c z x 9 j l 4 z J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 Q B R O O c 3 s 3 J a Y X c i l t o O h B g Z 3 / E = " > A A A B 6 H i c b V B N T 8 J A E J 3 6 i f i F e v S y k Z h 4 I i 0 X 5 U b 0 4 h G N F R J o y H a Z w o b t t t n d m p C G f + D F g x q v / i R v / h s X 6 E H B l 0 z y 8 t 5 M Z u a F q e D a u O 6 3 s 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 4 f d Z I p h j 5 L R K I 6 I d U o u E T f c C O w k y q k c S i w H Y 5 v Z n 7 7 C Z X m i X w w k x S D m A 4 l j z i j x k r 3 X q N f q b o 1 d w 6 y S r y C V K F A q 1 / 5 6 g 0 S l s U o D R N U 6 6 7 n p i b I q T K c C Z y W e 5 n G l L I x H W L X U k l j 1 E E + v 3 R K z q 0 y I F G i b E l D 5 u r v i Z z G W k / i 0 H b G 1 I z 0 s j c T / / O 6 m Y m u g p z L N D M o 2 W J R l A l i E j J 7 m w y 4 Q m b E x B L K F L e 3 E j a i i j J j w y n b E L z l l 1 e J X 6 8 1 a u 5 d v d q 8 L t I o w S m c w Q V 4 c A l N u I U W + M A g g m d 4 h T d n 7 L w 4 7 8 7 H o n X N K W Z O 4

A + c z x 9 j l 4 z J < / l a t e x i t > I < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 n 9 8 q f + l p c m y o 3 

C E 0 L c t o t O / E p I = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 3 p r w d h C G 8 p m O 2 n X b j Z h d y O U 0 F / g x Y O K V / + S N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 R R W V t f W N 4 q b p a 3 t n d 2 9 8 v 7 B g 4 5 T x d B j s Y h V O 6 A a B Z f o G W 4 E t h O F N A o E t o L R z d R v P a H S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p e d c r V 9 y q O w N Z J r W c V C B H o 1 f + 6 v Z j l k Y o D R N U 6 0 7 N T Y y f U W U 4 E z g p d V O N C W U j O s C O p Z J G q P 1 s d u i E n F i l T 8 J Y 2 Z K G z N T f E x m N t B 5 H g e 2 M q B n q R W 8 q / u d 1 U h N e + h m X S W p Q s v m i M B X E x G T 6 N e l z h c y I s S W U K W 5 v J W x I F W X G Z l O y I d Q W X 1 4 m 3 l n 1 q u o 2 z y v

V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 3 p r w d h C G 8 p m O 2 n X b j Z h d y O U 0 F / g x Y O K V / + S N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 R R W V t f W N 4 q b p a 3 t n d 2 9 8 v 7 B g 4 5 T x d B j s Y h V O 6 A a B Z f o G W 4 E t h O F N A o E t o L R z d R v P a H S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p e d c r V 9 y q O w N Z J r W c V C B H o 1 f + 6 v Z j l k Y o D R N U 6 0 7 N T Y y f U W U 4 E z g p d V O N C W U j O s C O p Z J G q P 1 s d u i E n F i l T 8 J Y 2 Z K G z N T f E x m N t B 5 H g e 2 M q B n q R W 8 q / u d 1 U h N e + h m X S W p Q s v m i M B X E x G T 6 N e l z h c y I s S W U K W 5 v J W x I F W X G Z l O y I d Q W X 1 4 m 3 l n 1 q u o 2 z y v

V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 3 p r w d h C G 8 p m O 2 n X b j Z h d y O U 0 F / g x Y O K V / + S N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 R R W V t f W N 4 q b p a 3 t n d 2 9 8 v 7 B g 4 5 T x d B j s Y h V O 6 A a B Z f o G W 4 E t h O F N A o E t o L R z d R v P a H S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p e d c r V 9 y q O w N Z J r W c V C B H o 1 f + 6 v Z j l k Y o D R N U 6 0 7 N T Y y f U W U 4 E z g p d V O N C W U j O s C O p Z J G q P 1 s d u i E n F i l T 8 J Y 2 Z K G z N T f E x m N t B 5 H g e 2 M q B n q R W 8 q / u d 1 U h N e + h m X S W p Q s v m i M B X E x G T 6 N e l z h c y I s S W U K W 5 v J W x I F W X G Z l O y I d Q W X 1 4 m 3 l n 1 q u o 2 z y v 1 6 z y N I h z B M Z x C D S 6 g D r f Q A A 8 Y I D z D K 7 w 5 j 8 6 L 8 + 5 8 z F s L T j 5 z C H / g f P 4 A D U C M o A = = < / l a t e x i t >

2N:

< l a t e x i t s h a 1 _ b a s e 6 4 = " 2 V N a 0 c g R / A j U X v 7 c i 8 9 / D u 3 S 9 g U = " >

A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m L o N 6 K X j x J F W M L b S i b 7 a Z d u t m E 3 Y l Q Q v + B F w 8 q X v 1 J 3 v w 3 b t s c t P X B w O O 9 G W b m B Y k U B l 3 3 2 y m s r K 6 t b x Q 3 S 1 v b O 7 t 7 5 f 2 D R x O n m n G P x T L W 7 Y A a L o X i H g q U v J 1 o T q N A 8 l Y w u p 7 6 r S e u j Y j V A 4 4 T 7 k d 0 o E Q o G E U r 3 d d v e + W K W 3 V n I M u k l p M K 5 G j 2 y l / d f s z S i C t k k h r T q b k J + h n V K J j k k 1 I 3 N T y h b E Q H v G O p o h E 3 f j a 7 d E J O r N I n Y a x t K S Q z 9 f d E R i N j x l F g O y O K Q 7 P o T c X / v E 6 K 4 Y W f C Z W k y B W b L w p T S T A m 0 7 d J X 2 j O U I 4 t o U w L e y t h Q 6 o p Q x t O y Y Z Q W 3 x 5 m X j 1 6 m X V v T u r N K 7 y N I p w B M d w C j U 4 h w b c Q B M 8 Y B D C M 7 z C m z N y X p x 3 5 2 P e W n D y m U P 4 A + f z B 4 V 6 j O E = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 V N a 0 c g R / A j U X v 7 c i 8 9 / D u 3 S 9 g U = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m L o N 6 K X j x J F W M L b S i b 7 a Z d u t m E 3 Y l Q Q v + B F w 8 q X v 1 J 3 v w 3 b t s c t P X B w O O 9 G W b m B Y k U B l 3 3 2 y m s r K 6 t b x Q 3 S 1 v b O 7 t 7 5 f 2 D R x O n m n G P x T L W 7 Y A a L o X i H g q U v J 1 o T q N A 8 l Y w u p 7 6 r S e u j Y j V A 4 4 T 7 k d 0 o E Q o G E U r 3 d d v e + W K W 3 V n I M u k l p M K 5 G j 2 y l / d f s z S i C t k k h r T q b k J + h n V K J j k k 1 I 3 N T y h b E Q H v G O p o h E 3 f j a 7 d E J O r N I n Y a x t K S Q z 9 f d E R i N j x l F g O y O K Q 7 P o T c X / v E 6 K 4 Y W f C Z W k y B W b L w p T S T A m 0 7 d J X 2 j O U I 4 t o U w L e y t h Q 6 o p Q x t O y Y Z Q W 3 x 5 m X j 1 6 m X V v T u r N K 7 y N I p w B M d w C j U 4 h w b c Q B M 8 Y B D C M 7 z C m

z N y X p x 3 5 2 P e W n D y m U P 4 A + f z B 4 V 6 j O E = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 V N a 0 c g R / A j U X v 7 c i 8 9 / D u 3 S 9 g U = " > ⌘ < l a t e x i t s h a 1 _ b a s e 6 4 = " M o v F e T 8 J H V 0 F B V e 6 5 0 Y T r S v q t t 4 = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b S b t 2 k 4 2 7 m 0 I p / Q 9 e P K h 4 9 Q d 5 8 9 + 4 b X P Q 1 g c D j / d m m J k X p o J r 4 7 r f z s r q 2 v r G Z m G r u L 2 z u 7 d f O j i s a 5 k p h j 6 T Q q p m S D U K n q B v u B H Y T B X S O B T Y C A e 3 U 7 8 x R K W 5 T B 7 M K M U g p r 2 E R 5 x R Y 6 V 6 G 5 8 y P u y U y m 7 F n Y E s E y 8 n Z c h R 6 5 S + 2 l 3 J s h g T w w T V u u W 5 q Q n G V B n O B E 6 K 7 U x j S t m A 9 r B l a U J j 1 M F 4 d u 2 E n F q l S y K p b C W G z N T f E 2 M a a z 2 K Q 9 s Z U 9 P X i 9 5 U / M 9 r Z S a 6 C s Y 8 S T O D C Z s v i j J B j C T T 1 0 m X K 2 R G j C y h T H F 7 K 2 F 9 q i g z N q C i D c F b f H m Z + O e V 6 4 p 7 f 1 G u 3 u R p F O A Y T u A M P L i E K t x B D X x g 8 A j P 8 A p v j n R e n H f n Y 9 6 6 4 u Q z R / A H z u c P L j y P D w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M o v F e T 8 J H V 0 F B V e 6 5 0 Y T r S v q t t 4 = " >

A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m L o N 6 K X j x J F W M L b S i b 7 a Z d u t m E 3 Y l Q Q v + B F w 8 q X v 1 J 3 v w 3 b t s c t P X B w O O 9 G W b m B Y k U B l 3 3 2 y m s r K 6 t b x Q 3 S 1 v b O 7 t 7 5 f 2 D R x O n m n G P x T L W 7 Y A a L o X i H g q U v J 1 o T q N A 8 l Y w u p 7 6 r S e u j Y j V A 4 4 T 7 k d 0 o E Q o G E U r 3 d d v e + W K W 3 V n I M u k l p M K 5 G j 2 y l / d f s z S i C t k k h r T q b k J + h n V K J j k k 1 I 3 N T y h b E Q H v G O p o h E 3 f j a 7 d E J O r N I n Y a x t K S Q z 9 f d E R i N j x l F g O y O K Q 7 P o T c X / v E 6 K 4 Y W f C Z W k y B W b L w p T S T A m 0 7 d J X 2 j O U I 4 t o U w L e y t h Q 6 o p Q x t O y Y Z Q W 3 x 5 m X j 1 6 m X V v T u

A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b S b t 2 k 4 2 7 m 0 I p / Q 9 e P K h 4 9 Q d 5 8 9 + 4 b X P Q 1 g c D j / d m m J k X p o J r 4 7 r f z s r q 2 v r G Z m G r u L 2 z u 7 d f O j i s a 5 k p h j 6 T Q q p m S D U K n q B v u B H Y T B X S O B T Y C A e 3 U 7 8 x R K W 5 T B 7 M K M U g p r 2 E R 5 x R Y 6 V 6 G 5 8 y P u y U y m 7 F n Y E s E y 8 n Z c h R 6 5 S + 2 l 3 J s h g T w w T V u u W 5 q Q n G V B n O B E 6 K 7 U x j S t m A 9 r B l a U J j 1 M F 4 d u 2 E n F q l S y K p b C W G z N T f E 2 M a a z 2 K Q 9 s Z U 9 P X i 9 5 U / M 9 r Z S a 6 C s Y 8 S T O D C Z s v i j J B j C T T 1 0 m X K 2 R G j C y h T H F 7 K 2 F 9 q i g z N q C i D c F b f H m Z + O e V 6 4 p 7 f 1 G u 3 u R p F O A Y T u A M P L i E K t x B D X

x g 8 A j P 8 A p v j n R e n H f n Y 9 6 6 4 u Q z R / A H z u c P L j y P D w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M o v F e T 8 J H V 0 F B V e 6 5 0 Y T r S v q t t 4 = " >

A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o k I 6 q 3 o x W M F 0 x b a U D b b S b t 2 k 4 2 7 m 0 I p / Q 9 e P K h 4 9 Q d 5 8 9 + 4 b X P Q 1 g c D j / d m m J k X p o J r 4 7 r f z s r q 2 v r G Z m G r u L 2 z u 7 d f O j i s a 5 k p h j 6 T Q q p m S D U K n q B v u B H Y T B X S O B T Y C A e 3 U 7 8 x R K W 5 T B 7 M K M U g p r 2 E R 5 x R Y 6 V 6 G 5 8 y P u y U y m 7 F n Y E s E y 8 n Z c h R 6 5 S + 2 l 3 J s h g T w w T V u u W 5 q Q n G V B n O B E 6 K 7 U x j S t m A 9 r B l a U J j 1 M F 4 d u 2 E n F q l S y K p b C W G z N T f E 2 M a a z 2 K Q 9 s Z U 9 P X i 9 5 U / M 9 r Z S a 6 C s Y 8 S T O D C Z s v i j J B j C T T 1 0 m X K 2 R G j C y h T H F 7 K 2 F 9 q i g z N q C i D c F b f H m Z + O e V 6 4 p 7 f 1 G u 3 u R p F O A Y T u A M P L i E K t x B D X x g 8 A j P 8 A p v j n R e n H f n Y 9 6 6 4 u Q z R / A H z u c P L j y P D w = = < / l a t e x i t > N < l a t e x i t s h a 1 _ b a s e 6 4 = " D 7 m a n c u c Q c C J 5 r 9 n h a E A x O 5 U V I U = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 5 6 k B W M L b S i b 7 a R d u 9 m E 3 Y 1 Q Q n + B F w 8 q X v 1 L 3 v w 3 b t s c t P X B w O O 9 G W b m B Y n g 2 r j u t 1 N Y W V 1 b 3 y h u l r a 2 d 3 b 3 y v s H D z p O F U O P x S J W 7 Y B q F F y i Z 7 g R 2 E 4 U 0 i g Q 2 A p G N 1 O / 9 Y R K 8 1 j e m 3 G C f k Q H k o e c U W O l 5 l 2 v X H G r 7 g x k m d R y U o E c j V 7 5 q 9 u P W R q h N E x Q r T s 1 N z F + R p X h T O C k 1 E 0 1 J p S N 6 A A 7 l k o a o f a z 2 a E T c m K V P g l j Z U s a M l N / T 2 Q 0 0 n o c B b Y z o m a o F 7 2 p + J / X S U 1 4 6 W d c J q l B y e a L w l Q Q E 5 P p 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 J h l B b f H m Z e G f V q 6 r b P K / U r / M 0 i n A E x 3 A K N b i A O t x C A z x g g P A M r / D m P D o v z r v z M W 8 t O P n M I f y B 8 / k D F M + M p Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D 7 m a n c u c Q c C J 5 r 9 n h a E A x O 5 U V I U = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 5 6 k B W M L b S i b 7 a R d u 9 m E 3 Y 1 Q Q n + B F w 8 q X v 1 L 3 v w 3 b t s c t P X B w O O 9 G W b m B Y n g 2 r j u t 1 N Y W V 1 b 3 y h u l r a 2 d 3 b 3 y v s H D z p O F U O P x S J W 7 Y B q F F y i Z 7 g R 2 E 4 U 0 i g Q 2 A p G N 1 O / 9 Y R K 8 1 j e m 3 G C f k Q H k o e c U W O l 5 l 2 v X H G r 7 g x k m d R y U o E c j V 7 5 q 9 u P W R q h N E x Q r T s 1 N z F + R p X h T O C k 1 E 0 1 J p S N 6 A A 7 l k o a o f a z 2 a E T c m K V P g l j Z U s a M l N / T 2 Q 0 0 n o c B b Y z o m a o F 7 2 p + J / X S U 1 4 6 W d c J q l B y e a L w l Q Q E 5 P p 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 J h l B b f H m Z e G f V q 6 r b P K / U r / M 0 i n A E x 3 A K N b i A O t x C A z x g g P A M r / D m P D o v z r v z M W 8 t O P n M I f y B 8 / k D F M + M p Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D 7 m a n c u c Q c C J 5 r 9 n h a E A x O 5 U V I U = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 5 6 k B W M L b S i b 7 a R d u 9 m E 3 Y 1 Q Q n + B F w 8 q X v 1 L 3 v w 3 b t s c t P X B w O O 9 G W b m B Y n g 2 r j u t 1 N Y W V 1 b 3 y h u l r a 2 d 3 b 3 y v s H D z p O F U O P x S J W 7 Y B q F F y i Z 7 g R 2 E 4 U 0 i g Q 2 A p G N 1 O / 9 Y R K 8 1 j e m 3 G C f k Q H k o e c U W O l 5 l 2 v X H G r 7 g x k m d R y U o E c j V 7 5 q 9 u P W R q h N E x Q r T s 1 N z F + R p X h T O C k 1 E 0 1 J p S N 6 A A 7 l k o a o f a z 2 a E T c m K V P g l j Z U s a M l N / T 2 Q 0 0 n o c B b Y z o m a o F 7 2 p + J / X S U 1 4 6 W d c J q l B y e a L w l Q Q E 5 P p 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 J h l B b f H m Z e G f V q 6 r b P K / U r / M 0 i n A E x 3 A K N b i A O t x C A z x g g P A M r / D m P D o v z r v z M W 8 t O P n M I f y B 8 / k D F M + M p Q = = < / l a t e x i t >

M:

< l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A

H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q

x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W / i S p q y l A W J 4 t h X p o 3 i Q x T S + D E w = " > A A A B 5 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l F U G 9 F L 1 6 E F o w t t K F s t p N 2 7 W Y T d j d C C f 0 F X j y o e P U v e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h Z X V t f a O 4 W d r a 3 t n d K + 8 f P O g 4 V Q w 9 F o t Y t Q O q U X C J n u F G Y D t R S K N A Y C s Y 3 U z 9 1 h M q z W N 5 b 8 Y J + h E d S B 5 y R o 2 V m n e 9 c s W t u j O Q Z V L L S Q V y N H r l r 2 4 / Z m m E 0 j B B t e 7 U 3 M T 4 G V W G M 4 G T U j f V m F A 2 o g P s W C p p h N r P Z o d O y I l V + i S M l S 1 p y E z 9 P Z H R S O t x F N j O i J q h X v S m 4 n 9 e J z X h p Z 9 x m a Q G J Z s v C l N B T E y m X 5 M + V 8 i M G F t C m e L 2 V s K G V F F m b D Y l G 0 J t 8 e V l 4 p 1 V r 6 p u 8 7 x S v 8 7 T K M I R H M M p 1 O A C 6 n A L D f C A A c I z v M K b 8 + i 8 O O / O x 7 y 1 4 O Q z h / A H z u c P E 0 y M p A = = < / l a t e x i t >

Figure 1 : A typical u-net module with outer residual connection. M is the number of input features. Across the u-net module, each layer has the same number of output feature maps (except for the final 1 × 1 filter), which we denote N . For better understanding the figure also includes the field of view (FoV) of each convolutional kernel (top) and the feature map size at the output of each filter (bottom), assuming a 64 × 64 input I. Best viewed in color.

fixed number of feature maps, N . To mitigate high frequency noise from the sampling operation, each strided conv/de-conv layer is followed by a convolution. Unlike traditional u-nets, the design of convolutional layers in our u-net module helps in retaining the original size of the feature maps at its output Consequently, multiple u-net modules can be stacked without loosing resolution.

In the following, we briefly highlight some of the design choices of the architecture. In comparison to traditional u-nets, the max-pooling operation is replaced with strided convolution for SUNets. The use of strided convolutions enables different filters in each u-net module to operate at different resolutions (see the discussion in Section 5). Moreover, the repeated use of max-pooling operations can cause gridding artifacts in dilated networks (Yu et al., 2017) .

Unlike the u-nets of Ronneberger et al. (2015) ; Newell et al. (2016) , our u-net module is comprised of only two levels of depth. We considered two major factors in choosing the depth: field of view (FoV) of the innermost conv filter and the total number of parameters in a single u-net module. The number of parameters influences the total number of stacks in SUNets. While keeping the total parameters of SUNet approximately constant, we experimented with a higher depth of three and four. We found that the increase in depth indeed led to a decline in performance for the image classification task. This may not be surprising, given that a SUNet with depth of two is able to stack more u-net modules. Moreover, deeper u-net modules make it harder to train the inner-most convolutional layers due to the vanishing gradients problem (Bengio et al., 1994) . For instance, in our current design, the maximum length of the gradient path is six. The popular classification networks of He et al. (2016a) ; Huang et al. (2017) are known to operate primarily on features with 28 2 and 14 2 resolution. At this scale, the effective FoV of 19 is more than sufficient to capture long-distance contextual information. Moreover, the stacking of multiple u-net modules will also serve to increase the effective FoV of higher layers.

SUNets train best when there is sufficient gradient flow to the bottom-most u-net layers. To avoid vanishing gradients, we include a skip connection (He et al., 2016a; Huang et al., 2017) around each u-net module. Also, inspired by the design of bottleneck blocks (He et al., 2016a) , we also include 1×1 convolutional layers. Bottleneck layers restricts the number of input features to a small number (N ), avoiding parameter inflation.

When stacking multiple u-nets it makes sense for each u-net module to reuse the raw feature maps from all the preceding u-net modules. Thus we also explored replacing the identity connection with dense connectivity (Huang et al., 2017) . This new network is memory intensive which in turn prevented proper learning of the batch-norm parameters. Instead, we chose to utilize dense connectivity only within each u-net, i.e., while reusing feature maps from E1 at D1. Thus the proposed u-net module leverages skip connectivity without getting burdened. 

  1 × 1 conv, 64 U-Net, N=64 1 × 1 conv, 256   × 2   1 × 1 conv, 128 U-Net, N=128 1 × 1 conv, 512   × 2   1 × 1 conv, 128 U-Net, N=128 1 × 1 conv, 512   × 2

Transition Layer 28 × 28 2 × 2 average pool, stride 2

UNets Block (2) 28 × 28

  1 × 1 conv, 64 U-Net, N=64 1 × 1 conv, 512   × 4   1 × 1 conv, 128 U-Net, N=128 1 × 1 conv, 1024   × 4   1 × 1 conv, 128 U-Net, N=128 1 × 1 conv, 1280   × 7

Transition Layer 14 × 14 2 × 2 average pool, stride 2

UNets Block (3) 14 × 14 Note that the building block in bracket refers to the integrated u-net module shown in Figure 1 .

  1 × 1 conv, 64 U-Net, N=64 1 × 1 conv, 768   × 4   1 × 1 conv, 128 U-Net, N=128 1 × 1 conv, 1536   × 4   1 × 1 conv, 128 U-Net, N=128 1 × 1 conv, 2048   × 7 Transition Layer 7 × 7 2 × 2 average pool, stride 2 UNets Block (4) 7 × 7   1 × 1 conv, 64 U-Net + , N=64 1 × 1 conv, 1024   × 1   1 × 1 conv, 128 U-Net + , N=128 1 × 1 conv, 2048   × 1   1 × 1 conv, 128 U-Net + , N=128 1 × 1 conv, 2304   × 1 Classification Layer 1 × 1 7 × 7

Before addressing segmentation, we describe a stacked u-net (SUNet) architecture that is appropriate for image classification. Because the amount of labeled data available for classification is much larger than for segmentation, classification tasks are often used to pre-train feature extraction networks, which are then adapted to perform segmentation.

The network design of SUNets for ImageNet classification is summarized in Table 1 . Note that each "conv" layer shown in the table corresponds to a sequence of "BN-ReLU-Conv" layers. The three listed configurations mainly differ in the number of output feature maps N of each convolutional layer and the total number of stacks in blocks 2 and 3. Input images are processed using a 7 × 7 conv filter followed by a residual block. Inspired by the work on dilated resnet (Yu et al., 2017) , the conventional max-pooling layer at this stage is replaced by a strided convolutional layer inside the residual block. Subsequently, the feature maps are processed bottom-up as well as top-down by multiple stacks of u-nets at different scales and with regularly decreasing resolutions. The feature map input size to block 4 is 7 × 7 and is further reduced to 2 × 2 at the input to the encoder E2 of the u-net module. At this resolution, it is not possible to have E2 and D2 layers, and hence a trimmed version of u-nets (u-net + ) are employed in block 4. The u-net + includes a single level of encoder and decoder (E1, D1) processing. Following block 4, a global average pooling is performed on features and transferred to a softmax classifier.

The residual connection in all but the first u-net in each block is implemented as an identity mapping. In the first u-nets the skip connection is implemented using an expansion layer i.e., a 1 × 1 conv filter. The number of feature map outputs from each block approximately equates to the total number of feature maps generated by all the preceding u-net modules. This arrangement allows flexibility for the network to retain all the raw feature maps of the preceding modules. Moreover among all other possibilities, the above architectures were picked because their performance on the image classification task is roughly equivalent to the ResNet-18, 50 and 101 network architectures (discussed in Section 6), albeit with fewer parameters. Finally, we note that, although SUNets were designed for pixel-level localization tasks, the selected models were chosen only based on their classification performance. By linking the model selection process to the primary task (segmentation) there is a possibility of improving performance.

Multigrid vs Downsampling::

We compare the performance of multigrid dilation (as shown in Figure 2 ) inside each u-net against the usual downsampling (Figure 1 ). We consider a dilated SUNet-7-128 network and report performance at different training output stride. The result is summarized in Table 4 . For a dilated network, replacing strided convolutional layers with the corresponding dilated layers is more logical as well as beneficial. This is because, when operating dilated convolutional layers with stride > 1, alternate feature points are dropped without being processed by any of the filters, leading to high frequency noise at the decoder output. Furthermore, due to a skip connection, the features from the lower layers are also corrupted. Due to error propagation, this effect is more prominent in a network with many dilated modules (for eg., output stride = 8).

Output Stride and Inference Strategy: Finally, we experiment with three different training output strides (8,16,32) and multi-scale inference at test time. For OS = 32, none of the layers are dilated and hence de-gridding layers were not used. Training with an OS = 32 is equivalent to fine-tuning a classification network with a global pooling layer removed. For multi-scale inference, each input image is scaled and tested using multiple scales {0.5, 0.75, 1.0, 1.25} and its left-right flipped image.

The average over all output maps is used in the final prediction. See results in Table 5 . We note that:

1. The network trained with OS = 32 performs 0.7 IoU better (with single scale) than the Resnet-101 and Resnet-152 models (Wu et al., 2016 ) each trained at OS = 8. This is significant as SUNet model outputs 16× fewer pixels at 4× faster training/inference with no performance drop. 2. The degridding layers do not improve performance at OS = 16. Since there is only small change in dilation factor between the final SUNet layer & classification layer, aliasing is not problematic. 3. The margin of performance improvement decreases with decrease in training OS. Given this and the above fact, subsequently we only report performance for models trained at OS = 16 without any degridding layers.

Pretraining with MS-COCO: Following common practice (Chen et al., 2017) , we pretrain SUNet-7-128 with the MS-COCO dataset (Lin et al., 2014) . The dataset contains pixel-level annotation for 80 object classes. Except for the PASCAL VOC classes, the pixel annotation for all other classes is set to the background class. Following this, the images having < 1000 annotated pixel labels were dropped yielding 90, 000 training images. After pretraining, the model is fine-tuned on "trainaug" for 5K iterations with 10× smaller initial learning rate. In the end, our model achieves 83.27% mIoU on the validation set. This performance is slightly better than the ResNet+ASPP model (Chen et al., 2017) (Li et al., 2017) 71.1 Piecewise (VGG16) (Lin et al., 2016) 71.6 RefineNet (Lin et al., 2017a) 73.6

Understanding Conv Wang et al. (2017) Cityscapes: A similar training strategy as in PASCAL is adopted except that the multi-scale inference is performed on additional scales {1.5, 1.75, 2.0, 2.25, 2.5}. Only the finer annotation set was used for training. The comparison on the Cityscapes test set results are displayed in Table 7 .

7. DISCUSSION AND CONCLUSION:

The fundamental structure of conventional bottom-up classification networks limits their efficacy on secondary tasks involving pixel-level localization or classification. To overcome this drawback, a new network architecture, stacked u-nets (SUNets), is discussed in this paper. SUNets leverage the information globalization power of u-nets in a deeper network architecture that is capable of handling the complexity of natural images. SUNets performs exceptionally well on semantic segmentation tasks while achieving fair performance on ImageNet classification. It generalizes well across tasks without any bells and whistles (difference between networks for two task is dilation). 

:



Figures and Tables:

1 6 z y N I h z B M Z x C D S 6 g D r f Q A A 8 Y I D z D K 7 w 5 j 8 6 L 8 + 5 8 z F s L T j 5 z C H / g f P 4 A D U C M o A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 n 9 8 q f + l p c m y o 3 C E 0 L c t o t O / E p I = " > A A A B 5 3 i c b

1 6 z y N I h z B M Z x C D S 6 g D r f Q A A 8 Y I D z D K 7 w 5 j 8 6 L 8 + 5 8 z F s L T j 5 z C H / g f P 4 A D U C M o A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 n 9 8 q f + l p c m y o 3 C E 0 L c t o t O / E p I = " > A A A B 5 3 i c b

r N K 7 y N I p w B M d w C j U 4 h w b c Q B M 8 Y B D C M 7 z C m z N y X p x 3 5 2 P e W n D y m U P 4 A + f z B 4 V 6 j O E = < / l a t e x i t >

Figure 3: Visualization of the segmentation output on PASCAL VOC 2012 val set when trained at an output stride = 16 using SUNet-7-128 network + MS-COCO. Final row shows couple of failure case which happens due to, ambiguous annotation and inability in detecting low resolution objects.

Figure 4: Visualization of the segmentation output on PASCAL VOC 2012 test set.

SUNet architectures for ImageNet. N denotes the number of filters per convolutional layer.

Table content: <table><tr><td>global average pool</td></tr><tr><td>1000D fully-connected, softmax</td></tr></table>

. The performance on each of these datasets is reported using intersection-over-union(IoU)   averaged over all classes. The experimental details are shared in Appendix B.Unless mentioned, for all our experiments we set output stride = 16. This means only the unet modules in the final block (4) operate at dilation factor of two; all other modules use the same stride as in the original classification model. The training and inference are 2× faster with output stride = 16 rather than 8. We compare the performance of the dilated SUNet architecture on semantic segmentation against the popular dilated ResNet-101 model. Models were fine-tuned on the "trainaug" set without the degridding layers and evaluated on the validation set.The performance of the plain dilated SUNets surpasses that of ResNet-101 by a wide margin (Table 3). In fact, the smallest SUNet model, SUNet-64 with 6.7M parameters, beats ResNet-101 (with 44.5M) by an absolute margin of 4.5% IoU while SUNet-7-128, the counterpart network to ResNet-101, improves by over 10.5% IoU. This is substantial, given that the gap between the ResNet-101 and VGG-16 models is ∼ 3%(Chen et al., 2016a)  (at output stride = 8). This contrasts with the negligible performance differences observed on classification, and suggests that specialized segmentation network architectures can surpass architectures adapted from classification.

Table content: <table><tr><td>Conv 7x7 stride 2</td><td>Residual Block</td><td>U-Net</td></tr><tr><td colspan="2">6.2.1 ABLATION STUDY</td><td/></tr><tr><td colspan="3">We experiment with different SUNet variants on the PASCAL VOC 2012 dataset.</td></tr><tr><td colspan="2">SUNets vs ResNet-101:</td><td/></tr></table>

(82.70%)  and equivalent to Xception+ASPP+Decoder model ofChen et al. (2018) (83.34%). Performance comparison at various output stride and inference strategies. MS: Multi-scale, DL: with Degridding Layers Methods mIoU Piecewise (VGG16) (Lin et al., 2016) 78.0 LRR+CRF (Ghiasi & Fowlkes, 2016) 77.3 DeepLabv2+CRF (Chen et al., 2016a) 79.7 Large-Kernel+CRF (Peng et al., 2017) 82.2 Deep Layer Cascade * (Li et al., 2017) 82.7 Understanding Conv(Wang et al., 2017) 83.1RefineNet(Lin et al., 2017a)   82.4 RefineNet-ResNet152(Lin et al., 2017a) 83.4

Table content: <table><tr><td/><td/><td/><td colspan="3">train OS eval OS DL MS Flip mIoU</td></tr><tr><td colspan="3">OS Strided conv Multigrid</td><td>32</td><td>32</td><td>76.03</td></tr><tr><td>8 16</td><td>65.99 78.25</td><td>78.64 78.95</td><td>32 32</td><td>32 32</td><td>77.58 77.57</td></tr><tr><td/><td/><td/><td>16</td><td>16</td><td>78.95</td></tr><tr><td colspan="3">Table 4: Performance com-</td><td>16</td><td>16</td><td>78.10</td></tr><tr><td colspan="3">parison of multigrid dilation</td><td>16</td><td>16</td><td>80.22</td></tr><tr><td colspan="3">against strided convolution</td><td>16</td><td>16</td><td>80.40</td></tr><tr><td colspan="3">inside each u-net module,</td><td>8</td><td>8</td><td>78.64</td></tr><tr><td>using</td><td>the</td><td>SUNet-7-128</td><td>8</td><td>8</td><td>78.88</td></tr><tr><td colspan="3">model and evaluated us-</td><td>8</td><td>8</td><td>80.37</td></tr><tr><td colspan="3">ing mean IoU. OS denotes</td><td>8</td><td>8</td><td>80.50</td></tr><tr><td colspan="3">output stride during train-</td><td/><td/><td/></tr><tr><td>ing.</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">PSPNet Zhao et al. (2017)</td><td>85.4</td><td/><td/></tr><tr><td/><td colspan="2">SUNet-7-128 (OS = 16)</td><td>84.3</td><td/><td/></tr></table>

Performance comparison on PASCAL VOC 2012 test set. For fair comparison, only the methods pre-trained using MS-COCO are displayed. PSPNet uses 82% more parameters.

Table content: <table><tr><td>Methods</td><td>mIoU</td></tr><tr><td colspan="2">LRR (VGG16) (Ghiasi &amp; Fowlkes, 2016) 69.7</td></tr><tr><td>DeepLabv2+CRF (Chen et al., 2016a)</td><td>70.4</td></tr><tr><td>Deep Layer Cascade  *</td><td/></tr></table>

Performance comparison on Cityscapes test set. All methods were trained only using the "fine" set. All nets utilize ResNet-101 as a base network, except if specified or marked with * .6.2.2 RESULTS ON TEST SETPASCAL VOC 2012: Before submitting test set output to an evaluation server, the above model was further fine-tuned on the "trainval" set with batch-norm parameters frozen and at 10× smaller initial learning rate. Table6compares the test set results against other state-of-the-art methods. PSPNet performs slightly better than SUNet, but at the cost of 29.1M more parameters while training at an output stride = 8 with 2× runtime. Specifically, the segmentation model of SUNet uses 35.4M parameters while PSPNet requires 64.5M -an increase of 82.2%. The qualitative results can be found in Appendix C.

Table content: <table><tr><td>77.6</td></tr></table>

Peer review (iclr2019):
 The paper proposes a network architecture for semantic image segmentation. The architecture is based on composing a stack of basic U-Net architectures. After an initial discriminative training of the first portion of the network, the training can be completed end-to-end for the entire network. The second portion of the network entails a dilation. The evaluation of the approach seems to be pretty extensive. The motivation is to build a network that does not heavily rely on pre-trained discriminative networks such as VGG or ResNet, but to build one that is fully targeting the segmentation problem. This approach has led to an architecture that reduces the number of parameters quite significantly while improving results. The paper is well organized and sufficiently clear. The results section is pretty rich. The technical novelty however is limited. The paper leverages many ideas that have been lingering around, including the most important idea, which is to stack U-Nets. The results do show an improvement (for segmentation) with a clear size decrease of the architecture, but this might not be enough. I would have appreciated a table showing the training times of the proposed approach in comparison with the previous works.