Local Image-to-Image Translation via Pixel-wise Highway Adaptive Instance Normalization

1 INTRODUCTION . Unpaired image-to-image translation ( or in short , image translation ) based on generative adversarial networks ( Goodfellow et al. , 2014 ) aims to transform an input image from one domain to another , without using paired data between different domains ( Zhu et al. , 2017a ; Liu et al. , 2017 ; Kim et al. , 2017 ; Choi et al. , 2018 ; Liu et al. , 2017 ) . An unpaired setting , however , is inherently multimodal , denoting a single input image can be mapped to multiple different outputs within a target domain . For example , when translating the hair color of a given image into a blonde , the detailed region ( e.g. , upper vs. lower , and partial vs. entire ) and color ( e.g. , golden , platinum , and silver ) may vary . Previous studies have achieved such multimodal outputs by adding a random noise sampled from a pre-defined prior distribution ( Zhu et al. , 2017b ) or taking a user-selected exemplar image as additional input , which contains the detailed information of an intended target style ( Chang et al. , 2018 ) . Recent studies ( Lin et al. , 2018 ; Ma et al. , 2018 ) including MUNIT ( Huang et al. , 2018 ) and DRIT ( Lee et al. , 2018 ) combine those two approaches , showing the state-of-the-art performance by separating ( i.e. , disentangling ) content and style information of a given image through two different encoder networks . However , existing exemplar-based image translation method has several limitations as follows . First , the style information is typically extracted and encoded from the entire region of a given exemplar , thus being potentially noisy due to those regions involved with respect to the target attribute to transfer . Suppose we translate the hair color of an image using an exemplar image . Since the hair color information is available only in the hair region of an image , the style information extracted from the entire region of the exemplar may contain the irrelevant information ( e.g. , color of the wall and edge pattern of the floor ) , which should not be reflected in the intended image translation . On the other hand , the extracted style is then applied to the entire region of the target image , even though particular regions should be kept as it is . Due to this limitation , some of the previous approaches ( Huang et al. , 2018 ; Lee et al. , 2018 ) often distort irrelevant regions of an input image such as the background . Furthermore , when multiple attributes are involved in an exemplar image , one has no choice but to impose all of them when translating a given image . For example , in a person ’ s facial image translation , if the exemplar image has two attributes , ( 1 ) a smiling expression and ( 2 ) a blonde hair , then both attributes have to be transferred with no other options . To tackle these issues , we propose a novel , LOcal Mask-based Image Translation approach , called LOMIT , which jointly generates a local , pixel-wise soft binary mask of an exemplar ( i.e. , the source region from which to extract out the style information ) and that of an input image to translate ( i.e. , the target region to which to apply the extracted style ) . This approach has something in common with those recent approaches that have attempted to leverage an attention mask in image translation ( Pumarola et al. , 2018 ; Chen et al. , 2018 ; Yang et al. , 2018 ; Ma et al. , 2018 ; Mejjati et al. , 2018 ) . In most approaches , the attention mask ( extracted from an input image ) plays a role of determining the target region to apply a translation . Note that we expand those approaches by jointly extracting two masks , from an input and an exemplar image , respectively , acting as the attention mask of an input and a relevant region ( foreground ) extractor of an exemplar . The main novelty of LOMIT is that to jointly obtain the local masks of two images , we utilize the cosegmentation networks ( Rother et al. , 2006 ) , which aim ( 1 ) to extract the targeted style information without noise introduced from irrelevant regions and ( 2 ) to translate only the necessary region of a target image while minimizing its distortion . While co-segmentation approaches were originally proposed to capture the regions of a common object existing in multiple input images ( Rother et al. , 2006 ; Li et al. , 2018 ) , we adopt and train co-segmentation networks for our own purpose . Once obtained local masks , LOMIT extends a recently proposed technique for image translation , called adaptive instance normalization , using highway networks ( Srivastava et al. , 2015 ) , which computes the weighted average of the input and the translated pixel values using the abovementioned pixel-wise local mask values as different linear combination weights per pixel location . LOMIT has an additional advantage of being able to manipulate the computed masks to selectively transfer an intended style , e.g. , choosing either a hair region ( to transfer the hair color ) or a facial region ( to transfer the facial expression ) . The effectiveness of LOMIT is evaluated on two facial datasets , via a user study and other quantitative measures such as the inception score and the classification accuracy . 2 BASIC SETUP . We define “ content ” as common features ( an underlying structure ) across all domains ( e.g. , the pose of a face , the location and the shape of eyes , a nose , a mouth , and hair ) , and “ style ” as a representation of the structure ( e.g. , background color , facial expression , skin tone , and hair color ) . As shown in Fig . 1 , we assume that an image x can be represented as x = c⊕ s , where c is a content code in a content space , and s is a style code in a style space . The operator⊕ combines and converts the content code c and the style code s into a complete image x . By considering the local mask indicating the relevant region ( or simply , the foreground ) to extract the style from or to apply it to , we further assume that s is decomposed into s = sf ⊕ sb , where sf is the style code extracted from the foreground region and sb is that from the background region . Separating an integrated style space S into a foreground style space Sf and a background style space Sb play a role of disentangling style feature representation1 . The pixel-wise soft binary mask m of an image x is represented as a matrix with the same spatial resolution of x . Each entry of m lies between 0 and 1 , which indicates the degree of the corresponding pixel belonging to the foreground . Then , the local foreground/background regions xf /xb of x is obtained as xf = m x , xb = ( 1−m ) x , ( 1 ) where is an element-wise multiplication . Finally , our assumption is extended to x = c⊕ sf ⊕ sb , where c , sf , and sb are obtained by the content encoder Ec : , the foreground style encoder Efs , and the background style encoder Ebs , respectively , which are all shared across multiple domains in LOMIT , i.e. , { cx , sfx , sbx } = { Ec ( x ) , Efs ( xf ) , Ebs ( xb ) } cx ∈ C , sfx ∈ Sf , sbx ∈ Sb ( 2 ) It is critical in LOMIT to properly learn to generate the local mask involved in image translation . To this end , we propose to combine the mask generation networks with our novel highway adaptive instance normalization , as will be described in Section 3.2 . 1 To verify the representations are properly disentangled , we refer the readers to the 2D embedding visualization of each space ( C , Sf , Sb ) in Fig . 8 in in the appendix . 3 LOCAL IMAGE TRANSLATION MODEL . We first denote x1 ∈ X1 and x2 ∈ X2 as images from domains X1 and X2 , respectively . As shown in Fig . 2 , LOMIT converts a given image x1 to X2 and vice versa , i.e. , x1→2 = G ( h ( Ec ( x1 ) , E f s ( x f 2 ) , E b s ( x b 1 ) ) ) , and x2→1 =G ( h ( Ec ( x2 ) , E f s ( x f 1 ) , E b s ( x b 2 ) ) ) , where G is decoder networks and h is our proposed , local mask-based highway adaptive instance normalization ( or in short , HAdaIN ) , as will be described in detail in Section 3.2 . For a brevity purpose , we omit the domain index notation in , say , m = { m1 , m2 } and x = { x1 , x2 } , unless needed for clarification . 3.1 LOCAL MASK EXTRACTION . LOMIT utilizes the local mask m to separate the image x into the foreground and the background regions , xf and xb . That is , we jointly extract the local masks of the input and the exemplar images , as those effectively involved in image translation , via co-segmentation networks . For example , given the input image and the exemplar , if LOMIT identifies the the hair color difference of a facial image , e.g. , blonde vs. black , then , our local masks should be obtained as the hair regions from two images . As shown in Fig . 3 , given two images x1 and x2 , the co-segmentation networks first encode the content of each as c1 and c2 via the content encoder Ec . Next , in the case of computing the segmentation of x2 , after average-pooling c1 globally , we forward it to an MLP to obtain the channel-wise soft binary mask cattn1 , which is then multiplied with c2 in a channel-wise manner , i.e. , c attn 1 c2 , where cattn1 = σ ( MLP ( c1 ) ) . This step works as transferring the objects information from x1 to x2 . Finally , we forward-propagate this output into the attention network A to obtain the local mask m2 of x2 , i.e. , m2 = A ( cattn1 c2 ) . The same process applies to the opposite case in a similar manner , resulting in m1 = A ( cattn2 c1 ) . Note that our co-segmentation networks are trained in an end-to-end manner with no direct supervision . 3.2 HIGHWAY ADAPTIVE INSTANCE NORMALIZATION . Adaptive instance normalization is an effective style transfer technique ( Huang & Belongie , 2017 ) . Generally , it matches the channel-wise statistics , e.g. , the mean and the variance , of the activation map of an input image with those of a style image . In the context of image translation , MUNIT ( Huang et al. , 2018 ) extends AdaIN in a way that the target mean and the variance are computed as the outputs of the trainable functions β and γ of a given style code , i.e. , AdaINx1→x2 ( c1 , s2 ) = γ ( s2 ) ( c1 − µ ( c1 ) σ ( c1 ) ) + β ( s2 ) , ( 3 ) As we pointed out earlier , such a transformation is applied globally over the entire region of an image , which may unnecessarily distort irrelevant regions . Hence , we formulate our local maskbased highway AdaIN ( HAdaIN ) as HAdaINx1→x2 ( m1 , c1 , s f 2 , s b 1 ) = m1 AdaINx1→x2 ( c1 , s f 2 ) + ( 1−m1 ) AdaINx1→x1 ( c1 , sb1 ) , ( 4 ) where each of β and γ in Eq . ( 3 ) is defined as a multi-layer perceptron ( MLP ) , i.e. , [ β ( sf ) ; γ ( sf ) ] = MLPf ( sf ) and [ β ( sb ) ; γ ( sb ) ] = MLPb ( sb ) . Note that we use different MLPs for the foreground and the background style code inputs . The first term of Eq . ( 4 ) corresponds to the local region of an input image translated by the foreground style , while the second corresponds to the complementary region where the original style of the input is kept as it is .

Summary: The paper tries to address an issue existing in current image-to-image translation at the point that different regions of the image should be treated differently. In other word, background should not be transferred while only foreground of interest should be transferred. The paper propose to use co-segmentation to find the common areas to for image translation. It reports the proposed method works through experiments.