Generalized rectifier wavelet covariance models for texture synthesis

1 INTRODUCTION . Textures ares spatially homogeneous images , consisting of similar patterns forming a coherent ensemble . In texture modeling , one of the standard approaches to synthesize textures relies on defining a maximum entropy model ( Jaynes , 1957 ) using a single observed image ( Raad et al. , 2018 ) . It consists of computing a set of prescribed statistics from the observed texture image , and then generating synthetic textures producing the same statistics as the observation . If the statistics correctly describe the structures present in the observation , then any new image with the same statistics should appear similar to the observation . A major challenge of such methods resides in finding a suitable set of statistics , that can generate both high-quality and diverse synthetic samples . This problem is fundamental as it is at the heart of many texture related problems . For example , in patch re-arrangement methods for texture modeling , these statistics are used to compute high-level similarities of image patches ( Li & Wand , 2016 ; Raad et al. , 2018 ) . Such statistics are also used in texture interpolation for probing visual perception ( Vacher et al. , 2020 ) , style transfer and image inpainting ( Gatys et al. , 2016 ; Laube et al. , 2018 ) . A key question along this line of research is to find what it takes to generate natural textures . This problem was originally posed in Julesz ( 1962 ) , in which the author looks for a statistical characterization of textures . In the classical work of Portilla & Simoncelli ( 2000 ) , the authors presented a model whose statistics are built on the wavelet transform of an input texture image . These statistics were carefully chosen , by showing that each of them captured a specific aspect of the structure of the image . This model produces satisfying results for a wide range of textures , but fails to reproduce complex geometric structures present in some natural texture images . Figure 1 presents a typical example composed of radishes , and synthetic images from three state-of-the-art models developed over the last few decades . To address this problem , the work of Gatys et al . ( 2015 ) proposes to use statistics built on the correlations between the feature maps of a deep CNN , pre-trained on the ImageNet classification problem ( Deng et al. , 2009 ; Simonyan & Zisserman , 2014 ) . While this model produces visually appealing images , these statistics are hard to interpret . The work of Ustyuzhaninov et al . ( 2017 ) made a significant simplification of such statistics , by using the feature maps of a one-layer rectifier CNN with random filters ( without learning ) . A crucial aspect of this simplification relies on using multi-scale filters , which are naturally connected to the wavelet transform . In this paper , we propose a wavelet-based model , more interpretable than CNN-based models ( with learned or random filters ) , to synthesize textures with complex geometric structures . It allows to bridge the gap between the classical work of Portilla & Simoncelli ( 2000 ) , and state-of-the-art models . This model is built on the recent development of the phase harmonics for image representations and non-Gaussian stationary process modeling ( Mallat et al. , 2020 ; Zhang & Mallat , 2021 ) . The phase harmonics are non-linear transformations that adjust the phase of a complex number . In Portilla & Simoncelli ( 2000 ) ; Zhang & Mallat ( 2021 ) , the authors illustrate that the phase dependencies between wavelet coefficients across scales contain important information about the geometric structures in textures and turbulent flows , and that they can be captured by applying the phase harmonics to complex wavelet coefficients . Remarkably , Mallat et al . ( 2020 ) show that the phase harmonics admit a dual representation , closely related to the rectifier non-linearity in CNNs . Our main contributions are : • We develop a family of texture models based on the wavelet transform and a generalized rectifier non-linearity , that significantly improves the visual quality of the classical waveletbased model of Portilla & Simoncelli ( 2000 ) on a wide range of textures . It relies on introducing spatial shift statistics across scales to capture geometric structures in textures . • By changing the number of statistics in our models , we show explicitly the trade-off on the quality and diversity of the synthesis . When there are too many statistics , our model tends to memorize image patches . We further investigate such memorization effects on non-stationary images and find that it sometimes relies on what statistics are chosen , rather than on how many . • Through the modeling of geometric structures in gray-scale textures , our model indicates the possibility of reducing significantly the number of statistics in the works of Gatys et al . ( 2015 ) and Ustyuzhaninov et al . ( 2017 ) , to achieve a similar visual quality . The rest of the paper is organized as follows : Section 2 reviews the framework of microcanonical maximum-entropy models , build upon a general family of covariance statistics . We then present our model for both gray-scale and color textures in Section 3 . Section 4 shows synthesis results of our model , compared with state-of-the-art models . Finally , in Section 5 , we discuss possible improvements of our model . Notations Throughout the paper , N denotes a positive integer . A gray-scale image x is an element of RN×N , i.e . x = x ( u ) , u ∈ ΩN , with ΩN : = { 0 , · · · , N−1 } 2 . A color image x = { xc } c=1,2,3 is an element of R3×N×N , or equivalently , each xc ∈ RN×N . We shall denote x̄ the observed texture ( observation ) , which is assumed to be a realisation a random vector X . For any complex number z ∈ C , z∗ is the complex conjugate of z , Real ( z ) its real part , |z| its modulus , and ϕ ( z ) its phase . 2 MICROCANONICAL COVARIANCE MODELS . We briefly review the standard framework of micro-canonical maximum-entropy models for textures . To reliably estimate the statistics in these models , we assume that a texture is a realization of a stationary and ergodic process X ( restricted to ΩN ) . We then review a special family of statistics that are used in the state-of-the-art texture models ( mentioned in Figure 1 ) , based on covariance statistics of an image representation . 1As the model from Ustyuzhaninov et al . ( 2017 ) uses on random filters , we shall use the abbreviation RF . 2.1 FRAMEWORK . Given a observation texture x̄ , we aim at generating new texture images , similar but different from x̄ . To that end , a classical method is to define a set of statistics Cx̄ , computed on the observation , and try to sample from the microcanonical set { x : ‖Cx− Cx̄‖ ≤ } , where ‖ · ‖ denotes the L2 norm . Under the stationary and ergodic assumption of X , one can construct Cx as a statistical estimator of E ( CX ) , from a complex-valued representation Rx.2 The set of covariance statistics Cx of a model can then be constructed by computing an averaging over the spatial variable u , i.e . Cx ( γ , γ′ , τ ) : = 1 |ΩN | ∑ u∈ΩN Rx ( γ , u ) Rx ( γ′ , u− τ ) ∗ , ( 1 ) for ( γ , γ′ , τ ) ∈ Υ ⊆ Γ × Γ × ΩN . The statistics Cx ( γ , γ′ , τ ) can be interpreted as estimating the covariance ( resp . correlations ) betweenRX ( γ , u ) andRX ( γ′ , u−τ ) for zero-meanRX ( resp . nonzero meanRX ) . The ergodicity assumption ensures that whenN is large enough , the approximation Cx̄ ' E ( CX ) over Υ should hold with high probability . Under these conditions , it makes sense to sample the microcanonical set in order to generate new texture samples . This framework encompasses a wide range of state-of-the-art texture models3 . In particular , the PS model developed in Portilla & Simoncelli ( 2000 ) takes inspiration from the human early visual system to define a multi-scale representation based on the wavelet transform of the image ( Heeger & Bergen , 1995 ) . We next review a family of covariance model which generalizes the statistics in the PS model . We write CM the statistics for a specific model M that uses the representation RM . 2.2 WAVELET PHASE HARMONIC COVARIANCE MODELS . We review a family of microcanonical covariance models defined by a representation built upon the wavelet transform and phase harmonics . It defines a class of covariance statistics that capture dependencies between wavelet coefficients across scales . 2.2.1 WAVELET TRANSFORM . The wavelet transform is a powerful tool in image processing to analyze signal structures , by defining a sparse representation ( Mallat , 2001 ) . For texture modeling , we consider oriented wavelets to model geometric structures in images at multiple scales . They include the Morlet wavelets and steerable wavelets , proposed in Goupillaud et al . ( 1984 ) ; Simoncelli & Freeman ( 1995 ) ; Unser & Chenouard ( 2013 ) . In particular , the Simoncelli steerable wavelets have been used to model a diverse variety of textures in Portilla & Simoncelli ( 2000 ) . Oriented wavelets are defined by the dilation and rotation of a complex function ψ : R2 7→ C on a plane . Let rθ denote the rotation by angle θ in R2 . They are derived from ψ with dilations by factors 2j , for j ∈ { 0 , 1 , · · · , J − 1 } , and rotations rθ over angles θ = ` π/L for 0 ≤ ` < L , where L is the number of angles in [ 0 , π ) . The wavelet at scale j and angle θ is defined by ψj , θ ( u ) = 2 −2jψ ( 2−jrθu ) , u ∈ R2 Scales equal or larger than J are carried by a low-pass filter φJ . The wavelet transform of an image x ∈ RN×N is a family of functions obtained by the convolution of x with discrete wavelets.4 Let Λ : = { 0 , · · · , J − 1 } × πL { 0 , · · · , L − 1 } be an index set . The wavelet coefficients are x ? ψj , θ ( u ) = ∑ v∈ΩN x ( u− v ) ψj , θ ( v ) , u ∈ ΩN , ( j , θ ) ∈ Λ . ( 2 ) The low-pass coefficients x ? φJ are defined similarly . 2The complex-valued representation Rx ( γ , u ) ∈ C is a function of ( γ , u ) in an index set Γ× ΩN . 3Such as Portilla & Simoncelli ( 2000 ) ; Gatys et al . ( 2015 ) ; Ustyuzhaninov et al . ( 2017 ) ; Zhang & Mallat ( 2021 ) 4The continuous wavelets are discretized with periodic boundary conditions on the spatial grid ΩN . 2.2.2 WAVELET PHASE HARMONICS AND THE PS MODEL . To model natural textures , it has been shown ( Portilla & Simoncelli , 2000 ; Zhang & Mallat , 2021 ) that it is crucial to capture statistical dependencies between wavelet coefficients across scales . This can be achieved by using a wavelet phase harmonic representation , which is defined by the composition of a linear wavelet transform of x , and a non-linear phase harmonic transform . In Mallat et al . ( 2020 ) , the authors introduce the phase harmonics to adjust the phase of a complex number z ∈ C. More precisely , the phase harmonics { [ z ] k } k∈Z of a complex number z ∈ C are defined by multiplying its phase ϕ ( z ) of z by integers k , while keeping the modulus constant , i.e . ∀ k ∈ Z , [ z ] k : = |z|eikϕ ( z ) . The wavelet phase harmonic representation ( WPH ) is then defined by RWPHx ( γ , u ) = [ x ? ψj , θ ( u ) ] k − µγ , γ = ( j , θ , k ) ∈ Γ = Λ× Z , ( 3 ) where µγ is defined as the spatial average of RWPHx̄ ( γ , · ) . It is shown in Zhang & Mallat ( 2021 ) that the PS model can be regarded as a low-order wavelet phase harmonics covariance model , which considers only a restricted number of pairs ( k , k′ ) ( see Appendix A for more details ) . In the next section , we shall use a dual representation of the phase harmonic operator to define a covariance model to capture high-order phase harmonics .

Summary: This paper presents a new image representation model based on wavelets and non-linear rectifiers that allows to synthesize complex geometric textures with a better visual quality than previous wavelet-based models.  The main interest of the paper is the usage of the mathematical results from Mallat et al 2020 and Zhang and Mallat 2021 on wavelet phase harmonics to image texture synthesis.   They also show that the PS model (Portilla & Simoncelli 2000) is a particular case of the wavelet phase harmonics-based (WPH) model. Both methods underline the importance of statistical dependencies between wavelet coefficients across scales. The specific choice of the parameters available for WPH is important to balance between reproducing good structural information and memorizing patterns from the observation.