Generalized rectifier wavelet covariance models for texture synthesis
Authors: Antoine Brochard
Sixin Zhang
Stéphane Mallat
Venue: ICLR 2022 Poster (2022)
Keywords: texture synthesis
generative models
wavelets

Abstract: State-of-the-art maximum entropy models for texture synthesis are built from statistics relying on image representations defined by convolutional neural networks (CNN). Such representations capture rich structures in texture images, outperforming wavelet-based representations in this regard. However, conversely to neural networks, wavelets offer meaningful representations, as they are known to detect structures at multiple scales (e.g. edges) in images. In this work, we propose a family of statistics built upon non-linear wavelet based representations, that can be viewed as a particular instance of a one-layer CNN, using a generalized rectifier non-linearity. These statistics significantly improve the visual quality of previous classical wavelet-based models, and allow one to produce syntheses of similar quality to state-of-the-art models, on both gray-scale and color textures. We further provide insights on memorization effects in these models. 


Published as a conference paper at ICLR 2022
GENERALIZED RECTIFIER WAVELET COVARIANCE MODELS FOR TEXTURE D SYNTHESIS
Antoine Brochard
Sixin Zhang
ENS, PSL University, Paris, France antoine.. .brochardeens fr
Université de Toulouse, INP, IRIT, Toulouse, France s i xin. x zhange irit...
Stéphane Mallat
Collège de France, Paris, France Flatiron Institute. New York, USA
ABSTRACT
State-of-the-art maximum entropy models for texture synthesis are built from statistics relying image representations defined by convolutional neural net- works (CNN). Such representations capture rich structures in texture images, out- performing wavelet-based representations in this regard. However, conversely neural networks, wavelets offer meaningful representations, as they are known detect structures at multiple scales (c.g. edges) in images. In this work, we propose a family of statistics built upon non-linear wavelet based representations, that can be viewed a particular instance of a one-layer CNN, using generalized rectifier non non-linearity. These statistics significantly improve the visual quality of previous classical ravelet-based models, and allow one to produce syntheses of similar quality to state-of-the-arr models, on both gray-scale and color textures. We further provide insights on memorization effects in these models.
INTRODUCTION
Textures ares spatially homogeneous images, consisting of similar patterns forming a coherent en- semble. In texture modeling, one of the standard approaches to synthesize textures relies on defining a maximum entropy model (Jaynes 1957) using a single observed image (Raad et al., 2018). It con- sists of computing set of prescribed statistics from the observed texture image, and then generating synthetic textures producing the same statistics as the observation. If the statisties correctly describe the structures present in the observation, then any new image with the same statistics should appear major challenge of such methods resides in finding a suitable set of
similar to the observation. statistics, that can generate both high-quality and diverse synthetic samples. This problem is funda- mental as is at the heart of many texture related problems. For example, in patch re- re-arrangement methods for texture modeling, these statistics used compute high-level similarities of image patches (Li & Wand 2016) Raad et al.] 2018). Such models are also used for visual perception (Freeman & Simoncelli 2011 Wallis al. 2019 Vacher al. (2020), style transfer (Gatys al., 2016 Deza et al. 2019) and image inpainting (Laube et al. 2018)).
A key question along this line of research is find what it takes generale natural textures. This problem was originally posed in Julesz (1962), in which the author looks for a statistical character- ization of textures. In the classical work of Portilla & Simoncelli (2000) (noted PS in this work), the authors presented model whose statistics are built on the wavelet transform of an input texture image. These statisties were carefully chosen, by showing that each of them captured a specific aspect of the structure of the image. This model produces satisfying results for a wide range of textures, but fails to reproduce complex geometric structures present some natural texture im- ages. Figure [ presents a typical example composed of radishes, and synthetic images from three state-of-tth--an models developed over the last few decades. To address this problem, the work of Gatys et al. (2015) proposes use statistics built the correlations between the feature maps of a deep CNN, pre-trained on the ImageNet classification problem (Deng et al. 2009) (Simonyan &
Zisserman 2014). While this model produces visually appealing images, these statistics hard to

---

Published as a conference paper at ICLR 2022
interpret. The work of Ustyuzhaninov et al. (2017) made significant simplificatin of such statis- tics, by using the feature maps of a one-layer rectifier CNN with random filters (without learning). A crucial aspect of this simplification relies using multi-scale filters, which are naturally connected to the wavelet transform. In this paper, we propose a wavelet-based model, more interpretable than CNN-based models (with learned or random filters), to synthesize textures with complex geometric structures. It allows to bridge the gap between the classical work of Portilla & Simoncelli (2000), and state-of-the-art models.
Observation
PS
VGG
RF
Figure 1: Example of syntheses from three texture models in chronological order. From left right: the observed texture in gray-scale, synthesis from PS PPottilla & Simoncelli 2000), from VGG (Gatys et 2015). and from RF Ustyuzhaninov et al. 2017).
This model is built on the recent development of the phase harmonics for image representations and non-Gaussian stationary process modeling (Mallat et al.) 2020 Zhang Mallat, 2021). The phase harmonics are non- linelin transformations that adjust the phase of complex number. In |Portilla & Simoncelli] (2000): Zhang Mallat (202T). the authors illustrate that the phase dependencies between wavelet coefficients across scales contain important information about the geometric struc- tures in textures and turbulent flows, and that they can be captured by applying the phase harmonics to complex wavelet coefficients. Remarkably, Mallat et al. (2020) show that the phase harmonics admit a dual representation, closely related to the rectifier non-linearity in CNNs. Our main contri- butions are:
We develop a family of texture models based on the wavelet transform and a generalized rectifier non-linearity, that significantly improves the visual quality of the classical wavelet- based model of PPortilla & Simoncelli (2000) on wide range of textures. It relies on introducing spatial shift statistics across scales to capture geometric structures in textures.
By changing the number of statistics in our models, we show explicitly the trade-off on the quality and diversity of the synthesis. When there are too many statistics, our model tends to memorize image patches. We further investigate such memorization effects on non-stationary images and find that it sometimes relies what statistics are chosen, rather than on how many.
Through the modeling of geometric structures in gray-scale textures, our model indicates the possibility of reducing significantly the number of statistics in the works of Gatys al.) (2015) and Ustyuzhaninov et al.) (0017), to achieve a similar visual quality.
The rest of the paper is organized as follows: Section reviews the framework of microcanonical maximum-entree entropy models, build upon a general family of covariance statistics. We then present our model for both gray-scale and color textures in Section 3 Section 4 shows synthesis results of our model, compared with state-of-the-art models. Finally, Section improvements of our mode
we discuss possible
Notations Throughout the paper, N denotes a positive integer. A gray-scale image x is an element of N i.e. I(u), Uy, with Ωx {0,- N --11 1)2. A color image f2r) 1,=1, is an element of Nx (observation). which is assumed to be a realisation a random vector X. For any complex number E C, 2" is the complex conjugate of 2, Real(z) its real part, |a| its modulus, and (o(2) its phase.
equivalently, each x N We shall denote the observed texture
1 'As the model from Ustyuzhaninov et al. 2017) uses on random filters, we shall use the abbreviation RF,
2 All calculations can he reproduced by Python software available at https: //gi thub. com/ abrochar/wave
syrthesis
2

---

Published as a conference paper at ICLR 2022
2 MICROCANONICAL COVARIANCE MODELS
We briefly review the standard framework of micro-canonical maximum-entropp models for tex- tures. To reliably estimate the statistics in these models, we assume that texture is a realization of a stationary and ergodic process X (restricted to O2x). We then review special family of statistics that are used in the state-of-the-art texture models (mentioned in Figure [], based on covariance statistics of an image representation.
2.1
FRAMEWORK
Given observation texture I. aim at generating new texture images, similar but different from J. To that end, a classical method is to define a set statisties Cr, computed on the observation, and try sample from the microcanonical set
tx |CCx Cãl S se),
where ||
denotes the 1 norm.
Under the stationary and ergodic assumption of X, one can construct Ca as statistical estimator of E(CX), from a complex-valued representation Rx R The set of covariance statistics Ca of model can then constructed by computing an averaging over the spatial variable u, i.e.
Co(())'sT))=
Σ WEΩN
R)
12
for (T) E T TTTxXX ΩN. The statistics Cz(y, 7', T) can be interpreted as estimating the covariance (resp. correlations) between RX u) and RX (7', -- for zero-mean R.X (resp. non- zero mean R.X). The ergodicity assumption ensures that when N is large enough, the approximation Ca ~ (CX) over Y should hold with high probability. Under these conditions, it makes sense to sample the microcanonical set in order to generate new texture samples.
This framework encompasses a wide range of state-of-the-art texture models") In particular, the PS model takes inspiration from the human early visual system to define multi-scale representation based on the wavelet transform of the image (Heeger & Bergen 1995). We next review a family of covariance model which generalizes the statistics in the PS model. We write C the statisties for specific model M that uses the representation RM
2.2 WAVELET PHASE HARMONIC COVARIANCE MODELS
We review a family of microcanonical covariance models defined by a representation built upon the wavelet transform and phase harmonics. It defines class of covariance statistics that capture dependencies between wavelet coefficients across scales.
2.2.1 WAVELET TRANSFORM
The wavelet transform is a powerful tool in image processing to analyze signal structures, by defin- ing sparse representation (Mallat 2001). For texture modeling. we consider oriented wavelets to model geometric structures in images at multiple scales. They include the Morlet wavelets and steerable wavelets, proposed in Goupillaud et al. (1984); [Simoncelli & Freeman) (1995); [nser & Chenouard 2013)) In particular, the Simoncelli stecrable wavelets have been used model diverse variety of textures in Portilla & Simoncelli (2000).
Oriented wavelets are defined by the dilation and rotation of complex function de IR - C plane. Let ra denote the rotation by angle 0 in IR2 They are derived from y with dilations by factors 2', for j E {0, 1,... ................................ JJ- 1). and rotations number angles in [0, RT. The wavelet at scale and angle 0 is defined by
angles 0 L for < l < L, where L is the
(5 2j 1/222 Prau),
WEE R
"The complex-valun representation Ra(y, x) E C is function of (r. u) in an index set I x ΩN c.g. Portilla & Simoncelli 20000; Gatys 2015 Ustyuzhaninov al. 2017 Zhang & Mallat 2021
3

---

Published as a conference paper at ICLR 2022
Scales equal larger than J are carried by low-pass filter ØJ.
The wavelet transform of an image I E IR N of z with discrete wavelets Let A == {0, wavelet coefficients are
N is a family of functions obtained by the convolution
1} x 100,... ,L 11 be an index set. The
* Vj,olu)=
Σ "ESty
z(u - v) ,,))) u E ΩN (j.0)
(2)
The low-pass coefficients I * ΦJ are defined similarly.
2.2.2 WAVELET PHASE HARMONICS AND THE PS MODEL
To model natural textures, it has been shown (Portilla & Simoncelli 2000 Zhang & Mallat 2021) that it is crucial to capture statistical dependencies between wavelet coefficients across scales. This can be achieved by using wavelet phase harmonic representation, which defined by the compo- sition of linear wavelet transform of x, and non-linear phase harmonic transform.
In Mallat al. (2020), the authors introduce the phase harmonics adjust the phase of complex number 2 E C. More precisely, the phase harmonics {[>]* /kez of a complex number E C are defined by multiplying phase (z2) of z by integers while keeping the modulus constant, i.c. k Z, [a]* := [2)c The wavelet phase harmonic representation (WPH) is then defined by
RWPH x(,, u) = [x * v3,)(1))** - Mag 7 - k) Ax=
(3)
where Byy is defined as the spatial average of [ v5,00*
It is shown in Zhang & Mallat (2021) that the PS model can be regarded a low-order wavelet phase harmonics covariance model, which considers only a restricted number of pairs (k, k') (see Appendix A for more details). In the next section, we shall use a dual representation of the phase harmonic operator to define covariance model to capture high-order phase harmonics.
3 GENERALIZED RECTIFIER WAVELET COVARIANCE MODEL
In the previous section, we presented a class models, built from the wavelet phase harmonic repre- sentation. A dual representation of the phase harmonic operator [-]k can be defined via a generalized rectified linear unit, that we review in Section 3.1 We then discuss in Section 3.2 how to define an appropriate index set of I' for gray-scale textures. Section 3.3 extends the model to color textures.
3.1
FROM PHASE HARMONICS TO THE GENERALIZED RECTIFIER
The generalized rectified linear unit of a complex number 2, with phase shifted by a E [0, 2n). is defined by
(=(2) = p(Real(e" =)),
(4)
where is a rectified linear unit, i.e. for any L E TR, p(() == max(0, 1). Mallat et al. 2020
it shown that applying Fourier transform on Pa (z) along the variable results the phase harmonics of z (up to some normalization constant). This suggests an alternative model, defined by coefficients of the form
RALPHAI(T: u) = Pa vyye(uu)) - Y (j,0,a),
(5)
for y E T = A x [0, 2m], and μ.y is defined as the spatial average of Pa (ō * (z)) over E Ω2
Relation with high-order phase harmonics Based on the duality between the phase harmonics k E & and the phase shift variable a E [0, 2T., we now present the relation between CALPHA and the high-order phase harmonics in CPP first proved in Mallat et al. (202)).
Proposition 1 There exists a complex-valued sequence {cr Cekk such that for all (j,0,a) E T, (i'0', o') r, and all E ΩN. CALPHA ae 0, a), (i', r)),
Σ
CE WPH (5' 00, K'), (e
(A.
'The continuous wavelets are discretized with periodic boundary conditions the spatial grid S2y
4

---

Published as a conference paper at ICLR 2022
The proof is given in Appendix B We remark that the sequence by the rectifier non-linearity p. and they are non-zero if is even (Mallat et al. 2020). This result shows that for a suitable choice of (a, a'), the covariance statisties CAL can implicitly capture CWPH& x with a wide range of k and k'.
{ck hsz is uniquely determined
Relation with second order statisties Using a simple decomposition of wavelet coefficients into their positive, negative, real and imaginary parts, we can further show that the covariance statistics C'ALHHA z capture the classical second order statistics of wavelet coefficients, also used in the PS model (with phase harmonic coefficients k =h'=
k'= ===11
Proposition 2 Let 1
{0, f. 3x (Wa,or (a. such that for all (i.0) A, and all t E &NN,
2
There exists a finite complex-valued sequence
Σ
aa CALPHA x(1, a),, (f', 0''0''.... d'),))= = E uERN
(o * (0x00,,())) (x by +))" (6)
(a.a")E12
The proof is given in Appendix This shows that using only four uniformly chosen between [0, 2T] is sufficient to capture second order statistics. Because the wavelet transform an invertible linear operator (on its range space), computing the r.h.s of eq. (6) for all (j, 0, r), well as the low-pass coefficients carried out by $. is equivalent to computing the correlation matrix of a.
Relation with the RF model Setting aside the subtraction by the spatial mean (4y the RF model can be viewed as a particular case of models defined by eq. (5). Indeed, the statistics of the RF model take the form of eq. (T), with
RRF zzf = p(a (5
where { being a family of multi-scal scale random filters. By writing Pa * 03,0(u -)) = p(x Real( ub oe (w)), with 30o denoting the translation of $5.0 by r, we that the models are similar, the difference being that our models use wavelet-based filters instead of random ones.
3.2 DEFINING AN APPROPRIAEE! Y
The choice of the covariance set T is of central importance in the definition of the model. Intuitively, a too small set of indices will induce a model that could miss important structural information about the texture that we want to synthesize. Conversely, if Y contains too many indices, the syntheses can have good visual quality, but the statistics of the model may have a large variance, leading to the memorization of some patterns of the observation. There is trade-off between these two aspects: one must capture enough information to get syntheses of good visual quality, but not much, so as not to reproduce parts of the original image. To illustrate this point, we shall study the model ALPHA defined with three different sets T: A smaller model ALPHAs with a limited amount of elements in Y, an intermediate model ALPHA, and larger model ALPHA
To precisely define these models, let us note 3 == -. -0. 0 - 2n
(.........--- .L-.,,111,.... and Aa = 9E BE (0... 22-1)-
A (0, A-13. Let us also define the set I = (c)s(())) sin from which the spatial shift shall be selected. Table summarizes the conditions that all parameters have satisfy to be contained in these sets. Additionally, these models include large scale informa- tion through the covariance of low-pass filter, i.e. the spatial average of x Φb ())x (-(-- r), for E T. To count the size of Y without redundancies, Appendix A..3 provides an upper bound on the non-redundant statistics in our models. This upper bound is used to count the number of statistics in our models. To keep this number from being too large, instead of taking all shifts in a square box, such as in Portilla & Simoncelli (2000), we choose to select only shifts of dyadic moduli, and with the same orientations as the wavelets.
ALPHAS vs. ALPHA The small model ALPHAs is inspired from the PS model, as it only takes into account of the interactions between nearby scales (i.e. | -j < 1), and the spatial shift cor- relations only considered for (j. () (j', 8'). There are two notable differences in the statistics included in the ALPHAş and ALPHA, models. The first one is the range of scales being corre- lated. It has been shown in Zhang & Mallat (2021) that constraining correlation between wider
5

---

Published as a conference paper at ICLR 2022
Table 1: List of indices in T for different ALPHA models.
Model
Scales
Angles
Phase shift
Spatial shift
Size of T
ALPHAs
Go b5-Ji1s1
(0,0') E @
(a, E A4 A1
TE T=0 otheroise.
(J|6) + J(
ALPHA
(1J')) e 3?
(0, (7,6") @²
(a, E AIxA1
TET
Ja |
ALPHAL
(i.5") E 3
(0, E @²
(a, E A
7 TE
range of scales induces better model for non-Gaussian stationary processes, and better estimation of cosmological parameters from observed data (Allys et al.) 2020). The second difference, which has significant impact on the number of statistics (it increases the model size by factor ~10), is the number of spatial shifts in the correlations. In the ALPHA, model, spatially shifted correlations are computed for all pairs of coefficient (') For both stationary textures and non non-stationary im- ages in gray-scale, shape and contours of salient structures and objects are better reproduced with ALPHA,, as illustrated in Figure 2 More examples are given in Appendix
ALPHA, vs. ALPHA As we observe in Figure 2 the ALPHA, model, containing 4 times less coefficients than the ALPHAL, suffers less from memorization effects, while still capturing most of the geometric information in the images. This small loss of information can be partially explained by the frequency transposition property of the phase harmonics operator (Mallat et al. (020), for compactly supported wavelets in the frequency domain, as detailed in Appendix E In order to avoid this memorization effect, we shall, in the rest the paper, consider only the intermediate model.
3.3 MODELLING COLOR INTERACTIONS
In order to generate color textures, the covariance model ALPHA, defined in Section 3.2 could be directly applied to each G and B color channel independently. However, it would not take into account the color coherence in the structures of the observation.
To capture color interactions in the observation image, shall impose the covariance between the coefficients of eq. (5) for all indices in Y and all color channels. More precisely, let r [x")e=122, be a color image, with the parameter e representing the color channel. The ALPHAc color model is defined by correlations between coefficients of the form:
RALPHACK(T, u) = pal +15,0(w) - Mhy 7= (j,0,,,c)).
(7)
The of indices is defined TALPHAC {(7,7/7) : (U,0,,),,,,,,,))))
Y YALPHAI, (c,c') E (1,2,333
Reduced ALPHAC The model ALPHAc has a large size it computes correlations between all coefficients for all color channels. This size can be significantly reduced by computing spatially shifted coefficients only for the same color channels (to capture their geometries). This reduced model contains three times less coefficients (~113k), without significant reduction of the visual quality of syntheses, as detailed in Appendix F
4 NUMERICAL RESULTS
In this section, we compare our intermediate model to the state-of-art models (PS, RF and VGG) on both scay- and color textures. We first specify the experimental setup. We then present the synthesis results of various examples, and discuss their quality through visual inspection. A quantitative evaluation of the quality of the syntheses, based on the synthesis loss of the VGG model, and proposed in Ustyuzhaninov et al. (2017), is discussed in Appendix G

---

Published as a conference paper at ICLR 2022
Observation
ALPHAs (3.5k)
ALPHA (35k)
ALPHA, (142K)
Figure 2: Examples of syntheses from the ALPHA models defined in Table I Visually similar image patches in textures are highlighted by red circles. The number of statistics is given in brackets next to each model name. From top to bottom: a Julesz counterexample, a stationary texture image, stationary turbulent field and non-stationary image. We see that the model ALPHA, achieves balance between the visual quality and diversity on these examples.
4.1 EXPERIMENTAL SETUP
For experiments, choose gray-scale and color textures with various visual structures In the gray-scale examples, we also include stationary turbulent field (vorticity), which is simulated from Navier-Stokes equations two dimensions (Schneider et al., 2006). These examples all contain complex geometric structures that are hard to model by the classical PS model.
The texture images presented in this work have a size of of ~65k. For all the ALPHA models, we use Morlet wavelets with maximal scale J 5 and number of orientations L. 4. This choice differs from the model, which uses Simoncelli steerable wavelets. In Appendix H we discuss the impact of these wavelets our model. To draw the samples, we follow gradicnt- based sampling algorithms, suitable high-dimension Mallat 2018), to minimize the objective function |C Czl starting from initial sample from a normal distribution. Similarly to Gatys et al. (2015); Ustyuzhaninov et al. (201)). we use the L- BFGS algorithm (Nocedal 1980) for the optimization of the objective. As in the VGG model (Gatys) et 2015),, we further apply histogram matching procedure as post-processing after optimization. The details of the PS, RF and VGG models, well detailed specifications of our models, are given in Appendix More synthesis examples can be found in Appendix
256, giving a number of pixels
(Bruna &
4.2 RESULTS
In Figure 3 we present examples of syntheses from the ALPHA, (or ALPHAc), PS, RF and VGG models, for both gray-scale and color textures, as well as for non-stationary images. We observe that
'The for the presented textures are given in Appendix

---

Published as a conference paper at ICLR 2022
model ALPHA, produces texture syntheses of similar visual quality to the RF and VGG models. also significantly outperforms the PS model in terms of the visual quality, without introducing visible memorization effects. As the model PS uses the statistics closer to ALPHAs compared to , the performance of PS is somehow expected.
Note that for the tiles example (the fifth row) in Figure the VGG model produces less convincing textures, because the long-range correlations present in the image (aligned tiles) are not reproduced. To remedy this issue, has been proposed in Berger & Memisevic (2017) add spatial shifts to the correlations of the network feature maps. These shift statistics are similar to the parameter T in our model. We also observe that, in the case of the sixth row example (flowers), all models fail to reproduce complex structures at object-level. Possible improvements of such models is further discussed in Section 5
For non-stationary images, we find that certain image patches can be more or less memorized by the RF, VGG and ALPHA models, as illustrated in the seventh row example. Understanding such memorization effect of non-stationary images is a subtle topic, as we find that in some binary images (#(u) E {0, 13), only the PS model can reproduce the observation, even though has a much smaller number of statistics (the last row example). We find that this is related to the spatial correlation statistics in PS (non-zero 7). By removing this constraint, 2 is no longer always reproduced The non-stationary nature also appear in some logo near the boundary of some textures (c.g. bottom left in the observation of the first and fourth rows). Although this logo is reproduced by RF, VGG, ALPHA; and ALPHAc, it is very local phenomenon, we do not find visible copies of the textures when there is no logo, and it is likely related to the way one addresses the boundary effect (see more in Appendix A.5
DISCUSSION
In this work, we presented new wavelet-based model for natural textures. This model incorpo- rates a wide range of statistics, by computing covariances between rectified wavelet coefficients, at different scales, phases and positions. We showed that this model is able to capture and reproduce complex geometric structures, present in natural textures or physical processes, producing syntheses of similar quality to state-of-the-art models that use CNN-based representations.
Being defined with wavelet family instead of multi-scale random filters, the proposed model uses less statisties than the RF model. For the gray-scale textures, our model has about 15 times less statistics, it focuses capturing the geometric structures present in images. Although our color model has a slightly larger number of statistics than VGG, the reduced color model, presented in Section 3.3 is three times smaller than ALPHAc. while achieving similar visual quality. It shows the potential to further reduce the size of the color model. In the PS model, a PCA on the color channels is performed (Vacher & Briand 202T). The same idea could also be applied to model.
Furthermore, there are examples where all the models may all fail to produce some geometric struc- tures at object-level, as illustrated in Figure 3 (sixth row). In this situation, we still need to find more informative statistics. One may for example consider to incorporate second layer of wavelet transform as in the wavelet scattering transform (Leonarduzzi et al., 2019 He & Hir 202T). An- other line of research is to introduce other kinds of losses (such as to encourage image smoothness) in order to improve the VGG model (Liu et al. 2016 [Sendik Cohen-Or 2017). These losses are complementary. and could thus also be added to our models. Integrating these models with learning-based approaches is another promising direction (Zhou et al. 2018 Zhu et al. 1997. Xic ct al. 2016 2018).
Finding minimal set of statistics to define a texture model remains important because a large number of statistics can result in a high variance of the estimators, and the associated model may suffer from memorization effects. This is a problem because the aim of the model is to approximate the underlying distribution of the observation, and therefore produce diverse textures. In this regard, the mere visual evaluation of the synthetic textures can fail to take this aspect of the model into account. Defining a quantitative evaluation of quality and diversity, coherent with visual perception, remains an open problem (Ustyuzhaninov et al. 2017 Yu et al. 2019))
Set the parameter A = in the PS model. See Appendix A for more details. This simple example suggests that sometimes is very important choose the right statistics capture specific geometric structures.
8

---

Published as a conference paper at ICLR 2022
Observation
PS (3.2k/17k)
RF (525k)
Ours (35k/320k)
VGG (177k)
Figure 3: Visual comparison between the gray-scale and color models. 'Ours' denotes the ALPHA model for gray-scale images and the ALPHAC model for color images.
Acknowledgments We thank all the reviewers for their insightful feedback. This work was par- tially supported by a grant from the PRAIRIE 31A Institute of the French ANR-19-P31A- 0001 pro-

---

Published as a conference paper at ICLR 2022
gram. Sixin Zhang acknowledges the support provided by 31A Artificial and Natural Intelligence Toulouse Institute, French "Investing for the Future PIA3" program under the Grant agreement ANR-19-PIA--006 and by Toulouse INP ETI and AAP unique CNRS 2022 under the Project LRGMD.
Reproducibility Statement In order to reproduce the experiments presented in this work, the main parameters to define the model can be found in Section 4.1 More details are given in Appendix A Additionally, the code is made publicly available.
Ethics Statement The authors acknowledge that no potential conflicts of interest, discrimination, bias, faimess concerns or research integrity issues have been raised during the completion of this work.
REFERENCES
Allys, T Marchand, J-F Cardoso, F Villaescusa-Navarro Ho, and Mallat. New interpretable statistics for large-scale structure analysis and generation. Physical Review D, 102(10):103506, 2020.
Guillaume Berger and Roland Memisevic. Incorporating long-range consistency in enn-based tex- ture generation. In International Conference Learning Representations, ICLR 2017. OpenRe- view.net, 2017.
Joan Bruna and Stephane Mallat, Multiscale Sparse Microcanonical Models. Mathematical Statis- fies and Learning, 1(5):257-315, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Arturo Deza, Aditya Jonnalagadda, and Miguel P. Eckstein. Towards metamerism via foveated style transfer. International Conference on Learning Representations, 2019.
Jeremy Freeman and Eero Simoncelli. Metamers of the ventral stream. Nature neuroscience, 14 19):1195-1201, 2011.
Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Advances in neural information processing systems, 28:262-270, 2015.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
P. Goupillaud, A. Grossmann, and J. Morlet. Cycle-octave and related transforms in seismic signal analysis. Geoexploration, 23(1):85-102 1984. ISSN 0016-7142. Seismic Signal Analysis and Discrimination III.
Jieqian He and Matthew Hir. Texture synthesis via projection onto multiscale, multilayer statisties. arXiv preprint arXiv:2105. 10825, 2021.
David J Heeger and James R Bergen. Pyramid-based texture analysis/synthesi.. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pp. 229-238, 1995.
Edwin Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Bela Julesz, Visual pattern discrimination. IRE transactions on Information Theory, 8(2):4-92, 1962.
P Laube, M Grunwald, MO Franz, and G Umlauf. Image inpainting for high-resolution textures using enn texture synthesis. In Proceedings of the Conference on Computer Graphics & Visual Computing, pp. 103-107, 2018.
10

---

Published as a conference paper at ICLR 2022
Roberto Leonarduzzi, Gaspar Rochette, entropy scattering models for financial time series. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1496-5500, IEEE, 2019.
Bouchaud, and Stéphane Mallat. Maximum-
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2479-2486, 2016.
Gang Liu, Yann Gousseau, and Gui-Song Xia. Texture synthesis through convolutional neural net- works and spectrum constraints. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 3234 12399.33..22 IEFF, 2016.
Stéphane Mallat, A Wavelet Tour of Signal Processing: The Sparse Way, 3rd Edition. Academic Press. 2001.
Stéphane Mallat, Sixin Zhang, and Gaspar Rochette. Phase harmonic correlations and convolutional neural networks. Information and Inference: A Journal of the IMA, 93):721-747, 2020.
Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation, 35(151):773-782 1980.
Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of com- plex wavelet coefficients. International journal of computer vision, 40(1):49-70, 2000.
Lara Raad, Axel Davy, Agnès Desolneux, and Jean-Michel Morel. A survey of exemplar-based texture synthesis. Annals of Mathematical Sciences and Applications, 3(1):89-144) 148, 2018.
Kai Schneider, Jörg Ziuber, Marie Farge, and Alexandre Azzalini. Coherent vortex extraction and simulation 2D isotropic turbulence. Journal of Turbulence, 7:N44, 2006.
Omry Sendik and Daniel Cohen-Or. Deep correlations for texture synthesis. ACM Transactions on Graphics (ToG), 36(5):1-15, 2017.
E. Simoncelli and Freeman. The steerable pyramid: a flexible architecture for multi-scale derivative computation. In Proceedings., International Conference on Image Processing, vol- ume 3, pp. 444-447 vol.3, Oct 1995. doi: 10.1109/CIP.1995533777...
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint trXiv:/409 1556, 2014.
M. Unser and N. Chenouard. A unifying parametric framework for 2d steerable wavelet transforms. SIAM Journal on Imaging Sciences, 6(1):102-135, 2013.
Ivan Ustyuzhaninov, Wieland Brendel, Leon A. Gatys, and Matthias Bethge. What does it take to generate natural textures? In International Conference Learning Representations, ICLR 20/7.
OpenReview.net, 2017.
Jonathan Vacher and Thibaud Briand. The portilla-simoneell texture model: towards understanding the early visual cortex. Image Processing On Line, 11:170-211, 2021.
Jonathan Vacher, Aida Davila, Adam Kohn, and Ruben Coen-Cagli. Texture interpolation for prob- ing visual perception, Advances in Neural Information Processing Systems, 33, 2020.
Thomas SA Wallis, Christina M Funke, Alexander S Ecker, Leon A Gatys, Felix A Wichmann, and Matthias Bethge. Image content is more important than bouma's law for scene metamers. ELife, 8:e42512, 2019.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644 PMLR, 2016.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45, 2018.
11

---

Published as a conference paper at ICLR 2022
Ning Yu, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, and Michal Lukac. Texture mixer: A network for controllable synthesis and interpolation texture. In Proceedings the the IEEE/CVF
Conference Computer Vision and Pattern Recognition, pp. 12164-12173, 2019.
Sixin Zhang and Stéphane Mallat. Maximum entropy models from phase harmonic covariances. Applied and Computational Harmonic Analysis, 53:199-230, 2021.
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Non- stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics (TOG), 37(4):1-13, 2018.
Song Chun Zhu, Ying Nian Wu, and David Mumford. Minimax entropy principle and its application to texture modeling. Neural computation, 98):1627-1660, 1997.
A MODEL AND ALGORITHMIC SPECIFICATION
We provide additional information needed to reproduce the numerical results of the models (for both gray-scale and color textures) considered this paper. First, we detail the models of PS, RF, VGG. We then give algorithmic parameters to obtain the synthesis images. For natural textures, we also propose a strategy to synthesize non-periodic images in our models. An image is non non-periodic if a periodic extension of the image to the domain outside Ωy create discontinuities at the contour of QN.
Sources of textures Our natural texture examples were obtained from the following three sources: CNS S textures,com Describable Textures Dataset mode 10 and the Github page of Berger & Memisevic (2017)
A.1 MODEL PARAMETERS
We specify the model parameters to synthesize both gray-scale and color textures. We also discuss how to extend the RF and the VGG model, originally designed for color textures, to model gray-scale textures.
PS: For both gcay-scale and color models, set the number of scales J 5, and the number of orientations = for the Simoncelli steerable wavelets. The spatial shift 7 (2)) E Ω2y chosen be in the range of max (7), 72) A 4. Note that this is different to the model parameters reported for the default PS model (which is / = 4, L =
4, A = 3) as we find that it results in a larger set of statistics and better visual quality. The synthesis results of this model can be reproduced by a Matlab software 12
RF: For gray-scale textures, we consider J x L. random convolutional filters (Wy)1s/SJL
Let f (j, f), the index j representing the scale of each filter, whose size is (W,, W,) for 1 < < J. For r = ( = (j,0)l < J, < L}, the representation is
RRF (RFx(7,u) = p( ()))()),,
=i.f)
For color image f2"Jsees. we use 3 x J x L random convolutional filters. The representation is
3
R x = Σ ves,e())), 7= e er.
The correlations CRF, x are defined for all pairs (7, 7) (Tx))xx x T
both the gray and color
cases, results correlation matrix CRP with _²1² statistics.
https://eettures.com
12

---

Published as a conference paper at ICLR 2022
Following the default setting of the RF model, set J and L 128 for filters whose
sizes are W1 = 3, W2 = 5, W3 = 7, W = 11. W3 = 15, W6 = 23, 7 = 37, Ws = 55. Each filter 3 w...je is generated randomly according to Gloro:Uniform in the software Lasagne.
VGG. For color image ( the VGG model computes correlation matrix CVGG between the features maps within different layers of a pre-trained CNN network. To adapt this model gray-scale textures, shall add one input layer which converts gray-scale image into a color image, by setting x y for each color channel c. This allows one use the same CVGG x to compute the gradient therefore to synthesise gray-scale texture. For both gray-scale and color textures, we use only five layers 'convl I', "pooll', 'pool2", 'pool3', 'pool4', as proposed in the original work.
the VGG loss with respect to y, and
ALPHA. See the main text.
A.2 ALGORITHMIC PARAMETERS
We specify the optimization parameters used synthesize both gray-scale and color textures.
PS: It utilizes iterative projections onto constraint sets generate textures. We the number of iterations to 200.
RF: It uses the L-BFGS procedure 14 with a memory size 20, and with maximal number of iterations 2000. The initialization for each pixel value of a gray-image is Uniform between [_., For the color image case, each RGB channel is initialized independently with Uniform distribution between [-1, 1]. To address non non-zero mean textures (i.e. E(X(w)) o
0), the empirical mean of z is subtracted from the input
compute the representation. is added back to the output of the optimization to produce a synthesis.
15 with memory size of 20, and with a maximal number of iterations of 2000.For both gray-scale and color images, bounds constraints are used for the optimization. The initialization of each pixel value is the standard normal distribution (zero mean, unit variance). To address non-zero mean textures (i.e. (E( * 0), the VGG mean is subtracted from the input of the representation 16 It is added back to the output after the optimization produce a synthesis (with an additional histogram matching post-processe proscesccsssng)
VGG: It uses the L-BFGS procedur
For all the models in ALPHA, restarts. Starting from the standard normal distribution, with mean and standard deviation estimated from the observation, runs for 500 iterations and then it restarted with an initialization obtained from the pre- vious L-BFGS result. This repeated 10 times obtain the synthesis (with an additional histogram matching post-processing)
ALPHA:
the L-BFGS optimization algorithm with
the L-BFGS procedure implemented in Pytorch. It
A.3 NUMBER OF STATISTICS IN THE ALPHA MODELS
In this section, we detail the number of statistics in the different ALPHA models. We begin by giving the formula for each model (note that we do not include the low-pass statistics, which numbers are negligible). To (partially) avoid redundancy in the coefficients, for all models, we compute only the correlations for indices in such that j2 2 f1- This gives us the following formulas for the number of statistics:
(ALPHAAs) (2. - 1)) 121- |
#(ALHHA)) = 1 [
https::/lassanne.eet htw.aaaanne.reddhhheccccoo /
Secipy.optimiz fmin. 1 in Python
Dacipy cetipy.cptimm mim ze Python
For the color image whose pixel value is between zero and one, the mean of BGR is 0.40760392, 0.45795686, 0.48501961. For the gray-scale, we simply take the average of the BGR mean.
13

---

Published as a conference paper at ICLR 2022
#(ALPHAL) 1) (ALPHHAc - 1J
Note that, for all ALPHA models, we also compute first order statistics jry, i.e. the spatial averages of RALPHAH(y, u). There are M| As| statistics of this sort in every model, which negligible with respect to the total number of second order statistics.
Note also that there are still some redundancies in these statistics, as for j = J', all correlations for 8, 0', a, a', are counted twice. The number of such statisties is for all model, superior to the number of first order statistics, which shows that our formula is in fact an upper bound for the exact number of statisties.
A.4 NUMBER OF STATISTICS IN THE PS MODELS
The PS model can be interpreted as a particular case of the WPH covariance model as it contains the following three key categories of statisties:
Raw coefficient correlations (A: = K' = 1): they capture 2nd order statistics of a stationary process X, i.e. the correlations between (u) and X (u') for (u,u') 32
Coefficient magnitude statistics (k: = k' = 0): they capture information of X beyond the 2nd order statistics. Very often, nearby scales and angles are considered in the model, such as and 6' = 0.
Cross-scale phase statistics (k: 1, 2): they capture local phase alignments of the wavelet coefficients at nearby scales j' = + 1, which are complementary to the magnitude correlations.
The major issue count the number of statistics of this model is avoid double counting the statistics which are the same. This is mostly due to the symmetries of the covariance matrices. We next detail how obtain the number of statistics for both the gray-scale and color model, by following the work Portilla & Simoncelli (000) and Vacher & Briand (2021).
As before, we assume the number of wavelet scales is J, the number of wavelet orientations L, and the spatial shift range is within square of size 12*+1) 1)xxxxxx (2A+1). Let us denote N₂ = 24 + 1. We next describe detail the number of statisties of each category counted our paper,
A.4.1 PS IN GRAY-SCALE
Marginal statistics of z: 6. They include mean, variance, skewness, etc.
Marginal statistics of wavelet coefficients: 2(J + 1)
Auto-correlation of wavelet coefficients (raw coefficient correlations): (J 1) + 1)/2.
Auto-correlation of magnitude of wavelet coefficients (coefficient magnitude statistics): JL(N?+ 1)/2 J - 1)/2 + (J 1
Mean of magnitude of wavelet coefficients: JL + 2. This not counted the paper of Portilla & Simoncelli 2000), but it used in the Matlab software.
Cross-correlation of phase wavelet coefficients (cross-scale phase statistics): 2(J-1)L3 JL2. The extra 2 coefficients counted in the paper, but used in the Matlab software.
cross-correlatio of real sub-band cousin coefficients, which not
check that the sum of these number is 792 when J = 4, = 4,A 3 (N, = 7). If we do not count the coefficients which are only counted in the Matlab software (JL + JI as reported in the original paper.
To compare with the model size of the original work of Portilla & Simoncelli (2000),
82), it results in 710
A.4.2 PS IN COLOR
Marginal statistics of x and PCA transform of r: 6 30.
17hhe number of such moments is of the order of ]|@| | mediate and large models, and ( [|,l² for the color model.
[Aal for the small model, ||A]? for the inter-
14

---

Published as a conference paper at ICLR 2022
Marginal statisties of wavelet coefficients: 6(J + +
Auto-correlation wavelet coefficients (raw coefficient correlations): 3(J + 2) + 11/2. This is called Central autoCorr of the PCA bands in the Matalb software (which includes lowband).
Auto-correlation of magnitude of wavelet coefficients (coefficient magnitude statistics): 3JL((N + 1)/2 (3L 1)/2 (J 1)(3
Mean of magnitude of wavelet coefficients: 3(JL + 2).
Cross-corrclation of phase of wavelet coefficients (cross-scale phase statistics): J(3L)² (J ()))
A.5 NON PERIODIC BOUNDARIES NATURAL IMAGES
The convolution operation in the wavelet transform (equation is performed using the Fast Fourier Transform. Additionally, recall from Section 2.Ithat that spatial shifts are defined with periodic bound- ary conditions. This implies periodicity of the imput image However, natural texture images not periodic, so one needs to adapt the computation of coefficients to take into account possible border effects. To that end, instead of averaging over all Ωn in eq. (T). each correlation coef- ficient is averaged over a sub-window inside ΩN,, which size depends on the scales of the coefficients being correlated. More precisely, let 7 (j, 0, a)) and 7 "0'00,a'' Note jn := max(j,j'). We 2 i = 1,2]. Then, for non periodic define Q,m.. == {u (u1, U2) E 22 2 I us < N
images, we compute
ALPHA 2 (T,7'TT)
Σ aelN
1a.m (u)d (u- (u- RALPHA x(7, HALPHAA 1 -)
where the spatial shifts are defined periodically. Note that the spatial averages My and fay are also performed on f
B
PROOF PROPOSITION
Using the fact that
p(Real(zee)) = =p((==lll)))) ( |=|p(cos(a + =(2)),
and computing the Fourier coefficients of the 2 Ax--perioii function Pa (2(2) in the variable a, we obtain
((p)(=))(k)
1
Pa (2)e ska da
2T [.22m]
| 2n 10,2al
p(cos(a ((2))e iko da
| dkp(t)
[2]*ck
where is the Fourier transform of f(.)) = p(cos(-)) at the frequency k. The function + P2(2) being periodic in a, have its decomposition in Fourier series
Pa(2) = Σ (k)
kez
Σ c* [=]=] ikk
kEE
We can then write, for any 2, E C, and a, a, E [0, 2T).
(2'))
Σ c
(2 - ((ka-Kaa),
k.
15

---

Published as a conference paper at ICLR 2022
Replacing and 2' by any two wavelet coefficients ())) and vo r), thus obtain the relation in Proposition
C PROOF OF PROPOSITION 2
Let e C, and recall from eq. 44, that Pa (z) = p(Real(ze"n)) Note that we have the following relation
2 = Po(2) - Px((2) -i(p (2) P2p (2)).
We can then write
zz" = (po(2) - Pr (z) - i(0g (2)pg()=- Pap (=)) - px i(pg( (z') - pap (21))
= Σ Pa
a
with / = {0, 2. π. Sn
2 J. Replacing a with x ( x' with æ By v T),, and injecting this gives the desired result, with Wa,a Sl
relation eq. [I]
D SUPPLEMENTARY RESULTS FOR THE GRAY-SCALE ALPHA MODELS
Here, we present further visual comparison between the different ALPHA models for gray-scale images, to illustrate the trade-off between quality and diversity.
E RELATION BETWEEN ALPHAL AND ALPHA|
Here, we informally explain why, under some conditions on the wavelet family, setting a' E (0) in the ALPHA models should not lose too much (but still some) information captured by the statistics.
First, remark that the simple linear relation
Σ
(ha
Σ pa (=)))((())edka aEλ,
tells us that computing all correlations for a E A4 and a' 0 gives us at least the information contained in the r.h.s. of eq. (9).
Furthermore, if a E As (0..... 3r), we make the following approximation
Σ e
[0.2 Pa (z)e ikas do 2TTcH[=]
Recall also from the proof of Proposition taken along the variable a. Therefore,
2 that ((pa = [2] where the Fourier transform
Po(2)= Σ c=(=)
KEZ
Therefore,
(p()) ~ ( Pa(z)e-she ika da.) Po(z)
Σ pa(z)e po(z) o
[TTc(22 Σ ow (2
keez
2TT Σ
cer[z] [=
ME
16

---

Published as a conference paper at ICLR 2022
Obs
ALPHAs (3.5k)
ALPHA (35k)
ALPHAL (142K)
Figure 4: Visual comparison of syntheses from the ALPHAs, ALPHAr and ALPHAr models.
Then, replacing z and with wavelet coefficients, we get
-iko CALPHA
2TCK
Σ
CPH ((,,0,)),,,,,,,,,,
aEda
K
Using Plancherel's theorem, write that
1
WPH OPPH2((,.00) (I.0, ('
F(I ((ax) (w),
WE
where the Fourier transform of image is defined by F(x)(w) = Every wwu)e-iwu twn, and fr denotes the translation by i.e. f-)( f(- 7).
Now, suppose that the wavelets ei, have disjoint compact frequency support, in balls B -2-2-iC") where À = 2- and is the central frequency of the mother wavelet ve (ef. ?). Suppose also
17

---

Published as a conference paper at ICLR 2022
that frequency transposition property of the phase harmonics operator (ef. Mallat al. (2020) is such that [a * dx has (approximately) frequency support in Bиx
Then, for all X, X, and all E Z. there exists only one k* such that the frequency supports of [x v/al and [z * sxx] are not disjoint, i.e. such that
(0i,
0', T) + O. This tells that
Σ aεAA
e iko -ii (i
f. a). (AAPHAA(A...O '''''''''''0000 0), CLCk C WPH (((,0,0)) ('
Thus, computing all correlations for a E AA, and a' gives us (approximately) all the informa- tion contained in WPH coefficients for any pair k. k'.
This result lies on several approximations, and strong assumptions about the wavelets, which are not fully met in practice. For this reason, setting a' = 0 instead of a' E A1 effectively reduced the amount of information captured by the statistics, and therefore icreassss the diversity of the model. However, as we observe in Section 3.2 there is not too much information lost, and the resulting model still captures most of the important geometric structures in texture images.
F REDUCED COLOR MODEL
One can reduce the number of statistics in the color model by selection the spatial shift parameter to be non-zero only for correlations between the same color channels. More precisely, it is defined by the following index set: : (())),T)) : (G.0..), (j' d')))).) E TALPHA .c d' e
(1,2,3) U ((,,''0) ((j,0,a) 0',0' o)),0) E TALPHA, (c, (1,2,3)²). This gives model of size ~ 113k, with little degradation of the visual quality, as shown in Figure
G VGG SCORE
In UUstuzhaninnv et al. (2117), the authors proposed to use the synthesis loss of the VGG model to evaluate the quality of syntheses from any model. The goal is to define a quantitative, and more objective evaluation method than mere visual inspection. Since the VGG model produces syntheses almost indistinguishable from real textures, it is natural to consider its loss to asses the quality of synthesis. We computed this loss for the first two examples of Figure 3 (radishes and cherries), and the frist two examples of Figure ( (gravel and Turbulence flow). Note however that this loss is not exactly the same as the one used in Ustyuzhaninov et al. (2017), as the layers selected to compute the loss are different. In this work, we chose to use the layers suggested in Gatys et al. (2015), (i.e. 'conyl_1" 'pooll', 'pool2', 'pool3", and 'pool4' of the VGG-19 19 network (Simonyan & Zisserman 2.14), and compute the relative VGG loss [8
We notice that this score is not always consistent with visual inspection, as there are texture examples and models for which the syntheses do not look much like the observation image, yet produce a small VGG loss (see e.g. the first and last rows of Figure 3 the RF model syntheses have the smallest loss). should also be noted that the VGG loss reported on the VGG syntheses is not the synthesis loss after optimization, as histogram matching (HM) procedure is performed as post-processing after optimization. We observed that the VGG loss of the syntheses from the VGG model after HM was considerably higher than the one for syntheses before it, while being visually very similar as illustrated in F These observations suggest that the VGG score suffers from instabilities after reaching certain level (that is, if the VGG loss is small enough, small perturbations of the values of the image pixels might have a strong impact on the loss).
H INFLUENCE OF THE CHOICE OF THE WAVELET TRANSFORM
H.1 INFLUENCE OF THE WAVELET FAMILY
In Section 3.2 we illustrated the importance of the set of indices T that define the wavelet coef- ficients being correlated. Another important role is played by the choice of the wavelets used in
Using the code from master/vee wester/vgg_lo los.. css.py .py (function style loss relative).
https: / ht///tttuuu thub. wttuu..comiiii
h/blo//
18

---

Published as a conference paper at ICLR 2022
Obs
ALPHAc (320k)
Reduced (113k)
Figure 5: Visual comparison of syntheses from ALPHAç model and its reduced version.
Table Relative VGG loss of gray-scale textures for the first two examples of Figure 3 and the first two examples of Figure 9
Data / Model ALPHA
VGG
PS
RF
Radishes
5.02e-05
1,87e-05
2.37e-04
1.13e-05
Cherries
4.86e-05
1.47e-06
6.68e-04
9.65e-06
Gravel
5.97e-05
3.08e-06
7.25e-04
1.29e-05
Turbulence
5.59e-05
5.97e-05
2.42e-04
3.95e-05
19

---

Published as a conference paper at ICLR 2022
Before HM
After HM
Figure 6: Visual comparison of syntheses from the VGG model, before and after histogram match- ing. Before HM, the relative VGG loss is 2.22c-08, while after HM, the loss is 5.38c-05.
equation 2 As illustrated in Figure 7 this choice can have a visible impact on the quality of the textures. We observe that, while on the first example, the coherence of the structures appear similar for the three wavelet families, the second example shows that the wavelets used in |Portilla & Si- moncelll (2000) are less efficient in reproducing the contours the objects (pebbles). While in our experiments, we chose to use the classical Morlet wavelets, an optimal choice for the wavelet family remains an open problem.
Observation
Simoncelli
Bump
Morlet
Comparison between different wavelets families used in the ALPHA, model. Central zooms of syntheses using the same covariance model, with three different wavelet families. From left to right: observation, Simoncelli steerable wavelets, bump steerable wavelets (Mallat et al. 2020), and Morlet wavelets.
Figure 7:
H.2 INFLUENCE OF SCALE PARAMETER
Recall from Section 2.2.1 the wavelet transform of an image x defined by
{ar w3s,, x du (oloosie #E (O."..I-))
The maximal scale parameter J also plays an important role in the definition of the wavelet trans- form. It determines the scales of the structures being captured by the transform. If this parameter is too small, large structures in the observation image might not be captured and reproduced in the model syntheses. Conversely, variance, inducing memorization effect in the syntheses. Figure 8 illustrates this point on two examples from Section 4.2 By setting J (i.e. the maximal range of structures captured by the wavelets is of size 2 = 16), we observe on the first example that the larger structures (bubbles) are not well reproduced. When J is set to 6, the observation is almost identically reproduced by the synthesis. Similarly the second examples, several parts of the synthesis appear very similar to ones the observation. We found that a suitable trade-off consists in setting = 5 for images of size N 256.
too large, then the large scale statistics may have a high
20

---

Published as a conference paper at ICLR 2022
Observation
4
J=5 5
Figure 8: Syntheses from the ALPHAc model defined with three different maximal scale parame-
ters J E 6, As in Section Morlet wavelets used.
I SUPPLEMENTARY RESULTS OF ATPHA//LPPAC VS. PS, RF, VGG
In Figure we present additional syntheses on various examples, from the PS, ALPHAc, RF and VGG models. These examples can be viewed as random (Turbulence flow, tree bark, porous stone), structured (gravel, paisley pattern, tree leaf, school text), or inhomogeneous (crafted pattern of third row, but also the porous stone).
We see that ALPHA again significantly improves the visual quality of the PS model on gray-scale textures such as the gravel and turbulence flow. The visual quality of RF and VGG seems also worse on Turbulence flow compared to ALPHA,. In some examples such as tree leaf, we find the synthesis of all the models are similar. On the inhomogeneous porous stone, non of the models give satisfying visual results.
Similarly, in Figure 10 are presented supplementary syntheses form the color models, for structured images (radishes, bubbles, flowers), quasi-periodic images (scales, honeycomb, bricks), and non- stationary images (feathers). As previously observed, for highly structured quasi-periodic images such as the bricks example, the VGG model fails to capture long-range correlation, which can be solved using the method of Berger & Memisevic (2017). Syntheses non-stationary images exhibit memorization effects, previously observed.
21

---

Published as a conference paper at ICLR 2022
Observation
PS (3.2k)
RF (525k)
ALPHA (35k)
VGG (177k)
Dappiaa to
Mare Iriin Man daged shaule dugondd paanoed Landonaata and in aho ai cu tanss redde hondor praegratonn lorn do ade of protiona ai bogueeee Dayy
Soa srdll levers in bi puansnn
Di Digascor sonnn Bailbnnanu, na molle teale d d'unn digon muceadaae dhe quadder manaaad dheule brann compuiiue by Dh Bondett
brakking
wnt I tan Eegen coae
fe word ae
2 yo
oy
dll Cd:
man
Lia anaeed
da banten Boaks aoe
10 laak dhe D50
Figure 9: Visual comparison between different texture models on gray-scale images.
22

---

Published as a conference paper at ICLR 2022
Observation
PS (17k)
RF (525k)
c (320k)
VGG (177k)
Figure 10: Visual comparison between different texture models on color images.
23


Reviews:
[{"review_id": "ytBwwamJO4", "reviewer": "Reviewer_Q8m3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Proving the relationship between the different sets of texture models, with the additional proof that ties ReLU's and phase harmonics together is nice work and a strong contribution.  Unfortunately, much of the rest of the paper is lacking and detracts from the overall effect.\n\nMain criticisms:\n1.  This paper is not suitable for ICLR as nothing within the paper is learned from data, these are hand constructed (and well though through) representations that are similar to those found in learned representations but are not here learned.  This is not a problem per se, it simply doesn't fit with the conference themes and as such may not be found useful by most of the conference attendees.\n\n2. The paper spends an inordinate amount of space laying out and explaining concepts and algorithms which are not novel contributions of this paper and would be better served with passing references for the reader to follow up with.  The novel contributions of the paper do not begin until the bottom of page 4.\n\n3. The examples shown reveal that all of the Alpha models (including AlphaI and AlphaC the chosen models for analysis in this paper) are memorizing patches of the image (any image that has the portilla simoncelli badge in the bottom left corner has this badge completely recreated somewhere else in the image as completely readable text), which a proper texture model would not do unless it was simply memorizing the image patches.  This is especially apparent in the color images which have more of examples of the name badge in the corner.  \n\n4. The authors state parameterizations of comparison models on page 8 that are at least partially incorrect (here stated that portilla simoncelli has ~7000 parameters when in reality it has 701 parameters, an order of magnitude difference).  A careful inspection of the others may find similar magnitude errors.\n", "summary_of_the_paper": "The paper presents a novel image texture model that is a hybrid of the traditional wavelet constructions, and the more modern machine learning constructions (incorporating only ReLU nonlinearities on the wavelet basis), proves that this class of models is actually akin to phase harmonic functions and that these are sufficient for capturing texture statistics, and claims to show competitive performance with state of the art with significantly fewer parameters.", "summary_of_the_review": "For these reasons above I am inclined to reject this paper for ICLR, and suggest the authors find a more suitable conference for this work, as well as reanalyzing some of the results and rethinking the overall presentation of the work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e1_L1peKpQf", "reviewer": "Reviewer_xfp4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a detailed review and explanation of wavelet and how to use the rectified wavelet coefficients for covariance statistics matching based texture synthesis. The motivation also seems to be clear: trying to use less number of statistics to achieve satisfying quality. However, I have several main concerns regarding to the comparisons with recent papers and the types of texture images this method can handle.\n\n1. as the main motivation is trying to achieve good quality with as few number of statistics as possible, from figure 3 and figure 4, the quality of “VGG” seems to be better than the proposed method and also uses less number of statistics (much less for color textures with 177k vs. 320 and similar for gray texture 177k vs. 142k)\n\n2. After Gatys et al. 2015 and Ustyuzhaninov et al. 2017, there are more recent methods on texture synthesis which have achieved many impressive results. For example:\n    Deep correlations for texture synthesis.  Sendik & Cohen-or et al. 2017.\n    Texture Synthesis Through Convolutional Neural Networks and Spectrum Constraints. 2016. \n    Non-Stationary Texture Synthesis by Adversarial Expansion. Zhou et al. 2018. \n    and many others.\n  This submission did not cite or compare with these approaches (and other more recent methods).\n \n3. The submission and supplementary file provide very limited number of results. The images are mostly stationary images. More examples of structure images and non-stationary images are needed to validate the proposed method. \n\n4. One main application of texture synthesis to generate a larger image output given a smaller input patch. It seems that the proposed can not generate a larger image output. \n", "summary_of_the_paper": "This work proposed a texture synthesis framework using the rectified wavelet coefficients. The paper claims that the proposed method cand achieve similar quality with the VGG feature based method (Gatys et al. 22015) and random filter based method (RF, Ustyuzhaninov et al 2017) and gets better quality than PS (Portilla & Simoncelli 2000, while requiring less number of statistics than RF. ", "summary_of_the_review": "Although the motivation of the proposed framework seems to be pretty specify and properly positioned, the experiments does not seem to validate the superiority over existing methods. The results are quite limited. It also missed the citation and comparisons with many recent works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ScFNCi7rzei", "reviewer": "Reviewer_Gwxo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positive points**: The results presented in the paper are very interesting because they shed light on what exactly are CNNs learning on images, since the presented model can produce similar results to state of the art CNNs with a much smaller number of parameters and a fixed filter structure (wavelets). This is, obviously very important given the usage of CNNs in the literature and the little current comprehension there is on the information they encode and it is relevant for image comprehension. In this sense, this paper shows that the rectifier wavelet operator is \n\n**Negative points**: The relationship with patch-based methods is not discussed at all. Even thought the proposed algorithm lays within the \"statistics-based methods\", in the numerical results it should be compared to other family of methods such as exemplary-based models (see Raad  et al 2018 for examples) ", "summary_of_the_paper": "This paper presents a new image representation model based on wavelets and non-linear rectifiers that allows to synthesize complex geometric textures with a better visual quality than previous wavelet-based models. \nThe main interest of the paper is the usage of the mathematical results from Mallat et al 2020 and Zhang and Mallat 2021 on wavelet phase harmonics to image texture synthesis. \n\nThey also show that the PS model (Portilla & Simoncelli 2000) is a particular case of the wavelet phase harmonics-based (WPH) model. Both methods underline the importance of statistical dependencies between wavelet coefficients across scales. The specific choice of the parameters available for WPH is important to balance between reproducing good structural information and memorizing patterns from the observation.\n ", "summary_of_the_review": "The paper presents the results on texture synthesis using a new image representation model based on wavelet phase harmonics. The results are interesting and show that these representations are powerful enough to encode complex image features. What this paper presents starts clarifying how CNNs are encoding image comprehension", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jp6u6L1toBT", "reviewer": "Reviewer_bssY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a paper that has a great idea going on with regards to qualitatively estimating the minimum number of constraints in a parametric texture model that revolves around wavelets. My main motivation for marginally rejecting this paper is that it comes up short in terms of the types of qualitative insights it can deliver, not capitalizing enough in the rich mathematical motivation for a wavelet-constrained model. Nevertheless, I think I am lenient on increasing my score if authors address some of the points I mention below and I’m also genuinely curious on what the other reviewers thoughts are.\n\n* Not enough highlight on the number of parameters of the wavelet based model. This should be emphasized more in Table 1, or in an extra table as it’s superficially mentioned at the end of section 4.1. The exploration of how the number of parameters affects the quality of the synthesized stimuli is a pivotal part of this paper that has not been highlighted enough (the results are there, but the presentation needs to improve).\n* Adding more examples of model failures, and singular successes. The figures in 2,3, and 4 are too basic and standard. I think they should be resumed in another separate figure  (perhaps just using figure 4 or a variation of this), and there should be *other new* figures that highlight how the models perform for very specific images. What would the synthesized textures look like for a scene? An abject? Julesz conjecture counter-examples? And does the quality change as the number of parameters is varied (Alpha S,I,L,C) Many of the previously mentioned images that do not provide a stationary signal can magnify the differences in terms of how each model computes its perceptual constraints. \n* Linking to the previous bullet point: Why does each model fail? What texture constraint has varied, such that the model all of a sudden breaks down or succeeds for some of these other images that are different from the canonical textures shown in the texture synthesis literature.\n\nOverall, the paper is well written, it provides a good mathematical overview of wavelets and also cites the relevant literature on this subject. I would also suggest authors to look into citing this paper as well that provides some connections of texture synthesis to neuroscience:\n\n* Texture Interpolation for Probing Visual Perception. Vacher et al. NeurIPS 2020.\n\nOnce again, I am looking forward to reading the author's rebuttal as I am eager to increase my score as this paper is already heading in the right direction.\n\n\n**Post-Rebuttal Notes**: I wanted to increase my score from 5 to 7, but since there is no 7 option I have decided to sin on the generous side and give this paper an 8. [I apologize to authors and/or reviewers if they have seen me toggle back and forth between 6 and 8. My overall vote is that this paper be accepted as a poster.]\n\n", "summary_of_the_paper": "This paper provides an attempt to characterize wavelet-based texture synthesis models contingent on the number of parameters and the nature of the parametrization. Proper baselines are added with respect to other relevant texture synthesis models.", "summary_of_the_review": "A solid first step, but more qualitative insights are needed given to make this paper shine as much as it should. Weak Reject with a possibility of increasing my score during rebuttal to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new wavelet-based model to represent textures. The model incorporates a wide range of statistics, by computing covariances between rectified wavelets coefficients, at different scales, phases and positions. The model can synthesize textures that have a similar quality to state-of-the-art texture models using CNN structure. Qualitative results are shown to demonstrate the effectiveness of the model.  The paper studies an important problem in computer vision and neuroscience, which is texture modeling. However, many important related works are missing. After rebuttal, three of four reviewers champion accepting the work because the proposed wavelet-based texture model, which produces competitive synthesis with much less parameters than the CNN-based model, will be beneficial to the fields of computer vision and neuroscience. One reviewer has critical comments on this paper because the paper lacks a comparison again more recent works both quantitatively and qualitatively. However, during rebuttal, the authors expressed their disagreement with it and pointed out that the goal of the paper is to bridge the gap between the classical work of Portilla and Simoncelli (2000), and the CNN-based models and to find what statistics are needed to describe the geometric structures in natural textures. Their discussion didn't reach an agreement after rebuttal. After an internal discussion, AC recommends accepting the paper but urges the authors to improve their paper by taking into account all the suggestions from reviewers, especially include the discussion or comparison with those related works mentioned in the rebuttal.
Decision: Accept (Poster)