Texture Representation and Synthesis Using Correlation of Complex Wavelet Coefficient Magnitudes

We present a statistical characterization of texture images in the context of an overcomplete complex wavelet transform. The characterization is based on empirical observatio ns of statistical regularities in such images, and parameterized by (1) the local auto-correlatio n of he coefficients in each subband; (2) both the local auto-correlation and cross-correlation o f coefficientmagnitudes at other orientations and spatial scales; and (3) the first few moments of th e image pixel histogram. We develop an efficient algorithm for synthesizing random im ages subject to these constraints using alternated projections, and demonstrate its effectivenes s o a wide range of synthetic and natural textures. We also show the flexibility of the repres entation, by applying to a variety of tasks which can be viewed as constrained image synthesis problems . Vision is arguably our most important sensory system, judgi ng from both the ubiquity of visual forms of communication, and the large proportion of the human brai n devoted to visual processing. Nevertheless, it has proven difficult to establish a good mathemat ical definition (in the form of a statistical model) for visual images. The set of visual images is enormou s, and yet only a small fraction of these are likely to be encountered in a natural setting [43, 2 5, 19, 54]. Thus, a statistical prior model, even one that partially captures these variations in likeli hood, can substantially benefit image processing and artificial vision systems. In addition, many aut hors have proposed that biological visual systems have evolved to take advantage of the statistical pr operties of the signals to which they are exposed [ e.g., 3, 4, 2], thus suggesting a direct link between image statis tics and visual processing. In order to characterize visual images statistically, one m ust make some sort of restriction on the probability model. The most common assumptions are localit y (the characterization is specified on local spatial neighborhoods), stationarity (the statisti cs depend only onrelative spatial position within the image), and parameterized forms of density ( e.g., Gaussian). The subclass of images that we commonly call “visual texture” seems most consistent with l ocal stationary density models. Traditional definitions of texture can be classified into “structu ral” and “statistical” [34], the first consisting of a set of repeated deterministic features, and the second c rresponding to a sample drawn from a JP is supported by a fellowship from the Consejo Superior de I nv stigaciones Cientificas (CSIC), and the Comision Interministerial de Ciencia y Tecnologia (CICYT, Spain), u nder grant TIC97-325. EPS is supported by an Alfred P. Sloan Research Fellowship, NSF CAREER grant MIP-9796040, and the Sloan Center for Theoretical Neurobiology at NYU. probability density. Such a distinction has turned out to be somewhat artificial: For example, Zhu et al.  have demonstrated that is possible to capture and repro duce structural elements in texture using purely statistical models. Furthermore, many real-wor ld textures seem to incorporate both aspects, in that they can be described as a set of repeating structural elements subject to some randomness in their location, size, color, orientation, etc. This observ ation leads us to seek a single method of representing textures. In this paper, we develop a fully statis tical description, and demonstrate that it is also able to capture and reproduce a wide variety of structur al elements. Julesz pioneered the statistical characterization of text ur s by hypothesizing that the N th-order joint empirical densities (for some unspecified N ) of neighborhoods of image pixels, could be used to partition textures into classes that are indistinguishabl e to a human observer [40]. This work thus established the use of both the locality and stationarity as sumptions, the goal of determining a minimal set of statistical constraints, and the validation of textu re models using human observers. Since then, researchers have explored a wide variety of approaches for t exture characterization and synthesis. One of the most basic distinctions between the various appro ches is the choice of representation. Starting with Julesz, many authors have worked directly on t he statistical attributes of local spatial neighborhoods of pixels, typically in the form of a Markov ra ndom field [e.g., 35, 42, 17]. But most others attempt to simplify the description of the densi ty by first processing with a set of linear filters, such as Gabor filters or wavelet bases. The use of loca lized multi-scale multi-orientation sets of bandpass filters is inspired by what is known of biolog ical visual processing, and justified by recent studies of the higher-order statistical properti es of such representations. We discuss this further in section 1. These filters may be held fixed, or chosen adaptively depending on the image statistics [ e.g., 10, 26, 22, 60, 51]. Assuming a preconditioning subband decomposition, most te xtur characterizations are defined by a set of statistical measurements that are used to constrain t he model. The majority are based only on marginal statistics. In particular, much of the work on text ure segmentation is based on estimates of variances of a set of subbands [ e.g., 61, 6, 7, 64]. The use of variance constraints implicitly as sumes an underlying Gaussian marginal density, but there is abund ant evidence that bandpass filtered natural images have highly non-Gaussian marginals [ e.g., 25, 18, 45]. Specifically, the densities tend to have sharp peaks at zero, and longer tails than a Gaussian. In addi tion, a number of researchers have shown that Gaussian statistics are not sufficient to constra i the appearance of visual textures [52, 24]. For purposes of synthesis, some authors have utilized filter ed non-Gaussian white noise [ .g., 24, 14] which have provided good results when based on a multi-sc ale representation [ e.g. 15]. Two recent highly successful algorithms (described below) are b sed on marginal densities of linear filter outputs [36, 70]. Joint densities are more difficult to characterize, even whe n limited to pairwise ( N = 2) interactions. The models discussed above capture joint statistics implicitly through the preconditioning linear transformation. Some authors have used covariance (or c orrelation) matrices to characterize the joint densities of local neighborhoods of subband coefficie nts for purposes of classification or segmentation [ e.g., 16, 8]. In the pixel domain, many authors have utilized the p airwise joint histograms (also known as “co-occurrence matrices”) [ e.g., 35, 17, 29, 31, 23]. Popat and Picard [50] have developed a clustering approach for representing densities o f local pixel neighborhoods (including those at different scales). They have applied this to compression , classification and restoration, and have demonstrated impressive synthesis results .