Wavelet Domain Multifractal Analysis for Static and Dynamic Texture Classification

In this paper, we propose a new texture descriptor for both static and dynamic textures. The new descriptor is built on the wavelet-based spatial-frequency analysis of two complementary wavelet pyramids: standard multiscale and wavelet leader. These wavelet pyramids essentially capture the local texture responses in multiple high-pass channels in a multiscale and multiorientation fashion, in which there exists a strong power-law relationship for natural images. Such a power-law relationship is characterized by the so-called multifractal analysis. In addition, two more techniques, scale normalization and multiorientation image averaging, are introduced to further improve the robustness of the proposed descriptor. Combining these techniques, the proposed descriptor enjoys both high discriminative power and robustness against many environmental changes. We apply the descriptor for classifying both static and dynamic textures. Our method has demonstrated excellent performance in comparison with the state-of-the-art approaches in several public benchmark datasets.

Related papers:

title: Representing and Recognizing the Visual Appearance of Materials using Three-dimensional Textons
abstract: We study the recognition of surfaces made from different materials such as concrete, rug, marble, or leather on the basis of their textural appearance. Such natural textures arise from spatial variation of two surface attributes: (1) reflectance and (2) surface normal. In this paper, we provide a unified model to address both these aspects of natural texture. The main idea is to construct a vocabulary of prototype tiny surface patches with associated local geometric and photometric properties. We call these 3D textons. Examples might be ridges, grooves, spots or stripes or combinations thereof. Associated with each texton is an appearance vector, which characterizes the local irradiance distribution, represented as a set of linear Gaussian derivative filter outputs, under different lighting and viewing conditions.Given a large collection of images of different materials, a clustering approach is used to acquire a small (on the order of 100) 3D texton vocabulary. Given a few (1 to 4) images of any material, it can be characterized using these textons. We demonstrate the application of this representation for recognition of the material viewed under novel lighting and viewing conditions. We also illustrate how the 3D texton model can be used to predict the appearance of materials under novel conditions.

title: Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
abstract: In this paper, we propose a computational model of the recognition of real world scenes that bypasses the segmentation and the processing of individual objects or regions. The procedure is based on a very low dimensional representation of the scene, that we term the Spatial Envelope. We propose a set of perceptual dimensions (naturalness, openness, roughness, expansion, ruggedness) that represent the dominant spatial structure of a scene. Then, we show that these dimensions may be reliably estimated using spectral and coarsely localized information. The model generates a multidimensional space in which scenes sharing membership in semantic categories (e.g., streets, highways, coasts) are projected closed together. The performance of the spatial envelope model shows that specific information about object shape or identity is not a requirement for scene categorization and that modeling a holistic representation of the scene informs about its probable semantic category.

title: Wavelet leaders and bootstrap for multifractal analysis of images
abstract: Multifractal analysis is considered a promising tool for image processing, notably for texture characterization. However, practical operational estimation procedures based on a theoretically well established multifractal analysis are still lacking for image (as opposed to signal) processing. Here, a wavelet leader based multifractal analysis, known to be theoretically strongly grounded, is described and assessed for 2D functions (images). By means of Monte Carlo simulations conducted over both self-similar and multiplicative cascade synthetic images, it is shown here to benefit from much better practical estimation performances than those obtained from a 2D discrete wavelet transform coefficient analysis. Furthermore, this is complemented by the original analysis and design of procedures aiming at practically assessing and handling the theoretical function space embedding requirements faced by multifractal analysis. In addition, a bootstrap based statistical approach developed in the wavelet domain is proposed and shown to enable the practical computation of accurate confidence intervals for multifractal attributes from a given image. It is based on an original joint time and scale block non-parametric bootstrap scheme. Performances are assessed by Monte Carlo simulations. Finally, the use and relevance of the proposed wavelet leader and bootstrap based tools are illustrated at work on real-world images.

title: Histogram model for 3D textures
abstract: Image texture can arise not only from surface albedo variations (2D texture) but also from surface height variations (3D texture). Since the appearance of 3D texture depends on the illumination and viewing direction in a complicated manner, such image texture can be called a bidirectional texture function. A fundamental representation of image texture is the histogram of pixel intensities. Since the histogram of 3D texture also depends on the illumination and viewing directions in a complex fashion, we refer to it as a bidirectional histogram. In this work, we present a concise analytical model for the bidirectional histogram of Lambertian, isotropic, randomly rough surfaces, which are common in real-world scenes. We demonstrate the accuracy of the histogram model by fitting to several samples from the Columbia-Utrecht texture database. The parameters obtained from the model fits are roughness measures which can be used in texture recognition schemes. In addition, the model has potential application in estimating illumination direction in scenes where surfaces of known tilt and roughness are visible. We demonstrate the usefulness of our model by employing it in a novel 3D texture synthesis procedure.

title: Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study
abstract: Recently, methods based on local image features have shown promise for texture and object recognition tasks. This paper presents a large-scale evaluation of an approach that represents images as distributions (signatures or histograms) of features extracted from a sparse set of keypoint locations and learns a Support Vector Machine classifier with kernels based on two effective measures for comparing distributions, the Earth Mover’s Distance and the χ2 distance. We first evaluate the performance of our approach with different keypoint detectors and descriptors, as well as different kernels and classifiers. We then conduct a comparative evaluation with several state-of-the-art recognition methods on four texture and five object databases. On most of these databases, our implementation exceeds the best reported results and achieves comparable performance on the rest. Finally, we investigate the influence of background correlations on recognition performance via extensive tests on the PASCAL database, for which ground-truth object localization information is available. Our experiments demonstrate that image representations based on distributions of local features are surprisingly effective for classification of texture and object images under challenging real-world conditions, including significant intra-class variations and substantial background clutter.

title: A sparse texture representation using local affine regions
abstract: This paper introduces a texture representation suitable for recognizing images of textured surfaces under a wide range of transformations, including viewpoint changes and nonrigid deformations. At the feature extraction stage, a sparse set of affine Harris and Laplacian regions is found in the image. Each of these regions can be thought of as a texture element having a characteristic elliptic shape and a distinctive appearance pattern. This pattern is captured in an affine-invariant fashion via a process of shape normalization followed by the computation of two novel descriptors, the spin image and the RIFT descriptor. When affine invariance is not required, the original elliptical shape serves as an additional discriminative feature for texture recognition. The proposed approach is evaluated in retrieval and classification tasks using the entire Brodatz database and a publicly available collection of 1,000 photographs of textured surfaces taken from different viewpoints.

title: Extended fractal analysis for texture classification and segmentation
abstract: The Hurst parameter for two-dimensional (2-D) fractional Brownian motion (fBm) provides a single number that completely characterizes isotropic textured surfaces whose roughness is scale-invariant. Extended self-similar (ESS) processes were previously introduced in order to provide a generalization of fBm. These new processes are described by a number of multiscale Hurst parameters. In contrast to the single Hurst parameter, the extended parameters are able to characterize a greater variety of natural textures where the roughness of these textures is not necessarily scale-invariant. In this work, we evaluate the effectiveness of multiscale Hurst parameters as features for texture classification and segmentation. For texture classification, the performance of the generalized Hurst features is compared to traditional Hurst and Gabor features. Our experiments show that classification accuracy for the generalized Hurst and Gabor features are comparable even though the generalized Hurst features lower the dimensionality by a factor of five. Next, the segmentation accuracy using generalized and standard Hurst features is evaluated on images of texture mosaics. For these experiments, the performance is evaluated with and without supplemental contrast and average grayscale features. Finally, we investigate the effectiveness of the Hurst features to segment real synthetic aperture radar (SAR) imagery.

title: Viewpoint Invariant Texture Description Using Fractal Analysis
abstract: Image texture provides a rich visual description of the surfaces in the scene. Many texture signatures based on various statistical descriptions and various local measurements have been developed. Existing signatures, in general, are not invariant to 3D geometric transformations, which is a serious limitation for many applications. In this paper we introduce a new texture signature, called the multifractal spectrum (MFS). The MFS is invariant under the bi-Lipschitz map, which includes view-point changes and non-rigid deformations of the texture surface, as well as local affine illumination changes. It provides an efficient framework combining global spatial invariance and local robust measurements. Intuitively, the MFS could be viewed as a “better histogram” with greater robustness to various environmental changes and the advantage of capturing some geometrical distribution information encoded in the texture. Experiments demonstrate that the MFS codes the essential structure of textures with very low dimension, and thus represents an useful tool for texture classification.

title: Locally Invariant Fractal Features for Statistical Texture Classification
abstract: We address the problem of developing discriminative, yet invariant, features for texture classification. Texture variations due to changes in scale are amongst the hardest to handle. One of the most successful methods of dealing with such variations is based on choosing interest points and selecting their characteristic scales [Lazebnik et al. PAMI 2005]. However, selecting a characteristic scale can be unstable for many textures. Furthermore, the reliance on an interest point detector and the inability to evaluate features densely can be serious limitations. Fractals present a mathematically well founded alternative to dealing with the problem of scale. However, they have not become popular as texture features due to their lack of discriminative power. This is primarily because: (a) fractal based classification methods have avoided statistical characterisations of textures (which is essential for accurate analysis) by using global features; and (b) fractal dimension features are unable to distinguish between key texture primitives such as edges, corners and uniform regions. In this paper, we overcome these drawbacks and develop local fractal features that are evaluated densely. The features are robust as they do not depend on choosing interest points or characteristic scales. Furthermore, it is shown that the local fractal dimension is invariant to local bi-Lipschitz transformations whereas its extension is able to correctly distinguish between fundamental texture primitives. Textures are characterised statistically by modelling the full joint PDF of these features. This allows us to develop a texture classification framework which is discriminative, robust and achieves state-of-the-art performance as compared to affine invariant and fractal based methods.

title: Combining powerful local and global statistics for texture description
abstract: A texture descriptor is proposed, which combines local highly discriminative features with the global statistics of fractal geometry to achieve high descriptive power, but also invariance to geometric and illumination transformations. As local measurements SIFT features are estimated densely at multiple window sizes and discretized. On each of the discretized measurements the fractal dimension is computed to obtain the so-called multifractal spectrum, which is invariant to geometric transformations and illumination changes. Finally to achieve robustness to scale changes, a multi-scale representation of the multifractal spectrum is developed using a framelet system, that is, a redundant tight wavelet frame system. Experiments on classification demonstrate that the descriptor outperforms existing methods on the UIUC as well as the UMD high-resolution dataset.

title: A wavelet-based method for multifractal image analysis: From theoretical concepts to experimental applications
abstract: Publisher Summary This chapter highlights a detailed description of the 2D wavelet transform modulus maxima (WTMM) methodology with some test applications to random monofractal and multifractal self-affine surfaces displaying isotropic as well as anisotropic (with respect to space variables) scale similarity properties. As an illustration of the efficiency and reliability of this method, the chapter explains the main results of its application to experimental 2D data in various domains—namely, geophysics, hydrodynamics, and medicine. For practical purposes, the WTMM representation is emphasized as a very efficient and accurate numerical tool for scanning the singularities of fractal landscapes. The 2D WTMM method is used for many purposes in image processing, including edge detection, pattern recognition, and image denoising. The chapter reviews the outcomes of the statistical analysis of high-resolution LANDSAT satellite images of cloudy scenes. This study brings into light the underlying multiplicative structure of marine stratocumulus clouds. The multifractal properties of the stratocumulus radiance fields are further compared to previous experimental estimates performed on velocity and temperature fluctuations in high Reynolds number turbulence. It has also been indicated that the texture discriminatory power of the 2D WTMM method may lead to significant improvement in computer-assisted diagnosis in digitized mammograms.

title: Wavelet Leader multifractal analysis for texture classification
abstract: Image classification often relies on texture characterization. Yet texture characterization has so far rarely been based on a true 2D multifractal analysis. Recently, a 2D wavelet Leader based multifractal formalism has been proposed. It allows to perform an accurate, complete and low computational and memory costs multifractal characterization of textures in images. This contribution describes the first application of such a formalism to a real large size (publicly available) image database, consisting of 25 classes of non traditional textures, with 40 high resolution images in each class. Multifractal attributes are estimated from each image and used as classification features within a standard k nearest neighbor classification procedure. The results reported here show that this Leader based multifractal analysis enables the effective discrimination of different textures, as performances in both classification scores and computational costs compare favorably against those of procedures previously proposed in the literature on the same database.

title: Wavelet-based texture retrieval using generalized Gaussian density and Kullback-Leibler distance
abstract: We present a statistical view of the texture retrieval problem by combining the two related tasks, namely feature extraction (FE) and similarity measurement (SM), into a joint modeling and classification scheme. We show that using a consistent estimator of texture model parameters for the FE step followed by computing the Kullback-Leibler distance (KLD) between estimated models for the SM step is asymptotically optimal in term of retrieval error probability. The statistical scheme leads to a new wavelet-based texture retrieval method that is based on the accurate modeling of the marginal distribution of wavelet coefficients using generalized Gaussian density (GGD) and on the existence a closed form for the KLD between GGDs. The proposed method provides greater accuracy and flexibility in capturing texture information, while its simplified form has a close resemblance with the existing methods which uses energy distribution in the frequency domain to identify textures. Experimental results on a database of 640 texture images indicate that the new method significantly improves retrieval rates, e.g., from 65% to 77%, compared with traditional approaches, while it retains comparable levels of computational complexity.

title: Texture classification using wavelet transform
abstract: Today, texture analysis plays an important role in many tasks, ranging from remote sensing to medical imaging and query by content in large image data bases. The main difficulty of texture analysis in the past was the lack of adequate tools to characterize different scales of textures effectively. The development in multi-resolution analysis such as Gabor and wavelet transform help to overcome this difficulty. This paper describes the texture classification using (i) wavelet statistical features, (ii) wavelet co-occurrence features and (iii) a combination of wavelet statistical features and co-occurrence features of one level wavelet transformed images with different feature databases. It is found that, the results of later method are promising.

title: Wavelet-based colour texture retrieval using the kullback-leibler divergence between bivariate generalized Gaussian models
abstract: We study the retrieval of coloured textures from a database. In a statistical framework we model the heavy-tailed wavelet histograms through a generalized Gaussian distribution (GGD). We choose the Kullback-Leibler divergence (KLD) as a similarity measure and we obtain a closed-form expression for the KLD between two zero-mean bivariate GGDs. This allows us to take into account the rich correlation structure between the colour bands two by two. We show that this results in a considerably improved retrieval rate and, in addition, we demonstrate the superior performance of the bivariate GGD, in comparison with the bivariate Gaussian.

title: Geodesics on the Manifold of Multivariate Generalized Gaussian Distributions with an Application to Multicomponent Texture Discrimination
abstract: We consider the Rao geodesic distance (GD) based on the Fisher information as a similarity measure on the manifold of zero-mean multivariate generalized Gaussian distributions (MGGD). The MGGD is shown to be an adequate model for the heavy-tailed wavelet statistics in multicomponent images, such as color or multispectral images. We discuss the estimation of MGGD parameters using various methods. We apply the GD between MGGDs to color texture discrimination in several classification experiments, taking into account the correlation structure between the spectral bands in the wavelet domain. We compare the performance, both in terms of texture discrimination capability and computational load, of the GD and the Kullback-Leibler divergence (KLD). Likewise, both uni- and multivariate generalized Gaussian models are evaluated, characterized by a fixed or a variable shape parameter. The modeling of the interband correlation significantly improves classification efficiency, while the GD is shown to consistently outperform the KLD as a similarity measure.

title: Efficient Texture Image Retrieval Using Copulas in a Bayesian Framework
abstract: In this paper, we investigate a novel joint statistical model for subband coefficient magnitudes of the dual-tree complex wavelet transform, which is then coupled to a Bayesian framework for content-based image retrieval. The joint model allows to capture the association among transform coefficients of the same decomposition scale and different color channels. It further facilitates to incorporate recent research work on modeling marginal coefficient distributions. We demonstrate the applicability of the novel model in the context of color texture retrieval on four texture image databases and compare retrieval performance to a collection of state-of-the-art approaches in the field. Our experiments further include a thorough computational analysis of the main building blocks, runtime measurements, and an analysis of storage requirements. Eventually, we identify a model configuration with low storage requirements, competitive retrieval accuracy, and a runtime behavior, which enables the deployment even on large image databases.

title: A new texture descriptor using multifractal analysis in multi-orientation wavelet pyramid
abstract: Based on multifractal analysis in wavelet pyramids of texture images, a new texture descriptor is proposed in this paper that implicitly combines information from both spatial and frequency domains. Beyond the traditional wavelet transform, a multi-oriented wavelet leader pyramid is used in our approach that robustly encodes the multi-scale information of texture edgels. Moreover, the resulting texture model shows empirically a strong power law relationship for nature textures, which can be characterized well by multifractal analysis. Combined with a statistics on affine invariant local patches, our proposed texture descriptor is robust to scale and rotation changes, more general geometrical transforms and illumination variations. In addition, the proposed texture descriptor is computationally efficient since it does not require many expensive processing steps, e.g., texton generation and cross-bin comparisons, which are often used by existing methods. As an application, the proposed descriptor is applied to texture classification and the experimental results on several public texture datasets verified the accuracy and efficiency of our descriptor.

title: Classifying Video with Kernel Dynamic Textures
abstract: The dynamic texture is a stochastic video model that treats the video as a sample from a linear dynamical system. The simple model has been shown to be surprisingly useful in domains such as video synthesis, video segmentation, and video classification. However, one major disadvantage of the dynamic texture is that it can only model video where the motion is smooth, i.e. video textures where the pixel values change smoothly. In this work, we propose an extension of the dynamic texture to address this issue. Instead of learning a linear observation function with PCA, we learn a non-linear observation function using kernel-PCA. The resulting kernel dynamic texture is capable of modeling a wider range of video motion, such as chaotic motion (e.g. turbulent water) or camera motion (e.g. panning). We derive the necessary steps to compute the Martin distance between kernel dynamic textures, and then validate the new model through classification experiments on video containing camera motion.

title: Dynamic texture recognition
abstract: Dynamic textures are sequences of images that exhibit some form of temporal stationarity, such as waves, steam, and foliage. We pose the problem of recognizing and classifying dynamic textures in the space of dynamical systems where each dynamic texture is uniquely represented. Since the space is non-linear, a distance between models must be defined We examine three different distances in the space of autoregressive models and assess their power.

title: Probabilistic kernels for the classification of auto-regressive visual processes
abstract: We present a framework for the classification of visual processes that are best modeled with spatio-temporal autoregressive models. The new framework combines the modeling power of a family of models known as dynamic textures and the generalization guarantees, for classification, of the support vector machine classifier. This combination is achieved by the derivation of a new probabilistic kernel based on the Kullback-Leibier divergence (KL) between Gauss-Markov processes. In particular, we derive the KL-kernel for dynamic textures in both 1) the image space, which describes both the motion and appearance components of the spatio-temporal process, and 2) the hidden state space, which describes the temporal component alone. Together, the two kernels cover a large variety of video classification problems, including the cases where classes can differ in both appearance and motion and the cases where appearance is similar for all classes and only motion is discriminant. Experimental evaluation on two databases shows that the new classifier achieves superior performance over existing solutions.

title: Temporal texture and activity recognition
abstract: The goal of this thesis is to demonstrate the utility of low-level motion features for the purpose of recognition. Although motion plays an important role in biological recognition tasks, motion recognition, in general, has received little attention in the literature compared to the volume of work on static object recognition. It has been shown that in some cases, motion information alone is sufficient for human visual system to achieve reliable recognition. Previous attempts at duplicating such capability in machine vision have been based on abstract higher-level models of objects, or have required building intermediate representations such as the trajectories of certain feature points of the object. In this work we demonstrate that motion recognition can be accomplished using lower-level motion features, without the use of abstract object models or trajectory representations. 
First, we show that certain statistical spatial and temporal features derived from the optic flow field have invariant properties, and can be used to classify regional motion patterns such as ripples on water, fluttering of leaves, and chaotic fluid flow. We then present a novel low-level computational approach for detecting and recognizing temporally repetitive movements, such as those characteristic of walking people, or flying birds, on the basis of the periodic nature of their motion signatures. We demonstrate the techniques on a number of real-world image sequences containing complex non-rigid motion patterns. We also show that the proposed techniques are reliable and efficient by implementing a real-time activity recognition system.

title: Dynamic Texture Recognition Using Normal Flow and Texture Regularity
abstract: The processing, description and recognition of dynamic (time-varying) textures are new exciting areas of texture analysis. Many real-world textures are dynamic textures whose retrieval from a video database should be based on both dynamic and static features. In this article, a method for extracting features revealing fundamental properties of dynamic textures is presented. These features are based on the normal flow and on the texture regularity though the sequence. Their discriminative ability is then successfully demonstrated on a full classification process.

title: Phase Based Modelling of Dynamic Textures
abstract: This paper presents a model of spatiotemporal variations in a dynamic texture (DT) sequence. Most recent work on DT modelling represents images in a DT sequence as the responses of a linear dynamical system (LDS) to noise. Despite its merits, this model has limitations because it attempts to model temporal variations in pixel intensities which do not take advantage of global motion coherence. We propose a model that relates texture dynamics to the variation of the Fourier phase, which captures the relationships among the motions of all pixels (i.e. global motion) within the texture, as well as the appearance of the texture. Unlike LDS, our model does not require segmentation or cropping during the training stage, which allows it to handle DT sequences containing a static background. We test the performance of this model on recognition and synthesis of DT's. Experiments with a dataset that we have compiled demonstrate that our phase based model outperforms LDS.

title: View-invariant dynamic texture recognition using a bag of dynamical systems
abstract: In this paper, we consider the problem of categorizing videos of dynamic textures under varying view-point. We propose to model each video with a collection of linear dynamics systems (LDSs) describing the dynamics of spatiotemporal video patches. This bag of systems (BoS) representation is analogous to the bag of features (BoF) representation, except that we use LDSs as feature descriptors. This poses several technical challenges to the BoF framework. Most notably, LDSs do not live in a Euclidean space, hence novel methods for clustering LDSs and computing codewords of LDSs need to be developed. Our framework makes use of nonlinear dimensionality reduction and clustering techniques combined with the Martin distance for LDSs for tackling these issues. Our experiments show that our BoS approach can be used for recognizing dynamic textures in challenging scenarios, which could not be handled by existing dynamic texture recognition methods.

title: Dynamic texture recognition based on distributions of spacetime oriented structure
abstract: This paper addresses the challenge of recognizing dynamic textures based on their observed visual dynamics. Typically, the term dynamic texture is used with reference to image sequences of various natural processes that exhibit stochastic dynamics (e.g., smoke, water and windblown vegetation); although, it applies equally well to images of simpler dynamics when analyzed in terms of aggregate region properties (e.g., uniform motion of elements in traffic video). In this paper, a novel approach to dynamic texture representation and an associated recognition method are proposed. The approach pursued here recognizes dynamic textures based on matching distributions (histograms) of spacetime orientation structure. Empirical evaluation on a standard database with controls to remove the effects of identical viewpoint demonstrates that the proposed approach achieves superior performance over alternative state-of-the-art methods.

title: Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
abstract: Dynamic texture (DT) is an extension of texture to the temporal domain. Description and recognition of DTs have attracted growing attention. In this paper, a novel approach for recognizing DTs is proposed and its simplifications and extensions to facial image analysis are also considered. First, the textures are modeled with volume local binary patterns (VLBP), which are an extension of the LBP operator widely used in ordinary texture analysis, combining motion and appearance. To make the approach computationally simple and easy to extend, only the co-occurrences of the local binary patterns on three orthogonal planes (LBP-TOP) are then considered. A block-based method is also proposed to deal with specific dynamic events such as facial expressions in which local information and its spatial locations should also be taken into account. In experiments with two DT databases, DynTex and Massachusetts Institute of Technology (MIT), both the VLBP and LBP-TOP clearly outperformed the earlier approaches. The proposed block-based method was evaluated with the Cohn-Kanade facial expression database with excellent results. The advantages of our approach include local processing, robustness to monotonic gray-scale changes, and simple computation

title: Qualitative Spatiotemporal Analysis Using an Oriented Energy Representation
abstract: This paper presents an approach to representing and analyzing spatiotemporal information in support of making qualitative, yet semantically meaningful distinctions at the earliest stages of processing. A small set of primitive classes of spatiotemporal structure are proposed that correspond to categories of stationary, coherently moving, incoherently moving, flickering, scintillating and "too unstructured to support further inference". It is shown how these classes can be represented and distinguished in a uniform fashion in terms of oriented energy signatures. Further, empirical results are presented that illustrate the use of the approach in application to natural imagery. The importance of the described work is twofold: (i) From a theoretical point of view a semantically meaningful decomposition of spatiotemporal information is developed. (ii) From a practical point of view, the developed approach has the potential to impact real world image understanding and analysis applications.  The approach could be used to support early focus of attention and cueing mechanisms that guide subsequent activities by an intelligent agent; the approach could provide the representational substrate for indexing video and other spatiotemporal data.

title: Clustering dynamic textures with the hierarchical EM algorithm
abstract: The dynamic texture (DT) is a probabilistic generative model, defined over space and time, that represents a video as the output of a linear dynamical system (LDS). The DT model has been applied to a wide variety of computer vision problems, such as motion segmentation, motion classification, and video registration. In this paper, we derive a new algorithm for clustering DT models that is based on the hierarchical EM algorithm. The proposed clustering algorithm is capable of both clustering DTs and learning novel DT cluster centers that are representative of the cluster members, in a manner that is consistent with the underlying generative probabilistic model of the DT. We then demonstrate the efficacy of the clustering algorithm on several applications in motion analysis, including hierarchical motion clustering, semantic motion annotation, and bag-of-systems codebook generation.

title: Maximum Margin Distance Learning for Dynamic Texture Recognition
abstract: The range space of dynamic textures spans spatiotemporal phenomena that vary along three fundamental dimensions: spatial texture, spatial texture layout, and dynamics. By describing each dimension with appropriate spatial or temporal features and by equipping it with a suitable distance measure, elementary distances (one for each dimension) between dynamic texture sequences can be computed. In this paper, we address the problem of dynamic texture (DT) recognition by learning linear combinations of these elementary distances. By learning weights to these distances, we shed light on how "salient" (in a discriminative manner) each DT dimension is in representing classes of dynamic textures. To do this, we propose an efficient maximum margin distance learning (MMDL) method based on the Pegasos algorithm , for both classindependent and class-dependent weight learning. In contrast to popular MMDL methods, which enforce restrictive distance constraints and have a computational complexity that is cubic in the number of training samples, we show that our method, called DL-PEGASOS, can handle more general distance constraints with a computational complexity that can be made linear. When class dependent weights are learned, we show that, for certain classes of DTs, spatial texture features are dominantly "salient", while for other classes, this "saliency" lies in their temporal features. Furthermore, DL-PEGASOS outperforms state-of-the-art recognition methods on the UCLA benchmark DT dataset. By learning class independent weights, we show that this benchmark does not offer much variety along the three DT dimensions, thus, motivating the proposal of a new DT dataset, called DynTex++.

title: Direct computation of shape cues using scale-adapted spatial derivative operators
abstract: This paper addresses the problem of computing cues to the three-dimensional structure of surfaces in the world directly from the local structure of the brightness pattern of either a single monocular image or a binocular image pair.It is shown that starting from Gaussian derivatives of order up to two at a range of scales in scale-space, local estimates of (i) surface orientation from monocular texture foreshortening, (ii) surface orientation from monocular texture gradients, and (iii) surface orientation from the binocular disparity gradient can be computed without iteration or search, and by using essentially the same basic mechanism.The methodology is based on a multi-scale descriptor of image structure called the windowed second moment matrix, which is computed with adaptive selection of both scale levels and spatial positions. Notably, this descriptor comprises two scale parameters; a local scale parameter describing the amount of smoothing used in derivative computations, and an integration scale parameter determining over how large a region in space the statistics of regional descriptors is accumulated.Experimental results for both synthetic and natural images are presented, and the relation with models of biological vision is briefly discussed.

title: Wavelet leader based multifractal analysis
abstract: We introduce a new multifractal formalism based on wavelet leaders and study its properties. Comparing it against previously formulated wavelet coefficient based multifractal formalism, we show first that this wavelet leader based formalism allows the multifractal spectrum to be obtained over its entire range, and second that it does not cease to hold when applied to processes embodying unusual chirp-type (or oscillating) singularities (as opposed to the more common cusp-type ones). We illustrate these results and properties on four examples of multifractal deterministic functions or stochastic processes containing a graduation of the major difficulties. We show that this new multifractal formalism benefits from excellent theoretical and practical performance. Matlab routines implementing it are available upon request.

title: A wavelet tour of signal processing
abstract: Introduction to a Transient World. Fourier Kingdom.  Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding.

title: Wavelet Techniques in Multifractal Analysis
abstract: We show how estimates on the Haudsdorff dimensions of the Hoelder singularities of a function can be derived from the properties of its expansion on an orthonormal wavelet basis. The validity of the multifractal formalism is examined and results concerning the genericity of multifractality in a given function space are deduced.

title: Techniques in fractal geometry
abstract: The Ergodic Theorem and Fractals. The Renewal Theorem and Fractals. Martingales and Fractals.  Dimensions of Measures. Some Multifractal Analysis. Fractals and Differential Equations.

title: Towards correct and informative evaluation methodology for texture classification under varying viewpoint and illumination
abstract: 3D texture classification under varying viewpoint and illumination has been a vivid research topic, and many methods have been developed. It is crucial that these methods be compared using an unbiased evaluation methodology. The most frequently employed methodologies use images from the Columbia-Utrecht Reflectance and Texture Database. These methodologies construct the training and test sets to be disjoint in the imaging parameters, but do not separate them spatially because they use images of the same surface patch for both. We perform a series of experiments which show that such practice leads to overestimation of classifier performance and distorts experimental findings. To correct that, we accurately register the images across all imaging conditions and split the surface patches to parts. The training and testing is then done on spatially disjoint parts. We show that such methodology gives a more realistic assessment of classifier performance. The sample annotations for all images are publicly available.

title: Robust wide-baseline stereo from maximally stable extremal regions
abstract: The wide-baseline stereo problem, i.e. the problem of establishing correspondences between a pair of images taken from different viewpoints is studied. A new set of image elements that are put into correspondence, the so called extremal regions , is introduced. Extremal regions possess highly desirable properties: the set is closed under (1) continuous (and thus projective) transformation of image coordinates and (2) monotonic transformation of image intensities. An efficient (near linear complexity) and practically fast detection algorithm (near frame rate) is presented for an affinely invariant stable subset of extremal regions, the maximally stable extremal regions (MSER). A new robust similarity measure for establishing tentative correspondences is proposed. The robustness ensures that invariants from multiple measurement regions (regions obtained by invariant constructions from extremal regions), some that are significantly larger (and hence discriminative) than the MSERs, may be used to establish tentative correspondences. The high utility of MSERs, multiple measurement regions and the robust metric is demonstrated in wide-baseline experiments on image pairs from both indoor and outdoor scenes. Significant change of scale (3.5×), illumination conditions, out-of-plane rotation, occlusion, locally anisotropic scale change and 3D translation of the viewpoint are all present in the test problems. Good estimates of epipolar geometry (average distance from corresponding points to the epipolar line below 0.09 of the inter-pixel distance) are obtained.

title: Scale & Affine Invariant Interest Point Detectors
abstract: In this paper we propose a novel approach for detecting interest points invariant to scale and affine transformations. Our scale and affine invariant detectors are based on the following recent results: (1) Interest points extracted with the Harris detector can be adapted to affine transformations and give repeatable results (geometrically stable). (2) The characteristic scale of a local structure is indicated by a local extremum over scale of normalized derivatives (the Laplacian). (3) The affine shape of a point neighborhood is estimated based on the second moment matrix.Our scale invariant detector computes a multi-scale representation for the Harris interest point detector and then selects points at which a local measure (the Laplacian) is maximal over scales. This provides a set of distinctive points which are invariant to scale, rotation and translation as well as robust to illumination changes and limited changes of viewpoint. The characteristic scale determines a scale invariant region for each point. We extend the scale invariant detector to affine invariance by estimating the affine shape of a point neighborhood. An iterative algorithm modifies location, scale and neighborhood of each point and converges to affine invariant points. This method can deal with significant affine transformations including large scale changes. The characteristic scale and the affine shape of neighborhood determine an affine invariant region for each point.We present a comparative evaluation of different detectors and show that our approach provides better results than existing methods. The performance of our detector is also confirmed by excellent matching results; the image is described by a set of scale/affine invariant descriptors computed on the regions associated with our points.

title: Feature Detection with Automatic Scale Selection
abstract: The fact that objects in the world appear in different ways depending on the scale of observation has important implications if one aims at describing them. It shows that the notion of scale is of utmost importance when processing unknown measurement data by automatic methods. In their seminal works, Witkin (1983) and Koenderink (1984) proposed to approach this problem by representing image structures at different scales in a so-called scale-space representation. Traditional scale-space theory building on this work, however, does not address the problem of how to select local appropriate scales for further analysis. This article proposes a systematic methodology for dealing with this problem. A framework is presented for generating hypotheses about interesting scale levels in image data, based on a general principle stating that local extrema over scales of different combinations of γ-normalized derivatives are likely candidates to correspond to interesting structures. Specifically, it is shown how this idea can be used as a major mechanism in algorithms for automatic scale selection, which adapt the local scales of processing to the local image structure.Support for the proposed approach is given in terms of a general theoretical investigation of the behaviour of the scale selection method under rescalings of the input pattern and by integration with different types of early visual modules, including experiments on real-world and synthetic data. Support is also given by a detailed analysis of how different types of feature detectors perform when integrated with a scale selection mechanism and then applied to characteristic model patterns. Specifically, it is described in detail how the proposed methodology applies to the problems of blob detection, junction detection, edge detection, ridge detection and local frequency estimation.In many computer vision applications, the poor performance of the low-level vision modules constitutes a major bottleneck. It is argued that the inclusion of mechanisms for automatic scale selection is essential if we are to construct vision systems to automatically analyse complex unknown environments.

title: On the Significance of Real-World Conditions for Material Classification
abstract: Classifying materials from their appearance is a challenging problem, especially if illumination and pose conditions are permitted to change: highlights and shadows caused by 3D structure can radically alter a sample’s visual texture. Despite these difficulties, researchers have demonstrated impressive results on the CUReT database which contains many images of 61 materials under different conditions. A first contribution of this paper is to further advance the state-of-the-art by applying Support Vector Machines to this problem. To our knowledge, we record the best results to date on the CUReT database.

title: Support Vector Machines for 3D Object Recognition
abstract: Support vector machines (SVMs) have been recently proposed as a new technique for pattern recognition. Intuitively, given a set of points which belong to either of two classes, a linear SVM finds the hyperplane leaving the largest possible fraction of points of the same class on the same side, while maximizing the distance of either class from the hyperplane. The hyperplane is determined by a subset of the points of the two classes, named support vectors, and has a number of interesting theoretical properties. In this paper, we use linear SVMs for 3D object recognition. We illustrate the potential of SVMs on a database of 7200 images of 100 different objects. The proposed system does not require feature extraction and performs recognition on images regarded as points of a space of high dimension without estimating pose. The excellent recognition rates achieved in all the performed experiments indicate that SVMs are well-suited for aspect-based recognition.

title: Contour and Texture Analysis for Image Segmentation
abstract: This paper provides an algorithm for partitioning grayscale images into disjoint regions of coherent brightness and texture. Natural images contain both textured and untextured regions, so the cues of contour and texture differences are exploited simultaneously. Contours are treated in the intervening contour framework, while texture is analyzed using textons. Each of these cues has a domain of applicability, so to facilitate cue combination we introduce a gating operator based on the texturedness of the neighborhood at a pixel. Having obtained a local measure of how likely two nearby pixels are to belong to the same region, we use the spectral graph theoretic framework of normalized cuts to find partitions of the image into regions of coherent texture and brightness. Experimental results on a wide range of images are shown.

title: Multifractals, texture, and image analysis
abstract: Image analysis using texture and multifractal paradigms is addressed. Multifractal theory and its application to image description are discussed, and it is shown that this approach allows the discrete signal to be worked on directly. A system for texture classification that is based on a learning scheme and does not make use of any a priori model is introduced. Image segmentation is then considered, and the notion of mixed classes, which allows accurate detection of texture boundaries on complex images, is introduced.<<ETX>>

title: Log-Polar Wavelet Energy Signatures for Rotation and Scale Invariant Texture Classification
abstract: Classification of texture images is important in image analysis and classification. This paper proposes an effective scheme for rotation and scale invariant texture classification using log-polar wavelet signatures. The rotation and scale invariant feature extraction for a given image involves applying a log-polar transform to eliminate the rotation and scale effects, but at same time produce a row shifted log-polar image, which is then passed to an adaptive row shift invariant wavelet packet transform to eliminate the row shift effects. So, the output wavelet coefficients are rotation and scale invariant. The adaptive row shift invariant wavelet packet transform is quite efficient with only O(n /spl middot/ log n) complexity. A feature vector of the most dominant log-polar wavelet energy signatures extracted from each subband of wavelet coefficients is constructed for rotation and scale invariant texture classification. In the experiments, we employed a Mahalanobis classifier to classify a set of 25 distinct natural textures selected from the Brodatz album. The experimental results, based on different testing data sets for images with different orientations and scales, show that the proposed classification scheme using log-polar wavelet signatures outperforms two other texture classification methods, its overall accuracy rate for joint rotation and scale invariance being 90.8 percent, demonstrating that the extracted energy signatures are effective rotation and scale invariant features. Concerning its robustness to noise, the classification scheme also performs better than the other methods.