nils-herrmann/clean_scirepeval huggingface dataset:
What are Textons?

Textons refer to fundamental micro-structures in natural images (and videos) and are considered as the atoms of pre-attentive human visual perception (Julesz, 1981). Unfortunately, the word "texton" remains a vague concept in the literature for lack of a good mathematical model. In this article, we first present a three-level generative image model for learning textons from texture images. In this model, an image is a superposition of a number of image bases selected from an over-complete dictionary including various Gabor and Laplacian of Gaussian functions at various locations, scales, and orientations. These image bases are, in turn, generated by a smaller number of texton elements, selected from a dictionary of textons. By analogy to the waveform-phoneme-word hierarchy in speech, the pixel-base-texton hierarchy presents an increasingly abstract visual description and leads to dimension reduction and variable decoupling. By fitting the generative model to observed images, we can learn the texton dictionary as parameters of the generative model. Then the paper proceeds to study the geometric, dynamic, and photometric structures of the texton representation by further extending the generative model to account for motion and illumination variations. (1) For the geometric structures, a texton consists of a number of image bases with deformable spatial configurations. The geometric structures are learned from static texture images. (2) For the dynamic structures, the motion of a texton is characterized by a Markov chain model in time which sometimes can switch geometric configurations during the movement. We call the moving textons as "motons". The dynamic models are learned using the trajectories of the textons inferred from video sequence. (3) For photometric structures, a texton represents the set of images of a 3D surface element under varying illuminations and is called a "lighton" in this paper. We adopt an illumination-cone representation where a lighton is a texton triplet. For a given light source, a lighton image is generated as a linear sum of the three texton bases. We present a sequence of experiments for learning the geometric, dynamic, and photometric structures from images and videos, and we also present some comparison studies with K-mean clustering, sparse coding, independent component analysis, and transformed component analysis. We shall discuss how general textons can be learned from generic natural images.
A generative model of human hair for hair sketching

Human hair is a very complex visual pattern whose representation is rarely studied in the vision literature despite its important role in human recognition. In this paper, we propose a generative model for hair representation and hair sketching, which is far more compact than the physically based models in graphics. We decompose a color hair image into three bands: a color band (a) (by Luv transform), a low frequency band (b) for lighting variations, and a high frequency band (c) for the hair pattern. Then we propose a three level generative model for the hair image (c). In this model, image (c) is generated by a vector field (d) that represents hair orientation, gradient strength, and directions; and this vector field is in turn generated by a hair sketch layer (e). We identify five types of primitives for the hair sketch each specifying the orientations of the vector field on the two sides of the sketch. With the five-layer representation (a-e) computed, we can reconstruct vivid hair images and generate hair sketches. We test our algorithm on a large data set of hairs and some results are reported in the experiments.
Cloth representation by shape from shading with shading primitives

Cloth is a complex visual pattern with flexible 3D shape and illumination variations. Computing the 3D shape of cloth from a single image is of great interest to both computer graphics and vision researches. However, the acquisition of 3D cloth shape by shape from shading (SFS) is still a challenge. In this paper, we present a two-layer generative model for representing both the 2D cloth image and the 3D cloth surface. The first layer represents all the folds on cloth, which are called "shading primitives" in (Haddon and Forsyth, 1998), and thus captures the overall "skeleton structures" of cloth. We learn a number of typical 3D fold primitives using some training images obtained through photometric stereo. The 3D fold primitives yield a dictionary of 2D shading primitives/or cloth images. The second layer represents non-fold parts with very smooth (often flat) surface or shading, which interpolates the primitives in the first layer with a smoothness prior like conventional SFS. Then we present an algorithm called "cloth sketching" to find all the shading primitives on cloth image and simultaneously recover their 3D shape by fitting to the 3D fold primitives. Our sketch representation can be viewed as a 2-layer Markov random field (MRF), and it introduces some prior knowledge on the folds and has lower dimension and is more robust than the traditional shape-fmm-shading representation which assumes a MRF model on pixels. We show a number of experiments with satisfactory results in comparison to previous work.
Learning Generic Prior Models For Visual Computation-Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference

This paper presents a novel theory f o r learning generic prior models f r o m a set of observed natural images based o n a minimax entropy theory that the authors studied in modeling textures. W e start by studying the statistics of natural images including the scale invariant properties, then generic prior models were learnt t o duplicate the observed statistics. The learned Gibbs distributions confirm and improve the forms of existing prior models. More interestingly inverted potentials are found to be necessary, and such potentials f o r m patterns and enhance preferred image features. The learned model is compared with existing prior models in experiments of image restoration.
Cluster Sampling and Its Applications in Image Analysis

Markov chain Monte Carlo (MCMC) methods have been used in many fields (physics, chemistry, biology, and computer science) for simulation, inference, and optimization. The essence of these methods is to simulate a Markov chain whose state X follows a target probability X ∼ π(X). In many applications, π(X) is defined on a graph G whose vertices represent elements in the system and whose edges represent the connectivity of the elements. X is a vector of variables on the vertices which often take discrete values called labels or colors. Designing rapid mixing Markov chain is a challenging task when the variables in the graph are strongly coupled. Methods, like the single-site Gibbs sampler, often experience long waiting time. A well-celebrated algorithm for sampling on graphs is the Swendsen-Wang (1987) (SW) method. The SWmethod finds a cluster of vertices as a connected component after turning off some edges probabilistically, and flips the color of the cluster as a whole. It is shown to mix rapidly under certain conditions. Unfortunately, the SW method is only applicable to the Ising/Potts models and slow down critically in the presence of "external fields" i.e. likelihood in Bayesian inference.In this paper, we present a general cluster sampling method which achieves the following objectives. Firstly, it extends the SW algorithm to general Bayesian inference on graphs. Especially we focus a number of image analysis problems where the graph sizes are in the order of O(103) — O(106) with O(1) connectivity. Secondly, the edge probability for clustering the vertices are discriminative probabilities computed from data. Empirically such data driven clustering leads to much improved efficiency. Thirdly, we present a generalized Gibbs sampler which samples the color of a cluster according to a conditional probability (like the Gibbs sampler) weighted by a product of edge probabilities. Fourthly, we design the algorithm to work on multi-grid and multi-level graphs. The algorithm is tested on typical problems in image analysis, such as image segmentation and motion analysis, In our experiments, the algorithm is O(102) orders faster than the single-site Gibbs sampler. In the literature, there are several ways for interpreting the SW-method which leads to various analyses or generalizations, including random cluster model (RCM), data augmentation, slice sampling, and partial decoupling. We take a different route by interpreting SW as a Metropolis-Hastings step with auxiliary variables for proposing the moves or we can view it as a generalized hit-and-run method.
Equivalence of Julesz Ensembles and FRAME Models

In the past thirty years, research on textures has been pursued along two different lines. The first line of research, pioneered by Julesz (1962, , seeks essential ingredients in terms of features and statistics in human texture perception. This leads us to a mathematical definition of textures in terms of Julesz ensembles (Zhu et al.,  2000). A Julesz ensemble is a set of images that share the same value of some basic feature statistics. Images in the Julesz ensemble are defined on a large image lattice (a mathematical idealization being Z2) so that exact constraint on feature statistics makes sense. The second line of research studies Markov random field (MRF) models that characterize texture patterns on finite (or small) image lattice in a statistical way. This leads us to a general class of MRF models called FRAME (Filter, Random field, And Maximum Entropy) (Zhu et al., Neural Computation, 9:1627–1660). In this article, we bridge the two lines of research by the fundamental principle of equivalence of ensembles in statistical mechanics (Gibbs, 1902, Elementary Principles of Statistical Mechanics.  We show that 1). As the size of the image lattice goes to infinity, a FRAME model concentrates its probability mass uniformly on a corresponding Julesz ensemble. Therefore, the Julesz ensemble characterizes the global statistical property of the FRAME model; 2). For a large image randomly sampled from a Julesz ensemble, any local patch of the image given its environment follows the conditional distribution specified by a corresponding FRAME model. Therefore, the FRAME model describes the local statistical property of the Julesz ensemble, and is an inevitable texture model on finite (or small) lattice if texture perception is decided by feature statistics. The key to derive these results is the large deviation estimate of the volume of (or the number of images in) the Julesz ensemble, which we call the entropy function. Studying the equivalence of ensembles provides deep insights into questions such as the origin of MRF models, typical images of statistical models, and error rates in various texture related vision tasks (Yuille and Coughlan,  2000). The second thrust of this paper is to study texture distance based on the texture models of both small and large lattice systems. We attempt to explain the asymmetry phenomenon observed in texture “pop-out” experiments by the asymmetry of Kullback-Leibler divergence. Our results generalize the traditional signal detection theory (Green and Swets, 1988, Signal Detection Theory and Psychophysics, Peninsula Publishing) for distance measures from iid cases to random fields. Our theories are verified by two groups of computer simulation experiments.
Modeling complex motion by tracking and editing hidden Markov graphs

We propose a generative model for representing complex motion, such as wavy river, dancing fire and dangling cloth. Our generative method consists of four components: (1) A photometric model using primal sketch[8] which transfers an image into an attribute graph representation. Each vertex of the graph is a scaled and oriented image patch selected from a dictionary. The graph connects and aligns these patches. (2) A geometric model which characterizes the deformation of the attribute graph. (3) A dynamic model, which specifies the motion dynamics of these vertices (patches) and their interactions in the form of coupled Markov chains. (4) A topological model, which interprets the graph topological changes over time. We learn this generative model by a stochastic gradient algorithm implemented by Markov Chain Monte Carlo (MCMC) sampling. This method is shown to be effective in handling the topological changes of graphs. The correctness of the learned model is verified by the low-dimension reconstruction of the original image as well as by the realistic motion sequences it synthesized.
Multigrid and multi-level Swendsen-Wang cuts for hierarchic graph partition

Many vision tasks can be formulated as partitioning an adjacency graph through optimizing a Bayesian posterior probability p defined on the partition-space. In this paper two approaches are proposed to generalize the Swendsen-Wang cut algorithm [A. Barbu and S.C. Zhu 2003] for sampling p. The first method is called multigrid SW-cut which runs SW-cut within a sequence of local "attentional" windows and thus simulates conditional probabilities of p in the partition space. The second method is called multi-level SW-cut which projects the adjacency graph into a hierarchical representation with each vertex in the high level graph corresponding to a sub-graph at the low level, and runs SW-cut at each level. Thus it simulates conditional probabilities of p at the higher level. Both methods are shown to observe the detailed balance equation and thus provide flexibilities in sampling the posterior probability p. We demonstrate the algorithms in image and motion segmentation with three levels (see Fig.1), and compare the speed improvement of the proposed methods.
Information Scaling Laws in Natural Scenes

In natural scenes, objects and patterns can appear at a wide variety of distances from the viewer. For the same visual pattern viewed at different distances, both the image and our perception of the pattern change over distance. We call the change of the image over distance as image scaling, and the change of our perception over distance as information scaling. While image scaling can be accounted for by the state space theory, information scaling has not been mathematically studied in computer vision. In this paper, we prove two information scaling laws: 1) the entropy rate of the image changes over distance, and 2) the entropy of the posterior distribution of the pattern also changes over distance. These two information scaling laws have deep implications in computer vision: they call for different models of the same visual pattern at different distances, as well as a model transition mechanism for switching models over different distance/scale regimes.
Title From Information Scaling of Natural Images to Regimes of Statistical Models Permalink

Computer vision can be considered a highly specialized data collection and data analysis problem. We need to understand the special properties of image data in order to construct statistical models for representing the wide variety of image patterns. One special property of vision that distinguishes itself from other sensory data such as speech data is that distance or scale plays a profound role in image data. More specifically, visual objects and patterns can appear at a wide range of distances or scales, and the same visual pattern appearing at different distances or scales produces different image data with different statistical properties, thus entails different regimes of statistical models. In particular, we show that the entropy rate of the image data changes over the viewing distance (as well as the camera resolution). Moreover, the inferential uncertainty changes with viewing distance too. We call these changes information scaling. From this perspective, we examine both empirically and theoretically two prominent and yet largely isolated research themes in image modeling literature, namely, wavelet sparse coding and Markov random fields. Our results indicate that the two models are appropriate on two different entropy regimes: sparse coding targets the low entropy regime, whereas the random fields are suitable for the high entropy regime. Because of information scaling, both models are necessary for representing and interpreting image intensity patterns in the whole entropy range, and information scaling triggers transitions between these two regimes of models. This motivates us to propose a full-zoom primal sketch model that integrates both sparse coding and Markov random fields. In this model, local image intensity patterns are classified into “sketchable regime” and “non-sketchable regime” by a sketchability criterion. In the sketchable regime, the image data are represented deterministically by highly parametrized sketch primitives. In the non-sketchable regime, the image data are characterized by Markov random fields whose sufficient statistics summarize computational results from failed attempts of sparse coding. The contribution of our work is two folded. First, information scaling provides a dimension to chart the space of natural images. Second, the full-zoom modeling scheme provides a natural integration of sparse coding and Markov random fields, thus enables us to develop a new and richer class of statistical models.
Modeling Visual Patterns by Integrating Descriptive and Generative Methods

This paper presents a class of statistical models that integrate two statistical modeling paradigms in the literature: (I) Descriptive methods, such as Markov random fields and minimax entropy learning (Zhu, S.C., Wu, Y.N., and Mumford, D. 1997. Neural Computation, 9(8)), and (II) Generative methods, such as principal component analysis, independent component analysis (Bell, A.J. and Sejnowski, T.J. 1997. ), transformed component analysis (Frey, B. and Jojic, N. 1999. ICCV), wavelet coding (Mallat, S. and Zhang, Z. 1993. ; Chen, S., Donoho, D., and Saunders, M.A. 1999. ), and sparse coding (Olshausen, B.A. and Field, D.J. 1996.  381:607–609; Lewicki, M.S. and Olshausen, B.A. 1999. JOSA, A. 16(7):1587–1601). In this paper, we demonstrate the integrated framework by constructing a class of hierarchical models for texton patterns (the term “texton” was coined by psychologist Julesz in the early 80s). At the bottom level of the model, we assume that an observed texture image is generated by multiple hidden “texton maps”, and textons on each map are translated, scaled, stretched, and oriented versions of a window function, like mini-templates or wavelet bases. The texton maps generate the observed image by occlusion or linear superposition. This bottom level of the model is generative in nature. At the top level of the model, the spatial arrangements of the textons in the texton maps are characterized by minimax entropy principle, which leads to embellished versions of Gibbs point process models (Stoyan, D., Kendall, W.S., and Mecke,  Stochastic Geometry and its Applications). The top level of the model is descriptive in nature. We demonstrate the integrated model by a set of experiments.
On the Relationship Between Image and Motion Segmentation

In this paper we present a generative model for image sequences, which can be applied to motion segmentation and tracking, and to image sequence compression. The model consists of regions of relatively constant color that have a motion model explaining their motion in time. At each frame, the model can allow accretion and deletion of pixels. We also present an algorithm for maximizing the posterior probability of the image sequence model, based on the recently introduced Swendsen-Wang Cuts algorithm. We show how one can use multiple cues and model switching in a reversible manner to make better bottom-up proposals. The algorithm works on the 3d spatiotemporal pixel volume to reassign entire trajectories of constant color in very few steps, while maintaining detailed balance.
Range image segmentation by an effective jump-diffusion method

This paper presents an effective jump-diffusion method for segmenting a range image and its associated reflectance image in the Bayesian framework. The algorithm works on complex real-world scenes (indoor and outdoor), which consist of an unknown number of objects (or surfaces) of various sizes and types, such as planes, conics, smooth surfaces, and cluttered objects (like trees and bushes). Formulated in the Bayesian framework, the posterior probability is distributed over a solution space with a countable number of subspaces of varying dimensions. The algorithm simulates Markov chains with both reversible jumps and stochastic diffusions to traverse the solution space. The reversible jumps realize the moves between subspaces of different dimensions, such as switching surface models and changing the number of objects. The stochastic Langevin equation realizes diffusions within each subspace. To achieve effective computation, the algorithm precomputes some importance proposal probabilities over multiple scales through Hough transforms, edge detection, and data clustering. The latter are used by the Markov chains for fast mixing. The algorithm is tested on 100 1D simulated data sets for performance analysis on both accuracy and speed. Then, the algorithm is applied to three data sets of range images under the same parameter setting. The results are satisfactory in comparison with manual segmentations.
Analysis and synthesis of textured motion: particles and waves

Natural scenes contain a wide range of textured motion phenomena which are characterized by the movement of a large amount of particle and wave elements, such as falling snow, wavy water, and dancing grass. In this paper, we present a generative model for representing these motion patterns and study a Markov chain Monte Carlo algorithm for inferring the generative representation from observed video sequences. Our generative model consists of three components. The first is a photometric model which represents an image as a linear superposition of image bases selected from a generic and overcomplete dictionary. The dictionary contains Gabor and LoG bases for point/particle elements and Fourier bases for wave elements. These bases compete to explain the input images and transfer them to a token (base) representation with an O(10/sup 2/)-fold dimension reduction. The second component is a geometric model which groups spatially adjacent tokens (bases) and their motion trajectories into a number of moving elements-called "motons". A moton is a deformable template in time-space representing a moving element, such as a falling snowflake or a flying bird. The third component is a dynamic model which characterizes the motion of particles, waves, and their interactions. For example, the motion of particle objects floating in a river, such as leaves and balls, should be coupled with the motion of waves. The trajectories of these moving elements are represented by coupled Markov chains. The dynamic model also includes probabilistic representations for the birth/death (source/sink) of the motons. We adopt a stochastic gradient algorithm for learning and inference. Given an input video sequence, the algorithm iterates two steps: 1) computing the motions and their trajectories by a number of reversible Markov chain jumps, and 2) learning the parameters that govern the geometric deformations and motion dynamics. Novel video sequences are synthesized from the learned models and, by editing the model parameters, we demonstrate the controllability of the generative model.