nils-herrmann/clean_scirepeval huggingface dataset:
Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling

This article presents a statistical theory for texture modeling. This theory combines filtering theory and Markov random field modeling through the maximum entropy principle, and interprets and clarifies many previous concepts and methods for texture analysis and synthesis from a unified point of view. Our theory characterizes the ensemble of images I with the same texture appearance by a probability distribution f(I) on a random field, and the objective of texture modeling is to make inference about f(I), given a set of observed texture examples.In our theory, texture modeling consists of two steps. (1) A set of filters is selected from a general filter bank to capture features of the texture, these filters are applied to observed texture images, and the histograms of the filtered images are extracted. These histograms are estimates of the marginal distributions of f( I). This step is called feature extraction. (2) The maximum entropy principle is employed to derive a distribution p(I), which is restricted to have the same marginal distributions as those in (1). This p(I) is considered as an estimate of f( I). This step is called feature fusion. A stepwise algorithm is proposed to choose filters from a general filter bank. The resulting model, called FRAME (Filters, Random fields And Maximum Entropy), is a Markov random field (MRF) model, but with a much enriched vocabulary and hence much stronger descriptive ability than the previous MRF models used for texture modeling. Gibbs sampler is adopted to synthesize texture images by drawing typical samples from p(I), thus the model is verified by seeing whether the synthesized texture images have similar visual appearances to the texture images being modeled. Experiments on a variety of 1D and 2D textures are described to illustrate our theory and to show the performance of our algorithms. These experiments demonstrate that many textures which are previously considered as from different categories can be modeled and synthesized in a common framework.
Title Multiscale Generative Model of Human Faces Permalink

In this paper, we propose a framework for modelling human faces over scales. As a person walks towards the camera, more details of the face will be revealed and thus more random variables and parameters have to be introduced. Accordingly, a series of existing generative models are organized as five regimes, which form nested probabilistic families. The generative model in higher regime is augmented by (1) adding more latent variables, features extractors, and (2) enlarging the dictionary of description, e.g. PCA bases, local parts or sketch patches. The minimum description length(MDL) is used as a criterion for the model selection and transition. As observed in our experiment, the optimal model switches among the different regimes when the scale changes. A sequence of tasks, such as face detection, recognition, sketching and super-resolution etc. can be accomplished based on the models in the different regimes.
Learning Generic Prior Models for Visual Computation

This paper presents a novel theory f o r learning generic prior models f r o m a set of observed natural images based o n a minimax entropy theory that the authors studied in modeling textures. W e start by studying the statistics of natural images including the scale invariant properties, then generic prior models were learnt t o duplicate the observed statistics. The learned Gibbs distributions confirm and improve the forms of existing prior models. More interestingly inverted potentials are found to be necessary, and such potentials f o r m patterns and enhance preferred image features. The learned model is compared with existing prior models in experiments of image restoration.
GRADE: Gibbs Reaction and Diffusion Equations

Recently there have been increasing interests in using nonlinear PDEs for applications in computer vision an.d image processing, in this paper, we propose a general statistical framework for designing a new class of PDEs. For a given application, a Markov random jield model p(1) as learned according to the minimax entropy principle studied in[25][26], so that p(1) should characterize the ensemble of images in our application. p(1) is a Gibbs distribution whose energy terms can be divided into two categories. Subsequently the partial differential equations given by gradient descent on the Gibbs potential are essentially reaction-diffusion equations, where the energy terms in one category produce anisotropic diffusion while the inverted energy terms in the second category produce reaction associated with pattern formation. W e call this new class of PDEs the Gibbs Reaction And’Diffusion Equations-GRADE and we demonstrate experiments where G R A D E are used for texture pattern formation, denoising, image enhancement, and clutter removal.
Automatic Single View Building Reconstruction by Integrating Segmentation

In this paper, we propose a stochastic algorithm using Markov chain Monte Carlo (MCMC) to automatically reconstruct buildings from a single image of architectural scenes by integrating segmentation and reconstruction. Buildings are modelled by two families of generative models: One is parameterized geometric primitives (e.g. boxes, prisms) for 3D structures of buildings. The other is image models for appearances of building surfaces. All the other objects except buildings in the scene are modelled as a 3D background plane with some appearance. Regarding one image of architectural scenes as the 2D projection of the appearances of all the component primitives in the buildings and the background plane to the image plane, we reconstruct buildings under the Bayesian statistical framework by inferring the 3D structure of its component primitives and image models of visible surfaces, which follow some spatial relation prior and reproduce the given image under some projection matrix. The aspect hierarchy is used to generate proposals in primitive space, which can greatly speed up the Markov chain search.
Perceptual Scaling

Vision can be posed as a statistical learning and inference problem. As an over-simplified account, let W be a description of the outside scene in terms of “what is where,” let I be the retina image, and let p(W, I) be the joint distribution ofW and I. Then visual learning is to learn p(W, I) from training data, and visual perception is to infer W from I based on p(W |I). There are two major schools on visual learning and perception. One school is operation-oriented and learns the inferential process defined by p(W ) directly, often in the form of an explicit transformation W ≈ F (I). This scheme is mostly used in supervised learning, whereW is object category, and is given in training data. The other school is representation-oriented and learns the generative process p(W ) and p(I|W ) explicitly, then perception is to invert the generative process by maximizing or sampling p(W |I) ∝ p(W )p(I|W ). This scheme is Bayesian in nature, the prior distribution p(W ) may also be accounted for by a regularization term such as smoothness or sparsity. This scheme is often used in unsupervised learning where W is not available in training data. In the literature, there are a number of statistical theories proposed for vision. In representation-oriented school, Grenander (1993) and Mumford (1994) proposed pattern theory as a paradigm for vision (see also Geman and Geman, 1984; Amit, Grenander and Piccioni, 1991; Grenander and Miller, 1994; and S. Geman, Potter, and Chi, 2002, for important contributions that
Title Bayesian Reconstruction of 3 D Shapes and Scenes From A Single Image Permalink

It’s common experience for human vision to perceive full 3D shape and scene from a single 2D image with the occluded parts “filled-in” by prior visual knowledge. In this paper we represent prior knowledge of 3D shapes and scenes by probabilistic models at two levels – both are defined on graphs. The first level model is built on a graph representation for single object, and it is a mixture model for both man-made block objects and natural objects such as trees and grasses. It assumes surface and boundary smoothness, 3D angle symmetry etc. The second level model is built on the relation graph of all objects in a scene. It assumes that objects should be supported for maximum stability with global bounding surfaces, such as ground, sky and walls. Given an input image, we extract the geometry and photometric structures through image segmentation and sketching, and represent them in a big graph. Then we partition the graph into subgraphs each being an object, infer the 3D shape and recover occluded surfaces, edges and vertices in each subgraph, and infer the scene structures between the recovered 3D sub-graphs. The inference algorithm samples from the prior model under the constraint that it reproduces the observed image/sketch under projective geometry.
Analysis and Synthesis of Textured Motion : Particle , Wave and Cartoon Sketch

Natural scenes contain a wide range of textured motion patterns which are characterized by the movement of a large amount of particle and wave elements, such as falling snow, water waves, dancing grass, etc. In this paper, we present a generative method for modeling these motion phenomena and our method consists of four components: (1). A photometric model which represents an image as a linear superposition of image bases selected from a generic and over-complete dictionary. The dictionary contains Gabor and LoG bases for point/particle-elements and Fourier bases for wave-elements. These bases compete to explain the input images and transfer a raw image to a base (token) representation with O(10)-fold of dimension reduction. (2). A geometric model which groups adjacent bases and their motion trajectories into a number of basic moving elements – called “motons”. A moton is a deformable template in space-time representing a moving element, such as a snowflake. (3). A dynamic model which characterizes the motion of particles, waves, and their interactions, e.g., balls/leaves floating on water. The trajectories of these elements are coupled Markov chains. Given an input video sequence, a statistical learning algorithm computes a set of motons with their trajectories as hidden variables. It also learns the parameters that govern the geometric deformations and motion dynamics by maximum likelihood estimation (MLE). Consequently, novel sequences are synthesized easily from the learnt models. (4). A sketch model replaces the dictionary of Gabor/LoG and Fourier bases with symbolic sketches (token symbols). With the same generative model, the sketch model can render realistic and stylish cartoon animation. In this paper, cartoon sketch is viewed as a symbolic visualization and simplification of the inner representation in visual perception. The success of the cartoon animation, in turn, suggests that the generative model captures the essence of visual perception of textured motion.
Stochastic Graph Partition: Generalizing the Swendsen-Wang Method

Author(s): Barbu, Adrian; Zhu, Song- Vision tasks, such as segmentation, grouping, recognition, and learning, have a "what-goes-with-what" component. It can be formulated as partitioning an adjacent graph into a number of subgraphs, each being a "coherent" visual pattern in the sense of optimizing a Bayesian posterior probability or minimizing an energy functional. In this paper, we generalize Swendsen-Wang (1987)- a well celebrated algorithm in statistical mechanics-for general graph partition. Our objective is to design reversible Markov chain moves in the space of all possible partitions to search for global optimum in the Bayesian framework. We start with an adjacency graph whose vertices are image elements, such as pixels, edgels, small regions, or image bases. For each edge in the graph, we compute a local discriminative probability or probability ratio for how likely the two vertices belong to an underlying visual pattern. These edge probabilities are computed in a bottom-up fasion through previous supervised learning techniques. By turning on/off the edges independently according to these edge probabilities, we obtain a partition of the graph into a number of connected subgraphs. This procedure is in fact a sample from the space of graph partitions. We use it as a proposal (hypothesis) in a probabilistic manner. Thus the algorithm picks up a connected subgraph and flips the label of all its vertices in a single reversible Markov chain jump. In comparison to the classic Gibbs sampler which flips a single vertex at a time, the proposed method achieves: 1) Fast mixing rate it can flip a large subgraph at a time and the acceptance probability can be made to be one. 2) Short burn-in period - it can walk at low temperature and does not need a long sumulated annealing procedure. Thus it is shown to be nearly 100 times faster than the Gibbs sampler and thus produce results in about 1 minute on a PC for image segmentation and curve grouping experiments. The algorithm is tested in image segmentation and curve grouping task, and it is general for many problems in vision and beyond.
Towards a mathematical theory of primal sketch and sketchability

In this paper, we present a mathematical theory for Marr's primal sketch. We first conduct a theoretical study of the descriptive Markov random field model and the generative wavelet/sparse coding model from the perspective of entropy and complexity. The competition between the two types of models defines the concept of "sketchability", which divides image into texture and geometry. We then propose a primal sketch model that integrates the two models and, in addition, a Gestalt field model for spatial organization. We also propose a sketching pursuit process that coordinates the competition between two pursuit algorithms: the matching pursuit (Mallat and Zhang, 1993) and the filter pursuit (Zhu, et al., 1997), that seek to explain the image by bases and filters respectively. The model can be used to learn a dictionary of image primitives, or textons in Julesz's language, for natural images. The primal sketch model is not only parsimonious for image representation, but produces meaningful sketches over a large number of generic images.
How Do Heuristics Expedite Markov Chain Search? Hitting-time Analysis of the Independence Metropolis Sampler

Author(s): Maciuca, Romeo; Zhu, Song- Solving vision problems often entails searching a solution space for optimal states that have maximum Bayesian posterior probability or minimum energy. When the volume of the space is huge, exhaustive search becomes infeasible. Generic stochastic search (e.g. Markov chain Monte Carlo) could be even worse than exhaustive search as it may visit a state repeatedly. To expedite the Markov chain search, one may use heuristics as proposal probability to guide the search in promising portions of the space. Empirically the recent data-driven Markov chain Monte Carlo (DDMCMC) scheme[14,12,2] achieves fast search in a number of vision tasks, attributed by two observations: (i). The posterior probabilities in vision tasks often have very low entropy and thus are narrowly focused on a tight portion of the state space; (ii). The proposal probability computed in bottom-up methods can approximate the posterior well. In this paper we study an independent Metropolis sampler which is a simple case used as components in designing complex MCMC algorithms. We obtain an analytic formula for the expected time to first hit a certain state ("first hitting-time"), as well as very tight lower and upper bounds which depend on the total variation between the target posterior and the heuristic probabilities. These results show, though in humble cases, that one can indeed reach optimal solutions in very few steps with good proposal probabilities regardless of the size of the original search space. This result is different from previous analysis on the Markov chain convergence rate which is bounded by the second largest eigen-value (modulus) and often corresponds to the worst case in the entire search space. In comparison our analysis bears more relevance to the optimization tasks in vision.
Modeling textured motion : particle, wave and sketch

We present a generative model for textured motion phenomena, such as falling snow, wavy river and dancing grass, etc. Firstly, we represent an image as a linear superposition of image bases selected from a generic and over-complete dictionary. The dictionary contains Gabor bases for point/particle elements and Fourier bases for wave-elements. These bases compete to explain the input images. The transform from a raw image to a base or a token representation leads to large dimension reduction. Secondly, we introduce a unified motion equation to characterize the motion of these bases and the interactions between waves and particles, e.g. a ball floating on water. We use statistical learning algorithm to identify the structure of moving objects and their trajectories automatically. Then novel sequences can be synthesized easily from the motion and image models. Thirdly, we replace the dictionary of Gabor and Fourier bases with symbolic sketches (also bases). With the same image and motion model, we can render realistic and stylish cartoon animation. In our view, cartoon and sketch are symbolic visualization of the inner representation for visual perception. The success of the cartoon animation, in turn, suggests that our image and motion models capture the essence of visual perception of textured motion.
Graph partition by Swendsen-Wang cuts

Vision tasks, such as segmentation, grouping, recognition, can be formulated as graph partition problems. The recent literature witnessed two popular graph cut algorithms: the Ncut using spectral graph analysis and the minimum-cut using the maximum flow algorithm. We present a third major approach by generalizing the Swendsen-Wang method - a well celebrated algorithm in statistical mechanics. Our algorithm simulates ergodic, reversible Markov chain jumps in the space of graph partitions to sample a posterior probability. At each step, the algorithm splits, merges, or regroups a sizable subgraph, and achieves fast mixing at low temperature enabling a fast annealing procedure. Experiments show it converges in 2-30 seconds on a PC for image segmentation. This is 400 times faster than the single-site update Gibbs sampler, and 20-40 times faster than the DDMCMC algorithm. The algorithm can optimize over the number of models and works for general forms of posterior probabilities, so it is more general than the existing graph cut approaches.
Bayesian reconstruction of 3D shapes and scenes from a single image

It is common experience for human vision to perceive full 3D shape and scene from a single 2D image with the occluded parts "filled-in" by prior visual knowledge. We represent prior knowledge of 3D shapes and scenes by probabilistic models at two levels - both are defined on graphs. The first level model is built on a graph representation for single objects, and it is a mixture model for both man-made block objects such as trees and grasses. It assumes surface and boundary smoothness, 3D angle symmetry etc. The second level model is built on the relation graph of all objects in a scene. It assumes that objects should be supported for maximum stability with global bounding surfaces, such as ground, sky and walls. Given an input image, we extract the geometry and photometric structures through image segmentation and sketching, and represent them in a big graph. Then we partition the graph into subgraphs each being an object, infer the 3D shape and recover occluded surfaces, edges and vertices in each subgraph, and infer the scene structures between the recovered 3D sub-graphs. The inference algorithm samples from the prior model under the constraint that it reproduces the observed image/sketch under projective geometry.
Statistical Modeling and Conceptualization of Visual Patterns

Natural images contain an overwhelming number of visual patterns generated by diverse stochastic processes. Defining and modeling these patterns is of fundamental importance for generic vision tasks, such as perceptual organization, segmentation, and recognition. The objective of this epistemological paper is to summarize various threads of research in the literature and to pursue a unified framework for conceptualization, modeling, learning, and computing visual patterns. This paper starts with reviewing four research streams: 1) the study of image statistics, 2) the analysis of image components, 3) the grouping of image elements, and 4) the modeling of visual patterns. The models from these research streams are then divided into four categories according to their semantic structures: 1) descriptive models, i.e., Markov random fields (MRF) or Gibbs, 2) variants of descriptive models (causal MRF and "pseudodescriptive" models), 3) generative models, and 4) discriminative models. The objectives, principles, theories, and typical models are reviewed in each category and the relationships between the four types of models are studied. Two central themes emerge from the relationship studies. 1) In representation, the integration of descriptive and generative models is the future direction for statistical modeling and should lead to richer and more advanced classes of vision models. 2) To make visual models computationally tractable, discriminative models are used as computational heuristics for inferring generative models. Thus, the roles of four types of models are clarified.
A multi-scale generative model for animate shapes and parts

We present a multiscale generative model for representing animate shapes and extracting meaningful parts of objects. The model assumes that animate shapes (2D simple dosed curves) are formed by a linear superposition of a number of shape bases. These shape bases resemble the multiscale Gabor bases in image pyramid representation, are well localized in both spatial and frequency domains, and form an over-complete dictionary. This model is simpler than the popular B-spline representation since it does not engage a domain partition. Thus it eliminates the interference between adjacent B-spline bases, and becomes a true linear additive model. We pursue the bases by reconstructing the shape in a coarse-to-fine procedure through curve evolution. These shape bases are further organized in a tree-structure, where the bases in each subtree sum up to an intuitive part of the object. To build probabilistic model for a class of objects, we propose a Markov random field model at each level of the tree representation to account for the spatial relationship between bases. Thus the final model integrates a Markov tree (generative) model over scales and a Markov random field over space. We adopt EM-type algorithm for learning the meaningful parts for a shape class, and show some results on shape synthesis.