self scaling fast rotations for stiff and equality constrained linear least squares problems abstract we present algorithms which apply self scaling fast plane rotations to the qr decomposition for stiff least squares problems we show that both fast and standard givens rotation based algorithms produce accurate results regardless of row sorting and even with extremely large weights when equality constrained least squares problems are solved by the weighting method numerical test results show that the householder qr decomposition algorithm is sensitive to row sorting and produces less accurate results when the weights are large and that the modified gram schmidt algorithm is less sensitive to row sorting this makes the fast plane rotation a method of choice for the qr decomposition of stiff matrices since it is also competitive in computational complexity based on the above results an efficient algorithm is also derived for the application where the least squares solutions are required for various constrained matrices for each fixed data matrix

maximum entropy density estimation with generalized regularization and an application to species distribution modeling we present a unified and complete account of maximum entropy density estimation subject to constraints represented by convex potential functions or alternatively by convex regularization we provide fully general performance guarantees and an algorithm with a complete convergence proof as special cases we easily derive performance guarantees for many known regularization types including l1 l2 l22 and l2 l22 style regularization we propose an algorithm solving a large and general subclass of generalized maximum entropy problems including all discussed in the paper and prove its convergence our approach generalizes and unifies techniques based on information geometry and bregman divergences as well as those based more directly on compactness our work is motivated by a novel application of maximum entropy to species distribution modeling an important problem in conservation biology and ecology in a set of experiments on real world data we demonstrate the utility of maximum entropy in this setting we explore effects of different feature types sample sizes and regularization levels on the performance of maxent and discuss interpretability of the resulting models

cooling schedules for optimal annealing a monte carlo optimization technique called simulated annealing is a descent algorithm modified by random ascent moves in order to escape local minima which are not global minima the level of randomization is determined by a control parameter t called temperature which tends to zero according to a deterministic cooling schedule we give a simple necessary and sufficient condition on the cooling schedule for the algorithm state to converge in probability to the set of globally minimum cost states in the special case that the cooling schedule has parametric form tt c log1 t the condition for convergence is that c be greater than or equal to the depth suitably defined of the deepest local minimum which is not a global minimum state

a continuous entropy rate estimator for spike trains using a k means based context tree entropy rate quantifies the change of information of a stochastic process cover thomas 2006 for decades the temporal dynamics of spike trains generated by neurons has been studied as a stochastic process barbieri quirk frank wilson brown 2001 brown frank tang quirk wilson 1998 kass ventura 2001 metzner koch wessel gabbiani 1998 zhang ginzburg mcnaughton sejnowski 1998 we propose here to estimate the entropy rate of a spike train from an inhomogeneous hidden markov model of the spike intervals the model is constructed by building a context tree structure to lay out the conditional probabilities of various subsequences of the spike train for each state in the markov chain we assume a gamma distribution over the spike intervals although any appropriate distribution may be employed as circumstances dictate the entropy and confidence intervals for the entropy are calculated from bootstrapping samples taken from a large raw data sequence the estimator was first tested on synthetic data generated by multiple order markov chains and it always converged to the theoretical shannon entropy rate except in the case of a sixth order model where the calculations were terminated before convergence was reached we also applied the method to experimental data and compare its performance with that of several other methods of entropy estimation

topographic mapping of large dissimilarity data sets topographic maps such as the self organizing map som or neural gas ng constitute powerful data mining techniques that allow simultaneously clustering data and inferring their topological structure such that additional features for example browsing become available both methods have been introduced for vectorial data sets they require a classical feature encoding of information often data are available in the form of pairwise distances only such as arise from a kernel matrix a graph or some general dissimilarity measure in such cases ng and som cannot be applied directly in this article we introduce relational topographic maps as an extension of relational clustering algorithms which offer prototype based representations of dissimilarity data to incorporate neighborhood structure these methods are equivalent to the standard vectorial techniques if a euclidean embedding exists while preventing the need to explicitly compute such an embedding extending these techniques for the general case of non euclidean dissimilarities makes possible an interpretation of relational clustering as clustering in pseudo euclidean space we compare the methods to well known clustering methods for proximity data based on deterministic annealing and discuss how far convergence can be guaranteed in the general case relational clustering is quadratic in the number of data points which makes the algorithms infeasible for huge data sets we propose an approximate patch version of relational clustering that runs in linear time the effectiveness of the methods is demonstrated in a number of examples

laplace s law of succession and universal encoding this correspondence shows that the known add half rule is not asymptotically optimal for predicting the n 1 st symbol after a sequence of n symbols whereas the add spl beta sub 0 rule spl beta sub 0 0 50922 spl middot spl middot spl middot is

unsupervised texture segmentation in a deterministic annealing framework we present a novel optimization framework for unsupervised texture segmentation that relies on statistical tests as a measure of homogeneity texture segmentation is formulated as a data clustering problem based on sparse proximity data dissimilarities of pairs of textured regions are computed from a multiscale gabor filter image representation we discuss and compare a class of clustering objective functions which is systematically derived from invariance principles as a general optimization framework we propose deterministic annealing based on a mean field approximation the canonical way to derive clustering algorithms within this framework as well as an efficient implementation of mean field annealing and the closely related gibbs sampler are presented we apply both annealing variants to brodatz like microtexture mixtures and real word images

a new point symmetry based fuzzy genetic clustering technique for automatic evolution of clusters in this paper a fuzzy point symmetry based genetic clustering technique fuzzy vgaps is proposed which can automatically determine the number of clusters present in a data set as well as a good fuzzy partitioning of the data the clusters can be of any size shape or convexity as long as they possess the property of symmetry here the membership values of points to different clusters are computed using the newly proposed point symmetry based distance a variable number of cluster centers are encoded in the chromosomes a new fuzzy symmetry based cluster validity index fsym index is first proposed here and thereafter it is utilized to measure the fitness of the chromosomes the proposed index can detect non convex as well as convex non hyperspherical partitioning with variable number of clusters it is mathematically justified via its relationship to a well defined hard cluster validity function the dunn s index for which the condition of uniqueness has already been established the results of the fuzzy vgaps are compared with those obtained by seven other algorithms including both fuzzy and crisp methods on four artificial and four real life data sets some real life applications of fuzzy vgaps to automatically cluster the gene expression data as well as segmenting the magnetic resonance brain image with multiple sclerosis lesions are also demonstrated

normalized cuts and image segmentation we propose a novel approach for solving the perceptual grouping problem in vision rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image we treat image segmentation as a graph partitioning problem and propose a novel global criterion the normalized cut for segmenting the graph the normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups we show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion we applied this approach to segmenting static images as well as motion sequences and found the results to be very encouraging

a nonparametric clustering algorithm with a quantile based likelihood estimator clustering is a representative of unsupervised learning and one of the important approaches in exploratory data analysis by its very nature clustering without strong assumption on data distribution is desirable information theoretic clustering is a class of clustering methods that optimize information theoretic quantities such as entropy and mutual information these quantities can be estimated in a nonparametric manner and information theoretic clustering algorithms are capable of capturing various intrinsic data structures it is also possible to estimate information theoretic quantities using a data set with sampling weight for each datum assuming the data set is sampled from a certain cluster and assigning different sampling weights depending on the clusters the cluster conditional information theoretic quantities are estimated in this letter a simple iterative clustering algorithm is proposed based on a nonparametric estimator of the log likelihood for weighted data sets the clustering algorithm is also derived from the principle of conditional entropy minimization with maximum entropy regularization the proposed algorithm does not contain a tuning parameter the algorithm is experimentally shown to be comparable to or outperform conventional nonparametric clustering methods

least squares quantization in pcm it has long been realized that in pulse code modulation pcm with a given ensemble of signals to handle the quantum values should be spaced more closely in the voltage regions where the signal amplitude is more likely to fall it has been shown by panter and dite that in the limit as the number of quanta becomes infinite the asymptotic fractional density of quanta per unit voltage should vary as the one third power of the probability density per unit voltage of signal amplitudes in this paper the corresponding result for any finite number of quanta is derived that is necessary conditions are found that the quanta and associated quantization intervals of an optimum finite quantization scheme must satisfy the optimization criterion used is that the average quantization noise power be a minimum it is shown that the result obtained here goes over into the panter and dite result as the number of quanta become large the optimum quautization schemes for 2 b quanta b 1 2 cdots 7 are given numerically for gaussian and for laplacian distribution of signal amplitudes

rademacher penalties and structural risk minimization we suggest a penalty function to be used in various problems of structural risk minimization this penalty is data dependent and is based on the sup norm of the so called rademacher process indexed by the underlying class of functions sets the standard complexity penalties used in learning problems and based on the vc dimensions of the classes are conservative upper bounds in a probabilistic sense uniformly over the set of all underlying distributions for the penalty we suggest thus for a particular distribution of training examples one can expect better performance of learning algorithms with the data driven rademacher penalties we obtain oracle inequalities for the theoretical risk of estimators obtained by structural minimization of the empirical risk with rademacher penalties the inequalities imply some form of optimality of the empirical risk minimizers we also suggest an iterative approach to structural risk minimization with rademacher penalties in which the hierarchy of classes is not given in advance but is determined in the data driven iterative process of risk minimization we prove probabilistic oracle inequalities for the theoretical risk of the estimators based on this approach as well

pac bayesian generalisation error bounds for gaussian process classification approximate bayesian gaussian process gp classification techniques are powerful non parametric learning methods similar in appearance and performance to support vector machines based on simple probabilistic models they render interpretable results and can be embedded in bayesian frameworks for model selection feature selection etc in this paper by applying the pac bayesian theorem of mcallester 1999a we prove distribution free generalisation error bounds for a wide range of approximate bayesian gp classification techniques we also provide a new and much simplified proof for this powerful theorem making use of the concept of convex duality which is a backbone of many machine learning techniques we instantiate and test our bounds for two particular gpc techniques including a recent sparse method which circumvents the unfavourable scaling of standard gp algorithms as is shown in experiments on a real world task the bounds can be very tight for moderate training sample sizes to the best of our knowledge these results provide the tightest known distribution free error bounds for approximate bayesian gpc methods giving a strong learning theoretical justification for the use of these techniques

information cut for clustering using a gradient descent approach we introduce a new graph cut for clustering which we call the information cut it is derived using parzen windowing to estimate an information theoretic distance measure between probability density functions we propose to optimize the information cut using a gradient descent based approach our algorithm has several advantages compared to many other graph based methods in terms of determining an appropriate affinity measure computational complexity memory requirements and coping with different data scales we show that our method may produce clustering and image segmentation results comparable or better than the state of the art graph based methods

robust right coprime factorization and robust stabilization of nonlinear feedback control systems robust right coprime factorization and robust stabilization of nonlinear feedback control systems are studied the concept of robust right coprime factorization of nonlinear operators for feedback control systems is introduced some conditions for the robustness of a right coprime factorization of a nonlinear plant under unknown but bounded perturbations are derived an example with a closed form solution is included to illustrate the general theory and a step by step construction of the robust factorization and the robust stabilization

estimating the temporal interval entropy of neuronal discharge to better understand the role of timing in the function of the nervous system we have developed a methodology that allows the entropy of neuronal discharge activity to be estimated from a spike train record when it may be assumed that successive interspike intervals are temporally uncorrelated the so called interval entropy obtained by this methodology is based on an implicit enumeration of all possible spike trains that are statistically indistinguishable from a given spike train the interval entropy is calculated from an analytic distribution whose parameters are obtained by maximum likelihood estimation from the interval probability distribution associated with a given spike train we show that this approach reveals features of neuronal discharge not seen with two alternative methods of entropy estimation the methodology allows for validation of the obtained data models by calculation of confidence intervals for the parameters of the analytic distribution and the testing of the significance of the fit between the observed and analytic interval distributions by means of kolmogorov smirnov and anderson darling statistics the method is demonstrated by analysis of two different data sets simulated spike trains evoked by either poissonian or near synchronous pulsed activation of a model cerebellar purkinje neuron and spike trains obtained by extracellular recording from spontaneously discharging cultured rat hippocampal neurons

generalization from observed to unobserved features by clustering we argue that when objects are characterized by many attributes clustering them on the basis of a random subset of these attributes can capture information on the unobserved attributes as well moreover we show that under mild technical conditions clustering the objects on the basis of such a random subset performs almost as well as clustering with the full attribute set we prove finite sample generalization theorems for this novel learning scheme that extends analogous results from the supervised learning setting we use our framework to analyze generalization to unobserved features of two well known clustering algorithms k means and the maximum likelihood multinomial mixture model the scheme is demonstrated for collaborative filtering of users with movie ratings as attributes and document clustering with words as attributes

vector quantization by deterministic annealing a deterministic annealing approach is suggested to search for the optimal vector quantizer given a set of training data the problem is reformulated within a probabilistic framework no prior knowledge is assumed on the source density and the principle of maximum entropy is used to obtain the association probabilities at a given average distortion the corresponding lagrange multiplier is inversely related to the temperature and is used to control the annealing process in this process as the temperature is lowered the system undergoes a sequence of phase transitions when existing clusters split naturally without use of heuristics the resulting codebook is independent of the codebook used to initialize the iterations

boundary detection by constrained optimization a statistical framework is used for finding boundaries and for partitioning scenes into homogeneous regions the model is a joint probability distribution for the array of pixel gray levels and an array of labels in boundary finding the labels are binary zero or one representing the absence or presence of boundary elements in partitioning the label values are generic two labels are the same when the corresponding scene locations are considered to belong to the same region the distribution incorporates a measure of disparity between certain spatial features of block pairs of pixel gray levels using the kolmogorov smirnov nonparametric measures of difference between the distributions of these features the number of model parameters is minimized by forbidding label configurations which are assigned probability zero the maximum a posteriori estimator of boundary placements and partitionings is examined the forbidden states introduce constraints into the calculation of these configurations stochastic relaxation methods are extended to accommodate constrained optimization

internally stable nonlinear systems with disturbances a parameterization this paper deals with the control of a nonlinear system whose output is subjected to an additive disturbance the main result is a simple parameterization of the set of all system responses that can be obtained through internally stable control of the given system the parameterization provides a clear indication of the effects of the disturbance on the response of the stabilized closed loop system the class of achievable responses is determined by the numerator of a right coprime fraction representation of the system being controlled

pac bayesian analysis of co clustering and beyond we derive pac bayesian generalization bounds for supervised and unsupervised learning models based on clustering such as co clustering matrix tri factorization graphical models graph clustering and pairwise clustering we begin with the analysis of co clustering which is a widely used approach to the analysis of data matrices we distinguish among two tasks in matrix data analysis discriminative prediction of the missing entries in data matrices and estimation of the joint probability distribution of row and column variables in co occurrence matrices we derive pac bayesian generalization bounds for the expected out of sample performance of co clustering based solutions for these two tasks the analysis yields regularization terms that were absent in the previous formulations of co clustering the bounds suggest that the expected performance of co clustering is governed by a trade off between its empirical performance and the mutual information preserved by the cluster variables on row and column ids we derive an iterative projection algorithm for finding a local optimum of this trade off for discriminative prediction tasks this algorithm achieved state of the art performance in the movielens collaborative filtering task our co clustering model can also be seen as matrix tri factorization and the results provide generalization bounds regularization terms and new algorithms for this form of matrix factorization the analysis of co clustering is extended to tree shaped graphical models which can be used to analyze high dimensional tensors according to the bounds the generalization abilities of tree shaped graphical models depend on a trade off between their empirical data fit and the mutual information that is propagated up the tree levels we also formulate weighted graph clustering as a prediction problem given a subset of edge weights we analyze the ability of graph clustering to predict the remaining edge weights the analysis of co clustering easily extends to this problem and suggests that graph clustering should optimize the trade off between empirical data fit and the mutual information that clusters preserve on graph nodes

combining pac bayesian and generic chaining bounds there exist many different generalization error bounds in statistical learning theory each of these bounds contains an improvement over the others for certain situations or algorithms our goal is first to underline the links between these bounds and second to combine the different improvements into a single bound in particular we combine the pac bayes approach introduced by mcallester 1998 which is interesting for randomized predictions with the optimal union bound provided by the generic chaining technique developed by fernique and talagrand see talagrand 1996 in a way that also takes into account the variance of the combined functions we also show how this connects to rademacher based bounds

fast givens rotations for orthogonal similarity transformations fast givens rotations with half as many multiplications are proposed for orthogonal similarity transformations and a matrix notation is introduced to describe them more easily applications are proposed and numerical results are examined for the jacobi method the reduction to hessenberg form and the qr algorithm for hessenberg matrices it can be seen that in general fast givens rotations are competitive with householder reflexions and offer distinct advantages for sparse matrices

stability based validation of clustering solutions data clustering describes a set of frequently employed techniques in exploratory data analysis to extract natural group structure in data such groupings need to be validated to separate the signal in the data from spurious structure in this context finding an appropriate number of clusters is a particularly important model selection question we introduce a measure of cluster stability to assess the validity of a cluster model this stability measure quantifies the reproducibility of clustering solutions on a second sample and it can be interpreted as a classification risk with regard to class labels produced by a clustering algorithm the preferred number of clusters is determined by minimizing this classification risk as a function of the number of clusters convincing results are achieved on simulated as well as gene expression data sets comparisons to other methods demonstrate the competitive performance of our method and its suitability as a general validation tool for clustering solutions in real world problems

how to avoid spurious cluster validation a methodological investigation on simulated and fmri data abstract this paper presents an evaluation of a common approach that has been considered as a promising option for exploratory fmri data analyses the approach includes two stages creating from the data a sequence of partitions with increasing number of subsets clustering and selecting the one partition in this sequence that exhibits the clearest indications of an existing structure cluster validation in order to achieve that the selected partition is actually the best characterization of the data structure previous studies were directed to find the most appropriate validity function s in our analysis protocol we first optimize the sequence of partitions according to the given objective function our study showed that an insufficient optimization of the partition for one or more numbers of clusters can easily yield a spurious validation result which in turn may lead the analyst to a misleading interpretation of the fmri experiment however a sufficient optimization for each included number of clusters provided the basis for a reliable adequate characterization of the data furthermore it enabled an adequate evaluation of the validity functions these findings were obtained independently for three clustering algorithms representing the hard and fuzzy clustering variant and three up to date cluster validity functions the findings were derived from analyses of gaussian clusters simulated data sets that mimic typical fmri response signals and real fmri data based on our results we propose a number of options of how to configure improved clustering tools

learning and generalization with the information bottleneck the information bottleneck is an information theoretic framework that finds concise representations for an input random variable that are as relevant as possible for an output random variable this framework has been used successfully in various supervised and unsupervised applications however its learning theoretic properties and justification remained unclear as it differs from standard learning models in several crucial aspects primarily its explicit reliance on the joint input output distribution in practice an empirical plug in estimate of the underlying distribution has been used so far without any finite sample performance guarantees in this paper we present several formal results that address these difficulties we prove several finite sample bounds which show that the information bottleneck can provide concise representations with good generalization based on smaller sample sizes than needed to estimate the underlying distribution the bounds are non uniform and adaptive to the complexity of the specific model chosen based on these results we also present a preliminary analysis on the possibility of analyzing the information bottleneck method as a learning algorithm in the familiar performance complexity tradeoff framework in addition we formally describe the connection between the information bottleneck and minimal sufficient statistics

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

a characterization of linkage based hierarchical clustering the class of linkage based algorithms is perhaps the most popular class of hierarchical algorithms we identify two properties of hierarchical algorithms and prove that linkage based algorithms are the only ones that satisfy both of these properties our characterization clearly delineates the difference between linkage based algorithms and other hierarchical methods we formulate an intuitive notion of locality of a hierarchical algorithm that distinguishes between linkage based and global hierarchical algorithms like bisecting k means and prove that popular divisive hierarchical algorithms produce clusterings that cannot be produced by any linkage based algorithm

estimating mutual information we present two classes of improved estimators for mutual information m x y from samples of random points distributed according to some joint probability density mu x y in contrast to conventional estimators based on binnings they are based on entropy estimates from k nearest neighbor distances this means that they are data efficient with k 1 we resolve structures down to the smallest possible scales adaptive the resolution is higher where data are more numerous and have minimal bias indeed the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale giving typically systematic errors which scale as functions of k n for n points numerically we find that both families become exact for independent distributions i e the estimator m x y vanishes up to statistical fluctuations if mu x y mu x mu y this holds for all tested marginal distributions and for all dimensions of x and y in addition we give estimators for redundancies between more than two random variables we compare our algorithms in detail with existing algorithms finally we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis ica for improving ica and for estimating the reliability of blind source separation

global parametrization of feedback systems with nonlinear plants for unity feedback systems with a plant p and controller c we show that given a nonlinear stable p the class of all controller stabilizing the feedback system is globally parametrized by a stable map q we also show how q is useful in studying the effects of modeling errors on the stability

kernel entropy component analysis we introduce kernel entropy component analysis kernel eca as a new method for data transformation and dimensionality reduction kernel eca reveals structure relating to the renyi entropy of the input space data set estimated via a kernel matrix using parzen windowing this is achieved by projections onto a subset of entropy preserving kernel principal component analysis kernel pca axes this subset does not need in general to correspond to the top eigenvalues of the kernel matrix in contrast to the dimensionality reduction using kernel pca we show that kernel eca may produce strikingly different transformed data sets compared to kernel pca with a distinct angle based structure a new spectral clustering algorithm utilizing this structure is developed with positive results furthermore kernel eca is shown to be an useful alternative for pattern denoising

unsupervised texture segmentation using gabor filters abstract this paper presents a texture segmentation algorithm inspired by the multi channel filtering theory for visual information processing in the early stages of human visual system the channels are characterized by a bank of gabor filters that nearly uniformly covers the spatial frequency domain and a systematic filter selection scheme is proposed which is based on reconstruction of the input image from the filtered images texture features are obtained by subjecting each selected filtered image to a nonlinear transformation and computing a measure of energy in a window around each pixel a square error clustering algorithm is then used to integrate the feature images and produce a segmentation a simple procedure to incorporate spatial information in the clustering process is proposed a relative index is used to estimate the true number of texture categories

a sequence based extension of mean field annealing using the forward backward algorithm application to image segmentation optimization based on mean field theory including mean field annealing mfa is widely used for discrete optimization label assignment problems defined on the pixel sites of an image one formulation of mfa is via maximum entropy where one seeks the joint distribution over the random assignments subject to an average level of cost mfa is obtained by assuming the assignments at each pixel are statistically independent given the observed image alternatively we make the less restrictive assumption of independent row labelings the independence assumption means that at each step mfa optimizes over only one pixel whereas our method jointly optimizes over an entire row i e our method is less greedy in principle an mfa extension could be developed that explicitly re estimates the row labeling distributions but such an approach is in practice infeasible even so we can indirectly implement this re estimation by re estimating quantities that determine the row labeling distributions these quantities are the a posteriori site probabilities re estimable via the well known forward backward f b algorithm thus our algorithm which descends in the me lagrangian free energy consists of iterative application of f b to the image rows columns at convergence maximum a posteriori site labeling is performed our method was applied to segmentation of both real and synthetic noise corrupted images it achieved lower markov random field model potentials and better segmentations compared with other methods and in high noise standard mfa

mib using mutual information for biclustering gene expression data result of any biclustering or clustering algorithm depends on the choice of the similarity measure most of the biclustering algorithms are based on euclidean distance or correlation coefficient these measures capture only linear relationships between the genes but nonlinear dependencies may exist amongst them in this paper we propose an approach using mutual information for biclustering gene expression data mutual information is a more general measure to investigate relationships positive negative correlation and nonlinear relationships as well to the best of our knowledge none of the existing algorithms for biclustering have used mutual information as a similarity measure between two genes we obtained biclusters from the gene expression data of arabidopsis thaliana and compared our biclusters with those obtained by two other algorithms namely isa and bimax biological significance of the biclusters was checked using go database it was found that the genes belonging to our biclusters were significantly enriched with go terms with better p values as compared to the genes of the biclusters obtained by the other two algorithms to further investigate the biclusters we studied the promoter regions of the genes belonging to a bicluster for common patterns transcription factor binding sites tfbs or motifs promoter regions of the genes of most of our biclusters were found to have a common motif patterns which existed in the motif database of arabidopsis thaliana also the motifs extracted from our biclusters had better e values than those of others thus reconfirming that use of mutual information as a similarity measure will produce better biclusters

biclustering of gene expression data based on related genes and conditions extraction biclustering is an important tool to find patterns in a microarray data matrix by simultaneous classification in two dimensions of genes and conditions unlike most existed biclustering algorithms where almost all genes and conditions are involved in the clustering process even if they contribute little to a bicluster we propose to perform the biclustering operation only in related genes and conditions of a given bicluster type in our algorithm the gene expression matrix is first partitioned to stable and unstable submatrices in both row and column directions by inspecting the similarity between the row or column vector and the full 1s vector then the related genes and conditions of a given type of biclusters are extracted by inspecting the row or column pairs in the corresponding stable or unstable submatrices finally the resulted biclusters of any type are obtained by performing clustering analysis in the extracted related genes and conditions additionally a novel strategy for estimating the missing data in the gene expression matrix is also presented based on the james stein and kernel estimation principle where the estimation matrix is obtained with the k means algorithm experimental results show excellent performance of our algorithm both in missing data estimation and biclustering

non linear systems stability and rationality the problem of when a non linear system can be represented as a quotient of two stable non linear systems is considered attention is mainly directed toward nonlinear discrete time recursive systems where recursive means that the relationship between an input sequence and the corresponding output sequence can be expressed in terms of a finite number of recursive equations necessary and sufficient conditions are derived for the existence of a fraction representation of a recursive system where the numerator and the denominator are stable recursive systems the explicit construction of such a fraction representation is described

a grouped ranking model for item preference parameter given a set of rating data for a set of items determining preference levels of items is a matter of importance various probability models have been proposed to solve this task one such model is the plackett luce model which parameterizes the preference level of each item by a real value in this letter the plackett luce model is generalized to cope with grouped ranking observations such as movie or restaurant ratings since it is difficult to maximize the likelihood of the proposed model directly a feasible approximation is derived and the em algorithm is adopted to find the model parameter by maximizing the approximate likelihood which is easily evaluated the proposed model is extended to a mixture model and two applications are proposed to show the effectiveness of the proposed model numerical experiments with real world data are carried out

geodesic entropic graphs for dimension and entropy estimation in manifold learning in the manifold learning problem one seeks to discover a smooth low dimensional surface i e a manifold embedded in a higher dimensional linear vector space based on a set of measured sample points on the surface in this paper we consider the closely related problem of estimating the manifold s intrinsic dimension and the intrinsic entropy of the sample points specifically we view the sample points as realizations of an unknown multivariate density supported on an unknown smooth manifold we introduce a novel geometric approach based on entropic graph methods although the theory presented applies to this general class of graphs we focus on the geodesic minimal spanning tree gmst to obtaining asymptotically consistent estimates of the manifold dimension and the re spl acute nyi spl alpha entropy of the sample density on the manifold the gmst approach is striking in its simplicity and does not require reconstruction of the manifold or estimation of the multivariate density of the samples the gmst method simply constructs a minimal spanning tree mst sequence using a geodesic edge matrix and uses the overall lengths of the msts to simultaneously estimate manifold dimension and entropy we illustrate the gmst approach on standard synthetic manifolds as well as on real data sets consisting of images of faces

the context tree weighting method basic properties describes a sequential universal data compression procedure for binary tree sources that performs the double mixture using a context tree this method weights in an efficient recursive way the coding distributions corresponding to all bounded memory tree sources and achieves a desirable coding distribution for tree sources with an unknown model and unknown parameters computational and storage complexity of the proposed procedure are both linear in the source sequence length the authors derive a natural upper bound on the cumulative redundancy of the method for individual sequences the three terms in this bound can be identified as coding parameter and model redundancy the bound holds for all source sequence lengths not only for asymptotically large lengths the analysis that leads to this bound is based on standard techniques and turns out to be extremely simple the upper bound on the redundancy shows that the proposed context tree weighting procedure is optimal in the sense that it achieves the rissanen 1984 lower bound

supervised learning of quantizer codebooks by information loss minimization this paper proposes a technique for jointly quantizing continuous features and the posterior distributions of their class labels based on minimizing empirical information loss such that the quantizer index of a given feature vector approximates a sufficient statistic for its class label informally the quantized representation retains as much information as possible for classifying the feature vector correctly we derive an alternating minimization procedure for simultaneously learning codebooks in the euclidean feature space and in the simplex of posterior class distributions the resulting quantizer can be used to encode unlabeled points outside the training set and to predict their posterior class distributions and has an elegant interpretation in terms of lossless source coding the proposed method is validated on synthetic and real data sets and is applied to two diverse problems learning discriminative visual vocabularies for bag of features image classification and image segmentation

the cal d q r algorithm basic theory convergence and conditional stability we describe a fast matrix eigenvalue algorithm that uses a matrix factorization and reverse order multiply technique involving three factors and that is based on the symmetric matrix factorization as well as on d orthogonal reduction techniques where d mbox diag pm1 is computed from the given matrix a it operates on a similarity reduction of a real matrix a nn to general tridiagonal form t and computes all of a s eigenvalues in frac 4 3 n 3 o n 2 operations where the o n 2 part of the operations is possibly performed over bbb c instead of the 7 8 n 3 real flops required by the qr eigenvalue algorithm potential breakdo wn of the dqr algorithm can occur in the reduction to tridiagonal form and in the d orthogonal dqr reductions both however can be monitored during the computations the former occurs rather rarely for dimensions n leq 300 and can essentially be bypassed while the latter is extremely rare and can be bypassed as well in our conditionally stable implementation of the dqr steps we prove an implicit dq theorem which allows implicit shifts give a convergence proof for the dqr algorithm and show that dqr is conditionally stable for general balanced tridiagonal matrices t rm b

quadratic convergence estimate of scaled iterates by j symmetric jacobi method abstract this paper estimates the quadratic convergence reduction of scaled iterates by j symmetric jacobi method numer math 64 1993 241 although the method is well defined for a general definite pair h j h ht j diag im in m the paper considers the most important case when h is positive definite in that case the method is an accurate floating point eigensolver for the pair h j as such it is used in a compound algorithm for accurate floating point computation of eigenvalues and eigenvectors of a non singular indefinite symmetric matrix the new result is proved for scaled diagonally dominant matrices in the general case of multiple eigenvalues it uses frobenius norm of the off diagonal part of symmetrically scaled iteration matrix and a relative gap in the spectrum of h j it can be effectively used in connection with stopping criterion of the method especially with its one sided version

a comparison of texture feature extraction using adaptive gabor filtering pyramidal and tree structured wavelet transforms in recent studies on image analysis an increasing effort has been carried out in the area of wavelet transform techniques for the discrimination and classification of textural images these methods compete with multichannel filtering techniques especially with the nonorthogonal and incomplete gabor filtering in this paper we introduce two feature extraction algorithms based on pyramidal and tree structured wavelet transforms and compare their performance with the feature extraction which employs adaptive gabor filtering this comparison is based on the segmentation results of several texture image examples using the identical segmentation algorithm for all three feature extraction methods the visible differences of the segmentation results are discussed and their algorithmic causes are analysed

on the statistical analysis of dirty pictures may 7th 1986 professor a f m smith in the chair summary a continuous two dimensional region is partitioned into a fine rectangular array of sites or pixels each pixel having a particular colour belonging to a prescribed finite set the true colouring of the region is unknown but associated with each pixel there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model the aim is to reconstruct the true scene with the additional knowledge that pixels close together tend to have the same or similar colours in this paper it is assumed that the local characteristics of the true scene can be represented by a nondegenerate markov random field such information can be combined with the records by bayes theorem and the true scene can be estimated according to standard criteria however the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field thus a simple iterative method of reconstruction is proposed which does not depend on these large scale characteristics the method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field some complications including parameter estimation are discussed potential applications are mentioned briefly

multivariate information bottleneck the information bottleneck ib method is an unsupervised model independent data organization technique given a joint distribution p x y this method constructs a new variable t that extracts partitions or clusters over the values of x that are informative about y algorithms that are motivated by the ib method have already been applied to text classification gene expression neural code and spectral analysis here we introduce a general principled framework for multivariate extensions of the ib method this allows us to consider multiple systems of data partitions that are interrelated our approach utilizes bayesian networks for specifying the systems of clusters and which information terms should be maintained we show that this construction provides insights about bottleneck variations and enables us to characterize the solutions of these variations we also present four different algorithmic approaches that allow us to construct solutions in practice and apply them to several real world problems

generalizing swendsen wang to sampling arbitrary posterior probabilities many vision tasks can be formulated as graph partition problems that minimize energy functions for such problems the gibbs sampler provides a general solution but is very slow while other methods such as ncut and graph cuts are computationally effective but only work for specific energy forms and are not generally applicable in this paper we present a new inference algorithm that generalizes the swendsen wang method to arbitrary probabilities defined on graph partitions we begin by computing graph edge weights based on local image features then the algorithm iterates two steps 1 graph clustering it forms connected components by cutting the edges probabilistically based on their weights 2 graph relabeling it selects one connected component and flips probabilistically the coloring of all vertices in the component simultaneously thus it realizes the split merge and regrouping of a chunk of the graph in contrast to gibbs sampler that flips a single vertex we prove that this algorithm simulates ergodic and reversible markov chain jumps in the space of graph partitions and is applicable to arbitrary posterior probabilities or energy functions defined on graphs we demonstrate the algorithm on two typical problems in computer vision image segmentation and stereo vision experimentally we show that it is 100 400 times faster in cpu time than the classical gibbs sampler and 20 40 times faster then the ddmcmc segmentation algorithm for stereo we compare performance with graph cuts and belief propagation we also show that our algorithm can automatically infer generative models and obtain satisfactory results better than the graphic cuts or belief propagation in the same amount of time

a generalized maximum entropy approach to bregman co clustering and matrix approximation co clustering or simultaneous clustering of rows and columns of a two dimensional data matrix is rapidly becoming a powerful data analysis technique co clustering has enjoyed wide success in varied application domains such as text clustering gene microarray analysis natural language processing and image speech and video analysis in this paper we introduce a partitional co clustering formulation that is driven by the search for a good matrix approximation every co clustering is associated with an approximation of the original data matrix and the quality of co clustering is determined by the approximation error we allow the approximation error to be measured using a large class of loss functions called bregman divergences that include squared euclidean distance and kl divergence as special cases in addition we permit multiple structurally different co clustering schemes that preserve various linear statistics of the original data matrix to accomplish the above tasks we introduce a new minimum bregman information mbi principle that simultaneously generalizes the maximum entropy and standard least squares principles and leads to a matrix approximation that is optimal among all generalized additive models in a certain natural parameter space analysis based on this principle yields an elegant meta algorithm special cases of which include most previously known alternate minimization based clustering algorithms such as kmeans and co clustering algorithms such as information theoretic dhillon et al 2003b and minimum sum squared residue co clustering cho et al 2004 to demonstrate the generality and flexibility of our co clustering framework we provide examples and empirical evidence on a variety of problem domains and also describe novel co clustering applications such as missing value prediction and compression of categorical data matrices

optimal cluster preserving embedding of nonmetric proximity data for several major applications of data analysis objects are often not represented as feature vectors in a vector space but rather by a matrix gathering pairwise proximities such pairwise data often violates metricity and therefore cannot be naturally embedded in a vector space concerning the problem of unsupervised structure detection or clustering in this paper a new embedding method for pairwise data into euclidean vector spaces is introduced we show that all clustering methods which are invariant under additive shifts of the pairwise proximities can be reformulated as grouping problems in euclidian spaces the most prominent property of this constant shift embedding framework is the complete preservation of the cluster structure in the embedding space restating pairwise clustering problems in vector spaces has several important consequences such as the statistical description of the clusters by way of cluster prototypes the generic extension of the grouping procedure to a discriminative prediction rule and the applicability of standard preprocessing methods like denoising or dimensionality reduction

axioms for graph clustering quality functions we investigate properties that intuitively ought to be satisfied by graph clustering quality functions that is functions that assign a score to a clustering of a graph graph clustering also known as network community detection is often performed by optimizing such a function two axioms tailored for graph clustering quality functions are introduced and the four axioms introduced in previous work on distance based clustering are reformulated and generalized for the graph setting we show that modularity a standard quality function for graph clustering does not satisfy all of these six properties this motivates the derivation of a new family of quality functions adaptive scale modularity which does satisfy the proposed axioms adaptive scale modularity has two parameters which give greater flexibility in the kinds of clusterings that can be found standard graph clustering quality functions such as normalized cut and unnormalized cut are obtained as special cases of adaptive scale modularity in general the results of our investigation indicate that the considered axiomatic framework covers existing good quality functions for graph clustering and can be used to derive an interesting new family of quality functions

a spike train probability model poisson processes usually provide adequate descriptions of the irregularity in neuron spike times after pooling the data across large numbers of trials as is done in constructing the peristimulus time histogram when probabilities are needed to describe the behavior of neurons within individual trials however poisson process models are often inadequate in principle an explicit formula gives the probability density of a single spike train in great generality but without additional assumptions the firing rate intensity function appearing in that formula cannot be estimated we propose a simple solution to this problem which is to assume that the time at which a neuron fires is determined probabilistically by and only by two quantities the experimental clock time and the elapsed time since the previous spike we show that this model can be fitted with standard methods and software and that it may used successfully to fit neuronal data

a note on one sided jacobi algorithm we propose a one sided or implicit variant of the jacobi diagonalization algorithm for positive definite matrices the variant is based on a previous cholesky decomposition and currently uses essentially one square array which on output contains the matrix of eigenvectors thus reaching the storage economy of the symmetric ql algorithm the current array is accessed only columnwise which makes the algorithm attractive for various parallelized and or vectorized implementations even on a serial computer our algorithm shows improved efficiency in particular if the cholesky step is made with diagonal pivoting on matrices of ordern 25 200 our algorithm is about 2 3 5 times slower than ql thus being almost on the halfway between the standard jacobi and ql algorithms the previous cholesky decomposition can be performed with higher precision without extra time or storage costs thus offering considerable gains in accuracy with highly conditioned input matrices

on the information and representation of non euclidean pairwise data two common data representations are mostly used in intelligent data analysis namely the vectorial and the pairwise representation pairwise data which satisfy the restrictive conditions of euclidean spaces can be faithfully translated into a euclidean vectorial representation by embedding non metric pairwise data with violations of symmetry reflexivity or triangle inequality pose a substantial conceptual problem for pattern recognition since the amount of predictive structural information beyond what can be measured by embeddings is unclear we show by systematic modeling of non euclidean pairwise data that there exists metric violations which can carry valuable problem specific information furthermore euclidean and non metric data can be unified on the level of structural information contained in the data stable component analysis selects linear subspaces which are particularly insensitive to data fluctuations experimental results from different domains support our pattern recognition strategy

annealing and the normalized n cut we describe an annealing procedure that computes the normalized n cut of a weighted graph g the first phase transition computes the solution of the approximate normalized 2 cut problem while the low temperature solution computes the normalized n cut the intermediate solutions provide a sequence of refinements of the 2 cut that can be used to split the data to k clusters with 2

right coprime factorizations and stabilization for nonlinear systems the problem of constructing right coprime factorizations which are based on the graph of an input output map instead of the bezout identity of nonlinear input output maps is considered the map is assumed to arise from a state variable realization with a fixed initial state the main result is that the existence of a stabilizing state feedback implies the existence of a right coprime factorization for the map the technique is illustrated by application to nonlinear systems which are affine in the control and have a controllable linear past and nonlinear systems which are feedback linearizable a notion of input output stability that requires a bound on the magnitude of the input signals is introduced methods of constructing such bounds are developed for a locally feedback linearizable system the problem of input bounds is transferred to the equivalent linear system this leads to a technique that allows state and input constraints for a feedback linearizable system to be mapped to the equivalent linear system

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

spike and slab biclustering abstract biclustering refers to the problem of simultaneously clustering the rows and columns of a given data matrix with the goal of obtaining submatrices where the selected rows present a coherent behaviour in the selected columns and vice versa to face this intrinsically difficult problem we propose a novel generative model where biclustering is approached from a sparse low rank matrix factorization perspective the main idea is to design a probabilistic model describing the factorization of a given data matrix in two other matrices from which information about rows and columns belonging to the sought for biclusters can be obtained one crucial ingredient in the proposed model is the use of a spike and slab sparsity inducing prior thus we term the approach spike and slab biclustering ssbi to estimate the parameters of the ssbi model we propose an expectation maximization em algorithm termed ssbiem which solves a low rank factorization problem at each iteration using a recently proposed augmented lagrangian algorithm experiments with both synthetic and real data show that the ssbi approach compares favorably with the state of the art

complexity optimized data clustering by competitive neural networks data clustering is a complex optimization problem with applications ranging from vision and speech processing to data transmission and data storage in technical as well as in biological systems we discuss a clustering strategy that explicitly reflects the tradeoff between simplicity and precision of a data representation the resulting clustering algorithm jointly optimizes distortion errors and complexity costs a maximum entropy estimation of the clustering cost function yields an optimal number of clusters their positions and their cluster probabilities our approach establishes a unifying framework for different clustering methods like k means clustering fuzzy clustering entropy constrained vector quantization or topological feature maps and competitive neural networks

a jacobi eigenreduction algorithm for definite matrix pairs we propose a jacobi eigenreduction algorithm for symmetric definite matrix pairsa j of small to medium size real symmetric matrices withj 2 i j diagonal neitherj nora itself need be definite our jacobi reduction works only on one matrix and usesj orthogonal elementary congruences which include both trigonometric and hyperbolic rotations and preserve the symmetry throughout the process for the rotation parameters only the pivotal elements of the current matrix are needed which facilitates parallelization we prove the global convergence of the method the quadratic convergence was observed in all experiments we apply our method in two situations i eigenreducing a single real symmetric matrix and ii eigenreducing an overdamped quadratic matrix pencil in both cases our method is preceded by a symmetric indefinite decomposition and performed in its one sided variant on the thus obtained factors our method outdoes the standard methods like standard jacobi orqr ql in accuracy in spite of the use of hyperbolic transformations which are not orthogonal a theoretical justification of this behaviour is made elsewhere the accuracy advantage of our method can be particularly drastic if the eigenvalues are of different order in addition in working with quadratic pencils our method is shown to either converge or to detect non overdampedness

a theory of proximity based clustering structure detection by optimization abstract in this paper a systematic optimization approach for clustering proximity or similarity data is developed starting from fundamental invariance and robustness properties a set of axioms is proposed and discussed to distinguish different cluster compactness and separation criteria the approach covers the case of sparse proximity matrices and is extended to nested partitionings for hierarchical data clustering to solve the associated optimization problems a rigorous mathematical framework for deterministic annealing and mean field approximation is presented efficient optimization heuristics are derived in a canonical way which also clarifies the relation to stochastic optimization by gibbs sampling similarity based clustering techniques have a broad range of possible applications in computer vision pattern recognition and data analysis as a major practical application we present a novel approach to the problem of unsupervised texture segmentation which relies on statistical tests as a measure of homogeneity the quality of the algorithms is empirically evaluated on a large collection of brodatz like micro texture mondrians and on a set of real word images to demonstrate the broad usefulness of the theory of proximity based clustering the performances of different criteria and algorithms are compared on an information retrieval task for a document database the superiority of optimization algorithms for clustering is supported by extensive experiments

a cutting plane algorithm for a clustering problem in this paper we consider a clustering problem that arises in qualitative data analysis this problem can be transformed to a combinatorial optimization problem the clique partitioning problem we have studied the latter problem from a polyhedral point of view and determined large classes of facets of the associated polytope these theoretical results are utilized in this paper we describe a cutting plane algorithm that is based on the simplex method and uses exact and heuristic separation routines for some of the classes of facets mentioned before we discuss some details of the implementation of our code and present our computational results we mention applications from e g zoology economics and the political sciences

hierarchical unsupervised learning with growing via phase transitions we address unsupervised learning subject to structural constraints with particular emphasis placed on clustering with an imposed decision tree structure most known methods are greedy optimizing one node of the tree at a time to minimize a local cost by constrast we develop a joint optimization method derived based on information theoretic principles and closely related to known methods in statistical physics the approach is inspired by the deterministic annealing algorithm for unstructured data clustering which was based on maximum entropy inference the new approach is founded on the principle of minimum cross entropy using informative priors to approximate the unstructured clustering solution while imposing the structural constraint the resulting method incorporates supervised learning principles applied in an unsupervised problem setting in our approach the tree grows by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales thus estimates of the tree size and structure are naturally obtained at each temperature in the process examples demonstrate considerable improvement over known methods

population stratification using a statistical model on hypergraphs population stratification is a problem encountered in several areas of natural science engineering and public health we tackle this problem by mapping a population and its element attributes onto a hypergraph a natural extension of the concept of graph or network to encode associations among any number of elements on this hypergraph we construct a statistical model reflecting our intuition about how the element attributes can emerge from a postulated population structure finally we introduce the concept of stratification representativeness as a mean to identify the simplest stratification already containing most of the information about the population structure we demonstrate the power of this framework stratifying an animal and a human population based on phenotypic and genotypic properties respectively

information theoretic clustering clustering is an important topic in pattern recognition since only the structure of the data dictates the grouping unsupervised learning information theory is an obvious criteria to establish the clustering rule the paper describes a novel valley seeking clustering algorithm using an information theoretic measure to estimate the cost of partitioning the data set the information theoretic criteria developed here evolved from a renyi entropy estimator a renyi 1960 that was proposed recently and has been successfully applied to other machine learning applications j c principe et al 2000 an improved version of the k change algorithm is used in optimization because of the stepwise nature of the cost function and existence of local minima even when applied to nonlinearly separable data the new algorithm performs well and was able to find nonlinear boundaries between clusters the algorithm is also applied to the segmentation of magnetic resonance imaging data mri with very promising results

estimating the number of clusters in a data set via the gap statistic we propose a method the gap statistic for estimating the number of clusters groups in a set of data the technique uses the output of any clustering algorithm e g k means or hierarchical comparing the change in within cluster dispersion with that expected under an appropriate reference null distribution some theory is developed for the proposal and a simulation study shows that the gap statistic usually outperforms other methods that have been proposed in the literature

estimation of entropy and mutual information we present some new results on the nonparametric estimation of entropy and mutual information first we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators the setup is related to grenander s method of sieves and places no assumptions on the underlying probability measure generating the data second we prove a converse to these consistency theorems demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information even given unlimited data this inconsistency theorem leads to an analytical approximation of the bias valid in surprisingly small sample regimes and more accurate than the usual 1 n formula of miller and madow over a large region of parameter space the two most practical implications of these results are negative 1 information estimates in a certain data regime are likely contaminated by bias even if bias corrected estimators are used and 2 confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods finally we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem by casting bias calculation problems in this approximation theory framework we obtain the best possible generalization of known asymptotic bias results more interesting this framework leads to an estimator with some nice properties the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions and this maximum error turns out to be surprisingly small we demonstrate the application of this new estimator on both real and simulated data

information based clustering in an age of increasingly large data sets investigators in many different disciplines have turned to clustering as a tool for data analysis and exploration existing clustering methods however typically depend on several nontrivial assumptions about the structure of data here we reformulate the clustering problem from an information theoretic perspective that avoids many of these assumptions in particular our formulation obviates the need for defining a cluster prototype does not require an a priori similarity metric is invariant to changes in the representation of the data and naturally captures nonlinear relations we apply this approach to different domains and find that it consistently produces clusters that are more coherent than those extracted by existing algorithms finally our approach provides a way of clustering based on collective notions of similarity rather than the traditional pairwise measures

lautum information a popular way to measure the degree of dependence between two random objects is by their mutual information defined as the divergence between the joint and product of marginal distributions we investigate an alternative measure of dependence the lautum information defined as the divergence between the product of marginal and joint distributions i e swapping the arguments in the definition of mutual information some operational characterizations and properties are provided for this alternative measure of information