relaxation dynamics of perturbed regular hyperbranched fractals we focus on perturbed regular hyperbranched fractals prhf which are rhf whose f coordinated centers fcc are traps we compute the mechanical properties storage and loss modulus and the average displacement in the framework of generalized gaussian structures by making use of the eigenvalue spectrum of the connectivity matrix we generalize the analysis to the case of a connectivity matrix perturbed by a diagonal and pure imaginary operator although the above cited observables in this new situation lose their original meaning they still give important information about the underlying structures and they could help to analyze other phenomena where complex operators are involved we obtain analytically the eigenvalue spectrum for prhf a drastic change was observed in the behavior of the studied quantities even for a very small perturbation strength however it is still possible to depict the scaling of the fractals in the intermediate time frequency domains

how many clusters are best an experiment this paper reports the results of a monte carlo study on the relative effectiveness of two internal indices in estimating the true number of clusters in multivariate data the two indices are the davies and bouldin 1 index and a new modification of the hubert statistic 2 data in d dimensions are clustered to create sequences of partitions estimates are based on plots of the indices as functions of the number of recovered clusters neither index uses a priori information the effects of sample size dimensionality cluster spread number of true clusters and sampling window are examined clustered data are generated to assure a given number of distinct clusters the degree of clustering in the data is verified by a separate monte carlo study based on the jaccard and corrected rand indices that exhibits the importance of correcting external indices for chance the modified hubert index proposed here for the first time is shown to perform better than the davies bouldin index under all experimental conditions recovery of the true number of clusters gets better as the number of true clusters decreases and as the number of dimensions increases no effect occurs due to sampling window the complete link clustering method and a square error clustering method recognize the true number of clusters consistently better than the single link method this study demonstrates the difficulty inherent in estimating the number of clusters

statistical mechanics in the extended gaussian ensemble the extended gaussian ensemble ege is introduced as a generalization of the canonical ensemble this ensemble is a further extension of the gaussian ensemble introduced by hetherington j low temp phys 66 145 1987 the statistical mechanical formalism is derived both from the analysis of the system attached to a finite reservoir and from the maximum statistical entropy principle the probability of each microstate depends on two parameters b and g which allow one to fix independently the mean energy of the system and the energy fluctuations respectively we establish the legendre transform structure for the generalized ther modynamic potential and propose a stability criterion we also compare the ege probability distribution with the q exponential distribution as an example an application to a system with few independent spins is presented the development of statistical mechanics based on en semble theory is founded on the postulate of equal a priori probabilities which is assumed to apply to all microstates consistent with the given macrostate of an isolated system 1 the corresponding statistical ensemble is the so called microcanonical ensemble a representative system in this en semble has all mechanical variables such as energy e volume v magnetization m etc fixed for convenience in calculations other ensembles are used which invariably sup pose the existence of a subsidiary system or reservoir in con tact with the actual system for instance in the canonical ensemble the walls of the system permit an exchange of en ergy with the reservoir while in the grand canonical en semble both energy and matter can be exchanged in gen eral the different ensembles are constructed by allowing one or more mechanical variables to fluctuate the exchange of each of these variables is controlled by a parameter which is a characteristic of the reservoir for instance in the case of the canonical ensemble this parameter is precisely the tem perature of the reservoir and determines the mean energy of the system actually this is adequate when the reservoir is a very large system that can exchange arbitrary amounts of energy without modification of its intensive properties in practical situations this is not always the case however very few studies have been devoted to analyze the conse quences of possible deviations from these ideal reservoir properties in this paper we develop the statistical mechanics of a system that can exchange energy with a finite reservoir char acterized by two parameters b and g these parameters con trol independently the mean energy of the system and its energy fluctuations respectively the corresponding statisti cal ensemble represents a generalization of the canonical en semble and will be called the extended gaussian ensemble ege a similar ensemble was already developed in a more restricted framework by hetherington 2 the author con sidered that the sample system was in contact with a finite reservoir with size dependent properties the so called gaussian ensemble was introduced so that it is equivalent to the canonical ensemble in the limit of large systems except in the energy range of a first order transition interestingly it enables a smooth interpolation between the microcanonical and the canonical ensembles taking into account these fea tures challa and hetherington 3 4 showed the interest of this ensemble for monte carlo simulation studies of phase transitions they demonstrated a significant reduction in computer time compared to standard simulations in the ca nonical ensemble and its adequacy for distinguishing second order from first order transitions compared to the ege introduced in the present paper the main difference arises from the fact that in the gaussian ensemble the sample and the reservoir are assumed to be statistically independent which implies the additivity of the corresponding entropies this is not assumed in our formalism the consequences are important and will be discussed in depth in this work the present formalism can be considered as an alternative to the statistical mechanics based on nonadditive generalized entropies actually the study of such generalized entropies has generated a lot of interest in the past 15 years the mo tivation for the so called tsallis statistical mechanics has been to extend the standard boltzmann gibbs framework to include nonextensive systems 5 among different interpre tations it has been suggested that tsallis formalism corre sponds to an ensemble describing a system attached to a finite reservoir 6 7 although a large number of papers have been published the physical meaning of many related issues is still open to discussion 8 9 the ege formalism that we propose in this paper provides a clear and consistent frame work for the statistical mechanics with a finite reservoir the paper is organized as follows in secs ii and iii the ege is founded from the analysis of a contact with a finite reservoir and from the maximum statistical entropy prin ciple respectively in sec iv the main thermodynamic rela tions are derived in sec v we highlight the nonadditive nature of the thermodynamic formalism in sec vi a stabil

evaluation of stability of k means cluster ensembles with respect to random initialization many clustering algorithms including cluster ensembles rely on a random component stability of the results across different runs is considered to be an asset of the algorithm the cluster ensembles considered here are based on k means clusterers each clusterer is assigned a random target number of clusters k and is started from a random initialization here we use 10 artificial and 10 real data sets to study ensemble stability with respect to random k and random initialization the data sets were chosen to have a small number of clusters two to seven and a moderate number of data points up to a few hundred pairwise stability is defined as the adjusted rand index between pairs of clusterers in the ensemble averaged across all pairs nonpairwise stability is defined as the entropy of the consensus matrix of the ensemble an experimental comparison with the stability of the standard k means algorithm was carried out for k from 2 to 20 the results revealed that ensembles are generally more stable markedly so for larger k to establish whether stability can serve as a cluster validity index we first looked at the relationship between stability and accuracy with respect to the number of clusters k we found that such a relationship strongly depends on the data set varying from almost perfect positive correlation 0 97 for the glass data to almost perfect negative correlation 0 93 for the crabs data we propose a new combined stability index to be the sum of the pairwise individual and ensemble stabilities this index was found to correlate better with the ensemble accuracy following the hypothesis that a point of stability of a clustering algorithm corresponds to a structure found in the data we used the stability measures to pick the number of clusters the combined stability index gave best results

finding the number of clusters in a dataset one of the most difficult problems in cluster analysis is identifying the number of groups in a dataset most previously suggested approaches to this problem are either somewhat ad hoc or require parametric assumptions and complicated calculations in this article we develop a simple yet powerful nonparametric method for choosing the number of clusters based on distortion a quantity that measures the average distance per dimension between each observation and its closest cluster center our technique is computationally efficient and straightforward to implement we demonstrate empirically its effectiveness not only for choosing the number of clusters but also for identifying underlying structure on a wide range of simulated and real world datasets in addition we give a rigorous theoretical justification for the method based on information theoretic ideas specifically results from the subfield of electrical engineering known as rate distortion theory allow us to describe the behavior of the dist

conjugate variables in continuous maximum entropy inference for a continuous maximum entropy distribution obtained from an arbitrary number of simultaneous constraints we derive a general relation connecting the lagrange multipliers and the expectation values of certain particularly constructed functions of the states of the system from this relation an estimator for a given lagrange multiplier can be constructed from derivatives of the corresponding constraining function these estimators sometimes lead to the determination of the lagrange multipliers by way of solving a linear system and in general they provide another tool to widen the applicability of jaynes s formalism this general relation especially well suited for computer simulation techniques also provides some insight into the interpretation of the hypervirial relations known in statistical mechanics and the recently derived microcanonical dynamical temperature we illustrate the usefulness of these new relations with several applications in statistics

external clustering validity index based on chi squared statistical test abstract clustering is one of the most commonly used techniques in data mining its main goal is to group objects into clusters so that each group contains objects that are more similar to each other than to objects in other clusters the evaluation of a clustering solution is a task carried out through the application of validity indices these indices measure the quality of the solution and can be classified as either internal that calculate the quality of the solution through the data of the clusters or as external indices that measure the quality by means of external information such as the class generally indices from the literature determine their optimal result through graphical representation whose results could be imprecisely interpreted the aim of this paper is to present a new external validity index based on the chi squared statistical test named chi index which presents accurate results that require no further interpretation chi index was analyzed using the clustering results of 3 clustering methods in 47 public datasets results indicate a better hit rate and a lower percentage of error against 15 external validity indices from the literature

emergence of tsallis statistics as a consequence of invariance abstract for non equilibrium systems in a steady state we present two necessary and sufficient conditions for the emergence of q canonical ensembles also known as tsallis statistics these conditions are invariance requirements over the definition of subsystem and environment and over the joint rescaling of temperature and energy our approach is complementary to the notions of tsallis non extensive statistics and superstatistics

silhouettes a graphical aid to the interpretation and validation of cluster analysis a new graphical display is proposed for partitioning techniques each cluster is represented by a so called silhouette which is based on the comparison of its tightness and separation this silhouette shows which objects lie well within their cluster and which ones are merely somewhere in between clusters the entire clustering is displayed by combining the silhouettes into a single plot allowing an appreciation of the relative quality of the clusters and an overview of the data configuration the average silhouette width provides an evaluation of clustering validity and might be used to select an appropriate number of clusters

bootstrap technique in cluster analysis abstract we define a method to estimate the number of clusters in a data set e using the bootstrap technique this approach involves the generation of several fake data sets by sampling patterns with replacement in e bootstrapping for each number k of clusters a measure of stability of the k cluster partitions over the bootstrap samples is used to characterize the significance of the k cluster partition for the original data set the value of k which provides the most stable partitions is the estimate of the number of clusters in e the performance of this new technique is demonstrated on both synthetic and real data and is applied to the segmentation of range images

information theoretic measures for clusterings comparison variants properties normalization and correction for chance information theoretic measures form a fundamental class of measures for comparing clusterings and have recently received increasing interest nevertheless a number of questions concerning their properties and inter relationships remain unresolved in this paper we perform an organized study of information theoretic measures for clustering comparison including several existing popular measures in the literature as well as some newly proposed ones we discuss and prove their important properties such as the metric property and the normalization property we then highlight to the clustering community the importance of correcting information theoretic measures for chance especially when the data size is small compared to the number of clusters present therein of the available information theoretic based measures we advocate the normalized information distance nid as a general measure of choice for it possesses concurrently several important properties such as being both a metric and a normalized measure admitting an exact analytical adjusted for chance form and using the nominal 0 1 range better than other normalized variants

on the statistics of generalized gaussian structures collapse and random external fields we consider the statistics of generalized gaussian structures ggs exposed to a random external field a ggs comprises n monomers connected to each other by harmonic potentials when the spectral dimension d of a ggs exceeds the value of two its radius of gyration r becomes independent of its mass n the cross over into this collapse can be treated continuously by cross linking m precursor chains of length n in the stretched state to an object which we call a polymer bundle we demonstrate that an external field f applied to each monomer can unfold such a collapsed state in the case where every monomer has an individual randomly distributed charge the critical spectral dimension for the collapse is raised to four r scales like fnalpha with alpha 4 ds 2ds for ds 4

performance evaluation of some clustering algorithms and validity indices in this article we evaluate the performance of three clustering algorithms hard k means single linkage and a simulated annealing sa based technique in conjunction with four cluster validity indices namely davies bouldin index dunn s index calinski harabasz index and a recently developed index i based on a relation between the index i and the dunn s index a lower bound of the value of the former is theoretically estimated in order to get unique hard k partition when the data set has distinct substructures the effectiveness of the different validity indices and clustering methods in automatically evolving the appropriate number of clusters is demonstrated experimentally for both artificial and real life data sets with the number of clusters varying from two to ten once the appropriate number of clusters is determined the sa based clustering technique is used for proper partitioning of the data into the said number of clusters

temperature is not an observable in superstatistics abstract superstatistics beck and cohen 2003 is a formalism that attempts to explain the presence of distributions other than the boltzmann gibbs distributions in nature typically power law behavior for systems out of equilibrium such as fluids under turbulence plasmas and gravitational systems superstatistics postulates that those systems are found in a superposition of canonical ensembles at different temperatures and sometimes the physical interpretation is one of local thermal equilibrium in the sense of an inhomogeneous temperature distribution in different regions of space or instants of time here we show that in order for superstatistics to be internally consistent it is impossible to define a phase space function or microscopic observable b p q corresponding one to one to the local value of 1 k b t thus unlike energy which is defined by a phase space function h p q the hamiltonian temperature is not a microscopic observable an important consequence of our proof is that in superstatistics the identification of temperature with the kinetic energy is limited to the expectation of and cannot be used to measure the different temperatures in local thermal equilibrium or its fluctuations

model based evaluation of clustering validation measures a cluster operator takes a set of data points and partitions the points into clusters subsets as with any scientific model the scientific content of a cluster operator lies in its ability to predict results this ability is measured by its error rate relative to cluster formation to estimate the error of a cluster operator a sample of point sets is generated the algorithm is applied to each point set and the clusters evaluated relative to the known partition according to the distributions and then the errors are averaged over the point sets composing the sample many validity measures have been proposed for evaluating clustering results based on a single realization of the random point set process in this paper we consider a number of proposed validity measures and we examine how well they correlate with error rates across a number of clustering algorithms and random point set models validity measures fall broadly into three classes internal validation is based on calculating properties of the resulting clusters relative validation is based on comparisons of partitions generated by the same algorithm with different parameters or different subsets of the data and external validation compares the partition generated by the clustering algorithm and a given partition of the data to quantify the degree of similarity between the validation indices and the clustering errors we use kendall s rank correlation between their values our results indicate that overall the performance of validity indices is highly variable for complex models or when a clustering algorithm yields complex clusters both the internal and relative indices fail to predict the error of the algorithm some external indices appear to perform well whereas others do not we conclude that one should not put much faith in a validity score unless there is evidence either in terms of sufficient data for model estimation or prior model knowledge that a validity measure is well correlated to the error rate of the clustering algorithm

possible generalization of boltzmann gibbs statistics with the use of a quantity normally scaled in multifractals a generalized form is postulated for entropy namelys q k 1 i 1 w p i q q 1 whereq characterizes the generalization andp i are the probabilities associated withw microscopic configurations w the main properties associated with this entropy are established particularly those corresponding to the microcanonical and canonical ensembles the boltzmann gibbs statistics is recovered as theq 1 limit

generalized adjusted rand indices for cluster ensembles in this paper adjusted rand index ari is generalized to two new measures based on matrix comparison i adjusted rand index between a similarity matrix and a cluster partition arimp to evaluate the consistency of a set of clustering solutions with their corresponding consensus matrix in a cluster ensemble and ii adjusted rand index between similarity matrices arimm to evaluate the consistency between two similarity matrices desirable properties of ari are preserved in the two new measures and new properties are discussed these properties include i detection of uncorrelatedness ii computation of arimp arimm in a distributed environment and iii characterization of the degree of uncertainty of a consensus matrix all of these properties are investigated from both the perspectives of theoretical analysis and experimental validation we have also performed a number of experiments to show the usefulness and effectiveness of the two proposed measures in practical applications

an examination of indexes for determining the number of clusters in binary data sets the problem of choosing the correct number of clusters is as old as cluster analysis itself a number of authors have suggested various indexes to facilitate this crucial decision one of the most extensive comparative studies of indexes was conducted by milligan and cooper 1985 the present piece of work pursues the same goal under different conditions in contrast to milligan and cooper s work the emphasis here is on high dimensional empirical binary data binary artificial data sets are constructed to reflect features typically encountered in real world data situations in the field of marketing research the simulation includes 162 binary data sets that are clustered by two different algorithms and lead to recommendations on the number of clusters for each index under consideration index results are evaluated and their performance is compared and analyzed

an extensive comparative study of cluster validity indices the validation of the results obtained by clustering algorithms is a fundamental part of the clustering process the most used approaches for cluster validation are based on internal cluster validity indices although many indices have been proposed there is no recent extensive comparative study of their performance in this paper we show the results of an experimental work that compares 30 cluster validity indices in many different environments with different characteristics these results can serve as a guideline for selecting the most suitable index for each possible application and provide a deep insight into the performance differences between the currently available indices

a method for comparing two hierarchical clusterings abstract this article concerns the derivation and use of a measure of similarity between two hierarchical clusterings the measure bk is derived from the matching matrix mij formed by cutting the two hierarchical trees and counting the number of matching entries in the k clusters in each tree the mean and variance of bk are determined under the assumption that the margins of mij are fixed thus bk represents a collection of measures for k 2 n 1 k bk plots are found to be useful in portraying the similarity of two clusterings bk is compared to other measures of similarity proposed respectively by baker 1974 and rand 1971 the use of k bk plots for studying clustering methods is explored by a series of monte carlo sampling experiments an example of the use of k bk on real data is given

an examination of procedures for determining the number of clusters in a data set a monte carlo evaluation of 30 procedures for determining the number of clusters was conducted on artificial data sets which contained either 2 3 4 or 5 distinct nonoverlapping clusters to provide a variety of clustering solutions the data sets were analyzed by four hierarchical clustering methods external criterion measures indicated excellent recovery of the true cluster structure by the methods at the correct hierarchy level thus the clustering present in the data was quite strong the simulation results for the stopping rules revealed a wide range in their ability to determine the correct number of clusters in the data several procedures worked fairly well whereas others performed rather poorly thus the latter group of rules would appear to have little validity particularly for data sets containing distinct clusters applied researchers are urged to select one or more of the better criteria however users are cautioned that the performance of some of the criteria may be data dependent

quantum walks and trapping on regular hyperbranched fractals the present work studies the behaviour of continuous time quantum walks on regular hyperbranched fractals whose centre is a trap we focus on the variations of the eigenvalue spectrum of the transfer operator by tuning the trap strength from zero to infinity we show that the degenerate eigenvalues are independent from the trap strength and can be obtained analytically due to this the mean survival probability is just in the intermediate range affected by the trap strength moreover because of the presence of real eigenvalues the asymptotical probability of being outside the trap is not zero

iterative shrinking method for clustering problems agglomerative clustering generates the partition hierarchically by a sequence of merge operations we propose an alternative to the merge based approach by removing the clusters iteratively one by one until the desired number of clusters is reached we apply local optimization strategy by always removing the cluster that increases the distortion the least data structures and their update strategies are considered the proposed algorithm is applied as a crossover method in a genetic algorithm and compared against the best existing clustering algorithms the proposed method provides best performance in terms of minimizing intra cluster variance

critique of q entropy for thermal statistics during the past 12 years there have been numerous papers on a relation between entropy and probability which is nonadditive and has a parameter q that depends on the nature of the thermodynamic system under consideration for q 1 this relation corresponds to the boltzmann gibbs entropy but for other values of q it is claimed that it leads to a formalism that is consistent with the laws of thermodynamics however it is shown here that the joint entropy for systems having different values of q is not defined in this formalism and consequently fundamental thermodynamic concepts such as temperature and heat exchange cannot be considered for such systems moreover for q ensuremath ne 1 the probability distribution for weakly interacting systems does not factor into the product of the probability distribution for the separate systems leading to spurious correlations and other unphysical consequences e g nonextensive energy that have been ignored in various applications given in the literature

cluster validation using graph theoretic concepts in this article we have generalized dunn s index and the davies bouldin index for cluster validation using graph structures such as gg rng and mst unlike dunn s index and the davies bouldin index the proposed indices are not sensitive to noisy points and are applicable to hyperspherical and structural clusters as well the relationships between various indices have also been established the effectiveness of the generalized indices and superiority over some existing cluster validity indices are established using eight data sets

fast diagonalization of large and dense complex symmetric matrices with applications to quantum reaction dynamics we present a new fast and efficient algorithm for computing the eigenvalues and eigenvectors of large size nondefective complex symmetric matrices our work was motivated by the emergence of this problem in recent methods for solving chemical reactive problems the algorithm we present is similiar to the qr ql algorithm for complex hermitian matrices but we use complex orthogonal not unitary transformations the new algorithm is faster by an order of magnitude than the corresponding eispack routine and is also more amenable for modern parallel and vector supercomputers we further present improved perturbation bounds for the householder transformation which lies at the basis of the whole transformation

clustering by competitive agglomeration we present a new clustering algorithm called competitive agglomeration ca which minimizes an objective function that incorporates the advantages of both hierarchical and partitional clustering the ca algorithm produces a sequence of partitions with a decreasing number of clusters the initial partition has an over specified number of clusters and the final one has the optimal number of clusters the update equation in the ca algorithm creates an environment in which clusters compete for feature points and only clusters with large cardinalities survive the algorithm can incorporate different distance measures in the objective function to f find an unknown number of clusters of various shapes

normality based validation for crisp clustering we introduce a new validity index for crisp clustering that is based on the average normality of the clusters unlike methods based on inter cluster and intra cluster distances this index emphasizes the cluster shape by using a high order characterization of its probability distribution the normality of a cluster is characterized by its negentropy a standard measure of the distance to normality which evaluates the difference between the cluster s entropy and the entropy of a normal distribution with the same covariance matrix the definition of the negentropy involves the distribution s differential entropy however we show that it is possible to avoid its explicit computation by considering only negentropy increments with respect to the initial data distribution where all the points are assumed to belong to the same cluster the resulting negentropy increment validity index only requires the computation of covariance matrices we have applied the new index to an extensive set of artificial and real problems where it provides in general better results than other indices both with respect to the prediction of the correct number of clusters and to the similarity among the real clusters and those inferred

fast agglomerative clustering using a k nearest neighbor graph we propose a fast agglomerative clustering method using an approximate nearest neighbor graph for reducing the number of distance calculations the time complexity of the algorithm is improved from o taun2 to o taun log n at the cost of a slight increase in distortion here tau denotes the lumber of nearest neighbor updates required at each iteration according to the experiments a relatively small neighborhood size is sufficient to maintain the quality close to that of the full search

centroid index cluster level similarity measure abstract in clustering algorithm one of the main challenges is to solve the global allocation of the clusters instead of just local tuning of the partition borders despite this all external cluster validity indexes calculate only point level differences of two partitions without any direct information about how similar their cluster level structures are in this paper we introduce a cluster level index called centroid index the measure is intuitive simple to implement fast to compute and applicable in case of model mismatch as well to a certain extent we expect it to generalize other clustering models beyond the centroid based k means as well

sep cop an efficient method to find the best partition in hierarchical clustering based on a new cluster validity index hierarchical clustering algorithms provide a set of nested partitions called a cluster hierarchy since the hierarchy is usually too complex it is reduced to a single partition by using cluster validity indices we show that the classical method is often not useful and we propose sep a new method that efficiently searches in an extended partition set furthermore we propose a new cluster validity index cop since many of the commonly used indices cannot be used with sep experiments performed with 80 synthetic and 7 real datasets confirm that sep cop is superior to the method currently used and furthermore it is less sensitive to noise

cluster ensembles a knowledge reuse framework for combining multiple partitions this paper introduces the problem of combining multiple partitionings of a set of objects into a single consolidated clustering without accessing the features or algorithms that determined these partitionings we first identify several application scenarios for the resultant knowledge reuse framework that we call cluster ensembles the cluster ensemble problem is then formalized as a combinatorial optimization problem in terms of shared mutual information in addition to a direct maximization approach we propose three effective and efficient techniques for obtaining high quality combiners consensus functions the first combiner induces a similarity measure from the partitionings and then reclusters the objects the second combiner is based on hypergraph partitioning the third one collapses groups of clusters into meta clusters which then compete for each object to determine the combined clustering due to the low computational costs of our techniques it is quite feasible to use a supra consensus function that evaluates all three approaches against the objective function and picks the best solution for a given situation we evaluate the effectiveness of cluster ensembles in three qualitatively different application scenarios i where the original clusters were formed based on non identical sets of features ii where the original clustering algorithms worked on non identical sets of objects and iii where a common data set is used and the main purpose of combining multiple clusterings is to improve the quality and robustness of the solution promising results are obtained in all three situations for synthetic as well as real data sets