hypothesis testing for automated community detection in networks summary community detection in networks is a key exploratory tool with applications in a diverse set of areas ranging from finding communities in social and biological networks to identifying link farms in the world wide web the problem of finding communities or clusters in a network has received much attention from statistics physics and computer science however most clustering algorithms assume knowledge of the number of clusters k we propose to determine k automatically in a graph generated from a stochastic block model by using a hypothesis test of independent interest our main contribution is twofold first we theoretically establish the limiting distribution of the principal eigenvalue of the suitably centred and scaled adjacency matrix and use that distribution for our test of the hypothesis that a random graph is of erd s renyi noise type secondly we use this test to design a recursive bipartitioning algorithm which naturally uncovers nested community structure using simulations and quantifiable classification tasks on real world networks with ground truth we show that our algorithm outperforms state of the art methods

asymptotics in directed exponential random graph models with an increasing bi degree sequence although asymptotic analyses of undirected network models based on degree sequences have started to appear in recent literature it remains an open problem to study the statistical properties of directed network models in this paper we provide for the first time a rigorous analysis of directed exponential random graph models using the in degrees and out degrees as sufficient statistics with binary and non binary weighted edges we establish the uniform consistency and the asymptotic normality of the maximum likelihood estimator when the number of parameters grows and only one realized observation of the graph is available one key technique in the proofs is to approximate the inverse of the fisher information matrix using a simple matrix with high accuracy along the way we also establish a geometrically fast rate of convergence for the newton iterative algorithm which is used to obtain the maximum likelihood estimate numerical studies confirm our theoretical findings

minimax rates of community detection in stochastic block models in statistics as well as in computer science probability and applied mathematics community detection for the stochastic block model sbm is probably the most studied topic in network analysis many methodologies have been proposed some beautiful and signicant phase transition results are obtained in various settings in this paper we provide a general minimax theory for community detection it gives minimax rates of the mis match ratio for a wide rage of settings including homogeneous and inhomogeneous sbms dense and sparse networks nite and growing number of communities the minimax rates are exponential dierent from polynomial rates we often see in statistical literature an immediate consequence of the result is to establish threshold phenomenon for strong consistency exact recovery as well as weak consistency partial recovery we obtain the upper bound by a range of penalized likelihood type approaches the lower bound is achieved by a novel reduction from a global mis match ratio to a local clustering problem for one node through an exchangeability property

network cross validation for determining the number of communities in network data abstractthe stochastic block model sbm and its variants have been a popular tool for analyzing large network data with community structures in this article we develop an efficient network cross validation ncv approach to determine the number of communities as well as to choose between the regular stochastic block model and the degree corrected block model dcbm the proposed ncv method is based on a block wise node pair splitting technique combined with an integrated step of community recovery using sub blocks of the adjacency matrix we prove that the probability of under selection vanishes as the number of nodes increases under mild conditions satisfied by a wide range of popular community recovery algorithms the solid performance of our method is also demonstrated in extensive simulations and two data examples supplementary materials for this article are available online

nonextensive entropies from information content and stabilities abstract we consider a family of nonextensive entropies derived from information contents which are a basic ingredient in information theory to give a deeper insight into a certain class of stability of generalized entropies and into recent discussion on it we present a criterion for stability which is satisfied by a minimum discrepancy against infinitesimal shift of distributions

general thermostatistical formalisms invariance under uniform spectrum translations and tsallis q additivity the recent proposal by tsallis of a generalized thermostatistics devised to treat systems endowed with long range interactions long range memory eects or a fractal like relevant phase space has raised interesting and profound issues concerning the properties of general thermostatistical formalisms in the present paper we identify families of thermostatistical formalisms that share some fundamental characteristics the canonical ensemble s invariance under uniform translations of the hamiltonian s energy spectrum is shown to be a universal property veried by any thermostatistical formalism based upon linear mean energy constraints we also provide multiparametric families of entropies exhibiting tsallis q additivity law c 1999 elsevier science b v all rights reserved

stochastic blockmodels and community structure in networks stochastic blockmodels have been proposed as a tool for detecting community structure in networks as well as for generating synthetic networks for use as benchmarks most blockmodels however ignore variation in vertex degree making them unsuitable for applications to real world networks which typically display broad degree distributions that can significantly affect the results here we demonstrate how the generalization of blockmodels to incorporate this missing element leads to an improved objective function for community detection in complex networks we also propose a heuristic algorithm for community detection using this objective function or its non degree corrected counterpart and show that the degree corrected version dramatically outperforms the uncorrected one in both real world and synthetic networks

community extraction for social networks analysis of networks and in particular discovering communities within networks has been a focus of recent work in several fields and has diverse applications most community detection methods focus on partitioning the entire network into communities with the expectation of many ties within communities and few ties between however many networks contain nodes that do not fit in with any of the communities and forcing every node into a community can distort results here we propose a new framework that extracts one community at a time allowing for arbitrary structure in the remainder of the network which can include weakly connected nodes the main idea is that the strength of a community should depend on ties between its members and ties to the outside world but not on ties between nonmembers the proposed extraction criterion has a natural probabilistic interpretation in a wide class of models and performs well on simulated and real networks for the case of the block model we establish asymptotic consistency of estimated node labels and propose a hypothesis test for determining the number of communities

finding community structure in networks using the eigenvectors of matrices we consider the problem of detecting communities or modules in networks groups of vertices with a higher than average density of edges connecting them previous work indicates that a robust approach to this problem is the maximization of the benefit function known as modularity over possible divisions of a network here we show that this maximization process can be written in terms of the eigenspectrum of a matrix we call the modularity matrix which plays a role in community detection similar to that played by the graph laplacian in graph partitioning calculations this result leads us to a number of possible algorithms for detecting community structure as well as several other results including a spectral measure of bipartite structure in networks and a centrality measure that identifies vertices that occupy central positions within the communities to which they belong the algorithms and measures proposed are illustrated with applications to a variety of real world complex networks

pseudo likelihood methods for community detection in large sparse networks many algorithms have been proposed for fitting network models with communities but most of them do not scale well to large networks and often fail on sparse networks here we propose a new fast pseudo likelihood method for fitting the stochastic block model for networks as well as a variant that allows for an arbitrary degree distribution by conditioning on degrees we show that the algorithms perform well under a range of settings including on very sparse networks and illustrate on the example of a network of political blogs we also propose spectral clustering with perturbations a method of independent interest which works well on sparse networks where regular spectral clustering fails and use it to provide an initial value for pseudo likelihood we prove that pseudo likelihood provides consistent estimates of the communities under a mild condition on the starting value for the case of a block model with two communities

normalized cuts and image segmentation we propose a novel approach for solving the perceptual grouping problem in vision rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image we treat image segmentation as a graph partitioning problem and propose a novel global criterion the normalized cut for segmenting the graph the normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups we show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion we applied this approach to segmenting static images as well as motion sequences and found the results to be very encouraging

nondistributive algebraic structures derived from nonextensive statistical mechanics we propose a two parametric nondistributive algebraic structure that follows from q q logarithm and q q exponential functions properties of generalized q q operators are analyzed we also generalize the proposal into a multiparametric structure generalization of logarithm and exponential functions and the corresponding algebraic operators all n parameter expressions recover n 1 generalization when the corresponding qn 1 nonextensive statistical mechanics has been the source of successive generalizations of entropic forms and mathematical structures in which this work is a consequence

asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications in this paper we extend our previous work on the stochastic block model a commonly used generative model for social and biological networks and the problem of inferring functional groups or communities from the topology of the network we use the cavity method of statistical physics to obtain an asymptotically exact analysis of the phase diagram we describe in detail properties of the detectability undetectability phase transition and the easy hard phase transition for the community detection problem our analysis translates naturally into a belief propagation algorithm for inferring the group memberships of the nodes in an optimal way i e that maximizes the overlap with the underlying group memberships and learning the underlying parameters of the block model finally we apply the algorithm to two examples of real world networks and discuss its performance

stochastic blockmodels for directed graphs abstract holland and leinhardt 1981 proposed the p 1 model for the analysis of binary directed graph data in network studies such a model provides information about the attractiveness and expansiveness of the individual nodes in the network as well as the tendency of a pair of nodes to reciprocate relational ties when the nodes are a priori partitioned into subgroups based on attributes such as race and sex the density of ties from one subgroup to another can differ considerably from that relating another pair of subgroups thus creating a situation called blocking in social networks the p 1 model completely ignores this extra piece of information and is therefore unable to explain the block structure blockmodels that are simple extensions of the p 1 model are proposed specifically for such data an iterative scaling algorithm is presented for fitting the model parameters by maximum likelihood the methodology is illustrated in detail on two empirical examples

on a general link between anomalous diffusion and nonextensivity stochastic processes of systems with composable entropy measures are studied the processes are described by fokker planck equations which are nonlinear with respect to their probability densities and whose diffusion coefficients are derived from the entropy measures in questions in nonextensive extensive systems anomalous normal diffusion is found

estimation and prediction for stochastic blockstructures a statistical approach to a posteriori blockmodeling for digraphs and valued digraphs is proposed the probability model assumes that the vertices of the digraph are partitioned into several unobserved latent classes and that the probability distribution of the relation between two vertices depends only on the classes to which they belong a bayesian estimator based on gibbs sampling is proposed the basic model is not identified because class labels are arbitrary the resulting identifiability problems are solved by restricting inference to the posterior distributions of invariant functions of the parameters and the vertex class membership in addition models are considered where class labels are identified by prior distributions for the class membership of some of the vertices the model is illustrated by an example from the social networks literature kapferer s tailor shop

consistency of community detection in networks under degree corrected stochastic block models community detection is a fundamental problem in network analysis with applications in many diverse areas the stochastic block model is a common tool for model based community detection and asymptotic tools for checking consistency of community detection under the block model have been recently developed however the block model is limited by its assumption that all nodes within a community are stochastically equivalent and provides a poor fit to networks with hubs or highly varying node degrees within communities which are common in practice the degree corrected stochastic block model was proposed to address this shortcoming and allows variation in node degrees within a community while preserving the overall block community structure in this paper we establish general theory for checking consistency of community detection under the degree corrected stochastic block model and compare several community detection criteria under both the standard and the degree corrected models we show which criteria are consistent under which models and constraints as well as compare their relative performance in practice we find that methods based on the degree corrected block model which includes the standard block model as a special case are consistent under a wider class of models and that modularity type methods require parameter constraints for consistency whereas likelihood based methods do not on the other hand in practice the degree correction involves estimating many more parameters and empirically we find it is only worth doing if the node degrees within communities are indeed highly variable we illustrate the methods on simulated networks and on a network of political blogs

on semidefinite relaxations for the block model the stochastic block model sbm is a popular tool for community detection in networks but fitting it by maximum likelihood mle involves a computationally infeasible optimization problem we propose a new semidefinite programming sdp solution to the problem of fitting the sbm derived as a relaxation of the mle we put ours and previously proposed sdps in a unified framework as relaxations of the mle over various subclasses of the sbm which also reveals a connection to the well known problem of sparse pca our main relaxation which we call sdp 1 is tighter than other recently proposed sdp relaxations and thus previously established theoretical guarantees carry over however we show that sdp 1 exactly recovers true communities over a wider class of sbms than those covered by current results in particular the assumption of strong assortativity of the sbm implicit in consistency conditions for previously proposed sdps can be relaxed to weak assortativity for our approach thus significantly broadening the class of sbms covered by the consistency results we also show that strong assortativity is indeed a necessary condition for exact recovery for previously proposed sdp approaches and not an artifact of the proofs our analysis of sdps is based on primal dual witness constructions which provides some insight into the nature of the solutions of various sdps in particular we show how to combine features from sdp 1 and already available sdps to achieve the most flexibility in terms of both assortativity and block size constraints as our relaxation has the tendency to produce communities of similar sizes this tendency makes it the ideal tool for fitting network histograms a method gaining popularity in the graphon estimation literature as we illustrate on an example of a social networks of dolphins we also provide empirical evidence that sdps outperform spectral methods for fitting sbms with a large number of blocks

possible generalization of boltzmann gibbs statistics with the use of a quantity normally scaled in multifractals a generalized form is postulated for entropy namelys q k 1 i 1 w p i q q 1 whereq characterizes the generalization andp i are the probabilities associated withw microscopic configurations w the main properties associated with this entropy are established particularly those corresponding to the microcanonical and canonical ensembles the boltzmann gibbs statistics is recovered as theq 1 limit

estimation of the number of communities in the stochastic block model in this article we introduce an estimator for the number of communities in the stochastic block model sbm based on the maximization of a penalized version of the so called krichevsky trofimov mixture distribution we prove its eventual almost sure convergence to the underlying number of communities without assuming a known upper bound on that quantity our results apply to both the dense and the sparse regimes to our knowledge this is the first consistency result for the estimation of the number of communities in the sbm in the unbounded case that is when the number of communities is allowed to grow with the same size

stochastic blockmodels with a growing number of classes we present asymptotic and finite sample results on the use of stochastic blockmodels for the analysis of network data we show that the fraction of misclassified network nodes converges in probability to zero under maximum likelihood fitting when the number of classes is allowed to grow as the root of the network size and the average network degree grows at least poly logarithmically in this size we also establish finite sample confidence bounds on maximum likelihood blockmodel parameter estimates from data comprising independent bernoulli random variates these results hold uniformly over class assignment we provide simulations verifying the conditions sufficient for our results and conclude by fitting a logit parameterization of a stochastic blockmodel with covariates to a network data example comprising self reported school friendships resulting in block estimates that reveal residual structure

optimization via low rank approximation for community detection in networks author s le can m levina elizaveta vershynin roman abstract community detection is one of the fundamental problems of network analysis for which a number of methods have been proposed most model based or criteria based methods have to solve an optimization problem over a discrete set of labels to find communities which is computationally infeasible some fast spectral algorithms have been proposed for specific methods or models but only on a case by case basis here we propose a general approach for maximizing a function of a network adjacency matrix over discrete labels by projecting the set of labels onto a subspace approximating the leading eigenvectors of the expected adjacency matrix this projection onto a low dimensional space makes the feasible set of labels much smaller and the optimization problem much easier we prove a general result about this method and show how to apply it to several previously proposed community detection criteria establishing its consistency for label estimation in each case and demonstrating the fundamental connection between spectral properties of the network and various model based approaches to community detection simulations and applications to real world data are included to demonstrate our method performs well for multiple problems over a wide range of parameters

general exact solution to the problem of the probability density for sums of random variables the exact explicit expression for the probability density p n x for a sum of n random arbitrary correlated summands is obtained the expression is valid for any number n and any distribution of the random summands most attention is paid to application of the developed approach to the case of independent and identically distributed summands the obtained results reproduce all known exact solutions valid for the so called stable distributions of the summands it is also shown that if the distribution is not stable the profile of p n x may be divided into three parts namely a core small x a tail large x and a crossover from the core to the tail moderate x the quantitative description of all three parts as well as that for the entire profile is obtained a number of particular examples are considered in detail

network cross validation by edge sampling summary while many statistical models and methods are now available for network analysis resampling of network data remains a challenging problem cross validation is a useful general tool for model selection and parameter tuning but it is not directly applicable to networks since splitting network nodes into groups requires deleting edges and destroys some of the network structure in this paper we propose a new network resampling strategy based on splitting node pairs rather than nodes that is applicable to cross validation for a wide range of network model selection tasks we provide theoretical justification for our method in a general setting and examples of how the method can be used in specific network model selection and parameter tuning tasks numerical results on simulated networks and on a statisticians citation network show that the proposed cross validation approach works well for model selection

statistical mechanics based on renyi entropy abstract in this work we show that it is possible to obtain a generalized statistical mechanics thermostatistics based on renyi entropy to be maximized with adequate constraints the equilibrium probability distribution thus obtained has a very interesting property indeed it reminds us the statistical distribution proposed by tsallis known to conveniently describe a variety of phenomena in nonextensive systems moreover some examples are worked out in order to illustrate the main features of the herein introduced formalism

network vector autoregression we consider here a large scale social network with a continuous response observed for each node at equally spaced time points the responses from different nodes constitute an ultra high dimensional vector whose time series dynamic is to be investigated in addition the network structure is also taken into consideration for which we propose a network vector autoregressive nar model the nar model assumes each node s response at a given time point as a linear combination of a its previous value b the average of its connected neighbors c a set of node specific covariates and d an independent noise the corresponding coefficients are referred to as the momentum effect the network effect and the nodal effect respectively conditions for strict stationarity of the nar models are obtained in order to estimate the nar model an ordinary least squares type estimator is developed and its asymptotic properties are investigated we further illustrate the usefulness of the nar model through a number of interesting potential applications simulation studies and an empirical example are presented

a block model for node popularity in networks with community structure summary the community structure that is observed in empirical networks has been of particular interest in the statistics literature with a strong emphasis on the study of block models we study an important network feature called node popularity which is closely associated with community structure neither the classical stochastic block model nor its degree corrected extension can satisfactorily capture the dynamics of node popularity as observed in empirical networks we propose a popularity adjusted block model for flexible and realistic modelling of node popularity we establish consistency of likelihood modularity for community detection as well as estimation of node popularities and model parameters and demonstrate the advantages of the new modularity over the degree corrected block model modularity in simulations by analysing the political blogs network the british members of parliament network and the digital bibliography and library project bibliographical network we illustrate that improved empirical insights can be gained through this methodology

on one parameter dependent generalizations of boltzmann gibbs statistical mechanics based on a previously postulated entropy that now becomes a particular case we show that there exists an infinite set of entropies with similar properties that reduce in a common limit to the boltzmann shannon form the probabilities for the microcanonical ensemble and for the canonical ensemble are obtained the method used to construct the set is quite simple and quite general and can be applied to generalizations of physical quantities and to other generalized entropies the existence of an infinite set of entropies with in principle similar properties could be a serious drawback for the actual utility of any of them and points to their utter uselessness unless some reason can be given for a special choice of one of them

spectral clustering and the high dimensional stochastic blockmodel networks or graphs can easily represent a diverse set of data sources that are characterized by interacting units or actors social ne tworks representing people who communicate with each other are one example communities or clusters of highly connected actors form an essential feature in the structure of several empirical networks spectral clustering is a popular and computationally feasi ble method to discover these communities the stochastic block model holland et al 1983 is a social network model with well defined communities each node is a member of one community for a network generated from the stochastic block model we bound the number of nodes misclus tered by spectral clustering the asymptotic results in th is paper are the first clustering results that allow the number of clusters in the model to grow with the number of nodes hence the name high dimensional in order to study spectral clustering under the stochastic block model we first show that under the more general latent space model the eigenvectors of the normalized graph laplacian asymptotically converge to the eigenvectors of a population normal ized graph laplacian aside from the implication for spectral clustering this provides insight into a graph visualization technique our method of studying the eigenvectors of random matrices is original ams 2000 subject classifications primary 62h30 62h25 secondary 60b20

hole probability for nodal sets of the cut off gaussian free field abstract let g be a closed connected surface equipped with a riemannian metric let n n n and n n n be the increasing sequence of eigenvalues and the sequence of corresponding l 2 normalized eigenfunctions of the laplacian on for each l 0 we consider l 0 n l n n n where the n are i i d centered gaussians with variance 1 as l l converges a s to the gaussian free field on in the sense of distributions we first compute the asymptotic behavior of the covariance function for this family of fields as l we then use this result to obtain the asymptotics of the probability that l is positive on a given open proper subset with smooth boundary in doing so we also prove the concentration of the supremum of l around 1 2 ln l

non linear kinetics underlying generalized statistics abstract the purpose of the present effort is threefold firstly it is shown that there exists a principle that we call kinetical interaction principle kip underlying the non linear kinetics in particle systems independently on the picture kramers boltzmann used to describe their time evolution secondly the kip imposes the form of the generalized entropy associated to the system and permits to obtain the particle statistical distribution both as stationary solution of the non linear evolution equation and as the state which maximizes the generalized entropy thirdly the kip allows on one hand to treat all the classical or quantum statistical distributions already known in the literature in a unifying scheme and on the other hand suggests how we can introduce naturally new distributions finally as a working example of the approach to the non linear kinetics here presented a new non extensive statistics is constructed and studied starting from a one parameter deformation of the exponential function holding the relation f x f x 1

a goodness of fit test for stochastic block models the stochastic block model is a popular tool for studying community structures in network data we develop a goodness of fit test for the stochastic block model the test statistic is based on the largest singular value of a residual matrix obtained by subtracting the estimated block mean effect from the adjacency matrix asymptotic null distribution is obtained using recent advances in random matrix theory the test is proved to have full power against alternative models with finer structures these results naturally lead to a consistent sequential testing estimate of the number of communities

tsallis versus renyi entropic form for systems with q exponential behaviour the case of dissipative maps abstract maximum entropy principle does not seem to distinguish between the use of tsallis and renyi entropies as either of them may be used to derive similar kind of q exponential distributions in this paper we address the question whether the renyi entropy is equally suitable to describe those systems with q exponential behaviour where the use of the tsallis entropy is relevant we discuss a specific class of dynamical systems namely one dimensional dissipative maps at chaos threshold and make our study from two aspects i power law sensitivity to the initial conditions and the rate of entropy increase ii generalized bit cumulants we present evidence that the tsallis entropy is more appropriate entropic form for such studies as compared to renyi form

using maximum entry wise deviation to test the goodness of fit for stochastic block models the stochastic block model is widely used for detecting community structures in network data how to test the goodness of fit of the model is one of the fundamental problems and has gained growing