hypothesis testing for automated community detection in networks summary community detection in networks is a key exploratory tool with applications in a diverse set of areas ranging from finding communities in social and biological networks to identifying link farms in the world wide web the problem of finding communities or clusters in a network has received much attention from statistics physics and computer science however most clustering algorithms assume knowledge of the number of clusters k we propose to determine k automatically in a graph generated from a stochastic block model by using a hypothesis test of independent interest our main contribution is twofold first we theoretically establish the limiting distribution of the principal eigenvalue of the suitably centred and scaled adjacency matrix and use that distribution for our test of the hypothesis that a random graph is of erd s renyi noise type secondly we use this test to design a recursive bipartitioning algorithm which naturally uncovers nested community structure using simulations and quantifiable classification tasks on real world networks with ground truth we show that our algorithm outperforms state of the art methods

distance dissimilarity index and network community structure we address the question of finding the community structure of a complex network in an earlier effort h zhou phys rev e 67 041908 2003 the concept of network random walking is introduced and a distance measure defined here we calculate based on this distance measure the dissimilarity index between nearest neighboring vertices of a network and design an algorithm to partition these vertices into communities that are hierarchically organized each community is characterized by an upper and a lower dissimilarity threshold the algorithm is applied to several artificial and real world networks and excellent results are obtained in the case of artificially generated random modular networks this method outperforms the algorithm based on the concept of edge betweenness centrality for yeast s protein protein interaction network we are able to identify many clusters that have well defined biological functions

link prediction based on local information considering preferential attachment abstract link prediction in complex networks has attracted much attention in many fields in this paper a common neighbors plus preferential attachment index is presented to estimate the likelihood of the existence of a link between two nodes based on local information of the nearest neighbors numerical experiments on six real networks demonstrated the high effectiveness and efficiency of the new index compared with five well known and widely accepted indices the common neighbors resource allocation index preferential attachment index local path index and katz index the new index provides competitively accurate prediction with local path index and katz index while has less computational complexity and is more accurate than the other two indices

a numerical study of the limited memory bfgs method and the truncated newton method for large scale optimization this paper examines the numerical performances of two methods for large scale optimization a limited memory quasi newton method l bfgs and a discrete truncated newton method tn various ways of classifying test problems are discussed in order to better understand the types of problems that each algorithm solves well the l bfgs and tn methods are also compared with the polak ribiere conjugate gradient method

a vector partitioning approach to detecting community structure in complex networks in recent years the problem of community structure detection has attracted more and more attention and many approaches have been proposed recently newman pointed out that this issue can be transformed into the problem of constrained maximization of the assignment matrix over possible divisions of a network he presents further that this maximization process can be written in terms of the eigenspectrum of the modularity matrix on the basis of this work and the vector partition approach in computer science we propose a kind of multiway division approach for detecting community structure of complex networks experimental results indicate that the algorithm works well and is effective at finding both good communities and the appropriate number of communities

network cross validation for determining the number of communities in network data abstractthe stochastic block model sbm and its variants have been a popular tool for analyzing large network data with community structures in this article we develop an efficient network cross validation ncv approach to determine the number of communities as well as to choose between the regular stochastic block model and the degree corrected block model dcbm the proposed ncv method is based on a block wise node pair splitting technique combined with an integrated step of community recovery using sub blocks of the adjacency matrix we prove that the probability of under selection vanishes as the number of nodes increases under mild conditions satisfied by a wide range of popular community recovery algorithms the solid performance of our method is also demonstrated in extensive simulations and two data examples supplementary materials for this article are available online

community extraction for social networks analysis of networks and in particular discovering communities within networks has been a focus of recent work in several fields and has diverse applications most community detection methods focus on partitioning the entire network into communities with the expectation of many ties within communities and few ties between however many networks contain nodes that do not fit in with any of the communities and forcing every node into a community can distort results here we propose a new framework that extracts one community at a time allowing for arbitrary structure in the remainder of the network which can include weakly connected nodes the main idea is that the strength of a community should depend on ties between its members and ties to the outside world but not on ties between nonmembers the proposed extraction criterion has a natural probabilistic interpretation in a wide class of models and performs well on simulated and real networks for the case of the block model we establish asymptotic consistency of estimated node labels and propose a hypothesis test for determining the number of communities

stochastic blockmodels and community structure in networks stochastic blockmodels have been proposed as a tool for detecting community structure in networks as well as for generating synthetic networks for use as benchmarks most blockmodels however ignore variation in vertex degree making them unsuitable for applications to real world networks which typically display broad degree distributions that can significantly affect the results here we demonstrate how the generalization of blockmodels to incorporate this missing element leads to an improved objective function for community detection in complex networks we also propose a heuristic algorithm for community detection using this objective function or its non degree corrected counterpart and show that the degree corrected version dramatically outperforms the uncorrected one in both real world and synthetic networks

finding community structure in networks using the eigenvectors of matrices we consider the problem of detecting communities or modules in networks groups of vertices with a higher than average density of edges connecting them previous work indicates that a robust approach to this problem is the maximization of the benefit function known as modularity over possible divisions of a network here we show that this maximization process can be written in terms of the eigenspectrum of a matrix we call the modularity matrix which plays a role in community detection similar to that played by the graph laplacian in graph partitioning calculations this result leads us to a number of possible algorithms for detecting community structure as well as several other results including a spectral measure of bipartite structure in networks and a centrality measure that identifies vertices that occupy central positions within the communities to which they belong the algorithms and measures proposed are illustrated with applications to a variety of real world complex networks

analysis of weighted networks the connections in many networks are not merely binary entities either present or not but have associated weights that record their strengths relative to one another recent studies of networks have by and large steered clear of such weighted networks which are often perceived as being harder to analyze than their unweighted counterparts here we point out that weighted networks can in many cases be analyzed using a simple mapping from a weighted network to an unweighted multigraph allowing us to apply standard techniques for unweighted graphs to weighted ones as well we give a number of examples of the method including an algorithm for detecting community structure in weighted networks and a simple proof of the maximum flow minimum cut theorem

latent space approaches to social network analysis network models are widely used to represent relational information among interacting units in studies of social networks recent emphasis has been placed on random graph models where the nodes usually represent individual social actors and the edges represent the presence of a specified relation between actors we develop a class of models where the probability of a relation between actors depends on the positions of individuals in an unobserved social space we make inference for the social space within maximum likelihood and bayesian frameworks and propose markov chain monte carlo procedures for making inference on latent positions and the effects of observed covariates we present analyses of three standard datasets from the social networks literature and compare the method to an alternative stochastic blockmodeling approach in addition to improving on model fit for these datasets our method provides a visual and interpretable model based spatial representation of social relationships and improv

statistical mechanics of community detection starting from a general ansatz we show how community detection can be interpreted as finding the ground state of an infinite range spin glass our approach applies to weighted and directed networks alike it contains the ad hoc introduced quality function from j reichardt and s bornholdt phys rev lett 93 218701 2004 and the modularity q as defined by newman and girvan phys rev e 69 026113 2004 as special cases the community structure of the network is interpreted as the spin configuration that minimizes the energy of the spin glass with the spin states being the community indices we elucidate the properties of the ground state configuration to give a concise definition of communities as cohesive subgroups in networks that is adaptive to the specific class of network under study further we show how hierarchies and overlap in the community structure can be detected computationally efficient local update rules for optimization procedures to find the ground state are given we show how the ansatz may be used to discover the community around a given node without detecting all communities in the full network and we give benchmarks for the performance of this extension finally we give expectation values for the modularity of random graphs which can be used in the assessment of statistical significance of community structure

identifying communities within energy landscapes potential energy landscapes can be represented as a network of minima linked by transition states the community structure of such networks has been obtained for a series of small lennard jones lj clusters this community structure is compared to the concept of funnels in the potential energy landscape two existing algorithms have been used to find community structure one involving removing edges with high betweenness the other involving optimization of the modularity the definition of the modularity has been refined making it more appropriate for networks such as these where multiple edges and self connections are not included the optimization algorithm has also been improved using monte carlo methods with simulated annealing and basin hopping both often used successfully in other optimization problems in addition to the small clusters two examples with known heterogeneous landscapes the 13 atom cluster lj13 with one labeled atom and the 38 atom cluster lj38 were studied with this approach the network methods found communities that are comparable to those expected from landscape analyses this is particularly interesting since the network model does not take any barrier heights or energies of minima into account for comparison the network associated with a two dimensional hexagonal lattice is also studied and is found to have high modularity thus raising some questions about the interpretation of the community structure associated with such partitions

normalized cuts and image segmentation we propose a novel approach for solving the perceptual grouping problem in vision rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image we treat image segmentation as a graph partitioning problem and propose a novel global criterion the normalized cut for segmenting the graph the normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups we show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion we applied this approach to segmenting static images as well as motion sequences and found the results to be very encouraging

community detection in complex networks using extremal optimization we propose a method to find the community structure in complex networks based on an extremal optimization of the value of modularity the method outperforms the optimal modularity found by the existing algorithms in the literature giving a better understanding of the community structure we present the results of the algorithm for computer simulated and real networks and compare them with other approaches the efficiency and accuracy of the method make it feasible to be used for the accurate identification of community structure in large complex networks

mixing patterns in networks we study assortative mixing in networks the tendency for vertices in networks to be connected to other vertices that are like or unlike them in some way we consider mixing according to discrete characteristics such as language or race in social networks and scalar characteristics such as age as a special example of the latter we consider mixing according to vertex degree i e according to the number of connections vertices have to other vertices do gregarious people tend to associate with other gregarious people we propose a number of measures of assortative mixing appropriate to the various mixing types and apply them to a variety of real world networks showing that assortative mixing is a pervasive phenomenon found in many networks we also propose several models of assortatively mixed networks both analytic ones based on generating function methods and numerical ones based on monte carlo graph generation techniques we use these models to probe the properties of networks as their level of assortativity is varied in the particular case of mixing by degree we find strong variation with assortativity in the connectivity of the network and in the resilience of the network to the removal of vertices

likelihood based model selection for stochastic block models the stochastic block model sbm provides a popular framework for modeling community structures in networks however more attention has been devoted to problems concerning estimating the latent node labels and the model parameters than the issue of choosing the number of blocks we consider an approach based on the log likelihood ratio statistic and analyze its asymptotic properties under model misspecification we show the limiting distribution of the statistic in the case of underfitting is normal and obtain its convergence rate in the case of overfitting these conclusions remain valid when the average degree grows at a polylog rate the results enable us to derive the correct order of the penalty term for model complexity and arrive at a likelihood based model selection criterion that is asymptotically consistent our analysis can also be extended to a degree corrected block model dcsbm in practice the likelihood function can be estimated using more computationally efficient variational methods or consistent label estimation algorithms allowing the criterion to be applied to large networks

eigenvectors of network complement reveal community structure more accurately abstract we propose a general spectral method to find communities of a network based on network complement and anti community concepts analytical and numerical results show that the eigenspace of matrices corresponding to a network complement reveals the community structure of a network more accurately than the eigenspace of matrices corresponding to the network itself it is shown that the laplacian eigenspace is the best candidate for spectral community detection especially in networks with a heterogeneous community structure the method is applied to some computer generated and real world networks with known community structures

spectral measures of bipartivity in complex networks we introduce a quantitative measure of network bipartivity as a proportion of even to total number of closed walks in the network spectral graph theory is used to quantify how close to bipartite a network is and the extent to which individual nodes and edges contribute to the global network bipartivity it is shown that the bipartivity characterizes the network structure and can be related to the efficiency of semantic or communication networks trophic interactions in food webs construction principles in metabolic networks or communities in social networks

a modularity based spectral method for simultaneous community and anti community detection abstract in a graph or complex network communities and anti communities are node sets whose modularity attains extremely large values positive and negative respectively we consider the simultaneous detection of communities and anti communities by looking at spectral methods based on various matrix based definitions of the modularity of a vertex set invariant subspaces associated to extreme eigenvalues of these matrices provide indications on the presence of both kinds of modular structure in the network the localization of the relevant invariant subspaces can be estimated by looking at particular matrix angles based on frobenius inner products

on network bipartivity systems with two types of agents with a preference for heterophilous interaction produces networks that are more or less close to bipartite we propose two measures quantifying the notion of bipartivity the two measures one well known and natural but computationally intractable one computationally less complex but also less intuitive are examined on model networks that continuously interpolates between bipartite graphs and graphs with many odd circuits we find that the bipartivity measures increase as we tune the control parameters of the test networks to intuitively increase the bipartivity and thus conclude that the measures are quite relevant we also measure and discuss the values of our bipartivity measures for empirical social networks constructed from professional collaborations internet communities and field surveys here we find as expected that networks arising from romantic online interaction have high and professional collaboration networks have low bipartivity values in some other cases probably due to low average degree of the network the bipartivity measures cannot distinguish between romantic and friendship oriented interaction

bootstrapping on undirected binary networks via statistical mechanics we propose a new method inspired from statistical mechanics for extracting geometric information from undirected binary networks and generating random networks that conform to this geometry in this method an undirected binary network is perceived as a thermodynamic system with a collection of permuted adjacency matrices as its states the task of extracting information from the network is then reformulated as a discrete combinatorial optimization problem of searching for its ground state to solve this problem we apply multiple ensembles of temperature regulated markov chains to establish an ultrametric geometry on the network this geometry is equipped with a tree hierarchy that captures the multiscale community structure of the network we translate this geometry into a parisi adjacency matrix which has a relative low energy level and is in the vicinity of the ground state the parisi adjacency matrix is then further optimized by making block permutations subject to the ultrametric geometry the optimal matrix corresponds to the macrostate of the original network an ensemble of random networks is then generated such that each of these networks conforms to this macrostate the corresponding algorithm also provides an estimate of the size of this ensemble by repeating this procedure at different scales of the ultrametric geometry of the network it is possible to compute its evolution entropy i e to estimate the evolution of its complexity as we move from a coarse to a fine description of its geometric structure we demonstrate the performance of this method on simulated as well as real data networks

estimation and prediction for stochastic blockstructures a statistical approach to a posteriori blockmodeling for digraphs and valued digraphs is proposed the probability model assumes that the vertices of the digraph are partitioned into several unobserved latent classes and that the probability distribution of the relation between two vertices depends only on the classes to which they belong a bayesian estimator based on gibbs sampling is proposed the basic model is not identified because class labels are arbitrary the resulting identifiability problems are solved by restricting inference to the posterior distributions of invariant functions of the parameters and the vertex class membership in addition models are considered where class labels are identified by prior distributions for the class membership of some of the vertices the model is illustrated by an example from the social networks literature kapferer s tailor shop

finding local community structure in networks although the inference of global community structure in networks has recently become a topic of great interest in the physics community all such algorithms require that the graph be completely known here we define both a measure of local community structure and an algorithm that infers the hierarchy of communities that enclose a given vertex by exploring the graph one vertex at a time this algorithm runs in time o k2d for general graphs when d is the mean degree and k is the number of vertices to be explored for graphs where exploring a new vertex is time consuming the running time is linear o k we show that on computer generated graphs the average behavior of this technique approximates that of algorithms that require global knowledge as an application we use this algorithm to extract meaningful local clustering information in the large recommender network of an online retailer

sampling promotes community structure in social and information networks abstract any network studied in the literature is inevitably just a sampled representative of its real world analogue additionally network sampling is lately often applied to large networks to allow for their faster and more efficient analysis nevertheless the changes in network structure introduced by sampling are still far from understood in this paper we study the presence of characteristic groups of nodes in sampled social and information networks we consider different network sampling techniques including random node and link selection network exploration and expansion we first observe that the structure of social networks reveals densely linked groups like communities while the structure of information networks is better described by modules of structurally equivalent nodes however despite these notable differences the structure of sampled networks exhibits stronger characterization by community like groups than the original networks irrespective of their type and consistently across various sampling techniques hence rich community structure commonly observed in social and information networks is to some extent merely an artifact of sampling

modularity and community detection in bipartite networks the modularity of a network quantifies the extent relative to a null model network to which vertices cluster into community groups we define a null model appropriate for bipartite networks and use it to define a bipartite modularity the bipartite modularity is presented in terms of a modularity matrix b some key properties of the eigenspectrum of b are identified and used to describe an algorithm for identifying modules in bipartite networks the algorithm is based on the idea that the modules in the two parts of the network are dependent with each part mutually being used to induce the vertices for the other part into the modules we apply the algorithm to real world network data showing that the algorithm successfully identifies the modular structure of bipartite networks

iterative and active graph clustering using trace norm minimization without cluster size constraints this paper investigates graph clustering under the planted partition model in the presence of small clusters traditional results dictate that for an algorithm to provably correctly recover the underlying clusters all clusters must be sufficiently large in particular the cluster sizes need to be n where n is the number of nodes of the graph we show that this is not really a restriction by a refined analysis of a convex optimization based recovery approach we prove that small clusters under certain mild assumptions do not hinder recovery of large ones based on this result we further devise an iterative algorithm to provably recover almost all clusters via a peeling strategy we recover large clusters first leading to a reduced problem and repeat this procedure these results are extended to the partial observation setting in which only a chosen part of the graph is observed the peeling strategy gives rise to an active learning algorithm in which edges adjacent to smaller clusters are queried more often after large clusters are learned and removed we expect that the idea of iterative peeling that is sequentially identifying a subset of the clusters and reducing the problem to a smaller one is useful more broadly beyond the specific implementations based on convex optimization used in this paper

backlund transformations for the ernst equation presented is a systematic approach to the transformation theories for the ernst equation from the viewpoint of the backlund transformation it is explicitly shown that the method of clairin gives a simple derivation of various transformations such as transformations found by ehlers neugebauer and harrison

consistency of community detection in networks under degree corrected stochastic block models community detection is a fundamental problem in network analysis with applications in many diverse areas the stochastic block model is a common tool for model based community detection and asymptotic tools for checking consistency of community detection under the block model have been recently developed however the block model is limited by its assumption that all nodes within a community are stochastically equivalent and provides a poor fit to networks with hubs or highly varying node degrees within communities which are common in practice the degree corrected stochastic block model was proposed to address this shortcoming and allows variation in node degrees within a community while preserving the overall block community structure in this paper we establish general theory for checking consistency of community detection under the degree corrected stochastic block model and compare several community detection criteria under both the standard and the degree corrected models we show which criteria are consistent under which models and constraints as well as compare their relative performance in practice we find that methods based on the degree corrected block model which includes the standard block model as a special case are consistent under a wider class of models and that modularity type methods require parameter constraints for consistency whereas likelihood based methods do not on the other hand in practice the degree correction involves estimating many more parameters and empirically we find it is only worth doing if the node degrees within communities are indeed highly variable we illustrate the methods on simulated networks and on a network of political blogs

the degree related clustering coefficient and its application to link prediction abstract link prediction plays a significant role in explaining the evolution of networks however it is still a challenging problem that has been addressed only with topological information in recent years based on the belief that network nodes with a great number of common neighbors are more likely to be connected many similarity indices have achieved considerable accuracy and efficiency motivated by the natural assumption that the effect of missing links on the estimation of a node s clustering ability could be related to node degree in this paper we propose a degree related clustering coefficient index to quantify the clustering ability of nodes unlike the classical clustering coefficient our new coefficient is highly robust when the observed bias of links is considered furthermore we propose a degree related clustering ability path dcp index which applies the proposed coefficient to the link prediction problem experiments on 12 real world networks show that our proposed method is highly accurate and robust compared with four common neighbor based similarity indices common neighbors cn adamic adar aa resource allocation ra and preferential attachment pa and the recently introduced clustering ability ca index

quadratic program based modularity maximization for fuzzy community detection in social networks one of the most important elements of social network analysis is community detection i e finding groups of similar people based on their traits in this paper we present the fuzzy modularity maximization fmm approach for community detection which finds overlapping that is fuzzy communities where appropriate by maximizing a generalized form of newman s modularity the first proposed fmm solution uses a tree based structure to find a globally optimal solution while the second proposed solution uses alternating optimization to efficiently search for a locally optimal solution both of these approaches are based on a proposed algorithm called one step modularity maximization osmm which computes the optimal cluster memberships for one person in the social network we prove that osmm can be formulated as a simplified quadratic knapsack optimization problem which is o n time complexity we then propose a tree based algorithm called fmm find best leaf node fmm fbln which represents sequences of osmm steps in a tree based structure it is proved that fmm fbln finds globally optimal solutions for fmm however the time complexity of fmm fbln is o n d d ge 2 thus it is impractical for most real world networks to combat this inefficiency we propose five heuristic based alternating optimization schemes i e fmm h1 h5 which are all shown to be o n 2 time complexity we compare the results of the fmm h solutions with those of state of the art community detection algorithms multicut spectral fcm msfcm and gals and with those of two fuzzy community detection algorithms called ga and vertex similarity based gradient descent method vsgd on ten real world datasets we conclude that one of the five heuristic algorithms fmm h2 is very competitive with gals and much more effective than msfcm ga and vsgd furthermore all of the fmm h schemes are at least two orders of magnitude faster than gals in run time finally fmm h unlike gals which only produces crisp partitions and msfcm which always finds fuzzy partitions is the only fuzzy community detection algorithm to date that can find the max modularity partition fuzzy or crisp

on the limited memory bfgs method for large scale optimization we study the numerical performance of a limited memory quasi newton method for large scale optimization which we call the l bfgs method we compare its performance with that of the method developed by buckley and lenir 1985 which combines cycles of bfgs steps and conjugate direction steps our numerical tests indicate that the l bfgs method is faster than the method of buckley and lenir and is better able to use additional storage to accelerate convergence we show that the l bfgs method can be greatly accelerated by means of a simple scaling we then compare the l bfgs method with the partitioned quasi newton method of griewank and toint 1982a the results show that for some problems the partitioned quasi newton method is clearly superior to the l bfgs method however we find that for other problems the l bfgs method is very competitive due to its low iteration cost we also study the convergence properties of the l bfgs method and prove global convergence on uniformly convex problems

normalized modularity optimization method for community identification with degree adjustment as a fundamental problem in network study community identification has attracted much attention from different fields representing a seminal work in this area the modularity optimization method has been widely applied and studied however this method has issues in resolution limit and extreme degeneracy and may not perform well for networks with unbalanced structures although several methods have been proposed to overcome these limitations they are all based on the original idea of defining modularity through comparing the total number of edges within the putative communities in the observed network with that in an equivalent randomly generated network in this paper we show that this modularity definition is not suitable to analyze some networks such as those with unbalanced structures instead we propose to define modularity through the average degree within the communities and formulate modularity as comparing the sum of average degree within communities of the observed network to that of an equivalent randomly generated network in addition we also propose a degree adjusted approach for further improvement when there are unbalanced structures we analyze the theoretical properties of our degree adjusted method numerical experiments for both artificial networks and real networks demonstrate that average degree plays an important role in network community identification and our proposed methods have better performance than existing ones

network ensemble comparisons with stochastic rewiring and von neumann entropy assessing whether a given network is typical or atypical for a random network ensemble i e network ensemble comparison has widespread applications ranging from null model selection and hypothesis testing to clustering and classifying networks we develop a framework for network ensemble comparison by subjecting the network to stochastic rewiring we study two rewiring processes uniform and degree preserved rewiring which yield random network ensembles that converge to the erdos renyi and configuration model ensembles respectively we study convergence through von neumann entropy vne a network summary statistic measuring information content based on the spectra of a laplacian matrix and develop a perturbation analysis for the expected effect of rewiring on vne our analysis yields an estimate for how many rewires are required for a given network to resemble a typical network from an ensemble offering a computationally efficient quantity for network ensemble comparison that does not require

unification of ernst equation backlund transformations using a modified wahlquist estabrook technique the three known backlund transformations for the ernst equation are derived using a modification of the wahlquist estabrook prolongation procedure the modification requires that the equation to be studied be cast into a set of differential forms and their exterior derivatives such that all coefficients are constant a cc ideal analysis of the resulting equations produces 16 solutions composed of the three basic transformations combined with identity and other essentially trivial transformations the group structure of the transformations is discussed a backlund transformation already known for the ernst maxwell equations can be found by the same method promising generalizations are mentioned

fast graph clustering with a new description model for community detection efficiently describing and discovering communities in a network is an important research concept for graph clustering in the paper we present a community description model that evaluates the local importance of a node in a community and its importance concentration in all communities to reflect its representability to the community based on the description model we propose a new evaluation criterion and an iterative search algorithm for community detection iscd the new algorithm can quickly discover communities in a large scale network due to the average linear time complexity with the number of edges furthermore we provide an initial method of input parameters including the number of communities and the initial partition before algorithm implementation which can enhance the local search quality of the iterative algorithm the proposed algorithm with the initial method is called iscd finally we compare the effectiveness and efficiency of the iscd algorithm with six representative algorithms on several real network data sets the experimental results illustrate that the proposed algorithm is suitable to address large scale networks

optimization via low rank approximation for community detection in networks author s le can m levina elizaveta vershynin roman abstract community detection is one of the fundamental problems of network analysis for which a number of methods have been proposed most model based or criteria based methods have to solve an optimization problem over a discrete set of labels to find communities which is computationally infeasible some fast spectral algorithms have been proposed for specific methods or models but only on a case by case basis here we propose a general approach for maximizing a function of a network adjacency matrix over discrete labels by projecting the set of labels onto a subspace approximating the leading eigenvectors of the expected adjacency matrix this projection onto a low dimensional space makes the feasible set of labels much smaller and the optimization problem much easier we prove a general result about this method and show how to apply it to several previously proposed community detection criteria establishing its consistency for label estimation in each case and demonstrating the fundamental connection between spectral properties of the network and various model based approaches to community detection simulations and applications to real world data are included to demonstrate our method performs well for multiple problems over a wide range of parameters

weight conserving characterization of complex functional brain networks complex functional brain networks are large networks of brain regions and functional brain connections statistical characterizations of these networks aim to quantify global and local properties of brain activity with a small number of network measures important functional network measures include measures of modularity measures of the goodness with which a network is optimally partitioned into functional subgroups and measures of centrality measures of the functional influence of individual brain regions characterizations of functional networks are increasing in popularity but are associated with several important methodological problems these problems include the inability to characterize densely connected and weighted functional networks the neglect of degenerate topologically distinct high modularity partitions of these networks and the absence of a network null model for testing hypotheses of association between observed nontrivial network properties and simple weighted connectivity properties in this study we describe a set of methods to overcome these problems specifically we generalize measures of modularity and centrality to fully connected and weighted complex networks describe the detection of degenerate high modularity partitions of these networks and introduce a weighted connectivity null model of these networks we illustrate our methods by demonstrating degenerate high modularity partitions and strong correlations between two complementary measures of centrality in resting state functional magnetic resonance imaging mri networks from the 1000 functional connectomes project an open access repository of resting state functional mri datasets our methods may allow more sound and reliable characterizations and comparisons of functional brain networks across conditions and subjects

personalized pagerank clustering a graph clustering algorithm based on random walks abstract graph clustering has been an essential part in many methods and thus its accuracy has a significant effect on many applications in addition exponential growth of real world graphs such as social networks biological networks and electrical circuits demands clustering algorithms with nearly linear time and space complexity in this paper we propose personalized pagerank clustering ppc that employs the inherent cluster exploratory property of random walks to reveal the clusters of a given graph we combine random walks and modularity to precisely and efficiently reveal the clusters of a graph ppc is a top down algorithm so it can reveal inherent clusters of a graph more accurately than other nearly linear approaches that are mainly bottom up it also gives a hierarchy of clusters that is useful in many applications ppc has a linear time and space complexity and has been superior to most of the available clustering algorithms on many datasets furthermore its top down approach makes it a flexible solution for clustering problems with different requirements

finding community structure in very large networks the discovery and analysis of community structure in networks is a topic of considerable recent interest within the physics community but most methods proposed so far are unsuitable for very large networks because of their computational cost here we present a hierarchical agglomeration algorithm for detecting community structure which is faster than many competing algorithms its running time on a network with n vertices and m edges is o md phantom rule 0 2em 0ex mathrm log phantom rule 0 2em 0ex n where d is the depth of the dendrogram describing the community structure many real world networks are sparse and hierarchical with m ensuremath sim n and d ensuremath sim mathrm log phantom rule 0 2em 0ex n in which case our algorithm runs in essentially linear time o n phantom rule 0 2em 0ex mathrm log 2 phantom rule 0 2em 0ex n as an example of the application of this algorithm we use it to analyze a network of items for sale on the web site of a large on line retailer items in the network being linked if they are frequently purchased by the same buyer the network has more than 400 000 vertices and 2 ifmmode times else texttimes fi 10 6 edges we show that our algorithm can extract meaningful communities from this network revealing large scale patterns present in the purchasing habits of customers

magnetic eigenmaps for community detection in directed networks communities in directed networks have often been characterized as regions with a high density of links or as sets of nodes with certain patterns of connection our approach for community detection combines the optimization of a quality function and a spectral clustering of a deformation of the combinatorial laplacian the so called magnetic laplacian the eigenfunctions of the magnetic laplacian which we call magnetic eigenmaps incorporate structural information hence using the magnetic eigenmaps dense communities including directed cycles can be revealed as well as role communities in networks with a running flow usually discovered thanks to mixture models furthermore in the spirit of the markov stability method an approach for studying communities at different energy levels in the network is put forward based on a quantum mechanical system at finite temperature

modularity from fluctuations in random graphs and complex networks the mechanisms by which modularity emerges in complex networks are not well understood but recent reports have suggested that modularity may arise from evolutionary selection we show that finding the modularity of a network is analogous to finding the ground state energy of a spin system moreover we demonstrate that due to fluctuations stochastic network models give rise to modular networks specifically we show both numerically and analytically that random graphs and scale free networks have modularity we argue that this fact must be taken into consideration to define statistically significant modularity in complex networks

robustness of community structure in networks the discovery of community structure is a common challenge in the analysis of network data many methods have been proposed for finding community structure but few have been proposed for determining whether the structure found is statistically significant or whether conversely it could have arisen purely as a result of chance in this paper we show that the significance of community structure can be effectively quantified by measuring its robustness to small perturbations in network structure we propose a suitable method for perturbing networks and a measure of the resulting change in community structure and use them to assess the significance of community structure in a variety of networks both real and computer generated

partition of networks into basins of attraction we study partition of networks into basins of attraction based on a steepest ascent search for the node of highest degree each node is associated with or attracted to its neighbor of maximal degree as long as the degree is increasing a node that has no neighbors of higher degree is a peak attracting all the nodes in its basin maximally random scale free networks exhibit different behavior based on their degree distribution exponent gamma for small gamma broad distribution networks are dominated by a giant basin whereas for large gamma narrow distribution there are numerous basins with peaks attracting mainly their nearest neighbors we derive expressions for the first two moments of the number of basins we also obtain the complete distribution of basin sizes for a class of hierarchical deterministic scale free networks that resemble random nets finally we generalize the problem to regular networks and lattices where all degrees are equal and thus the attractiveness of a node must be determined by an assigned weight rather than the degree we derive the complete distribution of basins of attraction resulting from randomly assigned weights in one dimensional chains

global optimization of statistical functions with simulated annealing abstract many statistical methods rely on numerical optimization to estimate a model s parameters unfortunately conventional algorithms sometimes fail even when they do converge there is no assurance that they have found the global rather than a local optimum we test a new optimization algorithm simulated annealing on four econometric problems and compare it to three common conventional algorithms not only can simulated annealing find the global optimum it is also less likely to fail on difficult functions because it is a very robust algorithm the promise of simulated annealing is demonstrated on the four econometric problems

statistical ensembles for economic networks economic networks share with other social networks the fundamental property of sparsity it is well known that the maximum entropy techniques usually employed to estimate or simulate weighted networks produce unrealistic dense topologies at the same time strengths should not be neglected since they are related to core economic variables like supply and demand to overcome this limitation the exponential bosonic model has been previously extended in order to obtain ensembles where the average degree and strength sequences are simultaneously fixed conditional geometric model in this paper a new exponential model which is the network equivalent of boltzmann ideal systems is introduced and then extended to the case of joint degree strength constraints conditional poisson model finally the fitness of these alternative models is tested against a number of networks while the conditional geometric model generally provides a better goodness of fit in terms of log likelihoods the conditional poisson model could nevertheless be preferred whenever it provides a higher similarity with original data if we are interested instead only in topological properties the simple bernoulli model appears to be preferable to the correlated topologies of the two more complex models

detecting communities in large networks we develop an algorithm to detect community structure in complex networks the algorithm is based on spectral methods and takes into account weights and link orientation since the method detects efficiently clustered nodes in large networks even when these are not sharply partitioned it turns to be specially suitable for the analysis of social and information networks we test the algorithm on a large scale data set from a psychological experiment of word association in this case it proves to be successful both in clustering words and in uncovering mental association patterns

discriminative metric learning for multi view graph partitioning we propose discriminative metric learning for multi view graph partitioning we envision the multi view graph as an adaptive dynamic system the intra view connections and the inter view couplings are interplayed extensive experiments have been conducted to show the effectiveness in recent years multi view graph partitioning has attracted more and more attention but most efforts have been made to develop graph partitioning approaches directly in the original topological structure in many real world applications graph may contain noisy links and the distance metric may not be so discriminative for revealing cluster structure this paper addresses the problem of discriminative metric learning for multi view graph partitioning in particular we propose a novel method called multi view dml abbr of multi view discriminative metric learning to transform the metric space in the original graph into a more discriminative metric space in which better graph partitioning results will be obtained we envision the multi view graph as an adaptive dynamic system where both the intra view connections and the inter view couplings are interplayed to gradually update the relation metric among nodes on the one hand the inter view coupling will be influenced by the intra view connection between two nodes on the other hand the inter view coupling will also affect the intra view connection such interplay eventually makes the whole graph reach a steady state which has a stronger cluster structure than the original graph extensive experiments are conducted on both synthetic and real world graphs to confirm that the proposed method is able to learn more discriminative metric

on the concepts of complex networks to quantify the difficulty in finding the way out of labyrinths abstract labyrinths have been a tradition and part of the imagination of the human kind for centuries and were probably built either as a challenge to make it difficult for someone to find the way out or for aesthetic purposes they are conventionally classified according to the country they were built to the style roman classic and contemporary or to the construction site in this study we show that labyrinths can be modeled as complex networks whose metrics can be used to classify them in terms of their difficulty to find the way out this is performed by calculating the absorption time defined as the time it takes for a particle on an internal node to reach an output node through a random walk the absorption time correlates well with the shortest paths and length of the networks as expected and has a very high correlation pearson coefficient of 0 97 with the betweenness therefore allowing one to quantify the level of complexity of any labyrinth it is shown that the conventional classification is inappropriate to distinguish between labyrinths because some with very similar properties exist in different countries or were built in distinct time periods a refined analysis in 77 famous labyrinths indicated that the majority were built for aesthetic purposes with relatively small absorption times furthermore with the expectation maximization algorithm we could combine the complex network metrics to identify four clusters of labyrinths that differ in terms of density and shape

community structure discovery method based on the gaussian kernel similarity matrix abstract community structure discovery in complex networks is a popular issue and overlapping community structure discovery in academic research has become one of the hot spots based on the gaussian kernel similarity matrix and spectral bisection this paper proposes a new community structure discovery method first by adjusting the gaussian kernel parameter to change the scale of similarity we can find the corresponding non overlapping community structure when the value of the modularity is the largest relatively second the changes of the gaussian kernel parameter would lead to the unstable nodes jumping off so with a slight change in method of non overlapping community discovery we can find the overlapping community nodes finally synthetic data karate club and political books datasets are used to test the proposed method comparing with some other community discovery methods to demonstrate the feasibility and effectiveness of this method

detecting community structure in complex networks based on a measure of information discrepancy abstract properties of complex networks such as small world property power law degree distribution network transitivity and network community structure which seem to be common to many real world networks have attracted great interest among researchers in this study global information of the networks is considered by defining the profile of any node based on the shortest paths between it and all the other nodes in the network then a useful iterative procedure for community detection based on a measure of information discrepancy and the popular modular function q is presented the new iterative method does not need any prior knowledge about the community structure and can detect an appropriate number of communities which can be hub communities or non hub communities the computational results of the method on real networks confirm its capability

community detection as an inference problem we express community detection as an inference problem of determining the most likely arrangement of communities we then apply belief propagation and mean field theory to this problem and show that this leads to fast accurate algorithms for community detection

identifying overlapping communities in networks using evolutionary method abstract community structure is a typical property of real world networks and has been recognized as a key to understand the dynamics of the networked systems in most of the networks overwhelming nodes apparently live in a community while there often exists a few nodes straddling several communities hence an ideal algorithm for community detection is that which can identify the overlapping communities in these networks we present an evolutionary method for detecting overlapping community structure in the network to represent an overlapping division of a network we develop an encoding scheme composed of two segments the first one represents a disjoint partition and the second one represents an extension of the partition that allows of multiple memberships we give two measures for the informativeness of a node and present a coevolutionary scheme between two segments over the population for solving the overlapping partition of the network experimental results show this method can give a better solution to a network it is also revealed that a best overlapping partition of the network might not be rooted from a best disjoint partition

goodness of fit of social network models we present a systematic examination of a real network data set using maximum likelihood estimation for exponential random graph models as well as new procedures to evaluate how well the models fit the observed networks these procedures compare structural statistics of the observed network with the corresponding statistics on networks simulated from the fitted model we apply this approach to the study of friendship relations among high school students from the national longitudinal study of adolescent health addhealth we focus primarily on one particular network of 205 nodes although we also demonstrate that this method may be applied to the largest network in the addhealth study with 2 209 nodes we argue that several well studied models in the networks literature do not fit these data well and demonstrate that the fit improves dramatically when the models include the recently developed geometrically weighted edgewise shared partner geometrically weighted dyadic shared partner and geometrically w

properties of euclidean and non euclidean distance matrices abstract a distance matrix d of order n is symmetric with elements 1 2 d ij 2 where dii 0 d is euclidean when the 1 2 n n 1 quantities dij can be generated as the distances between a set of n points x n p in a euclidean space of dimension p the dimensionality of d is defined as the least value of p rank x of any generating x in general p 1 and p 2 are also acceptable but may include imaginary coordinates even when d is euclidean basic properties of euclidean distance matrices are established in particular when rank d it is shown that depending on whether etd e is not or is zero the generating points lie in either p 1 dimensions in which case they lie on a hypersphere or in p 2 dimensions in which case they do not the notation e is used for a vector all of whose values are one when d is non euclidean its dimensionality p r s will comprise r real and s imaginary columns of x and r s are invariant for all generating x of minimal rank higher ranking representations can arise only from p 1 r 1 s or p 1 r s 1 or p 2 r 1 s 1 so that not only are r s invariant but they are both minimal for all admissible representations x

quantum behaved discrete multi objective particle swarm optimization for complex network clustering abstract complex network research has attracted lots of attention in both academic community and various application fields complex network clustering as one of the key issues in complex network explores the internal organization of the nodes in a complex network the discrete particle swarm optimization strategy has been successfully proposed for network clustering while the existing method works with weak robust in this paper we model the task of complex network clustering as a multi objective optimization problem and solve the problem with the quantum mechanism based particle swarm optimization algorithm which is a parallel algorithm to our knowledge this is the first attempt to apply the quantum mechanism based discrete particle swarm optimization algorithm into network clustering in addition the non dominant sorting selection operation is employed for individual replacement consequently a quantum behaved discrete multi objective particle swarm optimization algorithm is proposed for complex network clustering the experimental results demonstrate that the proposed algorithm performs effectively and achieves competitive performance with the state of the art approaches on the extension of girvan and newman benchmarks and real world networks especially on large scale networks

a goodness of fit test for stochastic block models the stochastic block model is a popular tool for studying community structures in network data we develop a goodness of fit test for the stochastic block model the test statistic is based on the largest singular value of a residual matrix obtained by subtracting the estimated block mean effect from the adjacency matrix asymptotic null distribution is obtained using recent advances in random matrix theory the test is proved to have full power against alternative models with finer structures these results naturally lead to a consistent sequential testing estimate of the number of communities

principal component analysis combined with truncated newton minimization for dimensionality reduction of chemical databases the similarity and diversity sampling problems are two challenging optimization tasks that arise in the analysis of chemical databases as a first step to their solution we propose an efficient projection refinement protocol based on the principal component analysis pca and the truncated newton minimization method implemented by our package tnpack pca tnpack we show that pca can provide the same initial guess as the singular value decomposition svd for the optimization task of solving the distance geometry optimization problem if each column of a database matrix has a mean of zero hence our pca tnpack approach is analogous to the svd tnpack projection refinement protocol that we developed recently for visualizing large chemical databases using pca tnpack and the merck mddr database mdl drug data report we further investigate the projection refinement procedure with regards to the preservation of the original clusters of chemical compounds the accuracy of similarity and diversity sampling of chemical compounds and the potential application in the study of structure activity relationships we also explore by simple experiments accuracy and efficiency aspects of the pca tnpack procedure compared to those of a global optimization algorithm simulated annealing as implemented by the program package simann in terms of producing the projection mapping of a database numerical results show that the 2d pca tnpack mapping can preserve the distance relationships of the original database and is thus valuable as a first step in similarity and diversity applications of course the generation of a global rather than local minimizer and its interpretation in terms of pharameceutical applications remains a challenge since all numerical tests are performed on the merck mddr database results are representative of realistic cases encountered in the field of drug design and may help analyze properties of medicinal compounds

random graphs with arbitrary degree distributions and their applications recent work on the structure of social networks and the internet has focused attention on graphs with distributions of vertex degree that are significantly different from the poisson degree distributions that have been widely studied in the past in this paper we develop in detail the theory of random graphs with arbitrary degree distributions in addition to simple undirected unipartite graphs we examine the properties of directed and bipartite graphs among other results we derive exact expressions for the position of the phase transition at which a giant component first forms the mean component size the size of the giant component if there is one the mean number of vertices a certain distance away from a randomly chosen vertex and the average vertex vertex distance within a graph we apply our theory to some real world graphs including the worldwide web and collaboration graphs of scientists and fortune 1000 company directors we demonstrate that in some cases random graphs with appropriate distributions of vertex degree predict with surprising accuracy the behavior of the real world while in others there is a measurable discrepancy between theory and reality perhaps indicating the presence of additional social structure in the network that is not captured by the random graph

on the equivalence of the label propagation method of community detection and a potts model approach abstract we show that the recently introduced label propagation method for detecting communities in complex networks is equivalent to finding the local minima of a simple potts model applying to empirical data the number of such local minima was found to be very high much larger than the number of nodes in the graph the aggregation method for combining information from more local minima shows a tendency to fragment the communities into very small pieces

method to find community structures based on information centrality community structures are an important feature of many social biological and technological networks here we study a variation on the method for detecting such communities proposed by girvan and newman and based on the idea of using centrality measures to define the community boundaries m girvan and m e j newman proc natl acad sci u s a 99 7821 2002 we develop an algorithm of hierarchical clustering that consists in finding and removing iteratively the edge with the highest information centrality we test the algorithm on computer generated and real world networks whose community structure is already known or has been studied by means of other methods we show that our algorithm although it runs to completion in a time o n4 is very effective especially when the communities are very mixed and hardly detectable by the other methods

parameterized centrality metric for network analysis a variety of metrics have been proposed to measure the relative importance of nodes in a network one of these alpha centrality p bonacich am j sociol 92 1170 1987 measures the number of attenuated paths that exist between nodes we introduce a normalized version of this metric and use it to study network structure for example to rank nodes and find community structure of the network specifically we extend the modularity maximization method for community detection to use this metric as the measure of node connectivity normalized alpha centrality is a powerful tool for network analysis since it contains a tunable parameter that sets the length scale of interactions studying how rankings and discovered communities change when this parameter is varied allows us to identify locally and globally important nodes and structures we apply the proposed metric to several benchmark networks and show that it leads to better insights into network structure than alternative metrics

integrating cross frequency and within band functional networks in resting state meg a multi layer network approach neuronal oscillations exist across a broad frequency spectrum and are thought to provide a mechanism of interaction between spatially separated brain regions since ongoing mental activity necessitates the simultaneous formation of multiple networks it seems likely that the brain employs interactions within multiple frequency bands as well as cross frequency coupling to support such networks here we propose a multi layer network framework that elucidates this pan spectral picture of network interactions our network consists of multiple layers frequency band specific networks that influence each other via inter layer cross frequency coupling applying this model to meg resting state data and using envelope correlations as connectivity metric we demonstrate strong dependency between within layer structure and inter layer coupling indicating that networks obtained in different frequency bands do not act as independent entities more specifically our results suggest that frequency band specific networks are characterised by a common structure seen across all layers superimposed by layer specific connectivity and inter layer coupling is most strongly associated with this common mode finally using a biophysical model we demonstrate that there are two regimes of multi layer network behaviour one in which different layers are independent and a second in which they operate highly dependent results suggest that the healthy human brain operates at the transition point between these regimes allowing for integration and segregation between layers overall our observations show that a complete picture of global brain network connectivity requires integration of connectivity patterns across the full frequency spectrum

estimating the number of clusters in a data set via the gap statistic we propose a method the gap statistic for estimating the number of clusters groups in a set of data the technique uses the output of any clustering algorithm e g k means or hierarchical comparing the change in within cluster dispersion with that expected under an appropriate reference null distribution some theory is developed for the proposal and a simulation study shows that the gap statistic usually outperforms other methods that have been proposed in the literature

a general method for deriving backlund transformations for the ernst equation a general method for deriving backlund transformations for the ernst equation 1968 is presented it is based on an ansatz of clairin s type and the consistency conditions of the resulting system of differential equations it is found that a simple ansatz gives ehler s transformation 1957 even though the functions appearing in the ansatz belong to a wide class of functions