minimax rates of community detection in stochastic block models in statistics as well as in computer science probability and applied mathematics community detection for the stochastic block model sbm is probably the most studied topic in network analysis many methodologies have been proposed some beautiful and signicant phase transition results are obtained in various settings in this paper we provide a general minimax theory for community detection it gives minimax rates of the mis match ratio for a wide rage of settings including homogeneous and inhomogeneous sbms dense and sparse networks nite and growing number of communities the minimax rates are exponential dierent from polynomial rates we often see in statistical literature an immediate consequence of the result is to establish threshold phenomenon for strong consistency exact recovery as well as weak consistency partial recovery we obtain the upper bound by a range of penalized likelihood type approaches the lower bound is achieved by a novel reduction from a global mis match ratio to a local clustering problem for one node through an exchangeability property

pseudo likelihood methods for community detection in large sparse networks many algorithms have been proposed for fitting network models with communities but most of them do not scale well to large networks and often fail on sparse networks here we propose a new fast pseudo likelihood method for fitting the stochastic block model for networks as well as a variant that allows for an arbitrary degree distribution by conditioning on degrees we show that the algorithms perform well under a range of settings including on very sparse networks and illustrate on the example of a network of political blogs we also propose spectral clustering with perturbations a method of independent interest which works well on sparse networks where regular spectral clustering fails and use it to provide an initial value for pseudo likelihood we prove that pseudo likelihood provides consistent estimates of the communities under a mild condition on the starting value for the case of a block model with two communities

statistical computational tradeoffs in planted problems and submatrix localization with a growing number of clusters and submatrices we consider two closely related problems planted clustering and submatrix localization in the planted clustering problem a random graph is generated based on an underlying cluster structure of the nodes the task is to recover these clusters given the graph the submatrix localization problem concerns locating hidden submatrices with elevated means inside a large real valued random matrix of particular interest is the setting where the number of clusters submatrices is allowed to grow unbounded with the problem size these formulations cover several classical models such as planted clique planted densest subgraph planted partition planted coloring and the stochastic block model which are widely used for studying community detection graph clustering and bi clustering for both problems we show that the space of the model parameters cluster submatrix size edge probabilities and the mean of the submatrices can be partitioned into four disjoint regions corresponding to decreasing statistical and computational complexities 1 the impossible regime where all algorithms fail 2 the hard regime where the computationally expensive maximum likelihood estimator mle succeeds 3 the easy regime where the polynomial time convexified mle succeeds 4 the simple regime where a local counting thresholding procedure succeeds moreover we show that each of these algorithms provably fails in the harder regimes our results establish the minimax recovery limits which are tight up to universal constants and hold even with a growing number of clusters submatrices and provide order wise stronger performance guarantees for polynomial time algorithms than previously known our study demonstrates the tradeoffs between statistical and computational considerations and suggests that the minimax limits may not be achievable by polynomial time algorithms

asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications in this paper we extend our previous work on the stochastic block model a commonly used generative model for social and biological networks and the problem of inferring functional groups or communities from the topology of the network we use the cavity method of statistical physics to obtain an asymptotically exact analysis of the phase diagram we describe in detail properties of the detectability undetectability phase transition and the easy hard phase transition for the community detection problem our analysis translates naturally into a belief propagation algorithm for inferring the group memberships of the nodes in an optimal way i e that maximizes the overlap with the underlying group memberships and learning the underlying parameters of the block model finally we apply the algorithm to two examples of real world networks and discuss its performance

information limits for recovering a hidden community we study the problem of recovering a hidden community of cardinality k from an n times n symmetric data matrix a where for distinct indices i j a ij sim p if i j both belong to the community and a ij sim q otherwise for two known probability distributions p and q depending on n if p mathrm bern p and q mathrm bern q with p q it reduces to the problem of finding a densely connected k subgraph planted in a large erdos renyi graph if p mathcal n mu 1 and q mathcal n 0 1 with mu 0 it corresponds to the problem of locating a k times k principal submatrix of elevated means in a large gaussian random matrix we focus on two types of asymptotic recovery guarantees as n to infty 1 weak recovery expected number of classification errors is o k and 2 exact recovery probability of classifying all indices correctly converges to one under mild assumptions on p and q and allowing the community size to scale sublinearly with n we derive a set of sufficient conditions and a set of necessary conditions for recovery which are asymptotically tight with sharp constants the results hold in particular for the gaussian case and for the case of bounded log likelihood ratio including the bernoulli case whenever p q and 1 p 1 q are bounded away from zero and infinity previous work has shown that if weak recovery is achievable then exact recovery is achievable in linear additional time by a simple voting procedure we provide a converse showing the condition for the voting procedure to succeed is almost necessary for exact recovery

high dimensional analysis of semidefinite relaxations for sparse principal components principal component analysis pca is a classical method for dimensionality reduction based on extracting the dominant eigenvectors of the sample covariance matrix however pca is well known to behave poorly in the large p small n setting in which the problem dimension p is comparable to or larger than the sample size n this paper studies pca in this high dimensional regime but under the additional assumption that the maximal eigenvector is sparse say with at most k nonzero components we consider a spiked covariance model in which a base matrix is perturbed by adding a k sparse maximal eigenvector and we analyze two computationally tractable methods for recovering the support set of this maximal eigenvector as follows a a simple diagonal thresholding method which transitions from success to failure as a function of the rescaled sample size dia n p k n k 2 log p k and b a more sophisticated semidefinite programming sdp relaxation which succeeds once the rescaled sample size sdp n p k n klog p k is larger than a critical threshold in addition we prove that no method including the best method which has exponential time complexity can succeed in recovering the support if the order parameter sdp n p k is below a threshold our results thus highlight an interesting trade off between computational and statistical efficiency in high dimensional inference

energy landscape for large average submatrix detection problems in gaussian random matrices the problem of finding large average submatrices of a real valued matrix arises in the exploratory analysis of data from a variety of disciplines ranging from genomics to social sciences in this paper we provide a detailed asymptotic analysis of large average submatrices of an n times n gaussian random matrix the first part of the paper addresses global maxima for fixed k we identify the average and the joint distribution of the k times k submatrix having largest average value as a dual result we establish that the size of the largest square sub matrix with average bigger than a fixed positive constant is with high probability equal to one of two consecutive integers that depend on the threshold and the matrix dimension n the second part of the paper addresses local maxima specifically we consider submatrices with dominant row and column sums that arise as the local optima of iterative search procedures for large average submatrices for fixed k we identify the limiting average value and joint distribution of a k times k submatrix conditioned to be a local maxima in order to understand the density of such local optima and explain the quick convergence of such iterative procedures we analyze the number l n k of local maxima beginning with exact asymptotic expressions for the mean and fluctuation behavior of l n k for fixed k the mean of l n k is theta n k log n k 1 2 while the standard deviation is theta n 2k 2 k 1 log n k 2 k 1 our principal result is a gaussian central limit theorem for l n k that is based on a new variant of stein s method

a spectral technique for coloring random 3 colorable graphs let g3n p 3 be a random 3 colorable graph on a set of 3n vertices generated as follows first split the vertices arbitrarily into three equal color classes and then choose every pair of vertices of distinct color classes randomly and independently to be edges with probability p we describe a polynomial time algorithm that finds a proper 3 coloring of g3n p 3 with high probability whenever p geq c n where c is a sufficiently large absolute constant this settles a problem of blum and spencer who asked if an algorithm can be designed that works almost surely for p geq polylog n n j algorithms 19 1995 pp 204 234 the algorithm can be extended to produce optimal k colorings of random k colorable graphs in a similar model as well as in various related models implementation results show that the algorithm performs very well in practice even for moderate values of c

computational and statistical tradeoffs via convex relaxation modern massive datasets create a fundamental problem at the intersection of the computational and statistical sciences how to provide guarantees on the quality of statistical inference given bounds on computational resources such as time or space our approach to this problem is to define a notion of algorithmic weakening in which a hierarchy of algorithms is ordered by both computational efficiency and statistical efficiency allowing the growing strength of the data at scale to be traded off against the need for sophisticated processing we illustrate this approach in the setting of denoising problems using convex relaxation as the core inferential tool hierarchies of convex relaxations have been widely used in theoretical computer science to yield tractable approximation algorithms to many computationally intractable tasks in the current paper we show how to endow such hierarchies with a statistical characterization and thereby obtain concrete tradeoffs relating algorithmic runtime to amount of data

foliations the ergodic theorem and brownian motion regarding the leaves of a compact foliated riemannian manifold as the orbits of a dynamical system we will prove a birkhoff ergodic theorem asserting that the spatial average of a function equals its time average along random paths in the leaves there are several unexpected developments in the foundations first there always are measures on any compact foliated riemannian manifold m f with respect to which an adequate brownian motion along the leaves ergodic theorem holds second relative to these measures which are called harmonic there is a liouville type theorem for any bounded bore1 function on the manifold which is harmonic on each leaf namely for almost all leaves relative to any one of these harmonic measures this leaf harmonic function is constant on each leaf third the harmonic measures have a very reasonable local characterization in any foliation coordinate system they are transversal sums of h l dx l where h l is a positive harmonic function on the leaf l and dx l is the riemannian measure on l these statements comprise theorem 1 which is stated more precisely later in the introduction the first fact is surprising because up to now the only measures attached to foliations were the holonomy invariant transverse measures of plante 141 and ruelle sullivan 151 and some foliations do not admit these the second fact is unexpected because there are foliations where all the leaves are hyperbolic planes and each hyperbolic plane being conformal to the disk admits 285 0022 1236183 3 00

reconstruction and estimation in the planted partition model the planted partition model also known as the stochastic blockmodel is a classical cluster exhibiting random graph model that has been extensively studied in statistics physics and computer science in its simplest form the planted partition model is a model for random graphs on n nodes with two equal sized clusters with an between class edge probability of q and a within class edge probability of p although most of the literature on this model has focused on the case of increasing degrees ie pn qn rightarrow infty as n rightarrow infty the sparse case p q o 1 n is interesting both from a mathematical and an applied point of view a striking conjecture of decelle krzkala moore and zdeborova based on deep non rigorous ideas from statistical physics gave a precise prediction for the algorithmic threshold of clustering in the sparse planted partition model in particular if p a n and q b n then decelle et al conjectured that it is possible to cluster in a way correlated with the true partition if a b 2 2 a b and impossible if a b 2 2 a b by comparison the best known rigorous result is that of coja oghlan who showed that clustering is possible if a b 2 c a b for some sufficiently large c we prove half of their prediction showing that it is indeed impossible to cluster if a b 2 2 a b furthermore we show that it is impossible even to estimate the model parameters from the graph when a b 2 2 a b on the other hand we provide a simple and efficient algorithm for estimating a and b when a b 2 2 a b following decelle et al our work establishes a rigorous connection between the clustering problem spin glass models on the bethe lattice and the so called reconstruction problem this connection points to fascinating applications and open problems

the dimension of the space of siegel eisenstein series of weight one in general it is difficult to determine the dimension of the space of siegel modular forms of low weights in particular the dimensions of the spaces of cusp forms are known in only a few cases in this paper we calculate the dimension of the space of siegel eisenstein series of weight 1 which is a certain subspace of a complement of the space of cusp forms

improved graph clustering graph clustering involves the task of dividing nodes into clusters so that the edge density is higher within clusters as opposed to across clusters a natural classic and popular statistical setting for evaluating solutions to this problem is the stochastic block model also referred to as the planted partition model in this paper we present a new algorithm a convexified version of maximum likelihood for graph clustering we show that in the classic stochastic block model setting it outperforms existing methods by polynomial factors when the cluster size is allowed to have general scalings in fact it is within logarithmic factors of known lower bounds for spectral methods and there is evidence suggesting that no polynomial time algorithm would do significantly better we then show that this guarantee carries over to a more general extension of the stochastic block model our method can handle the settings of semirandom graphs heterogeneous degree distributions unequal cluster sizes unaffiliated nodes partially observed graphs planted clique coloring and so on in particular our results provide the best exact recovery guarantees to date for the planted partition planted k disjoint cliques and planted noisy coloring models with general cluster sizes in other settings we match the best existing results up to logarithmic factors

detection of an anomalous cluster in a network we consider the problem of detecting whether or not in a given sensor network there is a cluster of sensors which exhibit an unusual behavior formally suppose we are given a set of nodes and attach a random variable to each node we observe a realization of this process and want to decide between the following two hypotheses under the null the variables are i i d standard normal under the alternative there is a cluster of variables that are i i d normal with positive mean and unit variance while the rest are i i d standard normal we also address surveillance settings where each sensor in the network collects information over time the resulting model is similar now with a time series attached to each node we again observe the process over time and want to decide between the null where all the variables are i i d standard normal and the alternative where there is an emerging cluster of i i d normal variables with positive mean and unit variance the growth models used to represent the emerging cluster are quite general and in particular include cellular automata used in modelling epidemics in both settings we consider classes of clusters that are quite general for which we obtain a lower bound on their respective minimax detection rate and show that some form of scan statistic by far the most popular method in practice achieves that same rate within a logarithmic factor our results are not limited to the normal location model but generalize to any one parameter exponential family when the anoma lous clusters are large enough

clustering partially observed graphs via convex optimization this paper considers the problem of clustering a partially observed unweighted graph i e one where for some node pairs we know there is an edge between them for some others we know there is no edge and for the remaining we do not know whether or not there is an edge we want to organize the nodes into disjoint clusters so that there is relatively dense observed connectivity within clusters and sparse across clusters we take a novel yet natural approach to this problem by focusing on finding the clustering that minimizes the number of disagreements i e the sum of the number of observed missing edges within clusters and observed present edges across clusters our algorithm uses convex optimization its basis is a reduction of disagreement minimization to the problem of recovering an unknown low rank matrix and an unknown sparse matrix from their partially observed sum we evaluate the performance of our algorithm on the classical planted partition stochastic block model our main theorem provides sufficient conditions for the success of our algorithm as a function of the minimum cluster size edge density and observation probability in particular the results characterize the tradeoff between the observation probability and the edge density gap when there are a constant number of clusters of equal size our results are optimal up to logarithmic factors

recovering a single community with side information we study the effect of the quality and quantity of side information on the recovery of a hidden community of size k o n in a graph of size n side information for each node in the graph is modeled by a random vector in which either the vector dimension or the llr of each component with respect to node labels is independent of n these two models represent the variation in quality and quantity of side information under maximum likelihood detection we calculate tight necessary and sufficient conditions for exact recovery of the labels we demonstrate how side information needs to evolve with n in terms of either its quantity or quality to improve the exact recovery threshold a similar set of results are obtained for weak recovery under belief propagation tight necessary and sufficient conditions for weak recovery are calculated when the llrs are constant and sufficient conditions when the llrs vary with n moreover we design and analyze a local voting procedure using side information that can achieve exact recovery when applied after belief propagation

matrix estimation by universal singular value thresholding consider the problem of estimating the entries of a large matrix when the observed entries are noisy versions of a small random fraction of the original entries this problem has received widespread attention in recent times especially after the pioneering works of emmanuel candes and collaborators this paper introduces a simple estimation procedure called universal singular value thresholding usvt that works for any matrix that has a little bit of structure surprisingly this simple estimator achieves the minimax error rate up to a constant factor the method is applied to solve problems related to low rank matrix estimation blockmodels distance matrix completion latent space models positive definite matrix completion graphon estimation and generalized bradley terry models for pairwise comparison

umbilical foliations and transversely holomorphic flows consider a codimension 1 foliation 9 on a closed oriented 3 manifold m equipped with a riemannian metric g and denote by jv the orthogonal one dimensional foliation there is an interesting relationship between the local geometry of the leaves of and the transverse structure of f more precisely assume for simplicity that jv is oriented by a unit speed flow denote by the orthogonal projection of the tangent space tm onto the tangent space tsf and by ii the second fundamental form at x of the leaf of through the point x then if v is any vector tangent to 9 at x one easily checks that

how hard is it to approximate the best nash equilibrium the quest for a polynomial time approximation scheme ptas for nash equilibrium in a two player game which emerged as a major open question in algorithmic game theory seeks to circumvent the ppad completeness of finding an exact nash equilibrium by finding an approximate equilibrium the closely related problem of finding an equilibrium maximizing a certain objective such as social welfare was shown to be np hard gilboa and zemel games econom behav 1 1989 pp 80 93 however this np hardness is unlikely to extend to approximate equilibria since the latter admits a quasi polynomial time algorithm lipton markakis and mehta in proceedings of the 4th acm conference on electronic commerce acm new york 2003 pp 36 41 we show that this optimization problem namely finding in a two player game an approximate equilibrium achieving a large social welfare is unlikely to have a polynomial time algorithm one interpretation of our results is that a ptas for nash equilibrium if it exists should not extend to a ptas for finding the best nash equilibrium technically our result is a reduction from the notoriously difficult problem in modern combinatorics of finding a planted but hidden clique in a random graph g n 1 2 our reduction starts from an instance with planted clique size o log n for comparison the currently known algorithms are effective only for a much larger clique size omega sqrt n

nuclear norm minimization for the planted clique and biclique problems we consider the problems of finding a maximum clique in a graph and finding a maximum edge biclique in a bipartite graph both problems are np hard we write both problems as matrix rank minimization and then relax them using the nuclear norm this technique which may be regarded as a generalization of compressive sensing has recently been shown to be an effective way to solve rank optimization problems in the special case that the input graph has a planted clique or biclique i e a single large clique or biclique plus diversionary edges our algorithm successfully provides an exact solution to the original instance for each problem we provide two analyses of when our algorithm succeeds in the first analysis the diversionary edges are placed by an adversary in the second they are placed at random in the case of random edges for the planted clique problem we obtain the same bound as alon krivelevich and sudakov as well as feige and krauthgamer but we use different techniques

the power of side information in subgraph detection in this paper we tackle the problem of hidden community detection we consider belief propagation bp applied to the problem of detecting a hidden erd s renyi er graph embedded in a larger and sparser er graph in the presence of side information we derive two related algorithms based on bp to perform subgraph detection in the presence of two kinds of side information the first variant of side information consists of a set of nodes called cues known to be from the subgraph the second variant of side information consists of a set of nodes that are cues with a given probability it was shown in past works that bp without side information fails to detect the subgraph correctly when a so called effective signal to noise ratio parameter falls below a threshold in contrast in the presence of nontrivial side information we show that the bp algorithm achieves asymptotically zero error for any value of a suitably defined phase transition parameter we validate our results on synthetic datasets and a few real world networks

spectral clustering and the high dimensional stochastic blockmodel networks or graphs can easily represent a diverse set of data sources that are characterized by interacting units or actors social ne tworks representing people who communicate with each other are one example communities or clusters of highly connected actors form an essential feature in the structure of several empirical networks spectral clustering is a popular and computationally feasi ble method to discover these communities the stochastic block model holland et al 1983 is a social network model with well defined communities each node is a member of one community for a network generated from the stochastic block model we bound the number of nodes misclus tered by spectral clustering the asymptotic results in th is paper are the first clustering results that allow the number of clusters in the model to grow with the number of nodes hence the name high dimensional in order to study spectral clustering under the stochastic block model we first show that under the more general latent space model the eigenvectors of the normalized graph laplacian asymptotically converge to the eigenvectors of a population normal ized graph laplacian aside from the implication for spectral clustering this provides insight into a graph visualization technique our method of studying the eigenvectors of random matrices is original ams 2000 subject classifications primary 62h30 62h25 secondary 60b20

global and local information in clustering labeled block models the stochastic block model is a classical cluster exhibiting random graph model that has been widely studied in statistics physics and computer science in its simplest form the model is a random graph with two equal sized clusters with intracluster edge probability p and intercluster edge probability q we focus on the sparse case i e p q o 1 n which is practically more relevant and also mathematically more challenging a conjecture of decelle krzakala moore and zdeborova based on ideas from statistical physics predicted a specific threshold for clustering the negative direction of the conjecture was proved by mossel neeman and sly 2012 and more recently the positive direction was independently proved by massoulie and mossel neeman and sly in many real network clustering problems nodes contain information as well we study the interplay between node and network information in clustering by studying a labeled block model where in addition to the edge information the true cluster labels of a small fraction of the nodes are revealed in the case of two clusters we show that below the threshold a small amount of node information does not affect recovery on the other hand we show that for any small amount of information efficient local clustering is achievable as long as the number of clusters is sufficiently large as a function of the amount of revealed information

exact recovery in the hypergraph stochastic block model a spectral algorithm abstract we consider the exact recovery problem in the hypergraph stochastic block model hsbm with k blocks of equal size more precisely we consider a random d uniform hypergraph h with n vertices partitioned into k clusters of size s n k hyperedges e are added independently with probability p if e is contained within a single cluster and q otherwise where 0 q p 1 we present a spectral algorithm which recovers the clusters exactly with high probability given mild conditions on n k p q and d our algorithm is based on the adjacency matrix of h which is a symmetric n n matrix whose u v th entry is the number of hyperedges containing both u and v to the best of our knowledge our algorithm is the first to guarantee exact recovery when the number of clusters k n

computational barriers in minimax submatrix detection this paper studies the minimax detection of a small submatrix of elevated mean in a large matrix contaminated by additive gaussian noise to investigate the tradeoff between statistical performance and computational cost from a complexity theoretic perspective we consider a sequence of discretized models which are asymptotically equivalent to the gaussian model under the hypothesis that the planted clique detection problem cannot be solved in randomized polynomial time when the clique size is of smaller order than the square root of the graph size the following phase transition phenomenon is established when the size of the large matrix p to infty if the submatrix size k theta p alpha for any alpha in 0 2 3 computational complexity constraints can incur a severe penalty on the statistical performance in the sense that any randomized polynomial time test is minimax suboptimal by a polynomial factor in p if k theta p alpha for any alpha in 2 3 1 minimax optimal detection can be attained within constant factors in linear time using schatten norm loss as a representative example we show that the hardness of attaining the minimax estimation rate can crucially depend on the loss function implications on the hardness of support recovery are also obtained

achieving exact cluster recovery threshold via semidefinite programming the binary symmetric stochastic block model deals with a random graph of n vertices partitioned into two equal sized clusters such that each pair of vertices is independently connected with probability p within clusters and q across clusters in the asymptotic regime of p a log n n and q b log n n for fixed a b and n to infty we show that the semidefinite programming relaxation of the maximum likelihood estimator achieves the optimal threshold for exactly recovering the partition from the graph with probability tending to one resolving a conjecture of abbe et al furthermore we show that the semidefinite programming relaxation also achieves the optimal recovery threshold in the planted dense subgraph model containing a single cluster of size proportional to n

convex optimization for the planted k disjoint clique problem we consider the k disjoint clique problem the input is an undirected graph g in which the nodes represent data items and edges indicate a similarity between the corresponding items the problem is to find within the graph k disjoint cliques that cover the maximum number of nodes of g this problem may be understood as a general way to pose the classical clustering problem in clustering one is given data items and a distance function and one wishes to partition the data into disjoint clusters of data items such that the items in each cluster are close to each other our formulation additionally allows noise nodes to be present in the input data that are not part of any of the cliques the k disjoint clique problem is np hard but we show that a convex relaxation can solve it in polynomial time for input instances constructed in a certain way the input instances for which our algorithm finds the optimal solution consist of k disjoint large cliques called planted cliques that are then obscured by noise edges inserted either at random or by an adversary as well as additional nodes not belonging to any of the k planted cliques

incoherence optimal matrix completion this paper considers the matrix completion problem we show that it is not necessary to assume joint incoherence which is a standard but unintuitive and restrictive condition that is imposed by previous studies this leads to a sample complexity bound that is orderwise optimal with respect to the incoherence parameter as well as to the rank r and the matrix dimension n up to a log factor as a consequence we improve the sample complexity of recovering a semidefinite matrix from o nr 2 log 2 n to o nr log 2 n and the highest allowable rank from theta sqrt n log n to theta n log 2 n the key step in proof is to obtain new bounds in terms of the ell infty 2 norm defined as the maximum of the row and column norms of a matrix to illustrate the applicability of our techniques we discuss extensions to singular value decomposition projection structured matrix completion and semisupervised clustering for which we provide orderwise improvements over existing results finally we turn to the closely related problem of low rank plus sparse matrix decomposition we show that the joint incoherence condition is unavoidable here for polynomial time algorithms conditioned on the planted clique conjecture this means it is intractable in general to separate a rank omega sqrt n positive semidefinite matrix and a sparse matrix interestingly our results show that the standard and joint incoherence conditions are associated respectively with the information statistical and computational aspects of the matrix decomposition problem