Information Limits for Recovering a Hidden Community

We study the problem of recovering a hidden community of cardinality <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> from an <inline-formula> <tex-math notation="LaTeX">$n \times n$ </tex-math></inline-formula> symmetric data matrix <inline-formula> <tex-math notation="LaTeX">$A$ </tex-math></inline-formula>, where for distinct indices <inline-formula> <tex-math notation="LaTeX">$i,j$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$A_{ij} \sim P$ </tex-math></inline-formula> if <inline-formula> <tex-math notation="LaTeX">$i, j$ </tex-math></inline-formula> both belong to the community and <inline-formula> <tex-math notation="LaTeX">$A_{ij} \sim Q$ </tex-math></inline-formula> otherwise, for two known probability distributions <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> depending on <inline-formula> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula>. If <inline-formula> <tex-math notation="LaTeX">$P={\mathrm{ Bern}}(p)$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$Q={\mathrm{ Bern}}(q)$ </tex-math></inline-formula> with <inline-formula> <tex-math notation="LaTeX">$p>q$ </tex-math></inline-formula>, it reduces to the problem of finding a densely connected <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula>-subgraph planted in a large Erdös–Rényi graph; if <inline-formula> <tex-math notation="LaTeX">$P=\mathcal {N}(\mu ,1)$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$Q=\mathcal {N}(0,1)$ </tex-math></inline-formula> with <inline-formula> <tex-math notation="LaTeX">$\mu >0$ </tex-math></inline-formula>, it corresponds to the problem of locating a <inline-formula> <tex-math notation="LaTeX">$K \times K$ </tex-math></inline-formula> principal submatrix of elevated means in a large Gaussian random matrix. We focus on two types of asymptotic recovery guarantees as <inline-formula> <tex-math notation="LaTeX">$n \to \infty $ </tex-math></inline-formula>: 1) weak recovery: expected number of classification errors is <inline-formula> <tex-math notation="LaTeX">$o(K)$ </tex-math></inline-formula> and 2) exact recovery: probability of classifying all indices correctly converges to one. Under mild assumptions on <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>, and allowing the community size to scale sublinearly with <inline-formula> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula>, we derive a set of sufficient conditions and a set of necessary conditions for recovery, which are asymptotically tight with sharp constants. The results hold, in particular, for the Gaussian case, and for the case of bounded log likelihood ratio, including the Bernoulli case whenever <inline-formula> <tex-math notation="LaTeX">$({p}/{q})$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$({1-p})/({1-q})$ </tex-math></inline-formula> are bounded away from zero and infinity. Previous work has shown that if weak recovery is achievable; then, exact recovery is achievable in linear additional time by a simple voting procedure. We provide a converse, showing the condition for the voting procedure to succeed is almost necessary for exact recovery.