Hypothesis Testing with Network Partition Quality Statistics

Scholars of social networks often rely on summary statistics to measure and compare the structures of their networks of interest. However, measuring the uncertainty inherent in these summaries can be challenging, thus making hypothesis testing for network summaries difficult. Computational and nonparametric procedures can over come these difficulties by allowing researchers to generate reference distributions for comparison directly from their data. In this research, I demonstrate the use of nonparametric hypothesis testing in networks using the popular network summary statistic modularity. I provide a method based on permutation testing for assessing whether a particular network modularity score is better than a researcher might expect due to random chance. I then create a simulation test of both network modularity and the simulated reference distribution I propose. Finally, I provide an empirical example of modularity and its reference distribution using cosponsorship networks from U.S. state legislatures. ∗Department of Political Science, University of North Carolina Chapel Hill The quantitative analysis of social networks has experienced a large increase in popularity over the last 10 to 15 years. As the tools for such analysis have become easier to access and more efficient to use, more social scientists have adopted a social networks perspective as a potentially useful lens through which to examine human behavior. With this advance in a networks perspective have come a number of new tools for analyzing and measuring constructs of interest in networks. Unfortunately, many of these new measures lack clear ways to assess the uncertainty inherent in any single sample of a network. This lack of clear measurements for uncertainty in network summary statistics is due, at least in part, to a lack of understanding regarding the distributional properties of networks and their summaries. Fortunately, as computational power increases, and nonparametric techniques like bootstrapping and permutation testing become more popular, the need for researchers to rely on established probability distributions as references for their statistics is diminishing. This holds particular promise for scholars of social networks who wish to engage in hypothesis testing on their network summaries. In this article, I demonstrate how a researcher might use permutation testing to assert whether or not a particularly popular network summary statistic, network modularity, is different than might be expected at random. The permutation test proves to be a useful way to assert when modularity is unusually large given the variance in a single measurement of a social network, and when an observed modularity score might be due to random chance. After introducing and providing a simulation-based analysis of the permutation test, I provide a demonstration of that test using real world data on the network of cosponsorships in U.S. state legislatures. This practical demonstration provides a quick and understandable measure of how the major political parties of the United States structure the collaborative activities of state legislators, and when that structuring is unusually powerful. Finally, I also provide an analysis of the properties of permutation tests for network modularity. The results of this analysis indicate that the number of connections and the size of sub-groups relative to 2 one another in a network influence the size of the reference distributions that emerge from my permutation test. This analysis also indicates that the degree of real community structure in a network influences the location of the permutation-based reference distribution. Measuring Partition Quality with Network Modularity Community detection is the act of partitioning or dividing a social network into distinct subgroups. Community detection algorithms attempt to partition networks so that there are dense connections between all the actors in a subgroup (or community) and sparse connections between actors of different subgroups. A community is then the set of actors within a single group in an optimal partition of the network with the implied assumption that something unique about that group of actors caused their within group cohesion and has set them apart from the other groups. Scholars have developed a variety of approaches to discover communities in an observed network (Newman 2003, Newman 2004, Newman and Girvan 2004, Pons and Latapy 2005, Newman 2006, Reichardt et al, 2006, Raghaven, Albert and Kumara 2007). Most, if not all, community detection routines rely on quality statistics to assert which potential network partition is the best, and thus, represents the “true” community structure. To measure the quality of network partitions, scholars of social networks typically use the popular quality statistic modularity (Newman and Girvan 2004, Newman 2006). Modularity is usually defined as the number of ties within a group compared to the number of ties that might be expected at random. With the most common definition of “at random” in place, The igraph package for social network analysis has six different algorithms for community detection (Csardi and Nepusz 2006). The initial article debuting modularity by Newman and Girvan (2004) has more than 2100 citations as of 2011. The precise definition of what “at random” means can vary a bit depending on the questions in which an analyst is interested. It is generally considered to be the number of connections (called edges) within groups if a network with the same communities had the same number edges placed between dyads randomly. 3 this simplifies to a measure of the number of ties within a group relative to the number of ties between any two other actors in the network. The mathematical formula for network modularity using the industry standard definition is: Mod = 1 2m ∑ ij [Aij − kikj 2m ] ∗ δ(ci, cj) (1) where m is the total number of connections in the network, Aij is the connection between actors i and j, ki is the total number of connections actor i has, and δ(ci, cj) is the Kroneker delta for communities of actors i and j. Thus, modularity examines all the actors in the same community (δ(ci, cj)), adds up their connections with one another (Aij), and subtracts the connections we might expect those actors to have in the network generally ( kikj 2m ). Finally, the sum of all these components is normalized by the total number of connections in the network ( 1 2m ). In networks with edge weights, where connections between actors take on more than a zero or a one, the calculation is very similar. In these weighted networks, Aij becomes the connection strength between i and j, m becomes the sum of all the edge weights in the network, and ki becomes the total weighted connections of actor i. Modularity values range between -1 and 1. A value of 1 occurs when no ties cross community lines and all the ties within a community are realized. A value of -1 occurs when no ties within a community are realized and all the possible ties across communties occur. It is possible for modularity to take on a value of precisely zero, which occurs when all the ties in a network are realized. However, as I will demonstrate in the next section, finite samples and stochastic social behaviors can generate non-zero modularity values even in networks where no real community structure exists. Kroneker’s delta is a mathematical operation which returns a value of 1 if the two numbers within the operation are equal to one another, and zero otherwise. Thus, all the ties within communities and across communities occur. This means that modularity is not a true probabilty distribution. Technically, it is a discrete measure, that can only take on a finite number of values because there are only a finite number of possible community structures in any particular network. 4 Simulating a Reference Distribution for Comparison While the point estimate of modularity is derived relative to some null model, the modularity score itself communicates nothing about the distribution of modularity under that null model. In order to provide a sense of the degree to which an observed modularity score reflects something unusual for a given network, analysts require some sense of the baseline level of modularity that might be observed at random in that network. By generating a reference distribution for modularity in a given network, scholars can develop a better sense of the degree to which a division partitions the network unusually well. A useful avenue for generating reference distributions for modularity is permutation (Erikson, Rader, and Pinto 2010, O’Gorman 2005, Edgington and Onghena 2007). Using permutations of actors’ attributes that could potentially divide a network, an analyst can simulate a random distribution of modularity for a particular social network and a particular attribute. The permutation test works as follows. First, an analyst measures a network and either through theoretical insight or community discovery develops a partition of interest for the network. That partition of interest provides a modularity score, but the analyst would like to know whether that modularity score is better than one might expect at random (implying that the partition is better than one might expect at random). The analyst then takes the community memberships defined by the partition and rearranges them amongst the actors, maintaining the size of groups, but randomly assigning each actor to a new community. This new partition with randomly assigned community memberships also produces a modularity score. This random assignment to communities is done hundreds (or perhaps thousands) of times and a distribution of modularity scores emerges. Table 1 provides an example of a partition and its implied membership permutations. Technically, the permutation approach does not develop a continuous reference distribution. Because there are only a fixed number of permutations