Ecometrics in the Age of Big Data: Measuring andAssessing "Broken Windows" Using Large-scale
Administrative Records
Citation
O’Brien, D. T., R. J. Sampson, and C. Winship. 2015. “Ecometrics in the Age of Big Data:
Measuring and Assessing ‘Broken Windows’ Using Large-Scale Administrative Records.”
Sociological Methodology (April 16). doi:10.1177/0081175015576601.

Published Version
10.1177/0081175015576601

Permanent link
http://nrs.harvard.edu/urn-3:HUL.InstRepos:17692600

Terms of Use
This article was downloaded from Harvard University’s DASH repository, and is made available
under the terms and conditions applicable to Open Access Policy Articles, as set forth at http://
nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#OAP

Share Your Story
The Harvard community has made this article openly available.
Please share how this access benefits you. Submit a story .
Accessibility

Citation: O’Brien, Daniel, Robert J. Sampson, and Christopher Winship. 2015. “Ecometrics in
the Age of Big Data: Measuring and Assessing ‘Broken Windows’ Using Large-scale
Administrative Records.” Sociological Methodology 45: Online first version—DOI:
10.1177/0081175015576601.

Ecometrics in the Age of Big Data:
Measuring and Assessing “Broken Windows” Using Large-scale Administrative Records

Daniel O’Brien, Robert J. Sampson and Christopher Winship
Harvard University

We thank the National Science Foundation (Grant # SMA 1338446), the John D. and Catherine
T. MacArthur Foundation (Grant # 13-105766-000-USP), and the Radcliffe Institute forAdvanced Study for funding assistance. We also thank the City of Boston’s “Office of New
Urban Mechanics” and “Department of Innovation and Technology” for supporting our
examination of government data, and the Editor and reviewers of Sociological Methodology for
helpful comments on earlier drafts. A version of this paper was presented at the annual meetingof the American Association for the Advancement of Science in Chicago, February 15th, 2014. 	
  Abstract:
The collection of large-scale administrative records in electronic form by many cities provides a
new opportunity for the measurement and longitudinal tracking of neighborhood characteristics,
but one that requires novel methodologies that convert such data into research-relevant measures.
The authors illustrate these challenges by developing measures of “broken windows” from
Boston’s constituent relationship management (CRM) system (aka 311 hotline). A 16-montharchive of the CRM database contains more than 300,000 address-based requests for city
services, many of which reference physical incivilities (e.g., graffiti removal). The authors carryout three ecometric analyses, each building on the previous one. Analysis 1 examines the content
of the measure, identifying 28 items that constitute two independent constructs, private neglect
and public denigration. Analysis 2 assesses the validity of the measure by using investigatorinitiated neighborhood audits to examine the “civic response rate” across neighborhoods.
Indicators of civic response were then extracted from the CRM database so that measurement
adjustments could be automated. These adjustments were calibrated against measures of litterfrom the objective audits. Analysis 3 examines the reliability of the composite measure ofphysical disorder at different spatiotemporal windows, finding that census tracts can be measured
at two-month intervals and census block groups at six-month intervals. The final measures arehighly detailed, can be tracked longitudinally, and are virtually costless. This framework thus
provides an example of how new forms of large-scale administrative data can yield ecometricmeasurement for urban science while illustrating the methodological challenges that must be
addressed.
Keywords: Ecometrics; urban sociology; “big data”; computational social science; physicaldisorder; broken windows; 311 hotlines

The global move towards digital technology has instigated a marked shift in the practice of
science over the last two decades. Surveys and experiments are now often conducted throughinternet platforms; GPS devices and other sensors allow us to track patterns of movement andbehavior; and computer processing technology has supported the development of new forms ofstatistical analysis. A recent consequence of this “digital revolution” is the availability of largescale administrative data that might prove useful in research. Many public agencies and privatecompanies systematically collect information on services and clients and compile it in digital
databases. Some of these are more detailed versions of familiar data, like crime reports, whileothers, like cell phone records or citizen requests for governmental services, are novel. These“big” or next-generation data offer the opportunity to paint a comprehensive picture of cities,
which has the potential to transform theoretical models of urban governance and social behavior(Lazer et al. 2009).
Despite considerable excitement at this prospect, big data have not yet becomecommonplace in contemporary social science research, in part, it seems, because researchers do
not entirely know what to make of them. Without a clear understanding of how these new data
sources contribute to our ongoing debates and the questions facing our fields, some mightreasonably consider their promise as being overblown. There is thus a need for methodologies
that can connect big data with the current practice of social science.
We offer one such “proof of concept” in the present paper, utilizing a database of over
300,000 citizen-generated requests for public services in Boston, MA to measure the conditionsof urban neighborhoods across space and time. Building on the methodology of ecometrics(Raudenbush and Sampson 1999), we construct and assess a measure of physical disorder, one of
the most widely used and popular concepts in urban sociology, criminology, and public policy.

	
  

1	
  

Although the idea of disorder has a long history in sociology, it has received increased attention
in recent decades owing to the influential “broken windows” theory of crime and urban decline(Raudenbush and Sampson 1999; Ross et al. 2001; Skogan 1992; Wilson and Kelling 1982),
making it an ideal test case for assessing the potential for ecometrics based on large-scale
administrative data.

1. AN ECOMETRIC APPROACH TO DISORDER
Almost fifteen years ago, Raudenbush and Sampson (1999) proposed a systematic approach tothe measurement of neighborhood social ecology, what they termed “ecometrics.” They
encouraged researchers to borrow three tools developed by psychometricians for themeasurement of behavior: 1) item-response models, which call for the use of scales whose
multiple items vary in their difficulty, allowing for greater precision in measurement acrossneighborhoods; 2) factor analysis, in order to address the interrelation between items, and to
identify one or a few latent constructs that the items reflect; and 3) generalizability theory, which
requires criteria for ensuring that a given measurement of a neighborhood is reflective of the“true” score on the characteristic of interest, and not overly influenced by either stochastic orconfounding processes. These guidelines, along with the illustrative examples that accompanied
them, provided researchers with a step-by-step methodology for developing survey andobservational protocols that could measure ecometrics, one that has been implemented byhundreds of researchers in dozens of cities.
The advent of large administrative data represents a new opportunity for ecometric study.
The giga- and terabytes of data being collected by both public and private sector entities are arich, low-cost resource for measuring the characteristics of neighborhoods, but using them in this

	
  

2	
  

manner poses clear methodological and substantive challenges. Administrative data are notcollected according to any research question or plan, and thus, in their raw state, lack some of thecharacteristics expected of researcher-collected data. These challenges are well suited to thetechniques common to ecometric study, which can act as a guide to both what is missing or
occluded in such data sets, as well as how a researcher might address such issues.
We focus here on one of the most influential concepts in the urban sciences—that ofphysical disorder, including the iconic “broken window,” the accumulation of litter, the presenceof graffiti, or other indications that a neighborhood is poorly maintained and monitored. Suchincivilities are often associated with elevated crime rates (Raudenbush and Sampson 1999;
Wilson and Kelling 1982) and lower mental, physical and behavioral health among residents(Burdette and Hill 2008; Caughy et al. 2008; Furr-Holden et al. 2012; Mujahid et al. 2008;
O'Brien and Kauffman 2013; Wen et al. 2006), attracting attention from a variety of disciplines(Caughy et al. 2001; Cohen et al. 2000; O'Brien and Wilson 2011; Taylor 2001). Physicaldisorder’s importance as a neighborhood characteristic is such that it was also one of the two test
cases that Raudenbush and Sampson (1999) used to illustrate their methodological approach toecometrics.
Physical disorder is traditionally measured either through surveys or detailed
neighborhood audits (e.g., Taylor, 2001; Sampson and Raudenbush 1999), but the effort and costassociated with such protocols have made whole-city assessments challenging and precise
longitudinal tracking nearly impossible. Modern technology used by city agencies, however, is
now recording similar information in real time. These databases have the potential to supplement
traditional ecometric protocols. One such database is a result of a recent policy innovation called
the constituent relationship management (CRM) system. Colloquially known as a Mayor’s

	
  

3	
  

Hotline or 311 line, these systems provide constituents with a variety of channels for directly
requesting services from the city government, using phone, internet or smart phone applications
that communicate requests to the appropriate department. The resultant database is a detailed
documentation of constituent needs, leading some of its initial implementers to refer to it as “theeyes and ears of the city”—Jane Jacobs (1961) meets “big data” as it were. Many of the requests
refer to individual instances of physical disorder, like graffiti or abandoned housing, giving thedatabase the ability to reflect their prevalence across neighborhoods.
Although the potential impacts of big data on science have been over-hyped (Pigliucci2009) and there have been highly visible failures of prediction based on large-scale data (Lazeret al. 2014), the CRM database offers a number of possibilities as an alternative or supplement to
expensive new data collection—especially in a time of declining research support. For one, thesystem receives hundreds of cases every day, each attributed to a particular address orintersection, giving researchers considerable flexibility in how they might geographically divide
the city. It also lends itself to the longitudinal tracking of physical disorder, a major advanceconsidering that no whole-city protocol to date has been conducted more than once in a five-year
period. Further, the database differentiates between dozens of case types, allowing greater
precision in defining the events that comprise disorder than has previously been possible.
The CRM database was not created for the purposes of disorder research, however, andhas three weaknesses that any methodology must address. First, the substantive content of thedatabase is noisy, and it is not immediately apparent what it can measure nor how it can do so.
Some cases, like requests for graffiti removal, are clear examples of physical disorder, but others,
like scheduling a bulk item pickup, are not. Second, there may be some aspect of data collectionthat creates systematic biases in measurement. For instance and quite importantly, the CRM

	
  

4	
  

system may suffer from skewed reporting in the incidence of disorder across neighborhoods.
Last, there is no information about what scale of geographical analysis the database can
support—for example, census block groups or tracts—nor over what time spans.
Whereas Raudenbush and Sampson (1999) forwarded criteria for survey- andobservation-based measurement across geographical units, the CRM database makes clear theneed for a new set of guidelines for the utilization of administrative data in the creation of
ecometrics. The current manuscript uses the CRM database from Boston to illustrate the multiple
analytic steps in the formulation of original ecometrics. This process is reported in three parts,
each requiring its own distinct logic, data sources, and analytical approach. Analysis 1 examines
the content within CRM database that reflects physical disorder, and uses correlational analyses
to identify an underlying factor structure. Analysis 2 then addresses the validity of any measureextracted from the CRM database by assessing biases in reporting through original datacollection involving neighborhood audits. A method is then developed for using auxiliary
measures from within the CRM database to estimate these biases, and to help account for overor underreporting. Analysis 3 then examines the reliability of these composite measures byidentifying the spatial and temporal ranges at which their measurement is consistent. In eachcase we spell out the necessary assumptions in our analysis and that are inherent to big data.

2. ANALYSIS 1: OPERATIONALIZING PHYSICAL DISORDER
When Raudenbush and Sampson (1999) developed their methodology for ecometrics, theyemphasized the development of item-response models, and their examination through factor
analysis, an approach that had been in common use in the field of psychometrics for decades.
When conducting a neighborhood audit, for example, a protocol might measure a variety of

	
  

5	
  

items that collectively capture an overall pattern. A factor analysis based on theirintercorrelations would then help determine which of these items in fact measured the desired
construct, while also testing whether they reflected one or multiple constructs regarding theneighborhood’s ecology. The challenge with next-generation data, however, is that it is not
immediately apparent what they can measure. Traditionally, research measures are derived fromprotocols written by the researchers themselves, and their items are based on an underlying
theoretical construct. Administrative data are not endowed with an a priori theoreticalorganization of this sort. The CRM database, for example, is a byproduct of a system intended totransmit the needs of constituents to the appropriate government agencies, and its organizationreflects this function, rather than a deliberate intent to measure neighborhood characteristics.
Nonetheless, with thousands of requests spanning over 150 case types, the CRM offers a richstore of information for measuring neighborhood characteristics. But before factor analysis canbe considered, it falls to researchers to use existing theory to identify those specific items that arelikely to be relevant.
Physical disorder is typically defined as any aspect of a neighborhood’s visual cues thatreflect a “breakdown of the local social order” (pg. 2, Skogan 1992), though this has come tomean two different things in practice. Raudenbush and Sampson’s (1999) measure focused
specifically on the artifacts of physical incivilities that were publicly visible and that denigrated
the public space according to broken windows theory, such as graffiti and various forms of litter
indicating illegal or typically problematic behavior (e.g., used condoms, empty beer bottles,
hypodermic needles). A variety of other researchers have expanded this definition to include any
item that might be evidence that “spaces are not being kept or used properly” (pg. 5, Taylor,2001). This had led to a variety of protocols that also include items that, while not the result of

	
  

6	
  

flagrant incivilities, reflect an overall pattern of neglect, including deteriorating or abandoned
housing, unkempt lawns or vegetation, and litters of all kinds (Caughy et al. 2001; Cohen et al.
2000; Furr-Holden et al. 2008; O'Brien and Wilson 2011; Ross and Mirowsky 1999; Rundle etal. 2011; Skogan 1992; Taylor 2001). One important consequence of this approach is that it
extends measurement to elements of the neighborhood that are technically private, but whose
appearance and use are a visible part of the local scenery, like front porches, lawns, and thefacades of houses. Despite this distinction, factor analyses on such protocols often identify asingle factor, though Ross and Mirowsky (1999) found evidence for two latent constructs theyreferred to as disorder and decay, approximating the dichotomy described here.
In order to make the greatest use of the CRM database, this study will identify case types
that reflect either private neglect or public denigration. Some will correspond directly to items inprevious methodologies, like a report of an abandoned house, or a request for graffiti removal.
But others will be novel, either because they are too uncommon to be measured through one-timeneighborhood audits (e.g., cars illegally parked on a lawn), or because they are more likely to beexperienced in private spaces, like a rodent infestation. This latter opportunity to “look” at theconditions inside houses could potentially add a new dimension to the measurement of disorder,
one that has been hinted at in previous protocols that examine visible deterioration, but has notbeen completely accessible. Altogether, it is possible to construct a battery of “items” that offersa greater breadth and depth than any previous measure of physical disorder. The second stage ofthe analysis will then use factor analysis to explore the dimensionality of these items. Given theirlarge number, it seems feasible that they will not describe a unitary construct, but one withmultiple aspects that are related but distinct.

	
  

7	
  

2.1. The CRM Database
Boston’s CRM system received 365,729 requests for service via its three channels(hotline calls, internet self-service portal and smartphone application) between March 1, 2010
and June 29, 2012, 334,874 of which had a geographic reference. March 1st was chosen as thestart date because that is when a standardized data entry form was implemented.
The requests for service included 178 different case types. A subset of types reflected
examples of physical disorder arising from either human negligence or denigration of theneighborhood (e.g., illegal dumping, abandoned bicycle). Other case types either did not indicate
physical disorder (e.g., general request, bulk item pickup) or indicated deterioration that was not
the fault of local residents (e.g., street light outage).
Each case record included the date of the request, the address or intersection whereservices were to be rendered, as well as the case type. These locations came from a master
geographical database of the addresses and intersections of Boston based on the City’s tax
assessor and roads data, with each address keyed to the appropriate census geographies (from the
2005-2009 ACS, the most recent census with socioeconomic data when the database was built).
The main measures for this analysis were counts of events that occurred in a neighborhood,which we operationalize as the census block group (CBG). CBGs are smaller than the more
typically used census tract (avg. population ≈ 1,000 vs. 4,000), but the volume of CRM calls
enables measurement and analysis at this finer scale. Boston contains 543 CBGs with asubstantial population. All CRM reports document an event at a parcel or intersection, and theseare each attributed to a CBG. From this, neighborhood-level counts for all case types can becalculated.

	
  

8	
  

2.2. Defining Physical Disorder from Case Types
An initial examination of the 178 case types produced a list of 33 that might be evidenceof human neglect or denigration in public spaces (see Table 1). Counts were tabulated for each ofthese 33 case types for each CBG over the period covered by the database. As a first step toidentifying an underlying factor structure, an exploratory factor analysis was run on the 33 countvariables (Tabachnick and Fidell 2006). The final solution produced 5 factors with an eigenvalue> 1. These factors, whose constituent types and loadings are in Table 1, might be described as:•

Housing issues, including eleven items referring to poor maintenance by landlords(e.g., poor heating, chronic dampness) and the presence of pests (e.g., bedbugs).

•“Uncivil” use of space, including seven items that reflect how private actions cannegatively impact the public sphere (e.g., illegal rooming house, poor conditionsof property, and abandoned building).

•

Big buildings complaints, including three different case types regarding problems
with the upkeep of big buildings, like condos.

•

Graffiti, including two different case types regarding graffiti, one generated byconstituents, the other by the Public Works Department.

•

Trash, including five items related to incivilities regarding trash disposal: illegal
dumping, improper storage of trash barrels, empty litter basket, abandoned
bicycle, and rodent activity. This last is not itself an incivility, but is aconsequence of poor trash storage.

Five items did not load on any factor and were discarded before the foregoing analyses.
Four other items that loaded at <.4, though, were maintained based on conceptual similarity:
abandoned buildings loaded at .36 on the factor of uncivil use; requests to empty a litter basket

	
  

9	
  

loaded >.3 on both trash and graffiti, and was maintained on the former factor based on its
substantive content; two items were added to the housing factor as they were conceptually
identical to the definition of the factor and likely did not load in the factor analysis because of
their low frequency.
2.3 Exploring the Dimensions of Physical Disorder
New measures were created from these five factors in order to evaluate their higher-orderfactor structure. We accomplished this by summing counts for each of the constituent case types
for each CBG over the period covered by the database. These measures had substantial outliers
and were all log-transformed before analysis. Correlations between them were all significant(except uncivil use and graffiti), although modest if they are considered to be manifestations of asingle super-construct (see Table 2); only two bivariate correlations were above r = .4 (housing
and uncivil use; graffiti and trash), and two others were above r = .3 (housing and big buildings).
Given both content and the pattern of correlations, the five factors appear to suggest two maingroupings: denigration of the public space, comprised of trash and graffiti; and poor care or
negligence for private space, comprised of big buildings, housing, and uncivil use.
Confirmatory factor analysis, via structural equation modeling, was used to compare this
two-factor structure to a one-factor structure in which all five measures loaded together on an
overarching measure of physical disorder. The two-factor model was superior by all measures. It
had better fit (CFI = .82 vs. .61; SRMR = .07 vs. .10; Δχ2df=1 = 89.27, p < .001), and accounted
for 42% of the variation across factors, as opposed to 26%. The model estimated the correlation
between the two factors at r = .38 (p < .001). One will note that although the two-factor model
was stronger, it still had a poor fit. Because the hypothesis in question was the efficacy of a oneor two-factor model, there were no assumptions that the components of each were completely

	
  

10	
  

independent. Thus, we took the exploratory step of examining modification indices, leading to
the addition of a covariance between uncivil use and trash was added to the model, greatly
improving fit (CFI = .95, SRMR = .05, χ2df=5 = 24.26, p < .001). The final parameter estimates
for this model are presented in Figure 1.
Analysis 1 thus suggests that the CRM database is at least in principle capable ofmeasuring two distinct but related aspects of physical disorder: private neglect and publicdenigration. This result provides a more nuanced measurement than existing scales of physicaldisorder, particularly with the ability to go beyond elements visible from the street, and to accessconditions inside of buildings. Many previous protocols for measuring disorder have combined
items from each of these categories (for example, abandoned or deteriorating housing withgraffiti), and thus it is not surprising that the two constructs are correlated. It may also explain
why previous longitudinal work has found that such items become uncoupled across time
(Taylor, 2001). It is important to note that correlational constructs of this sort reflect a sharedprocess, but it is not clear what this process actually is. It is possible, for example, that housing
issues and uncivil use of private space are generated by the same behavioral tendencies, but it isequally feasible that one of these causes the other, or even that they are mutually reinforcing.
These are questions that go beyond the scope of our paper and thus are ripe for future study. For
present purposes, the reliable co-occurrence of these elements across neighborhoods provides
two different sets of measures we subject to an ecometric analysis: two of a generalized sort,private neglect and public denigration; and five lower level categories that are more specific,
housing, uncivil use of space, big buildings, graffiti, and trash.

	
  

11	
  

3. ANALYSIS 2: VALIDITY AND BIAS IN ADMINISTRATIVE DATA
Although it is tempting to treat the CRM database as the “eyes and ears of the city,” and therebya direct reflection of neighborhood conditions across space and time, its accuracy in theseregards cannot be assumed because each case in the database is in fact the coincidence of two
events: the issue itself; and the decision of a resident or passer-by to report it. This fact suggeststhat assumptions must be imposed to analyze the data. In guiding this process we invoke asimple behavioral model for the distribution of calls defined not only by the probability of anissue in a given space (P1), but also the probability that it will be reported (P2).1 If P2 varies
across neighborhoods, it could in turn create systematic biases in any measure based on the CRM
system. For example, in regions where residents are not inclined to make such calls, an issue
might sit unnoted for a lengthy period, or even indefinitely, creating a gap or false negative in the
database. Conversely, the residents of some neighborhoods might be notable in their vigilance,
generating multiple reports for a single issue, leading to false positives that exaggerate the actual
prevalence of disorder. This variation in P2 might be referred to as the civic response rate, which
we thus account for in order to establish validity for the measures identified in Analysis 1.
In pursuing this goal, we look to develop a methodology that accounts for the local civic
response rate, producing final measures that more accurately reflect neighborhood conditions.
We focus particularly on issues in the public domain, like street light outages, as these are likely
to be the most vulnerable to such biases being that the responsibility for reporting them belongs
to no specific individual, but to the neighborhood as a whole. Developing this methodology
entails three steps, utilizing data from the CRM system and a series of neighborhood audits.
First, there must be an independent or “objective” measure of response rate that captures the
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
1
We thank an anonymous reviewer for spurring our thoughts on this issue.

	
  

12	
  

propensity of a neighborhood’s residents or visitors to report a given issue. We use two suchmeasures, one identifying street light outages, and the other evaluating sidewalk quality. Second,
it is necessary to create a measure of civic response rate that is based on measures from within
the CRM system. This is critical because such a measure would allow the continual estimation of
response rate, and in turn the production of valid measures of disorder, in lieu of regularneighborhood audits. In the next subsection we develop the theoretical basis for how particularpatterns in the CRM database might be reflective of civic response rate. By examining themultivariate relationships between these internal measures and the objective measures of
response rate, it is possible to construct a new measure from within the CRM system that can beused as an adjustment factor.
Third and last, we develop an equation that combines counts of cases with the adjustment
factor to calculate final measures of physical disorder. This requires a measure of objective
physical disorder, against which it is possible to calibrate the adjustment factor, determining how
heavy its influence should be. This is done through an additional neighborhood audit thatassessed loose litter on streets and sidewalks, an item that has been central to measures ofphysical disorder. In sum, this process produces a complete methodology for translating a raw
database of CRM calls into a measure of physical disorder across a city. To conclude, we
examine the construct validity of the measures produced by this methodology, comparing it to aseries of other demographic, economic, and social indicators traditionally associated withdisorder.
3.1. Sources of the Civic Response Rate
Reporting rates in the CRM database for public issues can be seen as having two distinct
elements. The first entails knowledge of the CRM system and willingness to use it. The second is

	
  

13	
  

a decision to take action or responsibility for the public space. To the former, a large part of thebattle for any public service agency is informing residents of available services, and making
them comfortable with utilizing them. The CRM system also requires direct interaction between
constituents and the government, something that those from disadvantaged or minority groups
are sometimes less inclined towards, either because they distrust the government in general, or
because they do not expect the requested services to actually be delivered (Putnam 1993; Verbaet al. 1995). The sum of these effects might be described as engagement, or the likelihood that aperson would use the CRM system in any case. Given the evidence that such patterns cluster
demographically, it is likely to vary across neighborhoods, potentially contributing tomeasurement bias.
Knowing of and being willing to use the CRM system is not sufficient for using it toreport a public issue, however. When calling in a report about something like graffiti or illegal
dumping, one is taking responsibility for the public space, something that might have a different
set of motivations than a call addressing one’s personal needs (e.g., a request for a bulk item
pick-up). There are a number of mechanisms that might cause this concern for public space tovary systematically across neighborhoods. First, such variation could be owed to differences in
the cognitive perception of disorder. One striking finding of city-wide neighborhood surveys isthat resident ratings of local disorder vary within the same neighborhood and only moderately
correlate with observational (e.g., video or research rating) measures (Franzini et al. 2008;
Sampson and Raudenbush 2004; Taylor 2001). This would indicate that individuals andcommunities vary in their definition of “disorder,” something that might play an important rolein how likely they are to feel compelled to report such issues. At the same time, it by definition
reveals that surveys reports are not “objective” measures of disorder either.

	
  

14	
  

Another factor could be the variation across individuals in the level of personal
responsibility they feel for the public space. For example, homeowners tend to be more engaged
with public maintenance (O'Brien 2012), likely because of the long-term investment they havemade by purchasing a house (Fischel 2005). Consistent with this, our preliminary analysis of theCRM data indicated that homeowners are four times more likely to report public issues thanrenters (O'Brien 2013). A third possibility is that the accumulation of physical disorder might
incline residents to see the act of reporting new issues as useless, as it will be unable to overcome
the consistent generation of such problems (Ross et al. 2001). The truth may involve any one ofthese mechanisms, or some combination thereof, but the point stands that concern for public
space could contribute to cross-neighborhood variations in the rate of reporting actual instancesof physical disorder.
There are two features of the CRM database that will prove useful in the development of
measures that reflect engagement and concern for public space. As noted, CRM case records
indicate the type of services requested. From these, there is a subset that indicates issues in the
public space. This subset overlaps with, but is not equivalent to, the subset of case types
regarding physical disorder. Second, users of the CRM system are able to register, creating an
account for tracking their reports.2 Reports made by a registered user are then attributed to theindividual’s account using an anonymous code, making it possible to determine how often an
individual uses the system, and to approximate the individual’s home location. Though thisignores those who have used the system but not established an account, this information still
provides insights on an individual’s calling patterns that we would not otherwise have.
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
2
This procedure is encouraged by the directors of the CRM system, who see followingup with constituents as central to their goal of establishing open communication between citizens
and the government.
	
  

15	
  

The most direct way to measure engagement would be to tabulate the number ofindividuals who do and do not know about the CRM system. This can be approximated as the
proportion of neighborhood residents who have an account with the CRM system. A less direct
approach would be to identify case types whose need might be even across the city, that is, forwhich P1 would be constant across neighborhoods. In these cases, measuring their geographic
distribution would then provide access to P2, the likelihood of utilizing the system. For example,
one might expect the need for general requests, which entail questions about city services andother government-related items, to be driven solely by interest and engagement with government.
Another example would be requests for sanitation services to pick up bulk items. It is reasonable
to assume that residents of all neighborhoods have a similar need for this service as it is notdetermined by external, neighborhood processes. A third example of an evenly distributed issue
is the need for snow plows during a snowstorm. During a snowstorm, all neighborhoods shouldhave a roughly equal need for snow plows, controlling for certain infrastructural characteristics(e.g., the total road length, dead ends). We then have four candidate measures of aneighborhood’s engagement: total registered users, general requests, bulk item pick-ups, andsnow plow requests.
Measuring concern for public space requires a focus on reports that document a case of
public deterioration, and, in turn, a constituent’s decision to take action regarding it. Thisrequires a list of case types that indicate a public issue. It is not possible to use any one of thesetypes as a benchmark, as done with general requests, bulk item pick-ups and snow plow requests,
because the very issue at hand is whether public issues are uniformly distributed across the city.
Instead, we focus on the other two techniques described for engagement. First, it is possible toidentify a subset of users who have made one or more reports of a public issue. This could be

	
  

16	
  

used to tabulate the number of individuals in each neighborhood that have used the CRM system
for such a purpose. Additionally, some of these “public reporters” make a disproportionate
number of reports. Given their zeal for neighborhood maintenance, these individuals might bereferred to as “exemplars.” Public issues in a neighborhood with either a greater number of
average or exemplar public reporters would be expected to instigate reports to the CRM system
more often and more quickly. Second, it is possible to measure the proportion of reports ofpublic issues that were made by registered users. This would indicate how consistently such calls
are part of a sustained relationship between a resident and government services. This amounts tothree measures of concern for public space: public reporters, exemplars, and proportion of calls
made by registered users. Importantly, none of these measures is fully independent ofengagement itself. For example, regardless of one’s inclination to report a street light outage, heor she must first know that the CRM system exists. Consequently, the following analysis allows
these measures to load on one or both of these constructs.
3.2. Estimating Civic Response Rate from the CRM Database
In order to be concurrent with the neighborhood audits (described below), the current
analysis uses only CRM reports from 2011, amounting to 161,703 cases with geographic
reference across 154 case types. This analysis incorporates two new ways of utilizing the CRM
database. First, similar to the identification of case types reflecting physical disorder in Analysis
1, 59 case types were identified as reflecting issues in the public space (e.g., street light outage,
pothole, graffiti removal; complete list in Appendix A). Such a report indicates a concern for the
maintenance of the public space on the part of the reporter. Other case types reflected personalneeds rather than public concerns (e.g., general request, bulk item pickup). Second, allindividuals who have registered with the CRM system have an anonymous ID code that is

	
  

17	
  

appended to each of their reports. In 2011, there were 29,439 constituent users, accounting for
38% of all requests for service.3 The ID code makes it possible to construct a database of users
with variables describing each individual’s pattern of reporting across time and space. This twopart database of calls and users was used to calculate the measures hypothesized to reflect a
CBG’s civic response rate.4
The call database was used to measure four of the seven proposed measures. Bulk item
pick-ups (bulk items) and general requests were measured as the number of such requests
occurring within a CBG. Proportion of public issues reported by registered users was measured
as the number of public issues reported in a CBG attributed to a registered user divided by the
total number of public issues reported in the CBG. Snow plow requests were first tabulated as acount for each CBG, but were then adjusted for the total population, road length, and the lengthof dead end roads.5
The other three measures were calculated from the database of registered users which
included three main pieces of information: 1) the total number of calls a user had made; 2) the
total number of calls a user had made regarding a public issue; 3) an estimate of the user’s homelocation, based on the locations at which they requested services.6 In 2011, 46% of registered
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
3
Users who made one or more reports as a department member at any time (including
2010 or 2012), were removed because city employees differ from other constituents in theirmotivation for making reports. This excluded five individuals, a number that is low because formany employee-specific case types, user ID’s were stripped before data sharing.
4
Two CBGs were excluded from analysis as there were concerns that calls from there
might not reflect usage of the CRM system by actual residents: 1) the CBG that contains City
Hall, because many reports without an address are attributed to that location; and 2) the CBG
that contains a large park, zoo and golf course, but includes the houses that ring the park.
5
The number of snow plow requests (log-transformed to adjust for a skewed distribution)
was regressed upon these three measures, accounting for 14% of the variation across CBGs.
6
The home location was estimated in two ways, depending on the geographic range of anindividual’s requests for service. If the individual reported cases over a range with diametersmaller than .5 miles (90% of users), location was defined as the centroid of all reports made,
	
  

18	
  

users were public reporters. Of these, 87% reported two or fewer public issues, though there
were those who were considerably more active (18 made over 100 reports). Given this
distribution, total users were measured as the number of registered users whose estimatedlocation fell within the CBG; average public reporters were measured as the number of a CBG’s
total users who had reported one or two public issues during the year of 2011; and exemplars
were measured as those who had reported 3 or more public issues.
3.3. Objective Measures from Neighborhood Audits
Objective neighborhood conditions were assessed through two separate audits. One
identified street light outages and the level of street garbage in 72 of Boston’s 156 census tracts(46%) between June 1 and August 31, 2011. In total, 4,239 street segments were assessed, and244 street light outages were identified. Street light outages were attributed to the nearest
address. Garbage was rated for each street block on a 5-point scale, with higher scores indicating
more and larger piles of garbage. More detail on this protocol is provided in Appendix B.
In the second audit a consulting group hired by the City of Boston’s Public Works Dept.
assessed the quality of all of the city’s sidewalks between November, 2009 and April, 2012. The
unit of analysis was each continuous stretch of sidewalk that ran from intersection to intersection(N = 27,388). For each sidewalk, the assessors noted the proportion of panels that required

	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
which was then attributed to the appropriate CBG. Because of the small range, this estimate canbe assumed to be reasonably precise. For those whose range had a diameter greater than .5 miles,
this precision was weaker. These individuals were attributed not to a centroid, but to the census
block group from which they made the most calls. This was done using the entire period of thedatabase (March 2010 – June 2012) in order to make the greatest use of available information.
This estimation technique was validated against a sample of 7,433 users for whom homelocations were known. Of these, 78% were attributed to the correct CBG. More importantly, thecounts generated by this process correlated with actual counts at r = .93. There is reason tobelieve that this correlation is underestimated. The sample used in the validation had an above
average number of calls per person, a subsample for which the estimates had greater error.
	
  

19	
  

replacement (i.e., cracked, broken), and subtracted this from the total. This generated a 0-100
measure of sidewalk quality (100 being a sidewalk with no panels requiring replacement).
Street light outages and sidewalks were each cross-referenced with the CRM database to
identify reports regarding them. For street light outages, we sought to identify the date on which
each was reported. This was defined as the earliest case of an outage reported on the street
segment in question that was fixed by the city after the date an auditor noted the outage.7 This
was then used to create a series of dichotomous measures indicating whether the outage had beenreported by a constituent within a certain time window (e.g., one month).8 For sidewalks, allrequests for sidewalk repair were joined to the nearest sidewalk polygon from the same road. We
were able to exclude those created by City employees as an additional code was included withsuch cases. The count of constituent reports for every sidewalk was then tabulated. Of the 27,388
sidewalk polygons, 1,168 generated requests for repair (4%; min = 1, max = 19).
Because the three audits described events or conditions on a single street segment withina neighborhood, multilevel models were run to create CBG-level measures (Raudenbush et al.
2004). These models controlled for micro-spatial characteristics of the street (e.g., zoning), andthe second-level residuals were then used as CBG-level measures. The outcome measures forthese models were: the likelihood of a sidewalk generating one or more requests; the likelihoodof a street light outage being reported within one month; and the continuous 1-to-5 measure of
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
7
Note that this means a street light might have been reported before the audit, as long as
the city had not completed the job until after.
8
It was possible to distinguish whether a report was made by a constituent or a City
employee. Thus a continuous measure of the time before reporting would not necessarily reflect
the strength of constituent response. Instead, the dichotomous measures were created so that
employee-reported outages could be considered not-reported until the date the employee report
appeared. Thereafter they were omitted from the data, as it is not possible to know whether aconstituent would have reported up to that point. For example, a street light outage reported 16
days later by a City employee takes the value “0” for the measure of being reported within two
weeks, but would take no value (omitted) for the measure one month.
	
  

20	
  

garbage. See Appendix C for more detail on these models and the specification of outcome
measures.
Two deviations from this approach are important to note. First, the number of outages per
CBG was low for a multilevel model (244 outages in 127 CBGs), so the models were run insteadwith tracts as the second-level (N = 56 tracts with outages). Each CBG then took the measure forits containing tract. Second, because sampling for the garbage audit occurred at the tract level,
CBGs varied in the number of street segments that were rated. In order to be certain thatneighborhood-level measures were reliable, the ensuing analysis was limited to the 196 CBGs
with 10 or more street segment measures (see also Sampson and Raudenbush, 1999).
3.4. Evaluating the Proposed Model of Civic Response Rate
Descriptive statistics for both objective measures of response rate and CRM-based
measures proposed to estimate response rate are reported in Table 3, as well as the correlations
among them. All tabular variables had a skewed distribution with a long tail of CBGs that used
the system extensively, leading us to log-transform them before correlational and regression
analyses. As hypothesized, all variables indicating use of the system (general requests, bulk item
pick-ups, all users, users reporting public issues, exemplary reporters) were strongly correlated(r’s = .36 - .93, all p-values < .001). Because of the very high correlation between all users andaverage users reporting public issues (r = .93 in the full sample and r = .95 in the subsample
with values for all measures), the two were deemed to be the same measure. The “all users”
measure was thus dropped from all proceeding analyses to avoid issues of multicollinearity.
Requests for sidewalk repairs and propensity to report street lights were modestly
correlated (r = .18, p < .05). Each also shared stronger correlations with those measures from theCRM database intended to measure concern for the public space (public reporters, exemplars,

	
  

21	
  

percentage of public issues reported by registered users) than those intended to measureengagement (general requests, bulk item pick-ups, all users). The reverse was true for requests
for snow plows, as predicted. They were significantly positively correlated with the sidewalk
measure (r = .14, p < .05) but not the street light outage measure (r = -.12, p = ns) and had astronger correlation with measures of engagement than with concern for public space.
Structural equation modeling was used to determine how well the proposed constructs fit
the data. The model analyzed those 195 CBGs with a measure for propensity to report street
lights. The best-fitting model, depicted in Figure 2, had good fit (CFI = .95, SRMR = .06, χ2df=9 =25.44, p < .01), and was quite similar to the model proposed in the introduction to this analysis.
The measures derived from the CRM system did indeed separate into the two proposed latentconstructs, engagement and concern for public disorder. It is notable, however, that the two
objective measures of civic response rate loaded on the latent construct of concern for publicdisorder (sidewalks: β = .34, p < .001; street light outages: β = .18, p < .05), but not on
engagement.
As with the models in Analysis 1, the novelty of the various measures required that wetake a partially exploratory approach, tweaking the theoretically-based model to specify the best
fit. Consequently there were a few alterations that bear mentioning. 1) With the removal of themeasure of all users, it was necessary to have average reporters of public issues load on bothlatent constructs (engagement: β = .50, p < .001; concern for the public space: β = .52, p < .001).
2) The measure of general requests was also removed as its strong correlation with othervariables made the factor structure unstable. 3) The percentage of public calls from registeredusers was discarded as doing so strengthened the model’s fit. 4) Total population was used as acontrol variable predicting average reporters of public issues (β = .13, p < .05) and exemplars (β

	
  

22	
  

= .16, p < .01). Modification indices for the final model suggested that no significant bivariate
relationships had been omitted.
3.5. Evaluating the Adjustment Factor
The results of the previous model suggest that the estimate of the civic response rate, andtherefore the desired adjustment factor, is based on measures of concern for the public space. A
composite measure for each CBG was created using the parameter estimates from Figure 2. We
then established its efficacy as an adjustment factor by examining how well it improved therelationship between the raw measures from Analysis 1 and objective measures of physicaldisorder, as indicated by street garbage. The analysis was performed in two parts. First, the raw
counts of case types in each category (log-transformed to better approximate normality) wereentered into five separate regressions predicting the level of street garbage in a CBG. Second, an
adjustment factor was created for each count as an interaction with the civic response rate, which
was then added to the corresponding regression.9 This analysis was limited to residential
neighborhoods (excluding regions dominated by institutions, parks, or downtown areas), as thepredictive relationship between local behavior and loose litter would be most clear in these areas;in others, litter would be subject to dynamics that would not necessarily influence the othercomponents of physical disorder (e.g., graffiti) in the same way (N = 135 residential CBGs).
The first set of regressions found that all but one of the raw measures (graffiti)
significantly predicted levels of street garbage (complete details in Table 4). The strongest
relationships were with housing (B = .63, p < .001) and uncivil use of space (B = .38, p < .001).
Big buildings (B = .21, p < .05) and trash (B = .18, p < .05) had more moderate relationships.
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
9
The response rate was standardized and centered before the interaction was calculated.
The physical disorder measures were left uncentered, with a minimum of zero, so that the
response rate would adjust up or down proportional to the total number of actual reports.	
  
	
  

23	
  

The fit of all five regressions increased significantly with the introduction of the adjustment
factor (see Table 4), with the strongest improvement occurring for trash (Δ R2 =.06, p < .01) anduncivil use (Δ R2 =.05, p < .05). Notably, the variance explained more than doubled for bothtrash and graffiti, which had the weakest initial relationships with street garbage.
3.5. Construct Validity for the Composite Measures
As a last step, we evaluated the construct validity of these final measures by examining
their relationship with other popular indicators of neighborhood conditions, drawn from threedifferent data sources: median income, homeownership, and measures of ethnic composition
from the census’ American Community Survey (ACS; 2005-2009); survey measures ofperceived physical disorder and collective efficacy (i.e., social cohesion and social control
between neighbors) from the Boston Neighborhood Survey (BNS; 2008-2010 estimates, N =3,428);10 and reports of gun-related incidents from Boston’s 911 call record (2011). Because thetime points of these data sources vary, we analyze their relationship to the CRM-based measurefor the most concurrent year: 2010 for the ACS and BNS, and 2011 for 911. As before, we focus
the analysis on residential neighborhoods, but in this case we analyze at the broader spatial scaleof census tracts rather than block groups (N = 121 residential census tracts). We do so because
the interpretation of the analysis depends in important ways on comparison with findings fromprevious studies, particularly Raudenbush and Sampson (1999) and Sampson and Raudenbush
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
10
The BNS was a telephone survey based on the methodology from Raudenbush andSampson (1999) with 3,428 participants in two waves (2008: N = 1710; 2010: N = 1718)
recruited by random-digit dial. The two waves were combined to provide a reasonable number of
respondents to create measurements at the scale of CBGs. Scales measuring physical disorderand collective efficacy were calculated first for each individual respondent. Neighborhood-levelmeasures were then calculated by fitting multilevel models that nested individuals within their
CBG and controlled for individual-level demographic characteristics (gender, age, ethnicity, andparental status). The Bayes residuals for the neighborhood-level model were then extracted asneighborhood measures adjusted for measurement error.
	
  

24	
  

(1999), which were conducted on census tracts and clusters of tracts. In addition, because of thesmaller sample size of the BNS compared to the Chicago study, the BNS has greater betweenneighborhood reliability for tracts than for block groups. For the sake of brevity, we conducted
this analysis on the higher-order measures of private neglect and public denigration. Results forthe five lower-order measures as well as block groups are available upon request.
The measure of private neglect was lower where there was higher median income (r = .59, p < .001), homeownership (r = -.36, p < .001), and collective efficacy (r = -.38, p < .001),
and greater where there were greater black (r = .61, p < .001) and Hispanic populations (r = .27,
p < .001). It also co-occurred with gun-related incidents (r = .68, p < .001). Further, it was higher
where residents perceived more disorder (r = .44, p < .001). The measure of public denigration
had largely similar relationships with these measures—it was lower in areas with more
homeowners (r = -.49, p < .001) collective efficacy (r = -.48, p < .001), and a higher medianincome (r = -.39, p = .001). Public denigration was higher where there was a greater Hispanic
population (r = .41, p < .001) and more gun-related incidents (r = .27, p < .01). It was also higher
where residents perceived more disorder (r = .48, p < .001). The one unexpected correlation wasthat it held no correlation with the proportion of black residents (r = -.05, p = ns).
These validation correlations are lower than those reported in Raudenbush and Sampson(1999: 31) for survey-reported disorder. For example, public denigration correlates withperceived disorder at .48 in Boston but .71 in Chicago. However, at least four factors differ
between studies beside the method (observation vs. CRM for non-survey based indicators ofdisorder)—the items in the measure, city, reliability of the surveys, and time period—making
direct comparability difficult. We would note though, that the correlations for structuralcharacteristics are similar—for example, the correlation between physical neglect and income in

	
  

25	
  

Boston is -.59 and in Chicago the correlation of observed disorder with poverty is .64. And thecorrelations for residential stability are -.36 in Boston and -.25 in Chicago. Moreover, the CRM
correlations are on par with previous comparisons between perceived and objective disorder inother studies (Brown et al. 2004; Franzini et al. 2008; Sampson and Raudenbush 2004; Taylor
2001).
Overall the results suggest that it is possible to construct a measure from within the CRM
database that adjusts counts of case types to better reflect neighborhood conditions, though there
are differences between the two classes of physical disorder that should be noted. In particular,
private neglect had a stronger relationship with street garbage, with two of its constituent metrics(housing and uncivil use) surpassing the threshold of ~15% shared variance typically seen
between domains of physical disorder (Taylor 2001). The relationships between the indicators of
public denigration and street garbage were a bit weaker, but the correlations with other indicatorsof disorder were of similar magnitude, even stronger in cases. This could be owed to one of two
possible explanations. The first is that issues of trash storage and graffiti are in fact less linked topatterns of litter than expected. The second is that these issues are more susceptible to reporter
bias and potentially in ways that audits of natural patterns in deterioration, like street lightoutages and sidewalk cracks, might not fully capture. The same norms that lead to garbage-laden
streets might also be responsible for diminished motivation to report graffiti or other issues in the
public space. If so then the assumption that a neighborhood’s civic response rate, P2, is consistent
for a given neighborhood across all case types is called into question. Future validation efforts
should carefully evaluate the most effective measures both for objective comparison and internal
adjustment, as these might differ depending on the particular set of conditions that are intended
to be the focus, a theme we return to below.

	
  

26	
  

4. ANALYSIS 3: ASSESSING RELIABILITY ACROSS SPACE AND TIME
Analyses 1 and 2 have provided a methodology for measuring physical disorder using the CRM
system, but without a guideline for how such measures should be bounded in space and time.
Thus far, measures have been developed for CBGs over the entire available time course (twoyears and four months). It is desirable, however, to assess measures for smaller time windows,
allowing researchers to examine local conditions at more precise intervals, and facilitating
longitudinal analysis. In addition to CBGs, it would be appropriate to determine the optimal time
window for census tracts, the unit at which most urban research is conducted.
Determining an “optimal” time window for measurement requires a balance of twocontrasting dimensions: smaller time windows are more precise, but are more sensitive torandom events. To do this, we must examine how consistent the multiple measures of a single
neighborhood are for different time intervals (using the intraclass correlation, or ICC), and theability to statistically distinguish between neighborhoods (using the reliability coefficient, or λ);these characteristics can be assessed using multilevel models. The goal is to identify the smallest
time interval for which measures within a neighborhood are sufficiently consistent, and not
overly sensitive to error or stochastic processes. Because the measures of interest are in factcomposites that combine counts of cases with the measures of concern for the public space, theestablishment of reliability requires two steps. First, we must identify a time interval for which
all of the constituent measures (e.g., instances of housing issues) attain a desired threshold forreliability and ICC. Once the appropriate time interval for the constituent measures isdetermined, it must be confirmed that the same time interval is appropriate for the composite
measure. One will note, however, that step one is not possible for the measure of exemplar
reporters, as they are defined by their behavior over the course of a complete year. For this

	
  

27	
  

reason, exemplars will always be calculated as the number of public reporters in a region
obtaining exemplar status over the previous 365 days.
The last question we seek to answer is that of longitudinal tracking. If the final timeintervals are small enough, it would be possible to examine patterns of change across time. The
multilevel models can assess the slope for a measure at both the global and neighborhood levels.
If the reliability for the slope is high enough, the model is capable of discerning varying
trajectories across neighborhoods, which could then be used in subsequent analyses.
4.1. Creating Measures for Spatio-Temporal Windows
The complete CRM database, including all requests for service received between March
1, 2010 and June 29, 2012, is the basis for the temporal analysis. All requests are categorized bycase type and include the date of the request and the address or intersection where services were
to be rendered, allowing all requests to be geocoded to the appropriate census geographies.
The focal variables are those that constitute the composite measures of physical disorder,
including both the raw counts of cases that reflect the five categories of physical disorder, andthe measures of response rate. Drawing from Analysis 1, the five categories of physical disorderwere housing, uncivil use of space, big buildings, graffiti and trash. Based on Analysis 2,
response rate was calculated as the number of individuals reporting public issues, divided into
two counts: those who made two or fewer calls in a year’s time, and those who made three ormore calls in a year’s time (i.e., exemplars).
Measures for each variable, excepting exemplars, were created for all CBGs and tracts
for eight temporal windows—1, 2, and 3 weeks, and 1, 2, 3, 4, and 6 months. For each, theoriginal database was split into intervals of the given size, starting with March 1, 2010 and

	
  

28	
  

ending with the last complete interval. A count was then produced for each interval for eachelement in the given level of analysis (i.e., block group or tract).11
4.2. Multilevel Models
Hierarchical Linear Modeling (Raudenbush et al. 2004) was used to compare theconsistency of counts within a CBG over time. A natural-log link was used to account for the
Poisson distributions of all outcome variables. The first level equation predicted the outcome fora given time point relative to other measures for that region, and included: the number of timeintervals elapsed since the start of the database, in order to estimate the rate and direction ofchange over time; dummy variables controlling for seasonal effects, based on the month of the
midpoint of the given time-interval:
𝑌!" =    𝛽!! + 𝛽!! ∗ 𝑡𝑖𝑚𝑒!" + 𝛽!! ∗ 𝑠𝑒𝑎𝑠𝑜𝑛!! + 𝑟!"
𝑟!" ~𝑁(0, 𝜎 ! )The second level equation was an intercepts-only model, estimating the average level of ameasure for a neighborhood across time. In addition, the parameter relating time to changes in ameasure, β1, was allowed to vary across CBGs, permitting the model to estimate different
trajectories of change for different CBGs:
𝛽!! =    𝛾!! + 𝜇!!
𝛽!! =    𝛾!" + 𝜇!!
𝜇!! ~𝑁(0, 𝜏! )𝜇!! ~𝑁(0, 𝜏! ),

	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
11
For example, there were 28 1-month time windows between 3/10 and 6/12, generatingthat many counts for each CBG. The resultant dataset then contained 15,176 counts (28*542),
each attributed to a CBG and a time window.	
  
	
  

29	
  

where τ0 is the measure of variation in the outcome measure between CBGs and τ1 is a measureof variation between CBGs in the linear relationship between time and the outcome variable.
Furthermore, σ2 is a measure of the variation in the outcome measure within CBGs (i.e.,
differences within a CBG across time).
The intraclass correlation coefficient (ICC) is then calculated as the proportion ofvariation that lies between groups:
𝜏!
!
𝜎 +𝜏

!

𝜏!
!
𝜏 +𝜎

𝑛

𝐼𝐶𝐶 =   Reliability is calculated as:
𝜆! =   !

where n is the number of observations per CBG. As one can see, this measure grows both with agreater ICC, but also with more observations.
Variation across CBGs in the linear relationship between time and the outcome measure
is assessed in two ways. First, the significance of the magnitude of τ1 is assessed using a χ2 test.
Second, its reliability is measured as:
𝜆! =   

𝜏!
𝜏! +

𝜎!

𝑆𝑆!"#$

where SSTime is the sums of squares for the measure of time.
4.3. Comparing Spatio-Temporal Windows
The reliabilities and ICCs from the multilevel models described above are reported inTables 5 (CBGs) and 6 (tracts). As expected, the proportion of variation attributable todifferences between both CBGs and tracts (measured by the ICC) increased monotonically astime windows became larger, owing both to greater consistency and fewer measures perneighborhood. As would also be expected, ICCs were higher when comparing tracts than CBGs.
	
  

30	
  

The six measures varied in their consistency, with housing, graffiti and trash holding thestrongest reliabilities and ICC’s among the different measures of physical disorder. These
differences seem largely attributable to the frequency of these categories. For example, there
were three times as many events reflecting housing issues than uncivil use of space. With a lower
frequency, counts of the latter will be more stochastic and therefore less consistent at smaller
time intervals. Interestingly, counts of public reporters, though far fewer in number than actual
calls, featured greater consistency within a region than any of the measures of physical disorder.
All ICCs in Tables 5 and 6 were significant at p < .001. The intent here, however, is not
to find significant between-region variation, but to identify spatio-temporal windows at which asingle measure is indicative of a region’s “actual” value on that measure. The ICC, in that case,is used as an evaluation of how strongly a single measure of a neighborhood correlates with allother measures of that neighborhood. If we elect .7 as a threshold for a reliable neighborhoodlevel measure, then there are acceptable spatio-temporal windows available for all of themeasures apart from big buildings. For those measures with greater consistency, the options aremany: housing, for example, could be measured at two-month intervals for tracts or four-monthintervals for CBGs. For others, like uncivil use, there is a need for six-month intervals at the tractlevel, and no time interval satisfies this criterion for CBGs.
The slope reliabilities in Tables 5 and 6 indicate the ability of the model to distinguishbetween the trajectories of different regions over time. Variation in slopes across CBGs andtracts were significant at p < .05 (or some lower threshold) in nearly all models, excepting all
those for big buildings, and those for public reporters of intervals longer than four months long.
This variation was somewhat more discernible in tract-level models.

	
  

31	
  

We then examined whether these cross-time consistencies hold for the composite
measures. For the sake of simplicity, this was done for all variables using six-month windows for
tracts. One will note that the generation of the composite measures requires the incorporation of
the number of exemplars, measured for the full year preceding the last day of the given time
window. Consequently, the first time window analyzed must be that which ends at or after theend of the 12th month of the available database, diminishing the number of measurements per
tract. For this reason, this analysis does not examine change over time.
Similar to above, multilevel models were run to examine the consistency of thecomposite measures across space and time. The reliabilities and ICC’s from these are reported in
the bottom row of Table 6. Across the board, reliabilities and ICC’s were lower for thecomposite measures, but not alarmingly so. All measures (other than big buildings) maintained
ICC’s around .6 or higher, and housing had an ICC greater than .7. Reliabilities were typicallyaround .8.
Last, we replicated the analysis for the two higher-order constructs, private neglect andpublic denigration. The statistical advantage is that the combination of multiple measuresamplifies the number of cases in the average time interval, thereby enabling higher reliabilities
and ICC’s at smaller time windows. This is particularly important when considering a measure
like big buildings. Though it has low reliability when measured on its own, it might beincorporated into a more comprehensive description of the neighborhood in this manner.
Reliabilities and ICC’s for these higher-order counts were higher than their constituent
categories. For each, the criterion of ICC = .7 was attained at six-month intervals for CBGs andtwo-month intervals for tracts (complete results available on request). This remained largely
consistent when they were combined with measures of concern for the public space to create

	
  

32	
  

composite measures, though the consistency in public denigration for small time windows was
somewhat diminished. For tracts with two-month intervals, public denigration had an ICC = .44,
and a reliability coefficient of .88. Private neglect had an ICC = .65 and a reliability coefficientof .94. For CBGs with six-month intervals, public denigration had an ICC = .51, and a reliability
coefficient of .76. Private neglect had an ICC = .68 and a reliability coefficient of .87.

5. SUMMARY AND IMPLICATIONS
The current study sought to demonstrate how a citizen-initiated administrative database might act
as “the eyes and ears of the city” in the spirit of Jane Jacobs (1961) while at the same time
providing a low-cost, real-time measure of physical disorder. To accomplish this goal we neededto address three major issues: the lack of interpretable constructs; the potential that the databasemight not objectively or accurately reflect real-world conditions; and the need for criteria forreliability when bounding measures in space and time. Creating a set of theoretically-guided
factors first required an item-response model, in this case 28 case types that reflected
deterioration or incivilities within a neighborhood. The subsequent factor analysis revealed five
separate categories of physical disorder. It is worth noting that these constructs were extant in thedata, but that it was necessary to distinguish them from the noise surrounding them. Skipping
forward to Analysis 3, once these measures were fully developed, criteria for reliability were
established both for one-time measures and cross-time trajectories using multilevel modeling.
In between these two steps, Analysis 2 addressed the question of validity, which is aperhaps underappreciated concern for ecometric study. Neighborhood audit protocols aredeveloped and administered to measure specific things as accurately as possible, meaning they
have an inherent validity for those items that they assess. In contrast, administrative data are the

	
  

33	
  

byproduct of processes whose idiosyncrasies might bias their reflection of ground-truth. The
CRM database is the product of constituent reports, and therefore is vulnerable to inconsistencies
in reporting across neighborhoods. Because the nature of the bias was known, however, it waspossible to account for it. The final methodology used indicators of civic response rate, derived
from the CRM database itself, to systematically adjust raw measures to better reflect objective
conditions. Reaching this point entailed considerable work, including two independent datacollections and a lengthy set of analyses. Nonetheless, that investment of cost and effort would
be necessary for any traditional protocol for measuring disorder, and in our case laid thegroundwork for a methodology that can be reproduced at little cost both within Boston acrosstime, and in other cities with their own CRM systems. It is also worth noting that cities
frequently conduct audit studies, so it is reasonable to assume that there will be an ongoingstream of potential sources of data from which to derive validation measures.
The final product was a multidimensional measure of physical disorder that is not only
nearly costless to the researcher, but also more comprehensive and precise than other measurescurrently available. Further, the programming code published along with this manuscript
facilitates reproduction of the measure wherever similar databases exist. Given these apparent
upsides to the use of administrative data, it seems appropriate to forward a new, three-stepprocess for carrying ecometrics into the age of big data:
1)

Extract constructs by identifying item-specific models that are reflective of thetheoretical concept of interest, and then examining their underlying factor structure2)

Validate the measure by identifying and adjusting for any bias that the

information source might impart to the data, and examining in conjunction with external
data.

	
  

34	
  

3)

Establish reliability in the measure’s ability to track information across space andtime.
With this methodology in hand, the opportunity before ecometric urban science isconsiderable, as there is a veritable trove of information on cities that sits largely untapped, of
which the CRM database is but one example. Cities collect and now make available many otherdata points—such as tax assessments, building permits, zoning decisions, restaurant inspections,environmental assessments, housing code violations, pedestrian flows, and bicycle collisions, toname a few—each providing their own insights on the social and physical ecology ofneighborhoods. Going further, there are private databases, such as Twitter, cell phone records,
and Flickr photo collections that are also geo-coded and might be equally informative in building
innovative measures of urban social processes. . These various resources could be used todevelop new versions of traditionally popular measures, like we have done here, or to explore
new ones that have not been previously accessible. An illustration of the latter comes from ourown analysis, where a byproduct of validation has provided two unanticipated behavioral
measures—one related to civic engagement and the other capturing attitudes towards disorder inthe public space. The potential of new forms of large-scale data underscores the central
inspiration of this manuscript: as the volume of data on urban areas continues to grow anddiversify, they provide new and distinctive ways to measure neighborhood characteristics, often
in ways not previously foreseen. Such advances can be appropriated to shed light on some of themost salient themes in urban science, from the structure and function of the social organization,
to the role of cognition and culture in generating local patterns, to the nascent examination ofrelationships between neighborhoods and the higher-order social structure of the city.

	
  

35	
  

Apart from its implications for ecometric science more broadly, the current methodology
represents an advance for the direct measurement of physical disorder in urban neighborhoods. It
incorporates a broad range of phenomena and is the first physical disorder measure to divide
these items into independent subcategories, suggesting new avenues for research. For example,
do the five subcategories relate differently to a neighborhood’s other social and demographic
characteristics? If so, do they each reflect a different set of processes occurring within theneighborhood? Further, what is the source of the higher-order constructs suggested by Analysis
1, private neglect and public denigration? Is it that their constituent types are all manifestationsof the same social and behavioral patterns, or do they share other causal relationships thatreinforce their correlation? It is crucial that we not over interpret this single case andinappropriately reify these particular constructs. It will be necessary to confirm their consistency
with data from other time points and cities, something that is likely to be possible with thecontinued proliferation of CRM systems throughout North America and Western Europe.
In addition, the measures enable a variety of analytical approaches that could prove
useful in the extension of research surrounding “broken windows” and other theories ofneighborhood well-being. All of the measures describe neighborhood conditions at the level ofcensus tracts, and some can be used for CBGs. Future work could likely find ways to measureand interpret patterns of disorder for streets or even individual buildings. The measures can also
be tracked across time, allowing for analyses that evaluate not only what a neighborhood’s
current level of physical disorder is, but whether it is on an upwards or downward trajectory.
Finally, the CRM data are continuously generated as part of administrative operations. A
new study with up-to-date data requires only a download and some data manipulation. In aneffort to assist others in initiating such work, we will be publishing the computer code for

	
  

36	
  

constructing the measures developed in the current manuscript (along with the data). As CRM
systems become more numerous around the world, typically in the form of 311 Hotlines, this sortof measurement is becoming possible in a variety of cities. Some of these cities have establishedcommon standards for publishing CRM data, meaning that the data are not only being madereadily available, but are compatible in ways that would support cross-city comparisons.

6. BALANCING LIMITATIONS AND OPPORTUNITIESWe have thus far focused predominantly on the opportunity presented by “big data,” but we mustalso take stock of the limitations that they carry and the challenges to be addressed. Indeed, themethodology presented here is only a first step—an illustration of what is possible—and future
work will need to refine it further, particularly in terms of the validation process. We would
likewise stress that traditional, well-established methods of urban data collection, such ascommunity surveys and social observation (Sampson 2012), will continue to play an essential
role in any future analysis. Claims to the contrary are merely “big data hubris” as aptly put by
Lazer et al. (2014). Each approach has its pros and cons, the balancing of which will depend on
the research question. Surveys and observation are expensive and cannot realistically be carriedout in real time, for example, but they can be calibrated to be representative of the population.
By contrast, the CRM data analyzed here are cheap and in principle can be measured at veryfine-grained geographic scales and almost in real time, but issues of reliability and what the data
are really measuring remain.
For example, complaints about big buildings were not particularly common in our
database, making it difficult to measure that construct reliably. Techniques that aggregate cases
at higher levels, by increasing the geographical range, the temporal window, or, as we did here,

	
  

37	
  

combining multiple related constructs, will be critical. Future research is needed to examine
these issues, especially in a context that can directly compare and contrast different
methodologies of data collection, such as systematic social observation. Perhaps more
important, although our validation process is promising we still cannot be entirely certain that wehave directly accessed the intended information, especially for those things that occur within
private spaces or out of the public observation. In particular, our measures of public denigration
were not as closely correlated with other indicators of urban social structure as might be
expected from past research. It may be, as we noted earlier, that the techniques for measuring
and accounting for bias in this case were not sufficient to fully calibrate public denigration.
Another potential weakness is our working assumption that reporting bias is consistentacross case types. Our data here seem to suggest that the situation is more nuanced, as reporting
rates for street light outages and broken sidewalks were only moderately correlated. This finding
points to important improvements for future versions of the measure, while also highlighting theneed to tailor the validation process to the specific measure of interest. In some cases, like that
presented here, there is a need to adjust for biases inherent in the data, and the objective
measures necessary for doing so will need to be carefully constructed and measured.
In other cases, however, such a process of construct validation or bias adjustment may beless necessary or not applicable even though reliability assessment by temporal and geographic
scale remains at issue. For example, building permits and zoning approvals or variances arelegal requirements for major building renovations and additions, meaning they should be largely
objective in the information they provide. Allocation of city resources (e.g., beautification
efforts or economic development) or distribution of the city budgets by amount and location are
also largely “bias free” in their measurement and now widely available electronically. The

	
  

38	
  

availability of such measures could provide new insight into processes such as gentrification andinequality in the delivery of city services (see e.g., Hwang and Sampson 2014).
Furthermore, in certain cases, the raw contents of the data are exactly what a researcher
wants, and their face validity is sufficient to offer concrete interpretation. In such situations,researchers do not actually want to adjust for any biases. For example, a recent paper used noise
complaints from New York City’s 311 hotline as a direct reflection of social conflict between
neighbors (Legewie 2014). Regardless of actual noise levels or norms of reactivity, each call inthis analysis reflects an objective case of one neighbor asking the government to regulate thebehavior of another. More generally, the electronic availability in many cities of citizen
reporting systems offers a wide variety of domains (in our Boston data, 178 unique types ofservice calls) to test Black’s (1976) theory of the behavior of law and citizen initiation of
government control.
Of course, the fundamental issues of measurement error and validity bear down at somelevel on all methodologies: survey reports can be skewed by other perceptual factors, including
implicit judgments of race and class (Sampson and Raudenbush 2004), and observational work is
dependent on inter-rater reliability that is always less than unity. The dominant investigatordriven research method is to conduct surveys or interviews, but even here there is continuing
controversy over the idea that researcher control leads to validity. For example, a recent critiqueargues that interviews are a weak basis for studying culture or inferring the motives for anindividual’s behaviors (Jerolmack and Khan 2014, and responses). While we would not go that
far, our point is that assumptions must always be invoked in the analysis of social science data.
It remains significant, however, that administrative data are outside of a researcher’s
direct control and that assumptions may sometimes be required that are uncheckable because of

	
  

39	
  

unobserved processes related to reporting or administrative filtering. It follows that validation isimperfect and should be viewed as a continuing process, one that will need to be undertaken for
new administrative data sets that become available. Some might argue that these caveats obviate
the usefulness of administrative data and other forms of “naturally occurring” digital
information. Our position is to recognize both strengths and weaknesses of the new data being
made available, using the most rigorous methods possible to address limitations. In defense of
our approach, we would also note the significant advantages of 311 data we analyze relative to
big data more generally. The CRM data are characterized by their richness and geographic
precision, and their longitudinal nature permits long-term tracking. In addition, some of thecontent that is most difficult to validate stems from the fact that it cannot be measured by directmeans and was thus previously unavailable, making it novel. If used properly, such data canbroaden the range of questions that we can examine, and the manner in which we do so. Our
hope is that future efforts will capitalize on the advantages of large-scale administrate records
and to combine them in meaningful ways with survey and observational protocols.
In sum, instead of an either/or approach, the debate between those who believe that
only data generated directly as part of the research process is valid and those that believe thatadministrative and other types of naturally occurring data can be of use pushes both sides toimprove the quality of their research, which certainly can only lead to better science. In the
meanwhile, owing to their increasing availability at little or no cost and at unprecedented
temporal and geographic scales, big data remain a resource to be tapped, and it is incumbent
upon researchers to develop methodologies that do so in ways that fulfill the expectations ofrigorous science.

	
  

40	
  

7. METHODOLOGY, THEORY, AND THE FUTURE OF BIG DATA
Although this manuscript was tailored to the specifics of ecometrics, the rise of big data
illustrates the challenges facing computational social science writ large. There is a clear need todemonstrate what these novel data sources can measure and how constructed metrics aretheoretically relevant. Further, they must be demonstrated to be both reliable and valid in theirmeasurement before modeling can begin, which unfortunately seems to be the default approachin many current approaches which emphasize “econometrics” over “ecometrics” or simply thepower to predict. However powerful predictive analytics may be, it does not answer thesubstantive questions about social processes and mechanisms that motivate most social scientists.
This paper therefore set out to accomplish a linked set of measurement goals that wasrooted in substantive concerns. We grounded our study in a measure that is influential in urban
research and theory, and we closely examined validity in a manner that goes beyond previous
work. Though others have used supplementary data to give context to the patterns in Facebook orcell phone calls (Eagle et al. 2009; Kosinski et al. 2013), ours is the only study that we know ofthat has gone a step further, using multiple internal measures to reduce measurement error andthen validating this technique with external sources and substantive theory—our approach was
not simply data driven. Given the size and novelty of aptly termed “big” data, there is thetemptation to allow them to guide analysis and, in turn, dictate theory. Indeed, some haveclaimed that the era of big data will eliminate the need for theory, as it will be derived from themassive size of the data available. The former Editor of Wired magazine was perhaps the mostbold, claiming “the end of theory” and that “the data deluge makes the scientific method
obsolete” (quoted in Pigliucci, 2009). We strongly disagree. Purely data driven approaches run
the risk of producing models and algorithms that are over fit to the idiosyncrasies of a particular

	
  

41	
  

data set, leading to new theoretical models that are artifactual or just plain wrong—somethingthat has been partially blamed for the failure to predict the crash of the housing bubble in 2008.
Big data hubris is indeed a problem (Lazer et al. 2014). Accordingly, we have had theory takethe lead throughout, determining the case types reflecting physical disorder and the measures ofcivic response rate.
A balance must nonetheless be maintained. As Lazer et al. (2009) rightfully point out, ourcurrent theories are not well-suited to the complexity of information contained in these sorts ofdata, and consequently are often unequipped to offer conjectures about them. In the current case,there was no model of disorder that was sufficiently articulated to predict a priori categories forthe 28 indicators that we identified. Thus, there is something to be learned from these data
about the causal dynamics that underpin disorder and its various manifestations, though such
insights will of course be subject to the same rigorous evaluation required of any new theory.
This “checks-and-balances” relationship between theory and empirics is instructive, and willprobably characterize the continued efforts of scientists to incorporate big data into their work, aswell as the emergence of a fully mature field of computational social science.

	
  

42	
  

REFERENCES
Black, Donald. 1976. The Behavior of Law. New York: Academic.
Brown, Barbara B., Douglas D. Perkins and Graham Brown. 2004. "Incivilities, Place
Attachment and Crime: Block and Individual Effects." Journal of Environmental
Psychology 24:359-371.
Burdette, A. M. and T. D. Hill. 2008. "An Examination of Processes Linking Perceived
Neighborhood Disorder and Obesity." Social science & medicine 67:38-46.
Caughy, M. O., S. M. Nettles and P. J. O'Campo. 2008. "The Effect of Residential Neighborhood
on Child Behavior Problems in First Grade." American Journal of Community
Psychology 42:39-50.
Caughy, Margaret O., Patricia J. O'Campo and Jacqueline Patterson. 2001. "A Brief
Observational Measure for Urban Neighborhoods." Health & Place 7:225-236.
Cohen, Deborah, Suzanne Spear, Richard Scribner, Patty Kissinger, Karen Mason and John
Widgen. 2000. ""Broken Windows" and the Risk of Gonorrhea." American Journal of
Public Health 90:230-236.
Eagle, Nathan, Alex Pentland and David Lazer. 2009. "Inferring Friendship Network Structure
by Using Mobile Phone Data." Proceedings of the National Academy of Sciences
106:15274-15278.
Fischel, William A. 2005. The Homevoter Hypothesis: How Home Values Influence Local
Government Taxation, School Finance, and Land-Use Policies. Cambridge, MA:
Harvard University Press.
Franzini, Luisa, Margaret O'Brien Caughy, Saundra Murray Nettles and Patricia O'Campo. 2008.
"Perceptions of Disorder: Contributions of Neighborhood Characteristics to Subjective
Perceptions of Disorder." Journal of Environmental Psychology 28:83-93.
	
  

43	
  

Furr-Holden, C. D. M., M. J. Smart, J. L. Pokorni, N. S. Ialongo, P. J. Leaf, H. D. Holder and J.
C. Anthony. 2008. "The Nifety Method for Environmental Assessment of NeighborhoodLevel Indicators of Violence, Alcohol, and Other Drug Exposure." Prevention Science
9:245-255.
Furr-Holden, C. Debra M., Adam J. Milam, Elizabeth K. Reynolds, Laura MacPherson and Carl
W. Lejuez. 2012. "Disordered Neighborhood Environments and Risk-Taking Propensity
in Late Childhood through Adolescence." Journal of Adolescent Health 50:100-102.
Hwang, Jackelyn and Robert J. Sampson. 2014. "Divergent Pathways of Gentrification: Racial
Inequality and the Social Order of Renewal in Chicago Neighborhoods." American
Sociological Review 79:726–751.
Jacobs, Jane. 1961. The Death and Life of Great American Cities. New York: Random House.
Kosinski, Michal, David Stillwell and Thore Graepel. 2013. "Private Traits and Attributes Are
Predictable from Digital Records of Human Behavior." Proceedings of the National
Academy of Sciences 110:5802-5805.
Lazer, David, Ryan Kennedy, Gary King and Alessandro Vespignani. 2014. "The Parable of
Google Flu: Traps in Big Data Analysis." Science 343:1203-1205.
Lazer, David, Alex Pentland, Lada Adamic, Siana Aral, Albert-Laszlo Barabasi, Devon Brewer,
Nicholas A. Christakis, Noshir Contractor, James Fowler, Myron Gutmann, Tony Jebara,
Gary King, Michael Macy, Deb Roy and Marshall Van Alstyne. 2009. "Computational
Social Science." Science 323:721-723.
Legewie, Joscha. 2014. "Contested Boundaries: Explaining Where Ethno-Racial Diversity
Provokes Neighborhood Conflict." New York University: Working Paper.

	
  

44	
  

Mujahid, M. S., A. V. D. Roux, M. W. Shen, D. Gowda, B. Sanchez, S. Shea, D. R. Jacobs and
S. A. Jackson. 2008. "Relation between Neighborhood Environments and Obesity in theMulti-Ethnic Study of Atherosclerosis." American Journal of Epidemiology 167:13491357.
O'Brien, Daniel Tumminelli. 2012. "Managing the Urban Commons: The Relative Influence ofIndividual and Social Inventives on the Treatment of Public Space." Human Nature
23:467-489.
---. 2013. "Custodians and Custodianship in Urban Neighborhoods: A Methodology Using
Reports of Public Issues Received by a City's 311 Hotline." Environment and Behavior.
O'Brien, Daniel Tumminelli and David S. Wilson. 2011. "Community Perception: The Ability toAssess the Safety of Unfamiliar Neighborhoods and Respond Adaptively." Journal of
Personality and Social Psychology 100:606-620.
O'Brien, Daniel Tumminelli and Richard A. Kauffman. 2013. "Broken Windows and Low
Adolescent Prosociality: Not Cause and Consequence but Co-Symptoms of Low
Collective Efficacy." American Journal of Community Psychology 51:359-369.
Pigliucci, Massimo 2009. "The End of Theory in Science?" European Molecular Biology
Organization Reports 10:534.
Putnam, Robert. 1993. Making Democracy Work: Civic Traditions in Modern Italy. Princeton,
NJ: Princeton University Press.
Raudenbush, Stephen W. and Robert J. Sampson. 1999. "Ecometrics: Toward a Science ofAssessing Ecological Settings, with Application to the Systematic Social Observation of
Neighborhoods." Sociological Methodology 29:1-41.

	
  

45	
  

---. 1999. "'Ecometrics': Toward a Science of Assessing Ecological Settings, with Application to
the Systematic Social Observation of Neighborhoods." Sociological Methodology 29:141.
Raudenbush, Stephen W., Anthony Bryk, Yuk Fai Cheong, Richard Congdon and Mathilda duToit. 2004. Hlm 6: Hierarchical Linear and Nonlinear Modeling. Lincolnwood, IL:
Scientific Software International.
Ross, Catherine E. and John Mirowsky. 1999. "Disorder and Decay: The Concept and
Measurement of Perceived Neighborhood Disorder." Urban Affairs Review 34:412-432.
Ross, Catherine E., J. Mirowsky and S. Pribesh. 2001. "Powerlessness and the Amplification of
Threat: Neighborhood Disadvantage, Disorder, and Mistrust." American Sociological
Review 66:568-591.
Rundle, A. G., M. D. M. Bader, C. A. Richards, K. M. Neckerman and J. O. Teitler. 2011.
"Using Google Street View to Audit Neighborhood Environments." American Journal ofPreventive Medicine 40:94-100.
Sampson, Robert J. 2012. Great American City: Chicago and the Enduring Neighborhood Effect.
Chicago: University of Chicago Press.
Sampson, Robert J. and Stephen W. Raudenbush. 2004. "Seeing Disorder: Neighborhood Stigma
and the Social Construction of "Broken Windows"." Social Psychology Quarterly
67:317-342.
Skogan, Wesley G. 1992. Disorder and Decline. Berkeley, CA: University of California Press.
Tabachnick, Barbara G. and Linda S. Fidell. 2006. Using Multivariate Statistics. New York:Allyn and Bacon.

	
  

46	
  

Taylor, Ralph B. 2001. Breaking Away from Broken Windows: Baltimore Neighborhoods andthe Nationwide Fight against Crime, Grime, Fear, and Decline. Boulder, CO: Westview.
Verba, Sidney, Kay Lehman Schlozman and Henry E. Brady. 1995. Voice and Equality: Civic
Volunteerism in American Politics. Cambridge, MA: Harvard University Press.
Wen, Ming, Lousie C. Hawkley and John T. Cacioppo. 2006. "Objective and Perceived
Neighborhood Environment, Individual Ses and Psychosocial Factors, and Self-Rated
Health: An Analysis of Older Adults in Cook County, Illinois." Social Science andMedicine 63:2575-2590.
Wilson, James Q. and George Kelling, L. 1982. "The Police and Neighborhood Safety: Broken
Windows." Atlantic Monthly 127:29-38.

	
  

47	
  

Table 1. Counts of case types that reflect human neglect or denigration of the neighborhood,including the factors and loadings from an exploratory factor analysis.
Case TypeCount

FactorLoading

Case Type

Housing Issues

Big Buildings
Big Buildings
Enforcement
Big Buildings Online
RequestBig Buildings
Resident ComplaintCount

FactorLoading

236

.68

274

.72

209

.60

Bed Bugs

871.49

Breathe Easy

590

.53

442

.44

2175

.62

687

.54

Graffiti Removal

8826

.83

796

.59

PWD Graffiti

847

.50

330

.52

26

—

Trash
Abandoned Bicycle

144

.45

128

—

Empty Litter Basketb

802

.30

8948

.85

Illegal Dumping

2292

.87

174

.41

4756

.91

3287

.40

105

—

236

—

Chronic
Dampness/Mold
Heat - Excessive,
Insufficient
MaintenanceComplaint –Residential
Mice Infestation –Residential
Pest Infestation –Residential
Poor Ventilationa
Squalid Living
Conditionsa
Unsatisfactory Living
Conditions
Unsatisfactory Utilities– Electrical, Plumbing
Uncivil Use of Space

Graffiti

Improper Storage ofTrash (Barrels)
Rodent Activity

Abandoned Building

238

.36

Illegal Occupancy

642

.42

Illegal Rooming House

471

.47

180

.41

Illegal Use

137

—

336

.42

Overflowing or Unkept Dumpstera

526

—

2438

.80

Pigeon Infestation

82

—

432

.57

Maintenance –Homeowner
Parking on Front/Back
Yards (Illegal Parking)
Poor Conditions ofProperty
Trash on Vacant Lot

No Factor (Discarded)
Illegal Auto Body
Shop
Illegal Posting of
Signs

a

– Items did not load on initial factor analysis, but were added based on content similar to factor
or one or more of its constituent items.
	
  

48	
  

b

– Item loaded at >.3 on both the trash and graffiti factors. It was maintained on the trash factor
for reasons of content.
Note: For factor analysis, N = 544 census block groups. An iterated principal factors estimation
was used with a promax rotation.

	
  

49	
  

Table 2.
Descriptive Statistics for and Correlations between Five Sub-Measures of Physical Disorder.
1.

2.

3.

4.

5.

1. Housing
2. Uncivil
Use
3. BigBuildings
4. Graffiti—.47***.34***.18***.20***

—

—

.10*

.04

.33***

—

—

—

.14**

.21***————.45***

5. TrashMedian(Range)—17.5(0 – 183)—
5(0 – 49)—
0(0 – 18)——
7
10(0 – 216) (0 – 279)

Note: N = 544 census block groups. All variables were log-transformed before correlations.
** - p < .01, *** - p < .001

	
  

50	
  

Table 3.
Descriptive statistics for and correlations between proposed indicators of response rate.

Exemplars% Pub
Issues
by
Users

Snow
Plowing

Sidewalk
Repairs

StreetLight
Outage
s

Total
Pop

.61***.53***.02.26***.18***.01.30***.78***.68***.36***-.24***.43***.19***-.09.07.78***—.93***.62***-.07+.44***.28***.04.27***.68***.68***.95***—.60***.03.42***.30***.13+.26***

Exemplary
Reporters of Public
Issuesa.51***.40***.66***.64***—.16***.29***.28***.09.25***

Percentage of
Public Issues
Reported byRegistered Users.04-.25***

-.08

.05

.22**

—

-.36***

.07+

.18*

.09*

Requests for Snow
Plowingb.31***.49***.51***.48***.35***-.36***—.14***-.12-.00

GeneralRequests

Bulk Item
Pick-Ups

Registered
Users

AveragePublicReporters

General Requestsa—.37***.67***

Bulk Item PickUpsa.37***—All RegisteredUsersa.69***

Registered Users
Reporting Public
Issuesa

	
  

51	
  

Propensity to
Request Sidewalk
Repairsb

.13+

.08

.18*

.26***.26***

.15*

.14+

—

.18*

.09*

Propensity toReport Street Light
Outages within One
Monthb

.01

-.09

.04

.13+

.09

.18*

-.12+

.18*

—

-.01

Total Population.25***

.00

.17*

.14+

.16*

.04

.04

.00

-.01

—

Mean (SD)47.13(34.07)56.28(36.28)53.49(30.68)21.37(13.33)3.06(2.99)

0.44(0.09)

0.00(1.00)

0.00(0.37)

0.00(0.60)

1153
(565)

Range

1 – 461

0 – 1992 – 232

0 – 104

0 – 290.17 –
0.76-3.29 –
2.64-.89 –
1.18-1.01 –
2.22

246 –
4719

+

- p < .10, * - p < .05, ** - p < .01, *** - p < .001

Note: N = 541 for all measures except propensity to report street light outages within one month (N = 195). Descriptive statistics
reported for all census block groups; correlations including all CBGs with measures on both variables reported above the diagonal,
correlations for those CBGs with values for all measures (N = 195) reported below the diagonal. See text for more details on thederivation of each measure.
a– Log-transformed before correlations to account for skewed distribution.
b– Deviation from regression equation controlling for key variables; see text for more detail.

	
  

52	
  

Table 4.
Comparison of results from regressions using the five categories of physical disorder derived
from the CRM database to predict objective measures of garbage, with and without the CRMbased adjustment factor.

Housing

Uncivil
Use

Big
Buildings

Graffiti

Trash

B

B

B

B

B

Raw
Measure.63***.38***

.21*

.13

.18*

R2

.40***.14***

.05*

.02

.03*

Raw
Measure.62***.39***

.27**

.29**

.33**

Adjustment
Factor

-.13+

-.19*

-.20*

-.27*

-.29**

Total R2

.42***.18***

.08**

.07*

.09**

Δ R2

.02*

.04*

.03*

.05*

.06**

Note: N = 135 census block groups classified as residential and with measures of garbage for ten
or more street segments. All CRM-based variables were log-transformed before regressions.
* - p < .05, ** - p < .01, *** - p < .001

	
  

53	
  

Table 5.
Intraclass correlations (ICC) and reliabilities (λ) for level (intercept) and cross-time change(slope) in measures of public denigration and private neglect across census block groups forvarious time windows.
Housing
Intercept

Uncivil Use
SlopeIntercept

Big Buildings

SlopeIntercept

Slop
e

ICC

λ

λ

IC
C

λ

λ

ICC

λ

λ

1 week

.13

.91

.36

.02

.78

.31

.01

.46

.10

2 weeks

.23

.91

.36

.03

.78

.30

.02

.46

.10

3 weeks

.31

.91

.35

.05

.78

.30

.03

.46

.10

1
month

.39

.91

.36

.07

.78

.30

.03

.46

.11

2
month

.56

.91

.37

.13

.78

.29

.04

.46

.10

3
month

.65

.91

.37

.19

.77

.30

.10

.46

.09

4
month

.72

.91

.37

.25

.78

.29

.19

.46

.12

6
month

.77

.90

.33

.39

.74

.25

.31

.48

.01

	
  

54	
  

Table 5 (cont).
GraffitiIntercept

Trash
Slope

Intercept

Public Reporters
Slope

ICC

λ

λ

IC
C

λ

λ

1 week

.07

.87

.51

.05

.88

2 weeks

.13

.87

.51

.09

3 weeks

.18

.87

.51

1 month.24.87

2 month.38

3 monthIntercept

Slope

ICC

λ

λ

.48

.20

.96

.47

.88

.48

.32

.96

.40

.13

.88

.48

.39

.96

.35

.51

.17

.88

.48

.47

.96

.30

.87

.51

.30

.88

.48

.63

.95

.21

.47

.86

.51

.41

.88

.45

.68

.95

.03

4 month

.56

.87

.51

.47

.88

.47

.75

.95

.04

6 month

.60

.84

.56

.63

.86

.39

.80

.93

.01

Note: N’s vary based on the number of time intervals possible for the 28 month period in the
database, nested in 541 census block groups. All ICCs significant at p < .001.

	
  

55	
  

Table 6.
Intraclass correlations (ICC) and reliabilities (λ) for level (intercept) and cross-time change(slope) in measures of public denigration and private neglect across census tracts for various time
windows.
Housing
Intercept

Uncivil Use
SlopeIntercept

Big Buildings

SlopeIntercept

Slope

IC
C

λ

Λ

IC
C

λ

λ

ICC

λ

λ

1 week

.30

.97

.44

.06

.92

.38

.02

.68

.26

2 weeks

.46

.97

.45

.12

.92

.37

.04

.68

.27

3 weeks

.56

.97

.45

.17

.92

.37

.07

.68

.26

1 month

.64

.97

.46

.22

.92

.37

.09

.68

.27

2 month

.78

.97

.46

.35

.92

.36

.12

.68

.27

3 month

.84

.97

.44

.46

.91

.40

.24

.68

.24

4 month

.88

.97

.45

.55

.92

.37

.40

.68

.29

6 month

.90

.96

.36

.75

.90

.31

.41

.70

.01

.78

.92

—

.64

.84

—

.20

.43

—

Composite
a

	
  

56	
  

Table 6 (cont).
GraffitiIntercept

Trash
Slope

Intercept

Public Reporters
Slope

Intercept

Slope

ICC

λ

λ

IC
C

λ

λ

ICC

λ

λ

1 week

.18

.95

.71

.14

.96

.68

.43

.99

.48

2 weeks

.31

.95

.71

.25

.96

.67

.59

.99

.39

3 weeks

.39

.95

.71

.32

.96

.67

.67

.99

.34

1 month

.49

.95

.71

.40

.96

.67

.73

.98

.28

2 month

.64

.95

.72

.59

.96

.67

.84

.98

.06

3 month

.73

.95

.72

.67

.96

.62

.88

.98

.07

4 month

.79

.95

.71

.74

.96

.66

.90

.98

.05

6 month

.86

.94

.77

.86

.95

.55

.93

.98

.00

Compositea

.53

.77

—

.59

.82

—

—

—

—

a

– A combination of the raw count and the measures of concern for the public space, calculated
for six-month windows only. See text for more details on construction.
Note: N’s vary based on the number of time intervals possible for the 28 month period in the
database, nested in 156 census tracts. All ICCs significant at p < .001.

	
  

57	
  

Figure 1.
Estimated relationships between categories of physical disorder with standardized parametersfrom best-fitting confirmatory factor analysis.

Note: CFI = .95; SRMR = .05; N = 543 census block groups. All parameters significant at p <
.001.

	
  

58	
  

Figure 2.
Relationships between objective and CRM-derived measures of response rate with standardized
parameters from the best-fitting structural equation model.

Note: CFI = .95, SRMR = .06; N = 195 census block groups with measures on all variables. All
parameters significant at p < .05.
a– Log-transformed before analysisb– Controlled for total population, total street length, and __dead end length before analysis.

	
  

59	
  

APPENDIX A
Table A1. Case Types that Reflect an Issue in the Public Space
Case TypeCount
Case Type
Abandoned Bicycle
71Park Safety Notifications
Abandoned Building
103
Parking Enforcement
Abandoned Vehicles
2233
Parking Meter Repairs
Bridge Maintenance29
Parking on Front/Back Yards(Illegal Parking)
Building Inspection Request 822
Parks General Request
Catchbasin
13
Parks Lighting Issues
Construction Debris
101
Pavement Marking
Maintenance
Empty Litter Basket
292
Pick up Dead Animal
Exceeding Terms of Permit
68
Pigeon Infestation
Fire Hydrant
8
PWD Graffiti
General Lighting Request
460
Request for Litter Basket
Installation
Graffiti Removal
3893
Request for Pothole Repair
Highway Maintenance
3297
Request for Snow Plowing
Illegal Auto Body Shop46
Requests for Street Cleaning
Illegal Dumping
831
Requests for Traffic Signal
Studies or Reviews
Illegal Occupancy
263
Roadway Repair
Illegal Posting of Signs
116
Rodent Activity
Illegal Rooming House
177
Sidewalk Cover / Manhole
Illegal Use
62
Sidewalk Repair
Illegal Vending
32
Sidewalk Repair (Make Safe)
Improper Storage of Trash
1745
Sign Repair(Barrels)
Install New Lighting
25
Snow Removal
Misc. Snow Complaint
1407
Street Light Knock Downs
Missed6211
Street Light Outages
Trash/Recycling/Yard
Waste/Bulk Item
Missing Sign
671
Traffic Signal Repair
New Sign, Crosswalk or
976
Trash on Vacant Lot
Pavement Marking
New Tree Requests
831
Tree Emergencies
Overflowing or Un-kept
149
Tree Maintenance Requests
DumpsterPark Improvement Requests 3
Upgrade Existing Lighting
Park Maintenance Requests
87

	
  

60	
  

Count
2
685
139
132
106
5
272
1374
29
160
80
4603
7270
953
96
306
1241
3
1294
2119
1172
2103
476
8127
2585
121
3446
3336
15

APPENDIX B
Neighborhood audits identifying street light outages and assessing levels of street garbage wereconducted in 72 of Boston’s 156 census tracts (46%) between June 1 and August 31, 2011 as partof an undergraduate seminar. The sample was constructed in a multistep process, intended tocover about half of the city, while capturing the full range of demographic, socioeconomic, andgeographic diversity.
First, tracts were attributed to one of Boston’s 16 planning districts, contiguous regionswith characteristic demographic and socioeconomic profiles.12 The population-weighted mean
for tract-level median income was calculated for each planning district. A stratified sample ofthree or four tracts was then created for each planning district (depending on the size), including
one tract more than a standard deviation above the local weighted mean for median income, one
more than a standard deviation below, and either one or two within a standard deviation of theweighted mean. Because planning districts vary in the number of tracts they contain (min = 1,
max = 24), the sample was completed by random selection from planning districts with a highnumber of tracts. The final sample was representative of the diversity across all Boston tracts,
both in terms of its central tendency and range (See Table B1).
During each audit teams of two walked the streets of a particular tract. Highways, service
roads, and other roads rarely used by pedestrians were omitted. One person walked each side of
the street. The goal was to cover all other roads, though this was sometimes not possible given
time constraints on audits. On each street segment (intersection-to-intersection or intersection–
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
12
A distinction created by the Boston Redevelopment Authority for administrative
purposes, but based on historically salient regions, many of which are once-independent
municipalities that were annexed. Using an ANOVA, the planning districts account for about
50% of the variation in ethnic composition and median income across census tracts.
	
  

61	
  

to-dead end), each person recorded the level of garbage and the presence of any street lightoutages on his or her side of the street. In total, 4,239 street segments were assessed. Garbage
was rated on a 5-point scale, with higher scores indicating larger piles of garbage and more ofthem, for both the street and the sidewalk (if present). These measures were then adjusted forstreet sweeping.
The date of data collection was used in conjunction with the City’s street sweeping
schedule to fit a linear model that used the number of days since that side of the street was swept
to predict level of garbage. The linear model indicated that streets swept within the past threedays had lower-than-expected litter at the rate of .06/day on our scale. After three days had
passed, there was no substantial difference in garbage ratings. Sidewalks were not adjusted inthis fashion as they are not swept. Following this, an average of the adjusted street measure andthe sidewalk measure on each side of the street was calculated as the total garbage rating for thestreet segment. Before data collection, inter-rater reliability was established through training
PowerPoint slides and neighborhood walks.

	
  

62	
  

Table B1. Comparison of Demographic Characteristics between Tracts Sampled in Street
Light Outage and Garbage Audits and All Tracts
All Tracts
Mean
Range
(SD)
$10,250
Median$52,572
Income($23,607)$143,819
Population
22.83
1.36 –a
Density(16.24)93.07%.36.00 - .88
Homeowners (.19).51% White.00 - .99(.31)
.21
% Black.00 - .92(.25)
.17
% Hispanic.00 - .84
(.16)
a

	
  

– Thousands per sq. mil

63	
  Sampled Tracts
Mean
Range
(SD)
$10,250$55,256($27,436)$143,819
23.96
3.18 –(17.55)93.07.38.00 - .88(.21)
.52
.00 - .98(.33)
.22
.00 - .92(.26)
.15
.00 - .62(.1f)

APPENDIX C
The street light outages, sidewalk reports, and garbage assessments all describe events orconditions on a single street segment within a CBG. To create CBG-level measures that
controlled for the microspatial effects of street characteristics, multilevel models (Raudenbush etal. 2004) were developed in which two simultaneous equations were estimated, one at the levelof streets (first-level), the second at the level of CBGs (second-level). The street-level equation
was defined as:
𝑌!" =    𝛽!! +𝛽! 𝑋!"# + 𝑟!"!

𝑟!" ~𝑁(0, 𝜎 ! )where Yjk represents the jth street in CBG k, and β0k is the estimated mean for neighborhood k.
Each Xi is a first-level predictor, and each βi is the corresponding regression parameter,
explaining differences between streets within the same CBG. The errors of measurement rij forstreet j in neighborhood k are assumed to be normally distributed with variance σ2. The estimatedmean for neighborhood k is modeled as:
𝛽!! =    𝛾!! + 𝜇!!
𝜇!! ~𝑁(0, 𝜏)
where γ00 is the estimated mean value for the neighborhood-level measure across neighborhoods,
and µ0k is the random neighborhood effect for neighborhood k. The latter can also be described as
the deviation of the average value in neighborhood k from the cross-neighborhood mean. These
random neighborhood effects are assumed to be normally distributed with variance τ, and are thevalues extracted for the desired CBG-level measure. For example, in the case of garbage, µ0k
indicates the extent to which the average street in CBG k has more or less loose garbage than the
average street in the average neighborhood. In addition, the magnitude of τ in relation to σ2 is
	
  

64	
  

valuable in determining how well Y captures differences between neighborhoods. This isevaluated with a χ2 test.
There were slight variations between the models for street light outages, sidewalks andgarbage, including in first-level predictors and the link function used. For sidewalks, thesidewalk care index was the lone first-level predictor. The binary outcome (whether a sidewalk
generated any reports) used a logit link, and the continuous outcome (how many reports a
sidewalk generated) used a zero-inflated Poisson link. The models for both garbage and street
light outages incorporated a dichotomous variables distinguishing between main and non-mainstreets, and between streets with different types of zoning. For garbage, dichotomous variables
for all non-residential zonings were included (i.e., commercial, industrial, exempt, and unzoned). To conserve degrees of freedom for the analysis of street light outages, this was
simplified to a single dichotomous variable distinguishing between residential and nonresidential zonings. The garbage model used a standard regression as garbage was a continuous,
normal variable. The street light outage outcomes were dichotomous, necessitating a logit link.
In determining the proper event-level outcome to use as the basis for the CBG-levelmeasure, there were multiple options for the street light outages and sidewalk reports. Multilevel
models were run using each option and their results were compared.
For sidewalks, there were two candidate measures: whether a sidewalk polygon generated
one or more reports (binary model); and if a polygon had generated any reports, how many it had
generated (continuous model). Since not all CBGs contained a sidewalk that generated a request
for repair, the continuous model only analyzed 416 CBGs. In each model, the sidewalk careindex was entered as the sole first-level predictor, in order to control for the objective need for
repair.

	
  

65	
  

Both models indicated significant CBG-level variation, with the binary measureappearing to do so more effectively (binary: χ2df=541 = 940.47, p < .001; continuous: χ2df=415 =
505.11, p < .01). In addition, the binary measure was predicted by the sidewalk care index in the
expected direction (i.e., a higher index predicts a lower likelihood of requests for repair), while
the continuous measure was not (binary: β = -0.003, p < .01; continuous: β = 0.001, p = ns),suggesting it to be the superior measure for the subsequent analyses. Neighborhood-level
residuals for this measure were extracted, with higher values indicating a CBG with a greater
likelihood of requesting a sidewalk repair, controlling for quality of the sidewalk.
For street light outages, it was necessary to run the models at the tract level, owing to the
low number of street light outages per CBG (244 in 127 CBGs and 56 tracts with outages13). The
model was used to predict the likelihood of an outage being reported by a constituent at six time
points after being identified: one week, two weeks, one month, two months, three months, andfour months. Of these, the one-month (χ2df=54 = 78.39, p < .05), two-month (χ2df=53 = 80.80, p <.01), three-month (χ2df=53 = 73.95, p < .05), and four-month (χ2df=53 = 73.30, p < .05) models
identified significant differences between tracts. Given the strong statistical similarity between
the one-month and two-month measures, the former was selected for subsequent analyses
because it indicates relatively quicker action on the part of constituents.
The continuous measure of garbage was also assessed for neighborhood-level variation.
The model indicated significant CBG-level variation (χ2df=350 = 4,765.56, p < .001). The
neighborhood-level residual was extracted as the measurement of garbage.

	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
13
This number diminishes with some measures that allow greater time between
identification of an outage and reporting, being that those that were reported by City employees
in that time span were removed.
	
  

66