Package: maup
maupmaupis the geospatial toolkit for redistricting data. The package streamlines
the basic workflows that arise when working with blocks, precincts, and
districts, such asAssigning precincts to districts,Aggregating block data to precincts,Disaggregating data from precincts down to blocks,Prorating data when units do not nest neatly,
andFixing topological issues, overlaps, and gapsThe project's priorities are to be efficient by using spatial indices whenever
possible and to integrate well with the existing ecosystem aroundpandas,geopandasandshapely. The package is distributed
under the MIT License.InstallationTo installmaupfrom PyPI, runpip install maupfrom your terminal.For development,maupusesPoetry.
To develop newmaupfeatures, clone this repository and runpoetry install.ExamplesHere are some basic situations where you might findmauphelpful. For these
examples, we use test data from Providence, Rhode Island, which you can find in
ourRhode Island shapefiles repo, or
in theexamplesfolder of this repo, reprojected to a non-geographic coordinate
reference system (CRS) optimized
for Rhode Island.** Many of maup's functions behave badly in geographic projections (i.e., lat/long
coordinates), which are the default for shapefiles from the U.S. Census bureau. In
order to find an appropriate CRS for a particular shapefile, consult the database
athttps://epsg.org. **>>>importgeopandas>>>importpandas>>>>>>blocks=geopandas.read_file("zip://./examples/blocks.zip").to_crs(32030)>>>precincts=geopandas.read_file("zip://./examples/precincts.zip").to_crs(32030)>>>districts=geopandas.read_file("zip://./examples/districts.zip").to_crs(32030)Assigning precincts to districtsTheassignfunction inmauptakes two sets of geometries calledsourcesandtargetsand returns a pandasSeries. The Series maps each geometry insourcesto the geometry intargetsthat covers it. (Here, geometryAcoversgeometryBif every point ofAand its boundary lies inBor its
boundary.) If a source geometry is not covered by one single target geometry, it
is assigned to the target geometry that covers the largest portion of its area.>>>importmaup>>>>>>precinct_to_district_assignment=maup.assign(precincts,districts)>>># Add the assigned districts as a column of the `precincts` GeoDataFrame:>>>precincts["DISTRICT"]=precinct_to_district_assignment>>>precinct_to_district_assignment.head()07152133641dtype:int64As an aside, you can use thatprecinct_to_district_assignmentobject to create agerrychainPartitionrepresenting this districting plan.Aggregating block data to precinctsPrecinct shapefiles usually come with election data, but not demographic data.
In order to study their demographics, we need to aggregate demographic data from
census blocks up to the precinct level. We can do this by assigning blocks to
precincts and then aggregating the data with a Pandasgroupbyoperation:>>>variables=["TOTPOP","NH_BLACK","NH_WHITE"]>>>>>>blocks_to_precincts_assignment=maup.assign(blocks,precincts)>>>precincts[variables]=blocks[variables].groupby(blocks_to_precincts_assignment).sum()>>>precincts[variables].head()TOTPOPNH_BLACKNH_WHITE05907886380156369241301265495844699360094351053449621563713If you want to move data from one set of geometries to another but your source
geometries do not nest cleanly into your target geometries, seeProrating data when units do not nest neatly.Disaggregating data from precincts down to blocksIt's common to have data at a coarser scale that you want to attach to
finer-scale geometries. For instance, this may happen when vote totals for a certain election are only reported at the county level, and we want to attach that data to precinct geometries.Let's say we want to prorate the vote totals in the columns"PRES16D","PRES16R"from ourprecinctsGeoDataFrame down to ourblocksGeoDataFrame.
The first crucial step is to decide how we want to distribute a precinct's data
to the blocks within it. Since we're prorating election data, it makes sense to
use a block's total population or voting-age population. Here's how we might
prorate by population ("TOTPOP"):>>>election_columns=["PRES16D","PRES16R"]>>>blocks_to_precincts_assignment=maup.assign(blocks,precincts)>>>>>># We prorate the vote totals according to each block's share of the overall>>># precinct population:>>>weights=blocks.TOTPOP/blocks_to_precincts_assignment.map(blocks.TOTPOP.groupby(blocks_to_precincts_assignment).sum())>>>prorated=maup.prorate(blocks_to_precincts_assignment,precincts[election_columns],weights)>>>>>># Add the prorated vote totals as columns on the `blocks` GeoDataFrame:>>>blocks[election_columns]=prorated>>>>>># We'll call .round(2) to round the values for display purposes, but note that the>>># actual values should NOT be rounded in order to avoid accumulation of rounding>>># errors.>>>blocks[election_columns].round(2).head()PRES16DPRES16R00.000.00112.261.70215.202.62315.502.6743.280.45Warning about areal interpolationWe strongly urge younotto prorate by area!The area of a census block isnota good predictor of its population. In fact, the correlation goes in the
other direction: larger census blocks arelesspopulous than smaller ones.Warnings about data anomalies(1) Many states contain Census blocks and precincts that have zero population. In the
example above, a zero-population precinct leads to division by zero in the
definition of the weights, which results in NaN values for some entries.Although it is not strictly necessary to resolve this in the example above, sometimes
this creates issues down the line.  One option is to replace NaN values with zeros,
using>>>weights=weights.fillna(0)(2) In some cases, zero-population precincts may have a small nonzero number of recorded
votes in some elections. The procedure outlined above will lose these votes in the
proration process due to the zero (or NaN) values for the weights corresponding to all
the blocks in those precincts. If it is crucial to keep vote totals perfectly accurate,
these votes will need to be assigned to the new units manually.Prorating data when units do not nest neatlySuppose you have a shapefile of precincts with some election results data and
you want to join that data onto a different, more recent precincts shapefile.
The two sets of precincts will have overlaps, and will not nest neatly like the
blocks and precincts did in the above examples. (Not that blocks and precincts
always nest neatly---in fact, they usually don't!)In most cases, election data should be prorated from each old precinct to the new
precincts with weights proportional to the population of the intersections between
the old precinct and each new precinct.  The most straightforward way to accomplish
this is to first disaggregate the data from the old precincts to Census blocks as in the example above, and then reaggregate from blocks to the new precincts.>>>old_precincts=precincts>>>new_precincts=geopandas.read_file("zip://./examples/new_precincts.zip").to_crs(32030)>>>>>>election_columns=["SEN18D","SEN18R"]>>>>>>blocks_to_old_precincts_assignment=maup.assign(blocks,old_precincts)>>>blocks_to_new_precincts_assignment=maup.assign(blocks,new_precincts)>>>>>># We prorate the vote totals according to each block's share of the overall>>># old precinct population:>>>weights=blocks.TOTPOP/blocks_to_old_precincts_assignment.map(blocks.TOTPOP.groupby(blocks_to_old_precincts_assignment).sum()).fillna(0)>>>prorated=maup.prorate(blocks_to_old_precincts_assignment,precincts[election_columns],weights)>>>>>># Add the prorated vote totals as columns on the `blocks` GeoDataFrame:>>>blocks[election_columns]=prorated>>>>>>new_precincts[election_columns]=blocks[election_columns].groupby(blocks_to_new_precincts_assignment).sum()>>>new_precincts[election_columns].round(2).head()SEN18DSEN18R0728.1749.381370.0021.00297.0017.00391.165.554246.0020.00As a sanity check, let's make sure that no votes were lost in either step.
Total votes in the old precincts, blocks, and new precincts:>>>old_precincts[election_columns].sum()SEN18D23401SEN18R3302dtype:float64>>>>>>blocks[election_columns].sum()SEN18D23401.0SEN18R3302.0dtype:float64>>>>>>new_precincts[election_columns].sum()SEN18D20565.656675SEN18R2947.046857dtype:float64Oh no - what happened??? All votes were successfully disaggregated to blocks, but a
significant percentage were lost when reaggregating to new precincts.It turns out that when blocks were assigned to both old and new precincts, many blocks were not assigned to any precincts.  We can count how many blocks were unassigned in each case:print(len(blocks))print(blocks_to_old_precincts_assignment.isna().sum())print(blocks_to_new_precincts_assignment.isna().sum())30148841227So, out of 3,014 total Census blocks, 884 were not assigned to any old precinct and
1,227 were not assigned to any new precinct.  If we plot the GeoDataFrames, we can see why:>>>blocks.plot()>>>old_precincts.plot()>>>new_precincts.plot()The boundaries of the regions covered by these shapefiles are substantially
different---and that doesn't even get into the possibility that the precinct shapefiles may have gaps between precinct polygons that some blocks may fall into.Once we know to look for this issue, we can see that it affected the previous example
as well:>>>blocks[variables].sum()TOTPOP178040NH_BLACK23398NH_WHITE66909dtype:int64>>>>>>precincts[variables].sum()TOTPOP140332NH_BLACK19345NH_WHITE46667dtype:int64Moral: Precinct shapefiles often haveterribletopological issues!These issues should be diagnosed and repaired to the greatest extent possible before
moving data around between shapefiles; seeFixing topological issues, overlaps, and gapsbelow for details about how maup can help with this.Progress barsFor long-running operations, the user might want to see a progress bar to
estimate how much longer a task will take (and whether to abandon it altogether).maupprovides an optional progress bar for this purpose. To temporarily activate
a progress bar for a certain operation, usewith maup.progress()::>>>withmaup.progress():...assignment=maup.assign(precincts,districts)...To turn on progress bars for all applicable operations (e.g. for an entire script),
setmaup.progress.enabled = True:>>>maup.progress.enabled=True>>># Now a progress bar will display while this function runs:>>>assignment=maup.assign(precincts,districts)>>># And this one too:>>>pieces=maup.intersections(old_precincts,new_precincts,area_cutoff=0)Fixing topological issues, overlaps, and gapsPrecinct shapefiles are often created by stitching together collections of
precinct geometries sourced from different counties or different years. As a
result, the shapefile often has gaps or overlaps between precincts where the
different sources disagree about the boundaries.  (And by "often," we mean "for almost every shapefile that isn't produced by the U.S. Census Burueau.")
As we saw in the examples above, these issues can pose problems when moving data between shapefiles.Even when working with a single shapefile, gaps and overlaps may cause problems if you are interested in working with the adjacency graph of the precincts.
This adjacency information is especially important when studying redistricting, because districts are almost always expected to be contiguous.Before doing anything else, it is wise to understand the current status of a shapefile with regard to topological issues.maupprovides adoctorfunction to diagnose gaps, overlaps, and invalid geometries.  If a shapefile has none of these issues,maup.doctorreturns a value ofTrue; otherwise it returnsFalsealong with a brief summary of the problems that it found.The blocks shapefile, like most shapefiles from the Census Bureau, is clean:>>>maup.doctor(blocks)TrueThe old precincts shapefile, however, has some minor issues:>>>maup.doctor(old_precincts)Thereare2overlaps.Thereare3holes.FalseAs of version 2.0.0,maupprovides two repair functions with a variety of options for fixing these issues:quick_repairis the new name for theautorepairfunction from version 1.x (andautorepairstill works as a synonym).  This function makes fairly simplistic repairs to gaps and overlaps:Any polygon $Q$ created by the overlapping intersection of two geometries $P_1$ and $P_2$ is removed from both polygons and reassigned to the one with which it shares the greatest perimeter.Any polygon $Q$ representing a gap between geometries $P_1,\ldots, P_n$ is assigned to the one with which it shares the greatest perimeter.This function is probably sufficient when gaps and overlaps are all very small in area relative to the areas of the geometries,ANDwhen the repaired file will only be used for operations like aggregating and prorating data.  But it shouldNOTbe relied upon when it is important for the repaired file to accurately represent adjacency relations between neighboring geometries, such as when a precinct shapefile is used as a basis for creating districting plans with contiguous districts.For instance, when a gap adjoins many geometries (which happens frequently along county boundaries in precinct shapefiles!), whichever geometry the gap is adjoined to becomes "adjacent" toallthe other geometries adjoining the gap, which can lead to the creation of discontiguous districts in plans based on the repaired shapefile.smart_repairis a more sophisticated repair function designed to reproduce the "true" adjacency relations between geometries as accurately as possible.  In the case of gaps that adjoin several geometries, this is accomplished by an algorithm that divides the gap into pieces to be assigned to different geometries instead of assigning the entire gap to a single geometry.In addition to repairing gaps and overlaps,smart_repairincludes two optional features:In many cases, the shapefile geometries are intended to nest cleanly into some larger units; e.g., in many states, precincts should nest cleanly into counties.smart_repairallows the user to optionally specify a second shapefile---e.g., a shapefile of county boundaries within a state---and then performs the repair process so that the repaired geometries nest cleanly into the units in the second shapefile.Whether as a result of inaccurate boundaries in the original map or as an artifact of the repair algorithm, it may happen that some units share boundaries with very short perimeter but should actually be considered "queen adjacent"---i.e., intersecting at only a single point---rather than "rook adjacent"---i.e., intersecting along a boundary of positive length.smart_repairincludes an optional step in which all rook adjacencies of length below a user-specified parameter are converted to queen adjacencies.smart_repaircan accept either a GeoSeries or GeoDataFrame as input, and the output type will be the same as the input type.  The input must be projected to a non-geographic coordinate reference system (CRS)---i.e.,notlat/long coordinates---in order to have sufficient precision for the repair.  One option is to reproject a GeoDataFrame calledgdfto a suitable UTM (Universal Transverse Mercator) projection viagdf=gdf.to_crs(gdf.estimate_utm_crs())At a minimum, all overlaps will be repaired in the output. Optional arguments include:snapped(default valueTrue): IfTrue, all polygon vertices are snapped to a grid of size no more than $10^{-10}$ times the maximum of width/height of the entire shapefile extent.HIGHLY RECOMMENDEDto avoid topological exceptions due to rounding errors.fill_gaps(default valueTrue): IfTrue, all simply connected gaps with area less thanfill_gaps_thresholdtimes the largest area of all geometries adjoining the gap are filled.  Default threshold is $0.1$; settingfill_gaps_threshold = Nonewill fill all simply connected gaps.nest_within_regions(default valueNone): Ifnest_within_regionsis a secondary GeoSeries or GeoDataFrame of region boundaries (e.g., counties within a state) then the repair will be performed so that repaired geometries nest cleanly into the region boundaries; specifically, each repaired geometry will be contained in the region with which the original geometry has the largest area of intersection.  Note that the CRS for the region GeoSeries/GeoDataFrame must be the same as that for the primary input.min_rook_length(default valueNone): Ifmin_rook_lengthis given a numerical value, all rook adjacencies with length below this value will be replaced with queen adjacencies.  Note that this is an absolute value and not a relative value, so make sure that the value provided is in the correct units with respect to the input GeoSeries/GeoDataFrame's CRS.ExamplesFirst, we'll useshapelyandgeopandasto create a GeoDataFrame of "toy precincts" from scratch.importrandomimportgeopandasimportmaupfromshapely.geometryimportPolygonrandom.seed(2023)# For reproducibilityppolys=[]foriinrange(4):forjinrange(4):poly=Polygon([(0.5*i+0.1*k,0.5*j+(random.random()-0.5)/12)forkinrange(6)]+[(0.5*(i+1)+(random.random()-0.5)/12,0.5*j+0.1*k)forkinrange(1,6)]+[(0.5*(i+1)-0.1*k,0.5*(j+1)+(random.random()-0.5)/12)forkinrange(1,6)]+[(0.5*i+(random.random()-0.5)/12,0.5*(j+1)-0.1*k)forkinrange(1,5)])ppolys.append(poly)toy_precincts_df=geopandas.GeoDataFrame(geometry=geopandas.GeoSeries(ppolys))toy_precincts_df.plot(cmap="tab20",alpha=0.7)Check for gaps and overlaps:>>>maup.doctor(old_precincts)Thereare28overlaps.Thereare23holes.FalseAll the gaps between geometries in this example are below the default threshold, so a basic application ofsmart_repairwill resolve all overlaps and fill all gaps:toy_precincts_repaired_df=maup.smart_repair(toy_precincts_df)toy_precincts_repaired_df.plot(cmap="tab20",alpha=0.7)We can check that the repair succeeded:>>>maup.doctor(old_precincts)TrueNow suppose that the precincts are intended to nest cleanly into the following "toy counties:"cpoly1=Polygon([(0,0),(1,0),(1,1),(0,1)])cpoly2=Polygon([(1,0),(2,0),(2,1),(1,1)])cpoly3=Polygon([(0,1),(1,1),(1,2),(0,2)])cpoly4=Polygon([(1,1),(2,1),(2,2),(1,2)])toy_counties_df=geopandas.GeoDataFrame(geometry=geopandas.GeoSeries([cpoly1,cpoly2,cpoly3,cpoly4]))toy_counties_df.plot(cmap='tab20')We can perform a "county-aware" repair as follows:toy_precincts_repaired_county_aware_df=maup.smart_repair(toy_precincts_df,nest_within_regions=toy_counties_df)toy_precincts_repaired_county_aware_df.plot(cmap="tab20",alpha=0.7)Next, suppose that we'd like to get rid of small rook adjacencies at corner points where 4 precincts meet.  We might reasonably estimate that these all have length less than $0.1$, so we can accomplish this as follows:toy_precincts_repaired_county_aware_rook_to_queen_df=maup.smart_repair(toy_precincts_df,nest_within_regions=toy_counties_df,min_rook_length=0.1)toy_precincts_repaired_county_aware_rook_to_queen_df.plot(cmap="tab20",alpha=0.7)The difference is hard to see, so let's zoom in on gap between the 4 original precincts in the upper left-hand corner.Original precincts:County-aware repair:County-aware repair with rook adjacency converted to queen:Modifiable areal unit problemThe name of this package comes from themodifiable areal unit problem (MAUP):
the same spatial data will look different depending on how you divide up the
space. Sincemaupis all about changing the way your data is aggregated and
partitioned, we have named it after the MAUP to encourage users to use the
toolkit thoughtfully and responsibly.