Package: the-census
The CensusWant to work with US Census data? Look no further.Getting startedView all datasetsIf you you're not sure what Census dataset you're interested in, the following code will take care of you:fromthe_censusimportCensusCensus.list_available_datasets()This will present you with a pandas DataFrame listing all available datasets from the US Census API. (This includes only aggregate datasets, as they other types [of which there are very few] don't play nice with the client).Help with terminologySome of the terms used in the data returned can be a bit opaque. To get a clearer sense of what some of those mean, run this:Census.help()This will print out links to documentation for various datasets, along with what their group/variable names mean, and how statistics were calculated.Selecting a datasetBefore getting started, you need toget a Census API key, and set the following the environment variableCENSUS_API_KEYto whatever that key is, either withexportCENSUS_API_KEY=<yourkey>or in a.envfile:CENSUS_API_KEY=<yourkey>Say you're interested in the American Community Survey 1-year estimates for 2019. Look up the dataset and survey name in the table provided bylist_available_datasets, and execute the following code:>>>fromthe_censusimportCensus>>>Census(year=2019,dataset="acs",survey="acs1")<Censusyear=2019dataset=acssurvey=acs1>Thedatasetobject will now let you query any census data for the the ACS 1-year estimates of 2019. We'll now dive into how to query this dataset with the tool. However, if you aren't familiar with dataset "architecture", check outthissection.Arguments toCensusThis is the signature ofCensus:classCensusdef__init__(self,year:int,dataset:str="acs",survey:str="acs1",cache_dir:str=CACHE_DIR,# cacheshould_load_from_existing_cache:bool=False,should_cache_on_disk:bool=False,replace_column_headers:bool=True,log_file:str=DEFAULT_LOG_FILE):# census.logpassyear: the year of the datasetdataset: type of the dataset, specified bylist_available_datasetssurvey: type of the survey, specified bylist_available_datasetscache_dir: if you opt in to on-disk caching (more on this below), the name of the directory in which to store cached datashould_load_from_existing_cache: if you have cached data from a previous session, this will reload cached data into theCensusobject, instead of hitting the Census API when that data is queriedshould_cache_on_disk: whether or not to cache data on disk, to avoid repeat API calls. The following data will be cached:Supported GeographiesGroup codesVariable codesreplace_column_headers: whether or not to replace column header names for variables with more intelligible names instead of their codeslog_file: name of the file in which to store logging informationA note on cachingWhile on-disk caching is optional, this tool, by design, performs in-memory caching. So a call tocensus.get_groups()will hit the Census API one time at most. All subsequent calls will retrieve the value cached in-memory.Making queriesSupported geographiesGetting thesupported geographiesfor a dataset as as simple as this:census.get_supported_geographies()This will output a DataFrame will all possible supported geographies (e.g., if I can query all school districts across all states).Supported geographies autocompleteIf you don't want to have to keep on typing supported geographies after this, you can use tab-completion in Jupyter by typing:census.supported_geographies.<TAB>Geography codesIf you decide you want to query a particular geography (e.g., a particular school district within a particular state), you'll need theFIPScodes for that school district and state.So, if you're interested in all school districts in Colorado, here's what you'd do:Get FIPS codes for all states:fromthe_censusimportGeoDomaincensus.get_geography_codes(GeoDomain("state","*"))Or, if you don't want to importGeoDomain, and prefer to use tuples:census.get_geography_codes(("state","*"))Get FIPS codes for all school districts within Colorado (FIPS code08):census.get_geography_codes(GeoDomain("school district","*"),GeoDomain("state","08"))Or, if you don't want to importGeoDomain, and prefer to use tuples:census.get_geography_codes(("school district","*"),("state","08"))Note that geography code queries must follow supported geography guidelines.GroupsWant to figure out what groups are available for your dataset? No problem. This will do the trick for ya:census.get_groups()...and you'll get a DataFrame with all groups for your census.Searching groupscensus.get_groups()will return a lot of data that might be difficult to slog through. In that case, run this:census.search_groups(regex=r"my regex")and you'll get a filtered DataFrame with matches to your regex.Groups autocompleteIf you're working in a Jupyter notebook and have autocomplete enabled, runningcensus.groups., followed by a tab, will trigger an autocomplete menu for possible groups by their name (as opposed to their code, which doesn't have any inherent meaning in and of itself).census.groups.SexByAge# code for this groupVariablesYou can either get a DataFrame of variables based on a set of groups:census.get_variables_by_group(census.groups.SexByAge,census.groups.MedianAgeBySex)Or, you can get a DataFrame with all variables for a given dataset:census.get_all_variables()This second operation, can, however, take a lot of time.Searching variablesSimilar to groups, you can search variables by regex:census.search_variables(r"my regex")And, you can limit that search to variables of a particular group or groups:census.search_variables(r"my regex",census.groups.SexByAge)Variables autocompleteVariables also support autocomplete for their codes, as with groups.census.variables.EstimateTotal_B01001# code for this variable(These names must be suffixed with the group code, since, while variable codes are unique across groups, their names are not unique across groups.)StatisticsOnce you have the variables you want to query, along with the geography you're interested in, you can now make statistics queries from your dataset:fromthe_censusimportGeoDomainvariables=census.get_variables_for_group(census.groups.SexByAge)census.get_stats(variables["code"].tolist(),GeoDomain("school district","*"),GeoDomain("state","08"))Or, if you'd rather use tuples instead ofGeoDomain:variables=census.get_variables_for_group(census.groups.SexByAge)census.get_stats(variables["code"].tolist(),("school district","*"),("state","08"))General notes on autocompleteJupyter notebook/lab has been having an issue with autocomplete lately (seethis GitHub issue), so running the following in your environment should help you take advantage of the autocomplete offerings of this package:pip install jedi==0.17.2Dataset "architecture"US Census datasets have 3 primary components:GroupsVariablesSupported GeographiesGroupsA group is a "category" of data gathered for a particular census. For example, theSEX BY AGEgroup would provide breakdowns of gender and age demographics in a given region in the United States.Some of these groups' names, however, are a not as clear asSEX BY AGE. In that case, I recommend heading over to the survey in question'stechnical documentationwhich elaborates on what certain terms mean with respect to particular groups. Unfortunately, the above link might be complicated to navigate, but if you're looking for ACS group documentation,here'sa handy link.(You can also get these links by runningCensus.help().)VariablesVariables measure a particular data-point. While they have their own codes, you might find variables which share the same name (e.g.,Estimate!!:Total:). This is because each variable belongs to agroup. So, theEstimate!!:Totalvariable forSEX BY AGEgroup is the total of all queried individuals in that group; but theEstimate!!:Totalvariable forPOVERTY STATUS IN THE PAST 12 MONTHS BY AGEgroup is the total of queried individuals forthatgroup. (It's important when calculating percentages that you work within the same group. So if I want the percent of men in the US, whose total number I got fromSEX BY AGEI should use theEstimate!!:Total:of that group as my denominator, and not theEstimate!!:Total:of thePOVERTY STATUSgroup).Variables on their own, however, do nothing. They mean something only when you query a particulargeographyfor them.Supported GeographiesSupported geographies dictate the kinds of queries you can make for a given census. For example, in the ACS-1, I might be interested in looking at stats across all school districts. The survey's supported geographies will tell me if I can actually do that; or, if I need to refine my query to look at school districts in a given state or smaller region.