Package: ukpopulation
ukpopulation: UK Demographic ProjectionsLatest news: 1.2 releaseadds support for custom subnational population projectionsCustom SNPP DataAn externally generated SNPP dataset (from e.g.simim) can be registered with theukpopulationpackage and used as if it was the standard ONS/StatsWales/NRScotland/NISRA projection:>>>importukpopulation.customsnppdataasCustomSNPPData>>>customdata=pd.read_csv("custom_snpp.csv")>>>customdata.head()GEOGRAPHY_CODEGENDERC_AGEOBS_VALUEPROJECTED_YEAR_NAME0E0600000510603.020181E0600000511600.020182E0600000512624.020183E0600000513636.020184E0600000514661.02018>>>CustomSNPPData.register_custom_projection("custom_snpp",customdata,"cache_directory")WritingcustomSNPPcustom_snpptocache/ukpopulation_custom_snpp_custom_snpp.csv>>>CustomSNPPData.list_custom_projections("cache_directory")['custom_snpp']>>>The external dataset must follow the format/column name conventions as above, but can also contain extra data if required for other use. TheGENDERcolumn should only take the values 1 (male) or 2 (female); theC_AGEcolumn should contain the range 0-90 inclusive (90 meaning 90 or over).1.1 releaseadds UK household projectionsinitial support for custom SNPP variantsbetter consistency across the MYE/NPP/SNPP APIs (breaks backwards compatibility)fixes some bugs/issuesHousehold ProjectionsVersion 1.1 adds functionality is aggregates household projection data for the UK at LAD (or equivalent level). Each country's statistical agency provides a disaggregation by household type, but as there is little or no consistency between them, no attempt has (yet) been made to provide a UK-wide unified disaggregation. The year ranges mirror the year ranges for the SNPP for each country (see below). Extrapolation, or application of a national projection variant to the data is not provided at this stage.Custom SNPP VariantsGiven externally-generated data describing variations to an official projection variant, by geography (LAD) and year, this new functionality generates a full SNPP dataset, disaggregated proportionately by age and gender. The custom variant can optionally be forced to nearest-integer values, preserving the original (rounded) total.API consistencyThe public methods of theMYEData,SNPPData,NPPDataandSNHPDataclasses now consistently order arguments: firstlycategories(where appropriate) thengeogs, thenyears.The statistical agencies of the United Kingdom, that is:ONS,StatsWales,NR Scotland, andNISRA, all produce annual population estimates and projection data. Although the data are essentially the same, the quantity, format, and availability varies between agencies and datasets. All of the population projection data is available by (single year of) age and gender. Household projection data is more varied, with each country/agency producing inconsistent disaggregations of households in terms of HRP age and/or household type. For now, this package does not disaggregate by age and leaves the household type disaggregations as-is, which are:Lowest common denominatorEngland (ONS)Wales (StatsWales)Scotland (NRScotland)N.Ireland (NISRA)Single person'One person households: Female''One person households: Male''1 person''1 adult: female''1 adult: male''One adult households'Adults and children'Households with one dependent child''Households with three or more dependent children''Households with two dependent children''2 person (1 adult, 1 child)''3 person (2 adults, 1 child)''3 person (1 adult, 2 children)''4 person (2+ adults, 1+ children)''4 person (1 adult, 3 children)''5+ person (2+ adults, 1+ children)''5+ person (1 adult, 4+ children)''1 adult 1+ children''2+ adults 1+ children''One adult households with children''Other households with children'Adults only'Other households with two or more adults''2 person (No children)''3 person (No children)''4 person (No children)''5+ person (No children)''2 adults''3+ adults''Two adults without children''Other households without children'National population projections (NPP) are the responsibility of ONS who provide the data for each country within the UK, including 15 variants covering a number of possible future scenarios. The current data is based on 2018 population estimates and project a century to 2118.Subnational population projections (SNPP) are the responsibility of each country's agencies (ONS for England), and project 25 years from a base year that depends on the country in question:CountryLatest SNPP year range (as of May 2020)England2018-2043Wales2018-2043Scotland2018-2043Northern Ireland2016-2041Mid-year population estimates (MYE) are available for the entire UK by local authority, single year of age and gender, from 1991 to 2016 inclusive.Projection CoverageThe countries within the UK produce their own SNPP data, and also produce some (patchy) variant projections. The ONS currently regard these (the England ones at least) as"experimental".Scenario/VariantCodeESWNNPPPrincipalpppxxxxxHigh fertilityhppxxxLow fertilitylppxxHigh life expectancyphpxxLow life expectancyplpxxModerately high life expectancypjpxModerately low life expectancyplpxHigh migrationpphxxLow migrationpplxxHigh populationhhhxxLow populationlllxx0% future EU migrationppqx50% future EU migrationpprx150% future EU migrationppsxZero net migrationppzxxxxYoung age structurehlhOld age structurelhlReplacement fertilityrppConstant fertilitycppNo mortality improvementpnpNo changecnpLong term balanced net migrationppbxAccessibilityNomiswebprovides an API which allows relatively easy programmatic access the to data, and by far the preferred source of data. Currently not all the data is available from this source butthis may change.Nomisweb currently hosts the ONS principal NPP data for the UK, the SNPP data for England, and all of the MYE data.All other data: ONS NPP variants, SNPP data for Wales, Scotland and Northern Ireland are available in different formats from the appropriate agency's website.RationaleThe purpose of this package is to provide a unified interface to both SNPP and NPP data, including variants:encapsulating the downloading, processing and caching of the NPP and SNPP data from the various sources.consistently differentiating by age (single year, up to 90) and gender over the various datasets.providing a unified format for all the data.providing a method of synthesising SNPP variant projections using SNPP principal and NPP principal/variant projectionsproviding a method of extrapolating SNPP data using NPP dataenabling easy filtering and aggregating of of the data, e.g. extracting projections of the working-age population.Methodology and DetailData SourcesNomisweb: UK NPP by country/age/gender, England SNPP by LAD/age/gender, UK MYE by LAD/age/gender.ONS: UK NPP variants by country/age/gender.Stats Wales: Wales SNPP by LAD/age/gender.National Records of Scotland: Scotland SNPP by LAD equivalent/age/gender.Northern Ireland Statistics and Research Agency: Northern Ireland SNPP by LAD equivalent/age/gender.Data ProcessingNote that the SNPP for each country, and the NPP data may not have the same reference year. (See table above).NPP data is broken down by country (England/Wales/Scotland/Northern Ireland), for all the variant projections indicated in the second table above.Column headings and category values follow the nomisweb/census conventions:GEOGRAPHY_CODE: ONS country, LAD, or LAD-equivalent codeGENDER: 1=Male, 2=FemaleC_AGE: 0-90, where 90 represents 90 or over. To avoid ambiguity, this is an exception - nomisweb census values are typically age+1)PROJECTED_YEAR_NAME: 2014-2116OBS_VALUE: count of personsAll data are cached for swift retrieval.ExtrapolationThe SNPP data can be extrapolated using the longer-term NPP data. This is done independently for each age and gender in order to try to capture the age-gender structure of the original population. Aggregation only takes place  on the extrapolated age-gender specific values. This means that the trends shown by SNPP geographies with different age-gender structures will differ.If, conversely, the extrapolation was done on the aggregated populations for each SNPP geography, then each SNPP geography would have an identical trend, which would be identical to that of the NPP data.This methodology can be more formally explained by the following equation for the aggregate SNPPS(g,y)for a given geography and year:whereNis the NPP,ais age,sis gender,y baris a reference year (typically the final year in the SNPP data), andc(g)represents a mapping from a SNPP geography (LAD) to a NPP one (country).Projection of VariantsThe extrapolation methodology above can equally be applied to synthesising SNPP variants from SNPP principal and NPP variant data. The equivalent expression to the above is:where the subscriptsVand0refer to the variant and the principal projections respectively.InstallationRequirementsAPI KeyThis package uses theUKCensusAPIpackage to obtain some of the projection data. The package requires an API key to function correctly, seeherefor details.PackageRequires Python 3.5 or higher. Dependenciesshouldresolve automatically, but if not seetroubleshootingPyPI$python3-mpipinstallukpopulationConda$condainstallukpopulationThis assumes you have added the conda-forge channel, which can be done with$condaconfig--addchannelsconda-forgeSome of the examples (see below) plot graphs and have a dependency onmatplotlib, which can be installed with either pip or conda as appropriate.Development versionTo clone the repo and install locally:$gitclonehttps://github.com/nismod/ukpopulation
$./setup.pyinstall(substituting the URL if a fork has been taken). The test data cache directory contains a fileNOMIS_API_KEYwhich defines a dummy key for testing purposes only. The test suite can be run from the (project root dir) using:$./setup.pytestTroubleshootingEnsure you are using the correct version (>=3) of pip:$pip--version
pip9.0.1from/usr/lib/python3/dist-packages(python3.6)If not replacepipwithpip3orpython3 -m pipIf the installation has missing dependencies, try:$pipinstall-rrequirements.txt
$./setup.pyinstallIf (with python 3.5?) you encounterAttributeError:module'html5lib.treebuilders'hasnoattribute'_base'then$pipinstallhtml5lib=0.9999999should fix it. But better solution is to upgrade to python3.6If matplotlib fails to install due to a missing dependency (tkinter), this can be fixed on Debian variants by$sudoaptinstallpython3-tkIf your problem isn't addressed above, please post an issue including as much supporting information as possible.Usage ExamplesRetrieve SNPP for specific LADsDetailed dataThis example fetches the 2018 projection for Newcastle by gender and age.>>>importukpopulation.snppdataasSNPPData>>>snpp=SNPPData.SNPPData()Cache directory:  ./raw_data/
using cached LAD codes: ./raw_data/lad_codes.json
Collating SNPP data for England...
./raw_data/NM_2006_1_metadata.json found, using cached metadata...
Using cached data: ./raw_data/NM_2006_1_56aba41fc0fab32f58ead6ae91a867b4.tsv
./raw_data/NM_2006_1_metadata.json found, using cached metadata...
Using cached data: ./raw_data/NM_2006_1_dbe6c087fb46306789f7d54b125482e4.tsv
Collating SNPP data for Wales...
Collating SNPP data for Scotland...
Collating SNPP data for Northern Ireland...>>>newcastle=snpp.filter("E08000021",2018)>>>newcastle.head()C_AGE  GENDER GEOGRAPHY_CODE  OBS_VALUE  PROJECTED_YEAR_NAME
0      0       1      E08000021     1814.0                 2018
1      1       1      E08000021     1780.0                 2018
2      2       1      E08000021     1770.0                 2018
3      3       1      E08000021     1757.0                 2018
4      4       1      E08000021     1747.0                 2018Aggregated dataThis example fetches the total population projections for Newcastle from 2018 to 2039.>>>importukpopulation.snppdataasSNPPData>>>snpp=SNPPData.SNPPData()Cache directory:  ./raw_data/
using cached LAD codes: ./raw_data/lad_codes.json
Collating SNPP data for England...
./raw_data/NM_2006_1_metadata.json found, using cached metadata...
Using cached data: ./raw_data/NM_2006_1_56aba41fc0fab32f58ead6ae91a867b4.tsv
./raw_data/NM_2006_1_metadata.json found, using cached metadata...
Using cached data: ./raw_data/NM_2006_1_dbe6c087fb46306789f7d54b125482e4.tsv
Collating SNPP data for Wales...
Collating SNPP data for Scotland...
Collating SNPP data for Northern Ireland...
```python
>>> newcastle=snpp.aggregate(["GENDER", "C_AGE"], "E08000021", range(2018,2039))
>>> newcastle.head()GEOGRAPHY_CODE  PROJECTED_YEAR_NAME  OBS_VALUE
0      E08000021                 2018   299132.0
1      E08000021                 2019   300530.0
2      E08000021                 2020   301699.0
3      E08000021                 2021   302729.0
4      E08000021                 2022   303896.0Retrieve NPP data filtered by ageHere's how to get the total working-age population by country from 2016 to 2050:>>>importukpopulation.nppdataasNPPData>>>npp=NPPData.NPPData()Cache directory:  ./raw_data/
using cached LAD codes: ./raw_data/lad_codes.json
Loading NPP principal (ppp) data for England, Wales, Scotland & Northern Ireland
./raw_data/NM_2009_1_metadata.json found, using cached metadata...
Using cached data: ./raw_data/NM_2009_1_444caf1f672f0646722e389963289973.tsv>>>uk_working_age=npp.aggregate(["GENDER","C_AGE"],"ppp",NPPData.NPPData.UK,range(2016,2051),ages=range(16,75))>>>uk_working_age.head()GEOGRAPHY_CODEPROJECTED_YEAR_NAMEOBS_VALUE0E920000012016402694701E920000012017404601182E920000012018405919653E920000012019407045214E92000001202040834471And this aggregates the figures for Great Britain:>>>gb_working_age=npp.aggregate(["GEOGRAPHY_CODE","GENDER","C_AGE"],"ppp",NPPData.NPPData.GB,range(2016,2051),ages=range(16,75))>>>gb_working_age.head()PROJECTED_YEAR_NAMEOBS_VALUE02016465900141201746801693220184694421932019470630694202047201882NB SNPP data can also be filtered by age and/or gender and/or geography in the same way.Retrieve NPP variants for England & WalesFirst detailed data (by age, gender and country), then aggregated by age and gender.>>>importukpopulation.nppdataasNPPData>>>npp=NPPData.NPPData()Cachedirectory:./raw_data/usingcachedLADcodes:./raw_data/lad_codes.jsonLoadingNPPprincipal(ppp)dataforEngland,Wales,Scotland&NorthernIreland./raw_data/NM_2009_1_metadata.jsonfound,usingcachedmetadata...Usingcacheddata:./raw_data/NM_2009_1_444caf1f672f0646722e389963289973.tsv>>>high_growth=npp.detail("hhh",NPPData.NPPData.EW)>>>high_growth.head()C_AGEGENDEROBS_VALUEPROJECTED_YEAR_NAMEGEOGRAPHY_CODE0013431982016E920000011013340252017E920000012013453322018E920000013013497962019E920000014013542742020E92000001>>>high_growth_agg=npp.aggregate(["GENDER","C_AGE"],"hhh",NPPData.NPPData.EW)>>>high_growth_agg.head()GEOGRAPHY_CODEPROJECTED_YEAR_NAMEOBS_VALUE0E920000012016552680671E920000012017556601552E920000012018561150273E920000012019565687954E92000001202057019007>>>Extrapolate MYE using SNPP and NPP dataSingle AreaConstruct aggregate data for Exeter from 2011-2065:use MYE data up to 2016, aggregated by age and gender.then use SNPP data up to 2041, aggregated by age and gender.extrapolate using NPP data and Exeter's (2041) age-gender structure.aggregrate the extrapolated data by age and genderplot the data.Source CodeBulk CalculationIn this example we extrapolate and aggregrate the SNPP for every LAD in Wales:for each area,extrapolate from 2039 to 2050 using the 2039 age-gender structure.aggregate the extrapolated datma by age and gender.append to full dataset.save Wales dataset as csv:GEOGRAPHY_CODEPROJECTED_YEAR_NAMEOBS_VALUEW060000112040262903.24103359133W060000112041262933.2340468692W060000112042263162.3661643687W060000112043263332.96819104964W060000112044263593.29826455784W060000112045263923.03553008236W060000112046264243.6253810904W060000112047264168.2113917932W060000112048264211.4576059673.........Source CodeConstruct an SNPP variant by applying NPP variant to a specific LADHere we apply the "hhh" (high growth) and "lll" (low growth) NPP variants to the SNPP data for Newcastle:calculate the principal ("ppp") projection by simply aggregrating the SNPP data for Newcastle, 2018-2039, by age and gender.calculate the variants by weighting the unaggregated data (i.e. by age and gender) by the ratio of the NPP variant/principal.aggregrate the variant data by age and gender.plot the results.Source CodeExtrapolating an SNPP variantHere we build on the examples above by not only applying the NPP variant, but extrapolating too. The process first involves extrapolating the SNPP by the NPP principal variant. The extrapolated data then has the variant adjustments applied to it.Source CodeComparing household and population projections for a single LADIn this example we simply plot the aggregate household projections for Newcastle against the (principal) population projection. You can see that population growth starts to tail off more than the household growth. This suggests a decrease in household size. Further inspection of the data should confirm this.Source CodeCode DocumentationPackage documentation can be viewed like so:importukpopulation.myedataasMYEDatahelp(MYEData)importukpopulation.nppdataasNPPDatahelp(NPPData)importukpopulation.snppdataasSNPPDatahelp(SNPPData)importukpopulation.snhpdataasSNHPDatahelp(SNHPData)ContributionsContributions to this package are welcomed via the usual pull request mechanism.SupportIf you encounter a bug, feel the documentation is incorrect or incomplete, or want to suggest new features, please post an issue in theissuestab.AcknowledgementsThis package was developed as a component of the EPSRC-fundedMISTRALprogramme, part of the Infrastructure Transitions Research Consortium.