Repository: john-sandall/maven

Issue:
Clean up 2015 results to use HoC data
The currently processed 2015 results data isn't even that nice. Integrate further cleaning and ID mapping from the [SixFifty model](https://github.com/six50/hackathon/blob/master/model/SixFifty%20GE2017%20-%201%20-%20Create%20Model%20Dataset.ipynb).

This currently uses the Electoral Commission's results dataset. Earlier this year, they stopped hosting results and this is now officially published by the House of Commons library at parliament.uk. We need to bring this code up-to-date so it uses the new data (which is in a different format!) for the 2015 results: https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-7186

The goal is the same, we want to turn this into a dataset with the same format as the current code with columns something like:
```
Press Association ID Number    Constituency ID    Constituency Name    Constituency Type    County    Region ID    Region    Country    Election Year    Electorate    Valid Votes    con    lab    ld    ukip    grn    snp    pc    other    con_pc    lab_pc    ld_pc    ukip_pc    grn_pc    snp_pc    pc_pc    other_pc    geo    winner
```
The new dataset also switches to using just ons_id (which is the same as Constituency ID in the previous cleaned results) and not Press Association ID Number, this is fine, you can drop that column.

Discussion:

Patch:
diff --git a/README.md b/README.md
index b38b5c4..27b5347 100644
--- a/README.md
+++ b/README.md
@@ -28,9 +28,10 @@ Data dictionaries for all datasets are available by clicking on the dataset's na
 
 | Dataset | Description | Date | Source | Licence |
 | -- | -- | -- | -- | -- |
+| [**`general-election/UK/2010/results`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | UK 2010 General Election results | 6th May 2010 | [House of Commons Library](https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-8647) | [Open Parliament Licence v3.0](https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/) |
+| [**`general-election/UK/2015/results`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | UK 2015 General Election results | 7th May 2015 | [House of Commons Library](https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-8647) | [Open Parliament Licence v3.0](https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/) |
+| [**`general-election/UK/2017/results`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | UK 2015 General Election results | 8th June 2017 | [House of Commons Library](https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-8647) | [Open Parliament Licence v3.0](https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/) |
 | [**`general-election/UK/2015/model`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | Model-ready datasets for forecasting the 2015 and 2017 UK General Elections | 2010, 2015 & 2017 data | [SixFifty](https://github.com/six50/) | Mixed |
-| [**`general-election/UK/2010/results`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | UK 2010 General Election results | 6th May 2010 | [Electoral Commission](http://www.electoralcommission.org.uk/our-work/our-research/electoral-data) | [Open Government Licence v2.0](http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/) |
-| [**`general-election/UK/2015/results`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | UK 2015 General Election results | 7th May 2015 | [Electoral Commission](http://www.electoralcommission.org.uk/our-work/our-research/electoral-data) | [Open Government Licence v2.0](http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/) |
 | [**`general-election/UK/polls`**](https://github.com/john-sandall/maven/tree/master/maven/datasets/general_election) | UK General Election opinion polling | May 2005 - June 2017 | [SixFifty](https://github.com/six50/pipeline/tree/master/data/polls/) | Unknown |
 
 
diff --git a/maven/__init__.py b/maven/__init__.py
index 2c54cca..100a5bd 100644
--- a/maven/__init__.py
+++ b/maven/__init__.py
@@ -1,1 +1,2 @@
+from . import utils
 from .get import get
diff --git a/maven/datasets/general_election/README.md b/maven/datasets/general_election/README.md
index 31e6725..bc2c737 100644
--- a/maven/datasets/general_election/README.md
+++ b/maven/datasets/general_election/README.md
@@ -46,76 +46,55 @@ We aim to source our data directly from the most authorative data provider, fall
 | `Region_West Midlands` | int | Dummied region | `0` |
 | `Region_Yorkshire and the Humber` | int | Dummied region | `0` |
 | `won_here_last` | int | Did this party win this constituency in 2010? | `0` |
-| `turnout Midlands` | float | Turnout in 2015 | `0.608953932` |
+| `turnout` | float | Turnout in 2015 | `0.608953932` |
 
 
 #### **`general-election/UK/2010/results`**
-| Column | Type | Description | Example |
-| -- | -- | -- | -- |
-| `Press Association Reference` | int | Standardised constituency identifier | `1` |
-| `Constituency Name` | str | Constituency name | `Aberavon` |
-| `Region` | str | Region:{`East Midlands`, `Eastern`, `London`, `North East`, `North West`, `Northern Ireland`, `Scotland`, `South East`, `South West`, `Wales`, `West Midlands`, `Yorkshire and The Humber`} | `Wales` |
-| `Election Year` | int | Year | Always `2010` |
-| `Electorate` | int | Electorate | `50838` |
-| `Votes` | int | Valid votes counted | `30958` |
-| `con` | int | Votes for the Conservative party in this constituency | `4411` |
-| `lab` | int | Votes for the Labour party in this constituency | `16073` |
-| `ld` | int | Votes for the Liberal Democrat party in this constituency | `5034` |
-| `ukip` | int | Votes for the UKIP party in this constituency | `489` |
-| `grn` | int | Votes for the Green party in this constituency | `0` |
-| `dup` | int | Votes for the DUP in this constituency | `0` |
-| `sf` | int | Votes for Sinn Féin in this constituency | `0` |
-| `sdlp` | int | Votes for the SDLP in this constituency | `0` |
-| `snp` | int | Votes for the SNP in this constituency | `0` |
-| `pc` | int | Votes for Plaid Cymru in this constituency | `2198` |
-| `other` | int | Votes for all other parties (combined) in this constituency | `2753` |
-| `con_pc` | float | Percentage voteshare for the Conservative party in this constituency | `0.142483365` |
-| `lab_pc` | float | Percentage voteshare for the Labour party in this constituency | `0.519187286` |
-| `ld_pc` | float | Percentage voteshare for the Liberal Democrat party in this constituency | `0.162607404` |
-| `ukip_pc` | float | Percentage voteshare for the UKIP party in this constituency | `0.015795594` |
-| `grn_pc` | float | Percentage voteshare for the Green party in this constituency | `0` |
-| `dup_pc` | float | Percentage voteshare for the DUP in this constituency | `0` |
-| `sf_pc` | float | Percentage voteshare for Sinn Féin party in this constituency | `0` |
-| `sdlp_pc` | float | Percentage voteshare for the SDLP in this constituency | `0` |
-| `snp_pc` | float | Percentage voteshare for the SNP in this constituency | `0` |
-| `pc_pc` | float | Percentage voteshare for Plaid Cymru in this constituency | `0.070999419` |
-| `other_pc` | float | Percentage voteshare for all other parties (combined) in this constituency | `0.088926933` |
-| `geo` | str | Model-specific regions: {`Wales`, `Scotland`, `England_not_london`, `NI`, `London`}  | `Wales` |
-| `winner` | str | Winning party with most votes in this constituency  | `lab` |
+| Column            | Type  | Description | Example |
+| --                | -- | -- | -- |
+| `ons_id`          | str   | Standardised constituency identifier | `E14000530` |
+| `constituency`    | str   | Constituency name | `ALDERSHOT` |
+| `county`          | str   | County name | `Hampshire` |
+| `region`          | str   | Region:{`East Midlands`, `Eastern`, `London`, `North East`, `North West`, `Northern Ireland`, `Scotland`, `South East`, `South West`, `Wales`, `West Midlands`, `Yorkshire and The Humber`} | `South East` |
+| `country`         | str   | Country:{`England`, `Northern Ireland`, `Scotland`, `Wales`} | `England` |
+| `electorate`      | int   | Electorate | `71465` |
+| `total_votes`     | int   | Total valid votes counted in this constituency | `45384` |
+| `turnout`         | float | Turnout in this constituency | `0.635052123` |
+| `party`           | str   | Name of political party (lower-cased & abbreviated) | `con` |
+| `votes`           | int   | Votes for this party | `21203` |
+| `voteshare`       | float | Vote share for this party within the constituency | `0.467191081` |
 
 
 #### **`general-election/UK/2015/results`**
-| Column | Type | Description | Example |
-| -- | -- | -- | -- |
-| `Press Association ID Number` | int | Standardised constituency identifier | `1` |
-| `Constituency ID` | str | ONS constituency identifier | `W07000049` |
-| `Constituency Name` | str | Name | `Aberavon` |
-| `Constituency Type` | str | Type: {`Borough`, `Burgh`, `County`} | `County` |
-| `County` | str | Country | `West Glamorgan` |
-| `Region ID` | str | ONS region identifier | `W92000004` |
-| `Region` | str | Region:{`East`, `East Midlands`, `London`, `North East`, `North West`, `Northern Ireland`, `Scotland`, `South East`, `South West`, `Wales`, `West Midlands`, `Yorkshire and The Humber`} | `Wales` |
-| `Country` | str | Country: {`England`, `Northern Ireland`, `Scotland`, `Wales`} | `Wales` |
-| `Election Year` | int | Year | Always `2015` |
-| `Electorate` | int | Electorate | `49821` |
-| `Valid Votes` | int | Valid votes counted | `31523` |
-| `con` | int | Votes for the Conservative party in this constituency | `3742` |
-| `lab` | int | Votes for the Labour party in this constituency | `15416` |
-| `ld` | int | Votes for the Liberal Democrat party in this constituency | `1397` |
-| `ukip` | int | Votes for the UKIP party in this constituency | `4971` |
-| `grn` | int | Votes for the Green party in this constituency | `711` |
-| `snp` | int | Votes for SNP in this constituency | `0` |
-| `pc` | int | Votes for Plaid Cymru in this constituency | `3663` |
-| `other` | int | Votes for all other parties (combined) in this constituency | `1623` |
-| `con_pc` | float | Percentage voteshare for the Conservative party in this constituency | `0.118707` |
-| `lab_pc` | float | Percentage voteshare for the Labour party in this constituency | `0.48904` |
-| `ld_pc` | float | Percentage voteshare for the Liberal Democrat party in this constituency | `0.0443168` |
-| `ukip_pc` | float | Percentage voteshare for the UKIP party in this constituency | `0.157694` |
-| `grn_pc` | float | Percentage voteshare for the Green party in this constituency | `0.022555` |
-| `snp_pc` | float | Percentage voteshare for SNP in this constituency | `0` |
-| `pc_pc` | float | Percentage voteshare for Plaid Cymru in this constituency | `0.116201` |
-| `other_pc` | float | Percentage voteshare for all other parties (combined) in this constituency | `0.0514862` |
-| `winner` | str | Winning party with most votes in this constituency  | `lab` |
-| `geo` | str | Model-specific regions: {`Wales`, `Scotland`, `England_not_london`, `NI`, `London`}  | `Wales` |
+| Column            | Type  | Description | Example |
+| --                | -- | -- | -- |
+| `ons_id`          | str   | Standardised constituency identifier | `E14000530` |
+| `constituency`    | str   | Constituency name | `ALDERSHOT` |
+| `county`          | str   | County name | `Hampshire` |
+| `region`          | str   | Region:{`East Midlands`, `Eastern`, `London`, `North East`, `North West`, `Northern Ireland`, `Scotland`, `South East`, `South West`, `Wales`, `West Midlands`, `Yorkshire and The Humber`} | `South East` |
+| `country`         | str   | Country:{`England`, `Northern Ireland`, `Scotland`, `Wales`} | `England` |
+| `electorate`      | int   | Electorate | `72430` |
+| `total_votes`     | int   | Total valid votes counted in this constituency | `46191` |
+| `turnout`         | float | Turnout in this constituency | `0.637732984` |
+| `party`           | str   | Name of political party (lower-cased & abbreviated) | `con` |
+| `votes`           | int   | Votes for this party | `23369` |
+| `voteshare`       | float | Vote share for this party within the constituency | `0.505921067` |
+
+
+#### **`general-election/UK/2017/results`**
+| Column            | Type  | Description | Example |
+| --                | -- | -- | -- |
+| `ons_id`          | str   | Standardised constituency identifier | `E14000530` |
+| `constituency`    | str   | Constituency name | `ALDERSHOT` |
+| `county`          | str   | County name | `Hampshire` |
+| `region`          | str   | Region:{`East Midlands`, `Eastern`, `London`, `North East`, `North West`, `Northern Ireland`, `Scotland`, `South East`, `South West`, `Wales`, `West Midlands`, `Yorkshire and The Humber`} | `South East` |
+| `country`         | str   | Country:{`England`, `Northern Ireland`, `Scotland`, `Wales`} | `England` |
+| `electorate`      | int   | Electorate | `76205` |
+| `total_votes`     | int   | Total valid votes counted in this constituency | `48950` |
+| `turnout`         | float | Turnout in this constituency | `0.642346303` |
+| `party`           | str   | Name of political party (lower-cased & abbreviated) | `con` |
+| `votes`           | int   | Votes for this party | `26950` |
+| `voteshare`       | float | Vote share for this party within the constituency | `0.550561798` |
 
 
 #### **`general-election/UK/polls`**
diff --git a/maven/datasets/general_election/__init__.py b/maven/datasets/general_election/__init__.py
index 8b1f584..39aed97 100644
--- a/maven/datasets/general_election/__init__.py
+++ b/maven/datasets/general_election/__init__.py
@@ -1,15 +1,7 @@
 from .uk_2010_results import UK2010Results
 from .uk_2015_model import UK2015Model
 from .uk_2015_results import UK2015Results
-from .uk_2015_results_HoC import UK2015ResultsHoC
-from .uk_2017_results_HoC import UK2017ResultsHoC
+from .uk_2017_results import UK2017Results
 from .uk_polls import UKPolls
 
-__all__ = [
-    'UK2010Results',
-    'UK2015Model',
-    'UK2015Results',
-    'UK2015ResultsHoC',
-    'UK2017ResultsHoC',
-    'UKPolls'
-]
+__all__ = ["UK2010Results", "UK2015Model", "UK2015Results", "UK2017Results", "UKPolls"]
diff --git a/maven/datasets/general_election/base.py b/maven/datasets/general_election/base.py
new file mode 100644
index 0000000..c63655d
--- /dev/null
+++ b/maven/datasets/general_election/base.py
@@ -0,0 +1,32 @@
+"""
+Base classes.
+"""
+
+import os
+import warnings
+from pathlib import Path
+
+import requests
+
+
+class UKResults:
+    """Handles results data for UK General Elections."""
+
+    def __init__(self, directory):
+        self.directory = Path(directory)
+        self.sources = []
+        self.verbose_name = ""
+
+    def retrieve(self):
+        target = self.directory / "raw"
+        os.makedirs(target, exist_ok=True)  # create directory if it doesn't exist
+        for url, filename in self.sources:
+            response = requests.get(url + filename)
+            if response.status_code == 200:
+                with open(target / filename, "wb") as file:
+                    file.write(response.content)
+                print(f"Successfully downloaded raw data into {target.resolve()}")
+                return
+            else:
+                warnings.warn(f"Received status 404 when trying to retrieve {url}{filename}")
+        raise RuntimeError(f"Unable to download {self.verbose_name} data.")
diff --git a/maven/datasets/general_election/uk_2010_results.py b/maven/datasets/general_election/uk_2010_results.py
index 3db32f0..81b4756 100644
--- a/maven/datasets/general_election/uk_2010_results.py
+++ b/maven/datasets/general_election/uk_2010_results.py
@@ -1,145 +1,53 @@
 """
 Results data for the United Kingdom's 2010 General Election.
 
+Usage:
+    >>> import maven
+    >>> maven.get('general-election/UK/2010/results', data_directory='./data/')
+
+
 Sources:
+    - http://researchbriefings.files.parliament.uk/documents/CBP-8647/1918-2017election_results.csv
+        - From https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-8647
+
+Deprecated sources:
     - http://www.electoralcommission.org.uk/__data/assets/excel_doc/0003/105726/GE2010-results-flatfile-website.xls
     - https://s3-eu-west-1.amazonaws.com/sixfifty/GE2010-results-flatfile-website.xls
 
-Usage:
-    > import maven
-    > maven.get('general-election/UK/2010/results', data_directory='./data/')
+Notes:
+    - GE2010-results-flatfile-website.xls is currently the only known source with a full list of votes for ALL parties.
 """
+
 import os
-import warnings
 from pathlib import Path
 
-import pandas as pd
-import requests
+from maven import utils
+from maven.datasets.general_election.base import UKResults
 
 
-class UK2010Results:
+class UK2010Results(UKResults):
     """Handles results data for the United Kingdom's 2010 General Election."""
 
-    def __init__(self, directory=Path('data/general-election/UK/2010/results')):
+    def __init__(self, directory=Path("data/general-election/UK/2010/results")):
         self.directory = Path(directory)
         self.sources = [
-            ('https://s3-eu-west-1.amazonaws.com/sixfifty/', 'GE2010-results-flatfile-website.xls',),
+            (
+                "http://researchbriefings.files.parliament.uk/documents/CBP-8647/",
+                "1918-2017election_results_by_pcon.xlsx",
+            ),
         ]
-
-    def retrieve(self):
-        """Retrieve results data for the United Kingdom's 2010 General Election."""
-        target = self.directory / 'raw'
-        os.makedirs(target, exist_ok=True)  # create directory if it doesn't exist
-        for url, filename in self.sources:
-            response = requests.get(url + filename)
-            if response.status_code == 200:
-                with open(target / filename, 'wb') as file:
-                    file.write(response.content)
-                print(f'Successfully downloaded raw data into {target.resolve()}')
-                return
-            warnings.warn(f'Received status 404 when trying to retrieve {url}{filename}')
-        raise RuntimeError('Unable to download UK 2010 General Election results data.')
+        self.verbose_name = "UK 2010 General Election results"
 
     def process(self):
         """Process results data for the United Kingdom's 2010 General Election."""
         filename = self.sources[0][1]
-        processed_results_filename = 'general_election-uk-2010-results.csv'
-        processed_results_full_filename = 'general_election-uk-2010-results-full.csv'
-        processed_results_location = self.directory / 'processed' / processed_results_filename
-        processed_results_full_location = self.directory / 'processed' / processed_results_full_filename
-        os.makedirs(self.directory / 'processed', exist_ok=True)  # create directory if it doesn't exist
-
-        ##########################
-        # GENERAL ELECTION RESULTS
-        ##########################
-        print(f'Read and clean {filename}')
-
-        # Import general election results
-        results = pd.read_excel(self.directory / 'raw' / filename, sheet_name='Party vote share')
-
-        # Remove rows where Constituency Name is blank (one row only - the last row of the sheet)
-        blank_rows = results['Constituency Name'].isnull()
-        results = results[-blank_rows].copy()
-
-        # Set NA vals to zero (NA => zero votes for that party within the constituency)
-        for party_vote_result in results.columns[6:]:  # first 6 cols are not party votes
-            results[party_vote_result] = results[party_vote_result].fillna(0)
-        assert results.shape == (650, 144)
-
-        # Save this for convenience
-        results_full = results.copy()
-
-        ############################
-        # ADDITIONAL TRANSFORMATIONS
-        ############################
-
-        # Filter to metadata cols + parties of interest (from ~139 parties to ~11).
-        parties_lookup = {
-            'Con': 'con',
-            'Lab': 'lab',
-            'LD': 'ld',
-            'UKIP': 'ukip',
-            'Grn': 'grn',
-            # Northern Ireland
-            'DUP': 'dup',
-            'SF': 'sf',
-            'SDLP': 'sdlp',
-            # Scotland
-            'SNP': 'snp',
-            # Wales
-            'PC': 'pc',
-            # Other
-            'Other': 'other',
-        }
-        other_parties = list(set(results.columns) - set(results.columns[:6]) - set(parties_lookup.keys()))
-
-        results['Other'] = results.loc[:, other_parties].sum(axis=1)
-        results = results.loc[:, list(results.columns[:6]) + list(parties_lookup.keys())]
-
-        # Rename parties (if in parties_lookup, else ignore)
-        # TODO: Cleaner with .rename()
-        results.columns = [parties_lookup[x] if x in parties_lookup else x for x in results.columns]
-
-        # Calculate constituency level vote share % (pc = percent)
-        for party in parties_lookup.values():
-            results[party + '_pc'] = results[party] / results['Votes']
-
-        # Create PANO -> geo lookup
-        results['geo'] = results.Region.map(
-            {
-                'East Midlands': 'England_not_london',
-                'Eastern': 'England_not_london',
-                'London': 'London',
-                'North East': 'England_not_london',
-                'North West': 'England_not_london',
-                'Northern Ireland': 'NI',
-                'Scotland': 'Scotland',
-                'South East': 'England_not_london',
-                'South West': 'England_not_london',
-                'Wales': 'Wales',
-                'West Midlands': 'England_not_london',
-                'Yorkshire and the Humber': 'England_not_london',
-            }
-        )
-        assert results.loc[237.0, 'geo'] == 'London'
-
-        # Who won?
-        def winner(row):
-            """Return winning party for given row of constituency outcomes."""
-            # Need to remove Other as this represents multiple parties so (usually) not the actual FPTP winner.
-            all_parties = set(results_full.columns[6:]) - set(['Other'])
-            winning_party = row[all_parties].sort_values(ascending=False).index[0]
-            if winning_party in parties_lookup.keys():
-                winning_party = parties_lookup[winning_party]
-            elif winning_party == 'Speaker':
-                winning_party = 'other'
-            return winning_party
+        processed_results_filename = "general_election-uk-2010-results.csv"
+        processed_results_location = self.directory / "processed" / processed_results_filename
+        os.makedirs(self.directory / "processed", exist_ok=True)  # create directory if it doesn't exist
 
-        results['winner'] = results_full.apply(winner, axis=1)
-        assert results.winner.value_counts()[0] == 306  # Check Conservatives won 306 seats in 2010
+        # Process
+        results = utils.process_hoc_sheet(input_file=filename, data_dir=self.directory, sheet_name="2010")
 
-        # EXPORT
-        print(f'Exporting dataset to {processed_results_location.resolve()}')
+        # Export
+        print(f"Exporting dataset to {processed_results_location.resolve()}")
         results.to_csv(processed_results_location, index=False)
-        print(f'Exporting dataset to {processed_results_full_location.resolve()}')
-        results_full.to_csv(processed_results_full_location, index=False)
diff --git a/maven/datasets/general_election/uk_2015_model.py b/maven/datasets/general_election/uk_2015_model.py
index 9b9a2e3..b560cb2 100644
--- a/maven/datasets/general_election/uk_2015_model.py
+++ b/maven/datasets/general_election/uk_2015_model.py
@@ -17,38 +17,38 @@ import maven
 class UK2015Model:
     """Generates model-ready data for the United Kingdom's 2015 General Election."""
 
-    def __init__(self, directory=Path('data/general-election/UK/2015/model')):
+    def __init__(self, directory=Path("data/general-election/UK/2015/model")):
         self.directory = Path(directory)
 
     def retrieve(self):
         """Will check to see if this already exists in directory tree, otherwise puts the
            datasets there by executing the necessary code from within this repo."""
-        destination_target = self.directory / 'raw'
+        destination_target = self.directory / "raw"
         os.makedirs(destination_target, exist_ok=True)  # create directory if it doesn't exist
-        data_directory = (self.directory / '..' / '..' / '..' / '..').resolve()  # sensible guess
+        data_directory = (self.directory / ".." / ".." / ".." / "..").resolve()  # sensible guess
         data = [
             # (identifier, type, filename)
-            ('general-election/UK/2010/results', 'processed', 'general_election-uk-2010-results.csv',),
-            ('general-election/UK/2010/results', 'processed', 'general_election-uk-2010-results-full.csv',),
-            ('general-election/UK/2015/results', 'processed', 'general_election-uk-2015-results.csv',),
-            ('general-election/UK/2015/results', 'processed', 'general_election-uk-2015-results-full.csv',),
-            ('general-election/UK/polls', 'processed', 'general_election-uk-polls.csv'),
-            ('general-election/UK/polls', 'processed', 'general_election-london-polls.csv',),
-            ('general-election/UK/polls', 'processed', 'general_election-scotland-polls.csv',),
-            ('general-election/UK/polls', 'processed', 'general_election-wales-polls.csv',),
-            ('general-election/UK/polls', 'processed', 'general_election-ni-polls.csv'),
+            ("general-election/UK/2010/results", "processed", "general_election-uk-2010-results.csv",),
+            ("general-election/UK/2010/results", "processed", "general_election-uk-2010-results-full.csv",),
+            ("general-election/UK/2015/results", "processed", "general_election-uk-2015-results.csv",),
+            ("general-election/UK/2015/results", "processed", "general_election-uk-2015-results-full.csv",),
+            ("general-election/UK/polls", "processed", "general_election-uk-polls.csv"),
+            ("general-election/UK/polls", "processed", "general_election-london-polls.csv",),
+            ("general-election/UK/polls", "processed", "general_election-scotland-polls.csv",),
+            ("general-election/UK/polls", "processed", "general_election-wales-polls.csv",),
+            ("general-election/UK/polls", "processed", "general_election-ni-polls.csv"),
         ]
         for identifier, data_type, filename in data:
-            source_target = f'{identifier}/{data_type}/{filename}'
+            source_target = f"{identifier}/{data_type}/{filename}"
             if not (data_directory / source_target).is_file():
-                print(f'Dataset {identifier} not found - retrieving now')
+                print(f"Dataset {identifier} not found - retrieving now")
                 maven.get(identifier, data_directory=data_directory)
             shutil.copyfile(src=data_directory / source_target, dst=destination_target / filename)
 
     def process(self):
         """Process results data from the United Kingdom's 2010 and 2015 General Elections
            into a single model-ready dataset for predicting the 2015 General Election."""
-        processed_directory = self.directory / 'processed'
+        processed_directory = self.directory / "processed"
         os.makedirs(processed_directory, exist_ok=True)  # create directory if it doesn't exist
 
         # TODO: Refactor these sections into functions to make it easier to read.
@@ -58,42 +58,42 @@ class UK2015Model:
         #############
 
         # Import general election results
-        ge_2010 = pd.read_csv(self.directory / 'raw' / 'general_election-uk-2010-results.csv')
-        ge_2010_full = pd.read_csv(self.directory / 'raw' / 'general_election-uk-2010-results-full.csv')
-        ge_2015 = pd.read_csv(self.directory / 'raw' / 'general_election-uk-2015-results.csv')
-        ge_2015_full = pd.read_csv(self.directory / 'raw' / 'general_election-uk-2015-results-full.csv')
-        polls = pd.read_csv(self.directory / 'raw' / 'general_election-uk-polls.csv')
+        ge_2010 = pd.read_csv(self.directory / "raw" / "general_election-uk-2010-results.csv")
+        ge_2010_full = pd.read_csv(self.directory / "raw" / "general_election-uk-2010-results-full.csv")
+        ge_2015 = pd.read_csv(self.directory / "raw" / "general_election-uk-2015-results.csv")
+        ge_2015_full = pd.read_csv(self.directory / "raw" / "general_election-uk-2015-results-full.csv")
+        polls = pd.read_csv(self.directory / "raw" / "general_election-uk-polls.csv")
 
         # Check constituencies are mergeable
         assert (
-            set(ge_2010['Press Association Reference']).difference(set(ge_2015['Press Association ID Number'])) == set()
+            set(ge_2010["Press Association Reference"]).difference(set(ge_2015["Press Association ID Number"])) == set()
         )
         assert (
-            set(ge_2015['Press Association ID Number']).difference(set(ge_2010['Press Association Reference'])) == set()
+            set(ge_2015["Press Association ID Number"]).difference(set(ge_2010["Press Association Reference"])) == set()
         )
-        assert len(ge_2010) == len(ge_2010['Press Association Reference']) == 650
-        assert len(ge_2015) == len(ge_2015['Press Association ID Number']) == 650
+        assert len(ge_2010) == len(ge_2010["Press Association Reference"]) == 650
+        assert len(ge_2015) == len(ge_2015["Press Association ID Number"]) == 650
 
         # Construct some lookups of the parties we want to model
         parties_lookup_2010 = {
-            'Con': 'con',
-            'Lab': 'lab',
-            'LD': 'ld',
-            'UKIP': 'ukip',
-            'Grn': 'grn',
-            'Other': 'other',
+            "Con": "con",
+            "Lab": "lab",
+            "LD": "ld",
+            "UKIP": "ukip",
+            "Grn": "grn",
+            "Other": "other",
         }
         parties_15 = list(parties_lookup_2010.values())
 
         parties_lookup_2015 = {
-            'C': 'con',
-            'Lab': 'lab',
-            'LD': 'ld',
-            'UKIP': 'ukip',
-            'Green': 'grn',
-            'SNP': 'snp',
-            'PC': 'pc',
-            'Other': 'other',
+            "C": "con",
+            "Lab": "lab",
+            "LD": "ld",
+            "UKIP": "ukip",
+            "Green": "grn",
+            "SNP": "snp",
+            "PC": "pc",
+            "Other": "other",
         }
         parties_17 = list(parties_lookup_2015.values())
 
@@ -102,28 +102,28 @@ class UK2015Model:
         ##############
 
         # Get 2015 polling
-        pollsters = polls[(polls.to >= '2015-04-04') & (polls.to <= '2015-05-04')].company.unique()
+        pollsters = polls[(polls.to >= "2015-04-04") & (polls.to <= "2015-05-04")].company.unique()
 
         # Use single last poll from each pollster in final week of polling then average out
-        polls = polls[(polls.to >= '2015-04-01') & (polls.to <= '2015-05-07')]
+        polls = polls[(polls.to >= "2015-04-01") & (polls.to <= "2015-05-07")]
         pop = polls.loc[:0]
         for p in pollsters:
             pop = pop.append(polls[polls.company == p].tail(1))
 
         # Create new polls dictionary by geo containing simple average across all pollsters
-        polls = {'UK': {}}
-        for p in ['con', 'lab', 'ld', 'ukip', 'grn']:
-            polls['UK'][p] = pop[p].mean()
-        polls['UK'] = pd.Series(polls['UK'])
+        polls = {"UK": {}}
+        for p in ["con", "lab", "ld", "ukip", "grn"]:
+            polls["UK"][p] = pop[p].mean()
+        polls["UK"] = pd.Series(polls["UK"])
 
         # Scotland, Wales, NI, London not available in 2015 data (we haven't extracted them yet!)
         # Add Other
-        for geo in ['UK']:
-            if 'other' not in polls[geo]:
-                polls[geo]['other'] = 1 - sum(polls[geo])
+        for geo in ["UK"]:
+            if "other" not in polls[geo]:
+                polls[geo]["other"] = 1 - sum(polls[geo])
 
         # Reweight to 100%
-        for geo in ['UK']:
+        for geo in ["UK"]:
             polls[geo] = polls[geo] / polls[geo].sum()
 
         ##############
@@ -132,114 +132,114 @@ class UK2015Model:
         # TODO: This is messy.
         # TODO: This should be in the polling processing pipeline.
         # Latest polling data
-        polls_17 = {'UK': {}}
-        polls_17_uk = pd.read_csv(self.directory / 'raw' / 'general_election-uk-polls.csv')
+        polls_17 = {"UK": {}}
+        polls_17_uk = pd.read_csv(self.directory / "raw" / "general_election-uk-polls.csv")
         # Filter to recent data
-        polls_17_uk = polls_17_uk[polls_17_uk.to >= '2017-06-06']
+        polls_17_uk = polls_17_uk[polls_17_uk.to >= "2017-06-06"]
         # Add parties
-        for p in ['con', 'lab', 'ld', 'ukip', 'grn', 'snp']:
-            polls_17['UK'][p] = (polls_17_uk.sample_size * polls_17_uk[p]).sum() / polls_17_uk.sample_size.sum()
-        polls_17['UK'] = pd.Series(polls_17['UK'], index=['con', 'lab', 'ld', 'ukip', 'snp', 'grn'])
+        for p in ["con", "lab", "ld", "ukip", "grn", "snp"]:
+            polls_17["UK"][p] = (polls_17_uk.sample_size * polls_17_uk[p]).sum() / polls_17_uk.sample_size.sum()
+        polls_17["UK"] = pd.Series(polls_17["UK"], index=["con", "lab", "ld", "ukip", "snp", "grn"])
 
         # Repeat for Scotland polling...
-        polls_17['Scotland'] = {}
-        polls_17_tmp = pd.read_csv(self.directory / 'raw' / 'general_election-scotland-polls.csv')
-        polls_17_tmp = polls_17_tmp[polls_17_tmp.to >= '2017-06-05']
-        for p in ['con', 'lab', 'ld', 'ukip', 'snp', 'grn']:
-            polls_17['Scotland'][p] = (
+        polls_17["Scotland"] = {}
+        polls_17_tmp = pd.read_csv(self.directory / "raw" / "general_election-scotland-polls.csv")
+        polls_17_tmp = polls_17_tmp[polls_17_tmp.to >= "2017-06-05"]
+        for p in ["con", "lab", "ld", "ukip", "snp", "grn"]:
+            polls_17["Scotland"][p] = (
                 polls_17_tmp.sample_size * polls_17_tmp[p]
             ).sum() / polls_17_tmp.sample_size.sum()
-        polls_17['Scotland'] = pd.Series(polls_17['Scotland'], index=['con', 'lab', 'ld', 'ukip', 'snp', 'grn'])
+        polls_17["Scotland"] = pd.Series(polls_17["Scotland"], index=["con", "lab", "ld", "ukip", "snp", "grn"])
 
         # ...and Wales
-        polls_17['Wales'] = {}
-        polls_17_tmp = pd.read_csv(self.directory / 'raw' / 'general_election-wales-polls.csv')
-        polls_17_tmp = polls_17_tmp[polls_17_tmp.to >= '2017-06-07']
-        for p in ['con', 'lab', 'ld', 'ukip', 'pc', 'grn']:
-            polls_17['Wales'][p] = (polls_17_tmp.sample_size * polls_17_tmp[p]).sum() / polls_17_tmp.sample_size.sum()
-        polls_17['Wales'] = pd.Series(polls_17['Wales'], index=['con', 'lab', 'ld', 'ukip', 'pc', 'grn'])
+        polls_17["Wales"] = {}
+        polls_17_tmp = pd.read_csv(self.directory / "raw" / "general_election-wales-polls.csv")
+        polls_17_tmp = polls_17_tmp[polls_17_tmp.to >= "2017-06-07"]
+        for p in ["con", "lab", "ld", "ukip", "pc", "grn"]:
+            polls_17["Wales"][p] = (polls_17_tmp.sample_size * polls_17_tmp[p]).sum() / polls_17_tmp.sample_size.sum()
+        polls_17["Wales"] = pd.Series(polls_17["Wales"], index=["con", "lab", "ld", "ukip", "pc", "grn"])
 
         # NI
-        polls_17['NI'] = (
-            pd.read_csv(self.directory / 'raw' / 'general_election-ni-polls.csv')
-            .sort_values(by='to', ascending=False)
+        polls_17["NI"] = (
+            pd.read_csv(self.directory / "raw" / "general_election-ni-polls.csv")
+            .sort_values(by="to", ascending=False)
             .iloc[0]
         )
         # Collate all NI parties under other
-        for k in polls_17['NI'].index:
+        for k in polls_17["NI"].index:
             if k not in parties_17:
-                del polls_17['NI'][k]
+                del polls_17["NI"][k]
 
-        del polls_17['NI']['other']
+        del polls_17["NI"]["other"]
 
         # London
-        polls_17['London'] = {}
-        polls_17_tmp = pd.read_csv(self.directory / 'raw' / 'general_election-london-polls.csv')
-        polls_17_tmp = polls_17_tmp[polls_17_tmp.to >= '2017-05-31']
-        for p in ['con', 'lab', 'ld', 'ukip', 'grn']:
-            polls_17['London'][p] = (polls_17_tmp.sample_size * polls_17_tmp[p]).sum() / polls_17_tmp.sample_size.sum()
-        polls_17['London'] = pd.Series(polls_17['London'], index=['con', 'lab', 'ld', 'ukip', 'grn'])
+        polls_17["London"] = {}
+        polls_17_tmp = pd.read_csv(self.directory / "raw" / "general_election-london-polls.csv")
+        polls_17_tmp = polls_17_tmp[polls_17_tmp.to >= "2017-05-31"]
+        for p in ["con", "lab", "ld", "ukip", "grn"]:
+            polls_17["London"][p] = (polls_17_tmp.sample_size * polls_17_tmp[p]).sum() / polls_17_tmp.sample_size.sum()
+        polls_17["London"] = pd.Series(polls_17["London"], index=["con", "lab", "ld", "ukip", "grn"])
 
         # Estimate polling for England excluding London
         survation_wts = {
             # from http://survation.com/wp-content/uploads/2017/06/Final-MoS-Post-BBC-Event-Poll-020617SWCH-1c0d4h9.pdf
-            'Scotland': 85,
-            'England': 881,
-            'Wales': 67,
-            'London': 137,
-            'NI': 16,
+            "Scotland": 85,
+            "England": 881,
+            "Wales": 67,
+            "London": 137,
+            "NI": 16,
         }
-        survation_wts['England_not_london'] = survation_wts['England'] - survation_wts['London']
-        survation_wts['UK'] = (
-            survation_wts['Scotland'] + survation_wts['England'] + survation_wts['Wales'] + survation_wts['NI']
+        survation_wts["England_not_london"] = survation_wts["England"] - survation_wts["London"]
+        survation_wts["UK"] = (
+            survation_wts["Scotland"] + survation_wts["England"] + survation_wts["Wales"] + survation_wts["NI"]
         )
 
         def calculate_england_not_london(party):
-            out = polls_17['UK'][party] * survation_wts['UK']
-            for geo in ['Scotland', 'Wales', 'NI', 'London']:
+            out = polls_17["UK"][party] * survation_wts["UK"]
+            for geo in ["Scotland", "Wales", "NI", "London"]:
                 if party in polls_17[geo]:
                     out = out - polls_17[geo][party] * survation_wts[geo]
-            out = out / survation_wts['England_not_london']
+            out = out / survation_wts["England_not_london"]
             return out
 
-        polls_17['England_not_london'] = {'pc': 0, 'snp': 0}
-        for party in ['con', 'lab', 'ld', 'ukip', 'grn']:
-            polls_17['England_not_london'][party] = calculate_england_not_london(party)
+        polls_17["England_not_london"] = {"pc": 0, "snp": 0}
+        for party in ["con", "lab", "ld", "ukip", "grn"]:
+            polls_17["England_not_london"][party] = calculate_england_not_london(party)
 
-        polls_17['England_not_london'] = pd.Series(polls_17['England_not_london'])
+        polls_17["England_not_london"] = pd.Series(polls_17["England_not_london"])
 
         # Fill in the gaps
-        for geo in ['UK', 'Scotland', 'Wales', 'NI', 'London', 'England_not_london']:
-            for party in ['con', 'lab', 'ld', 'ukip', 'grn', 'snp', 'pc']:
+        for geo in ["UK", "Scotland", "Wales", "NI", "London", "England_not_london"]:
+            for party in ["con", "lab", "ld", "ukip", "grn", "snp", "pc"]:
                 if party not in polls_17[geo]:
                     polls_17[geo][party] = 0
 
         # Fix PC (Plaid Cymru) for UK
-        polls_17['UK']['pc'] = polls_17['Wales']['pc'] * survation_wts['Wales'] / survation_wts['UK']
+        polls_17["UK"]["pc"] = polls_17["Wales"]["pc"] * survation_wts["Wales"] / survation_wts["UK"]
 
         # Add Other
-        for geo in ['UK', 'Scotland', 'Wales', 'NI', 'London', 'England_not_london']:
-            if 'other' not in polls_17[geo]:
-                polls_17[geo]['other'] = 1 - sum(polls_17[geo])
+        for geo in ["UK", "Scotland", "Wales", "NI", "London", "England_not_london"]:
+            if "other" not in polls_17[geo]:
+                polls_17[geo]["other"] = 1 - sum(polls_17[geo])
 
         # This doesn't work for UK or England_not_london; set current other polling to match 2015 result
-        polls_17['UK']['other'] = 0.03  # ge.other.sum() / ge['Valid Votes'].sum()
-        polls_17['England_not_london'][
-            'other'
+        polls_17["UK"]["other"] = 0.03  # ge.other.sum() / ge['Valid Votes'].sum()
+        polls_17["England_not_london"][
+            "other"
         ] = 0.01  # ge[ge.geo == 'England_not_london'].other.sum() / ge[ge.geo == 'England_not_london']['Valid Votes'].sum()
 
         # Reweight to 100%
-        for geo in ['UK', 'Scotland', 'Wales', 'NI', 'London', 'England_not_london']:
+        for geo in ["UK", "Scotland", "Wales", "NI", "London", "England_not_london"]:
             polls_17[geo] = polls_17[geo] / polls_17[geo].sum()
 
         # Export polling data
-        polls_15_csv = pd.DataFrame(columns=['con', 'lab', 'ld', 'ukip', 'grn', 'snp', 'pc', 'other'])
+        polls_15_csv = pd.DataFrame(columns=["con", "lab", "ld", "ukip", "grn", "snp", "pc", "other"])
         for geo in polls:
             for party in polls[geo].index:
                 polls_15_csv.loc[geo, party] = polls[geo].loc[party]
         # polls_15_csv.to_csv(polls_data_dir / 'final_polls_2015.csv', index=True)
 
-        polls_17_csv = pd.DataFrame(columns=['con', 'lab', 'ld', 'ukip', 'grn', 'snp', 'pc', 'other'])
+        polls_17_csv = pd.DataFrame(columns=["con", "lab", "ld", "ukip", "grn", "snp", "pc", "other"])
         for geo in polls_17:
             for party in polls_17[geo].index:
                 polls_17_csv.loc[geo, party] = polls_17[geo].loc[party]
@@ -249,56 +249,56 @@ class UK2015Model:
         # Calculate uplifts ("swing")
         #############################
 
-        parties_15 = ['con', 'lab', 'ld', 'ukip', 'grn', 'other']
-        parties_17 = ['con', 'lab', 'ld', 'ukip', 'grn', 'snp', 'pc', 'other']
+        parties_15 = ["con", "lab", "ld", "ukip", "grn", "other"]
+        parties_17 = ["con", "lab", "ld", "ukip", "grn", "snp", "pc", "other"]
 
         parties_lookup_2010 = {
-            'Con': 'con',
-            'Lab': 'lab',
-            'LD': 'ld',
-            'UKIP': 'ukip',
-            'Grn': 'grn',
-            'Other': 'other',
+            "Con": "con",
+            "Lab": "lab",
+            "LD": "ld",
+            "UKIP": "ukip",
+            "Grn": "grn",
+            "Other": "other",
         }
 
         parties_lookup_2015 = {
-            'C': 'con',
-            'Lab': 'lab',
-            'LD': 'ld',
-            'UKIP': 'ukip',
-            'Green': 'grn',
-            'SNP': 'snp',
-            'PC': 'pc',
-            'Other': 'other',
+            "C": "con",
+            "Lab": "lab",
+            "LD": "ld",
+            "UKIP": "ukip",
+            "Green": "grn",
+            "SNP": "snp",
+            "PC": "pc",
+            "Other": "other",
         }
 
         # Calculate national voteshare in 2010
-        ge_2010_totals = ge_2010.loc[:, ['Votes'] + parties_15].sum()
-        ge_2010_voteshare = ge_2010_totals / ge_2010_totals['Votes']
-        del ge_2010_voteshare['Votes']
+        ge_2010_totals = ge_2010.loc[:, ["Votes"] + parties_15].sum()
+        ge_2010_voteshare = ge_2010_totals / ge_2010_totals["Votes"]
+        del ge_2010_voteshare["Votes"]
         ge_2010_voteshare
 
         # Calculate swing between 2015 and latest smoothed polling
         swing = ge_2010_voteshare.copy()
         for party in parties_15:
-            swing[party] = polls_15_csv.loc['UK', party] / ge_2010_voteshare[party] - 1
-            ge_2010[party + '_swing'] = polls_15_csv.loc['UK', party] / ge_2010_voteshare[party] - 1
+            swing[party] = polls_15_csv.loc["UK", party] / ge_2010_voteshare[party] - 1
+            ge_2010[party + "_swing"] = polls_15_csv.loc["UK", party] / ge_2010_voteshare[party] - 1
 
         # Forecast is previous result multiplied by swing uplift
         for party in parties_15:
-            ge_2010[party + '_forecast'] = ge_2010[party + '_pc'] * (1 + swing[party])
+            ge_2010[party + "_forecast"] = ge_2010[party + "_pc"] * (1 + swing[party])
 
         def pred_15(row):
             return (
-                row[[p + '_forecast' for p in parties_15]]
+                row[[p + "_forecast" for p in parties_15]]
                 .sort_values(ascending=False)
                 .index[0]
-                .replace('_forecast', '')
+                .replace("_forecast", "")
             )
 
         # ge_2010['win_10'] = ge_2010_full.apply(win_10, axis=1)
         # ge_2015['win_15'] = ge_2015_full.apply(win_15, axis=1)
-        ge_2010['win_15'] = ge_2010.apply(pred_15, axis=1)
+        ge_2010["win_15"] = ge_2010.apply(pred_15, axis=1)
         # ge_2010.groupby('win_10').count()['Constituency Name'].sort_values(ascending=False)
 
         ########################################################
@@ -309,11 +309,11 @@ class UK2015Model:
         geos = list(ge_2015.geo.unique())
 
         # Calculate geo-level voteshare in 2015
-        ge_2015_totals = ge_2015.loc[:, ['Valid Votes', 'geo'] + parties_17].groupby('geo').sum()
+        ge_2015_totals = ge_2015.loc[:, ["Valid Votes", "geo"] + parties_17].groupby("geo").sum()
 
         # Convert into vote share
-        ge_2015_voteshare = ge_2015_totals.div(ge_2015_totals['Valid Votes'], axis=0)
-        del ge_2015_voteshare['Valid Votes']
+        ge_2015_voteshare = ge_2015_totals.div(ge_2015_totals["Valid Votes"], axis=0)
+        del ge_2015_voteshare["Valid Votes"]
         ge_2015_voteshare
 
         # Calculate geo-swing
@@ -328,125 +328,125 @@ class UK2015Model:
 
         # Apply swing
         for party in parties_17:
-            ge_2015[party + '_swing'] = ge_2015.apply(lambda row: swing_17.loc[row['geo']][party], axis=1)
-            ge_2015[party + '_2017_forecast'] = ge_2015.apply(
-                lambda x: x[party + '_pc'] * (1 + swing_17.loc[x['geo']][party]), axis=1
+            ge_2015[party + "_swing"] = ge_2015.apply(lambda row: swing_17.loc[row["geo"]][party], axis=1)
+            ge_2015[party + "_2017_forecast"] = ge_2015.apply(
+                lambda x: x[party + "_pc"] * (1 + swing_17.loc[x["geo"]][party]), axis=1
             )
 
         def win_17(row):
             return (
-                row[[p + '_2017_forecast' for p in parties_17]]
+                row[[p + "_2017_forecast" for p in parties_17]]
                 .sort_values(ascending=False)
                 .index[0]
-                .replace('_2017_forecast', '')
+                .replace("_2017_forecast", "")
             )
 
-        ge_2015['win_17'] = ge_2015.apply(win_17, axis=1)
+        ge_2015["win_17"] = ge_2015.apply(win_17, axis=1)
 
         ###########################
         # Create ML-ready dataframe
         ###########################
 
-        parties = ['con', 'lab', 'ld', 'ukip', 'grn']
-        act_15_lookup = {k: v for i, (k, v) in ge_2015[['Press Association ID Number', 'winner']].iterrows()}
-        ge_2010['act_15'] = ge_2010['Press Association Reference'].map(act_15_lookup)
+        parties = ["con", "lab", "ld", "ukip", "grn"]
+        act_15_lookup = {k: v for i, (k, v) in ge_2015[["Press Association ID Number", "winner"]].iterrows()}
+        ge_2010["act_15"] = ge_2010["Press Association Reference"].map(act_15_lookup)
         pc_15_lookup = {
-            p: {k: v for i, (k, v) in ge_2015[['Press Association ID Number', p + '_pc']].iterrows()} for p in parties
+            p: {k: v for i, (k, v) in ge_2015[["Press Association ID Number", p + "_pc"]].iterrows()} for p in parties
         }
         for p in parties:
-            ge_2010[p + '_actual'] = ge_2010['Press Association Reference'].map(pc_15_lookup[p])
+            ge_2010[p + "_actual"] = ge_2010["Press Association Reference"].map(pc_15_lookup[p])
 
-        df = ge_2010[['Press Association Reference', 'Constituency Name', 'Region', 'Electorate', 'Votes',] + parties]
+        df = ge_2010[["Press Association Reference", "Constituency Name", "Region", "Electorate", "Votes",] + parties]
         df = pd.melt(
             df,
-            id_vars=['Press Association Reference', 'Constituency Name', 'Region', 'Electorate', 'Votes',],
+            id_vars=["Press Association Reference", "Constituency Name", "Region", "Electorate", "Votes",],
             value_vars=parties,
-            var_name='party',
-            value_name='votes_last',
+            var_name="party",
+            value_name="votes_last",
         )
 
         # pc_last
         pc_last = pd.melt(
-            ge_2010[['Press Association Reference'] + [p + '_pc' for p in parties]],
-            id_vars=['Press Association Reference'],
-            value_vars=[p + '_pc' for p in parties],
-            var_name='party',
-            value_name='pc_last',
+            ge_2010[["Press Association Reference"] + [p + "_pc" for p in parties]],
+            id_vars=["Press Association Reference"],
+            value_vars=[p + "_pc" for p in parties],
+            var_name="party",
+            value_name="pc_last",
         )
-        pc_last['party'] = pc_last.party.apply(lambda x: x.replace('_pc', ''))
-        df = pd.merge(left=df, right=pc_last, how='left', on=['Press Association Reference', 'party'],)
+        pc_last["party"] = pc_last.party.apply(lambda x: x.replace("_pc", ""))
+        df = pd.merge(left=df, right=pc_last, how="left", on=["Press Association Reference", "party"],)
 
         # win_last
-        win_last = ge_2010[['Press Association Reference', 'winner']]
-        win_last.columns = ['Press Association Reference', 'win_last']
-        df = pd.merge(left=df, right=win_last, on=['Press Association Reference'])
+        win_last = ge_2010[["Press Association Reference", "winner"]]
+        win_last.columns = ["Press Association Reference", "win_last"]
+        df = pd.merge(left=df, right=win_last, on=["Press Association Reference"])
 
         # polls_now
-        df['polls_now'] = df.party.map(polls['UK'])
+        df["polls_now"] = df.party.map(polls["UK"])
 
         # swing_now
         swing_now = pd.melt(
-            ge_2010[['Press Association Reference'] + [p + '_swing' for p in parties]],
-            id_vars=['Press Association Reference'],
-            value_vars=[p + '_swing' for p in parties],
-            var_name='party',
-            value_name='swing_now',
+            ge_2010[["Press Association Reference"] + [p + "_swing" for p in parties]],
+            id_vars=["Press Association Reference"],
+            value_vars=[p + "_swing" for p in parties],
+            var_name="party",
+            value_name="swing_now",
         )
-        swing_now['party'] = swing_now.party.apply(lambda x: x.replace('_swing', ''))
+        swing_now["party"] = swing_now.party.apply(lambda x: x.replace("_swing", ""))
 
-        df = pd.merge(left=df, right=swing_now, how='left', on=['Press Association Reference', 'party'],)
+        df = pd.merge(left=df, right=swing_now, how="left", on=["Press Association Reference", "party"],)
 
         # swing_forecast_pc
         swing_forecast_pc = pd.melt(
-            ge_2010[['Press Association Reference'] + [p + '_forecast' for p in parties]],
-            id_vars=['Press Association Reference'],
-            value_vars=[p + '_forecast' for p in parties],
-            var_name='party',
-            value_name='swing_forecast_pc',
+            ge_2010[["Press Association Reference"] + [p + "_forecast" for p in parties]],
+            id_vars=["Press Association Reference"],
+            value_vars=[p + "_forecast" for p in parties],
+            var_name="party",
+            value_name="swing_forecast_pc",
         )
-        swing_forecast_pc['party'] = swing_forecast_pc.party.apply(lambda x: x.replace('_forecast', ''))
+        swing_forecast_pc["party"] = swing_forecast_pc.party.apply(lambda x: x.replace("_forecast", ""))
 
-        df = pd.merge(left=df, right=swing_forecast_pc, how='left', on=['Press Association Reference', 'party'],)
+        df = pd.merge(left=df, right=swing_forecast_pc, how="left", on=["Press Association Reference", "party"],)
 
         # swing_forecast_win
-        swing_forecast_win = ge_2010[['Press Association Reference', 'win_15']]
-        swing_forecast_win.columns = ['Press Association Reference', 'swing_forecast_win']
-        df = pd.merge(left=df, right=swing_forecast_win, on=['Press Association Reference'])
+        swing_forecast_win = ge_2010[["Press Association Reference", "win_15"]]
+        swing_forecast_win.columns = ["Press Association Reference", "swing_forecast_win"]
+        df = pd.merge(left=df, right=swing_forecast_win, on=["Press Association Reference"])
 
         # actual_win_now
-        actual_win_now = ge_2010[['Press Association Reference', 'act_15']]
-        actual_win_now.columns = ['Press Association Reference', 'actual_win_now']
-        df = pd.merge(left=df, right=actual_win_now, on=['Press Association Reference'])
+        actual_win_now = ge_2010[["Press Association Reference", "act_15"]]
+        actual_win_now.columns = ["Press Association Reference", "actual_win_now"]
+        df = pd.merge(left=df, right=actual_win_now, on=["Press Association Reference"])
 
         # actual_pc_now
         actual_pc_now = pd.melt(
-            ge_2010[['Press Association Reference'] + [p + '_actual' for p in parties]],
-            id_vars=['Press Association Reference'],
-            value_vars=[p + '_actual' for p in parties],
-            var_name='party',
-            value_name='actual_pc_now',
+            ge_2010[["Press Association Reference"] + [p + "_actual" for p in parties]],
+            id_vars=["Press Association Reference"],
+            value_vars=[p + "_actual" for p in parties],
+            var_name="party",
+            value_name="actual_pc_now",
         )
-        actual_pc_now['party'] = actual_pc_now.party.apply(lambda x: x.replace('_actual', ''))
+        actual_pc_now["party"] = actual_pc_now.party.apply(lambda x: x.replace("_actual", ""))
 
-        df = pd.merge(left=df, right=actual_pc_now, how='left', on=['Press Association Reference', 'party'],)
+        df = pd.merge(left=df, right=actual_pc_now, how="left", on=["Press Association Reference", "party"],)
 
         # dummy party
         df = pd.concat([df, pd.get_dummies(df.party)], axis=1)
 
         # dummy region
-        df = pd.concat([df, pd.get_dummies(df.Region, prefix='Region')], axis=1)
+        df = pd.concat([df, pd.get_dummies(df.Region, prefix="Region")], axis=1)
 
         # won_here_last
-        df['won_here_last'] = (df['party'] == df['win_last']).astype('int')
+        df["won_here_last"] = (df["party"] == df["win_last"]).astype("int")
 
         # turnout
-        df['turnout'] = df.Votes / df.Electorate
+        df["turnout"] = df.Votes / df.Electorate
 
         ########################################
         # Export final 2010 -> 2015 training set
         ########################################
-        print(f'Exporting 2010->2015 model dataset to {processed_directory.resolve()}')
-        df.to_csv(processed_directory / 'general_election-uk-2015-model.csv', index=False)
+        print(f"Exporting 2010->2015 model dataset to {processed_directory.resolve()}")
+        df.to_csv(processed_directory / "general_election-uk-2015-model.csv", index=False)
 
         ######################
         # REPEAT FOR 2015-2017
@@ -454,90 +454,90 @@ class UK2015Model:
         # Recreate this training dataset using same column names for 2015 -> 2017 for a GE2017 forecast
         # TODO: Needs refactoring!
         # Add SNP and Plaid Cymru
-        parties += ['snp', 'pc']
+        parties += ["snp", "pc"]
         df15 = ge_2015[
-            ['Press Association ID Number', 'Constituency Name', 'Region', 'geo', 'Electorate', 'Valid Votes',]
+            ["Press Association ID Number", "Constituency Name", "Region", "geo", "Electorate", "Valid Votes",]
             + parties
         ]
         df15.columns = [
-            'Press Association ID Number',
-            'Constituency Name',
-            'Region',
-            'geo',
-            'Electorate',
-            'Votes',
+            "Press Association ID Number",
+            "Constituency Name",
+            "Region",
+            "geo",
+            "Electorate",
+            "Votes",
         ] + parties
         df15 = pd.melt(
             df15,
-            id_vars=['Press Association ID Number', 'Constituency Name', 'Region', 'geo', 'Electorate', 'Votes',],
+            id_vars=["Press Association ID Number", "Constituency Name", "Region", "geo", "Electorate", "Votes",],
             value_vars=parties,
-            var_name='party',
-            value_name='votes_last',
+            var_name="party",
+            value_name="votes_last",
         )
 
         # pc_last
         pc_last = pd.melt(
-            ge_2015[['Press Association ID Number'] + [p + '_pc' for p in parties]],
-            id_vars=['Press Association ID Number'],
-            value_vars=[p + '_pc' for p in parties],
-            var_name='party',
-            value_name='pc_last',
+            ge_2015[["Press Association ID Number"] + [p + "_pc" for p in parties]],
+            id_vars=["Press Association ID Number"],
+            value_vars=[p + "_pc" for p in parties],
+            var_name="party",
+            value_name="pc_last",
         )
-        pc_last['party'] = pc_last.party.apply(lambda x: x.replace('_pc', ''))
+        pc_last["party"] = pc_last.party.apply(lambda x: x.replace("_pc", ""))
 
-        df15 = pd.merge(left=df15, right=pc_last, how='left', on=['Press Association ID Number', 'party'],)
+        df15 = pd.merge(left=df15, right=pc_last, how="left", on=["Press Association ID Number", "party"],)
 
         # win_last
-        win_last = ge_2015[['Press Association ID Number', 'winner']]
-        win_last.columns = ['Press Association ID Number', 'win_last']
-        df15 = pd.merge(left=df15, right=win_last, on=['Press Association ID Number'])
+        win_last = ge_2015[["Press Association ID Number", "winner"]]
+        win_last.columns = ["Press Association ID Number", "win_last"]
+        df15 = pd.merge(left=df15, right=win_last, on=["Press Association ID Number"])
 
         # polls_now <- USE REGIONAL POLLING! (Possibly a very bad idea, the regional UNS performed worse than national!)
-        df15['polls_now'] = df15.apply(lambda row: polls_17[row.geo][row.party], axis=1)
+        df15["polls_now"] = df15.apply(lambda row: polls_17[row.geo][row.party], axis=1)
 
         # swing_now
         swing_now = pd.melt(
-            ge_2015[['Press Association ID Number'] + [p + '_swing' for p in parties]],
-            id_vars=['Press Association ID Number'],
-            value_vars=[p + '_swing' for p in parties],
-            var_name='party',
-            value_name='swing_now',
+            ge_2015[["Press Association ID Number"] + [p + "_swing" for p in parties]],
+            id_vars=["Press Association ID Number"],
+            value_vars=[p + "_swing" for p in parties],
+            var_name="party",
+            value_name="swing_now",
         )
-        swing_now['party'] = swing_now.party.apply(lambda x: x.replace('_swing', ''))
+        swing_now["party"] = swing_now.party.apply(lambda x: x.replace("_swing", ""))
 
-        df15 = pd.merge(left=df15, right=swing_now, how='left', on=['Press Association ID Number', 'party'],)
+        df15 = pd.merge(left=df15, right=swing_now, how="left", on=["Press Association ID Number", "party"],)
 
         # swing_forecast_pc
         swing_forecast_pc = pd.melt(
-            ge_2015[['Press Association ID Number'] + [p + '_2017_forecast' for p in parties]],
-            id_vars=['Press Association ID Number'],
-            value_vars=[p + '_2017_forecast' for p in parties],
-            var_name='party',
-            value_name='swing_forecast_pc',
+            ge_2015[["Press Association ID Number"] + [p + "_2017_forecast" for p in parties]],
+            id_vars=["Press Association ID Number"],
+            value_vars=[p + "_2017_forecast" for p in parties],
+            var_name="party",
+            value_name="swing_forecast_pc",
         )
-        swing_forecast_pc['party'] = swing_forecast_pc.party.apply(lambda x: x.replace('_2017_forecast', ''))
+        swing_forecast_pc["party"] = swing_forecast_pc.party.apply(lambda x: x.replace("_2017_forecast", ""))
 
-        df15 = pd.merge(left=df15, right=swing_forecast_pc, how='left', on=['Press Association ID Number', 'party'],)
+        df15 = pd.merge(left=df15, right=swing_forecast_pc, how="left", on=["Press Association ID Number", "party"],)
 
         # swing_forecast_win
-        swing_forecast_win = ge_2015[['Press Association ID Number', 'win_17']]
-        swing_forecast_win.columns = ['Press Association ID Number', 'swing_forecast_win']
-        df15 = pd.merge(left=df15, right=swing_forecast_win, on=['Press Association ID Number'])
+        swing_forecast_win = ge_2015[["Press Association ID Number", "win_17"]]
+        swing_forecast_win.columns = ["Press Association ID Number", "swing_forecast_win"]
+        df15 = pd.merge(left=df15, right=swing_forecast_win, on=["Press Association ID Number"])
 
         # dummy party
         df15 = pd.concat([df15, pd.get_dummies(df15.party)], axis=1)
 
         # dummy region
-        df15 = pd.concat([df15, pd.get_dummies(df15.Region, prefix='Region')], axis=1)
+        df15 = pd.concat([df15, pd.get_dummies(df15.Region, prefix="Region")], axis=1)
 
         # won_here_last
-        df15['won_here_last'] = (df15['party'] == df15['win_last']).astype('int')
+        df15["won_here_last"] = (df15["party"] == df15["win_last"]).astype("int")
 
         # turnout
-        df15['turnout'] = df.Votes / df.Electorate
+        df15["turnout"] = df.Votes / df.Electorate
 
         ##########################################
         # Export final 2015 -> 2017 prediction set
         ##########################################
-        print(f'Exporting 2015->2017 model dataset to {processed_directory.resolve()}')
-        df15.to_csv(processed_directory / 'general_election-uk-2017-model.csv', index=False)
+        print(f"Exporting 2015->2017 model dataset to {processed_directory.resolve()}")
+        df15.to_csv(processed_directory / "general_election-uk-2017-model.csv", index=False)
diff --git a/maven/datasets/general_election/uk_2015_results.py b/maven/datasets/general_election/uk_2015_results.py
index 9d2bc9c..47010b5 100644
--- a/maven/datasets/general_election/uk_2015_results.py
+++ b/maven/datasets/general_election/uk_2015_results.py
@@ -1,204 +1,52 @@
 """
 Results data for the United Kingdom's 2015 General Election.
 
-Source: http://www.electoralcommission.org.uk/__data/assets/file/0004/191650/2015-UK-general-election-data-results-WEB.zip
-
 Usage:
-    > import maven
-    > maven.get('general-election/UK/2015/results', data_directory='./data/')
+    >>> import maven
+    >>> maven.get('general-election/UK/2015/results', data_directory='./data/')
+
+
+Sources:
+    - http://researchbriefings.files.parliament.uk/documents/CBP-8647/1918-2017election_results.csv
+        - From https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-8647
+
+Deprecated sources:
+    - http://www.electoralcommission.org.uk/__data/assets/file/0004/191650/2015-UK-general-election-data-results-WEB.zip
+
+Notes:
+    - 2015-UK-general-election-data-results-WEB.zip has a lot more detailed data.
 """
+
 import os
-import zipfile
 from pathlib import Path
 
-import pandas as pd
-import requests
+from maven import utils
+from maven.datasets.general_election.base import UKResults
 
 
-class UK2015Results:
+class UK2015Results(UKResults):
     """Handles results data for the United Kingdom's 2015 General Election."""
 
-    def __init__(self, directory=Path('data/general-election/UK/2015/results')):
+    def __init__(self, directory=Path("data/general-election/UK/2015/results")):
         self.directory = Path(directory)
-
-    def retrieve(self):
-        """Retrieve results data for the United Kingdom's 2015 General Election."""
-        url = 'https://www.electoralcommission.org.uk/sites/default/files/2019-08/'
-        filename = '2015-UK-general-election-data-results%20-%20CSV.zip'
-        target = self.directory / 'raw'
-        os.makedirs(target, exist_ok=True)  # create directory if it doesn't exist
-
-        print(f'Downloading into {target.resolve()}')
-        with open(target / filename, 'wb') as f:
-            response = requests.get(url + filename)
-            f.write(response.content)
-
-        print(f'Extracting into {target.resolve()}')
-        with zipfile.ZipFile(target / filename, 'r') as f:
-            f.extractall(target)
-
-        print('Cleaning up')
-        os.remove(target / filename)
+        self.sources = [
+            (
+                "http://researchbriefings.files.parliament.uk/documents/CBP-8647/",
+                "1918-2017election_results_by_pcon.xlsx",
+            ),
+        ]
+        self.verbose_name = "UK 2015 General Election results"
 
     def process(self):
         """Process results data for the United Kingdom's 2015 General Election."""
-        processed_results_filename = 'general_election-uk-2015-results.csv'
-        processed_results_full_filename = 'general_election-uk-2015-results-full.csv'
-        processed_results_location = self.directory / 'processed' / processed_results_filename
-        processed_results_full_location = self.directory / 'processed' / processed_results_full_filename
-        os.makedirs(self.directory / 'processed', exist_ok=True)  # create directory if it doesn't exist
-
-        # TODO: Refactor these sections into functions to make it easier to read.
-
-        ##########################
-        # GENERAL ELECTION RESULTS
-        ##########################
-        print('Read and clean RESULTS FOR ANALYSIS.csv')
-
-        # Import general election results
-        results = pd.read_csv(self.directory / 'raw' / 'RESULTS FOR ANALYSIS.csv')
-
-        # Remove 'Unnamed: 9' columnd
-        del results['Unnamed: 9']
-
-        # Fix bad column name (' Total number of valid votes counted ' to 'Valid Votes')
-        results.columns = list(results.columns[:8]) + ['Valid Votes'] + list(results.columns[9:])
-
-        # Remove rows where Constituency Name is blank
-        blank_rows = results['Constituency Name'].isnull()
-        results = results[-blank_rows].copy()
-
-        # Remove commas & coerce Electorate and Total number of valid votes counted
-        for col in ['Electorate', 'Valid Votes']:
-            results[col] = results[col].apply(lambda x: float(x.replace(',', '')))
-
-        # Set NA vals to zero
-        for col in results.columns[9:]:
-            results[col] = results[col].fillna(0)
-
-        # Checks
-        assert results.shape == (650, 146)
-
-        ###################
-        # CONSTITUENCY DATA
-        ###################
-        print('Read and clean CONSTITUENCY.csv')
-
-        # Import constituency data
-        constituency = pd.read_csv(self.directory / 'raw' / 'CONSTITUENCY.csv', encoding='latin1')
-
-        # Remove rows where Constituency Name is blank
-        blank_rows = constituency['Constituency Name'].isnull()
-        constituency = constituency[-blank_rows].copy()
-
-        # Remove 'Unnamed: 6' columnd
-        del constituency['Unnamed: 6']
-
-        # Checks
-        assert constituency.shape == (650, 10)
-
-        #######
-        # MERGE
-        #######
-        print('Merging in constituency identifiers')
-
-        # Pre-merge checks
-        match_col = 'Constituency ID'
-        assert len(set(constituency[match_col]).intersection(set(results[match_col]))) == 650
-        assert len(set(constituency[match_col]).difference(set(results[match_col]))) == 0
-        assert len(set(results[match_col]).difference(set(constituency[match_col]))) == 0
-
-        # Merge on Constituency ID
-        results = pd.merge(
-            left=results,
-            right=constituency[['Constituency ID', 'Region ID', 'County']],
-            how='left',
-            on='Constituency ID',
-        )
-        column_order = [
-            'Press Association ID Number',
-            'Constituency ID',
-            'Constituency Name',
-            'Constituency Type',
-            'County',
-            'Region ID',
-            'Region',
-            'Country',
-            'Election Year',
-            'Electorate',
-            'Valid Votes',
-        ] + list(results.columns[9:146])
-        results = results[column_order].copy()
-
-        ############################
-        # ADDITIONAL TRANSFORMATIONS
-        ############################
-        # Some MPs are members of both the Labour Party and the Co-operative Party, which plays havoc with modelling.
-        # We will therefore consider them all members of the Labour party.
-        results['Lab'] = results['Lab'] + results['Lab Co-op']
-        del results['Lab Co-op']
-
-        # Save this for convenience
-        results_full = results.copy()
-
-        # Filter to metadata cols + parties of interest
-        parties_lookup = {
-            'C': 'con',
-            'Lab': 'lab',
-            'LD': 'ld',
-            'UKIP': 'ukip',
-            'Green': 'grn',
-            'SNP': 'snp',
-            'PC': 'pc',
-            'Other': 'other',
-        }
-        other_parties = list(set(results.columns) - set(results.columns[:11]) - set(parties_lookup.keys()))
-        results['Other'] = results.loc[:, other_parties].sum(axis=1)
-        results = results.loc[:, list(results.columns[:11]) + list(parties_lookup.keys())]
-        # Rename parties
-        results.columns = [parties_lookup[x] if x in parties_lookup else x for x in results.columns]
-
-        # Calculate constituency level vote share
-        for party in parties_lookup.values():
-            results[party + '_pc'] = results[party] / results['Valid Votes']
-
-        # Create PANO -> geo lookup
-        geo_lookup = {x[1][0]: x[1][1] for x in results[['Press Association ID Number', 'Country']].iterrows()}
-        assert geo_lookup[14.0] == 'Northern Ireland'
-        # Add London boroughs
-        london_panos = results[results.County == 'London']['Press Association ID Number'].values
-        for pano in london_panos:
-            geo_lookup[pano] = 'London'
-        assert geo_lookup[237.0] == 'London'
-        # Rename other England
-        for k in geo_lookup:
-            if geo_lookup[k] == 'England':
-                geo_lookup[k] = 'England_not_london'
-            elif geo_lookup[k] == 'Northern Ireland':
-                geo_lookup[k] = 'NI'
-        results['geo'] = results['Press Association ID Number'].map(geo_lookup)
-
-        # Calculate geo-level vote share
-        # TODO: Do we use this?
-        results_by_geo = results.loc[:, ['Valid Votes', 'geo'] + list(parties_lookup.values())].groupby('geo').sum()
-        results_by_geo_voteshare = results_by_geo.div(results_by_geo['Valid Votes'], axis=0)
-        del results_by_geo_voteshare['Valid Votes']
-
-        # Who won?
-        def winner(row):
-            all_parties = set(results_full.columns[11:]) - set(['Other'])
-            winning_party = row[all_parties].sort_values(ascending=False).index[0]
-            if winning_party in parties_lookup.keys():
-                winning_party = parties_lookup[winning_party]
-            elif winning_party == 'Speaker':
-                winning_party = 'other'
-            return winning_party
+        filename = self.sources[0][1]
+        processed_results_filename = "general_election-uk-2015-results.csv"
+        processed_results_location = self.directory / "processed" / processed_results_filename
+        os.makedirs(self.directory / "processed", exist_ok=True)  # create directory if it doesn't exist
 
-        results['winner'] = results_full.apply(winner, axis=1)
-        # Check Conservatives won 330 seats in 2015.
-        assert results.groupby('winner').count()['Constituency Name'].sort_values(ascending=False)[0] == 330
+        # Process
+        results = utils.process_hoc_sheet(input_file=filename, data_dir=self.directory, sheet_name="2015")
 
-        # EXPORT
-        print(f'Exporting dataset to {processed_results_location.resolve()}')
+        # Export
+        print(f"Exporting dataset to {processed_results_location.resolve()}")
         results.to_csv(processed_results_location, index=False)
-        results_full.to_csv(processed_results_full_location, index=False)
diff --git a/maven/datasets/general_election/uk_2015_results_HoC.py b/maven/datasets/general_election/uk_2015_results_HoC.py
deleted file mode 100644
index 4c79472..0000000
--- a/maven/datasets/general_election/uk_2015_results_HoC.py
+++ /dev/null
@@ -1,119 +0,0 @@
-"""
-Results data for the United Kingdom's 2015 General Election.
-
-Source: https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-7186
-
-Usage:
-    > import maven
-    > maven.get('general-election/UK/2015/hoc_results', data_directory='./data/')
-"""
-import os
-from pathlib import Path
-
-import pandas as pd
-import requests
-import numpy as np
-
-
-class UK2015ResultsHoC:
-    """Handles results data from the HoC source for the United Kingdom's 2015 General Election."""
-
-    def __init__(self, directory=Path('data/general-election/UK/2015/hoc_results')):
-        self.directory = Path(directory)
-
-    def retrieve(self):
-        """Retrieve results data for the United Kingdom's 2015 General Election."""
-        url = "http://researchbriefings.files.parliament.uk/documents/CBP-7186"
-        filename = "hocl-ge2015-results-summary.csv"
-        target = self.directory / 'raw'
-        os.makedirs(target, exist_ok=True)  # create directory if it doesn't exist
-
-        print(f'Downloading into {target.resolve()}')
-        with open(target / filename, 'wb') as f:
-            response = requests.get(url + "/" + filename)
-            assert response.status_code == 200
-            f.write(response.content)
-
-    def process(self):
-        processed_results_filename = 'general_election-uk-2015-results.csv'
-        processed_results_full_filename = 'general_election-uk-2015-results-full.csv'
-        processed_results_location = self.directory / 'processed' / processed_results_filename
-        processed_results_full_location = self.directory / 'processed' / processed_results_full_filename
-
-        results = pd.read_csv(self.directory / 'raw' / "hocl-ge2015-results-summary.csv")
-
-        minor_parties = ['dup', 'sf', 'sdlp', 'uup', 'alliance']
-        major_parties = ['con', 'lab', 'ld', 'ukip', 'green', 'snp', 'pc']
-
-        results = results.rename({
-            'ons_id': 'Constituency ID', 
-            'constituency_name': 'Constituency Name',
-            'constituency_type': 'Constituency Type', 
-            'county_name': 'County', 
-            'region_name': 'Region',
-            'country_name': 'Country',
-            'electorate': 'Electorate',
-            'valid_votes': 'Valid Votes',
-            'first_party': 'winner'
-        }, axis=1)
-        results["Election Year"] = 2015
-        results["Press Association ID Number"] = np.nan
-
-        # Add minor parties to "other", then drop 
-        results['other'] += results[minor_parties].sum(axis=1)
-        results = results.drop(minor_parties, axis=1)
-
-        # Calculate voteshare % for major parties
-        for party in major_parties:
-            results[party + "_pc"] = (results[party] / results["Valid Votes"]) * 100
-
-        # Add 'geo' column to differentiate London from rest of England
-        def map_geo(row):
-            if row.Country == 'England':
-                if row.County == 'London':
-                    return 'London'
-                else:
-                    return 'England_not_london'
-            elif row.Country == "Northern Ireland":
-                return "NI"
-            else:
-                return row.Country
-
-        results['geo'] = results.apply(map_geo, axis=1)
-
-        results = results[[
-            "Press Association ID Number",
-            "Constituency ID",
-            "Constituency Name",
-            "Constituency Type",
-            "County",
-            # "Region ID",
-            "Region",
-            "Country",
-            "Election Year",
-            "Electorate",
-            "Valid Votes",
-            "con",
-            "lab",
-            "ld",
-            "ukip",
-            # "grn",
-            "snp",
-            "pc",
-            "other",
-            "con_pc",
-            "lab_pc",
-            "ld_pc",
-            "ukip_pc",
-            # "grn_pc",
-            "snp_pc",
-            "pc_pc",
-            # "other_pc",
-            "winner",
-            "geo"
-        ]]
-
-        os.makedirs(self.directory / 'processed', exist_ok=True)  # create directory if it doesn't exist
-
-        assert results.groupby('winner').count()['Constituency Name'].sort_values(ascending=False)[0] == 330
-        results.to_csv(processed_results_location, index=False)
diff --git a/maven/datasets/general_election/uk_2017_results.py b/maven/datasets/general_election/uk_2017_results.py
new file mode 100644
index 0000000..dcf4705
--- /dev/null
+++ b/maven/datasets/general_election/uk_2017_results.py
@@ -0,0 +1,50 @@
+"""
+Results data for the United Kingdom's 2017 General Election.
+
+Usage:
+    >>> import maven
+    >>> maven.get('general-election/UK/2017/results', data_directory='./data/')
+
+
+Sources:
+    - http://researchbriefings.files.parliament.uk/documents/CBP-8647/1918-2017election_results.csv
+        - From https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-8647
+
+Other sources:
+    - https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-7186
+    - http://researchbriefings.files.parliament.uk/documents/CBP-7979/HoC-GE2017-constituency-results.csv
+"""
+
+import os
+from pathlib import Path
+
+from maven import utils
+from maven.datasets.general_election.base import UKResults
+
+
+class UK2017Results(UKResults):
+    """Handles results data for the United Kingdom's 2017 General Election."""
+
+    def __init__(self, directory=Path("data/general-election/UK/2017/results")):
+        self.directory = Path(directory)
+        self.sources = [
+            (
+                "http://researchbriefings.files.parliament.uk/documents/CBP-8647/",
+                "1918-2017election_results_by_pcon.xlsx",
+            ),
+        ]
+        self.verbose_name = "UK 2017 General Election results"
+
+    def process(self):
+        """Process results data for the United Kingdom's 2017 General Election."""
+        filename = self.sources[0][1]
+        processed_results_filename = "general_election-uk-2017-results.csv"
+        processed_results_location = self.directory / "processed" / processed_results_filename
+        os.makedirs(self.directory / "processed", exist_ok=True)  # create directory if it doesn't exist
+
+        # Process
+        results = utils.process_hoc_sheet(input_file=filename, data_dir=self.directory, sheet_name="2017")
+
+        # Export
+        print(f"Exporting dataset to {processed_results_location.resolve()}")
+        results.to_csv(processed_results_location, index=False)
diff --git a/maven/datasets/general_election/uk_2017_results_HoC.py b/maven/datasets/general_election/uk_2017_results_HoC.py
deleted file mode 100644
index 7244a11..0000000
--- a/maven/datasets/general_election/uk_2017_results_HoC.py
+++ /dev/null
@@ -1,118 +0,0 @@
-import os
-from pathlib import Path
-
-import feather
-import numpy as np
-import pandas as pd
-import requests
-
-csv_filename = "HoC-GE2017-constituency-results.csv"
-
-
-class UK2017ResultsHoC:
-
-    def __init__(self, directory=Path("data/general-election/UK/2017/hoc_results")):
-        self.directory = Path(directory)
-
-    def retrieve(self):
-        # Config
-        url = "http://researchbriefings.files.parliament.uk/documents/CBP-7979/"
-        target = self.directory / "raw"
-        os.makedirs(target, exist_ok=True)  # create directory if it doesn't exist
-
-        # Download URL into local directory
-        print("Downloading into {}".format(target.resolve()))
-        with open(target / csv_filename, "wb") as f:
-            response = requests.get(url + csv_filename)
-            f.write(response.content)
-
-    def process(self):
-
-        processed_dir = self.directory / "processed"
-        processed_results_filename = Path('general_election-uk-2017-results.csv')
-        # GENERAL ELECTION RESULTS
-        print(f"Read and clean RESULTS FOR {csv_filename}")
-
-        # Import general election results
-        results = pd.read_csv(self.directory / "raw" / csv_filename)
-        assert results.shape == (650, 29)
-
-        results = results.rename(
-            columns={
-                "ons_id": "Constituency ID",
-                "constituency_name": "Constituency Name",
-                "constituency_type": "Constituency Type",
-                "county_name": "County",
-                "region_name": "Region",
-                "country_name": "Country",
-                "ons_region_id": "Region ID",
-                "electorate": "Electorate",
-                "valid_votes": "Valid Votes",
-                "first_party": "winner"
-            }
-        )
-        results["Election Year"] = 2017
-        results["Press Association ID Number"] = np.nan
-
-        minor_parties = ['dup', 'sf', 'sdlp', 'uup', 'alliance']
-        major_parties = ['con', 'lab', 'ld', 'ukip', 'green', 'snp', 'pc']
-
-        # Add minor parties to "other", then drop 
-        results['other'] += results[minor_parties].sum(axis=1)
-        results = results.drop(minor_parties, axis=1)
-
-        # Calculate voteshare % for major parties
-        for party in major_parties:
-            results[party + "_pc"] = (results[party] / results["Valid Votes"]) * 100
-
-        # Add 'geo' column to differentiate London from rest of England
-        def map_geo(row):
-            if row.Country == 'England':
-                if row.County == 'London':
-                    return 'London'
-                else:
-                    return 'England_not_london'
-            elif row.Country == "Northern Ireland":
-                return "NI"
-            else:
-                return row.Country
-
-        results['geo'] = results.apply(map_geo, axis=1)
-
-        results = results[[
-            "Press Association ID Number",
-            "Constituency ID",
-            "Constituency Name",
-            "Constituency Type",
-            "County",
-            "Region ID",
-            "Region",
-            "Country",
-            "Election Year",
-            "Electorate",
-            "Valid Votes",
-            "con",
-            "lab",
-            "ld",
-            "ukip",
-            # "grn",
-            "snp",
-            "pc",
-            "other",
-            "con_pc",
-            "lab_pc",
-            "ld_pc",
-            "ukip_pc",
-            # "grn_pc",
-            "snp_pc",
-            "pc_pc",
-            # "other_pc",
-            "winner",
-            "geo"
-        ]]
-
-        os.makedirs(processed_dir, exist_ok=True)  # create directory if it doesn't exist
-
-        # Export as both CSV and Feather
-        results.to_csv(processed_dir / processed_results_filename, index=False)
-        feather.write_dataframe(results, str(processed_dir / "general_election-uk-2017-results.feather"))
diff --git a/maven/datasets/general_election/uk_polls.py b/maven/datasets/general_election/uk_polls.py
index 6c1d1e3..a9b10f7 100644
--- a/maven/datasets/general_election/uk_polls.py
+++ b/maven/datasets/general_election/uk_polls.py
@@ -23,30 +23,30 @@ import requests
 class UKPolls:
     """Handles General Election polling data for the United Kingdom."""
 
-    def __init__(self, directory=Path('data/general-election/UK/polls')):
+    def __init__(self, directory=Path("data/general-election/UK/polls")):
         self.directory = Path(directory)
 
     def retrieve(self):
         """Retrieve General Election polling data for the United Kingdom."""
-        target_directory = self.directory / 'processed'
+        target_directory = self.directory / "processed"
         os.makedirs(target_directory, exist_ok=True)  # create directory if it doesn't exist
-        base_url = 'https://s3-eu-west-1.amazonaws.com/sixfifty/'
+        base_url = "https://s3-eu-west-1.amazonaws.com/sixfifty/"
         files = [
             # (source_filename, target_filename)
-            ('polls.csv', 'general_election-uk-polls.csv'),
-            ('polls_london.csv', 'general_election-london-polls.csv'),
-            ('polls_scotland.csv', 'general_election-scotland-polls.csv'),
-            ('polls_wales.csv', 'general_election-wales-polls.csv'),
-            ('polls_ni.csv', 'general_election-ni-polls.csv'),
+            ("polls.csv", "general_election-uk-polls.csv"),
+            ("polls_london.csv", "general_election-london-polls.csv"),
+            ("polls_scotland.csv", "general_election-scotland-polls.csv"),
+            ("polls_wales.csv", "general_election-wales-polls.csv"),
+            ("polls_ni.csv", "general_election-ni-polls.csv"),
         ]
         for source_filename, target_filename in files:
             response = requests.get(base_url + source_filename)
             if response.status_code == 200:
-                with open(target_directory / target_filename, 'wb') as f:
+                with open(target_directory / target_filename, "wb") as f:
                     f.write(response.content)
-                print(f'Successfully downloaded raw data into {target_directory.resolve()}')
+                print(f"Successfully downloaded raw data into {target_directory.resolve()}")
             else:
-                warnings.warn(f'Received status 404 when trying to retrieve {base_url}{source_filename}')
+                warnings.warn(f"Received status 404 when trying to retrieve {base_url}{source_filename}")
 
     def process(self):
         pass
diff --git a/maven/get.py b/maven/get.py
index 94a22bd..feb5387 100644
--- a/maven/get.py
+++ b/maven/get.py
@@ -11,7 +11,7 @@ from pathlib import Path
 from .datasets import general_election
 
 
-def get(name, data_directory=Path('.'), retrieve=True, process=True):
+def get(name, data_directory=Path("."), retrieve=True, process=True):
     """Core data getter function.
 
     Args:
@@ -24,12 +24,11 @@ def get(name, data_directory=Path('.'), retrieve=True, process=True):
     Returns: Nothing (datasets are placed into current working directory).
     """
     mapper = {
-        'general-election/UK/2010/results': general_election.UK2010Results,
-        'general-election/UK/2015/model': general_election.UK2015Model,
-        'general-election/UK/2015/results': general_election.UK2015Results,
-        'general-election/UK/2015/results_HoC': general_election.UK2015ResultsHoC,
-        'general-election/UK/2017/results_HoC': general_election.UK2017ResultsHoC,
-        'general-election/UK/polls': general_election.UKPolls,
+        "general-election/UK/2010/results": general_election.UK2010Results,
+        "general-election/UK/2015/model": general_election.UK2015Model,
+        "general-election/UK/2015/results": general_election.UK2015Results,
+        "general-election/UK/2017/results": general_election.UK2017Results,
+        "general-election/UK/polls": general_election.UKPolls,
     }
     if name not in mapper:
         raise KeyError(f"'{name}' not found in datasets.")
diff --git a/maven/utils.py b/maven/utils.py
new file mode 100644
index 0000000..e8ddbdf
--- /dev/null
+++ b/maven/utils.py
@@ -0,0 +1,81 @@
+"""
+Various helper functions.
+"""
+
+import pandas as pd
+
+
+def sanitise(x):
+    return x.lower().replace(" ", "_")
+
+
+def process_hoc_sheet(input_file, data_dir, sheet_name):
+    # Import general election results
+    print(f"Read and clean {input_file}")
+    parties = ["Con", "LD", "Lab", "UKIP", "Grn", "SNP", "PC", "DUP", "SF", "SDLP", "UUP", "APNI", "Other"]
+    results = pd.read_excel(
+        data_dir / "raw" / input_file, sheet_name=sheet_name, skiprows=4, header=None, skipfooter=19
+    )
+    assert results.shape == (650, 49)
+
+    # Specify columns (spread across multiple rows in Excel)
+    cols = ["", "id", "Constituency", "County", "Country/Region", "Country", "Electorate", ""]
+    for party in parties:
+        cols += [f"{party}_Votes", f"{party}_Voteshare", ""]
+    cols += ["Total votes", "Turnout"]
+    results.columns = cols
+
+    # Some basic data quality checks
+    for party in parties:
+        assert (results[f"{party}_Voteshare"] - results[f"{party}_Votes"] / results["Total votes"]).sum() == 0
+    assert (results[[f"{party}_Votes" for party in parties]].fillna(0.0).sum(axis=1) == results["Total votes"]).all()
+    assert ((results["Total votes"] / results["Electorate"]) == results["Turnout"]).all()
+
+    # Drop blank columns plus those that can be calculated
+    cols_to_drop = [""] + [c for c in cols if "Voteshare" in c] + ["Total votes", "Turnout"]
+    results = results.drop(columns=cols_to_drop)
+
+    # Sanitise column names
+    results.columns = [sanitise(c) for c in results.columns]
+    results = results.rename(columns={"id": "ons_id", "country/region": "region"})
+    results.columns = [c.replace("_votes", "") for c in results.columns]
+
+    # Reshape to long
+    results_long = pd.melt(
+        results,
+        id_vars=["ons_id", "constituency", "county", "region", "country", "electorate"],
+        var_name="party",
+        value_name="votes",
+    )
+    assert results.shape == (650, 19)
+    assert results_long.shape == (650 * len(parties), 19 - len(parties) + 2)
+
+    # Sort by (ons_id, party)
+    results_long["party"] = pd.Categorical(
+        results_long.party, categories=pd.Series(parties).apply(sanitise), ordered=True
+    )
+    results_long = results_long.sort_values(["ons_id", "party"]).reset_index(drop=True)
+
+    # Re-add total_votes & voteshare
+    results_long["total_votes"] = results_long.ons_id.map(results_long.groupby("ons_id").votes.sum().astype(int))
+    results_long["voteshare"] = results_long["votes"] / results_long["total_votes"]
+    results_long["turnout"] = results_long["total_votes"] / results_long["electorate"]
+
+    # Reorder cols for export
+    results_long = results_long[
+        [
+            "ons_id",
+            "constituency",
+            "county",
+            "region",
+            "country",
+            "electorate",
+            "total_votes",
+            "turnout",
+            "party",
+            "votes",
+            "voteshare",
+        ]
+    ].copy()
+
+    return results_long


Test patch:
diff --git a/tests/datasets/general_election/test_uk_results.py b/tests/datasets/general_election/test_uk_results.py
new file mode 100644
index 0000000..1ea0bc5
--- /dev/null
+++ b/tests/datasets/general_election/test_uk_results.py
@@ -0,0 +1,51 @@
+"""
+Running tests in development:
+    $ cd /path/to/repo
+    $ python -m pytest
+
+Running tests against installed version (either `pip install .` or `pip install maven`):
+    $ cd /path/to/repo
+    $ pytest
+"""
+
+from pathlib import Path
+
+import pandas as pd
+
+import maven
+
+
+def check_uk_hoc_results_data(identifier, processed_filename):
+    maven.get(identifier, data_directory="./data/")
+    df = pd.read_csv(Path("./data") / identifier / "processed" / processed_filename)
+    assert df.shape == (8450, 10)
+    assert df.columns.tolist() == [
+        "ons_id",
+        "constituency",
+        "county",
+        "region",
+        "country",
+        "electorate",
+        "party",
+        "votes",
+        "total_votes",
+        "voteshare",
+    ]
+
+
+def test_uk_2010_results():
+    check_uk_hoc_results_data(
+        identifier="general-election/UK/2010/results", processed_filename="general_election-uk-2010-results.csv"
+    )
+
+
+def test_uk_2015_results():
+    check_uk_hoc_results_data(
+        identifier="general-election/UK/2015/results", processed_filename="general_election-uk-2015-results.csv"
+    )
+
+
+def test_uk_2017_results():
+    check_uk_hoc_results_data(
+        identifier="general-election/UK/2017/results", processed_filename="general_election-uk-2017-results.csv"
+    )
diff --git a/tests/test_get.py b/tests/test_get.py
index 700543b..57eb15e 100644
--- a/tests/test_get.py
+++ b/tests/test_get.py
@@ -8,17 +8,15 @@ Running tests against installed version (either `pip install .` or `pip install
     $ pytest
 """
 
-import pytest
-
 import maven
+import pytest
 
 
 def test_nonexisting_identifier():
     with pytest.raises(KeyError):
-        maven.get('this-identifier-will-never-exist')
+        maven.get("this-identifier-will-never-exist", data_directory="./data/")
 
 
-def test_process_with_retrieve():
-    # TODO: This is a useful test for now but we should actually handle this explicitly with a better error message.
-    with pytest.raises(FileNotFoundError):
-        maven.get('general-election/UK/2015/results', retrieve=False, process=True)
+def test_nothing_happens():
+    """Setting retrieve=False and process=False should do nothing."""
+    maven.get("general-election/UK/2010/results", retrieve=False, process=False)
diff --git a/tests/test_pipelines_GE.py b/tests/test_pipelines_GE.py
deleted file mode 100644
index 1f03179..0000000
--- a/tests/test_pipelines_GE.py
+++ /dev/null
@@ -1,22 +0,0 @@
-import pytest
-from pathlib import Path
-
-import maven
-import pandas as pd
-
-
-def test_UK2017ResultsHoc():
-    name = 'general-election/UK/2017/results_HoC'
-    maven.get(name)
-    output = Path(name) / Path('processed') / Path('general_election-uk-2017-results.csv')
-    df = pd.read_csv(output)
-    print(list(df.columns))
-
-
-def test_UK2015ResultsHoc():
-    name = 'general-election/UK/2015/results_HoC'
-    maven.get(name)
-
-    output = Path(name) / Path('processed') / Path('general_election-uk-2015-results.csv')
-    df = pd.read_csv(output)
-    print(list(df.columns))