Repository: moj-analytical-services/data_linter (language: python)

Issue:
Tests for different scenarios
- `[init] -> [worker]*4 -> [closedown]` DONE
- `[worker]*4 -> [closedown]` TODO
- `[worker] -> [closedown]` ??? Maybe not needed as this is just a single validator run. No closedown required
- `[init] -> [worker] -> [closedown]` DONE


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/Lint.yml b/.github/workflows/Lint.yml
index 3a2e419..b433e92 100644
--- a/.github/workflows/Lint.yml
+++ b/.github/workflows/Lint.yml
@@ -40,7 +40,7 @@ jobs:
 
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
+          python -m pip install --upgrade pip==20.3.3
           pip install flake8 yamllint
           if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
 
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1812a7d..08b1e95 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,34 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## 5.0.0 - 2021-03-15
+
+- Created Pandas Validator (now replaces frictionless as the default validator) ([issue #120](https://github.com/moj-analytical-services/data_linter/issues/120), [issue #98](https://github.com/moj-analytical-services/data_linter/issues/98))
+- Enabled parallelisation of validators ([#122](https://github.com/moj-analytical-services/data_linter/issues/122))
+- Migrated to the new metadata schemas ([#140](https://github.com/moj-analytical-services/data_linter/issues/140))
+- Single process validator can run locally ([#121](https://github.com/moj-analytical-services/data_linter/issues/121))
+- Full log now writes to it's own folder ([#130](https://github.com/moj-analytical-services/data_linter/issues/130))
+
+```
+[ALL CLOSED ISSUES]
+- issue #140 
+- issue #139
+- issue #133 
+- issue #132
+- issue #131 
+- issue #130 
+- issue #129 
+- issue #128 
+- issue #125
+- issue #122 
+- issue #121 
+- issue #120 
+- issue #110 
+- issue #100 
+- issue #98
+- issue #87
+- issue #70
+```
 
 ## 4.1.0 - 2020-11-24
 
diff --git a/README.md b/README.md
index 4d4109a..82f0e4a 100644
--- a/README.md
+++ b/README.md
@@ -1,29 +1,40 @@
 # Data Linter
 
-A python package to to allow automatic validation of data as part of a Data Engineering pipeline. It is designed to automate the process of moving data from Land to Raw-History as described in the [ETL pipline guide](https://github.com/moj-analytical-services/etl-pipeline-example)
-
-This package implements different validators using different packages:
+A python package to to allow automatic validation of data as part of a Data Engineering pipeline. It is designed to read in and validate tabular data against a given schema for the data. The schemas provided adhere to [our metadata schemas standards](https://github.com/moj-analytical-services/mojap-metadata/) for data. This package can also be used to manage movement of data from a landing area (s3 or locally) to a new location based on the result of the validation.
 
+This package implements different validators using different packages based on the users preference:
+- `pandas`: (default) Uses our own lightweight pandas dataframe operations to run simple validation tests on the columns based on the datatype and additional tags in the metadata. Utilises pyarrow for reading data.
 - `frictionless`: Uses Frictionless data to validate the data against our metadata schemas. More information can be found [here](https://github.com/frictionlessdata/frictionless-py/)
-- `great-expectations`: Uses the Great Expectations data to validate the data against our metadata schemas. More information can be found [here](https://github.com/frictionlessdata/frictionless-py/)
+- `great-expectations`: Uses the Great Expectations data to validate the data against our metadata schemas. More information can be found [here](https://github.com/great-expectations/great_expectations)
 
 
 ## Installation
 
 ```bash
-pip install data_linter # frictionless only
+pip install data_linter # pandas validator only
 ```
 
+Or to install the necessary dependencies for the non-default validators.
+
 ```bash
+pip install data_linter[frictionless] # To include packages required for the frictionless validator
+
+# OR
+
 pip install data_linter[ge] # To include packages required for teh great-expectations validator
 ```
 
+
 ## Usage
 
-This package takes a `yaml` based config file written by the user (see example below), and validates data in the specified Land bucket against specified metadata. If the data conforms to the metadata, it is moved to the specified Raw bucket for the next step in the pipeline. Any failed checks are passed to a separate bucket for testing. The package also generates logs to allow you to explore issues in more detail.
+This package takes a `yaml` based config file written by the user (see example below), and validates data in the specified s3 folder path against specified metadata. If the data conforms to the metadata, it is moved to the specified s3 folder path for the next step in the pipeline (note can also provide local paths for running locally). Any failed checks are passed to a separate location for testing. The package also generates logs to allow you to explore issues in more detail.
+
+### Simple Use
 
 To run the validation, at most simple you can use the following:
 
+**In Python:**
+
 ```python
 from data_linter import run_validation
 
@@ -32,14 +43,20 @@ config_path = "config.yaml"
 run_validation(config_path)
 ```
 
-## Example config file
+**Via command line:**
+
+```bash
+data_linter --config_path config.yaml
+```
+
+### Example config file
 
 ```yaml
-land-base-path: s3://land-bucket/my-folder/  # Where to get the data from
-fail-base-path: s3://fail-bucket/my-folder/  # Where to write the data if failed
-pass-base-path: s3://pass-bucket/my-folder/  # Where to write the data if passed
-log-base-path: s3://log-bucket/my-folder/  # Where to write logs
-compress-data: true  # Compress data when moving elsewhere
+land-base-path: s3://testing-bucket/land/  # Where to get the data from
+fail-base-path: s3://testing-bucket/fail/  # Where to write the data if failed
+pass-base-path: s3://testing-bucket/pass/  # Where to write the data if passed
+log-base-path: s3://testing-bucket/logs/  # Where to write logs
+compress-data: true  # Compress data when moving elsewhere (only applicable from CSV/JSON)
 remove-tables-on-pass: true  # Delete the tables in land if validation passes
 all-must-pass: true  # Only move data if all tables have passed
 fail-unknown-files:
@@ -47,14 +64,15 @@ fail-unknown-files:
         - additional_file.txt
         - another_additional_file.txt
 
-validator-engine: frictionless # will default to this if unspecified
+validator-engine: pandas # will default to this if unspecified
+# (but other options are `frictionless` and `great-expectations`)
 
 # Tables to validate
 tables:
     table1:
         required: true  # Does the table have to exist
-        pattern: null  # Assumes file is called table1
-        metadata: meta_data/table1.json
+        pattern: null  # Assumes file is called table1 (same as key)
+        metadata: meta_data/table1.json # local path to metadata
 
     table2:
         required: true
@@ -98,8 +116,134 @@ for table in ["table1", "table2"]:
 run_validation(base_config) # Then watch that log go...
 ```
 
+### Validating a single file
+
+> Without all the bells and whistles
+
+If you do not need `data_linter` to match files to a specified config, log the process and then move data around based on the outcome of the validation you can just use the validators themselves:
+
+```python
+# Example using simple pandas validatior (without added data_linter features)
+import json
+from data_linter.validators import PandasValidator
+
+filepath = "tests/data/end_to_end1/land/table1.csv"
+table_params = {
+    "expect-header": True
+}
+with open("tests/data/end_to_end1/meta_data/table1.json") as f:
+    metadata = json.load(f)
+
+pv = PandasValidator(filepath, table_params, metadata)
+pv.read_data_and_validate()
+
+pv.valid  # True (data in table1.csv is valid against metadata)
+pv.response.get_result()  # Returns dict of all tests ran against data
+
+# The response object of for the PandasValidator in itself, and has it's own functions
+pv.get_names_of_column_failures()  # [], i.e. no cols failed
+```
+
+
+### Parallel Running
+
+Data Linter can also work in parallel to trigger multiple validations at once (only supports use of S3 atm). An example below:
+
+In this scenario we use the parallisation process to init the process split the job into 4 validators and then run the closedown.
+
+- **The init stage** splits the config into 4 chunks, based on the file size of the data sitting in the specified land path. It split configs are written to a temporary path in S3 for each validator to pick up and run in parallel.
+- **The validator stage** can be ran in parallel (for simplicity they are run sequentially in the example below). Each validator will take the config in the temp folder path and process the files given in that subsetting config.
+- **The closedown stage** will take all the logs all validator runs, conbine them then move the data based on the validators results. It will then finally clean up the temp folder.
+
+```python
+# Simple example running DL with multiple validators (in this case 4)
+# [init] -> [validator]x4 -> [closedown]
+import yaml
+from data_linter import validation
+from dataengineeringutils3.s3 import get_filepaths_from_s3_folder
+
+
+simple_yaml_config = """
+land-base-path: s3://land/
+fail-base-path: s3://fail/
+pass-base-path: s3://pass/
+log-base-path: s3://log/
+
+compress-data: false
+remove-tables-on-pass: true
+all-must-pass: true
+
+# Tables to validate
+tables:
+  table1:
+    required: true
+    metadata: tests/data/end_to_end1/meta_data/table1.json
+    expect-header: true
+
+  table2:
+    required: true
+    pattern: ^table2
+    metadata: tests/data/end_to_end1/meta_data/table2.json
+"""
+
+test_folder = "tests/data/end_to_end1/land/"
+config = yaml.safe_load(simple_yaml_config)
+
+# Init stage
+validation.para_run_init(4, config)
+
+# Validation stage (although ran sequencially this can be ran in parallel)
+for i in range(4):
+    validation.para_run_validation(i, config)
+
+# Closedown stage
+validation.para_collect_all_status(config)
+validation.para_collect_all_logs(config)
+```
+
+> There are more parallelisation examples, which can be found in the [test_simple_examples.py test module](tests/test_simple_examples.py)
+
 ## Validators
 
+### Pandas
+
+THis is the default validator used by data_linter as of the version 5 release.
+
+#### Dealing with timestamps and dates
+
+Timestamps are always a pain to deal with especially when using different file types. The Pandas Validator has tried to keep true to the file types based on the tests it runs.
+
+If the file type stores date/timestamp information as a string (i.e. CSV and JSONL) then the pandas Validator will read in the timestamp / date columns as strings. It will then apply validation tests against those columns checking if the string representation of the dates in the data is a valid date. For timestamp and date types these tests assume ISO standard string representation `%Y-%m-%d %H:%M:%S` and `%Y-%m-%d`. If your timestamp/date types are comming in as strings that do not conform to the ISO standard format then you can provide you column in the metadata with a `datetime_format` property that specifies the exected format e.g.
+
+```json
+...
+"columns": [
+    {
+        "name": "date_in_uk",
+        "type": "date64",
+        "datetime_format": "%d/%m/%Y"
+    },
+...
+```
+
+Often you might recieve data that is exported from a system that might encode your timestamp as a date but is written to a format that encodes the data as a timestamp. In this scenario you would expect your dates (in a str timestamp format) to always have a time component of `00:00:00`. You can also use data_linter to validate this by specifing the datetime_format of your column as the expected timestamp in format but still specify that the data type is a date e.g.
+
+```json
+...
+"columns": [
+    {
+        "name": "date_in_uk",
+        "type": "date64",
+        "datetime_format": "%d/%m/%Y 00:00:00"
+    },
+...
+```
+
+In the above data_linter will attempt to fist parse the column with the specified `datetime_format` and then as the column type is date it will check that it it truely a date (and not have a time component).
+
+If the file_format is `parquet` then timestamps are encoded in the filetype and there are just read in as is. Currently data_linter doesn't support minimum and maximum tests for timestamps/dates and also does not currently have tests for time types. 
+
+
 ### Frictionless
 
 Known errors / gotchas:
diff --git a/data_linter/__init__.py b/data_linter/__init__.py
index e40c72d..ba7be38 100644
--- a/data_linter/__init__.py
+++ b/data_linter/__init__.py
@@ -1,8 +1,1 @@
-import sys
-
-if sys.version_info[1] < 8:
-    from importlib_metadata import version
-else:
-    from importlib.metadata import version
-
-__version__ = version("data_linter")
+__version__ = "5.0.0"
diff --git a/data_linter/logging_functions.py b/data_linter/logging_functions.py
index fac3c3b..b22bd2a 100644
--- a/data_linter/logging_functions.py
+++ b/data_linter/logging_functions.py
@@ -1,7 +1,9 @@
 import logging
 import io
 import boto3
+import os
 from typing import Tuple
+from datetime import datetime
 
 from dataengineeringutils3.s3 import s3_path_to_bucket_key
 
@@ -28,7 +30,7 @@ def logging_setup() -> Tuple[logging.Logger, io.StringIO]:
     handler = logging.StreamHandler(log_stringio)
 
     log_formatter = logging.Formatter(
-        fmt="%(asctime)s | %(module)s | %(levelname)s | %(context)s | %(message)s",
+        fmt="%(asctime)s | %(funcName)s | %(levelname)s | %(context)s | %(message)s",
         datefmt="%Y-%m-%d %H:%M:%S",
     )
     handler.setFormatter(log_formatter)
@@ -46,13 +48,72 @@ def logging_setup() -> Tuple[logging.Logger, io.StringIO]:
     return log, log_stringio
 
 
-def upload_log(log: logging.Logger, log_stringio: io.StringIO, s3_path: str):
-    if s3_path:
-        s3_client = boto3.client("s3")
-        b, k = s3_path_to_bucket_key(s3_path)
-        s3_client.put_object(Body=log_stringio.getvalue(), Bucket=b, Key=k)
+def upload_log(log: logging.Logger, log_stringio: io.StringIO, log_path: str):
+    log_path_is_s3 = log_path.startswith("s3://")
+
+    if log_path:
+        if log_path_is_s3:
+            s3_client = boto3.client("s3")
+            b, k = s3_path_to_bucket_key(log_path)
+            s3_client.put_object(Body=log_stringio.getvalue(), Bucket=b, Key=k)
+        else:
+            dir_out = os.path.dirname(log_path)
+            if not os.path.exists(dir_out):
+                os.makedirs(dir_out, exist_ok=True)
+            with open(log_path, "w") as log_out:
+                log_out.write(log_stringio.getvalue())
     else:
         log.error(
             "An error occurred but no log path registered, "
-            "likely due to issue with config, so cannot upload to log S3."
+            "likely due to issue with config, so logs not saved."
         )
+
+
+def get_log_fn() -> str:
+    return f"data-linter-{int(datetime.utcnow().timestamp())}.log"
+
+
+def get_temp_log_basepath(config: dict) -> str:
+    """
+        Defines temp log basepath for parallel runs
+
+    Args:
+        config (dict): A data linter config
+
+    Returns:
+        str: tmp base path for log for a parallelised run
+    """
+    temp_log_basepath = os.path.join(
+        config["log-base-path"], "data_linter_temporary_fs/"
+    )
+    return temp_log_basepath
+
+
+def get_temp_log_path_from_config(config: dict) -> str:
+    """
+        Defines temp log path for parallel runs
+
+    Args:
+        config (dict): A data linter config
+
+    Returns:
+        str: tmp path for log for a parallelised run
+    """
+    temp_log_path = os.path.join(get_temp_log_basepath(config), "init", get_log_fn())
+    return temp_log_path
+
+
+def get_main_log_path_from_config(config: dict) -> str:
+    """
+        Defines main log file path for a data linter run
+
+    Args:
+        config (dict): A data linter config
+
+    Returns:
+        str: master log file path for a datalinter run
+    """
+    log_path = os.path.join(
+        config["log-base-path"], "data-linter-main-logs", get_log_fn()
+    )
+    return log_path
diff --git a/data_linter/schemas/config-schema.json b/data_linter/schemas/config-schema.json
index deb238e..fb95df6 100644
--- a/data_linter/schemas/config-schema.json
+++ b/data_linter/schemas/config-schema.json
@@ -31,14 +31,16 @@
             "type": "string",
             "title": "The validator-engine to use",
             "description": "If you want to use frictionless or great-expectations",
-            "default": "frictionless",
+            "default": "pandas",
             "examples": [
-                "frictionless",
-                "great-expectations"
+                "pandas",
+                "great-expectations",
+                "frictionless"
             ],
             "enum": [
-                "frictionless",
-                "great-expectations"
+                "pandas",
+                "great-expectations",
+                "frictionless"
             ]
         },
         "validator-engine-params": {
@@ -54,9 +56,9 @@
             "title": "The Land-base-path Schema",
             "description": "This is where data should arrive for a pipeline",
             "default": "",
-            "pattern": "^s3://",
             "examples": [
-                "s3://land-bucket/my-folder/"
+                "s3://land-bucket/my-folder/",
+                "data/land"
             ]
         },
         "fail-base-path": {
@@ -68,9 +70,9 @@
             "title": "The Fail-base-path Schema",
             "description": "Data which does not pass validation will end up here.",
             "default": "",
-            "pattern": "^s3://",
             "examples": [
-                "s3://fail-bucket/my-folder/"
+                "s3://fail-bucket/my-folder/",
+                "data/fail"
             ]
         },
         "pass-base-path": {
@@ -79,9 +81,9 @@
             "title": "The Pass-base-path Schema",
             "description": "Data which passes validation will end up here",
             "default": "",
-            "pattern": "^s3://",
             "examples": [
-                "s3://pass-bucket/my-folder/"
+                "s3://pass-bucket/my-folder/",
+                "data/pass"
             ]
         },
         "log-base-path": {
@@ -90,9 +92,9 @@
             "title": "The Log-base-path Schema",
             "description": "This validation will produce detailed logs. They will be stored here.",
             "default": "",
-            "pattern": "^s3://",
             "examples": [
-                "s3://log-bucket/my-folder/"
+                "s3://log-bucket/my-folder/",
+                "data/log"
             ]
         },
         "iam-role-name": {
@@ -135,6 +137,16 @@
                 true
             ]
         },
+        "run-paralell": {
+            "$id": "#/properties/run-paralell",
+            "type": "boolean",
+            "title": "The Run-paralell Schema",
+            "description": "Process tables in paralell",
+            "default": false,
+            "examples": [
+                true
+            ]
+        },
         "fail-unknown-files": {
             "$id": "#/properties/fail-unknown-files",
             "type": "object",
diff --git a/data_linter/utils.py b/data_linter/utils.py
index 4f04b73..a966b9e 100644
--- a/data_linter/utils.py
+++ b/data_linter/utils.py
@@ -10,6 +10,8 @@
 from dataengineeringutils3.s3 import (
     s3_path_to_bucket_key,
     write_local_file_to_s3,
+    copy_s3_object,
+    check_for_s3_file,
 )
 
 
@@ -22,16 +24,38 @@ def download_data(s3_path: str, local_path: str):
         s3_client.download_fileobj(b, o, f)
 
 
-def compress_data(s3_download_path: str, s3_upload_path: str):
-    s3_client = boto3.client("s3")
+def compress_data(download_path: str, upload_path: str):
+
+    download_path_is_s3 = download_path.startswith("s3://")
+    upload_path_is_s3 = upload_path.startswith("s3://")
+
+    if download_path_is_s3:
+        s3_client = boto3.client("s3")
+
+    if not upload_path_is_s3:
+        upload_path_dir = os.path.dirname(upload_path)
+        if not os.path.exists(upload_path_dir):
+            os.makedirs(upload_path_dir, exist_ok=True)
+
     with tempfile.TemporaryDirectory() as temp_dir:
-        bucket, key = s3_path_to_bucket_key(s3_download_path)
-        temp_file = os.path.join(temp_dir, key.split("/")[-1])
-        with open(temp_file, "wb") as opened_temp_file:
-            s3_client.download_fileobj(bucket, key, opened_temp_file)
+
+        if download_path_is_s3:
+            bucket, key = s3_path_to_bucket_key(download_path)
+            temp_file = os.path.join(temp_dir, key.split("/")[-1])
+            with open(temp_file, "wb") as opened_temp_file:
+                s3_client.download_fileobj(bucket, key, opened_temp_file)
+        else:
+            temp_file = os.path.join(temp_dir, download_path.split(os.path.sep)[-1])
+            with open(temp_file, "wb") as opened_temp_file:
+                shutil.copy(download_path, temp_file)
+
         with open(temp_file, "rb") as f_in, gzip.open(temp_file + ".gz", "wb") as f_out:
             shutil.copyfileobj(f_in, f_out)
-        write_local_file_to_s3(temp_file + ".gz", s3_upload_path, overwrite=True)
+
+        if upload_path_is_s3:
+            write_local_file_to_s3(temp_file + ".gz", upload_path, overwrite=True)
+        else:
+            shutil.copy(temp_file + ".gz", upload_path)
 
 
 def get_out_path(
@@ -43,6 +67,7 @@ def get_out_path(
     filenum: int = 0,
     timestamp_partition_name: Union[str, None] = None,
 ) -> str:
+
     filename_only, ext = filename.split(".", 1)
     final_filename = f"{filename_only}-{filenum}-{ts}.{ext}"
     if compress and not ext.endswith(".gz"):
@@ -57,7 +82,7 @@ def get_out_path(
     return out_path
 
 
-def get_log_path(basepath: str, table: str, ts: str, filenum: int = 0) -> str:
+def get_table_log_path(basepath: str, table: str, ts: str, filenum: int = 0) -> str:
     final_filename = f"log-{table}-{filenum}-{ts}.json"
 
     out_path = os.path.join(basepath, table, final_filename)
@@ -76,3 +101,84 @@ def local_file_to_s3(local_path: str, s3_path: str):
     b, o = s3_path_to_bucket_key(s3_path)
     with open(local_path, "rb") as f:
         s3_client.upload_fileobj(f, b, o)
+
+
+def s3_to_local(s3_path: str, local_path: str):
+    s3_client = boto3.client("s3")
+    bucket, key = s3_path_to_bucket_key(s3_path)
+
+    with open(local_path, "wb") as opened_file:
+        s3_client.download_fileobj(bucket, key, opened_file)
+
+
+def copy_data(src_path: str, dst_path: str):
+    src_path_is_s3 = src_path.startswith("s3://")
+    dst_path_is_s3 = dst_path.startswith("s3://")
+
+    if not dst_path_is_s3:
+        dst_path_dir = os.path.dirname(dst_path)
+        if not os.path.exists(dst_path_dir):
+            os.makedirs(dst_path_dir, exist_ok=True)
+
+    if src_path_is_s3 and dst_path_is_s3:
+        copy_s3_object(src_path, dst_path)
+    elif src_path_is_s3 and not dst_path_is_s3:
+        s3_to_local(src_path, dst_path)
+    elif not src_path_is_s3 and dst_path_is_s3:
+        write_local_file_to_s3(src_path, dst_path, overwrite=True)
+    elif not src_path_is_s3 and not dst_path_is_s3:
+        shutil.copyfile(src_path, dst_path)
+
+
+def get_filepaths_from_local_folder(
+    land_base_path: str,
+    file_extension: Union[None, str] = None,
+    exclude_zero_byte_files: bool = True,
+) -> list:
+
+    ret_file_paths = []
+
+    while land_base_path.endswith(os.path.sep):
+        land_base_path = land_base_path[:-1]
+
+    for curr_dir, _, file_names in os.walk(land_base_path):
+        file_names = [i for i in file_names if not i.startswith(".")]
+
+        if file_extension:
+            file_names = [i for i in file_names if i.endswith(file_extension)]
+
+        file_paths = [curr_dir + os.path.sep + i for i in file_names]
+
+        if exclude_zero_byte_files:
+            file_paths = [i for i in file_paths if os.stat(i).st_size != 0]
+
+        ret_file_paths.extend(file_paths)
+
+    return ret_file_paths
+
+
+def read_all_file_body(file_path: str) -> str:
+    """
+    Returns the text content of a file (will decode bytes if file read is bytes like)
+
+    Args:
+        file_path: A string specifying the location of the file to load text from.
+        can be s3 or local
+    """
+    file_path_is_s3 = file_path.startswith("s3://")
+
+    if file_path_is_s3:
+        s3_client = boto3.client("s3")
+        if not check_for_s3_file(file_path):
+            raise FileNotFoundError("Path to config: {file_path}. Not found.")
+        bucket, key = s3_path_to_bucket_key(file_path)
+        file_obj = s3_client.get_object(Bucket=bucket, Key=key)
+        file_obj_body = file_obj["Body"].read()
+    else:
+        with open(file_path) as f_in:
+            file_obj_body = f_in.read()
+
+    if isinstance(file_obj_body, bytes):
+        return file_obj_body.decode("utf-8")
+    else:
+        return file_obj_body
diff --git a/data_linter/validation.py b/data_linter/validation.py
index a56665f..b6e6376 100644
--- a/data_linter/validation.py
+++ b/data_linter/validation.py
@@ -2,79 +2,79 @@
 import yaml
 import json
 import re
-import logging
+import tempfile
+import boto3
+import shutil
+import io
 
-from typing import Union
+from typing import Union, List
 
 from datetime import datetime
 
 from jsonschema import validate as json_validate
 
+from copy import deepcopy
+
 from dataengineeringutils3.s3 import (
     get_filepaths_from_s3_folder,
-    copy_s3_object,
     delete_s3_object,
     write_json_to_s3,
+    s3_path_to_bucket_key,
+    delete_s3_folder_contents,
 )
 
-import boto3
-from botocore.client import Config
-
 from data_linter.constants import config_schema
 
 from data_linter.logging_functions import (
     upload_log,
     logging_setup,
+    get_temp_log_path_from_config,
+    get_main_log_path_from_config,
+    get_temp_log_basepath,
 )
 
 from data_linter.utils import (
     get_out_path,
-    get_log_path,
+    get_table_log_path,
     compress_data,
+    copy_data,
+    get_filepaths_from_local_folder,
+    local_file_to_s3,
+    read_all_file_body,
 )
 
 from data_linter.validators import (
     FrictionlessValidator,
     GreatExpectationsValidator,
+    PandasValidator,
 )
 
-boto3_config = Config(read_timeout=120)
-s3_client = boto3.client("s3", config=boto3_config)
-
-log = logging.getLogger("root")
+log, log_stringio = logging_setup()
 
 get_validator = {
+    "pandas": PandasValidator,
     "frictionless": FrictionlessValidator,
     "great-expectations": GreatExpectationsValidator,
 }
 
 
-def get_validator_name() -> str:
-    validator_name = os.getenv("VALIDATOR_NAME")
-    if not validator_name:
-        validator_name = "de-data-validator"
-    validator_name += f"-{int(datetime.utcnow().timestamp())}"
-    return validator_name
-
-
-def load_and_validate_config(config_path: str = "config.yaml") -> dict:
+def load_and_validate_config(config: Union[str, dict] = "config.yaml") -> dict:
     """
     Loads and validates the config
     """
 
-    # load yaml or json
-    if not os.path.isfile(config_path):
-        config_path = config_path.replace("yaml", "yml")
-        if not os.path.isfile(config_path):
-            raise FileNotFoundError(f"Expecting a file in path given {config_path}.")
-
-    with open(config_path, "r") as f:
-        config = yaml.safe_load(f)
+    if isinstance(config, str):
+        config_raw_text = read_all_file_body(config)
+        config = yaml.safe_load(config_raw_text)
+    elif isinstance(config, dict):
+        pass
+    else:
+        raise TypeError("Input 'config' must be a str or dict.")
 
-    return validate_and_clean_config(config)
+    return _validate_and_clean_config(config)
 
 
-def validate_and_clean_config(config: dict) -> dict:
+def _validate_and_clean_config(config: dict) -> dict:
     """Validates a config as a dict. And adds default
     properties to that config.
 
@@ -97,16 +97,20 @@ def validate_and_clean_config(config: dict) -> dict:
     return config
 
 
-def match_files_in_land_to_config(config) -> dict:
+def match_files_in_land_to_config(config: dict) -> dict:
     """
     Takes config and matches files in S3 to the corresponding table list in config.
     Checks against other config parameters and raise error if config params not met.
     """
+
     land_base_path = config["land-base-path"]
-    land_files = get_filepaths_from_s3_folder(land_base_path)
+    if land_base_path.startswith("s3://"):
+        land_files = get_filepaths_from_s3_folder(land_base_path)
+    else:
+        land_files = get_filepaths_from_local_folder(land_base_path)
 
     if not land_files and config.get("fail-no-files", False):
-        raise FileNotFoundError(f"No files found in the s3 path: {land_base_path}")
+        raise FileNotFoundError(f"No files found in the path: {land_base_path}")
     else:
         total_files = len(land_files)
         log.info(f"Found {total_files} in {land_base_path}")
@@ -158,27 +162,197 @@ def match_files_in_land_to_config(config) -> dict:
     return config
 
 
-def validate_data(config: dict):
+def run_validation(config: Union[str, dict] = "config.yaml"):
+    """
+    Runs end to end validation based on config.
+
+    Args:
+        config (Union[str, dict], optional): Either a string specifying the path to a
+        config yaml. Or a dict of a config in memory. Defaults to "config.yaml".
+
+    Raises:
+        Error: States where log is written if error is hit in validation and then raises
+        traceback.
+    """
+
+    log.info("Loading config")
+    log_path = None
+    try:
+        config = load_and_validate_config(config)
+        log_path = get_main_log_path_from_config(config)
+
+        log.info("Running validation")
+
+        config = match_files_in_land_to_config(config)
+
+        validate_data(config)
+        collect_all_status(config)
+
+    except Exception as e:
+        log_msg = f"Unexpected error. Uploading log to {log_path} before raising error."
+        error_msg = str(e)
+
+        log.error(log_msg)
+        log.error(error_msg)
+
+        upload_log(log, log_stringio, log_path)
+
+        raise e.with_traceback(e.__traceback__)
+    else:
+        upload_log(log, log_stringio, log_path)
+
+
+def bin_pack_configs(config: dict, max_bin_count: int):
+    """
+    creates up to max_bin_count of config files by splitting the files from the config
+    by size and grouping them into or below the average size of all the files
+
+    Args:
+        config: a config file specifying the files to be linted
+        max_bin_count: the maximum of bins to split the files up into - optimal number
+        is equal to the amount of workers available
+    """
+
+    log_base_path = config.get("log-base-path")
+    log_base_path_is_s3 = log_base_path.startswith("s3://")
+
+    if log_base_path_is_s3:
+        tmp_log_bp = get_temp_log_basepath(config)
+        s3_temp_path = os.path.join(tmp_log_bp, "configs")
+        file_list = []
+
+        # create a list of dictionaries, for each file with all attributes
+        for table_name, table in config["tables"].items():
+            table_sans_files = deepcopy(table)
+            mfiles = table_sans_files.pop("matched_files")
+
+            for file_name in mfiles:
+                table_sans_files["file-name"] = file_name
+                table_sans_files["table-name"] = table_name
+                file_list.append(deepcopy(table_sans_files))
+
+        # get the size of them all
+        acum_file_size = 0
+        for i, file_dict in enumerate(file_list):
+            s3_client = boto3.client("s3")
+            file_name = file_dict["file-name"]
+            bucket, key = s3_path_to_bucket_key(file_name)
+            obj = s3_client.get_object(Bucket=bucket, Key=key)
+            file_size = obj.get("ContentLength")
+            file_list[i]["file-size"] = file_size
+            acum_file_size += file_size
+
+        target_bin_size = acum_file_size / max_bin_count
+
+        # sort them in descending order
+        file_list.sort(key=lambda x: -x["file-size"])
+
+        bins = [None] * max_bin_count
+
+        offset = 0
+        for i in range(max_bin_count):
+            curr_bin = []
+            curr_bin_size = 0
+            has_been_binned = False
+            for j in range(offset, len(file_list)):
+                if len(curr_bin) == 0:
+                    curr_bin.append(file_list[j])
+                    curr_bin_size += file_list[j]["file-size"]
+                    offset += 1
+                else:
+                    if curr_bin_size <= target_bin_size:
+                        curr_bin.append(file_list[j])
+                        curr_bin_size += file_list[j]["file-size"]
+                        offset += 1
+                    else:
+                        bins[i] = curr_bin
+                        has_been_binned = True
+                        break
+            if not has_been_binned:
+                bins[i] = curr_bin
+
+        bins[i] = curr_bin
+        bins = [i for i in bins if i != []]
+
+        # create the configs for the given bins
+        for i, packed_bin in enumerate(bins):
+            config_n = deepcopy(config)
+            config_n.pop("tables")
+            config_n["tables"] = {}
+
+            for table in packed_bin:
+                curr_table_name = table.pop("table-name")
+
+                if config_n["tables"].get(curr_table_name):
+                    # it exists, so just add to matched files
+                    config_n["tables"][curr_table_name]["matched_files"].append(
+                        table["file-name"]
+                    )
+                else:
+                    # it doesn't exist, do a full copy of all attributes
+                    mfile = table.pop("file-name")
+                    table.pop("file-size")
+                    config_n["tables"][curr_table_name] = deepcopy(table)
+                    config_n["tables"][curr_table_name]["matched_files"] = []
+                    config_n["tables"][curr_table_name]["matched_files"].append(mfile)
+
+            # upload the config to temp storage, into config
+            with tempfile.NamedTemporaryFile(
+                suffix=".yml", prefix="config_"
+            ) as tmp_file:
+
+                with open(tmp_file.name, "w") as yaml_out:
+                    yaml.dump(config_n, yaml_out, default_flow_style=False)
+
+                tmp_file_name = tmp_file.name.split("/")[-1]
+                s3_out_path = os.path.join(s3_temp_path, str(i), tmp_file_name)
+                local_file_to_s3(tmp_file.name, s3_out_path)
+
+    else:
+        raise ValueError("Local land path not supported for parrallel running")
+
+
+def validate_from_chunked_configs(config: dict, config_num: int) -> bool:
 
-    utc_ts = int(datetime.utcnow().timestamp())
     land_base_path = config["land-base-path"]
-    all_must_pass = config.get("all-must-pass", False)
-    pass_base_path = config["pass-base-path"]
-    log_base_path = config["log-base-path"]
-    fail_base_path = config.get("fail-base-path")
-    remove_on_pass = config.get("remove-tables-on-pass")
-    compress = config.get("compress-data")
-    timestamp_partition_name = config.get("timestamp-partition-name")
-    validator_engine = config.get("validator-engine", "frictionless")
+    land_base_path_is_s3 = land_base_path.startswith("s3://")
+
+    if land_base_path_is_s3:
+        tmp_log_bp = get_temp_log_basepath(config)
+        s3_temp_path = os.path.join(tmp_log_bp, "configs", str(config_num))
+
+        config_file_paths = get_filepaths_from_s3_folder(s3_temp_path)
+        if not config_file_paths:
+            return False
+
+        s3_client = boto3.client("s3")
+
+        all_configs = []
+        for config_file_path in config_file_paths:
+            bucket, key = s3_path_to_bucket_key(config_file_path)
+            config_file_obj = s3_client.get_object(Bucket=bucket, Key=key)
+            all_configs.append(yaml.safe_load(config_file_obj["Body"].read()))
+
+        for config in all_configs:
+            validate_data(config)
+
+        return True
+
+    else:
+        raise ValueError("Local land path not supported for parrallel running")
+
+
+def validate_data(config: dict):
+
+    validator_engine = config.get("validator-engine", "pandas")
     validator_params = config.get("validator-engine-params", {})
-    config = match_files_in_land_to_config(config)
 
-    # If all the above passes lint each file
     all_table_responses = []
-    all_matched_files = []
-    overall_pass = True
+
     for table_name, table_params in config["tables"].items():
+
         table_params["lint-response"] = []
+
         if table_params["matched_files"]:
             log.info(f"Linting {table_name}")
 
@@ -190,173 +364,254 @@ def validate_data(config: dict):
                 metadata = json.load(sfile)
 
             for i, matched_file in enumerate(table_params["matched_files"]):
-                all_matched_files.append(matched_file)
-                log.info(f"...file {i+1} of {len(table_params['matched_files'])}")
-                file_basename = os.path.basename(matched_file)
 
+                log.info(f"...file {i+1} of {len(table_params['matched_files'])}")
                 validator = get_validator[validator_engine](
                     matched_file, table_params, metadata, **validator_params
                 )
+
                 validator.read_data_and_validate()
                 validator.write_validation_errors_to_log()
 
-                # response - needs to be standardised see issue #100
                 table_response = {
                     "valid": validator.valid,
                     "response": validator.get_response_dict(),
-                    "s3-original-path": matched_file,
+                    "original-path": matched_file,
                     "table-name": table_name,
                 }
 
-                # Write data to s3 on pass or elsewhere on fail
                 if table_response["valid"]:
-                    final_outpath = get_out_path(
-                        pass_base_path,
-                        table_name,
-                        utc_ts,
-                        file_basename,
-                        compress=compress,
-                        filenum=i,
-                        timestamp_partition_name=timestamp_partition_name,
-                    )
-
-                    table_response["archived-path"] = final_outpath
-                    if not all_must_pass:
-                        msg2 = f"...file passed. Writing to {final_outpath}"
-                        log.info(msg2)
-                        if compress:
-                            compress_data(matched_file, final_outpath)
-                        else:
-                            copy_s3_object(matched_file, final_outpath)
-                        if remove_on_pass:
-                            log.info(f"Removing {matched_file}")
-                            delete_s3_object(matched_file)
-                    else:
-                        log.info("File passed")
-
-                elif fail_base_path:
-                    overall_pass = False
-                    final_outpath = get_out_path(
-                        fail_base_path,
-                        table_name,
-                        utc_ts,
-                        file_basename,
-                        compress=compress,
-                        filenum=i,
-                        timestamp_partition_name=timestamp_partition_name,
-                    )
-                    table_response["archived-path"] = final_outpath
-                    if not all_must_pass:
-                        msg3 = f"...file failed. Writing to {final_outpath}"
-                        log.warning(msg3)
-                        if compress:
-                            compress_data(matched_file, final_outpath)
-                        else:
-                            copy_s3_object(matched_file, final_outpath)
-
+                    log.info("...file passed.")
                 else:
-                    overall_pass = False
-                    table_response["archived-path"] = None
-
-                # Write reponse log
-                log_outpath = get_log_path(log_base_path, table_name, utc_ts, filenum=i)
+                    log.info("...file failed.")
 
-                # Write log to s3
-                write_json_to_s3(table_response, log_outpath)
                 all_table_responses.append(table_response)
 
         else:
             msg4 = f"SKIPPING {table_name}. No files found."
             log.info(msg4)
 
-    if overall_pass:
-        log.info("All tables passed")
-        if all_must_pass:
-            msg5 = f"Copying data from {land_base_path} to {pass_base_path}"
-            log.info(msg5)
+    if all_table_responses:
+        save_completion_status(config, all_table_responses)
+
+
+def save_completion_status(config: dict, all_table_responses: List[dict]):
+    """
+    saves the status of the table linting to a file to be colleted later
+
+    Args:
+    config: A data linter config
+    all_table_responses: a list of dictionaries detailing whether it passed or failied
+    linting, the validator response, the file linted, and the table name
+    """
+
+    log_base_path_is_s3 = config["log-base-path"].startswith("s3://")
+
+    temp_status_basepath = os.path.join(get_temp_log_basepath(config), "status")
+    for table_response in all_table_responses:
+        if log_base_path_is_s3:
+            og_file_name = os.path.basename(table_response["original-path"]).split(".")[
+                0
+            ]
+
+            with tempfile.NamedTemporaryFile(
+                suffix=".json", prefix=og_file_name
+            ) as tmp_file:
 
-            for resp in all_table_responses:
-                msg6 = f"Copying {resp['s3-original-path']} to {resp['archived-path']}"
-                log.info(msg6)
+                with open(tmp_file.name, "w") as json_out:
+                    json.dump(table_response, json_out)
+
+                tmp_file_name = os.path.basename(tmp_file.name)
+                s3_temp_path = os.path.join(temp_status_basepath, tmp_file_name)
+
+                local_file_to_s3(tmp_file.name, s3_temp_path)
+
+        else:
+            if not os.path.exists(temp_status_basepath):
+                os.makedirs(temp_status_basepath, exist_ok=True)
+            tmp_file_resp = tempfile.mkstemp(suffix=".json", dir=temp_status_basepath)
+            tmp_filename = tmp_file_resp[1]
+            with open(tmp_filename, "w") as json_out:
+                json.dump(table_response, json_out)
+
+
+def collect_all_status(config: dict):
+    """
+    collects the status files saved and determines whether the linting was a succes or
+    not and copies/removes/compresses the files to and from the correct places
+
+    Args:
+    config: the config as given at the beggining with the paths of where to collect and
+    save data from as well as compression, remove-on-pass etc.
+    """
+
+    utc_ts = int(datetime.utcnow().timestamp())
+    land_base_path = config["land-base-path"]
+    all_must_pass = config.get("all-must-pass", False)
+    pass_base_path = config["pass-base-path"]
+    log_base_path = config["log-base-path"]
+    fail_base_path = config.get("fail-base-path")
+    remove_on_pass = config.get("remove-tables-on-pass")
+    compress = config.get("compress-data")
+    timestamp_partition_name = config.get("timestamp-partition-name")
+
+    land_base_path_is_s3 = land_base_path.startswith("s3://")
+    log_base_path_is_s3 = log_base_path.startswith("s3://")
+    temp_status_basepath = os.path.join(get_temp_log_basepath(config), "status")
+    if log_base_path_is_s3:
+        status_file_paths = get_filepaths_from_s3_folder(temp_status_basepath)
+
+        s3_client = boto3.client("s3")
+
+        all_table_response = []
+        for status_file_path in status_file_paths:
+            bucket, key = s3_path_to_bucket_key(status_file_path)
+            status_file_obj = s3_client.get_object(Bucket=bucket, Key=key)
+            all_table_response.append(json.loads(status_file_obj["Body"].read()))
+
+    else:
+        status_file_paths = get_filepaths_from_local_folder(temp_status_basepath)
+
+        all_table_response = []
+        for status_file_path in status_file_paths:
+            with open(status_file_path) as json_in:
+                all_table_response.append(json.load(json_in))
+
+    all_tables_passed = True
+
+    pass_count = sum([i["valid"] for i in all_table_response])
+
+    if pass_count != len(all_table_response):
+        all_tables_passed = False
+
+    there_was_a_fail = False
+    all_tables_to_fail = False
+    all_tables_to_respective = False
+
+    if all_tables_passed:
+        all_tables_to_respective = True
+    else:
+        if all_must_pass:
+            all_tables_to_fail = True
+        else:
+            all_tables_to_respective = True
+
+    for i, table_response in enumerate(all_table_response):
+        table_name = table_response.get("table-name")
+        matched_file = table_response.get("original-path")
+        file_basename = os.path.basename(matched_file)
+
+        if all_tables_to_fail:
+            there_was_a_fail = True
+            final_outpath = get_out_path(
+                fail_base_path,
+                table_name,
+                utc_ts,
+                file_basename,
+                compress=compress,
+                filenum=i,
+                timestamp_partition_name=timestamp_partition_name,
+            )
+            if compress:
+                log.info(f"Compressing file from {matched_file} to {final_outpath}")
+                compress_data(matched_file, final_outpath)
+            else:
+                log.info(f"Copying file from {matched_file} to {final_outpath}")
+                copy_data(matched_file, final_outpath)
+        elif all_tables_to_respective:
+            if table_response["valid"]:
+                final_outpath = get_out_path(
+                    pass_base_path,
+                    table_name,
+                    utc_ts,
+                    file_basename,
+                    compress=compress,
+                    filenum=i,
+                    timestamp_partition_name=timestamp_partition_name,
+                )
                 if compress:
-                    compress_data(resp["s3-original-path"], resp["archived-path"])
+                    log.info(f"Compressing file from {matched_file} to {final_outpath}")
+                    compress_data(matched_file, final_outpath)
                 else:
-                    copy_s3_object(resp["s3-original-path"], resp["archived-path"])
-
+                    log.info(f"Copying file from {matched_file} to {final_outpath}")
+                    copy_data(matched_file, final_outpath)
                 if remove_on_pass:
-                    log.info(f"Removing data in land: {resp['s3-original-path']}")
-                    delete_s3_object(resp["s3-original-path"])
-
-    elif all_must_pass:
-        if fail_base_path:
-            m0 = "Copying files"
-            log.info(m0)
-        for resp in all_table_responses:
-            if resp["archived-path"]:
+                    log.info(f"Removing data in land: {matched_file}")
+                    if land_base_path_is_s3:
+                        delete_s3_object(matched_file)
+                    else:
+                        os.remove(matched_file)
+
+            else:
+                there_was_a_fail = True
+                final_outpath = get_out_path(
+                    pass_base_path,
+                    table_name,
+                    utc_ts,
+                    file_basename,
+                    compress=compress,
+                    filenum=i,
+                    timestamp_partition_name=timestamp_partition_name,
+                )
                 if compress:
-                    log.info(
-                        f"Compressing file from {resp['s3-original-path']} to \
-                            {resp['archived-path']}"
-                    )
-                    compress_data(resp["s3-original-path"], resp["archived-path"])
+                    log.info(f"Compressing file from {matched_file} to {final_outpath}")
+                    compress_data(matched_file, final_outpath)
                 else:
-                    log.info(
-                        f"Copying file from {resp['s3-original-path']} to \
-                            {resp['archived-path']}"
-                    )
-                    copy_s3_object(resp["s3-original-path"], resp["archived-path"])
-            log.error("The following tables failed:")
-            if not resp["valid"]:
-                m1 = f"{resp['table-name']} failed"
-                m2 = f"... original path: {resp['s3-original-path']}"
-                m3 = f"... out path: {resp['archived-path']}"
-                log.error(m1)
-                log.error(m2)
-                log.error(m3)
-
-        m4 = f"Logs that show failed data: {log_base_path}"
-        m5 = (
-            "Tables that passed but not written due to other table failures"
-            f"are stored here: {land_base_path}"
-        )
-        log.info(m4)
-        log.info(m5)
+                    log.info(f"Copying file from {matched_file} to {final_outpath}")
+                    copy_data(matched_file, final_outpath)
+        table_response["archived-path"] = final_outpath
+
+        # write (table specific) log
+        log_outpath = get_table_log_path(log_base_path, table_name, utc_ts, filenum=i)
+        if log_base_path_is_s3:
+            write_json_to_s3(table_response, log_outpath)
+        else:
+            path_name = os.path.dirname(log_outpath)
+            os.makedirs(path_name, exist_ok=True)
+            with open(log_outpath, "w") as json_out:
+                json.dump(table_response, json_out)
+        log.info(f"log for {matched_file} uploaded to {log_outpath}")
+
+    if there_was_a_fail and all_must_pass:
+        log.info("The following tables have failed: ")
+        for failed_table in [i for i in all_table_response if not i["valid"]]:
+            log.info(f"{failed_table['table-name']} failed")
+            log.info(f"...original path: {failed_table['original-path']}")
+            log.info(f"...out path: {failed_table['archived-path']}")
         raise ValueError("Tables did not pass linter")
 
+    if not all_must_pass and there_was_a_fail:
+        msg6 = "Some tables failed but all_must_pass set to false."
+        msg6 += " Check logs for details"
+        log.info(msg6)
+
+    if log_base_path_is_s3:
+        delete_s3_folder_contents(temp_status_basepath)
     else:
-        m6 = "Some tables failed but all_must_pass set to false. Check logs for details"
-        log.info(m6)
+        shutil.rmtree(temp_status_basepath, ignore_errors=True)
 
 
-def run_validation(config: Union[str, dict] = "config.yaml"):
-    """
-    Runs end to end validation based on config.
+def para_run_init(max_bin_count: int, config: Union[str, dict] = "config.yaml"):
 
-    Args:
-        config (Union[str, dict], optional): Either a string specifying the path to a
-        config yaml. Or a dict of a config in memory. Defaults to "config.yaml".
+    log.info("Loading config for paralellisation")
+    log_path = None
+    try:
+        config = load_and_validate_config(config)
+        temp_log_path = get_temp_log_path_from_config(config)
+        if get_filepaths_from_s3_folder(temp_log_path):
+            log.info(
+                f"Found temp logs in {temp_log_path}."
+                "Deleting data in folder before run."
+            )
+            delete_s3_folder_contents(temp_log_path)
 
-    Raises:
-        Error: States where log is written if error is hit in validation and then raises
-        traceback.
-    """
+        log_path = get_main_log_path_from_config(config)
 
-    # set up logging
-    log, log_stringio = logging_setup()
+        config = match_files_in_land_to_config(config)
 
-    log.info("Loading config")
-    log_path = None
-    try:
-        if isinstance(config, str):
-            config = load_and_validate_config(config)
-        elif isinstance(config, dict):
-            config = validate_and_clean_config(config)
-        else:
-            raise TypeError("Input 'config' must be a str or dict.")
+        bin_pack_configs(config, max_bin_count)
 
-        log_path = os.path.join(config["log-base-path"], get_validator_name() + ".log")
         log.info("Running validation")
-        validate_data(config)
 
     except Exception as e:
         log_msg = f"Unexpected error. Uploading log to {log_path} before raising error."
@@ -369,4 +624,103 @@ def run_validation(config: Union[str, dict] = "config.yaml"):
 
         raise e.with_traceback(e.__traceback__)
     else:
+        upload_log(log, log_stringio, temp_log_path)
+
+
+def para_run_validation(config_num: int, config: Union[str, dict] = "config.yaml"):
+
+    try:
+        config = load_and_validate_config(config)
+
+        log.info(f"Worker {config_num} loading config for validaiton")
+        log_path = None
+
+        temp_log_path = get_temp_log_path_from_config(config)
+        log_path = get_main_log_path_from_config(config)
+        there_was_a_config = validate_from_chunked_configs(config, config_num)
+
+        if not there_was_a_config:
+            log.info(f"worker {config_num} had no work - moving on")
+
+    except Exception as e:
+        log_msg = f"Unexpected error. Uploading log to {log_path} before raising error."
+        error_msg = str(e)
+
+        log.error(log_msg)
+        log.error(error_msg)
+
         upload_log(log, log_stringio, log_path)
+
+        raise e.with_traceback(e.__traceback__)
+    else:
+        upload_log(log, log_stringio, temp_log_path)
+
+
+def para_collect_all_status(config: Union[str, dict] = "config.yaml"):
+
+    log_path = None
+    try:
+        config = load_and_validate_config(config)
+
+        temp_log_path = get_temp_log_path_from_config(config)
+        log_path = get_main_log_path_from_config(config)
+
+        log.info("collating table status")
+        collect_all_status(config)
+        log.info(f"collating all logs in {config['log-base-path']}")
+
+    except Exception as e:
+        log_msg = f"Unexpected error. Uploading log to {log_path} before raising error."
+        error_msg = str(e)
+
+        log.error(log_msg)
+        log.error(error_msg)
+
+        upload_log(log, log_stringio, log_path)
+
+        raise e.with_traceback(e.__traceback__)
+    else:
+        upload_log(log, log_stringio, temp_log_path)
+
+
+def para_collect_all_logs(config: Union[str, dict] = "config.yaml"):
+
+    config = load_and_validate_config(config)
+
+    log_base_path = config["log-base-path"]
+    log_path_fin = get_main_log_path_from_config(config)
+    log_base_path_is_s3 = log_base_path.startswith("s3://")
+
+    tmp_log_base_path = get_temp_log_basepath(config)
+    init_log_path = os.path.join(tmp_log_base_path, "init")
+    val_log_path = os.path.join(tmp_log_base_path, "val")
+    status_log_path = os.path.join(tmp_log_base_path, "status")
+
+    if log_base_path_is_s3:
+        init_log_paths = get_filepaths_from_s3_folder(init_log_path)
+        val_log_paths = get_filepaths_from_s3_folder(val_log_path)
+        status_log_paths = get_filepaths_from_s3_folder(status_log_path)
+    else:
+        init_log_paths = get_filepaths_from_local_folder(init_log_path)
+        val_log_paths = get_filepaths_from_local_folder(val_log_path)
+        status_log_paths = get_filepaths_from_local_folder(status_log_path)
+
+    log_string_list = []
+    for init_log_path in init_log_paths:
+        log_string_list.append(read_all_file_body(init_log_path))
+    for val_log_path in val_log_paths:
+        log_string_list.append(read_all_file_body(val_log_path))
+    for status_log_path in status_log_paths:
+        log_string_list.append(read_all_file_body(status_log_path))
+
+    log_io = io.StringIO()
+    for log_str in log_string_list:
+        log_io.write(log_str)
+    upload_log(log, log_io, log_path_fin)
+
+    log_path_del = os.path.join(log_base_path, "data_linter_temporary_fs")
+
+    if log_base_path_is_s3:
+        delete_s3_folder_contents(log_path_del)
+    else:
+        shutil.rmtree(log_path_del, ignore_errors=True)
diff --git a/data_linter/validators/__init__.py b/data_linter/validators/__init__.py
index 939e020..36a1c19 100644
--- a/data_linter/validators/__init__.py
+++ b/data_linter/validators/__init__.py
@@ -1,2 +1,3 @@
+from .pandas_validator import PandasValidator  # noqa
 from .frictionless_validator import FrictionlessValidator  # noqa
 from .great_expectations_validator import GreatExpectationsValidator  # noqa
diff --git a/data_linter/validators/base.py b/data_linter/validators/base.py
index 60f51d4..642feca 100644
--- a/data_linter/validators/base.py
+++ b/data_linter/validators/base.py
@@ -1,8 +1,108 @@
 import logging
 from copy import deepcopy
+from typing import List
 
+from mojap_metadata import Metadata
+from jsonschema.exceptions import ValidationError
 
-class BaseTableValidator(object):
+
+class ValidatorResult:
+    """
+    Little class to manage adding to validator dict
+    """
+
+    def __init__(self, result_dict=None, validator_valid_key_name=None):
+        if result_dict:
+            if not isinstance(result_dict, dict):
+                raise TypeError("result_dict must be a dict type")
+            self.result = result_dict
+        else:
+            self.result = {"valid": True}
+
+        if validator_valid_key_name:
+            if not isinstance(validator_valid_key_name, str):
+                raise TypeError("validator_valid_key_name must be a str type")
+            self.vvkn = validator_valid_key_name
+        else:
+            self.vvkn = "valid"
+
+    @property
+    def metadata(self):
+        return self._metadata
+
+    @metadata.setter
+    def metadata(self, meta_dict: dict):
+        try:
+            meta_obj = Metadata.from_dict(meta_dict)
+            meta_obj.set_col_type_category_from_types()
+            self._metadata = meta_obj.to_dict()
+
+            if "file_format" not in self.metadata:
+                raise ValidationError("metadata given must have a file_format property")
+        except ValidationError as e:
+            error_msg = (
+                "Pandas validator requires schemas that conform "
+                "to those found in the mojap_metadata package. "
+                f"Metadata given failed validation: {str(e)}"
+            )
+            raise ValidationError(error_msg)
+
+    def get_result(self, copy=True):
+        if copy:
+            return deepcopy(self.result)
+        else:
+            return self.result
+
+    def init_col(self, colname):
+        if colname not in self.result:
+            self.result[colname] = {"valid": True}
+
+    def get_names_of_column_failures(self, test_names: List[str] = []):
+        """
+
+        Return col names which have an overall fail. If test_names is given
+        only returns cols that failed those particular tests is given.
+        Args:
+            test_name (List[str], optional): [description]. List of tests to
+            check against Defaults to [].
+        """
+        failed_cols = []
+        for colname in self.result:
+            if colname == "valid" or colname.startswith("validator-table-test-"):
+                continue
+
+            if test_names:
+                overall_success = True
+                for k, v in self.result[colname].items():
+                    if k in test_names:
+                        overall_success = overall_success and v.get(self.vvkn, True)
+            else:
+                overall_success = self.result[colname].get("valid", True)
+
+            if not overall_success:
+                failed_cols.append(colname)
+
+        return failed_cols
+
+    def add_table_test(self, testname, test_result):
+        # Same setup - treats overall test as a colname
+        self.init_col(testname)
+        self.result[testname] = test_result
+        if self.vvkn in test_result:
+            self.result["valid"] = self.result["valid"] and test_result[self.vvkn]
+
+    def add_test_to_col(self, colname, testname, test_result):
+        self.init_col(colname)
+
+        self.result[colname][testname] = test_result
+        if self.vvkn in test_result:
+            self.result["valid"] = self.result["valid"] and test_result[self.vvkn]
+            self.result[colname]["valid"] = (
+                self.result[colname]["valid"] and test_result[self.vvkn]
+            )
+
+
+class BaseTableValidator:
     def __init__(self, filepath: str, table_params: dict, metadata: dict, **kwargs):
         """Base class for validators. Not a useable,
         but used to be inherited for other validators.
@@ -16,8 +116,11 @@ def __init__(self, filepath: str, table_params: dict, metadata: dict, **kwargs):
         self.filepath = filepath
         self.table_params = table_params
         self.metadata = metadata
-        self.valid = None
-        self.response = None
+
+        self.response = ValidatorResult(
+            result_dict=kwargs.get("result_dict"),
+            validator_valid_key_name=kwargs.get("validator_valid_key_name"),
+        )
 
     def write_validation_result_to_log(self, log: logging.Logger):
         """Writes a the validators response to log provided.
diff --git a/data_linter/validators/frictionless_validator.py b/data_linter/validators/frictionless_validator.py
index f7bd8dc..7eec5e7 100644
--- a/data_linter/validators/frictionless_validator.py
+++ b/data_linter/validators/frictionless_validator.py
@@ -23,6 +23,7 @@ class FrictionlessValidator(BaseTableValidator):
 
     def __init__(self, filepath: str, table_params: dict, metadata: dict):
         super().__init__(filepath, table_params, metadata)
+        self.response = None  # Not yet integrated with ValidatorResult class
         self.schema = convert_meta_to_goodtables_schema(metadata)
         if optional_import_errors:
             imp_err = (
@@ -47,7 +48,7 @@ def read_data_and_validate(self):
 
         # assert the correct dialect and checks
         header_case = not self.table_params.get("headers-ignore-case", False)
-        if self.metadata["data_format"] == "json":
+        if "json" in self.metadata["file_format"]:
             expected_headers = [
                 c["name"]
                 for c in self.metadata["columns"]
@@ -91,15 +92,15 @@ def read_data_and_validate(self):
         self.response = dict(resp.tables[0])
 
 
-def convert_meta_type_to_goodtable_type(meta_type: str) -> str:
+def convert_meta_type_to_goodtable_type(meta_col: dict) -> str:
     """
     Converts string name for etl_manager data type
     and converts it to a goodtables data type
 
     Parameters
     ----------
-    meta_type: str
-        Column type of the etl_manager metadata
+    meta_col: dict
+        Column dict from the metadata columns
 
     Returns
     -------
@@ -107,28 +108,33 @@ def convert_meta_type_to_goodtable_type(meta_type: str) -> str:
         Column type of the goodtables_type
         https://frictionlessdata.io/specs/table-schema/
     """
-    meta_type = meta_type.lower()
+    meta_type = meta_col["type"].lower()
+    meta_type_category = meta_col["type_category"].lower()
 
     lookup = {
-        "character": "string",
-        "int": "integer",
-        "long": "integer",
+        "string": "string",
+        "integer": "integer",
         "float": "number",
-        "double": "number",
-        "date": "date",
-        "datetime": "datetime",
         "boolean": "boolean",
     }
 
-    if meta_type in lookup:
-        gt_type = lookup[meta_type]
-    elif meta_type.startswith("array"):
+    if meta_type_category in lookup:
+        gt_type = lookup[meta_type_category]
+    elif meta_type_category.startswith("timestamp"):
+        if meta_type.startswith("date"):
+            gt_type = "date"
+        elif meta_type.startswith("timestamp"):
+            gt_type = "datetime"
+        else:
+            pass
+    elif meta_type_category == "list":
         gt_type = "array"
-    elif meta_type.startswith("struct"):
+    elif meta_type_category == "struct":
         gt_type = "object"
     else:
         raise TypeError(
-            f"Given meta_type: {meta_type} but this matches no goodtables equivalent"
+            f"Given meta with type:{meta_type} and type_category:"
+            f"{meta_type_category} but this matches no goodtables equivalent."
         )
 
     return gt_type
@@ -169,7 +175,7 @@ def convert_meta_to_goodtables_schema(meta: dict) -> dict:
     for col in meta["columns"]:
         gt_constraints = {}
 
-        gt_type = convert_meta_type_to_goodtable_type(col["type"])
+        gt_type = convert_meta_type_to_goodtable_type(col)
         gt_format = col.get("format", "default")
 
         if gt_type in ["date", "datetime"] and "format" not in col:
diff --git a/data_linter/validators/great_expectations_validator.py b/data_linter/validators/great_expectations_validator.py
index cb7952c..5fd0c92 100644
--- a/data_linter/validators/great_expectations_validator.py
+++ b/data_linter/validators/great_expectations_validator.py
@@ -1,7 +1,10 @@
 import logging
-from data_linter.validators.base import BaseTableValidator
+from data_linter.validators.base import (
+    BaseTableValidator,
+    ValidatorResult,
+)
+
 from copy import deepcopy
-from typing import List
 
 optional_import_errors = ""
 try:
@@ -22,91 +25,21 @@
     optional_import_errors += " " + str(e)
 
 numerical_conversion = {
-    "int": "Int64",
-    "long": "Int64",
-    "double": np.float,
+    "integer": "Int64",
     "float": np.float,
 }
 
 default_pd_in_type = "str"
 
 pd_conversion = deepcopy(numerical_conversion)
-pd_conversion["datetime"] = "str"
-pd_conversion["date"] = "str"
+pd_conversion["timestamp"] = "str"
 pd_conversion["boolean"] = "str"
-pd_conversion["character"] = "str"
+pd_conversion["string"] = "str"
 
 
 log = logging.getLogger("root")
 
 
-class ValidatorResult(object):
-    """
-    Little class to manage adding to validator dict
-    """
-
-    def __init__(self, result_dict={}):
-        if result_dict:
-            self.result = result_dict
-        else:
-            self.result = {"valid": True}
-
-    def get_result(self, copy=True):
-        if copy:
-            return deepcopy(self.result)
-        else:
-            return self.result
-
-    def init_col(self, colname):
-        if colname not in self.result:
-            self.result[colname] = {"valid": True}
-
-    def get_names_of_column_failures(self, test_names: List[str] = []):
-        """
-
-        Return col names which have an overall fail. If test_names is given
-        only returns cols that failed those particular tests is given.
-        Args:
-            test_name (List[str], optional): [description]. List of tests to
-            check against Defaults to [].
-        """
-        non_column_names = ["valid", "validator-table-test-"]
-        failed_cols = []
-        for colname in self.result:
-            if colname in non_column_names:
-                continue
-
-            if test_names:
-                overall_success = True
-                for k, v in self.result[colname].items():
-                    if k in test_names:
-                        overall_success = overall_success and v.get("success", True)
-            else:
-                overall_success = self.result[colname].get("valid", True)
-
-            if not overall_success:
-                failed_cols.append(colname)
-
-        return failed_cols
-
-    def add_table_test(self, testname, test_result):
-        # Same setup - treats overall test as a colname
-        self.init_col(testname)
-        self.result[testname] = test_result
-        if "success" in test_result:
-            self.result["valid"] = self.result["valid"] and test_result["success"]
-
-    def add_test_to_col(self, colname, testname, test_result):
-        self.init_col(colname)
-
-        self.result[colname][testname] = test_result
-        if "success" in test_result:
-            self.result["valid"] = self.result["valid"] and test_result["success"]
-            self.result[colname]["valid"] = (
-                self.result[colname]["valid"] and test_result["success"]
-            )
-
-
 class GreatExpectationsValidator(BaseTableValidator):
     """
     Great expectations data validator
@@ -120,7 +53,9 @@ def __init__(
         default_result_fmt="COMPLETE",
         ignore_missing_cols=False,
     ):
-        super().__init__(filepath, table_params, metadata)
+        super().__init__(
+            filepath, table_params, metadata, validator_valid_key_name="success"
+        )
 
         if optional_import_errors:
             imp_err = (
@@ -132,7 +67,6 @@ def __init__(
         self.default_result_fmt = default_result_fmt
         self.ignore_missing_cols = ignore_missing_cols
 
-        self.response = ValidatorResult()
         self.valid = self.response.result["valid"]
 
     def write_validation_errors_to_log(self):
@@ -159,7 +93,7 @@ def read_data_and_validate(self):
             nrows=self.table_params.get("row-limit"),
         )
 
-        if self.metadata["data_format"] != "parquet":
+        if self.metadata["file_format"] != "parquet":
             df = _convert_df_to_meta_for_testing(df, self.metadata, self.response)
 
         validate_df_with_ge(
@@ -182,10 +116,10 @@ def _convert_df_to_meta_for_testing(df, metadata, result: ValidatorResult):
 
     for c in cols:
         try:
-            if c["type"] in numerical_conversion:
+            if c["type_category"] in numerical_conversion:
                 df[c["name"]] = pd.to_numeric(df[c["name"]])
                 df[c["name"]] = df[c["name"]].astype(
-                    numerical_conversion[c["type"]]
+                    numerical_conversion[c["type_category"]]
                 )  # in case pandas converts to int rather than float
             else:
                 df[c["name"]] = df[c["name"]].astype("string")
@@ -193,7 +127,7 @@ def _convert_df_to_meta_for_testing(df, metadata, result: ValidatorResult):
                 c["name"], "test-type-conversion", {"success": True, "desc": None}
             )
         except Exception as e:
-            t = numerical_conversion.get(c["type"], "string")
+            t = numerical_conversion.get(c["type_category"], "string")
             e = (
                 f"Column {c['name']} could not be cast to pandas type {t}."
                 f"Error: {str(e)}"
@@ -218,7 +152,7 @@ def _parse_data_to_pandas(
 
     pandas_kwargs = table_params.get("pandas-kwargs", {})
 
-    if metadata["data_format"] == "csv":
+    if "csv" in metadata["file_format"]:
         names = None
         header = "infer" if table_params.get("expect-header", True) else None
         if header is None:
@@ -243,7 +177,7 @@ def _parse_data_to_pandas(
                 **pandas_kwargs,
             )
 
-    elif metadata["data_format"] == "json":
+    elif "json" in metadata["file_format"]:
         if filepath.startswith("s3://"):
             df = wr.s3.read_json(
                 [filepath],
@@ -261,16 +195,16 @@ def _parse_data_to_pandas(
                 **pandas_kwargs,
             )
 
-    elif metadata["data_format"] == "parquet":
+    elif "parquet" in metadata["file_format"]:
         if filepath.startswith("s3://"):
             df = wr.s3.read_parquet([filepath], nrows=nrows, **pandas_kwargs)
         else:
             df = pd.read_parquet(filepath, nrows=nrows, **pandas_kwargs)
 
     else:
-        data_fmt = metadata["data_format"]
         raise ValueError(
-            f"metadata data_format ({data_fmt}) not supported for GE validator."
+            f"metadata file_format ({metadata['file_format']})"
+            "not supported for GE validator."
         )
 
     if table_params.get("headers-ignore-case"):
@@ -346,7 +280,7 @@ def column_validation(dfe, metacol, result: ValidatorResult, result_format):
     if not metacol.get("nullable", True):
         result.add_test_to_col(n, "nullable", ge_nullable_test(dfe, n, result_format))
 
-    if metacol["type"] in ["date", "datetime"]:
+    if metacol["type"].startswith("date") or metacol["type"].startswith("timestamp"):
         result.add_test_to_col(
             n,
             "datetime-format",
@@ -386,7 +320,7 @@ def column_validation(dfe, metacol, result: ValidatorResult, result_format):
 
 def ge_test_datetime_format(dfe, colname, coltype, date_format, result_format=None):
     if date_format is None:
-        date_format = "%Y-%m-%d" if coltype == "date" else "%Y-%m-%d %H:%M:%S"
+        date_format = "%Y-%m-%d" if coltype.startswith("date") else "%Y-%m-%d %H:%M:%S"
     result = dfe.expect_column_values_to_match_strftime_format(
         colname,
         strftime_format=date_format,
diff --git a/data_linter/validators/pandas_validator.py b/data_linter/validators/pandas_validator.py
new file mode 100644
index 0000000..5de9dd4
--- /dev/null
+++ b/data_linter/validators/pandas_validator.py
@@ -0,0 +1,462 @@
+import logging
+import pandas as pd
+import inspect
+
+from functools import wraps
+
+from datetime import datetime
+
+from arrow_pd_parser.parse import (
+    pa_read_csv_to_pandas,
+    pa_read_json_to_pandas,
+)
+
+from typing import Union
+
+from arrow_pd_parser.pa_pd import arrow_to_pandas
+
+import pyarrow as pa
+from pyarrow import parquet as pq, fs, csv, json
+
+from data_linter.validators.base import (
+    BaseTableValidator,
+)
+
+from mojap_metadata import Metadata
+from mojap_metadata.converters.arrow_converter import ArrowConverter
+
+log = logging.getLogger("root")
+default_date_format = "%Y-%m-%d"
+default_datetime_format = "%Y-%m-%d %H:%M:%S"
+
+
+class PandasValidator(BaseTableValidator):
+    """
+    Validator using Pandas
+    """
+
+    def __init__(
+        self,
+        filepath: str,
+        table_params: dict,
+        metadata: dict,
+        ignore_missing_cols=False,
+    ):
+        super().__init__(filepath, table_params, metadata)
+
+        self.ignore_missing_cols = ignore_missing_cols
+
+    @property
+    def valid(self):
+        return self.response.result["valid"]
+
+    def write_validation_errors_to_log(self):
+        table_result = self.response.get_result()
+        if not table_result["valid"]:
+            failed_cols = self.response.get_names_of_column_failures()
+            err_msg = (
+                "Table failed validation. "
+                f"Col failures: {failed_cols}. "
+                "See response error log for more details."
+            )
+            log.error(err_msg, extra={"context": "VALIDATION"})
+            log.debug(str(table_result), extra={"context": "VALIDATION"})
+
+    def read_data_and_validate(self):
+        """Reads data from filepath and validates it.
+
+        Data is read using pd_arrow_parser.
+        """
+
+        df = _parse_data_to_pandas(self.filepath, self.table_params, self.metadata)
+        self.validate_df(df)
+
+    def get_response_dict(self):
+        return self.response.get_result()
+
+    def validate_df(self, df):
+
+        meta_cols = [col for col in self.metadata["columns"] if col["name"] in df]
+
+        for m in meta_cols:
+            self.validate_col(df[m["name"]], m)
+
+    def validate_col(self, col, meta_col):
+
+        self.min_max_test(col, meta_col)
+        self.min_max_length_test(col, meta_col)
+        self.pattern_test(col, meta_col)
+        self.enum_test(col, meta_col)
+        self.nullable_test(col, meta_col)
+        self.datetime_format_test(col, meta_col)
+        self.date_format_test(col, meta_col)
+
+    def min_max_test(self, col, meta_col):
+        res_dict = _min_max_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "min_max_test", res_dict)
+
+    def min_max_length_test(self, col, meta_col):
+        res_dict = _min_max_length_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "min_max_length_test", res_dict)
+
+    def pattern_test(self, col, meta_col):
+        res_dict = _pattern_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "pattern_test", res_dict)
+
+    def enum_test(self, col, meta_col):
+        res_dict = _enum_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "enum_test", res_dict)
+
+    def nullable_test(self, col, meta_col):
+        res_dict = _nullable_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "nullable_test", res_dict)
+
+    def datetime_format_test(self, col, meta_col):
+        res_dict = _datetime_format_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "datetime_format_test", res_dict)
+
+    def date_format_test(self, col, meta_col):
+        res_dict = _date_format_test(col, meta_col)
+        col_name = meta_col["name"]
+        if res_dict is not None:
+            self.response.add_test_to_col(col_name, "date_format_test", res_dict)
+
+
+def check_run_validation_for_meta(func):
+    """
+    Wrapper for each validation test. Will get inputs to function
+    and check if function should be called based on the inputs.
+    (Most likely this will be done based on the properties of the supplied
+    metadata).
+
+    Will return nothing if function should not be called.
+    """
+
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        sig = inspect.signature(func)
+        argmap = sig.bind_partial(*args, **kwargs).arguments
+        mc = argmap.get("meta_col")
+        if func.__name__ == "_min_max_test" and _check_meta_has_params(
+            ["minimum", "maximum"], mc
+        ):
+            return func(*args, **kwargs)
+        elif func.__name__ == "_min_max_length_test" and _check_meta_has_params(
+            ["minLength", "maxLength"], mc
+        ):
+            return func(*args, **kwargs)
+        elif func.__name__ == "_pattern_test" and _check_meta_has_params(
+            ["pattern"], mc
+        ):
+            return func(*args, **kwargs)
+        elif func.__name__ == "_enum_test" and _check_meta_has_params(["enum"], mc):
+            return func(*args, **kwargs)
+        elif func.__name__ == "_nullable_test" and not _check_meta_has_params(
+            [None, True], [mc.get("nullable")]
+        ):
+            return func(*args, **kwargs)
+        elif func.__name__ == "_date_format_test" and mc.get("type", "").startswith(
+            "date"
+        ):
+            return func(*args, **kwargs)
+        elif func.__name__ == "_datetime_format_test" and mc.get("type", "").startswith(
+            "timestamp"
+        ):
+            return func(*args, **kwargs)
+        else:
+            pass
+
+    return wrapper
+
+
+@check_run_validation_for_meta
+def _min_max_test(col: pd.Series, meta_col: dict) -> dict:
+
+    col_name = meta_col["name"]
+    mi = meta_col.get("minimum")
+    ma = meta_col.get("maximum")
+
+    test_inputs = {"column": col_name, "minimum_value": mi, "maximum_value": ma}
+    res_dict = _result_dict("min max numerical", test_inputs)
+
+    col_oob = _get_min_max_series_out_of_bounds_col(col, col_name, mi, ma)
+
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+@check_run_validation_for_meta
+def _min_max_length_test(col: pd.Series, meta_col: dict) -> dict:
+
+    col_name = meta_col["name"]
+    mil = meta_col.get("minLength")
+    mal = meta_col.get("maxLength")
+
+    test_inputs = {"column": col_name, "minimum_length": mil, "maximum_length": mal}
+    res_dict = _result_dict("min max length", test_inputs)
+
+    col_oob = _get_min_max_series_out_of_bounds_col(col.str.len(), col_name, mil, mal)
+
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+@check_run_validation_for_meta
+def _pattern_test(col: pd.Series, meta_col: dict) -> dict:
+
+    col_name = meta_col["name"]
+    pattern = meta_col.get("pattern")
+
+    test_inputs = {"column": col_name, "regex": pattern}
+
+    res_dict = _result_dict("regex", test_inputs)
+
+    col_oob = ~col.str.match(pattern)
+
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+@check_run_validation_for_meta
+def _enum_test(col: pd.Series, meta_col: dict) -> dict:
+
+    col_name = meta_col["name"]
+    enum = meta_col.get("enum")
+
+    test_inputs = {"column": col_name, "enum_value_set": enum}
+
+    res_dict = _result_dict("enum", test_inputs)
+
+    if meta_col.get("nullable", True):
+        col_oob = ~col.fillna(enum[0]).isin(enum)
+    else:
+        col_oob = ~col.isin(enum)
+
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+@check_run_validation_for_meta
+def _nullable_test(col: pd.Series, meta_col: dict) -> dict:
+
+    col_name = meta_col.get("name")
+
+    test_inputs = {
+        "column": col_name,
+    }
+
+    res_dict = _result_dict("nullable", test_inputs)
+
+    col_oob = col.isnull()
+
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+@check_run_validation_for_meta
+def _date_format_test(col: pd.Series, meta_col) -> dict:
+
+    col_name = meta_col["name"]
+    datetime_format = meta_col.get("datetime_format", default_date_format)
+    test_inputs = {"column": col_name, "datetime_format": datetime_format}
+
+    res_dict = _result_dict("datetime_format", test_inputs)
+
+    col_oob = ~col.apply(
+        lambda x: _valid_date_or_datetime_conversion(x, datetime_format, True)
+    )
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+@check_run_validation_for_meta
+def _datetime_format_test(col: pd.Series, meta_col):
+
+    col_name = meta_col["name"]
+    datetime_format = meta_col.get("datetime_format", default_datetime_format)
+    test_inputs = {"column": col_name, "datetime_format": datetime_format}
+
+    res_dict = _result_dict("datetime_format", test_inputs)
+
+    col_oob = ~col.apply(
+        lambda x: _valid_date_or_datetime_conversion(x, datetime_format)
+    )
+    return _fill_res_dict(col, col_oob, res_dict)
+
+
+def _valid_date_or_datetime_conversion(
+    date_or_datetime_str: str, dt_format: str, check_for_no_time_component=False
+):
+    if pd.isna(date_or_datetime_str) or not bool(date_or_datetime_str):
+        return True
+    else:
+        try:
+            dt = datetime.strptime(date_or_datetime_str, dt_format)
+            if check_for_no_time_component:
+                return _check_no_time_component_in_expected_date(dt)
+            else:
+                return True
+        except ValueError:
+            return False
+
+
+def _check_no_time_component_in_expected_date(dt: datetime):
+    result = dt.hour == 0 and dt.minute == 0 and dt.second == 0 and dt.microsecond == 0
+    return result
+
+
+def _result_dict(test_name: str, test_inputs: dict) -> dict:
+
+    d = {
+        "valid": None,
+        "test_inputs": test_inputs,
+    }
+
+    return d
+
+
+def _fill_res_dict(col, col_oob, res_dict) -> dict:
+
+    valid = not col_oob.any()
+    res_dict["valid"] = valid
+
+    if not valid:
+        col_oob = col_oob.fillna(False)
+        unexpected_index_list = col_oob.index[col_oob].tolist()
+        unexpected_list = col[unexpected_index_list].tolist()
+
+        res_dict["unexpected_index_list"] = unexpected_index_list
+        res_dict["unexpected_list"] = unexpected_list
+
+    return res_dict
+
+
+def _get_min_max_series_out_of_bounds_col(
+    col: pd.Series, colname: str, mi: Union[int, None], ma: Union[int, None]
+) -> pd.Series:
+
+    # Test if values out of bounds
+    if mi is not None and ma is None:
+        return col < mi
+    elif ma is not None and mi is None:
+        return col > ma
+    elif mi is not None and ma is not None:
+        return (col < mi) | (col > ma)
+    else:
+        raise ValueError(f"invalid min/max values for column: {colname}")
+
+
+def _check_meta_has_params(any_of: list, meta_col: dict):
+    return any([a in meta_col for a in any_of])
+
+
+def _parse_data_to_pandas(filepath: str, table_params: dict, metadata: dict):
+    """
+    Reads in the data from the given filepath and returns
+    a dataframe
+    """
+
+    meta_col_names = [
+        c["name"]
+        for c in metadata["columns"]
+        if c["name"] not in metadata.get("partitions", [])
+    ]
+
+    # For string based file types convert make arrow readers read them in as strings
+    # validators will still treat these as dates but will run validation against strings
+    # cols expecting values to match a timestamp format
+    if "json" in metadata["file_format"] or "csv" in metadata["file_format"]:
+        md_obj = Metadata.from_dict(metadata)
+        cols = md_obj.columns
+
+        cols_to_force_str_read_in = []
+        for c in cols:
+            if c["type"].startswith("time") or c["type"].startswith("date"):
+                c["type"] = "string"
+                c["type_category"] = "string"
+                cols_to_force_str_read_in.append(c["name"])
+
+        md_obj.columns = cols
+        ac = ArrowConverter()
+        arrow_schema = ac.generate_from_meta(md_obj)
+
+        ts_as_str_schema = pa.schema([])
+        for cname in cols_to_force_str_read_in:
+            ts_as_str_schema = ts_as_str_schema.append(arrow_schema.field(cname))
+
+    # Set the reader type
+    if filepath.startswith("s3://"):
+        reader_fs = fs.S3FileSystem(region="eu-west-1")
+        fp_for_file_reader = filepath.replace("s3://", "", 1)
+
+    else:
+        reader_fs = fs.LocalFileSystem()
+        fp_for_file_reader = filepath
+
+    with reader_fs.open_input_stream(fp_for_file_reader) as f:
+        if "csv" in metadata["file_format"]:
+
+            # Safer CSV load for newlines_in_values set to True
+            if table_params.get("expect-header", True):
+                po = csv.ParseOptions(newlines_in_values=True)
+            else:
+                po = csv.ParseOptions(
+                    newlines_in_values=True, column_names=meta_col_names
+                )
+
+            if ts_as_str_schema:
+                co = csv.ConvertOptions(column_types=ts_as_str_schema)
+            else:
+                co = None
+
+            df = pa_read_csv_to_pandas(
+                input_file=f,
+                schema=arrow_schema,
+                expect_full_schema=False,
+                parse_options=po,
+                convert_options=co,
+            )
+            # dates/datetimes == string
+
+        elif "json" in metadata["file_format"]:
+
+            po = json.ParseOptions(
+                newlines_in_values=True,
+                explicit_schema=ts_as_str_schema if ts_as_str_schema else None,
+            )
+
+            df = pa_read_json_to_pandas(
+                input_file=f,
+                schema=arrow_schema,
+                expect_full_schema=False,
+                parse_options=po,
+            )
+            # dates/datetimes == string
+
+        elif "parquet" in metadata["file_format"]:
+            df = arrow_to_pandas(pq.read_table(f))
+            # dates/datetimes == datetime / date
+
+        else:
+            raise ValueError(
+                f"Unknown file_format in metadata: {metadata['file_format']}."
+            )
+
+    if table_params.get("row-limit"):
+        df = df.sample(table_params.get("row-limit"))
+
+    if table_params.get("headers-ignore-case"):
+        df_cols = [c.lower() for c in df.columns]
+        df.columns = df_cols
+
+    if table_params.get("only-test-cols-in-metadata", False):
+        keep_cols = [c for c in df.columns if c in meta_col_names]
+        df = df[keep_cols]
+
+    return df
diff --git a/poetry.lock b/poetry.lock
index 370b6b5..3884a6f 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -27,8 +27,8 @@ python-versions = "*"
 
 [[package]]
 name = "appnope"
-version = "0.1.0"
-description = "Disable App Nap on OS X 10.9"
+version = "0.1.2"
+description = "Disable App Nap on macOS >= 10.9"
 category = "main"
 optional = false
 python-versions = "*"
@@ -50,6 +50,22 @@ dev = ["coverage[toml] (>=5.0.2)", "hypothesis", "pytest", "sphinx", "wheel", "p
 docs = ["sphinx"]
 tests = ["coverage[toml] (>=5.0.2)", "hypothesis", "pytest"]
 
+[[package]]
+name = "arrow-pd-parser"
+version = "0.1.0"
+description = "MoJ arrow-pd-parser"
+category = "main"
+optional = false
+python-versions = ">=3.7,<3.9"
+
+[package.dependencies]
+pandas = ">=1.1,<2.0"
+pyarrow = ">=2.0.0,<3.0.0"
+s3fs = ">=0.2.2,<0.5.0"
+
+[package.source]
+type = "url"
+url = "https://github.com/moj-analytical-services/mojap-arrow-pd-parser/archive/v0.1.0.zip"
 [[package]]
 name = "async-generator"
 version = "1.10"
@@ -82,7 +98,7 @@ tests_no_zope = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>
 
 [[package]]
 name = "aws-sam-translator"
-version = "1.31.0"
+version = "1.35.0"
 description = "AWS SAM Translator is a library that transform SAM templates into AWS CloudFormation templates"
 category = "dev"
 optional = false
@@ -94,7 +110,7 @@ jsonschema = ">=3.2,<4.0"
 six = ">=1.15,<2.0"
 
 [package.extras]
-dev = ["coverage (>=5.3,<6.0)", "flake8 (>=3.8.4,<3.9.0)", "tox (>=3.20.1,<3.21.0)", "pytest-cov (>=2.10.1,<2.11.0)", "pylint (>=1.7.2,<2.0)", "pyyaml (>=5.3.1,<5.4.0)", "mock (>=3.0.5,<4.0.0)", "parameterized (>=0.7.4,<0.8.0)", "requests (>=2.24.0,<2.25.0)", "docopt (>=0.6.2,<0.7.0)", "pytest (>=4.6.11,<4.7.0)", "pytest (>=6.1.1,<6.2.0)", "black (==20.8b1)"]
+dev = ["coverage (>=5.3,<6.0)", "flake8 (>=3.8.4,<3.9.0)", "tox (>=3.20.1,<3.21.0)", "pytest-cov (>=2.10.1,<2.11.0)", "pylint (>=1.7.2,<2.0)", "pyyaml (>=5.3.1,<5.4.0)", "mock (>=3.0.5,<4.0.0)", "parameterized (>=0.7.4,<0.8.0)", "click (>=7.1,<8.0)", "dateparser (>=0.7,<1.0)", "requests (>=2.24.0,<2.25.0)", "docopt (>=0.6.2,<0.7.0)", "pathlib2 (>=2.3.5)", "pytest (>=4.6.11,<4.7.0)", "pytest (>=6.1.1,<6.2.0)", "black (20.8b1)"]
 
 [[package]]
 name = "aws-xray-sdk"
@@ -112,7 +128,7 @@ wrapt = "*"
 
 [[package]]
 name = "awswrangler"
-version = "1.10.0"
+version = "2.3.0"
 description = "Pandas on AWS."
 category = "main"
 optional = true
@@ -122,12 +138,15 @@ python-versions = ">=3.6, <3.9"
 boto3 = ">=1.12.49,<2.0.0"
 botocore = ">=1.15.49,<2.0.0"
 numpy = ">=1.18.0,<1.20.0"
-pandas = ">=1.1.0,<1.2.0"
-psycopg2-binary = ">=2.8.0,<2.9.0"
+openpyxl = ">=3.0.0,<3.1.0"
+pandas = ">=1.1.0,<=1.2.0"
+pg8000 = ">=1.16.0,<1.17.0"
 pyarrow = ">=2.0.0,<2.1.0"
-pymysql = ">=0.9.0,<0.11.0"
-SQLAlchemy = ">=1.3.10,<1.4.0"
-sqlalchemy-redshift = ">=0.7.0,<0.9.0"
+pymysql = ">=0.9.0,<1.1.0"
+redshift-connector = ">=2.0.0,<2.1.0"
+
+[package.extras]
+sqlserver = ["pyodbc (>=4.0.30,<4.1.0)"]
 
 [[package]]
 name = "backcall"
@@ -137,6 +156,21 @@ category = "main"
 optional = false
 python-versions = "*"
 
+[[package]]
+name = "beautifulsoup4"
+version = "4.7.1"
+description = "Screen-scraping library"
+category = "main"
+optional = true
+python-versions = "*"
+
+[package.dependencies]
+soupsieve = ">=1.2"
+
+[package.extras]
+html5lib = ["html5lib"]
+lxml = ["lxml"]
+
 [[package]]
 name = "black"
 version = "20.8b1"
@@ -161,7 +195,7 @@ d = ["aiohttp (>=3.3.2)", "aiohttp-cors"]
 
 [[package]]
 name = "bleach"
-version = "3.2.1"
+version = "3.3.0"
 description = "An easy safelist-based HTML-sanitizing tool."
 category = "main"
 optional = false
@@ -182,33 +216,33 @@ python-versions = "*"
 
 [[package]]
 name = "boto3"
-version = "1.16.24"
+version = "1.17.27"
 description = "The AWS SDK for Python"
 category = "main"
 optional = false
-python-versions = "*"
+python-versions = ">= 2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
 
 [package.dependencies]
-botocore = ">=1.19.24,<1.20.0"
+botocore = ">=1.20.27,<1.21.0"
 jmespath = ">=0.7.1,<1.0.0"
 s3transfer = ">=0.3.0,<0.4.0"
 
 [[package]]
 name = "botocore"
-version = "1.19.24"
+version = "1.20.27"
 description = "Low-level, data-driven core of boto 3."
 category = "main"
 optional = false
-python-versions = "*"
+python-versions = ">= 2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
 
 [package.dependencies]
 jmespath = ">=0.7.1,<1.0.0"
 python-dateutil = ">=2.1,<3.0.0"
-urllib3 = {version = ">=1.25.4,<1.27", markers = "python_version != \"3.4\""}
+urllib3 = ">=1.25.4,<1.27"
 
 [[package]]
 name = "certifi"
-version = "2020.11.8"
+version = "2020.12.5"
 description = "Python package for providing Mozilla's CA Bundle."
 category = "main"
 optional = false
@@ -216,7 +250,7 @@ python-versions = "*"
 
 [[package]]
 name = "cffi"
-version = "1.14.4"
+version = "1.14.5"
 description = "Foreign Function Interface for Python calling C code."
 category = "main"
 optional = false
@@ -227,20 +261,20 @@ pycparser = "*"
 
 [[package]]
 name = "cfn-lint"
-version = "0.41.0"
+version = "0.47.2"
 description = "Checks CloudFormation templates for practices and behaviour that could potentially be improved"
 category = "dev"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [package.dependencies]
-aws-sam-translator = ">=1.25.0"
+aws-sam-translator = ">=1.34.0"
 jsonpatch = {version = "*", markers = "python_version != \"3.4\""}
 jsonschema = ">=3.0,<4.0"
 junit-xml = ">=1.9,<2.0"
 networkx = {version = ">=2.4,<3.0", markers = "python_version >= \"3.5\""}
 pyyaml = {version = "*", markers = "python_version != \"3.4\""}
-six = ">=1.11,<2.0"
+six = ">=1.11"
 
 [[package]]
 name = "chardet"
@@ -255,7 +289,7 @@ name = "click"
 version = "7.1.2"
 description = "Composable command line interface toolkit"
 category = "main"
-optional = false
+optional = true
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
 
 [[package]]
@@ -268,33 +302,33 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
 
 [[package]]
 name = "cryptography"
-version = "3.2.1"
+version = "3.4.6"
 description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
 category = "dev"
 optional = false
-python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*"
+python-versions = ">=3.6"
 
 [package.dependencies]
-cffi = ">=1.8,<1.11.3 || >1.11.3"
-six = ">=1.4.1"
+cffi = ">=1.12"
 
 [package.extras]
-docs = ["sphinx (>=1.6.5,!=1.8.0,!=3.1.0,!=3.1.1)", "sphinx-rtd-theme"]
+docs = ["sphinx (>=1.6.5,<1.8.0 || >1.8.0,<3.1.0 || >3.1.0,<3.1.1 || >3.1.1)", "sphinx-rtd-theme"]
 docstest = ["doc8", "pyenchant (>=1.6.11)", "twine (>=1.12.0)", "sphinxcontrib-spelling (>=4.0.1)"]
 pep8test = ["black", "flake8", "flake8-import-order", "pep8-naming"]
+sdist = ["setuptools-rust (>=0.11.4)"]
 ssh = ["bcrypt (>=3.1.5)"]
-test = ["pytest (>=3.6.0,!=3.9.0,!=3.9.1,!=3.9.2)", "pretend", "iso8601", "pytz", "hypothesis (>=1.11.4,!=3.79.2)"]
+test = ["pytest (>=6.0)", "pytest-cov", "pytest-subtests", "pytest-xdist", "pretend", "iso8601", "pytz", "hypothesis (>=1.11.4,<3.79.2 || >3.79.2)"]
 
 [[package]]
 name = "dataengineeringutils3"
-version = "1.1.0"
+version = "1.1.3"
 description = "Data engineering utils Python 3 version"
 category = "main"
 optional = false
-python-versions = ">=3.6,<4.0"
+python-versions = ">=3.6,<4"
 
 [package.dependencies]
-boto3 = ">=1.10,<2.0"
+boto3 = ">=1.10,<2.0.0"
 
 [[package]]
 name = "decorator"
@@ -306,7 +340,7 @@ python-versions = ">=2.6, !=3.0.*, !=3.1.*"
 
 [[package]]
 name = "defusedxml"
-version = "0.6.0"
+version = "0.7.1"
 description = "XML bomb protection for Python stdlib modules"
 category = "main"
 optional = false
@@ -314,14 +348,14 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
 
 [[package]]
 name = "docker"
-version = "4.4.0"
+version = "4.2.2"
 description = "A Python library for the Docker Engine API."
 category = "dev"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
 
 [package.dependencies]
-pywin32 = {version = "227", markers = "sys_platform == \"win32\""}
+pypiwin32 = {version = "223", markers = "sys_platform == \"win32\" and python_version >= \"3.6\""}
 requests = ">=2.14.2,<2.18.0 || >2.18.0"
 six = ">=1.4.0"
 websocket-client = ">=0.32.0"
@@ -354,29 +388,29 @@ name = "et-xmlfile"
 version = "1.0.1"
 description = "An implementation of lxml.xmlfile for the standard library"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
 name = "flake8"
-version = "3.8.4"
+version = "3.9.0"
 description = "the modular source code checker: pep8 pyflakes and co"
 category = "dev"
 optional = false
-python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,>=2.7"
+python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
 
 [package.dependencies]
 importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}
 mccabe = ">=0.6.0,<0.7.0"
-pycodestyle = ">=2.6.0a1,<2.7.0"
-pyflakes = ">=2.2.0,<2.3.0"
+pycodestyle = ">=2.7.0,<2.8.0"
+pyflakes = ">=2.3.0,<2.4.0"
 
 [[package]]
 name = "frictionless"
 version = "3.24.0"
 description = "Frictionless Framework is a framework to describe, extract, validate, and transform tabular data"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [package.dependencies]
@@ -416,6 +450,33 @@ spss = ["savReaderWriter (>=3.0)"]
 sql = ["sqlalchemy (>=1.3)"]
 tsv = ["linear-tsv (>=1.0)"]
 
+[[package]]
+name = "fsspec"
+version = "0.8.7"
+description = "File-system specification"
+category = "main"
+optional = false
+python-versions = ">3.6"
+
+[package.dependencies]
+importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}
+
+[package.extras]
+abfs = ["adlfs"]
+adl = ["adlfs"]
+dask = ["dask", "distributed"]
+dropbox = ["dropboxdrivefs", "requests", "dropbox"]
+gcs = ["gcsfs"]
+git = ["pygit2"]
+github = ["requests"]
+gs = ["gcsfs"]
+hdfs = ["pyarrow"]
+http = ["requests", "aiohttp"]
+s3 = ["s3fs"]
+sftp = ["paramiko"]
+smb = ["smbprotocol"]
+ssh = ["paramiko"]
+
 [[package]]
 name = "future"
 version = "0.18.2"
@@ -455,7 +516,7 @@ tzlocal = ">=1.2"
 
 [package.extras]
 airflow = ["apache-airflow[s3] (>=1.9.0)", "boto3 (>=1.7.3)"]
-gcp = ["google-cloud (>=0.34.0)", "google-cloud-storage (>=1.28.0)", "google-cloud-secret-manager (>=1.0.0)", "pybigquery (==0.4.15)"]
+gcp = ["google-cloud (>=0.34.0)", "google-cloud-storage (>=1.28.0)", "google-cloud-secret-manager (>=1.0.0)", "pybigquery (0.4.15)"]
 redshift = ["psycopg2 (>=2.8)"]
 s3 = ["boto3 (>=1.14)"]
 snowflake = ["snowflake-sqlalchemy (>=1.2)"]
@@ -484,30 +545,31 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "ijson"
-version = "3.1.3"
+version = "3.1.4"
 description = "Iterative JSON parser with standard Python iterator interfaces"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
 name = "importlib-metadata"
-version = "1.7.0"
+version = "3.7.3"
 description = "Read metadata from Python packages"
 category = "main"
 optional = false
-python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
+python-versions = ">=3.6"
 
 [package.dependencies]
+typing-extensions = {version = ">=3.6.4", markers = "python_version < \"3.8\""}
 zipp = ">=0.5"
 
 [package.extras]
-docs = ["sphinx", "rst.linker"]
-testing = ["packaging", "pep517", "importlib-resources (>=1.3)"]
+docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)"]
+testing = ["pytest (>=3.5,<3.7.3 || >3.7.3)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-cov", "pytest-enabler", "packaging", "pep517", "pyfakefs", "flufl.flake8", "pytest-black (>=0.3.7)", "pytest-mypy", "importlib-resources (>=1.3)"]
 
 [[package]]
 name = "ipykernel"
-version = "5.3.4"
+version = "5.5.0"
 description = "IPython Kernel for Jupyter"
 category = "main"
 optional = false
@@ -521,11 +583,11 @@ tornado = ">=4.2"
 traitlets = ">=4.1.0"
 
 [package.extras]
-test = ["pytest (!=5.3.4)", "pytest-cov", "flaky", "nose"]
+test = ["pytest (!=5.3.4)", "pytest-cov", "flaky", "nose", "jedi (<=0.17.2)"]
 
 [[package]]
 name = "ipython"
-version = "7.19.0"
+version = "7.21.0"
 description = "IPython: Productive Interactive Computing"
 category = "main"
 optional = false
@@ -536,7 +598,7 @@ appnope = {version = "*", markers = "sys_platform == \"darwin\""}
 backcall = "*"
 colorama = {version = "*", markers = "sys_platform == \"win32\""}
 decorator = "*"
-jedi = ">=0.10"
+jedi = ">=0.16"
 pexpect = {version = ">4.3", markers = "sys_platform != \"win32\""}
 pickleshare = "*"
 prompt-toolkit = ">=2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.1.0"
@@ -564,7 +626,7 @@ python-versions = "*"
 
 [[package]]
 name = "ipywidgets"
-version = "7.5.1"
+version = "7.6.3"
 description = "IPython HTML widgets for Jupyter"
 category = "main"
 optional = true
@@ -573,6 +635,7 @@ python-versions = "*"
 [package.dependencies]
 ipykernel = ">=4.5.1"
 ipython = {version = ">=4.0.0", markers = "python_version >= \"3.3\""}
+jupyterlab-widgets = {version = ">=1.0.0", markers = "python_version >= \"3.6\""}
 nbformat = ">=4.2.0"
 traitlets = ">=4.3.1"
 widgetsnbextension = ">=3.5.0,<3.6.0"
@@ -585,38 +648,30 @@ name = "isodate"
 version = "0.6.0"
 description = "An ISO 8601 date/time/duration parser and formatter"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [package.dependencies]
 six = "*"
 
-[[package]]
-name = "jdcal"
-version = "1.4.1"
-description = "Julian dates from proleptic Gregorian and Julian calendars."
-category = "main"
-optional = false
-python-versions = "*"
-
 [[package]]
 name = "jedi"
-version = "0.17.2"
+version = "0.18.0"
 description = "An autocompletion tool for Python that can be used for text editors."
 category = "main"
 optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
+python-versions = ">=3.6"
 
 [package.dependencies]
-parso = ">=0.7.0,<0.8.0"
+parso = ">=0.8.0,<0.9.0"
 
 [package.extras]
-qa = ["flake8 (==3.7.9)"]
-testing = ["Django (<3.1)", "colorama", "docopt", "pytest (>=3.9.0,<5.0.0)"]
+qa = ["flake8 (3.8.3)", "mypy (0.782)"]
+testing = ["Django (<3.1)", "colorama", "docopt", "pytest (<6.0.0)"]
 
 [[package]]
 name = "jinja2"
-version = "2.11.2"
+version = "2.11.3"
 description = "A very fast and expressive template engine."
 category = "main"
 optional = false
@@ -657,18 +712,15 @@ python-versions = "*"
 
 [[package]]
 name = "jsonlines"
-version = "1.2.0"
+version = "2.0.0"
 description = "Library with helpers for the jsonlines file format"
 category = "main"
-optional = false
-python-versions = "*"
-
-[package.dependencies]
-six = "*"
+optional = true
+python-versions = ">=3.6"
 
 [[package]]
 name = "jsonpatch"
-version = "1.27"
+version = "1.32"
 description = "Apply JSON-Patches (RFC 6902)"
 category = "main"
 optional = false
@@ -679,23 +731,23 @@ jsonpointer = ">=1.9"
 
 [[package]]
 name = "jsonpickle"
-version = "1.4.1"
+version = "2.0.0"
 description = "Python library for serializing any arbitrary object graph into JSON"
 category = "dev"
 optional = false
 python-versions = ">=2.7"
 
 [package.dependencies]
-importlib-metadata = "*"
+importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}
 
 [package.extras]
 docs = ["sphinx", "jaraco.packaging (>=3.2)", "rst.linker (>=1.9)"]
-testing = ["coverage (<5)", "pytest (>=3.5,!=3.7.3)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-black-multipy", "pytest-cov", "ecdsa", "feedparser", "numpy", "pandas", "pymongo", "sqlalchemy", "enum34", "jsonlib"]
+testing = ["coverage (<5)", "pytest (>=3.5,<3.7.3 || >3.7.3)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-black-multipy", "pytest-cov", "ecdsa", "feedparser", "numpy", "pandas", "pymongo", "sklearn", "sqlalchemy", "enum34", "jsonlib"]
 "testing.libs" = ["demjson", "simplejson", "ujson", "yajl"]
 
 [[package]]
 name = "jsonpointer"
-version = "2.0"
+version = "2.1"
 description = "Identify specific nodes in a JSON document (RFC 6901)"
 category = "main"
 optional = false
@@ -732,7 +784,7 @@ six = "*"
 
 [[package]]
 name = "jupyter-client"
-version = "6.1.7"
+version = "6.1.12"
 description = "Jupyter protocol implementation and client libraries"
 category = "main"
 optional = false
@@ -746,11 +798,12 @@ tornado = ">=4.1"
 traitlets = "*"
 
 [package.extras]
-test = ["ipykernel", "ipython", "mock", "pytest", "pytest-asyncio", "async-generator", "pytest-timeout"]
+doc = ["sphinx (>=1.3.6)", "sphinx-rtd-theme", "sphinxcontrib-github-alt"]
+test = ["async-generator", "ipykernel", "ipython", "mock", "pytest-asyncio", "pytest-timeout", "pytest", "jedi (<0.18)"]
 
 [[package]]
 name = "jupyter-core"
-version = "4.7.0"
+version = "4.7.1"
 description = "Jupyter core package. A base package on which Jupyter projects rely."
 category = "main"
 optional = false
@@ -807,6 +860,28 @@ requests = "*"
 [package.extras]
 test = ["pytest", "requests"]
 
+[[package]]
+name = "jupyterlab-widgets"
+version = "1.0.0"
+description = "A JupyterLab extension."
+category = "main"
+optional = true
+python-versions = ">=3.6"
+
+[[package]]
+name = "lxml"
+version = "4.6.2"
+description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API."
+category = "main"
+optional = true
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, != 3.4.*"
+
+[package.extras]
+cssselect = ["cssselect (>=0.7)"]
+html5 = ["html5lib"]
+htmlsoup = ["beautifulsoup4"]
+source = ["Cython (>=0.29.7)"]
+
 [[package]]
 name = "markupsafe"
 version = "1.1.1"
@@ -833,7 +908,7 @@ python-versions = "*"
 
 [[package]]
 name = "mock"
-version = "4.0.2"
+version = "4.0.3"
 description = "Rolling backport of unittest.mock for all Pythons"
 category = "dev"
 optional = false
@@ -842,11 +917,32 @@ python-versions = ">=3.6"
 [package.extras]
 build = ["twine", "wheel", "blurb"]
 docs = ["sphinx"]
-test = ["pytest", "pytest-cov"]
+test = ["pytest (<5.4)", "pytest-cov"]
+
+[[package]]
+name = "mojap-metadata"
+version = "1.1.0"
+description = "A python package to manage metadata"
+category = "main"
+optional = false
+python-versions = "^3.7"
+
+[package.dependencies]
+jsonschema = ">=3.0.0"
+parameterized = ">=0.7.0,<0.8.0"
+pyarrow = {version = ">=2.0,<3.0", optional = true, markers = "extra == \"arrow\""}
+PyYAML = ">=5,<6"
+
+[package.extras]
+etl-manager = ["etl-manager (>=7.4,<8.0)"]
+arrow = ["pyarrow (>=2.0,<3.0)"]
 
+[package.source]
+type = "url"
+url = "https://github.com/moj-analytical-services/mojap-metadata/archive/v1.1.0.zip"
 [[package]]
 name = "more-itertools"
-version = "8.6.0"
+version = "8.7.0"
 description = "More routines for operating on iterables, beyond itertools"
 category = "dev"
 optional = false
@@ -888,7 +984,7 @@ zipp = "*"
 
 [package.extras]
 acm = ["cryptography (>=2.3.0)"]
-all = ["cryptography (>=2.3.0)", "PyYAML (>=5.1)", "python-jose[cryptography] (>=3.1.0,<4.0.0)", "ecdsa (<0.15)", "docker (>=2.5.1)", "jsondiff (>=1.1.2)", "aws-xray-sdk (>=0.93,!=0.96)", "idna (>=2.5,<3)", "cfn-lint (>=0.4.0)", "sshpubkeys (>=3.1.0,<4.0)", "sshpubkeys (>=3.1.0)"]
+all = ["cryptography (>=2.3.0)", "PyYAML (>=5.1)", "python-jose[cryptography] (>=3.1.0,<4.0.0)", "ecdsa (<0.15)", "docker (>=2.5.1)", "jsondiff (>=1.1.2)", "aws-xray-sdk (>=0.93,<0.96 || >0.96)", "idna (>=2.5,<3)", "cfn-lint (>=0.4.0)", "sshpubkeys (>=3.1.0,<4.0)", "sshpubkeys (>=3.1.0)"]
 awslambda = ["docker (>=2.5.1)"]
 batch = ["docker (>=2.5.1)"]
 cloudformation = ["PyYAML (>=5.1)", "cfn-lint (>=0.4.0)"]
@@ -897,8 +993,8 @@ ec2 = ["cryptography (>=2.3.0)", "sshpubkeys (>=3.1.0,<4.0)", "sshpubkeys (>=3.1
 iam = ["cryptography (>=2.3.0)"]
 iotdata = ["jsondiff (>=1.1.2)"]
 s3 = ["cryptography (>=2.3.0)"]
-server = ["cryptography (>=2.3.0)", "PyYAML (>=5.1)", "python-jose[cryptography] (>=3.1.0,<4.0.0)", "ecdsa (<0.15)", "docker (>=2.5.1)", "jsondiff (>=1.1.2)", "aws-xray-sdk (>=0.93,!=0.96)", "idna (>=2.5,<3)", "cfn-lint (>=0.4.0)", "flask", "sshpubkeys (>=3.1.0,<4.0)", "sshpubkeys (>=3.1.0)"]
-xray = ["aws-xray-sdk (>=0.93,!=0.96)"]
+server = ["cryptography (>=2.3.0)", "PyYAML (>=5.1)", "python-jose[cryptography] (>=3.1.0,<4.0.0)", "ecdsa (<0.15)", "docker (>=2.5.1)", "jsondiff (>=1.1.2)", "aws-xray-sdk (>=0.93,<0.96 || >0.96)", "idna (>=2.5,<3)", "cfn-lint (>=0.4.0)", "flask", "sshpubkeys (>=3.1.0,<4.0)", "sshpubkeys (>=3.1.0)"]
+xray = ["aws-xray-sdk (>=0.93,<0.96 || >0.96)"]
 
 [[package]]
 name = "mypy-extensions"
@@ -910,11 +1006,11 @@ python-versions = "*"
 
 [[package]]
 name = "nbclient"
-version = "0.5.1"
+version = "0.5.3"
 description = "A client library for executing notebooks. Formerly nbconvert's ExecutePreprocessor."
 category = "main"
 optional = false
-python-versions = ">=3.6"
+python-versions = ">=3.6.1"
 
 [package.dependencies]
 async-generator = "*"
@@ -952,15 +1048,15 @@ testpath = "*"
 traitlets = ">=4.2"
 
 [package.extras]
-all = ["pytest", "pytest-cov", "pytest-dependency", "ipykernel", "ipywidgets (>=7)", "pyppeteer (==0.2.2)", "tornado (>=4.0)", "sphinx (>=1.5.1)", "sphinx-rtd-theme", "nbsphinx (>=0.2.12)", "ipython"]
+all = ["pytest", "pytest-cov", "pytest-dependency", "ipykernel", "ipywidgets (>=7)", "pyppeteer (0.2.2)", "tornado (>=4.0)", "sphinx (>=1.5.1)", "sphinx-rtd-theme", "nbsphinx (>=0.2.12)", "ipython"]
 docs = ["sphinx (>=1.5.1)", "sphinx-rtd-theme", "nbsphinx (>=0.2.12)", "ipython"]
 serve = ["tornado (>=4.0)"]
-test = ["pytest", "pytest-cov", "pytest-dependency", "ipykernel", "ipywidgets (>=7)", "pyppeteer (==0.2.2)"]
-webpdf = ["pyppeteer (==0.2.2)"]
+test = ["pytest", "pytest-cov", "pytest-dependency", "ipykernel", "ipywidgets (>=7)", "pyppeteer (0.2.2)"]
+webpdf = ["pyppeteer (0.2.2)"]
 
 [[package]]
 name = "nbformat"
-version = "5.0.8"
+version = "5.1.2"
 description = "The Jupyter Notebook format"
 category = "main"
 optional = false
@@ -974,11 +1070,11 @@ traitlets = ">=4.1"
 
 [package.extras]
 fast = ["fastjsonschema"]
-test = ["fastjsonschema", "testpath", "pytest", "pytest-cov"]
+test = ["check-manifest", "fastjsonschema", "testpath", "pytest", "pytest-cov"]
 
 [[package]]
 name = "nest-asyncio"
-version = "1.4.3"
+version = "1.5.1"
 description = "Patch asyncio to allow nested event loops"
 category = "main"
 optional = false
@@ -1010,7 +1106,7 @@ scipy = ["scipy"]
 
 [[package]]
 name = "notebook"
-version = "6.1.5"
+version = "6.2.0"
 description = "A web-based notebook environment for interactive computing"
 category = "main"
 optional = false
@@ -1027,38 +1123,38 @@ nbconvert = "*"
 nbformat = "*"
 prometheus-client = "*"
 pyzmq = ">=17"
-Send2Trash = "*"
+Send2Trash = ">=1.5.0"
 terminado = ">=0.8.3"
-tornado = ">=5.0"
+tornado = ">=6.1"
 traitlets = ">=4.2.1"
 
 [package.extras]
-docs = ["sphinx", "nbsphinx", "sphinxcontrib-github-alt"]
-test = ["nose", "coverage", "requests", "nose-warnings-filters", "nbval", "nose-exclude", "selenium", "pytest", "pytest-cov", "requests-unixsocket"]
+docs = ["sphinx", "nbsphinx", "sphinxcontrib-github-alt", "sphinx-rtd-theme"]
+json-logging = ["json-logging"]
+test = ["pytest", "coverage", "requests", "nbval", "selenium", "pytest-cov", "requests-unixsocket"]
 
 [[package]]
 name = "numpy"
-version = "1.19.4"
+version = "1.19.5"
 description = "NumPy is the fundamental package for array computing with Python."
 category = "main"
-optional = true
+optional = false
 python-versions = ">=3.6"
 
 [[package]]
 name = "openpyxl"
-version = "3.0.5"
+version = "3.0.7"
 description = "A Python library to read/write Excel 2010 xlsx/xlsm files"
 category = "main"
-optional = false
+optional = true
 python-versions = ">=3.6,"
 
 [package.dependencies]
 et-xmlfile = "*"
-jdcal = "*"
 
 [[package]]
 name = "packaging"
-version = "20.4"
+version = "20.9"
 description = "Core utilities for Python packages"
 category = "main"
 optional = false
@@ -1066,23 +1162,22 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [package.dependencies]
 pyparsing = ">=2.0.2"
-six = "*"
 
 [[package]]
 name = "pandas"
-version = "1.1.3"
+version = "1.2.0"
 description = "Powerful data structures for data analysis, time series, and statistics"
 category = "main"
-optional = true
-python-versions = ">=3.6.1"
+optional = false
+python-versions = ">=3.7.1"
 
 [package.dependencies]
-numpy = ">=1.15.4"
+numpy = ">=1.16.5"
 python-dateutil = ">=2.7.3"
-pytz = ">=2017.2"
+pytz = ">=2017.3"
 
 [package.extras]
-test = ["pytest (>=4.0.2)", "pytest-xdist", "hypothesis (>=3.58)"]
+test = ["pytest (>=5.0.1)", "pytest-xdist", "hypothesis (>=3.58)"]
 
 [[package]]
 name = "pandocfilters"
@@ -1094,7 +1189,7 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "parameterized"
-version = "0.7.4"
+version = "0.7.5"
 description = "Parameterized testing with any Python test framework"
 category = "main"
 optional = false
@@ -1105,14 +1200,15 @@ dev = ["jinja2"]
 
 [[package]]
 name = "parso"
-version = "0.7.1"
+version = "0.8.1"
 description = "A Python Parser"
 category = "main"
 optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+python-versions = ">=3.6"
 
 [package.extras]
-testing = ["docopt", "pytest (>=3.0.7)"]
+qa = ["flake8 (3.8.3)", "mypy (0.782)"]
+testing = ["docopt", "pytest (<6.0.0)"]
 
 [[package]]
 name = "pathspec"
@@ -1133,6 +1229,17 @@ python-versions = "*"
 [package.dependencies]
 ptyprocess = ">=0.5"
 
+[[package]]
+name = "pg8000"
+version = "1.16.6"
+description = "PostgreSQL interface library"
+category = "main"
+optional = true
+python-versions = ">=3.5"
+
+[package.dependencies]
+scramp = "1.2.0"
+
 [[package]]
 name = "pickleshare"
 version = "0.7.5"
@@ -1168,7 +1275,7 @@ twisted = ["twisted"]
 
 [[package]]
 name = "prompt-toolkit"
-version = "3.0.8"
+version = "3.0.17"
 description = "Library for building powerful interactive command lines in Python"
 category = "main"
 optional = false
@@ -1177,17 +1284,9 @@ python-versions = ">=3.6.1"
 [package.dependencies]
 wcwidth = "*"
 
-[[package]]
-name = "psycopg2-binary"
-version = "2.8.6"
-description = "psycopg2 - Python-PostgreSQL Database Adapter"
-category = "main"
-optional = true
-python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*"
-
 [[package]]
 name = "ptyprocess"
-version = "0.6.0"
+version = "0.7.0"
 description = "Run a subprocess in a pseudo terminal"
 category = "main"
 optional = false
@@ -1195,7 +1294,7 @@ python-versions = "*"
 
 [[package]]
 name = "py"
-version = "1.9.0"
+version = "1.10.0"
 description = "library with cross-python path, ini-parsing, io, code, log facilities"
 category = "main"
 optional = false
@@ -1206,7 +1305,7 @@ name = "pyarrow"
 version = "2.0.0"
 description = "Python library for Apache Arrow"
 category = "main"
-optional = true
+optional = false
 python-versions = ">=3.5"
 
 [package.dependencies]
@@ -1222,7 +1321,7 @@ python-versions = "*"
 
 [[package]]
 name = "pycodestyle"
-version = "2.6.0"
+version = "2.7.0"
 description = "Python style guide checker"
 category = "dev"
 optional = false
@@ -1238,7 +1337,7 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "pyflakes"
-version = "2.2.0"
+version = "2.3.0"
 description = "passive checker of Python programs"
 category = "dev"
 optional = false
@@ -1246,7 +1345,7 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "pygments"
-version = "2.7.2"
+version = "2.8.1"
 description = "Pygments is a syntax highlighting package written in Python."
 category = "main"
 optional = false
@@ -1254,11 +1353,11 @@ python-versions = ">=3.5"
 
 [[package]]
 name = "pymysql"
-version = "0.10.1"
+version = "1.0.2"
 description = "Pure Python MySQL Driver"
 category = "main"
 optional = true
-python-versions = "*"
+python-versions = ">=3.6"
 
 [package.extras]
 ed25519 = ["PyNaCl (>=1.4.0)"]
@@ -1272,6 +1371,17 @@ category = "main"
 optional = false
 python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
 
+[[package]]
+name = "pypiwin32"
+version = "223"
+description = ""
+category = "dev"
+optional = false
+python-versions = "*"
+
+[package.dependencies]
+pywin32 = ">=223"
+
 [[package]]
 name = "pyrsistent"
 version = "0.17.3"
@@ -1333,7 +1443,7 @@ name = "python-slugify"
 version = "4.0.1"
 description = "A Python Slugify application that handles Unicode"
 category = "main"
-optional = false
+optional = true
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
 
 [package.dependencies]
@@ -1344,7 +1454,7 @@ unidecode = ["Unidecode (>=1.1.1)"]
 
 [[package]]
 name = "pytz"
-version = "2020.4"
+version = "2020.1"
 description = "World timezone definitions, modern and historical"
 category = "main"
 optional = false
@@ -1352,7 +1462,7 @@ python-versions = "*"
 
 [[package]]
 name = "pywin32"
-version = "227"
+version = "300"
 description = "Python for Window Extensions"
 category = "main"
 optional = false
@@ -1368,23 +1478,43 @@ python-versions = "*"
 
 [[package]]
 name = "pyyaml"
-version = "5.3.1"
+version = "5.4.1"
 description = "YAML parser and emitter for Python"
 category = "main"
 optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
 
 [[package]]
 name = "pyzmq"
-version = "20.0.0"
+version = "22.0.3"
 description = "Python bindings for 0MQ"
 category = "main"
 optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+cffi = {version = "*", markers = "implementation_name == \"pypy\""}
+py = {version = "*", markers = "implementation_name == \"pypy\""}
+
+[[package]]
+name = "redshift-connector"
+version = "2.0.876"
+description = "Redshift interface library"
+category = "main"
+optional = true
 python-versions = ">=3.5"
 
 [package.dependencies]
-cffi = {version = "*", markers = "implementation_name === \"pypy\""}
-py = {version = "*", markers = "implementation_name === \"pypy\""}
+beautifulsoup4 = ">=4.7.0,<4.8.0"
+boto3 = ">=1.16.8,<2.0.0"
+botocore = ">=1.19.8,<2.0.0"
+lxml = ">=4.6.2"
+pytz = ">=2020.1,<2020.2"
+requests = ">=2.23.0,<2.24.0"
+scramp = ">=1.2.0,<1.3.0"
+
+[package.extras]
+full = ["numpy", "pandas"]
 
 [[package]]
 name = "regex"
@@ -1410,7 +1540,7 @@ urllib3 = ">=1.21.1,<1.25.0 || >1.25.0,<1.25.1 || >1.25.1,<1.26"
 
 [package.extras]
 security = ["pyOpenSSL (>=0.14)", "cryptography (>=1.3.4)"]
-socks = ["PySocks (>=1.5.6,!=1.5.7)", "win-inet-pton"]
+socks = ["PySocks (>=1.5.6,<1.5.7 || >1.5.7)", "win-inet-pton"]
 
 [[package]]
 name = "responses"
@@ -1433,7 +1563,7 @@ name = "rfc3986"
 version = "1.4.0"
 description = "Validating URI References per RFC 3986"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [package.extras]
@@ -1441,7 +1571,7 @@ idna2008 = ["idna"]
 
 [[package]]
 name = "rsa"
-version = "4.6"
+version = "4.7.2"
 description = "Pure-Python RSA implementation"
 category = "dev"
 optional = false
@@ -1452,14 +1582,14 @@ pyasn1 = ">=0.1.3"
 
 [[package]]
 name = "ruamel.yaml"
-version = "0.16.12"
+version = "0.16.13"
 description = "ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order"
 category = "main"
 optional = true
 python-versions = "*"
 
 [package.dependencies]
-"ruamel.yaml.clib" = {version = ">=0.1.2", markers = "platform_python_implementation == \"CPython\" and python_version < \"3.9\""}
+"ruamel.yaml.clib" = {version = ">=0.1.2", markers = "platform_python_implementation == \"CPython\" and python_version < \"3.10\""}
 
 [package.extras]
 docs = ["ryd"]
@@ -1473,9 +1603,21 @@ category = "main"
 optional = true
 python-versions = "*"
 
+[[package]]
+name = "s3fs"
+version = "0.4.2"
+description = "Convenient Filesystem interface over S3"
+category = "main"
+optional = false
+python-versions = ">= 3.5"
+
+[package.dependencies]
+botocore = ">=1.12.91"
+fsspec = ">=0.6.0"
+
 [[package]]
 name = "s3transfer"
-version = "0.3.3"
+version = "0.3.4"
 description = "An Amazon S3 Transfer Manager"
 category = "main"
 optional = false
@@ -1486,14 +1628,22 @@ botocore = ">=1.12.36,<2.0a.0"
 
 [[package]]
 name = "scipy"
-version = "1.5.4"
+version = "1.6.1"
 description = "SciPy: Scientific Library for Python"
 category = "main"
 optional = true
-python-versions = ">=3.6"
+python-versions = ">=3.7"
 
 [package.dependencies]
-numpy = ">=1.14.5"
+numpy = ">=1.16.5"
+
+[[package]]
+name = "scramp"
+version = "1.2.0"
+description = "An implementation of the SCRAM protocol."
+category = "main"
+optional = true
+python-versions = ">=3.5"
 
 [[package]]
 name = "send2trash"
@@ -1508,7 +1658,7 @@ name = "simpleeval"
 version = "0.9.10"
 description = "A simple, safe single expression evaluator library."
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
@@ -1516,7 +1666,7 @@ name = "simplejson"
 version = "3.17.2"
 description = "Simple, fast, extensible JSON encoder/decoder for Python"
 category = "main"
-optional = false
+optional = true
 python-versions = ">=2.5, !=3.0.*, !=3.1.*, !=3.2.*"
 
 [[package]]
@@ -1528,51 +1678,27 @@ optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"
 
 [[package]]
-name = "sqlalchemy"
-version = "1.3.20"
-description = "Database Abstraction Library"
+name = "soupsieve"
+version = "2.2"
+description = "A modern CSS selector implementation for Beautiful Soup."
 category = "main"
 optional = true
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
-
-[package.extras]
-mssql = ["pyodbc"]
-mssql_pymssql = ["pymssql"]
-mssql_pyodbc = ["pyodbc"]
-mysql = ["mysqlclient"]
-oracle = ["cx-oracle"]
-postgresql = ["psycopg2"]
-postgresql_pg8000 = ["pg8000"]
-postgresql_psycopg2binary = ["psycopg2-binary"]
-postgresql_psycopg2cffi = ["psycopg2cffi"]
-pymysql = ["pymysql"]
-
-[[package]]
-name = "sqlalchemy-redshift"
-version = "0.8.1"
-description = "Amazon Redshift Dialect for sqlalchemy"
-category = "main"
-optional = true
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
-
-[package.dependencies]
-packaging = "*"
-SQLAlchemy = ">=0.9.2,<2.0.0"
+python-versions = ">=3.6"
 
 [[package]]
 name = "stringcase"
 version = "1.2.0"
 description = "String case converter."
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
 name = "tabulate"
-version = "0.8.7"
+version = "0.8.9"
 description = "Pretty-print tabular data"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [package.extras]
@@ -1588,7 +1714,7 @@ python-versions = "*"
 
 [[package]]
 name = "terminado"
-version = "0.9.1"
+version = "0.9.2"
 description = "Tornado websocket backend for the Xterm.js Javascript terminal emulator library."
 category = "main"
 optional = false
@@ -1615,7 +1741,7 @@ name = "text-unidecode"
 version = "1.3"
 description = "The most basic Text::Unidecode port"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
@@ -1658,7 +1784,7 @@ test = ["pytest"]
 
 [[package]]
 name = "typed-ast"
-version = "1.4.1"
+version = "1.4.2"
 description = "a fork of Python 2 and 3 ast modules with type comment support"
 category = "main"
 optional = true
@@ -1669,7 +1795,7 @@ name = "typing-extensions"
 version = "3.7.4.3"
 description = "Backported and Experimental Type Hints for Python 3.5+"
 category = "main"
-optional = true
+optional = false
 python-versions = "*"
 
 [[package]]
@@ -1688,7 +1814,7 @@ name = "unicodecsv"
 version = "0.14.1"
 description = "Python2's stdlib csv module is nice, but it doesn't support unicode. This module is a drop-in replacement which *does*."
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
@@ -1702,7 +1828,7 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4"
 [package.extras]
 brotli = ["brotlipy (>=0.6.0)"]
 secure = ["pyOpenSSL (>=0.14)", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "certifi", "ipaddress"]
-socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]
+socks = ["PySocks (>=1.5.6,<1.5.7 || >1.5.7,<2.0)"]
 
 [[package]]
 name = "wcwidth"
@@ -1722,8 +1848,8 @@ python-versions = "*"
 
 [[package]]
 name = "websocket-client"
-version = "0.57.0"
-description = "WebSocket client for Python. hybi13 is supported."
+version = "0.58.0"
+description = "WebSocket client for Python with low level API options"
 category = "dev"
 optional = false
 python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
@@ -1764,18 +1890,23 @@ python-versions = "*"
 
 [[package]]
 name = "xlrd"
-version = "1.2.0"
-description = "Library for developers to extract data from Microsoft Excel (tm) spreadsheet files"
+version = "2.0.1"
+description = "Library for developers to extract data from Microsoft Excel (tm) .xls spreadsheet files"
 category = "main"
-optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+optional = true
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
+
+[package.extras]
+build = ["wheel", "twine"]
+docs = ["sphinx"]
+test = ["pytest", "pytest-cov"]
 
 [[package]]
 name = "xlwt"
 version = "1.3.0"
 description = "Library to create spreadsheet files compatible with MS Excel 97/2000/XP/2003 XLS files, on any platform, with Python 2.6, 2.7, 3.3+"
 category = "main"
-optional = false
+optional = true
 python-versions = "*"
 
 [[package]]
@@ -1788,23 +1919,24 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "zipp"
-version = "3.4.0"
+version = "3.4.1"
 description = "Backport of pathlib-compatible object wrapper for zip files"
 category = "main"
 optional = false
 python-versions = ">=3.6"
 
 [package.extras]
-docs = ["sphinx", "jaraco.packaging (>=3.2)", "rst.linker (>=1.9)"]
-testing = ["pytest (>=3.5,!=3.7.3)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-cov", "jaraco.test (>=3.2.0)", "jaraco.itertools", "func-timeout", "pytest-black (>=0.3.7)", "pytest-mypy"]
+docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)"]
+testing = ["pytest (>=4.6)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-cov", "pytest-enabler", "jaraco.itertools", "func-timeout", "pytest-black (>=0.3.7)", "pytest-mypy"]
 
 [extras]
-ge = ["pandas", "great-expectations", "awswrangler"]
+frictionless = ["frictionless"]
+ge = ["great-expectations", "awswrangler"]
 
 [metadata]
 lock-version = "1.1"
-python-versions = ">=3.7,<3.9"
-content-hash = "195f6ab1eb5d9bcc4e9dd0588aedfd88e293f00f770c4220cfd09cedccb779c9"
+python-versions = ">=3.7.1,<3.9"
+content-hash = "6cbd1cd1167ad0af1151a97ba00b7b3b8529b85802cdf3d0fac4f8d9241e3be4"
 
 [metadata.files]
 altair = [
@@ -1816,8 +1948,8 @@ appdirs = [
     {file = "appdirs-1.4.4.tar.gz", hash = "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41"},
 ]
 appnope = [
-    {file = "appnope-0.1.0-py2.py3-none-any.whl", hash = "sha256:5b26757dc6f79a3b7dc9fab95359328d5747fcb2409d331ea66d0272b90ab2a0"},
-    {file = "appnope-0.1.0.tar.gz", hash = "sha256:8b995ffe925347a2138d7ac0fe77155e4311a0ea6d6da4f5128fe4b3cbe5ed71"},
+    {file = "appnope-0.1.2-py2.py3-none-any.whl", hash = "sha256:93aa393e9d6c54c5cd570ccadd8edad61ea0c4b9ea7a01409020c9aa019eb442"},
+    {file = "appnope-0.1.2.tar.gz", hash = "sha256:dd83cd4b5b460958838f6eb3000c660b1f9caf2a5b1de4264e941512f603258a"},
 ]
 argon2-cffi = [
     {file = "argon2-cffi-20.1.0.tar.gz", hash = "sha256:d8029b2d3e4b4cea770e9e5a0104dd8fa185c1724a0f01528ae4826a6d25f97d"},
@@ -1837,6 +1969,7 @@ argon2-cffi = [
     {file = "argon2_cffi-20.1.0-cp38-cp38-win32.whl", hash = "sha256:77e909cc756ef81d6abb60524d259d959bab384832f0c651ed7dcb6e5ccdbb78"},
     {file = "argon2_cffi-20.1.0-cp38-cp38-win_amd64.whl", hash = "sha256:9dfd5197852530294ecb5795c97a823839258dfd5eb9420233c7cfedec2058f2"},
 ]
+arrow-pd-parser = []
 async-generator = [
     {file = "async_generator-1.10-py3-none-any.whl", hash = "sha256:01c7bf666359b4967d2cda0000cc2e4af16a0ae098cbffcb8472fb9e8ad6585b"},
     {file = "async_generator-1.10.tar.gz", hash = "sha256:6ebb3d106c12920aaae42ccb6f787ef5eefdcdd166ea3d628fa8476abe712144"},
@@ -1850,85 +1983,93 @@ attrs = [
     {file = "attrs-20.3.0.tar.gz", hash = "sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700"},
 ]
 aws-sam-translator = [
-    {file = "aws-sam-translator-1.31.0.tar.gz", hash = "sha256:3a1d73d098161e60966b0d53bb310c98e4f66101688cce3d1697903643782d79"},
-    {file = "aws_sam_translator-1.31.0-py2-none-any.whl", hash = "sha256:65749571042e704027bbcaabe6dddd5d13ab54959c46e60c347918d6b85551fd"},
-    {file = "aws_sam_translator-1.31.0-py3-none-any.whl", hash = "sha256:b87a61cffba8f29e4f1b5eb43a4abafdfb14b76aedc716517d48dbdf0b1f989a"},
+    {file = "aws-sam-translator-1.35.0.tar.gz", hash = "sha256:5cf7faab3566843f3b44ef1a42a9c106ffb50809da4002faab818076dcc7bff8"},
+    {file = "aws_sam_translator-1.35.0-py2-none-any.whl", hash = "sha256:2f8904fd4a631752bc441a8fd928c444ed98ceb86b94d25ed7b84982e2eff1cd"},
+    {file = "aws_sam_translator-1.35.0-py3-none-any.whl", hash = "sha256:c35075e7e804490d6025598ed4878ad3ab8668e37cafb7ae75120b1c37a6d212"},
 ]
 aws-xray-sdk = [
     {file = "aws-xray-sdk-2.6.0.tar.gz", hash = "sha256:abf5b90f740e1f402e23414c9670e59cb9772e235e271fef2bce62b9100cbc77"},
     {file = "aws_xray_sdk-2.6.0-py2.py3-none-any.whl", hash = "sha256:076f7c610cd3564bbba3507d43e328fb6ff4a2e841d3590f39b2c3ce99d41e1d"},
 ]
 awswrangler = [
-    {file = "awswrangler-1.10.0-py3-none-any.whl", hash = "sha256:0a0245b816c3e9ac9360b1ec841bd930c155cdc43f63d42eaa962cbc7cd15db3"},
-    {file = "awswrangler-1.10.0-py3.6.egg", hash = "sha256:0425618597214c6f8a77b4a879d4175ce925680e3878a79908e71a1198bf81fd"},
-    {file = "awswrangler-1.10.0.tar.gz", hash = "sha256:7aa3828ed6e7a88a5dee8ef4757682944213fc1705a67d2172e523e103812741"},
+    {file = "awswrangler-2.3.0-py3-none-any.whl", hash = "sha256:599e72937d87f84bce95bbd3e7490a49a065ff2c9844401562520994b31471af"},
+    {file = "awswrangler-2.3.0-py3.6.egg", hash = "sha256:147b7d6a1521dc0f3410e0b49c7449454fe27a4d20bb125c2ebc491b75fbc9a9"},
+    {file = "awswrangler-2.3.0.tar.gz", hash = "sha256:5f369fccf791e80efa7bb70496182857f35233275f66ae62d476e68be8892165"},
 ]
 backcall = [
     {file = "backcall-0.2.0-py2.py3-none-any.whl", hash = "sha256:fbbce6a29f263178a1f7915c1940bde0ec2b2a967566fe1c65c1dfb7422bd255"},
     {file = "backcall-0.2.0.tar.gz", hash = "sha256:5cbdbf27be5e7cfadb448baf0aa95508f91f2bbc6c6437cd9cd06e2a4c215e1e"},
 ]
+beautifulsoup4 = [
+    {file = "beautifulsoup4-4.7.1-py2-none-any.whl", hash = "sha256:ba6d5c59906a85ac23dadfe5c88deaf3e179ef565f4898671253e50a78680718"},
+    {file = "beautifulsoup4-4.7.1-py3-none-any.whl", hash = "sha256:034740f6cb549b4e932ae1ab975581e6103ac8f942200a0e9759065984391858"},
+    {file = "beautifulsoup4-4.7.1.tar.gz", hash = "sha256:945065979fb8529dd2f37dbb58f00b661bdbcbebf954f93b32fdf5263ef35348"},
+]
 black = [
     {file = "black-20.8b1.tar.gz", hash = "sha256:1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea"},
 ]
 bleach = [
-    {file = "bleach-3.2.1-py2.py3-none-any.whl", hash = "sha256:9f8ccbeb6183c6e6cddea37592dfb0167485c1e3b13b3363bc325aa8bda3adbd"},
-    {file = "bleach-3.2.1.tar.gz", hash = "sha256:52b5919b81842b1854196eaae5ca29679a2f2e378905c346d3ca8227c2c66080"},
+    {file = "bleach-3.3.0-py2.py3-none-any.whl", hash = "sha256:6123ddc1052673e52bab52cdc955bcb57a015264a1c57d37bea2f6b817af0125"},
+    {file = "bleach-3.3.0.tar.gz", hash = "sha256:98b3170739e5e83dd9dc19633f074727ad848cbedb6026708c8ac2d3b697a433"},
 ]
 boto = [
     {file = "boto-2.49.0-py2.py3-none-any.whl", hash = "sha256:147758d41ae7240dc989f0039f27da8ca0d53734be0eb869ef16e3adcfa462e8"},
     {file = "boto-2.49.0.tar.gz", hash = "sha256:ea0d3b40a2d852767be77ca343b58a9e3a4b00d9db440efb8da74b4e58025e5a"},
 ]
 boto3 = [
-    {file = "boto3-1.16.24-py2.py3-none-any.whl", hash = "sha256:f12fc470d170b283dcea921f0ae3744dd8d0fc5263dcc8974c891edefb008a17"},
-    {file = "boto3-1.16.24.tar.gz", hash = "sha256:7a76e9012bd7a47da0f75671d98a6cc2db0734b6f3ca17b34acfd381ebca2fb9"},
+    {file = "boto3-1.17.27-py2.py3-none-any.whl", hash = "sha256:6758751f1181b9363e4e7559dcbd5ac0fc7147b73f429c976ec5ecd1688c9ec7"},
+    {file = "boto3-1.17.27.tar.gz", hash = "sha256:fa41987f9f71368013767306d9522b627946a01b4843938a26fb19cc8adb06c0"},
 ]
 botocore = [
-    {file = "botocore-1.19.24-py2.py3-none-any.whl", hash = "sha256:7d4efbc621e3fe17226b356a65526210c9284f9a823ce7be7209108644823c40"},
-    {file = "botocore-1.19.24.tar.gz", hash = "sha256:5975b10dbc6a70c6789b95d083fbb93e4191f13afe25cd7a834cf2168fddbcd8"},
+    {file = "botocore-1.20.27-py2.py3-none-any.whl", hash = "sha256:57e45c9d443163da7312cae61bcc60382e6d0b3aecda68e850d6438162fe7b5b"},
+    {file = "botocore-1.20.27.tar.gz", hash = "sha256:4477803f07649f4d80b17d054820e7a09bb2cb0792d0decc2812108bc3759c4a"},
 ]
 certifi = [
-    {file = "certifi-2020.11.8-py2.py3-none-any.whl", hash = "sha256:1f422849db327d534e3d0c5f02a263458c3955ec0aae4ff09b95f195c59f4edd"},
-    {file = "certifi-2020.11.8.tar.gz", hash = "sha256:f05def092c44fbf25834a51509ef6e631dc19765ab8a57b4e7ab85531f0a9cf4"},
+    {file = "certifi-2020.12.5-py2.py3-none-any.whl", hash = "sha256:719a74fb9e33b9bd44cc7f3a8d94bc35e4049deebe19ba7d8e108280cfd59830"},
+    {file = "certifi-2020.12.5.tar.gz", hash = "sha256:1a4995114262bffbc2413b159f2a1a480c969de6e6eb13ee966d470af86af59c"},
 ]
 cffi = [
-    {file = "cffi-1.14.4-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:ebb253464a5d0482b191274f1c8bf00e33f7e0b9c66405fbffc61ed2c839c775"},
-    {file = "cffi-1.14.4-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:2c24d61263f511551f740d1a065eb0212db1dbbbbd241db758f5244281590c06"},
-    {file = "cffi-1.14.4-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:9f7a31251289b2ab6d4012f6e83e58bc3b96bd151f5b5262467f4bb6b34a7c26"},
-    {file = "cffi-1.14.4-cp27-cp27m-win32.whl", hash = "sha256:5cf4be6c304ad0b6602f5c4e90e2f59b47653ac1ed9c662ed379fe48a8f26b0c"},
-    {file = "cffi-1.14.4-cp27-cp27m-win_amd64.whl", hash = "sha256:f60567825f791c6f8a592f3c6e3bd93dd2934e3f9dac189308426bd76b00ef3b"},
-    {file = "cffi-1.14.4-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:c6332685306b6417a91b1ff9fae889b3ba65c2292d64bd9245c093b1b284809d"},
-    {file = "cffi-1.14.4-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:d9efd8b7a3ef378dd61a1e77367f1924375befc2eba06168b6ebfa903a5e59ca"},
-    {file = "cffi-1.14.4-cp35-cp35m-macosx_10_9_x86_64.whl", hash = "sha256:51a8b381b16ddd370178a65360ebe15fbc1c71cf6f584613a7ea08bfad946698"},
-    {file = "cffi-1.14.4-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:1d2c4994f515e5b485fd6d3a73d05526aa0fcf248eb135996b088d25dfa1865b"},
-    {file = "cffi-1.14.4-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:af5c59122a011049aad5dd87424b8e65a80e4a6477419c0c1015f73fb5ea0293"},
-    {file = "cffi-1.14.4-cp35-cp35m-win32.whl", hash = "sha256:594234691ac0e9b770aee9fcdb8fa02c22e43e5c619456efd0d6c2bf276f3eb2"},
-    {file = "cffi-1.14.4-cp35-cp35m-win_amd64.whl", hash = "sha256:64081b3f8f6f3c3de6191ec89d7dc6c86a8a43911f7ecb422c60e90c70be41c7"},
-    {file = "cffi-1.14.4-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:f803eaa94c2fcda012c047e62bc7a51b0bdabda1cad7a92a522694ea2d76e49f"},
-    {file = "cffi-1.14.4-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:105abaf8a6075dc96c1fe5ae7aae073f4696f2905fde6aeada4c9d2926752362"},
-    {file = "cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:0638c3ae1a0edfb77c6765d487fee624d2b1ee1bdfeffc1f0b58c64d149e7eec"},
-    {file = "cffi-1.14.4-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:7c6b1dece89874d9541fc974917b631406233ea0440d0bdfbb8e03bf39a49b3b"},
-    {file = "cffi-1.14.4-cp36-cp36m-win32.whl", hash = "sha256:155136b51fd733fa94e1c2ea5211dcd4c8879869008fc811648f16541bf99668"},
-    {file = "cffi-1.14.4-cp36-cp36m-win_amd64.whl", hash = "sha256:6bc25fc545a6b3d57b5f8618e59fc13d3a3a68431e8ca5fd4c13241cd70d0009"},
-    {file = "cffi-1.14.4-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a7711edca4dcef1a75257b50a2fbfe92a65187c47dab5a0f1b9b332c5919a3fb"},
-    {file = "cffi-1.14.4-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:00e28066507bfc3fe865a31f325c8391a1ac2916219340f87dfad602c3e48e5d"},
-    {file = "cffi-1.14.4-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:798caa2a2384b1cbe8a2a139d80734c9db54f9cc155c99d7cc92441a23871c03"},
-    {file = "cffi-1.14.4-cp37-cp37m-win32.whl", hash = "sha256:00a1ba5e2e95684448de9b89888ccd02c98d512064b4cb987d48f4b40aa0421e"},
-    {file = "cffi-1.14.4-cp37-cp37m-win_amd64.whl", hash = "sha256:9cc46bc107224ff5b6d04369e7c595acb700c3613ad7bcf2e2012f62ece80c35"},
-    {file = "cffi-1.14.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:df5169c4396adc04f9b0a05f13c074df878b6052430e03f50e68adf3a57aa28d"},
-    {file = "cffi-1.14.4-cp38-cp38-manylinux1_i686.whl", hash = "sha256:9ffb888f19d54a4d4dfd4b3f29bc2c16aa4972f1c2ab9c4ab09b8ab8685b9c2b"},
-    {file = "cffi-1.14.4-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:8d6603078baf4e11edc4168a514c5ce5b3ba6e3e9c374298cb88437957960a53"},
-    {file = "cffi-1.14.4-cp38-cp38-win32.whl", hash = "sha256:b4e248d1087abf9f4c10f3c398896c87ce82a9856494a7155823eb45a892395d"},
-    {file = "cffi-1.14.4-cp38-cp38-win_amd64.whl", hash = "sha256:ec80dc47f54e6e9a78181ce05feb71a0353854cc26999db963695f950b5fb375"},
-    {file = "cffi-1.14.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:840793c68105fe031f34d6a086eaea153a0cd5c491cde82a74b420edd0a2b909"},
-    {file = "cffi-1.14.4-cp39-cp39-manylinux1_i686.whl", hash = "sha256:b18e0a9ef57d2b41f5c68beefa32317d286c3d6ac0484efd10d6e07491bb95dd"},
-    {file = "cffi-1.14.4-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:045d792900a75e8b1e1b0ab6787dd733a8190ffcf80e8c8ceb2fb10a29ff238a"},
-    {file = "cffi-1.14.4-cp39-cp39-win32.whl", hash = "sha256:ba4e9e0ae13fc41c6b23299545e5ef73055213e466bd107953e4a013a5ddd7e3"},
-    {file = "cffi-1.14.4-cp39-cp39-win_amd64.whl", hash = "sha256:f032b34669220030f905152045dfa27741ce1a6db3324a5bc0b96b6c7420c87b"},
-    {file = "cffi-1.14.4.tar.gz", hash = "sha256:1a465cbe98a7fd391d47dce4b8f7e5b921e6cd805ef421d04f5f66ba8f06086c"},
+    {file = "cffi-1.14.5-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:bb89f306e5da99f4d922728ddcd6f7fcebb3241fc40edebcb7284d7514741991"},
+    {file = "cffi-1.14.5-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:34eff4b97f3d982fb93e2831e6750127d1355a923ebaeeb565407b3d2f8d41a1"},
+    {file = "cffi-1.14.5-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:99cd03ae7988a93dd00bcd9d0b75e1f6c426063d6f03d2f90b89e29b25b82dfa"},
+    {file = "cffi-1.14.5-cp27-cp27m-win32.whl", hash = "sha256:65fa59693c62cf06e45ddbb822165394a288edce9e276647f0046e1ec26920f3"},
+    {file = "cffi-1.14.5-cp27-cp27m-win_amd64.whl", hash = "sha256:51182f8927c5af975fece87b1b369f722c570fe169f9880764b1ee3bca8347b5"},
+    {file = "cffi-1.14.5-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:43e0b9d9e2c9e5d152946b9c5fe062c151614b262fda2e7b201204de0b99e482"},
+    {file = "cffi-1.14.5-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:cbde590d4faaa07c72bf979734738f328d239913ba3e043b1e98fe9a39f8b2b6"},
+    {file = "cffi-1.14.5-cp35-cp35m-macosx_10_9_x86_64.whl", hash = "sha256:5de7970188bb46b7bf9858eb6890aad302577a5f6f75091fd7cdd3ef13ef3045"},
+    {file = "cffi-1.14.5-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:a465da611f6fa124963b91bf432d960a555563efe4ed1cc403ba5077b15370aa"},
+    {file = "cffi-1.14.5-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:d42b11d692e11b6634f7613ad8df5d6d5f8875f5d48939520d351007b3c13406"},
+    {file = "cffi-1.14.5-cp35-cp35m-win32.whl", hash = "sha256:72d8d3ef52c208ee1c7b2e341f7d71c6fd3157138abf1a95166e6165dd5d4369"},
+    {file = "cffi-1.14.5-cp35-cp35m-win_amd64.whl", hash = "sha256:29314480e958fd8aab22e4a58b355b629c59bf5f2ac2492b61e3dc06d8c7a315"},
+    {file = "cffi-1.14.5-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:3d3dd4c9e559eb172ecf00a2a7517e97d1e96de2a5e610bd9b68cea3925b4892"},
+    {file = "cffi-1.14.5-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:48e1c69bbacfc3d932221851b39d49e81567a4d4aac3b21258d9c24578280058"},
+    {file = "cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:69e395c24fc60aad6bb4fa7e583698ea6cc684648e1ffb7fe85e3c1ca131a7d5"},
+    {file = "cffi-1.14.5-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:9e93e79c2551ff263400e1e4be085a1210e12073a31c2011dbbda14bda0c6132"},
+    {file = "cffi-1.14.5-cp36-cp36m-win32.whl", hash = "sha256:58e3f59d583d413809d60779492342801d6e82fefb89c86a38e040c16883be53"},
+    {file = "cffi-1.14.5-cp36-cp36m-win_amd64.whl", hash = "sha256:005a36f41773e148deac64b08f233873a4d0c18b053d37da83f6af4d9087b813"},
+    {file = "cffi-1.14.5-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:2894f2df484ff56d717bead0a5c2abb6b9d2bf26d6960c4604d5c48bbc30ee73"},
+    {file = "cffi-1.14.5-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:0857f0ae312d855239a55c81ef453ee8fd24136eaba8e87a2eceba644c0d4c06"},
+    {file = "cffi-1.14.5-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:cd2868886d547469123fadc46eac7ea5253ea7fcb139f12e1dfc2bbd406427d1"},
+    {file = "cffi-1.14.5-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:35f27e6eb43380fa080dccf676dece30bef72e4a67617ffda586641cd4508d49"},
+    {file = "cffi-1.14.5-cp37-cp37m-win32.whl", hash = "sha256:9ff227395193126d82e60319a673a037d5de84633f11279e336f9c0f189ecc62"},
+    {file = "cffi-1.14.5-cp37-cp37m-win_amd64.whl", hash = "sha256:9cf8022fb8d07a97c178b02327b284521c7708d7c71a9c9c355c178ac4bbd3d4"},
+    {file = "cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:8b198cec6c72df5289c05b05b8b0969819783f9418e0409865dac47288d2a053"},
+    {file = "cffi-1.14.5-cp38-cp38-manylinux1_i686.whl", hash = "sha256:ad17025d226ee5beec591b52800c11680fca3df50b8b29fe51d882576e039ee0"},
+    {file = "cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:6c97d7350133666fbb5cf4abdc1178c812cb205dc6f41d174a7b0f18fb93337e"},
+    {file = "cffi-1.14.5-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:8ae6299f6c68de06f136f1f9e69458eae58f1dacf10af5c17353eae03aa0d827"},
+    {file = "cffi-1.14.5-cp38-cp38-win32.whl", hash = "sha256:b85eb46a81787c50650f2392b9b4ef23e1f126313b9e0e9013b35c15e4288e2e"},
+    {file = "cffi-1.14.5-cp38-cp38-win_amd64.whl", hash = "sha256:1f436816fc868b098b0d63b8920de7d208c90a67212546d02f84fe78a9c26396"},
+    {file = "cffi-1.14.5-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:1071534bbbf8cbb31b498d5d9db0f274f2f7a865adca4ae429e147ba40f73dea"},
+    {file = "cffi-1.14.5-cp39-cp39-manylinux1_i686.whl", hash = "sha256:9de2e279153a443c656f2defd67769e6d1e4163952b3c622dcea5b08a6405322"},
+    {file = "cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:6e4714cc64f474e4d6e37cfff31a814b509a35cb17de4fb1999907575684479c"},
+    {file = "cffi-1.14.5-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:158d0d15119b4b7ff6b926536763dc0714313aa59e320ddf787502c70c4d4bee"},
+    {file = "cffi-1.14.5-cp39-cp39-win32.whl", hash = "sha256:afb29c1ba2e5a3736f1c301d9d0abe3ec8b86957d04ddfa9d7a6a42b9367e396"},
+    {file = "cffi-1.14.5-cp39-cp39-win_amd64.whl", hash = "sha256:f2d45f97ab6bb54753eab54fffe75aaf3de4ff2341c9daee1987ee1837636f1d"},
+    {file = "cffi-1.14.5.tar.gz", hash = "sha256:fd78e5fee591709f32ef6edb9a015b4aa1a5022598e36227500c8f4e02328d9c"},
 ]
 cfn-lint = [
-    {file = "cfn-lint-0.41.0.tar.gz", hash = "sha256:8706f91b8f97ffa36c49da94c379b8d36e58349f70a80c1be723d796ae7334af"},
-    {file = "cfn_lint-0.41.0-py3-none-any.whl", hash = "sha256:0fe08943ade00e832de08b94d5cb9736d4d920eb4df073fa4f57092bcb930c30"},
+    {file = "cfn-lint-0.47.2.tar.gz", hash = "sha256:91e323a87db35ec56e34fd627f0bb76eff80d089f2b947ae4785dce8873d34ac"},
+    {file = "cfn_lint-0.47.2-py3-none-any.whl", hash = "sha256:2f226f5d3c9aa4d6dd806d168618973dcb6c645c3aecbdaea596ae07068fbad4"},
 ]
 chardet = [
     {file = "chardet-3.0.4-py2.py3-none-any.whl", hash = "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"},
@@ -1940,46 +2081,37 @@ click = [
 ]
 colorama = [
     {file = "colorama-0.4.4-py2.py3-none-any.whl", hash = "sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2"},
+    {file = "colorama-0.4.4.tar.gz", hash = "sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b"},
 ]
 cryptography = [
-    {file = "cryptography-3.2.1-cp27-cp27m-macosx_10_10_x86_64.whl", hash = "sha256:6dc59630ecce8c1f558277ceb212c751d6730bd12c80ea96b4ac65637c4f55e7"},
-    {file = "cryptography-3.2.1-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:75e8e6684cf0034f6bf2a97095cb95f81537b12b36a8fedf06e73050bb171c2d"},
-    {file = "cryptography-3.2.1-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:4e7268a0ca14536fecfdf2b00297d4e407da904718658c1ff1961c713f90fd33"},
-    {file = "cryptography-3.2.1-cp27-cp27m-win32.whl", hash = "sha256:7117319b44ed1842c617d0a452383a5a052ec6aa726dfbaffa8b94c910444297"},
-    {file = "cryptography-3.2.1-cp27-cp27m-win_amd64.whl", hash = "sha256:a733671100cd26d816eed39507e585c156e4498293a907029969234e5e634bc4"},
-    {file = "cryptography-3.2.1-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:a75f306a16d9f9afebfbedc41c8c2351d8e61e818ba6b4c40815e2b5740bb6b8"},
-    {file = "cryptography-3.2.1-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:5849d59358547bf789ee7e0d7a9036b2d29e9a4ddf1ce5e06bb45634f995c53e"},
-    {file = "cryptography-3.2.1-cp35-abi3-macosx_10_10_x86_64.whl", hash = "sha256:bd717aa029217b8ef94a7d21632a3bb5a4e7218a4513d2521c2a2fd63011e98b"},
-    {file = "cryptography-3.2.1-cp35-abi3-manylinux1_x86_64.whl", hash = "sha256:efe15aca4f64f3a7ea0c09c87826490e50ed166ce67368a68f315ea0807a20df"},
-    {file = "cryptography-3.2.1-cp35-abi3-manylinux2010_x86_64.whl", hash = "sha256:32434673d8505b42c0de4de86da8c1620651abd24afe91ae0335597683ed1b77"},
-    {file = "cryptography-3.2.1-cp35-abi3-manylinux2014_aarch64.whl", hash = "sha256:7b8d9d8d3a9bd240f453342981f765346c87ade811519f98664519696f8e6ab7"},
-    {file = "cryptography-3.2.1-cp35-cp35m-win32.whl", hash = "sha256:d3545829ab42a66b84a9aaabf216a4dce7f16dbc76eb69be5c302ed6b8f4a29b"},
-    {file = "cryptography-3.2.1-cp35-cp35m-win_amd64.whl", hash = "sha256:a4e27ed0b2504195f855b52052eadcc9795c59909c9d84314c5408687f933fc7"},
-    {file = "cryptography-3.2.1-cp36-abi3-win32.whl", hash = "sha256:13b88a0bd044b4eae1ef40e265d006e34dbcde0c2f1e15eb9896501b2d8f6c6f"},
-    {file = "cryptography-3.2.1-cp36-abi3-win_amd64.whl", hash = "sha256:07ca431b788249af92764e3be9a488aa1d39a0bc3be313d826bbec690417e538"},
-    {file = "cryptography-3.2.1-cp36-cp36m-win32.whl", hash = "sha256:a035a10686532b0587d58a606004aa20ad895c60c4d029afa245802347fab57b"},
-    {file = "cryptography-3.2.1-cp36-cp36m-win_amd64.whl", hash = "sha256:d26a2557d8f9122f9bf445fc7034242f4375bd4e95ecda007667540270965b13"},
-    {file = "cryptography-3.2.1-cp37-cp37m-win32.whl", hash = "sha256:545a8550782dda68f8cdc75a6e3bf252017aa8f75f19f5a9ca940772fc0cb56e"},
-    {file = "cryptography-3.2.1-cp37-cp37m-win_amd64.whl", hash = "sha256:55d0b896631412b6f0c7de56e12eb3e261ac347fbaa5d5e705291a9016e5f8cb"},
-    {file = "cryptography-3.2.1-cp38-cp38-win32.whl", hash = "sha256:3cd75a683b15576cfc822c7c5742b3276e50b21a06672dc3a800a2d5da4ecd1b"},
-    {file = "cryptography-3.2.1-cp38-cp38-win_amd64.whl", hash = "sha256:d25cecbac20713a7c3bc544372d42d8eafa89799f492a43b79e1dfd650484851"},
-    {file = "cryptography-3.2.1.tar.gz", hash = "sha256:d3d5e10be0cf2a12214ddee45c6bd203dab435e3d83b4560c03066eda600bfe3"},
+    {file = "cryptography-3.4.6-cp36-abi3-macosx_10_10_x86_64.whl", hash = "sha256:57ad77d32917bc55299b16d3b996ffa42a1c73c6cfa829b14043c561288d2799"},
+    {file = "cryptography-3.4.6-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:4169a27b818de4a1860720108b55a2801f32b6ae79e7f99c00d79f2a2822eeb7"},
+    {file = "cryptography-3.4.6-cp36-abi3-manylinux2010_x86_64.whl", hash = "sha256:93cfe5b7ff006de13e1e89830810ecbd014791b042cbe5eec253be11ac2b28f3"},
+    {file = "cryptography-3.4.6-cp36-abi3-manylinux2014_aarch64.whl", hash = "sha256:5ecf2bcb34d17415e89b546dbb44e73080f747e504273e4d4987630493cded1b"},
+    {file = "cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl", hash = "sha256:fec7fb46b10da10d9e1d078d1ff8ed9e05ae14f431fdbd11145edd0550b9a964"},
+    {file = "cryptography-3.4.6-cp36-abi3-win32.whl", hash = "sha256:df186fcbf86dc1ce56305becb8434e4b6b7504bc724b71ad7a3239e0c9d14ef2"},
+    {file = "cryptography-3.4.6-cp36-abi3-win_amd64.whl", hash = "sha256:66b57a9ca4b3221d51b237094b0303843b914b7d5afd4349970bb26518e350b0"},
+    {file = "cryptography-3.4.6-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:066bc53f052dfeda2f2d7c195cf16fb3e5ff13e1b6b7415b468514b40b381a5b"},
+    {file = "cryptography-3.4.6-pp36-pypy36_pp73-manylinux2014_x86_64.whl", hash = "sha256:600cf9bfe75e96d965509a4c0b2b183f74a4fa6f5331dcb40fb7b77b7c2484df"},
+    {file = "cryptography-3.4.6-pp37-pypy37_pp73-manylinux2010_x86_64.whl", hash = "sha256:0923ba600d00718d63a3976f23cab19aef10c1765038945628cd9be047ad0336"},
+    {file = "cryptography-3.4.6-pp37-pypy37_pp73-manylinux2014_x86_64.whl", hash = "sha256:9e98b452132963678e3ac6c73f7010fe53adf72209a32854d55690acac3f6724"},
+    {file = "cryptography-3.4.6.tar.gz", hash = "sha256:2d32223e5b0ee02943f32b19245b61a62db83a882f0e76cc564e1cec60d48f87"},
 ]
 dataengineeringutils3 = [
-    {file = "dataengineeringutils3-1.1.0-py3-none-any.whl", hash = "sha256:96ad4c17988aa6c82208d4833a9887fd7a6c14f25ca0d9153e9d527052371b58"},
-    {file = "dataengineeringutils3-1.1.0.tar.gz", hash = "sha256:e2cd9e162724c4dc25efa85eccb49d79c0e74837c893606e3962f431bb546de7"},
+    {file = "dataengineeringutils3-1.1.3-py3-none-any.whl", hash = "sha256:14421b8084efbc5af3a09c6801f7195236791c8288a5b29e0340617072c9d594"},
+    {file = "dataengineeringutils3-1.1.3.tar.gz", hash = "sha256:a765d7251ee27b28ae99f9c567c1b0f5789f9cfeb1933af40df310616dc08adb"},
 ]
 decorator = [
     {file = "decorator-4.4.2-py2.py3-none-any.whl", hash = "sha256:41fa54c2a0cc4ba648be4fd43cff00aedf5b9465c9bf18d64325bc225f08f760"},
     {file = "decorator-4.4.2.tar.gz", hash = "sha256:e3a62f0520172440ca0dcc823749319382e377f37f140a0b99ef45fecb84bfe7"},
 ]
 defusedxml = [
-    {file = "defusedxml-0.6.0-py2.py3-none-any.whl", hash = "sha256:6687150770438374ab581bb7a1b327a847dd9c5749e396102de3fad4e8a3ef93"},
-    {file = "defusedxml-0.6.0.tar.gz", hash = "sha256:f684034d135af4c6cbb949b8a4d2ed61634515257a67299e5f940fbaa34377f5"},
+    {file = "defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61"},
+    {file = "defusedxml-0.7.1.tar.gz", hash = "sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69"},
 ]
 docker = [
-    {file = "docker-4.4.0-py2.py3-none-any.whl", hash = "sha256:317e95a48c32de8c1aac92a48066a5b73e218ed096e03758bcdd799a7130a1a1"},
-    {file = "docker-4.4.0.tar.gz", hash = "sha256:cffc771d4ea1389fc66bc95cb72d304aa41d1a1563482a9a000fba3a84ed5071"},
+    {file = "docker-4.2.2-py2.py3-none-any.whl", hash = "sha256:03a46400c4080cb6f7aa997f881ddd84fef855499ece219d75fbdb53289c17ab"},
+    {file = "docker-4.2.2.tar.gz", hash = "sha256:26eebadce7e298f55b76a88c4f8802476c5eaddbdbe38dbc6cce8781c47c9b54"},
 ]
 ecdsa = [
     {file = "ecdsa-0.14.1-py2.py3-none-any.whl", hash = "sha256:e108a5fe92c67639abae3260e43561af914e7fd0d27bae6d2ec1312ae7934dfe"},
@@ -1993,13 +2125,17 @@ et-xmlfile = [
     {file = "et_xmlfile-1.0.1.tar.gz", hash = "sha256:614d9722d572f6246302c4491846d2c393c199cfa4edc9af593437691683335b"},
 ]
 flake8 = [
-    {file = "flake8-3.8.4-py2.py3-none-any.whl", hash = "sha256:749dbbd6bfd0cf1318af27bf97a14e28e5ff548ef8e5b1566ccfb25a11e7c839"},
-    {file = "flake8-3.8.4.tar.gz", hash = "sha256:aadae8761ec651813c24be05c6f7b4680857ef6afaae4651a4eccaef97ce6c3b"},
+    {file = "flake8-3.9.0-py2.py3-none-any.whl", hash = "sha256:12d05ab02614b6aee8df7c36b97d1a3b2372761222b19b58621355e82acddcff"},
+    {file = "flake8-3.9.0.tar.gz", hash = "sha256:78873e372b12b093da7b5e5ed302e8ad9e988b38b063b61ad937f26ca58fc5f0"},
 ]
 frictionless = [
     {file = "frictionless-3.24.0-py2.py3-none-any.whl", hash = "sha256:1b66f20de3bcdf3ca0e89753edc76b1dcd4a198e0c8d3f8e5671c42c995ef64c"},
     {file = "frictionless-3.24.0.tar.gz", hash = "sha256:8b9b3228bf8142d6a5bc6ece93e2b4341072583e8164d4d020610b09f9e58345"},
 ]
+fsspec = [
+    {file = "fsspec-0.8.7-py3-none-any.whl", hash = "sha256:65dbf8244a3a3d23342109925f9f588c7551b2b01a5f47e555043b17e2b32d62"},
+    {file = "fsspec-0.8.7.tar.gz", hash = "sha256:4b11557a90ac637089b10afa4c77adf42080c0696f6f2771c41ce92d73c41432"},
+]
 future = [
     {file = "future-0.18.2.tar.gz", hash = "sha256:b1bead90b70cf6ec3f0710ae53a525360fa360d306a86583adc6bf83a4db537d"},
 ]
@@ -2016,103 +2152,99 @@ idna = [
     {file = "idna-2.10.tar.gz", hash = "sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6"},
 ]
 ijson = [
-    {file = "ijson-3.1.3-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:e9ab99967f54925d72f26ec332525d1270cb8b6027bd3a54475e6bb862c1ca2c"},
-    {file = "ijson-3.1.3-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:f9904ca4cee0ea52f4201accecd97a0208ece1508d00965ce1c669822caf1f6c"},
-    {file = "ijson-3.1.3-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:6acd0f5794cdd16d99c8df43f9b46052bf7c9d14c330f66aa41bb124130f24ba"},
-    {file = "ijson-3.1.3-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:0c520d462dd2b6de57354bfd6cfe378d11218e8a67d87cff2b1ffe46e421918b"},
-    {file = "ijson-3.1.3-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:86bef1aa97f1e2cf779dfdcdd1215d002c2eaf6fed56bbdaefc07d775c3f38a6"},
-    {file = "ijson-3.1.3-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:c654fefd2dc92b8f2c423a905b5b3d8762e09d2e3719ea94f139915fe13cb1b3"},
-    {file = "ijson-3.1.3-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:2f139f34d7d515ce0a1a79bc69975fee7d6cb3507ad2635b904afa0227a99c0d"},
-    {file = "ijson-3.1.3-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:dffa5bc8372845fcec2a6aa02cac15793f99c7fb37f1991bdd875e3695743a2f"},
-    {file = "ijson-3.1.3-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:479a234ec969d3293c2a9019d3cb6e8bbe307c9dcf83ce3602d85faba4ac7f62"},
-    {file = "ijson-3.1.3-cp35-cp35m-macosx_10_9_x86_64.whl", hash = "sha256:8acb45ccc58a19a1a6491b06c1031b1b620d7e3eb19025741b34eb453d8b8684"},
-    {file = "ijson-3.1.3-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:4f1055660065cf258ac0f64bd5a1371f3c052d812adfe996306bb56167868d1e"},
-    {file = "ijson-3.1.3-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:605b4a113281715a04230c404218a8513f87ad29fdf1c63176134dfd9cf46625"},
-    {file = "ijson-3.1.3-cp35-cp35m-manylinux2010_i686.whl", hash = "sha256:620188455c4baa15d6e61551321b483eff416473a29b55c6afd2640c08c8eadf"},
-    {file = "ijson-3.1.3-cp35-cp35m-manylinux2010_x86_64.whl", hash = "sha256:9d1839c0d1dd3cc8e3403260db891b60550fd9a03dd34b8e01c03ea79b972311"},
-    {file = "ijson-3.1.3-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:c4f592b4b715d0ea1b1307a8109404759ea5f73ae99ebdf5e000b8392dc9a888"},
-    {file = "ijson-3.1.3-cp35-cp35m-win32.whl", hash = "sha256:49847a440319a1adb0b1571c9545b29a7b83101c616be1afe50a976be028a701"},
-    {file = "ijson-3.1.3-cp35-cp35m-win_amd64.whl", hash = "sha256:756b615b5796c9c35148905d91fbf81999826509c437ba356bcf766e1a4992dd"},
-    {file = "ijson-3.1.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:985c8328d10b642de34fdb664cbfb920786220889dc8ba7b47c4b4d94804ebe0"},
-    {file = "ijson-3.1.3-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:aa01cb1c38b2875deca56a5116b91331c3ab24e2ffbc1e6e9c29e4922d469e8f"},
-    {file = "ijson-3.1.3-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:30011737ccb176886deda8a42fe90664e19411c6e60c6caa020c39042a0fee68"},
-    {file = "ijson-3.1.3-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:0ab2713f85c0cb150a4e15262e8574090c48f27dc99a2bd9db8a98698ad9110f"},
-    {file = "ijson-3.1.3-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:78dfa1cae1d68a7dcf834f8d25005be0ac8c9952cc7ae3808de123154c2904da"},
-    {file = "ijson-3.1.3-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:198a8ccd80dd3c45278692d64a8aab8f51df4297767d50839ab7a89af50c439e"},
-    {file = "ijson-3.1.3-cp36-cp36m-win32.whl", hash = "sha256:c2fd09ad20579699b4faacbdf9c51e049013ed1a3833d8223ea51829205debda"},
-    {file = "ijson-3.1.3-cp36-cp36m-win_amd64.whl", hash = "sha256:9a3494b1ee1b037d7cd874d9f3aafc685d5421e23ffc7c8ea3863b7b13c5f4e9"},
-    {file = "ijson-3.1.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:4efe4b91ddcfbd14792c73a3075bbe59fd834d57261f48b8be0e47f7891bbb97"},
-    {file = "ijson-3.1.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:e5f7d08df0968b0e42e2a741413af9b19f3f206484c9be0d01ebca632345fbbd"},
-    {file = "ijson-3.1.3-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:9e599f37422d5e4b4314eccb4880ab9dfa2669559ac6ac7da3a014f522fa84cc"},
-    {file = "ijson-3.1.3-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:388e7072f7c6bd1dd15551280d823232ba322f41d8c5c4165d890647aad77e89"},
-    {file = "ijson-3.1.3-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:e77d3f402e7aadd0decd0b7dc405f554050e230538c82d08d574ea132caeee47"},
-    {file = "ijson-3.1.3-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:4ba5fdad0a484294ffa3572c1fa0db42d876e6db6c292ceb6650ee823dc2854c"},
-    {file = "ijson-3.1.3-cp37-cp37m-win32.whl", hash = "sha256:03bb436afc44ec5db996d62019a0ed71c35b7d8d7cd730ac9606ff0b1ad78b95"},
-    {file = "ijson-3.1.3-cp37-cp37m-win_amd64.whl", hash = "sha256:4673586bfee9811a9f4a03ba13c4ce0eebdeb63da49176b40bf161f0a11500a3"},
-    {file = "ijson-3.1.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:c1da1662ef35421c74ef162aeadfc9320788eafef8f077b267236373d215525a"},
-    {file = "ijson-3.1.3-cp38-cp38-manylinux1_i686.whl", hash = "sha256:4c2c71dd8e347dd2094d4e8a9dc3b961af533278e337d4a0f50a139234d69d3c"},
-    {file = "ijson-3.1.3-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:7a76257a732ed9f0933918b8a327419a8b15978f71d5dde9112ba32d1bf1b671"},
-    {file = "ijson-3.1.3-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:250df6183298e00c5201d103c5b83d23472f7db47d3dc518d59f198b42ab9583"},
-    {file = "ijson-3.1.3-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:5baf497f74d6c9f6581150911560f7d608302ad5036375b2685e35b96d499b81"},
-    {file = "ijson-3.1.3-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:bb94b124dbe6a77618c02367ee8d0542c3bcc50d4360f45d7e060da980887000"},
-    {file = "ijson-3.1.3-cp38-cp38-win32.whl", hash = "sha256:484a4a117a9d4c16b4a60286fa1bb787d757d2ade6a398b1d3a669ba6376ac2d"},
-    {file = "ijson-3.1.3-cp38-cp38-win_amd64.whl", hash = "sha256:03329c9077ff4ee67047f8a5b02591ad638ec2a249f50a7e294a8890484f7141"},
-    {file = "ijson-3.1.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:479ed89d1c42fd45ffb9665e32047e30915ff06c78dacbae3abd708085849905"},
-    {file = "ijson-3.1.3-cp39-cp39-manylinux1_i686.whl", hash = "sha256:6490b1cd21ab42850a9f0322dd2b74a04742b7914429c7a3d648cb7e37711922"},
-    {file = "ijson-3.1.3-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:d2621331ec2e45d61c64d8ea956b91fc49ec60c537f23f9a675341b403fca636"},
-    {file = "ijson-3.1.3-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:779402d0ece200c988ae4d13c675e969cdef2202071ba6e019d1cd3365e82f60"},
-    {file = "ijson-3.1.3-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:a3751348538e554a7610dd6935a84951b60a2157190c58e1419d5a66f3584ab4"},
-    {file = "ijson-3.1.3-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:eee5210682138f6110d0afcae1668e4d5405a2a6f1cf4c1a44da37fd6dcf15fc"},
-    {file = "ijson-3.1.3-cp39-cp39-win32.whl", hash = "sha256:f77b58f6492633f0d2a6ed0ac298b0888ec3e7896459b46cf185a89f6c77caa3"},
-    {file = "ijson-3.1.3-cp39-cp39-win_amd64.whl", hash = "sha256:abc0069c16f484165b2d57429dff80e62844c29313248766fd26d0bb72595c6b"},
-    {file = "ijson-3.1.3-pp27-pypy_73-macosx_10_9_x86_64.whl", hash = "sha256:db2a2a0e07cf1303582ed85c4462e5dac44a7deeea397060ac90e5c100b6f007"},
-    {file = "ijson-3.1.3-pp27-pypy_73-manylinux1_x86_64.whl", hash = "sha256:4df28537f95d183ad73938430c8df1ed0949bf2620e36909f1ed571a66ee4e99"},
-    {file = "ijson-3.1.3-pp27-pypy_73-manylinux2010_x86_64.whl", hash = "sha256:b3d06b4aa4abbf1b7dc6de1555fcfdc84ed0ea7bbbc1054cdcad76750538d4f7"},
-    {file = "ijson-3.1.3-pp36-pypy36_pp73-macosx_10_9_x86_64.whl", hash = "sha256:c438a833ea4db15b29a897181809febbee1e6853e25fdb757f109757c293b984"},
-    {file = "ijson-3.1.3-pp36-pypy36_pp73-manylinux1_x86_64.whl", hash = "sha256:7d034a6b756013ad5f55f62f6e4bf9e3bfa1abda55bc0f7b0056fb6001167bb0"},
-    {file = "ijson-3.1.3-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:55052db59a338071db7d27f5f4658b0c57d4c925fe3da2f50c9eb0cc52e719ac"},
-    {file = "ijson-3.1.3-pp36-pypy36_pp73-win32.whl", hash = "sha256:f3a0f9738a05eb317a5c96719dfdfa458ecd9af9129b39d470e598fa2ecc6b90"},
-    {file = "ijson-3.1.3-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:ad0336810a3a47316f0cd423a2e625c52c034c03b01ee4daa6490f619eada39d"},
-    {file = "ijson-3.1.3-pp37-pypy37_pp73-manylinux1_x86_64.whl", hash = "sha256:281492f19ace95b8e57d7d418dff128d4cd7bc2615c6a51324c8a051df8f867b"},
-    {file = "ijson-3.1.3-pp37-pypy37_pp73-manylinux2010_x86_64.whl", hash = "sha256:28eb24e3b6c4e091f794fbcb63ca84e2e982ec704e76bb5f9de11986bc49aadb"},
-    {file = "ijson-3.1.3-pp37-pypy37_pp73-win32.whl", hash = "sha256:4bc833a02d619b3f2c481270edea480bb4f0983e2218be1818a69dc39a3725aa"},
-    {file = "ijson-3.1.3.tar.gz", hash = "sha256:d29977f7235b5bf83c372825c6abd8640ba0e3a8e031d3ffc3b63deaf6ae1487"},
+    {file = "ijson-3.1.4-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:6c1a777096be5f75ffebb335c6d2ebc0e489b231496b7f2ca903aa061fe7d381"},
+    {file = "ijson-3.1.4-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:475fc25c3d2a86230b85777cae9580398b42eed422506bf0b6aacfa936f7bfcd"},
+    {file = "ijson-3.1.4-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:f587699b5a759e30accf733e37950cc06c4118b72e3e146edcea77dded467426"},
+    {file = "ijson-3.1.4-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:339b2b4c7bbd64849dd69ef94ee21e29dcd92c831f47a281fdd48122bb2a715a"},
+    {file = "ijson-3.1.4-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:446ef8980504da0af8d20d3cb6452c4dc3d8aa5fd788098985e899b913191fe6"},
+    {file = "ijson-3.1.4-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:3997a2fdb28bc04b9ab0555db5f3b33ed28d91e9d42a3bf2c1842d4990beb158"},
+    {file = "ijson-3.1.4-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:fa10a1d88473303ec97aae23169d77c5b92657b7fb189f9c584974c00a79f383"},
+    {file = "ijson-3.1.4-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:9a5bf5b9d8f2ceaca131ee21fc7875d0f34b95762f4f32e4d65109ca46472147"},
+    {file = "ijson-3.1.4-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:81cc8cee590c8a70cca3c9aefae06dd7cb8e9f75f3a7dc12b340c2e332d33a2a"},
+    {file = "ijson-3.1.4-cp35-cp35m-macosx_10_9_x86_64.whl", hash = "sha256:3b98861a4280cf09d267986cefa46c3bd80af887eae02aba07488d80eb798afa"},
+    {file = "ijson-3.1.4-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:068c692efba9692406b86736dcc6803e4a0b6280d7f0b7534bff3faec677ff38"},
+    {file = "ijson-3.1.4-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:86884ac06ac69cea6d89ab7b84683b3b4159c4013e4a20276d3fc630fe9b7588"},
+    {file = "ijson-3.1.4-cp35-cp35m-manylinux2010_i686.whl", hash = "sha256:41e5886ff6fade26f10b87edad723d2db14dcbb1178717790993fcbbb8ccd333"},
+    {file = "ijson-3.1.4-cp35-cp35m-manylinux2010_x86_64.whl", hash = "sha256:24b58933bf777d03dc1caa3006112ec7f9e6f6db6ffe1f5f5bd233cb1281f719"},
+    {file = "ijson-3.1.4-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:13f80aad0b84d100fb6a88ced24bade21dc6ddeaf2bba3294b58728463194f50"},
+    {file = "ijson-3.1.4-cp35-cp35m-win32.whl", hash = "sha256:fa9a25d0bd32f9515e18a3611690f1de12cb7d1320bd93e9da835936b41ad3ff"},
+    {file = "ijson-3.1.4-cp35-cp35m-win_amd64.whl", hash = "sha256:c4c1bf98aaab4c8f60d238edf9bcd07c896cfcc51c2ca84d03da22aad88957c5"},
+    {file = "ijson-3.1.4-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:f0f2a87c423e8767368aa055310024fa28727f4454463714fef22230c9717f64"},
+    {file = "ijson-3.1.4-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:15507de59d74d21501b2a076d9c49abf927eb58a51a01b8f28a0a0565db0a99f"},
+    {file = "ijson-3.1.4-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:2e6bd6ad95ab40c858592b905e2bbb4fe79bbff415b69a4923dafe841ffadcb4"},
+    {file = "ijson-3.1.4-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:68e295bb12610d086990cedc89fb8b59b7c85740d66e9515aed062649605d0bf"},
+    {file = "ijson-3.1.4-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:3bb461352c0f0f2ec460a4b19400a665b8a5a3a2da663a32093df1699642ee3f"},
+    {file = "ijson-3.1.4-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:f91c75edd6cf1a66f02425bafc59a22ec29bc0adcbc06f4bfd694d92f424ceb3"},
+    {file = "ijson-3.1.4-cp36-cp36m-win32.whl", hash = "sha256:4c53cc72f79a4c32d5fc22efb85aa22f248e8f4f992707a84bdc896cc0b1ecf9"},
+    {file = "ijson-3.1.4-cp36-cp36m-win_amd64.whl", hash = "sha256:ac9098470c1ff6e5c23ec0946818bc102bfeeeea474554c8d081dc934be20988"},
+    {file = "ijson-3.1.4-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:dcd6f04df44b1945b859318010234651317db2c4232f75e3933f8bb41c4fa055"},
+    {file = "ijson-3.1.4-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:5a2f40c053c837591636dc1afb79d85e90b9a9d65f3d9963aae31d1eb11bfed2"},
+    {file = "ijson-3.1.4-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:f50337e3b8e72ec68441b573c2848f108a8976a57465c859b227ebd2a2342901"},
+    {file = "ijson-3.1.4-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:454918f908abbed3c50a0a05c14b20658ab711b155e4f890900e6f60746dd7cc"},
+    {file = "ijson-3.1.4-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:387c2ec434cc1bc7dc9bd33ec0b70d95d443cc1e5934005f26addc2284a437ab"},
+    {file = "ijson-3.1.4-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:179ed6fd42e121d252b43a18833df2de08378fac7bce380974ef6f5e522afefa"},
+    {file = "ijson-3.1.4-cp37-cp37m-win32.whl", hash = "sha256:26a6a550b270df04e3f442e2bf0870c9362db4912f0e7bdfd300f30ea43115a2"},
+    {file = "ijson-3.1.4-cp37-cp37m-win_amd64.whl", hash = "sha256:ff8cf7507d9d8939264068c2cff0a23f99703fa2f31eb3cb45a9a52798843586"},
+    {file = "ijson-3.1.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:09c9d7913c88a6059cd054ff854958f34d757402b639cf212ffbec201a705a0d"},
+    {file = "ijson-3.1.4-cp38-cp38-manylinux1_i686.whl", hash = "sha256:702ba9a732116d659a5e950ee176be6a2e075998ef1bcde11cbf79a77ed0f717"},
+    {file = "ijson-3.1.4-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:667841591521158770adc90793c2bdbb47c94fe28888cb802104b8bbd61f3d51"},
+    {file = "ijson-3.1.4-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:df641dd07b38c63eecd4f454db7b27aa5201193df160f06b48111ba97ab62504"},
+    {file = "ijson-3.1.4-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:9348e7d507eb40b52b12eecff3d50934fcc3d2a15a2f54ec1127a36063b9ba8f"},
+    {file = "ijson-3.1.4-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:93455902fdc33ba9485c7fae63ac95d96e0ab8942224a357113174bbeaff92e9"},
+    {file = "ijson-3.1.4-cp38-cp38-win32.whl", hash = "sha256:5b725f2e984ce70d464b195f206fa44bebbd744da24139b61fec72de77c03a16"},
+    {file = "ijson-3.1.4-cp38-cp38-win_amd64.whl", hash = "sha256:a5965c315fbb2dc9769dfdf046eb07daf48ae20b637da95ec8d62b629be09df4"},
+    {file = "ijson-3.1.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:b8ee7dbb07cec9ba29d60cfe4954b3cc70adb5f85bba1f72225364b59c1cf82b"},
+    {file = "ijson-3.1.4-cp39-cp39-manylinux1_i686.whl", hash = "sha256:d9e01c55d501e9c3d686b6ee3af351c9c0c8c3e45c5576bd5601bee3e1300b09"},
+    {file = "ijson-3.1.4-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:297f26f27a04cd0d0a2f865d154090c48ea11b239cabe0a17a6c65f0314bd1ca"},
+    {file = "ijson-3.1.4-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:9239973100338a4138d09d7a4602bd289861e553d597cd67390c33bfc452253e"},
+    {file = "ijson-3.1.4-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:2a64c66a08f56ed45a805691c2fd2e1caef00edd6ccf4c4e5eff02cd94ad8364"},
+    {file = "ijson-3.1.4-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:d17fd199f0d0a4ab6e0d541b4eec1b68b5bd5bb5d8104521e22243015b51049b"},
+    {file = "ijson-3.1.4-cp39-cp39-win32.whl", hash = "sha256:70ee3c8fa0eba18c80c5911639c01a8de4089a4361bad2862a9949e25ec9b1c8"},
+    {file = "ijson-3.1.4-cp39-cp39-win_amd64.whl", hash = "sha256:6bf2b64304321705d03fa5e403ec3f36fa5bb27bf661849ad62e0a3a49bc23e3"},
+    {file = "ijson-3.1.4-pp27-pypy_73-macosx_10_9_x86_64.whl", hash = "sha256:5d7e3fcc3b6de76a9dba1e9fc6ca23dad18f0fa6b4e6499415e16b684b2e9af1"},
+    {file = "ijson-3.1.4-pp27-pypy_73-manylinux1_x86_64.whl", hash = "sha256:a72eb0359ebff94754f7a2f00a6efe4c57716f860fc040c606dedcb40f49f233"},
+    {file = "ijson-3.1.4-pp27-pypy_73-manylinux2010_x86_64.whl", hash = "sha256:28fc168f5faf5759fdfa2a63f85f1f7a148bbae98f34404a6ba19f3d08e89e87"},
+    {file = "ijson-3.1.4-pp36-pypy36_pp73-macosx_10_9_x86_64.whl", hash = "sha256:2844d4a38d27583897ed73f7946e205b16926b4cab2525d1ce17e8b08064c706"},
+    {file = "ijson-3.1.4-pp36-pypy36_pp73-manylinux1_x86_64.whl", hash = "sha256:252defd1f139b5fb8c764d78d5e3a6df81543d9878c58992a89b261369ea97a7"},
+    {file = "ijson-3.1.4-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:15d5356b4d090c699f382c8eb6a2bcd5992a8c8e8b88c88bc6e54f686018328a"},
+    {file = "ijson-3.1.4-pp36-pypy36_pp73-win32.whl", hash = "sha256:6774ec0a39647eea70d35fb76accabe3d71002a8701c0545b9120230c182b75b"},
+    {file = "ijson-3.1.4-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:f11da15ec04cc83ff0f817a65a3392e169be8d111ba81f24d6e09236597bb28c"},
+    {file = "ijson-3.1.4-pp37-pypy37_pp73-manylinux1_x86_64.whl", hash = "sha256:ee13ceeed9b6cf81b3b8197ef15595fc43fd54276842ed63840ddd49db0603da"},
+    {file = "ijson-3.1.4-pp37-pypy37_pp73-manylinux2010_x86_64.whl", hash = "sha256:97e4df67235fae40d6195711223520d2c5bf1f7f5087c2963fcde44d72ebf448"},
+    {file = "ijson-3.1.4-pp37-pypy37_pp73-win32.whl", hash = "sha256:3d10eee52428f43f7da28763bb79f3d90bbbeea1accb15de01e40a00885b6e89"},
+    {file = "ijson-3.1.4.tar.gz", hash = "sha256:1d1003ae3c6115ec9b587d29dd136860a81a23c7626b682e2b5b12c9fd30e4ea"},
 ]
 importlib-metadata = [
-    {file = "importlib_metadata-1.7.0-py2.py3-none-any.whl", hash = "sha256:dc15b2969b4ce36305c51eebe62d418ac7791e9a157911d58bfb1f9ccd8e2070"},
-    {file = "importlib_metadata-1.7.0.tar.gz", hash = "sha256:90bb658cdbbf6d1735b6341ce708fc7024a3e14e99ffdc5783edea9f9b077f83"},
+    {file = "importlib_metadata-3.7.3-py3-none-any.whl", hash = "sha256:b74159469b464a99cb8cc3e21973e4d96e05d3024d337313fedb618a6e86e6f4"},
+    {file = "importlib_metadata-3.7.3.tar.gz", hash = "sha256:742add720a20d0467df2f444ae41704000f50e1234f46174b51f9c6031a1bd71"},
 ]
 ipykernel = [
-    {file = "ipykernel-5.3.4-py3-none-any.whl", hash = "sha256:d6fbba26dba3cebd411382bc484f7bc2caa98427ae0ddb4ab37fe8bfeb5c7dd3"},
-    {file = "ipykernel-5.3.4.tar.gz", hash = "sha256:9b2652af1607986a1b231c62302d070bc0534f564c393a5d9d130db9abbbe89d"},
+    {file = "ipykernel-5.5.0-py3-none-any.whl", hash = "sha256:efd07253b54d84d26e0878d268c8c3a41582a18750da633c2febfd2ece0d467d"},
+    {file = "ipykernel-5.5.0.tar.gz", hash = "sha256:98321abefdf0505fb3dc7601f60fc4087364d394bd8fad53107eb1adee9ff475"},
 ]
 ipython = [
-    {file = "ipython-7.19.0-py3-none-any.whl", hash = "sha256:c987e8178ced651532b3b1ff9965925bfd445c279239697052561a9ab806d28f"},
-    {file = "ipython-7.19.0.tar.gz", hash = "sha256:cbb2ef3d5961d44e6a963b9817d4ea4e1fa2eb589c371a470fed14d8d40cbd6a"},
+    {file = "ipython-7.21.0-py3-none-any.whl", hash = "sha256:34207ffb2f653bced2bc8e3756c1db86e7d93e44ed049daae9814fed66d408ec"},
+    {file = "ipython-7.21.0.tar.gz", hash = "sha256:04323f72d5b85b606330b6d7e2dc8d2683ad46c3905e955aa96ecc7a99388e70"},
 ]
 ipython-genutils = [
     {file = "ipython_genutils-0.2.0-py2.py3-none-any.whl", hash = "sha256:72dd37233799e619666c9f639a9da83c34013a73e8bbc79a7a6348d93c61fab8"},
     {file = "ipython_genutils-0.2.0.tar.gz", hash = "sha256:eb2e116e75ecef9d4d228fdc66af54269afa26ab4463042e33785b887c628ba8"},
 ]
 ipywidgets = [
-    {file = "ipywidgets-7.5.1-py2.py3-none-any.whl", hash = "sha256:13ffeca438e0c0f91ae583dc22f50379b9d6b28390ac7be8b757140e9a771516"},
-    {file = "ipywidgets-7.5.1.tar.gz", hash = "sha256:e945f6e02854a74994c596d9db83444a1850c01648f1574adf144fbbabe05c97"},
+    {file = "ipywidgets-7.6.3-py2.py3-none-any.whl", hash = "sha256:e6513cfdaf5878de30f32d57f6dc2474da395a2a2991b94d487406c0ab7f55ca"},
+    {file = "ipywidgets-7.6.3.tar.gz", hash = "sha256:9f1a43e620530f9e570e4a493677d25f08310118d315b00e25a18f12913c41f0"},
 ]
 isodate = [
     {file = "isodate-0.6.0-py2.py3-none-any.whl", hash = "sha256:aa4d33c06640f5352aca96e4b81afd8ab3b47337cc12089822d6f322ac772c81"},
     {file = "isodate-0.6.0.tar.gz", hash = "sha256:2e364a3d5759479cdb2d37cce6b9376ea504db2ff90252a2e5b7cc89cc9ff2d8"},
 ]
-jdcal = [
-    {file = "jdcal-1.4.1-py2.py3-none-any.whl", hash = "sha256:1abf1305fce18b4e8aa248cf8fe0c56ce2032392bc64bbd61b5dff2a19ec8bba"},
-    {file = "jdcal-1.4.1.tar.gz", hash = "sha256:472872e096eb8df219c23f2689fc336668bdb43d194094b5cc1707e1640acfc8"},
-]
 jedi = [
-    {file = "jedi-0.17.2-py2.py3-none-any.whl", hash = "sha256:98cc583fa0f2f8304968199b01b6b4b94f469a1f4a74c1560506ca2a211378b5"},
-    {file = "jedi-0.17.2.tar.gz", hash = "sha256:86ed7d9b750603e4ba582ea8edc678657fb4007894a12bcf6f4bb97892f31d20"},
+    {file = "jedi-0.18.0-py2.py3-none-any.whl", hash = "sha256:18456d83f65f400ab0c2d3319e48520420ef43b23a086fdc05dff34132f0fb93"},
+    {file = "jedi-0.18.0.tar.gz", hash = "sha256:92550a404bad8afed881a137ec9a461fed49eca661414be45059329614ed0707"},
 ]
 jinja2 = [
-    {file = "Jinja2-2.11.2-py2.py3-none-any.whl", hash = "sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035"},
-    {file = "Jinja2-2.11.2.tar.gz", hash = "sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0"},
+    {file = "Jinja2-2.11.3-py2.py3-none-any.whl", hash = "sha256:03e47ad063331dd6a3f04a43eddca8a966a26ba0c5b7207a9a9e4e08f1b29419"},
+    {file = "Jinja2-2.11.3.tar.gz", hash = "sha256:a6d58433de0ae800347cab1fa3043cebbabe8baa9d29e668f1c768cb87a333c6"},
 ]
 jmespath = [
     {file = "jmespath-0.10.0-py2.py3-none-any.whl", hash = "sha256:cdf6525904cc597730141d61b36f2e4b8ecc257c420fa2f4549bac2c2d0cb72f"},
@@ -2126,20 +2258,20 @@ jsondiff = [
     {file = "jsondiff-1.2.0.tar.gz", hash = "sha256:34941bc431d10aa15828afe1cbb644977a114e75eef6cc74fb58951312326303"},
 ]
 jsonlines = [
-    {file = "jsonlines-1.2.0-py2.py3-none-any.whl", hash = "sha256:0ebd5b0c3efe0d4b5018b320fb0ee1a7b680ab39f6eb853715859f818d386cc8"},
-    {file = "jsonlines-1.2.0.tar.gz", hash = "sha256:43b8d5588a9d4862c8a4a49580e38e20ec595aee7ad6fe469b10fb83fbefde88"},
+    {file = "jsonlines-2.0.0-py3-none-any.whl", hash = "sha256:bfb043d4e25fd894dca67b1f2adf014e493cb65d0f18b3a74a98bfcd97c3d983"},
+    {file = "jsonlines-2.0.0.tar.gz", hash = "sha256:6fdd03104c9a421a1ba587a121aaac743bf02d8f87fa9cdaa3b852249a241fe8"},
 ]
 jsonpatch = [
-    {file = "jsonpatch-1.27-py2.py3-none-any.whl", hash = "sha256:86eaaccfac8891c6a09ab48aec13fb5921ff32c838e7a19981f0d2b94915f668"},
-    {file = "jsonpatch-1.27.tar.gz", hash = "sha256:4d08af10d71723b5b2924da6ba90f273a4d1a5c6accfb605eb970cb2f9b29cf9"},
+    {file = "jsonpatch-1.32-py2.py3-none-any.whl", hash = "sha256:26ac385719ac9f54df8a2f0827bb8253aa3ea8ab7b3368457bcdb8c14595a397"},
+    {file = "jsonpatch-1.32.tar.gz", hash = "sha256:b6ddfe6c3db30d81a96aaeceb6baf916094ffa23d7dd5fa2c13e13f8b6e600c2"},
 ]
 jsonpickle = [
-    {file = "jsonpickle-1.4.1-py2.py3-none-any.whl", hash = "sha256:8919c166bac0574e3d74425c7559434062002d9dfc0ac2afa6dc746ba4a19439"},
-    {file = "jsonpickle-1.4.1.tar.gz", hash = "sha256:e8d4b7cd0bd6826001a74377df1079a76ad8bae0f909282de2554164c837c8ba"},
+    {file = "jsonpickle-2.0.0-py2.py3-none-any.whl", hash = "sha256:c1010994c1fbda87a48f8a56698605b598cb0fc6bb7e7927559fc1100e69aeac"},
+    {file = "jsonpickle-2.0.0.tar.gz", hash = "sha256:0be49cba80ea6f87a168aa8168d717d00c6ca07ba83df3cec32d3b30bfe6fb9a"},
 ]
 jsonpointer = [
-    {file = "jsonpointer-2.0-py2.py3-none-any.whl", hash = "sha256:ff379fa021d1b81ab539f5ec467c7745beb1a5671463f9dcc2b2d458bd361c1e"},
-    {file = "jsonpointer-2.0.tar.gz", hash = "sha256:c192ba86648e05fdae4f08a17ec25180a9aef5008d973407b581798a83975362"},
+    {file = "jsonpointer-2.1-py2.py3-none-any.whl", hash = "sha256:150f80c5badd02c757da6644852f612f88e8b4bc2f9852dcbf557c8738919686"},
+    {file = "jsonpointer-2.1.tar.gz", hash = "sha256:5a34b698db1eb79ceac454159d3f7c12a451a91f6334a4f638454327b7a89962"},
 ]
 jsonschema = [
     {file = "jsonschema-3.2.0-py2.py3-none-any.whl", hash = "sha256:4e5b3cf8216f577bee9ce139cbe72eca3ea4f292ec60928ff24758ce626cd163"},
@@ -2149,12 +2281,12 @@ junit-xml = [
     {file = "junit_xml-1.9-py2.py3-none-any.whl", hash = "sha256:ec5ca1a55aefdd76d28fcc0b135251d156c7106fa979686a4b48d62b761b4732"},
 ]
 jupyter-client = [
-    {file = "jupyter_client-6.1.7-py3-none-any.whl", hash = "sha256:c958d24d6eacb975c1acebb68ac9077da61b5f5c040f22f6849928ad7393b950"},
-    {file = "jupyter_client-6.1.7.tar.gz", hash = "sha256:49e390b36fe4b4226724704ea28d9fb903f1a3601b6882ce3105221cd09377a1"},
+    {file = "jupyter_client-6.1.12-py3-none-any.whl", hash = "sha256:e053a2c44b6fa597feebe2b3ecb5eea3e03d1d91cc94351a52931ee1426aecfc"},
+    {file = "jupyter_client-6.1.12.tar.gz", hash = "sha256:c4bca1d0846186ca8be97f4d2fa6d2bae889cce4892a167ffa1ba6bd1f73e782"},
 ]
 jupyter-core = [
-    {file = "jupyter_core-4.7.0-py3-none-any.whl", hash = "sha256:0a451c9b295e4db772bdd8d06f2f1eb31caeec0e81fbb77ba37d4a3024e3b315"},
-    {file = "jupyter_core-4.7.0.tar.gz", hash = "sha256:aa1f9496ab3abe72da4efe0daab0cb2233997914581f9a071e07498c6add8ed3"},
+    {file = "jupyter_core-4.7.1-py3-none-any.whl", hash = "sha256:8c6c0cac5c1b563622ad49321d5ec47017bd18b94facb381c6973a0486395f8e"},
+    {file = "jupyter_core-4.7.1.tar.gz", hash = "sha256:79025cb3225efcd36847d0840f3fc672c0abd7afd0de83ba8a1d3837619122b4"},
 ]
 jupyterlab = [
     {file = "jupyterlab-2.2.9-py3-none-any.whl", hash = "sha256:59af02c26a15ec2d2862a15bc72e41ae304b406a0b0d3f4f705eeb7caf91902b"},
@@ -2168,6 +2300,49 @@ jupyterlab-server = [
     {file = "jupyterlab_server-1.2.0-py3-none-any.whl", hash = "sha256:55d256077bf13e5bc9e8fbd5aac51bef82f6315111cec6b712b9a5ededbba924"},
     {file = "jupyterlab_server-1.2.0.tar.gz", hash = "sha256:5431d9dde96659364b7cc877693d5d21e7b80cea7ae3959ecc2b87518e5f5d8c"},
 ]
+jupyterlab-widgets = [
+    {file = "jupyterlab_widgets-1.0.0-py3-none-any.whl", hash = "sha256:caeaf3e6103180e654e7d8d2b81b7d645e59e432487c1d35a41d6d3ee56b3fef"},
+    {file = "jupyterlab_widgets-1.0.0.tar.gz", hash = "sha256:5c1a29a84d3069208cb506b10609175b249b6486d6b1cbae8fcde2a11584fb78"},
+]
+lxml = [
+    {file = "lxml-4.6.2-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:a9d6bc8642e2c67db33f1247a77c53476f3a166e09067c0474facb045756087f"},
+    {file = "lxml-4.6.2-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:791394449e98243839fa822a637177dd42a95f4883ad3dec2a0ce6ac99fb0a9d"},
+    {file = "lxml-4.6.2-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:68a5d77e440df94011214b7db907ec8f19e439507a70c958f750c18d88f995d2"},
+    {file = "lxml-4.6.2-cp27-cp27m-win32.whl", hash = "sha256:fc37870d6716b137e80d19241d0e2cff7a7643b925dfa49b4c8ebd1295eb506e"},
+    {file = "lxml-4.6.2-cp27-cp27m-win_amd64.whl", hash = "sha256:69a63f83e88138ab7642d8f61418cf3180a4d8cd13995df87725cb8b893e950e"},
+    {file = "lxml-4.6.2-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:42ebca24ba2a21065fb546f3e6bd0c58c3fe9ac298f3a320147029a4850f51a2"},
+    {file = "lxml-4.6.2-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:f83d281bb2a6217cd806f4cf0ddded436790e66f393e124dfe9731f6b3fb9afe"},
+    {file = "lxml-4.6.2-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:535f067002b0fd1a4e5296a8f1bf88193080ff992a195e66964ef2a6cfec5388"},
+    {file = "lxml-4.6.2-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:366cb750140f221523fa062d641393092813b81e15d0e25d9f7c6025f910ee80"},
+    {file = "lxml-4.6.2-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:97db258793d193c7b62d4e2586c6ed98d51086e93f9a3af2b2034af01450a74b"},
+    {file = "lxml-4.6.2-cp35-cp35m-win32.whl", hash = "sha256:648914abafe67f11be7d93c1a546068f8eff3c5fa938e1f94509e4a5d682b2d8"},
+    {file = "lxml-4.6.2-cp35-cp35m-win_amd64.whl", hash = "sha256:4e751e77006da34643ab782e4a5cc21ea7b755551db202bc4d3a423b307db780"},
+    {file = "lxml-4.6.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:681d75e1a38a69f1e64ab82fe4b1ed3fd758717bed735fb9aeaa124143f051af"},
+    {file = "lxml-4.6.2-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:127f76864468d6630e1b453d3ffbbd04b024c674f55cf0a30dc2595137892d37"},
+    {file = "lxml-4.6.2-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:4fb85c447e288df535b17ebdebf0ec1cf3a3f1a8eba7e79169f4f37af43c6b98"},
+    {file = "lxml-4.6.2-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:5be4a2e212bb6aa045e37f7d48e3e1e4b6fd259882ed5a00786f82e8c37ce77d"},
+    {file = "lxml-4.6.2-cp36-cp36m-win32.whl", hash = "sha256:8c88b599e226994ad4db29d93bc149aa1aff3dc3a4355dd5757569ba78632bdf"},
+    {file = "lxml-4.6.2-cp36-cp36m-win_amd64.whl", hash = "sha256:6e4183800f16f3679076dfa8abf2db3083919d7e30764a069fb66b2b9eff9939"},
+    {file = "lxml-4.6.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:d8d3d4713f0c28bdc6c806a278d998546e8efc3498949e3ace6e117462ac0a5e"},
+    {file = "lxml-4.6.2-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:8246f30ca34dc712ab07e51dc34fea883c00b7ccb0e614651e49da2c49a30711"},
+    {file = "lxml-4.6.2-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:923963e989ffbceaa210ac37afc9b906acebe945d2723e9679b643513837b089"},
+    {file = "lxml-4.6.2-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:1471cee35eba321827d7d53d104e7b8c593ea3ad376aa2df89533ce8e1b24a01"},
+    {file = "lxml-4.6.2-cp37-cp37m-win32.whl", hash = "sha256:2363c35637d2d9d6f26f60a208819e7eafc4305ce39dc1d5005eccc4593331c2"},
+    {file = "lxml-4.6.2-cp37-cp37m-win_amd64.whl", hash = "sha256:f4822c0660c3754f1a41a655e37cb4dbbc9be3d35b125a37fab6f82d47674ebc"},
+    {file = "lxml-4.6.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0448576c148c129594d890265b1a83b9cd76fd1f0a6a04620753d9a6bcfd0a4d"},
+    {file = "lxml-4.6.2-cp38-cp38-manylinux1_i686.whl", hash = "sha256:60a20bfc3bd234d54d49c388950195d23a5583d4108e1a1d47c9eef8d8c042b3"},
+    {file = "lxml-4.6.2-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:2e5cc908fe43fe1aa299e58046ad66981131a66aea3129aac7770c37f590a644"},
+    {file = "lxml-4.6.2-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:50c348995b47b5a4e330362cf39fc503b4a43b14a91c34c83b955e1805c8e308"},
+    {file = "lxml-4.6.2-cp38-cp38-win32.whl", hash = "sha256:94d55bd03d8671686e3f012577d9caa5421a07286dd351dfef64791cf7c6c505"},
+    {file = "lxml-4.6.2-cp38-cp38-win_amd64.whl", hash = "sha256:7a7669ff50f41225ca5d6ee0a1ec8413f3a0d8aa2b109f86d540887b7ec0d72a"},
+    {file = "lxml-4.6.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:e0bfe9bb028974a481410432dbe1b182e8191d5d40382e5b8ff39cdd2e5c5931"},
+    {file = "lxml-4.6.2-cp39-cp39-manylinux1_i686.whl", hash = "sha256:6fd8d5903c2e53f49e99359b063df27fdf7acb89a52b6a12494208bf61345a03"},
+    {file = "lxml-4.6.2-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:7e9eac1e526386df7c70ef253b792a0a12dd86d833b1d329e038c7a235dfceb5"},
+    {file = "lxml-4.6.2-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:7ee8af0b9f7de635c61cdd5b8534b76c52cd03536f29f51151b377f76e214a1a"},
+    {file = "lxml-4.6.2-cp39-cp39-win32.whl", hash = "sha256:2e6fd1b8acd005bd71e6c94f30c055594bbd0aa02ef51a22bbfa961ab63b2d75"},
+    {file = "lxml-4.6.2-cp39-cp39-win_amd64.whl", hash = "sha256:535332fe9d00c3cd455bd3dd7d4bacab86e2d564bdf7606079160fa6251caacf"},
+    {file = "lxml-4.6.2.tar.gz", hash = "sha256:cd11c7e8d21af997ee8079037fff88f16fda188a9776eb4b81c7e4c9c0a7d7fc"},
+]
 markupsafe = [
     {file = "MarkupSafe-1.1.1-cp27-cp27m-macosx_10_6_intel.whl", hash = "sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161"},
     {file = "MarkupSafe-1.1.1-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7"},
@@ -2212,12 +2387,13 @@ mistune = [
     {file = "mistune-0.8.4.tar.gz", hash = "sha256:59a3429db53c50b5c6bcc8a07f8848cb00d7dc8bdb431a4ab41920d201d4756e"},
 ]
 mock = [
-    {file = "mock-4.0.2-py3-none-any.whl", hash = "sha256:3f9b2c0196c60d21838f307f5825a7b86b678cedc58ab9e50a8988187b4d81e0"},
-    {file = "mock-4.0.2.tar.gz", hash = "sha256:dd33eb70232b6118298d516bbcecd26704689c386594f0f3c4f13867b2c56f72"},
+    {file = "mock-4.0.3-py3-none-any.whl", hash = "sha256:122fcb64ee37cfad5b3f48d7a7d51875d7031aaf3d8be7c42e2bee25044eee62"},
+    {file = "mock-4.0.3.tar.gz", hash = "sha256:7d3fbbde18228f4ff2f1f119a45cdffa458b4c0dee32eb4d2bb2f82554bac7bc"},
 ]
+mojap-metadata = []
 more-itertools = [
-    {file = "more-itertools-8.6.0.tar.gz", hash = "sha256:b3a9005928e5bed54076e6e549c792b306fddfe72b2d1d22dd63d42d5d3899cf"},
-    {file = "more_itertools-8.6.0-py3-none-any.whl", hash = "sha256:8e1a2a43b2f2727425f2b5839587ae37093f19153dc26c0927d1048ff6557330"},
+    {file = "more-itertools-8.7.0.tar.gz", hash = "sha256:c5d6da9ca3ff65220c3bfd2a8db06d698f05d4d2b9be57e1deb2be5a45019713"},
+    {file = "more_itertools-8.7.0-py3-none-any.whl", hash = "sha256:5652a9ac72209ed7df8d9c15daf4e1aa0e3d2ccd3c87f8265a0673cd9cbc9ced"},
 ]
 moto = [
     {file = "moto-1.3.16-py2.py3-none-any.whl", hash = "sha256:f51903b6b532f6c887b111b3343f6925b77eef0505a914138d98290cf3526df9"},
@@ -2228,109 +2404,102 @@ mypy-extensions = [
     {file = "mypy_extensions-0.4.3.tar.gz", hash = "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"},
 ]
 nbclient = [
-    {file = "nbclient-0.5.1-py3-none-any.whl", hash = "sha256:4d6b116187c795c99b9dba13d46e764d596574b14c296d60670c8dfe454db364"},
-    {file = "nbclient-0.5.1.tar.gz", hash = "sha256:01e2d726d16eaf2cde6db74a87e2451453547e8832d142f73f72fddcd4fe0250"},
+    {file = "nbclient-0.5.3-py3-none-any.whl", hash = "sha256:e79437364a2376892b3f46bedbf9b444e5396cfb1bc366a472c37b48e9551500"},
+    {file = "nbclient-0.5.3.tar.gz", hash = "sha256:db17271330c68c8c88d46d72349e24c147bb6f34ec82d8481a8f025c4d26589c"},
 ]
 nbconvert = [
     {file = "nbconvert-6.0.7-py3-none-any.whl", hash = "sha256:39e9f977920b203baea0be67eea59f7b37a761caa542abe80f5897ce3cf6311d"},
     {file = "nbconvert-6.0.7.tar.gz", hash = "sha256:cbbc13a86dfbd4d1b5dee106539de0795b4db156c894c2c5dc382062bbc29002"},
 ]
 nbformat = [
-    {file = "nbformat-5.0.8-py3-none-any.whl", hash = "sha256:aa9450c16d29286dc69b92ea4913c1bffe86488f90184445996ccc03a2f60382"},
-    {file = "nbformat-5.0.8.tar.gz", hash = "sha256:f545b22138865bfbcc6b1ffe89ed5a2b8e2dc5d4fe876f2ca60d8e6f702a30f8"},
+    {file = "nbformat-5.1.2-py3-none-any.whl", hash = "sha256:3949fdc8f5fa0b1afca16fb307546e78494fa7a7bceff880df8168eafda0e7ac"},
+    {file = "nbformat-5.1.2.tar.gz", hash = "sha256:1d223e64a18bfa7cdf2db2e9ba8a818312fc2a0701d2e910b58df66809385a56"},
 ]
 nest-asyncio = [
-    {file = "nest_asyncio-1.4.3-py3-none-any.whl", hash = "sha256:dbe032f3e9ff7f120e76be22bf6e7958e867aed1743e6894b8a9585fe8495cc9"},
-    {file = "nest_asyncio-1.4.3.tar.gz", hash = "sha256:eaa09ef1353ebefae19162ad423eef7a12166bcc63866f8bff8f3635353cd9fa"},
+    {file = "nest_asyncio-1.5.1-py3-none-any.whl", hash = "sha256:76d6e972265063fe92a90b9cc4fb82616e07d586b346ed9d2c89a4187acea39c"},
+    {file = "nest_asyncio-1.5.1.tar.gz", hash = "sha256:afc5a1c515210a23c461932765691ad39e8eba6551c055ac8d5546e69250d0aa"},
 ]
 networkx = [
     {file = "networkx-2.5-py3-none-any.whl", hash = "sha256:8c5812e9f798d37c50570d15c4a69d5710a18d77bafc903ee9c5fba7454c616c"},
     {file = "networkx-2.5.tar.gz", hash = "sha256:7978955423fbc9639c10498878be59caf99b44dc304c2286162fd24b458c1602"},
 ]
 notebook = [
-    {file = "notebook-6.1.5-py3-none-any.whl", hash = "sha256:508cf9dad7cdb3188f1aa27017dc78179029dfe83814fc505329f689bc2ab50f"},
-    {file = "notebook-6.1.5.tar.gz", hash = "sha256:3db37ae834c5f3b6378381229d0e5dfcbfb558d08c8ce646b1ad355147f5e91d"},
+    {file = "notebook-6.2.0-py3-none-any.whl", hash = "sha256:25ad93c982b623441b491e693ef400598d1a46cdf11b8c9c0b3be6c61ebbb6cd"},
+    {file = "notebook-6.2.0.tar.gz", hash = "sha256:0464b28e18e7a06cec37e6177546c2322739be07962dd13bf712bcb88361f013"},
 ]
 numpy = [
-    {file = "numpy-1.19.4-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:e9b30d4bd69498fc0c3fe9db5f62fffbb06b8eb9321f92cc970f2969be5e3949"},
-    {file = "numpy-1.19.4-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:fedbd128668ead37f33917820b704784aff695e0019309ad446a6d0b065b57e4"},
-    {file = "numpy-1.19.4-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:8ece138c3a16db8c1ad38f52eb32be6086cc72f403150a79336eb2045723a1ad"},
-    {file = "numpy-1.19.4-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:64324f64f90a9e4ef732be0928be853eee378fd6a01be21a0a8469c4f2682c83"},
-    {file = "numpy-1.19.4-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:ad6f2ff5b1989a4899bf89800a671d71b1612e5ff40866d1f4d8bcf48d4e5764"},
-    {file = "numpy-1.19.4-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:d6c7bb82883680e168b55b49c70af29b84b84abb161cbac2800e8fcb6f2109b6"},
-    {file = "numpy-1.19.4-cp36-cp36m-win32.whl", hash = "sha256:13d166f77d6dc02c0a73c1101dd87fdf01339febec1030bd810dcd53fff3b0f1"},
-    {file = "numpy-1.19.4-cp36-cp36m-win_amd64.whl", hash = "sha256:448ebb1b3bf64c0267d6b09a7cba26b5ae61b6d2dbabff7c91b660c7eccf2bdb"},
-    {file = "numpy-1.19.4-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:27d3f3b9e3406579a8af3a9f262f5339005dd25e0ecf3cf1559ff8a49ed5cbf2"},
-    {file = "numpy-1.19.4-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:16c1b388cc31a9baa06d91a19366fb99ddbe1c7b205293ed072211ee5bac1ed2"},
-    {file = "numpy-1.19.4-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:e5b6ed0f0b42317050c88022349d994fe72bfe35f5908617512cd8c8ef9da2a9"},
-    {file = "numpy-1.19.4-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:18bed2bcb39e3f758296584337966e68d2d5ba6aab7e038688ad53c8f889f757"},
-    {file = "numpy-1.19.4-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:fe45becb4c2f72a0907c1d0246ea6449fe7a9e2293bb0e11c4e9a32bb0930a15"},
-    {file = "numpy-1.19.4-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:6d7593a705d662be5bfe24111af14763016765f43cb6923ed86223f965f52387"},
-    {file = "numpy-1.19.4-cp37-cp37m-win32.whl", hash = "sha256:6ae6c680f3ebf1cf7ad1d7748868b39d9f900836df774c453c11c5440bc15b36"},
-    {file = "numpy-1.19.4-cp37-cp37m-win_amd64.whl", hash = "sha256:9eeb7d1d04b117ac0d38719915ae169aa6b61fca227b0b7d198d43728f0c879c"},
-    {file = "numpy-1.19.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:cb1017eec5257e9ac6209ac172058c430e834d5d2bc21961dceeb79d111e5909"},
-    {file = "numpy-1.19.4-cp38-cp38-manylinux1_i686.whl", hash = "sha256:edb01671b3caae1ca00881686003d16c2209e07b7ef8b7639f1867852b948f7c"},
-    {file = "numpy-1.19.4-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:f29454410db6ef8126c83bd3c968d143304633d45dc57b51252afbd79d700893"},
-    {file = "numpy-1.19.4-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:ec149b90019852266fec2341ce1db513b843e496d5a8e8cdb5ced1923a92faab"},
-    {file = "numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:1aeef46a13e51931c0b1cf8ae1168b4a55ecd282e6688fdb0a948cc5a1d5afb9"},
-    {file = "numpy-1.19.4-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:08308c38e44cc926bdfce99498b21eec1f848d24c302519e64203a8da99a97db"},
-    {file = "numpy-1.19.4-cp38-cp38-win32.whl", hash = "sha256:5734bdc0342aba9dfc6f04920988140fb41234db42381cf7ccba64169f9fe7ac"},
-    {file = "numpy-1.19.4-cp38-cp38-win_amd64.whl", hash = "sha256:09c12096d843b90eafd01ea1b3307e78ddd47a55855ad402b157b6c4862197ce"},
-    {file = "numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:e452dc66e08a4ce642a961f134814258a082832c78c90351b75c41ad16f79f63"},
-    {file = "numpy-1.19.4-cp39-cp39-manylinux1_i686.whl", hash = "sha256:a5d897c14513590a85774180be713f692df6fa8ecf6483e561a6d47309566f37"},
-    {file = "numpy-1.19.4-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:a09f98011236a419ee3f49cedc9ef27d7a1651df07810ae430a6b06576e0b414"},
-    {file = "numpy-1.19.4-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:50e86c076611212ca62e5a59f518edafe0c0730f7d9195fec718da1a5c2bb1fc"},
-    {file = "numpy-1.19.4-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:f0d3929fe88ee1c155129ecd82f981b8856c5d97bcb0d5f23e9b4242e79d1de3"},
-    {file = "numpy-1.19.4-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:c42c4b73121caf0ed6cd795512c9c09c52a7287b04d105d112068c1736d7c753"},
-    {file = "numpy-1.19.4-cp39-cp39-win32.whl", hash = "sha256:8cac8790a6b1ddf88640a9267ee67b1aee7a57dfa2d2dd33999d080bc8ee3a0f"},
-    {file = "numpy-1.19.4-cp39-cp39-win_amd64.whl", hash = "sha256:4377e10b874e653fe96985c05feed2225c912e328c8a26541f7fc600fb9c637b"},
-    {file = "numpy-1.19.4-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:2a2740aa9733d2e5b2dfb33639d98a64c3b0f24765fed86b0fd2aec07f6a0a08"},
-    {file = "numpy-1.19.4.zip", hash = "sha256:141ec3a3300ab89c7f2b0775289954d193cc8edb621ea05f99db9cb181530512"},
+    {file = "numpy-1.19.5-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:cc6bd4fd593cb261332568485e20a0712883cf631f6f5e8e86a52caa8b2b50ff"},
+    {file = "numpy-1.19.5-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:aeb9ed923be74e659984e321f609b9ba54a48354bfd168d21a2b072ed1e833ea"},
+    {file = "numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:8b5e972b43c8fc27d56550b4120fe6257fdc15f9301914380b27f74856299fea"},
+    {file = "numpy-1.19.5-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:43d4c81d5ffdff6bae58d66a3cd7f54a7acd9a0e7b18d97abb255defc09e3140"},
+    {file = "numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:a4646724fba402aa7504cd48b4b50e783296b5e10a524c7a6da62e4a8ac9698d"},
+    {file = "numpy-1.19.5-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:2e55195bc1c6b705bfd8ad6f288b38b11b1af32f3c8289d6c50d47f950c12e76"},
+    {file = "numpy-1.19.5-cp36-cp36m-win32.whl", hash = "sha256:39b70c19ec771805081578cc936bbe95336798b7edf4732ed102e7a43ec5c07a"},
+    {file = "numpy-1.19.5-cp36-cp36m-win_amd64.whl", hash = "sha256:dbd18bcf4889b720ba13a27ec2f2aac1981bd41203b3a3b27ba7a33f88ae4827"},
+    {file = "numpy-1.19.5-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:603aa0706be710eea8884af807b1b3bc9fb2e49b9f4da439e76000f3b3c6ff0f"},
+    {file = "numpy-1.19.5-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:cae865b1cae1ec2663d8ea56ef6ff185bad091a5e33ebbadd98de2cfa3fa668f"},
+    {file = "numpy-1.19.5-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:36674959eed6957e61f11c912f71e78857a8d0604171dfd9ce9ad5cbf41c511c"},
+    {file = "numpy-1.19.5-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:06fab248a088e439402141ea04f0fffb203723148f6ee791e9c75b3e9e82f080"},
+    {file = "numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:6149a185cece5ee78d1d196938b2a8f9d09f5a5ebfbba66969302a778d5ddd1d"},
+    {file = "numpy-1.19.5-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:50a4a0ad0111cc1b71fa32dedd05fa239f7fb5a43a40663269bb5dc7877cfd28"},
+    {file = "numpy-1.19.5-cp37-cp37m-win32.whl", hash = "sha256:d051ec1c64b85ecc69531e1137bb9751c6830772ee5c1c426dbcfe98ef5788d7"},
+    {file = "numpy-1.19.5-cp37-cp37m-win_amd64.whl", hash = "sha256:a12ff4c8ddfee61f90a1633a4c4afd3f7bcb32b11c52026c92a12e1325922d0d"},
+    {file = "numpy-1.19.5-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:cf2402002d3d9f91c8b01e66fbb436a4ed01c6498fffed0e4c7566da1d40ee1e"},
+    {file = "numpy-1.19.5-cp38-cp38-manylinux1_i686.whl", hash = "sha256:1ded4fce9cfaaf24e7a0ab51b7a87be9038ea1ace7f34b841fe3b6894c721d1c"},
+    {file = "numpy-1.19.5-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:012426a41bc9ab63bb158635aecccc7610e3eff5d31d1eb43bc099debc979d94"},
+    {file = "numpy-1.19.5-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:759e4095edc3c1b3ac031f34d9459fa781777a93ccc633a472a5468587a190ff"},
+    {file = "numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:a9d17f2be3b427fbb2bce61e596cf555d6f8a56c222bd2ca148baeeb5e5c783c"},
+    {file = "numpy-1.19.5-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:99abf4f353c3d1a0c7a5f27699482c987cf663b1eac20db59b8c7b061eabd7fc"},
+    {file = "numpy-1.19.5-cp38-cp38-win32.whl", hash = "sha256:384ec0463d1c2671170901994aeb6dce126de0a95ccc3976c43b0038a37329c2"},
+    {file = "numpy-1.19.5-cp38-cp38-win_amd64.whl", hash = "sha256:811daee36a58dc79cf3d8bdd4a490e4277d0e4b7d103a001a4e73ddb48e7e6aa"},
+    {file = "numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:c843b3f50d1ab7361ca4f0b3639bf691569493a56808a0b0c54a051d260b7dbd"},
+    {file = "numpy-1.19.5-cp39-cp39-manylinux1_i686.whl", hash = "sha256:d6631f2e867676b13026e2846180e2c13c1e11289d67da08d71cacb2cd93d4aa"},
+    {file = "numpy-1.19.5-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:7fb43004bce0ca31d8f13a6eb5e943fa73371381e53f7074ed21a4cb786c32f8"},
+    {file = "numpy-1.19.5-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:2ea52bd92ab9f768cc64a4c3ef8f4b2580a17af0a5436f6126b08efbd1838371"},
+    {file = "numpy-1.19.5-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:400580cbd3cff6ffa6293df2278c75aef2d58d8d93d3c5614cd67981dae68ceb"},
+    {file = "numpy-1.19.5-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:df609c82f18c5b9f6cb97271f03315ff0dbe481a2a02e56aeb1b1a985ce38e60"},
+    {file = "numpy-1.19.5-cp39-cp39-win32.whl", hash = "sha256:ab83f24d5c52d60dbc8cd0528759532736b56db58adaa7b5f1f76ad551416a1e"},
+    {file = "numpy-1.19.5-cp39-cp39-win_amd64.whl", hash = "sha256:0eef32ca3132a48e43f6a0f5a82cb508f22ce5a3d6f67a8329c81c8e226d3f6e"},
+    {file = "numpy-1.19.5-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:a0d53e51a6cb6f0d9082decb7a4cb6dfb33055308c4c44f53103c073f649af73"},
+    {file = "numpy-1.19.5.zip", hash = "sha256:a76f502430dd98d7546e1ea2250a7360c065a5fdea52b2dffe8ae7180909b6f4"},
 ]
 openpyxl = [
-    {file = "openpyxl-3.0.5-py2.py3-none-any.whl", hash = "sha256:f7d666b569f729257082cf7ddc56262431878f602dcc2bc3980775c59439cdab"},
-    {file = "openpyxl-3.0.5.tar.gz", hash = "sha256:18e11f9a650128a12580a58e3daba14e00a11d9e907c554a17ea016bf1a2c71b"},
+    {file = "openpyxl-3.0.7-py2.py3-none-any.whl", hash = "sha256:46af4eaf201a89b610fcca177eed957635f88770a5462fb6aae4a2a52b0ff516"},
+    {file = "openpyxl-3.0.7.tar.gz", hash = "sha256:6456a3b472e1ef0facb1129f3c6ef00713cebf62e736cd7a75bcc3247432f251"},
 ]
 packaging = [
-    {file = "packaging-20.4-py2.py3-none-any.whl", hash = "sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181"},
-    {file = "packaging-20.4.tar.gz", hash = "sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8"},
+    {file = "packaging-20.9-py2.py3-none-any.whl", hash = "sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"},
+    {file = "packaging-20.9.tar.gz", hash = "sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5"},
 ]
 pandas = [
-    {file = "pandas-1.1.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:882012763668af54b48f1412bab95c5cc0a7ccce5a2a8221cfc3839a6e3394ef"},
-    {file = "pandas-1.1.3-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:206d7c3e5356dcadf082e64dc25c24bc8541718045826074f96346e9d6d05a20"},
-    {file = "pandas-1.1.3-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:ca31ac8578d48da354cf66a473d4d5ff99277ca71d321dc7ea4e6fad3c6bb0fd"},
-    {file = "pandas-1.1.3-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:fd6f05b6101d0e76f3e5c26a47be5be7be96ed84ef3981dc1852e76898e73594"},
-    {file = "pandas-1.1.3-cp36-cp36m-win32.whl", hash = "sha256:ca71a5aa9eeb3ef5b31feca7d9b6369d6b3d0b2e9c85d7a89abe3ecb013f1e86"},
-    {file = "pandas-1.1.3-cp36-cp36m-win_amd64.whl", hash = "sha256:54f5f564058b0280d588c3758abde82e280702c440db5faf0c686b80336096f9"},
-    {file = "pandas-1.1.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:3a038cd5da602b955d335aa80cbaa0e5774f68501ff47b9c21509906981478da"},
-    {file = "pandas-1.1.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:24f61f40febe47edac271eda45d683e42838b7db2bd0f82574d9800259d2b182"},
-    {file = "pandas-1.1.3-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:427be9938b2f79ab298de84f87693914cda238a27cf10580da96caf3dff64115"},
-    {file = "pandas-1.1.3-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:5a8a84b75ca3a29bb4263b35d5ed9fcaae2b062f014feed8c5daa897339c7d85"},
-    {file = "pandas-1.1.3-cp37-cp37m-win32.whl", hash = "sha256:c22e40f1b4d162ca18eb6b2c572e63eef220dbc9cc3de0241cefb77972621bb7"},
-    {file = "pandas-1.1.3-cp37-cp37m-win_amd64.whl", hash = "sha256:920d30fdff65a079f071db635d282b4f583c2b26f2b58d5dca218aac7c59974d"},
-    {file = "pandas-1.1.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:d6b1f9d506dc23da2915bcae5c5968990049c9cec44108bd9855d2c7c89d91dc"},
-    {file = "pandas-1.1.3-cp38-cp38-manylinux1_i686.whl", hash = "sha256:b11b496c317dbe007898de699fd59eaf687d0fe8c1b7dad109db6010155d28ae"},
-    {file = "pandas-1.1.3-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:d89dbc58aec1544722a8d5046f880b597c497ef8a82c5fe695b4b2effafac5ec"},
-    {file = "pandas-1.1.3-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:df43ea0e9fd9f9672b0de9cac26d01255ad50481994bf3cb4687c21eec2d7bbc"},
-    {file = "pandas-1.1.3-cp38-cp38-win32.whl", hash = "sha256:a605054fbca71ed1d08bb2aef6f73c84a579bbac956bfe8f9718d5e84cb41248"},
-    {file = "pandas-1.1.3-cp38-cp38-win_amd64.whl", hash = "sha256:84a4ffe668df357e31f98c829536e3a7142c3036c82f996e639f644c5d32eda1"},
-    {file = "pandas-1.1.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:147162568b1242355290341baf281926cfac66ada07e634f3fc521ac967e4653"},
-    {file = "pandas-1.1.3-cp39-cp39-manylinux1_i686.whl", hash = "sha256:2999adc6736f8cb4c69d65a6e2b25a11bcb395da5b048342b8e4d6fe055e57ae"},
-    {file = "pandas-1.1.3-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:f4cb8252ae71f093f4a6b847adf0bc9330f109c48f08363c2071f189f1c89c87"},
-    {file = "pandas-1.1.3-cp39-cp39-win32.whl", hash = "sha256:b026e913d88fad3a74eea8ed5a5f98e8823080ea02f8d9bb0ec19e92552daad6"},
-    {file = "pandas-1.1.3-cp39-cp39-win_amd64.whl", hash = "sha256:11c284769f41e95f7d16a327eb555989c5f29418aad075fa80c97ef3aa8fb885"},
-    {file = "pandas-1.1.3.tar.gz", hash = "sha256:babbeda2f83b0686c9ad38d93b10516e68cdcd5771007eb80a763e98aaf44613"},
+    {file = "pandas-1.2.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:cba93d4fd3b0a42858b2b599495aff793fb5d94587979f45a14177d1217ba446"},
+    {file = "pandas-1.2.0-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:9e18631d996fe131de6cb31a8bdae18965cc8f39eb23fdfbbf42808ecc63dabf"},
+    {file = "pandas-1.2.0-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:7b54c14130a3448d81eed1348f52429c23e27188d9db6e6d4afeae792bc49c11"},
+    {file = "pandas-1.2.0-cp37-cp37m-win32.whl", hash = "sha256:6c1a57e4d0d6f9633a07817c44e6b36d81c265fe4c52d0c0505513a2d0f7953c"},
+    {file = "pandas-1.2.0-cp37-cp37m-win_amd64.whl", hash = "sha256:43482789c55cbabeed9482263cfc98a11e8fcae900cb63ef038948acb4a72570"},
+    {file = "pandas-1.2.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0be6102dd99910513e75ed6536284743ead810349c51bdeadd2a5b6649f30abb"},
+    {file = "pandas-1.2.0-cp38-cp38-manylinux1_i686.whl", hash = "sha256:9c6692cea6d56da8650847172bdb148622f545e7782d17995822434c79d7a211"},
+    {file = "pandas-1.2.0-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:272675a98fa4954b9fc0933df775596fc942e50015d7e75d8f19548808a2bfdf"},
+    {file = "pandas-1.2.0-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:33318fa24b192b1a4684347ff76679a7267fd4e547da9f71556a5914f0dc10e7"},
+    {file = "pandas-1.2.0-cp38-cp38-win32.whl", hash = "sha256:3bc6d2be03cb75981d8cbeda09503cd9d6d699fc0dc28a65e197165ad527b7b8"},
+    {file = "pandas-1.2.0-cp38-cp38-win_amd64.whl", hash = "sha256:7904ee438549b5223ce8dc008772458dd7c5cf0ccc64cf903e81202400702235"},
+    {file = "pandas-1.2.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:f8b87d2f541cd9bc4ecfe85a561abac85c33fe4de4ce70cca36b2768af2611f5"},
+    {file = "pandas-1.2.0-cp39-cp39-manylinux1_i686.whl", hash = "sha256:91fd0b94e7b98528177a05e6f65efea79d7ef9dec15ee48c7c69fc39fdd87235"},
+    {file = "pandas-1.2.0-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:8f92b07cdbfa3704d85b4264e52c216cafe6c0059b0d07cdad8cb29e0b90f2b8"},
+    {file = "pandas-1.2.0-cp39-cp39-win32.whl", hash = "sha256:2d8b4f532db37418121831a461fd107d826c240b098f52e7a1b4ab3d5aaa4fb2"},
+    {file = "pandas-1.2.0-cp39-cp39-win_amd64.whl", hash = "sha256:616478c1bd8fe1e600f521ae2da434e021c11e7a4e5da3451d02906143d3629a"},
+    {file = "pandas-1.2.0.tar.gz", hash = "sha256:e03386615b970b8b41da6a68afe717626741bb2431cec993640685614c0680e4"},
 ]
 pandocfilters = [
     {file = "pandocfilters-1.4.3.tar.gz", hash = "sha256:bc63fbb50534b4b1f8ebe1860889289e8af94a23bff7445259592df25a3906eb"},
 ]
 parameterized = [
-    {file = "parameterized-0.7.4-py2.py3-none-any.whl", hash = "sha256:59ab908e31c01505a987a2be78854e19cb1630c047bbab7848169c371d614d56"},
-    {file = "parameterized-0.7.4.tar.gz", hash = "sha256:190f8cc7230eee0b56b30d7f074fd4d165f7c45e6077582d0813c8557e738490"},
+    {file = "parameterized-0.7.5-py2.py3-none-any.whl", hash = "sha256:b36f709fcfd9e8c144c1374b16a40fc0b5984efe93e41d53e6d228ae2a1540c5"},
+    {file = "parameterized-0.7.5.tar.gz", hash = "sha256:b5e6af67b9e49485e30125b1c8f031ffa81a265ca08bfa73f31551bf03cf68c4"},
 ]
 parso = [
-    {file = "parso-0.7.1-py2.py3-none-any.whl", hash = "sha256:97218d9159b2520ff45eb78028ba8b50d2bc61dcc062a9682666f2dc4bd331ea"},
-    {file = "parso-0.7.1.tar.gz", hash = "sha256:caba44724b994a8a5e086460bb212abc5a8bc46951bf4a9a1210745953622eb9"},
+    {file = "parso-0.8.1-py2.py3-none-any.whl", hash = "sha256:15b00182f472319383252c18d5913b69269590616c947747bc50bf4ac768f410"},
+    {file = "parso-0.8.1.tar.gz", hash = "sha256:8519430ad07087d4c997fda3a7918f7cfa27cb58972a8c89c2a0295a1c940e9e"},
 ]
 pathspec = [
     {file = "pathspec-0.8.1-py2.py3-none-any.whl", hash = "sha256:aa0cb481c4041bf52ffa7b0d8fa6cd3e88a2ca4879c533c9153882ee2556790d"},
@@ -2340,6 +2509,10 @@ pexpect = [
     {file = "pexpect-4.8.0-py2.py3-none-any.whl", hash = "sha256:0b48a55dcb3c05f3329815901ea4fc1537514d6ba867a152b581d69ae3710937"},
     {file = "pexpect-4.8.0.tar.gz", hash = "sha256:fc65a43959d153d0114afe13997d439c22823a27cefceb5ff35c2178c6784c0c"},
 ]
+pg8000 = [
+    {file = "pg8000-1.16.6-py3-none-any.whl", hash = "sha256:66fa16a402f38f8ba664206b4ba4040f24ea9641c4205b2b96a1ff3a613de3be"},
+    {file = "pg8000-1.16.6.tar.gz", hash = "sha256:8fc1e6a62ccb7c9830f1e7e9288e2d20eaf373cc8875b5c55b7d5d9b7717be91"},
+]
 pickleshare = [
     {file = "pickleshare-0.7.5-py2.py3-none-any.whl", hash = "sha256:9649af414d74d4df115d5d718f82acb59c9d418196b7b4290ed47a12ce62df56"},
     {file = "pickleshare-0.7.5.tar.gz", hash = "sha256:87683d47965c1da65cdacaf31c8441d12b8044cdec9aca500cd78fc2c683afca"},
@@ -2353,53 +2526,16 @@ prometheus-client = [
     {file = "prometheus_client-0.9.0.tar.gz", hash = "sha256:9da7b32f02439d8c04f7777021c304ed51d9ec180604700c1ba72a4d44dceb03"},
 ]
 prompt-toolkit = [
-    {file = "prompt_toolkit-3.0.8-py3-none-any.whl", hash = "sha256:7debb9a521e0b1ee7d2fe96ee4bd60ef03c6492784de0547337ca4433e46aa63"},
-    {file = "prompt_toolkit-3.0.8.tar.gz", hash = "sha256:25c95d2ac813909f813c93fde734b6e44406d1477a9faef7c915ff37d39c0a8c"},
-]
-psycopg2-binary = [
-    {file = "psycopg2-binary-2.8.6.tar.gz", hash = "sha256:11b9c0ebce097180129e422379b824ae21c8f2a6596b159c7659e2e5a00e1aa0"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl", hash = "sha256:d14b140a4439d816e3b1229a4a525df917d6ea22a0771a2a78332273fd9528a4"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:1fabed9ea2acc4efe4671b92c669a213db744d2af8a9fc5d69a8e9bc14b7a9db"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:f5ab93a2cb2d8338b1674be43b442a7f544a0971da062a5da774ed40587f18f5"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27m-win32.whl", hash = "sha256:b4afc542c0ac0db720cf516dd20c0846f71c248d2b3d21013aa0d4ef9c71ca25"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27m-win_amd64.whl", hash = "sha256:e74a55f6bad0e7d3968399deb50f61f4db1926acf4a6d83beaaa7df986f48b1c"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:0deac2af1a587ae12836aa07970f5cb91964f05a7c6cdb69d8425ff4c15d4e2c"},
-    {file = "psycopg2_binary-2.8.6-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:ad20d2eb875aaa1ea6d0f2916949f5c08a19c74d05b16ce6ebf6d24f2c9f75d1"},
-    {file = "psycopg2_binary-2.8.6-cp34-cp34m-win32.whl", hash = "sha256:950bc22bb56ee6ff142a2cb9ee980b571dd0912b0334aa3fe0fe3788d860bea2"},
-    {file = "psycopg2_binary-2.8.6-cp34-cp34m-win_amd64.whl", hash = "sha256:b8a3715b3c4e604bcc94c90a825cd7f5635417453b253499664f784fc4da0152"},
-    {file = "psycopg2_binary-2.8.6-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl", hash = "sha256:d1b4ab59e02d9008efe10ceabd0b31e79519da6fb67f7d8e8977118832d0f449"},
-    {file = "psycopg2_binary-2.8.6-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:ac0c682111fbf404525dfc0f18a8b5f11be52657d4f96e9fcb75daf4f3984859"},
-    {file = "psycopg2_binary-2.8.6-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:7d92a09b788cbb1aec325af5fcba9fed7203897bbd9269d5691bb1e3bce29550"},
-    {file = "psycopg2_binary-2.8.6-cp35-cp35m-win32.whl", hash = "sha256:aaa4213c862f0ef00022751161df35804127b78adf4a2755b9f991a507e425fd"},
-    {file = "psycopg2_binary-2.8.6-cp35-cp35m-win_amd64.whl", hash = "sha256:c2507d796fca339c8fb03216364cca68d87e037c1f774977c8fc377627d01c71"},
-    {file = "psycopg2_binary-2.8.6-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl", hash = "sha256:ee69dad2c7155756ad114c02db06002f4cded41132cc51378e57aad79cc8e4f4"},
-    {file = "psycopg2_binary-2.8.6-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:e82aba2188b9ba309fd8e271702bd0d0fc9148ae3150532bbb474f4590039ffb"},
-    {file = "psycopg2_binary-2.8.6-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:d5227b229005a696cc67676e24c214740efd90b148de5733419ac9aaba3773da"},
-    {file = "psycopg2_binary-2.8.6-cp36-cp36m-win32.whl", hash = "sha256:a0eb43a07386c3f1f1ebb4dc7aafb13f67188eab896e7397aa1ee95a9c884eb2"},
-    {file = "psycopg2_binary-2.8.6-cp36-cp36m-win_amd64.whl", hash = "sha256:e1f57aa70d3f7cc6947fd88636a481638263ba04a742b4a37dd25c373e41491a"},
-    {file = "psycopg2_binary-2.8.6-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl", hash = "sha256:833709a5c66ca52f1d21d41865a637223b368c0ee76ea54ca5bad6f2526c7679"},
-    {file = "psycopg2_binary-2.8.6-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:ba28584e6bca48c59eecbf7efb1576ca214b47f05194646b081717fa628dfddf"},
-    {file = "psycopg2_binary-2.8.6-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:6a32f3a4cb2f6e1a0b15215f448e8ce2da192fd4ff35084d80d5e39da683e79b"},
-    {file = "psycopg2_binary-2.8.6-cp37-cp37m-win32.whl", hash = "sha256:0e4dc3d5996760104746e6cfcdb519d9d2cd27c738296525d5867ea695774e67"},
-    {file = "psycopg2_binary-2.8.6-cp37-cp37m-win_amd64.whl", hash = "sha256:cec7e622ebc545dbb4564e483dd20e4e404da17ae07e06f3e780b2dacd5cee66"},
-    {file = "psycopg2_binary-2.8.6-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl", hash = "sha256:ba381aec3a5dc29634f20692349d73f2d21f17653bda1decf0b52b11d694541f"},
-    {file = "psycopg2_binary-2.8.6-cp38-cp38-manylinux1_i686.whl", hash = "sha256:a0c50db33c32594305b0ef9abc0cb7db13de7621d2cadf8392a1d9b3c437ef77"},
-    {file = "psycopg2_binary-2.8.6-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:2dac98e85565d5688e8ab7bdea5446674a83a3945a8f416ad0110018d1501b94"},
-    {file = "psycopg2_binary-2.8.6-cp38-cp38-win32.whl", hash = "sha256:bd1be66dde2b82f80afb9459fc618216753f67109b859a361cf7def5c7968729"},
-    {file = "psycopg2_binary-2.8.6-cp38-cp38-win_amd64.whl", hash = "sha256:8cd0fb36c7412996859cb4606a35969dd01f4ea34d9812a141cd920c3b18be77"},
-    {file = "psycopg2_binary-2.8.6-cp39-cp39-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl", hash = "sha256:89705f45ce07b2dfa806ee84439ec67c5d9a0ef20154e0e475e2b2ed392a5b83"},
-    {file = "psycopg2_binary-2.8.6-cp39-cp39-manylinux1_i686.whl", hash = "sha256:42ec1035841b389e8cc3692277a0bd81cdfe0b65d575a2c8862cec7a80e62e52"},
-    {file = "psycopg2_binary-2.8.6-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:7312e931b90fe14f925729cde58022f5d034241918a5c4f9797cac62f6b3a9dd"},
-    {file = "psycopg2_binary-2.8.6-cp39-cp39-win32.whl", hash = "sha256:6422f2ff0919fd720195f64ffd8f924c1395d30f9a495f31e2392c2efafb5056"},
-    {file = "psycopg2_binary-2.8.6-cp39-cp39-win_amd64.whl", hash = "sha256:15978a1fbd225583dd8cdaf37e67ccc278b5abecb4caf6b2d6b8e2b948e953f6"},
+    {file = "prompt_toolkit-3.0.17-py3-none-any.whl", hash = "sha256:4cea7d09e46723885cb8bc54678175453e5071e9449821dce6f017b1d1fbfc1a"},
+    {file = "prompt_toolkit-3.0.17.tar.gz", hash = "sha256:9397a7162cf45449147ad6042fa37983a081b8a73363a5253dd4072666333137"},
 ]
 ptyprocess = [
-    {file = "ptyprocess-0.6.0-py2.py3-none-any.whl", hash = "sha256:d7cc528d76e76342423ca640335bd3633420dc1366f258cb31d05e865ef5ca1f"},
-    {file = "ptyprocess-0.6.0.tar.gz", hash = "sha256:923f299cc5ad920c68f2bc0bc98b75b9f838b93b599941a6b63ddbc2476394c0"},
+    {file = "ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35"},
+    {file = "ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220"},
 ]
 py = [
-    {file = "py-1.9.0-py2.py3-none-any.whl", hash = "sha256:366389d1db726cd2fcfc79732e75410e5fe4d31db13692115529d34069a043c2"},
-    {file = "py-1.9.0.tar.gz", hash = "sha256:9ca6883ce56b4e8da7e79ac18787889fa5206c79dcc67fb065376cd2fe03f342"},
+    {file = "py-1.10.0-py2.py3-none-any.whl", hash = "sha256:3b80836aa6d1feeaa108e046da6423ab8f6ceda6468545ae8d02d9d58d18818a"},
+    {file = "py-1.10.0.tar.gz", hash = "sha256:21b81bda15b66ef5e1a777a21c4dcd9c20ad3efd0b3f817e7a809035269e1bd3"},
 ]
 pyarrow = [
     {file = "pyarrow-2.0.0-cp35-cp35m-macosx_10_13_intel.whl", hash = "sha256:6afc71cc9c234f3cdbe971297468755ec3392966cb19d3a6caf42fd7dbc6aaa9"},
@@ -2426,7 +2562,6 @@ pyarrow = [
     {file = "pyarrow-2.0.0-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:cf9bf10daadbbf1a360ac1c7dab0b4f8381d81a3f452737bd6ed310d57a88be8"},
     {file = "pyarrow-2.0.0-cp38-cp38-manylinux2014_x86_64.whl", hash = "sha256:dd661b6598ce566c6f41d31cc1fc4482308613c2c0c808bd8db33b0643192f84"},
     {file = "pyarrow-2.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:14b02a629986c25e045f81771799e07a8bb3f339898c111314066436769a3dd4"},
-    {file = "pyarrow-2.0.0.tar.gz", hash = "sha256:b5e6cd217457e8febcc98a6c279b96f72d5c31a24cd2bffd8d3b2da701d2025c"},
 ]
 pyasn1 = [
     {file = "pyasn1-0.4.8-py2.4.egg", hash = "sha256:fec3e9d8e36808a28efb59b489e4528c10ad0f480e57dcc32b4de5c9d8c9fdf3"},
@@ -2444,29 +2579,33 @@ pyasn1 = [
     {file = "pyasn1-0.4.8.tar.gz", hash = "sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba"},
 ]
 pycodestyle = [
-    {file = "pycodestyle-2.6.0-py2.py3-none-any.whl", hash = "sha256:2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367"},
-    {file = "pycodestyle-2.6.0.tar.gz", hash = "sha256:c58a7d2815e0e8d7972bf1803331fb0152f867bd89adf8a01dfd55085434192e"},
+    {file = "pycodestyle-2.7.0-py2.py3-none-any.whl", hash = "sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068"},
+    {file = "pycodestyle-2.7.0.tar.gz", hash = "sha256:c389c1d06bf7904078ca03399a4816f974a1d590090fecea0c63ec26ebaf1cef"},
 ]
 pycparser = [
     {file = "pycparser-2.20-py2.py3-none-any.whl", hash = "sha256:7582ad22678f0fcd81102833f60ef8d0e57288b6b5fb00323d101be910e35705"},
     {file = "pycparser-2.20.tar.gz", hash = "sha256:2d475327684562c3a96cc71adf7dc8c4f0565175cf86b6d7a404ff4c771f15f0"},
 ]
 pyflakes = [
-    {file = "pyflakes-2.2.0-py2.py3-none-any.whl", hash = "sha256:0d94e0e05a19e57a99444b6ddcf9a6eb2e5c68d3ca1e98e90707af8152c90a92"},
-    {file = "pyflakes-2.2.0.tar.gz", hash = "sha256:35b2d75ee967ea93b55750aa9edbbf72813e06a66ba54438df2cfac9e3c27fc8"},
+    {file = "pyflakes-2.3.0-py2.py3-none-any.whl", hash = "sha256:910208209dcea632721cb58363d0f72913d9e8cf64dc6f8ae2e02a3609aba40d"},
+    {file = "pyflakes-2.3.0.tar.gz", hash = "sha256:e59fd8e750e588358f1b8885e5a4751203a0516e0ee6d34811089ac294c8806f"},
 ]
 pygments = [
-    {file = "Pygments-2.7.2-py3-none-any.whl", hash = "sha256:88a0bbcd659fcb9573703957c6b9cff9fab7295e6e76db54c9d00ae42df32773"},
-    {file = "Pygments-2.7.2.tar.gz", hash = "sha256:381985fcc551eb9d37c52088a32914e00517e57f4a21609f48141ba08e193fa0"},
+    {file = "Pygments-2.8.1-py3-none-any.whl", hash = "sha256:534ef71d539ae97d4c3a4cf7d6f110f214b0e687e92f9cb9d2a3b0d3101289c8"},
+    {file = "Pygments-2.8.1.tar.gz", hash = "sha256:2656e1a6edcdabf4275f9a3640db59fd5de107d88e8663c5d4e9a0fa62f77f94"},
 ]
 pymysql = [
-    {file = "PyMySQL-0.10.1-py2.py3-none-any.whl", hash = "sha256:44f47128dda8676e021c8d2dbb49a82be9e4ab158b9f03e897152a3a287c69ea"},
-    {file = "PyMySQL-0.10.1.tar.gz", hash = "sha256:263040d2779a3b84930f7ac9da5132be0fefcd6f453a885756656103f8ee1fdd"},
+    {file = "PyMySQL-1.0.2-py3-none-any.whl", hash = "sha256:41fc3a0c5013d5f039639442321185532e3e2c8924687abe6537de157d403641"},
+    {file = "PyMySQL-1.0.2.tar.gz", hash = "sha256:816927a350f38d56072aeca5dfb10221fe1dc653745853d30a216637f5d7ad36"},
 ]
 pyparsing = [
     {file = "pyparsing-2.4.7-py2.py3-none-any.whl", hash = "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"},
     {file = "pyparsing-2.4.7.tar.gz", hash = "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1"},
 ]
+pypiwin32 = [
+    {file = "pypiwin32-223-py3-none-any.whl", hash = "sha256:67adf399debc1d5d14dffc1ab5acacb800da569754fafdc576b2a039485aa775"},
+    {file = "pypiwin32-223.tar.gz", hash = "sha256:71be40c1fbd28594214ecaecb58e7aa8b708eabfa0125c8a109ebd51edbd776a"},
+]
 pyrsistent = [
     {file = "pyrsistent-0.17.3.tar.gz", hash = "sha256:2e636185d9eb976a18a8a8e96efce62f2905fea90041958d8cc2a189756ebf3e"},
 ]
@@ -2486,22 +2625,20 @@ python-slugify = [
     {file = "python-slugify-4.0.1.tar.gz", hash = "sha256:69a517766e00c1268e5bbfc0d010a0a8508de0b18d30ad5a1ff357f8ae724270"},
 ]
 pytz = [
-    {file = "pytz-2020.4-py2.py3-none-any.whl", hash = "sha256:5c55e189b682d420be27c6995ba6edce0c0a77dd67bfbe2ae6607134d5851ffd"},
-    {file = "pytz-2020.4.tar.gz", hash = "sha256:3e6b7dd2d1e0a59084bcee14a17af60c5c562cdc16d828e8eba2e683d3a7e268"},
+    {file = "pytz-2020.1-py2.py3-none-any.whl", hash = "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed"},
+    {file = "pytz-2020.1.tar.gz", hash = "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"},
 ]
 pywin32 = [
-    {file = "pywin32-227-cp27-cp27m-win32.whl", hash = "sha256:371fcc39416d736401f0274dd64c2302728c9e034808e37381b5e1b22be4a6b0"},
-    {file = "pywin32-227-cp27-cp27m-win_amd64.whl", hash = "sha256:4cdad3e84191194ea6d0dd1b1b9bdda574ff563177d2adf2b4efec2a244fa116"},
-    {file = "pywin32-227-cp35-cp35m-win32.whl", hash = "sha256:f4c5be1a293bae0076d93c88f37ee8da68136744588bc5e2be2f299a34ceb7aa"},
-    {file = "pywin32-227-cp35-cp35m-win_amd64.whl", hash = "sha256:a929a4af626e530383a579431b70e512e736e9588106715215bf685a3ea508d4"},
-    {file = "pywin32-227-cp36-cp36m-win32.whl", hash = "sha256:300a2db938e98c3e7e2093e4491439e62287d0d493fe07cce110db070b54c0be"},
-    {file = "pywin32-227-cp36-cp36m-win_amd64.whl", hash = "sha256:9b31e009564fb95db160f154e2aa195ed66bcc4c058ed72850d047141b36f3a2"},
-    {file = "pywin32-227-cp37-cp37m-win32.whl", hash = "sha256:47a3c7551376a865dd8d095a98deba954a98f326c6fe3c72d8726ca6e6b15507"},
-    {file = "pywin32-227-cp37-cp37m-win_amd64.whl", hash = "sha256:31f88a89139cb2adc40f8f0e65ee56a8c585f629974f9e07622ba80199057511"},
-    {file = "pywin32-227-cp38-cp38-win32.whl", hash = "sha256:7f18199fbf29ca99dff10e1f09451582ae9e372a892ff03a28528a24d55875bc"},
-    {file = "pywin32-227-cp38-cp38-win_amd64.whl", hash = "sha256:7c1ae32c489dc012930787f06244426f8356e129184a02c25aef163917ce158e"},
-    {file = "pywin32-227-cp39-cp39-win32.whl", hash = "sha256:c054c52ba46e7eb6b7d7dfae4dbd987a1bb48ee86debe3f245a2884ece46e295"},
-    {file = "pywin32-227-cp39-cp39-win_amd64.whl", hash = "sha256:f27cec5e7f588c3d1051651830ecc00294f90728d19c3bf6916e6dba93ea357c"},
+    {file = "pywin32-300-cp35-cp35m-win32.whl", hash = "sha256:1c204a81daed2089e55d11eefa4826c05e604d27fe2be40b6bf8db7b6a39da63"},
+    {file = "pywin32-300-cp35-cp35m-win_amd64.whl", hash = "sha256:350c5644775736351b77ba68da09a39c760d75d2467ecec37bd3c36a94fbed64"},
+    {file = "pywin32-300-cp36-cp36m-win32.whl", hash = "sha256:a3b4c48c852d4107e8a8ec980b76c94ce596ea66d60f7a697582ea9dce7e0db7"},
+    {file = "pywin32-300-cp36-cp36m-win_amd64.whl", hash = "sha256:27a30b887afbf05a9cbb05e3ffd43104a9b71ce292f64a635389dbad0ed1cd85"},
+    {file = "pywin32-300-cp37-cp37m-win32.whl", hash = "sha256:d7e8c7efc221f10d6400c19c32a031add1c4a58733298c09216f57b4fde110dc"},
+    {file = "pywin32-300-cp37-cp37m-win_amd64.whl", hash = "sha256:8151e4d7a19262d6694162d6da85d99a16f8b908949797fd99c83a0bfaf5807d"},
+    {file = "pywin32-300-cp38-cp38-win32.whl", hash = "sha256:fbb3b1b0fbd0b4fc2a3d1d81fe0783e30062c1abed1d17c32b7879d55858cfae"},
+    {file = "pywin32-300-cp38-cp38-win_amd64.whl", hash = "sha256:60a8fa361091b2eea27f15718f8eb7f9297e8d51b54dbc4f55f3d238093d5190"},
+    {file = "pywin32-300-cp39-cp39-win32.whl", hash = "sha256:638b68eea5cfc8def537e43e9554747f8dee786b090e47ead94bfdafdb0f2f50"},
+    {file = "pywin32-300-cp39-cp39-win_amd64.whl", hash = "sha256:b1609ce9bd5c411b81f941b246d683d6508992093203d4eb7f278f4ed1085c3f"},
 ]
 pywinpty = [
     {file = "pywinpty-0.5.7-cp27-cp27m-win32.whl", hash = "sha256:b358cb552c0f6baf790de375fab96524a0498c9df83489b8c23f7f08795e966b"},
@@ -2516,46 +2653,64 @@ pywinpty = [
     {file = "pywinpty-0.5.7.tar.gz", hash = "sha256:2d7e9c881638a72ffdca3f5417dd1563b60f603e1b43e5895674c2a1b01f95a0"},
 ]
 pyyaml = [
-    {file = "PyYAML-5.3.1-cp27-cp27m-win32.whl", hash = "sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f"},
-    {file = "PyYAML-5.3.1-cp27-cp27m-win_amd64.whl", hash = "sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76"},
-    {file = "PyYAML-5.3.1-cp35-cp35m-win32.whl", hash = "sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2"},
-    {file = "PyYAML-5.3.1-cp35-cp35m-win_amd64.whl", hash = "sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c"},
-    {file = "PyYAML-5.3.1-cp36-cp36m-win32.whl", hash = "sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2"},
-    {file = "PyYAML-5.3.1-cp36-cp36m-win_amd64.whl", hash = "sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648"},
-    {file = "PyYAML-5.3.1-cp37-cp37m-win32.whl", hash = "sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a"},
-    {file = "PyYAML-5.3.1-cp37-cp37m-win_amd64.whl", hash = "sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf"},
-    {file = "PyYAML-5.3.1-cp38-cp38-win32.whl", hash = "sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97"},
-    {file = "PyYAML-5.3.1-cp38-cp38-win_amd64.whl", hash = "sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee"},
-    {file = "PyYAML-5.3.1.tar.gz", hash = "sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d"},
+    {file = "PyYAML-5.4.1-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:3b2b1824fe7112845700f815ff6a489360226a5609b96ec2190a45e62a9fc922"},
+    {file = "PyYAML-5.4.1-cp27-cp27m-win32.whl", hash = "sha256:129def1b7c1bf22faffd67b8f3724645203b79d8f4cc81f674654d9902cb4393"},
+    {file = "PyYAML-5.4.1-cp27-cp27m-win_amd64.whl", hash = "sha256:4465124ef1b18d9ace298060f4eccc64b0850899ac4ac53294547536533800c8"},
+    {file = "PyYAML-5.4.1-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:bb4191dfc9306777bc594117aee052446b3fa88737cd13b7188d0e7aa8162185"},
+    {file = "PyYAML-5.4.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:6c78645d400265a062508ae399b60b8c167bf003db364ecb26dcab2bda048253"},
+    {file = "PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc"},
+    {file = "PyYAML-5.4.1-cp36-cp36m-win32.whl", hash = "sha256:3bd0e463264cf257d1ffd2e40223b197271046d09dadf73a0fe82b9c1fc385a5"},
+    {file = "PyYAML-5.4.1-cp36-cp36m-win_amd64.whl", hash = "sha256:e4fac90784481d221a8e4b1162afa7c47ed953be40d31ab4629ae917510051df"},
+    {file = "PyYAML-5.4.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:5accb17103e43963b80e6f837831f38d314a0495500067cb25afab2e8d7a4018"},
+    {file = "PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:e1d4970ea66be07ae37a3c2e48b5ec63f7ba6804bdddfdbd3cfd954d25a82e63"},
+    {file = "PyYAML-5.4.1-cp37-cp37m-win32.whl", hash = "sha256:dd5de0646207f053eb0d6c74ae45ba98c3395a571a2891858e87df7c9b9bd51b"},
+    {file = "PyYAML-5.4.1-cp37-cp37m-win_amd64.whl", hash = "sha256:08682f6b72c722394747bddaf0aa62277e02557c0fd1c42cb853016a38f8dedf"},
+    {file = "PyYAML-5.4.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:d2d9808ea7b4af864f35ea216be506ecec180628aced0704e34aca0b040ffe46"},
+    {file = "PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:8c1be557ee92a20f184922c7b6424e8ab6691788e6d86137c5d93c1a6ec1b8fb"},
+    {file = "PyYAML-5.4.1-cp38-cp38-win32.whl", hash = "sha256:fa5ae20527d8e831e8230cbffd9f8fe952815b2b7dae6ffec25318803a7528fc"},
+    {file = "PyYAML-5.4.1-cp38-cp38-win_amd64.whl", hash = "sha256:0f5f5786c0e09baddcd8b4b45f20a7b5d61a7e7e99846e3c799b05c7c53fa696"},
+    {file = "PyYAML-5.4.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:294db365efa064d00b8d1ef65d8ea2c3426ac366c0c4368d930bf1c5fb497f77"},
+    {file = "PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:74c1485f7707cf707a7aef42ef6322b8f97921bd89be2ab6317fd782c2d53183"},
+    {file = "PyYAML-5.4.1-cp39-cp39-win32.whl", hash = "sha256:49d4cdd9065b9b6e206d0595fee27a96b5dd22618e7520c33204a4a3239d5b10"},
+    {file = "PyYAML-5.4.1-cp39-cp39-win_amd64.whl", hash = "sha256:c20cfa2d49991c8b4147af39859b167664f2ad4561704ee74c1de03318e898db"},
+    {file = "PyYAML-5.4.1.tar.gz", hash = "sha256:607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e"},
 ]
 pyzmq = [
-    {file = "pyzmq-20.0.0-cp35-cp35m-macosx_10_9_intel.whl", hash = "sha256:523d542823cabb94065178090e05347bd204365f6e7cb260f0071c995d392fc2"},
-    {file = "pyzmq-20.0.0-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:225774a48ed7414c0395335e7123ef8c418dbcbe172caabdc2496133b03254c2"},
-    {file = "pyzmq-20.0.0-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:bc7dd697356b31389d5118b9bcdef3e8d8079e8181800c4e8d72dccd56e1ff68"},
-    {file = "pyzmq-20.0.0-cp35-cp35m-win32.whl", hash = "sha256:d81184489369ec325bd50ba1c935361e63f31f578430b9ad95471899361a8253"},
-    {file = "pyzmq-20.0.0-cp35-cp35m-win_amd64.whl", hash = "sha256:7113eb93dcd0a5750c65d123ed0099e036a3a3f2dcb48afedd025ffa125c983b"},
-    {file = "pyzmq-20.0.0-cp36-cp36m-macosx_10_9_intel.whl", hash = "sha256:b62113eeb9a0649cebed9b21fd578f3a0175ef214a2a91dcb7b31bbf55805295"},
-    {file = "pyzmq-20.0.0-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:f0beef935efe78a63c785bb21ed56c1c24448511383e3994927c8bb2caf5e714"},
-    {file = "pyzmq-20.0.0-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:46250789730489009fe139cbf576679557c070a6a3628077d09a4153d52fd381"},
-    {file = "pyzmq-20.0.0-cp36-cp36m-win32.whl", hash = "sha256:bf755905a7d30d2749079611b9a89924c1f2da2695dc09ce221f42122c9808e3"},
-    {file = "pyzmq-20.0.0-cp36-cp36m-win_amd64.whl", hash = "sha256:2742e380d186673eee6a570ef83d4568741945434ba36d92b98d36cdbfedbd44"},
-    {file = "pyzmq-20.0.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1e9b75a119606732023a305d1c214146c09a91f8116f6aff3e8b7d0a60b6f0ff"},
-    {file = "pyzmq-20.0.0-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:03638e46d486dd1c118e03c8bf9c634bdcae679600eac6573ae1e54906de7c2f"},
-    {file = "pyzmq-20.0.0-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:63ee08e35be72fdd7568065a249a5b5cf51a2e8ab6ee63cf9f73786fcb9e710b"},
-    {file = "pyzmq-20.0.0-cp37-cp37m-win32.whl", hash = "sha256:c95dda497a7c1b1e734b5e8353173ca5dd7b67784d8821d13413a97856588057"},
-    {file = "pyzmq-20.0.0-cp37-cp37m-win_amd64.whl", hash = "sha256:cc09c5cd1a4332611c8564d65e6a432dc6db3e10793d0254da9fa1e31d9ffd6d"},
-    {file = "pyzmq-20.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6e24907857c80dc67692e31f5bf3ad5bf483ee0142cec95b3d47e2db8c43bdda"},
-    {file = "pyzmq-20.0.0-cp38-cp38-manylinux1_i686.whl", hash = "sha256:895695be380f0f85d2e3ec5ccf68a93c92d45bd298567525ad5633071589872c"},
-    {file = "pyzmq-20.0.0-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:d92c7f41a53ece82b91703ea433c7d34143248cf0cead33aa11c5fc621c764bf"},
-    {file = "pyzmq-20.0.0-cp38-cp38-win32.whl", hash = "sha256:309d763d89ec1845c0e0fa14e1fb6558fd8c9ef05ed32baec27d7a8499cc7bb0"},
-    {file = "pyzmq-20.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:0e554fd390021edbe0330b67226325a820b0319c5b45e1b0a59bf22ccc36e793"},
-    {file = "pyzmq-20.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:cfa54a162a7b32641665e99b2c12084555afe9fc8fe80ec8b2f71a57320d10e1"},
-    {file = "pyzmq-20.0.0-cp39-cp39-manylinux1_i686.whl", hash = "sha256:5efe02bdcc5eafcac0aab531292294298f0ab8d28ed43be9e507d0e09173d1a4"},
-    {file = "pyzmq-20.0.0-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:0af84f34f27b5c6a0e906c648bdf46d4caebf9c8e6e16db0728f30a58141cad6"},
-    {file = "pyzmq-20.0.0-cp39-cp39-win32.whl", hash = "sha256:c63fafd2556d218368c51d18588f8e6f8d86d09d493032415057faf6de869b34"},
-    {file = "pyzmq-20.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:f110a4d3f8f01209eec304ed542f6c8054cce9b0f16dfe3d571e57c290e4e133"},
-    {file = "pyzmq-20.0.0-pp36-pypy36_pp73-macosx_10_9_x86_64.whl", hash = "sha256:4d9259a5eb3f71abbaf61f165cacf42240bfeea3783bebd8255341abdfe206f1"},
-    {file = "pyzmq-20.0.0.tar.gz", hash = "sha256:824ad5888331aadeac772bce27e1c2fbcab82fade92edbd234542c4e12f0dca9"},
+    {file = "pyzmq-22.0.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:c0cde362075ee8f3d2b0353b283e203c2200243b5a15d5c5c03b78112a17e7d4"},
+    {file = "pyzmq-22.0.3-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:ff1ea14075bbddd6f29bf6beb8a46d0db779bcec6b9820909584081ec119f8fd"},
+    {file = "pyzmq-22.0.3-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:26380487eae4034d6c2a3fb8d0f2dff6dd0d9dd711894e8d25aa2d1938950a33"},
+    {file = "pyzmq-22.0.3-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:3e29f9cf85a40d521d048b55c63f59d6c772ac1c4bf51cdfc23b62a62e377c33"},
+    {file = "pyzmq-22.0.3-cp36-cp36m-win32.whl", hash = "sha256:4f34a173f813b38b83f058e267e30465ed64b22cd0cf6bad21148d3fa718f9bb"},
+    {file = "pyzmq-22.0.3-cp36-cp36m-win_amd64.whl", hash = "sha256:30df70f81fe210506aa354d7fd486a39b87d9f7f24c3d3f4f698ec5d96b8c084"},
+    {file = "pyzmq-22.0.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:7026f0353977431fc884abd4ac28268894bd1a780ba84bb266d470b0ec26d2ed"},
+    {file = "pyzmq-22.0.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:6d4163704201fff0f3ab0cd5d7a0ea1514ecfffd3926d62ec7e740a04d2012c7"},
+    {file = "pyzmq-22.0.3-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:763c175294d861869f18eb42901d500eda7d3fa4565f160b3b2fd2678ea0ebab"},
+    {file = "pyzmq-22.0.3-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:61e4bb6cd60caf1abcd796c3f48395e22c5b486eeca6f3a8797975c57d94b03e"},
+    {file = "pyzmq-22.0.3-cp37-cp37m-win32.whl", hash = "sha256:b25e5d339550a850f7e919fe8cb4c8eabe4c917613db48dab3df19bfb9a28969"},
+    {file = "pyzmq-22.0.3-cp37-cp37m-win_amd64.whl", hash = "sha256:3ef50d74469b03725d781a2a03c57537d86847ccde587130fe35caafea8f75c6"},
+    {file = "pyzmq-22.0.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:60e63577b85055e4cc43892fecd877b86695ee3ef12d5d10a3c5d6e77a7cc1a3"},
+    {file = "pyzmq-22.0.3-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:f5831eff6b125992ec65d973f5151c48003b6754030094723ac4c6e80a97c8c4"},
+    {file = "pyzmq-22.0.3-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:9221783dacb419604d5345d0e097bddef4459a9a95322de6c306bf1d9896559f"},
+    {file = "pyzmq-22.0.3-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:b62ea18c0458a65ccd5be90f276f7a5a3f26a6dea0066d948ce2fa896051420f"},
+    {file = "pyzmq-22.0.3-cp38-cp38-win32.whl", hash = "sha256:81e7df0da456206201e226491aa1fc449da85328bf33bbeec2c03bb3a9f18324"},
+    {file = "pyzmq-22.0.3-cp38-cp38-win_amd64.whl", hash = "sha256:f52070871a0fd90a99130babf21f8af192304ec1e995bec2a9533efc21ea4452"},
+    {file = "pyzmq-22.0.3-cp39-cp39-macosx_10_15_universal2.whl", hash = "sha256:c5e29fe4678f97ce429f076a2a049a3d0b2660ada8f2c621e5dc9939426056dd"},
+    {file = "pyzmq-22.0.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:d18ddc6741b51f3985978f2fda57ddcdae359662d7a6b395bc8ff2292fca14bd"},
+    {file = "pyzmq-22.0.3-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:4231943514812dfb74f44eadcf85e8dd8cf302b4d0bce450ce1357cac88dbfdc"},
+    {file = "pyzmq-22.0.3-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:23a74de4b43c05c3044aeba0d1f3970def8f916151a712a3ac1e5cd9c0bc2902"},
+    {file = "pyzmq-22.0.3-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:532af3e6dddea62d9c49062ece5add998c9823c2419da943cf95589f56737de0"},
+    {file = "pyzmq-22.0.3-cp39-cp39-win32.whl", hash = "sha256:33acd2b9790818b9d00526135acf12790649d8d34b2b04d64558b469c9d86820"},
+    {file = "pyzmq-22.0.3-cp39-cp39-win_amd64.whl", hash = "sha256:a558c5bc89d56d7253187dccc4e81b5bb0eac5ae9511eb4951910a1245d04622"},
+    {file = "pyzmq-22.0.3-pp36-pypy36_pp73-macosx_10_9_x86_64.whl", hash = "sha256:581787c62eaa0e0db6c5413cedc393ebbadac6ddfd22e1cf9a60da23c4f1a4b2"},
+    {file = "pyzmq-22.0.3-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:38e3dca75d81bec4f2defa14b0a65b74545812bb519a8e89c8df96bbf4639356"},
+    {file = "pyzmq-22.0.3-pp36-pypy36_pp73-win32.whl", hash = "sha256:2f971431aaebe0a8b54ac018e041c2f0b949a43745444e4dadcc80d0f0ef8457"},
+    {file = "pyzmq-22.0.3-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:da7d4d4c778c86b60949d17531e60c54ed3726878de8a7f8a6d6e7f8cc8c3205"},
+    {file = "pyzmq-22.0.3-pp37-pypy37_pp73-manylinux2010_x86_64.whl", hash = "sha256:13465c1ff969cab328bc92f7015ce3843f6e35f8871ad79d236e4fbc85dbe4cb"},
+    {file = "pyzmq-22.0.3-pp37-pypy37_pp73-win32.whl", hash = "sha256:279cc9b51db48bec2db146f38e336049ac5a59e5f12fb3a8ad864e238c1c62e3"},
+    {file = "pyzmq-22.0.3.tar.gz", hash = "sha256:f7f63ce127980d40f3e6a5fdb87abf17ce1a7c2bd8bf2c7560e1bbce8ab1f92d"},
+]
+redshift-connector = [
+    {file = "redshift_connector-2.0.876-py3-none-any.whl", hash = "sha256:32ce26ef141874cf3d968744d50dc3ba39ee5597b8b16d4423b4f4b356ed7591"},
 ]
 regex = [
     {file = "regex-2020.11.13-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:8b882a78c320478b12ff024e81dc7d43c1462aa4a3341c754ee65d857a521f85"},
@@ -2614,12 +2769,12 @@ rfc3986 = [
     {file = "rfc3986-1.4.0.tar.gz", hash = "sha256:112398da31a3344dc25dbf477d8df6cb34f9278a94fee2625d89e4514be8bb9d"},
 ]
 rsa = [
-    {file = "rsa-4.6-py3-none-any.whl", hash = "sha256:6166864e23d6b5195a5cfed6cd9fed0fe774e226d8f854fcb23b7bbef0350233"},
-    {file = "rsa-4.6.tar.gz", hash = "sha256:109ea5a66744dd859bf16fe904b8d8b627adafb9408753161e766a92e7d681fa"},
+    {file = "rsa-4.7.2-py3-none-any.whl", hash = "sha256:78f9a9bf4e7be0c5ded4583326e7461e3a3c5aae24073648b4bdfa797d78c9d2"},
+    {file = "rsa-4.7.2.tar.gz", hash = "sha256:9d689e6ca1b3038bc82bf8d23e944b6b6037bc02301a574935b2dd946e0353b9"},
 ]
 "ruamel.yaml" = [
-    {file = "ruamel.yaml-0.16.12-py2.py3-none-any.whl", hash = "sha256:012b9470a0ea06e4e44e99e7920277edf6b46eee0232a04487ea73a7386340a5"},
-    {file = "ruamel.yaml-0.16.12.tar.gz", hash = "sha256:076cc0bc34f1966d920a49f18b52b6ad559fbe656a0748e3535cf7b3f29ebf9e"},
+    {file = "ruamel.yaml-0.16.13-py2.py3-none-any.whl", hash = "sha256:64b06e7873eb8e1125525ecef7345447d786368cadca92a7cd9b59eae62e95a3"},
+    {file = "ruamel.yaml-0.16.13.tar.gz", hash = "sha256:bb48c514222702878759a05af96f4b7ecdba9b33cd4efcf25c86b882cef3a942"},
 ]
 "ruamel.yaml.clib" = [
     {file = "ruamel.yaml.clib-0.2.2-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:28116f204103cb3a108dfd37668f20abe6e3cafd0d3fd40dba126c732457b3cc"},
@@ -2629,61 +2784,56 @@ rsa = [
     {file = "ruamel.yaml.clib-0.2.2-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:73b3d43e04cc4b228fa6fa5d796409ece6fcb53a6c270eb2048109cbcbc3b9c2"},
     {file = "ruamel.yaml.clib-0.2.2-cp35-cp35m-macosx_10_6_intel.whl", hash = "sha256:53b9dd1abd70e257a6e32f934ebc482dac5edb8c93e23deb663eac724c30b026"},
     {file = "ruamel.yaml.clib-0.2.2-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:839dd72545ef7ba78fd2aa1a5dd07b33696adf3e68fae7f31327161c1093001b"},
-    {file = "ruamel.yaml.clib-0.2.2-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:1236df55e0f73cd138c0eca074ee086136c3f16a97c2ac719032c050f7e0622f"},
     {file = "ruamel.yaml.clib-0.2.2-cp35-cp35m-win32.whl", hash = "sha256:b1e981fe1aff1fd11627f531524826a4dcc1f26c726235a52fcb62ded27d150f"},
     {file = "ruamel.yaml.clib-0.2.2-cp35-cp35m-win_amd64.whl", hash = "sha256:4e52c96ca66de04be42ea2278012a2342d89f5e82b4512fb6fb7134e377e2e62"},
     {file = "ruamel.yaml.clib-0.2.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:a873e4d4954f865dcb60bdc4914af7eaae48fb56b60ed6daa1d6251c72f5337c"},
     {file = "ruamel.yaml.clib-0.2.2-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:ab845f1f51f7eb750a78937be9f79baea4a42c7960f5a94dde34e69f3cce1988"},
-    {file = "ruamel.yaml.clib-0.2.2-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:2fd336a5c6415c82e2deb40d08c222087febe0aebe520f4d21910629018ab0f3"},
     {file = "ruamel.yaml.clib-0.2.2-cp36-cp36m-win32.whl", hash = "sha256:e9f7d1d8c26a6a12c23421061f9022bb62704e38211fe375c645485f38df34a2"},
     {file = "ruamel.yaml.clib-0.2.2-cp36-cp36m-win_amd64.whl", hash = "sha256:2602e91bd5c1b874d6f93d3086f9830f3e907c543c7672cf293a97c3fabdcd91"},
     {file = "ruamel.yaml.clib-0.2.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:44c7b0498c39f27795224438f1a6be6c5352f82cb887bc33d962c3a3acc00df6"},
     {file = "ruamel.yaml.clib-0.2.2-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:8e8fd0a22c9d92af3a34f91e8a2594eeb35cba90ab643c5e0e643567dc8be43e"},
-    {file = "ruamel.yaml.clib-0.2.2-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:75f0ee6839532e52a3a53f80ce64925ed4aed697dd3fa890c4c918f3304bd4f4"},
     {file = "ruamel.yaml.clib-0.2.2-cp37-cp37m-win32.whl", hash = "sha256:464e66a04e740d754170be5e740657a3b3b6d2bcc567f0c3437879a6e6087ff6"},
     {file = "ruamel.yaml.clib-0.2.2-cp37-cp37m-win_amd64.whl", hash = "sha256:52ae5739e4b5d6317b52f5b040b1b6639e8af68a5b8fd606a8b08658fbd0cab5"},
     {file = "ruamel.yaml.clib-0.2.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:4df5019e7783d14b79217ad9c56edf1ba7485d614ad5a385d1b3c768635c81c0"},
     {file = "ruamel.yaml.clib-0.2.2-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:5254af7d8bdf4d5484c089f929cb7f5bafa59b4f01d4f48adda4be41e6d29f99"},
-    {file = "ruamel.yaml.clib-0.2.2-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:8be05be57dc5c7b4a0b24edcaa2f7275866d9c907725226cdde46da09367d923"},
     {file = "ruamel.yaml.clib-0.2.2-cp38-cp38-win32.whl", hash = "sha256:74161d827407f4db9072011adcfb825b5258a5ccb3d2cd518dd6c9edea9e30f1"},
     {file = "ruamel.yaml.clib-0.2.2-cp38-cp38-win_amd64.whl", hash = "sha256:058a1cc3df2a8aecc12f983a48bda99315cebf55a3b3a5463e37bb599b05727b"},
     {file = "ruamel.yaml.clib-0.2.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:c6ac7e45367b1317e56f1461719c853fd6825226f45b835df7436bb04031fd8a"},
     {file = "ruamel.yaml.clib-0.2.2-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:b4b0d31f2052b3f9f9b5327024dc629a253a83d8649d4734ca7f35b60ec3e9e5"},
-    {file = "ruamel.yaml.clib-0.2.2-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:1f8c0a4577c0e6c99d208de5c4d3fd8aceed9574bb154d7a2b21c16bb924154c"},
-    {file = "ruamel.yaml.clib-0.2.2-cp39-cp39-win32.whl", hash = "sha256:46d6d20815064e8bb023ea8628cfb7402c0f0e83de2c2227a88097e239a7dffd"},
-    {file = "ruamel.yaml.clib-0.2.2-cp39-cp39-win_amd64.whl", hash = "sha256:6c0a5dc52fc74eb87c67374a4e554d4761fd42a4d01390b7e868b30d21f4b8bb"},
     {file = "ruamel.yaml.clib-0.2.2.tar.gz", hash = "sha256:2d24bd98af676f4990c4d715bcdc2a60b19c56a3fb3a763164d2d8ca0e806ba7"},
 ]
+s3fs = [
+    {file = "s3fs-0.4.2-py3-none-any.whl", hash = "sha256:91c1dfb45e5217bd441a7a560946fe865ced6225ff7eb0fb459fe6e601a95ed3"},
+    {file = "s3fs-0.4.2.tar.gz", hash = "sha256:2ca5de8dc18ad7ad350c0bd01aef0406aa5d0fff78a561f0f710f9d9858abdd0"},
+]
 s3transfer = [
-    {file = "s3transfer-0.3.3-py2.py3-none-any.whl", hash = "sha256:2482b4259524933a022d59da830f51bd746db62f047d6eb213f2f8855dcb8a13"},
-    {file = "s3transfer-0.3.3.tar.gz", hash = "sha256:921a37e2aefc64145e7b73d50c71bb4f26f46e4c9f414dc648c6245ff92cf7db"},
+    {file = "s3transfer-0.3.4-py2.py3-none-any.whl", hash = "sha256:1e28620e5b444652ed752cf87c7e0cb15b0e578972568c6609f0f18212f259ed"},
+    {file = "s3transfer-0.3.4.tar.gz", hash = "sha256:7fdddb4f22275cf1d32129e21f056337fd2a80b6ccef1664528145b72c49e6d2"},
 ]
 scipy = [
-    {file = "scipy-1.5.4-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:4f12d13ffbc16e988fa40809cbbd7a8b45bc05ff6ea0ba8e3e41f6f4db3a9e47"},
-    {file = "scipy-1.5.4-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:a254b98dbcc744c723a838c03b74a8a34c0558c9ac5c86d5561703362231107d"},
-    {file = "scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:368c0f69f93186309e1b4beb8e26d51dd6f5010b79264c0f1e9ca00cd92ea8c9"},
-    {file = "scipy-1.5.4-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:4598cf03136067000855d6b44d7a1f4f46994164bcd450fb2c3d481afc25dd06"},
-    {file = "scipy-1.5.4-cp36-cp36m-win32.whl", hash = "sha256:e98d49a5717369d8241d6cf33ecb0ca72deee392414118198a8e5b4c35c56340"},
-    {file = "scipy-1.5.4-cp36-cp36m-win_amd64.whl", hash = "sha256:65923bc3809524e46fb7eb4d6346552cbb6a1ffc41be748535aa502a2e3d3389"},
-    {file = "scipy-1.5.4-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:9ad4fcddcbf5dc67619379782e6aeef41218a79e17979aaed01ed099876c0e62"},
-    {file = "scipy-1.5.4-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:f87b39f4d69cf7d7529d7b1098cb712033b17ea7714aed831b95628f483fd012"},
-    {file = "scipy-1.5.4-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:25b241034215247481f53355e05f9e25462682b13bd9191359075682adcd9554"},
-    {file = "scipy-1.5.4-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:fa789583fc94a7689b45834453fec095245c7e69c58561dc159b5d5277057e4c"},
-    {file = "scipy-1.5.4-cp37-cp37m-win32.whl", hash = "sha256:d6d25c41a009e3c6b7e757338948d0076ee1dd1770d1c09ec131f11946883c54"},
-    {file = "scipy-1.5.4-cp37-cp37m-win_amd64.whl", hash = "sha256:2c872de0c69ed20fb1a9b9cf6f77298b04a26f0b8720a5457be08be254366c6e"},
-    {file = "scipy-1.5.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:e360cb2299028d0b0d0f65a5c5e51fc16a335f1603aa2357c25766c8dab56938"},
-    {file = "scipy-1.5.4-cp38-cp38-manylinux1_i686.whl", hash = "sha256:3397c129b479846d7eaa18f999369a24322d008fac0782e7828fa567358c36ce"},
-    {file = "scipy-1.5.4-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:168c45c0c32e23f613db7c9e4e780bc61982d71dcd406ead746c7c7c2f2004ce"},
-    {file = "scipy-1.5.4-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:213bc59191da2f479984ad4ec39406bf949a99aba70e9237b916ce7547b6ef42"},
-    {file = "scipy-1.5.4-cp38-cp38-win32.whl", hash = "sha256:634568a3018bc16a83cda28d4f7aed0d803dd5618facb36e977e53b2df868443"},
-    {file = "scipy-1.5.4-cp38-cp38-win_amd64.whl", hash = "sha256:b03c4338d6d3d299e8ca494194c0ae4f611548da59e3c038813f1a43976cb437"},
-    {file = "scipy-1.5.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:3d5db5d815370c28d938cf9b0809dade4acf7aba57eaf7ef733bfedc9b2474c4"},
-    {file = "scipy-1.5.4-cp39-cp39-manylinux1_i686.whl", hash = "sha256:6b0ceb23560f46dd236a8ad4378fc40bad1783e997604ba845e131d6c680963e"},
-    {file = "scipy-1.5.4-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:ed572470af2438b526ea574ff8f05e7f39b44ac37f712105e57fc4d53a6fb660"},
-    {file = "scipy-1.5.4-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:8c8d6ca19c8497344b810b0b0344f8375af5f6bb9c98bd42e33f747417ab3f57"},
-    {file = "scipy-1.5.4-cp39-cp39-win32.whl", hash = "sha256:d84cadd7d7998433334c99fa55bcba0d8b4aeff0edb123b2a1dfcface538e474"},
-    {file = "scipy-1.5.4-cp39-cp39-win_amd64.whl", hash = "sha256:cc1f78ebc982cd0602c9a7615d878396bec94908db67d4ecddca864d049112f2"},
-    {file = "scipy-1.5.4.tar.gz", hash = "sha256:4a453d5e5689de62e5d38edf40af3f17560bfd63c9c5bd228c18c1f99afa155b"},
+    {file = "scipy-1.6.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a15a1f3fc0abff33e792d6049161b7795909b40b97c6cc2934ed54384017ab76"},
+    {file = "scipy-1.6.1-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:e79570979ccdc3d165456dd62041d9556fb9733b86b4b6d818af7a0afc15f092"},
+    {file = "scipy-1.6.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:a423533c55fec61456dedee7b6ee7dce0bb6bfa395424ea374d25afa262be261"},
+    {file = "scipy-1.6.1-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:33d6b7df40d197bdd3049d64e8e680227151673465e5d85723b3b8f6b15a6ced"},
+    {file = "scipy-1.6.1-cp37-cp37m-win32.whl", hash = "sha256:6725e3fbb47da428794f243864f2297462e9ee448297c93ed1dcbc44335feb78"},
+    {file = "scipy-1.6.1-cp37-cp37m-win_amd64.whl", hash = "sha256:5fa9c6530b1661f1370bcd332a1e62ca7881785cc0f80c0d559b636567fab63c"},
+    {file = "scipy-1.6.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:bd50daf727f7c195e26f27467c85ce653d41df4358a25b32434a50d8870fc519"},
+    {file = "scipy-1.6.1-cp38-cp38-manylinux1_i686.whl", hash = "sha256:f46dd15335e8a320b0fb4685f58b7471702234cba8bb3442b69a3e1dc329c345"},
+    {file = "scipy-1.6.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:0e5b0ccf63155d90da576edd2768b66fb276446c371b73841e3503be1d63fb5d"},
+    {file = "scipy-1.6.1-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:2481efbb3740977e3c831edfd0bd9867be26387cacf24eb5e366a6a374d3d00d"},
+    {file = "scipy-1.6.1-cp38-cp38-win32.whl", hash = "sha256:68cb4c424112cd4be886b4d979c5497fba190714085f46b8ae67a5e4416c32b4"},
+    {file = "scipy-1.6.1-cp38-cp38-win_amd64.whl", hash = "sha256:5f331eeed0297232d2e6eea51b54e8278ed8bb10b099f69c44e2558c090d06bf"},
+    {file = "scipy-1.6.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:0c8a51d33556bf70367452d4d601d1742c0e806cd0194785914daf19775f0e67"},
+    {file = "scipy-1.6.1-cp39-cp39-manylinux1_i686.whl", hash = "sha256:83bf7c16245c15bc58ee76c5418e46ea1811edcc2e2b03041b804e46084ab627"},
+    {file = "scipy-1.6.1-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:794e768cc5f779736593046c9714e0f3a5940bc6dcc1dba885ad64cbfb28e9f0"},
+    {file = "scipy-1.6.1-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:5da5471aed911fe7e52b86bf9ea32fb55ae93e2f0fac66c32e58897cfb02fa07"},
+    {file = "scipy-1.6.1-cp39-cp39-win32.whl", hash = "sha256:8e403a337749ed40af60e537cc4d4c03febddcc56cd26e774c9b1b600a70d3e4"},
+    {file = "scipy-1.6.1-cp39-cp39-win_amd64.whl", hash = "sha256:a5193a098ae9f29af283dcf0041f762601faf2e595c0db1da929875b7570353f"},
+    {file = "scipy-1.6.1.tar.gz", hash = "sha256:c4fceb864890b6168e79b0e714c585dbe2fd4222768ee90bc1aa0f8218691b11"},
+]
+scramp = [
+    {file = "scramp-1.2.0-py3-none-any.whl", hash = "sha256:74815c25aad1fe0b5fb994e96c3de63e8695164358a80138352aaadfa4760350"},
+    {file = "scramp-1.2.0.tar.gz", hash = "sha256:d6865ed1d135ddb124a619d7cd3a5b505f69a7c92e248024dd7e48bc77752af5"},
 ]
 send2trash = [
     {file = "Send2Trash-1.5.0-py3-none-any.whl", hash = "sha256:f1691922577b6fa12821234aeb57599d887c4900b9ca537948d2dac34aea888b"},
@@ -2743,63 +2893,23 @@ six = [
     {file = "six-1.15.0-py2.py3-none-any.whl", hash = "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"},
     {file = "six-1.15.0.tar.gz", hash = "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259"},
 ]
-sqlalchemy = [
-    {file = "SQLAlchemy-1.3.20-cp27-cp27m-macosx_10_14_x86_64.whl", hash = "sha256:bad73f9888d30f9e1d57ac8829f8a12091bdee4949b91db279569774a866a18e"},
-    {file = "SQLAlchemy-1.3.20-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:e32e3455db14602b6117f0f422f46bc297a3853ae2c322ecd1e2c4c04daf6ed5"},
-    {file = "SQLAlchemy-1.3.20-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:5cdfe54c1e37279dc70d92815464b77cd8ee30725adc9350f06074f91dbfeed2"},
-    {file = "SQLAlchemy-1.3.20-cp27-cp27m-win32.whl", hash = "sha256:2e9bd5b23bba8ae8ce4219c9333974ff5e103c857d9ff0e4b73dc4cb244c7d86"},
-    {file = "SQLAlchemy-1.3.20-cp27-cp27m-win_amd64.whl", hash = "sha256:5d92c18458a4aa27497a986038d5d797b5279268a2de303cd00910658e8d149c"},
-    {file = "SQLAlchemy-1.3.20-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:53fd857c6c8ffc0aa6a5a3a2619f6a74247e42ec9e46b836a8ffa4abe7aab327"},
-    {file = "SQLAlchemy-1.3.20-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:0a92745bb1ebbcb3985ed7bda379b94627f0edbc6c82e9e4bac4fb5647ae609a"},
-    {file = "SQLAlchemy-1.3.20-cp35-cp35m-macosx_10_14_x86_64.whl", hash = "sha256:b6f036ecc017ec2e2cc2a40615b41850dc7aaaea6a932628c0afc73ab98ba3fb"},
-    {file = "SQLAlchemy-1.3.20-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:3aa6d45e149a16aa1f0c46816397e12313d5e37f22205c26e06975e150ffcf2a"},
-    {file = "SQLAlchemy-1.3.20-cp35-cp35m-manylinux2010_x86_64.whl", hash = "sha256:ed53209b5f0f383acb49a927179fa51a6e2259878e164273ebc6815f3a752465"},
-    {file = "SQLAlchemy-1.3.20-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:d3b709d64b5cf064972b3763b47139e4a0dc4ae28a36437757f7663f67b99710"},
-    {file = "SQLAlchemy-1.3.20-cp35-cp35m-win32.whl", hash = "sha256:950f0e17ffba7a7ceb0dd056567bc5ade22a11a75920b0e8298865dc28c0eff6"},
-    {file = "SQLAlchemy-1.3.20-cp35-cp35m-win_amd64.whl", hash = "sha256:8dcbf377529a9af167cbfc5b8acec0fadd7c2357fc282a1494c222d3abfc9629"},
-    {file = "SQLAlchemy-1.3.20-cp36-cp36m-macosx_10_14_x86_64.whl", hash = "sha256:0157c269701d88f5faf1fa0e4560e4d814f210c01a5b55df3cab95e9346a8bcc"},
-    {file = "SQLAlchemy-1.3.20-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:7cd40cb4bc50d9e87b3540b23df6e6b24821ba7e1f305c1492b0806c33dbdbec"},
-    {file = "SQLAlchemy-1.3.20-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:c092fe282de83d48e64d306b4bce03114859cdbfe19bf8a978a78a0d44ddadb1"},
-    {file = "SQLAlchemy-1.3.20-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:166917a729b9226decff29416f212c516227c2eb8a9c9f920d69ced24e30109f"},
-    {file = "SQLAlchemy-1.3.20-cp36-cp36m-win32.whl", hash = "sha256:632b32183c0cb0053194a4085c304bc2320e5299f77e3024556fa2aa395c2a8b"},
-    {file = "SQLAlchemy-1.3.20-cp36-cp36m-win_amd64.whl", hash = "sha256:bbc58fca72ce45a64bb02b87f73df58e29848b693869e58bd890b2ddbb42d83b"},
-    {file = "SQLAlchemy-1.3.20-cp37-cp37m-macosx_10_14_x86_64.whl", hash = "sha256:b15002b9788ffe84e42baffc334739d3b68008a973d65fad0a410ca5d0531980"},
-    {file = "SQLAlchemy-1.3.20-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:9e379674728f43a0cd95c423ac0e95262500f9bfd81d33b999daa8ea1756d162"},
-    {file = "SQLAlchemy-1.3.20-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:2b5dafed97f778e9901b79cc01b88d39c605e0545b4541f2551a2fd785adc15b"},
-    {file = "SQLAlchemy-1.3.20-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:fcdb3755a7c355bc29df1b5e6fb8226d5c8b90551d202d69d0076a8a5649d68b"},
-    {file = "SQLAlchemy-1.3.20-cp37-cp37m-win32.whl", hash = "sha256:bca4d367a725694dae3dfdc86cf1d1622b9f414e70bd19651f5ac4fb3aa96d61"},
-    {file = "SQLAlchemy-1.3.20-cp37-cp37m-win_amd64.whl", hash = "sha256:f605f348f4e6a2ba00acb3399c71d213b92f27f2383fc4abebf7a37368c12142"},
-    {file = "SQLAlchemy-1.3.20-cp38-cp38-macosx_10_14_x86_64.whl", hash = "sha256:84f0ac4a09971536b38cc5d515d6add7926a7e13baa25135a1dbb6afa351a376"},
-    {file = "SQLAlchemy-1.3.20-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:2909dffe5c9a615b7e6c92d1ac2d31e3026dc436440a4f750f4749d114d88ceb"},
-    {file = "SQLAlchemy-1.3.20-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:c3ab23ee9674336654bf9cac30eb75ac6acb9150dc4b1391bec533a7a4126471"},
-    {file = "SQLAlchemy-1.3.20-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:009e8388d4d551a2107632921320886650b46332f61dc935e70c8bcf37d8e0d6"},
-    {file = "SQLAlchemy-1.3.20-cp38-cp38-win32.whl", hash = "sha256:bf53d8dddfc3e53a5bda65f7f4aa40fae306843641e3e8e701c18a5609471edf"},
-    {file = "SQLAlchemy-1.3.20-cp38-cp38-win_amd64.whl", hash = "sha256:7c735c7a6db8ee9554a3935e741cf288f7dcbe8706320251eb38c412e6a4281d"},
-    {file = "SQLAlchemy-1.3.20-cp39-cp39-macosx_10_14_x86_64.whl", hash = "sha256:4bdbdb8ca577c6c366d15791747c1de6ab14529115a2eb52774240c412a7b403"},
-    {file = "SQLAlchemy-1.3.20-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:ce64a44c867d128ab8e675f587aae7f61bd2db836a3c4ba522d884cd7c298a77"},
-    {file = "SQLAlchemy-1.3.20-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:be41d5de7a8e241864189b7530ca4aaf56a5204332caa70555c2d96379e18079"},
-    {file = "SQLAlchemy-1.3.20-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:1f5f369202912be72fdf9a8f25067a5ece31a2b38507bb869306f173336348da"},
-    {file = "SQLAlchemy-1.3.20-cp39-cp39-win32.whl", hash = "sha256:0cca1844ba870e81c03633a99aa3dc62256fb96323431a5dec7d4e503c26372d"},
-    {file = "SQLAlchemy-1.3.20-cp39-cp39-win_amd64.whl", hash = "sha256:d05cef4a164b44ffda58200efcb22355350979e000828479971ebca49b82ddb1"},
-    {file = "SQLAlchemy-1.3.20.tar.gz", hash = "sha256:d2f25c7f410338d31666d7ddedfa67570900e248b940d186b48461bd4e5569a1"},
-]
-sqlalchemy-redshift = [
-    {file = "sqlalchemy-redshift-0.8.1.tar.gz", hash = "sha256:588bf1e54ca41a0411c2f8f4a3ca14026b637e867dfdabb60b3572319a5b1ce5"},
-    {file = "sqlalchemy_redshift-0.8.1-py2.py3-none-any.whl", hash = "sha256:22518011c50f2fbe7e06d804aec90b693175b540152dd4a2c8c7acd78618fafe"},
+soupsieve = [
+    {file = "soupsieve-2.2-py3-none-any.whl", hash = "sha256:d3a5ea5b350423f47d07639f74475afedad48cf41c0ad7a82ca13a3928af34f6"},
+    {file = "soupsieve-2.2.tar.gz", hash = "sha256:407fa1e8eb3458d1b5614df51d9651a1180ea5fedf07feb46e45d7e25e6d6cdd"},
 ]
 stringcase = [
     {file = "stringcase-1.2.0.tar.gz", hash = "sha256:48a06980661908efe8d9d34eab2b6c13aefa2163b3ced26972902e3bdfd87008"},
 ]
 tabulate = [
-    {file = "tabulate-0.8.7-py3-none-any.whl", hash = "sha256:ac64cb76d53b1231d364babcd72abbb16855adac7de6665122f97b593f1eb2ba"},
-    {file = "tabulate-0.8.7.tar.gz", hash = "sha256:db2723a20d04bcda8522165c73eea7c300eda74e0ce852d9022e0159d7895007"},
+    {file = "tabulate-0.8.9-py3-none-any.whl", hash = "sha256:d7c013fe7abbc5e491394e10fa845f8f32fe54f8dc60c6622c6cf482d25d47e4"},
+    {file = "tabulate-0.8.9.tar.gz", hash = "sha256:eb1d13f25760052e8931f2ef80aaf6045a6cceb47514db8beab24cded16f13a7"},
 ]
 termcolor = [
     {file = "termcolor-1.1.0.tar.gz", hash = "sha256:1d6d69ce66211143803fbc56652b41d73b4a400a2891d7bf7a1cdf4c02de613b"},
 ]
 terminado = [
-    {file = "terminado-0.9.1-py3-none-any.whl", hash = "sha256:c55f025beb06c2e2669f7ba5a04f47bb3304c30c05842d4981d8f0fc9ab3b4e3"},
-    {file = "terminado-0.9.1.tar.gz", hash = "sha256:3da72a155b807b01c9e8a5babd214e052a0a45a975751da3521a1c3381ce6d76"},
+    {file = "terminado-0.9.2-py3-none-any.whl", hash = "sha256:23a053e06b22711269563c8bb96b36a036a86be8b5353e85e804f89b84aaa23f"},
+    {file = "terminado-0.9.2.tar.gz", hash = "sha256:89e6d94b19e4bc9dce0ffd908dfaf55cc78a9bf735934e915a4a96f65ac9704c"},
 ]
 testpath = [
     {file = "testpath-0.4.4-py2.py3-none-any.whl", hash = "sha256:bfcf9411ef4bf3db7579063e0546938b1edda3d69f4e1fb8756991f5951f85d4"},
@@ -2865,36 +2975,36 @@ traitlets = [
     {file = "traitlets-5.0.5.tar.gz", hash = "sha256:178f4ce988f69189f7e523337a3e11d91c786ded9360174a3d9ca83e79bc5396"},
 ]
 typed-ast = [
-    {file = "typed_ast-1.4.1-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3"},
-    {file = "typed_ast-1.4.1-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb"},
-    {file = "typed_ast-1.4.1-cp35-cp35m-win32.whl", hash = "sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919"},
-    {file = "typed_ast-1.4.1-cp35-cp35m-win_amd64.whl", hash = "sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01"},
-    {file = "typed_ast-1.4.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75"},
-    {file = "typed_ast-1.4.1-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652"},
-    {file = "typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"},
-    {file = "typed_ast-1.4.1-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:fcf135e17cc74dbfbc05894ebca928ffeb23d9790b3167a674921db19082401f"},
-    {file = "typed_ast-1.4.1-cp36-cp36m-win32.whl", hash = "sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1"},
-    {file = "typed_ast-1.4.1-cp36-cp36m-win_amd64.whl", hash = "sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa"},
-    {file = "typed_ast-1.4.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614"},
-    {file = "typed_ast-1.4.1-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41"},
-    {file = "typed_ast-1.4.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b"},
-    {file = "typed_ast-1.4.1-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:f208eb7aff048f6bea9586e61af041ddf7f9ade7caed625742af423f6bae3298"},
-    {file = "typed_ast-1.4.1-cp37-cp37m-win32.whl", hash = "sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe"},
-    {file = "typed_ast-1.4.1-cp37-cp37m-win_amd64.whl", hash = "sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355"},
-    {file = "typed_ast-1.4.1-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6"},
-    {file = "typed_ast-1.4.1-cp38-cp38-manylinux1_i686.whl", hash = "sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907"},
-    {file = "typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d"},
-    {file = "typed_ast-1.4.1-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:7e4c9d7658aaa1fc80018593abdf8598bf91325af6af5cce4ce7c73bc45ea53d"},
-    {file = "typed_ast-1.4.1-cp38-cp38-win32.whl", hash = "sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c"},
-    {file = "typed_ast-1.4.1-cp38-cp38-win_amd64.whl", hash = "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4"},
-    {file = "typed_ast-1.4.1-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34"},
-    {file = "typed_ast-1.4.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:92c325624e304ebf0e025d1224b77dd4e6393f18aab8d829b5b7e04afe9b7a2c"},
-    {file = "typed_ast-1.4.1-cp39-cp39-manylinux1_i686.whl", hash = "sha256:d648b8e3bf2fe648745c8ffcee3db3ff903d0817a01a12dd6a6ea7a8f4889072"},
-    {file = "typed_ast-1.4.1-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:fac11badff8313e23717f3dada86a15389d0708275bddf766cca67a84ead3e91"},
-    {file = "typed_ast-1.4.1-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:0d8110d78a5736e16e26213114a38ca35cb15b6515d535413b090bd50951556d"},
-    {file = "typed_ast-1.4.1-cp39-cp39-win32.whl", hash = "sha256:b52ccf7cfe4ce2a1064b18594381bccf4179c2ecf7f513134ec2f993dd4ab395"},
-    {file = "typed_ast-1.4.1-cp39-cp39-win_amd64.whl", hash = "sha256:3742b32cf1c6ef124d57f95be609c473d7ec4c14d0090e5a5e05a15269fb4d0c"},
-    {file = "typed_ast-1.4.1.tar.gz", hash = "sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b"},
+    {file = "typed_ast-1.4.2-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:7703620125e4fb79b64aa52427ec192822e9f45d37d4b6625ab37ef403e1df70"},
+    {file = "typed_ast-1.4.2-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:c9aadc4924d4b5799112837b226160428524a9a45f830e0d0f184b19e4090487"},
+    {file = "typed_ast-1.4.2-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:9ec45db0c766f196ae629e509f059ff05fc3148f9ffd28f3cfe75d4afb485412"},
+    {file = "typed_ast-1.4.2-cp35-cp35m-win32.whl", hash = "sha256:85f95aa97a35bdb2f2f7d10ec5bbdac0aeb9dafdaf88e17492da0504de2e6400"},
+    {file = "typed_ast-1.4.2-cp35-cp35m-win_amd64.whl", hash = "sha256:9044ef2df88d7f33692ae3f18d3be63dec69c4fb1b5a4a9ac950f9b4ba571606"},
+    {file = "typed_ast-1.4.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:c1c876fd795b36126f773db9cbb393f19808edd2637e00fd6caba0e25f2c7b64"},
+    {file = "typed_ast-1.4.2-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:5dcfc2e264bd8a1db8b11a892bd1647154ce03eeba94b461effe68790d8b8e07"},
+    {file = "typed_ast-1.4.2-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:8db0e856712f79c45956da0c9a40ca4246abc3485ae0d7ecc86a20f5e4c09abc"},
+    {file = "typed_ast-1.4.2-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:d003156bb6a59cda9050e983441b7fa2487f7800d76bdc065566b7d728b4581a"},
+    {file = "typed_ast-1.4.2-cp36-cp36m-win32.whl", hash = "sha256:4c790331247081ea7c632a76d5b2a265e6d325ecd3179d06e9cf8d46d90dd151"},
+    {file = "typed_ast-1.4.2-cp36-cp36m-win_amd64.whl", hash = "sha256:d175297e9533d8d37437abc14e8a83cbc68af93cc9c1c59c2c292ec59a0697a3"},
+    {file = "typed_ast-1.4.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:cf54cfa843f297991b7388c281cb3855d911137223c6b6d2dd82a47ae5125a41"},
+    {file = "typed_ast-1.4.2-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:b4fcdcfa302538f70929eb7b392f536a237cbe2ed9cba88e3bf5027b39f5f77f"},
+    {file = "typed_ast-1.4.2-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:987f15737aba2ab5f3928c617ccf1ce412e2e321c77ab16ca5a293e7bbffd581"},
+    {file = "typed_ast-1.4.2-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:37f48d46d733d57cc70fd5f30572d11ab8ed92da6e6b28e024e4a3edfb456e37"},
+    {file = "typed_ast-1.4.2-cp37-cp37m-win32.whl", hash = "sha256:36d829b31ab67d6fcb30e185ec996e1f72b892255a745d3a82138c97d21ed1cd"},
+    {file = "typed_ast-1.4.2-cp37-cp37m-win_amd64.whl", hash = "sha256:8368f83e93c7156ccd40e49a783a6a6850ca25b556c0fa0240ed0f659d2fe496"},
+    {file = "typed_ast-1.4.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:963c80b583b0661918718b095e02303d8078950b26cc00b5e5ea9ababe0de1fc"},
+    {file = "typed_ast-1.4.2-cp38-cp38-manylinux1_i686.whl", hash = "sha256:e683e409e5c45d5c9082dc1daf13f6374300806240719f95dc783d1fc942af10"},
+    {file = "typed_ast-1.4.2-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:84aa6223d71012c68d577c83f4e7db50d11d6b1399a9c779046d75e24bed74ea"},
+    {file = "typed_ast-1.4.2-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:a38878a223bdd37c9709d07cd357bb79f4c760b29210e14ad0fb395294583787"},
+    {file = "typed_ast-1.4.2-cp38-cp38-win32.whl", hash = "sha256:a2c927c49f2029291fbabd673d51a2180038f8cd5a5b2f290f78c4516be48be2"},
+    {file = "typed_ast-1.4.2-cp38-cp38-win_amd64.whl", hash = "sha256:c0c74e5579af4b977c8b932f40a5464764b2f86681327410aa028a22d2f54937"},
+    {file = "typed_ast-1.4.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:07d49388d5bf7e863f7fa2f124b1b1d89d8aa0e2f7812faff0a5658c01c59aa1"},
+    {file = "typed_ast-1.4.2-cp39-cp39-manylinux1_i686.whl", hash = "sha256:240296b27397e4e37874abb1df2a608a92df85cf3e2a04d0d4d61055c8305ba6"},
+    {file = "typed_ast-1.4.2-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:d746a437cdbca200622385305aedd9aef68e8a645e385cc483bdc5e488f07166"},
+    {file = "typed_ast-1.4.2-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:14bf1522cdee369e8f5581238edac09150c765ec1cb33615855889cf33dcb92d"},
+    {file = "typed_ast-1.4.2-cp39-cp39-win32.whl", hash = "sha256:cc7b98bf58167b7f2db91a4327da24fb93368838eb84a44c472283778fc2446b"},
+    {file = "typed_ast-1.4.2-cp39-cp39-win_amd64.whl", hash = "sha256:7147e2a76c75f0f64c4319886e7639e490fee87c9d25cb1d4faef1d8cf83a440"},
+    {file = "typed_ast-1.4.2.tar.gz", hash = "sha256:9fc0b3cb5d1720e7141d103cf4819aea239f7d136acf9ee4a69b047b7986175a"},
 ]
 typing-extensions = [
     {file = "typing_extensions-3.7.4.3-py2-none-any.whl", hash = "sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f"},
@@ -2921,8 +3031,8 @@ webencodings = [
     {file = "webencodings-0.5.1.tar.gz", hash = "sha256:b36a1c245f2d304965eb4e0a82848379241dc04b865afcc4aab16748587e1923"},
 ]
 websocket-client = [
-    {file = "websocket_client-0.57.0-py2.py3-none-any.whl", hash = "sha256:0fc45c961324d79c781bab301359d5a1b00b13ad1b10415a4780229ef71a5549"},
-    {file = "websocket_client-0.57.0.tar.gz", hash = "sha256:d735b91d6d1692a6a181f2a8c9e0238e5f6373356f561bb9dc4c7af36f452010"},
+    {file = "websocket_client-0.58.0-py2.py3-none-any.whl", hash = "sha256:44b5df8f08c74c3d82d28100fdc81f4536809ce98a17f0757557813275fbb663"},
+    {file = "websocket_client-0.58.0.tar.gz", hash = "sha256:63509b41d158ae5b7f67eb4ad20fecbb4eee99434e73e140354dc3ff8e09716f"},
 ]
 werkzeug = [
     {file = "Werkzeug-1.0.1-py2.py3-none-any.whl", hash = "sha256:2de2a5db0baeae7b2d2664949077c2ac63fbd16d98da0ff71837f7d1dea3fd43"},
@@ -2936,8 +3046,8 @@ wrapt = [
     {file = "wrapt-1.12.1.tar.gz", hash = "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"},
 ]
 xlrd = [
-    {file = "xlrd-1.2.0-py2.py3-none-any.whl", hash = "sha256:e551fb498759fa3a5384a94ccd4c3c02eb7c00ea424426e212ac0c57be9dfbde"},
-    {file = "xlrd-1.2.0.tar.gz", hash = "sha256:546eb36cee8db40c3eaa46c351e67ffee6eeb5fa2650b71bc4c758a29a1b29b2"},
+    {file = "xlrd-2.0.1-py2.py3-none-any.whl", hash = "sha256:6a33ee89877bd9abc1158129f6e94be74e2679636b8a205b43b85206c3f0bbdd"},
+    {file = "xlrd-2.0.1.tar.gz", hash = "sha256:f72f148f54442c6b056bf931dbc34f986fd0c3b0b6b5a58d013c9aef274d0c88"},
 ]
 xlwt = [
     {file = "xlwt-1.3.0-py2.py3-none-any.whl", hash = "sha256:a082260524678ba48a297d922cc385f58278b8aa68741596a87de01a9c628b2e"},
@@ -2948,6 +3058,6 @@ xmltodict = [
     {file = "xmltodict-0.12.0.tar.gz", hash = "sha256:50d8c638ed7ecb88d90561beedbf720c9b4e851a9fa6c47ebd64e99d166d8a21"},
 ]
 zipp = [
-    {file = "zipp-3.4.0-py3-none-any.whl", hash = "sha256:102c24ef8f171fd729d46599845e95c7ab894a4cf45f5de11a44cc7444fb1108"},
-    {file = "zipp-3.4.0.tar.gz", hash = "sha256:ed5eee1974372595f9e416cc7bbeeb12335201d8081ca8a0743c954d4446e5cb"},
+    {file = "zipp-3.4.1-py3-none-any.whl", hash = "sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"},
+    {file = "zipp-3.4.1.tar.gz", hash = "sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76"},
 ]
diff --git a/pyproject.toml b/pyproject.toml
index 998edc3..f5083f5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,25 +1,30 @@
 [tool.poetry]
 name = "data_linter"
-version = "4.1.0"
+version = "5.0.0"
 description = "data linter"
 authors = ["Thomas Hirsch <thomas.hirsch@digital.justice.gov.uk>",
            "George Kelly <george.kelly@digital.justice.gov.uk>",
            "Tapan Perkins <tapan.perkins@digital.justice.gov.uk>",
-           "Karik Isichei <karik.isichei@digital.justice.gov.uk>"]
+           "Karik Isichei <karik.isichei@digital.justice.gov.uk>",
+           "Stephen Bias <stephen.bias@digital.justice.gov.uk>"]
+
 readme = "README.md"
+repository = "https://github.com/moj-analytical-services/data_linter"
+homepage = "https://github.com/moj-analytical-services/data_linter"
 
 [tool.poetry.dependencies]
-python = ">=3.7,<3.9"
+python = ">=3.7.1,<3.9"
 dataengineeringutils3 = "^1.0.1"
-frictionless = "3.24.0"
 boto3 = "^1.14.7"
 jsonschema = "^3.2.0"
 pyyaml = "^5.3.1"
 iam_builder = "^3.7.0"
-importlib-metadata = "^1.7"
-pandas = { version = "1.1.3", optional = true }
+pandas = "^1.2"
+mojap-metadata = {url = "https://github.com/moj-analytical-services/mojap-metadata/archive/v1.1.0.zip", extras = ["arrow"]}
+arrow-pd-parser = { url = "https://github.com/moj-analytical-services/mojap-arrow-pd-parser/archive/v0.1.0.zip"}
 great-expectations = { version = "0.12.9", optional = true }
-awswrangler = {version = "1.10.0", optional = true}
+awswrangler = {version = "2.3.0", optional = true}
+frictionless = {version = "3.24.0", optional = true}
 
 [tool.poetry.dev-dependencies]
 pytest = "^3.4"
@@ -28,7 +33,8 @@ flake8 = "^3.8.4"
 jupyterlab = "^2.2.8"
 
 [tool.poetry.extras]
-ge = ["pandas", "great-expectations", "awswrangler"]
+ge = ["great-expectations", "awswrangler"]
+frictionless = ["frictionless"]
 
 [build-system]
 requires = ["poetry>=0.12"]


Test patch:
diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml
index 96e8096..260c4f2 100644
--- a/.github/workflows/run-tests.yml
+++ b/.github/workflows/run-tests.yml
@@ -17,10 +17,10 @@ jobs:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
+        python -m pip install --upgrade pip==20.3.3
         pip install poetry
         poetry config virtualenvs.create false \
-          && poetry install --extras "ge" --no-interaction --no-ansi
+          && poetry install --extras "ge frictionless" --no-interaction --no-ansi
         pytest tests/ -vv
 
   docker-run-tests:
diff --git a/tests/Dockerfile b/tests/Dockerfile
index a2b1d2d..0f45334 100644
--- a/tests/Dockerfile
+++ b/tests/Dockerfile
@@ -2,12 +2,12 @@ FROM python:3.7-slim-buster
 
 WORKDIR /validator
 
-RUN python -m pip install --upgrade pip
+RUN python -m pip install --upgrade pip==20.3.3
 RUN pip install poetry
 
 COPY . /validator
 
 RUN poetry config virtualenvs.create false \
-  && poetry install --extras "ge" --no-interaction --no-ansi
+  && poetry install --extras "ge frictionless" --no-interaction --no-ansi
 
 ENTRYPOINT pytest tests/ -vv
diff --git a/tests/Dockerfile_no_lock b/tests/Dockerfile_no_lock
index 0091674..045897c 100644
--- a/tests/Dockerfile_no_lock
+++ b/tests/Dockerfile_no_lock
@@ -2,13 +2,13 @@ FROM python:3.7-slim-buster
 
 WORKDIR /validator
 
-RUN python -m pip install --upgrade pip
+RUN python -m pip install --upgrade pip==20.3.3
 RUN pip install poetry
 
 COPY . /validator
 RUN rm /validator/poetry.lock
 
 RUN poetry config virtualenvs.create false \
-  && poetry install --extras "ge" --no-interaction --no-ansi
+  && poetry install --extras "ge frictionless" --no-interaction --no-ansi
 
 ENTRYPOINT pytest tests/ -vv
diff --git a/tests/conftest.py b/tests/conftest.py
index 155913c..81b7b18 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -8,10 +8,19 @@
 @pytest.fixture(scope="function")
 def aws_credentials():
     """Mocked AWS Credentials for moto."""
-    os.environ["AWS_ACCESS_KEY_ID"] = "testing"
-    os.environ["AWS_SECRET_ACCESS_KEY"] = "testing"
-    os.environ["AWS_SECURITY_TOKEN"] = "testing"
-    os.environ["AWS_SESSION_TOKEN"] = "testing"
+    mocked_envs = [
+        "AWS_ACCESS_KEY_ID",
+        "AWS_SECRET_ACCESS_KEY",
+        "AWS_SECURITY_TOKEN",
+        "AWS_SESSION_TOKEN",
+    ]
+    for menv in mocked_envs:
+        os.environ[menv] = "testing"
+
+    yield  # Allows us to close down envs on exit
+
+    for menv in mocked_envs:
+        del os.environ[menv]
 
 
 @pytest.fixture(scope="function")
diff --git a/tests/data/end_to_end1/bin_pack/config_1_0.yml b/tests/data/end_to_end1/bin_pack/config_1_0.yml
new file mode 100644
index 0000000..d34032c
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_1_0.yml
@@ -0,0 +1,26 @@
+land-base-path: s3://land/
+fail-base-path: s3://fail/
+pass-base-path: s3://pass/
+log-base-path: s3://log/
+
+compress-data: true
+remove-tables-on-pass: true
+all-must-pass: false
+
+# Tables to validate
+tables:
+  table1:
+    required: true
+    metadata: tests/data/end_to_end1/meta_data/table1.json
+    expect-header: true
+    matched_files:
+      - s3://land/table1.csv
+
+  table2:
+    required: true
+    pattern: ^table2
+    metadata: tests/data/end_to_end1/meta_data/table2.json
+    matched_files:
+      - s3://land/table2.jsonl
+      - s3://land/table2_extra.jsonl
+      - s3://land/table2_reordered_cols.jsonl
diff --git a/tests/data/end_to_end1/bin_pack/config_2_0.yml b/tests/data/end_to_end1/bin_pack/config_2_0.yml
new file mode 100644
index 0000000..2e19f99
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_2_0.yml
@@ -0,0 +1,15 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2.jsonl
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_extra.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_2_1.yml b/tests/data/end_to_end1/bin_pack/config_2_1.yml
new file mode 100644
index 0000000..e93d281
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_2_1.yml
@@ -0,0 +1,20 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table1:
+    linter: goodtables
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table1.csv
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table1.json
+    required: true
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_reordered_cols.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_3_0.yml b/tests/data/end_to_end1/bin_pack/config_3_0.yml
new file mode 100644
index 0000000..ecb6a77
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_3_0.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_3_1.yml b/tests/data/end_to_end1/bin_pack/config_3_1.yml
new file mode 100644
index 0000000..522d901
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_3_1.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_extra.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_3_2.yml b/tests/data/end_to_end1/bin_pack/config_3_2.yml
new file mode 100644
index 0000000..e93d281
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_3_2.yml
@@ -0,0 +1,20 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table1:
+    linter: goodtables
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table1.csv
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table1.json
+    required: true
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_reordered_cols.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_4_0.yml b/tests/data/end_to_end1/bin_pack/config_4_0.yml
new file mode 100644
index 0000000..ecb6a77
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_4_0.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_4_1.yml b/tests/data/end_to_end1/bin_pack/config_4_1.yml
new file mode 100644
index 0000000..522d901
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_4_1.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_extra.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_4_2.yml b/tests/data/end_to_end1/bin_pack/config_4_2.yml
new file mode 100644
index 0000000..e93d281
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_4_2.yml
@@ -0,0 +1,20 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table1:
+    linter: goodtables
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table1.csv
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table1.json
+    required: true
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_reordered_cols.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_5_0.yml b/tests/data/end_to_end1/bin_pack/config_5_0.yml
new file mode 100644
index 0000000..ecb6a77
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_5_0.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_5_1.yml b/tests/data/end_to_end1/bin_pack/config_5_1.yml
new file mode 100644
index 0000000..522d901
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_5_1.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_extra.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_5_2.yml b/tests/data/end_to_end1/bin_pack/config_5_2.yml
new file mode 100644
index 0000000..e93d281
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_5_2.yml
@@ -0,0 +1,20 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table1:
+    linter: goodtables
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table1.csv
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table1.json
+    required: true
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_reordered_cols.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_5_3.yml b/tests/data/end_to_end1/bin_pack/config_5_3.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_5_4.yml b/tests/data/end_to_end1/bin_pack/config_5_4.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_0.yml b/tests/data/end_to_end1/bin_pack/config_6_0.yml
new file mode 100644
index 0000000..ecb6a77
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_6_0.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_6_1.yml b/tests/data/end_to_end1/bin_pack/config_6_1.yml
new file mode 100644
index 0000000..522d901
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_6_1.yml
@@ -0,0 +1,14 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_extra.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_6_2.yml b/tests/data/end_to_end1/bin_pack/config_6_2.yml
new file mode 100644
index 0000000..e93d281
--- /dev/null
+++ b/tests/data/end_to_end1/bin_pack/config_6_2.yml
@@ -0,0 +1,20 @@
+all-must-pass: true
+compress-data: true
+fail-base-path: s3://alpha-mojap-de-courts-team/stephen/fail/
+land-base-path: s3://alpha-mojap-de-courts-team/stephen/land/
+log-base-path: s3://alpha-mojap-de-courts-team/stephen/log/
+pass-base-path: s3://alpha-mojap-de-courts-team/stephen/pass/
+remove-tables-on-pass: false
+tables:
+  table1:
+    linter: goodtables
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table1.csv
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table1.json
+    required: true
+  table2:
+    matched_files:
+      - s3://alpha-mojap-de-courts-team/stephen/land/table2_reordered_cols.jsonl
+    metadata: /Users/stephen/Documents/data_linter/tests/data/end_to_end1/meta_data/table2.json
+    pattern: ^table2
+    required: true
diff --git a/tests/data/end_to_end1/bin_pack/config_6_3.yml b/tests/data/end_to_end1/bin_pack/config_6_3.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_4.yml b/tests/data/end_to_end1/bin_pack/config_6_4.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_5.yml b/tests/data/end_to_end1/bin_pack/config_6_5.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_6.yml b/tests/data/end_to_end1/bin_pack/config_6_6.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_7.yml b/tests/data/end_to_end1/bin_pack/config_6_7.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_8.yml b/tests/data/end_to_end1/bin_pack/config_6_8.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/bin_pack/config_6_9.yml b/tests/data/end_to_end1/bin_pack/config_6_9.yml
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/end_to_end1/config.yaml b/tests/data/end_to_end1/config.yaml
index e41dcc0..d4d1bf6 100644
--- a/tests/data/end_to_end1/config.yaml
+++ b/tests/data/end_to_end1/config.yaml
@@ -2,8 +2,9 @@ land-base-path: s3://land/
 fail-base-path: s3://fail/
 pass-base-path: s3://pass/
 log-base-path: s3://log/
+
 compress-data: true
-remove-tables-on-pass: true
+remove-tables-on-pass: false
 all-must-pass: false
 
 # Tables to validate
diff --git a/tests/data/end_to_end1/config_matched_files.yml b/tests/data/end_to_end1/config_matched_files.yml
new file mode 100644
index 0000000..d34032c
--- /dev/null
+++ b/tests/data/end_to_end1/config_matched_files.yml
@@ -0,0 +1,26 @@
+land-base-path: s3://land/
+fail-base-path: s3://fail/
+pass-base-path: s3://pass/
+log-base-path: s3://log/
+
+compress-data: true
+remove-tables-on-pass: true
+all-must-pass: false
+
+# Tables to validate
+tables:
+  table1:
+    required: true
+    metadata: tests/data/end_to_end1/meta_data/table1.json
+    expect-header: true
+    matched_files:
+      - s3://land/table1.csv
+
+  table2:
+    required: true
+    pattern: ^table2
+    metadata: tests/data/end_to_end1/meta_data/table2.json
+    matched_files:
+      - s3://land/table2.jsonl
+      - s3://land/table2_extra.jsonl
+      - s3://land/table2_reordered_cols.jsonl
diff --git a/tests/data/end_to_end1/table1.csv b/tests/data/end_to_end1/land/table1.csv
similarity index 100%
rename from tests/data/end_to_end1/table1.csv
rename to tests/data/end_to_end1/land/table1.csv
diff --git a/tests/data/end_to_end1/table2.jsonl b/tests/data/end_to_end1/land/table2.jsonl
similarity index 100%
rename from tests/data/end_to_end1/table2.jsonl
rename to tests/data/end_to_end1/land/table2.jsonl
diff --git a/tests/data/end_to_end1/table2_extra.jsonl b/tests/data/end_to_end1/land/table2_extra.jsonl
similarity index 100%
rename from tests/data/end_to_end1/table2_extra.jsonl
rename to tests/data/end_to_end1/land/table2_extra.jsonl
diff --git a/tests/data/end_to_end1/table2_reordered_cols.jsonl b/tests/data/end_to_end1/land/table2_reordered_cols.jsonl
similarity index 100%
rename from tests/data/end_to_end1/table2_reordered_cols.jsonl
rename to tests/data/end_to_end1/land/table2_reordered_cols.jsonl
diff --git a/tests/data/end_to_end1/meta_data/table1.json b/tests/data/end_to_end1/meta_data/table1.json
index 79c4629..349b0c5 100644
--- a/tests/data/end_to_end1/meta_data/table1.json
+++ b/tests/data/end_to_end1/meta_data/table1.json
@@ -1,34 +1,39 @@
 {
-    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/table/v1.0.0.json",
+    "$schema": "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.1.0.json",
     "name": "table1",
     "description": "test meta for table1",
-    "data_format": "csv",
+    "file_format": "csv",
     "location": "table1/",
     "columns": [
         {
             "name": "my_int",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "minimum": 10,
             "maximum": 20,
             "nullable": false
         },
         {
             "name": "animal",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "enum": ["cat", "dog", "fish", "chicken"],
             "nullable": false
         },
         {
             "name": "my_email",
-            "type": "character"
+            "type": "string",
+            "type_category": "string"
         },
         {
             "name": "my_datetime",
-            "type": "datetime"
+            "type": "timestamp(s)",
+            "type_category": "timestamp"
         },
         {
             "name": "my_date",
-            "type": "date"
+            "type": "date64",
+            "type_category": "timestamp"
         }
     ]
-}
\ No newline at end of file
+}
diff --git a/tests/data/end_to_end1/meta_data/table2.json b/tests/data/end_to_end1/meta_data/table2.json
index b7ba602..35ca8aa 100644
--- a/tests/data/end_to_end1/meta_data/table2.json
+++ b/tests/data/end_to_end1/meta_data/table2.json
@@ -1,13 +1,14 @@
 {
-    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/table/v1.0.0.json",
+    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.1.0.json",
     "name": "table2",
     "description": "test meta for table2",
-    "data_format": "json",
+    "file_format": "jsonl",
     "location": "table2/",
     "columns": [
         {
             "name": "employee_id",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "description": "an ID for each employee",
             "nullable": false,
             "minimum": 100000,
@@ -15,25 +16,29 @@
         },
         {
             "name": "employee_name",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "name of the employee"
         },
         {
             "name": "employee_dob",
-            "type": "date",
+            "type": "date64",
+            "type_category": "timestamp",
             "description": "date of birth for the employee"
         },
         {
             "name": "employee_email",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "date of birth for the employee"
         },
         {
             "name": "fte",
-            "type": "double",
+            "type": "float64",
+            "type_category": "float",
             "description": "FTE of employee",
-            "minimum": 0,
-            "maximum": 1
+            "minimum": 0.0,
+            "maximum": 1.0
         }
     ]
 }
\ No newline at end of file
diff --git a/tests/data/great_expectations/meta_data/table1.json b/tests/data/great_expectations/meta_data/table1.json
index acb81bb..d6cb2e0 100644
--- a/tests/data/great_expectations/meta_data/table1.json
+++ b/tests/data/great_expectations/meta_data/table1.json
@@ -1,34 +1,39 @@
 {
-    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/table/v1.0.0.json",
+    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.1.0.json",
     "name": "table1",
     "description": "test meta for table1",
-    "data_format": "csv",
+    "file_format": "csv",
     "location": "table1/",
     "columns": [
         {
             "name": "my_int",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "minimum": 10,
             "maximum": 20,
             "nullable": false
         },
         {
             "name": "animal",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "enum": ["cat", "dog", "fish", "chicken", "NA"],
             "nullable": false
         },
         {
             "name": "my_email",
-            "type": "character"
+            "type": "string",
+            "type_category": "string"
         },
         {
             "name": "my_datetime",
-            "type": "datetime"
+            "type": "timestamp(s)",
+            "type_category": "timestamp"
         },
         {
             "name": "my_date",
-            "type": "date"
+            "type": "date64",
+            "type_category": "timestamp"
         }
     ]
 }
\ No newline at end of file
diff --git a/tests/data/headers/meta_data/table1.json b/tests/data/headers/meta_data/table1.json
index 79c4629..33f7bda 100644
--- a/tests/data/headers/meta_data/table1.json
+++ b/tests/data/headers/meta_data/table1.json
@@ -1,34 +1,39 @@
 {
-    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/table/v1.0.0.json",
+    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.1.0.json",
     "name": "table1",
     "description": "test meta for table1",
-    "data_format": "csv",
+    "file_format": "csv",
     "location": "table1/",
     "columns": [
         {
             "name": "my_int",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "minimum": 10,
             "maximum": 20,
             "nullable": false
         },
         {
             "name": "animal",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "enum": ["cat", "dog", "fish", "chicken"],
             "nullable": false
         },
         {
             "name": "my_email",
-            "type": "character"
+            "type": "string",
+            "type_category": "string"
         },
         {
             "name": "my_datetime",
-            "type": "datetime"
+            "type": "timestamp(s)",
+            "type_category": "timestamp"
         },
         {
             "name": "my_date",
-            "type": "date"
+            "type": "date64",
+            "type_category": "timestamp"
         }
     ]
 }
\ No newline at end of file
diff --git a/tests/data/headers/meta_data/table2.json b/tests/data/headers/meta_data/table2.json
index b7ba602..ab4e038 100644
--- a/tests/data/headers/meta_data/table2.json
+++ b/tests/data/headers/meta_data/table2.json
@@ -1,13 +1,14 @@
 {
-    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/table/v1.0.0.json",
+    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.1.0.json",
     "name": "table2",
     "description": "test meta for table2",
-    "data_format": "json",
+    "file_format": "json",
     "location": "table2/",
     "columns": [
         {
             "name": "employee_id",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "description": "an ID for each employee",
             "nullable": false,
             "minimum": 100000,
@@ -15,22 +16,26 @@
         },
         {
             "name": "employee_name",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "name of the employee"
         },
         {
             "name": "employee_dob",
-            "type": "date",
+            "type": "date64",
+            "type_category": "timestamp",
             "description": "date of birth for the employee"
         },
         {
             "name": "employee_email",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "date of birth for the employee"
         },
         {
             "name": "fte",
-            "type": "double",
+            "type": "float64",
+            "type_category": "float",
             "description": "FTE of employee",
             "minimum": 0,
             "maximum": 1
diff --git a/tests/data/inputs/basic_meta_data.json b/tests/data/inputs/basic_meta_data.json
index c3e957e..ff14277 100644
--- a/tests/data/inputs/basic_meta_data.json
+++ b/tests/data/inputs/basic_meta_data.json
@@ -1,5 +1,5 @@
 {
-    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/table/v1.0.0.json",
+    "$schema" : "https://moj-analytical-services.github.io/metadata_schema/mojap_metadata/v1.1.0.json",
     "name": "teams",
     "description": "month snapshot of which employee with working in what team",
     "data_format": "parquet",
@@ -7,7 +7,8 @@
     "columns": [
         {
             "name": "team_id",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "ID given to each team",
             "nullable" : false,
             "unique": true,
@@ -15,26 +16,30 @@
         },
         {
             "name": "team_name",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "name of the team"
         },
         {
             "name": "employee_id",
-            "type": "character",
+            "type": "string",
+            "type_category": "string",
             "description": "primary key for each employee in the employees table",
             "minLength": 8,
             "maxLength": 8
         },
         {
             "name": "snapshot_year",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "description": "year at which snapshot of workforce was taken",
             "minimum": 2010,
             "maximum": 2020
         },
         {
             "name": "snapshot_month",
-            "type": "int",
+            "type": "int64",
+            "type_category": "integer",
             "description": "month at which snapshot of workforce was taken",
             "enum" : [1,2,3,4,5,6,7,8,9,10,11,12]
         }
diff --git a/tests/data/inputs/example_config_fail_bucket.yaml b/tests/data/inputs/example_config_fail_enum.yaml
similarity index 62%
rename from tests/data/inputs/example_config_fail_bucket.yaml
rename to tests/data/inputs/example_config_fail_enum.yaml
index ee75b7a..b698901 100644
--- a/tests/data/inputs/example_config_fail_bucket.yaml
+++ b/tests/data/inputs/example_config_fail_enum.yaml
@@ -1,4 +1,5 @@
-land-base-path: s4://land-bucket/my-folder/  # Where to get the data from
+# vaildator-engine will fail schema validation as not one of the enum vals
+land-base-path: s3://land-bucket/my-folder/  # Where to get the data from
 fail-base-path: s3://fail-bucket/my-folder/  # Where to write the data if failed
 pass-base-path: s3://pass-bucket/my-folder/  # Where to write the data if passed
 log-base-path: s3://log-bucket/my-folder/  # Where to write logs
@@ -6,6 +7,7 @@ iam-role-name: role-name
 compress-data: true  # Compress data when moving elsewhere
 remove-tables-on-pass: true  # Delete the tables if pass
 all-must-pass: true  # Only move data if all tables have passed
+validator-engine: a-really-good-validator
 fail-unknown-files:
     exceptions:
         - additional_file.txt
@@ -13,7 +15,7 @@ fail-unknown-files:
 
 # Tables to validate
 tables:
-    - table_blah:
-          required: true  # Does the table have to exist
-          pattern: null  # Assumes file is called table1
-          linter: goodtables  # jsonschema?
+    table_blah:
+        required: true  # Does the table have to exist
+        pattern: null  # Assumes file is called table1
+        linter: goodtables  # jsonschema?
diff --git a/tests/data/inputs/example_config_fail_required.yaml b/tests/data/inputs/example_config_fail_required.yaml
new file mode 100644
index 0000000..9568299
--- /dev/null
+++ b/tests/data/inputs/example_config_fail_required.yaml
@@ -0,0 +1,17 @@
+# required field "validator-engine" missing causing schema validation failuire
+land-base-path: s3://land-bucket/my-folder/  # Where to get the data from
+fail-base-path: s3://fail-bucket/my-folder/  # Where to write the data if failed
+pass-base-path: s3://pass-bucket/my-folder/  # Where to write the data if passed
+log-base-path: s3://log-bucket/my-folder/  # Where to write logs
+iam-role-name: role-name
+compress-data: true  # Compress data when moving elsewhere
+remove-tables-on-pass: true  # Delete the tables if pass
+all-must-pass: true  # Only move data if all tables have passed
+fail-unknown-files:
+
+# Tables to validate
+tables:
+    table_blah:
+        required: true  # Does the table have to exist
+        pattern: null  # Assumes file is called table1
+        linter: goodtables  # jsonschema?
diff --git a/tests/data/inputs/example_config_fail_table.yaml b/tests/data/inputs/example_config_fail_type.yaml
similarity index 72%
rename from tests/data/inputs/example_config_fail_table.yaml
rename to tests/data/inputs/example_config_fail_type.yaml
index bd97119..4bf1829 100644
--- a/tests/data/inputs/example_config_fail_table.yaml
+++ b/tests/data/inputs/example_config_fail_type.yaml
@@ -1,3 +1,4 @@
+# the tables parameter will fail as it is type array (list) not object (dict)
 land-base-path: s3://land-bucket/my-folder/  # Where to get the data from
 fail-base-path: s3://fail-bucket/my-folder/  # Where to write the data if failed
 pass-base-path: s3://pass-bucket/my-folder/  # Where to write the data if passed
@@ -13,5 +14,7 @@ fail-unknown-files:
 
 # Tables to validate
 tables:
-    table_blah:
-        pattern: null
+    - table_blah:
+      required: true  # Does the table have to exist
+      pattern: null  # Assumes file is called table1
+      linter: goodtables  # jsonschema?
diff --git a/tests/helpers.py b/tests/helpers.py
new file mode 100644
index 0000000..417b102
--- /dev/null
+++ b/tests/helpers.py
@@ -0,0 +1,70 @@
+import os
+from contextlib import contextmanager
+import io
+import boto3
+from dataengineeringutils3.s3 import s3_path_to_bucket_key
+
+
+def set_up_s3(mocked_s3, test_folder, config):
+    """
+    Used to setup mocked s3 before a run that expects data in S3
+    """
+    from dataengineeringutils3.s3 import s3_path_to_bucket_key
+
+    land_base_path = config.get("land-base-path", "s3://land/")
+    fail_base_path = config.get("fail-base-path", "s3://fail/")
+    pass_base_path = config.get("pass-base-path", "s3://pass/")
+    log_base_path = config.get("log-base-path", "s3://log/")
+
+    land_base_path_is_s3 = land_base_path.startswith("s3://")
+    fail_base_path_is_s3 = fail_base_path.startswith("s3://")
+    pass_base_path_is_s3 = pass_base_path.startswith("s3://")
+    log_base_path_is_s3 = log_base_path.startswith("s3://")
+
+    buckets = []
+
+    if land_base_path_is_s3:
+        land_bucket, _ = s3_path_to_bucket_key(land_base_path)
+        buckets.append(land_bucket)
+    if fail_base_path_is_s3:
+        fail_bucket, _ = s3_path_to_bucket_key(fail_base_path)
+        buckets.append(fail_bucket)
+    if pass_base_path_is_s3:
+        pass_bucket, _ = s3_path_to_bucket_key(pass_base_path)
+        buckets.append(pass_bucket)
+    if log_base_path_is_s3:
+        log_bucket, _ = s3_path_to_bucket_key(log_base_path)
+        buckets.append(log_bucket)
+
+    for b in buckets:
+        mocked_s3.meta.client.create_bucket(
+            Bucket=b,
+            CreateBucketConfiguration={"LocationConstraint": "eu-west-1"},
+        )
+
+    files = [
+        f for f in os.listdir(test_folder) if f.endswith(".csv") or f.endswith(".jsonl")
+    ]
+
+    if land_base_path_is_s3:
+        for filename in files:
+            full_path = os.path.join(test_folder, filename)
+            mocked_s3.meta.client.upload_file(full_path, land_bucket, filename)
+
+
+class MockS3FilesystemReadInputStream:
+    @staticmethod
+    @contextmanager
+    def open_input_stream(s3_file_path_in: str) -> io.BytesIO:
+        s3_resource = boto3.resource("s3")
+        bucket, key = s3_path_to_bucket_key(s3_file_path_in)
+        obj_bytes = s3_resource.Object(bucket, key).get()["Body"].read()
+        obj_io_bytes = io.BytesIO(obj_bytes)
+        try:
+            yield obj_io_bytes
+        finally:
+            obj_io_bytes.close()
+
+
+def mock_get_file(*args, **kwargs):
+    return MockS3FilesystemReadInputStream()
diff --git a/tests/test_configs.py b/tests/test_configs.py
index 2d7f32b..184fc36 100644
--- a/tests/test_configs.py
+++ b/tests/test_configs.py
@@ -1,4 +1,6 @@
 import pytest
+import toml
+import data_linter as dl
 from data_linter.validation import load_and_validate_config
 from jsonschema.exceptions import ValidationError
 import json
@@ -8,14 +10,14 @@
 @pytest.mark.parametrize(
     "test_input, expected",
     [
-        ("example_config_fail_bucket.yaml", "pattern"),
-        ("example_config_fail_table.yaml", "required"),
+        ("example_config_fail_required.yaml", r"required"),
+        ("example_config_fail_enum.yaml", r"enum"),
+        ("example_config_fail_type.yaml", r"type"),
     ],
 )
 def test_load_and_validate_config_fail(test_input, expected):
-    try:
+    with pytest.raises(ValidationError, match=expected) as e:
         _ = load_and_validate_config(os.path.join("tests/data/inputs", test_input))
-    except ValidationError as e:
         assert e.validator == expected
 
 
@@ -24,3 +26,9 @@ def test_load_and_validate_config_pass():
         expected_pass = json.load(f)
     c = load_and_validate_config("tests/data/inputs/example_config_pass.yaml")
     assert c == expected_pass
+
+
+def test_pyproject_toml_matches_version():
+    with open("pyproject.toml") as f:
+        proj = toml.load(f)
+    dl.__version__ == proj["tool"]["poetry"]["version"]
diff --git a/tests/test_end_to_end_run.py b/tests/test_end_to_end_run.py
index 219cadd..f2ab624 100644
--- a/tests/test_end_to_end_run.py
+++ b/tests/test_end_to_end_run.py
@@ -2,85 +2,77 @@
 import yaml
 import gzip
 import tempfile
-
 import pytest
 
+from pyarrow import fs
 
-def set_up_s3(mocked_s3, test_folder, config):
-    from dataengineeringutils3.s3 import s3_path_to_bucket_key
-
-    land_bucket, _ = s3_path_to_bucket_key(config.get("land-base-path", "s3://land/"))
-    fail_bucket, _ = s3_path_to_bucket_key(config.get("fail-base-path", "s3://fail/"))
-    pass_bucket, _ = s3_path_to_bucket_key(config.get("pass-base-path", "s3://pass/"))
-    log_bucket, _ = s3_path_to_bucket_key(config.get("log-base-path", "s3://log/"))
-
-    buckets = [
-        land_bucket,
-        fail_bucket,
-        pass_bucket,
-        log_bucket,
-    ]
-    for b in buckets:
-        mocked_s3.meta.client.create_bucket(
-            Bucket=b,
-            CreateBucketConfiguration={"LocationConstraint": "eu-west-1"},
-        )
+from tests.helpers import (
+    set_up_s3,
+    mock_get_file,
+)
 
-    files = [
-        f for f in os.listdir(test_folder) if f.endswith(".csv") or f.endswith(".jsonl")
-    ]
-    for filename in files:
-        full_path = os.path.join(test_folder, filename)
-        mocked_s3.meta.client.upload_file(full_path, land_bucket, filename)
 
+def test_end_to_end(s3, monkeypatch):
 
-def test_end_to_end(s3):
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
 
     from data_linter.validation import run_validation
 
     test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
+
     config_path = os.path.join(test_folder, "config.yaml")
+
     with open(config_path) as f:
         config = yaml.safe_load(f)
-    set_up_s3(s3, test_folder, config)
+
+    # if config.get("validator-engine", "pandas") == "pandas":
+    #     config_path = os.path.join(test_folder, "config2.yaml")
+
+    set_up_s3(s3, land_folder, config)
     run_validation(config_path)
     os.system(f"python data_linter/command_line.py --config-path {config_path}")
 
 
-@pytest.mark.parametrize("validator", ["frictionless", "great-expectations"])
-def test_end_to_end_ge(s3, validator):
+@pytest.mark.parametrize("validator", ["pandas", "frictionless", "great-expectations"])
+def test_end_to_end_all_validators(s3, monkeypatch, validator):
+
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
 
     from data_linter.validation import run_validation
 
     test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
     config_path = os.path.join(test_folder, "config.yaml")
     with open(config_path) as f:
         config = yaml.safe_load(f)
     config["validator-engine"] = validator
-    set_up_s3(s3, test_folder, config)
+    set_up_s3(s3, land_folder, config)
     run_validation(config)
 
 
 def test_end_to_end_no_creds_error():
 
     from data_linter.validation import run_validation
-    from botocore.exceptions import ClientError
+    from botocore.exceptions import NoCredentialsError
 
     test_folder = "tests/data/end_to_end1/"
     config_path = os.path.join(test_folder, "config.yaml")
 
-    with pytest.raises(ClientError):
+    with pytest.raises(NoCredentialsError):
         run_validation(config_path)
 
 
 def test_compression(s3):
+
     from data_linter.utils import compress_data
 
     test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
     config_path = os.path.join(test_folder, "config.yaml")
     with open(config_path) as f:
         config = yaml.safe_load(f)
-    set_up_s3(s3, test_folder, config)
+    set_up_s3(s3, land_folder, config)
     test_file_uncompressed = "table2.jsonl"
     test_file_compressed = "table2.jsonl.gz"
     uncompressed_location = os.path.join(
@@ -99,7 +91,218 @@ def test_compression(s3):
 
     compressed_json_str = json_bytes.decode("utf-8")
 
-    with open(os.path.join(test_folder, test_file_uncompressed)) as uncompressed_json:
+    with open(os.path.join(land_folder, test_file_uncompressed)) as uncompressed_json:
         assert (
             compressed_json_str == uncompressed_json.read()
         ), "uncompressed json doesn't contain the same data as compressed json"
+
+
+@pytest.mark.parametrize("land_path", ["s3://land/", "tests/data/end_to_end1/land/"])
+@pytest.mark.parametrize("fail_path", ["s3://fail/", "fail"])
+@pytest.mark.parametrize("pass_path", ["s3://pass/", "pass"])
+@pytest.mark.parametrize("log_path", ["s3://log/", "log"])
+def test_end_to_end_full_path_spectrum(
+    s3, tmpdir_factory, monkeypatch, land_path, fail_path, pass_path, log_path
+):
+
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    from data_linter.validation import run_validation
+
+    test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
+    config_path = os.path.join(test_folder, "config.yaml")
+
+    with open(config_path) as yml:
+        config = yaml.safe_load(yml)
+
+    if not fail_path.startswith("s3://"):
+        fail_path = tmpdir_factory.mktemp(fail_path)
+        fail_path = fail_path.strpath
+    if not pass_path.startswith("s3://"):
+        pass_path = tmpdir_factory.mktemp(pass_path)
+        pass_path = pass_path.strpath
+    if not log_path.startswith("s3://"):
+        log_path = tmpdir_factory.mktemp(log_path)
+        log_path = log_path.strpath
+
+    config["land-base-path"] = land_path
+    config["fail-base-path"] = fail_path
+    config["pass-base-path"] = pass_path
+    config["log-base-path"] = log_path
+
+    set_up_s3(s3, land_folder, config)
+
+    run_validation(config)
+
+
+@pytest.mark.parametrize("land_path", ["s3://land/"])
+@pytest.mark.parametrize("fail_path", ["s3://fail/", "fail"])
+@pytest.mark.parametrize("pass_path", ["s3://pass/", "pass"])
+@pytest.mark.parametrize("log_path", ["s3://log/", "log"])
+def test_end_to_end_full_path_spectrum_parallel(
+    s3,
+    monkeypatch,
+    tmpdir_factory,
+    land_path,
+    fail_path,
+    pass_path,
+    log_path,
+):
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    from data_linter import validation
+
+    test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
+    config_path = os.path.join(test_folder, "config.yaml")
+    max_bin_count = 3
+
+    with open(config_path) as yml:
+        config = yaml.safe_load(yml)
+
+    if not fail_path.startswith("s3://"):
+        fail_path = tmpdir_factory.mktemp(fail_path)
+    if not pass_path.startswith("s3://"):
+        pass_path = tmpdir_factory.mktemp(pass_path)
+    if not log_path.startswith("s3://"):
+        log_path = tmpdir_factory.mktemp(log_path)
+
+    config["land_path"] = land_path
+    config["fail_path"] = fail_path
+    config["pass_path"] = pass_path
+    config["log_path"] = log_path
+
+    set_up_s3(s3, land_folder, config)
+
+    validation.para_run_init(max_bin_count, config_path)
+    for i in range(max_bin_count):
+        validation.para_run_validation(i, config_path)
+    validation.para_collect_all_status(config_path)
+    validation.para_collect_all_logs(config_path)
+
+
+@pytest.mark.parametrize("max_bin_count", [1, 3, 10])
+def test_bin_count(s3, monkeypatch, max_bin_count):
+
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    from data_linter import validation
+
+    test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
+    config_path = os.path.join(test_folder, "config.yaml")
+
+    with open(config_path) as yml:
+        config = yaml.safe_load(yml)
+
+    set_up_s3(s3, land_folder, config)
+
+    validation.para_run_init(max_bin_count, config_path)
+    for i in range(max_bin_count):
+        validation.para_run_validation(i, config_path)
+    validation.para_collect_all_status(config_path)
+    validation.para_collect_all_logs(config_path)
+
+
+def test_end_to_end_single_file_config(s3, monkeypatch):
+
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    from data_linter import validation
+
+    test_folder = "tests/data/end_to_end1/land/"
+
+    config = {
+        "land-base-path": "s3://land/",
+        "fail-base-path": "s3://fail/",
+        "pass-base-path": "s3://pass/",
+        "log-base-path": "s3://log/",
+        "compress-data": True,
+        "remove-tables-on-pass": True,
+        "all-must-pass": True,
+        "tables": {
+            "table1": {
+                "required": True,
+                "metadata": "tests/data/end_to_end1/meta_data/table1.json",
+                "expect-header": True,
+                "matched-files": ["s3://land/table1.csv"],
+            }
+        },
+    }
+
+    set_up_s3(s3, test_folder, config)
+
+    validation.para_run_init(1, config)
+    validation.para_run_validation(0, config)
+    validation.para_collect_all_status(config)
+    validation.para_collect_all_logs(config)
+
+
+@pytest.mark.parametrize("max_bin_count", [1, 2, 3, 4, 10])
+def test_bin_pack_configs(s3, max_bin_count):
+
+    from data_linter import validation
+    from data_linter.utils import read_all_file_body
+    from dataengineeringutils3.s3 import get_filepaths_from_s3_folder
+    from botocore.exceptions import ClientError
+
+    test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
+    config_path = os.path.join(test_folder, "config_matched_files.yml")
+
+    with open(config_path) as yml:
+        config = yaml.safe_load(yml)
+
+    set_up_s3(s3, land_folder, config)
+
+    validation.bin_pack_configs(config, max_bin_count)
+
+    land_base_path = config["land-base-path"]
+
+    all_bin_packed_configs = get_filepaths_from_s3_folder(
+        f"{land_base_path}/data_linter_temporary_storage/configs"
+    )
+
+    for i, file_path in enumerate(all_bin_packed_configs):
+        bin_pack_path = os.path.join(
+            test_folder, f"bin_pack/config_{max_bin_count}_{i}.yml"
+        )
+        with open(bin_pack_path) as yml:
+            pre_bin_packed = yaml.safe_load(yml)
+
+        try:
+            actual_bin_pack = yaml.safe_load(read_all_file_body(file_path))
+        except ClientError as e:
+            if e.response["Error"]["Code"] == "NoSuchKey":
+                assert pre_bin_packed is None
+        else:
+            assert actual_bin_pack == pre_bin_packed
+
+
+@pytest.mark.parametrize("land_path", ["s3://land/", "tests/data/end_to_end1/land/"])
+def test_read_all_file_body(s3, land_path):
+
+    from data_linter.utils import read_all_file_body
+
+    test_folder = "tests/data/end_to_end1/"
+    land_folder = "tests/data/end_to_end1/land/"
+    config_path = os.path.join(test_folder, "config.yaml")
+    table_1_path = os.path.join(land_folder, "table1.csv")
+
+    with open(config_path) as yml:
+        config = yaml.safe_load(yml)
+    with open(table_1_path) as f_in:
+        table_1_body_actual = f_in.read()
+
+    config["land-base-path"] = land_path
+
+    set_up_s3(s3, land_folder, config)
+    land_base_path = config["land-base-path"]
+    table_1_body = read_all_file_body(f"{land_base_path}table1.csv")
+
+    # Unix new line is \r\n, Mac new line is \n
+    table_1_body = table_1_body.replace("\r\n", "\n")
+    table_1_body_actual = table_1_body_actual.replace("\r\n", "\n")
+
+    assert table_1_body == table_1_body_actual
diff --git a/tests/test_moto.py b/tests/test_moto.py
index 41f37b5..8c12681 100644
--- a/tests/test_moto.py
+++ b/tests/test_moto.py
@@ -13,7 +13,7 @@ def test_download_fileobj(s3_client):
     bucket, key = s3_path_to_bucket_key(s3_download_path)
 
     table1 = "table1.csv"
-    test_path = "tests/data/end_to_end1/"
+    test_path = "tests/data/end_to_end1/land/"
     full_path = os.path.join(test_path, table1)
     s3_client.upload_file(full_path, bucket, table1)
 
diff --git a/tests/test_pandas_validator.py b/tests/test_pandas_validator.py
new file mode 100644
index 0000000..1c068aa
--- /dev/null
+++ b/tests/test_pandas_validator.py
@@ -0,0 +1,304 @@
+import pytest
+from data_linter.validators import pandas_validator as pv
+import pandas as pd
+
+int_not_null = pd.Series([1, 2, 3, 4, 5], dtype=pd.Int64Dtype())
+int_is_null = pd.Series([1, 2, None, 4, 5], dtype=pd.Int64Dtype())
+
+double_not_null = pd.Series([1.0, 2.23545, 3.532513, 4.35, 5.93567])
+double_is_null = pd.Series([1.0, 2.23545, 3.532513, None, 5.93567])
+
+str_not_null = pd.Series(
+    ["cat", "dog", "fish", "apple", "pineapple"], dtype=pd.StringDtype()
+)
+str_is_null = pd.Series(
+    ["cat", "dog", None, "apple", "pineapple"], dtype=pd.StringDtype()
+)
+
+date_str_not_null = pd.Series(
+    ["01/01/2020", "29/12/3000", "25/06/1903"], dtype=pd.StringDtype()
+)
+date_str_is_null = pd.Series([None, "29/12/3000", "25/06/1903"], dtype=pd.StringDtype())
+
+datetime_str_not_null = pd.Series(
+    ["2020-01-01 10:00:31", "3000-12-29 09:12:23", "1903-06-25 12:00:00"],
+    dtype=pd.StringDtype(),
+)
+datetime_str_is_null = pd.Series(
+    [None, "3000-12-29 09:12:23", "1903-06-25 12:00:00"], dtype=pd.StringDtype()
+)
+
+
+@pytest.mark.parametrize(
+    "col,expected_valid",
+    [
+        (int_is_null, False),
+        (int_not_null, True),
+        (double_is_null, False),
+        (double_not_null, True),
+        (str_is_null, False),
+        (str_not_null, True),
+    ],
+)
+def test_nullable_validation(col, expected_valid):
+    """
+    Check that nullable validation test gives expected result T/F
+    """
+    res = pv._nullable_test(col, {"name": "test_col", "nullable": False})
+    assert isinstance(res, dict)
+    assert expected_valid == res["valid"]
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        int_is_null,
+        int_not_null,
+        double_is_null,
+        double_not_null,
+    ],
+)
+@pytest.mark.parametrize(
+    "meta_col",
+    [
+        {"name": "test_col", "minimum": 0, "maximum": 6},
+        {"name": "test_col", "minimum": -6},
+        {"name": "test_col", "maximum": 6},
+        {"name": "test_col", "minimum": 0.0, "maximum": 6.0},
+        {"name": "test_col", "minimum": -6.1},
+        {"name": "test_col", "maximum": 6.2},
+    ],
+)
+def test_min_max_validation_pass(col, meta_col):
+    """
+    Check that every combination of pandas series and
+    metadata col passes min/max validation test
+    """
+    res = pv._min_max_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"]
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        int_is_null,
+        int_not_null,
+        double_is_null,
+        double_not_null,
+    ],
+)
+@pytest.mark.parametrize(
+    "meta_col",
+    [
+        {"name": "test_col", "minimum": 5, "maximum": 6},
+        {"name": "test_col", "minimum": 5},
+        {"name": "test_col", "maximum": 3},
+    ],
+)
+def test_min_max_validation_fail(col, meta_col):
+    """
+    Check that every combination of pandas series and
+    metadata col fails min/max validation test
+    """
+    res = pv._min_max_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"] is False
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        str_is_null,
+        str_not_null,
+    ],
+)
+@pytest.mark.parametrize(
+    "meta_col",
+    [
+        {"name": "test_col", "minLength": 0, "maxLength": 10},
+        {"name": "test_col", "minLength": 0},
+        {"name": "test_col", "maxLength": 10},
+    ],
+)
+def test_min_max_length_test_pass(col, meta_col):
+    """
+    Check that every combination of pandas series and
+    metadata col passes min/max length validation test
+    """
+    res = pv._min_max_length_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"]
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        str_is_null,
+        str_not_null,
+    ],
+)
+@pytest.mark.parametrize(
+    "meta_col",
+    [
+        {"name": "test_col", "minLength": 5, "maxLength": 6},
+        {"name": "test_col", "minLength": 5},
+        {"name": "test_col", "maxLength": 6},
+    ],
+)
+def test_min_max_length_test_fail(col, meta_col):
+    """
+    Check that every combination of pandas series and
+    metadata col passes min/max length validation test
+    """
+    res = pv._min_max_length_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"] is False
+
+
+@pytest.mark.parametrize(
+    "col_values,expected_valid",
+    [
+        (["abc-1235", "xyz-4468", "xyz-0284", "acx-8936"], True),
+        (["abc-1235", None, "xyz-0284", "acx-8936"], True),
+        (["abc-1235", "xyz-4468", "xyz-0284", "1acx-8936"], False),
+        (["abc-1235", "xyz-4468", None, "cx-8936"], False),
+    ],
+)
+def test_pattern_test_pass(col_values, expected_valid):
+    """
+    Check that pattern validation test gives expected result T/F
+    """
+    col = pd.Series(col_values, dtype=pd.StringDtype())
+    meta_col = {"name": "c", "pattern": "^\\D{3}-\\d{4}$"}
+    res = pv._pattern_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"] == expected_valid
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        str_is_null,
+        str_not_null,
+    ],
+)
+@pytest.mark.parametrize(
+    "meta_col,expected_valid",
+    [
+        (
+            {"name": "test_col", "enum": ["cat", "dog", "fish", "apple", "pineapple"]},
+            True,
+        ),
+        ({"name": "test_col", "enum": ["cat", "dog", "fish", "robot"]}, False),
+    ],
+)
+def test_enum_test(col, meta_col, expected_valid):
+    """
+    Check that pattern validation test gives expected meta and result T/F
+    For all nullable and non-nullable strings.
+    """
+    res = pv._enum_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"] == expected_valid
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        date_str_is_null,
+        date_str_not_null,
+    ],
+)
+def test_date_format_test_pass(col):
+    meta_col = {"name": "test_col", "type": "date64", "datetime_format": "%d/%m/%Y"}
+    res = pv._date_format_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"]
+
+    # Test datetime format can be safely cast to dates
+    meta_col2 = {
+        "name": "test_col",
+        "type": "date32",
+        "datetime_format": "%d/%m/%Y %H:%M:%S",
+    }
+    # Add zeros to date str
+    col2 = col.apply(lambda x: x + " 00:00:00" if not pd.isna(x) else x)
+    res = pv._date_format_test(col2, meta_col2)
+    assert isinstance(res, dict)
+    assert res["valid"]
+
+
+@pytest.mark.parametrize(
+    "col,meta_col",
+    [
+        (
+            # expects iso by default
+            date_str_is_null,
+            {"name": "test_col", "type": "date64"},
+        ),
+        (
+            # expects iso by default
+            date_str_not_null,
+            {"name": "test_col", "type": "date64"},
+        ),
+        (
+            # datetime strs with time component not date
+            datetime_str_is_null,
+            {
+                "name": "test_col",
+                "type": "date64",
+                "datetime_format": "%Y/%m/%d %H:%M:%S",
+            },
+        ),
+        (
+            # datetime strs with time component not date
+            datetime_str_not_null,
+            {
+                "name": "test_col",
+                "type": "date64",
+                "datetime_format": "%Y/%m/%d %H:%M:%S",
+            },
+        ),
+    ],
+)
+def test_date_format_test_fail(col, meta_col):
+    res = pv._date_format_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert not res["valid"]
+
+
+@pytest.mark.parametrize(
+    "col",
+    [
+        datetime_str_is_null,
+        datetime_str_not_null,
+    ],
+)
+@pytest.mark.parametrize(
+    "datetime_format,expected_valid",
+    [
+        (None, True),
+        ("%Y-%m-%d %H:%M:%S", True),
+        ("%d/%m/%Y %H:%M:%S", False),
+    ],
+)
+def test_datetime_format_test(col, datetime_format, expected_valid):
+    meta_col = {"name": "test_col", "type": "timestamp(s)"}
+    if datetime_format:
+        meta_col["datetime_format"] = datetime_format
+
+    res = pv._datetime_format_test(col, meta_col)
+    assert isinstance(res, dict)
+    assert res["valid"] == expected_valid
+
+
+def test_validation_function_skips():
+    assert pv._nullable_test(str_is_null, {"name": "c"}) is None
+    assert pv._nullable_test(str_is_null, {"name": "c", "nullable": True}) is None
+    assert pv._min_max_test(str_is_null, {"name": "c"}) is None
+    assert pv._min_max_length_test(str_is_null, {"name": "c"}) is None
+    assert pv._pattern_test(str_is_null, {"name": "c"}) is None
+    assert pv._enum_test(str_is_null, {"name": "c"}) is None
+    assert pv._datetime_format_test(str_is_null, {"name": "c"}) is None
+    assert pv._date_format_test(str_is_null, {"name": "c"}) is None
diff --git a/tests/test_simple_examples.py b/tests/test_simple_examples.py
new file mode 100644
index 0000000..64ed58b
--- /dev/null
+++ b/tests/test_simple_examples.py
@@ -0,0 +1,165 @@
+import os
+import yaml
+from copy import deepcopy
+from pyarrow import fs
+from tests.helpers import (
+    set_up_s3,
+    mock_get_file,
+)
+
+simple_yaml_config = """
+land-base-path: s3://land/
+fail-base-path: s3://fail/
+pass-base-path: s3://pass/
+log-base-path: s3://log/
+
+compress-data: false
+remove-tables-on-pass: true
+all-must-pass: true
+
+# Tables to validate
+tables:
+  table1:
+    required: true
+    metadata: tests/data/end_to_end1/meta_data/table1.json
+    expect-header: true
+
+  table2:
+    required: true
+    pattern: ^table2
+    metadata: tests/data/end_to_end1/meta_data/table2.json
+"""
+
+
+def test_validation_single_worker(s3, monkeypatch):
+    """
+    Simple example on how to run DL for a single worker.
+
+    [init] -> [worker]x1 -> [closedown]
+    """
+
+    # Need to mock S3 read for pyarrow (only for testing)
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    from data_linter import validation
+    from dataengineeringutils3.s3 import get_filepaths_from_s3_folder
+
+    land_folder = "tests/data/end_to_end1/land/"
+    config = yaml.safe_load(simple_yaml_config)
+
+    # Only required for mocked tests
+    set_up_s3(s3, land_folder, config)
+
+    validation.para_run_init(1, config)
+    validation.para_run_validation(0, config)
+    validation.para_collect_all_status(config)
+    validation.para_collect_all_logs(config)
+
+    # Assert that files have moved from land -> pass and nothing failed
+    land_files = get_filepaths_from_s3_folder(config["land-base-path"])
+    pass_files = get_filepaths_from_s3_folder(config["pass-base-path"])
+    fail_files = get_filepaths_from_s3_folder(config["fail-base-path"])
+    assert (not land_files and not fail_files) and pass_files
+
+
+def test_validation_multiple_workers(s3, monkeypatch):
+    """
+    Simple example on how to run DL for multiple worker.
+
+    [init] -> [worker]x4 -> [closedown]
+    """
+
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    from data_linter import validation
+    from dataengineeringutils3.s3 import get_filepaths_from_s3_folder
+
+    test_folder = "tests/data/end_to_end1/land/"
+    config = yaml.safe_load(simple_yaml_config)
+
+    # Only required for mocked tests
+    set_up_s3(s3, test_folder, config)
+
+    validation.para_run_init(4, config)
+
+    # although ran sequencially this can be ran in parallel
+    for i in range(4):
+        validation.para_run_validation(i, config)
+
+    validation.para_collect_all_status(config)
+    validation.para_collect_all_logs(config)
+
+    # Assert that files have moved from land -> pass and nothing failed
+    land_files = get_filepaths_from_s3_folder(config["land-base-path"])
+    pass_files = get_filepaths_from_s3_folder(config["pass-base-path"])
+    fail_files = get_filepaths_from_s3_folder(config["fail-base-path"])
+    assert (not land_files and not fail_files) and pass_files
+
+
+def test_validation_multiple_workers_no_init(s3, monkeypatch):
+    """
+    Simple example on how to run DL for multiple workers.
+    But without using the init. You would want to do this
+    if you want to specify which worker works on what specific dataset.
+    In the example below we run 1 worker per table validation
+
+    [worker]x2 -> [closedown]
+    """
+
+    monkeypatch.setattr(fs, "S3FileSystem", mock_get_file)
+
+    import boto3
+    from data_linter import validation
+    from data_linter.logging_functions import get_temp_log_basepath
+
+    from dataengineeringutils3.s3 import (
+        s3_path_to_bucket_key,
+        get_filepaths_from_s3_folder,
+    )
+
+    s3_client = boto3.client("s3")
+
+    land_folder = "tests/data/end_to_end1/land/"
+    config = yaml.safe_load(simple_yaml_config)
+
+    # Only required for mocked tests
+    set_up_s3(s3, land_folder, config)
+
+    worker_config_path = os.path.join(get_temp_log_basepath(config), "configs")
+    log_bucket, worker_base_key = s3_path_to_bucket_key(worker_config_path)
+
+    config = validation.load_and_validate_config(config)
+    config = validation.match_files_in_land_to_config(config)
+
+    # Create a config for worker 0 to only process table1
+    # (aka drop other tables in config)
+    # and write to worker 0 config to s3
+    worker0_conf = deepcopy(config)
+    del worker0_conf["tables"]["table2"]
+    s3_client.put_object(
+        Body=yaml.dump(worker0_conf).encode("utf-8"),
+        Bucket=log_bucket,
+        Key=f"{worker_base_key}/0/config.yml",
+    )
+
+    # Create a config for worker 1 to only process table2
+    # and write to worker 1 config to s3
+    worker1_conf = deepcopy(config)
+    del worker1_conf["tables"]["table1"]
+    s3_client.put_object(
+        Body=yaml.dump(worker1_conf).encode("utf-8"),
+        Bucket=log_bucket,
+        Key=f"{worker_base_key}/1/config.yml",
+    )
+
+    validation.para_run_validation(0, config)
+    validation.para_run_validation(1, config)
+
+    validation.para_collect_all_status(config)
+    validation.para_collect_all_logs(config)
+
+    # Assert that files have moved from land -> pass and nothing failed
+    land_files = get_filepaths_from_s3_folder(config["land-base-path"])
+    pass_files = get_filepaths_from_s3_folder(config["pass-base-path"])
+    fail_files = get_filepaths_from_s3_folder(config["fail-base-path"])
+    assert (not land_files and not fail_files) and pass_files