Repository: moj-analytical-services/mojap-arrow-pd-parser (language: python)

Issue:
changes



Discussion / hints:

Fix patch:
diff --git a/.flake8 b/.flake8
index f38a5e7..3f37cff 100644
--- a/.flake8
+++ b/.flake8
@@ -5,7 +5,7 @@ count = True
 show-source = True
 statistics = True
 # ignore E203 (disagrees with PEP8 and Black) and W503 (outdated - replaced by W504)
-ignore = E203, W503
+ignore = E203, W503, C901
 # potential to only check a specific list of rules not covered by Black
 # select = E9, F63, F7, F82
 exclude = 
@@ -19,5 +19,5 @@ exclude =
 	venv/,
 	env/,
 	.github
-	__init__.py
-
+	__init__.py,
+	*.ipynb_checkpoints/*
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 94c093e..c527fd1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## 1.0.0 - 2021-09-22
+
+First Major Release
+
+- Changed API for reader and writer to simplify entry point to readers and writers
+- Reduced support for multiple readers currently only uses Pandas for CSV and JSONL and Arrow for Parquet. Codebase can be expanded in the future.
+- Automatically reads from S3 or local depending on string filepath
+- README.md discusses use of package. Changes are not backwards compatible.
+- Pushed arrow dependency to at least version `5.0.0`. This is to reflect changes made in the parquet reader that is necessary for packages in our other eco systems.
+
 ## 0.4.4 - 2021-06-08
 
 - made it so partition data is not cast
diff --git a/README.md b/README.md
index 7aa5252..5802417 100644
--- a/README.md
+++ b/README.md
@@ -2,9 +2,9 @@
 
 Using pyArrow/Pandas to read CSV, JSONL and Parquet ensuring that you get the best representation of the column types in the resulting Pandas dataframe. Also ensures data type conformance by maintaining column types when reading and writing data back into Pandas (even when round tripping across multiple data types).
 
-This package also can read in data given a pyArrow schema or [MoJ-Metadata](https://github.com/moj-analytical-services/mojap-metadata) schema again to result in a Pandas dataframe that best represents the provided schema.
+This package also can read in data given an [MoJ-Metadata](https://github.com/moj-analytical-services/mojap-metadata) schema again to result in a Pandas dataframe that best represents the provided schema.
 
-Can also be used to write data back to supported formats using Pandas (for CSV/JSONL) and Pyarrow (for Parquet).
+Can also be used to write data back to supported formats using Pandas (for CSV/JSONL) and PyArrow (for Parquet).
 
 ## Installation
 
@@ -18,20 +18,15 @@ or via GitHub
 pip install arrow-pd-parser @ git+https://github.com/moj-analytical-services/mojap-arrow-pd-parser
 ```
 
-## Usage
+## Basic Usage
 
-This package uses `pyArrow` and/or Pandas to parse CSVs, JSONL and Parquet files and convert them to a Pandas Dataframe that are the best representation of those datatypes and ensure conformance between them. Also can write data back into the above formats to still maintain conformance to the provided schema.
+This package uses `PyArrow` and/or `Pandas` to parse CSVs, JSONL and Parquet files and convert them to a Pandas Dataframe that are the best representation of those datatypes and ensure conformance between them. Also can write data back into the above formats to still maintain conformance to the provided schema.
 
 ```python
-from arrow_pd_parser.parse import (
-    pa_read_csv_to_pandas,
-    pd_read_csv,
-)
-
-from arrow_pd_parser.pandas import pd_parser
+from arrow_pd_parser import reader, writer
 
-df1 = pa_read_csv_to_pandas("tests/data/all_types.csv")
-df1.dtypes()
+df1 = reader.read("tests/data/all_types.csv")
+df1.dtypes
 
 # i                     Int64
 # my_bool             boolean
@@ -41,288 +36,186 @@ df1.dtypes()
 # my_int                Int64
 # my_string            string
 
-df2 = pd_read_csv("tests/data/all_types.csv")
-df2.dtypes() == df1.dtypes()  # True
+df2 = reader.read("tests/data/all_types.jsonl")
+df2.dtypes # Note that Pandas struggles with nullable booleans when it reads JSONL (hence it is an Int64)
+
+# Write the dataframe to parquet
+# note deafult settings for the parquet writer is to
+# compress using snappy. (compression is not inferred from filepath but may do in future releases)
+writer.write(df1, "new_output.snappy.parquet")
 ```
 
 Note that the default behavior of this package is to utilse the new pandas datatypes for Integers, Booleans and Strings that represent Nulls as `pd.NA()`. Dates are returned as nullable objects of `datetime.date()` type and timestamps are `datetime.datetime()`. By default we enforce these types instead of the native pandas timestamp as the indexing for the Pandas timestamp is nanoseconds and can cause dates to be out of bounds. See the [timestamps](#timestamps) section for more details.
 
-If unsure on what parser to use we would suggest:
+The `reader.read()` method will infer what the file format is based on the extension of your filepath failing that it will take the format from your metadata if provided. By default the reader.read() will use the following readers for the prescribed file format.
 
-| Data Type | Parser |
-|-----------|--------|
-| CSV       | Pandas |
-| JSONL     | Pandas |
-| Parquet   | Arrow  |
+| Data Type | Reader | Writer |
+|-----------|--------|--------|
+| CSV       | Pandas | Pandas |
+| JSONL     | Pandas | Pandas |
+| Parquet   | Arrow  | Arrow  |
 
+You can also specify the file format in the reader and writer function or specify the reader type:
 
-### Advanced Usage
+```python
+from arrow_pd_parser import reader, writer
 
-This package will read in tabular data using pyarrow and then convert it to a pandas dataset. The full order of operations are:
-- Read in data to an arrow dataset
-- Cast the arrow dataset to the specified schema (if provided)
-- Convert the arrow dataset to Pandas
+# Specifying the reader
+# Both reader statements are equivalent and call the same readers
+# under the hood
+df1 = reader.read("tests/data/all_types.csv", file_format="csv")
+df2 = reader.csv.read("tests/data/all_types.csv")
 
-#### Reading and Schema Casting (Arrow)
+# You can also pass the reader args to the reader as kwargs
+df3 = reader.csv.read("tests/data/all_types.csv", nrows = 2)
 
-You can split up the previous code example (which used `pa_read_csv_to_pandas`) into two parts to get the exact same result (in case you wanted to do some transformations to the arrow dataset first).
+# The writer API has the same functionality
+writer.write(df1, file_format="parquet")
+writer.parquet.write(df1)
+```
 
-```python
-from arrow_pd_parser.parse import pa_read_csv
-from arrow_pd_parser.pa_pd import arrow_to_pandas
+### Use of Metadata
 
-# Read in the data first then convert it to a pandas dataframe
-df_arrow = pa_read_csv("tests/data/all_types.csv")
-df = arrow_to_pandas(df_arrow)
-```
+The main usefulness of this repo comes from specifying metadata for the data you want. We are going to focus on CSV and JSONL in this section as Parquet becomes a bit more nuanced as it encodes data types (unlike CSV and JSON that can be more vague).
 
-You can specify the additional parameters used by the pyarrow readers when reading in the data. 
+When reading and writing you can specify metadata (using our [metadata schema definitions](https://github.com/moj-analytical-services/mojap-metadata)) to ensure the data read into pandas conforms.
 
 ```python
-from io import BytesIO
-from pyarrow import csv
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
-
-csv_data = b"""
-a;b
-1;"This is text with a 
-new line"
-2;some more text
-"""
-test_file = BytesIO(csv_data)
-csv_parse_options = csv.ParseOptions(delimiter=";", newlines_in_values=True)
-df = pa_read_csv_to_pandas(test_file, parse_options=csv_parse_options)
-df.head()
-```
+from arrow_pd_parser import reader
 
-You can also provide an arrow schema to try and cast the data.
+# If no meta is provided we let the reader infer type
+df1 = reader.read("tests/data/all_types.csv")
+df1.dtypes
 
-```python
-from io import BytesIO
-import pyarrow as pa
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
+# i                     Int64
+# my_bool             boolean
+# my_nullable_bool    boolean
+# my_date              object
+# my_datetime          object
+# my_int                Int64
+# my_string            string
 
+# If metadata is provided we ensure conformance
+meta = {
+    "columns": [
+        {"name": "my_bool", "type": "string"},
+        {"name": "my_nullable_bool", "type": "bool"},
+        {"name": "my_date", "type": "string"},
+        {"name": "my_datetime", "type": "string"},
+        {"name": "my_int", "type": "float64"},
+        {"name": "my_string", "type": "string"}
+    ]
+}
+df2 = reader.read("tests/data/all_types.csv", metadata=meta)
+df2.dtypes
 
-csv_data = b"""
-a,b
-1,1.24
-2,7.81
-"""
-test_file = BytesIO(csv_data)
-from io import BytesIO
-import pyarrow as pa
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
-
-csv_data = b"""
-a,b
-1,1.24
-2,7.81
-"""
+# i                     Int64
+# my_bool              string
+# my_nullable_bool    boolean
+# my_date              string
+# my_datetime          string
+# my_int              float64
+# my_string            string
 
-# By default the data above would read as
-# a:Int64, b:float64 for a Pandas dataframe
-# Instead tell pa_pd_parser to treat these with the following schema
-schema = pa.schema([("a", pa.string()), ("b", pa.decimal128(3,2))])
-test_file = BytesIO(csv_data)
-df = pa_read_csv_to_pandas(test_file, schema=schema)
-df.types # a: String, b:object (each value is a decimal.Decimal)
+df3 = reader.read("tests/data/all_types.jsonl", metadata=meta)
+df3.dtypes
+
+assert df2.dtypes.to_list() == df3.dtypes.to_list()
 ```
 
-Note there are currently some issues around pyarrow not being able to cast timestamps to strings (see this see repo's issues for more details). For example:
+As of the v1 release we expect the API for reading and writing will remain the same but will still be lacking in how the caster works (what is called under the hood when casting the data to the prescribed metadata). The caster should improve with subsequent releases.
 
-```python
-from io import BytesIO
-import pyarrow as pa
-from pyarrow import csv
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
-
-csv_data = b"""
-a,b
-1,2020-01-01 00:00:00
-2,2021-01-01 23:59:59
-"""
+## Advanced Usage
 
-# note can also provide partial schema and get package to infer a's type by also setting `expect_full_schema=False`
-schema = pa.schema([("b", pa.string())])
-test_file = BytesIO(csv_data)
-
-# The following line will raise an ArrowNotImplementedError.
-# This is because there is currently no implementation to casting timestamps to str.
-df = pa_read_csv_to_pandas(test_file, schema=schema, expect_full_schema=False)
-
-# By default Arrow will read in str representations of timestamps as
-# timestamps if they conform to ISO standard format.
-# Then you get the error when you try and cast that timestamp to str. To
-# get around this you can force pyarrow to read in the data as a string
-# when it parses it as a CSV (note that ConvertOptions is not currently
-# available for the JSON reader)
-co = csv.ConvertOptions(column_types=schema)
-df = pa_read_csv_to_pandas(test_file, schema=schema, expect_full_schema=False, convert_options=co)
-```
+### Parameterising your reader and writer
 
-#### Reading and Schema Casting (Pandas)
+When using the `reader.read()` or `writer.write()` it pulls the default method with their default settings. However, if you want to customise your reader/writer you can do.
 
-In the same way you can seperate the reading and casting in the arrow example above you can do the same for the pandas parser.
 
-```python
-import pandas as pd
-from arrow_pd_parser.parse import (
-    pd_read_csv,
-    cast_pandas_table_to_schema
-)
+#### Passing arguments to the reader
 
-# Read in the data first then convert it to a pandas dataframe
+You can pass arguments to the underlying reader that is used to read data in. In the example below we use the nrows arg in pd.`pd.read_csv` that is used for our underlying reader.
 
-# pandas parsing/casting only needs type_category except for
-# "timestamp" type_categories where both type and type_categories
-# are required
-meta = {
-    "columns": [
-        {"name": "my_bool", "type_category": "boolean"},
-        {"name": "my_nullable_bool", "type_category": "boolean"},
-        {"name": "my_date", "type": "date32", "type_category": "timestamp"},
-        {"name": "my_datetime", "type": "timestamp(s)", "type_category": "timestamp"},
-        {"name": "my_int", "type_category": "integer"},
-        {"name": "my_string", "type_category": "string"},
-    ]
-}
+```python
+from arrow_pd_parser import reader
 
-df_str = pd.read_csv("tests/data/all_types.csv", dtype=str, low_memory=False)  # Best type conversion when reading in types as strings
-df_cast = cast_pandas_table_to_schema(df_str, meta)
+# Passing args to the read function.
+df = reader.csv.read("tests/data/all_types.jsonl", nrows=1000)
 ```
 
-The pandas parser functions that require metadata (like `pd_read_csv` and `cast_pandas_table_to_schema`) takes a `dict` that is compliant with a `Metadata` schema or a `Metadata` object. You can use the metadata object to set type_categories based of column types for metadata that only has the latter:
+#### Reader and Writer settings
+
+The readers and writers have some settings that you can use to configure how it reads/writes data. One of the main settings is how we deal with Pandas types. We default to the new pandas Series types: `StringDtype` for `string`, `BooleanDtype` for `bool` and `Int64Dtype` for `integer`. We also force dates and timestamps to be a series of objects (see the Timestamp section below)[#Reader-Pandas-Timestamps]. To change what pandas types to use you can change the reader settings:
 
 ```python
+from arrow_pd_parser.reader import csv
 from io import StringIO
-import pandas as pd
-from arrow_pd_parser.parse import (
-    pd_read_csv,
-)
+
+csv.pd_integer = False
+csv.pd_string = False
+csv.bool_map = {"Yes": True, "No": False}
 
 data = """
-my_nullable_bool,my_date,my_datetime,my_int
-True,True,2013-06-13,2013-06-13 05:11:07,
-True,,1995-04-30,1995-04-30 10:23:29,16
-False,False,2017-10-15,2017-10-15 20:25:05,0
+int_col,str_col,bool_col
+1,"Hello, mate",Yes
+2,Hi,No
 """
-
-# Set type categories in metadata object
 meta = {
     "columns": [
-        {"name": "my_nullable_bool", "type": "bool_"},
-        {"name": "my_date", "type": "date32"},
-        {"name": "my_datetime", "type": "timestamp(s)"},
-        {"name": "my_int", "type": "int64"},
+        {"name": "int_col", "type": "int64"},
+        {"name": "str_col", "type": "string"},
+        {"name": "bool_col", "type": "bool"},
     ]
 }
-metadata_instance = Metadata.from_dict(meta)
-df = pd_read_csv(StringIO(data), metadata_instance)
+f = StringIO(data)
+df = csv.read(f, metadata = meta)
 df.dtypes
-# i                     Int64
-# my_nullable_bool    boolean
-# my_date              object
-# my_datetime          object
-# my_int                Int64
-
+# int_col       int64
+# str_col      object
+# bool_col    boolean
 ```
 
+#### Using your own reader
 
-#### Exporting data to CSV/JSON
-
-You can also use the export module of this package to write data back (to CSV and JSON to ensure the same datatype will be read back in). This is useful when having to constantly read/write data between different storage systems and/or pipelines.
+If you wanted to create your own instance of a reader that wasn't the default provided you can.
 
 ```python
-from arrow_pd_parser.parse import (
-    pa_read_csv_to_pandas,
-    pa_read_json_to_pandas,
-)
-import pyarrow as pa
-from io import StringIO, BytesIO
-from arrow_pd_parser.export import pd_to_json
-
-s = pa.schema(
-    [
-        ("i", pa.int8()),
-        ("my_bool", pa.bool_()),
-        ("my_nullable_bool", pa.bool_()),
-        ("my_date", pa.date32()),
-        ("my_datetime", pa.timestamp("s")),
-        ("my_int", pa.uint8()),
-        ("my_string", pa.string()),
-    ]
-)
-
-# Read in original table
-original = pa_read_csv_to_pandas(
-        "tests/data/all_types.csv",
-        s,
-    )
+from arrow_pd_parser.reader import PandasCsvReader, csv
 
-# Write the table back out to a JSONL file
-f = StringIO()
-pd_to_json(original, f)
 
-# Read it back in and check it matches original
-new_f = BytesIO(f.getvalue().encode("utf8"))
-new = pa_read_json_to_pandas(new_f, s)
+# But you can also create your own and init the settings
+specific_csv = PandasCsvReader(
+    pd_integer = False,
+    pd_string = False,
+    bool_map = {"Yes": True, "No": False}
+)
 
-original == new # note that the two False values are where datetime is None in both tables
+# This may come in handy if you want to have two instances of a CSV reader with different settings
+df1 = csv.read("tests/data/all_types.csv") # default reader
+df2 = specific_csv.read("tests/data/all_types.csv")
 ```
 
-#### Integration with mojap-metadata (Arrow)
-
-The arrow modules can also be used alongside the `mojap-metadata` package which is already installed. In the example below you will need to install the package with the arrow dependencies:
-
-```
-pip install mojap-metadata[arrow]
-```
+#### Reading and Writing Parquet
 
+The default parquet reader and writer uses Arrow under the hood. This also means it uses a different method for casting datatypes (it is done in Arrow not with Pandas). Because parquet is stricter most likely if there is a deviation from your metadata it will fail to cast. The when provided metadata reader and writer API forces the data to that metadata, this makes sense for CSV and JSON but may make less sense for parquet. Therefore when using the parquet reader and writer you might want to consider if you want to provide your metadata (this may still be advantagous to check that the parquet matches metadata schema you have for the data).
 
 ```python
-from io import BytesIO
-from mojap_metadata import Metadata
-from mojap_metadata.converters.arrow_converter import (
-    ArrowConverter
-)
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
-
-csv_data = b"""
-a,b
-1,2020-01-01 00:00:00
-2,2021-01-01 23:59:59
-"""
-test_file = BytesIO(csv_data)
-
-# Define our metadata for the data
-md = {
-    "name": "test_data",
-    "columns": [
-        {
-            "name": "a",
-            "type": "string"
-        },
-        {
-            "name": "b",
-            "type": "timestamp(ms)"
-        }
-    ] 
-}
-meta = Metadata.from_dict(md)
+from arrow_pd_parser import reader, writer
 
-# Convert our schema to an arrow schema
-ac = ArrowConverter()
-arrow_schema = ac.generate_from_meta(meta)
+# Note all readers (CSV, JSON and Parquet) support S3 paths in the following way
+df = reader.parquet.read("s3://bucket/in.snappy.parquet")
+writer.parquet.write("s3://bucket/out.snappy.parquet")
 
-# Use the arrow_schema with arrow_pd_parser
-df = pa_read_csv_to_pandas(test_file, schema=arrow_schema)
+print(writer.parquet.compression, writer.parquet.version)
+# The default settings for the Parquet writer are snappy compression and version = '2.0'
+# These are passed to the arrow parquet write method but can either be changed by setting the
+# properties or by passing them as kwargs (which will super seed the default properties) e.g.
+writer.parquet.write("s3://bucket/out.parquet", compression=None, version='1.0')
 ```
 
-## Data Type Conformance
+#### Pandas Timestamps
 
-### Timestamps
-
-Pandas timestamps (currently) only support nanosecond resolution which is not ideal for a lot of timestamps as the range can be often too small.
+When metadata is provided (or for all Arrow Readers without metadata) we default to dates and datetimes as a series of objects rather than the Pandas timestamps this is because Pandas timestamps (currently) only support nanosecond resolution which is not ideal for a lot of timestamps as the range can be often too small.
 
 ```python
 import pandas as pd
@@ -336,31 +229,15 @@ Whereas Spark 3.0 (for example) allows timestamps from `0001-01-01 00:00:00` to
 When setting `pd_timestamp_type=pd_period` pd_arrow_parser will identify the correct pandas period resolution based on the arrow column type.
 
 ```python
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
-import pyarrow as pa
-
-# The Pandas Period type resolution is determined by the arrow col type's resolution
-schema = pa.schema([("my_datetime", pa.timestamp("s"))])
-df = pa_read_csv_to_pandas(
-    "tests/data/datetime_type.csv",
-    pd_timestamp_type="pd_period",
-)
-df.my_datetime.dtype # "period[S]"
-
-schema = pa.schema([("my_datetime", pa.timestamp("ms"))])
-df = pa_read_csv_to_pandas(
-    "tests/data/datetime_type.csv",
-    pd_timestamp_type="pd_period",
-)
-df.my_datetime.dtype # "period[L]"
+from arrow_pd_parser.reader import csv
 
+# The Pandas Period
+csv.pd_timestamp_type="pd_period"
+df = csv.read("tests/data/datetime_type.csv")
+df.my_datetime.dtype # "period[S]"
 
-# Using timestamp type
-df = pa_read_csv_to_pandas(
-    "tests/data/datetime_type.csv",
-    pd_timestamp_type="pd_timestamp"
-)
+# Use pandas timestamp
+csv.pd_timestamp_type="pd_timestamp"
+df = csv.read("tests/data/datetime_type.csv")
 df.my_datetime.dtype # dtype('<M8[ns]')
 ```
-
-
diff --git a/arrow_pd_parser/__init__.py b/arrow_pd_parser/__init__.py
index e69de29..5becc17 100644
--- a/arrow_pd_parser/__init__.py
+++ b/arrow_pd_parser/__init__.py
@@ -0,0 +1,1 @@
+__version__ = "1.0.0"
diff --git a/arrow_pd_parser/parse/arrow_parser.py b/arrow_pd_parser/_arrow_parsers.py
similarity index 91%
rename from arrow_pd_parser/parse/arrow_parser.py
rename to arrow_pd_parser/_arrow_parsers.py
index 50418d5..9050108 100644
--- a/arrow_pd_parser/parse/arrow_parser.py
+++ b/arrow_pd_parser/_arrow_parsers.py
@@ -1,3 +1,5 @@
+from mojap_metadata.metadata.metadata import Metadata
+from mojap_metadata.converters.arrow_converter import ArrowConverter
 import pyarrow as pa
 from pyarrow import csv, json, parquet
 
@@ -6,6 +8,21 @@
 from typing import Union, IO
 
 
+def _get_arrow_schema(schema: Union[pa.schema, Metadata, dict]):
+    ac = ArrowConverter()
+    if isinstance(schema, Metadata):
+        schema = ac.generate_to_meta(schema)
+    elif isinstance(schema, dict):
+        schema = Metadata.from_dict(schema)
+        schema = ac.generate_to_meta(schema)
+    elif isinstance(schema, pa.Schema):
+        pass
+    else:
+        raise TypeError(f"schema type not allowed: {type(schema)}")
+
+    return schema
+
+
 def update_existing_schema(
     current_schema: pa.Schema, new_schema: pa.Schema
 ) -> pa.Schema:
@@ -16,12 +33,10 @@ def update_existing_schema(
     If current_schema has fields that also exist in new_schema then
     the field in new_schema is chosen. If fields exist in new_schema
     but not in current, these will be ignored.
-
     Args:
         current_schema (pa.Schema): Schema to update
         new_schema (pa.Schema): Schema with fields that you wish to be
           used to update current_schema
-
     Returns:
         pa.Schema: Returns a schema with the same column order as
         current_schema but with the fields updated for any fields
@@ -44,7 +59,6 @@ def cast_arrow_table_to_schema(
     expect_full_schema: bool = True,
 ):
     """Casts an arrow schema to a new or partial schema
-
     Args:
         tab (pa.Table): An arrow table
         schema (Union[pa.Schema, None], optional): [description]. Defaults to None.
@@ -69,10 +83,9 @@ def pa_read_csv(
     input_file: Union[IO, str],
     schema: Union[pa.Schema, None] = None,
     expect_full_schema: bool = True,
-    **kwargs
+    **kwargs,
 ):
     """Read a csv file into an Arrow table.
-
     Args:
         input_file (Union[IO, str]): the CSV you want to read. string, path or
             file-like object.
@@ -84,11 +97,13 @@ def pa_read_csv(
             are listed in the schema, leaving all other columns to their
             default type on read.
         **kwargs (optional): Additional kwargs are passed to pyarrow.csv.read_csv
-
     Returns:
         pyarrow.Table: the csv file in pyarrow format.
     """
 
+    if schema:
+        schema = _get_arrow_schema(schema)
+
     pa_csv_table = csv.read_csv(input_file=input_file, **kwargs)
     if schema:
         pa_csv_table = cast_arrow_table_to_schema(
@@ -100,17 +115,16 @@ def pa_read_csv(
 
 def pa_read_csv_to_pandas(
     input_file: Union[IO, str],
-    schema: pa.Schema = None,
+    schema: Union[pa.Schema, Metadata, dict] = None,
     expect_full_schema: bool = True,
     pd_boolean: bool = True,
     pd_integer: bool = True,
     pd_string: bool = True,
     pd_date_type: str = "datetime_object",
     pd_timestamp_type: str = "datetime_object",
-    **kwargs
+    **kwargs,
 ):
     """Read a csv file into an Arrow table and convert it to a Pandas DataFrame.
-
     Args:
         input_file (Union[IO, str]): the CSV you want to read. string, path or
             file-like object.
@@ -131,7 +145,6 @@ def pa_read_csv_to_pandas(
         pd_timestamp_type (str, optional): specifies the datetime type. Can be one of:
             "datetime_object", "pd_timestamp" or "pd_period".
         **kwargs (optional): Additional kwargs are passed to pyarrow.csv.read_csv
-
     Returns:
         Pandas DataFrame: the csv data as a dataframe, with the specified data types
     """
@@ -151,12 +164,11 @@ def pa_read_csv_to_pandas(
 
 def pa_read_json(
     input_file: Union[IO, str],
-    schema: pa.Schema = None,
+    schema: Union[pa.Schema, Metadata, dict] = None,
     expect_full_schema: bool = True,
-    **kwargs
+    **kwargs,
 ):
     """Read a jsonlines file into an Arrow table.
-
     Args:
         input_file (Union[IO, str]): the JSONL you want to read. string, path or
             file-like object.
@@ -168,11 +180,13 @@ def pa_read_json(
             are listed in the schema, leaving all other columns to their
             default type on read.
         **kwargs (optional): Additional kwargs are passed to pyarrow.json.read_json
-
     Returns:
         pyarrow.Table: the jsonl file in pyarrow format casted to the specified schema
     """
 
+    if schema:
+        schema = _get_arrow_schema(schema)
+
     pa_json_table = json.read_json(input_file, **kwargs)
 
     if schema:
@@ -185,17 +199,16 @@ def pa_read_json(
 
 def pa_read_json_to_pandas(
     input_file: Union[IO, str],
-    schema: pa.Schema = None,
+    schema: Union[pa.Schema, Metadata, dict] = None,
     expect_full_schema: bool = True,
     pd_boolean: bool = True,
     pd_integer: bool = True,
     pd_string: bool = True,
     pd_date_type: str = "datetime_object",
     pd_timestamp_type: str = "datetime_object",
-    **kwargs
+    **kwargs,
 ):
     """Read a jsonlines file into an Arrow table and convert it to a Pandas DataFrame.
-
     Args:
         input_file (Union[IO, str]): the JSONL you want to read. string, path or
             file-like object.
@@ -217,7 +230,6 @@ def pa_read_json_to_pandas(
         pd_timestamp_type (str, optional): specifies the datetime type. Can be one of:
             "datetime_object", "pd_timestamp" or "pd_period".
         **kwargs (optional): Additional kwargs are passed to pyarrow.json.read_json
-
     Returns:
         Pandas DataFrame: the jsonl data as a dataframe, with the specified data types
     """
@@ -237,26 +249,26 @@ def pa_read_json_to_pandas(
 
 def pa_read_parquet(
     input_file: str,
-    schema: pa.Schema = None,
+    schema: Union[pa.Schema, Metadata, dict] = None,
     expect_full_schema: bool = True,
-    **kwargs
+    **kwargs,
 ):
 
     """
     reads parquet file to in memory arrow table
-
     Args:
         input_file (str): path (s3 or local) to the parquet file to read in
         schema (pa.Schema, optional): schema to cast the data to. Defaults to None.
         expect_full_schema (bool, optional): expect full schema. Defaults to True.
         kwargs (optional): kwargs to pass to pyarrow.parquet.read_table
-
     Returns:
         pyarrow table: data in an in memory arrow table
     """
 
     if not isinstance(input_file, str):
         raise TypeError("currently only supports string paths for input")
+    if schema:
+        schema = _get_arrow_schema(schema)
 
     pa_parquet_table = parquet.read_table(input_file, **kwargs)
 
@@ -270,18 +282,17 @@ def pa_read_parquet(
 
 def pa_read_parquet_to_pandas(
     input_file: str,
-    schema: pa.Schema = None,
+    schema: Union[pa.Schema, Metadata, dict] = None,
     expect_full_schema: bool = True,
     pd_boolean: bool = True,
     pd_integer: bool = True,
     pd_string: bool = True,
     pd_date_type: str = "datetime_object",
     pd_timestamp_type: str = "datetime_object",
-    **kwargs
+    **kwargs,
 ):
     """
     reads a parquet file to pandas dataframe with various type casting options
-
     Args:
         input_file (str): path (s3 or local) to the parquet file to read in
         schema (pa.Schema, optional): schema to cast the data to. Defaults to None.
@@ -292,7 +303,6 @@ def pa_read_parquet_to_pandas(
         pd_date_type (str, optional): [description]. Defaults to "datetime_object".
         pd_timestamp_type (str, optional): [description]. Defaults to "datetime_object".
         kwargs (optional) : kwargs to pass to pyarrow.parquet.read_table
-
     Returns:
         pandas dataframe: pandas dataframe of the given input data
     """
diff --git a/arrow_pd_parser/export.py b/arrow_pd_parser/_export.py
similarity index 85%
rename from arrow_pd_parser/export.py
rename to arrow_pd_parser/_export.py
index f6fc850..237a299 100644
--- a/arrow_pd_parser/export.py
+++ b/arrow_pd_parser/_export.py
@@ -1,9 +1,12 @@
 import datetime
+import warnings
+
 import numpy as np
 import pandas as pd
 import pyarrow as pa
-import warnings
 
+from mojap_metadata import Metadata
+from arrow_pd_parser._arrow_parsers import _get_arrow_schema
 from typing import Union, IO
 
 
@@ -77,13 +80,12 @@ def pd_to_json(
 def pd_to_parquet(
     df: pd.DataFrame,
     output_file: Union[str, pa.lib.NativeFile],
-    from_pandas_kwargs={},
-    write_table_kwargs={},
-    arrow_schema: pa.lib.Schema = None,
+    from_pandas_kwargs=None,
+    write_table_kwargs=None,
+    schema: Union[pa.Schema, Metadata, dict] = None,
 ):
     """
     Export a data frame as parquet
-    Does no conversion, and adheres to the parquet schema
 
     Args:
         df (pd.DataFrame): a pandas dataframe
@@ -91,11 +93,17 @@ def pd_to_parquet(
         from_pandas_kwargs (optional, dict): kwargs to pass to pyarrow.Table.from_pandas
         write_table_kwargs (optional, dicr):
             kwargs to pass to pyarrow.parquet.write_table
-        arrow_schema (optional, pyarrow.lib.schema): schema to cast the dataframe to
-            during conversion
+        schema (optional, pyarrow.lib.schema, Metadata, dict):
+            schema to cast the dataframe to during writing
     """
-
-    if from_pandas_kwargs.get("schema") and arrow_schema is not None:
+    if not from_pandas_kwargs:
+        from_pandas_kwargs = {}
+    if not write_table_kwargs:
+        write_table_kwargs = {}
+    if schema:
+        schema = _get_arrow_schema(schema)
+
+    if from_pandas_kwargs.get("schema") and schema is not None:
         warnings.warn(
             "schema specified twice, dropping schema specified in from_pandas_kwargs"
         )
@@ -104,9 +112,6 @@ def pd_to_parquet(
     if type(output_file) != str and type(output_file) != pa.lib.NativeFile:
         raise TypeError(f"unsupported output type: {type(output_file)}")
 
-    table = pa.Table.from_pandas(df, **from_pandas_kwargs, schema=arrow_schema)
-
-    # there was some date converting here, but I think parquet is ok as is
-    # ... I think ...
+    table = pa.Table.from_pandas(df, **from_pandas_kwargs, schema=schema)
 
     pa.parquet.write_table(table, output_file, **write_table_kwargs)
diff --git a/arrow_pd_parser/_readers.py b/arrow_pd_parser/_readers.py
new file mode 100644
index 0000000..96e9a0c
--- /dev/null
+++ b/arrow_pd_parser/_readers.py
@@ -0,0 +1,224 @@
+from abc import ABC, abstractmethod
+from typing import List, Union, Dict, IO
+import warnings
+from dataclasses import dataclass
+
+import pandas as pd
+import awswrangler as wr
+from pyarrow import parquet as pq
+
+from mojap_metadata import Metadata
+from mojap_metadata.converters.arrow_converter import ArrowConverter
+
+from arrow_pd_parser.utils import (
+    FileFormat,
+    is_s3_filepath,
+    EngineNotImplementedError,
+    validate_and_enrich_metadata,
+)
+from arrow_pd_parser.caster import cast_pandas_table_to_schema
+from arrow_pd_parser.pa_pd import arrow_to_pandas
+from arrow_pd_parser._arrow_parsers import cast_arrow_table_to_schema
+
+
+@dataclass
+class DataFrameFileReader(ABC):
+    """
+    Abstract class for reader functions used by reader API
+    Should just have a read method.
+    """
+
+    ignore_columns: List = None
+    drop_columns: List = None
+    pd_integer: bool = True
+    pd_string: bool = True
+    pd_boolean: bool = True
+    pd_date_type: str = "datetime_object"
+    pd_timestamp_type: str = "datetime_object"
+    bool_map: Dict = None
+
+    @abstractmethod
+    def read(
+        self, input_path: str, metadata: Union[Metadata, dict] = None, **kwargs
+    ) -> pd.DataFrame:
+        """reads the file into pandas DataFrame"""
+
+    def _cast_pandas_table_to_schema(
+        self, df: pd.DataFrame, metadata: Union[Metadata, dict]
+    ):
+        metadata = validate_and_enrich_metadata(metadata)
+        df = cast_pandas_table_to_schema(
+            df=df,
+            metadata=metadata,
+            ignore_columns=self.ignore_columns,
+            drop_columns=self.drop_columns,
+            pd_integer=self.pd_integer,
+            pd_string=self.pd_string,
+            pd_boolean=self.pd_boolean,
+            pd_date_type=self.pd_date_type,
+            pd_timestamp_type=self.pd_timestamp_type,
+            bool_map=self.bool_map,
+        )
+        return df
+
+
+@dataclass
+class PandasCsvReader(DataFrameFileReader):
+    """reader for CSV files"""
+
+    def read(
+        self,
+        input_path: Union[IO, str],
+        metadata: Union[Metadata, dict] = None,
+        **kwargs,
+    ) -> pd.DataFrame:
+        """
+        Reads a CSV file and returns a Pandas DataFrame
+        input_path: File to read either local or S3.
+        metadata: A metadata object or dict
+        **kwargs (optional): Additional kwargs are passed to pandas or awswrangler
+            read_csv. Note if metadata is not None then kwargs: low_memory=False
+            and dtype=str are set in order to properly cast CSV to metadata schema.
+        """
+        if metadata:
+            # If metadata is provided force
+            # str read in ready for type conversion
+            if "low_memory" not in kwargs:
+                kwargs["low_memory"] = False
+            if "dtype" not in kwargs:
+                kwargs["dtype"] = str
+
+        if is_s3_filepath(input_path):
+            df = wr.s3.read_csv(input_path, **kwargs)
+        else:
+            df = pd.read_csv(input_path, **kwargs)
+
+        if metadata is None and (
+            self.pd_string or self.pd_integer or self.pd_boolean
+        ):
+            df = df.convert_dtypes(
+                infer_objects=True,
+                convert_string=self.pd_string,
+                convert_integer=self.pd_integer,
+                convert_boolean=self.pd_boolean,
+                convert_floating=False,
+            )
+        else:
+            df = self._cast_pandas_table_to_schema(df, metadata)
+        return df
+
+
+@dataclass
+class PandasJsonReader(DataFrameFileReader):
+    """reader for json files"""
+
+    def read(
+        self,
+        input_path: Union[IO, str],
+        metadata: Union[Metadata, dict] = None,
+        **kwargs,
+    ) -> pd.DataFrame:
+        """
+        Reads a JSONL file and returns a Pandas DataFrame
+        input_path: File to read either local or S3.
+        metadata: A metadata object or dict
+        **kwargs (optional): Additional kwargs are passed to pandas or awswrangler
+            read_json. Note orient and lines will be ignored as always set to
+            lines=True and orient="records".
+        """
+
+        if not kwargs.get("lines", True):
+            warnings.warn("Ignoring lines in kwargs. Setting to lines=True.")
+        kwargs["lines"] = True
+
+        if kwargs.get("orient", "records") != "records":
+            warnings.warn('Ignoring orient in kwargs. Setting to orient="records"')
+        kwargs["orient"] = "records"
+
+        if is_s3_filepath(input_path):
+            df = wr.s3.read_json(input_path, **kwargs)
+        else:
+            df = pd.read_json(input_path, **kwargs)
+
+        if metadata is None and (
+            self.pd_string or self.pd_integer or self.pd_boolean
+        ):
+            df = df.convert_dtypes(
+                infer_objects=True,
+                convert_string=self.pd_string,
+                convert_integer=self.pd_integer,
+                convert_boolean=self.pd_boolean,
+                convert_floating=False,
+            )
+        else:
+            df = self._cast_pandas_table_to_schema(df, metadata)
+
+        return df
+
+
+@dataclass
+class ArrowParquetReader(DataFrameFileReader):
+    """reader for parquet files"""
+
+    expect_full_schema: bool = True
+
+    def read(
+        self, input_path: str, metadata: Metadata = None, **kwargs
+    ) -> pd.DataFrame:
+        """
+        Reads a Parquet file and returns a Pandas DataFrame
+        input_path: File to read either local or S3.
+        metadata: A metadata object or dict
+        **kwargs (optional): Additional kwargs are passed to the arrow reader
+            arrow.parquet.read_table
+        """
+
+        arrow_tab = pq.read_table(input_path, **kwargs)
+
+        if metadata:
+            meta = validate_and_enrich_metadata(metadata)
+            schema = ArrowConverter().generate_from_meta(meta)
+            arrow_tab = cast_arrow_table_to_schema(
+                arrow_tab,
+                schema=schema,
+                expect_full_schema=self.expect_full_schema,
+            )
+
+        df = arrow_to_pandas(
+            arrow_tab,
+            pd_boolean=self.pd_boolean,
+            pd_integer=self.pd_integer,
+            pd_string=self.pd_string,
+            pd_date_type=self.pd_date_type,
+            pd_timestamp_type=self.pd_timestamp_type,
+        )
+
+        return df
+
+
+def get_default_reader_from_file_format(
+    file_format: Union[FileFormat, str],
+    engine: str = None,
+) -> DataFrameFileReader:
+    # Convert to enum
+    if isinstance(file_format, str):
+        file_format = FileFormat.from_string(file_format)
+
+    if engine is not None:
+        raise EngineNotImplementedError(
+            "We plan to support engine choice in the future. "
+            "For now we only support one engine per file type. "
+            "CSV: Pandas, JSON: Pandas, Parquet: Arrow"
+        )
+
+    # Get reader
+    if file_format == FileFormat.CSV:
+        reader = PandasCsvReader()
+    elif file_format == FileFormat.JSON:
+        reader = PandasJsonReader()
+    elif file_format == FileFormat.PARQUET:
+        reader = ArrowParquetReader()
+    else:
+        raise ValueError(f"Unsupported file_format {file_format}")
+
+    return reader
diff --git a/arrow_pd_parser/_writers.py b/arrow_pd_parser/_writers.py
new file mode 100644
index 0000000..2658833
--- /dev/null
+++ b/arrow_pd_parser/_writers.py
@@ -0,0 +1,261 @@
+from abc import ABC, abstractmethod
+from typing import List, Union, Dict, IO
+import warnings
+from dataclasses import dataclass
+import datetime
+
+
+import pandas as pd
+import numpy as np
+
+import awswrangler as wr
+
+import pyarrow as pa
+from pyarrow import parquet as pq
+
+from mojap_metadata import Metadata
+from mojap_metadata.converters.arrow_converter import ArrowConverter
+
+from arrow_pd_parser.utils import (
+    FileFormat,
+    is_s3_filepath,
+    EngineNotImplementedError,
+    validate_and_enrich_metadata,
+)
+
+
+@dataclass
+class DataFrameFileWriter(ABC):
+    """
+    Abstract class for writer functions used by writer API
+    Should just have a write method.
+    """
+
+    copy = False
+    ignore_columns: List = None
+    drop_columns: List = None
+    pd_integer: bool = True
+    pd_string: bool = True
+    pd_boolean: bool = True
+    pd_date_type: str = "datetime_object"
+    pd_timestamp_type: str = "datetime_object"
+    bool_map: Dict = None
+
+    @abstractmethod
+    def write(
+        self,
+        df: pd.DataFrame,
+        output_path: Union[IO, str],
+        metadata: Union[Metadata, dict] = None,
+        **kwargs,
+    ) -> pd.DataFrame:
+        """writes the Dataframe to the output file"""
+
+
+@dataclass
+class PandasCsvWriter(DataFrameFileWriter):
+    """write for CSV files"""
+
+    drop_index = True
+
+    def write(
+        self,
+        df: pd.DataFrame,
+        output_path: Union[IO, str],
+        metadata: Union[Metadata, dict] = None,
+        **kwargs,
+    ):
+        """
+        Writes a pandas DataFrame to CSV
+        output_path: File to read either local or S3.
+        metadata: A metadata object or dict to cast the dataframe to before writing
+          (not necessarily needed for writing especially for CSV)
+        **kwargs (optional): Additional kwargs are passed to pandas or awswrangler
+            to_csv method.
+        """
+        if self.copy:
+            df_out = df.copy()
+        else:
+            df_out = df
+
+        if kwargs is None:
+            kwargs = {}
+
+        for col in df_out.columns:
+            # Convert period columns to strings so they're exported in a way
+            # Arrow can read
+            if pd.api.types.is_period_dtype(df_out[col]):
+                df_out[col] = df_out[col].dt.strftime("%Y-%m-%d %H:%M:%S")
+
+        kwargs_index = kwargs.get("index", (not self.drop_index))
+        if kwargs_index != (not self.drop_index):
+            warning_msg = (
+                f"Your kwargs for index ({kwargs_index}) mismatches the writer's "
+                f"settings self.drop_index ({self.drop_index}). "
+                "In this instance kwargs superseeds the writer settings."
+            )
+            warnings.warn(warning_msg)
+        else:
+            kwargs["index"] = not self.drop_index
+
+        if is_s3_filepath(output_path):
+            wr.s3.to_csv(df_out, output_path, **kwargs)
+        else:
+            df_out.to_csv(output_path, **kwargs)
+
+
+@dataclass
+class PandasJsonWriter(DataFrameFileWriter):
+    """write for CSV files"""
+
+    def write(
+        self,
+        df: pd.DataFrame,
+        output_path: Union[IO, str],
+        metadata: Union[Metadata, dict] = None,
+        **kwargs,
+    ):
+        """
+        Writes a pandas DataFrame to CSV
+        output_path: File to read either local or S3.
+        metadata: A metadata object or dict to cast the dataframe to before writing
+          (not necessarily needed for writing especially for CSV)
+        **kwargs (optional): Additional kwargs are passed to pandas or awswrangler
+            to_csv method.
+        """
+
+        if self.copy:
+            df_out = df.copy()
+        else:
+            df_out = df
+
+        if kwargs is None:
+            kwargs = {}
+
+        # Convert date-related columns to strings Arrow can read consistently
+        for col in df_out.columns:
+            if pd.api.types.is_period_dtype(df_out[col]):
+                df_out[col] = df_out[col].dt.strftime("%Y-%m-%d %H:%M:%S")
+            elif any(
+                [
+                    pd.api.types.is_datetime64_any_dtype(df_out[col]),
+                    isinstance(
+                        df_out[col][df_out[col].notnull()].iloc[0],
+                        (datetime.datetime, datetime.date),
+                    ),
+                ]
+            ):
+                df_out[col] = df_out[col].astype(pd.StringDtype())
+                # Convert pd_timestamp string 'NaT' to NaN so PyArrow can read them
+                df_out[col].replace("NaT", np.nan, regex=False, inplace=True)
+
+        for col in df_out.columns:
+            # Convert period columns to strings so they're exported in a way
+            # Arrow can read
+            if pd.api.types.is_period_dtype(df_out[col]):
+                df_out[col] = df_out[col].dt.strftime("%Y-%m-%d %H:%M:%S")
+
+        if kwargs.get("orient", "records") != "records":
+            error_msg = (
+                "You are not allowed to specify orient in your kwargs "
+                "as anything other than `orient='records'`. This is a "
+                "jsonl writer and requires this setting."
+                ""
+            )
+            raise ValueError(error_msg)
+
+        if not kwargs.get("lines", True):
+            error_msg = (
+                "You are not allowed to specify lines in your kwargs "
+                "as anything other than `lines='records'`. This is a "
+                "jsonl writer and requires this setting."
+            )
+            raise ValueError(error_msg)
+
+        if is_s3_filepath(output_path):
+            wr.s3.to_json(df_out, output_path, orient="records", lines=True, **kwargs)
+        else:
+            df_out.to_json(output_path, orient="records", lines=True, **kwargs)
+
+
+@dataclass
+class ArrowParquetWriter(DataFrameFileWriter):
+    """write for Parquet files"""
+
+    version: str = "2.0"
+    compression: str = "snappy"
+
+    def write(
+        self,
+        df: pd.DataFrame,
+        output_path: str,
+        metadata: Union[Metadata, dict] = None,
+        **kwargs,
+    ):
+        """
+        Writes a pandas DataFrame to CSV
+        output_path: File to read either local or S3.
+        metadata: A metadata object or dict to cast the dataframe to before writing
+          (not necessarily needed for writing especially for CSV)
+        **kwargs (optional): Additional kwargs are passed to
+          pyarrow.parquet.write_table
+        """
+        if kwargs is None:
+            kwargs = {}
+
+        if metadata:
+            meta = validate_and_enrich_metadata(metadata)
+            arrow_schema = ArrowConverter().generate_from_meta(meta)
+
+        else:
+            arrow_schema = None
+
+        kwargs["version"] = kwargs.get("version", self.version)
+        kwargs["compression"] = kwargs.get("compression", self.compression)
+
+        if kwargs["version"] != self.version:
+            warning_msg = (
+                f"Your kwargs for version ({kwargs.get('version')}) mismatches "
+                f"the writer's settings self.version ({self.version})."
+                "In this instance kwargs superseeds the writer settings."
+            )
+            warnings.warn(warning_msg)
+
+        if kwargs["compression"] != self.compression:
+            warning_msg = (
+                f"Your kwargs for compression ({kwargs.get('compression')}) mismatches "
+                f"the writer's settings self.compression ({self.compression}). "
+                "In this instance kwargs superseeds the writer settings."
+            )
+            warnings.warn(warning_msg)
+
+        table = pa.Table.from_pandas(df, schema=arrow_schema)
+        pq.write_table(table, output_path, **kwargs)
+
+
+def get_default_writer_from_file_format(
+    file_format: Union[FileFormat, str],
+    engine: str = None,
+) -> DataFrameFileWriter:
+    # Convert to enum
+    if isinstance(file_format, str):
+        file_format = FileFormat.from_string(file_format)
+
+    if engine is not None:
+        raise EngineNotImplementedError(
+            "We plan to support engine choice in the future. "
+            "For now we only support one engine per file type. "
+            "CSV: Pandas, JSON: Pandas, Parquet: Arrow"
+        )
+
+    # Get writer
+    if file_format == FileFormat.CSV:
+        writer = PandasCsvWriter()
+    elif file_format == FileFormat.JSON:
+        writer = PandasJsonWriter()
+    elif file_format == FileFormat.PARQUET:
+        writer = ArrowParquetWriter()
+    else:
+        raise ValueError(f"Unsupported file_format {file_format}")
+
+    return writer
diff --git a/arrow_pd_parser/parse/pandas_parser.py b/arrow_pd_parser/caster.py
similarity index 58%
rename from arrow_pd_parser/parse/pandas_parser.py
rename to arrow_pd_parser/caster.py
index 81f3076..4d0c99c 100644
--- a/arrow_pd_parser/parse/pandas_parser.py
+++ b/arrow_pd_parser/caster.py
@@ -1,13 +1,12 @@
 import warnings
-
 import numpy as np
 import pandas as pd
 
-from typing import List, Union, IO, Callable
 from copy import deepcopy
 from datetime import datetime
-from pandas.api.types import is_numeric_dtype
 from mojap_metadata import Metadata
+from pandas.api.types import is_numeric_dtype
+from typing import List, Union, Callable
 
 _allowed_type_categories = [
     "integer",
@@ -25,15 +24,21 @@ class PandasCastError(Exception):
 
 
 def _convert_str_to_ns_timestamp_series(
-    s: pd.Series, str_datetime_format=None,
+    s: pd.Series,
+    str_datetime_format=None,
 ) -> pd.Series:
-    s = pd.to_datetime(s, format=str_datetime_format,)
+    s = pd.to_datetime(
+        s,
+        format=str_datetime_format,
+    )
 
     return s
 
 
 def _convert_str_to_datetime_obj_series(
-    s: pd.Series, is_date, str_datetime_format=None,
+    s: pd.Series,
+    is_date,
+    str_datetime_format=None,
 ) -> pd.Series:
 
     if str_datetime_format is None:
@@ -109,7 +114,11 @@ def convert_to_float_series(s: pd.Series, num_errors: str) -> pd.Series:
     return s
 
 
-def convert_to_bool_series(s: pd.Series, pd_boolean, bool_map=None,) -> pd.Series:
+def convert_to_bool_series(
+    s: pd.Series,
+    pd_boolean,
+    bool_map=None,
+) -> pd.Series:
     """
     Reads a pandas Series and casts to a bool. If type is already boolean like
     i.e. an object of bools and nulls, bool dtype or boolean dtype then conversion
@@ -243,6 +252,7 @@ def cast_pandas_table_to_schema(
     pd_date_type: str = "datetime_object",
     pd_timestamp_type: str = "datetime_object",
     bool_map: Union[Callable, dict] = None,
+    num_error_map: dict = None,
 ):
     """
     Casts the columns in dataframe provided to the meta data dictionary provided.
@@ -259,7 +269,7 @@ def cast_pandas_table_to_schema(
         _default_str_bool_mapper.
     """
 
-    num_errors = "raise"
+    default_num_errors = "raise"
 
     if ignore_columns is None:
         ignore_columns = []
@@ -284,8 +294,9 @@ def cast_pandas_table_to_schema(
 
     all_exclude_cols = ignore_columns + drop_columns
     meta_cols_to_convert = [
-        c for c in meta["columns"] if c["name"] not in all_exclude_cols
-        or c["name"] not in meta["partitions"]
+        c
+        for c in meta["columns"]
+        if c["name"] not in all_exclude_cols or c["name"] not in meta["partitions"]
     ]
 
     for c in meta_cols_to_convert:
@@ -294,6 +305,14 @@ def cast_pandas_table_to_schema(
             raise ValueError(f"Column '{c['name']}' not in df")
 
         else:
+            # must get num_errors from either meta or num_error_map. Meta has precedence
+            if c.get("num_errors"):
+                num_errors = c.get("num_errors")
+            elif isinstance(num_error_map, dict):
+                num_errors = num_error_map.get(c["name"])
+            else:
+                num_errors = default_num_errors
+
             df[c["name"]] = cast_pandas_column_to_schema(
                 df[c["name"]],
                 metacol=c,
@@ -307,144 +326,10 @@ def cast_pandas_table_to_schema(
             )
 
     final_cols = [
-        c["name"] for c in meta["columns"] if c["name"] not in drop_columns
-        or c["name"] not in meta["partitions"]
+        c["name"]
+        for c in meta["columns"]
+        if c["name"] not in drop_columns or c["name"] not in meta["partitions"]
     ]
     df = df[final_cols]
 
     return df
-
-
-def pd_read_csv(
-    input_file: Union[IO, str],
-    metadata: Union[Metadata, dict],
-    ignore_columns: List = None,
-    drop_columns: List = None,
-    pd_integer: bool = True,
-    pd_string: bool = True,
-    pd_boolean: bool = True,
-    pd_date_type: str = "datetime_object",
-    pd_timestamp_type: str = "datetime_object",
-    bool_map=None,
-    **kwargs,
-) -> pd.DataFrame:
-    """Read a csv file into a Pandas dataframe casting cols based on Metadata.
-
-    Args:
-        input_file (Union[IO, str]): the CSV you want to read. string, path or
-            file-like object.
-        metadata Union[Metadata, dict]: what you want the column to be cast to.
-        ignore_columns: (List, optional): a list of column names to not cast to
-            the meta data dictionary. These columns are remained unchanged.
-        drop_columns:  (List, optional): a list of column names you want to drop
-            from the dataframe.
-        pd_boolean: whether to use the new pandas boolean format. Defaults to True.
-            When set to False, uses a custom boolean format to coerce object type.
-        pd_integer: if True, converts integers to Pandas int64 format.
-            If False, uses float64. Defaults to True.
-        pd_string: Defaults to True.
-        pd_date_type (str, optional): specifies the timestamp type. Can be one of:
-            "datetime_object", "pd_timestamp" or "pd_period" ("pd_period" not yet
-            implemented).
-        pd_timestamp_type (str, optional): specifies the timestamp type. Can be one of:
-            "datetime_object", "pd_timestamp" or "pd_period" ("pd_period" not yet
-            implemented).
-        bool_map (Callable, dict, optional): A custom mapping function that is applied
-            to str cols to be converted to booleans before conversion to boolean type.
-            e.g. {"Yes": True, "No": False}. If not set bool values are inferred by the
-            _default_str_bool_mapper.
-        **kwargs (optional): Additional kwargs are passed to pandas.read_csv
-
-    Returns:
-        Pandas DataFrame: the csv data as a dataframe, with the specified data types
-    """
-    if "low_memory" not in kwargs:
-        kwargs["low_memory"] = False
-    if "dtype" not in kwargs:
-        kwargs["dtype"] = str
-
-    df = pd.read_csv(input_file, **kwargs)
-    df = cast_pandas_table_to_schema(
-        df=df,
-        metadata=metadata,
-        ignore_columns=ignore_columns,
-        drop_columns=drop_columns,
-        pd_integer=pd_integer,
-        pd_string=pd_string,
-        pd_boolean=pd_boolean,
-        pd_date_type=pd_date_type,
-        pd_timestamp_type=pd_timestamp_type,
-        bool_map=bool_map,
-    )
-
-    return df
-
-
-def pd_read_json(
-    input_file: Union[IO, str],
-    metadata: Union[Metadata, dict],
-    ignore_columns: List = None,
-    drop_columns: List = None,
-    pd_integer: bool = True,
-    pd_string: bool = True,
-    pd_boolean: bool = True,
-    pd_date_type: str = "datetime_object",
-    pd_timestamp_type: str = "datetime_object",
-    bool_map=None,
-    **kwargs,
-) -> pd.DataFrame:
-    """Read a csv file into a Pandas dataframe casting cols based on Metadata.
-
-    Args:
-        input_file (Union[IO, str]): the CSV you want to read. string, path or
-            file-like object.
-        metadata Union[Metadata, dict]: what you want the column to be cast to.
-        ignore_columns: (List, optional): a list of column names to not cast to
-            the meta data dictionary. These columns are remained unchanged.
-        drop_columns:  (List, optional): a list of column names you want to drop
-            from the dataframe.
-        pd_boolean: whether to use the new pandas boolean format. Defaults to True.
-            When set to False, uses a custom boolean format to coerce object type.
-        pd_integer: if True, converts integers to Pandas int64 format.
-            If False, uses float64. Defaults to True.
-        pd_string: Defaults to True.
-        pd_date_type (str, optional): specifies the timestamp type. Can be one of:
-            "datetime_object", "pd_timestamp" or "pd_period" ("pd_period" not yet
-            implemented).
-        pd_timestamp_type (str, optional): specifies the timestamp type. Can be one of:
-            "datetime_object", "pd_timestamp" or "pd_period" ("pd_period" not yet
-            implemented).
-        bool_map (Callable, dict, optional): A custom mapping function that is applied
-            to str cols to be converted to booleans before conversion to boolean type.
-            e.g. {"Yes": True, "No": False}. If not set bool values are inferred by the
-            _default_str_bool_mapper.
-        **kwargs (optional): Additional kwargs are passed to pandas.read_csv. Params
-            orient and lines will be ignored as always set to lines=True and
-            orient="records".
-
-    Returns:
-        Pandas DataFrame: the csv data as a dataframe, with the specified data types
-    """
-
-    if "lines" in kwargs:
-        warnings.warn("Ignoring lines in kwargs. Setting to lines=True.")
-        kwargs.pop("lines")
-    if "orient" in kwargs:
-        warnings.warn('Ignoring orient in kwargs. Setting to orient="records"')
-        kwargs.pop("orient")
-
-    df = pd.read_json(input_file, lines=True, orient="records", **kwargs)
-    df = cast_pandas_table_to_schema(
-        df=df,
-        metadata=metadata,
-        ignore_columns=ignore_columns,
-        drop_columns=drop_columns,
-        pd_integer=pd_integer,
-        pd_string=pd_string,
-        pd_boolean=pd_boolean,
-        pd_date_type=pd_date_type,
-        pd_timestamp_type=pd_timestamp_type,
-        bool_map=bool_map,
-    )
-
-    return df
diff --git a/arrow_pd_parser/parse/__init__.py b/arrow_pd_parser/parse/__init__.py
deleted file mode 100644
index e8db9ee..0000000
--- a/arrow_pd_parser/parse/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-from .arrow_parser import (
-    update_existing_schema,
-    cast_arrow_table_to_schema,
-    pa_read_csv,
-    pa_read_csv_to_pandas,
-    pa_read_json,
-    pa_read_json_to_pandas,
-    pa_read_parquet_to_pandas,
-    pa_read_parquet,
-)
-
-from .pandas_parser import (
-    cast_pandas_table_to_schema,
-    cast_pandas_column_to_schema,
-    pd_read_csv,
-    pd_read_json,
-    PandasCastError,
-)
diff --git a/arrow_pd_parser/reader.py b/arrow_pd_parser/reader.py
new file mode 100644
index 0000000..b5967b3
--- /dev/null
+++ b/arrow_pd_parser/reader.py
@@ -0,0 +1,53 @@
+import pandas as pd
+from mojap_metadata import Metadata
+from typing import Union
+
+from arrow_pd_parser._readers import (
+    ArrowParquetReader,
+    PandasCsvReader,
+    PandasJsonReader,
+    get_default_reader_from_file_format,
+)
+from arrow_pd_parser.utils import infer_file_format, FileFormat
+
+
+def read(
+    input_path: str,
+    metadata: Union[Metadata, dict] = None,
+    file_format: Union[FileFormat, str] = None,
+    parquet_expect_full_schema: bool = True,
+    **kwargs,
+) -> pd.DataFrame:
+    """
+    A function wrapper for csv.read(), json.read() or parquet.read().
+    Has an additional parameter "file_format". Will use the appropriate
+    reader based on the value of file_format (can be a str or FileFormat
+    Enum (PARQUET, CSV or JSON)).
+
+    If file_format=None, then will try to infer file format from input_path
+    and failing that metadata. Will error if no file type can be achieved.
+
+    See csv.read(), json.read() or parquet.read() for docsctring on
+    other params.
+    """
+    if file_format is None:
+        file_format = infer_file_format(input_path, metadata)
+    elif isinstance(file_format, str):
+        file_format = FileFormat.from_string(file_format)
+    else:
+        pass
+
+    reader = get_default_reader_from_file_format(file_format=file_format)
+    if file_format == FileFormat.PARQUET:
+        reader.expect_full_schema = parquet_expect_full_schema
+
+    return reader.read(
+        input_path=input_path,
+        metadata=metadata,
+        **kwargs,
+    )
+
+
+csv = PandasCsvReader()
+json = PandasJsonReader()
+parquet = ArrowParquetReader()
diff --git a/arrow_pd_parser/utils.py b/arrow_pd_parser/utils.py
new file mode 100644
index 0000000..81468da
--- /dev/null
+++ b/arrow_pd_parser/utils.py
@@ -0,0 +1,101 @@
+import os
+from copy import deepcopy
+from enum import Enum, auto
+from typing import Union, IO
+from mojap_metadata import Metadata
+
+
+class FileFormatNotFound(Exception):
+    pass
+
+
+class EngineNotImplementedError(Exception):
+    pass
+
+
+class FileFormat(Enum):
+    PARQUET = auto()
+    JSON = auto()
+    CSV = auto()
+
+    @classmethod
+    def from_string(cls, string: str):
+        s = string.strip().upper()
+
+        if s in ["JSON", "JSONL", "NDJSON"]:
+            s = "JSON"
+
+        return cls[s]
+
+
+def is_s3_filepath(input_file: Union[IO, str]) -> bool:
+    if isinstance(input_file, str):
+        return input_file.startswith("s3://")
+    else:
+        return False
+
+
+def match_file_format_to_str(s: str, raise_error=False) -> Union[FileFormat, None]:
+    for file_format in FileFormat.__members__.keys():
+        if file_format in s.upper():
+            return FileFormat[file_format]
+    if raise_error:
+        raise FileFormatNotFound(f"Could not determine file format from {s}")
+    else:
+        return None
+
+
+def infer_format_from_filepath(input_file) -> FileFormat:
+    fn = os.path.basename(input_file)
+    _, ext = fn.split(".", 1)
+    file_format = match_file_format_to_str(ext)
+    if file_format:
+        return file_format
+    else:
+        raise FileFormatNotFound(f"Could not infer file format from: {input_file}")
+
+
+def infer_file_format_from_meta(metadata: Union[Metadata, dict]):
+    if isinstance(metadata, Metadata):
+        file_format_str = metadata.file_format
+    else:
+        file_format_str = metadata.get("file_format", "")
+    file_format = match_file_format_to_str(file_format_str)
+    if file_format:
+        return file_format
+    else:
+        raise FileFormatNotFound("Could not infer file format from metadata")
+
+
+def infer_file_format(input_file, metadata: Union[Metadata, dict] = None):
+    file_format = None
+    try:
+        file_format = infer_format_from_filepath(input_file)
+    except FileFormatNotFound:
+        if metadata:
+            try:
+                file_format = infer_file_format_from_meta(metadata)
+            except FileFormatNotFound:
+                pass
+
+    if file_format:
+        return file_format
+    else:
+        raise FileFormatNotFound(
+            "Could not infer file_format from input_file or metadata"
+        )
+
+
+def validate_and_enrich_metadata(metadata: Union[Metadata, dict]) -> Metadata:
+    if isinstance(metadata, dict):
+        m = Metadata.from_dict(metadata)
+        m.set_col_type_category_from_types()
+        return m
+    elif isinstance(metadata, Metadata):
+        m = deepcopy(metadata)
+        m.set_col_type_category_from_types()
+        return metadata
+    else:
+        raise TypeError(
+            "Expecting metadata to be type Metadata or dict " f"got {type(metadata)}"
+        )
diff --git a/arrow_pd_parser/writer.py b/arrow_pd_parser/writer.py
new file mode 100644
index 0000000..9b94d4a
--- /dev/null
+++ b/arrow_pd_parser/writer.py
@@ -0,0 +1,53 @@
+from typing import Union
+
+import pandas as pd
+from mojap_metadata import Metadata
+
+from arrow_pd_parser._writers import (
+    ArrowParquetWriter,
+    PandasCsvWriter,
+    PandasJsonWriter,
+    get_default_writer_from_file_format,
+)
+from arrow_pd_parser.utils import infer_file_format, FileFormat
+
+
+def write(
+    df: pd.DataFrame,
+    output_path: str,
+    metadata: Union[Metadata, dict] = None,
+    file_format: Union[FileFormat, str] = None,
+    **kwargs,
+) -> pd.DataFrame:
+    """
+    A function wrapper for csv.write(), json.write() or parquet.write().
+    Has an additional parameter "file_format". Will use the appropriate
+    reader based on the value of file_format (can be a str or FileFormat
+    Enum (PARQUET, CSV or JSON)).
+
+    If file_format=None, then will try to infer file format from output_path
+    and failing that metadata. Will error if no file type can be achieved.
+
+    See csv.write(), json.write() or parquet.write() for docsctring on
+    other params.
+    """
+    if file_format is None:
+        file_format = infer_file_format(output_path, metadata)
+    elif isinstance(file_format, str):
+        file_format = FileFormat.from_string(file_format)
+    else:
+        pass
+
+    writer = get_default_writer_from_file_format(file_format=file_format)
+
+    return writer.write(
+        df,
+        output_path,
+        metadata=metadata,
+        **kwargs,
+    )
+
+
+csv = PandasCsvWriter()
+json = PandasJsonWriter()
+parquet = ArrowParquetWriter()
diff --git a/githooks/pre-commit b/githooks/pre-commit
index 38071e9..5fe1980 100755
--- a/githooks/pre-commit
+++ b/githooks/pre-commit
@@ -2,53 +2,6 @@
 #
 # commit without checks by adding --no-verify to your commit command
 
-# run Black on all .py files
-printf "\n\e[1mRunning Black Python code formatter\e[0m\n"
-hash black 2>/dev/null || { echo >&2 "Black not installed in this environment. Halting commit."; exit 1; }
-black . --check
-black_exit=$?
-
-# run flake8 on all .py files
-printf "\n\e[1mRunning flake8 Python linter\e[0m\n"
-hash flake8 2>/dev/null || { echo >&2 "flake8 not installed in this environment. Halting commit."; exit 1; }
-flake8 . --count
-flake_exit=$?
-
-# run yamllint on all .yaml and .yml files
-printf "\n\e[1mRunning yamllint .yaml linter\e[0m\n"
-hash yamllint 2>/dev/null || { echo >&2 "yamllint not installed in this environment. Halting commit."; exit 1; }
-yamllint .
-yaml_exit=$?
-
-printf "\n\e[1mRunning git-crypt status and checking for warnings\e[0m\n"
-# git-crypt check for unencrypted file warnings adapted from https://www.gitmemory.com/issue/AGWA/git-crypt/45/614607121
-if [ -d ".git-crypt" ]; then
-    hash git-crypt status | grep "\*\*\* WARNING"; test $? -eq 1
-fi
-printf "git-crypt fine\n\n"
-
-exec < /dev/tty
-
-if [ $black_exit -gt 0 ]; then 
-    printf "\e[1mBLACK WARNING\e[0m: not all of your Python files comply with Black\n"
-    printf "You might want to run Black on the files listed above before committing\n"
-fi
-
-if [ $flake_exit -gt 0 ]; then 
-    printf "\e[1mFLAKE8 WARNING\e[0m: not all of your Python files comply with Flake8\n"
-    printf "Please check if you want to make changes before committing\n"
-fi
-
-if [ $yaml_exit -gt 0 ]; then 
-    printf "\e[1mYAML WARNING\e[0m: not all of your .yaml files comply with yamllint\n"
-    printf "Please check if you want to make changes before committing\n"
-fi
-
-echo ""
-while read -p "Continue with commit? (Y/n) " yn; do
-    case $yn in
-        [Yy] ) printf "Continuing with commit\n"; break;;
-        [Nn] ) printf "Cancelling commit\n"; exit 1;;
-        * ) echo "Please answer y (yes) or n (no):" && continue;
-    esac
-done
+# add .version
+poetry version | awk '{print "__version__ = \"" $2 "\""}' > arrow_pd_parser/__init__.py
+git add arrow_pd_parser/__init__.py
diff --git a/poetry.lock b/poetry.lock
index c8f2784..bf52ae7 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1,3 +1,11 @@
+[[package]]
+name = "asn1crypto"
+version = "1.4.0"
+description = "Fast ASN.1 parser and serializer with definitions for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS#3, PKCS#7, PKCS#8, PKCS#12, PKCS#5, X.509 and TSP"
+category = "main"
+optional = false
+python-versions = "*"
+
 [[package]]
 name = "atomicwrites"
 version = "1.4.0"
@@ -20,6 +28,128 @@ docs = ["furo", "sphinx", "zope.interface", "sphinx-notfound-page"]
 tests = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "mypy", "pytest-mypy-plugins", "zope.interface"]
 tests_no_zope = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "mypy", "pytest-mypy-plugins"]
 
+[[package]]
+name = "awswrangler"
+version = "2.11.0"
+description = "Pandas on AWS."
+category = "main"
+optional = false
+python-versions = ">=3.6.2,<3.10"
+
+[package.dependencies]
+boto3 = ">=1.16.8,<2.0.0"
+botocore = ">=1.19.8,<2.0.0"
+numpy = ">=1.18.0,<2.0.0"
+openpyxl = ">=3.0.0,<3.1.0"
+pandas = {version = ">=1.2.0,<2.0.0", markers = "python_full_version >= \"3.7.1\" and python_full_version < \"4.0.0\""}
+pg8000 = ">=1.16.0,<1.22.0"
+pyarrow = ">=2.0.0,<5.1.0"
+pymysql = ">=0.9.0,<1.1.0"
+redshift-connector = ">=2.0.884,<2.1.0"
+
+[package.extras]
+sqlserver = ["pyodbc (>=4.0.30,<4.1.0)"]
+
+[[package]]
+name = "beautifulsoup4"
+version = "4.10.0"
+description = "Screen-scraping library"
+category = "main"
+optional = false
+python-versions = ">3.0.0"
+
+[package.dependencies]
+soupsieve = ">1.2"
+
+[package.extras]
+html5lib = ["html5lib"]
+lxml = ["lxml"]
+
+[[package]]
+name = "black"
+version = "21.9b0"
+description = "The uncompromising code formatter."
+category = "dev"
+optional = false
+python-versions = ">=3.6.2"
+
+[package.dependencies]
+click = ">=7.1.2"
+mypy-extensions = ">=0.4.3"
+pathspec = ">=0.9.0,<1"
+platformdirs = ">=2"
+regex = ">=2020.1.8"
+tomli = ">=0.2.6,<2.0.0"
+typed-ast = {version = ">=1.4.2", markers = "python_version < \"3.8\""}
+typing-extensions = ">=3.10.0.0"
+
+[package.extras]
+colorama = ["colorama (>=0.4.3)"]
+d = ["aiohttp (>=3.6.0)", "aiohttp-cors (>=0.4.0)"]
+jupyter = ["ipython (>=7.8.0)", "tokenize-rt (>=3.2.0)"]
+python2 = ["typed-ast (>=1.4.2)"]
+uvloop = ["uvloop (>=0.15.2)"]
+
+[[package]]
+name = "boto3"
+version = "1.18.47"
+description = "The AWS SDK for Python"
+category = "main"
+optional = false
+python-versions = ">= 3.6"
+
+[package.dependencies]
+botocore = ">=1.21.47,<1.22.0"
+jmespath = ">=0.7.1,<1.0.0"
+s3transfer = ">=0.5.0,<0.6.0"
+
+[package.extras]
+crt = ["botocore[crt] (>=1.21.0,<2.0a0)"]
+
+[[package]]
+name = "botocore"
+version = "1.21.47"
+description = "Low-level, data-driven core of boto 3."
+category = "main"
+optional = false
+python-versions = ">= 3.6"
+
+[package.dependencies]
+jmespath = ">=0.7.1,<1.0.0"
+python-dateutil = ">=2.1,<3.0.0"
+urllib3 = ">=1.25.4,<1.27"
+
+[package.extras]
+crt = ["awscrt (==0.11.24)"]
+
+[[package]]
+name = "certifi"
+version = "2021.5.30"
+description = "Python package for providing Mozilla's CA Bundle."
+category = "main"
+optional = false
+python-versions = "*"
+
+[[package]]
+name = "chardet"
+version = "4.0.0"
+description = "Universal encoding detector for Python 2 and 3"
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
+
+[[package]]
+name = "click"
+version = "8.0.1"
+description = "Composable command line interface toolkit"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+colorama = {version = "*", markers = "platform_system == \"Windows\""}
+importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}
+
 [[package]]
 name = "colorama"
 version = "0.4.4"
@@ -28,6 +158,14 @@ category = "dev"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
 
+[[package]]
+name = "et-xmlfile"
+version = "1.1.0"
+description = "An implementation of lxml.xmlfile for the standard library"
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
 [[package]]
 name = "flake8"
 version = "3.9.2"
@@ -42,9 +180,17 @@ mccabe = ">=0.6.0,<0.7.0"
 pycodestyle = ">=2.7.0,<2.8.0"
 pyflakes = ">=2.3.0,<2.4.0"
 
+[[package]]
+name = "idna"
+version = "2.10"
+description = "Internationalized Domain Names in Applications (IDNA)"
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+
 [[package]]
 name = "importlib-metadata"
-version = "4.5.0"
+version = "4.8.1"
 description = "Read metadata from Python packages"
 category = "main"
 optional = false
@@ -56,7 +202,8 @@ zipp = ">=0.5"
 
 [package.extras]
 docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)"]
-testing = ["pytest (>=4.6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "packaging", "pep517", "pyfakefs", "flufl.flake8", "pytest-black (>=0.3.7)", "pytest-mypy", "importlib-resources (>=1.3)"]
+perf = ["ipython"]
+testing = ["pytest (>=4.6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "packaging", "pep517", "pyfakefs", "flufl.flake8", "pytest-perf (>=0.9.2)", "pytest-black (>=0.3.7)", "pytest-mypy", "importlib-resources (>=1.3)"]
 
 [[package]]
 name = "iniconfig"
@@ -66,6 +213,14 @@ category = "dev"
 optional = false
 python-versions = "*"
 
+[[package]]
+name = "jmespath"
+version = "0.10.0"
+description = "JSON Matching Expressions"
+category = "main"
+optional = false
+python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
+
 [[package]]
 name = "jsonschema"
 version = "3.2.0"
@@ -84,6 +239,20 @@ six = ">=1.11.0"
 format = ["idna", "jsonpointer (>1.13)", "rfc3987", "strict-rfc3339", "webcolors"]
 format_nongpl = ["idna", "jsonpointer (>1.13)", "webcolors", "rfc3986-validator (>0.1.0)", "rfc3339-validator"]
 
+[[package]]
+name = "lxml"
+version = "4.6.3"
+description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API."
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, != 3.4.*"
+
+[package.extras]
+cssselect = ["cssselect (>=0.7)"]
+html5 = ["html5lib"]
+htmlsoup = ["beautifulsoup4"]
+source = ["Cython (>=0.29.7)"]
+
 [[package]]
 name = "mccabe"
 version = "0.6.1"
@@ -94,56 +263,75 @@ python-versions = "*"
 
 [[package]]
 name = "mojap-metadata"
-version = "1.2.1"
+version = "1.2.2"
 description = "A python package to manage metadata"
 category = "main"
 optional = false
-python-versions = ">=3.7,<4.0"
+python-versions = ">=3.7,<4"
 
 [package.dependencies]
 jsonschema = ">=3.0.0"
 parameterized = ">=0.7.0,<0.8.0"
-pyarrow = {version = ">=2.0.0,<5.0.0", optional = true, markers = "extra == \"arrow\""}
-PyYAML = ">=5,<6"
+pyarrow = {version = ">=3.0.0", optional = true, markers = "extra == \"arrow\""}
+PyYAML = ">=5.4.1"
 
 [package.extras]
-arrow = ["pyarrow (>=2.0.0,<5.0.0)"]
+arrow = ["pyarrow (>=3.0.0)"]
 etl-manager = ["etl-manager (>=7.4,<8.0)"]
 
+[[package]]
+name = "mypy-extensions"
+version = "0.4.3"
+description = "Experimental type system extensions for programs checked with the mypy typechecker."
+category = "dev"
+optional = false
+python-versions = "*"
+
 [[package]]
 name = "numpy"
-version = "1.20.3"
+version = "1.21.2"
 description = "NumPy is the fundamental package for array computing with Python."
 category = "main"
 optional = false
-python-versions = ">=3.7"
+python-versions = ">=3.7,<3.11"
+
+[[package]]
+name = "openpyxl"
+version = "3.0.9"
+description = "A Python library to read/write Excel 2010 xlsx/xlsm files"
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+et-xmlfile = "*"
 
 [[package]]
 name = "packaging"
-version = "20.9"
+version = "21.0"
 description = "Core utilities for Python packages"
-category = "dev"
+category = "main"
 optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+python-versions = ">=3.6"
 
 [package.dependencies]
 pyparsing = ">=2.0.2"
 
 [[package]]
 name = "pandas"
-version = "1.2.4"
+version = "1.3.3"
 description = "Powerful data structures for data analysis, time series, and statistics"
 category = "main"
 optional = false
 python-versions = ">=3.7.1"
 
 [package.dependencies]
-numpy = ">=1.16.5"
+numpy = ">=1.17.3"
 python-dateutil = ">=2.7.3"
 pytz = ">=2017.3"
 
 [package.extras]
-test = ["pytest (>=5.0.1)", "pytest-xdist", "hypothesis (>=3.58)"]
+test = ["hypothesis (>=3.58)", "pytest (>=6.0)", "pytest-xdist"]
 
 [[package]]
 name = "parameterized"
@@ -156,19 +344,51 @@ python-versions = "*"
 [package.extras]
 dev = ["jinja2"]
 
+[[package]]
+name = "pathspec"
+version = "0.9.0"
+description = "Utility library for gitignore style pattern matching of file paths."
+category = "dev"
+optional = false
+python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
+
+[[package]]
+name = "pg8000"
+version = "1.21.2"
+description = "PostgreSQL interface library"
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+scramp = ">=1.4.1"
+
+[[package]]
+name = "platformdirs"
+version = "2.3.0"
+description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[package.extras]
+docs = ["Sphinx (>=4)", "furo (>=2021.7.5b38)", "proselint (>=0.10.2)", "sphinx-autodoc-typehints (>=1.12)"]
+test = ["appdirs (==1.4.4)", "pytest (>=6)", "pytest-cov (>=2.7)", "pytest-mock (>=3.6)"]
+
 [[package]]
 name = "pluggy"
-version = "0.13.1"
+version = "1.0.0"
 description = "plugin and hook calling mechanisms for python"
 category = "dev"
 optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+python-versions = ">=3.6"
 
 [package.dependencies]
 importlib-metadata = {version = ">=0.12", markers = "python_version < \"3.8\""}
 
 [package.extras]
 dev = ["pre-commit", "tox"]
+testing = ["pytest", "pytest-benchmark"]
 
 [[package]]
 name = "py"
@@ -180,7 +400,7 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "pyarrow"
-version = "3.0.0"
+version = "5.0.0"
 description = "Python library for Apache Arrow"
 category = "main"
 optional = false
@@ -205,25 +425,37 @@ category = "dev"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
+[[package]]
+name = "pymysql"
+version = "1.0.2"
+description = "Pure Python MySQL Driver"
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[package.extras]
+ed25519 = ["PyNaCl (>=1.4.0)"]
+rsa = ["cryptography"]
+
 [[package]]
 name = "pyparsing"
 version = "2.4.7"
 description = "Python parsing module"
-category = "dev"
+category = "main"
 optional = false
 python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
 
 [[package]]
 name = "pyrsistent"
-version = "0.17.3"
+version = "0.18.0"
 description = "Persistent/Functional/Immutable data structures"
 category = "main"
 optional = false
-python-versions = ">=3.5"
+python-versions = ">=3.6"
 
 [[package]]
 name = "pytest"
-version = "6.2.4"
+version = "6.2.5"
 description = "pytest: simple powerful testing with Python"
 category = "dev"
 optional = false
@@ -236,7 +468,7 @@ colorama = {version = "*", markers = "sys_platform == \"win32\""}
 importlib-metadata = {version = ">=0.12", markers = "python_version < \"3.8\""}
 iniconfig = "*"
 packaging = "*"
-pluggy = ">=0.12,<1.0.0a1"
+pluggy = ">=0.12,<2.0"
 py = ">=1.8.2"
 toml = "*"
 
@@ -245,7 +477,7 @@ testing = ["argcomplete", "hypothesis (>=3.56)", "mock", "nose", "requests", "xm
 
 [[package]]
 name = "python-dateutil"
-version = "2.8.1"
+version = "2.8.2"
 description = "Extensions to the standard Python datetime module"
 category = "main"
 optional = false
@@ -270,6 +502,78 @@ category = "main"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
 
+[[package]]
+name = "redshift-connector"
+version = "2.0.887"
+description = "Redshift interface library"
+category = "main"
+optional = false
+python-versions = ">=3.5"
+
+[package.dependencies]
+beautifulsoup4 = ">=4.7.0,<5.0.0"
+boto3 = ">=1.16.8,<2.0.0"
+botocore = ">=1.19.8,<2.0.0"
+lxml = ">=4.6.2"
+packaging = "*"
+pytz = ">=2020.1,<2021.9"
+requests = ">=2.23.0,<2.25.2"
+scramp = ">=1.2.0,<1.5.0"
+
+[package.extras]
+full = ["numpy", "pandas"]
+
+[[package]]
+name = "regex"
+version = "2021.8.28"
+description = "Alternative regular expression module, to replace re."
+category = "dev"
+optional = false
+python-versions = "*"
+
+[[package]]
+name = "requests"
+version = "2.25.1"
+description = "Python HTTP for Humans."
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
+
+[package.dependencies]
+certifi = ">=2017.4.17"
+chardet = ">=3.0.2,<5"
+idna = ">=2.5,<3"
+urllib3 = ">=1.21.1,<1.27"
+
+[package.extras]
+security = ["pyOpenSSL (>=0.14)", "cryptography (>=1.3.4)"]
+socks = ["PySocks (>=1.5.6,!=1.5.7)", "win-inet-pton"]
+
+[[package]]
+name = "s3transfer"
+version = "0.5.0"
+description = "An Amazon S3 Transfer Manager"
+category = "main"
+optional = false
+python-versions = ">= 3.6"
+
+[package.dependencies]
+botocore = ">=1.12.36,<2.0a.0"
+
+[package.extras]
+crt = ["botocore[crt] (>=1.20.29,<2.0a.0)"]
+
+[[package]]
+name = "scramp"
+version = "1.4.1"
+description = "An implementation of the SCRAM protocol."
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+asn1crypto = ">=1.4.0"
+
 [[package]]
 name = "six"
 version = "1.16.0"
@@ -278,6 +582,14 @@ category = "main"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"
 
+[[package]]
+name = "soupsieve"
+version = "2.2.1"
+description = "A modern CSS selector implementation for Beautiful Soup."
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
 [[package]]
 name = "toml"
 version = "0.10.2"
@@ -286,17 +598,58 @@ category = "dev"
 optional = false
 python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
 
+[[package]]
+name = "tomli"
+version = "1.2.1"
+description = "A lil' TOML parser"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "typed-ast"
+version = "1.4.3"
+description = "a fork of Python 2 and 3 ast modules with type comment support"
+category = "dev"
+optional = false
+python-versions = "*"
+
 [[package]]
 name = "typing-extensions"
-version = "3.10.0.0"
+version = "3.10.0.2"
 description = "Backported and Experimental Type Hints for Python 3.5+"
 category = "main"
 optional = false
 python-versions = "*"
 
+[[package]]
+name = "urllib3"
+version = "1.26.7"
+description = "HTTP library with thread-safe connection pooling, file post, and more."
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4"
+
+[package.extras]
+brotli = ["brotlipy (>=0.6.0)"]
+secure = ["pyOpenSSL (>=0.14)", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "certifi", "ipaddress"]
+socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]
+
+[[package]]
+name = "yamllint"
+version = "1.26.3"
+description = "A linter for YAML files."
+category = "dev"
+optional = false
+python-versions = ">=3.5"
+
+[package.dependencies]
+pathspec = ">=0.5.3"
+pyyaml = "*"
+
 [[package]]
 name = "zipp"
-version = "3.4.1"
+version = "3.5.0"
 description = "Backport of pathlib-compatible object wrapper for zip files"
 category = "main"
 optional = false
@@ -304,14 +657,18 @@ python-versions = ">=3.6"
 
 [package.extras]
 docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)"]
-testing = ["pytest (>=4.6)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-cov", "pytest-enabler", "jaraco.itertools", "func-timeout", "pytest-black (>=0.3.7)", "pytest-mypy"]
+testing = ["pytest (>=4.6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "jaraco.itertools", "func-timeout", "pytest-black (>=0.3.7)", "pytest-mypy"]
 
 [metadata]
 lock-version = "1.1"
-python-versions = ">=3.7.1,<=3.9"
-content-hash = "dd9502b39e8e3df99cd19462fabea8849ed5273aed31d293ab74cd5dfe05631b"
+python-versions = ">=3.7.1,<3.10"
+content-hash = "f876c29ef158e4a31f6fcd849d0b23b23a0d2b483cf80eedd3387fd9996548ac"
 
 [metadata.files]
+asn1crypto = [
+    {file = "asn1crypto-1.4.0-py2.py3-none-any.whl", hash = "sha256:4bcdf33c861c7d40bdcd74d8e4dd7661aac320fcdf40b9a3f95b4ee12fde2fa8"},
+    {file = "asn1crypto-1.4.0.tar.gz", hash = "sha256:f4f6e119474e58e04a2b1af817eb585b4fd72bdd89b998624712b5c99be7641c"},
+]
 atomicwrites = [
     {file = "atomicwrites-1.4.0-py2.py3-none-any.whl", hash = "sha256:6d1784dea7c0c8d4a5172b6c620f40b6e4cbfdf96d783691f2e1302a7b88e197"},
     {file = "atomicwrites-1.4.0.tar.gz", hash = "sha256:ae70396ad1a434f9c7046fd2dd196fc04b12f9e91ffb859164193be8b6168a7a"},
@@ -320,116 +677,246 @@ attrs = [
     {file = "attrs-21.2.0-py2.py3-none-any.whl", hash = "sha256:149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"},
     {file = "attrs-21.2.0.tar.gz", hash = "sha256:ef6aaac3ca6cd92904cdd0d83f629a15f18053ec84e6432106f7a4d04ae4f5fb"},
 ]
+awswrangler = [
+    {file = "awswrangler-2.11.0-py3-none-any.whl", hash = "sha256:413e38c62aca7f52517347a92ee17353c9d6ab238ac8ebeca9778f8268f20ef9"},
+    {file = "awswrangler-2.11.0.tar.gz", hash = "sha256:b6cc7374a23447aadb7eb3c66b141ea020153a92e9cec82cb0f387413e9532b1"},
+]
+beautifulsoup4 = [
+    {file = "beautifulsoup4-4.10.0-py3-none-any.whl", hash = "sha256:9a315ce70049920ea4572a4055bc4bd700c940521d36fc858205ad4fcde149bf"},
+    {file = "beautifulsoup4-4.10.0.tar.gz", hash = "sha256:c23ad23c521d818955a4151a67d81580319d4bf548d3d49f4223ae041ff98891"},
+]
+black = [
+    {file = "black-21.9b0-py3-none-any.whl", hash = "sha256:380f1b5da05e5a1429225676655dddb96f5ae8c75bdf91e53d798871b902a115"},
+    {file = "black-21.9b0.tar.gz", hash = "sha256:7de4cfc7eb6b710de325712d40125689101d21d25283eed7e9998722cf10eb91"},
+]
+boto3 = [
+    {file = "boto3-1.18.47-py3-none-any.whl", hash = "sha256:52d87c94511226ccacd9e94335eb8f8f980b97964f63db92b97148b5486330a7"},
+    {file = "boto3-1.18.47.tar.gz", hash = "sha256:07260ec42a21c31a13b07ffbf219811a79be60d2eb9a99cedfec65c2a5f2155d"},
+]
+botocore = [
+    {file = "botocore-1.21.47-py3-none-any.whl", hash = "sha256:282951e8cde2c93fa436e5a7acfeb02f815de823bc3dac745ea51f971a1052e4"},
+    {file = "botocore-1.21.47.tar.gz", hash = "sha256:57637772b837da290239b1a9a6676ae2dd8a64833ab7770dc67d169d09bc80b3"},
+]
+certifi = [
+    {file = "certifi-2021.5.30-py2.py3-none-any.whl", hash = "sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8"},
+    {file = "certifi-2021.5.30.tar.gz", hash = "sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee"},
+]
+chardet = [
+    {file = "chardet-4.0.0-py2.py3-none-any.whl", hash = "sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5"},
+    {file = "chardet-4.0.0.tar.gz", hash = "sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa"},
+]
+click = [
+    {file = "click-8.0.1-py3-none-any.whl", hash = "sha256:fba402a4a47334742d782209a7c79bc448911afe1149d07bdabdf480b3e2f4b6"},
+    {file = "click-8.0.1.tar.gz", hash = "sha256:8c04c11192119b1ef78ea049e0a6f0463e4c48ef00a30160c704337586f3ad7a"},
+]
 colorama = [
     {file = "colorama-0.4.4-py2.py3-none-any.whl", hash = "sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2"},
     {file = "colorama-0.4.4.tar.gz", hash = "sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b"},
 ]
+et-xmlfile = [
+    {file = "et_xmlfile-1.1.0-py3-none-any.whl", hash = "sha256:a2ba85d1d6a74ef63837eed693bcb89c3f752169b0e3e7ae5b16ca5e1b3deada"},
+    {file = "et_xmlfile-1.1.0.tar.gz", hash = "sha256:8eb9e2bc2f8c97e37a2dc85a09ecdcdec9d8a396530a6d5a33b30b9a92da0c5c"},
+]
 flake8 = [
     {file = "flake8-3.9.2-py2.py3-none-any.whl", hash = "sha256:bf8fd333346d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907"},
     {file = "flake8-3.9.2.tar.gz", hash = "sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b"},
 ]
+idna = [
+    {file = "idna-2.10-py2.py3-none-any.whl", hash = "sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"},
+    {file = "idna-2.10.tar.gz", hash = "sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6"},
+]
 importlib-metadata = [
-    {file = "importlib_metadata-4.5.0-py3-none-any.whl", hash = "sha256:833b26fb89d5de469b24a390e9df088d4e52e4ba33b01dc5e0e4f41b81a16c00"},
-    {file = "importlib_metadata-4.5.0.tar.gz", hash = "sha256:b142cc1dd1342f31ff04bb7d022492b09920cb64fed867cd3ea6f80fe3ebd139"},
+    {file = "importlib_metadata-4.8.1-py3-none-any.whl", hash = "sha256:b618b6d2d5ffa2f16add5697cf57a46c76a56229b0ed1c438322e4e95645bd15"},
+    {file = "importlib_metadata-4.8.1.tar.gz", hash = "sha256:f284b3e11256ad1e5d03ab86bb2ccd6f5339688ff17a4d797a0fe7df326f23b1"},
 ]
 iniconfig = [
     {file = "iniconfig-1.1.1-py2.py3-none-any.whl", hash = "sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3"},
     {file = "iniconfig-1.1.1.tar.gz", hash = "sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32"},
 ]
+jmespath = [
+    {file = "jmespath-0.10.0-py2.py3-none-any.whl", hash = "sha256:cdf6525904cc597730141d61b36f2e4b8ecc257c420fa2f4549bac2c2d0cb72f"},
+    {file = "jmespath-0.10.0.tar.gz", hash = "sha256:b85d0567b8666149a93172712e68920734333c0ce7e89b78b3e987f71e5ed4f9"},
+]
 jsonschema = [
     {file = "jsonschema-3.2.0-py2.py3-none-any.whl", hash = "sha256:4e5b3cf8216f577bee9ce139cbe72eca3ea4f292ec60928ff24758ce626cd163"},
     {file = "jsonschema-3.2.0.tar.gz", hash = "sha256:c8a85b28d377cc7737e46e2d9f2b4f44ee3c0e1deac6bf46ddefc7187d30797a"},
 ]
+lxml = [
+    {file = "lxml-4.6.3-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:df7c53783a46febb0e70f6b05df2ba104610f2fb0d27023409734a3ecbb78fb2"},
+    {file = "lxml-4.6.3-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:1b7584d421d254ab86d4f0b13ec662a9014397678a7c4265a02a6d7c2b18a75f"},
+    {file = "lxml-4.6.3-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:079f3ae844f38982d156efce585bc540c16a926d4436712cf4baee0cce487a3d"},
+    {file = "lxml-4.6.3-cp27-cp27m-win32.whl", hash = "sha256:bc4313cbeb0e7a416a488d72f9680fffffc645f8a838bd2193809881c67dd106"},
+    {file = "lxml-4.6.3-cp27-cp27m-win_amd64.whl", hash = "sha256:8157dadbb09a34a6bd95a50690595e1fa0af1a99445e2744110e3dca7831c4ee"},
+    {file = "lxml-4.6.3-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:7728e05c35412ba36d3e9795ae8995e3c86958179c9770e65558ec3fdfd3724f"},
+    {file = "lxml-4.6.3-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:4bff24dfeea62f2e56f5bab929b4428ae6caba2d1eea0c2d6eb618e30a71e6d4"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:74f7d8d439b18fa4c385f3f5dfd11144bb87c1da034a466c5b5577d23a1d9b51"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:f90ba11136bfdd25cae3951af8da2e95121c9b9b93727b1b896e3fa105b2f586"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux2010_i686.whl", hash = "sha256:4c61b3a0db43a1607d6264166b230438f85bfed02e8cff20c22e564d0faff354"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux2014_x86_64.whl", hash = "sha256:5c8c163396cc0df3fd151b927e74f6e4acd67160d6c33304e805b84293351d16"},
+    {file = "lxml-4.6.3-cp35-cp35m-win32.whl", hash = "sha256:f2380a6376dfa090227b663f9678150ef27543483055cc327555fb592c5967e2"},
+    {file = "lxml-4.6.3-cp35-cp35m-win_amd64.whl", hash = "sha256:c4f05c5a7c49d2fb70223d0d5bcfbe474cf928310ac9fa6a7c6dddc831d0b1d4"},
+    {file = "lxml-4.6.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:d2e35d7bf1c1ac8c538f88d26b396e73dd81440d59c1ef8522e1ea77b345ede4"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:289e9ca1a9287f08daaf796d96e06cb2bc2958891d7911ac7cae1c5f9e1e0ee3"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:bccbfc27563652de7dc9bdc595cb25e90b59c5f8e23e806ed0fd623755b6565d"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:d916d31fd85b2f78c76400d625076d9124de3e4bda8b016d25a050cc7d603f24"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:820628b7b3135403540202e60551e741f9b6d3304371712521be939470b454ec"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux2014_x86_64.whl", hash = "sha256:c47ff7e0a36d4efac9fd692cfa33fbd0636674c102e9e8d9b26e1b93a94e7617"},
+    {file = "lxml-4.6.3-cp36-cp36m-win32.whl", hash = "sha256:5a0a14e264069c03e46f926be0d8919f4105c1623d620e7ec0e612a2e9bf1c04"},
+    {file = "lxml-4.6.3-cp36-cp36m-win_amd64.whl", hash = "sha256:92e821e43ad382332eade6812e298dc9701c75fe289f2a2d39c7960b43d1e92a"},
+    {file = "lxml-4.6.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:efd7a09678fd8b53117f6bae4fa3825e0a22b03ef0a932e070c0bdbb3a35e654"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:efac139c3f0bf4f0939f9375af4b02c5ad83a622de52d6dfa8e438e8e01d0eb0"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:0fbcf5565ac01dff87cbfc0ff323515c823081c5777a9fc7703ff58388c258c3"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:36108c73739985979bf302006527cf8a20515ce444ba916281d1c43938b8bb96"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:122fba10466c7bd4178b07dba427aa516286b846b2cbd6f6169141917283aae2"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux2014_x86_64.whl", hash = "sha256:cdaf11d2bd275bf391b5308f86731e5194a21af45fbaaaf1d9e8147b9160ea92"},
+    {file = "lxml-4.6.3-cp37-cp37m-win32.whl", hash = "sha256:3439c71103ef0e904ea0a1901611863e51f50b5cd5e8654a151740fde5e1cade"},
+    {file = "lxml-4.6.3-cp37-cp37m-win_amd64.whl", hash = "sha256:4289728b5e2000a4ad4ab8da6e1db2e093c63c08bdc0414799ee776a3f78da4b"},
+    {file = "lxml-4.6.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:b007cbb845b28db4fb8b6a5cdcbf65bacb16a8bd328b53cbc0698688a68e1caa"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux1_i686.whl", hash = "sha256:76fa7b1362d19f8fbd3e75fe2fb7c79359b0af8747e6f7141c338f0bee2f871a"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:26e761ab5b07adf5f555ee82fb4bfc35bf93750499c6c7614bd64d12aaa67927"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:e1cbd3f19a61e27e011e02f9600837b921ac661f0c40560eefb366e4e4fb275e"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:66e575c62792c3f9ca47cb8b6fab9e35bab91360c783d1606f758761810c9791"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux2014_x86_64.whl", hash = "sha256:1b38116b6e628118dea5b2186ee6820ab138dbb1e24a13e478490c7db2f326ae"},
+    {file = "lxml-4.6.3-cp38-cp38-win32.whl", hash = "sha256:89b8b22a5ff72d89d48d0e62abb14340d9e99fd637d046c27b8b257a01ffbe28"},
+    {file = "lxml-4.6.3-cp38-cp38-win_amd64.whl", hash = "sha256:2a9d50e69aac3ebee695424f7dbd7b8c6d6eb7de2a2eb6b0f6c7db6aa41e02b7"},
+    {file = "lxml-4.6.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ce256aaa50f6cc9a649c51be3cd4ff142d67295bfc4f490c9134d0f9f6d58ef0"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux1_i686.whl", hash = "sha256:7610b8c31688f0b1be0ef882889817939490a36d0ee880ea562a4e1399c447a1"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:f8380c03e45cf09f8557bdaa41e1fa7c81f3ae22828e1db470ab2a6c96d8bc23"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:3082c518be8e97324390614dacd041bb1358c882d77108ca1957ba47738d9d59"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:884ab9b29feaca361f7f88d811b1eea9bfca36cf3da27768d28ad45c3ee6f969"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux2014_x86_64.whl", hash = "sha256:6f12e1427285008fd32a6025e38e977d44d6382cf28e7201ed10d6c1698d2a9a"},
+    {file = "lxml-4.6.3-cp39-cp39-win32.whl", hash = "sha256:33bb934a044cf32157c12bfcfbb6649807da20aa92c062ef51903415c704704f"},
+    {file = "lxml-4.6.3-cp39-cp39-win_amd64.whl", hash = "sha256:542d454665a3e277f76954418124d67516c5f88e51a900365ed54a9806122b83"},
+    {file = "lxml-4.6.3.tar.gz", hash = "sha256:39b78571b3b30645ac77b95f7c69d1bffc4cf8c3b157c435a34da72e78c82468"},
+]
 mccabe = [
     {file = "mccabe-0.6.1-py2.py3-none-any.whl", hash = "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42"},
     {file = "mccabe-0.6.1.tar.gz", hash = "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"},
 ]
 mojap-metadata = [
-    {file = "mojap-metadata-1.2.1.tar.gz", hash = "sha256:734d3f724fb719390b34d256f741b4b11bf1857dbddb8ad0f8c710afb47fe839"},
-    {file = "mojap_metadata-1.2.1-py3-none-any.whl", hash = "sha256:d8e47deff5f40d007a77debb9db0712cc45ea2f61fc6934400029c9e70a23727"},
+    {file = "mojap-metadata-1.2.2.tar.gz", hash = "sha256:4d1997edc0159cd8740ecb325d2804f1afef4c70fe2d3b688fd941a458cf11de"},
+    {file = "mojap_metadata-1.2.2-py3-none-any.whl", hash = "sha256:1e33660e463f45c98bd4b7d3b3117b3a78e169a38c679d7e67a79e1f96072337"},
+]
+mypy-extensions = [
+    {file = "mypy_extensions-0.4.3-py2.py3-none-any.whl", hash = "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d"},
+    {file = "mypy_extensions-0.4.3.tar.gz", hash = "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"},
 ]
 numpy = [
-    {file = "numpy-1.20.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:70eb5808127284c4e5c9e836208e09d685a7978b6a216db85960b1a112eeace8"},
-    {file = "numpy-1.20.3-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:6ca2b85a5997dabc38301a22ee43c82adcb53ff660b89ee88dded6b33687e1d8"},
-    {file = "numpy-1.20.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:c5bf0e132acf7557fc9bb8ded8b53bbbbea8892f3c9a1738205878ca9434206a"},
-    {file = "numpy-1.20.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:db250fd3e90117e0312b611574cd1b3f78bec046783195075cbd7ba9c3d73f16"},
-    {file = "numpy-1.20.3-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:637d827248f447e63585ca3f4a7d2dfaa882e094df6cfa177cc9cf9cd6cdf6d2"},
-    {file = "numpy-1.20.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:8b7bb4b9280da3b2856cb1fc425932f46fba609819ee1c62256f61799e6a51d2"},
-    {file = "numpy-1.20.3-cp37-cp37m-win32.whl", hash = "sha256:67d44acb72c31a97a3d5d33d103ab06d8ac20770e1c5ad81bdb3f0c086a56cf6"},
-    {file = "numpy-1.20.3-cp37-cp37m-win_amd64.whl", hash = "sha256:43909c8bb289c382170e0282158a38cf306a8ad2ff6dfadc447e90f9961bef43"},
-    {file = "numpy-1.20.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:f1452578d0516283c87608a5a5548b0cdde15b99650efdfd85182102ef7a7c17"},
-    {file = "numpy-1.20.3-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:6e51534e78d14b4a009a062641f465cfaba4fdcb046c3ac0b1f61dd97c861b1b"},
-    {file = "numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:e515c9a93aebe27166ec9593411c58494fa98e5fcc219e47260d9ab8a1cc7f9f"},
-    {file = "numpy-1.20.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c1c09247ccea742525bdb5f4b5ceeacb34f95731647fe55774aa36557dbb5fa4"},
-    {file = "numpy-1.20.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:66fbc6fed94a13b9801fb70b96ff30605ab0a123e775a5e7a26938b717c5d71a"},
-    {file = "numpy-1.20.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:ea9cff01e75a956dbee133fa8e5b68f2f92175233de2f88de3a682dd94deda65"},
-    {file = "numpy-1.20.3-cp38-cp38-win32.whl", hash = "sha256:f39a995e47cb8649673cfa0579fbdd1cdd33ea497d1728a6cb194d6252268e48"},
-    {file = "numpy-1.20.3-cp38-cp38-win_amd64.whl", hash = "sha256:1676b0a292dd3c99e49305a16d7a9f42a4ab60ec522eac0d3dd20cdf362ac010"},
-    {file = "numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:830b044f4e64a76ba71448fce6e604c0fc47a0e54d8f6467be23749ac2cbd2fb"},
-    {file = "numpy-1.20.3-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:55b745fca0a5ab738647d0e4db099bd0a23279c32b31a783ad2ccea729e632df"},
-    {file = "numpy-1.20.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:5d050e1e4bc9ddb8656d7b4f414557720ddcca23a5b88dd7cff65e847864c400"},
-    {file = "numpy-1.20.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a9c65473ebc342715cb2d7926ff1e202c26376c0dcaaee85a1fd4b8d8c1d3b2f"},
-    {file = "numpy-1.20.3-cp39-cp39-win32.whl", hash = "sha256:16f221035e8bd19b9dc9a57159e38d2dd060b48e93e1d843c49cb370b0f415fd"},
-    {file = "numpy-1.20.3-cp39-cp39-win_amd64.whl", hash = "sha256:6690080810f77485667bfbff4f69d717c3be25e5b11bb2073e76bb3f578d99b4"},
-    {file = "numpy-1.20.3-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:4e465afc3b96dbc80cf4a5273e5e2b1e3451286361b4af70ce1adb2984d392f9"},
-    {file = "numpy-1.20.3.zip", hash = "sha256:e55185e51b18d788e49fe8305fd73ef4470596b33fc2c1ceb304566b99c71a69"},
+    {file = "numpy-1.21.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:52a664323273c08f3b473548bf87c8145b7513afd63e4ebba8496ecd3853df13"},
+    {file = "numpy-1.21.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:51a7b9db0a2941434cd930dacaafe0fc9da8f3d6157f9d12f761bbde93f46218"},
+    {file = "numpy-1.21.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:9f2dc79c093f6c5113718d3d90c283f11463d77daa4e83aeeac088ec6a0bda52"},
+    {file = "numpy-1.21.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:a55e4d81c4260386f71d22294795c87609164e22b28ba0d435850fbdf82fc0c5"},
+    {file = "numpy-1.21.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:426a00b68b0d21f2deb2ace3c6d677e611ad5a612d2c76494e24a562a930c254"},
+    {file = "numpy-1.21.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:298156f4d3d46815eaf0fcf0a03f9625fc7631692bd1ad851517ab93c3168fc6"},
+    {file = "numpy-1.21.2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:09858463db6dd9f78b2a1a05c93f3b33d4f65975771e90d2cf7aadb7c2f66edf"},
+    {file = "numpy-1.21.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:805459ad8baaf815883d0d6f86e45b3b0b67d823a8f3fa39b1ed9c45eaf5edf1"},
+    {file = "numpy-1.21.2-cp37-cp37m-win32.whl", hash = "sha256:f545c082eeb09ae678dd451a1b1dbf17babd8a0d7adea02897a76e639afca310"},
+    {file = "numpy-1.21.2-cp37-cp37m-win_amd64.whl", hash = "sha256:b160b9a99ecc6559d9e6d461b95c8eec21461b332f80267ad2c10394b9503496"},
+    {file = "numpy-1.21.2-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:a5109345f5ce7ddb3840f5970de71c34a0ff7fceb133c9441283bb8250f532a3"},
+    {file = "numpy-1.21.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:209666ce9d4a817e8a4597cd475b71b4878a85fa4b8db41d79fdb4fdee01dde2"},
+    {file = "numpy-1.21.2-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:c01b59b33c7c3ba90744f2c695be571a3bd40ab2ba7f3d169ffa6db3cfba614f"},
+    {file = "numpy-1.21.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:e42029e184008a5fd3d819323345e25e2337b0ac7f5c135b7623308530209d57"},
+    {file = "numpy-1.21.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:7fdc7689daf3b845934d67cb221ba8d250fdca20ac0334fea32f7091b93f00d3"},
+    {file = "numpy-1.21.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:550564024dc5ceee9421a86fc0fb378aa9d222d4d0f858f6669eff7410c89bef"},
+    {file = "numpy-1.21.2-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:bf75d5825ef47aa51d669b03ce635ecb84d69311e05eccea083f31c7570c9931"},
+    {file = "numpy-1.21.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:a9da45b748caad72ea4a4ed57e9cd382089f33c5ec330a804eb420a496fa760f"},
+    {file = "numpy-1.21.2-cp38-cp38-win32.whl", hash = "sha256:e167b9805de54367dcb2043519382be541117503ce99e3291cc9b41ca0a83557"},
+    {file = "numpy-1.21.2-cp38-cp38-win_amd64.whl", hash = "sha256:466e682264b14982012887e90346d33435c984b7fead7b85e634903795c8fdb0"},
+    {file = "numpy-1.21.2-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:dd0e3651d210068d13e18503d75aaa45656eef51ef0b261f891788589db2cc38"},
+    {file = "numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:92a0ab128b07799dd5b9077a9af075a63467d03ebac6f8a93e6440abfea4120d"},
+    {file = "numpy-1.21.2-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:fde50062d67d805bc96f1a9ecc0d37bfc2a8f02b937d2c50824d186aa91f2419"},
+    {file = "numpy-1.21.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:640c1ccfd56724f2955c237b6ccce2e5b8607c3bc1cc51d3933b8c48d1da3723"},
+    {file = "numpy-1.21.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:5de64950137f3a50b76ce93556db392e8f1f954c2d8207f78a92d1f79aa9f737"},
+    {file = "numpy-1.21.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b342064e647d099ca765f19672696ad50c953cac95b566af1492fd142283580f"},
+    {file = "numpy-1.21.2-cp39-cp39-win32.whl", hash = "sha256:30fc68307c0155d2a75ad19844224be0f2c6f06572d958db4e2053f816b859ad"},
+    {file = "numpy-1.21.2-cp39-cp39-win_amd64.whl", hash = "sha256:b5e8590b9245803c849e09bae070a8e1ff444f45e3f0bed558dd722119eea724"},
+    {file = "numpy-1.21.2-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:d96a6a7d74af56feb11e9a443150216578ea07b7450f7c05df40eec90af7f4a7"},
+    {file = "numpy-1.21.2.zip", hash = "sha256:423216d8afc5923b15df86037c6053bf030d15cc9e3224206ef868c2d63dd6dc"},
+]
+openpyxl = [
+    {file = "openpyxl-3.0.9-py2.py3-none-any.whl", hash = "sha256:8f3b11bd896a95468a4ab162fc4fcd260d46157155d1f8bfaabb99d88cfcf79f"},
+    {file = "openpyxl-3.0.9.tar.gz", hash = "sha256:40f568b9829bf9e446acfffce30250ac1fa39035124d55fc024025c41481c90f"},
 ]
 packaging = [
-    {file = "packaging-20.9-py2.py3-none-any.whl", hash = "sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"},
-    {file = "packaging-20.9.tar.gz", hash = "sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5"},
+    {file = "packaging-21.0-py3-none-any.whl", hash = "sha256:c86254f9220d55e31cc94d69bade760f0847da8000def4dfe1c6b872fd14ff14"},
+    {file = "packaging-21.0.tar.gz", hash = "sha256:7dc96269f53a4ccec5c0670940a4281106dd0bb343f47b7471f779df49c2fbe7"},
 ]
 pandas = [
-    {file = "pandas-1.2.4-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:c601c6fdebc729df4438ec1f62275d6136a0dd14d332fc0e8ce3f7d2aadb4dd6"},
-    {file = "pandas-1.2.4-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:8d4c74177c26aadcfb4fd1de6c1c43c2bf822b3e0fc7a9b409eeaf84b3e92aaa"},
-    {file = "pandas-1.2.4-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:b730add5267f873b3383c18cac4df2527ac4f0f0eed1c6cf37fcb437e25cf558"},
-    {file = "pandas-1.2.4-cp37-cp37m-win32.whl", hash = "sha256:2cb7e8f4f152f27dc93f30b5c7a98f6c748601ea65da359af734dd0cf3fa733f"},
-    {file = "pandas-1.2.4-cp37-cp37m-win_amd64.whl", hash = "sha256:2111c25e69fa9365ba80bbf4f959400054b2771ac5d041ed19415a8b488dc70a"},
-    {file = "pandas-1.2.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:167693a80abc8eb28051fbd184c1b7afd13ce2c727a5af47b048f1ea3afefff4"},
-    {file = "pandas-1.2.4-cp38-cp38-manylinux1_i686.whl", hash = "sha256:612add929bf3ba9d27b436cc8853f5acc337242d6b584203f207e364bb46cb12"},
-    {file = "pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:971e2a414fce20cc5331fe791153513d076814d30a60cd7348466943e6e909e4"},
-    {file = "pandas-1.2.4-cp38-cp38-win32.whl", hash = "sha256:68d7baa80c74aaacbed597265ca2308f017859123231542ff8a5266d489e1858"},
-    {file = "pandas-1.2.4-cp38-cp38-win_amd64.whl", hash = "sha256:bd659c11a4578af740782288cac141a322057a2e36920016e0fc7b25c5a4b686"},
-    {file = "pandas-1.2.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9db70ffa8b280bb4de83f9739d514cd0735825e79eef3a61d312420b9f16b758"},
-    {file = "pandas-1.2.4-cp39-cp39-manylinux1_i686.whl", hash = "sha256:298f0553fd3ba8e002c4070a723a59cdb28eda579f3e243bc2ee397773f5398b"},
-    {file = "pandas-1.2.4-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:52d2472acbb8a56819a87aafdb8b5b6d2b3386e15c95bde56b281882529a7ded"},
-    {file = "pandas-1.2.4-cp39-cp39-win32.whl", hash = "sha256:d0877407359811f7b853b548a614aacd7dea83b0c0c84620a9a643f180060950"},
-    {file = "pandas-1.2.4-cp39-cp39-win_amd64.whl", hash = "sha256:2b063d41803b6a19703b845609c0b700913593de067b552a8b24dd8eeb8c9895"},
-    {file = "pandas-1.2.4.tar.gz", hash = "sha256:649ecab692fade3cbfcf967ff936496b0cfba0af00a55dfaacd82bdda5cb2279"},
+    {file = "pandas-1.3.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:68408a39a54ebadb9014ee5a4fae27b2fe524317bc80adf56c9ac59e8f8ea431"},
+    {file = "pandas-1.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:86b16b1b920c4cb27fdd65a2c20258bcd9c794be491290660722bb0ea765054d"},
+    {file = "pandas-1.3.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:37d63e78e87eb3791da7be4100a65da0383670c2b59e493d9e73098d7a879226"},
+    {file = "pandas-1.3.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:53e2fb11f86f6253bb1df26e3aeab3bf2e000aaa32a953ec394571bec5dc6fd6"},
+    {file = "pandas-1.3.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7326b37de08d42dd3fff5b7ef7691d0fd0bf2428f4ba5a2bdc3b3247e9a52e4c"},
+    {file = "pandas-1.3.3-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ed2f29b4da6f6ae7c68f4b3708d9d9e59fa89b2f9e87c2b64ce055cbd39f729e"},
+    {file = "pandas-1.3.3-cp37-cp37m-win32.whl", hash = "sha256:3f5020613c1d8e304840c34aeb171377dc755521bf5e69804991030c2a48aec3"},
+    {file = "pandas-1.3.3-cp37-cp37m-win_amd64.whl", hash = "sha256:c399200631db9bd9335d013ec7fce4edb98651035c249d532945c78ad453f23a"},
+    {file = "pandas-1.3.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:a800df4e101b721e94d04c355e611863cc31887f24c0b019572e26518cbbcab6"},
+    {file = "pandas-1.3.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3334a5a9eeaca953b9db1b2b165dcdc5180b5011f3bec3a57a3580c9c22eae68"},
+    {file = "pandas-1.3.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49fd2889d8116d7acef0709e4c82b8560a8b22b0f77471391d12c27596e90267"},
+    {file = "pandas-1.3.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7557b39c8e86eb0543a17a002ac1ea0f38911c3c17095bc9350d0a65b32d801c"},
+    {file = "pandas-1.3.3-cp38-cp38-win32.whl", hash = "sha256:629138b7cf81a2e55aa29ce7b04c1cece20485271d1f6c469c6a0c03857db6a4"},
+    {file = "pandas-1.3.3-cp38-cp38-win_amd64.whl", hash = "sha256:45649503e167d45360aa7c52f18d1591a6d5c70d2f3a26bc90a3297a30ce9a66"},
+    {file = "pandas-1.3.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ebbed7312547a924df0cbe133ff1250eeb94cdff3c09a794dc991c5621c8c735"},
+    {file = "pandas-1.3.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a9f1b54d7efc9df05320b14a48fb18686f781aa66cc7b47bb62fabfc67a0985c"},
+    {file = "pandas-1.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e9bc59855598cb57f68fdabd4897d3ed2bc3a3b3bef7b868a0153c4cd03f3207"},
+    {file = "pandas-1.3.3-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4def2ef2fb7fcd62f2aa51bacb817ee9029e5c8efe42fe527ba21f6a3ddf1a9f"},
+    {file = "pandas-1.3.3-cp39-cp39-win32.whl", hash = "sha256:f7d84f321674c2f0f31887ee6d5755c54ca1ea5e144d6d54b3bbf566dd9ea0cc"},
+    {file = "pandas-1.3.3-cp39-cp39-win_amd64.whl", hash = "sha256:e574c2637c9d27f322e911650b36e858c885702c5996eda8a5a60e35e6648cf2"},
+    {file = "pandas-1.3.3.tar.gz", hash = "sha256:272c8cb14aa9793eada6b1ebe81994616e647b5892a370c7135efb2924b701df"},
 ]
 parameterized = [
     {file = "parameterized-0.7.5-py2.py3-none-any.whl", hash = "sha256:b36f709fcfd9e8c144c1374b16a40fc0b5984efe93e41d53e6d228ae2a1540c5"},
     {file = "parameterized-0.7.5.tar.gz", hash = "sha256:b5e6af67b9e49485e30125b1c8f031ffa81a265ca08bfa73f31551bf03cf68c4"},
 ]
+pathspec = [
+    {file = "pathspec-0.9.0-py2.py3-none-any.whl", hash = "sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a"},
+    {file = "pathspec-0.9.0.tar.gz", hash = "sha256:e564499435a2673d586f6b2130bb5b95f04a3ba06f81b8f895b651a3c76aabb1"},
+]
+pg8000 = [
+    {file = "pg8000-1.21.2-py3-none-any.whl", hash = "sha256:e5cffa2a561c5196a69118cdb6e94b6915bf6cc069790a902652a8f52799f586"},
+    {file = "pg8000-1.21.2.tar.gz", hash = "sha256:36a3b517408334967c1fa0d29656da03608d63122a372ec92c85f49aed2d24e3"},
+]
+platformdirs = [
+    {file = "platformdirs-2.3.0-py3-none-any.whl", hash = "sha256:8003ac87717ae2c7ee1ea5a84a1a61e87f3fbd16eb5aadba194ea30a9019f648"},
+    {file = "platformdirs-2.3.0.tar.gz", hash = "sha256:15b056538719b1c94bdaccb29e5f81879c7f7f0f4a153f46086d155dffcd4f0f"},
+]
 pluggy = [
-    {file = "pluggy-0.13.1-py2.py3-none-any.whl", hash = "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"},
-    {file = "pluggy-0.13.1.tar.gz", hash = "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0"},
+    {file = "pluggy-1.0.0-py2.py3-none-any.whl", hash = "sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3"},
+    {file = "pluggy-1.0.0.tar.gz", hash = "sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159"},
 ]
 py = [
     {file = "py-1.10.0-py2.py3-none-any.whl", hash = "sha256:3b80836aa6d1feeaa108e046da6423ab8f6ceda6468545ae8d02d9d58d18818a"},
     {file = "py-1.10.0.tar.gz", hash = "sha256:21b81bda15b66ef5e1a777a21c4dcd9c20ad3efd0b3f817e7a809035269e1bd3"},
 ]
 pyarrow = [
-    {file = "pyarrow-3.0.0-cp36-cp36m-macosx_10_13_x86_64.whl", hash = "sha256:03e2435da817bc2b5d0fad6f2e53305eb36c24004ddfcb2b30e4217a1a80cf22"},
-    {file = "pyarrow-3.0.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:2be3a9eab4bfd00024dc3c83fa03de1c1d04a0f47ebaf3dc483cd100546eacbf"},
-    {file = "pyarrow-3.0.0-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:a76031ef19d11db2fef79a97cc69997c97bea35aa07efbe042a177c7e3b1a390"},
-    {file = "pyarrow-3.0.0-cp36-cp36m-manylinux2014_x86_64.whl", hash = "sha256:a07e286e81ceb20f8f0c45f69760d2ebc434fe83794d5f9b44f89fc2dc6dc24d"},
-    {file = "pyarrow-3.0.0-cp36-cp36m-win_amd64.whl", hash = "sha256:cfea99a01d844c3db5e25374a6cdcf3b5ba1698bfe95d41272c295a4581e884c"},
-    {file = "pyarrow-3.0.0-cp37-cp37m-macosx_10_13_x86_64.whl", hash = "sha256:d5666a7fa2668f3ff95df028c2072d59e8b17e73d682068e8505dafa2688f3cc"},
-    {file = "pyarrow-3.0.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:3ea6574d1ae2d9bff7e6e1715f64c31bdc01b42387a5c78311a8ce9c09cfe135"},
-    {file = "pyarrow-3.0.0-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:2d5c95eb04a3d2e786e097b53534893eade6c8b3faf10f53a06143384b4446b1"},
-    {file = "pyarrow-3.0.0-cp37-cp37m-manylinux2014_x86_64.whl", hash = "sha256:31e6fc0868963aba4e6b8a3e218c9a5ff347bca870d622da0b3d58269d0c5398"},
-    {file = "pyarrow-3.0.0-cp37-cp37m-win_amd64.whl", hash = "sha256:960a9b0fd599601ddac42f16d5acf049637ec08957359c6741d6eb2bf0dbae97"},
-    {file = "pyarrow-3.0.0-cp38-cp38-macosx_10_13_x86_64.whl", hash = "sha256:2c3353d38d137f1158595b3b18dcef711f3d8fdb57cf7ae2d861d07235064bc1"},
-    {file = "pyarrow-3.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:72206cde1857d5420601feae75f53921cffab4326b42262a858c7b8be67982b7"},
-    {file = "pyarrow-3.0.0-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:dec007a0f7adba86bd170252140ede01646b45c3a470d5862ce00d8e40cd29bd"},
-    {file = "pyarrow-3.0.0-cp38-cp38-manylinux2014_x86_64.whl", hash = "sha256:bf6684fe9e38f8ddb696e38901461eab783ec1d565974ebd5862270320b3e27f"},
-    {file = "pyarrow-3.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:3b46487c45faaea8d1a5aa65002e2832ae2e1c9e68ecb461cda4fa59891cf490"},
-    {file = "pyarrow-3.0.0-cp39-cp39-macosx_10_13_x86_64.whl", hash = "sha256:978bbe8ec9090d1133a25f00f32ed92600f9d315fbfa29a17952bee01f0d7fe5"},
-    {file = "pyarrow-3.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:b7a8903f2b8a80498725ef5d4a35cd7dd5a98b74e080d42692545e61a6cbfbe4"},
-    {file = "pyarrow-3.0.0-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:b1cf92df9f336f31706249e543dc0ffce3c67a78204ce540f1173c6c07dfafec"},
-    {file = "pyarrow-3.0.0-cp39-cp39-manylinux2014_x86_64.whl", hash = "sha256:b08c119cc2b9fcd1567797fedb245a2f4352a3084a22b7298272afe7cf7a4730"},
-    {file = "pyarrow-3.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:5faa2dc73444bdcf042f121383965a47362be1f946303d46e8fd80f8d26cd90c"},
-    {file = "pyarrow-3.0.0.tar.gz", hash = "sha256:4bf8cc43e1db1e0517466209ee8e8f459d9b5e1b4074863317f2a965cf59889e"},
+    {file = "pyarrow-5.0.0-cp36-cp36m-macosx_10_13_x86_64.whl", hash = "sha256:e9ec80f4a77057498cf4c5965389e42e7f6a618b6859e6dd615e57505c9167a6"},
+    {file = "pyarrow-5.0.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:b1453c2411b5062ba6bf6832dbc4df211ad625f678c623a2ee177aee158f199b"},
+    {file = "pyarrow-5.0.0-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:9e04d3621b9f2f23898eed0d044203f66c156d880f02c5534a7f9947ebb1a4af"},
+    {file = "pyarrow-5.0.0-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:64f30aa6b28b666a925d11c239344741850eb97c29d3aa0f7187918cf82494f7"},
+    {file = "pyarrow-5.0.0-cp36-cp36m-manylinux2014_x86_64.whl", hash = "sha256:99c8b0f7e2ce2541dd4c0c0101d9944bb8e592ae3295fe7a2f290ab99222666d"},
+    {file = "pyarrow-5.0.0-cp36-cp36m-win_amd64.whl", hash = "sha256:456a4488ae810a0569d1adf87dbc522bcc9a0e4a8d1809b934ca28c163d8edce"},
+    {file = "pyarrow-5.0.0-cp37-cp37m-macosx_10_13_x86_64.whl", hash = "sha256:c5493d2414d0d690a738aac8dd6d38518d1f9b870e52e24f89d8d7eb3afd4161"},
+    {file = "pyarrow-5.0.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1832709281efefa4f199c639e9f429678286329860188e53beeda71750775923"},
+    {file = "pyarrow-5.0.0-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:b6387d2058d95fa48ccfedea810a768187affb62f4a3ef6595fa30bf9d1a65cf"},
+    {file = "pyarrow-5.0.0-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:bbe2e439bec2618c74a3bb259700c8a7353dc2ea0c5a62686b6cf04a50ab1e0d"},
+    {file = "pyarrow-5.0.0-cp37-cp37m-manylinux2014_x86_64.whl", hash = "sha256:5c0d1b68e67bb334a5af0cecdf9b6a702aaa4cc259c5cbb71b25bbed40fcedaf"},
+    {file = "pyarrow-5.0.0-cp37-cp37m-win_amd64.whl", hash = "sha256:6e937ce4a40ea0cc7896faff96adecadd4485beb53fbf510b46858e29b2e75ae"},
+    {file = "pyarrow-5.0.0-cp38-cp38-macosx_10_13_x86_64.whl", hash = "sha256:7560332e5846f0e7830b377c14c93624e24a17f91c98f0b25dafb0ca1ea6ba02"},
+    {file = "pyarrow-5.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:53e550dec60d1ab86cba3afa1719dc179a8bc9632a0e50d9fe91499cf0a7f2bc"},
+    {file = "pyarrow-5.0.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:2d26186ca9748a1fb89ae6c1fa04fb343a4279b53f118734ea8096f15d66c820"},
+    {file = "pyarrow-5.0.0-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:7c4edd2bacee3eea6c8c28bddb02347f9d41a55ec9692c71c6de6e47c62a7f0d"},
+    {file = "pyarrow-5.0.0-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:601b0aabd6fb066429e706282934d4d8d38f53bdb8d82da9576be49f07eedf5c"},
+    {file = "pyarrow-5.0.0-cp38-cp38-manylinux2014_x86_64.whl", hash = "sha256:ff21711f6ff3b0bc90abc8ca8169e676faeb2401ddc1a0bc1c7dc181708a3406"},
+    {file = "pyarrow-5.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:ed135a99975380c27077f9d0e210aea8618ed9fadcec0e71f8a3190939557afe"},
+    {file = "pyarrow-5.0.0-cp39-cp39-macosx_10_13_universal2.whl", hash = "sha256:6e1f0e4374061116f40e541408a8a170c170d0a070b788717e18165ebfdd2a54"},
+    {file = "pyarrow-5.0.0-cp39-cp39-macosx_10_13_x86_64.whl", hash = "sha256:4341ac0f552dc04c450751e049976940c7f4f8f2dae03685cc465ebe0a61e231"},
+    {file = "pyarrow-5.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:c3fc856f107ca2fb3c9391d7ea33bbb33f3a1c2b4a0e2b41f7525c626214cc03"},
+    {file = "pyarrow-5.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:357605665fbefb573d40939b13a684c2490b6ed1ab4a5de8dd246db4ab02e5a4"},
+    {file = "pyarrow-5.0.0-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:f4db312e9ba80e730cefcae0a05b63ea5befc7634c28df56682b628ad8e1c25c"},
+    {file = "pyarrow-5.0.0-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:1d9485741e497ccc516cb0a0c8f56e22be55aea815be185c3f9a681323b0e614"},
+    {file = "pyarrow-5.0.0-cp39-cp39-manylinux2014_x86_64.whl", hash = "sha256:b3115df938b8d7a7372911a3cb3904196194bcea8bb48911b4b3eafee3ab8d90"},
+    {file = "pyarrow-5.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:4d8adda1892ef4553c4804af7f67cce484f4d6371564e2d8374b8e2bc85293e2"},
+    {file = "pyarrow-5.0.0.tar.gz", hash = "sha256:24e64ea33eed07441cc0e80c949e3a1b48211a1add8953268391d250f4d39922"},
 ]
 pycodestyle = [
     {file = "pycodestyle-2.7.0-py2.py3-none-any.whl", hash = "sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068"},
@@ -439,20 +926,44 @@ pyflakes = [
     {file = "pyflakes-2.3.1-py2.py3-none-any.whl", hash = "sha256:7893783d01b8a89811dd72d7dfd4d84ff098e5eed95cfa8905b22bbffe52efc3"},
     {file = "pyflakes-2.3.1.tar.gz", hash = "sha256:f5bc8ecabc05bb9d291eb5203d6810b49040f6ff446a756326104746cc00c1db"},
 ]
+pymysql = [
+    {file = "PyMySQL-1.0.2-py3-none-any.whl", hash = "sha256:41fc3a0c5013d5f039639442321185532e3e2c8924687abe6537de157d403641"},
+    {file = "PyMySQL-1.0.2.tar.gz", hash = "sha256:816927a350f38d56072aeca5dfb10221fe1dc653745853d30a216637f5d7ad36"},
+]
 pyparsing = [
     {file = "pyparsing-2.4.7-py2.py3-none-any.whl", hash = "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"},
     {file = "pyparsing-2.4.7.tar.gz", hash = "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1"},
 ]
 pyrsistent = [
-    {file = "pyrsistent-0.17.3.tar.gz", hash = "sha256:2e636185d9eb976a18a8a8e96efce62f2905fea90041958d8cc2a189756ebf3e"},
+    {file = "pyrsistent-0.18.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:f4c8cabb46ff8e5d61f56a037974228e978f26bfefce4f61a4b1ac0ba7a2ab72"},
+    {file = "pyrsistent-0.18.0-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:da6e5e818d18459fa46fac0a4a4e543507fe1110e808101277c5a2b5bab0cd2d"},
+    {file = "pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:5e4395bbf841693eaebaa5bb5c8f5cdbb1d139e07c975c682ec4e4f8126e03d2"},
+    {file = "pyrsistent-0.18.0-cp36-cp36m-win32.whl", hash = "sha256:527be2bfa8dc80f6f8ddd65242ba476a6c4fb4e3aedbf281dfbac1b1ed4165b1"},
+    {file = "pyrsistent-0.18.0-cp36-cp36m-win_amd64.whl", hash = "sha256:2aaf19dc8ce517a8653746d98e962ef480ff34b6bc563fc067be6401ffb457c7"},
+    {file = "pyrsistent-0.18.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:58a70d93fb79dc585b21f9d72487b929a6fe58da0754fa4cb9f279bb92369396"},
+    {file = "pyrsistent-0.18.0-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:4916c10896721e472ee12c95cdc2891ce5890898d2f9907b1b4ae0f53588b710"},
+    {file = "pyrsistent-0.18.0-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:73ff61b1411e3fb0ba144b8f08d6749749775fe89688093e1efef9839d2dcc35"},
+    {file = "pyrsistent-0.18.0-cp37-cp37m-win32.whl", hash = "sha256:b29b869cf58412ca5738d23691e96d8aff535e17390128a1a52717c9a109da4f"},
+    {file = "pyrsistent-0.18.0-cp37-cp37m-win_amd64.whl", hash = "sha256:097b96f129dd36a8c9e33594e7ebb151b1515eb52cceb08474c10a5479e799f2"},
+    {file = "pyrsistent-0.18.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:772e94c2c6864f2cd2ffbe58bb3bdefbe2a32afa0acb1a77e472aac831f83427"},
+    {file = "pyrsistent-0.18.0-cp38-cp38-manylinux1_i686.whl", hash = "sha256:c1a9ff320fa699337e05edcaae79ef8c2880b52720bc031b219e5b5008ebbdef"},
+    {file = "pyrsistent-0.18.0-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:cd3caef37a415fd0dae6148a1b6957a8c5f275a62cca02e18474608cb263640c"},
+    {file = "pyrsistent-0.18.0-cp38-cp38-win32.whl", hash = "sha256:e79d94ca58fcafef6395f6352383fa1a76922268fa02caa2272fff501c2fdc78"},
+    {file = "pyrsistent-0.18.0-cp38-cp38-win_amd64.whl", hash = "sha256:a0c772d791c38bbc77be659af29bb14c38ced151433592e326361610250c605b"},
+    {file = "pyrsistent-0.18.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:d5ec194c9c573aafaceebf05fc400656722793dac57f254cd4741f3c27ae57b4"},
+    {file = "pyrsistent-0.18.0-cp39-cp39-manylinux1_i686.whl", hash = "sha256:6b5eed00e597b5b5773b4ca30bd48a5774ef1e96f2a45d105db5b4ebb4bca680"},
+    {file = "pyrsistent-0.18.0-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:48578680353f41dca1ca3dc48629fb77dfc745128b56fc01096b2530c13fd426"},
+    {file = "pyrsistent-0.18.0-cp39-cp39-win32.whl", hash = "sha256:f3ef98d7b76da5eb19c37fda834d50262ff9167c65658d1d8f974d2e4d90676b"},
+    {file = "pyrsistent-0.18.0-cp39-cp39-win_amd64.whl", hash = "sha256:404e1f1d254d314d55adb8d87f4f465c8693d6f902f67eb6ef5b4526dc58e6ea"},
+    {file = "pyrsistent-0.18.0.tar.gz", hash = "sha256:773c781216f8c2900b42a7b638d5b517bb134ae1acbebe4d1e8f1f41ea60eb4b"},
 ]
 pytest = [
-    {file = "pytest-6.2.4-py3-none-any.whl", hash = "sha256:91ef2131a9bd6be8f76f1f08eac5c5317221d6ad1e143ae03894b862e8976890"},
-    {file = "pytest-6.2.4.tar.gz", hash = "sha256:50bcad0a0b9c5a72c8e4e7c9855a3ad496ca6a881a3641b4260605450772c54b"},
+    {file = "pytest-6.2.5-py3-none-any.whl", hash = "sha256:7310f8d27bc79ced999e760ca304d69f6ba6c6649c0b60fb0e04a4a77cacc134"},
+    {file = "pytest-6.2.5.tar.gz", hash = "sha256:131b36680866a76e6781d13f101efb86cf674ebb9762eb70d3082b6f29889e89"},
 ]
 python-dateutil = [
-    {file = "python-dateutil-2.8.1.tar.gz", hash = "sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c"},
-    {file = "python_dateutil-2.8.1-py2.py3-none-any.whl", hash = "sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a"},
+    {file = "python-dateutil-2.8.2.tar.gz", hash = "sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86"},
+    {file = "python_dateutil-2.8.2-py2.py3-none-any.whl", hash = "sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"},
 ]
 pytz = [
     {file = "pytz-2021.1-py2.py3-none-any.whl", hash = "sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798"},
@@ -465,44 +976,141 @@ pyyaml = [
     {file = "PyYAML-5.4.1-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:bb4191dfc9306777bc594117aee052446b3fa88737cd13b7188d0e7aa8162185"},
     {file = "PyYAML-5.4.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:6c78645d400265a062508ae399b60b8c167bf003db364ecb26dcab2bda048253"},
     {file = "PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc"},
-    {file = "PyYAML-5.4.1-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:72a01f726a9c7851ca9bfad6fd09ca4e090a023c00945ea05ba1638c09dc3347"},
-    {file = "PyYAML-5.4.1-cp36-cp36m-manylinux2014_s390x.whl", hash = "sha256:895f61ef02e8fed38159bb70f7e100e00f471eae2bc838cd0f4ebb21e28f8541"},
     {file = "PyYAML-5.4.1-cp36-cp36m-win32.whl", hash = "sha256:3bd0e463264cf257d1ffd2e40223b197271046d09dadf73a0fe82b9c1fc385a5"},
     {file = "PyYAML-5.4.1-cp36-cp36m-win_amd64.whl", hash = "sha256:e4fac90784481d221a8e4b1162afa7c47ed953be40d31ab4629ae917510051df"},
     {file = "PyYAML-5.4.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:5accb17103e43963b80e6f837831f38d314a0495500067cb25afab2e8d7a4018"},
     {file = "PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:e1d4970ea66be07ae37a3c2e48b5ec63f7ba6804bdddfdbd3cfd954d25a82e63"},
-    {file = "PyYAML-5.4.1-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:cb333c16912324fd5f769fff6bc5de372e9e7a202247b48870bc251ed40239aa"},
-    {file = "PyYAML-5.4.1-cp37-cp37m-manylinux2014_s390x.whl", hash = "sha256:fe69978f3f768926cfa37b867e3843918e012cf83f680806599ddce33c2c68b0"},
     {file = "PyYAML-5.4.1-cp37-cp37m-win32.whl", hash = "sha256:dd5de0646207f053eb0d6c74ae45ba98c3395a571a2891858e87df7c9b9bd51b"},
     {file = "PyYAML-5.4.1-cp37-cp37m-win_amd64.whl", hash = "sha256:08682f6b72c722394747bddaf0aa62277e02557c0fd1c42cb853016a38f8dedf"},
     {file = "PyYAML-5.4.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:d2d9808ea7b4af864f35ea216be506ecec180628aced0704e34aca0b040ffe46"},
     {file = "PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:8c1be557ee92a20f184922c7b6424e8ab6691788e6d86137c5d93c1a6ec1b8fb"},
-    {file = "PyYAML-5.4.1-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:fd7f6999a8070df521b6384004ef42833b9bd62cfee11a09bda1079b4b704247"},
-    {file = "PyYAML-5.4.1-cp38-cp38-manylinux2014_s390x.whl", hash = "sha256:bfb51918d4ff3d77c1c856a9699f8492c612cde32fd3bcd344af9be34999bfdc"},
     {file = "PyYAML-5.4.1-cp38-cp38-win32.whl", hash = "sha256:fa5ae20527d8e831e8230cbffd9f8fe952815b2b7dae6ffec25318803a7528fc"},
     {file = "PyYAML-5.4.1-cp38-cp38-win_amd64.whl", hash = "sha256:0f5f5786c0e09baddcd8b4b45f20a7b5d61a7e7e99846e3c799b05c7c53fa696"},
     {file = "PyYAML-5.4.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:294db365efa064d00b8d1ef65d8ea2c3426ac366c0c4368d930bf1c5fb497f77"},
     {file = "PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:74c1485f7707cf707a7aef42ef6322b8f97921bd89be2ab6317fd782c2d53183"},
-    {file = "PyYAML-5.4.1-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:d483ad4e639292c90170eb6f7783ad19490e7a8defb3e46f97dfe4bacae89122"},
-    {file = "PyYAML-5.4.1-cp39-cp39-manylinux2014_s390x.whl", hash = "sha256:fdc842473cd33f45ff6bce46aea678a54e3d21f1b61a7750ce3c498eedfe25d6"},
     {file = "PyYAML-5.4.1-cp39-cp39-win32.whl", hash = "sha256:49d4cdd9065b9b6e206d0595fee27a96b5dd22618e7520c33204a4a3239d5b10"},
     {file = "PyYAML-5.4.1-cp39-cp39-win_amd64.whl", hash = "sha256:c20cfa2d49991c8b4147af39859b167664f2ad4561704ee74c1de03318e898db"},
     {file = "PyYAML-5.4.1.tar.gz", hash = "sha256:607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e"},
 ]
+redshift-connector = [
+    {file = "redshift_connector-2.0.887-py3-none-any.whl", hash = "sha256:88a3d18f65228bf1a369440dd10331788d7c6842077350cac34fbf0be1c11301"},
+]
+regex = [
+    {file = "regex-2021.8.28-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9d05ad5367c90814099000442b2125535e9d77581855b9bee8780f1b41f2b1a2"},
+    {file = "regex-2021.8.28-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f3bf1bc02bc421047bfec3343729c4bbbea42605bcfd6d6bfe2c07ade8b12d2a"},
+    {file = "regex-2021.8.28-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5f6a808044faae658f546dd5f525e921de9fa409de7a5570865467f03a626fc0"},
+    {file = "regex-2021.8.28-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:a617593aeacc7a691cc4af4a4410031654f2909053bd8c8e7db837f179a630eb"},
+    {file = "regex-2021.8.28-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:79aef6b5cd41feff359acaf98e040844613ff5298d0d19c455b3d9ae0bc8c35a"},
+    {file = "regex-2021.8.28-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:0fc1f8f06977c2d4f5e3d3f0d4a08089be783973fc6b6e278bde01f0544ff308"},
+    {file = "regex-2021.8.28-cp310-cp310-win32.whl", hash = "sha256:6eebf512aa90751d5ef6a7c2ac9d60113f32e86e5687326a50d7686e309f66ed"},
+    {file = "regex-2021.8.28-cp310-cp310-win_amd64.whl", hash = "sha256:ac88856a8cbccfc14f1b2d0b829af354cc1743cb375e7f04251ae73b2af6adf8"},
+    {file = "regex-2021.8.28-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:c206587c83e795d417ed3adc8453a791f6d36b67c81416676cad053b4104152c"},
+    {file = "regex-2021.8.28-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e8690ed94481f219a7a967c118abaf71ccc440f69acd583cab721b90eeedb77c"},
+    {file = "regex-2021.8.28-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:328a1fad67445550b982caa2a2a850da5989fd6595e858f02d04636e7f8b0b13"},
+    {file = "regex-2021.8.28-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:c7cb4c512d2d3b0870e00fbbac2f291d4b4bf2634d59a31176a87afe2777c6f0"},
+    {file = "regex-2021.8.28-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:66256b6391c057305e5ae9209941ef63c33a476b73772ca967d4a2df70520ec1"},
+    {file = "regex-2021.8.28-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:8e44769068d33e0ea6ccdf4b84d80c5afffe5207aa4d1881a629cf0ef3ec398f"},
+    {file = "regex-2021.8.28-cp36-cp36m-win32.whl", hash = "sha256:08d74bfaa4c7731b8dac0a992c63673a2782758f7cfad34cf9c1b9184f911354"},
+    {file = "regex-2021.8.28-cp36-cp36m-win_amd64.whl", hash = "sha256:abb48494d88e8a82601af905143e0de838c776c1241d92021e9256d5515b3645"},
+    {file = "regex-2021.8.28-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:b4c220a1fe0d2c622493b0a1fd48f8f991998fb447d3cd368033a4b86cf1127a"},
+    {file = "regex-2021.8.28-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d4a332404baa6665b54e5d283b4262f41f2103c255897084ec8f5487ce7b9e8e"},
+    {file = "regex-2021.8.28-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c61dcc1cf9fd165127a2853e2c31eb4fb961a4f26b394ac9fe5669c7a6592892"},
+    {file = "regex-2021.8.28-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:ee329d0387b5b41a5dddbb6243a21cb7896587a651bebb957e2d2bb8b63c0791"},
+    {file = "regex-2021.8.28-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f60667673ff9c249709160529ab39667d1ae9fd38634e006bec95611f632e759"},
+    {file = "regex-2021.8.28-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:b844fb09bd9936ed158ff9df0ab601e2045b316b17aa8b931857365ea8586906"},
+    {file = "regex-2021.8.28-cp37-cp37m-win32.whl", hash = "sha256:4cde065ab33bcaab774d84096fae266d9301d1a2f5519d7bd58fc55274afbf7a"},
+    {file = "regex-2021.8.28-cp37-cp37m-win_amd64.whl", hash = "sha256:1413b5022ed6ac0d504ba425ef02549a57d0f4276de58e3ab7e82437892704fc"},
+    {file = "regex-2021.8.28-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:ed4b50355b066796dacdd1cf538f2ce57275d001838f9b132fab80b75e8c84dd"},
+    {file = "regex-2021.8.28-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:28fc475f560d8f67cc8767b94db4c9440210f6958495aeae70fac8faec631797"},
+    {file = "regex-2021.8.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bdc178caebd0f338d57ae445ef8e9b737ddf8fbc3ea187603f65aec5b041248f"},
+    {file = "regex-2021.8.28-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:999ad08220467b6ad4bd3dd34e65329dd5d0df9b31e47106105e407954965256"},
+    {file = "regex-2021.8.28-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:808ee5834e06f57978da3e003ad9d6292de69d2bf6263662a1a8ae30788e080b"},
+    {file = "regex-2021.8.28-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:d5111d4c843d80202e62b4fdbb4920db1dcee4f9366d6b03294f45ed7b18b42e"},
+    {file = "regex-2021.8.28-cp38-cp38-win32.whl", hash = "sha256:473858730ef6d6ff7f7d5f19452184cd0caa062a20047f6d6f3e135a4648865d"},
+    {file = "regex-2021.8.28-cp38-cp38-win_amd64.whl", hash = "sha256:31a99a4796bf5aefc8351e98507b09e1b09115574f7c9dbb9cf2111f7220d2e2"},
+    {file = "regex-2021.8.28-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:04f6b9749e335bb0d2f68c707f23bb1773c3fb6ecd10edf0f04df12a8920d468"},
+    {file = "regex-2021.8.28-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9b006628fe43aa69259ec04ca258d88ed19b64791693df59c422b607b6ece8bb"},
+    {file = "regex-2021.8.28-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:121f4b3185feaade3f85f70294aef3f777199e9b5c0c0245c774ae884b110a2d"},
+    {file = "regex-2021.8.28-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:a577a21de2ef8059b58f79ff76a4da81c45a75fe0bfb09bc8b7bb4293fa18983"},
+    {file = "regex-2021.8.28-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1743345e30917e8c574f273f51679c294effba6ad372db1967852f12c76759d8"},
+    {file = "regex-2021.8.28-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:e1e8406b895aba6caa63d9fd1b6b1700d7e4825f78ccb1e5260551d168db38ed"},
+    {file = "regex-2021.8.28-cp39-cp39-win32.whl", hash = "sha256:ed283ab3a01d8b53de3a05bfdf4473ae24e43caee7dcb5584e86f3f3e5ab4374"},
+    {file = "regex-2021.8.28-cp39-cp39-win_amd64.whl", hash = "sha256:610b690b406653c84b7cb6091facb3033500ee81089867ee7d59e675f9ca2b73"},
+    {file = "regex-2021.8.28.tar.gz", hash = "sha256:f585cbbeecb35f35609edccb95efd95a3e35824cd7752b586503f7e6087303f1"},
+]
+requests = [
+    {file = "requests-2.25.1-py2.py3-none-any.whl", hash = "sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e"},
+    {file = "requests-2.25.1.tar.gz", hash = "sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804"},
+]
+s3transfer = [
+    {file = "s3transfer-0.5.0-py3-none-any.whl", hash = "sha256:9c1dc369814391a6bda20ebbf4b70a0f34630592c9aa520856bf384916af2803"},
+    {file = "s3transfer-0.5.0.tar.gz", hash = "sha256:50ed823e1dc5868ad40c8dc92072f757aa0e653a192845c94a3b676f4a62da4c"},
+]
+scramp = [
+    {file = "scramp-1.4.1-py3-none-any.whl", hash = "sha256:93c9cc2ffe54a451e02981c07a5a23cbd830701102789939cfb4ff91efd6ca8c"},
+    {file = "scramp-1.4.1.tar.gz", hash = "sha256:f964801077be9be2a1416ffe255d2d78834b3d9d5c8ce5d28f76a856f209f70e"},
+]
 six = [
     {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"},
     {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"},
 ]
+soupsieve = [
+    {file = "soupsieve-2.2.1-py3-none-any.whl", hash = "sha256:c2c1c2d44f158cdbddab7824a9af8c4f83c76b1e23e049479aa432feb6c4c23b"},
+    {file = "soupsieve-2.2.1.tar.gz", hash = "sha256:052774848f448cf19c7e959adf5566904d525f33a3f8b6ba6f6f8f26ec7de0cc"},
+]
 toml = [
     {file = "toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b"},
     {file = "toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"},
 ]
+tomli = [
+    {file = "tomli-1.2.1-py3-none-any.whl", hash = "sha256:8dd0e9524d6f386271a36b41dbf6c57d8e32fd96fd22b6584679dc569d20899f"},
+    {file = "tomli-1.2.1.tar.gz", hash = "sha256:a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442"},
+]
+typed-ast = [
+    {file = "typed_ast-1.4.3-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:2068531575a125b87a41802130fa7e29f26c09a2833fea68d9a40cf33902eba6"},
+    {file = "typed_ast-1.4.3-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:c907f561b1e83e93fad565bac5ba9c22d96a54e7ea0267c708bffe863cbe4075"},
+    {file = "typed_ast-1.4.3-cp35-cp35m-manylinux2014_aarch64.whl", hash = "sha256:1b3ead4a96c9101bef08f9f7d1217c096f31667617b58de957f690c92378b528"},
+    {file = "typed_ast-1.4.3-cp35-cp35m-win32.whl", hash = "sha256:dde816ca9dac1d9c01dd504ea5967821606f02e510438120091b84e852367428"},
+    {file = "typed_ast-1.4.3-cp35-cp35m-win_amd64.whl", hash = "sha256:777a26c84bea6cd934422ac2e3b78863a37017618b6e5c08f92ef69853e765d3"},
+    {file = "typed_ast-1.4.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:f8afcf15cc511ada719a88e013cec87c11aff7b91f019295eb4530f96fe5ef2f"},
+    {file = "typed_ast-1.4.3-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:52b1eb8c83f178ab787f3a4283f68258525f8d70f778a2f6dd54d3b5e5fb4341"},
+    {file = "typed_ast-1.4.3-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:01ae5f73431d21eead5015997ab41afa53aa1fbe252f9da060be5dad2c730ace"},
+    {file = "typed_ast-1.4.3-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:c190f0899e9f9f8b6b7863debfb739abcb21a5c054f911ca3596d12b8a4c4c7f"},
+    {file = "typed_ast-1.4.3-cp36-cp36m-win32.whl", hash = "sha256:398e44cd480f4d2b7ee8d98385ca104e35c81525dd98c519acff1b79bdaac363"},
+    {file = "typed_ast-1.4.3-cp36-cp36m-win_amd64.whl", hash = "sha256:bff6ad71c81b3bba8fa35f0f1921fb24ff4476235a6e94a26ada2e54370e6da7"},
+    {file = "typed_ast-1.4.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:0fb71b8c643187d7492c1f8352f2c15b4c4af3f6338f21681d3681b3dc31a266"},
+    {file = "typed_ast-1.4.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:760ad187b1041a154f0e4d0f6aae3e40fdb51d6de16e5c99aedadd9246450e9e"},
+    {file = "typed_ast-1.4.3-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:5feca99c17af94057417d744607b82dd0a664fd5e4ca98061480fd8b14b18d04"},
+    {file = "typed_ast-1.4.3-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:95431a26309a21874005845c21118c83991c63ea800dd44843e42a916aec5899"},
+    {file = "typed_ast-1.4.3-cp37-cp37m-win32.whl", hash = "sha256:aee0c1256be6c07bd3e1263ff920c325b59849dc95392a05f258bb9b259cf39c"},
+    {file = "typed_ast-1.4.3-cp37-cp37m-win_amd64.whl", hash = "sha256:9ad2c92ec681e02baf81fdfa056fe0d818645efa9af1f1cd5fd6f1bd2bdfd805"},
+    {file = "typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:b36b4f3920103a25e1d5d024d155c504080959582b928e91cb608a65c3a49e1a"},
+    {file = "typed_ast-1.4.3-cp38-cp38-manylinux1_i686.whl", hash = "sha256:067a74454df670dcaa4e59349a2e5c81e567d8d65458d480a5b3dfecec08c5ff"},
+    {file = "typed_ast-1.4.3-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:7538e495704e2ccda9b234b82423a4038f324f3a10c43bc088a1636180f11a41"},
+    {file = "typed_ast-1.4.3-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:af3d4a73793725138d6b334d9d247ce7e5f084d96284ed23f22ee626a7b88e39"},
+    {file = "typed_ast-1.4.3-cp38-cp38-win32.whl", hash = "sha256:f2362f3cb0f3172c42938946dbc5b7843c2a28aec307c49100c8b38764eb6927"},
+    {file = "typed_ast-1.4.3-cp38-cp38-win_amd64.whl", hash = "sha256:dd4a21253f42b8d2b48410cb31fe501d32f8b9fbeb1f55063ad102fe9c425e40"},
+    {file = "typed_ast-1.4.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:f328adcfebed9f11301eaedfa48e15bdece9b519fb27e6a8c01aa52a17ec31b3"},
+    {file = "typed_ast-1.4.3-cp39-cp39-manylinux1_i686.whl", hash = "sha256:2c726c276d09fc5c414693a2de063f521052d9ea7c240ce553316f70656c84d4"},
+    {file = "typed_ast-1.4.3-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:cae53c389825d3b46fb37538441f75d6aecc4174f615d048321b716df2757fb0"},
+    {file = "typed_ast-1.4.3-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:b9574c6f03f685070d859e75c7f9eeca02d6933273b5e69572e5ff9d5e3931c3"},
+    {file = "typed_ast-1.4.3-cp39-cp39-win32.whl", hash = "sha256:209596a4ec71d990d71d5e0d312ac935d86930e6eecff6ccc7007fe54d703808"},
+    {file = "typed_ast-1.4.3-cp39-cp39-win_amd64.whl", hash = "sha256:9c6d1a54552b5330bc657b7ef0eae25d00ba7ffe85d9ea8ae6540d2197a3788c"},
+    {file = "typed_ast-1.4.3.tar.gz", hash = "sha256:fb1bbeac803adea29cedd70781399c99138358c26d05fcbd23c13016b7f5ec65"},
+]
 typing-extensions = [
-    {file = "typing_extensions-3.10.0.0-py2-none-any.whl", hash = "sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497"},
-    {file = "typing_extensions-3.10.0.0-py3-none-any.whl", hash = "sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84"},
-    {file = "typing_extensions-3.10.0.0.tar.gz", hash = "sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342"},
+    {file = "typing_extensions-3.10.0.2-py2-none-any.whl", hash = "sha256:d8226d10bc02a29bcc81df19a26e56a9647f8b0a6d4a83924139f4a8b01f17b7"},
+    {file = "typing_extensions-3.10.0.2-py3-none-any.whl", hash = "sha256:f1d25edafde516b146ecd0613dabcc61409817af4766fbbcfb8d1ad4ec441a34"},
+    {file = "typing_extensions-3.10.0.2.tar.gz", hash = "sha256:49f75d16ff11f1cd258e1b988ccff82a3ca5570217d7ad8c5f48205dd99a677e"},
+]
+urllib3 = [
+    {file = "urllib3-1.26.7-py2.py3-none-any.whl", hash = "sha256:c4fdf4019605b6e5423637e01bc9fe4daef873709a7973e195ceba0a62bbc844"},
+    {file = "urllib3-1.26.7.tar.gz", hash = "sha256:4987c65554f7a2dbf30c18fd48778ef124af6fab771a377103da0585e2336ece"},
+]
+yamllint = [
+    {file = "yamllint-1.26.3.tar.gz", hash = "sha256:3934dcde484374596d6b52d8db412929a169f6d9e52e20f9ade5bf3523d9b96e"},
 ]
 zipp = [
-    {file = "zipp-3.4.1-py3-none-any.whl", hash = "sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"},
-    {file = "zipp-3.4.1.tar.gz", hash = "sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76"},
+    {file = "zipp-3.5.0-py3-none-any.whl", hash = "sha256:957cfda87797e389580cb8b9e3870841ca991e2125350677b2ca83a0e99390a3"},
+    {file = "zipp-3.5.0.tar.gz", hash = "sha256:f5812b1e007e48cff63449a5e9f4e7ebea716b4111f9c4f9a645f91d579bf0c4"},
 ]
diff --git a/pyproject.toml b/pyproject.toml
index 59755af..c34408a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "arrow_pd_parser"
-version = "0.4.4"
+version = "1.0.0"
 description = "MoJ arrow-pd-parser"
 authors = ["Karik Isichei <karik.isichei@digital.justice.gov.uk>",
            "Alec Johnson <alec.johnson@digital.justice.gov.uk>",
@@ -18,14 +18,17 @@ repository = "https://github.com/moj-analytical-services/mojap-arrow-pd-parser"
 homepage = "https://github.com/moj-analytical-services/mojap-arrow-pd-parser"
 
 [tool.poetry.dependencies]
-python = ">=3.7.1,<=3.9"
-pandas = "^1.2"
-pyarrow = ">=2.0.0,<4.0.0"
-mojap-metadata = {version = "^1.2.1", extras = ["arrow"]}
+python = ">=3.7.1,<3.10" # wrangler dependency
+pandas = ">=1.2"
+pyarrow = ">=5.0.0"
+mojap-metadata = {version = "^1.2.2", extras = ["arrow"]}
+awswrangler = "^2.3.0"
 
 [tool.poetry.dev-dependencies]
-pytest = "^6.1"
+pytest = ">=6.1"
 flake8 = "^3.8.4"
+black = ">=21.8b0"
+yamllint = ">=1.26.3"
 
 [build-system]
 requires = ["poetry>=0.12"]


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 5ae65a3..d9ba70d 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -1,32 +1,33 @@
-  
-name: Test
-on:
-  - pull_request
+name: Run tests with package dependencies
+
+on: pull_request
 
 jobs:
-  preview:
-    name: Test
+  python-version-run-tests:
     runs-on: ubuntu-latest
     strategy:
       max-parallel: 4
       matrix:
         python-version: [3.7, 3.8, 3.9]
+        poetry-install: [from-lock-file, fresh-install]
     steps:
-      - uses: actions/checkout@v2
-        with:
-          fetch-depth: 1
-
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ matrix.python-version }}
-
-      - name: Install Python dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install poetry
-          poetry install
-
-      - name: Run tests with pytest
-        run: |
-          if [ -d tests ]; then poetry run pytest tests/ -vv; fi
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies ${{ matrix.poetry-install }}
+      run: |
+        if [ "${{ matrix.poetry-install }}" == "from-lock-file" ]; then
+          echo "Installing dependencies from lock file"
+        else
+          echo "Removing lock file and installing latest possible dependencies"
+          rm poetry.lock
+        fi
+        python -m pip install --upgrade pip
+        pip install poetry
+        poetry config virtualenvs.create false \
+          && poetry install --no-interaction --no-ansi
+    - name: Run Tests
+      run: |
+        pytest tests/ -vv
diff --git a/tests/test_arrow_parser.py b/tests/test_arrow_parser.py
index 0b1a89a..9290e30 100644
--- a/tests/test_arrow_parser.py
+++ b/tests/test_arrow_parser.py
@@ -2,7 +2,7 @@
 import pyarrow as pa
 import pandas as pd
 from pandas.testing import assert_frame_equal
-from arrow_pd_parser.parse import (
+from arrow_pd_parser._arrow_parsers import (
     pa_read_csv_to_pandas,
     pa_read_json_to_pandas,
     cast_arrow_table_to_schema,
diff --git a/tests/test_bool_conformance.py b/tests/test_bool_conformance.py
index f020631..a921791 100644
--- a/tests/test_bool_conformance.py
+++ b/tests/test_bool_conformance.py
@@ -1,7 +1,7 @@
 import pytest
 import pyarrow as pa
 
-from arrow_pd_parser.parse import pa_read_csv_to_pandas, pa_read_json_to_pandas
+from arrow_pd_parser._arrow_parsers import pa_read_csv_to_pandas, pa_read_json_to_pandas
 
 
 args = (
diff --git a/tests/test_date_timestamp_conformance.py b/tests/test_date_timestamp_conformance.py
index 7870c4b..cd99216 100644
--- a/tests/test_date_timestamp_conformance.py
+++ b/tests/test_date_timestamp_conformance.py
@@ -3,7 +3,7 @@
 import pandas as pd
 import pyarrow as pa
 import numpy as np
-from arrow_pd_parser.parse import (
+from arrow_pd_parser._arrow_parsers import (
     pa_read_csv_to_pandas,
     pa_read_json_to_pandas,
 )
diff --git a/tests/test_decimal_conformance.py b/tests/test_decimal_conformance.py
index 676fc9f..9900ef6 100644
--- a/tests/test_decimal_conformance.py
+++ b/tests/test_decimal_conformance.py
@@ -1,7 +1,7 @@
 import pytest
 import pyarrow as pa
 from pandas.testing import assert_frame_equal
-from arrow_pd_parser.parse import pa_read_csv_to_pandas, pa_read_json_to_pandas
+from arrow_pd_parser._arrow_parsers import pa_read_csv_to_pandas, pa_read_json_to_pandas
 
 
 @pytest.mark.parametrize(
diff --git a/tests/test_export.py b/tests/test_export.py
index 43f5e9a..6baf5cf 100644
--- a/tests/test_export.py
+++ b/tests/test_export.py
@@ -3,12 +3,12 @@
 import pytest
 import pyarrow as pa
 
-from arrow_pd_parser.parse import (
+from arrow_pd_parser._arrow_parsers import (
     pa_read_csv_to_pandas,
     pa_read_json_to_pandas,
     pa_read_parquet_to_pandas,
 )
-from arrow_pd_parser.export import pd_to_csv, pd_to_json, pd_to_parquet
+from arrow_pd_parser._export import pd_to_csv, pd_to_json, pd_to_parquet
 from pandas.testing import assert_frame_equal
 
 schemas = [
diff --git a/tests/test_int_conformance.py b/tests/test_int_conformance.py
index 238e408..ed45d89 100644
--- a/tests/test_int_conformance.py
+++ b/tests/test_int_conformance.py
@@ -1,7 +1,10 @@
 import pytest
 import pyarrow as pa
 
-from arrow_pd_parser.parse import pa_read_csv_to_pandas, pa_read_json_to_pandas
+from arrow_pd_parser._arrow_parsers import (
+    pa_read_csv_to_pandas,
+    pa_read_json_to_pandas,
+)
 
 
 # Parameters used across tests for both json and csv
diff --git a/tests/test_pandas_parser.py b/tests/test_pandas_parser.py
index 351993d..f570e14 100644
--- a/tests/test_pandas_parser.py
+++ b/tests/test_pandas_parser.py
@@ -6,24 +6,22 @@
 import pandas as pd
 import numpy as np
 from pandas.testing import assert_frame_equal, assert_series_equal
-from arrow_pd_parser.parse import (
-    pd_read_csv,
-    pd_read_json,
-    cast_pandas_table_to_schema,
-    cast_pandas_column_to_schema,
-    PandasCastError,
-)
 
-from arrow_pd_parser.parse.pandas_parser import (
+from arrow_pd_parser.caster import (
     _infer_bool_type,
     convert_to_bool_series,
     convert_str_to_timestamp_series,
+    cast_pandas_table_to_schema,
+    PandasCastError,
+    cast_pandas_column_to_schema,
 )
 
 from jsonschema.exceptions import ValidationError
 
 from mojap_metadata import Metadata
 
+from arrow_pd_parser import reader
+
 
 def test_file_reader_returns_df():
     csv_meta = {
@@ -33,12 +31,12 @@ def test_file_reader_returns_df():
         ]
     }
 
-    df = pd_read_csv("tests/data/example_data.csv", metadata=csv_meta)
+    df = reader.csv.read("tests/data/example_data.csv", metadata=csv_meta)
     assert isinstance(df, pd.DataFrame)
     assert [str(df[c].dtype) for c in df.columns] == ["string", "string"]
 
-    # df = pa_read_json("tests/data/example_data.jsonl")
-    # assert isinstance(df, pd.DataFrame)
+    df = reader.json.read("tests/data/example_data.jsonl")
+    assert isinstance(df, pd.DataFrame)
 
 
 def test_file_reader_works_with_both_meta_types():
@@ -48,15 +46,14 @@ def test_file_reader_works_with_both_meta_types():
             {"name": "a_column", "type_category": "string"},
         ]
     }
-    df_csv1 = pd_read_csv("tests/data/example_data.csv", csv_meta)
+
+    df_csv1 = reader.csv.read("tests/data/example_data.csv", csv_meta)
     csv_meta = Metadata.from_dict(csv_meta)
-    df_csv2 = pd_read_csv("tests/data/example_data.csv", csv_meta)
+    df_csv2 = reader.csv.read("tests/data/example_data.csv", csv_meta)
     assert_frame_equal(df_csv1, df_csv2)
-    with pytest.raises(KeyError):
-        pd_read_csv("tests/data/example_data.jsonl", metadata={})
     with pytest.raises(ValidationError):
         m = {"columns": [{"name": "broken", "type": "not-a-type"}]}
-        pd_read_csv("tests/data/example_data.jsonl", metadata=m)
+        reader.csv.read("tests/data/example_data.jsonl", metadata=m)
 
 
 @pytest.mark.parametrize("data_format", ["jsonl", "csv"])
@@ -103,12 +100,11 @@ def test_basic_end_to_end(data_format):
     assert actual_dtypes == expected_dtypes
 
     if data_format == "jsonl":
-        df2 = pd_read_json(test_data_path, meta)
-        with pytest.warns(UserWarning):
-            df3 = pd_read_json(test_data_path, meta, orient="records", lines=True)
+        df2 = reader.json.read(test_data_path, meta)
+        df3 = reader.json.read(test_data_path, meta, orient="records", lines=True)
     else:
-        df2 = pd_read_csv(test_data_path, meta)
-        df3 = pd_read_csv(test_data_path, meta, dtype=str, low_memory=False)
+        df2 = reader.csv.read(test_data_path, meta)
+        df3 = reader.csv.read(test_data_path, meta, dtype=str, low_memory=False)
 
     assert_frame_equal(dfn, df2)
     assert_frame_equal(df2, df3)
@@ -180,18 +176,15 @@ def test_timestamp_conversion(s, dt_fmt, is_date, pd_date_type):
         assert_series_equal(pd.to_datetime(s, format=dt_fmt), pd.to_datetime(s_))
 
 
-@pytest.mark.parametrize(
-    "col_type",
-    ["date64", "date32", "timestamp(s)"]
-)
+@pytest.mark.parametrize("col_type", ["date64", "date32", "timestamp(s)"])
 def test_timestamp_conversion_in_df(col_type):
     meta = {
         "name": "test",
         "columns": [
             {"name": "datelong", "datetime_format": "%d-%b-%Y"},
             {"name": "dateshort", "datetime_format": "%d-%b-%y"},
-            {"name": "date_uk", "datetime_format": "%d/%m/%Y"}
-        ]
+            {"name": "date_uk", "datetime_format": "%d/%m/%Y"},
+        ],
     }
     for c in meta["columns"]:
         c["type"] = col_type
@@ -206,7 +199,7 @@ def test_timestamp_conversion_in_df(col_type):
     if col_type.startswith("date"):
         expected_col_values = [v.date() for v in expected_col_values]
 
-    df = pd_read_csv(StringIO(data), metadata=meta)
+    df = reader.csv.read(StringIO(data), metadata=meta)
     for c in df.columns:
         assert expected_col_values == df[c].to_list()
 
@@ -217,7 +210,7 @@ def test_cast_error():
         "name": "bad_format",
         "type": "date64()",
         "type_category": "timestamp",
-        "datetime_format": "%d/%m/%Y %H:%M:%S"
+        "datetime_format": "%d/%m/%Y %H:%M:%S",
     }
     with pytest.raises(PandasCastError) as exec_info:
         cast_pandas_column_to_schema(col, mc)
@@ -230,18 +223,15 @@ def test_cast_error():
 
 
 @pytest.mark.parametrize(
-    "row,t,tc", [
+    "row,t,tc",
+    [
         ({"a": 0, "b": "string"}, "struct<a:int64, b:string>", "struct"),
         ([0, 1, 2], "list<int64>", "list"),
         ([0, 1, 2], "large_list<int64>", "list"),
-    ]
+    ],
 )
 def test_complex_cast_warning(row, t, tc):
     col = pd.Series({"complex_col": [row, row]})
-    mc = {
-        "name": "complex_col",
-        "type": t,
-        "type_category": tc
-    }
+    mc = {"name": "complex_col", "type": t, "type_category": tc}
     with pytest.warns(UserWarning):
         cast_pandas_column_to_schema(col, mc)
diff --git a/tests/test_project_version.py b/tests/test_project_version.py
new file mode 100644
index 0000000..4d71110
--- /dev/null
+++ b/tests/test_project_version.py
@@ -0,0 +1,8 @@
+import toml
+import arrow_pd_parser
+
+
+def test_pyproject_toml_matches_version():
+    with open("pyproject.toml") as f:
+        proj = toml.load(f)
+    assert arrow_pd_parser.__version__ == proj["tool"]["poetry"]["version"]
diff --git a/tests/test_reader_api.py b/tests/test_reader_api.py
new file mode 100644
index 0000000..b1e5a66
--- /dev/null
+++ b/tests/test_reader_api.py
@@ -0,0 +1,105 @@
+import pytest
+import tempfile
+
+import numpy as np
+import pandas as pd
+from pandas.testing import assert_frame_equal
+
+from arrow_pd_parser import reader, writer
+
+
+@pytest.mark.parametrize("data_format", ["jsonl", "csv"])
+def test_inferred_cols_pandas_types(data_format):
+    df = reader.read(f"tests/data/all_types.{data_format}")
+    test = df.dtypes.to_dict()
+    assert isinstance(test["i"], pd.core.arrays.integer.Int64Dtype)
+    assert isinstance(test["my_float"], type(np.dtype("float64")))
+    assert isinstance(test["my_bool"], pd.core.arrays.boolean.BooleanDtype)
+    if data_format == "jsonl":
+        pytest.skip("Pandas cannot infer bool with nulls from JSON datasets")
+    else:
+        assert isinstance(test["my_nullable_bool"], pd.core.arrays.boolean.BooleanDtype)
+
+    assert isinstance(test["my_string"], pd.core.arrays.string_.StringDtype)
+
+
+@pytest.mark.parametrize("data_format", ["jsonl", "csv"])
+@pytest.mark.parametrize("use_meta", [True, False])
+def test_read(data_format, use_meta):
+    test_data_path = f"tests/data/all_types.{data_format}"
+
+    if use_meta:
+        meta = {
+            "columns": [
+                {"name": "my_float", "type": "float64", "type_category": "float"},
+                {"name": "my_bool", "type": "bool_", "type_category": "boolean"},
+                {
+                    "name": "my_nullable_bool",
+                    "type": "bool_",
+                    "type_category": "boolean",
+                },
+                {"name": "my_date", "type": "date32", "type_category": "timestamp"},
+                {
+                    "name": "my_datetime",
+                    "type": "timestamp(s)",
+                    "type_category": "timestamp",
+                },
+                {"name": "my_int", "type": "int64", "type_category": "integer"},
+                {"name": "my_string", "type": "string", "type_category": "string"},
+            ]
+        }
+    else:
+        meta = None
+
+    df1 = reader.read(test_data_path, meta)
+
+    if data_format == "csv":
+        df2 = reader.csv.read(test_data_path, meta)
+    elif data_format == "jsonl":
+        df2 = reader.json.read(test_data_path, meta)
+    else:
+        raise ValueError(f"Test wasn't expecting: {data_format}")
+
+    assert_frame_equal(df1, df2)
+
+
+def test_round_trip():
+    meta = {
+        "columns": [
+            {"name": "my_float", "type": "float64", "type_category": "float"},
+            {"name": "my_bool", "type": "bool_", "type_category": "boolean"},
+            {"name": "my_nullable_bool", "type": "bool_", "type_category": "boolean"},
+            {"name": "my_date", "type": "date32", "type_category": "timestamp"},
+            {
+                "name": "my_datetime",
+                "type": "timestamp(s)",
+                "type_category": "timestamp",
+            },
+            {"name": "my_int", "type": "int64", "type_category": "integer"},
+            {"name": "my_string", "type": "string", "type_category": "string"},
+        ]
+    }
+
+    # Create parquet temp file
+    with tempfile.NamedTemporaryFile(suffix=".parquet") as f:
+        tmp_out_file = f.name
+    original = reader.csv.read("tests/data/all_types.csv", meta)
+    writer.parquet.write(original, tmp_out_file)
+
+    data_paths = {
+        "csv": "tests/data/all_types.csv",
+        "json": "tests/data/all_types.jsonl",
+        "parquet": tmp_out_file,
+    }
+
+    for type1 in ["csv", "json", "parquet"]:
+        for type2 in ["csv", "json", "parquet"]:
+            df1 = reader.read(
+                input_path=data_paths[type1],
+                metadata=meta,
+            )
+            df2 = reader.read(
+                input_path=data_paths[type2],
+                metadata=meta,
+            )
+            assert_frame_equal(df1, df2)
diff --git a/tests/test_round_trip.py b/tests/test_round_trip.py
new file mode 100644
index 0000000..4e609c0
--- /dev/null
+++ b/tests/test_round_trip.py
@@ -0,0 +1,46 @@
+import pytest
+import tempfile
+
+from pandas.testing import assert_frame_equal
+
+from arrow_pd_parser import reader, writer
+from arrow_pd_parser.utils import FileFormat
+
+
+all_formats = [FileFormat.CSV, FileFormat.JSON, FileFormat.PARQUET]
+
+
+@pytest.mark.parametrize("trip1_file_format", all_formats)
+@pytest.mark.parametrize("trip2_file_format", all_formats)
+def test_round_trip(trip1_file_format, trip2_file_format):
+    meta = {
+        "columns": [
+            {"name": "my_float", "type": "float64", "type_category": "float"},
+            {"name": "my_bool", "type": "bool_", "type_category": "boolean"},
+            {"name": "my_nullable_bool", "type": "bool_", "type_category": "boolean"},
+            {"name": "my_date", "type": "date32", "type_category": "timestamp"},
+            {
+                "name": "my_datetime",
+                "type": "timestamp(s)",
+                "type_category": "timestamp",
+            },
+            {"name": "my_int", "type": "int64", "type_category": "integer"},
+            {"name": "my_string", "type": "string", "type_category": "string"},
+        ]
+    }
+    original = reader.csv.read("tests/data/all_types.csv", meta)
+    orig_copy = original.copy()
+
+    # Trip 1
+    with tempfile.NamedTemporaryFile() as f:
+        tmp_out_file1 = f.name
+    writer.write(orig_copy, tmp_out_file1, file_format=trip1_file_format, metadata=meta)
+    df_mid = reader.read(tmp_out_file1, file_format=trip1_file_format, metadata=meta)
+
+    # Trip 2
+    with tempfile.NamedTemporaryFile() as f:
+        tmp_out_file2 = f.name
+    writer.write(df_mid, tmp_out_file2, file_format=trip2_file_format, metadata=meta)
+    final = reader.read(tmp_out_file2, file_format=trip2_file_format, metadata=meta)
+
+    assert_frame_equal(original, final)
diff --git a/tests/test_str_conformance.py b/tests/test_str_conformance.py
index 2f0e99a..3fa66e3 100644
--- a/tests/test_str_conformance.py
+++ b/tests/test_str_conformance.py
@@ -1,13 +1,14 @@
 import pytest
 import pyarrow as pa
 from pyarrow import csv
-from arrow_pd_parser.parse import pa_read_csv_to_pandas
+from arrow_pd_parser._arrow_parsers import pa_read_csv_to_pandas
 from pandas.testing import assert_series_equal
 from pandas import Series
 
 
 @pytest.mark.parametrize(
-    "in_type,pd_old_type,pd_new_type", [("string", "object", "string")],
+    "in_type,pd_old_type,pd_new_type",
+    [("string", "object", "string")],
 )
 def test_string(in_type, pd_old_type, pd_new_type):
 
@@ -24,7 +25,8 @@ def test_string(in_type, pd_old_type, pd_new_type):
 
 
 @pytest.mark.parametrize(
-    "in_type,pd_old_type,pd_new_type", [("string", "object", "string")],
+    "in_type,pd_old_type,pd_new_type",
+    [("string", "object", "string")],
 )
 def test_csv_options(in_type, pd_old_type, pd_new_type):
     schema = pa.schema([("string_col", pa.string())])
diff --git a/tests/test_utils.py b/tests/test_utils.py
new file mode 100644
index 0000000..a4e1278
--- /dev/null
+++ b/tests/test_utils.py
@@ -0,0 +1,109 @@
+import pytest
+from io import StringIO, BytesIO
+
+from mojap_metadata import Metadata
+
+from arrow_pd_parser.utils import (
+    FileFormat,
+    is_s3_filepath,
+    infer_format_from_filepath,
+    infer_file_format_from_meta,
+    infer_file_format,
+    FileFormatNotFound,
+)
+
+
+@pytest.mark.parametrize(
+    "input_str,expected",
+    [
+        ("json", FileFormat.JSON),
+        ("jsonl", FileFormat.JSON),
+        ("ndjson", FileFormat.JSON),
+        ("csv", FileFormat.CSV),
+        ("parquet", FileFormat.PARQUET),
+    ],
+)
+def test_file_format(input_str, expected):
+    actual = FileFormat.from_string(input_str)
+    assert actual == expected
+
+    actual = FileFormat.from_string(input_str.upper())
+    assert actual == expected
+
+
+def test_is_s3_filepath():
+    assert is_s3_filepath("s3://bucket/object.csv") is True
+    assert is_s3_filepath("local/file.csv") is False
+    assert is_s3_filepath(StringIO()) is False
+    assert is_s3_filepath(BytesIO()) is False
+
+
+@pytest.mark.parametrize(
+    "filepath,expected",
+    [
+        ("a/file.csv", FileFormat.CSV),
+        ("a/file.csv.gzip", FileFormat.CSV),
+        ("a/file.json", FileFormat.JSON),
+        ("a/file.ndjson", FileFormat.JSON),
+        ("a/file.jsonl", FileFormat.JSON),
+        ("a/file.parquet", FileFormat.PARQUET),
+        ("a/file.snappy.parquet", FileFormat.PARQUET),
+    ],
+)
+def test_infer_format_from_filepath(filepath, expected):
+    actual = infer_format_from_filepath(filepath)
+    assert actual == expected
+
+
+@pytest.mark.parametrize(
+    "file_format,expected",
+    [
+        ("csv", FileFormat.CSV),
+        ("csv.gzip", FileFormat.CSV),
+        ("json", FileFormat.JSON),
+        ("ndjson", FileFormat.JSON),
+        ("jsonl", FileFormat.JSON),
+        ("parquet", FileFormat.PARQUET),
+        ("snappy.parquet", FileFormat.PARQUET),
+    ],
+)
+def test_infer_file_format_from_meta(file_format, expected):
+    meta = {
+        "name": "test",
+        "columns": [{"name": "a", "type": "string"}],
+        "file_format": file_format,
+    }
+    actual = infer_file_format_from_meta(meta)
+    assert actual == expected
+
+    metadata = Metadata.from_dict(meta)
+    actual = infer_file_format_from_meta(metadata)
+    assert actual == expected
+
+
+def test_infer_file_format():
+    filepath = "test.csv"
+    meta = {
+        "name": "test",
+        "columns": [{"name": "a", "type": "string"}],
+        "file_format": "parquet",
+    }
+    actual = infer_file_format(filepath, meta)
+    assert actual == FileFormat.CSV
+
+    actual = infer_file_format(filepath, None)
+    assert actual == FileFormat.CSV
+
+    actual = infer_file_format("file.unknown", meta)
+    assert actual == FileFormat.PARQUET
+
+    with pytest.raises(FileFormatNotFound):
+        _ = infer_file_format("file.unknown", None)
+
+    meta["file_format"] = "bob"
+    with pytest.raises(FileFormatNotFound):
+        _ = infer_file_format("file.unknown", meta)
+
+    _ = meta.pop("file_format")
+    with pytest.raises(FileFormatNotFound):
+        _ = infer_file_format("file.unknown", meta)
diff --git a/tests/test_writer_api.py b/tests/test_writer_api.py
new file mode 100644
index 0000000..db09526
--- /dev/null
+++ b/tests/test_writer_api.py
@@ -0,0 +1,58 @@
+import pytest
+import tempfile
+
+from arrow_pd_parser import writer, reader
+
+
+@pytest.mark.parametrize("data_format", ["jsonl", "csv", "snappy.parquet"])
+@pytest.mark.parametrize("use_meta", [True, False])
+def test_write(data_format, use_meta):
+
+    if use_meta:
+        meta = {
+            "columns": [
+                {"name": "my_float", "type": "float64", "type_category": "float"},
+                {"name": "my_bool", "type": "bool_", "type_category": "boolean"},
+                {
+                    "name": "my_nullable_bool",
+                    "type": "bool_",
+                    "type_category": "boolean",
+                },
+                {"name": "my_date", "type": "date32", "type_category": "timestamp"},
+                {
+                    "name": "my_datetime",
+                    "type": "timestamp(s)",
+                    "type_category": "timestamp",
+                },
+                {"name": "my_int", "type": "int64", "type_category": "integer"},
+                {"name": "my_string", "type": "string", "type_category": "string"},
+            ]
+        }
+    else:
+        meta = None
+
+    in_data_path = "tests/data/all_types.csv"
+    df = reader.read(in_data_path, meta)
+
+    # Create temp files
+    with tempfile.NamedTemporaryFile(suffix=f".{data_format}") as f:
+        tmp_out1 = f.name
+    with tempfile.NamedTemporaryFile(suffix=f".{data_format}") as f:
+        tmp_out2 = f.name
+
+    writer.write(df, tmp_out1, meta)
+    if data_format == "csv":
+        writer.csv.write(df, tmp_out2, meta)
+    elif data_format == "jsonl":
+        writer.json.write(df, tmp_out2, meta)
+    elif data_format == "snappy.parquet":
+        writer.parquet.write(df, tmp_out2, meta)
+    else:
+        raise ValueError(f"Test wasn't expecting: {data_format}")
+
+    with open(tmp_out1, "rb") as f:
+        b1 = f.read()
+    with open(tmp_out2, "rb") as f:
+        b2 = f.read()
+
+    assert b1 == b2