Package: arrow_pd_parser
mojap-arrow-pd-parserUsing pyArrow/Pandas to read CSV, JSONL and Parquet ensuring that you get the best representation of the column types in the resulting Pandas dataframe. Also ensures data type conformance by maintaining column types when reading and writing data back into Pandas (even when round tripping across multiple data types).This package also can read in data given anMoJ-Metadataschema again to result in a Pandas dataframe that best represents the provided schema.Can also be used to write data back to supported formats using Pandas (for CSV/JSONL) and PyArrow (for Parquet).Installationpip install arrow-pd-parseror via GitHubpip install arrow-pd-parser @ git+https://github.com/moj-analytical-services/mojap-arrow-pd-parserBasic UsageThis package usesPyArrowand/orPandasto parse CSVs, JSONL and Parquet files and convert them to a Pandas Dataframe that are the best representation of those datatypes and ensure conformance between them. Also can write data back into the above formats to still maintain conformance to the provided schema.fromarrow_pd_parserimportreader,writerdf1=reader.read("tests/data/all_types.csv")df1.dtypes# i                     Int64# my_bool             boolean# my_nullable_bool    boolean# my_date              object# my_datetime          object# my_int                Int64# my_string            stringdf2=reader.read("tests/data/all_types.jsonl")df2.dtypes# Note that Pandas struggles with nullable booleans when it reads JSONL (hence it is an Int64)# Write the dataframe to parquet# note deafult settings for the parquet writer is to# compress using snappy. (compression is not inferred from filepath but may do in future releases)writer.write(df1,"new_output.snappy.parquet")Note that the default behavior of this package is to utilise the new pandas datatypes for Integers, Booleans and Strings that represent Nulls aspd.NA(). Dates are returned as nullable objects ofdatetime.date()type and timestamps aredatetime.datetime(). By default we enforce these types instead of the native pandas timestamp as the indexing for the Pandas timestamp is nanoseconds and can cause dates to be out of bounds. See thetimestampssection for more details.Thereader.read()method will infer what the file format is based on the extension of your filepath failing that it will take the format from your metadata if provided. By default the reader.read() will use the following readers for the prescribed file format.Data TypeReaderWriterCSVPandasPandasJSONLPandasPandasParquetArrowArrowYou can also specify the file format in the reader and writer function or specify the reader type:fromarrow_pd_parserimportreader,writer# Specifying the reader# Both reader statements are equivalent and call the same readers# under the hooddf1=reader.read("tests/data/all_types.csv",file_format="csv")df2=reader.csv.read("tests/data/all_types.csv")# You can also pass the reader args to the reader as kwargsdf3=reader.csv.read("tests/data/all_types.csv",nrows=2)# The writer API has the same functionalitywriter.write(df1,file_format="parquet")writer.parquet.write(df1)Use of MetadataThe main usefulness of this repo comes from specifying metadata for the data you want. We are going to focus on CSV and JSONL in this section as Parquet becomes a bit more nuanced as it encodes data types (unlike CSV and JSONL that can be more vague).When reading and writing you can specify metadata (using ourmetadata schema definitions) to ensure the data read into pandas conforms.fromarrow_pd_parserimportreader# If no meta is provided we let the reader infer typedf1=reader.read("tests/data/all_types.csv")df1.dtypes# i                     Int64# my_bool             boolean# my_nullable_bool    boolean# my_date              object# my_datetime          object# my_int                Int64# my_string            string# If metadata is provided we ensure conformancemeta={"columns":[{"name":"my_bool","type":"string"},{"name":"my_nullable_bool","type":"bool"},{"name":"my_date","type":"string"},{"name":"my_datetime","type":"string"},{"name":"my_int","type":"float64"},{"name":"my_string","type":"string"}]}df2=reader.read("tests/data/all_types.csv",metadata=meta)df2.dtypes# i                     Int64# my_bool              string# my_nullable_bool    boolean# my_date              string# my_datetime          string# my_int              float64# my_string            stringdf3=reader.read("tests/data/all_types.jsonl",metadata=meta)df3.dtypesassertdf2.dtypes.to_list()==df3.dtypes.to_list()As of the v1 release we expect the API for reading and writing will remain the same but will still be lacking in how the caster works (what is called under the hood when casting the data to the prescribed metadata). The caster should improve with subsequent releases.Advanced UsageParameterising your reader and writerWhen using thereader.read()orwriter.write()it pulls the default method with their default settings. However, if you want to customise your reader/writer you can do.Passing arguments to the readerYou can pass arguments to the underlying reader that is used to read data in. In the example below we use the nrows arg in pd.pd.read_csvthat is used for our underlying reader.fromarrow_pd_parserimportreader# Passing args to the read function.df=reader.csv.read("tests/data/all_types.jsonl",nrows=1000)Reader and Writer settingsThe readers and writers have some settings that you can use to configure how it reads/writes data. One of the main settings is how we deal with Pandas types. We default to the new pandas Series types:StringDtypeforstring,BooleanDtypeforboolandInt64Dtypeforinteger. We also force dates and timestamps to be a series of objects (see the Timestamp section below)[#Reader-Pandas-Timestamps]. To change what pandas types to use you can change the reader settings:fromarrow_pd_parser.readerimportcsvfromioimportStringIOcsv.pd_integer=Falsecsv.pd_string=Falsecsv.bool_map={"Yes":True,"No":False}data="""int_col,str_col,bool_col1,"Hello, mate",Yes2,Hi,No"""meta={"columns":[{"name":"int_col","type":"int64"},{"name":"str_col","type":"string"},{"name":"bool_col","type":"bool"},]}f=StringIO(data)df=csv.read(f,metadata=meta)df.dtypes# int_col       int64# str_col      object# bool_col    boolean#### Using your own readerIf you wanted to create your own instance of a reader that wasn't the default provided you can.fromarrow_pd_parser.readerimportPandasCsvReader,csv# But you can also create your own and init the settingsspecific_csv=PandasCsvReader(pd_integer=False,pd_string=False,bool_map={"Yes":True,"No":False})# This may come in handy if you want to have two instances of a CSV reader with different settingsdf1=csv.read("tests/data/all_types.csv")# default readerdf2=specific_csv.read("tests/data/all_types.csv")Reading and Writing ParquetThe default parquet reader and writer uses Arrow under the hood. This also means it uses a different method for casting datatypes (it is done in Arrow not with Pandas). Because parquet is stricter, most likely if there is a deviation from your metadata it will fail to cast. When you provide metadata to the reader and writer, the API forces the data to that metadata. This makes sense for CSV and JSONL, but may make less sense for parquet. Therefore when using the parquet reader and writer you might want to consider if you want to provide your metadata. This may still be advantagous to check that the parquet matches metadata schema you have for the data.In some cases (seethis issue) pyarrow will silently convert some data types when writing a parquet file, overriding the provided metadata. Thewriterwill raise a warning if this happens, and you may wish to update your metdata so it matches the type that arrow expects.fromarrow_pd_parserimportreader,writer# Note all readers (CSV, JSONL and Parquet) support S3 paths in the following waydf=reader.parquet.read("s3://bucket/in.snappy.parquet")writer.parquet.write("s3://bucket/out.snappy.parquet")print(writer.parquet.compression,writer.parquet.version)# The default settings for the Parquet writer are snappy compression and version = '2.0'# These are passed to the arrow parquet write method but can either be changed by setting the# properties or by passing them as kwargs (which will super seed the default properties) e.g.writer.parquet.write("s3://bucket/out.parquet",compression=None,version='1.0')Pandas TimestampsWhen metadata is provided (or for all Arrow Readers without metadata) we default to dates and datetimes as a series of objects rather than the Pandas timestamps this is because Pandas timestamps (currently) only support nanosecond resolution which is not ideal for a lot of timestamps as the range can be often too small.importpandasaspdpd.Timestamp.min# Timestamp('1677-09-22 00:12:43.145225')pd.Timestamp.max# Timestamp('2262-04-11 23:47:16.854775807')Whereas Spark 3.0 (for example) allows timestamps from0001-01-01 00:00:00to9999-12-31 23:59:59.999999(source). By default we do not allow Timestamps for this reason instead we use the python native datetime class as default for our types (wrapped in a Pandas object column type). Users can specify other Pandas date/timestamp using thepd_timestamp_typeparameter which can either beobject(default),pd_timestamporpd_period.When settingpd_timestamp_type=pd_periodpd_arrow_parser will identify the correct pandas period resolution based on the arrow column type.fromarrow_pd_parser.readerimportcsv# The Pandas Periodcsv.pd_timestamp_type="pd_period"df=csv.read("tests/data/datetime_type.csv")df.my_datetime.dtype# "period[S]"# Use pandas timestampcsv.pd_timestamp_type="pd_timestamp"df=csv.read("tests/data/datetime_type.csv")df.my_datetime.dtype# dtype('<M8[ns]')Reading and writing large datasetsDatasets that are too large to fit into memory can be read in chunks. If thechunksizeparameter is given toreader.readthen an iterator of dataframes is returned rather than a single dataframe.chunksizecan be an integer indicating the number of rows each chunk contains, or a string indicating the amount of memory each chunk should fill, e.g. "1 GB". Note that the memory size should not fill available memory as some overhead is required for reading and writing. Thewriter.writefunction can then use these iterators instead of a dataframe.fromarrow_pd_parserimportreader,writerdf_iter=reader.read("s3://my_bucket/csv_data/my_table.csv",chunksize=10000)writer.write(df_iter,"s3://my_bucket/parquet_data/my_table.parquet")If the dataframe needs transforming before writing then use a generator.fromarrow_pd_parserimportreader,writerdeftransform(df):# do somethingreturndfdf_iter=reader.read("s3://my_bucket/csv_data/my_table.csv",chunksize="500MB")# Using a generator comprehensiondf_transformed_iter=(transform(df)fordfindf_iter)writer.write(df_transformed_iter,"s3://my_bucket/parquet_data/my_transformed_table.parquet")