Package: diligent
diligent: a transformer between database and Parquet fileIntroduction:diligent is a python tool library based on Python pyarrow which supports multithread and asynchronous calls. It can help
users transform data between database and Parquet files.Features:Multithread: support batch reading/writeing and multithread functions an database table and Parquet files.Read directory: reading all Parquet files in the giving directory and transform into database. One directory maps to
one database table.Mock data: create Parquet files or database tables in a customized schema.Workflow: giving a yaml file including your customized configurations, diligent can create a workflow to execute
multiple jobs.Limits:Only support Postgresql DB and Oracle DB by now.Installationsource codegitclonehttps://gitee.com/lipicoder/diligent.gitcddiligent
pipinstall-e.Pypi$python-mbuildpip$pipinstalldiligentUsageInput 'diligent' in command line. The usage lint will be displayed.(.context)%diligent
Usage:diligent[OPTIONS]COMMAND[ARGS]...diligentcommandlineinterface.

Options:-h,--helpShowthismessageandexit.

Commands:mockMockdatatodirectory,file,ordatabase.parquetWriteorreadParquetfileordirectoryworkflowDatabaseandparquetdatatransformworkflow.mockExample:# mock data workflow# workflow listsmocks:-name:"mock-to-directory"row-number:2048direction:"directory"# directory ,file or  tabledirectory:"./datasets/mock_data/mock_stocks"columns:{# stock code as a type"sec_code":"code",# "600001""date":"date",# 2022-08-24"close":"float",# 16.87"open":"float",# 16.65"high":"float",# 16.95"low":"float",# 16.55"hold":"decimal(10,5)",# 123.25515"time":"timestample(unit=s,tz=Asia/Shanghai)","volume":"int",# 1530231"amount":"int",# 2571196416"memo":"string",# hello}-name:"mock-to-file"row-number:2048direction:"file"# directory, file or tablefilepath:"./datasets/mock_data/mock_stocks.parquet"columns:{# stock code as a type"sec_code":"code",# "600001""date":"date",# 2022-08-24"close":"float",# 16.87"open":"float",# 16.65"high":"float",# 16.95"low":"float",# 16.55"hold":"decimal(10,5)",# 123.25515"time":"timestample(unit=s,tz=Asia/Shanghai)","volume":"int",# 1530231"amount":"int",# 2571196416"memo":"string",# hello}-name:"mock-to-table"row-number:204800direction:"table"# directory ,file or tableengine:"postgresql"dsn:"postgresql://test:test123@localhost:5432/diligent"table-name:"mock_stocks"columns:{# stock code as a type"sec_code":"code",# "600001""date":"date",# 2022-08-24"close":"float",# 16.87"open":"float",# 16.65"high":"float",# 16.95"low":"float",# 16.55"hold":"decimal(10,5)",# 123.25515# datetime.datetime(2022,10,25).timestamp()"time":"timestample(unit=s,tz=Asia/Shanghai)","volume":"int",# 1530231"amount":"int",# 2571196416}directiontransform to which format. 'directory': a directory path. 'file': a file path. 'table': a database
table name.columnsdefination of the database table. Support standard data types of Postgresql or Oracle db, based on which
database you choose. diligent will auto map the database data type to pyarrow row data type. 'code' means stock code,
which would be removed later.execute mock workflow:(.context)%diligentmocktests/yaml/mock_file.yaml
writedirectoryfinished
writeparquetfinishedparquetRead data in database table and write data into a Parquet file or Parquet files in a directory. Or read data in a
Parquet file or Parquet files in a directory and write data into a database table.(.context)%diligentparquet
Usage:diligentparquet[OPTIONS]COMMAND[ARGS]...WriteorreadParquetfileordirectory.

Options:-h,--helpShowthismessageandexit.

Commands:readReadparquetfileandwritetodatabase.writeReadfromdatabaseandwritetoparquetfile.parquet readRead data in a Parquet file or Parquet files in a directory and write data into a database table.(.context)%diligentparquetread-h
Usage:diligentparquetread[OPTIONS][-eENGINE]PATHDSNTABLE_NAMEReadparquetfile,writetodatabase.dsnsample:postgresql://user:password@host:port/dbname.engine[postgresql/oracle]Options:-h,--helpShowthismessageandexit.Example:diligentparquetreaddatasets/mock_data/mock_stocks.parquetpostgresql://test:test123@localhost:5432/diligentmock_stocks-epostgresqlparquet write:Read data in database table and write data into a Parquet file or Parquet files in a directory.directory.(.context)%diligentparquetwrite-h
Usage:diligentparquetwrite[OPTIONS][-eENGINE]PATHDSNTABLE_NAMEReadfromdatabaseandwritetoparquetfile.dsnsample:postgresql://user:password@host:port/dbname.engine[postgresql/oracle]Options:-s,--batch_sizeINTEGER-h,--helpShowthismessageandexit.Example:diligentparquetwritedatasets/mock_data/mock_stocks_tmppostgresql://test:test123@localhost:5432/diligentmock_stocks-epostgresqlparquet write supports configuration of batch sizediligentparquetwrite-s1024-epostgresqldatasets/mock_data/mock_stockspostgresql://test:test123@localhost:5432/diligentmock_stocksIn this case, the max row number of a single parquet file is 1024, we can see four files in the directory.(.context)%lsdatasets/mock_data/mock_stocks
ea6c445914824cae8ef171bbafd3a58f.parquet
604a63ccf14343c39bcc5bc0d1b3907d.parquet
9c7150d9821c46c78054d87ae23d900f.parquet
2ba1952316344b01a2a2f8e6faf41c31.parquetfilediligentparquetwrite-epostgresqldatasets/mock_data/mock_stocks_tmp.parquetpostgresql://test:test123@localhost:5432/diligentmock_stocks;Consider of reducing the memory usage and speed up the writing efficiency. write file can also support configuration of
batch size.workflowcreate and exectue a workflow using a yaml configuration file.(.context)%diligentworkflow-h
Usage:diligentworkflow[OPTIONS]YAML_FILEExecuteyamlworkflow.

Options:-h,--helpShowthismessageandexit.Example:workflows:-name:"read-workflow"# workflow namedirection:"table"# table or file or directorytable-name:"mock_stocks"# table nameengine:"postgresql"dsn:"postgresql://test:test123@localhost:5432/diligent"read-type:"file"# file or directoryfilepath:"./datasets/mock_data/mock_stocks.parquet"# filepath-name:"write-directory-workflow"# workflow namedirection:"directory"table-name:"mock_stocks"# table nameengine:"postgresql"dsn:"postgresql://test:test123@localhost:5432/diligent"directory:"./datasets/mock_data/mock_stocks"# directory pathbatch-size:1024# batch size-name:"write-file-workflow"# workflow namedirection:"file"table-name:"mock_stocks"# table nameengine:"postgresql"dsn:"postgresql://test:test123@localhost:5432/diligent"filepath:"./datasets/mock_data/mock_stocks_rewrite.parquet"TODOremove 'code' type from diligent mock. add new supported types such as random string and random digit string.Dependencypyarrow >=9.0.0python >= 3.10LicenseApache License 2.0