Package: etlhelper
etlhelperetlhelper is a Python ETL library to simplify data transfer into and out of databases.Note: There are a number of breaking changes planned foretlhelperversion 1.0.
Please pin the version number in your dependency list to avoid disruption and
watch the project on GitHub for notification of new releases (in Custom section).Overviewetlhelpermakes it easy to run a SQL query via Python and return the results.
It is built upon theDBAPI2
specificationand takes care of
importing drivers, formatting connection strings and cursor management.
This reduces the amount of boilerplate code required to query a relational
database with Python.Featuressetup_oracle_clientscript installs Oracle Instant Client on Linux systemsDbParamsobjects provide consistent way to connect to different database types (currently Oracle, PostgreSQL, SQLite and MS SQL Server)get_rows,iter_rows,fetchoneand other functions for querying databaseexecute,executemany, andloadfunctions to insert datacopy_rowsandcopy_table_rowsto transfer data from one database to anotheron_errorfunction to process rows that fail to insertSupport for parameterised queries and in-flight transformation of dataOutput results as namedtuple or dictionaryTimestamped log messages for tracking long-running data transfersHelpful error messages display the failed query SQLThese tools can create easy-to-understand, lightweight, versionable and testable Extract-Transform-Load (ETL) workflows.etlhelperis not a tool for coordinating ETL jobs (useApache Airflow), for
converting GIS data formats (useogr2ogrorfiona), for translating between SQL dialects or providing Object Relation Mapping (useSQLAlchemy).
However, it can be used in conjunction with each of these.The documentation below explains how the main features are used.
See the individual function docstrings for full details of parameters and
options.For a high level introduction toetlhelper, see the FOSS4GUK 2019 presentationOpen Source Spatial ETL with Python and Apache Airflow:video(20 mins),slides.DocumentationInstallationConnect to databasesTransfer dataUtilitiesRecipesDevelopmentReferencesInstallationPython packagespipinstalletlhelperDatabase driver packages are not included by default and should be specified in
square brackets.
Options areoracle(installs cx_Oracle),mssql(installs pyodbc) andpostgres(installs psycopg2).
Multiple values can be separated by commas.pip install etlhelper[oracle,postgres]Thesqlite3driver is included within Python's Standard Library.Database driver dependenciesSome database drivers have additional dependencies.
On Linux, these can be installed via the system package manager.cx_Oracle (for Oracle):sudo apt install libaio1(Debian/Ubuntu) orsudo dnf install libaio(CentOS, RHEL, Fedora)pyodbc (for MS SQL Server):Follow instructions onMicrosoft SQL Docs websiteOracle Instant ClientOracle Instant Client libraries are required to connect to Oracle databases.
On Linux,etlhelperprovides a script to download and unzip them from theOracle
website.
Once the drivers are installed, their location must be added to LD_LIBRARY_PATH
environment variable before they can be used.setup_oracle_clientwrites
a file that can then be "sourced" to do this for the current shell.  These two steps
can be executed in a single command as:source$(setup_oracle_client)This command must be run in each new shell session.
Seesetup_oracle_client --helpfor further command line flags, including
specifying an alternative URL or filesystem path for the zipfile location.Connect to databasesDbParamsDatabase connection details are defined byDbParamsobjects.
Connections are made via theirconnectfunctions (see below).DbParamsobjects are created as follows or from environment variables using thefrom_environment()function.
The class initialisation function checks that the correct attributes have been provided for
a givendbtype.frometlhelperimportDbParamsORACLEDB=DbParams(dbtype='ORACLE',host="localhost",port=1521,dbname="mydata",user="oracle_user")POSTGRESDB=DbParams(dbtype='PG',host="localhost",port=5432,dbname="mydata",user="postgres_user")SQLITEDB=DbParams(dbtype='SQLITE',filename='/path/to/file.db')MSSQLDB=DbParams(dbtype='MSSQL',host="localhost",port=1433,dbname="mydata",user="mssql_user",odbc_driver="ODBC Driver 17 for SQL Server")DbParams objects have a function to check if a given database can be reached
over the network.  This does not require a username or password.ifnotORACLEDB.is_reachable():raiseETLHelperError("Network problems")Other methods/properties areget_connection_string,get_sqlalchemy_connection_string,paramstyleandcopy.
See function docstrings for details.connectfunctionTheDbParams.connect()function returns a DBAPI2 connection as provided by the
underlying driver.
Using context-manager syntax as below ensures that the connection is closed
after use.withSQLITEDB.connect()asconn1:withPOSTGRESDB.connect('PGPASSWORD')asconn2:do_something()A standaloneconnectfunction provides backwards-compatibility with
previous releases ofetlhelper:frometlhelperimportconnectconn3=connect(ORACLEDB,'ORACLE_PASSWORD')Both versions accept additional keyword arguments that are passed to theconnectfunction of the underlying driver.  For example, the following sets the character
encoding used by cx_Oracle to ensure that values are returned as UTF-8:conn4=connect(ORACLEDB,'ORACLE_PASSWORD',encoding="UTF-8",nencoding="UTF8")The above is a solution when special characters are scrambled in the returned data.Disabling fast_executemany for SQL Server and other pyODBC connectionsBy default anetlhelperpyODBC connection uses a cursor with itsfast_executemanyattribute set toTrue. This setting improves the
performance of theexecutemanywhen performing bulk inserts to a
SQL Server database. However, this overides the default behaviour
of pyODBC and there are some limitations in doing this. Importantly,
it is only recommended for applications that use Microsoft's ODBC Driver for
SQL Server. SeepyODBC fast_executemany.Usingfast_executemanymay raise aMemoryErrorif query involves columns of typesTEXTandNTEXT, which are now deprecated.
Under these circumstances,etlhelperfalls back onfast_executemanybeing set toFalseand produces a warning output. SeeInserting into SQL server with
fast_executemany results in MemoryError.If required, thefast_executemanyattribute can be set toFalsevia theconnectfunction:conn5=connect(MSSQLDB,'MSSQL_PASSWORD',fast_executemany=False)This keyword argument is used byetlhelper, any further keyword arguments are
passed to theconnectfunction of the underlying driver.PasswordsDatabase passwords must be specified via an environment variable.
This reduces the temptation to store them within scripts.
This can be done on the command line via:export ORACLE_PASSWORD=some-secret-passwordon Linuxset ORACLE_PASSWORD=some-secret-passwordon WindowsOr in a Python terminal via:importosos.environ['ORACLE_PASSWORD']='some-secret-password'No password is required for SQLite databases.Transfer dataGet rowsTheget_rowsfunction returns a list of named tuples containing data as
native Python objects.frommy_databasesimportORACLEDBfrometlhelperimportget_rowssql="SELECT * FROM src"withORACLEDB.connect("ORA_PASSWORD")asconn:get_rows(sql,conn)returns[Row(id=1, value=1.234, simple_text='text', utf8_text='Öæ°\nz',
     day=datetime.date(2018, 12, 7),
     date_time=datetime.datetime(2018, 12, 7, 13, 1, 59)),
 Row(id=2, value=2.234, simple_text='text', utf8_text='Öæ°\nz',
     day=datetime.date(2018, 12, 8),
     date_time=datetime.datetime(2018, 12, 8, 13, 1, 59)),
 Row(id=3, value=2.234, simple_text='text', utf8_text='Öæ°\nz',
     day=datetime.date(2018, 12, 9),
     date_time=datetime.datetime(2018, 12, 9, 13, 1, 59))]Data are accessible via index (row[4]) or name (row.day).Other functions are provided to select data.fetchone,fetchmanyandfetchallare equivalent to the cursor methods specified in the DBAPI v2.0.dump_rowspasses each row to a function (default isprint).iter_rowsIt is recommended to useiter_rowsfor looping over large result sets.  It
is a generator function that only yields data as requested.  This ensures that
the data are not all loaded into memory at once.with ORACLEDB.connect("ORA_PASSWORD") as conn:
    for row in iter_rows(sql, conn):
        do_something(row)ParametersVariables can be inserted into queries by passing them as parameters.
These "bind variables" are sanitised by the underlying drivers to preventSQL
injection attacks.
The requiredparamstylecan be checked withMY_DB.paramstyle.
A tuple is used for positional placeholders, or a dictionary for named
placeholders.select_sql="SELECT * FROM src WHERE id = :id"withORACLEDB.connect("ORA_PASSWORD")asconn:get_rows(sql,conn,parameters={'id':1})Row factoriesRow factories control the output format of returned rows.For example return each row as a dictionary, use the following:frometlhelperimportget_rowsfrometlhelper.row_factoriesimportdict_row_factorysql="SELECT * FROM my_table"withORACLEDB.connect('ORACLE_PASSWORD')asconn:forrowinget_rows(sql,conn,row_factory=dict_row_factory):print(row['id'])Thedict_row_factoryis useful when data are to be serialised to JSON/YAML,
as those formats use dictionaries as input.Four different row_factories are included, based in built-in Python types:Row FactoryAttribute accessMutableParameter placeholdernamedtuple_row_factory (default)row.idorrow[0]NoPositionaldict_row_factoryrow["id"]YesNamedtuple_row_factoryrow[0]NoPositionallist_row_factoryrow[0]YesPositionalThe choice of row factory depends on the use case.  In general named tuples
and dictionaries are best for readable code, while using tuples or lists can
give a slight increase in performance.
Mutable rows are convenient when used with transform functions because they
can be modified without need to create a whole new output row.When usingcopy_rows, it is necessary to use approriate parameter placeholder
style for the chosen row factory in the INSERT query.
Using thedict_row_factoryrequires a switch from named to positional
parameter placeholders (e.g.%(id)sinstead of%sfor PostgreSQL,:idinstead of:1for Oracle).
Thepyodbcdriver for MSSQL only supports positional placeholders.When using theloadfunction in conjuction withiter_chunksdata must be
either named tuples or dictionaries.TransformThetransformparameter allows passing of a function to transform the data
before returning it.
The function must take a list of rows and return a list of modified rows.
Seecopy_rowsfor more details.Chunk sizeAll data extraction functions useiter_chunksbehind the scenes.
This reads rows from the database in chunks to prevent them all being loaded
into memory at once.
The defaultchunk_sizeis 5000 and this can be set via keyword argument.Insert rowsexecutecan be used to insert a single row or to execute other single
statements e.g. "CREATE TABLE ...".
Theexecutemanyfunction is used to insert multiple rows of data.
Large datasets are broken into chunks and inserted in batches to reduce the
number of queries.
The INSERT query must container placeholders with an appropriate format for
the input data e.g. positional for tuples, named for dictionaries.
The number of rows that were processed and the number that failed is returned.frometlhelperimportexecutemanyrows=[(1,'value'),(2,'another value')]insert_sql="INSERT INTO some_table (col1, col2) VALUES (%s,%s)"withPOSTGRESDB.connect('PGPASSWORD')asconn:processed,failed=executemany(insert_sql,conn,rows,chunk_size=1000)Thechunk_sizedefault is 5,000 and it can be set with a keyword argument.
Thecommit_chunksflag defaults toTrue.
This ensures that an error during a large data transfer doesn't require all the
records to be sent again.
Some work may be required to determine which records remain to be sent.
Settingcommit_chunkstoFalsewill roll back the entire transfer in case
of an error.Some database engines can return autogenerated values (e.g. primary key IDs)
after INSERT statements.
To capture these values, use thefetchonemethod to execute the SQL command
instead.insert_sql="INSERT INTO my_table (message) VALUES ('hello') RETURNING id"withPOSTGRESDB.connect('PGPASSWORD')asconn:result=fetchone(insert_sql,conn)print(result.id)Theloadfunction is similar toexecutemanyexcept that it autogenerates
an insert query based on the data provided. It usesgenerate_insert_queryto remove the need to explicitly write the query for simple cases. By
calling this function manually, users can create a base insert query that can
be extended with clauses such asON CONFLICT DO NOTHING.NOTE: theloadfunction uses the first row of data to generate the list of column for the insert query. If later items in the data contain extra columns, those columns will not be inserted and no error will be raised.Asgenerate_insert_querycreates SQL statements from user-provided input,
it checks the table and column names to ensure that they only contain valid
characters.Handling insert errorsThe default behaviour ofetlhelperis to raise an exception on the first
error and abort the transfer.
Sometimes it is desirable to ignore the errors and to do something else with
the failed rows.
Theon_errorparameter allows a function to be passed that is applied to the
failed rows of each chunk.
The input is a list of (row, exception) tuples.Different examples are given here.  The simplest approach is to collect all the
errors into a list to process at the end.errors=[]executemany(sql,conn,rows,on_error=errors.extend)iferrors:do_something()Errors can be logged to theetlhelperlogger.frometlhelperimportloggerdeflog_errors(failed_rows):forrow,exceptioninfailed_rows:logger.error(exception)executemany(sql,conn,rows,on_error=log_errors)The IDs of failed rows can be written to a file.defwrite_bad_ids(failed_rows):withopen('bad_ids.txt','at')asout_file:forrow,exceptioninfailed_rows:out_file.write(f"{row.id}\n")executemany(sql,conn,rows,on_error=write_bad_ids)executemany,load,copy_rowsandcopy_table_rowscan all take anon_errorparameter.  They each return a tuple containing the number of rows
processed and the number of rows that failed.Copy table rowscopy_table_rowsprovides a simple way to copy all the data from one table to
another.
It can take atransformfunction in case some modification of the data, e.g.
change of case of column names, is required.frommy_databasesimportPOSTGRESDB,ORACLEDBfrometlhelperimportcopy_table_rowswithORACLEDB.connect("ORA_PASSWORD")assrc_conn:withPOSTGRESDB.connect("PG_PASSWORD")asdest_conn:copy_table_rows('my_table',src_conn,dest_conn)Thechunk_size,commit_chunksandon_errorparameters can all be set.
A tuple with counts of rows processed and failed is returned.Note that the target table must already exist.
If it doesn't, you can useexecutewith aCREATE TABLE IF NOT EXISTS ...statement to create it first.
See the recipes for examples.Combiningiter_rowswithloadFor extra control selecting the data to be transferred,iter_rowscan be
combined withload.frommy_databasesimportPOSTGRESDB,ORACLEDBfrometlhelperimportiter_rows,loadselect_sql="""SELECT id, name, value FROM my_tableWHERE value > :min_value"""withORACLEDB.connect("ORA_PASSWORD")assrc_conn:withPOSTGRESDB.connect("PG_PASSWORD")asdest_conn:rows=iter_rows(select_sql,src_conn,parameters={'min_value':99})load('my_table',dest_conn,rows)Copy rowsCustomising both queries gives the greatest control on data selection and loading.copy_rowstakes the results from a SELECT query and applies them as parameters
to an INSERT query.
The source and destination tables must already exist.
For example, here we use GROUP BY and WHERE in the SELECT query and insert extra
auto-generated values via the INSERT query.frommy_databasesimportPOSTGRESDB,ORACLEDBfrometlhelperimportcopy_rowsselect_sql="""SELECTcustomer_id,SUM (amount) AS total_amountFROM paymentWHERE id > 1000GROUP BY customer_id"""insert_sql="""INSERT INTO dest (customer_id, total_amount, loaded_by, load_time)VALUES (%s,%s, current_user, now())"""withORACLEDB.connect("ORA_PASSWORD")assrc_conn:withPOSTGRESDB.connect("PG_PASSWORD")asdest_conn:copy_rows(select_sql,src_conn,insert_sql,dest_conn)parameterscan be passed to the SELECT query as before and thecommit_chunks,chunk_sizeandon_erroroptions can be set.A tuple of rows processed and failed is returned.TransformData can be transformed in-flight by applying a transform function.  This is
any Python callable (e.g. function) that takes an iterator (e.g. list) and returns
another iterator.
Transform functions are applied to data as they are read from the database and
can be used withget_rows-type methods and withcopy_rows.The following code demonstrates that the returned chunk can have a different number
of rows, and be of different length, to the input.
When used withcopy_rows, the INSERT query must contain the correct placeholders for the
transform result.
Extra data can result from a calculation, a call to a webservice or another database.importrandomdefmy_transform(chunk):# Append random integer (1-10), filter if <5.new_chunk=[]forrowinchunk:# each row is a namedtupleextra_value=random.randrange(10)ifextra_value>=5:new_chunk.append((*row,extra_value))returnnew_chunkcopy_rows(select_sql,src_conn,insert_sql,dest_conn,transform=my_transform)It can be easier to modify individual columns when using thedict_row_factory(see above).frometlhelper.row_factoriesimportdict_row_factorydefmy_transform(chunk):# Add prefix to id, remove newlines, set lower case email addressesnew_chunk=[]forrowinchunk:# each row is a dictionaryrow['id']+=1000row['description']=row['description'].replace('\n',' ')row['email']=row['email'].lower()new_chunk.append(row)returnnew_chunkget_rows(select_sql,src_conn,row_factory=dict_row_factory,transform=my_transform)Theiter_chunksanditer_rowsfunctions that are used internally return
generators.  Each chunk or row of data is only accessed when it is required.
The transform function can also be written to return a generator instead of
a list.  Data transformation can then be performed viamemory-efficient
iterator-chains.Aborting running jobsWhen running as a script,etlhelperjobs can be stopped by pressingCTRL-C.
This option is not available when the job is running as a background process,
e.g. in a GUI application.
Theabort_etlhelper_threads()function is provided to cancel jobs running in
a separate thread by raising anETLHelperAbortexception within the thread.The state of the data when the job is cancelled (or crashes) depends on the
arguments passed toexecutemany(or the functions that call it e.g.load,copy_rows).Ifcommit_chunksisTrue(default), all chunks up to the one where the
error occured are committed.Ifcommit_chunksisFalse, everything is rolled back and the database is
unchanged.If anon_errorfunction is defined, all rows without errors are committed.UtilitiesThe following utility functions provide useful database metadata.Table infoThetable_infofunction provides basic metadata for a table. An optional schema
can be used. Note that forsqlitethe schema value is currently ignored.frometlhelper.utilsimporttable_infowithORACLEDB.connect("ORA_PASSWORD")asconn:columns=table_info('my_table',conn,schema='my_schema')The returned value is a list of named tuples of four values. Each tuple represents
one column in the table, giving its name, type, if it has a NOT NULL constraint
and if is has a DEFAULT value constraint. For example,[Column(name='ID',type='NUMBER',not_null=1,has_default=0),Column(name='VALUE',type='VARCHAR2',not_null=0,has_default=1),]the ID column is of type NUMBER and has a NOT NULL constraint but not a DEFAULT value,
while the VALUE column is of type VARCHAR2, can be NULL but does have a DEFAULT value.RecipesThe following recipes demonstrate howetlhelpercan be used.Debug SQL and monitor progress with loggingETL Helper provides a custom logging handler.
Time-stamped messages indicating the number of rows processed can be enabled by
setting the log level to INFO.
Setting the level to DEBUG provides information on the query that was run,
example data and the database connection.importloggingfrometlhelperimportloggerlogger.setLevel(logging.INFO)Output from a call tocopy_rowswill look like:2019-10-07 15:06:22,411 iter_chunks: Fetching rows
2019-10-07 15:06:22,413 executemany: 1 rows processed
2019-10-07 15:06:22,416 executemany: 2 rows processed
2019-10-07 15:06:22,419 executemany: 3 rows processed
2019-10-07 15:06:22,420 iter_chunks: 3 rows returned
2019-10-07 15:06:22,420 executemany: 3 rows processed in totalNote: errors on database connections output messages that include login
credentials in clear text.Database to database copy ETL script templateThe following is a template for an ETL script.
It copies copy all the sensor readings from the previous day from an Oracle
source to PostgreSQL destination.# copy_readings.pyimportdatetimeasdtfrometl_helperimportcopy_rowsfrommy_databasesimportORACLEDB,POSTGRESDBCREATE_SQL=dedent("""CREATE TABLE IF NOT EXISTS sensordata.readings(sensor_data_id bigint PRIMARY KEY,measure_id bigint,time_stamp timestamp without time zone,meas_value double precision)""").strip()DELETE_SQL=dedent("""DELETE FROM sensordata.readingsWHERE time_stamp BETWEEN%(startdate)sAND%(enddate)s""").strip()SELECT_SQL=dedent("""SELECT id, measure_id, time_stamp, readingFROM sensor_dataWHERE time_stamp BETWEEN :startdate AND :enddateORDER BY time_stamp""").strip()INSERT_SQL=dedent("""INSERT INTO sensordata.readings (sensor_data_id, measure_id, time_stamp,meas_value)VALUES (%s,%s,%s,%s)""").strip()defcopy_readings(startdate,enddate):params={'startdate':startdate,'enddate':enddate}withORACLEDB.connect("ORA_PASSWORD")assrc_conn:withPOSTGRESDB.connect("PG_PASSWORD")asdest_conn:execute(CREATE_SQLdest_conn)execute(DELETE_SQL,dest_conn,parameters=params)copy_rows(SELECT_SQL,src_conn,INSERT_SQL,dest_conn,parameters=params)if__name__=="__main__":# Copy data from 00:00:00 yesterday to 00:00:00 todaytoday=dt.combine(dt.date.today(),dt.time.min)yesterday=today-dt.timedelta(1)copy_readings(yesterday,today)It is valuable to createidempotentscripts to ensure that they can be rerun without problems.
In this example, the "CREATE TABLE IF NOT EXISTS" command can be called repeatedly.
The DELETE_SQL command clears existing data prior to insertion to prevent duplicate key errors.
SQL syntax such as "INSERT OR UPDATE", "UPSERT" or "INSERT ... ON CONFLICT" may be more efficient, but the the exact commands depend on the target database type.Calling ETL Helper scripts from Apache AirflowThe following is anApache Airflow
DAGthat uses thecopy_readingsfunction
defined in the script above.
The Airflow scheduler will create tasks for each day since 1 August 2019 and
callcopy_readingswith the appropriate start and end times.# readings_dag.pyimportdatetimeasdtfromairflowimportDAGfromairflow.operators.python_operatorimportPythonOperatorimportcopy_readingsdefcopy_readings_with_args(**kwargs):# Set arguments for copy_readings from contextstart=kwargs.get('prev_execution_date')end=kwargs.get('execution_date')copy_readings.copy_readings(start,end)dag=DAG('readings',schedule_interval=dt.timedelta(days=1),start_date=dt.datetime(2019,8,1),catchup=True)t1=PythonOperator(task_id='copy_readings',python_callable=copy_readings_with_args,provide_context=True,dag=dag)Spatial ETLNo specific drivers are required for spatial data if they are transferred as
Well Known Text.select_sql_oracle="""SELECTid,SDO_UTIL.TO_WKTGEOMETRY(geom)FROM src"""insert_sql_postgis="""INSERT INTO dest (id, geom) VALUES (%s,ST_Transform(ST_GeomFromText(%s, 4326), 27700))"""Other spatial operations e.g. coordinate transforms, intersections and
buffering can be carried out in the SQL.
Transform functions can manipulate geometries using theShapelylibrary.Database to API / NoSQL copy ETL script templateetlhelpercan be combined with Python'saiohttplibrary to create an ETL
for posting data from a database into an HTTP API.
The API could be a NoSQL document store (e.g. ElasticSearch, Cassandra) or some other
web service.This example transfers data from Oracle to ElasticSearch.
It usesiter_chunksto fetch data from the database without loading it all into
memory at once.
A custom transform function creates a dictionary structure from each row
of data.
This is "dumped" into JSON and posted to the API viaaiohttp.aiohttpallows the records in each chunk to be posted to the API
asynchronously.
The API is often the bottleneck in such pipelines and we have seen significant
speed increases (e.g. 10x) using asynchronous transfer as opposed to posting
records in series.# copy_sensors_async.pyimportasyncioimportdatetimeasdtimportjsonimportloggingimportaiohttpfrometlhelperimportiter_chunksfromdbimportORACLE_DBlogger=logging.getLogger("copy_sensors_async")SELECT_SENSORS="""SELECT CODE, DESCRIPTIONFROM BGS.DIC_SEN_SENSORWHERE date_updated BETWEEN :startdate AND :enddateORDER BY date_updated"""BASE_URL="http://localhost:9200/"HEADERS={'Content-Type':'application/json'}defcopy_sensors(startdate,enddate):"""Read sensors from Oracle and post to REST API."""logger.info("Copying sensors with timestamps from%sto%s",startdate.isoformat(),enddate.isoformat())row_count=0withORACLE_DB.connect('ORACLE_PASSWORD')asconn:# chunks is a generator that yields lists of dictionarieschunks=iter_chunks(SELECT_SENSORS,conn,parameters={"startdate":startdate,"enddate":enddate},transform=transform_sensors)forchunkinchunks:result=asyncio.run(post_chunk(chunk))row_count+=len(result)logger.info("%sitems transferred",row_count)logger.info("Transfer complete")deftransform_sensors(chunk):"""Transform rows to dictionaries suitable for converting to JSON."""new_chunk=[]forrowinchunk:new_row={'sample_code':row.CODE,'description':row.DESCRIPTION,'metadata':{'source':'ORACLE_DB',# fixed value'transferred_at':dt.datetime.now().isoformat()# dynamic value}}logger.debug(new_row)new_chunk.append(new_row)returnnew_chunkasyncdefpost_chunk(chunk):"""Post multiple items to API asynchronously."""asyncwithaiohttp.ClientSession()assession:# Build list of taskstasks=[]foriteminchunk:tasks.append(post_one(item,session))# Process tasks in parallel.  An exception in any will be raised.result=awaitasyncio.gather(*tasks)returnresultasyncdefpost_one(item,session):"""Post a single item to API using existing aiohttp Session."""# Post the itemresponse=awaitsession.post(BASE_URL+'sensors/_doc',headers=HEADERS,data=json.dumps(item))# Log responses before throwing errors because error info is not included# in generated Exceptions and so cannot otherwise be seen for debugging.ifresponse.status>=400:response_text=awaitresponse.text()logger.error('The following item failed:%s\nError message:\n(%s)',item,response_text)awaitresponse.raise_for_status()returnresponse.statusif__name__=="__main__":# Configure logginghandler=logging.StreamHandler()formatter=logging.Formatter('%(asctime)s%(levelname)-8s%(message)s')handler.setFormatter(formatter)logger.setLevel(logging.INFO)logger.addHandler(handler)# Copy data from 1 January 2000 to 00:00:00 todaytoday=dt.datetime.combine(dt.date.today(),dt.time.min)copy_sensors(dt.datetime(2000,1,1),today)In this example, failed rows will fail the whole job.  Removing theraise_for_status()call will let them just be logged instead.CSV load script templateThe following script is an example of using theloadfunction to import data
from a CSV file into a database.
It shows how atransformfunction can perform common parsing tasks such as
renaming columns and converting timestamps into datetime objects.
The database has aCHECKconstraint that rejects any rows with an ID
divisible by 1000.
An exampleon_errorfunction prints the IDs of rows that fail to insert."""Script to create database and load observations data from csv file. It alsodemonstrates how an `on_error` function can handle failed rows.Generate observations.csv with:curl 'https://sensors.bgs.ac.uk/FROST-Server/v1.1/Observations?$select=@iot.id,result,phenomenonTime&$top=20000&$resultFormat=csv' -o observations.csv"""importcsvimportdatetimeasdtfromtypingimportIterable,List,Tuplefrometlhelperimportexecute,load,DbParamsdefload_observations(csv_file,conn):"""Load observations from csv_file to db_file."""# Drop table (helps with repeated test runs!)drop_table_sql="""DROP TABLE IF EXISTS observations"""execute(drop_table_sql,conn)# Create table (reject ids with no remainder when divided by 1000)create_table_sql="""CREATE TABLE IF NOT EXISTS observations (id INTEGER PRIMARY KEY CHECK (id % 1000),time TIMESTAMP,result FLOAT)"""execute(create_table_sql,conn)# Load datawithopen(csv_file,'rt')asf:reader=csv.DictReader(f)load('observations',conn,transform(reader),on_error=on_error)# The on_error function is called after each chunk with all the failed rowsdefon_error(failed_rows:List[Tuple[dict,Exception]])->None:"""Print the IDs of failed rows"""rows,exceptions=zip(*failed_rows)failed_ids=[row['id']forrowinrows]print(f"Failed IDs:{failed_ids}")# A transform function that takes an iterable and yields one row at a time# returns a "generator".  The generator is also iterable, and records are# processed as they are read so the whole file is never held in memory.deftransform(rows:Iterable[dict])->Iterable[dict]:"""Rename time column and convert to Python datetime."""forrowinrows:row['time']=row.pop('phenomenonTime')row['time']=dt.datetime.strptime(row['time'],"%Y-%m-%dT%H:%M:%S.%fZ")yieldrowif__name__=="__main__":importloggingfrometlhelperimportloggerlogger.setLevel(logging.INFO)db=DbParams(dbtype="SQLITE",filename="observations.sqlite")withdb.connect()asconn:load_observations('observations.csv',conn)Export data to CSVThePandaslibrary can connect to databases via SQLAlchemy.
It has powerful tools for manipulating tabular data.
ETL Helper makes it easy to prepare the SQL Alchemy connection.importpandasaspdfromsqlalchemyimportcreate_enginefrommy_databasesimportORACLEDBengine=create_engine(ORACLEDB.get_sqlalchemy_connection_string("ORACLE_PASSWORD"))sql="SELECT * FROM my_table"df=pd.read_sql(sql,engine)df.to_csv('my_data.csv',header=True,index=False,float_format='%.3f')DevelopmentMaintainersETL Helper was created by and is maintained by British Geological Survey Informatics.John A Stevenson (volcan01010)Jo Walsh (metazool)Declan Valters (dvalters)Colin Blackburn (ximenesuk)Daniel Sutton (kerberpolis)Development statusThe code is still under active development and breaking changes are possible.
Users should pin the version in their dependency lists andwatchthe repository for new releases.
SeeCONTRIBUTING.mdfor details on how to contribute.LicenceETL Helper is distributed under theLGPL v3.0 licence.
Copyright: © BGS / UKRI 2019ReferencesPEP249 DB API2psycopg2cx_Oraclepyodbcsqlite3