Package: ooetl
ooetlis a minimal opinionated object oriented ETL framework.The class-based nature of the framework allows to build complex dedicated classes,
starting from simple abstract ones.InstallationPython versions from 3.7.1 are supported.The package is hosted on pypi, and can be installed, for example using pip:pip install --upgrade "ooetl[all]"
pip install "ooetl[elastic]==1.1.2"or poetry:poetry add ooetl -Eall
poetry add ooetl -EelasticUsageThis is a quick tutorial on how to use theooetlmodule in order to fetch
data from a SQL source and store them into a CSV destination.The ETL process is performed by invoking theetl()method on aooetl.ETLinstance.Theetl()method is a shortcut to the sequenceooetl.ETL::extract().transform().load(),
which is possible, as each method returns a pointer to theooetl.ETLinstance.When theooetl.ETLinstance invokes theooetl.ETL::extract()method, it invokes the correspongingooetl.Extractor::extract()method of theextractor. The method extracts the data from the source
into theooetl.ETL::original_dataattribute of the ooetl.ETLinstance.Theooetl.ETL::transform()method is overridden in the instance and may be used to apply
custom data transformation, before the loading phase.
The data fromooetl.ETL::original_dataare then transformed intoooetl.ETL::processed_data.Theooetl.ETL::load()method invokes theooetl.Loader::load()method storing the data fromooetl.ETL::processed_datainto the defined destination.The package provides a series of simple Extractors and Loaders, derived from common abstract classes.Extractors:CSVExctractor - extracts data from a remote CSVZIPCSVExctractor - extracts data from a remote zipped CSVHTMLParserExtractor - extracts data from a remote HTML page (requires html extra and needs to be extended)SparqlExtractor - extracts data from a remote SPARQL endpoint (requires sparql extra)SqlExtractor - extracts data from a RDBMS (requires mysql or postgresql extra)XSLExtractor - extracts data from a remote Excel fileZIPXLSExctractor - extracts data from an excel file within a remote zipped archiveLoaders:CSVLoader - loads data into a CSVJsonLoader - loads data into a json fileESLoader - loads data into an ES instance (requires elastic extra)DjangoBulkLoader - adds data in bulk to a django model (only works inside a django project)DjangoUpdateOrCreateLoader - adds data with an update or create logic into a django model (slow, only works within a django project)Theooetl.ETLabstract class is defined in the__init__.pyfile of theooetlpackage.ETL classes implement a pipeline of extraction, transformation and load logic.Aa a very basic example, here is how to extract data from a postgresql query, into a CSV file.fromooetlimportETLfromooetl.extractorsimportSqlExtractorfromooetl.loadersimportCSVLoaderETL(extractor=SqlExtractor(conn_url="postgresql://postgres:@localhost:5432/opdm",sql="select id, name, inhabitants from popolo_area where istat_classification='COM' ""order by inhabitants desc"),loader=CSVLoader(csv_path="./",label='opdm_areas'))()Extractors (and Loaders) may be easily extended within the projects using theooetlpackage.
As an example, consider the following snippet, extending the abstractooetl.HTMLParserExtractor, that parser
the Italian government's site and extracts the list of officials, as CSV.This example requires the html extra to be installed.importrequestsfromlxmlimporthtmlfromlxml.cssselectimportCSSSelectorfromooetlimportETLfromooetl.extractorsimportHTMLParserExtractorfromooetl.loadersimportCSVLoaderclassGovernoExtractor(HTMLParserExtractor):defparse(self,html_content):list_tree=html.fromstring(html_content)items=[]foreinCSSSelector('div.content div.box_text a')(list_tree):item_name=e.text_content().strip()item_url=e.get('href').strip()item_page=requests.get(item_url)item_tree=html.fromstring(item_page.content)item_par=CSSSelector('div.content div.field')(item_tree)[0]item_charge=CSSSelector('blockquote p')(item_par)[0].text_content().strip()item_descr=" ".join([e.text_content()foreinCSSSelector('p')(item_par)[1:]if\e.text_content()isnotNone])items.append({'nome':item_name,'url':item_url,'incarico':item_charge,'descrizione':item_descr})print(item_name)returnitemsETL(extractor=GovernoExtractor("https://www.governo.it/it/ministri-e-sottosegretari"),loader=CSVLoader(csv_path="./",label='governo'))()Other, more complex examples can be found in theexamplesdirectory.SupportThere is no guaranteed support available, but authors will try to keep up with issues
and merge proposed solutions into the code base.Project StatusThis project is currently being developed by theOpenpolis Foundationand is being used interanally.Currently extras for elasticsearch and sparql have been developed.Should more be needed, you can either ask to increase the coverage, or try to contribute, following instructions below.ContributingIn order to contribute to this project:verify that python 3.7.1+ is being used (or usepyenv)verify or installpoetry, to handle packages and dependencies in a leaner way,
with respect to pip and requirementsclone the projectgit clone git@github.com:openpolis/ooetl.gitinstall the dependencies in the virtualenv, withpoetry install -Eall,
this will also install the dev dependencies and all extrasdevelop and testcreate apull requestwait for the maintainers to review and eventually merge your pull request into the main repositoryTestingTests are under the tests folder, and can be launched withpytestRequests and responses from ATOKA's API are mocked, in order to avoid having to connect to
the remote service during tests (slow and needs an API key).Coverage is installed as a dev dependency and can be used to see how much of the package's code is covered by tests:coverage run -m pytest

# sends coverage report to terminal
coverage report -m 

# generate and open a web page with interactive coverage report
coverage html
open htmlcov/index.htmlSyntax can be checked withflake8.Coverage and flake8 configurations are in their sections withinsetup.cfg.AuthorsGuglielmo Celata -guglielmo@openpolis.itLicensingThis package is released under an MIT License, see details in the LICENSE.txt file.