Package: toucan-connectors
Toucan ConnectorsToucan Tocodata connectors are plugins to the Toucan Toco platform. Their role is to returnPandas DataFramesfrom many different sources.Each connector is dedicated to a single type of source (PostrgeSQL, Mongo, Salesforce, etc...) and is made of two classes:Connectorwhich contains all the necessary information touse a data provider(e.g. hostname,  auth method and details, etc...).DataSourcewhich contains all the information toget a dataframe(query, path, etc...) using theConnectorclass above.The Toucan Toco platform instantiates these classes using values provided by Toucan admin and app designers, it then uses the following methods to get data and metadata:Connector._retrieve_datareturning an instance ofpandas.DataFrame, method used to return data to a Toucan Toco end userConnector.get_slicereturning an instance ofDataSlice, method used to return data to a Toucan Toco application designer when building a query.Connector.get_statusreturning an instance ofConnectorStatus, method used to inform an admin or Toucan Toco application designer of the status of its connection to a third party data service. Is it reachable from our servers? Are the authentication details and method working? etc...Installing for developmentWe usepoetryfor packaging and development. Use the following command to install the project for development:poetry install -E allDependenciesThis project usesmakeandPython 3.8. Install the main dependencies :pipinstall-e.We are using thesetuptoolsconstructextra_requiresto define each connector's dependencies separately. For example to install the MySQL connector dependencies:pipinstall-e".[mysql]"There is a shortcut calledallto install all the dependencies for all the connectors. I do not recommend that you use this as a contributor to this package, but if you do, use the section below to install the necessary system packages.pipinstall-e".[all]"You may face issues when instally the repo locally due to dependencies.
That's why a dev container is available to be used with visual studio.
Refer tothis docto use it.System packagesSome connectors dependencies require specific system packages. As each connector can define its dependencies separatly you do not need this until you want to use these specific connectors.ODBCOnlinux, you're going to need bindings forunixodbcto installpyodbcfrom the requirements, and to install that (using apt), just follow:sudoapt-getupdate
sudoapt-getinstallunixodbc-devMSSSQLTo test and usemssql(andazure_mssql) you need to install the Microsoft ODBC driver for SQL Server forLinuxorMacOSPostgreSQLOn macOS, to test thepostgresconnector, you need to installpostgresqlby running for instancebrew install postgres.
You can then install the library withenv LDFLAGS='-L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib' pip install psycopg2TestingWe are usingpytestand various packages of its ecosystem.
To install the testing dependencies, run:pipinstall-rrequirements-testing.txtAs each connector is an independant plugin, its tests are written independently from the rest of the codebase.
Run the tests for a specifc connector (http_apiin this example) like this:pytesttests/http_apiNote: running the tests above implies that you have installed the specific dependencies of thehttp_apiconnector (using thepip install -e .[http_api]command)Our CI does run all the tests for all the connectors, like this:pip install -e ".[all]"
make testSome connectors are tested using mocks (cf.trello), others are tested by making calls to data providers (cf.elasticsearch) running on the system in docker containers. The required images are in thetests/docker-compose.ymlfile, they need to be pulled (cf.pytest --pull) to run the relevant tests.ContributingThis is an open source repository under theBSD 3-Clause Licence. The Toucan Toco tech team are the maintainers of this repository, we welcome contributions.At the moment the main use of this code is its integration into Toucan Toco commercially licenced software, as a result our dev and maintenance efforts applied here are mostly driven by Toucan Toco internal priorities.The starting point of a contribution should be anIssue, either one you create or an existing one. This allows us (maintainers) to discuss the contribution before it is produced and avoids back and forth in reviews or stalled pull requests.Step 1: Generate base classes and tests filesTo generate the connector and test modules from boilerplate, run:make new_connector type=mytypemytypeshould be the name of a system we would like to build a connector for,
such asMySQLorMagento.Open the folder intestsfor the new connector. You can start writing your tests before implementing it.Some connectors are tested with calls to the actual data systems that they target,
for exampleelasticsearch,mongo,mssql.Others are tested with mocks of the
classes or functions returning data that you are wrapping (see :HttpAPI, ormicrostrategy).If you have a container for your target system, add a docker image in
thedocker-compose.yml, then use thepytestfixtureservice_containerto automatically
start the docker and shut it down for you when you are running tests.The fixture will not pull the image for you for each test runs, you need to pull the image on your machine (at least once) using thepytest --pulloption.Step 2: New connectorOpen the foldermytypeintoucan_connectorsfor your new connector and create your classes.importpandasaspd# Careful here you need to import ToucanConnector from the deep path, not the __init__ path.fromtoucan_connectors.toucan_connectorimportToucanConnector,ToucanDataSourceclassMyTypeDataSource(ToucanDataSource):"""Model of my datasource"""query:strclassMyTypeConnector(ToucanConnector,data_source_model=MyTypeDataSource):"""Model of my connector"""host:strport:intdatabase:strdef_retrieve_data(self,data_source:MyTypeDataSource)->pd.DataFrame:...defget_slice(self,...)->DataSlice:...defget_status(self)->ConnectorStatus:...Step 3: Register your connector, add documentationAdd your connector intoucan_connectors/__init__.py.
The key is what we call thetypeof the connector, which
is an id used to retrieve it when used in Toucan Toco platform.CONNECTORS_CATALOGUE={...,'MyType':'mytype.mytype_connector.MyTypeConnector',...}Add you connector requirements to thesetup.pyin theextras_requiredictionary:extras_require={...'mytype':['my_dependency_pkg1==x.x.x', 'my_dependency_pkg2>=x.x.x']}If you need to add testing dependencies, add them to therequirements-testing.txtfile.You can now generate and edit the documentation page for your connector:# Example: PYTHONPATH=. python doc/generate.py github > doc/connectors/github.mdPYTHONPATH=.pythondoc/generate.pymyconnectormodule>doc/connectors/mytypeconnector.mdStep 4 : Create a pull requestMake sure your new code is properly formatted by runningmake lint. If it's not, please usemake format. You can now create a pull request.Publishing a releaseCreate a pull request updatingonlythechangelogand theversionattribute of the[tool.poetry]section in thepyproject.tomlfile.Once the pull request is approved, merge it using thesquash and mergestrategy.Create an annotated tag for the release commit. it should be in thevX.Y.Zformat, whereX.Y.Zis the semver
version defined inpyproject.toml. Example:git tag -a v1.23.45 -m v1.23.45 ea3768a
git push origin v1.23.45In the project'sReleasespage, click on theDraft a new releasebutton. Pick the tag you just pushed,
and click onGenerate release notes. Adapt the releases notes if needed, and click onPublish release.A GitHub action in charge of publishing the required artifacts to PyPI should now be running. Make sure
the action is successful.