Python ETL vs. ETL Tools
By: Sean Knight
Tags: Etl, Python, Data Science, Programming, Big Data

Python ETL vs. ETL Tools

Should you code it yourself or license an ETL tool?

Pick your direction: coding your ETL pipeline yourself or using an existing ETL tool (image by author)

If you’re researching ETL solutions you are going to have to decide between using an existing ETL tool, or building your own using one of the Python ETL libraries. In this article, we look at some of the factors to consider when making that decision.

ETL (Extract Transform Load) is the most important aspect of creating data pipelines for data warehouses. The market offers various ready-to-use ETL tools that can be implemented in the data warehouse very easily. However, recently Python has also emerged as a great option for creating custom ETL pipelines. In this article, we shall give a quick comparison between Python ETL vs ETL tools to help you choose between the two for your project.

ETL Tools

There are many ready-to-use ETL tools available in the market for building easy-to-complex data pipelines. Most offer friendly graphical user interfaces, have rich pipeline building features, support various databases and data formats, and sometimes even include some limited business intelligence features. The best thing about it is that all of this is available out of the box.

These tools can be either licensed or open-sourced. Most of them are priced on a subscription model that ranges from anywhere between a few hundred dollars per month to thousands of dollars per month. On the other hand, the open-source tools are free, and they also offer some of the features that the licensed tools provide, but there is often much more development required to reach a similar result.

A few of the ETL tools available in the market are as follows.

Avik Cloud

Avik Cloud is a relatively new ETL platform designed with a cloud-first approach. This means it’s created specifically to be used in Azure, AWS, and Google Cloud and is available in all three market places. Similar to the cloud-based pricing structure of those platforms, Avik Cloud charges on a pay-for-what-you-use model.

Avik Cloud’s ETL process is built on Spark to achieve low latency continuous processing. Avik Cloud also features an easy-to-use visual pipeline builder.

Informatica PowerCenter

Informatica’s ETL solution is currently the most common data integration tool used for connecting and retrieving data from different datasources. Informatica has been in the industry a long time and is an established player in this space. They have data integration products for ETL, data masking, data quality, data replication, data management, and more.

Alooma

Alooma is a licensed ETL tool focused on data migration to data warehouses in the cloud. Alooma seemed to be a great solution for a lot of businesses with its automated data pipelines and its easy integrations for Amazon Redshift, Microsoft Azure, and Google BigQuery. However, after getting acquired by Google in 2019, Alooma has largely dropped support for non-Google data warehousing solutions. This may cause problems for companies that are relying on multiple cloud platforms.

Xplenty

Xplenty is a cloud-based ETL and ELT (extract, load, transform) tool. It uses a visual interface for building data pipelines and connects to more than 100 common datasources.

AWS Glue

AWS Glue is Amazon’s serverless ETL solution based on the AWS platform. If you are already entrenched in the AWS ecosystem, AWS Glue may be a good choice.

Python ETL

Python needs no introduction. Every year Python becomes ubiquitous in more-and-more fields ranging from astrophysics to search engine optimization. So it’s no surprise that Python has solutions for ETL.

For ETL, Python offers a handful of robust open-source libraries. Thanks to the ever-growing Python open-source community, these ETL libraries offer loads of features to develop a robust end-to-end data pipeline.

If you are all-in on Python, you can create complex ETL pipelines similar to what can be done with ETL tools. But be ready to burn some development hours. These libraries are feature-rich but are not ready out-of-the-box like some of the ETL platforms listed above.

Some of the popular python ETL libraries are:

Pandas

Luigi

Petl

Bonobo

Bubbles

These libraries have been compared in other posts on Python ETL options, so we won’t repeat that discussion here. Instead, we’ll focus on whether to use those or use the established ETL platforms.

Python ETL vs ETL tools

The strategy of ETL has to be carefully chosen when designing a data warehousing strategy. Once you have chosen an ETL process, you are somewhat locked in, since it would take a huge expenditure of development hours to migrate to another platform. This is especially true of enterprise data warehouses with many schemas and complex architectures.

So, let’s compare the usefulness of both custom Python ETL and ETL tools to help inform that choice.

Cost

The license cost of ETL tools (especially for big enterprise data warehouse) can be high–but this expense may be offset by how much time it saves your engineers to work on other things. Smaller companies or startups may not always be able to afford the licensing cost of ETL platforms. In such a scenario, creating a custom Python ETL may be a good option. But it’s also important to consider whether that cost savings is worth the delay it would cause in your product going to market. One other consideration for startups is that platforms with more flexible pricing like Avik Cloud keep the cost proportional to use–which would make it much more affordable for early-stage startups with limited ETL needs.

Size and Complexity of Data Warehouse

If it is a big data warehouse with complex schema, writing a custom Python ETL process from scratch might be challenging, especially when the schema changes more frequently. In this case, you should explore the options from various ETL tools that fit your requirements and budget.

Simplicity and Flexibility

If the data warehouse is small, you may not require all the features of enterprise ETL tools. It might be a good idea to write a custom light-weighted Python ETL process, as it will be both simple and give you better flexibility to customize it as per your needs.

Scalability

The initial size of the database might not be big. But if you anticipate growth in the near future, you should make a judgment about whether your custom Python ETL pipeline will also be able to scale with an increase in data throughput. If in doubt, you might want to look more closely at some of the ETL tools as they will scale more easily.

User Friendliness

To use Python for your ETL process, as you might guess, it requires expertise in Python. But ETL tools generally have user-friendly GUIs which make it easy to operate even for a non-technical person to work. So again, it is a choice to make as per the project requirements.

Value Addition and Support

ETL tools, especially the paid ones, give more value adds in terms of multiple features and compatibilities. They also offer customer support–which seems like an unimportant consideration until you need it. However, the open-source tools do have good documentation and plenty of online communities that can also offer support.

You will miss out on these things if you go with the custom Python ETL. It will be a challenging work to incorporate so many features of market ETL tools in the custom Python ETL process with the same robustness.

Surprise Third Option: Data Virtualization

One other option that has been gaining some traction is foregoing ETL altogether in favor of data virtualization. Data virtualization is a logical data layer that integrates data across disparate silos. Data virtualization manages the unified data, and delivers it to users with centralized security and governance.

Denodo offers data virtualization as a data management/engineering option.

Knowi takes it one step further and uses data virtualization to power their business intelligence platform. Data virtualization’s ability to connect to disparate data sources natively allows Knowi to connect to and visualize data in real-time without first having to use ETL to move thing to a data warehouse.

Conclusion

There is no clear winner when it comes to Python ETL vs ETL tools, they both have their own advantages and disadvantages. Finally, it all comes down to making a choice based on various parameters that we discussed above. But if you are strongly considering using Python for ETL, at least take a look at the platform options out there. And if you’re feeling especially bold, consider going down the rabbit hole and learning a bit about data virtualization. Likewise, if you’re thinking about using an ETL tool for a simple ETL use-case, it may be worth giving Python a closer look.

About the Author

Sean Knight works as Head of Growth Knowi, a SAAS analytics company. He has degrees in both physics and data science and has worked at particle accelerators, NASA JPL, a research nuclear reactor, and is now in the startup world. He is a data geek who enjoys contributing to Towards Data Science, The Startup, and Hackernoon.

Find him on Twitter and Linkedin.