You may face many critical challenges for continuous integration/continuous delivery while building a data pipeline. However, some challenges are critical and worth mentioning such as   Data exploration: you may find it difficult to explore data while building a data pipeline when multiple users collaborate on the same project.  Iterating unit tests: Changing code and writing unit cases iteratively pose a challenge as the process becomes cumbersome.  Continuous build and integration: New code is continuously merged. The build server needs to pull the latest changes and perform unit testing for multiple components and publish the artifacts on an ongoing basis.   Staging data pipelines: Pushing data pipelines to the staging environment poses a challenge as the pipeline is tested against a much larger data set (which is similar to production data) for data quality and performance.  Pushing data pipelines to production: Pushing data pipelines from the staging environment to the production environment may often pose challenges.