Data Pipelines — Design Patterns for Reusability, Extensibility | by Mukul Sood | Towards Data Science

Designing extensible, modular, reusable Data Pipelines is a larger topic and very relevant in Data Engineering as the type of work involves dealing with constant change across different layers such as data sources, ingest, validation, processing, security, logging, monitoring. The changes happen at varying rates across these layers and they impact the data pipelines differently based on the level of abstraction and design of the pipeline. In this post, we will discuss Cross Cutting Concerns, specifically Logging and Monitoring and discuss some use cases and patterns that may provide the ground to build modularity and reusability into the pipelines
To get some context into layers of a Data Pipeline and to start to map the config, here’s a conceptual view:
This is a simplified view, as the layers could be represented in many different ways however in a distilled form the pipeline can be thought of as Ingest, Processing and Result layers. For each layer, we can think in terms of functions, as the image shows the functional blocks. The content of the blocks change depending on the layer requirements. This helps us think in terms of templates and config that could represent the Pipeline DAG. As an example, the yaml config below:
The Ingest, Processing and Result layers could be mapped to different loggers and monitors based on requirements. For example, at the Ingest layer the file log could be S3 and event log could be custom and monitors could be Stackdriver and Redash. However, the Results layer could be mapped to event log DataDog and event monitor as DataDog.
If we take a generic pipeline, specifically looking at Logging and Monitoring, this representation would get implemented as Dag methods where loggers and monitors would be coupled with Dag code. This involves more coding, it is brittle, difficult to reuse and a violation of design principles SRP, DRY, Open Closed making the overall pipeline unstable and unreliable. If we extend this problem statement beyond Monitoring and Logging, we will see similar problems in different functional blocks — Data Quality/Validation, Security/Privacy etc.
When we see similarities in problems, that is an indicator to recognize the common themes. For example, here we have DAG specific and function specific (Logging, Monitoring) responsibilities. In addition, we want to remove the coupling between these and increase cohesion within these. This gives us enough context to start thinking in terms of Design Patterns. Refer Design Patterns: Elements of Reusable Object-Oriented Software by the “Gang of Four” (Gamma et al.). For our context, we will show how these patterns play a role in our design. Schematic below shows a high level view of Pipeline DAG with the patterns applicable to different layers:
The first set of patterns are Creational or Structural. They allow us to separate creation and structure of cross cutting concerns such as Logging, Monitoring from Dag specific areas. The Factory and AbstractFactory help in abstracting and decoupling the different Loggers and Monitors from the Dag code allowing the Dag codebase to evolve without dependencies on Logging, Monitoring code base
from loglib import LogConfigConstfrom abc import ABC, abstractmethodfrom boto3 import Sessionclass Logger(metaclass=ABC.ABCMeta):    @abstractmethod    def log(self,message):        passclass S3Logger(Logger):    """ log to AWS S3 """    _initialized = False    _instance = None    _s3session = None    def __new__(cls, *args, **kwargs):        if not cls._instance:            cls._instance = super(S3Logger, cls).__new__(cls, *args, **kwargs)        return cls._instance    def __init__(self, *args, **kwargs):        if self._initialized:            return        super(S3Logger, self).__init__(*args, **kwargs)        #get the kwarg config that has details of S3 connection params -- Region, Public Key, Secret Key        logconfig=kwargs["logconfig"]        logConfigConst=LogConfigConst()        #establish s3 session, use boto3        self.__class__._s3session=Session(aws_access_key_id=logconfig[logConfigConst.AWS_ACCESS_KEY_ID],                                          aws_secret_access_key=logconfig[logConfigConst.AWS_SECRET_ACCESS_KEY],                                          region_name=logconfig[logConfigConst.AWS_REGION])        self.__class__._initialized = True    @classmethod    def log(cls, message):        #log to S3        return
Code sample showing an example of Logger and LogFactory classes:
class LogFactory:    _dictlogger={}    @classmethod    def create_logger(config):        logger=None        if config.name == LogConfigConst.filelog.names.S3:            logger=S3Logger(logconfig=config)        elif config.name == LogConfigConst.eventlog.names.DataDog:            logger=DataDogLogger(logconfig=config)        elif config.name == LogConfigConst.databaselog.names.PostGresDB:            logger=PostGresDBLogger(logconfig=config)        return logger    @classmethod    def get_logger(cls,config):        if cls._dictlogger[config.name] is None:            cls.dict_logger[config.name] = cls.create_logger(config)        return cls.dict_logger[config.name]
The second set of patterns are Behavioral. They allow us to specify behavior while maintaining SRP, DRY and SOLID principles. The Decorator pattern is widely used to modify or add behavior to existing functions. Logging and Monitoring are immediately applicable as we see below.
import functools,sys,tracebackfrom loglib import LoggingFacadeclass LoggingDecorator:    def __init__(self,config):        self.logFacade=LoggingFacade(config_template=config)    def log_error(self,func=None):        def error_log(func):            @functools.wraps(func)            def wrapper(*args, **kwargs):                try:                    # Execute the called function                    return func(*args, **kwargs)                except Exception as e:                    error_msg = 'And error has occurred in '.join(func.__name__)                    self.logFacade.log_filestore(error_msg)                    raise e            return wrapper        return error_log@LoggingDecorator.log_errordef push_exception_to_xcom(kwargs):    """To push exception occuring from dag task to xcom ; this will be used in alert/reporting"""    exc_type, exc_value, exc_traceback = sys.exc_info()    exception_details= ''.join(traceback.format_exception(etype=type(exc_type),value=exc_value, tb=exc_traceback))    kwargs['ti'].xcom_push(key='exception_details', value=exception_details)
Code sample showing an example of LoggingDecorator:
The Facade pattern is useful when a narrower or specific api is needed for client or consumer. For example, the broad set of api’s or methods exposed by the different Loggers and Monitors do not need to be exposed to the Dag layer. The Facade pattern helps define an access or interface to the Logging, Monitoring layer, as we see below
from loglib import LogConfigConstfrom loglib.LoggerFactory import LogFactoryclass LoggingFacade:    _file_logger, _event_logger, _database_logger = None    _initialized = False    def __new__(cls, *args, **kwargs):        if not cls._instance:            cls._instance = super(LoggingFacade, cls).__new__(cls, *args, **kwargs)        return cls._instance    def __init__(self, *args, **kwargs):        if self._initialized:            return        super(LoggingFacade, self).__init__(*args, **kwargs)        #get the kwarg config that has config for all the loggers        config=kwargs["config_template"]        logconfig=LogConfigConst()        for conf in config.loggers:            if conf.type == logconfig.FILELOG:                self.__class__._file_logger = LogFactory().get_logger(conf)            elif conf.type == logconfig.EVENTLOG:                self.__class__._event_logger = LogFactory().get_logger(conf)            elif conf.type == logconfig.DATABASELOG:                self.__class__._database_logger = LogFactory().get_logger(conf)        self.__class__._initialized = True    #filestore could be S3 or other providers    @classmethod    def log_filestore(cls,message):        cls._file_logger.log(message)    #event logger could be DataDog or other providers    @classmethod    def log_event(cls,message):        cls.event_logger.log(message)    @classmethod    def log_database(cls,message):        cls.database_logger.log(message)
Code sample showing an example of LoggingFacade:
When we combine these patterns, we realize the benefits of design principles as the separation of responsibilities allows the modularization of code base at multiple levels: Dag level, Cross Cutting Concerns level (separate packages for Monitoring, Logging), Cross Dag level (common templates could be abstracted for example domain specific). What this provides is building blocks for generalizing the Data pipelines to move away from writing custom code to more generic modules, templates, config. The different pieces follow their development cycles and deployment packages completely decoupled from Dag codebase. While adding these design principles does increase the abstraction and some level of complexity however that is a small price to pay for scaling up the pipeline development while maintaining quality and velocity
While the world of Data Engineering is continuously evolving and DAG Pipeline design and architecture has its own set of challenges, there are some key principles that can be applied to increase stability, reliability, maintainability of the pipelines. I hope you enjoyed reading this article and found some useful information to apply in your Data Engineering initiatives. I look forward to your comments and feedback.