Data Pipelines — Design Patterns for Reusability, Extensibility
By: Mukul Sood
Tags: Big Data Analytics, Data Engineering, Design Patterns, Data Pipeline

Data Pipelines — Design Patterns for Reusability, Extensibility

Designing extensible, modular, reusable Data Pipelines is a larger topic and very relevant in Data Engineering as the type of work involves dealing with constant change across different layers such as data sources, ingest, validation, processing, security, logging, monitoring. The changes happen at varying rates across these layers and they impact the data pipelines differently based on the level of abstraction and design of the pipeline. In this post, we will discuss Cross Cutting Concerns, specifically Logging and Monitoring and discuss some use cases and patterns that may provide the ground to build modularity and reusability into the pipelines

To get some context into layers of a Data Pipeline and to start to map the config, here’s a conceptual view:

Image: Author Data Pipeline High Level Architecture

This is a simplified view, as the layers could be represented in many different ways however in a distilled form the pipeline can be thought of as Ingest, Processing and Result layers. For each layer, we can think in terms of functions, as the image shows the functional blocks. The content of the blocks change depending on the layer requirements. This helps us think in terms of templates and config that could represent the Pipeline DAG. As an example, the yaml config below:

Image: Author Conceptual Yaml config for a DAG layer

The Ingest, Processing and Result layers could be mapped to different loggers and monitors based on requirements. For example, at the Ingest layer the file log could be S3 and event log could be custom and monitors could be Stackdriver and Redash. However, the Results layer could be mapped to event log DataDog and event monitor as DataDog.

If we take a generic pipeline, specifically looking at Logging and Monitoring, this representation would get implemented as Dag methods where loggers and monitors would be coupled with Dag code. This involves more coding, it is brittle, difficult to reuse and a violation of design principles SRP, DRY, Open Closed making the overall pipeline unstable and unreliable. If we extend this problem statement beyond Monitoring and Logging, we will see similar problems in different functional blocks — Data Quality/Validation, Security/Privacy etc.

When we see similarities in problems, that is an indicator to recognize the common themes. For example, here we have DAG specific and function specific (Logging, Monitoring) responsibilities. In addition, we want to remove the coupling between these and increase cohesion within these. This gives us enough context to start thinking in terms of Design Patterns. Refer Design Patterns: Elements of Reusable Object-Oriented Software by the “Gang of Four” (Gamma et al.). For our context, we will show how these patterns play a role in our design. Schematic below shows a high level view of Pipeline DAG with the patterns applicable to different layers:

Image: Author Patterns applicable to different DAG layers

The first set of patterns are Creational or Structural. They allow us to separate creation and structure of cross cutting concerns such as Logging, Monitoring from Dag specific areas. The Factory and AbstractFactory help in abstracting and decoupling the different Loggers and Monitors from the Dag code allowing the Dag codebase to evolve without dependencies on Logging, Monitoring code base

from loglib import LogConfigConst

from abc import ABC, abstractmethod

from boto3 import Session



class Logger(metaclass=ABC.ABCMeta):

@abstractmethod

def log(self,message):

pass



class S3Logger(Logger):

""" log to AWS S3 """

_initialized = False

_instance = None

_s3session = None

def __new__(cls, *args, **kwargs):

if not cls._instance:

cls._instance = super(S3Logger, cls).__new__(cls, *args, **kwargs)

return cls._instance



def __init__(self, *args, **kwargs):

if self._initialized:

return

super(S3Logger, self).__init__(*args, **kwargs)

#get the kwarg config that has details of S3 connection params -- Region, Public Key, Secret Key

logconfig=kwargs["logconfig"]

logConfigConst=LogConfigConst()

#establish s3 session, use boto3

self.__class__._s3session=Session(aws_access_key_id=logconfig[logConfigConst.AWS_ACCESS_KEY_ID],

aws_secret_access_key=logconfig[logConfigConst.AWS_SECRET_ACCESS_KEY],

region_name=logconfig[logConfigConst.AWS_REGION])

self.__class__._initialized = True



@classmethod

def log(cls, message):

#log to S3

return

Code sample showing an example of Logger and LogFactory classes:

class LogFactory:

_dictlogger={}

@classmethod

def create_logger(config):

logger=None

if config.name == LogConfigConst.filelog.names.S3:

logger=S3Logger(logconfig=config)

elif config.name == LogConfigConst.eventlog.names.DataDog:

logger=DataDogLogger(logconfig=config)

elif config.name == LogConfigConst.databaselog.names.PostGresDB:

logger=PostGresDBLogger(logconfig=config)

return logger

@classmethod

def get_logger(cls,config):

if cls._dictlogger[config.name] is None:

cls.dict_logger[config.name] = cls.create_logger(config)

return cls.dict_logger[config.name]

The second set of patterns are Behavioral. They allow us to specify behavior while maintaining SRP, DRY and SOLID principles. The Decorator pattern is widely used to modify or add behavior to existing functions. Logging and Monitoring are immediately applicable as we see below.

import functools,sys,traceback

from loglib import LoggingFacade



class LoggingDecorator:



def __init__(self,config):

self.logFacade=LoggingFacade(config_template=config)



def log_error(self,func=None):

def error_log(func):

@functools.wraps(func)

def wrapper(*args, **kwargs):

try:

# Execute the called function

return func(*args, **kwargs)

except Exception as e:

error_msg = 'And error has occurred in '.join(func.__name__)

self.logFacade.log_filestore(error_msg)

raise e

return wrapper

return error_log



@LoggingDecorator.log_error

def push_exception_to_xcom(kwargs):

"""To push exception occuring from dag task to xcom ; this will be used in alert/reporting"""

exc_type, exc_value, exc_traceback = sys.exc_info()

exception_details= ''.join(traceback.format_exception(etype=type(exc_type),value=exc_value, tb=exc_traceback))

kwargs['ti'].xcom_push(key='exception_details', value=exception_details)

Code sample showing an example of LoggingDecorator:

The Facade pattern is useful when a narrower or specific api is needed for client or consumer. For example, the broad set of api’s or methods exposed by the different Loggers and Monitors do not need to be exposed to the Dag layer. The Facade pattern helps define an access or interface to the Logging, Monitoring layer, as we see below

from loglib import LogConfigConst

from loglib.LoggerFactory import LogFactory



class LoggingFacade:

_file_logger, _event_logger, _database_logger = None

_initialized = False

def __new__(cls, *args, **kwargs):

if not cls._instance:

cls._instance = super(LoggingFacade, cls).__new__(cls, *args, **kwargs)

return cls._instance



def __init__(self, *args, **kwargs):

if self._initialized:

return

super(LoggingFacade, self).__init__(*args, **kwargs)

#get the kwarg config that has config for all the loggers

config=kwargs["config_template"]

logconfig=LogConfigConst()

for conf in config.loggers:

if conf.type == logconfig.FILELOG:

self.__class__._file_logger = LogFactory().get_logger(conf)

elif conf.type == logconfig.EVENTLOG:

self.__class__._event_logger = LogFactory().get_logger(conf)

elif conf.type == logconfig.DATABASELOG:

self.__class__._database_logger = LogFactory().get_logger(conf)

self.__class__._initialized = True



#filestore could be S3 or other providers

@classmethod

def log_filestore(cls,message):

cls._file_logger.log(message)



#event logger could be DataDog or other providers

@classmethod

def log_event(cls,message):

cls.event_logger.log(message)



@classmethod

def log_database(cls,message):

cls.database_logger.log(message)

Code sample showing an example of LoggingFacade:

When we combine these patterns, we realize the benefits of design principles as the separation of responsibilities allows the modularization of code base at multiple levels: Dag level, Cross Cutting Concerns level (separate packages for Monitoring, Logging), Cross Dag level (common templates could be abstracted for example domain specific). What this provides is building blocks for generalizing the Data pipelines to move away from writing custom code to more generic modules, templates, config. The different pieces follow their development cycles and deployment packages completely decoupled from Dag codebase. While adding these design principles does increase the abstraction and some level of complexity however that is a small price to pay for scaling up the pipeline development while maintaining quality and velocity

While the world of Data Engineering is continuously evolving and DAG Pipeline design and architecture has its own set of challenges, there are some key principles that can be applied to increase stability, reliability, maintainability of the pipelines. I hope you enjoyed reading this article and found some useful information to apply in your Data Engineering initiatives. I look forward to your comments and feedback.