Package: pipeline-penguin
DescriptionPipeline Penguin is a versatile python library for data quality.DocumentationReferenceGetting StartedHow to installYou can use PyPI Test to install the early develoment build by executing this command:pip install pipeline-penguinCore ConceptsBefore you start executing validations on you data, firstly you need to understand a few concepts:Data NodeAny database your pipeline needs to process. It has a Node Type, that identifies what is the source of that data, like BigQuery.Data PremiseAny premise on a Data Node, for example, the column "name" on the database "employers" must not have a number on it. Data Premises also have a type called Premise Type. If an Premise Type is SQL, you cannot execute a validation on a Data Node with a Node Type that does not have a SQL engine to process that query.ConnectorThe way you access an Data Node to check a Data Premise.Premise OutputIs the result of a Data Premise validation on a Data Node.Implementing a testImporting and instantiating PPfrompipeline_penguinimportPipelinePenguinpp=PipelinePenguin()Defining the default connectorbq_connector=ConnectorSQLBigQuery('/config/service_account.json')pp.connectors.define_default(bq_connector)Creating a Data Nodenode=pp.nodes.create_node('Node Name',DataNodeBigQuery,project_id='example',dataset_id='example',table_id='example')pp.nodes.list_nodes()Creating a Data Premisenode.insert_premise('Premise Name',DataPremiseSQLCheckIsNull,"Column Name")Executing a validationpp.nodes.run_premises()Checking Logslog_formatter=OutputFormatterLog()outputs.format_outputs(log_formatter)Implementing a custom Data PremiseImplementing a new DataPremise classfrompipeline_penguin.core.data_premise.sqlimportDataPremiseSQLfrompipeline_penguin.core.premise_output.premise_outputimportPremiseOutputfrompipeline_penguin.data_node.sql.bigqueryimportDataNodeBigQueryclassCheckBanana(DataPremiseSQL):def__init__(self,name:str,data_node:DataNodeBigQuery,column:str):self.query_template="SELECT * result FROM `{project}.{dataset}.{table}` WHERE LOWER({column}) = 'banana')"super().__init__(name,data_node,column)defquery_args(self):"""Method for returning the arguments to be passed on the query template of thisvalidation.Returns:A `dictionary` with the query parameters."""return{"project":self.data_node.project_id,"dataset":self.data_node.dataset_id,"table":self.data_node.table_id,"column":self.column}defvalidate(self)->PremiseOutput:"""Method for executing the validation over the DataNode.Returns:PremiseOutput: Object storeing the results for this validation."""query=self.query_template.format(**self.query_args())connector=self.data_node.get_connector(self.type)data_frame=connector.run(query)failed_count=len(data_frame["result"])passed=failed_count==0output=PremiseOutput(self,self.data_node,self.column,passed,failed_count,data_frame)returnoutputTesting a DataNode with a custom Data Premisefrompipeline_penguinimportPipelinePenguinimportCheckBananapp=PipelinePenguin()bq_connector=ConnectorSQLBigQuery('/config/service_account.json')pp.connectors.define_default(bq_connector)node=pp.nodes.create_node('Node Name',DataNodeBigQuery,project_id='example',dataset_id='example',table_id='example')node.insert_premise('Check Null',DataPremiseSQLCheckIsNull,"Column Name")node.insert_premise('Check Contains Banana',CheckBanana,"Column Name")log_formatter=OutputFormatterLog()outputs.format_outputs(log_formatter)CollaborateInstallationpipenv installTestspipenv install --devRunning testspipenv run testStyle formatRunning formatpipenv run formatChecking formatpipenv run format-checkDeveloping documentationRunning local buildpipenv run docsBulding docspipenv run pipenv run build-docsSupport or ContactHaving trouble with PP? Check out ourdocumentationor contact support and we’ll help you sort it out.DP6 Koopa-Troopa Team
e-mail:koopas@dp6.com.br