Package: datayoga-core
DataYoga CoreIntroductiondatayoga-coreis the transformation engine used inDataYoga, a framework for building and generating data pipelines.Installationpipinstalldatayoga-coreQuick StartThis demonstrates how to transform data using a DataYoga job.Create a JobUse thisexample.yaml:steps:-uses:add_fieldwith:fields:-field:full_namelanguage:jmespathexpression:concat([fname, ' ' , lname])-field:countrylanguage:sqlexpression:country_code || ' - ' || UPPER(country_name)-uses:rename_fieldwith:fields:-from_field:fnameto_field:first_name-from_field:lnameto_field:last_name-uses:remove_fieldwith:fields:-field:credit_card-field:country_name-field:country_code-uses:mapwith:expression:{first_name:first_name,last_name:last_name,greeting:"'Hello'||CASEWHENgender='F'THEN'Ms.'WHENgender='M'THEN'Mr.'ELSE'N/A'END||''||full_name",country:country,full_name:full_name}language:sqlTransform Data Usingdatayoga-coreUse this code snippet to transform a data record using the job definedabove. The transform method returns a tuple of processed, filtered, and rejected records:importdatayoga_coreasdyfromdatayoga_core.jobimportJobfromdatayoga_core.resultimportResult,Statusfromdatayoga_core.utilsimportread_yamljob_settings=read_yaml("example.yaml")job=dy.compile(job_settings)assertjob.transform([{"fname":"jane","lname":"smith","country_code":1,"country_name":"usa","credit_card":"1234-5678-0000-9999","gender":"F"}]).processed==[Result(status=Status.SUCCESS,payload={"first_name":"jane","last_name":"smith","country":"1 - USA","full_name":"jane smith","greeting":"Hello Ms. jane smith"})]The job can also be provided as a parsed json inline:importdatayoga_coreasdyfromdatayoga_core.jobimportJobfromdatayoga_core.resultimportResult,Statusimportyamlimporttextwrapjob_settings=textwrap.dedent("""steps:- uses: add_fieldwith:fields:- field: full_namelanguage: jmespathexpression: concat([fname, ' ' , lname])- field: countrylanguage: sqlexpression: country_code || ' - ' || UPPER(country_name)- uses: rename_fieldwith:fields:- from_field: fnameto_field: first_name- from_field: lnameto_field: last_name- uses: remove_fieldwith:fields:- field: credit_card- field: country_name- field: country_code- uses: mapwith:expression:{first_name: first_name,last_name: last_name,greeting: "'Hello ' || CASE WHEN gender = 'F' THEN 'Ms.' WHEN gender = 'M' THEN 'Mr.' ELSE 'N/A' END || ' ' || full_name",country: country,full_name: full_name}language: sql""")job=dy.compile(yaml.safe_load(job_settings))assertjob.transform([{"fname":"jane","lname":"smith","country_code":1,"country_name":"usa","credit_card":"1234-5678-0000-9999","gender":"F"}]).processed==[Result(status=Status.SUCCESS,payload={"first_name":"jane","last_name":"smith","country":"1 - USA","full_name":"jane smith","greeting":"Hello Ms. jane smith"})]As can be seen, the record has been transformed based on the job:fnamefield renamed tofirst_name.lnamefield renamed tolast_name.countryfield added based on an SQL expression.full_namefield added based on aJMESPathexpression.greetingfield added based on an SQL expression.ExamplesAdd a new fieldcountryout of an SQL expression that concatenatescountry_codeandcountry_namefields after upper case the later:uses:add_fieldwith:field:countrylanguage:sqlexpression:country_code || ' - ' || UPPER(country_name)Renamefnamefield tofirst_nameandlnamefield tolast_name:uses:rename_fieldwith:fields:-from_field:fnameto_field:first_name-from_field:lnameto_field:last_nameRemovecredit_cardfield:uses:remove_fieldwith:field:credit_cardFor a full list of supported block typessee reference.Expression LanguageDataYoga supports both SQL andJMESPathexpressions. JMESPath are especially useful to handle nested JSON data, while SQL is more suited to flat row-like structures.For more information about custom functions and supported expression language syntaxsee reference.