Package: pipelinewise-transform-field
pipelinewise-transform-fieldTransformation component betweenSingertaps and targets.This is aPipelineWisecompatible component.How to use itThe recommended method of running this component is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files, and most of things are automated.
Please check the related documentation atTransformationsIf you want to run thisSingercompatible component independently please read further.InstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-transform-fieldorpython3-mvenvvenv.venv/bin/activatepipinstall--upgradepipsetuptoolspipinstall.To validate transformationstransform-field --validate --config [config.json] --catalog [catalog.json]To runPut it between a tap and a target with simple unix pipes:some-singer-tap | transform-field --config [config.json] | some-singer-targetIt's reading incoming messages from STDIN and usingconfig.jsonto transform incoming RECORD messages.Note: To avoid version conflicts runtap,transformandtargetsin separate virtual environments.Transformation typesThe following are the transformation types supported bypipelinewise-transform-field:SET-NULL: Transforms any input to NULLHASH: Transforms string input to hashHASH-SKIP-FIRST-n: Transforms string input to hash skipping first n characters, e.g. HASH-SKIP-FIRST-2MASK-DATE: Replaces the months and day parts of date columns to be always 1st of JanMASK-NUMBER: Transforms any numeric value to zeroMASK-HIDDEN: Transforms any string to 'hidden'MASK-STRING-SKIP-ENDS-n: Transforms string input to masked version skipping first and last n characters, e.g. MASK-STRING-SKIP-ENDS-3PS: 1 =< n =< 9Conditional transformationsIt is possible to transform a record's property based on some given condition(s), the transformation will only take place when all conditions are met.A condition is a combination of:column [required]: the field to look up tooperation [required]: the comparison type to use, the supported ones areequalsandregex_match.value [required]: the column value to look for in records.An equality condition on a column{"column":"<some column name>","equals":<someimportantvalue>}A regex condition on a column{"column":"<some column name>","regex_match":"<some regex pattern>"}A condition on a property within a JSON-type column{"column":"<some column name>","field_path":"<xpath to property within 'column' object>","equals":<someimportantvalue>}ConfigurationYou need to define which columns have to be transformed by which method and in which condition the transformation needs to be applied.Basic transformationA basic transformation is where a field in all a stream records will be transformed can be achieved with:{"tap_stream_name":"<stream ID>","field_id":"<Name of the field to transform in the record>","type":"<Transformation type>"}Transformation within JSONIn order to transform property(ies) within a JSON type field, you can make use offield_pathsproperty:{"tap_stream_name":"<stream ID>","field_id":"<Name of the field to transform in the record>","field_paths":["xpath to property 1","xpath to property 2"],"type":"<Transformation type>"}Conditional TransformationTo apply transformation conditionally, you can make use of the propertywhenwhich can have one or many conditions:{"tap_stream_name":"<stream ID>","field_id":"<Name of the field to transform in the record>","type":"<Transformation type>","when":[{"column":"string_col_1","equals":"some value"},{"column":"string_col_2","regex_match":".*PII.*"},{"column":"numeric_col_1","equals":33},{"column":"json_column","field_path":"metadata/comment","regex_match":"sensitive"}]}Sample configconfig.json(Tip: PipelineWise generating this for you from a more readable YAML format)To check code style:Install python dependencies in a virtual envpython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .[test]Run pylintpylinttransform_fieldTo run tests:Install python dependencies in a virtual env and run unit and integration testspython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .[test]Run tests:Unit testspytest -v tests/unitIntegration testspytest -v tests/integrationAll testspytest -v testsLicenseApache License Version 2.0SeeLICENSEto see the full text.