Project: Beam (App Definition and Development / Streaming & Messaging)
File: mltransform.md

title: "MLTransform" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< localstorage language language-py >}} <table> <tr> <td> <a> {{< button-pydoc path="apache_beam.ml.transforms" class="MLTransform" >}} </a> </td> </tr> </table> Use `MLTransform` to apply common machine learning (ML) processing tasks on keyed data. Apache Beam provides ML data processing transformations that you can use with `MLTransform`. For the full list of available data processing transformations, see the in GitHub. To define a data processing transformation by using `MLTransform`, create instances of data processing transforms with `columns` as input parameters. The data in the specified `columns` is transformed and outputted to the `beam.Row` object. The following example demonstrates how to use `MLTransform` to normalize your data between 0 and 1 by using the minimum and maximum values from your entire dataset. `MLTransform` uses the `ScaleTo01` transformation. ``` scaletozscoretransform = ScaleToZScore(columns=['x', 'y']) with beam.Pipeline() as p: (data | MLTransform(writeartifactlocation=artifactlocation).withtransform(scaletozscoretransform)) ``` In this example, `MLTransform` receives a value for `writeartifactlocation`. `MLTransform` then uses this location value to write artifacts generated by the transform. To pass the data processing transform, you can use either the `with_transform` method of `MLTransform` or a list. ``` MLTransform(transforms=transforms, writeartifactlocation=writeartifactlocation) ``` The transforms passed to `MLTransform` are applied sequentially on the dataset. `MLTransform` expects a dictionary and returns a transformed row object with NumPy arrays. The following examples demonstrate how to to create pipelines that use `MLTransform` to preprocess data. `MLTransform` can do a full pass on the dataset, which is useful when you need to transform a single element only after analyzing the entire dataset. The first two examples require a full pass over the dataset to complete the data transformation. For the `ComputeAndApplyVocabulary` transform, the transform needs access to all of the unique words in the dataset. For the `ScaleTo01` transform, the transform needs to know the minimum and maximum values in the dataset. This example creates a pipeline that uses `MLTransform` to scale data between 0 and 1. The example takes a list of integers and converts them into the range of 0 to 1 using the transform `ScaleTo01`. {{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/mltransform.py" class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/mltransform.py" mltransformscaleto01 >}} {{</ highlight >}} {{< paragraph class="notebook-skip" >}} Output: {{< /paragraph >}} {{< highlight class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/mltransformtest.py" mltransformscaleto0_1 >}} {{< /highlight >}} This example creates a pipeline that use `MLTransform` to compute vocabulary on the entire dataset and assign indices to each unique vocabulary item. It takes a list of strings, computes vocabulary over the entire dataset, and then applies a unique index to each vocabulary item. {{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/mltransform.py" class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/mltransform.py" mltransformcomputeandapplyvocabulary >}} {{</ highlight >}} {{< paragraph class="notebook-skip" >}} Output: {{< /paragraph >}} {{< highlight class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/mltransformtest.py" mltransformcomputeandapply_vocab >}} {{< /highlight >}} This example creates a pipeline that uses `MLTransform` to compute vocabulary on the entire dataset and assign indices to each unique vocabulary item. This pipeline takes a single element as input instead of a list of elements. {{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/mltransform.py" class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/mltransform.py" mltransformcomputeandapplyvocabularywithscalar >}} {{</ highlight >}} {{< paragraph class="notebook-skip" >}} Output: {{< /paragraph >}} {{< highlight class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/mltransformtest.py" mltransformcomputeandapplyvocabularywith_scalar >}} {{< /highlight >}}