Project: Beam (App Definition and Development / Streaming & Messaging)
File: bqml.md

title: "BigQuery ML integration" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> With the samples on this page we will demonstrate how to integrate models exported from into your Apache Beam pipeline using . Roughly, the sections below will go through the following steps in more detail: Create and train your BigQuery ML model Export your BigQuery ML model Create a transform that uses the brand-new BigQuery ML model {{< language-switcher java py >}} To be able to incorporate your BQML model into an Apache Beam pipeline using tfx_bsl, it has to be in the format. An overview that maps different model types to their export model format for BQML can be found . For the sake of simplicity, we'll be training a (simplified version of the) logistic regression model in the , using the publicly available Google Analytics sample dataset (which is a - alternatively, you might encounter ). An overview of all models you can create using BQML can be found . After creating a BigQuery dataset, you continue to create the model, which is fully defined in SQL: ``` CREATE MODEL IF NOT EXISTS `bqmltutorial.samplemodel` OPTIONS(modeltype='logisticreg', inputlabelcols=["label"]) AS SELECT IF(totals.transactions IS NULL, 0, 1) AS label, IFNULL(geoNetwork.country, "") AS country FROM

WHERE TABLESUFFIX BETWEEN '20160801' AND '20170630' ``` The model will predict if a purchase will be made given the country of the visitor on data gathered between 2016-08-01 and 2017-06-30. In order to incorporate your model in an Apache Beam pipeline, you will need to export it. Prerequisites to do so are and to store your exported model. Export the model using the following command: ``` bq extract -m bqmltutorial.samplemodel gs://some/gcs/path ``` In this section we will construct an Apache Beam pipeline that will use the BigQuery ML model we just created and exported. The model can be served using Google Cloud AI Platform Prediction - for this please refer to the . In this case, we'll be illustrating how to use the tfx_bsl library to do local predictions (on your Apache Beam workers). First, the model needs to be downloaded to a local directory where you will be developing the rest of your pipeline (e.g. to `servingdir/samplemodel/1`). Then, you can start developing your pipeline like you would normally do. We will be using the `RunInference` PTransform from the library, and we will point it to our local directory where the model is stored (see the `modelpath` variable in the code example). The transform takes elements of the type `tf.train.Example` as inputs and outputs elements of the type . Depending on the signature of your model, you can extract values from the output; in our case we extract `labelprobs`, `labelvalues` and the `predictedlabel` as per the in the `extract_prediction` function. {{< highlight py >}} import apache_beam import tensorflow as tf from google.protobuf import text_format from tensorflow.python.framework import tensor_util from tfxbsl.beam import runinference from tfx_bsl.public.beam import RunInference from tfxbsl.public.proto import modelspec_pb2 inputs = tf.train.Example(features=tf.train.Features( feature={ 'os': tf.train.Feature(bytes_list=tf.train.BytesList(b"Microsoft")) }) ) modelpath = "servingdir/sample_model/1" def extract_prediction(response): yield response.predictlog.response.outputs['labelvalues'].string_val, tensorutil.MakeNdarray(response.predictlog.response.outputs['label_probs']), response.predictlog.response.outputs['predictedlabel'].string_val with beam.Pipeline() as p: res = ( p | beam.Create([inputs]) | RunInference( modelspecpb2.InferenceSpecType( savedmodelspec=modelspecpb2.SavedModelSpec( modelpath=modelpath, signaturename=['servingdefault']))) | beam.ParDo(extract_prediction) {{< /highlight >}} {{< highlight java >}} Implemented in Python. {{< /highlight >}}