text: {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Perform A/B Test using REST Endpoints"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can test and deploy new models behind a single SageMaker Endpoint with a concept called “production variants.” These variants can differ by hardware (CPU/GPU), by data (comedy/drama movies), or by region (US West or Germany North). You can shift traffic between the models in your endpoint for canary rollouts and blue/green deployments. You can split traffic for A/B tests. And you can configure your endpoint to automatically scale your endpoints out or in based on a given metric like requests per second. As more requests come in, SageMaker will automatically scale the model prediction API to meet the demand."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<img src=\"img/model_ab.png\" width=\"80%\" align=\"left\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can use traffic splitting to direct subsets of users to different model variants for the purpose of comparing and testing different models in live production. The goal is to see which variants perform better. Often, these tests need to run for a long period of time (weeks) to be statistically significant. The figure shows 2 different recommendation models deployed using a random 50-50 traffic split between the 2 variants."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import boto3\n",
      "import sagemaker\n",
      "import pandas as pd\n",
      "\n",
      "sess   = sagemaker.Session()\n",
      "bucket = sess.default_bucket()\n",
      "role = sagemaker.get_execution_role()\n",
      "region = boto3.Session().region_name\n",
      "\n",
      "sm = boto3.Session().client(service_name='sagemaker', region_name=region)\n",
      "cw = boto3.Session().client(service_name='cloudwatch', region_name=region)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Clean Up Previous Endpoints to Save Resources"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%store -r autopilot_endpoint_name"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "try: \n",
      "    autopilot_endpoint_name\n",
      "    sm.delete_endpoint(\n",
      "        EndpointName=autopilot_endpoint_name\n",
      "    )\n",
      "    print('Autopilot Endpoint has been deleted to save resources.  This is good.')    \n",
      "except:\n",
      "    print('Endpoints are cleaned up.  This is good.  Keep moving forward!')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%store -r training_job_name"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "print(training_job_name)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Copy the Model to the Notebook"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {
      "scrolled": true
     },
     "outputs": [],
     "source": [
      "!aws s3 cp s3://$bucket/$training_job_name/output/model.tar.gz ./model.tar.gz"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "!mkdir -p ./model/\n",
      "!tar -xvzf ./model.tar.gz -C ./model/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Show the Prediction Signature"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "!saved_model_cli show --all --dir ./model/tensorflow/saved_model/0/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Show `inference.py`"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "!pygmentize ./model/code/inference.py"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Create Variant A Model From the Training Job in a Previous Section\n",
      "\n",
      "Notes:\n",
      "* `primary_container_image` is required because the inference and training images are different.\n",
      "* By default, the training image will be used, so we need to override it.  \n",
      "* See https://github.com/aws/sagemaker-python-sdk/issues/1379\n",
      "* If you are not using a US-based region, you may need to adapt the container image to your current region using the following table:\n",
      "\n",
      "https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import time\n",
      "timestamp = '{}'.format(int(time.time()))\n",
      "\n",
      "model_a_name = '{}-{}-{}'.format(training_job_name, 'varianta', timestamp)\n",
      "\n",
      "sess.create_model_from_job(name=model_a_name,\n",
      "                           training_job_name=training_job_name,\n",
      "                           role=role,\n",
      "                           image_uri='763104351884.dkr.ecr.{}.amazonaws.com/tensorflow-inference:2.3.0-cpu-py37-ubuntu18.04'.format(region))"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Create Variant B Model From the Training Job in a Previous Section\n",
      "Notes:\n",
      "* `primary_container_image` is required because the inference and training images are different.\n",
      "* By default, the training image will be used, so we need to override it.  See https://github.com/aws/sagemaker-python-sdk/issues/1379\n",
      "* If you are not using a US-based region, you may need to adapt the container image to your current region using the following table:\n",
      "\n",
      "https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "model_b_name = '{}-{}-{}'.format(training_job_name, 'variantb', timestamp)\n",
      "\n",
      "sess.create_model_from_job(name=model_b_name,\n",
      "                           training_job_name=training_job_name,\n",
      "                           role=role,\n",
      "                           image_uri='763104351884.dkr.ecr.{}.amazonaws.com/tensorflow-inference:2.3.0-cpu-py37-ubuntu18.04'.format(region))"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Canary Rollouts and A/B Testing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Canary rollouts are used to release new models safely to only a small subset of users such as 5%. They are useful if you want to test in live production without affecting the entire user base. Since the majority of traffic goes to the existing model, the cluster size of the canary model can be relatively small since it’s only receiving 5% traffic."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Instead of `deploy()`, we can create an `Endpoint Configuration` with multiple variants for canary rollouts and A/B testing."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from sagemaker.session import production_variant\n",
      "\n",
      "timestamp = '{}'.format(int(time.time()))\n",
      "\n",
      "endpoint_config_name = '{}-{}-{}'.format(training_job_name, 'abtest', timestamp)\n",
      "\n",
      "variantA = production_variant(model_name=model_a_name,\n",
      "                              instance_type='ml.m5.4xlarge',\n",
      "                              initial_instance_count=1,\n",
      "                              variant_name='VariantA',\n",
      "                              initial_weight=50)\n",
      "\n",
      "variantB = production_variant(model_name=model_b_name,\n",
      "                              instance_type='ml.m5.4xlarge',\n",
      "                              initial_instance_count=1,\n",
      "                              variant_name='VariantB',\n",
      "                              initial_weight=50)\n",
      "\n",
      "endpoint_config = sm.create_endpoint_config(\n",
      "    EndpointConfigName=endpoint_config_name,\n",
      "    ProductionVariants=[variantA, variantB]\n",
      ")"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from IPython.core.display import display, HTML\n",
      "\n",
      "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpointConfig/{}\">REST Endpoint Configuration</a></b>'.format(region, endpoint_config_name)))\n"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "model_ab_endpoint_name = '{}-{}-{}'.format(training_job_name, 'abtest', timestamp)\n",
      "\n",
      "endpoint_response = sm.create_endpoint(\n",
      "    EndpointName=model_ab_endpoint_name,\n",
      "    EndpointConfigName=endpoint_config_name)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Store Endpoint Name for Next Notebook(s)"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%store model_ab_endpoint_name"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Track the Deployment Within our Experiment"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%store -r experiment_name"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "print(experiment_name)"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%store -r trial_name"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "print(trial_name)"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from smexperiments.trial import Trial\n",
      "\n",
      "timestamp = '{}'.format(int(time.time()))\n",
      "\n",
      "trial = Trial.load(trial_name=trial_name)\n",
      "print(trial)"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from smexperiments.tracker import Tracker\n",
      "\n",
      "tracker_deploy = Tracker.create(display_name='deploy', \n",
      "                                sagemaker_boto_client=sm)\n",
      "\n",
      "deploy_trial_component_name = tracker_deploy.trial_component.trial_component_name\n",
      "print('Deploy trial component name {}'.format(deploy_trial_component_name))"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Attach the `deploy` Trial Component and Tracker as a Component to the Trial"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "trial.add_trial_component(tracker_deploy.trial_component)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Track the Endpoint Name"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "tracker_deploy.log_parameters({\n",
      "    'endpoint_name': model_ab_endpoint_name,\n",
      "})\n",
      "\n",
      "# must save after logging\n",
      "tracker_deploy.trial_component.save()"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from sagemaker.analytics import ExperimentAnalytics\n",
      "\n",
      "lineage_table = ExperimentAnalytics(\n",
      "    sagemaker_session=sess,\n",
      "    experiment_name=experiment_name,\n",
      "    metric_names=['validation:accuracy'],\n",
      "    sort_by=\"CreationTime\",\n",
      "    sort_order=\"Ascending\",\n",
      ")\n",
      "\n",
      "lineage_df = lineage_table.dataframe()\n",
      "lineage_df.shape"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "lineage_df"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from IPython.core.display import display, HTML\n",
      "\n",
      "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">REST Endpoint</a></b>'.format(region, model_ab_endpoint_name)))\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# _Wait Until the Endpoint is Deployed_"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "waiter = sm.get_waiter('endpoint_in_service')\n",
      "waiter.wait(EndpointName=model_ab_endpoint_name)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# _Wait Until the ^^ Endpoint ^^ is Deployed_"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Simulate a Prediction from an Application"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from sagemaker.tensorflow.model import TensorFlowPredictor\n",
      "\n",
      "predictor = TensorFlowPredictor(endpoint_name=model_ab_endpoint_name,\n",
      "                                sagemaker_session=sess,\n",
      "                                model_name='saved_model',\n",
      "                                model_version=0)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Waiting for the Endpoint to be ready to Serve Predictions"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import time\n",
      "time.sleep(30)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Predict the `star_rating` with `review_body` Samples from our TSV's"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import csv\n",
      "\n",
      "df_reviews = pd.read_csv('./data/amazon_reviews_us_Digital_Software_v1_00.tsv.gz', \n",
      "                                delimiter='\\t', \n",
      "                                quoting=csv.QUOTE_NONE,\n",
      "                                compression='gzip')\n",
      "df_sample_reviews = df_reviews[['review_body', 'star_rating']].sample(n=50)\n",
      "df_sample_reviews = df_sample_reviews.reset_index()\n",
      "df_sample_reviews.shape"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import pandas as pd\n",
      "\n",
      "def predict(review_body):\n",
      "    return predictor.predict([review_body])[0]\n",
      "\n",
      "df_sample_reviews['predicted_class'] = df_sample_reviews['review_body'].map(predict)\n",
      "df_sample_reviews.head(5)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Predict the `star_rating` with Ad Hoc `review_body` Samples"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "reviews = [\"This is great!\", \n",
      "           \"This is not good.\"]\n",
      "\n",
      "predicted_classes = predictor.predict(reviews)\n",
      "\n",
      "for predicted_class, review in zip(predicted_classes, reviews):\n",
      "    print('[Predicted Star Rating: {}]'.format(predicted_class), review)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Review the REST Endpoint Performance Metrics in CloudWatch"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from IPython.core.display import display, HTML\n",
      "\n",
      "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">REST Endpoint Performance Metrics</a></b>'.format(region, model_ab_endpoint_name)))\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Review the REST Endpoint Performance Metrics in a Dataframe\n",
      "\n",
      "Amazon SageMaker emits metrics such as Latency and Invocations (full list of metrics [here](https://alpha-docs-aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html)) for each variant in Amazon CloudWatch. Let’s query CloudWatch to get the InvocationsPerVariant to show how invocations are split across variants."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from datetime import datetime, timedelta\n",
      "\n",
      "import boto3\n",
      "import pandas as pd\n",
      "\n",
      "def get_invocation_metrics_for_endpoint_variant(endpoint_name,\n",
      "                                                namespace_name,\n",
      "                                                metric_name,\n",
      "                                                variant_name,\n",
      "                                                start_time,\n",
      "                                                end_time):\n",
      "    metrics = cw.get_metric_statistics(\n",
      "        Namespace=namespace_name,\n",
      "        MetricName=metric_name,\n",
      "        StartTime=start_time,\n",
      "        EndTime=end_time,\n",
      "        Period=60,\n",
      "        Statistics=[\"Sum\"],\n",
      "        Dimensions=[\n",
      "            {\n",
      "                \"Name\": \"EndpointName\",\n",
      "                \"Value\": endpoint_name\n",
      "            },\n",
      "            {\n",
      "                \"Name\": \"VariantName\",\n",
      "                \"Value\": variant_name\n",
      "            }\n",
      "        ]\n",
      "    )\n",
      "\n",
      "    if metrics['Datapoints']:\n",
      "        return pd.DataFrame(metrics[\"Datapoints\"])\\\n",
      "                .sort_values(\"Timestamp\")\\\n",
      "                .set_index(\"Timestamp\")\\\n",
      "                .drop(\"Unit\", axis=1)\\\n",
      "                .rename(columns={\"Sum\": variant_name})\n",
      "    else:\n",
      "        return pd.DataFrame()\n",
      "\n",
      "\n",
      "def plot_endpoint_metrics_for_variants(endpoint_name,\n",
      "                                       namespace_name,\n",
      "                                       metric_name,\n",
      "                                       start_time=None):\n",
      "    try:\n",
      "        start_time = start_time or datetime.now() - timedelta(minutes=60)\n",
      "        end_time = datetime.now()\n",
      "\n",
      "        metrics_variantA = get_invocation_metrics_for_endpoint_variant(endpoint_name=model_ab_endpoint_name, \n",
      "                                                                       namespace_name=namespace_name,\n",
      "                                                                       metric_name=metric_name,\n",
      "                                                                       variant_name=variantA[\"VariantName\"], \n",
      "                                                                       start_time=start_time, \n",
      "                                                                       end_time=end_time)\n",
      "\n",
      "        metrics_variantB = get_invocation_metrics_for_endpoint_variant(endpoint_name=model_ab_endpoint_name,\n",
      "                                                                       namespace_name=namespace_name,\n",
      "                                                                       metric_name=metric_name,                                                                   \n",
      "                                                                       variant_name=variantB[\"VariantName\"], \n",
      "                                                                       start_time=start_time, \n",
      "                                                                       end_time=end_time)\n",
      "\n",
      "        metrics_variants = metrics_variantA.join(metrics_variantB, how=\"outer\")\n",
      "        metrics_variants.plot()\n",
      "    except:\n",
      "        pass\n",
      "    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Show the Metrics for Each Variant\n",
      "If you see `Metrics not yet available`, please be patient as metrics may take a few mins to appear in CloudWatch.\n",
      "\n",
      "Also, make sure the predictions ran successfully above."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(20)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='/aws/sagemaker/Endpoints',\n",
      "                                   metric_name='CPUUtilization')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='Invocations')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='InvocationsPerInstance')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='ModelLatency')"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Shift All Traffic to Variant B\n",
      "_**No downtime** occurs during this traffic-shift activity._\n",
      "\n",
      "This may take a few minutes.  Please be patient."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "updated_endpoint_config = [\n",
      "    {\n",
      "        'VariantName': variantA['VariantName'],\n",
      "        'DesiredWeight': 0,\n",
      "    },\n",
      "    {\n",
      "        'VariantName': variantB['VariantName'],\n",
      "        'DesiredWeight': 100,\n",
      "    }\n",
      "]"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "sm.update_endpoint_weights_and_capacities(\n",
      "    EndpointName=model_ab_endpoint_name,\n",
      "    DesiredWeightsAndCapacities=updated_endpoint_config\n",
      ")"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {
      "scrolled": true
     },
     "outputs": [],
     "source": [
      "from IPython.core.display import display, HTML\n",
      "\n",
      "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">REST Endpoint</a></b>'.format(region, model_ab_endpoint_name)))\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# _Wait for the ^^ Endpoint Update ^^ to Complete Above_\n",
      "This may take a few minutes.  Please be patient."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "waiter = sm.get_waiter('endpoint_in_service')\n",
      "waiter.wait(EndpointName=model_ab_endpoint_name)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Run Some More Predictions"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import pandas as pd\n",
      "\n",
      "def predict(review_body):\n",
      "    return predictor.predict([review_body])[0]\n",
      "\n",
      "df_sample_reviews['predicted_class'] = df_sample_reviews['review_body'].map(predict)\n",
      "df_sample_reviews"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Show the Metrics for Each Variant\n",
      "If you see `Metrics not yet available`, please be patient as metrics may take a few mins to appear in CloudWatch.\n",
      "\n",
      "Also, make sure the predictions ran successfully above."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(20)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='/aws/sagemaker/Endpoints',\n",
      "                                   metric_name='CPUUtilization')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='Invocations')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='InvocationsPerInstance')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='ModelLatency')"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Remove Variant A to Reduce Cost\n",
      "Modify the Endpoint Configuration to only use variant B.\n",
      "\n",
      "_**No downtime** occurs during this scale-down activity._\n",
      "\n",
      "This may take a few mins.  Please be patient."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import time\n",
      "timestamp = '{}'.format(int(time.time()))\n",
      "\n",
      "updated_endpoint_config_name = '{}-{}'.format(training_job_name, timestamp)\n",
      "\n",
      "updated_endpoint_config = sm.create_endpoint_config(\n",
      "    EndpointConfigName=updated_endpoint_config_name,\n",
      "    ProductionVariants=[\n",
      "        {\n",
      "         'VariantName': variantB['VariantName'],\n",
      "         'ModelName': model_b_name,  # Only specify variant B to remove variant A\n",
      "         'InstanceType':'ml.m5.4xlarge',\n",
      "         'InitialInstanceCount': 1,\n",
      "         'InitialVariantWeight': 100\n",
      "        }\n",
      "    ])"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {
      "scrolled": true
     },
     "outputs": [],
     "source": [
      "sm.update_endpoint(\n",
      "    EndpointName=model_ab_endpoint_name,\n",
      "    EndpointConfigName=updated_endpoint_config_name\n",
      ")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# _If You See An ^^ Error ^^ Above, Please Wait Until the Endpoint is Updated_"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "from IPython.core.display import display, HTML\n",
      "\n",
      "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">REST Endpoint</a></b>'.format(region, model_ab_endpoint_name)))\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# _Wait for the ^^ Endpoint Update ^^ to Complete Above_\n",
      "This may take a few minutes.  Please be patient."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {
      "scrolled": true
     },
     "outputs": [],
     "source": [
      "waiter = sm.get_waiter('endpoint_in_service')\n",
      "waiter.wait(EndpointName=model_ab_endpoint_name)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Run Some More Predictions"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import pandas as pd\n",
      "\n",
      "def predict(review_body):\n",
      "    return predictor.predict([review_body])[0]\n",
      "\n",
      "df_sample_reviews['predicted_class'] = df_sample_reviews['review_body'].map(predict)\n",
      "df_sample_reviews"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Show the Metrics for Each Variant\n",
      "If you see `Metrics not yet available`, please be patient as metrics may take a few mins to appear in CloudWatch.\n",
      "\n",
      "Also, make sure the predictions ran successfully above."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(20)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='/aws/sagemaker/Endpoints',\n",
      "                                   metric_name='CPUUtilization')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='Invocations')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='InvocationsPerInstance')"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "%config InlineBackend.figure_format='retina'\n",
      "\n",
      "time.sleep(5)\n",
      "plot_endpoint_metrics_for_variants(endpoint_name=model_ab_endpoint_name,\n",
      "                                   namespace_name='AWS/SageMaker',                                   \n",
      "                                   metric_name='ModelLatency')"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Delete Endpoint\n",
      "To save money, we should delete the endpoint."
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "sm.delete_endpoint(\n",
      "     EndpointName=model_ab_endpoint_name\n",
      ")"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# More Links\n",
      "* Optimize Cost with TensorFlow and Elastic Inference\n",
      "https://aws.amazon.com/blogs/machine-learning/optimizing-costs-in-amazon-elastic-inference-with-amazon-tensorflow/\n",
      "\n",
      "* Using API Gateway with SageMaker Endpoints\n",
      "https://aws.amazon.com/blogs/machine-learning/creating-a-machine-learning-powered-rest-api-with-amazon-api-gateway-mapping-templates-and-amazon-sagemaker/"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%store"
     ]
    },
    {
     "cell_type": "code",
     "execution_count": null,
     "metadata": {},
     "outputs": [],
     "source": [
      "%%javascript\n",
      "Jupyter.notebook.save_checkpoint();\n",
      "Jupyter.notebook.session.delete();"
     ]
    }
   ],
   "metadata": {
    "kernelspec": {
     "display_name": "conda_python3",
     "language": "python",
     "name": "conda_python3"
    },
    "language_info": {
     "codemirror_mode": {
      "name": "ipython",
      "version": 3
     },
     "file_extension": ".py",
     "mimetype": "text/x-python",
     "name": "python",
     "nbconvert_exporter": "python",
     "pygments_lexer": "ipython3",
     "version": "3.6.10"
    }
   },
   "nbformat": 4,
   "nbformat_minor": 4
  }
  