Repository: GoogleCloudPlatform/professional-services

Issue:
Composer Code Lab: Issue with creating URL
User reported: 
I'm running the lab now, and I got stuck trying to construct the Endpoint URL.

Here are the instructions and command (I'm assuming you don't actually replace anything since environment variables were created). 
Construct your endpoint by running the following in Cloud Shell, replacing and updating the with the information you saved:
```
python dag_trigger.py \
--url="${AIRFLOW_URI}/api/experimental/dags/average-speed/dag_runs" \
--iapClientId=$CLIENT_ID \
--raw_path=gs://$PROJECT/cloud-composer-lab/raw-$EXPORT_TS/
```
And this is the error:
```
(venv) student_00_f3a493604a22@cloudshell:~/professional-services/examples/cloud-composer-examples/composer_http_post_example (qwiklabs-gcp-00-ab87f60eadec)$ python dag_trigger.py \
> --url="${AIRFLOW_URI}/api/experimental/dags/average-speed/dag_runs" \
> --iapClientId=$CLIENT_ID \
> --raw_path=gs://$PROJECT/cloud-composer-lab/raw-$EXPORT_TS/
Traceback (most recent call last):
  File "/home/student_00_f3a493604a22/professional-services/examples/cloud-composer-examples/composer_http_post_example/venv/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 289, in _call_metadata_identity_endpoint
    self._target_audience
  File "/home/student_00_f3a493604a22/professional-services/examples/cloud-composer-examples/composer_http_post_example/venv/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 172, in get
    response,
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=350744924973-km59f4t2vvj8ii3cluaq26tflodev8bn.apps.googleusercontent.com&format=full from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7f19cbd32cf8>)
```
The above exception was the direct cause of the following exception:
```
Traceback (most recent call last):
  File "dag_trigger.py", line 100, in <module>
    main()
  File "dag_trigger.py", line 96, in main
    data=json.dumps(payload))
```


Discussion:

Resolving patch:
diff --git a/examples/cloud-composer-examples/composer_http_post_example/ephemeral_dataproc_spark_dag.py b/examples/cloud-composer-examples/composer_http_post_example/ephemeral_dataproc_spark_dag.py
--- a/examples/cloud-composer-examples/composer_http_post_example/ephemeral_dataproc_spark_dag.py
+++ b/examples/cloud-composer-examples/composer_http_post_example/ephemeral_dataproc_spark_dag.py
@@ -64,7 +64,7 @@
     # POST to the REST API.
     # Alternatively, we could set this to yesterday and the dag will be triggered upon upload to the
     # dag folder.
-    'start_date': datetime.utcnow(),
+    'start_date': datetime(2020, 1, 1),
     'email_on_failure': False,
     'email_on_retry': False,
     'retries': 1,  # Retry once before failing the task.
@@ -73,12 +73,12 @@
     # We only want the DAG to run when we POST to the api.
     # Alternatively, this could be set to '@daily' to run the job once a day.
     # more options at https://airflow.apache.org/scheduler.html#dag-runs
-    'schedule_interval': None
 }
 
 # Create Directed Acyclic Graph for Airflow
-with DAG('average-speed', default_args=DEFAULT_DAG_ARGS
-         ) as dag:  # Here we are using dag as context.
+with DAG('average-speed',
+         default_args=DEFAULT_DAG_ARGS,
+         schedule_interval=None) as dag:  # Here we are using dag as context.
     # Create the Cloud Dataproc cluster.
     # Note: this operator will be flagged a success if the cluster by this name already exists.
     create_cluster = DataprocClusterCreateOperator(
@@ -87,6 +87,7 @@
         # in YYYYMMDD format. See docs https://airflow.apache.org/code.html?highlight=macros#macros
         cluster_name='ephemeral-spark-cluster-{{ ds_nodash }}',
         num_workers=2,
+        storage_bucket=Variable.get('dataproc_bucket'),
         zone=Variable.get('gce_zone'))
 
     # Submit the PySpark job.
diff --git a/examples/cloud-composer-examples/composer_http_post_example/make_iap_request.py b/examples/cloud-composer-examples/composer_http_post_example/make_iap_request.py
deleted file mode 100644
--- a/examples/cloud-composer-examples/composer_http_post_example/make_iap_request.py
+++ /dev/null
@@ -1,151 +0,0 @@
-# Copyright 2016 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Example use of a service account to authenticate to Identity-Aware Proxy."""
-
-# [START iap_make_request]
-import google.auth
-import google.auth.app_engine
-import google.auth.compute_engine.credentials
-import google.auth.iam
-from google.auth.transport.requests import Request
-import google.oauth2.credentials
-import google.oauth2.service_account
-import requests
-import requests_toolbelt.adapters.appengine
-
-IAM_SCOPE = 'https://www.googleapis.com/auth/iam'
-OAUTH_TOKEN_URI = 'https://www.googleapis.com/oauth2/v4/token'
-
-
-def make_iap_request(url, client_id, method='GET', **kwargs):
-    """Makes a request to an application protected by Identity-Aware Proxy.
-
-    Args:
-      url: The Identity-Aware Proxy-protected URL to fetch.
-      client_id: The client ID used by Identity-Aware Proxy.
-      method: The request method to use
-              ('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
-      **kwargs: Any of the parameters defined for the request function:
-                https://github.com/requests/requests/blob/master/requests/api.py
-
-    Returns:
-      The page body, or raises an exception if the page couldn't be retrieved.
-    """
-    # Figure out what environment we're running in and get some preliminary
-    # information about the service account.
-    bootstrap_credentials, _ = google.auth.default(scopes=[IAM_SCOPE])
-    if isinstance(bootstrap_credentials,
-                  google.oauth2.credentials.Credentials):
-        raise Exception('make_iap_request is only supported for service '
-                        'accounts.')
-    elif isinstance(bootstrap_credentials, google.auth.app_engine.Credentials):
-        requests_toolbelt.adapters.appengine.monkeypatch()
-
-    # For service account's using the Compute Engine metadata service,
-    # service_account_email isn't available until refresh is called.
-    bootstrap_credentials.refresh(Request())
-
-    signer_email = bootstrap_credentials.service_account_email
-    if isinstance(bootstrap_credentials,
-                  google.auth.compute_engine.credentials.Credentials):
-        # Since the Compute Engine metadata service doesn't expose the service
-        # account key, we use the IAM signBlob API to sign instead.
-        # In order for this to work:
-        #
-        # 1. Your VM needs the https://www.googleapis.com/auth/iam scope.
-        #    You can specify this specific scope when creating a VM
-        #    through the API or gcloud. When using Cloud Console,
-        #    you'll need to specify the "full access to all Cloud APIs"
-        #    scope. A VM's scopes can only be specified at creation time.
-        #
-        # 2. The VM's default service account needs the "Service Account Actor"
-        #    role. This can be found under the "Project" category in Cloud
-        #    Console, or roles/iam.serviceAccountActor in gcloud.
-        signer = google.auth.iam.Signer(Request(), bootstrap_credentials,
-                                        signer_email)
-    else:
-        # A Signer object can sign a JWT using the service account's key.
-        signer = bootstrap_credentials.signer
-
-    # Construct OAuth 2.0 service account credentials using the signer
-    # and email acquired from the bootstrap credentials.
-    service_account_credentials = google.oauth2.service_account.Credentials(
-        signer,
-        signer_email,
-        token_uri=OAUTH_TOKEN_URI,
-        additional_claims={'target_audience': client_id})
-
-    # service_account_credentials gives us a JWT signed by the service
-    # account. Next, we use that to obtain an OpenID Connect token,
-    # which is a JWT signed by Google.
-    google_open_id_connect_token = get_google_open_id_connect_token(
-        service_account_credentials)
-
-    # Fetch the Identity-Aware Proxy-protected URL, including an
-    # Authorization header containing "Bearer " followed by a
-    # Google-issued OpenID Connect token for the service account.
-    resp = requests.request(
-        method,
-        url,
-        headers={
-            'Authorization': 'Bearer {}'.format(google_open_id_connect_token)
-        },
-        **kwargs)
-    if resp.status_code == 403:
-        raise Exception(
-            'Service account {} does not have permission to '
-            'access the IAP-protected application.'.format(signer_email))
-    elif resp.status_code != 200:
-        raise Exception(
-            'Bad response from application: {!r} / {!r} / {!r}'.format(
-                resp.status_code, resp.headers, resp.text))
-    else:
-        return resp.text
-
-
-def get_google_open_id_connect_token(service_account_credentials):
-    """Get an OpenID Connect token issued by Google for the service account.
-
-    This function:
-
-      1. Generates a JWT signed with the service account's private key
-         containing a special "target_audience" claim.
-
-      2. Sends it to the OAUTH_TOKEN_URI endpoint. Because the JWT in #1
-         has a target_audience claim, that endpoint will respond with
-         an OpenID Connect token for the service account -- in other words,
-         a JWT signed by *Google*. The aud claim in this JWT will be
-         set to the value from the target_audience claim in #1.
-
-    For more information, see
-    https://developers.google.com/identity/protocols/OAuth2ServiceAccount .
-    The HTTP/REST example on that page describes the JWT structure and
-    demonstrates how to call the token endpoint. (The example on that page
-    shows how to get an OAuth2 access token; this code is using a
-    modified version of it to get an OpenID Connect token.)
-    """
-
-    service_account_jwt = (
-        service_account_credentials._make_authorization_grant_assertion())
-    request = google.auth.transport.requests.Request()
-    body = {
-        'assertion': service_account_jwt,
-        'grant_type': google.oauth2._client._JWT_GRANT_TYPE,
-    }
-    token_response = google.oauth2._client._token_endpoint_request(
-        request, OAUTH_TOKEN_URI, body)
-    return token_response['id_token']
-
-
-# [END iap_make_request]


Test patch: