Repository: GoogleCloudPlatform/professional-services

Issue:
asset inventory dataflow jobs failing with "The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING." and "A hot key was detected in step"
8 days ago we started getting the following dataflow errors in the asset-inventory tool:

"google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27'] === Source Location Trace: === dist_proc/dax/workflow/worker/fnapi_sdk_harness.cc:178"

"A hot key was detected in step 'group_by_key_before_enforce/Read+enforce_schema+group_by_key_before_write/Write' with age of 1m57.513412938s. This is a symptom of key distribution being skewed. To fix, please inspect your data and pipeline to ensure that elements are evenly distributed across your key space."
 
The asset-inventory is set to overwrite the BigQuery tables on each run and no changes were made to the resources prior to seeing the error.  

Any assistance or suggestions at fixing this issue would be much appreciated.

Cheers,
Dante

**Error Logs:**

`{
  "insertId": "2598849684236876365:261148:0:172781",
  "jsonPayload": {
    "thread": "244",
    "line": "hot_key_logger.cc:169",
    "message": "A hot key was detected in step 'group_by_key_before_enforce/Read+enforce_schema+group_by_key_before_write/Write' with age of 1m57.513412938s. This is a symptom of key distribution being skewed. To fix, please inspect your data and pipeline to ensure that elements are evenly distributed across your key space."
  },
  "resource": {
    "type": "dataflow_step",
    "labels": {
      "project_id": "****",
      "step_id": "",
      "job_name": "cloud-asset-import-2022-10-06t07-32-18-998964",
      "job_id": "2022-10-06_00_32_19-16094098302119097546",
      "region": "us-central1"
    }
  },
  "timestamp": "2022-10-06T07:54:30.004049Z",
  "severity": "WARNING",
  "labels": {
    "dataflow.googleapis.com/job_name": "cloud-asset-import-2022-10-06t07-32-18-998964",
    "dataflow.googleapis.com/log_type": "system",
    "compute.googleapis.com/resource_id": "2598849684236876365",
    "dataflow.googleapis.com/region": "us-central1",
    "compute.googleapis.com/resource_name": "cloud-asset-import-2022-1-10060032-hd41-harness-p6z1",
    "dataflow.googleapis.com/job_id": "2022-10-06_00_32_19-16094098302119097546",
    "compute.googleapis.com/resource_type": "instance"
  },
  "logName": "projects/*****/logs/dataflow.googleapis.com%2Fharness",
  "receiveTimestamp": "2022-10-06T07:54:58.087488001Z"
}`


Workflow failed. Causes: S19:group_written_objects_by_key/Read+delete_tables+load_to_bigquery failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: 

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']
,

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']
,

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']
,

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']

asset inventory dataflow jobs failing with "The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING." and "A hot key was detected in step"
8 days ago we started getting the following dataflow errors in the asset-inventory tool:

"google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27'] === Source Location Trace: === dist_proc/dax/workflow/worker/fnapi_sdk_harness.cc:178"

"A hot key was detected in step 'group_by_key_before_enforce/Read+enforce_schema+group_by_key_before_write/Write' with age of 1m57.513412938s. This is a symptom of key distribution being skewed. To fix, please inspect your data and pipeline to ensure that elements are evenly distributed across your key space."
 
The asset-inventory is set to overwrite the BigQuery tables on each run and no changes were made to the resources prior to seeing the error.  

Any assistance or suggestions at fixing this issue would be much appreciated.

Cheers,
Dante

**Error Logs:**

`{
  "insertId": "2598849684236876365:261148:0:172781",
  "jsonPayload": {
    "thread": "244",
    "line": "hot_key_logger.cc:169",
    "message": "A hot key was detected in step 'group_by_key_before_enforce/Read+enforce_schema+group_by_key_before_write/Write' with age of 1m57.513412938s. This is a symptom of key distribution being skewed. To fix, please inspect your data and pipeline to ensure that elements are evenly distributed across your key space."
  },
  "resource": {
    "type": "dataflow_step",
    "labels": {
      "project_id": "****",
      "step_id": "",
      "job_name": "cloud-asset-import-2022-10-06t07-32-18-998964",
      "job_id": "2022-10-06_00_32_19-16094098302119097546",
      "region": "us-central1"
    }
  },
  "timestamp": "2022-10-06T07:54:30.004049Z",
  "severity": "WARNING",
  "labels": {
    "dataflow.googleapis.com/job_name": "cloud-asset-import-2022-10-06t07-32-18-998964",
    "dataflow.googleapis.com/log_type": "system",
    "compute.googleapis.com/resource_id": "2598849684236876365",
    "dataflow.googleapis.com/region": "us-central1",
    "compute.googleapis.com/resource_name": "cloud-asset-import-2022-1-10060032-hd41-harness-p6z1",
    "dataflow.googleapis.com/job_id": "2022-10-06_00_32_19-16094098302119097546",
    "compute.googleapis.com/resource_type": "instance"
  },
  "logName": "projects/*****/logs/dataflow.googleapis.com%2Fharness",
  "receiveTimestamp": "2022-10-06T07:54:58.087488001Z"
}`


Workflow failed. Causes: S19:group_written_objects_by_key/Read+delete_tables+load_to_bigquery failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: 

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']
,

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']
,

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']
,

  cloud-asset-import-2022-1-10060032-hd41-harness-p6z1
      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
    response = task()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 423, in process
  File "/home/bmenasha/workspace/professional-services/tools/asset-inventory/./asset_inventory/import_pipeline.py", line 412, in process
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 2350, in load_table_from_uri
    load_job._begin(retry=retry, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 510, in _begin
    api_response = client._call_api(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api
    return call()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 484, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/security-asset-inventory/jobs?prettyPrint=false: The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING. [while running 'load_to_bigquery-ptransform-27']



Discussion:
Is this error consistently reproducible? 
If you wish to provide your asset export file, I can try to recreate myself. (grant bmenasha@google.com read access to the gcs object). 
When you say that you overwrite tables, I assume you mean that write_disposition is WRITE_EMPTY so tables are deleted on every import?
thanks

@bmenasha I have given access to the assets exports in gcs and sent the specifics to your email.

The errors are consistently reproducible, I've had the same errors since the September 29th on all dataflow runs. 
You're correct, the write_disposition is WRITE_EMPTY so table are deleted every time. (Sorry, should have been more specific)

Thanks,
Dante 
Is this error consistently reproducible? 
If you wish to provide your asset export file, I can try to recreate myself. (grant bmenasha@google.com read access to the gcs object). 
When you say that you overwrite tables, I assume you mean that write_disposition is WRITE_EMPTY so tables are deleted on every import?
thanks

@bmenasha I have given access to the assets exports in gcs and sent the specifics to your email.

The errors are consistently reproducible, I've had the same errors since the September 29th on all dataflow runs. 
You're correct, the write_disposition is WRITE_EMPTY so table are deleted every time. (Sorry, should have been more specific)

Thanks,
Dante 

Resolving patch:
diff --git a/tools/asset-inventory/asset_inventory/bigquery_schema.py b/tools/asset-inventory/asset_inventory/bigquery_schema.py
--- a/tools/asset-inventory/asset_inventory/bigquery_schema.py
+++ b/tools/asset-inventory/asset_inventory/bigquery_schema.py
@@ -144,7 +144,6 @@ def _merge_fields(destination_field, source_field):
     Returns:
         A `google.cloud.bigquery.SchemaField` dict.
     """
-    field = copy.deepcopy(destination_field)
 
     dd = destination_field.get('description', None)
     sd = source_field.get('description', None)
@@ -152,11 +151,15 @@ def _merge_fields(destination_field, source_field):
     sft = source_field.get('field_type', None)
     # use the field with more information.
     if ((not dd and sd) or (sd and dd and len(dd) < len(sd))):
-        field['description'] = sd
-        field['field_type'] = sft
+        destination_field['description'] = sd
+        destination_field['field_type'] = sft
+
     # use the less specific type. and join fields
-    elif ((dft != 'RECORD' and dft != 'STRING') and sft == 'STRING'):
-        field['field_type'] = sft
+    # but don't overwrite the timestamp field as per
+    # https://github.com/GoogleCloudPlatform/professional-services/issues/900
+    elif (source_field.get('name', None) != 'timestamp' and
+          (dft != 'RECORD' and dft != 'STRING') and sft == 'STRING'):
+        destination_field['field_type'] = sft
 
     # https://github.com/GoogleCloudPlatform/professional-services/issues/614
     # Use the schema with the additonalProperties overrides. See
@@ -174,23 +177,23 @@ def merge_additional_properties_fields(apf, fields):
     sf = source_field.get('fields', [])
     df = destination_field.get('fields', [])
     if is_additonal_properties(sf) and not is_additonal_properties(df):
-        field['mode'] = 'REPEATED'
+        destination_field['mode'] = 'REPEATED'
         sf = copy.deepcopy(sf)
         merge_additional_properties_fields(sf, df)
-        field['fields'] = sf
+        destination_field['fields'] = sf
     elif is_additonal_properties(df) and not is_additonal_properties(sf):
-        field['mode'] = 'REPEATED'
+        destination_field['mode'] = 'REPEATED'
         merge_additional_properties_fields(df, sf)
-        field['fields'] = df
+        destination_field['fields'] = df
     elif is_additonal_properties(df) and is_additonal_properties(sf):
-        field['mode'] = 'REPEATED'
+        destination_field['mode'] = 'REPEATED'
         merge_additional_properties_fields(df, sf)
-        field['fields'] = df
+        destination_field['fields'] = df
     else:
         mf = _merge_schema(df, sf)
         if mf:
-            field['fields'] = mf
-    return field
+            destination_field['fields'] = mf
+    return destination_field
 
 
 def _merge_schema(destination_schema, source_schema):
@@ -401,11 +404,15 @@ def sanitize_property_value(property_value, depth=0, num_properties=0):
     if isinstance(property_value, list):
         for i in range(len(property_value)):
             if isinstance(property_value[i], (dict, list)):
-                sanitize_property_value(property_value[i], depth, num_properties)
+                sanitize_property_value(property_value[i],
+                                        depth, num_properties)
             else:
-                # if the list element has a primitive type, we need to re-affect the sanitized value
-                property_value[i] = sanitize_property_value(property_value[i], depth, num_properties)
-    
+                # if the list element has a primitive type, we need to
+                # re-affect the sanitized value
+                property_value[i] = sanitize_property_value(
+                    property_value[i],
+                    depth, num_properties)
+
     # and each nested json object.
     if isinstance(property_value, dict):
         remove_duplicates(property_value)
diff --git a/tools/asset-inventory/asset_inventory/import_pipeline.py b/tools/asset-inventory/asset_inventory/import_pipeline.py
--- a/tools/asset-inventory/asset_inventory/import_pipeline.py
+++ b/tools/asset-inventory/asset_inventory/import_pipeline.py
@@ -81,14 +81,15 @@ class AssignGroupByKey(beam.DoFn):
     def __init__(self, group_by, num_shards):
         if isinstance(group_by, string_types):
             group_by = StaticValueProvider(str, group_by)
+
         if isinstance(num_shards, str):
             num_shards = StaticValueProvider(str, num_shards)
-
         self.num_shards = num_shards
         self.group_by = group_by
         self.shard_map = None
 
     def apply_shard(self, key):
+        # initialize shard_map from num_shard
         if self.shard_map is None:
             self.shard_map = {
                 k: int(v) for (k, v) in
@@ -169,24 +170,31 @@ def __init__(self, group_by):
             group_by = StaticValueProvider(str, group_by)
         self.group_by = group_by
 
+    def create_resource_copy(self, element):
+        # Write out a json consistenly structured row called 'resource' to the
+        # 'resource' table. returns a copy of element without data field which
+        # is grouped by the 'resource' key.
+        resource_data = element.get('resource', {}).pop('data', None)
+        resource_copy = copy.deepcopy(element)
+        # add it back after the copy operation.
+        if resource_data is not None:
+            element['resource']['data'] = resource_data
+        # override the group by field so it's written to the `resource` table
+        resource_copy['_group_by'] = 'resource'
+        return resource_copy
+
     def process(self, element):
+        # add the resource.json_data property if we have resource.data
         if ('resource' in element and
             'data' in element['resource']):
-            resource = element['resource']
-            # add json_data property.
-            resource['json_data'] = json.dumps(resource['data'])
-            resource_element = copy.deepcopy(element)
-            resource_element['resource'].pop('data')
-            resource_element['_group_by'] = 'resource'
-            yield resource_element
-            if self.group_by.get() != 'NONE':
-                yield element
-        else:
-            resource_element = copy.deepcopy(element)
-            resource_element['_group_by'] = 'resource'
-            yield resource_element
-            if self.group_by.get() != 'NONE':
-                yield element
+            element['resource']['json_data'] = json.dumps(
+                element['resource']['data'])
+        # yeild the resource copy.
+        yield self.create_resource_copy(element)
+        # and the element if we are grouping by asset_type or
+        # asset_type_version.
+        if self.group_by.get() != 'NONE':
+            yield element
 
 
 class AddLoadTime(beam.DoFn):
@@ -213,10 +221,11 @@ def process(self, element, schemas):
         elements = element[1]
         schema = schemas[key_name]
         for elem in elements:
-            resource_data = elem.get('resource', {}).get('data', {})
-            if resource_data:
-                bigquery_schema.enforce_schema_data_types(elem, schema)
-            yield (key_name, elem)
+            if elem.get('resource', {}).get('data', {}):
+                yield (element[0], bigquery_schema.enforce_schema_data_types(
+                    elem, schema))
+            else:
+                yield (element[0], elem)
 
 
 class CombinePolicyResource(beam.DoFn):
@@ -284,7 +293,11 @@ def process(self, element):
             file_handle.write(json.dumps(asset_line).encode())
             file_handle.write(b'\n')
         if created_file_path:
-            yield (element[0], created_file_path)
+            # key is bigquery table name so each table deleted and created
+            # independently
+            # value is sharded key and gcs filepath.
+            yield (AssignGroupByKey.remove_shard(element[0]),
+                       (element[0], created_file_path))
 
     def finish_bundle(self):
         for _, file_handle in self.open_files.items():
@@ -292,6 +305,16 @@ def finish_bundle(self):
             file_handle.close()
 
 
+class AssignShardedKeyForLoad(beam.DoFn):
+    """Element is a tuple keyed by table, value is iterable of sharded key and
+       gcs file path. The transform unbundle the iterable and return a tuples of
+       sharded key and gcs file path to be loaded in parallel to bigquery.
+    """
+    def process(self, element):
+        for (sharded_key, created_file_path) in element[1]:
+            yield (sharded_key, created_file_path)
+
+
 class BigQueryDoFn(beam.DoFn):
     """Superclass for a DoFn that requires BigQuery dataset information."""
 
@@ -341,8 +364,8 @@ def start_bundle(self):
 class DeleteDataSetTables(BigQueryDoFn):
     """Delete tables when truncating and not appending.
 
-    If we are not keeping old data around, it safer to delete all tables in the
-    dataset before loading so that no old asset types remain.
+    If we are not keeping old data around, it safer to delete tables in the
+    dataset before loading so that no old records remain.
     """
 
     def __init__(self, dataset, add_load_date_suffix, load_time,
@@ -356,18 +379,21 @@ def __init__(self, dataset, add_load_date_suffix, load_time,
         self.write_disposition = write_disposition
 
     def process(self, element):
+        # If we are appending to the table, no need to Delete first.
         if self.write_disposition.get() == 'WRITE_APPEND':
             yield element
-        else:
-            key_name = AssignGroupByKey.remove_shard(element[0])
-            table_name = self.asset_type_to_table_name(key_name)
-            table_ref = self.get_dataset_ref().table(
-                table_name)
-            try:
-                self.bigquery_client.delete_table(table_ref)
-            except NotFound:
-                pass
-            yield element
+            return
+
+        # Delete the BigQuery table prior to loading to it.
+        key_name = element[0]
+        table_name = self.asset_type_to_table_name(key_name)
+        table_ref = self.get_dataset_ref().table(
+            table_name)
+        try:
+            self.bigquery_client.delete_table(table_ref)
+        except NotFound:
+            pass
+        yield element
 
 
 class LoadToBigQuery(BigQueryDoFn):
@@ -500,7 +526,7 @@ def run(argv=None):
     # Joining all iam_policy objects with resources of the same name.
     merged_iam = (
         sanitized | 'assign_name_key' >> beam.ParDo(
-            AssignGroupByKey('NAME', options.num_shards))
+            AssignGroupByKey('NAME', ''))
         | 'group_by_name' >> beam.GroupByKey()
         | 'combine_policy' >> beam.ParDo(CombinePolicyResource()))
 
@@ -517,9 +543,9 @@ def run(argv=None):
     # pylint: disable=expression-not-assigned
     (keyed_assets
      | 'add_load_time' >> beam.ParDo(AddLoadTime(options.load_time))
-     | 'group_by_key_before_enforce' >> beam.GroupByKey()
+     | 'group_by_sharded_key_for_enfoce' >> beam.GroupByKey()
      | 'enforce_schema' >> beam.ParDo(EnforceSchemaDataTypes(), pvalue_schemas)
-     | 'group_by_key_before_write' >> beam.GroupByKey()
+     | 'group_by_sharded_key_for_write' >> beam.GroupByKey()
      | 'write_to_gcs' >> beam.ParDo(
          WriteToGCS(options.stage, options.load_time))
      | 'group_written_objects_by_key' >> beam.GroupByKey()
@@ -527,6 +553,8 @@ def run(argv=None):
          DeleteDataSetTables(options.dataset, options.add_load_date_suffix,
                              options.load_time,
                              options.write_disposition))
+     | 'assign_sharded_key_for_load' >> beam.ParDo(AssignShardedKeyForLoad())
+     | 'group_by_sharded_key_for_load' >> beam.GroupByKey()
      | 'load_to_bigquery' >> beam.ParDo(
          LoadToBigQuery(options.dataset, options.add_load_date_suffix,
                         options.load_time),


Test patch: