Repository: GoogleCloudPlatform/professional-services

Issue:
CAI Export Pipeline is broken: KeyError: "definitions [while running 'assign_group_by_key']"
Hello @bmenasha,
pipeline is broken since yesterday evening.
I imagine something has changed on google side?
Can you help please?
thanks a lot in advanced
```
{
  "insertId": "8652472802243659974:124764:0:75485",
  "jsonPayload": {
    "step": "s5-read-shuffle43",
    "exception": "Traceback (most recent call last):\n  File \"apache_beam/runners/common.py\", line 1213, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 569, in apache_beam.runners.common.SimpleInvoker.invoke_process\n  File \"apache_beam/runners/common.py\", line 1374, in apache_beam.runners.common._OutputProcessor.process_outputs\n  File \"apache_beam/runners/worker/operations.py\", line 155, in apache_beam.runners.worker.operations.ConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 1006, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 1035, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"./asset_inventory/import_pipeline.py\", line 150, in add_input\n    resource_schema = self.element_to_schema(element)\n  File \"./asset_inventory/import_pipeline.py\", line 147, in element_to_schema\n    'iam_policy' in element)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 454, in bigquery_schema_for_resource\n    resource_name)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in _get_schema_for_resource\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in <listcomp>\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 280, in _translate_resource_to_schema\n    resources = cls._get_document_resources(document)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 272, in _get_document_resources\n    return document['definitions']\nKeyError: 'definitions'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py\", line 649, in do_work\n    work_executor.execute()\n  File \"/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py\", line 179, in execute\n    op.start()\n  File \"dataflow_worker/shuffle_operations.py\", line 63, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 64, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 79, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 80, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 84, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"apache_beam/runners/worker/operations.py\", line 356, in apache_beam.runners.worker.operations.Operation.output\n  File \"apache_beam/runners/worker/operations.py\", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive\n  File \"dataflow_worker/shuffle_operations.py\", line 261, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.process\n  File \"dataflow_worker/shuffle_operations.py\", line 268, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 356, in apache_beam.runners.worker.operations.Operation.output\n  File \"apache_beam/runners/worker/operations.py\", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 703, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 704, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/common.py\", line 1215, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 1279, in apache_beam.runners.common.DoFnRunner._reraise_augmented\n  File \"apache_beam/runners/common.py\", line 1213, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 569, in apache_beam.runners.common.SimpleInvoker.invoke_process\n  File \"apache_beam/runners/common.py\", line 1374, in apache_beam.runners.common._OutputProcessor.process_outputs\n  File \"apache_beam/runners/worker/operations.py\", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 703, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 704, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/common.py\", line 1215, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented\n  File \"/usr/local/lib/python3.7/site-packages/future/utils/__init__.py\", line 446, in raise_with_traceback\n    raise exc.with_traceback(traceback)\n  File \"apache_beam/runners/common.py\", line 1213, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 569, in apache_beam.runners.common.SimpleInvoker.invoke_process\n  File \"apache_beam/runners/common.py\", line 1374, in apache_beam.runners.common._OutputProcessor.process_outputs\n  File \"apache_beam/runners/worker/operations.py\", line 155, in apache_beam.runners.worker.operations.ConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 1006, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 1035, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"./asset_inventory/import_pipeline.py\", line 150, in add_input\n    resource_schema = self.element_to_schema(element)\n  File \"./asset_inventory/import_pipeline.py\", line 147, in element_to_schema\n    'iam_policy' in element)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 454, in bigquery_schema_for_resource\n    resource_name)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in _get_schema_for_resource\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in <listcomp>\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 280, in _translate_resource_to_schema\n    resources = cls._get_document_resources(document)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 272, in _get_document_resources\n    return document['definitions']\nKeyError: \"definitions [while running 'assign_group_by_key']\"\n",
    "worker": "sre-relaunch-2021-06-29-06292141-1qo8-harness-f4fb",
    "job": "2021-06-29_21_41_43-2985006778211866139",
    "message": "An exception was raised when trying to execute the workitem 8833639980881219882 : Traceback (most recent call last):\n  File \"apache_beam/runners/common.py\", line 1213, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 569, in apache_beam.runners.common.SimpleInvoker.invoke_process\n  File \"apache_beam/runners/common.py\", line 1374, in apache_beam.runners.common._OutputProcessor.process_outputs\n  File \"apache_beam/runners/worker/operations.py\", line 155, in apache_beam.runners.worker.operations.ConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 1006, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 1035, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"./asset_inventory/import_pipeline.py\", line 150, in add_input\n    resource_schema = self.element_to_schema(element)\n  File \"./asset_inventory/import_pipeline.py\", line 147, in element_to_schema\n    'iam_policy' in element)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 454, in bigquery_schema_for_resource\n    resource_name)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in _get_schema_for_resource\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in <listcomp>\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 280, in _translate_resource_to_schema\n    resources = cls._get_document_resources(document)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 272, in _get_document_resources\n    return document['definitions']\nKeyError: 'definitions'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py\", line 649, in do_work\n    work_executor.execute()\n  File \"/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py\", line 179, in execute\n    op.start()\n  File \"dataflow_worker/shuffle_operations.py\", line 63, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 64, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 79, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 80, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"dataflow_worker/shuffle_operations.py\", line 84, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start\n  File \"apache_beam/runners/worker/operations.py\", line 356, in apache_beam.runners.worker.operations.Operation.output\n  File \"apache_beam/runners/worker/operations.py\", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive\n  File \"dataflow_worker/shuffle_operations.py\", line 261, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.process\n  File \"dataflow_worker/shuffle_operations.py\", line 268, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 356, in apache_beam.runners.worker.operations.Operation.output\n  File \"apache_beam/runners/worker/operations.py\", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 703, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 704, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/common.py\", line 1215, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 1279, in apache_beam.runners.common.DoFnRunner._reraise_augmented\n  File \"apache_beam/runners/common.py\", line 1213, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 569, in apache_beam.runners.common.SimpleInvoker.invoke_process\n  File \"apache_beam/runners/common.py\", line 1374, in apache_beam.runners.common._OutputProcessor.process_outputs\n  File \"apache_beam/runners/worker/operations.py\", line 218, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 703, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 704, in apache_beam.runners.worker.operations.DoOperation.process\n  File \"apache_beam/runners/common.py\", line 1215, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented\n  File \"/usr/local/lib/python3.7/site-packages/future/utils/__init__.py\", line 446, in raise_with_traceback\n    raise exc.with_traceback(traceback)\n  File \"apache_beam/runners/common.py\", line 1213, in apache_beam.runners.common.DoFnRunner.process\n  File \"apache_beam/runners/common.py\", line 569, in apache_beam.runners.common.SimpleInvoker.invoke_process\n  File \"apache_beam/runners/common.py\", line 1374, in apache_beam.runners.common._OutputProcessor.process_outputs\n  File \"apache_beam/runners/worker/operations.py\", line 155, in apache_beam.runners.worker.operations.ConsumerSet.receive\n  File \"apache_beam/runners/worker/operations.py\", line 1006, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"apache_beam/runners/worker/operations.py\", line 1035, in apache_beam.runners.worker.operations.PGBKCVOperation.process\n  File \"./asset_inventory/import_pipeline.py\", line 150, in add_input\n    resource_schema = self.element_to_schema(element)\n  File \"./asset_inventory/import_pipeline.py\", line 147, in element_to_schema\n    'iam_policy' in element)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 454, in bigquery_schema_for_resource\n    resource_name)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in _get_schema_for_resource\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 102, in <listcomp>\n    for document in discovery_documents\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 280, in _translate_resource_to_schema\n    resources = cls._get_document_resources(document)\n  File \"/usr/local/lib/python3.7/site-packages/asset_inventory/api_schema.py\", line 272, in _get_document_resources\n    return document['definitions']\nKeyError: \"definitions [while running 'assign_group_by_key']\"\n",
    "logger": "root:batchworker.py:do_work",
    "thread": "106:139688331458368",
    "stage": "s06"
  },
  "resource": {
    "type": "dataflow_step",
    "labels": {
      "step_id": "s5-read-shuffle43",
      "job_id": "2021-06-29_21_41_43-2985006778211866139",
      "region": "europe-west1",
      "job_name": "sre_relaunch_2021-06-29",
      "project_id": "dfdp-sre-data"
    }
  },
  "timestamp": "2021-06-30T04:51:14.832150220Z",
  "severity": "ERROR",
  "labels": {
    "dataflow.googleapis.com/region": "europe-west1",
    "dataflow.googleapis.com/job_id": "2021-06-29_21_41_43-2985006778211866139",
    "compute.googleapis.com/resource_name": "sre-relaunch-2021-06-29-06292141-1qo8-harness-f4fb",
    "compute.googleapis.com/resource_id": "8652472802243659974",
    "compute.googleapis.com/resource_type": "instance",
    "dataflow.googleapis.com/job_name": "sre_relaunch_2021-06-29"
  },
  "logName": "projects/dfdp-sre-data/logs/dataflow.googleapis.com%2Fworker",
  "receiveTimestamp": "2021-06-30T04:51:24.572961966Z"
}
```



Discussion:
Hi,

What specific tool are you talking about?
Hi I'm speaking about [asset inventory](https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/asset-inventory).
export from CAI to json dump file then a dataflow pipeline to ingest data into Big Query with per object split.

Ok Just so you know this repo is maintained on a best effort basis. If you have a fix feel free to contribute it

@bmenasha any idea what could cause this ?
I know, but I'm forced to use these tools because CAI PM doesn't want to export CAI information with historization or column descriptions.


I understand, But please understand that this is a best effort maintained repo.

If ben has time to have a look it will be good. Otherwise i'm afraid we cannot support.

Resolving patch:
diff --git a/tools/asset-inventory/asset_inventory/api_schema.py b/tools/asset-inventory/asset_inventory/api_schema.py
--- a/tools/asset-inventory/asset_inventory/api_schema.py
+++ b/tools/asset-inventory/asset_inventory/api_schema.py
@@ -269,7 +269,7 @@ def _get_cache_key(cls, resource_name, document):
     def _get_document_resources(cls, document):
         if document.get('schemas'):
             return document['schemas']
-        return document['definitions']
+        return document.get('definitions', [])
 
     @classmethod
     def _translate_resource_to_schema(cls, resource_name, document):


Test patch: