Repository: dbt-athena/dbt-athena
Pull request #77 (2022-12-05 02:51:14)

fix: fix to KeyError for tables without TableType
### Description
I got an KeyError exception during dbt-athena run.

In rare cases, such as [tables not migrated to the AWS Glue Data Catalog](https://docs.aws.amazon.com/quicksight/latest/user/troubleshoot-athena-glue-table-not-upgraded.html), Get Tables API returns `TableList` without `TableType` attribute.

So I suggest ignoring such tables in this PR.

Another option is assuming it is EXTERNAL_TABLE.
However, since we don’t know detail of tables without TableType attribute, I think any assumption shouldn’t be appropriate.

Here is a stacktrace of `KeyError` on `dbt-core==1.3.1` `dbt-athena-community==1.3.2`:

```
22:44:30  Encountered an error: 'TableType'
22:44:30  Traceback (most recent call last):
  File "/project/.venv/lib/python3.9/site-packages/dbt/main.py", line 135, in main
    results, succeeded = handle_and_check(args)
...
...
  File "/project/.venv/lib/python3.9/site-packages/dbt/utils.py", line 480, in connected
    return func(*args, **kwargs)
  File "/project/.venv/lib/python3.9/site-packages/dbt/adapters/athena/impl.py", line 254, in list_relations_without_caching
    _type = table["TableType"]
KeyError: 'TableType'
```


<!--- Add a little description on what you plan to tackle with this PR -->
<!--- Add resolve #issue-number in case you resolve an open issue -->

## Models used to test - Optional
<!--- Add here the models that you use to test the changes -->


## Checklist
- [x] You followed [contributing section](https://github.com/dbt-athena/dbt-athena#contributing)
- [x] You added unit testing when necessary
- [ ] You added functional testing when necessary


nicor88: @t-yuki Could you verify that this solution works when table type is `ICEBERG`?
t-yuki: > @t-yuki Could you verify that this solution works when table type is `ICEBERG` or if in general we don't have breaking changes.

I've checked Glue API response and `dbt run` command works well with this PR.

Glue API response for iceberg table has 'EXTERNAL_TABLE' TableType attribute like:

```
{
  "TableList": {
    ...
    "TableType": "EXTERNAL_TABLE",
    "Parameters": {
        "metadata_location": "s3://my-bucket/metadata/00000-0a57eaa2-5c95-4229-90cd-097153b46b6d.metadata.json",
        "table_type": "ICEBERG"
    }
  }
}
```

So this PR doesn't break functions for iceberg table.
flaviomax: We are also experiencing this error and need this fix released :)

Patch:
diff --git a/dbt/adapters/athena/impl.py b/dbt/adapters/athena/impl.py
index 157a90b..eb5a8db 100755
--- a/dbt/adapters/athena/impl.py
+++ b/dbt/adapters/athena/impl.py
@@ -289,6 +289,9 @@ class AthenaAdapter(SQLAdapter):
         for page in page_iterator:
             tables = page["TableList"]
             for table in tables:
+                if "TableType" not in table:
+                    logger.debug(f"Table '{table['Name']}' has no TableType attribute - Ignoring")
+                    continue
                 _type = table["TableType"]
                 if _type == "VIRTUAL_VIEW":
                     _type = self.Relation.View


Test patch:
diff --git a/tests/unit/test_adapter.py b/tests/unit/test_adapter.py
index 2ce13a8..f2e21a6 100644
--- a/tests/unit/test_adapter.py
+++ b/tests/unit/test_adapter.py
@@ -434,6 +434,7 @@ class TestAthenaAdapter:
         self.mock_aws_service.create_table("table")
         self.mock_aws_service.create_table("other")
         self.mock_aws_service.create_view("view")
+        self.mock_aws_service.create_table_without_table_type("without_table_type")
         schema_relation = self.adapter.Relation.create(
             database=DATA_CATALOG_NAME,
             schema=DATABASE_NAME,
@@ -450,6 +451,7 @@ class TestAthenaAdapter:
         self.mock_aws_service.create_table("table")
         self.mock_aws_service.create_table("other")
         self.mock_aws_service.create_view("view")
+        self.mock_aws_service.create_table_without_table_type("without_table_type")
         schema_relation = self.adapter.Relation.create(
             database=data_catalog_name,
             schema=DATABASE_NAME,
diff --git a/tests/unit/utils.py b/tests/unit/utils.py
index 0104154..180ed6d 100644
--- a/tests/unit/utils.py
+++ b/tests/unit/utils.py
@@ -200,6 +200,24 @@ class MockAWSService:
             },
         )
 
+    def create_table_without_table_type(self, table_name: str):
+        glue = boto3.client("glue", region_name=AWS_REGION)
+        glue.create_table(
+            DatabaseName=DATABASE_NAME,
+            TableInput={
+                "Name": table_name,
+                "StorageDescriptor": {
+                    "Columns": [
+                        {
+                            "Name": "id",
+                            "Type": "string",
+                        },
+                    ],
+                    "Location": f"s3://{BUCKET}/tables/{table_name}",
+                },
+            },
+        )
+
     def add_data_in_table(self, table_name: str):
         s3 = boto3.client("s3", region_name=AWS_REGION)
         s3.create_bucket(Bucket=BUCKET, CreateBucketConfiguration={"LocationConstraint": AWS_REGION})