Repository: dbt-labs/dbt-core
Pull request: Fix for non-json-serializable values in BigQuery nested columns
resolves #2336

### Description

Use the Decimal-aware json encoder when constructing an agate table from a nested BigQuery response


### Checklist
 - [x] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
 - [x] I have run this code in development and it appears to resolve the stated issue
 - [x] This PR includes tests, or tests are not required/relevant for this PR
 - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt next" section.

Problem statement:
run_sql command via dbt rpc returns Object of type Decimal is not JSON serializable
### Describe the bug
Executing a `run_sql` command via `dbt rbc` against a BigQuery profile returns `Object of type Decimal is not JSON serializable`. The equivalent command against Redshift, Snowflake, and PostgreSQL connections works correctly.

### Steps To Reproduce
A model exists with the body `with source as ( select * from {{source('recharge', 'orders')}} ) select * from source`. 

The query of the model is wrapped in a CTE limiting its results to 100 records:
```
with run_cte as (with source as ( select * from {{source('recharge', 'orders')}} )) select * from run_cte limit 100
```

And converted to base64:

```
d2l0aCBydW5fY3RlIGFzICh3aXRoIHNvdXJjZSBhcyAoIHNlbGVjdCAqIGZyb20ge3tzb3VyY2UoJ3JlY2hhcmdlJywgJ29yZGVycycpfX0gKSkgc2VsZWN0ICogZnJvbSBydW5fY3RlIGxpbWl0IDEwMA==
```

A request is made to the RPC Server with the `run_sql` command, with the following body:

```
{
    jsonrpc: "2.0",
    method: 'run_sql',
    id: 123,
    params:  {
      name: "query",
      timeout: 60,
      sql: d2l0aCBydW5fY3RlIGFzICh3aXRoIHNvdXJjZSBhcyAoIHNlbGVjdCAqIGZyb20ge3tzb3VyY2UoJ3JlY2hhcmdlJywgJ29yZGVycycpfX0gKSkgc2VsZWN0ICogZnJvbSBydW5fY3RlIGxpbWl0IDEwMA==
    }
```

### Expected behavior
I would expect this to return the result set of the underlying sql statement.

### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [x] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
0.16.01
```

**The operating system you're using:** macOS 10.13.6

**The output of `python --version`:** 3.7

### Additional context
Add any other context about the problem here.

Hints from issue discussion:
Thanks for the report @brettgriffin - this error occurs if a `numeric` column is found inside of a struct or array column in BigQuery

Commit message: (#2336) Fix for non-json-serializable values in BigQuery nested columns

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 758f59aee8a..6d8390465d5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,6 +23,7 @@
 - Fix skipped node count in stdout at the end of a run ([#2095](https://github.com/fishtown-analytics/dbt/issues/2095), [#2310](https://github.com/fishtown-analytics/dbt/pull/2310))
 - Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. ([#2188](https://github.com/fishtown-analytics/dbt/issues/2188), [#2325](https://github.com/fishtown-analytics/dbt/pull/2325))
 - Fix "dbt deps" command so it respects the "--project-dir" arg if specified. ([#2338](https://github.com/fishtown-analytics/dbt/issues/2338), [#2339](https://github.com/fishtown-analytics/dbt/issues/2339))
+- Fix Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures ([#2336](https://github.com/fishtown-analytics/dbt/issues/2336), [#2348](https://github.com/fishtown-analytics/dbt/pull/2348))
 
 ### Under the hood
 - Added more tests for source inheritance ([#2264](https://github.com/fishtown-analytics/dbt/issues/2264), [#2291](https://github.com/fishtown-analytics/dbt/pull/2291))
diff --git a/core/dbt/clients/agate_helper.py b/core/dbt/clients/agate_helper.py
index 9e26dc88a71..29f285edac3 100644
--- a/core/dbt/clients/agate_helper.py
+++ b/core/dbt/clients/agate_helper.py
@@ -4,6 +4,7 @@
 import datetime
 import isodate
 import json
+import dbt.utils
 from typing import Iterable, List, Dict, Union, Optional, Any
 
 from dbt.exceptions import RuntimeException
@@ -92,7 +93,7 @@ def table_from_data_flat(data, column_names: Iterable[str]) -> agate.Table:
         row = []
         for value in list(_row.values()):
             if isinstance(value, (dict, list, tuple)):
-                row.append(json.dumps(value))
+                row.append(json.dumps(value, cls=dbt.utils.JSONEncoder))
             else:
                 row.append(value)
         rows.append(row)
diff --git a/test/integration/022_bigquery_test/test_bigquery_repeated_records.py b/test/integration/022_bigquery_test/test_bigquery_repeated_records.py
index 17c529291b0..06b22fceec0 100644
--- a/test/integration/022_bigquery_test/test_bigquery_repeated_records.py
+++ b/test/integration/022_bigquery_test/test_bigquery_repeated_records.py
@@ -26,8 +26,8 @@ def test__bigquery_fetch_nested_records(self):
             cast('Stonebreaker' as string) as lname
           ) as user,
           [
-            struct(1 as val_1, 2 as val_2),
-            struct(3 as val_1, 4 as val_2)
+            struct(1 as val_1, cast(2.12 as numeric) as val_2),
+            struct(3 as val_1, cast(4.83 as numeric) as val_2)
           ] as val
 
         union all
@@ -38,8 +38,8 @@ def test__bigquery_fetch_nested_records(self):
             cast('Brickmaker' as string) as lname
           ) as user,
           [
-            struct(7 as val_1, 8 as val_2),
-            struct(9 as val_1, 0 as val_2)
+            struct(7 as val_1, cast(8 as numeric) as val_2),
+            struct(9 as val_1, cast(null as numeric) as val_2)
           ] as val
         """
 
@@ -54,8 +54,8 @@ def test__bigquery_fetch_nested_records(self):
                 '{"fname": "Johnny", "lname": "Brickmaker"}'
             ],
             "val": [
-                '[{"val_1": 1, "val_2": 2}, {"val_1": 3, "val_2": 4}]',
-                '[{"val_1": 7, "val_2": 8}, {"val_1": 9, "val_2": 0}]'
+                '[{"val_1": 1, "val_2": 2.12}, {"val_1": 3, "val_2": 4.83}]',
+                '[{"val_1": 7, "val_2": 8}, {"val_1": 9, "val_2": null}]'
             ]
         }
 


Code review comments:
diff_hunk: @@ -23,6 +23,7 @@
 - Fix skipped node count in stdout at the end of a run ([#2095](https://github.com/fishtown-analytics/dbt/issues/2095), [#2310](https://github.com/fishtown-analytics/dbt/pull/2310))
 - Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. ([#2188](https://github.com/fishtown-analytics/dbt/issues/2188), [#2325](https://github.com/fishtown-analytics/dbt/pull/2325))
 - Fix "dbt deps" command so it respects the "--project-dir" arg if specified. ([#2338](https://github.com/fishtown-analytics/dbt/issues/2338), [#2339](https://github.com/fishtown-analytics/dbt/issues/2339))
+- Fix Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures ([#2336](https://github.com/fishtown-analytics/dbt/issues/2336), [#2348](https://github.com/fishtown-analytics/dbt/pull/2348))
path: CHANGELOG.md
text: @user1:
```suggestion
- Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures ([#2336](https://github.com/fishtown-analytics/dbt/issues/2336), [#2348](https://github.com/fishtown-analytics/dbt/pull/2348))
```