Repository: googleapis/python-bigquery

Issue:
load_table_from_json interpolates string as int
#### Issue

using `load_table_from_json()` to load data into a string column fails when using a value that can be interpolated as an int (eg. "123")

see code snippet, which shows inconsistency with the other data loading apis

#### Environment details

  - OS type and version: macos 12.2.1
  - Python version: 3.9.5 (also errors on 3.8)
  - pip version: `pip --version`
  - `google-cloud-bigquery` version: 2.31.0 & 3.0.1

#### Code example

```python
import io
import json

from google.api_core.exceptions import BadRequest
from google.cloud.bigquery import Client, Table, SchemaField

data = [{"col_1": "123"}]

project = 'my-project'
bq = Client(project)

table = Table(f'{project}.aaa.data_load_test', schema=[SchemaField("col_1", "STRING"), ])
bq.delete_table(table, not_found_ok=True)
bq.create_table(table)


# WORKS: bq load from byte stream
with io.StringIO() as fp:
    json.dump(data, fp)
    fp.seek(0)
    bq.load_table_from_file(fp, table).result()


# WORKS: streaming api
bq.insert_rows_json(table, data)


try:
    # FAILS: bq load from python dict.
    # 123 is interpolated as an int, even though i supply it as a string
    bq.load_table_from_json(data, table).result()
except BadRequest as e:
    print("ERROR")
    print(e)


# WORKS: bq load from python dict, but 123d cannot be interpolated as an int
bq.load_table_from_json([{"col_1": "123d"}], table).result()

```

#### Stack trace

```
Traceback (most recent call last):
  File "/Users/ben.marengo/code/scratch/bq_load.py", line 28, in <module>
    bq.load_table_from_json(data, table).result()
  File "/Users/ben.marengo/code/pyenvs/scratch/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 728, in result
    return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
  File "/Users/ben.marengo/code/pyenvs/scratch/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in result
    raise self._exception
google.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table my-project:aaa.data_load_test. Field col_1 has changed type from STRING to INTEGER
```





Discussion:
This is a tough one. One possibility would be to omit any local schema interpolation (if there is any, I forget) and use an autodetect schema, instead.

I suspect you should be able to work around this issue by manually supplying the correct schema in a job configuration object.
New theory: I see we don't have any client-side schema generation. Instead, the client adds `autodetect = True` if no schema is supplied.

https://github.com/googleapis/python-bigquery/blob/66500b4cc4d03873118cf40fc33615064842f06d/google/cloud/bigquery/client.py#L2760-L2761

Perhaps omitting this flag when the table already exists will help?

I suspect that the backend first attempts to generate a schema from the provided data and then compares that generated schema with the existing table. If this is the case, we should also file a backend issue (possibly in addition to adding a workaround in the client if the table already exists).
I have been coming across the same issue when trying to load json files into BigQuery.  I thought the problem was already fixed (https://github.com/googleapis/python-bigquery/pull/1248) but the issue is still around.

How can we request that it is solved? This is the code that I am using to load data.

```
job_config = LoadJobConfig(create_disposition='CREATE_IF_NEEDED',
                                        write_disposition='WRITE_APPEND',
                                        autodetect=True)

bq_table = f'{project_id}.{dataset_id}.{table_id}'

job = bqclient.load_table_from_json(json_data, bq_table, job_config=job_config)
```

#1248 was never merged, so perhaps that fix is still relevant. 

Re: https://github.com/googleapis/python-bigquery/issues/1228#issuecomment-1119996488 — if this turns out to be a backend issue we should absolutely file a bug with the correct team (and I'd propose not adding a client-side workaround since we'd have to undo it later and that might cause even more problems for folks). 
I think this is less of a backend bug, since we are sending conflicting information to the backend: `autodetect` shouldn't be true when the table already exists. The problem is the client does not know if the table exists without making an API call to verify it. Adding this call may make the method take longer, but will reduce user confusion and time spent on errors like this. We may also update the documentation to ask people to make sure local schema aligns with existing table. WDYT? @tswast 
> Adding this call may make the method take longer, but will reduce user confusion and time spent on errors like this.

We already do this in some other methods, such as `list_rows`:

https://github.com/googleapis/python-bigquery/blob/39f33b210ecbe9c2fd390825d29393c2d80257f5/google/cloud/bigquery/client.py#L3928-L3932

Note: We should only do this call to `get_table` if all of the following is true:

* The job config specifies "APPEND" mode (or is not set, as this is the default, I believe)
* The job config does not have a schema or autodetect manually set already.
Awesome, I'll go fix this issue.
Truth table: under which condition we should check table existence, and whether client should set autodetect value for user.

| Schema Provided | (user) Autodetect | writeDisposition | Check Table Existence | (client) Autodetect |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| T | T/F/not provided | * | F | same as user value |
| F   | T/F | * | F | same as user value | 
| F   | not provided | WRITE_APPEND / not provided | T | F if table exists, otherwise T |
| F   | not provided | WRITE_TRUNCATE / WRITE_EMPTY | F | T |
~~After playing around, I'm starting to feel like we just don't have to set `autodetect = True` in the first place. For example, when I use `load_table_from_json` with schema and autodetect not set, the backend is still able to detect the schema. As these two lines of code were added five years ago, I guess maybe we had to do it in the past, but the backend has updated since? @tswast I wanted to double check the reason why we added it in the past.~~

I think my test sample was wrong, and the backend still throws an error if no schema is provided and `autodetect = False/None`.
If a table doesn't already exist, it used to be the case that either you had to specify a `schema` or you had to specify `autodetect = True`. With neither specified, it would error. It could be that now BigQuery automatically autodetects the schema in this case?
If the table does exist, you don't have to set anything.
> It could be that now BigQuery automatically autodetects the schema in this case?

I think so. I'll just remove these two lines then.

Resolving patch:
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -2833,8 +2833,22 @@ def load_table_from_json(
 
         new_job_config.source_format = job.SourceFormat.NEWLINE_DELIMITED_JSON
 
-        if new_job_config.schema is None:
-            new_job_config.autodetect = True
+        # In specific conditions, we check if the table alread exists, and/or
+        # set the autodetect value for the user. For exact conditions, see table
+        # https://github.com/googleapis/python-bigquery/issues/1228#issuecomment-1910946297
+        if new_job_config.schema is None and new_job_config.autodetect is None:
+            if new_job_config.write_disposition in (
+                job.WriteDisposition.WRITE_TRUNCATE,
+                job.WriteDisposition.WRITE_EMPTY,
+            ):
+                new_job_config.autodetect = True
+            else:
+                try:
+                    self.get_table(destination)
+                except core_exceptions.NotFound:
+                    new_job_config.autodetect = True
+                else:
+                    new_job_config.autodetect = False
 
         if project is None:
             project = self.project


Test patch:
diff --git a/tests/system/test_client.py b/tests/system/test_client.py
--- a/tests/system/test_client.py
+++ b/tests/system/test_client.py
@@ -994,6 +994,45 @@ def test_load_table_from_json_schema_autodetect(self):
         self.assertEqual(tuple(table.schema), table_schema)
         self.assertEqual(table.num_rows, 2)
 
+    # Autodetect makes best effort to infer the schema, but situations exist
+    # when the detected schema is wrong, and does not match existing schema.
+    # Thus the client sets autodetect = False when table exists and just uses
+    # the existing schema. This test case uses a special case where backend has
+    # no way to distinguish int from string.
+    def test_load_table_from_json_schema_autodetect_table_exists(self):
+        json_rows = [
+            {"name": "123", "age": 18, "birthday": "2001-10-15", "is_awesome": False},
+            {"name": "456", "age": 79, "birthday": "1940-03-10", "is_awesome": True},
+        ]
+
+        dataset_id = _make_dataset_id("bq_system_test")
+        self.temp_dataset(dataset_id)
+        table_id = "{}.{}.load_table_from_json_basic_use".format(
+            Config.CLIENT.project, dataset_id
+        )
+
+        # Use schema with NULLABLE fields, because schema autodetection
+        # defaults to field mode NULLABLE.
+        table_schema = (
+            bigquery.SchemaField("name", "STRING", mode="NULLABLE"),
+            bigquery.SchemaField("age", "INTEGER", mode="NULLABLE"),
+            bigquery.SchemaField("birthday", "DATE", mode="NULLABLE"),
+            bigquery.SchemaField("is_awesome", "BOOLEAN", mode="NULLABLE"),
+        )
+        # create the table before loading so that the column order is predictable
+        table = helpers.retry_403(Config.CLIENT.create_table)(
+            Table(table_id, schema=table_schema)
+        )
+        self.to_delete.insert(0, table)
+
+        # do not pass an explicit job config to trigger automatic schema detection
+        load_job = Config.CLIENT.load_table_from_json(json_rows, table_id)
+        load_job.result()
+
+        table = Config.CLIENT.get_table(table)
+        self.assertEqual(tuple(table.schema), table_schema)
+        self.assertEqual(table.num_rows, 2)
+
     def test_load_avro_from_uri_then_dump_table(self):
         from google.cloud.bigquery.job import CreateDisposition
         from google.cloud.bigquery.job import SourceFormat
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -8951,6 +8951,8 @@ def test_load_table_from_dataframe_w_higher_scale_decimal128_datatype(self):
             SchemaField("x", "BIGNUMERIC", "NULLABLE", None),
         )
 
+    # With autodetect specified, we pass the value as is. For more info, see
+    # https://github.com/googleapis/python-bigquery/issues/1228#issuecomment-1910946297
     def test_load_table_from_json_basic_use(self):
         from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
         from google.cloud.bigquery import job
@@ -8962,12 +8964,28 @@ def test_load_table_from_json_basic_use(self):
             {"name": "Two", "age": 22, "birthday": "1997-08-09", "adult": True},
         ]
 
+        job_config = job.LoadJobConfig(autodetect=True)
+
         load_patch = mock.patch(
             "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
         )
 
-        with load_patch as load_table_from_file:
-            client.load_table_from_json(json_rows, self.TABLE_REF)
+        # mock: remote table already exists
+        get_table_reference = {
+            "projectId": "project_id",
+            "datasetId": "test_dataset",
+            "tableId": "test_table",
+        }
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            return_value=mock.Mock(table_reference=get_table_reference),
+        )
+
+        with load_patch as load_table_from_file, get_table_patch:
+            client.load_table_from_json(
+                json_rows, self.TABLE_REF, job_config=job_config
+            )
 
         load_table_from_file.assert_called_once_with(
             client,
@@ -9066,6 +9084,174 @@ def test_load_table_from_json_w_invalid_job_config(self):
         err_msg = str(exc.value)
         assert "Expected an instance of LoadJobConfig" in err_msg
 
+    # When all following are true:
+    # (1) no schema provided;
+    # (2) no autodetect value provided;
+    # (3) writeDisposition == WRITE_APPEND or None;
+    # (4) table already exists,
+    # client sets autodetect == False
+    # For more details, see https://github.com/googleapis/python-bigquery/issues/1228#issuecomment-1910946297
+    def test_load_table_from_json_wo_schema_wo_autodetect_write_append_w_table(self):
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.job import WriteDisposition
+
+        client = self._make_client()
+
+        json_rows = [
+            {"name": "One", "age": 11, "birthday": "2008-09-10", "adult": False},
+            {"name": "Two", "age": 22, "birthday": "1997-08-09", "adult": True},
+        ]
+
+        job_config = job.LoadJobConfig(write_disposition=WriteDisposition.WRITE_APPEND)
+
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        # mock: remote table already exists
+        get_table_reference = {
+            "projectId": "project_id",
+            "datasetId": "test_dataset",
+            "tableId": "test_table",
+        }
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            return_value=mock.Mock(table_reference=get_table_reference),
+        )
+
+        with load_patch as load_table_from_file, get_table_patch:
+            client.load_table_from_json(
+                json_rows, self.TABLE_REF, job_config=job_config
+            )
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            size=mock.ANY,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=client.location,
+            project=client.project,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.source_format == job.SourceFormat.NEWLINE_DELIMITED_JSON
+        assert sent_config.schema is None
+        assert not sent_config.autodetect
+
+    # When all following are true:
+    # (1) no schema provided;
+    # (2) no autodetect value provided;
+    # (3) writeDisposition == WRITE_APPEND or None;
+    # (4) table does NOT exist,
+    # client sets autodetect == True
+    # For more details, see https://github.com/googleapis/python-bigquery/issues/1228#issuecomment-1910946297
+    def test_load_table_from_json_wo_schema_wo_autodetect_write_append_wo_table(self):
+        import google.api_core.exceptions as core_exceptions
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.job import WriteDisposition
+
+        client = self._make_client()
+
+        json_rows = [
+            {"name": "One", "age": 11, "birthday": "2008-09-10", "adult": False},
+            {"name": "Two", "age": 22, "birthday": "1997-08-09", "adult": True},
+        ]
+
+        job_config = job.LoadJobConfig(write_disposition=WriteDisposition.WRITE_APPEND)
+
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        # mock: remote table doesn't exist
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            side_effect=core_exceptions.NotFound(""),
+        )
+
+        with load_patch as load_table_from_file, get_table_patch:
+            client.load_table_from_json(
+                json_rows, self.TABLE_REF, job_config=job_config
+            )
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            size=mock.ANY,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=client.location,
+            project=client.project,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.source_format == job.SourceFormat.NEWLINE_DELIMITED_JSON
+        assert sent_config.schema is None
+        assert sent_config.autodetect
+
+    # When all following are true:
+    # (1) no schema provided;
+    # (2) no autodetect value provided;
+    # (3) writeDisposition == WRITE_TRUNCATE or WRITE_EMPTY;
+    # client sets autodetect == True
+    # For more details, see https://github.com/googleapis/python-bigquery/issues/1228#issuecomment-1910946297
+    def test_load_table_from_json_wo_schema_wo_autodetect_others(self):
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.job import WriteDisposition
+
+        client = self._make_client()
+
+        json_rows = [
+            {"name": "One", "age": 11, "birthday": "2008-09-10", "adult": False},
+            {"name": "Two", "age": 22, "birthday": "1997-08-09", "adult": True},
+        ]
+
+        job_config = job.LoadJobConfig(
+            write_disposition=WriteDisposition.WRITE_TRUNCATE
+        )
+
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        with load_patch as load_table_from_file:
+            client.load_table_from_json(
+                json_rows, self.TABLE_REF, job_config=job_config
+            )
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            size=mock.ANY,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=client.location,
+            project=client.project,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.source_format == job.SourceFormat.NEWLINE_DELIMITED_JSON
+        assert sent_config.schema is None
+        assert sent_config.autodetect
+
     def test_load_table_from_json_w_explicit_job_config_override(self):
         from google.cloud.bigquery import job
         from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
@@ -9190,8 +9376,19 @@ def test_load_table_from_json_unicode_emoji_data_case(self):
         load_patch = mock.patch(
             "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
         )
+        # mock: remote table already exists
+        get_table_reference = {
+            "projectId": "project_id",
+            "datasetId": "test_dataset",
+            "tableId": "test_table",
+        }
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            return_value=mock.Mock(table_reference=get_table_reference),
+        )
 
-        with load_patch as load_table_from_file:
+        with load_patch as load_table_from_file, get_table_patch:
             client.load_table_from_json(json_rows, self.TABLE_REF)
 
         load_table_from_file.assert_called_once_with(