Repository: googleapis/google-cloud-python

Issue:
BigQuery: Potentially more Pythonic options for the API?
I _thought_ I'd discussed this with @tswast, and he'd even generously added a PR; but after ten mins of searching GH I've come up empty on both issues and PRs, so posting here. 
@tswast lmk if my memory is correct and we have discussed this!

I appreciate API design is difficult and there are tradeoffs between verbosity vs explicitness, and consistency within a product vs language-specific adjustments. The bar should be high for people offering criticism when they can only see a subset of the relevant information.
But I do frequently use the BQ python API and find it fairly awkward and un-pythonic, as though I were writing Java.

Here's an example, straight from the docs:

```
from google.cloud import bigquery
client = bigquery.Client()
dataset_id = 'my_dataset'  # replace with your dataset ID
table_id = 'my_table'  # replace with your table ID
table_ref = client.dataset(dataset_id).table(table_id)
table = client.get_table(table_ref)  # API request
```

I would love to be able to write this, and receive a table object. 

```
client = bigquery.Client()
client.table(dataset='ds', table='tbl')
```

There are some points a level down (the `.table` object doesn't return a `Table` object while a dataset object does return a `Dataset` object / do we need a `TableReference` class in place of a string, etc), but they all center around the ergonomics of the API, particularly in respect to Python.

Thank you as ever for a wonderful product, and appreciate any thoughts on whether I'm making mistakes here.


Discussion:
In our previous conversation, I added a `from_string()` method to the `DatasetReference` and `TableReference` classes. https://github.com/GoogleCloudPlatform/google-cloud-python/pull/5255

You're right that the BigQuery client does still feel overly verbose. I find it a problem that `from_string()` can't account for the default project on the client.

The reason we have the `*Reference` classes is that the BigQuery API has such resources in the REST responses. I wanted to be super clear when the API only returns a pointer to a table rather than a full table resource.

It's a historical artifact that `client.dataset()` and `dataset.table()` return a reference and not an actual Dataset or Table. In retrospect, I probably should have renamed or removed those methods in the 1.0 redesign project.

An idea:

- Whereever Client methods accept a `*Reference`, also allow a string. This gets a little muddy in the library code, but I think so long as it's documented well it'd be okay.

PRs welcome!
cc @shollyman @alixhami 
Hi Tim!

Thanks for your thoughtful reply, as ever! I had the sense there might be some influence from the REST design.

> Whereever Client methods accept a *Reference, also allow a string. This gets a little muddy in the library code, but I think so long as it's documented well it'd be okay.

That makes a lot of sense. Particularly if this could be `project[:.]dataset.table` _or_ `dataset.table` and fall back to the default project, then that could surmount the issue you point out with `.from_string`. I think that would accomplish most of the goals - it would be easy & intuitive to get tables, and users wouldn't need to learn the ref objects for 80% of cases. A step further would be to attempt to coerce strings / `Tableref` objects to `Table`s, though maybe that's a step too far.

I'm trying to focus on xarray & pandas-gbq re OSS work (and a touch of rust!) so while I'd love to tinker here, please don't wait for me on this one. I hope the comments are still productive even from the peanut gallery.
Why not a step further and introducing `default_dataset` at `Client()`-level. It could propagate to similar property e.g. in `QueryJobConfig()`, or act as default wherever a dataset is required, including in `.from_string` if only table is passed...

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/client.py b/bigquery/google/cloud/bigquery/client.py
--- a/bigquery/google/cloud/bigquery/client.py
+++ b/bigquery/google/cloud/bigquery/client.py
@@ -285,8 +285,14 @@ def create_dataset(self, dataset):
         https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert
 
         Args:
-            dataset (google.cloud.bigquery.dataset.Dataset):
-                A ``Dataset`` populated with the desired initial state.
+            dataset (Union[ \
+                :class:`~google.cloud.bigquery.dataset.Dataset`, \
+                :class:`~google.cloud.bigquery.dataset.DatasetReference`, \
+                str, \
+            ]):
+                A :class:`~google.cloud.bigquery.dataset.Dataset` to create.
+                If ``dataset`` is a reference, an empty dataset is created
+                with the specified ID and client's default location.
 
         Returns:
             google.cloud.bigquery.dataset.Dataset:
@@ -300,6 +306,12 @@ def create_dataset(self, dataset):
             >>> dataset = client.create_dataset(dataset)
 
         """
+        if isinstance(dataset, str):
+            dataset = DatasetReference.from_string(
+                dataset, default_project=self.project)
+        if isinstance(dataset, DatasetReference):
+            dataset = Dataset(dataset)
+
         path = '/projects/%s/datasets' % (dataset.project,)
 
         data = dataset.to_api_repr()
@@ -317,12 +329,27 @@ def create_table(self, table):
         See
         https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert
 
-        :type table: :class:`~google.cloud.bigquery.table.Table`
-        :param table: A ``Table`` populated with the desired initial state.
+        Args:
+            table (Union[ \
+                :class:`~google.cloud.bigquery.table.Table`, \
+                :class:`~google.cloud.bigquery.table.TableReference`, \
+                str, \
+            ]):
+                A :class:`~google.cloud.bigquery.table.Table` to create.
+                If ``table`` is a reference, an empty table is created
+                with the specified ID. The dataset that the table belongs to
+                must already exist.
 
-        :rtype: ":class:`~google.cloud.bigquery.table.Table`"
-        :returns: a new ``Table`` returned from the service.
+        Returns:
+            google.cloud.bigquery.table.Table:
+                A new ``Table`` returned from the service.
         """
+        if isinstance(table, str):
+            table = TableReference.from_string(
+                table, default_project=self.project)
+        if isinstance(table, TableReference):
+            table = Table(table)
+
         path = '/projects/%s/datasets/%s/tables' % (
             table.project, table.dataset_id)
         api_response = self._connection.api_request(


Test patch:
diff --git a/bigquery/tests/unit/test_client.py b/bigquery/tests/unit/test_client.py
--- a/bigquery/tests/unit/test_client.py
+++ b/bigquery/tests/unit/test_client.py
@@ -711,6 +711,109 @@ def test_create_dataset_w_client_location_w_dataset_location(self):
                 'location': OTHER_LOCATION,
             })
 
+    def test_create_dataset_w_reference(self):
+        path = '/projects/%s/datasets' % self.PROJECT
+        resource = {
+            'datasetReference':
+                {'projectId': self.PROJECT, 'datasetId': self.DS_ID},
+            'etag': "etag",
+            'id': "%s:%s" % (self.PROJECT, self.DS_ID),
+            'location': self.LOCATION,
+        }
+        creds = _make_credentials()
+        client = self._make_one(
+            project=self.PROJECT, credentials=creds, location=self.LOCATION)
+        conn = client._connection = _make_connection(resource)
+
+        dataset = client.create_dataset(client.dataset(self.DS_ID))
+
+        self.assertEqual(dataset.dataset_id, self.DS_ID)
+        self.assertEqual(dataset.project, self.PROJECT)
+        self.assertEqual(dataset.etag, resource['etag'])
+        self.assertEqual(dataset.full_dataset_id, resource['id'])
+        self.assertEqual(dataset.location, self.LOCATION)
+
+        conn.api_request.assert_called_once_with(
+            method='POST',
+            path=path,
+            data={
+                'datasetReference': {
+                    'projectId': self.PROJECT,
+                    'datasetId': self.DS_ID,
+                },
+                'labels': {},
+                'location': self.LOCATION,
+            })
+
+    def test_create_dataset_w_fully_qualified_string(self):
+        path = '/projects/%s/datasets' % self.PROJECT
+        resource = {
+            'datasetReference':
+                {'projectId': self.PROJECT, 'datasetId': self.DS_ID},
+            'etag': "etag",
+            'id': "%s:%s" % (self.PROJECT, self.DS_ID),
+            'location': self.LOCATION,
+        }
+        creds = _make_credentials()
+        client = self._make_one(
+            project=self.PROJECT, credentials=creds, location=self.LOCATION)
+        conn = client._connection = _make_connection(resource)
+
+        dataset = client.create_dataset(
+            '{}.{}'.format(self.PROJECT, self.DS_ID))
+
+        self.assertEqual(dataset.dataset_id, self.DS_ID)
+        self.assertEqual(dataset.project, self.PROJECT)
+        self.assertEqual(dataset.etag, resource['etag'])
+        self.assertEqual(dataset.full_dataset_id, resource['id'])
+        self.assertEqual(dataset.location, self.LOCATION)
+
+        conn.api_request.assert_called_once_with(
+            method='POST',
+            path=path,
+            data={
+                'datasetReference': {
+                    'projectId': self.PROJECT,
+                    'datasetId': self.DS_ID,
+                },
+                'labels': {},
+                'location': self.LOCATION,
+            })
+
+    def test_create_dataset_w_string(self):
+        path = '/projects/%s/datasets' % self.PROJECT
+        resource = {
+            'datasetReference':
+                {'projectId': self.PROJECT, 'datasetId': self.DS_ID},
+            'etag': "etag",
+            'id': "%s:%s" % (self.PROJECT, self.DS_ID),
+            'location': self.LOCATION,
+        }
+        creds = _make_credentials()
+        client = self._make_one(
+            project=self.PROJECT, credentials=creds, location=self.LOCATION)
+        conn = client._connection = _make_connection(resource)
+
+        dataset = client.create_dataset(self.DS_ID)
+
+        self.assertEqual(dataset.dataset_id, self.DS_ID)
+        self.assertEqual(dataset.project, self.PROJECT)
+        self.assertEqual(dataset.etag, resource['etag'])
+        self.assertEqual(dataset.full_dataset_id, resource['id'])
+        self.assertEqual(dataset.location, self.LOCATION)
+
+        conn.api_request.assert_called_once_with(
+            method='POST',
+            path=path,
+            data={
+                'datasetReference': {
+                    'projectId': self.PROJECT,
+                    'datasetId': self.DS_ID,
+                },
+                'labels': {},
+                'location': self.LOCATION,
+            })
+
     def test_create_table_w_day_partition(self):
         from google.cloud.bigquery.table import Table
         from google.cloud.bigquery.table import TimePartitioning
@@ -991,6 +1094,97 @@ def test_create_table_w_external(self):
                          SourceFormat.CSV)
         self.assertEqual(got.external_data_configuration.autodetect, True)
 
+    def test_create_table_w_reference(self):
+        path = 'projects/%s/datasets/%s/tables' % (
+            self.PROJECT, self.DS_ID)
+        creds = _make_credentials()
+        client = self._make_one(project=self.PROJECT, credentials=creds)
+        resource = {
+            'id': '%s:%s:%s' % (self.PROJECT, self.DS_ID, self.TABLE_ID),
+            'tableReference': {
+                'projectId': self.PROJECT,
+                'datasetId': self.DS_ID,
+                'tableId': self.TABLE_ID
+            },
+        }
+        conn = client._connection = _make_connection(resource)
+
+        got = client.create_table(self.TABLE_REF)
+
+        conn.api_request.assert_called_once_with(
+            method='POST',
+            path='/%s' % path,
+            data={
+                'tableReference': {
+                    'projectId': self.PROJECT,
+                    'datasetId': self.DS_ID,
+                    'tableId': self.TABLE_ID
+                },
+                'labels': {},
+            })
+        self.assertEqual(got.table_id, self.TABLE_ID)
+
+    def test_create_table_w_fully_qualified_string(self):
+        path = 'projects/%s/datasets/%s/tables' % (
+            self.PROJECT, self.DS_ID)
+        creds = _make_credentials()
+        client = self._make_one(project=self.PROJECT, credentials=creds)
+        resource = {
+            'id': '%s:%s:%s' % (self.PROJECT, self.DS_ID, self.TABLE_ID),
+            'tableReference': {
+                'projectId': self.PROJECT,
+                'datasetId': self.DS_ID,
+                'tableId': self.TABLE_ID
+            },
+        }
+        conn = client._connection = _make_connection(resource)
+
+        got = client.create_table(
+            '{}.{}.{}'.format(self.PROJECT, self.DS_ID, self.TABLE_ID))
+
+        conn.api_request.assert_called_once_with(
+            method='POST',
+            path='/%s' % path,
+            data={
+                'tableReference': {
+                    'projectId': self.PROJECT,
+                    'datasetId': self.DS_ID,
+                    'tableId': self.TABLE_ID
+                },
+                'labels': {},
+            })
+        self.assertEqual(got.table_id, self.TABLE_ID)
+
+    def test_create_table_w_string(self):
+        path = 'projects/%s/datasets/%s/tables' % (
+            self.PROJECT, self.DS_ID)
+        creds = _make_credentials()
+        client = self._make_one(project=self.PROJECT, credentials=creds)
+        resource = {
+            'id': '%s:%s:%s' % (self.PROJECT, self.DS_ID, self.TABLE_ID),
+            'tableReference': {
+                'projectId': self.PROJECT,
+                'datasetId': self.DS_ID,
+                'tableId': self.TABLE_ID
+            },
+        }
+        conn = client._connection = _make_connection(resource)
+
+        got = client.create_table('{}.{}'.format(self.DS_ID, self.TABLE_ID))
+
+        conn.api_request.assert_called_once_with(
+            method='POST',
+            path='/%s' % path,
+            data={
+                'tableReference': {
+                    'projectId': self.PROJECT,
+                    'datasetId': self.DS_ID,
+                    'tableId': self.TABLE_ID
+                },
+                'labels': {},
+            })
+        self.assertEqual(got.table_id, self.TABLE_ID)
+
     def test_get_table(self):
         path = 'projects/%s/datasets/%s/tables/%s' % (
             self.PROJECT, self.DS_ID, self.TABLE_ID)