BUG: Unhandled ValueError when Bigquery called through io.gbq returns zero rows #10273
State: closed

closes #10273 


Labels:
name: Bug

Comments:
Issue number in form of comment? Also, why is 

```
assert_frame_equal
```

better for checking empty dataframe? I am asking just for my understanding. 

oh, because the frame is not exempty, it has column names (or should have), and an index (which has 0-len). You are guaranteeing a certain return type and meta-data to the user (e.g. a query that returns rows has these things, so an empty one should as well).

OK. Here it is.

ok, pls add a release note in the whatsnew for 0.16.2.

pls squash as well.

Can you check-in the base template for doc/source/whatsnew/v0.16.2.txt please? Or should I copy it from 5ebf521eac528f5a4b772e4677f70b689b8b1826 ?

rebase on master. Its already there.

Added release note. Integrated with master.

See contributing docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

pls squash.

I am checking-in Squashed commit in few minutes.

What about docs? I checked in release note in dadf5c2bf71cf8c2a8e85cdf0e8664148e778a11. Is it missing something?
I read the documentation you mentioned above but could not understand  specific problem.

Squashed commit submitted.

cc @jacobschaer

can you test this out and lmk?

cc @jacobschaer
cc @sean-schaefer

@ssaumitra 

can you show

`nosetests  pandas/io/tests/test_gbq.py  -v`

on your system (as this is not testsed with actual credentials on travis). 

@jreback I am away from work. I will upload the output next week as soon as I can.

The test output is as follows. I am also updating documentation file to mark changes in 0.17, not in 0.16.2.

```
$ nosetests pandas/io/tests/test_gbq.py -v
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_bad_project_id (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_bad_table_name (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order_plus_index (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_download_dataset_larger_than_200k_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_index_column (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_malformed_query (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_arbitrary_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_empty_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_false_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_timestamp_unix_epoch (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_true_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_unicode_string_conversion_and_normalization (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_zero_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_read_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_booleans_as_python_booleans (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_floats_as_python_floats (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_integers_as_python_floats (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_strings_as_python_strings (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_timestamps_as_numpy_datetime (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_that_parse_data_works_properly (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_to_gbq_should_fail_if_invalid_table_name_passed (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_to_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_generate_bq_schema (pandas.io.tests.test_gbq.TestToGBQIntegration) ... SKIP: Cannot run to_gbq tests without bq command line client
test_google_upload_errors_should_raise_exception (pandas.io.tests.test_gbq.TestToGBQIntegration) ... SKIP: Cannot run to_gbq tests without bq command line client
test_upload_data (pandas.io.tests.test_gbq.TestToGBQIntegration) ... SKIP: Cannot run to_gbq tests without bq command line client
pandas.io.tests.test_gbq.test_requirements ... ok

----------------------------------------------------------------------
Ran 40 tests in 51.413s

OK (SKIP=3)
```

@jreback  Any news?

@jreback I have merged latest changes. Please let me know whether any changes are needed in this commit. I am available to make changes in next week. But I would be away few weeks after that.

cc @jacobschaer
cc @sean-schaefer

@ssaumitra can you rebase.

@jreback rebase done.

@jreback @sean-schaefer I will be away from work from next week. So I won't be able to respond. It would be great if we can conclude it this week. Is any more input required from my side here?

this looks fine. Ideally i'd like

cc @jacobschaer
cc @sean-schaefer

to give a test

Looked fine, thanks. All tests passed when I ran it:

```
Successfully installed numpy pandas
Cleaning up...
+ pip freeze
Cython==0.23.1
argparse==1.2.1
bigquery==2.0.17
ez-setup==0.9
google-api-python-client==1.2
google-apputils==0.4.2
httplib2==0.9.1
nose==1.3.7
numpy==1.9.2
oauth2client==1.2
-e git+https://github.com/ssaumitra/pandas.git@cf6025e6ccd2a4bf79fe0b85e852bc3fe0ef50ff#egg=pandas-origin/bugfix-bigquery
pyasn1==0.1.8
pyasn1-modules==0.0.7
python-dateutil==2.4.2
python-gflags==2.0
pytz==2015.4
rsa==3.2
simplejson==3.8.0
six==1.9.0
uritemplate==0.6
wsgiref==0.1.2
+ python pandas/io/tests/test_gbq.py

nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
test_should_be_able_to_get_a_bigquery_service (__main__.TestGBQConnectorIntegration) ... ok

test_should_be_able_to_get_results_from_query (__main__.TestGBQConnectorIntegration) ... ok

test_should_be_able_to_get_schema_from_query (__main__.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_valid_credentials (__main__.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_make_a_connector (__main__.TestGBQConnectorIntegration) ... ok

test_bad_project_id (__main__.TestReadGBQIntegration) ... ok
test_bad_table_name (__main__.TestReadGBQIntegration) ... ok

test_column_order (__main__.TestReadGBQIntegration) ... ok

test_column_order_plus_index (__main__.TestReadGBQIntegration) ... ok

test_download_dataset_larger_than_200k_rows (__main__.TestReadGBQIntegration) ... ok

test_index_column (__main__.TestReadGBQIntegration) ... ok
test_malformed_query (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_arbitrary_timestamp (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_empty_strings (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_false_boolean (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_null_boolean (__main__.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_floats (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_null_integers (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_null_strings (__main__.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_timestamp (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_timestamp_unix_epoch (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_true_boolean (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_valid_floats (__main__.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_integers (__main__.TestReadGBQIntegration) ... ok

test_should_properly_handle_valid_strings (__main__.TestReadGBQIntegration) ... ok

test_unicode_string_conversion_and_normalization (__main__.TestReadGBQIntegration) ... ok

test_zero_rows (__main__.TestReadGBQIntegration) ... ok
test_read_gbq_with_no_project_id_given_should_fail (__main__.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_booleans_as_python_booleans (__main__.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_floats_as_python_floats (__main__.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_integers_as_python_floats (__main__.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_strings_as_python_strings (__main__.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_timestamps_as_numpy_datetime (__main__.TestReadGBQUnitTests) ... ok
test_that_parse_data_works_properly (__main__.TestReadGBQUnitTests) ... ok
test_to_gbq_should_fail_if_invalid_table_name_passed (__main__.TestReadGBQUnitTests) ... ok
test_to_gbq_with_no_project_id_given_should_fail (__main__.TestReadGBQUnitTests) ... ok

Dataset 'serene-epsilon-769:pydata_pandas_bq_testing' successfully created.

Table 'serene-epsilon-769:pydata_pandas_bq_testing.new_test' successfully created.
test_generate_bq_schema (__main__.TestToGBQIntegration) ... 
ok
test_google_upload_errors_should_raise_exception (__main__.TestToGBQIntegration) ...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...
Job not yet complete...



Streaming Insert is 100% Completeok
test_upload_data (__main__.TestToGBQIntegration) ...
__main__.test_requirements ... ok

----------------------------------------------------------------------
Ran 40 tests in 565.830s

OK
Job not yet complete...
[Output Exception BugFix] $ /bin/sh -xe /tmp/hudson8412146074501557652.sh
+ cp bigquery_credentials.dat /var/lib/jenkins/bigquery_credentials.dat
Finished: SUCCESS
```

@jacobschaer gr8 thanks!

merged via https://github.com/pydata/pandas/commit/53a6830b30248865aeddadcd4e888eb06e88cdea

thanks!

Thanks all :)