Gbq service account
State: closed

This adds service account authentication while still supports standard web auth method.
It also adds some useful std out messages: progress with elapsed time and percentage ~~\+ price calculation~~.

In RTBHouse we're using this service account auth since may. It works perfectly with remote jupyter server (iPython notebooks).

fixes #8489


Labels:

Comments:
the `private_key` should be allowed to be a sequence of bytes as well. (and rename)

OK, I've added `private_key` contents support. I understand it's better design decision.

pls add a whatsnew entry (enhancements).
can you post a sample session with verbose=True.

cc @parthea 
cc @andrewryno
cc @andrioni 
cc @jacobschaer 

pls have a look

pls rebase / squash

ok, rabased and squashed

cc @parthea 
cc @andrewryno
cc @andrioni 
cc @jacobschaer

any comments

I'm receiving the following exception related to the `time.monotonic()` function when running the gbq integration tests.

`AttributeError: 'module' object has no attribute 'monotonic'`

I believe Travis will also have this issue. Travis is currently skipping integration tests that require a BigQuery project id, so the issue is not reported by Travis. 

Should we add a dependency for `monotonic`? 
https://pypi.python.org/pypi/monotonic/0.5

```
======================================================================
ERROR: test_should_properly_handle_valid_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tony/pandas-gbq_service_account/pandas/io/tests/test_gbq.py", line 322, in test_should_properly_handle_valid_integers
    df = gbq.read_gbq(query, project_id=PROJECT_ID)
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 503, in read_gbq
    schema, pages = connector.run_query(query)
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 269, in run_query
    self._start_timer()
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 185, in _start_timer
    self.start = time.monotonic()
AttributeError: 'module' object has no attribute 'monotonic'
```

we don't need any more deps

use time.time()

Ad FileNotFoundError) I'm running tests on python 3 only, where this exception is built-in, hence I've missed this out. After adding support for json contents handling this exception is not needed any more. See line 168 in gbq.py, where I'm checking file exists and is regular file. So file not found exception can not happend. I'm removing it from except clause.

Doing this I've also rethinked, rewritten and tested invalid auth scenarios.

Ad json.load) it works for me. I cant reproduce it under python3 :(
I don't understand json magic in pandas.json module but I think it is imported here.
What I can do is to invoke json.loads which was used here before my changes... but its stupid, because python's built-in json module can read files... 

```
nosetests test_gbq.py -v 
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_bad_project_id (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_bad_table_name (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order_plus_index (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_download_dataset_larger_than_200k_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_index_column (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_malformed_query (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_arbitrary_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_empty_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_false_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_timestamp_unix_epoch (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_true_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_read_as_service_account_with_key_contents (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_read_as_service_account_with_key_path (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_unicode_string_conversion_and_normalization (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_zero_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_read_gbq_when_private_key_json_values_has_wrong_types_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_read_gbq_with_corrupted_private_key_json_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_read_gbq_with_empty_private_key_file_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_read_gbq_with_empty_private_key_json_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_read_gbq_with_invalid_private_key_json_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_read_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_booleans_as_python_booleans (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_floats_as_python_floats (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_integers_as_python_floats (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_strings_as_python_strings (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_should_return_bigquery_timestamps_as_numpy_datetime (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_that_parse_data_works_properly (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_to_gbq_should_fail_if_invalid_table_name_passed (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_to_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... ok
test_create_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_create_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_dataset_does_not_exist (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_dataset_exists (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_delete_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_delete_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_generate_schema (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_google_upload_errors_should_raise_exception (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_table_zero_results (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_table_does_not_exist (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_append (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_fail (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_replace (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_as_service_account_with_key_contents (pandas.io.tests.test_gbq.TestToGBQIntegrationServiceAccountKeyContents) ... ok
test_upload_data_as_service_account_with_key_path (pandas.io.tests.test_gbq.TestToGBQIntegrationServiceAccountKeyPath) ... ok
pandas.io.tests.test_gbq.test_requirements ... ok
pandas.io.tests.test_gbq.test_generate_bq_schema_deprecated ... ok

----------------------------------------------------------------------
Ran 73 tests in 330.985s

OK
```

Sample session `gbq.read_gbq` verbose output:

```
Requesting query... ok.
Query running...
  Elapsed 11.55 s. Waiting...
Query done.
Processed: 37.5 Mb

Retrieving results...
  Got page: 1; 10% done. Elapsed 17.25 s.
  Got page: 2; 19% done. Elapsed 20.32 s.
  Got page: 3; 29% done. Elapsed 24.03 s.
  Got page: 4; 39% done. Elapsed 28.5 s.
  Got page: 5; 48% done. Elapsed 33.15 s.
  Got page: 6; 58% done. Elapsed 37.19 s.
  Got page: 7; 67% done. Elapsed 40.6 s.
  Got page: 8; 77% done. Elapsed 43.95 s.
  Got page: 9; 87% done. Elapsed 48.27 s.
  Got page: 10; 96% done. Elapsed 185.45 s.
  Got page: 11; 100% done. Elapsed 187.93 s.
Got 1038579 rows.

Total time taken 191.67 s.
Finished at 2016-01-15 16:16:07.
```

pls `git diff master | flake8 --diff` and fix the issues

The following test is failing. All others are passing with python 2.7:

```
nosetests test_gbq.py:TestReadGBQUnitTests.test_read_gbq_with_corrupted_private_key_json_should_fail -v --nocapture
test_read_gbq_with_corrupted_private_key_json_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests) ... Requesting query... ERROR

======================================================================
ERROR: test_read_gbq_with_corrupted_private_key_json_should_fail (pandas.io.tests.test_gbq.TestReadGBQUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tony/pandas-gbq_service_account/pandas/io/tests/test_gbq.py", line 289, in test_read_gbq_with_corrupted_private_key_json_should_fail
    gbq.read_gbq('SELECT 1', project_id='x', private_key=re.sub('[0-9]', '9', PRIVATE_KEY_JSON_CONTENTS))
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 519, in read_gbq
    schema, pages = connector.run_query(query)
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 283, in run_query
    query_reply = job_collection.insert(projectId=self.project_id, body=job_data).execute()
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/tony/anaconda/lib/python2.7/site-packages/googleapiclient/http.py", line 722, in execute
    body=self.body, headers=self.headers)
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/client.py", line 565, in new_request
    self._refresh(request_orig)
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/client.py", line 835, in _refresh
    self._do_refresh_request(http_request)
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/client.py", line 862, in _do_refresh_request
    body = self._generate_refresh_request_body()
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/client.py", line 1541, in _generate_refresh_request_body
    assertion = self._generate_assertion()
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/client.py", line 1670, in _generate_assertion
    private_key, self.private_key_password), payload)
  File "/home/tony/anaconda/lib/python2.7/site-packages/oauth2client/_openssl_crypt.py", line 117, in from_string
    pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, parsed_pem_key)
  File "/home/tony/anaconda/lib/python2.7/site-packages/OpenSSL/crypto.py", line 2323, in load_privatekey
    _raise_current_error()
  File "/home/tony/anaconda/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
Error: []
```

I fixed this PY3 compat issue and few other minor things: verbosity, exception comment.
I also fixed import error reported by travis.
Please review my last commit. I'm not python master :) and not sure if those super constructor calls are correct.

But all tests are passing! (under python 3).

Ad test_read_gbq_with_corrupted_private_key_json_should_fail) 
I dont see exception name there. Under Python 3 I get `ValueError: RSA key format is not supported`. Can you tell me how to catch this specific exception under py2.7?

I receive the following error with the latest code. 

```
======================================================================
ERROR: test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tony/pandas-gbq_service_account/pandas/io/tests/test_gbq.py", line 168, in setUp
    self.sut = gbq.GbqConnector(PROJECT_ID, private_key=PRIVATE_KEY_JSON_PATH)
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 143, in __init__
    self.credentials = self.get_credentials()
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 149, in get_credentials
    return self.get_service_account_credentials()
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 193, in get_service_account_credentials
    raise InvalidPrivateKeyFormat("Service account private key should be valid JSON (file path or string contents) "
InvalidPrivateKeyFormat: Service account private key should be valid JSON (file path or string contents) with at least two keys: 'client_email' and 'private_key'. Can be obtained from google developers console. 
```

If I remove the exception handling, I get the following error:

```
======================================================================
ERROR: test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tony/pandas-gbq_service_account/pandas/io/tests/test_gbq.py", line 168, in setUp
    self.sut = gbq.GbqConnector(PROJECT_ID, private_key=PRIVATE_KEY_JSON_PATH)
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 143, in __init__
    self.credentials = self.get_credentials()
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 149, in get_credentials
    return self.get_service_account_credentials()
  File "/home/tony/pandas-gbq_service_account/pandas/io/gbq.py", line 181, in get_service_account_credentials
    json_key = json.loads(self.private_key.replace('\n', '   '))
ValueError: Expected object or value
```

Can you try running the unit tests under python 2.7 to confirm that you also get this issue?  To configure separate environments goto "Creating a development environment" at http://pandas.pydata.org/pandas-docs/stable/contributing.html

Ok, I've installed separate env for py2.7. All tests are passing.

```
$ python --version
Python 2.7.11 :: Continuum Analytics, Inc.
$ nosetests test_gbq.py -v
...
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
...
----------------------------------------------------------------------
Ran 73 tests in 361.659s

OK
```

You had probably provided wrong private key format ;)

what crypto lib had you installed? It must be `pip install PyCrypto`

I think add a section to the docs w.r.t. the `private_key` (e.g. what deps you may need, where they typically live, and how are generated). 

I've fixed this test.  All is passing even w/o crypto lib.

All tests passing in python 2.7

```
nosetests test_gbq.py -v
test_read_gbq_when_private_key_json_values_has_wrong_types_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_corrupted_private_key_json_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_empty_private_key_file_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_empty_private_key_json_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_invalid_private_key_json_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_booleans_as_python_booleans (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_floats_as_python_floats (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_integers_as_python_floats (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_strings_as_python_strings (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_timestamps_as_numpy_datetime (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_that_parse_data_works_properly (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_to_gbq_should_fail_if_invalid_table_name_passed (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_to_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_bad_project_id (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_bad_table_name (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order_plus_index (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_download_dataset_larger_than_200k_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_index_column (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_malformed_query (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_arbitrary_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_empty_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_false_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_timestamp_unix_epoch (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_true_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_read_as_service_account_with_key_contents (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_read_as_service_account_with_key_path (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_unicode_string_conversion_and_normalization (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_zero_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_create_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_create_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_dataset_does_not_exist (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_dataset_exists (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_delete_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_delete_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_generate_schema (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_google_upload_errors_should_raise_exception (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_table_zero_results (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_table_does_not_exist (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_append (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_fail (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_replace (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_as_service_account_with_key_contents (pandas.io.tests.test_gbq.TestToGBQIntegrationServiceAccountKeyContents) ... ok
test_upload_data_as_service_account_with_key_path (pandas.io.tests.test_gbq.TestToGBQIntegrationServiceAccountKeyPath) ... ok
pandas.io.tests.test_gbq.test_requirements ... ok
pandas.io.tests.test_gbq.test_generate_bq_schema_deprecated ... ok

----------------------------------------------------------------------
Ran 73 tests in 293.282s

OK
```

All tests passed on python 3.4, with exception of a warning caused by a deprecated gbq method

```
nosetests test_gbq.py -v
test_read_gbq_when_private_key_json_values_has_wrong_types_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_corrupted_private_key_json_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_empty_private_key_file_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_empty_private_key_json_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_invalid_private_key_json_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_read_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_booleans_as_python_booleans (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_floats_as_python_floats (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_integers_as_python_floats (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_strings_as_python_strings (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_return_bigquery_timestamps_as_numpy_datetime (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_that_parse_data_works_properly (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_to_gbq_should_fail_if_invalid_table_name_passed (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_to_gbq_with_no_project_id_given_should_fail (pandas.io.tests.test_gbq.GBQUnitTests) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorIntegration) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyContentsIntegration) ... ok
test_should_be_able_to_get_a_bigquery_service (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_results_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_schema_from_query (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_get_valid_credentials (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_should_be_able_to_make_a_connector (pandas.io.tests.test_gbq.TestGBQConnectorServiceAccountKeyPathIntegration) ... ok
test_bad_project_id (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_bad_table_name (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_column_order_plus_index (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_download_dataset_larger_than_200k_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_index_column (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_malformed_query (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_arbitrary_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_empty_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_false_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_null_timestamp (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_timestamp_unix_epoch (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_true_boolean (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_floats (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_integers (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_properly_handle_valid_strings (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_read_as_service_account_with_key_contents (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_should_read_as_service_account_with_key_path (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_unicode_string_conversion_and_normalization (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_zero_rows (pandas.io.tests.test_gbq.TestReadGBQIntegration) ... ok
test_create_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_create_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_dataset_does_not_exist (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_dataset_exists (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_delete_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_delete_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_generate_schema (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_google_upload_errors_should_raise_exception (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_dataset (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_table (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_list_table_zero_results (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_table_does_not_exist (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_append (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_fail (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_if_table_exists_replace (pandas.io.tests.test_gbq.TestToGBQIntegration) ... ok
test_upload_data_as_service_account_with_key_contents (pandas.io.tests.test_gbq.TestToGBQIntegrationServiceAccountKeyContents) ... ok
test_upload_data_as_service_account_with_key_path (pandas.io.tests.test_gbq.TestToGBQIntegrationServiceAccountKeyPath) ... ok
pandas.io.tests.test_gbq.test_requirements ... ok
pandas.io.tests.test_gbq.test_generate_bq_schema_deprecated ... FAIL

======================================================================
FAIL: pandas.io.tests.test_gbq.test_generate_bq_schema_deprecated
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tony/anaconda/envs/py34/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tony/pandas-gbq_service_account/pandas/io/tests/test_gbq.py", line 130, in test_generate_bq_schema_deprecated
    gbq.generate_bq_schema(df)
  File "/home/tony/anaconda/envs/py34/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/tony/pandas-gbq_service_account/pandas/util/testing.py", line 2067, in assert_produces_warning
    % extra_warnings)
AssertionError: Caused unexpected warning(s): ['ResourceWarning', 'ResourceWarning', 'ResourceWarning', 'ResourceWarning'].

----------------------------------------------------------------------
Ran 73 tests in 284.332s

FAILED (failures=1)
```

When I try to authenticate on the remote server without providing a service key or path, the code does not execute successfully (note it also currently does in the production pandas version). Should we add a note to the docs regarding this known issue?

When I press the stop button in jupyter, I see the following message:

```
Your browser has been opened to visit:
If your browser is on a different machine then exit and re-run this
application with the command-line parameter 

  --noauth_local_webserver
```

I think we should add some hint in that case, to use private_key. What do you think?

@parthea something ELSE is showing a deprecation warning if the test to specifically catching it is failing.

set the env variable `PANDAS_TESTING_MODE='deprecate'` to make these show up 

> I think we should add some hint in that case, to use private_key. What do you think?

I'm not sure if we can detect that the user is on a remote server. I think it will be sufficient to add a note to the docs. Something like:

```
.. note::

Authentication via a remote server is only supported with a service account. 
```

needs a rebase; most of the codebase underwent a PEP cleanup very recently.

- We need to update the `to_gbq()` definition in `frame.py` to also support the `'private_key'` parameter.

```
data_frame.to_gbq(DESTINATION_TABLE, project_id = PROJECT_ID, if_exists='append',  private_key = PRIVATE_KEY)

TypeError: to_gbq() got an unexpected keyword argument 'private_key'
```

@tworec just a couple of more stylistic changes to conform to the rest of the code base.

almost there!

gr8! all style fixes done. i'm running local tests and pushing it. please review.

All 73 tests have passed both on py3.5 and py2.7