pandas.io.gbq Version 2
State: closed

closes #5840 (as new interface makes it obsolete)
closes #6096 

@jreback : We still have some documentation to work on, but we would like your initial thoughts on what we have so far. The key change for this version is the removal of bq.py as a dependency (except as a setup method for a test case). Instead, we rely entirely on the BigQuery python API. We also simplified to_gbq() significantly. Though it cost a few features, the code is much more manageable and less error prone (thanks Google!). Test cases are much more granular and run significantly faster. To use the test cases fully, a BigQuery project_id is still required, though there are some unittests offline. 


Labels:

Comments:
can u rebase on master?

also flip github switch for Travis

I rebased to master, and Travis was turned on. I'm still working to get Travis passing - mostly just a matter of getting the right dependencies in place and being sure integration tests are skipped.

this going to close #5840 ?

@jreback : I believe #5840 was no longer an issue due to a backend change with bigquery. However, it's no longer an issue with our code, since we track the page tokens we've seen so far and ensure we don't get a duplicate.

@jacobschaer you are doing small doc change to close #6096 as well?

@jacobschaer like to get this in soon...how's coming?

@jacobschaer how's this coming?

@jacobschaer pls rebase on master

OK, all rebased and mostly squashed. Were there any documentation things we're missing?

@jacobschaer 

the 3.4 and numpy-dev builds are failing because they are not installing `httplib2` which defacto is a requirement of gbq now. those tests should be skipping (because of the import failure).

those builds need to pass (they are experimental just because the build matrix comes back faster)

@jreback Seems 3.4 and numpy-dev builds now pass.

@jacobschaer pls squash down to a small number of commits (and remove merge branches), and rebase on master

@jacobschaer can you rebase and squash?

All rebased and squashed. 

@jacobschaer thanks for this!

pls review the docs when they are built in case a followup is needed. http://pandas-docs.github.io/pandas-docs-travis/

@jacobschaer If you have a chance. Can you break the big paragraph into say bullet points:

```
Finally, you can append data to a BigQuery table from a pandas DataFrame using the to_gbq() 
function. This function uses the Google streaming API which requires that your destination table exists in
 BigQuery. Given the BigQuery table already exists, your DataFrame should match the destination table 
in column order, structure, and data types. DataFrame indexes are not supported. By default, rows are 
streamed to BigQuery in chunks of 10,000 rows, but you can pass other chuck values via the chunksize 
argument. You can also see the progess of your post via the verbose flag which defaults to True. The 
http response code of Google BigQuery can be successful (200) even if the append failed. For this 
reason, if there is a failure to append to the table, the complete error response from BigQuery is 
returned which can be quite long given it provides a status for each row. You may want to start with 
smaller chuncks to test that the size and types of your dataframe match your destination table to make 
debugging simpler.
```

http://pandas-docs.github.io/pandas-docs-travis/io.html#io-bigquery