BUG: fix read_gbq lost precision for longs above 2^53 and floats above 10k
State: closed

fixes:
- lost precision for longs above 2^53
- and floats above 10^4

Also contains  `test_gbq.py` clean up
- [x] closes #14020, #14305
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] updated docs
- [x] whatsnew entry


Labels:

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/14064?src=pr&el=h1) Report
> Merging [#14064](https://codecov.io/gh/pandas-dev/pandas/pull/14064?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/153da508a536a3ef203d9cd315c67b5fd3022a51?src=pr&el=desc) will **not impact** coverage by `-0.01%`.


```diff
@@            Coverage Diff             @@
##           master   #14064      +/-   ##
==========================================
- Coverage   86.32%   86.32%   -0.01%     
==========================================
  Files         141      141              
  Lines       51165    51170       +5     
==========================================
+ Hits        44169    44170       +1     
- Misses       6996     7000       +4
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/14064?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...788ccee2d77bff474e14224234f4f6f13ba75daa?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `17.21% <ø> (-0.19%)` | :x: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...788ccee2d77bff474e14224234f4f6f13ba75daa?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.36% <ø> (+0.33%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/14064?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/14064?src=pr&el=footer). Last update [153da50...788ccee](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...788ccee2d77bff474e14224234f4f6f13ba75daa?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
you are changing lots of things, but have minimal tests for this. pls add some.

is this meant to close #14020 ?

I didn't see #14020 before, but this PR seems to answer @aschmolck request 

needs more tests for datetimes

this PR is all about numeric precision loss. I'll restore my datetime improvement. It can be added as separate PR I think.

Ok, I've rebased and added full tests for this change (+ test clean up)

cc @parthea 

@jreback, @parthea please review

I think it might be better to force the user to specify a `dtype` keyword to override / set behavior for particular columns (like we do in `read_csv`; we sort of do this with `coerce_float` in `read_sql`.

@jorisvandenbossche 

I offer solution with default behavior. 
Specifing `dtype` is cool, but it's new feature which I do not want to focus on right now. This PR is ment to fix things up, not to put new features.

Can we stop here?

@jreback ping

@jreback ok, I've rebased, moved to 0.20.0 and added install page ref to gbq deps

@tworec Added some minor doc comments. 
But, we also still have to decide about object vs float (https://github.com/pandas-dev/pandas/pull/14064#discussion_r82159392), I think? I find it a bit strange that we would take here a different approach than in all other places in pandas.

@jorisvandenbossche I've corrected docs.

regarding float storage: my proposition is described in comment and implemented with tests
whats your proposition?

gentelmen @jreback @jorisvandenbossche :smiley: 
please decide if we go in current shape or we need to handle it in other way 

@tworec sorry we got a bit lost in the shuffle.

can you rebase and we can get this in.
@jreback ok, I've rebased
@parthea thoughts?
hi @jreback , can we finally merge this? Please decide if you want this change or drop it.
I was waiting for @parthea to come back on this.
oh, I c. We'll wait then. 
@parthea can you please check this out? :)
@parthea please review, because every time I look at this change it need manual rebase (conflicts).
@parthea my Travis CI setup is broken, it causes errors with pickles eg.
`ERROR: pandas.io.tests.test_pickle.TestPickle.test_pickles('0.14.0',)`

https://travis-ci.org/RTBHOUSE/pandas/jobs/199354963

Can you help?
@tworec I'm sorry for the delay. The latest commit looks good. 

Could you try running `git push --tags` on your master branch to see if that resolves the issue ?  Let me know if this works. I will submit a PR to update the BigQuery integration testing steps in the contributing docs to make it clear that you need to push tags to your master branch.
@parthea thx
@jreback can you merge it? :)


I've done it. Output is below. Why is this needed -- tags should not affect building right?
I guess this 141 commits is/was the problem. We'll see.

```
$ g co master 
Switched to branch 'master'
Your branch is ahead of 'origin/master' by 141 commits.
  (use "git push" to publish your local commits)

$ g ps --tags
To git@github.com:RTBHOUSE/pandas.git
 * [new tag]         v0.16.2 -> v0.16.2
 * [new tag]         v0.17.0 -> v0.17.0
 * [new tag]         v0.17.0rc1 -> v0.17.0rc1
 * [new tag]         v0.17.0rc2 -> v0.17.0rc2
 * [new tag]         v0.17.1 -> v0.17.1
 * [new tag]         v0.18.0 -> v0.18.0
 * [new tag]         v0.18.0rc1 -> v0.18.0rc1
 * [new tag]         v0.18.0rc2 -> v0.18.0rc2
 * [new tag]         v0.18.1 -> v0.18.1
 * [new tag]         v0.19.0 -> v0.19.0
 * [new tag]         v0.19.0rc1 -> v0.19.0rc1
 * [new tag]         v0.19.1 -> v0.19.1
 * [new tag]         v0.19.2 -> v0.19.2

```

i've squashed
@tworec when git clones a repo it doesn't clone the tags, so they are from *your* master whenever it was cloned, NOT the current one. This behavior baffles me, but that's how it works :<
ok, now my travis build works well
thx for explanation
thanks for the patience @tworec 

you can also put up a PR to: https://github.com/pydata/pandas-gbq (will need to take out the documentation changes, but outherwise should be clean).