solve "Int64 with null value mangles large-ish integers" problem
State: closed

- [x] closes #30268 
- [ ] tests added/passed
- [x] passes black pandas
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [x] whatsnew entry
The `pd.Series` was giving less precise output when inputting a list containing large integers and np.nan values and passing `dtype="Int64"`. Which was due to converting it to `np.array`. This PR aims to solve that bug.

Labels:
name: Bug
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@jorisvandenbossche 
I have added conditions for `np.ndarray` and `list` as you told in the issue. Please tell me which other datatypes should I add as conditions and tell me if any other changes are needed.
@rushabh-v Thanks for the PR. can you add a whatsnew and test.
The Tests are failing.
It says `TypeError: object cannot be converted to an IntegerDtype`.
[see here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=23169&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=9365e6ae-bf62-55c3-8ba1-d11b71b622f9&l=110).
@rushabh-v you will need to accept "boolean" as well as inferred dtype
Yes, I'll add tests once the checks pass and I have added the whatsnew note above(If it should be somewhere else then please tell me.). @TomAugspurger.
Oh Okay, I am adding tests.
Hello @rushabh-v! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-30 18:54:09 UTC
@rushabh-v This may also close #26259?
> This may also close #26259?

Yes It's related to it. But I am not understanding why
`pd.read_csv(t, dtype={'timestamp': object})` is calling `integer_array`,
 whereas `pd.read_csv(t, dtype={'timestamp': pd.Int64Dtype()})` is not calling `integer_array`. 

I have checked this locally.
The tests are failing. It says,
```
AssertionError: Attributes of Series are different           
           Attribute "dtype" are different
           [left]:  bool
           [right]: Int64
```
[see here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=23393&view=logs&j=a67b4c4c-cd2e-5e3c-a361-de73ac9c05f9&t=a98a35ac-8614-5ad9-3f72-a150628735a8&l=67).
What mistake have I made while adding whatsnew?
It says "These conflicts are too complex to resolve in the web editor".
There's a merge conflict (you edited the same line as another pull request). You'll need to fix that as described in https://dev.pandas.io/docs/development/contributing.html#updating-your-pull-request.
>The tests are failing. It says,
AssertionError: Attributes of Series are different           
           Attribute "dtype" are different
           [left]:  bool
           [right]: Int64
see here.

They have started appearing after  [adding "boolean" as inffered type](https://github.com/pandas-dev/pandas/pull/30282/commits/a9479ccd14a67b50e75c1700c9faca13c1ddf4d0).
Only CI checks are failing!
It says,
```
In linting:
    Check import format using isort
    ##[error]Skipped 2 files


and in Publish benchmarks artifact
    Run actions/upload-artifact@master
    ##[error]Path does not exist /home/runner/work/pandas/pandas/asv_bench/benchmarks.log
    ##[error]Exit code 1 returned from process: file name 
    '/home/runner/runners/2.163.1/bin/Runner.PluginHost', arguments 'action 
    "GitHub.Runner.Plugins.Artifact.PublishArtifact, Runner.Plugins"'.
```
Can somebody please guide me to get rid of these failing CI checks? I have written the error message above.
You need to run black on the files specified here

https://github.com/pandas-dev/pandas/pull/30282/checks?check_run_id=363015447#step:6:10
>You need to run black on the files specified here

Thanks @WillAyd 
It's giving `ImportError: Cannot import name ExtensionArray`. I think I haven't touched anything wrong there. [Find logs here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=24090&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=9365e6ae-bf62-55c3-8ba1-d11b71b622f9&l=29)
All greens!
@rushabh-v thanks for the link where tests are failing. I have been looking a bit more into it, and discovered several issues related to this: https://github.com/pandas-dev/pandas/issues/31108, https://github.com/pandas-dev/pandas/issues/31104, https://github.com/pandas-dev/pandas/issues/31102

I am not yet fully sure how to solve it though. But we need to find a better way than the specific dtype checks you now did in https://github.com/pandas-dev/pandas/pull/30282/files#r361776673
>@rushabh-v thanks for the link where tests are failing. I have been looking a bit more into it, and discovered several issues related to this: #31108, #31104, #31102

Oh, thanks for informing that. I was also wondering why it was failing only due to the changes in this PR.
>@rushabh-v thanks for the link where tests are failing.

@jorisvandenbossche Can we ignore those tests for this PR, Because the error is general and not caused by this PR? Or we should wait for  #31108 to be fixed?
> > @rushabh-v thanks for the link where tests are failing.
> 
> @jorisvandenbossche Can we ignore those tests for this PR, Because the error is general and not caused by this PR? Or we should wait for #31108 to be fixed?

no cannot ignore tests.
np.asarray raises `TypeError: data type not understood` when dtypes are pandas dtypes(i.e, Int8Dtype(),Int16Dtype(), Int64Dtype(), UInt8Dtype(), etc).
See,
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=26801&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9&l=516
@rushabh-v IIUC can just map to the numpy equivalent dtype before the asarray call
@rushabh-v can you address comments and try to get green?
I think this is still blocked by the issues I mentioned in https://github.com/pandas-dev/pandas/pull/30282#issuecomment-575686004 ? 
I will try to take a look at those the coming week.
@rushabh-v i am not sure what the status of this as we have had many moving parts under the hood. can you merge master and update.
@rushabh-v is this still active?