BrandonFors/pandas-github-issues huggingface dataset:
BUG/ERR: raise correct error when sql driver is not installed
State: closed

Closes #14513

When the driver was not installed, but sqlalchemy itself was, when passing a URI string, you got an error indicating that SQLAlchemy was not installed, instead of the driver not being installed. This was because the import error for the driver was captured as import error for sqlalchemy.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14527?src=pr) is 85.26% (diff: 100%)

> No coverage report found for **master** at 096d886.
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last update [096d886...67f6189](https://codecov.io/gh/pandas-dev/pandas/compare/096d8866a90c8cbb44ab8243320e811fc24190bd...67f61896d8606673b23f1c0d0505c54cd05c732b?src=pr)
ENH: Added optional pickle protocol version argument to pandas.to_pickle()
State: closed

- [ ] closes #14488 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

… :issue:'14488'


Labels:
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
can you rebase
closing as stale, but if you would like to update, pls comment.
If the original author @beverast lost interest in the PR, I would like to take over.
I am also interested in having an optional parameter `protocol` to the `to_pickle` function.
cc @jreback 
sure u can start with the commits here if u like 
open a new PR
BUG: don't close user-provided file handles in C parser (GH14418)
State: closed

Closes #14418


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14520?src=pr) is 85.26% (diff: 100%)

> No coverage report found for **master** at 096d886.
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last update [096d886...73510b6](https://codecov.io/gh/pandas-dev/pandas/compare/096d8866a90c8cbb44ab8243320e811fc24190bd...73510b68589a66f62f3cf3d3723042334908186d?src=pr)
MAINT: Expand lint for *.py
State: closed

Expand lint to include all Python files, excluding the directories `pandas/rpy` (deprecated and soon to be removed) and `pandas/src` (C code).


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
are flake8 options not strict enough? IOW why were these not caught before?

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14516?src=pr) is 85.26% (diff: 91.66%)

> Merging [#14516](https://codecov.io/gh/pandas-dev/pandas/pull/14516?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14516   diff @@
==========================================
  Files           140        140          
  Lines         50672      50672          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43207      43207          
  Misses         7465       7465          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [d7fb5bd...e7429aa](https://codecov.io/gh/pandas-dev/pandas/compare/d7fb5bd310edcab9875c1f9339b62e92baae8291...e7429aafce4d3136d71bdbbb60caf68792b0b520?src=pr)

@jreback : TBH, I have no idea.  I didn't really change anything with the command flags.  But if it works, that's work counts I guess. :smile: 

@jreback : Is this good to merge since Travis is passing?

@jorisvandenbossche : Any reason why you moved it to 0.20.0?  This isn't an API change.

It doesn't matter to merge now or not, that just indicates that it is not essential to backport

Ah, okay.  Good to know!

@gfyoung thanks!
BUG: Don't parse inline quotes in skipped lines
State: closed

Title is self-explanatory.  Closes #14459.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
@gfyoung Thanks a lot! 
Looks good to me. 

Do we have benchmarks for skiprows?

@jorisvandenbossche : Yes, we do in `io_bench.py`.  I didn't see any major performance hits.

OK, thanks.

@jreback or @chris-b1 Can somebody else give a look as well?

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14514?src=pr) is 85.26% (diff: 100%)

> Merging [#14514](https://codecov.io/gh/pandas-dev/pandas/pull/14514?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14514   diff @@
==========================================
  Files           140        140          
  Lines         50672      50672          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43208      43208          
  Misses         7464       7464          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [7f5a45c...2e41dab](https://codecov.io/gh/pandas-dev/pandas/compare/7f5a45c1b388b3f7f309f82bfa0733b7b9980c3a...2e41dab22b1c96583b618d9102c2feb7260a4aae?src=pr)

lgtm.

@gfyoung can you just do a run of the csv asv's just for a quick check.

@jreback : As I alluded to <a href="https://github.com/pandas-dev/pandas/pull/14514#issuecomment-256664268">above</a>, I saw no major performance hits for the `csv` benchmarks.

@gfyoung Thanks!
ENH : First attempt to add sparse_pivot
State: closed

- [ ] Resolves #14493 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

``` python
sparse_series = pivot_sparse(index=df['a'], columns=df['b'], values=df['c'])

```


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
this would need to be an actual sparse pivot implementation. converting to dense then pivotting doesn't really work here. If you have some thoughts on that, pls reopen / comment.
hmm.. i didn't convert to dense though...

the idea is just transforming the index instead of data/values.   it's like keep the data in a series and update the new index which is after pivot
CI: add conda-forge as lowest priority channel
State: closed

xref #14495 


Labels:
name: CI
description: Continuous Integration

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14509?src=pr) is 85.27% (diff: 100%)

> Merging [#14509](https://codecov.io/gh/pandas-dev/pandas/pull/14509?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14509   diff @@
==========================================
  Files           140        140          
  Lines         50689      50689          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43225      43225          
  Misses         7464       7464          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [1d95179...a8e60c2](https://codecov.io/gh/pandas-dev/pandas/compare/1d951794187b72841d59b3dfde0a98309a64dec7...a8e60c2fa6716974e8ec493ab57fd766cf9bfeea?src=pr)

@jreback The failing test for the python 3.4 build is due to a very old version of sqlalchemy that is installed (0.7.1). Not sure where this is coming from, as the requirements file do not specify a version.

Ah, it's coming from the `pandas` conda channel. I suppose the conda-forge channel had priority over that before. 
Is there a reason you want conda-forge as the lowest priority?

@jorisvandenbossche yeah maybe I should just take this out of the pandas channel. The idea with conda-forge is that we can pick up things that are _not_ in the default channel (quicker), but if they are already there, use them. The reasoning is that most people are using _defaults_, so we want to be exposed to that pipeline first.

The case here is `bottleneck==1.2.0` which is only on conda-forge (ATM).

That sounds good as a reasoning. 
Are there actually still packages in the pandas channel we need? I suppose eg `psycopg2` is in the meantime also available in either defaults or conda-forge

Is specifying a version in a requirements file enough to trigger it to get it from conda-forge?

https://github.com/conda-forge/bottleneck-feedstock/pull/7 is merged in the meantime, so you can specify 1.2.0 for one of the builds?

yeah I need to add the new version on a build (unfort have to manually specify it).

yes, can prob remove the need for the pandas channel entirely (if we have conda-forge), or maybe will add that one _after_. will try later

closing for now. have better ways to do this esp when you need to specify more things than a channel, e.g. 34_SLOW build. / 3.6 build
BUG: to_numeric downcast = 'unsigned' would not un-sign a 0 value.
State: closed

- [ ] closes #14401
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Have to figure out whats going wrong with the build still. Will let you know as soon as I do.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can you rebase this

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14504?src=pr) is 85.28% (diff: 100%)

> Merging [#14504](https://codecov.io/gh/pandas-dev/pandas/pull/14504?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14504   diff @@
==========================================
  Files           140        140          
  Lines         50693      50693          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43235      43235          
  Misses         7458       7458          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [726efc7...07264cd](https://codecov.io/gh/pandas-dev/pandas/compare/726efc7cd55744eb88636dd6f11293f18355a10a...07264cd9bfb3b49b8a1b9769991a155c7b92cdde?src=pr)

@verhalenn There is still a test failing. Do you have time to look into this?

@jorisvandenbossche I have been but I'm having trouble replicating it.

@verhalenn Apparently this is a bug in older numpy versions:

```
>>> np.__version__
'1.8.2'
>>> arr = np.array([-128, 127])
>>> arr2 = arr.astype('int8')
>>> np.allclose(arr, arr2)
False
```

```
>>> np.__version__
'1.11.2'
>>> arr = np.array([-128, 127])
>>> arr2 = arr.astype('int8')
>>> np.allclose(arr, arr2)
True
```

So you can skip the failing test for numpy versions of 1.8 or lower.

That should do. If there is anything you would like different, let me know.

Also thanks @jorisvandenbossche for the help.

@verhalenn See here for an example how to skip a test for a numpy version: https://github.com/pandas-dev/pandas/blob/52f31d470d779204e4c1388cdb56351c68332c3f/pandas/tests/series/test_quantile.py#L76

Can you also move the whatsnew note to the 0.19.2 file ?

thanks!
TST: simplify tests for GH14346
State: closed

Addresses my remaining comment from https://github.com/pandas-dev/pandas/pull/14346#issuecomment-255520873


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14502?src=pr) is 85.26% (diff: 100%)

> Merging [#14502](https://codecov.io/gh/pandas-dev/pandas/pull/14502?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14502   diff @@
==========================================
  Files           140        140          
  Lines         50670      50670          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43205      43206     +1   
+ Misses         7465       7464     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [e3d943d...061484b](https://codecov.io/gh/pandas-dev/pandas/compare/e3d943d1876e8914036e9c323f1876e52da34a04...061484bd3802a967e750aa9ba897ca15d7bee0cd?src=pr)

lgtm. you _might_ even want to put them in `TestDatetimeLike` (and the index is generic), but might be slightly complicated / not worth it. 

Yeah, the fact that the test is slightly different (freq should be None or not) will indeed make it more complicated I think
BUG: fix DatetimeIndex._maybe_cast_slice_bound for empty index (GH14354)
State: closed

Closes #14354


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14501?src=pr) is 85.26% (diff: 100%)

> Merging [#14501](https://codecov.io/gh/pandas-dev/pandas/pull/14501?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14501   diff @@
==========================================
  Files           140        140          
  Lines         50670      50670          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43205      43206     +1   
+ Misses         7465       7464     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [e3d943d...304ca64](https://codecov.io/gh/pandas-dev/pandas/compare/e3d943d1876e8914036e9c323f1876e52da34a04...304ca6419d86c9e97ac290ef208fe9906ecf9a6e?src=pr)

lgtm. (though travis is failing)
COMPAT/TST: fix test for range testing of negative integers to neg powers
State: closed

xref https://github.com/numpy/numpy/pull/8127

closes #14489


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14498?src=pr) is 85.26% (diff: 100%)

> Merging [#14498](https://codecov.io/gh/pandas-dev/pandas/pull/14498?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14498   diff @@
==========================================
  Files           140        140          
  Lines         50670      50670          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43205      43205          
  Misses         7465       7465          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [e3d943d...882872e](https://codecov.io/gh/pandas-dev/pandas/compare/e3d943d1876e8914036e9c323f1876e52da34a04...882872e64ff7c4a50e54f1fb1a14dcad911633ab?src=pr)

FWIW I think this one is not complete as for resolving the issue with numpy beta starting to  error on negative powers for ints, see my reply on numpy mailing list:  https://www.mail-archive.com/numpy-discussion@scipy.org/msg51230.html
@yarikoptic I believe you are using a 32-bit machine, which we have talked about before, would accept patches (IIRC you have an open PR for just that). but we don't test regularly on that.
Details escaped me but I think this issue was not solely 32bit-related. Unfortunately discussion on the numpy-discussion mailing list (https://www.mail-archive.com/numpy-discussion@scipy.org/msg51233.html) died off without resolution AFAIK.

And yes -- there is a still open https://github.com/pandas-dev/pandas/pull/15044  ;-)
Cheers
TST: do not use int to negative int power in tests (GH14489)
State: closed

Closes #14489

Numpy starts to error on those cases from 1.12 https://github.com/numpy/numpy/pull/8127

This just fixes the test for now, but other option is to coerce to floats (but that is certainly not for 0.19.1)


Labels:

Comments:
hah I think we duped on this one.......

Yep, I also just see it :-) Yours is cleaner, mine still tests negative ints to positive powers

hahah, ok.
Use old Cython build_ext until includes are fixed.
State: closed

This change in Cython 0.25

The distutils extension Cython.Distutils.build_ext has now been updated to use cythonize which properly handles dependencies. The old extension can still be found in Cython.Distutils.old_build_ext and is now deprecated.

seems to have broken the include of auto-generated algos_common_helper.pxi. As a temporary workaround, the old build_ext can be used.


Labels:
name: Build
description: Library building on various platforms

Comments:
It looks like the difficulties are due to
1. The new `Cython.Distutils.build_ext` runs `cythonize()` in `finalize_options()`, before the `.pxi` files are generated, and
2. The extension objects use the `include_dirs` property, which is only inspected by the old build_ext. 

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14496?src=pr) is 85.26% (diff: 100%)

> Merging [#14496](https://codecov.io/gh/pandas-dev/pandas/pull/14496?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14496   diff @@
==========================================
  Files           140        140          
  Lines         50667      50670     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43203      43205     +2   
- Misses         7464       7465     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [d1d75d7...3d977c3](https://codecov.io/gh/pandas-dev/pandas/compare/d1d75d7fcb6b5d090cf96d328d0136c3959a82b8...3d977c3093f486aae36da99017f1009454a56a7d?src=pr)

lgtm. can you add a release note, say compat with cython 0.25 builds.

was just about to say this isn't tested on travis, but cython JUST updated and broke our builds :>

so going to merge.

thanks!

I fixed the 3.4 build on the most recent cython. https://github.com/pandas-dev/pandas/commit/6ac759d5e24f8a1b8eb9f39f08b139079cad401e

If you would like to submit a PR to actually fix the way cython builds for >= 0.25 would be great! (need compat of course for older cythons). We still support >= 0.19.1, but should prob move that up.
BUG: don't allow users to move from an interned string
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

I was thinking about code paths that give away borrowed references to strings and realized that interned strings can be accessed even when their refcount is 1. This is because the intern table holds a "weak ref" to the string. This allows the string to be garbage collected normally; however, if a user creates an interned string with exactly one reference and feeds it to move, you could later get a second reference to the **same** string object by calling intern with a copy of the old string. The data backing the old string may have been mutated so you will get very strange results. I am almost certain that no on has run into this since `move_into_mutable_buffer` was added but it doesn't hurt to add this quick check to prevent broken usage like this in the future.

This only affects python 2 because bytes objects cannot be interned in python 3.


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
I read somewhere that PY3 _only_ interns short strings. Is that not right?

how did you find this issue (out of curiosity)? IOW, what broke?

I am not sure what gets interned by default, but users may call `sys.intern` on any string to intern it I believe. This is not a concern for `move` in py3 because strings are unicode type, and this function only accepts bytes type. I found this by thinking about ways to mutate string objects with tools like `np.frombuffer` and thought that I could try to craft bad inputs to this which would give me a mutable string. Nothing broke, this was just the result of some experiments.

Also, it looks like only one of the py3.4 builds failed:
`FAIL: test_downcast_limits (pandas.tools.tests.test_util.TestToNumeric)`

I am not sure how my change would have affected this, and only in the particular configuration. Does anyone have any ideas?

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14494?src=pr) is 85.28% (diff: 100%)

> Merging [#14494](https://codecov.io/gh/pandas-dev/pandas/pull/14494?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14494   diff @@
==========================================
  Files           140        140          
  Lines         50693      50693          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43234      43234          
  Misses         7459       7459          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [1d6dbb4...ae8aa64](https://codecov.io/gh/pandas-dev/pandas/compare/1d6dbb41b26a39121ec8c4f19f5da78bb0ab4af7...ae8aa648c8a8c388af45dc1d8be6e11680dfc001?src=pr)

this looks fine. wait till we merge #14631 then push again to force rebuilds.

Tests are now passing after a rebase

ping

thanks!

Thank you!