BrandonFors/pandas-github-issues huggingface dataset:
ENH Add check for inferred compression before `get_filepath_or_buffer`
State: closed

When reading CSVs, if `compression='infer'`, check the input before calling `get_filepath_or_buffer` in the `_read` function. This way we can catch compresion extensions on S3 files. Partially resolves issue #11070 .

Checking for the file extension in the `_read` function should make the checks inside the parsers redundant. When I tried to remove them, however, I discovered that there's tests which assume the parsers can take an "infer" compression, so I left their checks.

I also discovered that the URL-reading code has a test which reads a URL ending in "gz" but which appears not to be gzip encoded, so this PR attempts to preserve its verdict in that case.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
pls change tests which are incorrect as well
iow this should cause them to fail 

I made this PR so that it didn't break any tests. Are the parsers ever accessed outside of the `_read` function? Do they need to be able to infer the compression type on their own? If not, I can remove that code from the parsers and change the tests.

the infer param can be moved higher up in the stack (eg in the get_filepath_or_buffer) - makes the readers simpler in that respect

Found it. I actually didn't need to change any tests. Now the only check for file extensions happens in the `_read` function, instead of separately inside each of the two parsers. By the time the parsers get called, any compression inference has already taken place.

gr8

Added a test using the new files in s3://pandas-test/.

Should I do anything else for this PR?

can you add a whatsnew note for this

pls rebase. ping when green.

@jreback , green! I found had to tweak `get_filepath_or_buffer` with an extra check for 'infer' compression. 

thanks!
ENH Enable bzip2 streaming for Python 3
State: closed

This is the one modification related to issue #11070 which affects non-S3 interactions with `read_csv`. The Python 3 standard library has an improved capability for handling bz2 compression, so a simple change will let `read_csv` stream bz2-compressed files.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO CSV
description: read_csv, to_csv

Comments:
tests!

I added a test for reading from an open file with the C parser. It fails on the master branch and passes here. How's that?

do you have exactly the same deps

Yes, exactly the same dependencies. This PR works because the standard library `bz2` module was upgraded to accept file pointers in 3.3.

ok, this looks good. pls add a note in whatsnew for 0.17.0 (just released the rc1 yesterday, but this is ok). reference both the original issue and this PR number I think.

squash & ping when green.

Note added. It doesn't look like anything else references a PR; should I leave that reference in?

@jreback , tests are green!

thanks!
ENH Recognize 's3n' and 's3a' as an S3 address
State: closed

This PR allows `read_csv` to recognize that "s3n://" designates a valid AWS S3 address. Partially addresses issue #11070 .


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Fixed. I also realized (reading the links I added for issue #11070 ) that "s3a" is a valid S3 designator as well, so I added that too.

The one test failure appears unrelated to the code change in this PR:

```
FAIL: test_numeric_column_names (pandas.io.tests.test_stata.TestStata)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/io/tests/test_stata.py", line 480, in test_numeric_column_names
    original.to_stata(path)
  File "/home/travis/miniconda/envs/pandas/lib/python3.3/contextlib.py", line 55, in __exit__
    next(self.gen)
  File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 2021, in assert_produces_warning
    % extra_warnings)
AssertionError: Caused unexpected warning(s): ['ResourceWarning'].
```

no it's just hitting the actual error there
you have something failing

add a whatsnew note in 0.17.0 (enhancements section)

@jreback , green here too!

thanks!
DOC/CI: include api docs on travis
State: closed

Continued in #12002

---

Closes #6100
Closes #3800

Just to check how this takes on travis


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: CI
description: Continuous Integration

Comments:
might simply want to create a doc-build (e.g. this issue https://github.com/pydata/pandas/issues/3800)

@jreback yes, I know, but I was first just experimenting a bit to get it working (not very familiar with travis/bash syntax details). Once I have something working the idea was indeed to put it in a separate travis build.

The problem I have now, is that for some reason `make.py` does not output its progress to the standard output of travis (which is the case if you run it locally in a terminal). And then, because of this, travis thinks nothing is happening anymore and aborts the build .. 
Do you know a solution for this?

http://docs.travis-ci.com/user/build-timeouts/

might work

is make.py writing to stderr? then could just redirect

maybe it's buffering?

https://coderwall.com/p/wws2uq/have-travis-ci-test-your-sphinx-docs

?

https://github.com/Syntaf/travis-sphinx

`make.py` calls `sphinx-build`, which AFAIU outputs to both stdout as stderr? And also, I would think travis outputs both stdout as stderr in the logs? But it is not really clear to me how it is specified the output of the doc build is logged in a file that is then later shown, and why for tests it is outputted directly.

The `travis_wait` is probably not a solution, as this just raises the timeout from 10 to 20 minutes, but I think the doc build will even take longer.

aha, hadn't notices that there is also a separate `run_build_docs.sh` script next to `build_docs.sh`, and that is of course capturing that output ..

@jreback With the last commit the full docs build OK, although it took 37 minutes. So I put it in a separate travis build, and include the full api docs (there is still a temporary commit to let the others fail).

Can you have a look at the travis ci changes? (it also included somes hacks to numpydoc to prevent some warnings, they are not very nice hacks, but the warnings had to be eliminated to get it running on travis (otherwise the log was too big ..)).

@jorisvandenbossche all looks cool to me then. I know the doc-build takes a while but so what its in the  lower section.

note - the builds run in the order that they are defined in the top section (no matter if they are in th ebottom section or not). So you prob want to have the doc-build run last? (I think you have it that way). That way other workers can finish the other jobs. we can have 5 simultaneously i think.

The problem with this approach is that the time it takes to build on Travis seems rather variable (maybe depending on its load?). In any case, sometimes it succeeded to finish it in ca 45 min, but other times it goes up to 50min and then travis aborts it. 
But now trying if parallel build speeds it up

@jorisvandenbossche closing, but reopen if updating

in theory this looks like a great idea....but yeh it takes quite some time to build the API docs

Continued in #12002
BUG: matplotlib inset-axis #10407
State: closed

This is the issue: https://github.com/pydata/pandas/issues/10407 
I just added a bunch of try-catch AttributeError around the offending code - it seemed to do the trick for me.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
this would need a test

Thanks, this is also a side effect of #9740. 

However, example in #10407 looks work on current master and matplotlib 1.4.3, maybe fixed by #10879. Can you show another example which causes a problem? Of course it is very nice to add an explicit tests for this.

@d1manson can you update according to comments

@d1manson pls update

Here is some information from pd.show_versions()
python: 3.4.0.final.0
pandas: 0.17.0
matplotlib: 1.4.3

I want to use add_axes instead of subplots.  The problem with subplots is that all subplot are the same height.  The advantage of add_axes is that subplots can be created easily with different heights.  For my use case I need  the first subplot to be smaller in height than subsequent subplots.

Here is a simple test case to reproduce the problem on in pandas 17.0:
df_test = pd.DataFrame([1,2,3,4]) # create DataFrame
fig_test = plt.figure() # create the figure instance

#add_axes : Add an axes at position rect [left, bottom, width, height]
#where all quantities are in fractions of figure width and height.
axes_a =  fig_test.add_axes([0.1, 0.1, 0.8, 0.4]) 
df_test.plot(ax = axes_a, title = 'Lower')
axes_b =  fig_test.add_axes([0.1, 0.6, 0.8, 0.2]) 
df_test.plot(ax = axes_b, title = 'Higher')

This causes error AttributeError: 'Axes' object has no attribute 'rowNum'

I really need this functionality - looks like I need to downgrade pandas.

I think this one is trying to solve the same underlying issue as what I tried to solve with https://github.com/pydata/pandas/pull/11561, which is now merged.

@patclaffey normally your example should work in the master version of pandas now.

@d1manson the approach I took in my fix was to just not call this function (`_handle_shared_axes`) when not having a SubplotAxes, instead of catching the AttributeError. Sorry I didn't see this before, but thanks for working on it!
Adds the option display.escape_notebook_repr_html
State: closed

Adds the option `display.escape_notebook_repr_html` for controlling HTML escaping when rendering DataFrames in IPython notebooks. Unset, this argument does not change the default behaviour.  

Fixes #11062


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
open to these types of things, but:
- use a `display.html` name space for these types of options. (or maybe `display.notebook`, but would have to rename some of the existing options possibly)
- I think we need a more general soln.

pls update

closing, but if you would like to continue working on this pls reopen
BUG: Fixed bug in groupby(axis=1) with filter() throws IndexError, #11041
State: closed

Fixed #11041 


Labels:
name: Bug
name: Groupby

Comments:
On green now. Can you merge?

merged via https://github.com/pydata/pandas/commit/ba5106ec753d820c1d61b330645b40bd43c97e62

thanks

note that we always to compare the results with an expected, and not just the index as this is a more general comparison.
Add capability to handle Path/LocalPath objects
State: closed

fixes #11033


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
ok, looks reasonable

the `pathlib` test should pass on 3.4 and skip on all others (pls confirm via the travis log)
however, pls add `py` to the requirements-2.7.txt so conda will install it (so it will test)

Docs would be great as well, in the docstring and in the `doc/source/io.rst`.

can you add `pathlib` to one of the py2 builds 
pls add a whatsnew note for 0.17.1

all done!

anaconda can’t install pathlib. wat do?

add to `ci/requirements-2.7.pip`

pls squash. otherwise lgtm.

squashed :+1: 

done, sorry for the holdup

merged via https://github.com/pydata/pandas/commit/82f0033369f417d0234990578c666947c50ee211

thanks!

:fireworks: yeah!
BUG: GH10645 and GH10692 where operation on large Index would error
State: closed

closes #10645
closes #10692 
replaces #10675 
Do I need `@slow` for these tests?


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
no need for slow these are < 1s total.

minor correction, ping on green

@kawochen thank you sir!
DOC: correct quoting constants order
State: closed

Small correction, QUOTE_NONE and QUOTE_NONNUMERIC have to be switched:

```
In [1]: import csv

In [2]: csv.QUOTE_NONNUMERIC
Out[2]: 2

In [3]: csv.QUOTE_NONE
Out[3]: 3
```


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
lgtm
BUG: Fix Timestamp __ne__ comparison issue
State: closed

closes #11034 


Labels:
name: Bug
name: Blocker
description: Blocking issue or pull request for an upcoming release

Comments:
merged via https://github.com/pydata/pandas/commit/ccaf040a35d6edadad59da019001c723c58af6f9

@cpcloud I add some more tests for completeness.

ty
BUG: Fixed bug in len(DataFrame.groupby) causing IndexError when there's a NaN-only column (issue11016)
State: closed

This is the new PR for Fixed issue #11016 


Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
generally we don't make _new_ PR's. just `push -f` the old ones. ok for now. but pls have a look at:

http://pandas.pydata.org/pandas-docs/stable/contributing.html

@jreback Got it.  I will take a closer look at it. 

So when I run `nosetests pandas/tests/test_groupby.py`, it gave me a lot of errors. This is on master branch from recent pydata/pandas. I think the errors are from recent commits rather than this PR. 

you need to rebuild the c extensions
after u pull from master do this

merged via https://github.com/pydata/pandas/commit/066783c6ea16115d2cbc75dca443e973e81ad754

thanks

Thanks for fixing!
BUG: Fixed bug in len(DataFrame.groupby) causing IndexError when there's a NaN-only column (issue11016)
State: closed

Fixed #11016 


Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@jreback Could you help me with this conflict? 

don't put the whatsnew note at the very end of bug fixes. stick it where their is blank space. at the end always causes conflicts, much much more rare when its not.

Thanks. I had trouble rebasing it. It has conflict that I failed to fix. I've created a new PR. Sorry for inconvenience.
Fix for DataFrame.hist() with by- and weights-keyword
State: closed

closes https://github.com/pydata/pandas/issues/9540

for example:

``` python
import pandas as pd
d = {'one' : ['A', 'A', 'B', 'C'],
     'two' : [4., 3., 2., 1.],
     'three' : [10., 8., 5., 7.]}     
df = pd.DataFrame(d)
df.hist('two', by='one', weights='three', bins=range(0, 10))
```

does not seem to break anything, but this is my first meddling in the pandas library, so a review would be nice


Labels:
name: Visualization
description: plotting

Comments:
Just gave a quick look through here. This is a good idea that we should support.

We'll need tests for this. Put them in `pandas/tests/test_graphics.py`. Make sure to cover cases where
- `by` is None or a column
- `weights` is an array of weights or a string column
- `values` has missing data and / or `weights` has missing data

The plotting stuff is being refactored a bunch currently, so I've tagged this for the 0.18. You might want to hold off on making more changes, but in the meantime you can write tests for this.

FYI the checking is quite similar to how weights are checked for `DataFrame.sample`, so would want to make this a common function (could be a private function on a dataframe)

Thanks jreback, I will try my best to add some tests. I will also check what can be done with synching dropna with the weighs.

I was thinking in the lines of
drow rows in weighs, that are na in group
drop rows na in group
fillna in weighs with zero, so they do not count to anything

would this be ok?

Or more logical to drop all rows that are NA in group or in weights?

I'd say your lest method. Something like `data.dropna(subset=['group', 'weight'])`

@TomAugspurger can you review

Sorry I have not had time to continue with the tests, my limited git knowledge made it tough interacting with the repo. I planned to look how my patch worked with the new release

Added a new pull request here: https://github.com/pydata/pandas/pull/11441

for the new version, sorry for not using git correctly :(

Superseded by #11441
BUG: DatetimeIndex.freq can defer by ufunc
State: closed

Follow up of #10638. `DatetimeIndex.freq` also can be changed when other arg is an ndarray.

As the same as #10638, freq is re-infered only when it exists. Otherwise, leave it as `None` even if the result can be on specific freq. This is based on other functions behavior as below:

```
idx = pd.DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03'])
idx
# DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03'], dtype='datetime64[ns]', freq=None)
idx.take([0, 2])
# DatetimeIndex(['2011-01-01', '2011-01-03'], dtype='datetime64[ns]', freq=None)

idx = pd.DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03'], freq='D')
idx.take([0, 2])
# DatetimeIndex(['2011-01-01', '2011-01-03'], dtype='datetime64[ns]', freq='2D')
```


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
lgtm. merge on green.
PERF: use NaT comparisons in int64/datetimelikes #11010
State: closed

closes #11010 

```
In [10]: %timeit gr['3rd'].count()
100 loops, best of 3: 15.1 ms per loop

In [11]: %timeit gr['3rd'].min()
100 loops, best of 3: 8.4 ms per loop

In [12]: %timeit gr['3rd'].max()
100 loops, best of 3: 8.47 ms per loop

In [13]: %timeit gr['3rd'].first()
100 loops, best of 3: 10.5 ms per loop

In [14]: %timeit gr['3rd'].last()
100 loops, best of 3: 10.3 ms per loop
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
cc @larvian 
cc @behzadnouri
BUG: Exception when setting a major- or minor-axis slice of a Panel with RHS a DataFrame
State: closed

Fixes GH #11014

This isn't actually a recent regression. The bug was introduced in 0.15.0 with this commit: https://github.com/pydata/pandas/commit/30246a7d2b113b7115fd0708655ae8e1c3f2e6c5; it was just made more common (i.e., it happens on panels without mixed dtype) in 0.16.2 by this commit: https://github.com/pydata/pandas/commit/5c394677e10dac38fe3375e2564bb76cf8bbffca.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@evanpw gr8 thanks!
BUG: Unable to infer negative freq
State: closed

Fixed 2 issues:
- `date_range` can't handle negative calendar-based freq (like `A`, `Q` and `M`).

```
import pandas as pd

# OK
pd.date_range('2011-01-01', freq='-1D', periods=3)
# DatetimeIndex(['2011-01-01', '2010-12-31', '2010-12-30'], dtype='datetime64[ns]', freq='-1D')

# NG
pd.date_range('2011-01-01', freq='-1M', periods=3)
# DatetimeIndex([], dtype='datetime64[ns]', freq='-1M')
```
- Unable to infer negative freq.

```
pd.DatetimeIndex(['2011-01-05', '2011-01-03', '2011-01-01'], freq='infer')
# DatetimeIndex(['2011-01-05', '2011-01-03', '2011-01-01'], dtype='datetime64[ns]', freq=None)

pd.TimedeltaIndex(['-1 days', '-3 days', '-5 days'], freq='infer')
# TimedeltaIndex(['-1 days', '-3 days', '-5 days'], dtype='timedelta64[ns]', freq=None)
```


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
thanks @sinhrks
BUG: Index dtype may not be applied properly
State: closed

Fixed 2 problems:
- Specified `dtype` is not applied to other iterables.

```
import numpy as np
import pandas as pd

pd.Index([1, 2, 3], dtype=int)
# Index([1, 2, 3], dtype='object')
```
- Specifying `category` to ndarray-like results in `TypeError`

```
pd.Index(np.array([1, 2, 3]), dtype='category')
# TypeError: data type "category" not understood
```


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
pls run perf check on this - 

@sinhrks can you see if this affects perf?

Following is a asv result. Will consider a better path.

```
    before     after       ratio
  [789e07d9] [ce89f9d1]
    33.94μs    44.57μs      1.31  ctors.index_from_series_ctor.time_index_from_series_ctor
   121.86μs   173.21μs      1.42  frame_methods.frame_get_dtype_counts.time_frame_get_dtype_co
    16.45μs    27.07μs      1.65  index_object.index_float64_construct.time_index_float64_construct
```

this is prob just from the imports (e.g. a Float64Index) doesn't care about Datetimeindex, so the import adds the extra time (or the check for the import anyhow)

@jreback Thanks, suggested changes improve perf a little. I assume other slowness is caused by `categorical` condition moved to the top.

```
All benchmarks:

    before     after       ratio
  [4d4a2e33] [c3eaeb07]
    33.85μs    41.24μs      1.22  ctors.index_from_series_ctor.time_index_from_series_ctor
   129.48μs   160.86μs      1.24  frame_methods.frame_get_dtype_counts.time_frame_get_dtype_counts
    16.36μs    22.88μs      1.40  index_object.index_float64_construct.time_index_float64_construct
```

master (this is `index_object.index_float64_construct.time_index_float64_construct`) benchmark

```
In [1]: arr = np.arange(1000000.0)
In [2]: %timeit Index(arr)
The slowest run took 12.23 times longer than the fastest. This could mean that an intermediate result is being cached 
100000 loops, best of 3: 15.7 Âµs per loop
```

this branch

```
In [2]: %timeit Index(arr)
The slowest run took 7.49 times longer than the fastest. This could mean that an intermediate result is being cached 
10000 loops, best of 3: 19.8 Âµs per loop
```

fix https://github.com/jreback/pandas/commit/29c03255978e81db2d0149f25e5412949eeddf39

```
In [2]: %timeit Index(arr)
The slowest run took 11.74 times longer than the fastest. This could mean that an intermediate result is being cached 
100000 loops, best of 3: 13.8 Âµs per loop
```

will merge in a bit, thanks @sinhrks 

Wow, great. Thanks!

merged via https://github.com/pydata/pandas/commit/ead3ca85b42e6d12855acecd347ef7cc94792c9d (my change in another commit)

thanks!

I don't believe we had an issue assosicated, correct?

Though this was based on gitter chat, #5196 refers to the same issue. Closed.

@sinhrks awesome thanks!
BUG: .ix with PeriodIndex is fixed
State: closed

Solves https://github.com/pydata/pandas/issues/4125

It doesn't feel like an elegant solution, but I think it works. 
I think when Period gets its own dtype, these will all go away?


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Period
description: Period data type

Comments:
pls add a whatsnew note

merged via https://github.com/pydata/pandas/commit/879f5e90bea3b8c4849fd9f83a834e858dccd407

thanks!

(I changed this slightly in another commit). `is_array_period_like` does inference on the object and thus could be slow potentially

Cheers @jreback!
DOC: Docstring Redesign to fix the problem of unexpected keyword arg (issue10888)
State: closed

Fixed issue #10888 


Labels:
name: Docs

Comments:
@jreback  Is this one ok to merge? 

needs tests

So this is simply a DOC problem where users use the wrong parameters specified in Documentation but they don't belong to those methods. I've fixed the documentation so users won't get confused again. I don't think unit tests are needed for this PR. 

can you post before-after of the doc-strings

Apart from the few comments, looks good!

@jreback @jorisvandenbossche Thanks! They look better now! 

@jreback Thanks. Anything else I should change before I add a new burden to Travis?

thanks!
ENH: add Series.astype with the new tz dtype
State: closed

add ability to astype using the new dtype


Labels:
name: Enhancement
name: Docs

Comments:
@jreback I am not really sure about this. For me, it is not very clear if this would be doing a tz_convert or a tz_localize. 

And since we already have `tz_convert` and  `tz_localize`, I don't really see the added value of advertising the use of `astype('datetime64[ns, tz=...]')` as an alternative

@jorisvandenbossche is just for compat really. I think people would try to do it. its actually very natural once you try it! (think about tz as part of the dtype, then this makes perfect sense).

and we allow `s_aware.astype('datetime64[ns]')` so this is just the other direction