BrandonFors/pandas-github-issues huggingface dataset:
Moved caching in AbstractHolidayCalendar to the instance level
State: closed

Closes #9552


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
![image](https://cloud.githubusercontent.com/assets/881019/6389208/060f4ecc-bd9a-11e4-8e20-6af34fb90c3d.png)

@dhirschfeld this looks good. Can you add a release note in Bug fixes, and squash. ping me when pushed.

@jreback - added a comment in `whatsnew/v0.16.1.txt`. Let me know if that's ok.

@dhirschfeld looks good. pls ping when green.

@jreback - looks like travis finally got there in the end!

@jreback - rebased after conflicts in `whatsnew/v0.16.1.txt`. Will ping when travis says ok

thanks @dhirschfeld
`read_hdf` opens the store in `rw` mode by default.
State: closed

Currently the `HDFStore` in `read_hdf` is opened in `rw` mode by default, leading to permission
errors when reading a non-writeable file.

This patch changes the default mode to `r`.


Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
This seems like a big change since any code like 

```
store = pd.HDFStore('my_store.h5')
store.put(df,'key')
store.close()
```

will now not work.  The default is `a` I believe (append).

I updated the description, this doesn't actually change the behaviour on `HDFStore` but just on `read_hdf`.

I see - seems reasonable to read in `'r'`.

Is it possible to test this change?

Either way, this needs a release note for "what's new".

Create a hdf file and change it to read only using os.chmod.  The use read_hdf...it should pass now and would have failed before.

On Feb 25, 2015 3:06 PM, Stephan Hoyer notifications@github.com wrote:

Is it possible to test this change?

Either way, this needs a release note for "what's new".

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/9551#issuecomment-76044621.

the default mode is 'a', which is the `PyTables` default, and IMHO the most correct thing to do. It doesn't actually write the file if you don't do any write operations so its safe for read-only systems as well.

You can write a test to validate, but since the mode is determined by `.open()` I believe it is already correct.

this was a bug, but fixed in 0.13: https://github.com/pydata/pandas/pull/4513

Hmm, indeed it works locally, but if the `h5` file to be opened is on a network share it fails with a permission error (that's where this appeared). I'll try to verify it, but it's going to be difficult to test this ...

Indeed, when I deny the `Write` permission in Windows' security settings I can not read anymore with `mode="a"` but it works with `mode="r"`.

It sounds like it is a platform-specific bug, but it also looks innocuous to open `'r'` in `read_hdf` since the store is immediately closed.

@filmor I don't think this is going to be possible as the default is append (and makes more sense)

What? For `*read*_hdf` `append` is more sensible than `read`? I disagree.

Just to make this clear, this PR does _not_ change the default on `HDFStore`, nor does it change how `read_hdf` works if you pass in a store directly.

Please reconsider, the patch is trivial, doesn't break anything because behaviour doesn't change and this issue is causing real problems in our systems, as people are not regularly allowed to change the shared HDF5 files. I hate to sprinkle `mode='r'` all over the place, in particular as `read_hdf(f, mode='r')` just sounds broken ...

you can simply use a function internal to what you do. Even if I agreed that this should be changed, this would take a month or more to make it to the next release, then you would have to update, etc.

```
def read_hdf(fname, key, mode=None, **kwargs):
    if mode is None:
         mode = 'r'
    return pd.read_hdf(fname, key, mode=mode, **kwargs)
```
ENH + BUG: insert of new values for axis parameter on pandas.DataFrame.quantile
State: closed

closes #9543

I added the values "index" and "columns" for the axis parameter of the method pandas.DataFrame.quantile().
Also, there was no check on the value inserted for axis. E.g. if 2 or "foo" was inserted, no ValueError() was raised.
I added a basic check.

Thanks 


Labels:
name: Bug
name: API Design

Comments:
you don't need all this.

`axis = self._get_axis_number(axis)` will suffice

mainly need to add a test

You are right. I am correcting it.
Thanks

I corrected it and I implemented a basic test. In case tell me if I did any mistake.
Thanks

I finally had the time to complete my PR with the Travis CI build. Now what have I to do? (it is my first PR)

this looks fine. can you add a release note for 0.17.0 in the API section? ping when travis is green.

Hi @jreback 
Have I to modify this file: https://github.com/pydata/pandas/blob/master/doc/source/release.rst
?
Second, when you say "can you add the issue number here" (see comment above) it means I should write something like:

`# Unit test for #9544`

Thanks

@fbrundu this the file to which to add release notes: https://github.com/pydata/pandas/blob/master/doc/source/whatsnew/v0.16.2.txt

(we're actually going to release 0.16.2 before 0.17, and this can go in there)

As for the issue number, yes that would be fine. Something as simple as  `# GH9544` would be enough.

@jreback travis is green but git tells me merge conflicts
let me know

merged via https://github.com/pydata/pandas/commit/c0917de763d0c1369124f52bcaa18fb3fb36f05f

thanks!
BUG: declare and use self.unique_check
State: closed

... unless I'm really missing something trivial...


Labels:
name: Performance
description: Memory or execution speed performance
name: API Design

Comments:
u aren't using it
is it used anywhere? 
is there any perf impact?

The attribute is used at line
https://github.com/pydata/pandas/blob/a064c7d976f5e93ea91a23f97690f1d180a03948/pandas/index.pyx#L209
The performance impact is virtually null: after the test fails, _ensure_mapping_populated ends up being called, it sees the index is already initialized and stops. So unless there are plans to make more fine grained initializations, I think one could just remove anything related to it, and make "__get__" of "is_unique" simply check self.initialized and in case call self.initialize.

closing pls reopen if/when updated
Update tslib.pyx
State: closed

closes https://github.com/pydata/pandas/issues/9513
pd.NaT.date() was returning datetime.date(1, 255, 255)
it raises now a ValueError


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
needs a test. further needs to validate that other appropriate methods either raise/return something reasonable.

closing pls reopen if/when updated
BUG: multiple level unstack with nulls
State: closed

closes https://github.com/pydata/pandas/issues/9497

```
>>> mi
                        jim    joe
1st 2nd        3rd
1   2014-02-01 -1 days  100 -20.87
2   NaT        NaT      101   5.76
1   2014-02-03 1 days   102  -4.94
2   NaT        2 days   103  -0.79
1   2014-02-05 NaT      104 -12.51
2   2014-02-06 4 days   105  -9.89

>>> mi.unstack(['2nd', '3rd']).fillna('.')
           jim                                                     joe
2nd 2014-02-01  NaT 2014-02-03    NaT 2014-02-05 2014-02-06 2014-02-01   NaT 2014-02-03    NaT 2014-02-05 2014-02-06
3rd    -1 days  NaT     1 days 2 days        NaT     4 days    -1 days   NaT     1 days 2 days        NaT     4 days
1st
1          100    .        102      .        104          .     -20.87     .      -4.94      .     -12.51          .
2            .  101          .    103          .        105          .  5.76          .  -0.79          .      -9.89
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @behzadnouri
FIX: Fix some instances where idx[0] not in idx
State: closed

`DatetimeIndex.__contains__` and `TimedeltaIndex.__contains__` were failing to see duplicated elements in some circumstances.

Fixes #9512


Labels:
name: Bug

Comments:
awesome!

can you add a release note to the bug-fixes section of `doc/source/whatsnew/0.16.0.txt` and squash the commits into one?

merged via #9525. Thanks!
PERF: Cython optimizations for period module, round one
State: closed

Each commit passes `test_fast.sh`, so performance can be measured between commits.

I can break any of these commits into separate PRs if needed.


Labels:
name: Performance
description: Memory or execution speed performance
name: Period
description: Period data type

Comments:
vbench results

https://gist.github.com/blbradley/3b0706bb2efe908c7026

I removed one commit because I thought it was a bad decision. I don't think it will affect the performance results too much.

Looks good to me.

You'll need to add types if you want significant speedups for things like the `Period` constructor, though.

Current build failure is a problem with Travis CI and PyPI package resolution. Happening with other builds too.

@blbradley thanks for the report, I opened a conda issue for that: https://github.com/conda/conda/issues/1166

can you post an updated vbench?

@blbradley The build did pass when I restarted it (after conda deployed their fix)

@shoyer fyi, I turned off `ssl_verify` to fix this (subsequently conda DID fix requests though)

lgtm, @shoyer ?

Yes, looks good to me!
WIP/TST: Added test of unstacking two levels with NaN.
State: closed

Added a test to `test_unstack_nan_index()` for unstacking two levels simultaneously when one of them contains a `NaN`.

Should delete the `print()` statements before merging...

This is the output I get:

```
======================================================================
FAIL: test_unstack_nan_index (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\seth\github\pandas2\build\lib.win-amd64-3.4\pandas\tests\test_frame.py", line 12439, in test_unstack_nan_index
    assert_series_equal(left, right)
  File "c:\Users\seth\github\pandas2\build\lib.win-amd64-3.4\pandas\util\testing.py", line 674, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "das\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2745)
  File "das\src\testing.pyx", line 93, in pandas._testing.assert_almost_equal (pandas\src\testing.c:1830)
  File "das\src\testing.pyx", line 135, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2514)
nose.proxy.AssertionError: (very low values) expected 3.00000 but got 0.00000, with decimal 5
-------------------- >> begin captured stdout << ---------------------
dfs=
       C
A B     
a 0    0
  1    1
  2    2
  NaN  3
b 4    4
  5    5
  6    6
  7    7

left=
   A  B
C  a  1    0
      2    1
      4    2
      0    3
   b  6    4
      7    5
   a  0    6
      1    7
dtype: int32

right=
   A  B  
C  a  NaN    3
      0      0
      1      1
      2      2
   b  4      4
      5      5
      6      6
      7      7
dtype: int32

--------------------- >> end captured stdout << ----------------------
```


Labels:

Comments:
cc: @behzadnouri

@jreback, @behzadnouri: In view of  @behzadnouri's observation https://github.com/pydata/pandas/pull/9023#issuecomment-74323885 that implementing `DataFrame.unstack()` in terms of (my modified) `DataFrame.stack()` slowed things down tremendously, I tried going the other way and implementing `DataFrame.stack()` in terms of `DataFrame.unstack()` (after making a few simple modifications to the latter to support `dropna` and `sequentially`). I noticed that then some of my `stack()` tests were failing, and isolated this case as an example.

For comparison, this is the result in 0.15.2 in the absence of `NaN`s.

```
In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'A': list('aaaabbbb'),'B':range(8), 'C':range(8)})

In [3]: dfs = df.set_index(['A', 'B'])

In [4]: dfs.unstack([0, 1])
Out[4]:
   A  B
C  a  0    0
      1    1
      2    2
      3    3
   b  4    4
      5    5
      6    6
      7    7
dtype: int32
```

[Recommitted code with correct expected `right`.]

@seth-p pls submit this as a separate issue report 

Created #9497.
BUG: unstack with nulls & Timedelta/DateTime index
State: closed

xref https://github.com/pydata/pandas/pull/9292#discussion_r24620887

on branch:

```
>>> df
           a       b    c
0 2014-02-01 -1 days  100
1        NaT     NaT  101
2 2014-02-03  1 days  102
3        NaT  2 days  103
4 2014-02-05     NaT  104
5 2014-02-06  4 days  105

>>> df.pivot('a', 'b', 'c').fillna('-')
b           NaT -1 days 1 days 2 days 4 days
a
NaT         101       -      -    103      -
2014-02-01    -     100      -      -      -
2014-02-03    -       -    102      -      -
2014-02-05  104       -      -      -      -
2014-02-06    -       -      -      -    105

>>> df.pivot('b', 'a', 'c').fillna('-')
a        NaT 2014-02-01 2014-02-03 2014-02-05 2014-02-06
b
NaT      101          -          -        104          -
-1 days    -        100          -          -          -
1 days     -          -        102          -          -
2 days   103          -          -          -          -
4 days     -          -          -          -        105
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
- can you add a release note
- pls make the test on a mixed frame e.g. with ints/datetimes/timedeltas in it (with datetimes/timedeltas) have NaT's. for a full test

what do u mean? the added test is already a mixed frame with ints, datetimes & timedelta

sorry, was looking at some things else....ok just the release note and good to go (and put a comment on the test that refers to issue as well)

added the release note

@behzadnouri thanks as always!
CLN: import packers/gbq/html only on-demand (GH9482)
State: closed

closes #9482 
so a 30% reduction in import time

```
[jreback-~/pandas] time python -c 'import pandas'
0.788u 0.120s 0:00.91 98.9%     0+0k 0+7io 0pf+0w
[jreback-~/pandas] time python -c 'import pandas'
0.782u 0.115s 0:00.91 97.8%     0+0k 0+0io 0pf+0w
[jreback-~/pandas] time python -c 'import pandas'
0.790u 0.121s 0:00.91 100.0%    0+0k 0+0io 0pf+0w
```

```
[jreback-~/pandas] time python -c 'import pandas'
0.270u 0.091s 0:00.37 97.2%     0+0k 10+20io 0pf+0w
[jreback-~/pandas] time python -c 'import pandas'
0.262u 0.087s 0:00.35 97.1%     0+0k 0+0io 0pf+0w
[jreback-~/pandas] time python -c 'import pandas'
0.264u 0.088s 0:00.36 94.4%     0+0k 0+0io 0pf+0w
```


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
This _seems_ to work (meaning that it doesn't import these except on-demand), can people give this a try

@TomAugspurger @shoyer @jorisvandenbossche @cpcloud 
cc @rockg

from the related issues
cc @thomasj02 
cc @mpenning
cc @ndc33
cc @steve3141

any comments on this?

Looks good to me! (gave it a quick test and glance over the code)
BUG: bug in sort for grouping with a categorical columns (GH8868)
State: closed

Closes #8868

hack. I can add tests if this passes the CI and we think this should be taken further.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
well you would need tests. Not sure this 'hack' is correct.

@cottrell ok, other PR is merged in. Pls add tests and update. See if it can be done more naturally.

@cottrell thanks

(this was correct)
BUG: fix common.is_hashable for NumPy scalars on Python 3
State: closed

Fixes #9276

This now relies entirely on the result of calling `hash` on the argument.

Note: I had to change the test for old style classes on Python 2 -- these are
now considered hashable by `is_hashable`, because they don't fail `hash`.

CC @aevri


Labels:
name: Bug

Comments:
lgtm
TST: Remove assertEquals
State: closed

Same as #7165. I've also added some of these tests :(

One idea to prevent this is to override ``tm.TestCase.assertEquals` to raise more explicit warning or exception.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@sinhrks while don't we just make `assertEquals` == `assertEqual` (with a warning) though

Made suggested changes based on the [unittest doc](https://docs.python.org/3/library/unittest.html#deprecated-aliases). Used `deprecate` decorator as a function because it can't used in the class definition.

Followings are not changed.
- `assertRegexpMatches`: Inroduced in python 3.1 and not used in the script. No need to be cared.
- `assertRaisesRegexp`: Introduced in python 3.1, but uses original implementation in `util.testing`. Thus untouched.

thanks you sir!
BUG: subplots with layout kw may show unnecessary warning
State: closed

Related to #8877 and #9278. There is remaining case which shows unnecessary warning.

Occurrence condition:
- `subplots=True`.
- `layout` is specified larger than the number of required subplot axes.

In this case, `legend` is called against blank axes and results in warning.


Labels:
name: Visualization
description: plotting

Comments:
thanks @sinhrks
Fix bug in multiindex series groupby where sort argument is ignored
State: closed

PR as per. 

One test added. Test fails without the fix.

closes https://github.com/pydata/pandas/issues/9444


Labels:
name: Bug
name: MultiIndex

Comments:
looks good
ping when green

@cottrell looks good now. thanks. ping when green.

does this also fix #8868  ?

@jreback It does not seem to in my test of #8868. Probably a similar issue but elsewhere. I can probably have a look later this week.

ok gr8 thanks

thanks!
Groupby sort
State: closed

PR as per issue 9444. One test added.
https://github.com/pydata/pandas/issues/9444


Labels:

Comments:
Continued in #9461

By the way, @cottrell, it is not needed to open an new PR. Just push force your cleaned up branch, and the PR will be updated automatically (as long as it is the same branch name)

@jorisvandenbossche ah thanks ... I see it messes up the timeline a bit. I mistakenly pushed junk the first time and didn't want anybody to spend time on it.
BUG: handle NotImplementedError in Index comparisons
State: closed

Fixes issue #9432

A very specific fix that is not anticipated to impact performance.  
The test performed is safe and documents that the origin of the 
problem is actually numpy.  When this numpy bug is fixed it will 
be easy to remove this code. 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
can u add a test for same (eg use your example)

Of course :)

I am not familiar enough with pandas package development to choose a good
place to put an _indexOp test within pandas/tests/test_index.py

Cheers,
David

On Mon, Feb 9, 2015 at 2:50 PM, jreback notifications@github.com wrote:

> can I add a test for same (eg use your example)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9451#issuecomment-73503279.

you can just put this in `test_index/Base`. It might break some other index types (e.g. DatetimeIndex), as they are expecting different types of comparisions here but do conversions, so the test is a bit tricky. Or you can put it in `TestIndex` is prob ok. This is just an edge case. 

Because this address an external issue (numpy bug), I added a test that includes instructions to remove both the test and the try except block in _indexOp when it fails (presumably after the numpy bug is fixed, but perhaps in a future change).

closing pls reopen if/when updated
PERF: performance improvement in MultiIndex.sortlevel
State: closed

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
multiindex_sortlevel_int64                   | 664.1977 | 2443.2590 |   0.2718 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [dd31bae] : performance improvement in MultiIndex.sortlevel
Base   [c4a996a] : Merge pull request #9411 from dashesy/add_sql_test

test sql table name
```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
looks good otherwise, ping when green after that change.

@behzadnouri thanks!
BUG: scatter_matrix draws incorrect axis
State: closed

Closes #5662. Refactored a little.

Made examples to confirm 1st and 2nd row's yaxis are identical.

```
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 3))
df[0] = df[1]
pd.tools.plotting.scatter_matrix(df)
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/6092365/63f4ce98-af1e-11e4-9175-eba45b483bba.png)

```
df[1] = df[1] / 3 - 2
df[0] = df[1]
pd.tools.plotting.scatter_matrix(df)
```

![figure_2](https://cloud.githubusercontent.com/assets/1696302/6092366/68041dea-af1e-11e4-8ecd-cfcb33aaf1a9.png)


Labels:
name: Visualization
description: plotting

Comments:
@sinhrks is there actually anything that prevents this from being merged?
cc @TomAugspurger 

No, I think this is ready for review. Because it is categorized as a bug fix, is it possible to include it in 0.16.1 (not next major release)?

I'll rebase this later to fix the release note.

Thanks. Ping me when the rebase is done and I'll get it merged. 

> On Mar 30, 2015, at 22:18, Sinhrks notifications@github.com wrote:
> 
> No, I think this is ready for review. Because it is categorized as a bug fix, is it possible to include it in 0.16.1 (not next major release)?
> 
> I'll rebase to in this later to fix the release note.
> 
> —
> Reply to this email directly or view it on GitHub.

@TomAugspurger Rebased, and now green.

OK, thanks!
DOC: Clarify definition of kurtosis used
State: closed

Added explation that kurtosis returned is Fisher's kurtosis, also
known as excess kurtosis.

Also cleans up the kurtosis function to remove some unused computations.


Labels:
name: Docs
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Trivial PR to better explain how kurtosis is computed.  When checking the calculation, I noticed 2 lines that were unused to also removes those.

Indeed on the first use of `C`.

@bashtage thanks
PERF: performance improvements in multi-key groupby
State: closed

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_multi_index                          | 1008.6970 | 1861.1110 |   0.5420 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [907e1b7] : performance improvements in multi-key groupby
Base   [5dc8009] : Merge pull request #9418 from sinhrks/rn_20150205

DOC: Fix release note format
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
pls add a release note in perf section, otherwise looks good.

@jreback added the release note

thanks!

we should prob rerun these benchmarks to see how much better things are now

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping
ENH: Don't infer WOM-5MON if we don't support it (#9425)
State: closed

closes #9425 


Labels:
name: Frequency
description: DateOffsets

Comments:
pls add a release note. you can put in bug fixes for 0.17.0. looks good. pls ping when travis is green.

@jreback build passed. let me know if there's anything else i can do

thanks!