BrandonFors/pandas-github-issues huggingface dataset:
ENH: #10143 Function to walk the group hierarchy of a PyTables HDF5 file
State: closed

closes #10143 

This implementation is inspired by os.walk and follows the interface as much as possible.


Labels:
name: Enhancement
name: API Design
name: IO HDF5
description: read_hdf, HDFStore

Comments:
any reason not to call this just walk?

pls add a note in whatsnew/0.17.0 in enhancements

I've renamed walk() and included Series objects.  However, I think you have a different API in mind.  I am deliberately not yielding each Pandas object individually but yielding each PyTables group name, along with a tuple of its contents.  This follows the os.walk API.  I.e. each yield is 

  `(group_path, [subgroup_name, ...], [subobj_name, ...])`

I think there are several advantages:
1. The consumer can see the difference between groups and Pandas objects
2. Future extension could allow pruning of the search space by mutating the yielded lists, as is possible with os.walk.

Note also:
- Some testing of node type is necessary during walk because a Pandas object is also a group to PyTables.
- All non-pandas leaves are ignored.  walk() will only yield groups and Pandas objects.

Please let me know what you think before I write something in whatsnew/0.17.0

ok, looks reasonable. Pls add a whatsnew (and add this to the docs in io.rst), separate-sub-section in the HDF5 section. 

closing, pls reopen if you can fix according to comments
Add tests to ensure sort preserved by groupby, add docs
State: closed

xref #9651
closes #8588 

Adds test to ensure the sort of a target object is preserved within `groupby()` groups, modifies docs to make it clear sort is preserved within groups. 


Labels:
name: Docs
name: Groupby

Comments:
@jreback  updated!

@jorisvandenbossche @shoyer ?

The basic part on sort=True/False and sort preserved within groups looks good!

However, I don't really seem to grasp the need to explain the sorting after groupby: _"Users should be careful about re-sorting their data after execution of `groupby()`"_. Why should they be careful? Is there anything not-expected happening?

@jorisvandenbossche @shoyer @jreback OK, warning dropped! 

@jorisvandenbossche great, added. 

@nickeubank thanks!
fixed bug in DataFrame.diff - issue #10907
State: closed

Fixed issue #10907 


Labels:
name: Bug
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
This is my first PR here. Could someone help me pass this Travis test? After I made the changes in the code, what should I do?

try running the tests locally first 

eg

nosetests pandas/test_frame.py

the consolidate has to go in core/internals.py (where diff is defined)

When I run tests by calling `nosetests pandas/tests/test_frame.py`, the following error occurred: `ImportError: No module named test_frame`. 

Is there anything else I should do before test it? 

see the docs here: http://pandas.pydata.org/pandas-docs/stable/contributing.html

Thanks so much for your help! I have made the changes. Now I understand the code structure better. Hopefully I'll be able to contribute more in the future. 

Done. Should be able to merge now since it passes tests locally. There are too many Travis build tasks pending now.

needs a whatsnew note
pls squash 

This should do it now, right?

pls squash as well.

Thanks. Done. 

ok, pls squash to a single commit. ping when green.

It has conflicts that I don't know how to solve if I squash...could we leave it as it is this time? Both commits are descriptive to me. 

ping me when green. typically we use just a single commit on things like this. this is just the way pandas has always worked.

@jreback could you merge?

will get to soon

This is the same as the original. Now in the testing code, I've created the same DataFrame that's originally created via DataFrame.insert. The core problem rises from DataFrame.diff instead of DataFrame.insert. Please take a quick comparison between the outdated diff in this PR and the new push in this PR. Thanks.

If you do a assertDataFrameEqual to the original test code and the current test code. You'll see those DataFrames are equivalent. 

your test passes on 0.16.2
so you have not replicated it exactly

you haven't proven that you fixed the bug (you did) but your test doesn't work 

Ok didn't know that. I'll revise the test soon. Thanks for double checking.
On Aug 31, 2015 3:44 PM, "Jeff Reback" notifications@github.com wrote:

> your test passes on 0.16.2
> so you have not replicated it exactly
> 
> you haven't proven that you fixed the bug (you did) but your test doesn't
> work
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10930#issuecomment-136479652.

@jreback Could you double check new push?

merged via https://github.com/pydata/pandas/commit/219ad757a36f96110810282257e3cf2633f1f5bb

thanks!

generally you want to create a branch each time you make a change (and not do it on master)

see [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

@jreback I don't think this fix works if you have mixed types, since consolidate can't consolidate those blocks.  This is much less common, but I could see someone with mixed ints and floats that wants to shift

@hhuuggoo yes, this is prob just wrong. it needs to create the indexer then do a take.

Can you create another issue with an example?

thxs

yup!
Revert "Revert "Merge pull request #10727 from jorisvandenbossche/sph…
State: closed

Reverts pydata/pandas#10916


Labels:
name: Docs

Comments:
This is becoming a nice title Revert of revert of ... :-)

Just opening this, so I can grab this and only change the syntaxerror to a code-block, to see if that lets travis build the docs correctly.

Hmm, still not working ..

@jorisvandenbossche status for this?

@jorisvandenbossche status for this?

@jorisvandenbossche close this?

@jorisvandenbossche keep open?

@jorisvandenbossche status of this?

@jorisvandenbossche closing as a PR. maybe create an issue for this?
DEPR: Series.nlargest/nsmallest take_last.
State: closed

Closes #10792. 

Because using `deprecate_kwargs` hides original impl, used ugly workaround for `GroupBy`...


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jreback Yes, updated.

thank you sir!
Revert "Merge pull request #10727 from jorisvandenbossche/sphinx-traceback
State: closed

This reverts commit d40627398cf78347f5a49fa060bad1c40514908a, reversing
changes made to bd804aa75ac9b7a55c00101dbea01571cbdfd068.

@jorisvandenbossche reverting so that the docs build

pls rebsumit when you can


Labels:
name: Docs

Comments:
Yep, sure!

The problem is, it just builds fine locally ... Tested it both on linux and windows. 

I think it has to do with this example:

```
In [41]: a, b = 1, 2

In [42]: pd.eval('@a + b')
  File "<string>", line unknown
SyntaxError: The '@' prefix is not allowed in top-level eval calls, 
please refer to your variables by name without the '@' prefix
```

Because it is a syntax-error, which seems to have some different error traceback than other errors
explain how to skip rows between header & data
State: closed

This is the result from
https://github.com/pydata/pandas/issues/10898


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
can you update

> can you update

What is missing?

@jorisvandenbossche what do you think?

Seems OK! @dacoex Could you squash the commits? (http://pandas-docs.github.io/pandas-docs-travis/contributing.html#combining-commits)

@jorisvandenbossche  done / see above.

Thanks!
PERF: imporves performance in SeriesGroupBy.value_counts
State: closed

``` -------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_ngroups_10000_value_counts           |  11.4880 | 6698.5627 |   0.0017 |
groupby_ngroups_100_value_counts             |   1.3397 |  69.8457 |   0.0192 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [d587be8] : PERF: imporves performance in SeriesGroupBy.value_counts
Base   [a3c4b59] : TST: pythonxs link seems to have changed in test_html.py, skip tests
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
wow, you made this crazy fast! thanks!
API: Closes #7879: (drops not nan in panel.to_frame() by default)
State: closed

closes #7879 by changing API (filter_observations=True now deprecated and replaced with dropna=False)


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
this will need a sub-section in whatsnew/0.17.0. showing what the original behavior is and what the new is. 

can you update according to comments

@jreback: updated the commit according to your comments!

- need a test for the deprecation itself.
- need a separate sub-section in the whatsnew explaining this change

added/fixed tests and added sub-section

this is pushed to 0.18.0 pls update when you can

closing, but if you'd like to update, pls reopen
PERF: improves SeriesGroupBy.nunique performance
State: closed

closes https://github.com/pydata/pandas/issues/10820

on master:

``` ipython
In [2]: df = pd.DataFrame({'a': np.random.randint(10000, size=100000),
   ...:                    'b': np.random.randint(10, size=100000)})

In [3]: %timeit df.groupby('a')['b'].nunique()
1 loops, best of 3: 1.66 s per loop

In [4]: %timeit df.groupby(['a', 'b'])['b'].first().groupby(level=0).size()
10 loops, best of 3: 36.3 ms per loop
```

on branch:

``` ipython
In [2]: %timeit df.groupby('a')['b'].nunique()
10 loops, best of 3: 29.2 ms per loop
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
benchmarks:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_ngroups_10000_nunique                |   6.8550 | 1701.2453 |   0.0040 |
groupby_ngroups_100_nunique                  |   0.5283 |  18.2413 |   0.0290 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [091c73d] : PERF: improves SeriesGroupBy.nunique performance
Base   [54f02df] : COMPAT: value_counts always return int64 dtype, xref #10876
```

@behzadnouri looks great. can you add a benchmark as well. ping when green.

@jreback there already are 2 benchmarks

oh, right, ok then.

@behzadnouri thanks!

That's awesome, sort of what I was thinking about but didn't have time to hack together :/ Thanks @behzadnouri!
DEPR: Bunch o deprecation removals part 2
State: closed

- Remove the table keyword in HDFStore.put/append, in favor of using format= #4645
- Remove unused keyword `kind` in `read_excel/ExcelFile` #4712 
- Remove `infer_type` keyword from `pd.read_html` as its unused, #4770, #7032
- Remove `offset` and `timeRule` keywords from `Series.tshift/shift`, in favor of `freq`, #4853, #4864
- Remove `pd.load/pd.save` aliases in favor of `pd.to_pickle/pd.read_pickle`, #3787
- Deprecate `WidePanel/LongPanel`


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
+1 !
BUG: encoding of categoricals in hdf serialization
State: closed

closes #10366 
replaces #10454 


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore
name: Categorical
description: Categorical Data Type

Comments:
cc @cottrell 

I changed this a bit to explicity check for bytes, and doced this a bit.

Seems ok to me, but can you check for 

nosetests -v pandas.io.tests.test_pytables passed on my local python 3.4.3 build. I see other errors popping up as in the travis tests.

thanks @cottrell
BUG: GH10885 where an edge case in date_range produces an extra point
State: closed

closes #10885 


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
lgtm ping when green.

Updated. Had to change a little because of a possible bug in numpy.

```
 In [14]: len(numpy.arange(0, 10000000000000001, 100000000000000))
Out[14]: 101

In [15]: len(numpy.arange(0, 100000000000000000001, 1000000000000000000))
Out[15]: 100
```

thanks

might want to report that to numpy - looks like an unhandled overflow issue

for reference:  numpy/numpy#6238
ERR: better error reporting for failing parsing in timedelta/timeseries #10720
State: closed

closes #10720 


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Timedelta
description: Timedelta data type

Comments:
Why did you close this one?

Some advice:
- always work on a branch and not in master for a PR (so this one is better than #10884, I can reopen this one if you want)
- always update with the latest upstream before starting a new branch (you have still some old commits of a previous pr):
  
  ```
  git checkout master
  git fetch upstream
  git merge upstream/master --ff-only
  git checkout -b new_branch
  ```
  
  or now if you already started a branch, rebase it on master:
  
  ```
  git fetch upstream
  git rebase upstream/master
  ```

Ohh i closed the other one because i wanted to rebase. Thanks for the tip

On Saturday, 22 August 2015, Joris Van den Bossche notifications@github.com
wrote:

> Some advice:
> - always work on a branch and not in master for a PR (so this one is
>   better than #10884 https://github.com/pydata/pandas/pull/10884, I
>   can reopen this one if you want)
> - always update with the latest upstream before starting a new branch
>   (you have still some old commits of a previous pr): git checkout
>   master git fetch upstream git merge upstream/master --ff-only git checkout
>   -b new_branch
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10883#issuecomment-133674560.

## 

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

Can you reopen this one instead and close the other one?

Ok - so that attempt passed, any got any feedback about whether it's ready to be merged or does it need some more work?

Seems ready 

@springcoil this doesn't do anything. as you didn't actually change any code except for the tests. So this fails. And it still is not rebased on master.

```
In [1]: td = pd.Series(['May 04', 'Jun 02', ''], index=[1,2,3])

In [3]: pd.to_datetime(td,format='%b %y', errors='raise') 
ValueError: time data '' does not match format '%b %y' (match)
```

Ok will fix that

On Saturday, 22 August 2015, Jeff Reback notifications@github.com wrote:

> @springcoil https://github.com/springcoil this doesn't do anything. as
> you didn't actually change any code except for the tests. So this fails.
> And it still is not rebased on master.
> 
> In [1]: td = pd.Series(['May 04', 'Jun 02', ''], index=[1,2,3])
> 
> In [3]: pd.to_datetime(td,format='%b %y', errors='raise')
> ValueError: time data '' does not match format '%b %y' (match)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10883#issuecomment-133755403.

## 

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

I'm going to close this for now.
ERR/DEPR: Fix quantile error message / remove percentile_width
State: closed

Currently, error is raised from `numpy` and incorrect for `pandas`

```
s = pd.Series([1, 2, 3])
s.quantile(2)
#   File ".../numpy/lib/function_base.py", line 3078, in _percentile
#     raise ValueError("Percentiles must be in the range [0,100]")
# ValueError: Percentiles must be in the range [0,100]
```

`describe` using `percentiles` option outputs better error message, but not check lower limit.

```
# OK
s.describe(percentiles=[2])
# Traceback (most recent call last):
#   File "pandas/core/generic.py", line 4167, in describe
#    raise ValueError(msg.format(list(percentiles)))
# ValueError: percentiles should all be in the interval [0, 1]. Try [0.02] instead.

# NG
s.describe(percentiles=[-2])
#   File ".../numpy/lib/function_base.py", line 3078, in _percentile
#     raise ValueError("Percentiles must be in the range [0,100]")
# ValueError: Percentiles must be in the range [0,100]
```

This PR fix both error messages. Also, remove deprecated `percentile_width` option in `describe`. 


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Deprecate
description: Functionality to remove in pandas

Comments:
Ah, found an existing bug which drops `Series.name`. Fixed in the same PR.

```
# current master
pd.Series([1, 2, 3], name='x').quantile([0.5])
# 0.5    2
# dtype: float64
```

thanks!
BUG: pd.Series.interpolate(method='spline') Errort Msg, #10633
State: closed

closed #10633 

This might be able to be merged quicker than the other one.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
merged via https://github.com/pydata/pandas/commit/e581e1e07574b2cc6a346007f53f1f902c8a7cac

thanks!
BUG: DataFrame.plot may raise IndexError / show unnessesary minor ticklabels
State: closed

Closes #10657. Closes #10819. Both are related to the logic how `pandas` detects axes layout. Use current logic as much to support cases like #7457, and added error handing logic for gridspec.

Following is an example notebook how the fix works.
- https://gist.github.com/sinhrks/2d29e2e8ef26f2757e92

CC @TomAugspurger @JanSchulz @heelancd @williamsmj


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
thank you sir!
BUG: GH10747 where 'timestamp' is not inferred to be datetime column
State: closed

addresses second point of #10747 


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
lgtm. can you add a small doc update in `io.rst` to specify what names are auto-inferred for this.

Updated. 

updated
PERF: uses bincount instead of hash table in categorical value counts
State: closed

closes https://github.com/pydata/pandas/issues/10804

``` ipython
In [1]: np.random.seed(2718281)

In [2]: n = 500000

In [3]: u = int(0.1*n)

In [4]: arr = ["s%04d" % i for i in np.random.randint(0, u, size=n)]

In [5]: ts = pd.Series(arr).astype('category')

In [6]: %timeit ts.value_counts()
10 loops, best of 3: 82.7 ms per loop
```

on branch:

``` ipython
In [6]: %timeit ts.value_counts()
10 loops, best of 3: 31.3 ms per loop
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
I think the soln in the issues is faster than this no?

I get

``` ipython
In [9]: %timeit Series(np.arange(len(ts.cat.categories)),ts.cat.categories).map(ts.cat.codes.value_counts()).order(ascending=False)
10 loops, best of 3: 28.3 ms per loop
```

but this does not check for nulls, and the index is not categorical.

with `dropna=False` the branch performs better:

``` ipython
In [9]: %timeit ts.value_counts(dropna=True)
10 loops, best of 3: 32.3 ms per loop

In [10]: %timeit ts.value_counts(dropna=False)
10 loops, best of 3: 25.7 ms per loop

In [11]: %timeit Series(np.arange(len(ts.cat.categories)),ts.cat.categories).map(ts.cat.codes.value_counts()).order(ascending=False) 
10 loops, best of 3: 29.5 ms per loop
```

wow, this does even better!

```
In [7]: %timeit ts.value_counts(dropna=True)
100 loops, best of 3: 11 ms per loop

In [8]: %timeit ts.value_counts(dropna=False)
100 loops, best of 3: 9.53 ms per loop

In [9]: %timeit Series(np.arange(len(ts.cat.categories)),ts.cat.categories).map(ts.cat.codes.value_counts()).order(ascending=False) 
100 loops, best of 3: 17.4 ms per loop
```

ping when green

Maybe worth adding a benchmark?

I will add benchmark later today

added the benchmark, all green.

thank you sir!
BUG: import of maybe_convert_indices in pandas.core.index.py, #10610
State: closed

closes #10610 

I fixed the import statement and added a test to check for proper behavior when accessing a mixed-integer index with a list of values.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
I removed the index checking if-statement, and added a line that would change the -1 indices to equal the length of the Index being evaluated, so that the IndexError would be raised in maybe_convert_indices.

ok, this is fine.

pls squash to a single commit. ping when green.

merged via https://github.com/pydata/pandas/commit/12df3ea39b28f752a0dcb43b87c4121387114cdf

thanks!
DEPR: deprecate the data reader package, #10861
State: closed

closed #10861 


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jorisvandenbossche ?

Yep! Merged with https://github.com/pydata/pandas/commit/dacae1039485634d77f0cc5422330268f467653d (I slightly changed the wording and added the imports changes)

And did the same for wb.py: https://github.com/pydata/pandas/commit/a44e4b376f2a882763230fd264193c1d75b063dc

awesome thxs!

The original idea was to just replace pandas data/web with an import statement, that way no code actually needs to be updated... tests could remain etc.

If there's a warning (not sure there has to be if we change the import) it should include, "install pandas-datareader and replace `pandas.io.data` with `pandas_datareader.data`".

Thoughts?

that's how it is; see joris commit above

Great, sorry didn't see that!

But why warn here... if we're later to just replace this in pandas as the import statement, I thought the idea was that users don't need to change any code; they just need to install pandas-datareader.

@hayd You are correct that that was one of the ideas in the discussion, but in my impression we didn't really converged to a final decision there yet (but maybe that is because I am not fully convinced of that position .. :-))

So this can certainly still be discussed/adapted (but indeed, I changed the wording a bit in the commit, so the import change is noted).

See my comment here with my reasons: https://github.com/pydata/pandas/issues/8961#issuecomment-70367939 + my last comment.

But if we discuss it further, maybe lets do it in the original issue over there: #8961

I have to agree with joris
this should be very explicit that you are now importing a new package (though it's highly related)

people will see the deprecation if they use the data package - they can then choose to update or not
DEPR: iterkv change Deprecation to FutureWarning
State: closed

So this is visible to the users (already long time deprecated in the docs)


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
what exactly is the difference between a `Deprecation` and `Future` warning?

DeprecationWarning is not visible to users (by default), FutureWarning is

That is the reason we use mostly FutureWarnings

oh ok, good to know.

Problem is that you actually want a combination of both: only show the warning if you call the function/method triggering it, and not when a library that you are using calls such a method. But AFAIK that is not possible.