BrandonFors/pandas-github-issues huggingface dataset:
fix crash on unicode level names
State: closed

This fixed my problem when calling DataFrame.stack(<integer level number>). Not really familiar with pandas internals, but couldn't find any reason why unicode strings won't be accepted here.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Unicode
description: Unicode strings
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
tests would be needed

To be honest, I don't know pandas internals/tests and I don't use python3. Please consider this as a bug report. Let me know if you need help with reproducing it.

@alavrik 

can you show some code which reproduces the error?

the point of a bug report is to say, here is what I did, here's how to reproduce it, its doing x, but I think it should by doing y

we cannot simply change code because it 'looks right'. then what happens when someone decided that some other way looks right?

this is not about knowing pandas internals or not, its about giving a bug report that can lead to tests that can lead to fixed code

sorry for being harsh, but I find lots of people writing code w/o even thinking about testing it.

@jreback 
You are right. Here's the test (Python 2.7.5, Pandas 0.15.2):

``` python
import pandas as pd
i = pd.MultiIndex(
    levels=[[u'foo', u'bar'], [u'one', u'two'], [u'a', u'b']],
    labels=[[0, 0, 1, 1], [0, 1, 0, 1], [1, 0, 1, 0]],
    names=[u'first', u'second', u'third'])
s = pd.Series(0, index=i)
s.unstack([1, 2]).stack(0)
```

and this is the last couple of frames of the error output:

```
/Users/alavrik/devel/ipython/ve/lib/python2.7/site-packages/pandas/core/index.pyc in sortlevel(self, level, ascending, sort_remaining)
   3868         if isinstance(level, (str, int)):
   3869             level = [level]
-> 3870         level = [self._get_level_number(lev) for lev in level]
   3871 
   3872         # partition labels and shape

/Users/alavrik/devel/ipython/ve/lib/python2.7/site-packages/pandas/core/index.pyc in _get_level_number(self, level)
   3166         except ValueError:
   3167             if not isinstance(level, int):
-> 3168                 raise KeyError('Level %s not found' % str(level))
   3169             elif level < 0:
   3170                 level += self.nlevels

KeyError: 'Level t not found'
```

It is easy to see what's going on. Instead of iterating over `[level]`, it is trying to iterate over the characters of the unicode `level` name. In this case, it starts with the character `t` of the level `two`.

closing in favor of #9856 bug report
PERF: optimize memory usage for to_hdf
State: closed

from [here](http://stackoverflow.com/questions/29016093/pandas-pytable-memory-overhead-when-writing-to-hdf/29035532#29035532)

reduce memeory usage necessary for using `to_hdf`
- was copying always in astyping
- was ravelling then reshaping
- was constantly allocating a new chunked buffer, now re-uses the same buffer

```
In [1]: df = pd.DataFrame(np.random.rand(1000000,500))
df.info()

In [2]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Columns: 500 entries, 0 to 499
dtypes: float64(500)
memory usage: 3.7 GB
```

Previously

```
In [3]: %memit -r 1 df.to_hdf('test.h5','df',format='table',mode='w')
peak memory: 11029.49 MiB, increment: 7130.57 MiB
```

With PR

```
In [2]: memit -r 1 df.to_hdf('test.h5','df',format='table',mode='w')
peak memory: 4669.21 MiB, increment: 794.57 MiB
```


Labels:
name: Performance
description: Memory or execution speed performance
name: IO HDF5
description: read_hdf, HDFStore

Comments:
If you happen to find yourself in new york I'm buying you a beer for this fix.

hahha I figured I broke it I should fix it

in Nyc, so anytime!

Thanks a lot for fixing this! It helps a lot.

the bug is back

```
df = pd.DataFrame(np.random.rand(1000000,500))
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000000 entries, 0 to 999999
Columns: 500 entries, 0 to 499
dtypes: float64(500)
memory usage: 3.7 GB

%memit -r 1 df.to_hdf('test.h5','df',format='table',mode='w')
peak memory: 7934.20 MiB, increment: 3823.80 MiB

pd.__version__
'0.24.2'
```

With a more complex structure, everything is much worse.
```
data_ifa.info()
<class 'pandas.core.frame.DataFrame'>
Index: 100000 entries, b88d3b87-3432-43cc-8219-f45d97389d8f to eb705297-94e8-4ccf-a910-5f3b9734d572
Data columns (total 2 columns):
bundles        100000 non-null object
bundles_len    100000 non-null int64
dtypes: int64(1), object(1)
memory usage: 2.3+ MB

%memit -r 1 data_ifa.to_hdf(full_file_name_hd5, key='data_ifa', encoding='utf-8', complevel=9, mode='w', format='table')
peak memory: 22106.07 MiB, increment: 21324.53 MiB
```
Allow clip{,_lower,_upper} to use array-like thresholds (GH 6966)
State: closed

closes #6966 


Labels:
name: Bug

Comments:
Sorry, first time contributing. My local testing wasn't getting to test_frame and test_series like I thought it was. This second commit should fix the problems.

pls add a release note in whatnew (0.16.1), which will be available after we release 0.16.0 (prob this weekend)

whats new is now available for 0.16.1 https://github.com/pydata/pandas/commit/5ebf521eac528f5a4b772e4677f70b689b8b1826

I'm pretty sure the test failure is caused by a bug which is independent of my changes: pandas/io/data.py lines 174-182 should be indented and part of the else clause.

@evanpw ok, if you can fix the file modes, looks good otherwise

Mode is fixed, rebased, squashed, and tests are green.

ty sir!
DOC: reorg whatsnew file v0.16.0
State: closed

No real edits, just some rearranging.

I added an automatic 'contents' in the beginning, so this gives all headers automatically, and in this way restricted the list in the beginning to just the highlights (then it can be copied exactly like this to the release notes in release.rst


Labels:
name: Docs

Comments:
@jreback ok for you?

yep, lgtm.

will only be a 2-level menu on the left hand bar, but that's ok
DOC: fix some doc errors/warnings
State: closed

- `pd.read_csv('tmp2.sv', sep=None)` was given the warning "_ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'_"  
  Is this correct? (assuming so I just added what the warning message was saying)
- with the ordered -> unordered change in the Categoricals, the example on comparing them was failing (see http://pandas-docs.github.io/pandas-docs-travis/categorical.html#comparisons)


Labels:
name: Docs

Comments:
`sep=None` should default back to sniffing mode, which IIRC is only in `engine='python'`. So I think this is correct.

Maybe create an issue to investigate this and see if it really _should_ show the warning and/or should implement in `c` engine (this just involved using the sniffer so it should be just a bit of pre-processing)

This warning was already there in previous pandas releases, it was just not in the docs because the example was incorrect there (I fixed it recently). But indeed, let's open an issue to see if this could be added to the c engine.

Opened an issue here: #9645
Rearrange into logical sections and add relevant links
State: closed

This rearranges the Contributing page into more logical sections and provides some more useful links to wiki pages.  I think this should be the place where all relevant documentation on pandas development can be found.  For example, both this page and the pandas homepage mention `vbench` but the information on running tests was lacking and it turns out that the wiki has this information but it was difficult to find.  Hopefully this is a step in the right direction.


Labels:

Comments:
This looks like a nice improvement! Merging...

@rockg Thanks a lot for this! Just a comment about the numpydoc reference, but maybe I missed it.

It is at the top of the "Documentation / Commit Messages" section (I felt it was important).

Ah, so I missed it! :-)
Patch 4
State: closed


Labels:

Comments:
@nickeubank There is not need to close the PR, you can always just update your branch and push again. So even if you want to update the PR, change the implementation etc, just push again.

Ok -- that was just an error caused by me getting used to git hub. I'll
send a real pull request soon!
On Fri, Mar 13, 2015 at 6:10 AM Joris Van den Bossche <
notifications@github.com> wrote:

> @nickeubank https://github.com/nickeubank There is not need to close
> the PR, you can always just update your branch and push again. So even if
> you want to update the PR, change the implementation etc, just push again.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9638#issuecomment-78963772.

No problem!
fixed header=list (to create a MultiIndex) for read_excel
State: closed

closes #4679 

This is a very simple fix to allow passing a list go the header argument of read_excel so that it creates a MultiIndex for columns. Given what is said in issue #4679, I am not really sure it is supposed to work, but with this fix, it works for my files. One thing discussed in issue #4679, which I did not do anything about (and thus probably still does not work) is if the header cells are merged, but my opinion is that this should be a separate issue, as it is already useful to be able to read files with the header duplicated (the files I receive are usually like this).  If this PR is accepted (at least in spirit), the docstring for read_excel should probably be updated to reflect the "new" capability and the changelog updated, but I wanted to check first what you think.

As a side note, I am not entirely sure that _trim_excel_header is a good thing to do anyway, but that is another debate entirely.


Labels:
name: IO Excel
description: read_excel, to_excel
name: MultiIndex

Comments:
needs tests

@gdementen can you rebase this. What are all of the changes in index.pyx for?

First, sorry for letting this rot for so long.

I have since then realized that for my use cases, the code this PR changes (_trim_excel_header) should be removed altogether rather than changed. I have postponed proposing to do that because this code was probably put there for a reason and thus removing it will break backward compatibility for someone somewhere even though this seems like a more sensible way to do things (it makes the behavior the same than for csv files). If I knew what the use case this code is meant to handle, I would probably add an option for this, but I don't.

The hashtable.pyx changes are completely unrelated. I simply forgot to create a branch for my changes before sending the pull request... I will close this PR and create two separate ones.
BLD: add conda recipe (GH8934)
State: closed

closes #8934 


Labels:
name: CI
description: Continuous Integration

Comments:
so this recipe install the necessary deps for building, even on windows (this is what the dep on `libpython` does, it installs mingw which compiles everything.

Nice! 

I think it would make sense to create a 'pandas' organization (instead of under your name now), and start providing dev builds (if this is possible? not yet very familiar with binstar) so people who want to try it out don't have to build pandas themselves. So you could do something like:

```
conda install -c https://conda.binstar.org/pandas/channel/dev pandas
```
Return the right class, when subclassing a DataFrame with multi-level index
State: closed

Subclassing pandas.DataFrame had some issues, when a multi-level index was used. This PR should fix that.


Labels:
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
needs tests

What to test for, @jreback ? I can test, if subclassing works correctly, but there are other open issues on this topic and I understand it's work in progress.

well you need to assert that the correct class is returned

you create a dummy class do the operation and assert

not sure what you mean by the rest of your statement - exactly what doesn't work?

You're right. That's something that can be tested. I'll just add the use cases that were broken before. Thanks for the pointer.

whats new is now available for 0.16.1 https://github.com/pydata/pandas/commit/5ebf521eac528f5a4b772e4677f70b689b8b1826

Great. Now it pulled in some other commits as well. Is that an issue?

@manuelRiel If you do:

```
git fetch upstream
git rebase upstream/master
```

this should solve the extra commits

Yes. That did the trick. Usually I do a feature branch, but I forgot this time. So the PR is ok now?

@manuelRiel you need to `git push gitusername branchname -f`
in order to reflect these changes here.

Cleaned out the extra commits. We're good now?

pls add a release note in v0.16.1 (you can put in API changes).

@sinhrks can you take a look

Thanks for the heads up, guys. I have added the reference to the PR, as well as the release note. Let me know, if there is anything else to change.

pls rebase / squash, see [here for help](http://pandas-docs.github.io/pandas-docs-travis/contributing.html)

Done @jreback

@manuelRiel Thanks for the fix. Can you prepare test cases when the column is `MultiIndex`, because you're fixing `_getitem_multilevel`?  Pls check followings can return correct classes.

```
# comments are results of current master
mcol = pd.MultiIndex.from_tuples([('A', 'A'), ('A', 'B')])
df = CustomDataFrame([[0, 1], [2, 3]], columns=mcol)

type(df['A'])
# <class 'pandas.core.frame.DataFrame'>

mcol = pd.MultiIndex.from_tuples([('A', ''), ('B', '')])
df = CustomDataFrame([[0, 1], [2, 3]], columns=mcol)
type(df['A'])
# <class 'pandas.core.series.Series'>
```

Sure. I'll look into that. We're using multi-levels extensively and didn't run into issues. But it's possible I overrode this function in the subclass. I'll make sure it works properly.

Added the tests, @sinhrks . Returns custom class as expected.

@manuelRiel this looks fine. Can you rebase and ping on green.

Rebased and passing, @jreback 

@manuelRiel thanks!
DEPR: DatetimeIndex/PeriodsIndex +/- ops (GH9094)
State: closed

closes #9094
closes #9095 

deprecate `DatetimeIndex/PeriodIndex` `+/-` ops which are set ops. In a future version `-` will produce a `TimeDeltaIndex` and `+` a `TypeError`


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
surprise surprise, this was not actually tested anywhere AT ALL.

Should the same be done for the other operations?

Eg addition also still does a set operation:

```
In [12]: dti + dti
Out[12]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2012-01-01, 2012-01-02]
Length: 2, Freq: D, Timezone: None
```

Addition is not a usefull operation for datetimes as a numeric operation, but better to be consistent in the fact it will not be a set operation in the future I think.

@jorisvandenbossche yes, `dti + dti` should be a `TypeError` (in 0.17.0) to be consistent. (as is `dti + dt` now). will update.

This should then also close #9095 I think.

hmm, I guess we changed these all to `NotImplementedError` (which is actually more correct). but most are still `TypeErrors`...hmm

```
In [3]: pd.date_range('20130101',periods=5)+Timestamp('20140101')
NotImplementedError: 
```

```
In [4]: pd.date_range('20130101',periods=5)+Timestamp('20140101').to_datetime64()

TypeError: ufunc add cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')
```

@shoyer IIRC did you change these? (I know you suggested we did....)

`NotImplementedError` is wrong -- `pd.date_range('20130101',periods=5)+Timestamp('20140101')` should be a `TypeError`.

I believe I suggested changing some operations like `__add__` to return `NotImplemented`, but that's not `NotImplementedError` and it's something that the user should never see -- Python should translate it into a `TypeError` before it reaches the user.

@shoyer yeh I just looked at the code and everything is tested as a  `TypeError`, so maybe this odd that it is showing up like this.

I think this is implemented as an `NotImplementedError`, because this is within `DatetimeIndexOpsMixin` its method `_add_datelike`, and this method should be overwritten in `DatetimeIndex`? (because this should work for `TimedeltaIndex`)? 

And indeed, in `TimedeltaIndex` it is overwritten (https://github.com/pydata/pandas/blob/899212982af5978a5991dfd9517f3a0212bcdaa8/pandas/tseries/tdi.py#L316), but not in DatetimeIndex
TST/DOC: Fix tests and docs for .cat raising AttributeError if invalid
State: closed

Follow-up on #9617

Fixes #8814 -- is it fair to say that `hasattr(s, 'cat')` is probably
the best solution we're going to come up with for checking for categorical
data?

CC @jreback @jorisvandenbossche @JanSchulz


Labels:
name: Build
description: Library building on various platforms
name: Docs
name: Categorical
description: Categorical Data Type

Comments:
FYI this commit passed on my fork so we can safely merge this: https://travis-ci.org/shoyer/pandas/builds/53882855

ok, looks ok to me. go ahead and merge.
improve error message when importing pandas from source directory
State: closed

Currently if one imports from the source directory without building the C extension the error message is not very helpful:
`ImportError: cannot import name 'hashtable'`

I see a few previous PRs (https://github.com/pydata/pandas/pull/3827, https://github.com/pydata/pandas/pull/3821) that tried to address this but somehow this is still the behavior on `master`.

With this PR the error message would be a lot more informative:

```
ImportError: C extension: 'hashtable' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
```


Labels:
name: Build
description: Library building on various platforms

Comments:
@jreback @cpcloud please take a look

looks ok to me. @shoyer @jorisvandenbossche @cpcloud 
?

Indeed, looks good to me

@mortada thanks!
DOC: Shrink image asset for assign
State: closed

Let's try this again. The sphinx directive didn't seem to work. I just shrunk the static asset.

I'm not at my regular computer and I haven't been able to get the docs to build on this one, so I'm exactly sure how it will look in the page :)


Labels:
name: Docs

Comments:
Looks like the pyMySQL issue caused the failure. Cool with merging still?

yep go ahead

@TomAugspurger looks much bettter! thanks!
DEPR: deprecate pandas.rpy (GH9602)
State: closed

closes #9602 


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jorisvandenbossche pls have a look (failures are the Mysql issue #9608)

For me it is OK to already deprecate it, but I think we still should make sure that all existing functionality in pandas is ported to rpy2 (if any, but is should be checked)

And, also provide in the docs some explanation about how to move from the pandas api to rpy2

@jorisvandenbossche agreed. Nothing has really changed. But I think for example having `rpy2` add Categorical is a good idea.

ok that we keep the #9602 issue open for this
ENH: Add days_in_month property to Timestamp/DatetimeIndex/... (GH9572)
State: closed

closes #9572 

Added days_in_month property (compatibility alias daysinmonth) to Timestamp, DatetimeIndex, Period, PeriodIndex,
Series.dt. 

Although not mentioned in the GH issue, I added the property to Period and PeriodIndex to be consistent with the others: dayofweek, dayofyear, etc.


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype

Comments:
@josham looks gr8.

after I wrote up the issue I realized that this should prob be named `daysinmonth` (even though I like `days_in_month` better), for compat with the other field naming schema.

we _could_ change these to a proper naming convention, but that's a different issue.

see #9606 
actually might as well simply add both `days_in_month/daysinmonth` and call it a day. (and maybe just leave the tests alone).

OK, I'll add daysinmonth

thanks @josham 

keep em coming!
Test added and patch to fix python-version-dependent issues when len ro...
State: closed

...w/col_levels is 1.

The docs for sparse to_coo methods failed to build. There was some case (row_levels len 1) that failed in python 2.7 only that I failed to test (and I have been building docs in python 3). Have added test and patched. Also needed to expand the interator (list(map(... ) in the "# squish" line as there was some tupleizing differences between python 3 and 2.

Perhaps there is a better way to avoid these issues? Waiting for Travis.


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
Put the lmap in. Will check the docs again once this is merged.

@cottrell thanks!
Deprecation for 0.16 (#6581)
State: closed

These are the deprecations that are to be removed in 0.16, per #6581, except for the deprecations around the boxplot function.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@shoyer @jorisvandenbossche look ok to you? (other than the typo)

yep, looks ok!

@jsexauer thanks!
allow method=nearest for fillna. see issue #9471.
State: closed

closes #9471 


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: API Design

Comments:
This needs tests. Does it work on non-numeric dtypes? (e.g., strings?)

The latest commit should support all the methods from `interpolate()` and adds a test which checks for the same result as `interpolate()`. I'm not sure if this is a reasonable way to test this. The test is only for the DataFrame.

Unfortunately, it does not seem to work on non-numeric types, since scipy interpolation does not seem to support that case. I'm also unsure what distance metric should be used for strings.

Regarding your code comment, I changed the `axis=1` override to only happen if the incoming axis is None and one of the interpolate methods is used.

Regarding 'nearest', I can see these different scenarios:
1. index and values are both numeric.
   This should be mostly taken care of with calling out to `interpolate`.
2. index is numeric and values are non-numeric (in particular strings).
   This does not work with the above patch and should certainly be supported.
3. index is non-numeric and values are either numeric or non-numeric.
   The index still needs a distance metric (probably user defineable).
   This case seems more experimental and I'm not sure if it is even worthwhile doing.

Regarding case (2), this is what I am currently playing with:

``` python
import numpy as np
import pandas as pd
from pandas.util.testing import assert_frame_equal

df = pd.DataFrame({'A': ["a",np.NaN,"c",np.NaN,"e"]}, index=[1,2,5,9,10])
expected = pd.DataFrame({'A': ["a","a","c","e","e"]}, index=[1,2,5,9,10])

nan_inds = df[df.isnull().any(axis=1)].index.values
df_no_nan = df.dropna()

result = df.copy()
result.loc[result.isnull().any(axis=1), 'A'] = [result.loc[df_no_nan.index[df_no_nan.index.get_loc(i, method='nearest')]].values[0] for i in nan_inds]

assert_frame_equal(result, expected)
```

This seems very convoluted. I would be grateful for hints.

closing pls reopen if/when updated
BUG: Fix read_csv on S3 files for python 3
State: closed

Closes https://github.com/pydata/pandas/issues/9452

Needed to pass along encoding parameter.

No tests... Kind of hard since it's only s3 only.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks like there was already an existing test for reading from S3 :).

Good thing too! I was going to test this at work on py2, so I would have caught it :)

The failure cam from us using an old version of boto. The `encoding` parameter was added since 2.26.1. I bumped the version for our tests to the latest (2.36.0).

thanks @TomAugspurger