BrandonFors/pandas-github-issues huggingface dataset:
PERF/DOC:  Option to .info() and .memory_usage() to provide for deep introspection of memory consumption #11595
State: closed

closes #11595


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design

Comments:
cc @mrocklin

Does this descend into categories and the index?

wondering if you were going to ask that....it DOES do the index. not the categories, but I can fix this ......

Now includes embedded usage for Index & Categorical

```
In [5]:    df = DataFrame({'A' : ['foo']*1000})

In [6]:    df['B'] = df['A'].astype('category')

In [8]:    df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000 entries, 0 to 999
Data columns (total 2 columns):
A    1000 non-null object
B    1000 non-null category
dtypes: category(1), object(1)
memory usage: 16.6+ KB

In [9]:    df.info(deep=True)
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000 entries, 0 to 999
Data columns (total 2 columns):
A    1000 non-null object
B    1000 non-null category
dtypes: category(1), object(1)
memory usage: 55.7 KB

In [11]: df.memory_usage()
Out[11]: 
A    8000
B    1008
dtype: int64

In [12]: df.memory_usage(deep=True)
Out[12]: 
A    48000
B     1048
dtype: int64
```

And providing on `Series` as well

```
In [6]: df['A'].memory_usage()
Out[6]: 8000

In [7]: df['A'].memory_usage(index=True)
Out[7]: 16000

In [8]: df['A'].memory_usage(index=True,deep=True)
Out[8]: 56000
```

BTW, I'm glad that `memory_usage_of_objects` is usable on numpy arrays as well.  I may end up using that outside of pandas.

right `dask.array` could certainly introspect here as well

I'll wait until this is merged before adding `__getsize__`. 
Is there a reason `index` is `False` by default? I'd have thought that would be 'part of the package'. 

@maximilianr I don't recall the discussion, but I think we _should_ change the default. Note that this is just for a direct call to `memory_usage` and not for `.info` where it is included.

why don't you post an issue and we'll change in 0.18 (as its a small API change).
BUG/PERF: Stata value labels
State: closed

closes #12014 

This PR fixes a minor bug and introduces some performance enhancements, all related to value label reading in Stata files.  

The bug is that when reading a Stata file incrementally, the value labels will be read even when specifying convert_categoricals=False (this does not happen when reading the entire file at once).

The performance enhancements are:
1. Read key/value information as an ndarray using np.frombuffer rather than as a Python loop.
2. When splitting the value labels at the offsets, we currently pass txt[off[i]:] to null_terminate, which creates many copies of large segments of a potentially large byte array.  I changed it so that
   only the relevant part of the string is copied.

Relating to 2, further performance improvements might be possible since there is no trailing null byte to remove except for the last element of `txt` (thus some of the work in `_null_terminate` is superfluous).

Background:  This is an issue when processing large Stata files with millions of distinct value labels.


Labels:
name: Performance
description: Memory or execution speed performance
name: IO Stata
description: read_stata, to_stata

Comments:
ok, sounds gr8!.

just confirm that we have reasonable benchmarks in `asv` (the `packers.py`) file is the only one for stata.

Is there a place to put data files for ASV to read in?  The stata writer is more limited than the stata reader, so it's hard to test some features like strls that are not part of dta version 114.

you can just make a directory under the `asv_benchmarks`

@kshedden want to update

I added a test but haven't been able to get asv to run it.  I don't have conda so switched the asv conf file to use virtualenv.  It fails when using pip to install pytables:

`(py3)-bash-4.1$ pip install --upgrade pytables
Collecting pytables
  Could not find a version that satisfies the requirement pytables (from versions: )
No matching distribution found for pytables`

I'm using python 3.4.2rc1 (and also switched asv conf to use python 3.4)

asv _can_ work with pip but needs a slighty modified config file (e.g. a different one) as pip uses `tables` (not that _conda_ accepts both of these). 

pls add a whatsnew note (in performance). you don't necessarily need to include an asv benchmark (though nice if its easy), but post a perf comparison at the top of the issue.

Re conda vs. virtualenv, this may be of interest: https://github.com/spacetelescope/asv/issues/322#issuecomment-150848137 https://github.com/spacetelescope/asv/pull/329

@pv that's a nice feature....care to do a PR to upgrade `pandas/asv.conf.json`

@kshedden can you update

@kshedden can you update

Sorry for the delay... I wasn't able to test this in ASV.  I updated whatsnew, not sure what else needs to be done.

looks good
can u run flake8 on the diff and fix any issue

minor comments. pls squash. ping when green.

@jreback should be good to go

merged via https://github.com/pydata/pandas/commit/449ab6b6646812c21e859f71d4aabda2288e8ec4

thanks!
TST: Enable Index dtype comparison by default
State: closed

Enable `check_index_type` and `check_columns_type`  by default to detect dtype related problems.

Fixed following 2 minor bugs at the same time, but I'm willing to split them to separate PR if required.
### 1. `Series.quantile` with empty list results in `object` index rather than `float`.

```
pd.Series([1, 2, 3]).quantile([]).index
# Index([], dtype='object')
```
### 2. `pd.merge` includes empty data may result in `Int64Index` rather than `object`

Sorry, #10826 was incomplete.

When fixing #11497, #11586, we can enable corresponding `check_index_type`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks!
ENH: #3335 Pivot table support for setting name of margins column.
State: closed

closes #3335.

Adds margin_column parameter to pivot_table so that user can set it to
something other than 'All'.
Raises ValueError exception if there is a conflict between the value of
margin_column and one of the other values appearing in the indices of
the pivot table.

take into account notes from closed pull request: #10296


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
pls also add a whatsnew note (API changes)

done

you have to do `message {0}".format('blah')` so 2.6 doesn't complain

.format() change made, waiting on Travis.

Travis tests green ;)

thanks!
VIS: only apply shared axes handling on actual SubplotAxes
State: closed

closes #11556
closes #11520


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@sinhrks @TomAugspurger What do you think of this approach?

I think your fix looks fine, but I need to reread those issues to better understand what's going on. It could be a bit before I get to that, so I'm comfortable with merging this if you or @sinhrks are.

Looks nice! It's great to add tests of these axes for future cases.

```
# divider
from mpl_toolkits.axes_grid1 import make_axes_locatable
divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="5%", pad=0.05)
hasattr(cax, 'is_first_col')
# False

# insert_ax
from mpl_toolkits.axes_grid.inset_locator import inset_axes
iax = inset_axes(ax, width="30%", height=1., loc=3)
hasattr(iax, 'is_first_col')
# False
```

@sinhrks added those examples to the tests as well

lgtm. merge when ready.

merged via https://github.com/pydata/pandas/commit/9ee21391b59de46222c0a9cf4ae27cfebcaa79e3
BUG: Fix bug for kendall corr when in DF num and bool
State: closed

Hi,
1. When DataFrame contain Numerics and Booleans, than numpy will have type object,
   so `np.isfinite(mat)` will raise Exception.

I've fixed this by using `com._ensure_float64` like for other correlation.
1. I've skipped half of computation, because correlation is symmetrical


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@jreback Jeff, 
I've fixed handling NaN and added test for that.

couple of comments. pls add a whatsnew (put in bug fixes), use this PR number as the issue number. squash, then ping when green.

@jreback Done

thanks!

@jreback Thank you for pandas!
BUG: multi-index to_native_types is not passing thru parameters
State: closed

closes #7791
xref #6797


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: MultiIndex

Comments:
couple of comments. pls also add a whatsnew note (bug fix section)

Hum, `na_rep` has its own difficulty:

For a plain index:

```
In [79]: pd.DataFrame({'a': np.NaN, 'b':[2,3]}).set_index('a').index
Out[79]: Float64Index([nan, nan], dtype='float64', name=u'a')
```

and that's cool, but for a multi-index:

```
In [80]: pd.DataFrame({'a': np.NaN, 'b':[2,3]}).set_index(['a','b']).index
Out[80]: 
MultiIndex(levels=[[], [2, 3]],
           labels=[[-1, -1], [0, 1]],
           names=[u'a', u'b'])
```

so there are no `nan` values stored, and thus:

```
In [81]: pd.DataFrame({'a': np.NaN, 'b':[2,3]}).set_index(['a','b']).index.levels[0]
Out[81]: Float64Index([], dtype='float64', name=u'a')
```

You are setting everything to the index, which collapses the frame.

```
In [10]: pd.DataFrame({'a': np.NaN, 'b':[2,3], 'c' : [5,6]}).set_index(['a','b'])      
Out[10]: 
       c
a   b   
NaN 2  5
    3  6
```

having nans in an index is quite odd and only semi-supported, meaning some edge cases exist.

Even when there is a third column `'c'` that stays outside the index, we still get an empty `Float64Index`:

```
In [5]: pd.DataFrame({'a': np.NaN, 'b':[2,3], 'c' : [5,6]}).set_index(['a','b']).index
Out[5]: 
MultiIndex(levels=[[], [2, 3]],
           labels=[[-1, -1], [0, 1]],
           names=[u'a', u'b'])

In [6]: pd.DataFrame({'a': np.NaN, 'b':[2,3], 'c' : [5,6]}).set_index('a').index
Out[6]: Float64Index([nan, nan], dtype='float64', name=u'a')
```

If this is an "edge case", I'm pushing what I have done so far:
- what's new
- a short explanation for the test for `date_format`
- add `names=self.names` where it should be
- test if `quoting` is also passed to the multi-indexes

What I have NOT done:
- test if `decimal`, `float_format` (issue #11553), and `na_rep` are passed

ok, this looks good. pls squash and ping when green.

All done

merged via https://github.com/pydata/pandas/commit/9cbe8b966d4b58f271625dd3415534304f32a5cd

thanks!
ENH: allow dataframe get to take an axis argument
State: closed

closes #6703

I would have like to have a solution also working for panels, but I've never used them, so I'm not sure how it should behave.


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Ok, I used your suggestion and put the tests in `tests/test_generic.py`... but it still is the same test. I should probably make more tests, in particular to test what happens for Panel.

(Sorry, Atom couldn't help but by itself remove some spaces that were at the end of their lines)

I don't understand, why is the build failing?

If you go to 'Details', you can get the build's output. In this case:

```
======================================================================
FAIL: test_getitem_get (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_series.py", line 1339, in test_getitem_get
    self.series.get(-1), self.series.get(self.series.index[-1]))
AssertionError: None != -0.2267325702945841
```

Uh, yes, thanks.

I think I've identified what is going on, but I'm not sure on how to proceed:

```
In [56]: pd.Series([0,1], index=[0,1]).get(-1)

In [57]: pd.Series([0,1], index=['a','b']).get(-1)
Out[57]: 1
```

So, basically, the result of the `get` method for a series depends on the type of the index... Which is not the case for a dataframe:

```
In [63]: pd.DataFrame([[0,1],[2,3]], columns=['a','b']).get(-1) is None
Out[63]: True
```

I could redefine `Series.get` to supersede `NDFrame.get`... but I don't know if that's a good solution. Maybe there are other classes inheriting from `NDFrame` that work like `Series`?

Here is what I'm proposing: if the solution based on the slice and `loc` fail, and `axis` is `None`, we revert to the old implementation. Thus we maintain a backward compatibility...

I'm not sure this is a good idea. Get is a standard method for mapping types and we already have a lot of indexing methods.

yeh, this is getting into too many edge cases. Lets just keep `.get` simple.
BUG: GH11517 add multiindex column names after describe()
State: closed

Hi,

This fixes the bug mentioned in https://github.com/pydata/pandas/issues/11517
I have added column names after the describe operation. 
Request you to review the change and merge it. 

Thanks 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Please add a test for this :)

Hi,

I have added test case for that . 
Request you to review the change . 

pls add a whatsnew note for 0.17.1

Thanks for comments.
I have modified test case and also added comment to whatsnew 0.17.1
Please review.. 

couple of comments. pls squash when done. ping on green.

I have made the changes. 
Request you to please review and merge. 

looks good. pls squash to a single commit. ping when green.

Done. Please check.

pls squash: http://pandas-docs.github.io/pandas-docs-travis/contributing.html#combining-commits

Apologies for not getting it right the first time
Please check.

thank you sir!

@varun-kr @jreback 

Thanks for fixing this bug. :+1:  I'll have to get on the bandwagon and submit some pull requests also.
BUG: df.join(df2, how='right') TypeError
State: closed

Issue #11519

Somehow right joins had been forgotten in a previous bugfix. There were tests already written that should have seen the problem, but they had been commented out because the expected results were wrong, because of a subtlety in the way non-unique index are handled.

Thanks for the labels "difficulty novice" and "effort low": I'm using pandas every day, and I'm glad I could contribute that easily.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Ok, the commented test is redundant with what is just before, so I'm removing it.

pls add a whatsnew note (bug fixes) & squash

Done

can you add all of the examples from the issue as well. (in `tests/tools/test_merge.py`) would be a good location

I've put them in what I thought was the most appropriate test method, I hope that's all right.

merged via https://github.com/pydata/pandas/commit/d78266eb396143438d187eb8d5a8cb750c2ba737

thanks!

@nbonnotte @jreback 

Thanks for closing this out! :+1:
BUG: fix incorrect xticklabels when specifying xticks (GH11529)
State: closed

Fixes #11529

@sinhrks in PR #10717 you moved the setting of the ticklabels from `_adorn_subplots` to its own function `_post_plot_logic_common`. But this is called before `_adorn_subplots`, and the specified `xticks` are only set in `_adorn_subplots`, so now it does not use the new xticks to determine the xticklabels. 
I just switched the order of the two functions. At first sight this seems OK, but do you know if there could be any side effects of switching the order?


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Thanks for the fix. The fix looks valid (I've overlooked the side effect in #10717).
Updated read_excel docstring to include parse_dates and date_parser
State: closed


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
I'm not sure those keywords actually do anything with the excel parser?

``` python
In [12]: dti = pd.date_range('2014-1-1', periods=10)

In [13]: df = pd.DataFrame({'dates':dti, 'strings':dti.strftime('%m/%d/%Y')})

In [14]: df.dtypes
Out[14]: 
dates      datetime64[ns]
strings            object
dtype: object

In [15]: df.to_excel('test.xlsx')

In [16]: pd.read_excel('test.xlsx').dtypes
Out[16]: 
dates      datetime64[ns]
strings            object
dtype: object

In [17]: pd.read_excel('test.xlsx', parse_dates=True).dtypes
Out[17]: 
dates      datetime64[ns]
strings            object
dtype: object

In [18]: pd.read_excel('test.xlsx', parse_dates=False).dtypes
Out[18]: 
dates      datetime64[ns]
strings            object
dtype: object
```

yeh, I don't think these are valid keywords. Actually what we really need here is a check on non-implemented keywords. Closing this and I will create another issue.

@chris-b1 It actually even gives an error and is not just ignored. With your example (as `parse_dates=True` if for parsing the index, so if you want to see if it can parse the string column, you have to pass its name):

```
In [37]: pd.read_excel('test.xlsx', parse_dates=['strings'])

....

C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in _should_parse_dates(self,
 i)
    812             return self.parse_dates
    813         else:
--> 814             name = self.index_names[i]
    815             j = self.index_col[i]
    816

TypeError: 'NoneType' object has no attribute '__getitem__'
```

But if you set the `strings` column as the index, `parse_dates=True` is indeed ignored.

@chris-b1 This error actually only happens if you have an implicit index due to the structure of the excel file. If you don't have this, `parse_dates` works as expected:

```
In [46]: df.to_excel('test.xlsx', index=False)

In [47]: pd.read_excel('test.xlsx').dtypes
Out[47]:
dates      datetime64[ns]
strings            object
dtype: object

In [48]: pd.read_excel('test.xlsx', parse_dates=['strings']).dtypes
Out[48]:
dates      datetime64[ns]
strings    datetime64[ns]
dtype: object
```
fix _is_view for multi-block frames
State: closed

_is_view previously only testing whether first block was a view, when should return `_is_view = True` if _any_ block is a view. 


Labels:

Comments:
read my comment in the code

if u think his is necessary then it should go in your other pr

@jreback OK -- need to be more conservative for copy-on-write. Will move into other PR
BUG: GH10355 groupby std() doesnt sqrt grouping cols
State: closed

New attempt at #10355 Hopefully should address the issues raised in #11300

Previously, grouping columns were square rooted when as_index=False
We now test whether the grouping keys are in the columns, and
if so don't square root those columns.

Note that we squash TypeError which occurs when self.keys is not
Hashable, and so we can't check for existence in columns.


Labels:
name: Bug
name: Groupby

Comments:
pls add a whatsnew note in 0.17.1 bug-fixes

@jreback 

My if tests logic finds columns used to make the group. This is to ensure their values are not square rooted.
How can I use self._selected_obj to discover which columns are being used in the grouping?

```
In [27]: df = pandas.DataFrame({
               'a' : [1,1,1,2,2,2,3,3,3],
               'b' : [1,2,3,4,5,6,7,8,9],
})

In [21]: g = df.groupby('a',as_index=False)

In [22]: g._set_selection_from_grouper()

In [23]: g._selected_obj
Out[23]: 
   a  b
0  1  1
1  1  2
2  1  3
3  2  4
4  2  5
5  2  6
6  3  7
7  3  8
8  3  9

In [24]: g = df.groupby('a',as_index=True)

In [25]: g._set_selection_from_grouper()

In [26]: g._selected_obj
Out[26]: 
   b
0  1
1  2
2  3
3  4
4  5
5  6
6  7
7  8
8  9
```

these `_set_selection_from_grouper()` functions are called when functions are actually run (e.g. you actually call `.std()`). So you can then use the `._selected_obj` for what the actual data (excluding the groupings is).

@jreback 

Thanks for your quick reply.

As you can see in your output on line [23] _selected_obj doesn't exclude the grouping when as_index=False
It only excludes the grouping when as_index=True, and that isn't the case we're dealing with.

Sorry if I'm missing something.

use this, though maybe something more sophisticated in there. you have to test using levels as well. Everything is there in the grouper objects, you just have to look for it. Don't reinvent the wheel.

```
In [36]: g.grouper.names 
Out[36]: ['a']
```

pls rebase / update according to comments

can you rebase / update according to comments

can you update?

can you update

closing. but pls reopen if you'd like to update

https://github.com/pandas-dev/pandas/pull/25315
BUG: .loc with duplicated label may have incorrect index dtype
State: closed

`.loc` result with duplicated keys may have incorred `Index` dtype.

```
import pandas as pd

ser = pd.Series([0.1, 0.2], index=pd.Index([1, 2], name='idx'))

# OK
ser.loc[[2, 2, 1]].index
# Int64Index([2, 2, 1], dtype='int64', name=u'idx')

# NG, Int64Index(dtype=object) 
ser.loc[[3, 2, 3]].index 
# Int64Index([3, 2, 3], dtype='object', name=u'idx')
ser.loc[[3, 2, 3, 'x']].index 
# Int64Index([3, 2, 3, u'x'], dtype='object', name=u'idx')

idx = pd.date_range('2011-01-01', '2011-01-02', freq='D', name='idx')
ser = pd.Series([0.1, 0.2], index=idx, name='s')

# OK
ser.loc[[pd.Timestamp('2011-01-02'), pd.Timestamp('2011-01-02'), pd.Timestamp('2011-01-01')]].index
# DatetimeIndex(['2011-01-02', '2011-01-02', '2011-01-01'], dtype='datetime64[ns]', name=u'idx', freq=None)

# NG, ValueError
ser.loc[[pd.Timestamp('2011-01-03'), pd.Timestamp('2011-01-02'), pd.Timestamp('2011-01-03')]].index
# ValueError: Inferred frequency None from passed dates does not conform to passed frequency D
```
## After the PR:

Above OK results are unchanged.

```
import pandas as pd
ser = pd.Series([0.1, 0.2], index=pd.Index([1, 2], name='idx'))

ser.loc[[3, 2, 3]].index 
# Int64Index([3, 2, 3], dtype='int64', name=u'idx')
ser.loc[[3, 2, 3, 'x']].index 
# Index([3, 2, 3, u'x'], dtype='object', name=u'idx')

idx = pd.date_range('2011-01-01', '2011-01-02', freq='D', name='idx')
ser = pd.Series([0.1, 0.2], index=idx, name='s')

ser.loc[[pd.Timestamp('2011-01-03'), pd.Timestamp('2011-01-02'), pd.Timestamp('2011-01-03')]].index
# DatetimeIndex(['2011-01-03', '2011-01-02', '2011-01-03'], dtype='datetime64[ns]', name=u'idx', freq=None)
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Updated and now green.

@sinhrks can you update

@sinhrks yeh, prob requires some playing around to get this to work nicely. Like your separation of concerns though. We should document this prob at top of core/index.py

How about adding `_infer_new` as below?
- `_infer_new`: The same as current `_shallow_copy(infer=True)`. Always tries to infer correct type.
- `_shallow_copy`: Remove `infer` kw. Always use the same type as the caller.

We can replace `_shallow_copy` by `_get_attribute_dict` + `_simple_new` completely, but I didn't do because adding `_get_attribute_dict` is redundant / easy to be missed.

If OK, I'll squash.

@sinhrks seems reasonable call it `_shallow_copy_with_infer`? to make the connection with `_shallow_copy`

@jreback OK, renamed. Also added general explanation about `simple_new`, `shallow_copy` and `shallow_copy_with_infer`. Could you check?

thanks @sinhrks and nice docs!
ENH: support Akima 1D interpolation
State: closed

I went hunting for some low-hanging fruit in the area of missing data and came across #7588. I had to implement it a little differently than krogh, pchip, etc. because there is no convenience function for Akima like there are for the already supported methods.

Once #11445 is merged I can rebase.


Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
ok, can you rebase on master

if you can do this on master (and not depend on the other branch), we can take it.

pls reopen if you wish to do so

I was just working with akima and I think that there is an issue.  I've posted a bug report here:   [https://github.com/pandas-dev/pandas/issues/21276](https://github.com/pandas-dev/pandas/issues/21276)