BrandonFors/pandas-github-issues huggingface dataset:
Deprecate detection of IPython frontends
State: closed

The check for the Qt console won't be necessary in IPython 3 and above: we decided that the Qt console's display of HTML reprs was so bad that it shouldn't attempt to show them. We also want to remove the `parent_appname` config value (ipython/ipython#4980), which will break both of these functions (making them always return False).

The assignment to `ipnbh` was redundant - nothing used that variable. I guess it was used previously and didn't get cleaned up during some changes (quite possibly my own omission).


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
About the qtconsole, then this issue #7372 can maybe closed? (which is a nice illustration of how bad it can be ...)

I've rebased this pull request - I think it's ready for review.

is their a min ipython version that this requires? (we don't test this per se), e.g. will this break with an older version of ipython but pandas 0.14.1/master? 

I've not actually changed anything functionally (the assignment I removed is an unused variable), so no, it won't break.

When IPython 3.0 has been out for a while, I'll make another PR to remove this code. This deprecation is just to ensure pandas doesn't start using this functionality for anything new while we're removing it.

so if someone uses ipython 0.11 it will still work the same?

Yes - this PR should not change how anything actually works whatsoever. All it does is:
- Add deprecation notes in comments and docstrings
- Remove an assignment to an unused variable

thanks!
BUG: Bug in .loc performing fallback integer indexing with object dtype indices (GH7496)
State: closed

closes #7496

```


In [4]: s = Series()

In [5]: s.loc[1] = 1

In [6]: s.loc['a'] = 2

In [7]: s.loc[-1]
KeyError: 'the label [-1] is not in the [index]'

In [8]: s.loc[[-1, -2]]
Out[8]: 
-1   NaN
-2   NaN
dtype: float64

In [9]: s.loc[['4']]
Out[9]: 
4   NaN
dtype: float64

In [10]: s.loc[-1] = 3

In [11]: s.loc[[-1,-2]]
Out[11]: 
-1     3
-2   NaN
dtype: float64

In [12]: s['a'] = 2

In [13]: s.loc[[-2]]
Out[13]: 
-2   NaN
dtype: float64

In [14]: del s['a']

In [15]: s.loc[[-2]] = 0
KeyError: '[-2] not in index'
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
```
s.loc[[-2]] = 0
```

shouldn't `-2` to be added to index after that?

this one looks similar https://github.com/pydata/pandas/issues/5420

for the `s.loc[[-2]] = 0`.

Setting with a single indexer (e.g. `s.loc[-2] = 0` is fine as this is enlargement.
However, _setting_ with a list (either on a Series or a DataFrame) was deemed to be
more 'error prone', so it was disallowed.

e.g. both `s.loc[[-2]] = 0` and `df.loc[[-2]] = 0` are not allowed unless the location already exists.
though `s.loc[-2] = 0` and `df.loc[-2] = 0` will work
DOC: Fix index error for remote data docs.
State: closed

Fixes the index error caused by #7484.  Also removes duplicate example in v0.14.1.txt


Labels:

Comments:
This fixes the index error @jorisvandenbossche.  

OK. Another small thing, can you change ``#5602``  in v0.14.1.txt to use the issue linking? (-> `:issue:`5602``

Done and squashed into one commit.

Thanks!
ENH/BUG: Period/PeriodIndex supports NaT
State: closed

Closes #7228. Closes #4731.

`Period` and `PeriodIndex` now can contain `NaT` using `iNaT` as its internal value.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Period
description: Period data type

Comments:
looks fine, ping when green

@jreback Thanks, get green!

excellent thanks!
DOC: Clean up docs for io.data.Options.
State: closed

Per conversation on #5602, adjusted some of the docs.
Removed from release.rst, changed example strike price for AAPL, fixed backticks.


Labels:
name: Docs

Comments:
@jorisvandenbossche 
@jreback 

@jorisvandenbossche wondering if should somehow limit the columns so the display is not so wide...

?

We could do that - maybe cut it off at volume so it doesn't wrap? 

The downside being that you wouldn't see all the available information that is downloaded - ie: open int, underlying price, quote time, etc.  We could add that detail in the text.

the doc-string is pretty comprehensive, this is just an example

Good point.  Changed it to show only to volume.

@jreback we could set `pd.options.display.max_columns` to a lower number, but otherwise, for the docs maybe better the user sees the same thing when they try it out themselves.

no that's ok, only thing may want to revist the screen width in general in the docs? shouldn't it be larger?

We have had that discussion before #4165, I think in the first place we should make our examples so that the output is not too wide.

@dstephens99 There is now an indexing error: http://pandas-docs.github.io/pandas-docs-travis/remote_data.html#yahoo-finance-options
BUG/WIP: fix pivot with nan indexes
State: closed

closes #7466
- [ ] vbench
- [ ] make sure there isn't a better way to do this


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
seems ok...(though didn't look real close :)

haven't found the time to do the vbench ... not reeally a whole of time when i don't have abunch of things running (except when i'm sleeping :)

I moved these to 0.15.0, but if you are ok then can throw in 0.14.1

ok just not too comfortable with this fix yet, we can wait until 0.15.0

@cpcloud status? (if not move to 15.1)

I think #9292 (and #9061) fixed all of these
BUG: Make copies of certain interpolate arguments (GH7295)
State: closed

closes #7295
On the alt_methods branch in interpolate_1d, make copies if x, y, and new_x don't have their `writeable` attribute set.  It seems a little weird that we need to do this, but it'll work around the issue so that the test can pass, anyway.


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
any specific tests that need to add (aside from the 32-but failures)
pls add release note in 0.14.1
can u rebase everything always and squash to a single commit

thxs

Yep, was looking up my "how to rebase and squash" tutorial right now.  I don't think there are any new tests, though.  Are minor and obscure notes like this really worth a release note, though?

right - forgot the release note then
ok on tests (assume it passes your system!)

thanks!
Fix cache key collision and add test for cache key distinctness.
State: closed

Fixes #7420.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timezones
description: Timezone data dtype

Comments:
tests ok in windows?

I ran tests for python 3.4.

I can run tests for earlier versions tomorrow if you like.

On 16 June 2014 22:01, jreback notifications@github.com wrote:

> tests ok in windows?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7478#issuecomment-46236814.

this looks good....all tests green on windows now!

thanks!

Fantastic, at last! Thanks very much for your help.

no thank you!
BUG: Bug Multiindex assignment with an aligned rhs (but at different levels) (GH7475)
State: closed

closes #7475


Labels:
name: Bug
name: Enhancement
name: MultiIndex

Comments:
@jreback Can you document why you closed it for future reference (issue is still open)?

I don't think is really possible in any meaningful way (and its a badly posed problem).

IOW the user is responsible.
ENH: Cast ndarray-like datetime64 arrays to Index properly
State: closed

It turns out that the ndarray-like arrays of dtype `datetime64` were
not being properly cast to an `Index`, because -- due to a bug with
`np.datetime64` -- calling `np.asarray(x, dtype=object)` if x is an
ndarray of type `datetime64` results in an _integer_ array.

This PR adds tests and a work around to `pd.Index.__new__`.

Related #5460


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
thanks

seem inocuous enough....famous last words though :)
BUG: Some offsets.apply cannot handle tz properly
State: closed

There are some offsets which cannot handle input with `tz` properly

```
pd.offsets.Day().apply(pd.Timestamp('2010-01-01 9:00', tz='US/Eastern'))
#2010-01-02 09:00:00-05:00 (Expected)

pd.offsets.CustomBusinessDay().apply(pd.Timestamp('2010-01-01 9:00', tz='US/Eastern'))
#2010-01-04 09:00:00 (tzinfo lost)

pd.offsets.CustomBusinessMonthEnd().apply(pd.Timestamp('2010-01-01 9:00', tz='US/Eastern'))
# ValueError: Cannot compare tz-naive and tz-aware timestamps
```
### Affected Offsets
- pandas.tseries.offsets.CustomBusinessDay'
- 'pandas.tseries.offsets.CustomBusinessMonthEnd'
- 'pandas.tseries.offsets.CustomBusinessMonthBegin'
- 'pandas.tseries.offsets.BusinessMonthBegin'
- 'pandas.tseries.offsets.YearBegin'
- 'pandas.tseries.offsets.BYearBegin'
- 'pandas.tseries.offsets.YearEnd'
- 'pandas.tseries.offsets.BYearEnd'
- 'pandas.tseries.offsets.BQuarterBegin'
- 'pandas.tseries.offsets.LastWeekOfMonth'
- 'pandas.tseries.offsets.FY5253Quarter'
- 'pandas.tseries.offsets.FY5253'
- 'pandas.tseries.offsets.Week'
- 'pandas.tseries.offsets.WeekOfMonth'
- 'pandas.tseries.offsets.Easter'


Labels:
name: Bug
name: Frequency
description: DateOffsets
name: Timezones
description: Timezone data dtype

Comments:
thanks
BUG: unstack fails for mixed dtype subset
State: closed

closes #7405


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback ok with this?

weird that works - 

there might be a better way to fix it ... i'll look a little deeper

this way makes more sense, since we want the result of an unstack op to return the requested levels taking into account the labels ... before this fix, wasn't doing that if the unstack index result was a single level (coming from a multiindex)

do u have an indexer? this is better done with take (though what u r doing should work if the level is unique I think)

if i understand what you mean by indexer then yes `result_labels` is a list of labels for each level that are indexer for the level

@jreback this ok? the length check is really only for the one case ... feels a bit strange, but i couldn't see another place to put the fix ... welcome a suggestion

you can't just use `take` because there could be `nan`s in `result_labels`, in all other cases (so far) the length is equal, i guess in theory u could still have a `nan` if the lengths aren't equal

let me see if i can come up with a test for that last case

I guess it's fine

though maybe could add some tests to try to break it

looks ok...squash and merge

i'm not totally sure this is the best way to go about this .. i'll spend some time this weekend trying to break my impl

@cpcloud status?

was originally closed by #9292
BUG/CLN: LinePlot uses incorrect xlim when secondary_y=True
State: closed

`xlim` is not set properly when `secondary_y=True`. This is different issue from #7322.

Also, refactored `secondary_y` to be handled only by `MPLPlot`, because it will be required in future fix to pass multiple axes to `df.plot`, like #7069 (I'm willing to work soon..).

```
df = pd.DataFrame(np.random.randn(5, 5), columns=['A', 'B', 'C', 'D', 'E'])
df.plot(secondary_y=True)
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/3278423/15d4ba78-f3be-11e3-9d8e-172c9d0b754b.png)

```
df = pd.DataFrame(np.random.randn(5, 5), columns=['A', 'B', 'C', 'D', 'E'])
df.plot(secondary_y=['A', 'B'], subplots=True)
```

![figure_2](https://cloud.githubusercontent.com/assets/1696302/3278424/1b2dae44-f3be-11e3-8b8c-b19f5ae7fef3.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@cpcloud 

@cpcloud ?

@sinhrks ?

@jreback Updated and rebased based on #7322 and #7515. 
cc @rosnfeld

@TomAugspurger 

cc @rosnfeld ok with this?

Not sure how deep a review I can do as I am just about to head out abroad again (more or less offline until August), but this does seem to behave properly in various ad-hoc tests, and seems to clean up a few things that surprised me when I first started looking at plotting code. A somewhat larger/more dramatic change than I would have expected, but glad to see that larger changes are possible/being done to the somewhat thorny (IMO) plotting code.

@sinhrks thanks! @rosnfeld thanks for the commentary

if you guys find addtl issues, pls report!
BUG: Better axis label handling for partial layout
State: closed

Closes #5897.

Handle `ticklabels` and `labels` based on below rules. Currently `labels` are always displayed even if `ticklabels` are hidden and causes confusion.
- If `sharex` is `True`, display only most bottom `xticklabels` on each columns. (Because #7035 hides the bottom-right axes). Hide `xlabel` as the same manner as `xticklabels`
- If `sharey` is True, display most left `yticklabels` (no change). Hide `ylabel` as the same manner as `yticklabels` (changed)

```
import pandas as pd
from numpy.random import randn
import matplotlib.pyplot as plt

d = pd.DataFrame({'one':randn(5), 'two':randn(5), 'three':randn(5), 'label':['label'] * 5},
        columns = ['one','two','three', 'label'])
bp= d.boxplot(by='label', rot=45)
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/3074667/70d267ea-e34c-11e3-8996-24db3e31ed67.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@cpcloud 

@cpcloud ?

@cpcloud ? @sinhrks can you rebase

thanks!
FIX: Enable fixed width strings to be read from Stata 13 (117) files
State: closed

Fixes a bug which prevented files containing fixed width string data from being read.  Stata 13 files also allow variable length strings, which are not supported in the current version, and an explicit exception regarding this type is now given.  Added tests which cover these cases, and Stata 13 format files. 

fixes #7360


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
I think this fix is ready - it is a bit ugly due to some existing PY3 compatibility code that probably has string encoding in the wrong place, but it is working.

thanks!
_sanitize_column now reports proper duplicate error
State: closed

closes #7432
A very simple fix for GH7432. `internals.py` (line 3240) actually raises the correct exception (`ValueError: cannot reindex from a duplicate axis`) but the except: doesn't allow it to be raised. Here I've just allowed the exception itself to be raised since it's more useful to the user than the weird `TypeError` message.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Here's what the situation in [Issue 7432](https://github.com/pydata/pandas/issues/7432) now looks like:

```
In [4]: df['newcol'] = ser
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-acaa0ccb832e> in <module>()
----> 1 df['hello'] = ser

/home/rob/pandas_roblevy/pandas/core/frame.py in __setitem__(self, key, value)
   1853         else:
   1854             # set column
-> 1855             self._set_item(key, value)
   1856 
   1857     def _setitem_slice(self, key, value):

/home/rob/pandas_roblevy/pandas/core/frame.py in _set_item(self, key, value)
   1921         is_existing = key in self.columns
   1922         self._ensure_valid_index(value)
-> 1923         value = self._sanitize_column(key, value)
   1924         NDFrame._set_item(self, key, value)
   1925 

/home/rob/pandas_roblevy/pandas/core/frame.py in _sanitize_column(self, key, value)
   1964                     value = value.reindex(self.index).values
   1965                 except ValueError as e:
-> 1966                     raise e
   1967                 except:
   1968                     raise TypeError('incompatible index of inserted column '

ValueError: cannot reindex from a duplicate axis
```

yep..looks trivial...can you rebase (so that travis will then run).

thanks!

the importance of testing! turns out their are some other error messages here that are effectively raising as TypeError, fixed up here: https://github.com/pydata/pandas/commit/7ef4efa306bbd8685f849c9b65070523b10ecb15

Amazing. Next time I do a pull request I'll add a test for whatever I add. This is a good learning experience for me...
make nanops work when ndim==1 and axis==0 ( issue #7354 )
State: closed

This fixes issue #7354, where some `nanops` functions fail for 1-dimensional arrays for the argument `axis=0`.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
as an aside do you test for invalid axis (>ndim)

No, currently there is no exception testing.

ok easy enough to add assert statements inline - when u have a chance

The travis build is passing and the pull request should be up-to-date, so this should be ready to merge.

merged via https://github.com/pydata/pandas/commit/586e3173cce75373f9bf395d9c0709c67613956a

thanks!
API: Improved inference of datetime/timedelta with mixed null objects. (GH7431)
State: closed

- regression from 0.13.1 in interpretation of an object Index
- additional tests to validate datetimelike inferences

closes #7431


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@cpcloud @jorisvandenbossche have a look

this inference biz is pretty complicated; hopefully sightly less complicated now (issue is that you can easily mess up perf if you keep trying to infer object dtypes).

technically I did change one API here, in that 

In 0.14.0

```
In [1]: from datetime import timedelta

In [2]: Series([timedelta(1)])
Out[2]: 
0   1 days
dtype: timedelta64[ns]

In [3]: DataFrame([timedelta(1)])
Out[3]: 
                0
0  1 day, 0:00:00

In [4]: DataFrame([timedelta(1)]).dtypes
Out[4]: 
0    object
dtype: object
```

Now

```
In [2]: DataFrame([timedelta(1)])
Out[2]: 
       0
0 1 days

In [3]: DataFrame([timedelta(1)]).dtypes
Out[3]: 
0    timedelta64[ns]
dtype: object
```

So was not tested before

I can't comment on the code, but for the API change, it was indeed no problem to merge I think as it was actually more a bug that is fixed. And not one that too many people will rely upon I think, as eg with `DataFrame({'A':[timedelta(1), timedelta(2)]})` it already did work as expected before.
make nanops._maybe_null_out work with complex numbers ( issue #7353 )
State: closed

This fixes #7353 where `nanops._maybe_null_out`, and thus other functions that call on it, don't work with complex numbers.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
excellent! 

The travis build test passes. I just rebased it, so it should be ready to merge.

thanks!
add unit tests for nanops
State: closed

This adds unit tests for `pandas.core.nanops`.  It also moves existing nanops tests from `test_common` to `test_nanops`.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@toddrjen nice....

@toddrjen take a look and see if you can divine what's wrong

https://github.com/pydata/pandas/issues/7520

these are from debian systems

if you need more info pls ask on that issue
FIX value_counts should skip NaT
State: closed

fixes #7423  
fixes #5569.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
Hmm this seems to be explicitly tested for in `test_base` and `test_timeseries`. I still think correct behaviour is to drop the NaTs...

The test was added based on the discussion in #6734. I agree it will be clearer if `NaT` is treated as the same as `nan`. 

so this makes this consistent with `nan` (which is excluded), separately, maybe have an option to `value_counts` to _include_ `nan/NaT`?

@jreback yes, shouldn't be too trick to put something together. I was thinking the same when I was doing this... of some analogous `include_na` (maybe `get_dummies`... called `dummy_na`)... I suppose `dropna` is more standard kwarg.

yes, `dropna=True` (as the default) is prob correct. Pls add this. I think we might want to change the default to `dropna=False` in 0.15.0 but that is a separate issue.

@hayd looks good...squash and merge

thanks!

@hayd some comments, but will fix it in a PR

@jorisvandenbossche thanks! good points!
TST7337: Fix more tests in windows
State: closed

Hopefully final attempt at fixing the windows test issues from dateutil timezone work. Related discussion on previous pull request [here](https://github.com/pydata/pandas/pull/7362).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Windows
description: Windows OS
name: Timezones
description: Timezone data dtype

Comments:
Thanks for sticking it out!

Np :)  Thanks for your patience. I didn't expect getting the tests passing on windows to be so difficult..

@dbew thanks for the update.....I'll check the windows results later tonight.
BUG: fix repring of nat multiindex and fix neg indexing in datetimeindex
State: closed

closes #7406, #7408, #7409


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@jreback good 2 go?

3x issues +1!

Neat!
to_stata uint16
State: closed

Simple changes to io/stata.py to write unsigned integers to Stata files. Sorry about the poor commit messages--I'm still new to git and wasn't able to correct it cleanly.

(See issue #7365)


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO Stata
description: read_stata, to_stata

Comments:
you need to add some tests in io/tests/test_stata.py

you create a frame with several dtypes then round trip and compare with the original

these tests should fail (with uint16 dtypes) before your fix and pass after

I'm not sure this is a good idea since Stata doesn't support any unsigned data types.  In particular some of the code which uses the numpy datatypes in this commit will result in data loss.  

The only place where uint\* handling should be added is in the casting code.  Once the columns have been cast to supported Stata datatypes then only the supported Stata datatypes should be used when writing the Stata data types (e.g. `char(253)`)

Also noticed a clear bug in (my) the code

https://github.com/dmsul/pandas/blob/tostata-uint16/pandas/io/stata.py#L244

should be 

```
if data[col].max() <= 2 ** 53 or data[col].min() >= -2 ** 53:
```

The first comparison has `*`, not `**`.  Would be good to fix this.

@dmsul ok, then this needs to be changed to coerce `uint` types to `int` (and if they are out of range, then I would raise (e.g. a >`int64` value from a `uint64`)
BUG: grouped hist and scatter use old figsize default
State: closed

Grouped `hist` and `scatter` plots use `figsize` default which is different from pandas current and mpl.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@TomAugspurger ?

@sinhrks what's the default figure size after this PR?

regardless, this is ok, just rebase and i'll put this in

@cpcloud Default will be either the mpl default, or pandas default (`pd.options.display.mpl_style`).

I rebased.

@cpcloud ?