BrandonFors/pandas-github-issues huggingface dataset:
BUG: pd.unique should respect datetime64 and timedelta64 dtypes (GH9431)
State: closed

To fix [GH9431](https://github.com/pydata/pandas/issues/9431).

Previously `pd.unique` would return an array of  `object` dtype when passed a 1D array, Series or Index with a `datetime64` or `timedelta64` dtype. This PR should remedy that behaviour.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
I can add the extra tests and make the changes there, no problem.

I did originally put the `if`/`else` logic in `_hashtable_algo`. There it works great for `unique`, but I realised it messed up `pd.match` for datetime64 arrays (which also calls `_hashtable_algo`). Here instead of returning an array of integer indices, `pd.match` returns a view of those indices as datetime64 because `_hashtable_algo` is only told what hashtable needs to be built, not what the dtype of the returned array needs to be. 

Putting the logic in `pd.unique` itself looked to be by far the most straightforward way to solve this, but I'm happy to try and see if any other approaches are viable.

@ajcr `.match` looks up using an `ObjectHashTable` for non-integer/float, (changing is trivial for `datetime64/timedelta64` but prob not worth the effort.

So add a return type to `_hashtable_algo` and have it do the casting

@jreback thanks for the guidance - I've made changes to the tests and to `_hashtable_algo`. Happy to do more work if anything should be improved further.

@jreback I fixed the variable name and everything's green - as ever, please do let me know if there's anything more I should do.

@ajcr thanks!

its funny that these didn't exist, though people just use `.unique()` method on `Indexes`, but this is nice for compat
Clarify docstring: Series.sort_index() creates new instance (rather than sorting in-place)
State: closed


Labels:

Comments:
all pandas objects return new objects except for indexing operators. the fact that Series.sort is actually in place is a historical fact and will be deprecated (in favor of a new method .sorted soon)

so this is superfluous 

Thanks for the clarification (and the super-quick response!), and apologies for the noise. I just saw that this is indeed mentioned in the docs (for reference: http://pandas.pydata.org/pandas-docs/stable/basics.html#copying). It's good to know that `Series.sort()` will be deprecated soon. I think it is the fact that this method does an in-place modification (but most/all others don't) which has led to quite a bit of confusion amongst some colleagues of mine. Glad to hear that it will be consistent soon.

this has been a wart for a while and I am going to fix it soon
it comes from the fact that np.sort is in place (for efficiency but odd for semantics)
DEPR: deprecate irow,icol,iget_value,iget in Series/DataFrame, #10711
State: closed

closes #10711 


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
This looks good to me. :+1: 

Yep, looks good!

Just a few minor comments, and one other thing: for `irow` on a GroupBy, the alternative is not `iloc` (as this will raise), but `nth` or `apply(lambda x: x.iloc[]`). Maybe you can add that one to GroupBy to have another depr warning (instead of the one of `irow` on the dataframe/series)?

I think all points addressed.

yes, thanks!
BUG: Categorical doesn't show tzinfo properly
State: closed

Closes #10713.
- [x] Check timedelta
- [x] Refactor not to import `DatetimeIndex` and `PeriodIndex` if possible.
- [x] Fix `categories` line break location
- [x] Check `Categorical` `Series` formatting
- [x] Check `Categorical` `DataFrame` formatting 
- [x] Fix `CategoricalIndex` formatting 
## CategoricalIndex problem (befor this PR)

```
import pandas as pd
idx = pd.period_range('2011-01-01 09:00', freq='H', periods=5)
idx = pd.Index(pd.Categorical(idx))
idx
# CategoricalIndex([359409, 359410, 359411, 359412, 359413], categories=[2011-01-01 09:00, 2011-01-01 10:00, 2011-01-01 11:00, 2011-01-01 12:00, 2011-01-01 13:00], ordered=False, dtype='category')
```
## Format differences (after this PR)

I left ordered category representation of `CategoricalIndex` as below because it has `ordered` repr separately.

```
pd.Categorical([1, 2, 3], ordered=True)
# [1, 2, 3]
# Categories (3, int64): [1 < 2 < 3]

pd.Series(pd.Categorical([1, 2, 3], ordered=True))
#0    1
#1    2
#2    3
# dtype: category
# Categories (3, int64): [1 < 2 < 3]

pd.CategoricalIndex([1, 2, 3], ordered=True)
# CategoricalIndex([1, 2, 3], categories=[1, 2, 3], ordered=True, dtype='category')
```


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@jreback Thanks, fixed your points and it's ready for review. Checked `Categorical`, `Index`, `Series` and `DataFrame` outputs. 

Thanks for comments. Added `CategoricalArrayFormatter` as `CategoricalFormatter` is already occupied, and also added `PeriodArrayFormatter` which doesn't exist. Pls check whether it meets your suggestion.

minor comments
looks good
merge when ready
CLN: plotting cleanups for groupby plotting
State: closed

Related to #8018. Because it touches lots of codes, I've split some cleanups required for #8018 which does NOT changes current functionality.


Labels:
name: Visualization
description: plotting
name: Clean

Comments:
@sinhrks status?

@TomAugspurger Can you take a look? It's nice if #8018 can be included in v0.17.

this lgtm. so when you guys are ok, pls merge.

Let me merge this if there is no objections in this weekend.
BUG: Series.align with MultiIndex may be inverted
State: closed

Closes #10665. Closes #10741.

I'm adding some more tests for `Index.join`, and found a skeptic behavior.

For `right` join, all levels / labels of right side remains (OK).

```
import pandas as pd
midx = pd.MultiIndex.from_product([[0, 1], [0, 1, 2]], names=['a', 'b'])
idx = pd.Index([1, 2 ,3], name='b')

idx.join(midx, how='right')
# MultiIndex(levels=[[0, 1], [0, 1, 2]],
#            labels=[[0, 0, 0, 1, 1, 1], [0, 1, 2, 0, 1, 2]],
#            names=[u'a', u'b'])
idx.join(midx, how='right').values
# array([(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2)], dtype=object)
```

For `left` join, all levels are remain but not used as levels. Thus, values only on left side is dropped (NG).

```
idx.join(midx, how='left')
# MultiIndex(levels=[[0, 1], [1, 2, 3]],
#            labels=[[0, 0, 1, 1], [0, 1, 0, 1]],
#            names=[u'a', u'b'])
idx.join(midx, how='left').values
array([(0, 1), (0, 2), (1, 1), (1, 2)], dtype=object)
```

I understand the last result must be the following. If my understanding is correct, I'll include the fix to this PR.

```
idx.join(midx, how='left')
# MultiIndex(levels=[[0, 1], [1, 2, 3]],
#            labels=[[0, 0, 1, 1], [0, 1, 2, 0, 1, 2]],
#            names=[u'a', u'b'])
idx.join(midx, how='left').values
# array([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2), (1, 3)], dtype=object)
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
thanks @sinhrks
added axvlines_color parameter to parallel_coordinates
State: closed

parallel_coordinates are in pandas.tools.plotting
'black' color was hard-coded for vertical lines, I suggest to make it an option


Labels:
name: Visualization
description: plotting

Comments:
@TomAugspurger @sinhrks ?

No strong opinion here. It's an easy addition but I don't think our parallel coordinate plot is worth expanding.

> On Oct 11, 2015, at 10:56, Jeff Reback notifications@github.com wrote:
> 
> @TomAugspurger @sinhrks ?
> 
> —
> Reply to this email directly or view it on GitHub.

should we deprecate `parallel_coordinates` ? (is their a direct replacement say in seaborn/mpl/bokeh)? can we simply dispatch?

No replacement in seaborn / mpl. The best I've seen is [here](https://github.com/Geosyntec/wqio/pull/37/files#diff-dd00120ecd13d10566a50c6ffc3de9c4R535).

Though we don't set a high value on maintaining other plotting funcs than `.plot` method, this kind of small fix looks OK to be applied. Otherwise, may better to deprecate it as unmaintained.

ok can u add a test, make sure the keyworkd are added in the .plot accessor and add a whatsnew note 

Hi guys!
Sorry for the late reply - was traveling a little bit
Thank you for considering my request!

Ehm.... how about  passing keyword arguments for avx line potting insted of
just color?
Because well, I needed another color - somebody else might need another
line thickness etc

can make it like this:

def parallel_coordinates(frame, class_column, cols=None, ax=None,
color=None,
                         use_columns=False, xticks=None, colormap=None,
                         axvlines=True,
axvlines_kwds={'linewidth':1,'color':'black'}, **kwds):

the default {'linewidth':1,'color':'black'}, ensure that backward
compatibility

Cheers!
Alex

2015-10-19 1:34 GMT+09:00 Jeff Reback notifications@github.com:

> ok can u add a test, make sure the keyworkd are added in the .plot
> accessor and add a whatsnew note
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10709#issuecomment-149028586.

Oh, right, I already committed this :)
Anyway, just wanted to add the parallel_coordinates is useful in Pandas - at least I did not see any other "out of the box" alternatives 

@undertherain ok this is fine. can you squash. ping when done.

actually, pls add a whatnew note (enhancement section), use this PR number as the issue number

I added the  whatsnew note with issue number
but for the "squashing" I can use some hint

@undertherain See here for some explanation: http://pandas-docs.github.io/pandas-docs-travis/contributing.html#combining-commits

@jorisvandenbossche yep, thank you! I've just got a bit confused as I made few changes, than merged with main version than make few more changes. Anyway, I hope I sorted it out and it should be ready now.

@jreback oh, in case github doesn''t send notifications to people in the thread not being  mentioned explicitly  - ping.

merged via https://github.com/pydata/pandas/commit/4b6fcc08f7831c2a59d75534f47a078120dac284

thanks!
adding plotly to ecosystem
State: closed


Labels:
name: Docs

Comments:
Thanks!

Thanks for adding us @jorisvandenbossche; we're delighted to be included and a pandas tool :hammer:

@msund No problem! 
Just one question (which I didn't think of before merging it ..): I don't think plotly is fully open source, but you have a free (limited?) version? Is that correct?

If so, would you mind to add one sentence explaining the situation a little bit? (something in the line of "The free version includes .." or "The cloud based environment is free for public files", but you will know that better :-))

Good idea, thanks. Does https://github.com/pydata/pandas/pull/10714 do it in the way you had in mind?

yes, that looks good! thanks
ENH: Index optional pivot
State: closed

Closes #3962.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
Addressed comments.

Failures look unrelated?

lgtm. can you add a release note refing the original issue. squash, then ping when green.

Done

- can you update the doc-strings of both `pd.pivot` and `DataFrame.pivot` (saying `index` is optional/can be `None`).
- can you add a test using `pd.pivot`?

Are there any tests for `pd.pivot`? It's just a pass-through.

**Edit** Ah, you mean `pivot_simple`, which is different than `pivot` but imported into the namespace as `pivot`? Yeah, I didn't touch this code. I suppose we should just create a default index in this case?

looks `pd.pivot_table == DataFrame.pivot_table`. While `pd.pivot == DataFrame.pivot`.

So ok on `.pivot`. Maybe have a look at `.pivot_table`?

It appears to work.

```
In [8]: df
Out[8]:
     A    B      C  D
0  foo  one  small  1
1  foo  one  large  2
2  foo  one  large  2
3  foo  two  small  3
4  foo  two  small  3
5  bar  one  large  4
6  bar  one  small  5
7  bar  two  small  6
8  bar  two  large  7

In [9]: pd.pivot_table(df, values='D', index=None, columns=['C'], aggfunc=np.sum)
Out[9]:
C
large    15
small    18
Name: D, dtype: int64

In [10]: pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc=np.sum)
Out[10]:
C        large  small
A   B
bar one      4      5
    two      7      6
foo one      4      1
    two    NaN      6
```

ok gr8. test pretty pretty pls.

It's tested. https://github.com/pydata/pandas/blob/master/pandas/tools/tests/test_pivot.py#L60

ic, I guess that already accepted `None` then, ok.

ok, just fix the doc-string and push again.

Done.

ok cool. ping when green.

thank you sir!
TST: fix usage of assert_produces_warning
State: closed

@jreback I just notices some tests were not using `assert_produces_warning` as a context manager, therefore these would never fail (you could put whatever in there).

I changed it to using `with`, as I don't think there is way to use that function not as a context?

@bashtage I also changed some of your tests. I am not familiar with this code, but I think this should still be correct


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Looks correct to me.
Empty subtypes of Index return their type, rather than Index
State: closed

Resolves #10596 

Prior was in #10599 FYI

I think this has a PyTables error which I need to resolve


Labels:

Comments:
did you resubmit for some reason?

#10599 was merged (I merged it directly rather than thru github). I just closed the issued (forgot to before)

@jreback Great - didn't know it was merged. Cheers!

no thank you!
Fix Visual Studio 2015 compile errors
State: closed

Together with #10681 this allows building Pandas for Python 3.5b4 using Visual Studio 2015.


Labels:
name: Windows
description: Windows OS
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
merged via https://github.com/pydata/pandas/commit/3db0e8271eb8d352c97f06beddf81fdc4f74a16f

thanks!
Remove duplicate code
State: closed

Use pandas internal [stdint.h](https://github.com/pydata/pandas/blob/master/pandas/src/headers/stdint.h).


Labels:
name: Windows
description: Windows OS
name: Clean

Comments:
merged via https://github.com/pydata/pandas/commit/bd73cd0149b7b7adaea6117664fecdc94a3f7dcf

thanks!
DOC: improve docs on iteration
State: closed

As this comes up once in a while, I tried to improve/clarify the docs somewhat.

Further suggestions to add or make it clearer are welcome!

xref #10334 

closes #7194


Labels:
name: Docs

Comments:
@shoyer thanks, I forgot to note that, added now!

@jreback @shoyer I added some further warnings / alternatives (also about setting in a loop, so this should now also address #7194). You can have a new look.

looks great! (minor comment on the reference to enhancingperf)
DEPR: add stacklevel to FutureWarnings (GH9584)
State: closed

Closes #9584


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Deprecate
description: Functionality to remove in pandas

Comments:
should revamp this....its pretty useful :)

yes, on my todo list! 
The problem I encountered is that for certain Warnings that are raised deeper into the pandas internals, the needed stacklevel is not always the same for different end user use cases. So it will not be possible to fix all of them. But for many (all warnings of deprecated functions or keywords, which are raised rather top-level), it is also simple to fix.

I now patched the `assert_produced_warning` to check for this, but should make it option so you can turn if off when it is not possible to get it right.

@jorisvandenbossche sounds good. Well at least the top-level ones would be a step up. 

@jreback updated this. Added a check to `assert_produces_warning` that the stacklevel is set correctly, but for now only check it for FutureWarning/DeprecationWarning (all other warnings could be checked as well, but it was a bit too much work to get these all right for now).

lgtm.

what we actually need is a warning that then traverses the stack and finds the user level code as opposed to using an explict stack level, but I suspect thats a bit tricky.

> what we actually need is a warning that then traverses the stack and finds the user level code as opposed to using an explict stack level, but I suspect thats a bit tricky.

Hmm, that goes beyond my python skills, so for somebody else :-) (doubt if this is even possible)

Will merge tomorrow if no further comments

hahha 

cc @njsmith since you reported this. This should solve almost all FutureWarnings we have in pandas.

Doing that stack traversal thing is totally doable. Really it should be added as a feature upstream, like `warn(..., not_package="pandas.*")` to warn at the first level where the code isn't inside pandas (and there's some somewhat-relevant discussion in http://bugs.python.org/file40211). But in the mean time, woot, awesome.

Inferring the stacklevel is indeed not that difficult :-)

Something like this did it:

```
import warnings
import inspect

frame = inspect.currentframe()
stacklevel = 1

while '\\pandas\\' in frame.f_code.co_filename:
    frame = frame.f_back
    stacklevel += 1

warnings.warn('test!', stacklevel=stacklevel)
```

(the `'\\pandas\\'` is because I was testing on Windows)
But don't how robust something like this would be.

FYI I think u have to do a 

del frame at the end to break the gc cycles 

@jorisvandenbossche let's merge this. I think I already changed a few as I thought you had merged this.

@jorisvandenbossche pls rebase and merge
BUG: #10645 in using MultiIndex.__contains__
State: closed

This PR fix a BUG #10645 
@sinhrks would you review my PR?


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
needs tests

Thanks for mentioning about missing test. Just added. :)

I revisit this issue and Yes, you're right @jreback I did further investigation and it seems a odd behavior of Cython. so this PR doesn't solve the issue.

I'd like to finish this issue. Do you have any idea with this? Anything would be appreciated.

so the indexes have different behavior if > 1M elements on lookups. so that is a different path that is taken.

This isn't Cython issue.

https://github.com/pydata/pandas/blob/master/pandas/index.pyx#L141-L148
If > 1M elements on lookups, `IndexEngine.get_loc()` do `_bin_search()` and try to `get_value_at()` the `loc`.
If the `loc` is equal to `len(values)` (like this error case), it looks ok to raise `IndexError` in `util.get_value_at()` because it means the `loc` is out of bounds.

So we should handle this `IndexError` somewhere.
IMHO, `get_loc()` itself has a 'key' parameter and if it fail key lookup, it's okay to raise `KeyError` here.

How's your thought? @jreback 

This PR also fixes #10692 
@jreback Any comments would be appreciated.

@ scari Gr8! Could you add test cases and release note for #10692? Then, pls squash to a single commit.

this is not correct 
you are fixing a symptom not the problem
I don't think the indexer is setup correctly

Okay. I'll take another look at this.

@scari I'll submit a PR if you don't have time to come back to this. 

replaced by #11049 

@kawochen Thanks!
API: #10636, changing default of to_datetime to raise, deprecating coerce in favor of errors
State: closed

closes #10636 

changes `to_timedelta` API as well to similar effect.


Labels:
name: Datetime
description: Datetime data dtype
name: API Design
name: Timedelta
description: Timedelta data type

Comments:
@jorisvandenbossche updated if you want to have a look again.

@jorisvandenbossche @shoyer @sinhrks @TomAugspurger any other comments?

Looks good.

Looks good!

One thing I was thinking: it would be nice if in the error message you get when there is a parsing error, that it also says something like "to coerce errors as NaN use "errors='coerce'" " if possible

@jorisvandenbossche your last point is nice, but I think a separate PR is in order. If you would like to open an issue gr8.

@jorisvandenbossche is there a way to limit tracebacks automatically in Sphinx (ones that are on purpose), other than using a code block

eg http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#changes-to-to-datetime-and-to-timedelta

no, I don't think so (in any case, the `okexcept` does not do that), but that is already a long time something I wanted to fix. Will try to do that for 0.17

And for the error message suggestion, certainly, that was no blocker for merging this!
read_sql/to_sql can accept database URI as con parameter
State: closed

It should fix https://github.com/pydata/pandas/issues/10654


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
I should have remove

```
engine = sqlalchemy.create_engine(db_uri)
```

in `test_sql.py` `test_to_sql_read_sql_with_database_uri`

Looking good!
- We will have to remove the file 'iris.db' again afterwards. You can have a look at `tm.ensure_clean()` to use this.
- the `import sqlalchemy` will have to be wrapped in a try except block, as it is not guaranteed that this is installed. You have a `_SQLALCHEMY_INSTALLED` constant, but the problem is that this is only initialized the first time `_is_sqlalchemy_connectable` is used, and the `engine_builder` is called before that in case of `read_sql_table`

@scls19fr To change things, you can just add more commits in the same branch and push again, then this PR will be automatically updated.

I think that's ok now.

And we'd be grateful if you could squash these down to one commit. Let us know if you need any help: https://github.com/pydata/pandas/wiki/Using-Git#fetch-and-then-rebase-interactively-to-squash-reword-fix-otherwise-change-some-commits

I'm facing some issues to squash. Any help is welcome.

Normally, the following should do it:

```
git fetch upstream
git rebase -i upstream/master
```

Then you will see a pop-up window (or a vim session), where you can change occurences of 'pick' to 'squash' or 'fixup' 
And force push the branch again: `git push -f origin issue_10654`

But no worries, if you can't get it working, I can do it for you before merging

That's nice I'm with Tom on Gitter

@jorisvandenbossche I think we're good on the rebase.

@scls19fr Thanks a lot for your contribution!
Small doc update for Stata 118 support
State: closed


Labels:
name: Docs
name: IO Stata
description: read_stata, to_stata

Comments:
right, can you also add a "::note:" to io.rst about what versions are supported.

Do we know that versions < 113 work?  There are no tests for any of these formats so it would be more accurate to start support at 113.

ty!
BUG: #10565 Series.name lost in rolling_* funcions
State: closed

Addresses #10565


Labels:
name: Bug

Comments:
Looks good to me... anyone else?

Sure. I think @sinhrks has been closing a bunch of these losing names issues. Is there a master list somewhere?

@chris-b1 thanks!

@TomAugspurger There is no master tracker for `Series.name`, but only for `Index.name` (#9862). I'll change this to list both `Series.name` and `Index.name`.