BrandonFors/pandas-github-issues huggingface dataset:
ENH: Add duplicated/drop_duplicates to Index
State: closed

Closes #4060.

```
idx = pd.Index([1, 2, 3, 4, 1, 2])
idx.duplicated()
# Index([False, False, False, False, True, True], dtype='bool')
idx.drop_duplicates()
# Int64Index([1, 2, 3, 4], dtype='int64')

idx.duplicated(take_last=True)
# Index([True, True, False, False, False, False], dtype='bool')
idx.drop_duplicates(take_last=True)
# Int64Index([3, 4, 1, 2], dtype='int64')
```


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
@jreback, @jorisvandenbossche Considering both comments and fixed.

Defining `update_inplace` in `IndexOpsMixin` results in `Series` to refer it because of inheritance order. Thus, functions which uses `update_inplace` must be overridden eventually.
https://github.com/pydata/pandas/blob/master/pandas/core/series.py#L77

So defined common logic in `IndexOpsMixin` and override in `Index` and `Series` each. As a result, `Index` no longer need `update_inplace` because it will not accept `inplace` keyword.

Just as a usage question, what do we envisage as the 'recommended' way to drop duplicate indices from a DataFrame (where you now had to say the somewhat unintuitive `df.groupby(level=0).first()`)?:

```
df[~df.index.duplicated()]
```

or

```
df.reindex(df.index.drop_duplicates())
```

although these are even longer than the groupby ..

unchanged, the first is best (this is for the Index to be compatible)

@jorisvandenbossche 's point is #2825. Though I feel  `df[~df.index.duplicated()]` is simple enough.

the doc string for `Series.duplicated/drop_duplicated` does not have `inplace`?

I would change this around. Why don't you just have `_duplicated/_drop_duplicates` in Base (w/o the inplace argument, and are private (no doc strings)).

Then in Index/Series put in the doc-strings (and inplace for Series)?

No, `Series` has. I expect docstrings are rendered as expected.
![pandas index drop_duplicates pandas 0 14 1-201-g54d3e4d documentation 2014-08-15 21-40-19](https://cloud.githubusercontent.com/assets/1696302/3933177/edc4108c-2479-11e4-94b2-e57273d55fb2.png)
![pandas series drop_duplicates pandas 0 14 1-201-g54d3e4d documentation 2014-08-15 21-39-58](https://cloud.githubusercontent.com/assets/1696302/3933178/f0e6d524-2479-11e4-84a2-95b52c1fff9f.png)

Agreed to remove docstring from `IndexOpsMixin`, and I feel no need to make them private (little confusing).

ok, that's fine then. ping hwne ready

Thanks to confirm. Now green.
BUG: Allow __name__less callables as groupby hows (GH7929)
State: closed

Allow `functools.partial` objects (and similar callables without `__name__` attributes) to be used as `groupby` functions, which also adds support along the tree (e.g. `resample`, which is specifically tested.)

Closes #7929.

This does not address the potential enhancements allowing automatic naming of duplicate-named functions (relatively minor) and the use of a Series to specify names (more useful).


Labels:
name: Bug
name: Groupby
name: API Design

Comments:
@dsm054 minor comment, otherwise looks good

Explicit `return None` and justification for it added; all green.

thanks!
CI/WIP: Use conda for most deps
State: closed

closes #6870 
- [x] use conda.binstar.org/cpcloud channel for versions/packages not in conda defaults
- [x] possibly bring back the ccache, not totally sure what this is doing

this doesn't remove maintenance, but it sure does simplify things, since conda recipes are very easy to create


Labels:
name: CI
description: Continuous Integration

Comments:
the cache is to avoid regenerating the cython files each time 

should be unaffected by using conda

ah ok cool

FYI should email continuum to have then setup a pandas binstar 

I am sure they would

i tried to but i couldn't find a way to upload packages to an organization

hmm should be a way - ok 

can do later anyhow

ok looks trivial let me try

https://groups.google.com/a/continuum.io/forum/#!topic/conda/uYtVRGW--iU

nice, it works like a charm

only major issue is that there's no python 3.2 in continuum's repos and the one i built doesn't seem to work

i'll scour the interwebz

@jreback if you sign up for binstar i can add u to the group

hmm I thought I did will check later

okay sounds good

closing until i make more progress

FYI the statsmodels guys use miniconda

look at their repo

yeah i checked it out there's some path stuff re travis that i don't fully undersatnd yet

alright i got at least one build passing!

https://travis-ci.org/cpcloud/pandas/builds/32118580

@jreback what is your feeling about these older versions of pytables + numexpr + numpy?

conda doesn't let me install them together, it complains about unsatisfiable packages probably because of some of the compat issues that we came up against earlier

i haven't investigated yet but i was wondering if you think it would be too drastic to simply use whatever the latest versions of the deps are on conda (keeping older ones where we can).

hmm we could drop support for pytables < 3.0.0 (and consequently Numexpr < 2.1)

in fact need to switch to the new API for pytables in 3.0.0
(which is really just making the function calls pep 8 nothing actually changes)

I think it would be ok to do this

want to create  an issue for that?

what older versions do they have these (that seem 2 work)?
still want to test as much as possible of course

could do pytables=3.0.0 + numexpr=2.2.2 + numpy=1.7.0

ok

gr8

so u specify just pytables==3.0.0 and let it figure out the others? (or u specify all ?)

also if u can specificy pip/setup tools versions try to have an oldest possible that works too 

so i can specify or if i don't then it will use the latest versions of each

@jreback would you accept a pr that has a `./fake_ci.sh` script that essentially does the conda stuff and runs tests?

what so u mean?

do

i have script that essentially does all the conda install stuff but you can run it locally eg to test requirements without having to wait on travis

creates an environemnt based on the sha1 hash of the req file so no need to recreate if you want to run it again

a very poor man's tox

that's useful too

we should drop 3.2 if we do this yes? (alternatively could leave the experimental builds in place?
and use conda for the main builds, but leave the numpy_master/3.2 build )

?
ENH Warn about panel.to_frame() discarding NaN GH7879
State: closed

Also there was unimported module warnings with calls to it. In pandas/core/panel.py lines 718, 748 for example.

closes #7879 


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
@m-novikov couple of minor comments here, otherwise looks good

Fixed according to your comments, also added suppressing warning statement to pandas/io/tests/test_pytables.py as it's shows up on test run.

if u take those extra lines out ( the filtering in setup and the setattr) what happens? I am puzzled why they are in here in the first place

that is the point of assert_produces_warning it does the filtering necessary to assert the warning

more to the point is the rationale behind this pr
you are warning on a valid option

that seems a little odd no ?

i think this warning is ok, it's a bit of a surprise to throw away data by default.

why not just change the default then?

well that would be nice but would break existing code

If I lose 
`warnings.filterwarnings(action='ignore', category=UserWarning)`
then during test run there will be this warning raised.
If I lose
`setattr` then I will be dependent on order in which tests are run, warning raised only once by default, and if it was raised in some other test module I will not catch it.

I say just a change the default as an API change

or do nothing 

i'd be ok with that, maybe a nice warning in the docs that says "you have to call dropna now"

Removed warnings suppression

@m-novikov if u change the result does anything break in the test suite (aside from the actual test for this)

@jreback if I leave it with warnings nothing breaks, just occasional message in some test will be displayed and so without it but as was stated it's not expected behaviour.
`setattr(panelm, '__warningregistry__', {})` breaks nothing, but warnings of this module may be displayed twice during test suite run. Possibly I should store state of registry and restore it after test.

no I mean if u change the default to False
instead if warning

@jreback  There is [SparsePanel.to_frame()](https://github.com/pydata/pandas/blob/master/pandas/sparse/panel.py#L284) which doesn't support filter_observations=False, if change signature only for Panel class then it will be inconsistent.
Also it breaks a few methods like `Panel.count` for example.

@m-novikov can you put in a short doc note. Also Let's change this to a `FutureWarning`. Then change the default in 0.17.0

@jreback updated

I would like to repurpose this and simply change the default here (no need to warn then). Can you update?

Yes, so this means all related test to be fixed too, I will get to it

@m-novikov gr8! lmk

I think we can simply change this for 0.17.0.

going to close this, but pls reopen if you want address that.
ENH/BUG: Period and PeriodIndex ops supports timedelta-like
State: closed

Must be revisited after #7954 to remove `_skip_if_not_numpy17_friendly`

Closes #7740.

Allow `Period` and `PeriodIndex` `add` and `sub` to support `timedelta`-like. If period freq is `offsets.Tick`, offsets can be added if the result can have same freq. Otherwise, `ValueError` will be raised.

```
idx = pd.period_range('2014-07-01 09:00', periods=5, freq='H')
idx + pd.offsets.Hour(2)
# <class 'pandas.tseries.period.PeriodIndex'>
# [2014-07-01 11:00, ..., 2014-07-01 15:00]
# Length: 5, Freq: H

idx + datetime.timedelta(minutes=120)
# <class 'pandas.tseries.period.PeriodIndex'>
# [2014-07-01 11:00, ..., 2014-07-01 15:00]
# Length: 5, Freq: H

idx + np.timedelta64(7200, 's')
# <class 'pandas.tseries.period.PeriodIndex'>
# [2014-07-01 11:00, ..., 2014-07-01 15:00]
# Length: 5, Freq: H

idx + pd.offsets.Minute(5)
# ValueError: Input has different freq from PeriodIndex(freq=H)
```

 If period freq isn't `Tick`, only the same offset can be added. Otherwise, `ValueError` will be raised.

```
idx = pd.period_range('2014-07', periods=5, freq='M')
idx + pd.offsets.MonthEnd(3)
# <class 'pandas.tseries.period.PeriodIndex'>
# [2014-10, ..., 2015-02]
# Length: 5, Freq: M
idx + pd.offsets.MonthBegin(3)
# ValueError: Input has different freq from PeriodIndex(freq=M)
```


Labels:
name: Bug
name: Timedelta
description: Timedelta data type
name: Period
description: Period data type

Comments:
thanks @sinhrks
CI: Drop numpy 1.6 support
State: closed

closes #7711 


Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
well that was cathartic

fyi, your 2.6/2.7_locale build are prob taking forever as they have to compie numpy (until the wheel is there)

yeah i know .. there's some fails too

i think we might need to require 1.7.1, there's an issue with type coercion in ufuncs in 1.7.0 not totally sure what the issue is taking a look now

arg nope 1.7.1 still fails

ok wheels are up for 1.7.0/1.7.1 on both 2.6 and 2.7_LOCALE

enjoy!

I'm going to close this until i make some more progress

whoops wrong pr

This will pass but there's a weird bug where 

``` python
with tm.assertRaises(ValueError):
    o.item()
```

will fail but

``` python
self.assertRaises(ValueError, o.item)
```

will pass

there might be a weird py26 bug in this decorator ... let me see if i can't dig around and fix it

@jreback @jorisvandenbossche Any comments? The assert raises issue only shows up on Travis, even when I use the exact same deps. 

yeh I think their might be something in the decorator

I always use:

`self.assertRaises(ValueError, lambda : o.item())`

need to update `install.rst` and big fat warning in v0.15.0 and highlites (also may need to update else where in docs)

Cool. Will do. 

You actually don't need a lambda in those cases because the function is already a no argument callable. A bit easier to read IMO 

@jreback good to go?

i'll squash if you watn

version is also mentioned in the readme

minor comments (I would squash). ok otherwise

@jorisvandenbossche 

alright if this next build passes going to merge

@jorisvandenbossche thanks for the heads up about the readme

ok sure

this is passing on my fork, travis is going to take forever to get to this i guess can merge tomorrow

if passing then ok to merge
DOC/TST: index followup
State: closed

TST: argmin/max impl, tests, and doc-strings
DOC: release notes corrections
API: raise on `__nonzero__` for Indexes, closes  #7897

partial close of #7904 


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
@jorisvandenbossche this fixes the missing properties from api docs and such, however

some properties, that are `cached_read_only` are not having the doc-strings picked up

e.g. `Index.is_monotonic,is_unique`....

do you know what the problem is here?

Nope, I don't really know what the exact cause is, it's a long standing issue, see #6100 (it happens in two cases: when an attribute is initialized as class attribute to None (eg Index.name), or when cache_readonly is used for properties (eg Index.is_unique))

@jorisvandenbossche hmm, ok then
BUG: Fix for to_excel +/- infinity
State: closed

BUG: Previously, a negative sign was being prepended for positive infinity, not for negative infinity. (GH6812)

xref #6812 


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
- need a tests for this (write a frame with both pos/neg inf, read back and check that they match)
  should fail w/o the fix and pass after
- need a release note in v0.15.0.txt (bug fix section), reference this issue number

status of this? (did you see my comments above)?

There was an existing test (`test_inf_roundtrip()` in test_excel.py) that wrote/read a frame with positive and negative infinity, but it was not catching the error due to an oversight in the underlying comparison. In testing.pyx, `assert_almost_equal()` was only looking for infinite values, regardless of +/-. Adding the check for +/- there allows `test_inf_roundtrip()` to work as expected. 

@maxchang ok, makes sense.

pls rebase and squash and then I think good to go

@jreback Ok, should be ready now.

thanks!
Add warning about HDFStore concurrent reads
State: closed

Not too much to say; this is a documentation PR.


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
@yikelu can you update according to above comments?

@jreback I've been incredibly busy, and I haven't figured out the
documentation.

On Sun, Jan 18, 2015 at 3:31 PM, jreback notifications@github.com wrote:

> @yikelu https://github.com/yikelu can you update according to above
> comments?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7944#issuecomment-70427037.

@yikelu can you update, pls.

closing as stale, but if you would like to reopen and fixup pls do.
BUG/API: Consistency in .where() when setting with None for both inplace in a Series (GH7939)
State: closed

closes #7939


Labels:
name: Bug
name: API Design

Comments:
@jreback not fully clear, but which of the two behaviours is it after this PR? The inplace (convert to NaN) or the non-inplace (keep None) ?

I closed this because not really sure what (if anything) to do at this point. You can see the reference discussion on the issue. `None` is kind of a mess. But changing anything is extremely tricky as their really aren't enough tests to validate current or proposed behavior. and it is often ambiguous on what to do.

`np.nan` is the missing value indicator, so `None` should NOT be treated specially, but it IS in some places for convience. It is a can of worms. 

ah, yes, I thought this was merged, my mistake. 
I know it is a mess, I just added some more of it :-) (https://github.com/pydata/pandas/issues/7939#issuecomment-51353012)
DOC: mention that stack/unstack implicicly sort
State: closed

It took me a while to discover what was causing this TypeError... I guess this can be useful to other people.

(Yes, I know there is a note somewhere warning users that some operations depend on the index being sorted, but maybe this is a sligthly different case, since the operations do work after all, the problem is they do something more than what is documented)


Labels:

Comments:
Oops, done... by the way: maybe this info could go also/instead in the docs for the two methods, I'm not sure.

Yes, I think you can add a note in the docstring.

thanks!
BUG: rolling/expanding_* treatment of center
State: closed

Closes https://github.com/pydata/pandas/issues/7925.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@seth-p is this a bug / api change? pls add a release note

I'm not sure whether it's a bug, API change, or both. I want to do a little bit more testing -- want to try to figure out vbench, perhaps for Python 3.4 -- and then will update release notes when I think it's ready.

I've added a release note under "API changes".

The odds of me getting vbench up and running in a finite amount of time are pretty low, so I'd suggest going ahead and merging this, if it looks ok to you. My guess is that most people use the rolling/expanding function with `center=False` (the default), so will not be affected by this.

hahah....will look soon

Actually, I just added a comment to https://github.com/pydata/pandas/issues/7925#issuecomment-51277322, suggesting that we remove the `center` argument from all the `expanding_*` functions. If you agree, probably best for me to try to include that in this PR.

I've updated the code to remove `center` from all the `expanding_*` functions (and updated the notes under "API changes"). Should I create a separate issue for this change, as distinct from https://github.com/pydata/pandas/issues/7925#issuecomment-51277322?

If it looks ok to you, I think this is ready to be merged.
DOC: update docs to show construction of periodindex when needing out-of bounds spans
State: closed


Labels:
name: Docs
name: Period
description: Period data type

Comments:
cc @CarstVaartjes

this what you had in mind?

great improvement in terms of clarity! 

While in general I do understand the period_range, what made it click for me was the applymap with Period (as it made me understand how to create a Period column that you can use from your existing data).
I'll try to make nice docstrings for the to_periods that also explains that concisely
BUG: define empty product on Series and DataFrame to be 1
State: closed

closes #7889 


Labels:
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
looks good. nice refactor on bottleneck decorator...simpler

maybe just need a release note on API section

sure thing

IIRC this was a can of worms. pls reopen if you'd like to address.

@jreback : I think re-incorporating it would be more complicated now because the `zero_value` attribute isn't even respected (or returned) with the `bottleneck_switch` decorator.  Not sure what happened there.
BUG/DOC: Categorical fixes (GH7918)
State: closed

Categoricals now raise NotImplementedError when writing to HDFStore with a Fixed type store
Slicing bug with a single-dtyped category and a possible view

closes #7918


Labels:
name: Bug
name: Docs
name: IO HDF5
description: read_hdf, HDFStore
name: Categorical
description: Categorical Data Type

Comments:
https://github.com/pydata/pandas/issues/7621 will fix this in a future version (or if someone gets to it sooner). Its not very hard to actually write the categorical at all, the issue is retaining the meta-data (e.g. levels) needs to be done for recreation (and query).

cc @JanSchultz

seems good!
BUG: Fix Grouper with multi-level index and frequency (GH7885)
State: closed

close #7885 


Labels:
name: Bug
name: Groupby

Comments:
need to have a test (you can put in test in the issue almost directly)

test for level name and by level number

write unit tests for both using the level number and name

then u will know of the code works and/or needs modification to handle level numbers/names. (I think it does)

Thanks for clarifying. Will do.

you specify levels like `level=1` (e.g. an integer)

`ilevel_0` is ONLY for `.query` and doesn't apply here

I get what you are saying. It's just that I see, as it states in the docs, that it is possible to construct a MultiIndex that has level names set to None. In this case, unless I am misunderstanding things, an integer index name will not work.

of course, an integer when the names are `None` is a simply the level. Can you clarify what you mean?

My mistake. You're right.

ok gr8, was puzzled because your tests passed (which should not have)

Okay. How does that look?

I have applied your suggestions. 

@jmorris0x0 thanks, merged in.

next time it IS easier!
ENH: New `level` argument for DataFrame.tz_localize and DataFrame.tz_convert (GH7846)
State: closed

Closes #7846

New `level` argument for `DataFrame.tz_localize()` and `DataFrame.tz_convert()`, needed for a DataFrame with MultiIndex:

```
tz_convert(self, tz, axis=0, level=None, copy=True)
tz_localize(self, tz, axis=0, level=None, copy=True, infer_dst=False)
```

@jreback Not sure if `test_generic.py` is the right place to add the tests.


Labels:
name: Enhancement
name: API Design
name: Timezones
description: Timezone data dtype
name: MultiIndex

Comments:
@jreback I have made some changes to the test so that it does not test PeriodIndex if not implemented. Let me know if you like it or not.

ok, looks good, pls squash down to a single commit.

If you want (could be here or another PR). If you want to make a factory function to generate tz_localize/tz_convert in core/generic (see _make_stat_function for how you would do this). To avoid some duplicate code. Its not a big deal (becuase its only 2 functions). lmk if you want to do this (or if its not worth it0

Squashed into a single commit.

I don't think it is worth doing it at this point, as the function signature of tz_localize and tz_convert are slightly different (tz_localize has infer_dst but the other one does not). Let me think about it more and will create another PR if I can think of something that makes more sense.

ok, ping when green

Tests passed!

thanks!

nice fix!
update to holiday to help with GH#7070
State: closed

I am working on a financial calendar and wanted to add the following functions to pandas.tseries.holiday in order to properly handle half days.


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype

Comments:
looks reasonable (test failures though)

The right syntax would be `h = Holiday("Easter Monday", month=1, day=1, offset=[Easter(), Day(1)])`.  However, I'm seeing a problem in python3 with the map objects.  I think if we just wrap list() around the map in Holiday as it iterates through the offsets everything will work.

Thank you very much. Should I keep a test for defined holidays?

I don't see any harm in that.

Thanks for the help on this.

@rockg, I am still crashing on a python3 build.  How did you intend to handle the two observations?

I will setup a python3 environment to test both going forward.

Yeah, I had to do the same.  So many of my changes in the past needed
python3 specific fixes.

@MichaelWS looks fine, can you add a release note (just to document)

easy enough, should I put it in v0.15.0.txt?

yep

all green, I added to the init so its clear

@jreback , thanks for the help.

oh i see, ok then.

thanks for the PR!