BrandonFors/pandas-github-issues huggingface dataset:
ENH: support StringMethods index and rindex
State: closed

as a part of https://github.com/pydata/pandas/issues/9111

@sinhrks @jreback 


Labels:
name: Strings
description: String extension data type and string data

Comments:
are there any other routines that have an `array` as a return type?

@jreback good catch, return type of `array` is used in the old docstrings, I'll update it

I also added to the release note 

@jreback do you think this can make 0.16.1? seems like it'd be nice to get all the string methods in

@jreback i updated the release note

the only other PR remaining is for `str.translate()` here: https://github.com/pydata/pandas/pull/10052/ 

was also passing before, right?

yes this was passing too, just changing docs
BUG: Resample BM/BQ adds extra index point #9756
State: closed

closes #9756 


Labels:
name: Bug
name: Resample
description: resample method

Comments:
thanks, minor doc point. ping when you have pushed.

@jreback Thank you for reviewing!

merged via https://github.com/pydata/pandas/commit/c0d4339b18ab29d3d692901e19393a3ea10d7cf3

thanks!
BUG: numericlike set ops on unsupported Indexes
State: closed

closes #10038 
closes #10039
- `CategoricalIndex` now raise `TypeError` on ops like : `idx - idx` (we are moving all non-numeric Indexes to this, but just deprecated for now on existing index types)
- Index ops with lists will now show the same warning (e.g. `idx - ['a','b']`)


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
There is also an inconsistency between the behaviour of `__add__` and `__sub__` (I see now, this was already before this PR).

```
def __add__(self, other):
    if is_list_like(other):
        warn ...
        return self.union(other)
    return Index(np.array(self) + other)
```

while

```
def __sub__(self, other):
    warn ...
    return self.union(other)
```

Shouldn't this last one also not do the list_like check (so you don't do a set operation when subtracting a scalar)?

I can do a PR if this seems OK

I guess technically the warning is showing up when using `__sub__`, but then it immediately raises anyhow as you cannot use a non-iterable (while for `__add__` you can).

This is concatenation, and IS convenient

```
In [1]: i = Index(['foo','bar','baz'])

In [2]: i + ' 2'
Out[2]: Index([u'foo 2', u'bar 2', u'baz 2'], dtype='object')

In [3]: i - ' 2'
pandas/core/index.py:1193: FutureWarning: using '-' to provide set differences with Indexes is deprecated, use .difference()
  "use .difference()",FutureWarning)
TypeError: Input must be iterable!
```

If you want to change `__sub__` to not show the warning (even though it raises), ok by me

yes, that one as well. 
But the inline comment I made was about this:

```
In [1]: idx = pd.Index(['a', 'b', 'c', 'd'])

In [2]: idx + ['a', 'e']
pandas\core\index.py:1184: FutureWarning: using '+' to provide set union with In
dexes is deprecated, use '|' or .union()
  "use '|' or .union()",FutureWarning)
---------------------------------------------------------------------------
...
ValueError: operands could not be broadcast together with shapes (4,) (2,)

In [3]: idx + pd.Index(['a', 'e'])
Out[3]: Index([u'a', u'b', u'c', u'd', u'e'], dtype='object')

```

So for the list, it displays both a warning and an error, and those two contradict each other (and I think it is also more logical to have the same behaviour as with an Index)

@jorisvandenbossche ahh, ok,  I see. I think the example I used broadcast correctly, so yes this needs cleaning up. be my guest!
DOC: use _shared_docs for fillna methods
State: closed

currently the docstring for `DataFrame.fillna` is shown for `Series.fillna` and `Panel.fillna`. http://pandas.pydata.org/pandas-docs/version/0.16.0/generated/pandas.Panel.fillna.html?highlight=fillna#pandas.Panel.fillna 

this is not accurate in particular for the `axis` values. This PR makes it such that they share the same template with different `axis` descriptions. 


Labels:
name: Docs

Comments:
@sinhrks updated according to your comments 

@mortada thanks you sir!
DOC: added fredapi module to ecosystem list
State: closed

kind of a shameless plug here, but would love to have my pandas-based project on this page :)
please let me know what you think. Thanks. 


Labels:
name: Docs

Comments:
Certainly OK!

Yep, looks good to me.

As we move towards making pandas-datareader an independent project, we should try to give alternative  data reader projects (like fredapi) equal billing in the pandas docs.
pd.lib.infer_dtype infers bytes in Python3
State: closed


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
ping!

?

@jreback ok with this?

oh whoops, need to add tests for strops

and add to this line: https://github.com/pydata/pandas/blob/master/pandas/core/base.py#L522

(test if you can)

squash as well. thxs

ok, in progress

hm, so to really be solid about this we'd have to test every single str operation against a bytes series

e.g., `cat` was failing because it has a `sep` argument that is `''` if  `None` is passed in. That string will be a unicode string in python 3 which can't be used with a `bytes` object

Do we have to support `bytes` operations? I really just wanted the ability to tell if a `Series` contained `bytes`. I wasn't looking to add any new features.

no I don't think bytes are necessarily a big deal to support - in fact prob should raise on str type ops?

ok, i'll test that we raise on `str` access

ok, looks good. just add a release note and merge. ty.
ENH: Added str.normalize to use unicodedata.normalize
State: closed

Derived from #9111.  Can this be considered in v0.16.1? Otherwise will change the milestone.

> [`unicodedata.normalize`](https://docs.python.org/2/library/unicodedata.html?highlight=unicodedata#unicodedata.normalize) is quite useful to standardize multi-bytes characters. I think it is nice if `StringMethods.normalize` can perform this.

```
import pandas as pd
s = pd.Series([u'ＡＢＣＤＥ', u'１２３４５'])
s
#0    ＡＢＣＤＥ
#1    １２３４５
# dtype: object

s.str.normalize()
#0    ABCDE
#1    12345
# dtype: object
```

Another point I'd like to discuss here is the condition `Index.str` can be used. Currently, `inferred_type` must be `string`. I think the preferable condition is:
- `Index` must be normal Index, not `MultiIndex`.
- Its `inferred_type` should be either `string`,  `unicode` or `mixed`.

This PR adds `unicode` currently, not `mixed`.

```
pd.Index([u'a', u'B']).inferred_type
# unicode
pd.Index(['a', u'B']).inferred_type
# mixed

# when we allow "mixed" to show str, we should exclude MultiIndex case.
pd.MultiIndex.from_tuples([('a', 'a'), ('a', 'b')]).inferred_type
# mixed
```

CC: @mortada


Labels:
name: Strings
description: String extension data type and string data

Comments:
I never used this, but I can see this could be a useful string method (and I think it is OK for 0.16.1)

And good catch for the unicode index! (for series this did work as it allows all object types) 
Maybe add an explicit test for this? (it is now tested in your normalize tests, but for clarity?)

> Maybe add an explicit test for this? 

Yes, added tests for `Index` to confirm it should work as the same as `Series`.

@sinhrks thanks!
DOC: mention other packages in IO docs (for now, just xray for netCDF)
State: closed

As discussed in #5487.

This would also be a nice place to mention other packages that connect the
pandas DataFrames with other file formats. Right now, the only one I can think
of off-hand is `root_pandas` (#9378, CC @ibab), but I'm sure there are more.


Labels:
name: Docs
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
@cpcloud should add `odo` here

Looks good to me! 
Maybe adding a small snippet of a typical code example how you read in a file and convert to dataframe? (just a code-block, not executing it)

Could add `impyla`

thanks @shoyer 

Thanks for the suggestions, guys. I'm going to do a followup adding a small example for netCDF/xray, but I'll leave odo and impyla examples to others who are more familiar with those tools.
Fix TimedeltaIndex constructor fastpath when name is set
State: closed

Fixes #10025


Labels:
name: Timedelta
description: Timedelta data type

Comments:
For the record, I don't think it is actually a good idea to return an existing object from an index constructor. This leads to weird things like:

```
new_idx = pd.Index(old_idx)
new_index.name = 'foo'
# also sets old_idx.name!
```

We should preserve the old data and hashtable, if possible, but not everything on an Index object is immutable (name/names is the main exception).

you are setting the name directly, use `set_names` to get a new object

```
In [24]: i = Index(range(5),name='foo')

In [25]: i2 = Index(i,name='bar')

In [26]: i3 = i2.set_names('baz')

In [27]: i.name
Out[27]: 'foo'

In [28]: i2.name
Out[28]: 'bar'

In [29]: i3.name
Out[29]: 'baz'
```

@jreback do you really think it is a good idea for the `Index` constructor to return the same object that was passed in? I agree that `set_names` is cleaner here...

@shoyer its prob IS a bug that it is doing that. I _think_ that it should be a shallow copy always. See what the other indexes do. Certainly no tests for this type of aliasing I'll bet.

thanks!
Speed up max_len_string_array
State: closed

Before:

``` python
In [5]: x = np.array(['abcd', 'abcde', 'abcdef', 'abcdefg'] * int(1e7), object)

In [2]: f = pd.lib.max_len_string_array

In [6]: %timeit f(x)
1 loops, best of 3: 501 ms per loop
```

After:

``` python
In [1]: x = np.array(['abcd', 'abcde', 'abcdef', 'abcdefg'] * int(1e7), object)

In [2]: f = pd.lib.max_len_string_array

In [3]: %timeit f(x)
10 loops, best of 3: 68.4 ms per loop
```


Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance

Comments:
looks good. release note in perf section & squash. merge when ready.

my naive quick look does show _some_ improvements to `to_hdf` when you have `object` dtypes, order is 10-15% impv. (only used in `to_stata/to_hdf`).

@shoyer I didn't have a particular pandas use case for this. I'm going to start using it in some often called paths in odo and I wanted to see if I could squeeze out some more perf. 

I think @jreback had some ideas about using cython memoryviews in some of re csv code similar to how I use them here. IIRC he said there are quite a few places where we don't take full advantage of what cython has to offer. For example if you type a variable as just `ndarray` you incur the overhead of the fully general get item c API whereas if you type it as `ndarray[type]` the getitem syntax goes directly to the underlying raw pointer array. 

ok squashed. merging on pass
read_csv newline fix
State: closed

Slight change to the logic in the `tokenize_delimited()` and `tokenize_delim_customterm()` functions of the C parser.

Fixes #10022.

I believe the new logic is correct, but perhaps someone with more familiarity can double-check.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
you need a test. As not really sure what you are fixing.

Ok, test forthcoming.

There is a small self-contained test in the comments on issue #10022. Would it be desirable to make it into a unit test? It takes about a second to run on my machine.

yep
the idea would be to add the test ; it fails on master; after your fix everything passes

Unit test added. Any further comments?

even though the fix is only in the c-parser, IIRC, this should work in python parser as well? hence pls move the tests to `test_parser.py` which will tests with all parsers.

@jblackburne looks good. 
- pls add a release note in v0.16.1
- pls squash 
- ping when green.
  [here](http://pandas-docs.github.io/pandas-docs-travis/contributing.html) for detailed git instructions if you need

Squash all into a single commit?

@jblackburne Yes, that's what @jreback is asking for

pls take a look

cc @evanpw
cc @mdmueller
cc @selasley

The logic looks right to me.

@jblackburne pls ping when this is green.

Ok, green.
ENH: support __radd__ operation on Index
State: closed

closed #10083

Currently the following would raise a `TypeError`:

```
In [1]: import pandas as pd
In [2]: idx = pd.Index(list('abc'))
In [3]: idx + '1'
Out[3]: Index(['a1', 'b1', 'c1'], dtype='object')
In [4]: '1' + idx
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-5e2fbeff7266> in <module>()
----> 1 '1' + idx

TypeError: Can't convert 'Index' object to str implicitly
```

While for `Series` this works fine:

```
In [5]: s = pd.Series(list('abc'))

In [6]: '1' + s
Out[6]:
0    1a
1    1b
2    1c
dtype: object

In [7]: s + '1'
Out[7]:
0    a1
1    b1
2    c1
dtype: object
```

This PR implements the `__radd__` method so that adding from the left would work for `Index`. 


Labels:
name: API Design

Comments:
Also, sorry about the noise because of my PEP8 fixes. There's a lot of PEP8 violations around white spaces in this file; you may want to just view this diff with `?w=1`.

Thanks. 

I'm removing the pep8 cleanup for now

This looks good to me -- @jreback can we get this in 0.16.1?

merged via https://github.com/pydata/pandas/commit/58e72b661d746c240f7e397f853bba159cbc8814

thanks!
Only add the ga: prefix if it is required
State: closed

This checks if ga keys are already prefixed with ga: and only adds the
prefix if required. The previous behaviour was particularly anoying for
complex filters where keys within the filter would need a ga: prefix but
the first one could not have the prefix.

eg. Both the following would fail:
`ga:pagepath=/foo;ga:city=London`
`pagepath=/foo;city=London`

Now the following works
`ga:pagepath=/foo;ga:city=London`


Labels:

Comments:
needs a test, however I don't think the `ga` infrastructure is currently tested on ci at all though. 

closing as stale, but if you would like to reopen and fixup pls do.
kwarg for Categorical.fillna should be value instead of fill_value
State: closed

It seems inconsistent that all the other `fillna()` methods (`DataFrame.fillna`, `Series.fillna`, etc) have `value` as the first kwarg but `Categorical` has `fill_value` instead


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Looks good to me

@mortada thanks!
DOC: improve fillna() doc for limit keyword (fixes #10002)
State: closed

fixes #10002


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Actually, in the way I interpret "maximum size gap to forward or backward fill", this is not fully correct.

I interpret it as "only fill that gap if the gap is not larger as 'limit'", but it is "per gap, fill a maximum numbers of 'limit'":

```
In [10]: s=pd.Series(range(4), index=[1,2,5,9])

In [11]: s = s.reindex(range(10))

In [12]: s
Out[12]:
0   NaN
1     0
2     1
3   NaN
4   NaN
5     2
6   NaN
7   NaN
8   NaN
9     3
dtype: float64

In [13]: s.fillna(method='ffill', limit=2)
Out[13]:
0   NaN
1     0
2     1
3     1
4     1
5     2
6     2
7     2
8   NaN
9     3
dtype: float64
```

What if we rephrase it to say something like:

"maximum number of consecutive holes to forward or backward fill"

lgtm 

It is a bit subjective, but I would personally still interpret that not fully correct (I see a hole as possibly multiple NaN values if they are consecutive, but maybe this is not correct language?). 
Otherwise: "maximum number of consecutive NaN values to forward/backward file per hole" 

I think an example in the docstring to show this behaviour would possibly be the most clarifying.

Ah yeah I suppose you could interpret a "hole" as multiple NaNs. @jorisvandenbossche please see the updated changes in the diff, I wonder if it's now clear to you. 

Thanks!
pivot table with margins (GH3335)
State: closed

closes #3335 

Changed code to raise an exception when margin=True and All was being used in the table


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@TheInan you need a tests to validate what you are doing. First construct a tests which shows the unwanted/incorrect behavior (not even sure what the original issue points 2, so that is the first goal).

closing if/when updated, pls reopen
Descriptive Statististics Example Added_Rahman_Shadman_NE24_Spring2015
State: closed

I added an example of how to compare two sets of data through measures of center and spread and created methods which print out statements and create tables to illustrate this.


Labels:

Comments:
I'm not quite sure what your idea is here, but these aren't appropriate additions to the pandas codebase.

If you're just using pandas, you should write code in your own modules instead of modifying the pandas codebase itself.

If you'd like to improve the pandas documentation with new examples, that's great, but you should make additions to the [actual documentation](https://github.com/pydata/pandas/tree/master/doc) instead of the source code for the pandas library itself.
BUGFIX: length_of_indexer() can return incorrect values that break slice assignments
State: closed

... assignments.

BUG DESCRIPTION:
length_of_indexer() (defined in pandas/core/indexing.py) returns incorrect result
if (stop-start) is not divisible by step. As a consequence some slice assignments
throw exceptions. Affected panda versions: 0.16.x, 0.15.x, current git master.

HOW TO REPRODUCE:

import pandas as pd
sr= pd.Series([-1]*6) # series with 6 elements indexed from 0 to 5.
sr[0::2]= [0,2,4] # setting even elements works fine!
sr[1::2]= [1,3,5] # setting odd elements results in error:

.../pandas/core/internals.pyc in setitem(self, indexer, value)
    568             if is_list_like(value) and l:
    569                 if len(value) != length_of_indexer(indexer, values):
--> 570                     raise ValueError("cannot set using a slice indexer with a "
    571                                      "different length than the value")
    572
ValueError: cannot set using a slice indexer with a different length than the value


Labels:

Comments:
Looks great to me, thanks! Could you please add a brief bug fix note to what's new for 0.16.1?

@shoyer : Stephan, I just added a whatsnew bugfix note you requested. Please, pull. --Leo
BUG: transform and filter misbehave when grouping on categorical data
State: closed

Fixes GH #9921
closes #9700 


Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
That small change fixed most of the performance difference:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_transform_series2                    | 124.1800 | 124.5937 |   0.9967 |
groupby_transform_series                     |  19.8309 |  19.8836 |   0.9974 |
groupby_transform                            | 151.5836 | 151.8770 |   0.9981 |
groupby_transform_ufunc                      | 112.3510 | 110.2444 |   1.0191 |
groupby_transform_multi_key4                 | 143.6467 | 137.7186 |   1.0430 |
groupby_transform_multi_key3                 | 802.0917 | 753.2163 |   1.0649 |
groupby_transform_multi_key2                 |  49.4637 |  46.3487 |   1.0672 |
groupby_transform_multi_key1                 |  73.4100 |  68.7597 |   1.0676 |
```

lgtm. pls squash, and ping when green.

Conditional is cleaned up, squashed, and tests are green.

@evanpw thanks!
ENH: improve extract and get_dummies methods for Index.str (fix for #9980)
State: closed

Closes #9980


Labels:
name: API Design
name: Strings
description: String extension data type and string data

Comments:
Thanks! Added "Closes ... " description to the body to automatically close the original issue.

I'll update #9843 once behavior is decided.

pls add a release note. lgtm otherwise

@jreback updated with release note and issue number as inline comment as you suggested 

@mortada can you update/rebase. I believe you can use `_wrap_result_expand` to simplify the code a bit.

@jreback updated, I can squash the commits later but for now I want to showcase the difference between the two implementations. To simplify the code inside `str_extract()`, we'd need to pass the `name` argument down into `_wrap_result()` 

merged via https://github.com/pydata/pandas/commit/e68638778b627f898ad3dd63dabdff9442e18e76

thanks!
DOC: Add composition example to internals.rst
State: closed

Follow-up of #9802. Added an example of composition to internal.rst. 

Also, fixed broken section name for subclass document.


Labels:
name: Docs

Comments:
I made a few grammar/language suggestions.

But actually, I'm not sure this example makes the best case for why composition is usually a better idea than subclassing. Using `__getattr__` makes this `ComposedSeries` example a sort of limited version of subclassing.

So many we should show an example of how subclassing can lead to unexpected behavior or a more realistic version of composition that simply makes use of a series or frame to hold internal state.

status?

@sinhrks going to move this to next version.

@sinhrks can you update

closing, but if you'd like to update, pls reopen
BUG: .iloc and .loc behavior not consistent on empty dataframe
State: closed

Fixes #9964 .

Notice that `assert_frame_equal` now fails for empty dataframes with different dtypes (as, I think, it should).  However, this means some tests need to be patched now.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
What other changes is this making? (there are a lot of edge cases to empty frames!)

For instance the tests in test_groupby looked correct before...

The problem is that due to the fact that `.iloc` looses datatypes on an empty dataframe, dtypes are not checked on empty dataframes.  E.g., right now on master:

```
In [1]: import pandas as pd

In [2]: df1=pd.DataFrame(columns=["col1","col2"])

In [3]: df1["col1"] = df1["col1"].astype('int64')

In [4]: df2=pd.DataFrame(columns=["col1","col2"])

In [5]: print df1.dtypes
col1     int64
col2    object
dtype: object

In [6]: print df2.dtypes
col1    object
col2    object
dtype: object

In [7]: pd.util.testing.assert_frame_equal(df1, df2, check_dtype=True)

In [8]: 
```

Where on PR this assert fails. So a couple of tests that compare empty dataframes with different dtypes need to be patched.

Please test the corrected behaviour of assert_frame_equal in test_testing.

IIUC This means previously fleshed out behaviour for empty edge case dtypes has been broken for some time?

OK, added test to `test_testing`.

Yes, I think assert_frame_equal was broken for empty dataframes.  It would still compare number/names of columns, but not their dtypes.

@jreback Made changes, except for moving test out of test_testing --- are you sure about that?
Also, came across another weird bug --- see comments above.

@jreback OK, fixed `test_frame.py`

@artemyk ok looks good. ping on green (if you want to rebase on master, prob a release note conflict, ok, otherwise will do it on merging).

@jreback Ready to go.

@artemyk thanks!

proof that sometimes trivial looking things (in indexing), actually can have lots of cases / need lots of testing