BrandonFors/pandas-github-issues huggingface dataset:
Pickle subclass metadata
State: closed

Fixes #10553


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
might be able to remove setstate from Series as well (it will use the inherited then)

> might be able to remove setstate from Series as well (it will use the inherited then)

Apparently so. Done.

awesome!

can u add a note in whatsnew (u can put under API changes)

and squash

ping when green

@jreback green and squashed like mashed peas.

thank you sir!

SciPy was really amazing btw

really cool that so many from industry, academia, and technology get together!
CLN: cleanup up platform / python version checks. fix GB10151
State: closed

Notes:
- I did not use `is_platform_xxx` on `pandas/util/clipboard.py` because this module is a fork. So I guess better not deviate from the original, right?
- I added the helper `_skip_if_windows` function in `pandas/util/testing.py`, I added an underscore before `skip` to maintain consistency with other `skip` functions (although none of them should have this underscore...)

> I think we could use an update on the wiki on the same 

I didnt find any references on the wiki or docs.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks! big help!

thanks, pandas looks pretty cool. I like how you guys organize issue tags and have good issue descriptions. I planning to contribute more :) 

I guess you can close the original issue #10151 

@schettino72 that would be gr8 thanks!
BUG: get_dummies not returning SparseDataFrame
State: closed

Fixes #10531 .


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
@jreback Perhaps this would be more elegantly fixed by having `concat` drop empty passed-in DataFrames?

xref #10536 

I think #10536 should be fixed first. But this introduces the problem of when/how to convert a concat of series (that happen to be all SparseSeries) into a SparseDataFrame (as opposed to a DataFrame). 

@jreback Not sure why #10536 needs to be fixed first --- it does not affect this issue (we are concating SparseDataFrames here)

merged via https://github.com/pydata/pandas/commit/d06537478486887042ef89141707f6b05695995e

thanks!
BUG : read_csv() twice decodes stream on URL file #10424
State: closed

As described in #10424 , reading non-utf-8 csv files from URL leads to decoding problems, i.e. a decoding may first be made in io.common.get_filepath_or_buffer() when file is URL.
Modification done makes this function read stream from URL without decoding (this is done later, at the same place as for local files). 
It's also used by io.common.read_stata(), io.common.read_json() and io.common.read_msgpack(). Similar problems  reading stata and msgpack URL files may also be solved using this modification.

Thanks for reviewing.


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: Unicode
description: Unicode strings

Comments:
needs some tests
u can download s small file and use a file:////url
for json/msgpack create a file encode and access like the above 

@BotoKopo can you add some tests and update

@BotoKopo closing. pls reopen if you would like to update with some tests
BUG: GH9618 in read_msgpack where DataFrame has duplicate column names
State: closed

To close #9618
Note I modified `encode`, so it's not backward compatible.


Labels:
name: Bug

Comments:
@kawochen can you rebase? I just applied the fix for #10525 in a14f513d5b7104bf5a16bfecf85da495ac39053c

@kawochen this needs a msgpack from 0.16.2 as a regression test
in fact I think that u can use the https://github.com/pydata/pandas/blob/master/pandas/io/tests/generate_legacy_pickles.py
to create it 

@jreback Do you mean creating a similar script to generate msgpack files (and using them in tests)? I changed `decode` in a way such that old msgpack wouldn't work, but we could test whether a msgpack is in the old format or new format and decode correspondingly. Is that desirable? 

you can just that script, modify it to generate either a msgpack or a pickle (give a cmd line arg), and rename it as well.

Then needs some regression tests to make sure that we CAN read back-compat msgpack. If you are actually breaking a prior API then this needs to be noted, the msgpack version number bumped, and then back-compat reading is ok.

@jreback I just made the changes. 

we are referring to #10525 in comments by @kawochen and @shoyer but is that the correct issue? 

Updated.

@kawochen lgtm, just a minor documentation note. ping when you push.

updated

thanks!
Fix a typo 'does' -> 'do'
State: closed

This came up during the SciPy 2015 tutorial.


Labels:

Comments:
thanks!

OK, I'm an idiot and merged this before letting CI run... turns out there was a test for this error message!

patch coming up...

Fixed in a14f513d5b7104bf5a16bfecf85da495ac39053c
Fix and tests for issue #10154 inconsistent behavior with invalid dates
State: closed

closes #10154 

This addresses the original issue and another discovered with the test cases. 
Please review the tests to be sure you agree with the assertions.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
merged via https://github.com/pydata/pandas/commit/5a76b44f89922a3c3c8656fa039c1434e633da57

thanks!
DOC: Clarified PyTables "natural" names
State: closed

The HDF5 documentation "Notes & Caveats" defines a PyTables NaturalNameWarning vaguely, and incorrectly states that a natural name may not begin with an underscore. I clarified the definition of a natural identifier. The relevant regex is defined [here](https://github.com/PyTables/PyTables/blob/2fde39957c7b5263dd4d0b11aa797900250f16bc/tables/path.py#L39) in the PyTables source.


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
thanks!
TST: Deprecate assert_numpy_array_equivalent
State: closed

Closes #10427.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
add this to the deprecation issue for removal in 0.18

Sure. Also added `testing.assert_isinstance` #10458 to the list.

Is there a reason we don't use `np.testing.assert_array_equal` (but our own `array_equivalent` implementation)?

`array_equivalent` is much more general as it handles all nulls for any dtypes (while numpy is restricted to floats). I may do nat's but I don't think so.

@sinhrks Can you then add a note in the docstring of `assert_numpy_array_equal` that it is similar to `np.testing.assert_array_equal` but more flexible regarding NaNs (so it is clear why this function exists/is used)

OK, added docstring. Also I found some test case uses `numpy.testing.assert_array_equal` imported to `pandas.util.testing` namespace. Fixed them and deprecate `assert_array_equal` also.

Ah,  `np.testing.assert_array_equal` behaves slightly different from current `assert_numpy_array_equal`. I'm going to fix test cases to pass `assert_numpy_array_equal` which is more strict. Lmk if I should leave `np.testing.assert_array_equal`.

```
np.testing.assert_array_equal(np.array([1, 1]), 1)
# no error 

pd.util.testing.assert_numpy_array_equal(np.array([1, 1]), 1)
# AssertionError: [1 1] is not equivalent to 1.
```

Found some cases are affected by  #10546. Revisit later.

Applied a fix done in #10546. I think this is ready for review.

@sinhrks looks good to me! (just needs a rebase)

Do we have a general rule for ourselves whether to use `tm.assert_...` or `self.assert_...`? They are available on both, but I saw you now changed some from self to tm.

the usual issue are we promoting using `self.assert_numpy_array_equal` or `tm.assert_numpy_array_equal`. prob don't care so as long as both work.

pls rebase right before you merge as this touches lots of code.

@jorisvandenbossche There is no policy, sometimes `self`, `tm.` or directly import each funcs. 

Merging this.

Super!

very nice

now all we need is some kind of checker script that we can run in Travis to flag things 
kind of like a linter

eg might be as simple as greping for
np.testing and such
BUG: Groupby(sort=False) with datetime-like Categorical raises ValueError
State: closed

Closes #10505.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
Fixing `Categorical.unique` to return unique `Categorical`. Would like to confirm the expected behavior based on current impl.
- unordered category: values and categories are sorted by appearance order, excluding unnecessary ones.
- ordered category: values are sorted by appearance order, categories keeps existing order excluding unnecessary ones.

```
import pandas as pd
import numpy as np

# unordered
cat = pd.Categorical(["b","b","a","a"], categories=["a","b","c"])
cat.unique()
# [b, a]
# Categories (2, object): [b, a]

cat = pd.Categorical(["b", "b", np.nan, "a"], categories=["a","b","c"]) 
cat.unique()
# [b, a]
# Categories (2, object): [b, a]

# ordered
cat = pd.Categorical(["b","b","a","a"], categories=["a","b","c"], ordered=True)
cat.unique()
# [b, a]
# Categories (2, object): [a < b]

cat = pd.Categorical(["b", "b", np.nan, "a"], categories=["a","b","c"], ordered=True) 
cat.unique()
# [b, a]
# Categories (2, object): [a < b]
```

Looks `groupby` needs some logic for `Categorical(ordered=True)` and `sort=False`.

the uniques should not have `nan` in them. Further I think the `ordered=True` should order according to the categories (e.g. the present ones), this might be a small API change.

Ah, correct. Fixed and updated the example above.

I understand `groupby` result must follows the same rule as above. It means `groupby(Categorical(ordered=True), sorted=False)` will have the index sorted by the order of categories, was previously ordered by appearance of values (unsorted).

> uniques should not have nan

I have misunderstand this with `value_counts`. Currently, `unique` returns array including `NaN` both normal and categorical.

```
pd.Series([1., 2., 1., np.nan]).unique()
# array([  1.,   2.,  nan])
pd.Series([1., 2., 1., np.nan], dtype='category').unique()
# array([  1.,   2.,  nan])
```

So, the fixed categorical result should preserve `NaN` in codes and not in categories as below? 

```
>>> pd.Series([1., 2., 1., np.nan], dtype='category').unique()
# [1, 2, NaN]
# Categories (2, float64): [1, 2]
```

lgtm. merge when ready.

Thanks, let me merge this once rebase and gets green.
TST: Fix test for datetime categorical
State: closed

Related to #10465, but different part.

Current `test_groupby_datetime_categorical` in `test_groupby.py` is incorrect, the actual result returns `CategoricalIndex` as level 0, otherwise expected result uses `DatetimeIndex` as level 0. Changed to use the same dtype and added explicit comparison.
### Actual Result (current test case)

```
levels = pd.date_range('2014-01-01', periods=4)
codes = np.random.randint(0, 4, size=100)
cats = pd.Categorical.from_codes(codes, levels, name='myfactor', ordered=True)
data = pd.DataFrame(np.random.randn(100, 4))
grouped = data.groupby(cats)
desc_result = grouped.describe()
desc_result.index.get_level_values(0)
# CategoricalIndex([2014-01-01T09:00:00.000000000+0900,
#                  ...
#                   2014-01-04T09:00:00.000000000+0900],
#                  categories=[2014-01-01 00:00:00, 2014-01-02 00:00:00, 2014-01-03 00:00:00, 2014-01-04 00:00:00],
# ordered=True, name=u'myfactor', dtype='category')
```
### Expected Result (current test case)

It must be `CategoricalIndex`.

```
idx = cats.codes.argsort()
ord_labels = np.asarray(cats).take(idx)
ord_data = data.take(idx)
expected = ord_data.groupby(ord_labels, sort=False).describe()
expected.index.names = ['myfactor', None]
expected.index.get_level_values(0)
# DatetimeIndex(['2014-01-01', '2014-01-01', '2014-01-01', '2014-01-01',
#               ...
#               '2014-01-04', '2014-01-04', '2014-01-04', '2014-01-04'],
#               dtype='datetime64[ns]', name=u'myfactor', freq=None, tz=None)
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type

Comments:
lgtm, merge when ready

Thanks, merging.
TST: DataFrame.quantile should have Float64Index
State: closed

The result of `DataFrame.quantile` should have `Float64Index` as below, but current test case doesn't check it.

```
import pandas as pd
pd.DataFrame([1, 2, 3, 4]).quantile([0.5, 0.75]).index
# Float64Index([0.5, 0.75], dtype='float64')
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
:+1:

Thanks for checking. Merging.
Add test and fix for categorical series .shift #10495.
State: closed

#10495

Waiting for Travis and will review further.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
Pls add release note to `doc/source/whatsnew/v0.17.0.txt`

#10497 fixes this i believe. But pls check.

#10497 is a much better way to do this
BUG: CategoricalBlock shift GH9416
State: closed

Should resolve #9416.

CategoricalBlocks always seem to have ndim=1, even if multiple
categoricals are in a frame with the same categories. This simplifies
the axis shift logic somewhat.

Note that dataframe shift with axis=1 still doesn't work with multiple
categorical columns, since they are each a different block.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
@jreback updated as requested

pls add a whatsnew note as well (you can call this a bug fix)

@jreback updates green on travis

thanks!
ENH: GH10485 'Frequency' label for Series.plot
State: closed

More informative label for histogram.

Conversation in #10485 


Labels:
name: Visualization
description: plotting

Comments:
Thanks! Please do:
- Fix the test cases under `pandas/tests/test_graphics.py`
- Add release note to `doc/source/whatsnew/v0.17.0.txt`

@sinhrks Tests pass locally now. Is there a way to trigger another build on Travis?

If you've updated this branch, please once rebase then push with force flag (`-f`). Travis test runs automatically.

https://github.com/pydata/pandas/wiki/Git-Workflows

Ok, rebased. Build log shows this failure, which I believe is unrelated:

```
FAIL: test_metadata_immutable (pandas.tests.test_index.TestMultiIndex)
```

Indeed, build failure was unrelated -- I just retriggered your tests

@sinhrks build looks good now.
Series.from_csv not loading header names
State: closed

Series.from_csv at the moment does not load the series.name and series.index.name when the header keyword argument is stated.

This was introduced in 6078fba9410918baa486ca008cc9e3ba066c03ec.  The issue was that `Series.from_csv` uses `DataFrame.from_csv`, which automatically indexes columns (in practice, the index column gets labelled 0 and the values column gets labelled 1).  Converting this to a series will then cause the `series.name` to be 1, and the `series.index.name` to be 0.  The fix was to explicitly set the names in Series.from_csv to `None`.

This caused the headers to be deleted even if they were provided by setting the header kwarg.

The fix is simple, to check if the headers are provided, and only setting the names to None if they are not.

I included some tests, please let me know if this is enough as I am new to open-source and pandas.

Thanks!


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
looks good

pls add a whatsnew note in bug-fixes (for 0.17.0)
squash and ping when green

Hopefully this worked, still new to git!

I think this is ready to be merged if everything looks ok?

merged via https://github.com/pydata/pandas/commit/3382e12b0d5aa990540a4b4fa8a84f70186d0631

thanks!
Fix docstring for na_values in parsers
State: closed

I discovered entirely by accident that the `na_values` in `pd.read_*` can be a string. According to the docstring, it can only be a sequence or a dict. This PR fixes that.


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
thanks!

@jaidevd There are some other places in the docs were this is described (eg in io.rst). Would you like to adapt this as well?

@jorisvandenbossche Sure. Give me a couple of hours.

@jorisvandenbossche https://github.com/pydata/pandas/pull/10488
BUG: Enable complex values to be written to HDF
State: closed

Enable table format to be used to store complex values
in DataFrames, Panels and Panel4Ds.
Add tests for both fixed and panel.
Add exception when attempting to write Series with complex values.

closes #10447


Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Complex
description: Complex Numbers

Comments:
I added an `append` and `select`, and now provide a more meaningful warning when attempting to index a complex column.

looks really good!

IIRC we support fixed stores with complex? pls move those tests next to these (in a separate class for all of the complex tests as well would be gr8)

did u mean to close? this looked fine

Oops, no. 

thank you sir!
ENH: Make group_var_ use Welford's algorithm.
State: closed

closes #10448 

This PR reimplements the Cython functions `group_var_float64` and `group_var_float32` to use Welford's algorithm, rather than the sum-of-squares method, which is numerically unstable. This came up in #10448; see also #10242 for more discussion.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@jreback Would you be willing to give this a review? I can probably tackle `_nanvar` at some point too, but not before the weekend.

looks good. pls add:
- release note in 0.17.0 whatsnew (you can put in bug fix section)
- add a test that should match before this PR (to assure there isn't a calculation regression), e.g. something that isn't an edge case 
- squash

@jreback Thanks for your feedback. I added some tests and squashed everything onto one commit. I also took the liberty of fixing up the "closes" trigger in the PR description: this addresses #10448 --- I have some ideas for #10242 but would like to address that in a separate PR.

couple of style issues, otherwise looks good. ping when pushed / green

@jreback Done and Travis shows green. Thanks for the review!

thanks! this looks gr8!
BUG: Fix csv_read bugs when using empty input. GH10467 & GH10413
State: closed

closes #10413 
closes #10467 


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
pls add a whatsnew note (you can put both issues in a single entry)

@jreback - build is green.

Let me know if you'd like me to squash the commits or if you want a whatsnew note for the last commit.

Hi @jreback - just to clarify what's required. Are you happy to merge this if I add some extra tests to explicitly verify different `index_col` types (i.e. `str`, `int`, `list` of `str`, `list` of `int`)?

@santegoeds yes, just want to try to cover the bases on tests.

@jreback, build passing again.  Is this in line with your expectations?

very nice
comprehensive! thanks
BUG: Series.map using categorical Series raises AttributeError
State: closed

Closes #10324. Closes #10460.

Based on #9848, using `.get_values` should be avoided?


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Categorical
description: Categorical Data Type

Comments:
get_values is fine 
in fact this is the point of it 

Thanks. Another point I forgot to refer is below case results in `object` dtype. Should it be `category`?

```
a = Series([1, 2, 3, 4])
b = Series(["even", "odd", "even", "odd"], dtype="category")
a.map(b)
# 0     odd
# 1    even
# 2     odd
# 3     NaN
# dtype: object
```

Applied a fix to `common.take_nd`. I will add more tests later.

BTW, I've noticed `Categorical` has its own `take_nd`. I think we can use this, but it raises `IndexError` for non-existing key.

```
import pandas as pd
c = pd.Categorical(['A', 'B', 'A', 'A'])
c.take_nd([2, 1])
# [A, B]
# Categories (2, object): [A, B]

c.take_nd([2, 1, 4])
# IndexError: Out of bounds on buffer access (axis 0)
```

So another point of the original issue is the behavior when the mapped category doesn't contain all the mappings. Should raise or padding with np.nan (create a new category implicitly)?

```
# this should work because all values are mapped to existing categories
pd.Series([0, 1, 2, 3]).map(pd.Series(["even", "odd", "even", "odd"], dtype="category"))
# 0    even
# 1     odd
# 2    even
# 3     odd
# dtype: category 

pd.Series([1, 2, 3, 4]).map(pd.Series(["even", "odd", "even", "odd"], dtype="category"))
# raise or padding with np.nan?
```

These should be the same as a post-convert to category
The nans are NOT part of the categorical, rather missing in the Series themselves.

```
In [5]: a = pd.Series([0, 1, 2, 3]).map(pd.Series(["even", "odd", "even", "odd"]))                   

In [6]: b = pd.Series([1, 2, 3, 4]).map(pd.Series(["even", "odd", "even", "odd"]))

In [7]: a.astype('category')
Out[7]: 
0    even
1     odd
2    even
3     odd
dtype: category
Categories (2, object): [even, odd]

In [8]: b.astype('category')
Out[8]: 
0     odd
1    even
2     odd
3     NaN
dtype: category
Categories (2, object): [even, odd]
```

I see. Then, current impl works as expected. I'll consider if it can be better.

```
b = pd.Series([1, 2, 3, 4]).map(pd.Series(["even", "odd", "even", "odd"], dtype='category'))
b
# 0     odd
# 1    even
# 2     odd
# 3     NaN
# dtype: category
# Categories (2, object): [even, odd]
```

looks good. go ahead and merge when green.

@jreback Thanks for review. Merging.