BrandonFors/pandas-github-issues huggingface dataset:
ENH: read_csv dialect parameter can take a string (GH8703)
State: closed

closes #8703


Labels:

Comments:
I closed this PR and opened a new one by mistake, sorry for the noise. The new PR #8719 should reflect the test changes.

@thrasibule For the future, it is not needed to open a new PR. Just push the new commit to the same branch on your github fork, and the PR will be automatically updated.
BUG: fix reverse comparison operations for Categorical
State: closed

This PR fixes #8658 and also adds `pd.lib.unbox_if_zerodim` that extracts values from zerodim arrays and adds detection of `pd.Period`, `datetime.date` and `datetime.time` in pd.lib.isscalar.

I tried making `unbox_if_zerodim` usage more implicit but there's just too much to test when this is introduced (think every method that accepts a scalar must have another test to check it also accepts zerodim array).  Definitely too much to add for a single PR, but overall zerodim support could be added later on class-by-class basis.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
hmm
what about nan/NaT ? 

pls also create an issue to change use of np.isscalar to use lib.isscalar

> what about nan/NaT? 

nan is good as long as it's some sort of float.
NaT is good as long as it's an instance of `datetime.date`:

``` python
In [4]: type(pd.NaT).mro()
Out[4]: 
[pandas.tslib.NaTType,
 pandas.tslib._NaT,
 pandas.tslib._Timestamp,
 datetime.datetime,
 datetime.date,
 object]
```

But OK, I'll add the tests 

I'm also not sure about the name of `unbox_if_zerodim`, it may create an impression that it's related to timestamp boxing/unboxing, but I couldn't come up with a better yet concise name.

how about `_maybe_to_scalar` ?

> how about _maybe_to_scalar ?

It kind of loses the zerodim part and, besides, array may contain a non-scalar, e.g. `np.array({}).item() == {}`. Maybe `item_if_zerodim`?

Or, given there's a `values_from_object`, `item_from_zerodim`?

`item_from_zerodim` sounds good (maybe doc better that I did `values_from_object`  :)

merged via https://github.com/pydata/pandas/commit/02de8530419e253eb51ca8ec17e195deb455cd75

I changed it very slightly to make the period checking more consistent with internals. (and IIRC need this somewhere else anyhow).

thanks!

these subtle changes are excellent! all towards better code!
BUG: setitem fails on mixed-type Panel4D
State: closed

Trying to set a value on a mixed-type Panel4D currently fails:

```
In [1]: from pandas.util import testing as tmd

In [2]: p4d = tmd.makePanel4D()

In [3]: p4d['foo'] = 'bar'

In [4]: p4d.iloc[0,0,0,0] = 0
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-bb74b95f7244> in <module>()
----> 1 p4d.iloc[0,0,0,0] = 0

/home/stahlous/pd_dev/pandas/pandas/core/indexing.py in __setitem__(self, key, value)
    119                 indexer = self._convert_to_indexer(key, is_setter=True)
    120
--> 121         self._setitem_with_indexer(indexer, value)
    122
    123     def _has_valid_type(self, k, axis):

/home/stahlous/pd_dev/pandas/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value)
    356             # indexer here
    357             if (len(labels) == 1 and
--> 358                     isinstance(self.obj[labels[0]].index, MultiIndex)):
    359                 item = labels[0]
    360                 obj = self.obj[item]

/home/stahlous/pd_dev/pandas/pandas/core/generic.py in __getattr__(self, name)
   1927         """
   1928         if name in self._internal_names_set:
-> 1929             return object.__getattribute__(self, name)
   1930         elif name in self._metadata:
   1931             return object.__getattribute__(self, name)

AttributeError: 'Panel' object has no attribute 'index'
```

Fortunately, it looks like a simple fix. If this looks OK. I'll add a note to `v0.15.1.txt` (doesn't have a GH issue number yet).

This fix is needed for my latest attempt at the fillna bug #8395.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
tests!

Ah... yeah. There is a test that highlights this in my fillna bug branch, but I guess that doesn't do any good here. I'll add a test as well to this branch.

I added in a test which captures this issue. I also added in some tests for the non-mixed-type case since it looks like there weren't any tests of setting by index at all for `Panel4D`.

It looks like this only fixes setting by index for scalar values. Setting by index to a `Series` or `DataFrame` is still broken for mixed-type `Panel4D`. That fix is unfortunately not as trivial. I can work on the more comprehensive fix if you want to hold off merging in this partial fix. If you decide to merge in this PR, I can open an issue for the comprehensive fix.

keep things in separate PRs 
open issues for things you believe are bugs

it's faster/easier to review s single fix at a time

Thanks. I added the issue number to `test_setitem_by_indexer_mixed_type` but not `test_setitem_by_indexer` since non-mixed-types were working as expected but just not tested.

merged via https://github.com/pydata/pandas/commit/1041dff59a3626ecfd311e7ab8ba8502e6214fd6

thanks!
BUG: various Categorical fixes (GH8623)
State: closed

BUG: bug in selecting from a Categorical with iloc (GH8623)
BUG: bug in groupby-transform with a Categorical (GH8623)
BUG: bug in duplicated/drop_duplicates with a Categorical (GH8623)

closes #8623 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
Thanks.
API/BUG: return np.nan rather than -1 for invalid datetime accessors values (GH8689)
State: closed

BUG: millisecond Timestamp/DatetimeIndex accessor fixed

closes #8689 


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@jorisvandenbossche note that this is a slight API change, but fixes a long-standing oddity (the .milliseconds).

@jorisvandenbossche I removed the millisecond stuff...still save for later 

yep, good to split it. The `-1/nan` thing can certainly be regarded as a bug
CLN: fix grammar in extract_index error message
State: closed


Labels:

Comments:
The unit test result showed me I need to change the unit test also if it is grammar error. So, I will request it again.

Yes, that is looking good! BTW, you don't have to open a new PR to update it, just push again and the PR will be updated automatically.

Yes, sorry about it. Thanks for merging it. :)

for future reference, this is merged in #8691
BUG: fix Categorical comparison to work with datetime
State: closed

`pd.Timestamp` is one of types that don't pass `np.isscalar` test previously
used in _cat_compare_op, `pd.lib.isscalar` should be used instead.

``` python
In [1]: cat = pd.Categorical(pd.date_range('2014', periods=5))

In [2]: cat > cat[0]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-77883e4783fe> in <module>()
----> 1 cat > cat[0]

/home/dshpektorov/sources/pandas/pandas/core/categorical.py in f(self, other)
     52             msg = "Cannot compare a Categorical for op {op} with type {typ}. If you want to \n" \
     53                   "compare values, use 'np.asarray(cat) <op> other'."
---> 54             raise TypeError(msg.format(op=op,typ=type(other)))
     55 
     56     f.__name__ = op

TypeError: Cannot compare a Categorical for op __gt__ with type <class 'pandas.tslib.Timestamp'>. If you want to 
compare values, use 'np.asarray(cat) <op> other'.

```


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
can u add a simple doc note ref this pr

Ah, indeed, I've forgotten about that.

thanks!
BUG: fix writing of Categorical with to_sql (GH8624)
State: closed

Closes #8624 

Use .get_values() instead of .values on the block,
this ensures that NonConsolidatable blocks (non-dense
blocks like categorical or sparse) are densified
instead of just returning the raw .values.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
needs big doc note that these are converted to object on serialization (to reading back does NOT generate a categorical)

yes, OK, in the io.rst docs for sql?

needs big doc note that these are converted to object on serialization (to reading back does NOT generate a categorical)

@jreback added a doc note.
BUG: years-only in date_range uses current date (GH6961)
State: closed

closes #6961 this was probably fixed by #7907 but just adding an explicit test and release note.
(in a sense, it should really be the 0.15.0 release note, but that release has shipped...)


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
@rosnfeld thanks!
BUG: fix concat to work with more iterables (GH8645)
State: closed

closes #8645

Enhances `pd.concat` to work with any iterable (except specifically undesired ones like pandas objects and strings). A new test is included covering tuples, lists, generator expressions, deques, and custom sequences, and all existing tests still pass. Finally, a "what's new" entry is included (not sure this last is correct - but [pull requests guidelines](https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md) mentioned it)


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
@waveform80 minor comment
otherwise excellent pr!

lmk after you add that test and its green

Updated the test to include `Iterable` (I've pulled it in from `collections` rather than `collections.abc` for compatibility with 2.x and 3.2); I also moved the import into the test itself (seems more sensible given it's the only place `deque` and `Iterable` are used). Tests still pass on my machine here - hopefully Travis'll confirm the same in a bit!

Done and done (common code in "what's new" pulled out into new section above, and bug reference moved to top of the new test)

perfect - ping on green

wish all PRs were this easy to review!

pick some more issues!

ping :)

see easy peasy :)

thanks for the effort!
ENH: Series.str.split can return a DataFrame instead of Series of lists
State: closed

closes  #8428.

Adds a flag which when True returns a DataFrame with columns being the index of the lists generated by the string splitting operation.  When False, it returns a 1D numpy array, as before. Defaults to false to not break compatibility. 

In the case with no splits, returns a single column DataFrame rather than squashing to a Series.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design
name: Strings
description: String extension data type and string data

Comments:
looks good. pls add a release note in v0.15.1.txt

I am not sure I love the option `to_df`, @jorisvandenbossche @TomAugspurger ?

maybe `as_frame`? (side issue, its possible we want to deprecate this in 0.16.0 and change it to have `as_frame=True` by default) - or whatever it is called

for the name, I would certainly use `frame` instead of `df` (in line with the `to_frame` method), but `as_frame` or `to_frame` are both good for me

edit: maybe `as_frame` sounds better!

:+1: for `as_frame`.

Eventually changing the default to True (to return a frame) sounds good. I think you're saying start deprecating in 0.16 right? And make the change later?

As a pure "what if" option, in `DataFrame.to_dict(self, outtype)` method, outtype is `{'dict', 'list', 'series'}`, so `outtype={'list', 'frame'}` might be better consistency-wise.

@immerrr But this `outtype` kwarg is recently deprecated in favour of `orient` (which was more in line with `from_dict`) ... 
https://github.com/pydata/pandas/issues/7840, http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_dict.html

I don't know if there are other examples in pandas of similar things?

`orient` sounds good

ah, that is not what I meant :-)

How does the word 'orient' relate to the fact it is a series or expanded to dataframe?

@jorisvandenbossche that is true. ok, `outtype` (sematically correct here), the other deprecation was more for consistency.

in [boxplot](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.boxplot.html) you have the `return_type` kwarg

ohh, I like that. @billletson want to change to that kw?

@TomAugspurger yes, the idea is to change this in 0.16.0 (or maybe just deprecate and change the default later)

Speaking of making it the default, it makes a lot of sense to return frames by default when `n is not None`, i.e. when you have an upper boundary on number of created columns. When the number of splits is unknown in advance, it seems safer to return lists instead, or it may blow up on a single unfortunate value...

Revised the kw, added a release note, as well as a couple more test cases.

looks good to me, @jorisvandenbossche ?

@billletson can you also create a new issue to have the default changed to frame? pls reference this issue (i will mark it as 0.16.0)

yep, looks good. @billletson Thanks a lot!
BUG: cut/qcut on Series with "retbins" (GH8589)
State: closed

Fixes #8589.

I centralized a bit of logic into tile.py's `_bins_to_cuts()` so that this change could be made in one place, hope that's okay.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
thanks!

FYI, if you are interested in creating an `Interval` class would be gr8!

e.g. `(-1.5,0]` -> `Interval(left=(-1.5,'open),right=(0,'closed'))`

(which repr's to the above)!
Fix timezone comparison in DatetimeIndex._generate
State: closed

This fix is just a band-aid-- a real fix will have to wait for PEP 431 to unite all the time zone implementations.

Also refine a bare `except:` in the same method.

Fixes #8616 


Labels:
name: Timezones
description: Timezone data dtype

Comments:
need to have the tests from the referenced issue

Derf.  Will do.

@ischwabacher can you revisit/rebase

closing pls reopen if/when updated
Fix shape attribute for MultiIndex
State: closed

This fix ensures that `idx.shape == (len(idx),)` if `idx` is a MultiIndex.

Also added tests for other ndarray compat properties.


Labels:
name: API Design
name: MultiIndex

Comments:
hmm these should be the same
is something failing?

@jreback here's what I found on master:

```
In [1]: import pandas as pd

In [2]: idx = pd.MultiIndex.from_arrays([['a', 'b'], [0, 1]])

In [3]: idx
Out[3]:
MultiIndex(levels=[[u'a', u'b'], [0, 1]],
           labels=[[0, 1], [0, 1]])

In [4]: idx.shape
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-f9305225184b> in <module>()
----> 1 idx.shape

/Users/shoyer/dev/pandas/pandas/core/base.pyc in shape(self)
    298     def shape(self):
    299         """ return a tuple of the shape of the underlying data """
--> 300         return self._data.shape
    301
    302     @property

AttributeError: 'NoneType' object has no attribute 'shape'
```

this is a little odd because a MultiIndex is 1-d. So this could be a bit confusing. `len(idx) == shape(idx)[0]`

IMO MultiIndex can be unambiguously interpreted as a 1d array of tuples. I don't think it's surprising to have `len(idx) == shape(idx)[0]` -- that's exactly the case for any 1d numpy array. All the other indexes already have shape defined (in the same way).

I know but you want to return `(5,20)` or something right? (e.g. tuple of length of levels)?
maybe could make a new attribute? thought this prob doesn't break anything

Nope, I just want to return (len(idx),)

On Fri, Oct 24, 2014 at 12:04 PM, jreback notifications@github.com
wrote:

> I know but you want to return `(5,20)` or something right? (e.g. tuple of length of levels)?
> 
> ## maybe could make a new attribute? thought this prob doesn't break anything
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/8609#issuecomment-60434245

ahh, then this is indeed a missing component. ok, rebase and will put in

rebased and green
ENH: Raise error in certain unhandled _reduce cases.
State: closed


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Oh, I’d thought there was possibly interest in supporting axis 1 (single element reduction along a perpendicular axis) because the preexisting group by tests I modified had been attempting to sum a series on axis 1.

I changed to make this a value error per your comment.

pls rebase and add a doc-note in v0.15.1 enhancements (reffing this pr as it doesn't have a separate issue)

Ok, rebased and added to whatsnew.

@staple sorry, you will have to rebase again, I moved the whatsnew file to a subfolder

@staple pls squash as well, otherwise looks ok

Ok, rebased and squashed.

minor comment: is this error message only supposed to be raised internally? Or will users also see this in some cases? 
If that is the case (users will see it), I don't think it is clear as users don't know what `_reduce` is. If it is not the case, the enhancement entry seems not very interesting for users (not to say that the PR would not be interesting of course!)

@jorisvandenbossche Users can see this error message. What would make more sense for a user message? Would something like 'Series aggregation does not implement numeric_only' make sense?

Would it be possible to detect in which kind of aggregation function this is called? 

It looks like 'name' (the name of the aggregation function) is generally passed to _reduce. But it's an optional argument. I could try to make it a non optional argument, and use the value there in the error message. Would that make sense?

I just wanted to suggest the same!

Ok, cool.

It is indeed an optional argument, bug eg in https://github.com/staple/pandas/blob/reduce/pandas/core/generic.py#L3924 where it is used in `_make_stat_function`, `name` is not an optional argument, so it would always be present (but you should look if it is used in other places)

Ok, I made the requested changes. Left them in separate commits for now, for ease of review. But if you like I can squash them when you're ready. Let me know if the change to make 'name' a required argument to _reduce should be a separate commit.

Ok changed the whats new text.

Yep, better! 
@jreback OK with you that `name` is now a required positional arg of `_reduce`?

let me have a look

Another remark: shouldn't this rather be a UserWarning instead of an error? (I don't know how similar things are handled in other places in pandas) 
Also, if we raise an error and not a warning, we should maybe keep this for 0.16 instead of 0.15.1?

I think this is ok (only thing, maybe need a couple of specific cases that actually trigger this error, rather than a constructed case). can't think of any off-hand though. pls rebase / squash.

Ok, I rebased and squashed.

I think this is fine. @jorisvandenbossche ?

thanks!
BUG: LooseVersion was used incorrectly
State: closed

closes #8581 

Version comparison had a formatting error resulting in "TypeError: expected string or buffer" and preventing Spyder from loading (and probably other issues). Fixed.


Labels:
name: Bug

Comments:
can you add a note in v0.15.1 (refing the issue). In what cases would this fail? (e.g. if you have no deps then it bypasses that whole try,except), but when would it hit that error?

Not sure if you mean just leave a comment here? -- I found the issue when the offending code prevented Spyder from launching after I installed the Google Python API. It was also causing the pandas library to think that the API could not be imported, or an error when one wrote "from  pandas.io import gbq"

this is not very well tested (really hard as we don't have an authentication setup). Just trying to see if only having _some_ deps caused this to fail.

merged via https://github.com/pydata/pandas/commit/403f38da35ab04d3997f61db3c98134fe02910d3
thanks!

I'll see about rerunning the integration tests this coming weekend.

I'm having some permissions issues with our Bigquery automated test suite. However, as with https://github.com/pydata/pandas/pull/8590 it seems that gbq's test_requirements() method is failing if the user has `bigquery` installed (which is a requirement for the automated tests of gbq). 

```
__main__.test_requirements ... SKIP: pandas requires google-api-python-client >= 1.2.0 for Google BigQuery support, current version 1.2
```

Of course, this may be unrelated to this patch - just thought I'd mention it. It looks like this can be resolved by simply correcting the test_imports() method:

```
           if LooseVersion(_GOOGLE_API_CLIENT_VERSION) >= '1.2':
                _GOOGLE_API_CLIENT_VALID_VERSION = True
```

I'm not sure what the consequences of doing this are though...
DOC/REL: clean-up and restructure v0.15.0 whatsnew file (GH8477)
State: closed

Closes #8477 

What did I change:
- some general clean-up (blank lines, ..)
- put the different new feature sections in front (as also in the highlights section)
- the 2-level menu structure as discussed in #8477 
- API section: move some entries to the 'enhancements' section (entries that were not 'real' (in sense of breaking) API changes):
  - describe keywords include/exclude
  - new 'level' keyword for Index.set_names/set_levels
  - new 'level' keyword for Index isin
  - histogram/boxplot also via `plot`
  - index new methods duplicated and drop_duplicates

Still work in progress!


Labels:
name: Docs

Comments:
you are going to incorporate this in 0.15.0 release? (ok by me)

@jreback it can just be merged here (so it is in history and in the repo, but only after the 0.15 tag. So not really in the 0.15 release, but not a problem I think). I can just update the docs by checking out the 0.15 tag and applying this patch but without committing.

@jorisvandenbossche yep that's what I was going to do.

ok, will announce (everything is posted already, so need to do it). Then merge / change docs as needed.

thanks!

yes, go ahead with announcing, not need to wait on this (everything is already there, it is just a bit restructuring). And thank you!

@jorisvandenbossche merge this when ready (then will close 0.15.0 milestone)

@jorisvandenbossche isn't this done with released docs? (or you just need to merge this in).

prob need to change the template for 0.15.1 a bit as well

yes, I will try to finish this tomorrow and then merge (most changes are already in the online docs now, only the last ones not yet, but will then rebuild the released docs tomorrow as well). And will also model the 0.15.1 whatsnew file accordingly.

@jorisvandenbossche perfect thxs

OK, finished this up. Updated docs that reflect these changes here are also online.

@jreback  and I added a wiki page on building/uploading the docs for a release: https://github.com/pydata/pandas/wiki/Building-and-uploading-the-official-documentation
TST/PERF: optimize tm.makeStringIndex
State: closed

As a performance junkie I benchmark a lot and that involves frequent creation of temporary arrays of significant sizes.  I couldn't but notice that `tm.makeStringIndex` is somewhat slow in that regard and this PR should fix that.

It also does:

```
* add tm.rands_array(nchars, size) function to generate string arrays

* add tm.randu_array(nchars, size) function to generate unicode arrays

* replace [rands(N) for _ in range(X)] idioms:
  - with makeStringIndex in benchmarks to maintain backward compatibility
  - with rands_array in tests to maintain 1-to-1 type correspondence
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
looks finer I'll merge after 0.15 release (today or tomorrow)

thanks!