BrandonFors/pandas-github-issues huggingface dataset:
BUG: fixes pd.Grouper for non-datetimelike groupings #8866
State: closed

closes #8866 

The bug was caused by a couple of things, firstly when specifying only a level, the axis default was None. This meant it failed to create the internal grouper breaking on line 254 of groupby.py: 

``` python
ax = obj._get_axis(self.axis)
```

Secondly, the aggregate function is set up to take a basegrouper class whereas for datetime resampling we need to pass a datetime index. I split the `_set_grouper` method to return a datetime index when a resampling frequency is specified and a `BaseGrouper` class otherwise. I opted to split that out into two functions to reduce complexity with much of the `_set_basegrouper` function being borrowed from the `_get_grouper` function. Let me know if there's a better way to fix this.


Labels:
name: Bug
name: Groupby

Comments:
this needs a test. This should be a much simpler change.

I thought it was going to be simpler, maybe I've missed something but I couldn't get around that I needed to pass a basegrouper to aggregate and a datetime index to resample. 

no, this just needs a small change in `TimeGrouper` to handle the `level` arg. 

when `pd.Grouper(freq='....'....)` happens, it is immeditately returned as a `TimeGrouper`. (see `__new__`). Its not the best design, but `TimeGrouper` is a valid class.

Ah I see, I presumed TimeGrouper should only be used for datetimes, (was that originally the case?). So anything that gets passed into groupby as a grouper will be handled by the `TimeGrouper` then. I guess that simplifies things, I'll look into making that change instead.

I've had a look again and it's easy enough to return a `TimeGrouper` when level is specified, however the problem is that the groupby calls the `TimeGrouper._get_grouper` function which returns a `BinGrouper`. I'm not sure it makes sense to convert a non-datetime index to a `BinGrouper` does it? Only  `BinGrouper` and `BaseGrouper` are currently valid groupers for aggregating.

no this should be virtually identical to setting the index then resampling (in fact that can be w test)

how can we resample on a non-datetime index?

I mean groupby.

turns out this was a trivial issue, see #9008 

thanks for the effort
ENH: add contextmanager to HDFStore (#8791)
State: closed

Added a context manager to HDFStore

closes #8791


Labels:
name: Enhancement
name: IO HDF5
description: read_hdf, HDFStore

Comments:
this is fine

pls add a release note in the enhancements section
pls review the docs in io.rst to see if anything needs to be changed

@jreback I've updated the docs. Not sure if I've got it right.

pls squash, otherwise looks ok.

`get_store` now looks superfluous. Maybe we should deprecate in future versions?

@jorisvandenbossche 

@jreback I've added warning that get_store is deprecated

On the deprecation, I don't know if it is needed to really deprecate it, maybe we could only do "deprecation by documentation" (-> keeping it as an alias (and indicate this in the docstring of `get_store`) but use `HDFStore` everywhere in the documentation)

@jorisvandenbossche @jreback so I've made those changes. Sorry misunderstood that you guys wanted it deprecated. 

ok, pls add a release note (put in enhancements section)

looks ok to me. ping on green.

@jreback it's green

@vikram thanks!
Cookbook text fix
State: closed

Closes #8944


Labels:
name: Docs

Comments:
@SteveSimmons Make a new branch off of master, then use `git cherry-pick 1b0db93`.

@SteveSimmons yep, good catch (`size` was only introduced recently for DataFrame, see #8847, this is the reason it was working before)

@SteveSimmons need to rebase/squash this on master

@SteveSimmons I guess you deleted this branch? want to resubmit?

Yes, i will redo that branch. I am still learning about git, making clean commits, rebasing branches against upstream head, and have made some mistakes along the way. e.g. i am still not clear if I should worry a push is x commits behind head.

On 3 December 2014 22:29:29 GMT+00:00, jreback notifications@github.com wrote:

> @SteveSimmons I guess you deleted this branch? want to resubmit?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/8957#issuecomment-65503320

## 

Sent from my Android device with K-9 Mail. Please excuse my brevity.
Applying max_colwidth to the DataFrame index (#7856)
State: closed

closes #7856 

reports a problem with the application of max_colwidth to groupby.

The real problem is that max_colwidth is not applied to the index. This patch solves that.

I am not applying self.justify to the formatting of the index (as that would break a test). But maybe it makes more sense to change the test?


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
needs a test

cc @bjonen @onesandzeroes

I have added a test case. Tell me if you want anything changed, or if anything is missing. I will do a final squash when everything is up to standard.

@jorisvandenbossche I now have time time to finalize this. I have merged with the current head. If there is still interest in dealing with this issue just say. I could imagine that there is still a few things to do (like adding an entry to whatsnew)...

@tiagoantao Certainly interest!

Remark on the test: I don't think it is testing the correct thing (but you can also leave it in as it is now for testing the width of the columns), as it is testing the width of the columns which is already working. You can eg set some columns as the index, and then test again. That is what now is failing:

```
In [25]: pd.set_option('max_colwidth', 20)

In [26]: df
Out[26]: 
     a    b                    c  d
0  foo  bar  uncomfortably lo...  1
1  foo  bar                stuff  1

In [27]: df.set_index(['a', 'b', 'c'])  <-- failing in the index
Out[27]: 
                                                    d
a   b   c                                            
foo bar uncomfortably long line with lots of stuff  1
        stuff                                       1
```

pls add a release note that explains this. if you can do this asap i can put in 0.16.0

And the test needs an addition, see my comment above.

I will do this over the weekend.

On 6 March 2015 at 09:21, Joris Van den Bossche notifications@github.com
wrote:

> And the test needs an addition, see my comment above.
> 
> ## 
> 
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/8954#issuecomment-77529226.

## 

When the senses are shaken, and the soul is driven to madness, Who can
stand? - William Blake

I have tried to sort this out. My sincere apologies for the delay. I will be monitoring this during the next day or so. I will try to react as fast as possible if you find anything in need of intervention.

Did all the corrections. I just hope the way I committed (i.e. in terms of logs and squash) is acceptable. My git-fu is a bit substandard.

I apologize for all these mistakes. I hope in the future, with this experience, I am able to do a few contributions will less noise.

merged via https://github.com/pydata/pandas/commit/e4cb0f8a6cbb5f0c89b24783baa44326e4b2cccb

thanks!
Updating generic.py error message #8618 - New branch
State: closed

closes #8015

I deleted the previous branch that was causing problems and submitted this.
This is my first attempt at a PR. I just made a little change to allow an error message. Would appreciate some feedback.
@jreback @jorisvandenbossche @hayd 


Labels:
name: Groupby
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@springcoil That are not the errors that I see on travis. These tests are failing:

```
======================================================================
ERROR: test_cumcount_empty (pandas.tests.test_groupby.TestGroupBy)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/pydata/pandas/pandas/tests/test_groupby.py", line 3703, in test_cumcount_empty
ge = DataFrame().groupby()
File "/home/travis/build/pydata/pandas/pandas/core/generic.py", line 2878, in groupby
raise TypeError('You have to specify at least one of "by" and "level"')

TypeError: You have to specify at least one of "by" and "level"

======================================================================
ERROR: test_groupby_args (pandas.tests.test_groupby.TestGroupBy)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/pydata/pandas/pandas/tests/test_groupby.py", line 2011, in test_groupby_args
self.assertRaisesRegexp(TypeError, k, "You have to supply one of 'by' and 'level'")
File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 1596, in assertRaisesRegexp
manager = _AssertRaisesContextmanager(exception=_exception, regexp=_regexp)
File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 1609, in __init__
regexp = re.compile(regexp)
File "/home/travis/miniconda/envs/pandas/lib/python2.7/re.py", line 190, in compile
return _compile(pattern, flags)
File "/home/travis/miniconda/envs/pandas/lib/python2.7/re.py", line 240, in _compile
raise TypeError, "first argument must be string or compiled pattern"

TypeError: first argument must be string or compiled pattern
```

The tests pass :) Can we merge now?

Ok second attempt :)

@springcoil Could you squash?

You can do this with:

```
git fetch upstream
git rebase -i upstream/master
git push -f origin PR8015_springcoil
```

and then change 'pick' into 'fixup' apart for the first commit. 

Hi @jorisvandenbossche thanks for the squash reminder. I Squashed it and commited the new code. 
I hope all is well now. 
Thanks also for confirmation it was ok to change a few of the tests, as I needed to get them to pass. 

merged via https://github.com/pydata/pandas/commit/2ed182e41d059c21fab5de832da9bfcbc5bbe1fb

thanks! I know it was an experience, but the first PR is always the hardest.

Thanks :)
BUG: preserve left frame order in left merge
State: closed

closes https://github.com/pydata/pandas/issues/7331

on master:

```
>>> left
      dates states
0  20140101     CA
1  20140102     NY
2  20140103     CA
>>> right
   stateid states
0        1     CA
1        2     NY
>>> pd.merge(left, right, how='left', on='states', sort=False)
      dates states  stateid
0  20140101     CA        1
1  20140103     CA        1
2  20140102     NY        2
```

`DataFrame.join` already works fine:

```
>>> left.join(right.set_index('states'), on='states', how='left')
      dates states  stateid
0  20140101     CA        1
1  20140102     NY        2
2  20140103     CA        1
```

on branch:

```
>>> pd.merge(left, right, how='left', on='states', sort=False)
      dates states  stateid
0  20140101     CA        1
1  20140102     NY        2
2  20140103     CA        1

>>> pd.merge(left, right, how='left', on='states', sort=True)
      dates states  stateid
0  20140101     CA        1
1  20140103     CA        1
2  20140102     NY        2
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
ok then!

btw, if you are looking for something to do......

https://github.com/pydata/pandas/issues/4088 ( I did this a while back, https://github.com/pydata/pandas/pull/6647 but it doesn't quite fix this)
API: Allow equality comparisons of Series with a categorical dtype and object type are allowed (GH8938)
State: closed

closes #8938 


Labels:
name: Bug
name: API Design
name: Categorical
description: Categorical Data Type

Comments:
@jorisvandenbossche 
cc @shoyer
cc @JanSchulz
cc @immerr

?

Probably want to make sure `!=` works, too.

@shoyer good point, fixed up.

This should also get some updates in categorical.rst. Want to get a patch?

@JanSchulz if you want to post a patch all ears! (for docs)....i'll fix the other in a minute

Patch is at #8952

What I started to wonder when I did the docs: if `cat > scalar` is allowed and `cat == list` also because it basically is doing a comparison of each line as if it was the scalar case, then by that logic, `cat > list` should also be allowed: each row in that comparison would treat the element from the list as a scalar. 

On the other hand a scalar comparison with the categorical makes only sense if the scalar can be treated as a category (for any other value, it's basically a "not of the same type" comparison, which would raise on python3), so the scalar must be in `categories` and this should not work:

```
In[4]: df = pd.DataFrame({"a":[1,3,3,3,np.nan]})
In[6]: df["b"] = df.a.astype("category")
In[7]: df.b
Out[7]: 
0     1
1     3
2     3
3     3
4   NaN
Name: b, dtype: category
Categories (2, float64): [1 < 3]
In[8]: df.b > 2
Out[8]: 
0    False
1     True
2     True
3     True
4    False
Name: b, dtype: bool
```

@JanSchulz I can kind of buy that last example we would 'like' to work (as it seems natural), but it does violate the Categorical principles.

Ok, how about we merge this to fix the equalitiy inconsistency with merging then open a new issue for discussion about non-equal comparisons.

@shoyer 
@jorisvandenbossche 

?

@JanSchulz ok on this?

Jep, I'll open a new issue with that..

Oh, one more thing: according to that thought, `df.b == 2` (-> The "equality" case) should also NOT work, because `2` is not in `categories` and therefore a "different type".

@JanSchulz gr8 thanks

```
In [5]: df.b==2
Out[5]: 
0    False
1    False
2    False
3    False
4    False
Name: b, dtype: bool
```

this is actually consistent (e.g. it returns `False`). On a comparison it shouldn't raise so this is a reasonable result. I think this is de-facto like the following and is useful.

```
In [7]: Series(['a','b','c'])==2
Out[7]: 
0    False
1    False
2    False
dtype: bool
```
round function added to DatetimeIndex
State: closed

closes #4314 


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
- needs a release note, but this I think will have to wait for 0.16.0 as we need to fixup round for `NumericIndex`.
- needs tests & adjustment for `NaT` (wrap the `.round` using `self._mask_missing`)
- add in tests (including `NaT`)  & impl for `TimedeltaIndex` as well (impl is easy here as well, you can just `self._mask_missing(lib.map_infer, Timedelta.round)`)

closing pls reopen if/when updated
BUG: Resample across multiple days
State: closed

Fixes an issue where resampling over multiple days causes a ValueError
when a number of days between the normalized first and normalized last
days is not a multiple of the frequency.

Added test TestSeries.test_resample

Closes #8683


Labels:
name: Bug
name: Resample
description: resample method

Comments:
@hkleynhans I think test_resample.py is a better place to put the test (https://github.com/hkleynhans/pandas/blob/fix/8683_resample_value_error/pandas/tseries/tests/test_resample.py)

Thanks for the feedback.  The issues have been addressed.

@jreback can you have a look at this? (or is there someone with great knowledge of the resample code?) But the change seems logic, and the tests pass

will have a look

looks good.

@jorisvandenbossche merge on green.

@hkleynhans Thanks a lot! It was nice debugging
BUG: Fixed font size (set it on both x and y axis). #8765
State: closed

Fix for https://github.com/pydata/pandas/issues/8765 
The font size was only set on x axis if vertical or the y axis if horizontal.


Labels:
name: Visualization
description: plotting

Comments:
@TomAugspurger can you review this?

@papaloizouc ready to merge if you could add a short note to `doc/source/whatsnew/v0.15.2`.

Thanks!
Categorical: let unique only return used categories
State: closed

Discussion see here: https://github.com/pydata/pandas/issues/8559#issuecomment-64189570


Labels:
name: API Design
name: Categorical
description: Categorical Data Type

Comments:
put a release note referincing that issue. I think this is already in the API.rst ? (maybe just check doc-string too).

`Categorical.unique` is not in api.rst, only methods under `Series.cat` are.

re release note: under API change or bug fix?

ahh, so maybe add to `Series.cat` methods? I forgot how we are calling this

`s.unique()` of course does it, but I suppose `s.cat.unique()` should be allowed as well?

This 'changes' the API slightly so API change it is. (maybe put a short explanation), example if you feel it is waranted.

ok, added it under api change (without an example)

I don't think any more methods should be added to `Series.cat`: simple calling `s.unique()` (or `s.min()` or `s.describe()`) is enough. After the last discussion about what is API and what is not, I started to think of all methods in `Categorical` as "for compatibility with numpy" and I think that's the right way to think about them. If a users knows better they can look at the docstrings, but I don't think pandas should do any API guarantee whether these methods stay or change or be removed. 

thanks!
ENH: Adds gcs module for IO with Google Cloud Storage
State: closed

Hi,

This isn't ready for primetime yet, but I wanted to see if there was feedback, since it would be a new io module.

Basic usage:

```
import pandas.io.gcs as gcs
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'a': np.random.normal(size=1e4),
    'b': np.random.normal(size=1e4)
})

gcs.to_gcs(df, "th-code", "pandas2.csv", "eighth-physics-623", to_gcs_kwargs={'resumable': True}, to_csv_kwargs={'index': False})
i = gcs.from_gcs("th-code", "pandas2.csv", "eighth-physics-623")
```

This borrows some of the connection patterns from `gbq`, though it can do service account auth in addition to OAuth2WebServer auth.  Might be nice at some point to centralize google api connection stuff.  If this is accepted the code to connect to google's APIs will be in 3 places.

I'm not sure everything this would need to be accepted, besides tests...


Labels:

Comments:
for sure want to fix the connection API / testing issues (that is a separate / distinct PR).

this seems like a seemless-file-transfer mechanism? can you post a link to the service

You mean like: https://cloud.google.com/storage/?

yep. You are moving a csv to the storage? (or a pickle?) do they have an API for it? (the format I mean).

Yeah, right now I have it defaulting to `text/csv`.  Through the API it comes back from - and goes to - GCS as a string, so I'm using `StringIO` to go between. 

I have looked at other formats.  I believe it's possible to change the format by setting the `contentEncoding` parameter when inserting the object.  To me gzip is more appealing than pickle, but both would be nice.

closing, pls reopen if/when this is ready
BUG: moving the utf encoding line to the first line (before from __futur...
State: closed

```
ERROR: Failure: SyntaxError (Non-ASCII character '\xc3' in file /home/fvia/bloom/pandas/pandas/tests/test_format.py on line 825, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details (test_format.py, line 825))
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unicode
description: Unicode strings

Comments:
looks good, can you audit all the files to see if this is case anywhere else?

is it recommended that EACH file have an encoding line? (or ONLY where we actually use unicode)?

where does this failure show up?

This failure shows up in python2,  when somebody using python3 writes a non ASCII char. 
Adding an encoding line in every file can reduce the number of bugs as more people are moving to python3.  

I also think it is just good practice to specify that in each file

I will add a encoding line to all files in test/

thanks!
BUG: allow numpy.array as c values to scatterplot
State: closed

Ensure that we can pass an np.array as 'c' straight through to
matplotlib, this functionality was accidentally removed previously.

Add tests.

Closes #8852


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
pls squash, otherwise looks ok

@TomAugspurger ?

I'm fine with this being two commits since it separates the addition of `is_hashable` and the plotting issue.

Could you add a test or two for `is_hashable` too? Thanks.

@aevri pls add test for `is_hashable` in `tests/test_common`

I've updated with these changes:
- added tests for `is_hashable` in `tests/test_common`
- modified `is_hashable` to first reject based on collections.Hashable, this is to avoid broadening the definition of hashable to include old-style classes and anything else I can't think of
- changed whatsnew entry to say `DataFrame.plot(kind='scatter')`
- In scatter_with_c tests, supplied RGBA colors as 'c' and added test for face colors to be identical to supplied values
- rebased to catch up with upstream

Was considering how to test the face colors of the case where we supply a colormap, don't know how to query the colormap ahead of time though.

Cheers!

@aevri By the way, David (@cournape) fixed it in numpy today: https://github.com/numpy/numpy/pull/5326

And the tests are failing on python 3 (and on numpy master, because of Davids fix)

Have updated:
- rebased on master to fix merge conflicts
- wrapped Python2-specific tests in sys.version_info check
- made numpy.array test work with or without numpy fix

Cheers!

Rebased again for new merge conflicts

Afaik all points are addressed and this is ready to land, please let me know if there's something I've missed :O)

Looks OK for me! Lets wait until travis is green

@aevri thanks for the fix!

Yay! Thanks everyone :)
Implement Categorical.searchsorted(v, side, sorter)
State: closed

Continued in #8972

This closes #8420, with code that supports searchsorted in Categorical, for both side='left'/'right' and with an optional sorter. The case of side='right' was tricky, as we look to the right only when matching the category name, not when matching the codes. The docstring and unit tests cover the various cases.

One thing I wasn't sure about was forcing scalar input to a Series. I copied this behaviour from Series.searchsorted to match its docstring examples. To avoid circular imports from series.py importing categorical.py, I got Series via a local import. Probably there is a better way to do this.


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
@SteveSimmons can you rebase and take out the print statements

Dealt with all your comments in pull request #8928. Changing the wrapping of v from pd.Series(v).values to np.asarray(v) changed the scalar case's output from array to scalar. I decided to keep this as it matches numpy.searchsorted's behaviour. The case of scalar input is very much an edge case, anyway. What do you think is the right behaviour?

Continued in #8972
BUG: fixed chunksize guessed to 0 (py3 only). #8621
State: closed

Fixes https://github.com/pydata/pandas/issues/8621
As mentioned in the github issue its a py3 only bug. I copied the test @rmorgans made


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
need a release note in v0.15.2 referencing the original issue.

@papaloizouc thanks for the fix!
astype checks notnull Fixes: #8732
State: closed

As suggested in issue #8732 if there are no nulls, we skip the nan_safe bit.


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
needs a vbench for perf comparisons 

@jreback there are a lot of checks for different dtypes (datetime, timedelta, etc) in the `_astype_nansafe` (https://github.com/pydata/pandas/blob/master/pandas/core/common.py#L2541). I suppose they can't just be skipped at all if there are no nans?

I think the original purpose was to render object types safe (as in an actual python object)
so for example datetime should be ok 

but needs some investigation

You are right needs a bit more work.