BrandonFors/pandas-github-issues huggingface dataset:
BUG: Add squeeze keyword to groupby to allow reduction in returned type
State: closed

This will allow a reduction in the returned type from DataFrame -> Series
if the groups are unique.

This is a fix for a regression from 0.10.1.

Allows functionaility in #2893, by specifying `squeeze=True` 
in groupby call. #3596 functionaility is back as the default

This returns a Series because we are passing `squeeze=True`

```
In [9]: df2 = DataFrame([{"val1": 1, "val2" : 20}, 
{"val1":1, "val2": 19},{"val1":1, "val2": 27}, {"val1":1, "val2": 12}])

In [10]: df2
Out[10]: 
   val1  val2
0     1    20
1     1    19
2     1    27
3     1    12

In [11]: def func(dataf):
   ....:     return dataf["val2"]  - dataf["val2"].mean()
   ....: 

In [12]: df2.groupby("val1", squeeze=True).apply(func)
Out[12]: 
0    0.5
1   -0.5
2    7.5
3   -7.5
Name: 1, dtype: float64
```

Traditionally returns a DataFrame (even though have unique groups)
Implicity (`squeeze=False`)

```
In [13]: df = DataFrame([[1,1],[1,1]],columns=['X','Y'])

In [14]: df
Out[14]: 
   X  Y
0  1  1
1  1  1

In [15]: df.groupby('X').count()
Out[15]: 
   X  Y
X      
1  2  2
```


Labels:

Comments:
@wesm pls take a look. I think the prior behavior is correct, most often you would not want the reduction (which is data dependent). The new keyword allows it.

This is a good feature/fix.  
Note: `squeeze` is what's used in [`read_csv`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.parsers.read_csv.html) to do this... perhaps we should use the same name here?

thats a good idea! will update. thankxs

@hayd better? (and is the example in v0.11.1 adequate?)
raise on fillna passed a list or tuple
State: closed

fixes #3435.


Labels:

Comments:
pls add a release notes mention (and in general if its worth of the whatsnew a mention of that too)

@jreback Okay. Rule of thumb for when to add a rls notes mention?

pretty much anything non-trivial should get in release notes (or trivial but its a API change or bug)

if its a separate issue (e.g. documented), I always put in release notes with an issue link. Docs don't bother (unless again it has an actual issue attached, e.g. #3566) whatsnew if it is an non-trivial API change, or you want to 'announce' it, e.g. your regex stuff will certainly need a whatsnew (and docs), as its useful and new behavior. I just do a quick ipython (usually based off the tests), or you have doc strings, whatever works.

so this needs a mention (as its an issue). If you had just discovered this (and not an op), then maybe doesn't make much difference as a future user would then read the doc string and not be suprised (or get the exception).

cool. great explntn.

this looks ready for merge, yes?

@cpcloud can you rebase....this is the annoyance with release notes. the fact that I merged something in a different order causes these things to not be auto mergable.....thxs

@jreback rebased

better commit message
Moving pandasjson back into mainline pandas
State: closed

This extension works perfectly fine on Python 2.x and all platforms to my knowledge except win32 on MinGW (it's fine with VC2008+ AFAIK). It might have problems on some of @yarikoptic's esoteric Debian platforms, we should identify which ones those are and disable the extension in the setup.py until we have a chance to sort out what's wrong in the C code. I don't think that's a bad compromise for now in the interest of having a usable extension for a lot of folks. 


Labels:

Comments:
also cc @Komnomnomnom

Ok, so iirc the problems on the esoteric platforms were solved, at least on SPARC 32+64, if memory serves there was some memory alignment and buffer overflow problems and the fixes were pushed upstream to ujson.

As for windows, that was indeed a bit of a nightmare, the problems that occurred were intermittent  segmentation faults which were hard to reproduce and pin down, and I neither had the windows tools or experience to debug it thoroughly. Here's what I concluded at the time (EPD = Enthought Python Distribution):

> With an independent version of mingw -> no segmentation faults. Compiled a python debug version on win32 and compiled numpy and pandas using EPD's mingw -> no segmentation faults. From the little bit of a stack trace I've been able to get it appears that the problem occurs when ujson calls numpy C-API functions (take that with a healthy dose of salt however as the stack trace also looked pretty corrupted.)

I think the problem occurred when mixing VS compiled ujson and mingw compiled numpy, but it will take more investigation to know for sure. Ideally it would be great if someone with a proper windows setup could take a look, but maybe having it in pandas but disabled might motivate some brave soul...

On another note I think there have been some minor fixes and enhancements to ujson since it was last merged with pandasjson. Once / if pandasjson is re-included in mainline pandas I'll see about merging them in and send a pull request through?

BTW the extension should work fine in Python 3.x too, or are there some issues I'm not aware of?

Yes I think it should be working with python 3, I need to run the CI tests (I didn't have travis enabled when I opened the PR). If you could merge in the upstream changes sometime that would be great

@wesm travis is broken :(

@wesm @Komnomnomnom @hayd want to give #3804 a test run?

closed via #3804
ENH: plot only numeric data and raise an exception *before* plotting if there is no numeric data
State: closed

Raise a `TypeError` alerting the user to the fact that they are trying to plot nonnumeric data, or if there are any numeric data plot those. closes #1818.


Labels:

Comments:
I think this should raise before plotting anything... @jreback @y-p thoughts? It's annoying for a plot to come up only to discover that it won't work anyway. 

@cpcloud I think there is a flag `raise_on_error` which does this, @changhiskhan put this is a while back...?

`raise_on_error` only gets called after a plot comes up.

oh
thought that was the point of it
ok then

raise_on_error is a way to catch an exception and show the message, it was only being called for nonnumeric data, so i essentially just replaced it with a typeerror which i thought was clearer.

ok...so user can still set `raise_on_error` to `False` if they know what they are doing I guess

pls add release notes!

@jreback @y-p any thoughts here? i could do something like

``` python
if isinteractive():
    ioff()
    the_one_true_plotting_function() # catch the error here before drawing
    ion()
draw_if_interactive()
```

I dont' use this enough to give you an opinion here, but looks reasonable

nvm. i can use the currently-unused-method `_compute_plot_data` to check...

i think plotting only numeric data would be a better solution to this problem. have to check if any issues w/ `groupby.plot()`. will leave open until 0.12, because `Series` needs to be an `NDFrame` (it needs the blkmgr method `_get_numeric_data`)

this ready to merge?

not just yet

FYI this could be done by  checking the series dtype but prob better off in. 0.12

:) that's what i did, but since it's not critical it might be better to wait

also that doesn't cover the case of an object array of numbers which means you have to check the whole array for numbers which is linear in the length of the array which is probably not desirable if u want to plot something gigantic although maybe not the worst thing.

actually since u already know the dtype that shouldn't be a problem except in the case of object
where you can just astype in a try except

ah good call didn't think of that

should i do that and submit? that should work in 0.12 too right? there will still be `astype` i presume

sure  

guess that allows `Series(['1.1', '1.2']).plot()` to work as well. that ok?

i could the `'safe'` argument to disallow that

maybe should put a warning on trying to plot an object type in general (if it succeeds that is); otherwise it will raise
in general object type is bad

I think object type should warn or raise

probably raise to make the user explicitly choose to cast it if they want

also raise on error should probably be taken out since that's more of a stop gap than this, although that is an api change

yep raise on object type is prob easiest thing to do

fyi...after this merge the `raise_on_error` argument will be unreachable

then go ahead and take it out and add a note in API 

@jreback this is ready 2 go after teh travis buildz
DOC: document non-preservation of dtypes across rows with iterrows
State: closed

iterrows doesn't preserve dtypes across rows, document it, #3566. i learned that you can nest ..ipython:: blocks in ..note:: blocks in sphinx. sweet.


Labels:

Comments:
@cpcloud u r quick! thanks

i try. :) thanks. there's lots of low-hanging fruit for PRs here, just trying to eat it all up
document read_csv's usecols argument.
State: closed

read_csv's usecols is currently not documented here:

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.parsers.read_csv.html

This patch fixes that.


Labels:

Comments:
I think this also needs to be added to io.rst, can you update?

@jreback, I've updated my patch ;)

thanks!
Parse raised exceptions and their error messages sans interpolation/format spec
State: closed

This PR partially addresses #3024. It also provides a shiny new script useful looking at the current state of the messages in raised exceptions. Here's an example of some output:
# Example

![pandas-shiny-new-parse-asserts](https://f.cloud.github.com/assets/417981/457480/70282b60-b39e-11e2-8db1-f5cfb4f0c5bf.png)

You can reproduce this with

``` sh
scripts/parse_except.py --type empty --module pandas --kind assert
```

from the top level pandas directory.
# Description

The above line searches for all empty raises in the module `pandas` whose constructor matches the regular expression `'.*assert.*'`--not case sensitive.

Note: **This assumes you have installed pandas with `python setup.py develop`**. This will not work without that (well, that's not strictly true, but it will search for pandas code that's installed wherever you installed it `/usr/*/site-packages/*/pandas` or `$VIRTUAL_ENV/*/pandas`, for example, which is probably not what you want).

To see all of the options and a short description you can do the usual:

``` sh
scripts/parse_except.py --help
```


Labels:

Comments:
This could probably use a `NodeVisitor` subclass...but I wanted to hack this together before optimizing.

@jreback this could be merged if y'all think it's useful...all the nan's in the above pic are gone now since i filled all those missing or vague assert messages in and a few others as well. rls notes here?

@jreback @y-p i will close this if there's no chance of it getting in. no sense in keeping things open unnecessarily...

I didnt' realize you changed a bunch of no-message AssertionErrors.....so this is good...

does this pass travis?

> I didnt' realize you changed a bunch of no-message AssertionErrors.....so this is good...

i was hoping that the script would be useful (alas, i probably spent too much time creating the assertion-table-lister thing because i think asts are fun) it should pass...it wasn't before because i did 1e9 rebases in a row and travis doen't like that

@cpcloud 
you might want to join pandas-dev ml:

http://mail.python.org/mailman/listinfo/pandas-dev

this passes
BUG/CLN: datetime64/timedelta64
State: closed

Various bugs related to datetime64s
- Properly convert np.datetime64 objects in a Series, #3416

This would convert to object dtype previously

```
In [1]: dates = [
   ...:     np.datetime64(datetime.date(2013, 1, 1)),
   ...:     np.datetime64(datetime.date(2013, 1, 2)),
   ...:     np.datetime64(datetime.date(2013, 1, 3)),
   ...: ]

In [2]: s = pd.Series(dates)

In [3]: s
Out[3]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2013-01-03 00:00:00
dtype: datetime64[ns]
```
- Do not allow astypes on `datetime64[ns]` except to `object`, and
  `timedelta64[ns]` to `object/int`, #3425
- Do not allow datetimelike/timedeltalike creation except with valid types
  (e.g. cannot pass `datetime64[ms]`, #3423

Any creation/astype of a datetimelike to a non accepted dtype will raise

```
In [6]: Series([Timestamp('20130101'),Timestamp('20130102')],dtype='datetime64[s]')
TypeError: cannot convert datetimelike to dtype [datetime64[s]]
```


Labels:

Comments:
@wesm @y-p These are various datetime fixes, really minor. They basically disallow datetime operations that can cause weird issues, e.g. trying to convert a datetime64[ns] to something which we don't support now will raise
ENH: add mode.mangle_dupe_cols option GH3468
State: closed

#3468

as discussed, an added option to disable th mangling behavior
of dupe columns taken by pd.read_csv going back to olden days.

Not sure what the default should be, currently the default is compatible
with existing scripts.

Tested with both c and python parser engines.


Labels:

Comments:
what about doing this as an argument to `read_csv`, say `duplicate_column_names=False`, default should prob be `False` but that breaks API ......

yeah, I'll go for that. still not sure about the default though. Still compatible right now.
if it breaks back compat, I'll push this to 0.12.

actually I think we should put in the option now
but leave the default = True so doesn't break API compat
(then maybe break in 0.12)?

works for me, added note to RELEASE.rst.
Fixed Unbound Variable `edge` access when BinGrouper is empty
State: closed

Incorrectly accessed variable `edge` instead of pre-filled value `start`.

Signed-off-by: Kevin Stone kevinastone@gmail.com


Labels:

Comments:
This bug surfaced when I was using a grouper on an empty array.  Since the for loop ends without assign edge, it leads to the UnboundVariableError.

can post an example that reproduces (or even better, add a test for this error?)
and provide a release notes mention. thanks!

Ask and you shall receive.

can u setup Travis as well, see CONTRIBUTING

rebase into one commit and then push to have Travis run the tests

let me know if u need any help

thxs

Lock and load.

I can't take it with these RELEASE.rst conflicts. Merged manually via 99137af.

Thanks!
ENH: Support reading from S3
State: closed

Tests not written. Will need a way to mock boto.


Labels:

Comments:
IMO this doesn't belong in pandas (neither are other API storage services), 
users can implement that in their own scripts.

Thanks and I understand your point. 

Retrieving files from S3 offers a powerful io experience in pandas given the way people use S3 for logs and other historical record and pandas' handling of compressed files. Perhaps this would be better as its own IO tool similar to the sql wrappers. It would aid the user in retrieving from and setting to S3.

Feel free to close this or move the discussion to a better resource (mailing list, issue tracker).  

This has me really stumped for some reason. But I guess I should err on the side of "it's useful",
and we do want to make it as easy to get data into pandas as possible.

Please add a note in RELEASE.rst, and mention boto as an optional dependency in README.md
for this functionality, I'll merge this for 0.11.1.

That's the spirit all!

@y-p Thank you. Everything should be up to date.

@pjob , can you please add a blurb to io.rst in the docs?

@y-p sure.

Thanks.
My first PR ever.  Adding documentation to Cookbook.
State: closed

Just adding some more description to Cookbook documentation per J. Reback's request.  If I didn't mess up anything here, I will go through the rest of the items and add some description.

Please let me know if I messed anything up.


Labels:

Comments:
Hi!  very glad to be on the receiving side of your very first PR.

The first commit solves a problem that I'm not sure exists, and it basically rewrites the entire file,
although I don't see any dangling whitespace in it.
if you're on windows, it's possible that your git repo is setup to automatically
convert line endings.

Hi y-p,  

Thanks for the feedback.  I noticed that most of the other `.rst` docs did not have the line ending characters from a file created on Windows*.  When cookbook.rst was first created, I'm guessing the line endings were not handled by git.

I removed all `^M` 's just to make it cleaner for editing because I wasn't sure if mixing the line endings might have unwanted side effects.

Let me know what you think...but I'm happy to change it back.  

*http://stackoverflow.com/questions/5843495/what-does-m-character-mean-in-vim

Thanks, had to resort to hexdump to realize the problem. I rewrote the file
for line endings as @jreback, and incorporated your additions. Thanks for the PR.

aside, consider setting up you git email to a real address using `git config`,
and try to avoid introducing dangling whitespace into patches, your editor
might have a tweakable for that.

I will make those changes accordingly!  I intend to complete documentation in the cookbook in my next PR.
BUG: GH3493 fix Cannot append DataFrames with uint dtypes to HDFStore
State: closed

Fix for self-reported uint bug in HDFStore. #3493
Travis results (pass): https://travis-ci.org/jmellen/pandas/builds/6764554


Labels:

Comments:
pls add a mention in RELEASE.rst referencing the bug and issue

Sorry for the delay this afternoon; here's the release note update.  Thanks!

thanks for the PR!
Failing Unit Test and Patch for Fixing MonthEnd DateRange Unions with Timezones
State: closed

This patch preserves the timezone information when applying an offset to `DatetimeIndex` with a timezone applied.  The `MonthEnd` Offset wasn't retaining the timezone information resulting in an error even when both indexes had timezones.

```
TypeError: can't compare offset-naive and offset-aware datetimes
```


Labels:

Comments:
merged as ce8a15d5ea1f6c.

Thanks.
Support mrecarrays in DataFrame constructor
State: closed

This is intended as a fix for #3478. ~~Sorry I do not have the time to write rigorous tests.~~


Labels:

Comments:
To illustrate my point, this is what the condition where all `MaskedArray` stuff is handled together looks like:

``` python
        elif isinstance(data, ma.MaskedArray):
            def unmask(arr):
                mask = ma.getmaskarray(data)
                if mask.any():
                    data, fill_value = _maybe_upcast(data, copy=True)
                    data[mask] = fill_value
                    return data
                return data.copy()

            if data.dtype.names:
                data_columns, data = _rec_to_dict(data)
                data = dict((k, unmask(v)) for k, v in data.iteritems())
                if columns is None:
                    columns = data_columns
                mgr = self._init_dict(data, index, columns, dtype=dtype)
            else:
                mgr = self._init_ndarray(unmask(data), index, columns, dtype=dtype,
                                         copy=copy)
```

Doesn't that second half look stupendously like the `if isinstance(..., ndarray)` case?? With an unnecessary dict creation thrown in for fun?

I still contend this is a case where code quality should not be sacrificed for marginal efficiency.

You could easily make a function to combine best of both worlds here (also would put `unmask` outside of Frame in any event). certainly try if you would like. Still need tests though. There are lots of cases and need to avoid breaking anything

The point is: you _couldn't_ easily factor out the is/isn't rec-array case, because in unmasking already requires a non-rec array. You can easily factor out the unmasking, which is what I did.

Of course it requires testing (or alternatively, test smaller units like `_unmask` and `_rec_to_dict` and you can get away with fewer tests for the same coverage). But I don't have the time to put to that.

what is your use case for masked record arrays? why u creating them in the first place (rather than directly in pandas)? existing code?

I am not sure the use-case is relevant, seeing as it would be natural for pandas to support them, being the equivalent numpy data structure to what pandas works with, and given that pandas already supports both numpy masking and numpy records / structured arrays. But I am considering mrecarray output from a model selection procedure in scikit-learn: to handle some cases (where the set of parameters set in each iteration is not constant) masking or equivalent is required; leaving it in a numpy structure avoids binding users to pandas and avoids the project including a pandas dependency.

Okay. Here is a test that ensures input with mrecarray is identical to equivalent input with dicts.

closing in favor of #4836

I don't get why you prefer more specialisation rather than code reuse, but I'm glad the feature is now supported.

your solution forced unmasking on evey ndarray which is not good

also the style was different than the rest

your tests were gr8!
ENH: Bring Series.dot up to par with DataFrame.dot
State: closed

If second argument is Series or DataFrame, ensures alignment of indices.
If second argument is some type of ndarray, just fall back to np.dot on the values.
This mirrors the behavior in DataFrame.dot

Includes unittest


Labels:

Comments:
@dieterv77 , by chance, I updated [CONTRIBUTING.md](https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md) today with instructions
on getting Travis-CI working, can you try it out and let me know if it works out for you?

@y-p I actually already configured it after i saw your instructions that you committed today.  I pushed an old local branch to my fork and it triggered a build on travis.  Any way that i can get it to trigger on this PR?  

thanks

I have these aliases defined in .bashrc:

```
alias gca='git commit --amend'
alias gcac='gca -C HEAD'
```

so doing `gcac` gives the current branch a new hash without touching anything else.
if you then force push that to the PR branch, it should trigger a new build on travis for this PR.

Looks like that worked.  thanks

I just hit this and fixed with 66cbdc0.
fyi @dieterv77
BUG: GH3455 Duplicate indexes with getitem will return items in the correct order
State: closed

BUG: GH3457 to_csv writing duplicate columns incorrectly

closes #3455, #3457


Labels:

Comments:
I'll redo #3458 after you merge this.

ok...any other tests that we should run against this? 

I don't think #3457 is fixed by this. I'm happy to just leave it to raise an exception if tried,
engine='python' is just a fallback.

```
In [11]: df= mkdf(10, 3)
    ...: df.columns = ['a','a','b']
    ...: df.to_csv("/tmp/1.csv",engine='python')
    ...: !cat "/tmp/1.csv"
R0,a,a,b
R_l0_g0,R0C1,R0C1,R0C2
R_l0_g1,R1C1,R1C1,R1C2
R_l0_g2,R2C1,R2C1,R2C2
R_l0_g3,R3C1,R3C1,R3C2
R_l0_g4,R4C1,R4C1,R4C2
R_l0_g5,R5C1,R5C1,R5C2
R_l0_g6,R6C1,R6C1,R6C2
R_l0_g7,R7C1,R7C1,R7C2
R_l0_g8,R8C1,R8C1,R8C2
R_l0_g9,R9C1,R9C1,R9C2
```

**edit:** ok, we'll just let #3458 have it raise a warning, and users should transition to the new csv writer code.
BUG: Removing a nonexistent store raises a KeyError (GH3346)
State: closed


Labels:

Comments:
can you hook up travis? (instructions in CONTRIBUTING)

Yes. I think it is now enabled. Thanks for walking me through this a bit.

did you turn it on travis too? (you have to turn on github, then login on travis to do this), then force an update!

I logged into Travis, flipped the switch next to pandas-danielballan, and verified that the Service Hook is there in my github settings. (When I "Test Hook," it says "Payload deployed." Success?) And I forced a push, if that's what you meant.

daniel, wrong branch. you need to force push on `remove-raises-keyerror`

https://travis-ci.org/danielballan/pandas/builds

OK, I see that it is building the master branch, and that the last commit happens to be yours. I'm sorry if this is generating spam for you guys or clogging up the works.

I did `git push -f origin remove-raises-keyerror`. Yes?

perhaps http://stackoverflow.com/questions/6089294/git-why-do-i-need-to-do-set-upstream-all-the-time

dallan@dielectric-pc:~/pandas-danielballan$ git push -u origin remove-raises-keyerror
Branch remove-raises-keyerror set up to track remote branch remove-raises-keyerror from origin.
Everything up-to-date

That looks fine.

now try 

```
git commit --amend -C HEAD # same state, new commit hash
git push -f
```