BrandonFors/pandas-github-issues huggingface dataset:
BUG: allow conversion of Timestamp and Timedelta to string in astype
State: closed

Fixes GH #9757


Labels:
name: Datetime
description: Datetime data dtype
name: API Design
name: Timedelta
description: Timedelta data type

Comments:
this needs some discussion
pls post in the issue why this should actually be allowed
this is string formatting and not the meaning of astype 

ok, seems reasonable. can you rebase / squash. ping on green.

Added a test for `astype(unicode)` on Python 2, and a test for DataFrame.

OK, this seems good to from my perspective. I'll merge unless there are any further objections....

I've totally borked this branch with an accidental force push. I'll fix it tonight. On the positive side, I finally learned what push.default=simple does :).

Should be fixed.

Thanks @evanpw
Start combining various development documentation into one place.
State: closed

Closes #6232 

In working with the various "Contributing" documents recently it was clear that we needed to combine them.  This is a stab at doing so.  I took the various components from pydata.org, CONTRIBUTING.md, various wiki pages, and the already included Documentation documents and created this file.  I think it's a good start.


Labels:
name: Docs

Comments:
@rockg this is pretty awesome!

I would like to emphasize test driven development (TDD)
eg write a test to reproduce where test fails , fix code, then test passes
so maybe move testing a bit higher

I tried to keep things in order of how they are done.  I think it makes sense where it is in the document--after building the code and before committing.  I can put some mention of it at the top in the "working with code" section and make it more explicit in the testing section.  Would that work?

lgtm. @jorisvandenbossche @shoyer 

Indeed, thanks @rockg !

With this I would like to redirect http://pandas.pydata.org/developers.html and https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md to this page.  Does that sound good? I think CONTRIBUTING.md is what is linked when making a new pull request (something like "Please read the following guidelines...").  Is it possible to have something else show up in that message, e.g., a list of things, or can it only point to this file?

I'm pretty sure the GitHub link can only point to the file with that name.

On Thu, Apr 2, 2015 at 4:40 PM, rockg notifications@github.com wrote:

> With this I would like to redirect
> http://pandas.pydata.org/developers.html and
> https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md to this
> page. Does that sound good? I think CONTRIBUTING.md is what is linked when
> making a new pull request (something like "Please read the following
> guidelines..."). Is it possible to have something else show up in that
> message, e.g., a list of things, or can it only point to this file?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9754#issuecomment-89080159.

I can change the developers links to just about anything

ok let's merge and I'll fix the link tonight

then we can review everything and fix if needed

I think we might need to link CONTRIBUTING.md to the dev page for this to work

"dev page" is the page added for this PR, right?  If so, yes, I think this is what we want.  And then we llink the developers from pydata to the docs page as well.

yep

Better yet, probably link to "Contributing to the code base"

@rockg Wow, this is great!

http://pandas.pydata.org/index.html

is updated (the About Pandas / Contributing Guidelines link), currently goes to the dev docs.
BUG: where gives incorrect results when upcasting (GH 9731)
State: closed

closes #9731

The main issue is when the destination and source arrays have different lengths, `np.putmask` doesn't behave like `arr[mask] = values`:

"Sets `a.flat[n] = values[n]` for each n where `mask.flat[n]==True`"

We have to use `np.place` instead. A secondary issue is that `np.place` doesn't automatically convert an integer to a `datetime64` like `np.putmask` does (I created a numpy issue for this), so we need an additional check for that case. The rest of the commit is just cleaning up `_maybe_upcast_putmask`, which had some parameters that were never used, and a confusing docstring.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@evanpw thanks!

this was subtle!

I assume you changed to use `np.place` because some OTHER tests were initially failing on your change. Or do we need some tests for the same operation but with dtype = `M8[ns]` or `m8[ns]`?

if so, can you do a pr with those?

The tests that were failing after I changed `np.putmask` to `np.place` were the ones testing `idxmax` and `idxmin`, but the new tests in GH #9804 are more direct.
BUG: Bring pandas up to date with pandas-datareader
State: closed

Two bug fixes from pandas-datareader.
pydata/pandas-datareader#25
pydata/pandas-datareader#24


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Added fix per pydata/pandas-datareader#35

@dstephens99 ty!
Plotting: don't change visibility of xaxis labels and ticklabels if passing in a axis
State: closed

This does two things:
- fix for changing the wrong xlabels/xticklabels if a gridspec is used and the ax is passed in (not sure about using subplot without gridspec)
- changes the default behaviour for sharex if an axis is passed in: before the visibility changes were applied per default, now they are only applied if explicitly requested via `sharex=True`

Closes: #9737


Labels:
name: Visualization
description: plotting

Comments:
Ok, seems that there is a place where the plotting code is tested... will redo...

Ok, fix incoming... rebasing this will be interesting :-(

ok, updated the code and the tests... Let's see...

Updated docstrings and fixed whatsnew

Thanks @JanSchulz. I agree that a user passing in an `ax` should mean they handle more things with the axes labels and we don't try to mess with that.

@sinhrks  look good to you?

Squashed and fixed the above comments

@JanSchulz @TomAugspurger Looks great!

Ok, thanks all.
ENH: Add an axis parameter to DataFrame.diff
State: closed


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
+1 on adding this parameter. I think the axis should be flipped though, right? `axis=0` is a difference along the index, i.e. `df.iloc[1] - df.iloc[0]`. Similar to how `cumsum` operates along the index.

See also `np.diff`.

@TomAugspurger: You're right about `np.diff` having a different convention. All of the internal diff machinery (in core.internals.Block, core.common, and C implementations in algos) has this opposite convention as well, but maybe I'll just invert the axis in this public facing `DataFrame.diff` method.

@evanpw you need to pass the axis as `self._get_block_manager_axis(axis)` which handles this. frames (only) are represented with inverted axes.

Ahh, that explains it.

@evanpw thanks!
TST: Fix dateutil version check
State: closed

This allows the test to pass when using dateutil version 2.4.0

```
In [2]: '2.4.0' <= LooseVersion('2.4')
Out[2]: False

In [3]: '2.4.0' <= LooseVersion('2.4.0')
Out[3]: True
```


Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks

Thank _you_!
Moved imports into functions that use them
State: closed

closes #9713
closes #8327

Changes to be committed:
modified: pandas/io/gbq.py

Moved google api imports into functions that call them, removed unused imports. This fixes #9713 


Labels:

Comments:
cc @jreback @jacobschaer

I've tested this change and successfully pulled data from and pushed to GBQ with it.

@jreback good point. I've moved the Python3 check into the function that checks for compatibility with the google API

- can you rebase
- pls add a release note to 0.16.1

cc @jacobschaer

can you give a test out pls.

Rebase done and release note added.

@cgrin looks like you merged in something else.

see instruction on git-fu [here](http://pandas-docs.github.io/pandas-docs-travis/contributing.html)

Alright, that should take care of that. Not sure what happened yesterday. This closes #9713 and #8327 

@cgrin the `_test_imports` is still called somewhere (and is failing)

@jreback Looks like it's called in the test suite. Since the `_test_imports` function itself gone due to the imports happening on demand, should we remove this test? It's the `test_requirements` function (lines 32-36 of pandas/io/tests/test_gbq.py) raising the error, which is in turn called by each class in the file.

We could also bring that function into test_gbq.py, removing the imports that are no longer necessary.

you _can_ change the tests. The idea is to have a series of tests:
- that will allow importing of gbq and not call anything (e.g. to test that if the deps are not installed simply importing will not fail)
- use `to/read_gbq` with deps installed (which IIRC the ci does), but since we don't have credentials, they skip gracefully (e.g. the expected expections happen)
- 'real' testing, currently done offline only (as credentials are necessary) that actually tests functionaility

Thanks. I opted to move this function into the test suite and remove the no longer needed imports. (Also rebased again as there has been some movement on the upstream repo). Hopefully this gets us where we need to be!

Looks like I missed a couple imports there. Apologies, I'm having issues getting the tests to run at all locally. 686ee7e should fix the compatibility tests.

@cgrin ok, looks good. if you could squash to a single commit would be gr8.

anything beyond the release note for #8327. e.g. would a user seeing this need to do anything? 
this is prob not back-compat, but 0.16.0 is broken anyhow.....

Rebased one more time and squashed everything.

Regarding #8327, the `run` function has been [deprecated since at least June 2013](https://groups.google.com/forum/#!topic/google-api-python-client/s4r7NRmKE3w), so it's very unlikely that anyone would need to take any action. `google-api-python-client` already has a version requirement of 1.2.0, which was released in July 2013 and includes the `run_flow` function, so this is more of a cleanup to get rid of the warning on auth than anything else.

merged via https://github.com/pydata/pandas/commit/2cf4132c6c30d0a26dd4fa316aaa7579a4fcc1d3

thanks!
DOC: add DataFrame.assign to API docs
State: closed

@TomAugspurger does this look like the right place to put this?

@jorisvandenbossche can we possibly merge this into the docs for v0.16?


Labels:
name: Docs

Comments:
yep, will do when I do an update of the docs (planned to do that anyway one of the coming days when I fix the Accessor thing, so there are several things I have to include)
ENH Make year and quarter dash-separatable
State: closed

This closes #9688 by making years and quarters separable by dashes.


Labels:
name: Enhancement
name: Period
description: Period data type

Comments:
@jreback I think I've adjusted them accordingly, although I'm not quite sure about the how you want the test-case class. Anyhow, tell me if it looks OK so I can rebase it :)

minor comments, pls squash as well. thanks. ping when green.

@jreback That should do it, commits are now squashed.

thank you!

Happy to help, pandas has saved my ass too many times to count.

William Rudenmalm
tel. +46 70 263 86 33
www.whn.se

On Thu, Mar 26, 2015 at 10:50 AM, jreback notifications@github.com wrote:

> thank you!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9701#issuecomment-86428206.
BUG: ensure we use group sizes, not group counts, in transform (GH9697)
State: closed

Switch count() to size(), so that when we build the expanded values we're using the size of the groups and not simply the number of non-null values they have.  Fixes #9697.


Labels:
name: Bug
name: Groupby

Comments:
This looks great to me! Can you add a bug fix note to `0.16.1.txt`?

@dsm054 thanks!
BUG: fix for interp with axis=1 and inplace
State: closed

Closes https://github.com/pydata/pandas/issues/9687

There's still some weird issues with dtypes. In my test case there's a DataFrame with dtypes [int, float, int]. After the interpolation they should be downcasts to [int, int, int] when the missing value is filled. But along the way a transpose changes things to `[float, float, float]` before filling. And then (I think) `self._update_inplace` doesn't change dtypes, so the actual result is `[float, float, float]`.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
closing pls reopen if/when updated
BUG: datetime/timedelta Series quantile() call
State: closed

Changes to be committed:
    modified:   pandas/core/series.py
    modified:   pandas/tests/test_series.py

Fixes global reference to iNaT (should be tslib.iNaT) in series._maybe_box.
Adds test `test_datetime_timedelta_quantiles` to check for proper return value
in test_series.py.

Issue #9675


Labels:
name: Bug

Comments:
pls add a release note for what's new/0.16.1 (bug fix section)

pls squash to a single commit

merged via https://github.com/pydata/pandas/commit/a4772028fef7f660cf6d94129a1a9a6f3852e441

thanks!
Start python3 compatibility
State: closed

Start for #9660

These are changes to get vbench running under python3.  There are quite a few tests that need modification, but with these changes the code will parse and the suite can run (along with [vbench#36](https://github.com/pydata/vbench/pull/36)).


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@rockg Really love you are working on this! Just wanted to mention that I put a consideration about this in the issue #9660 (but of course, that does not prevent us improving vbench as well, certainly on the short term).

@rockg let's see if this can be fixed in 0.17.0; note that there are other PR's for using asv (but nice to fix this in the interim).

closing as stale, but if you would like to reopen and fixup pls do.
Fixes groupby.apply() error when no returns #9684
State: closed

Closes issue: #9684

Returns empty object even if all entries are missing. 


Labels:
name: Groupby
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Sorry, don't quite follow. Would you mind elaborating?

OK -- I'd then need to extract the first item from the list below that. That still preferable?

@nickeubank hmm, maybe leave it the original (and put in a try except for StopIteration like you did).
then it doesn't materialzie it 

whats new is now available for 0.16.1 https://github.com/pydata/pandas/commit/5ebf521eac528f5a4b772e4677f70b689b8b1826

pls add a test and a mention in v0.16.1 whatsnew.

merged via https://github.com/pydata/pandas/commit/990972bf2e4cc14c2a10765b4ce1ba51ab5b7271

thanks!
Bug: Access unknown attribute of timedeltaindexed series used to raise ValueError
State: closed

closes #9680

And now it raises AttributeError.


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
whats new is now available for 0.16.1 https://github.com/pydata/pandas/commit/5ebf521eac528f5a4b772e4677f70b689b8b1826

merged via  https://github.com/pydata/pandas/commit/cf6203793cc2dd9d3de571ec3def57726244abd4

thanks!
BUG: workaround PyTables 319, but not setting expected rows (GH8265, GH9676)
State: closed

closes #8265
closes #9676 


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
cc @rockg 
cc @alexfields 

give this a try and see if it fixed the issues you guys reported. lmk and can put this in 0.16.0

That did not work for me and actually returned fewer records than previously.  With this change, 2804 records were returned versus 2892 previously (the actual number should be 2972 records).

Maybe important to note that I tested this on 0.14.0 and putting your change in as that is what I have at work.  I don't know if there have been other substantive changes that might impact my test.

My test agrees with @rockg - resaving my "problem" hdf via this commit and then reading with where statements gives slightly fewer lines than before (192334 vs 193757 previously, full file is 202836). Sorry @jreback! BTW in my case I am not using chunks or start/stop, I am selecting on the full file, and it still fails.

In the meantime I've just been calling ptrepack every time I save an HDF. I don't know if this always solves the problem but it has solved it every time I've tested, and gives an added bonus of speeding up on-disk selects anyway. As long as the tables saved by ptrepack are OK, this doesn't seem like such a bad workaround for now.

hmm ok
can u guys test with master on your dataset when u have a chance and lmk?

Master 026a1221dee770597b5438a8a43f154bbdf891df is giving me the same 193757 number as in 0.15.2 (different from the 7480a4b0411400acc86bd9ca92c902b4081921a0 commit you sent earlier).

ok, seems that even though MY test worked, something else is going on. Ok will close this and can bug the PyTables guys to see if they can fix.
DataFrame.plot error when both 'color' and 'style' arguments are passed (GH9671)
State: closed

closes #9671

`self.style` can be a list of strings or a string. If it is a string, we make a list containing that string, so that we only have to deal with a list of strings. We check every element in that list and raise an error if it has a color symbol.
The test makes sure that calling `plot()` and passing both `color` and `style` arguments is allowed if there is no color symbol in the style string(s), and that passing both `color` and `style` arguments is not allowed if there is a color symbol in the style string(s). This test is done for a dataframe and for a series.


Labels:
name: Visualization
description: plotting

Comments:
Thanks. In the future, you don't need to open up a new PR. You can push changes to your original pull request and github will pick them up. I made a couple comments inline. Please let me know if you've got any questions or if anything I wrote doesn't make sense.

The tests look great! I'm glad you included one that _should_ raise an exception (I always forget these). I just had a few things to make them fit into our test framework.

Once you make those changes you can test these specific ones with `nosetests test_graphics:TestDataFramePlots.<test_name>`. You also might want to make new tests instead of attaching them to `test_plot`. That way if they fail in the future we can easily see which test failed.

Thanks for your comments Tom. I made the changes you suggested.

@davidbrochart could you rebase on master and add a release note in `doc/source/whatsnew/v0.16.1.txt`? Thanks.

You can also [squash](https://github.com/pydata/pandas/wiki/Using-Git#fetch-and-then-rebase-interactively-to-squash-reword-fix-otherwise-change-some-commits) your commits down to one. Let me know if you have any troubles.

@TomAugspurger I rebased on master and added a release note. Hope this is fine.

Merged via https://github.com/pydata/pandas/commit/f00d6bba5a7ee42569b9d6f7848962fd04a3360d

Thanks @davidbrochart
Fix for issue #9671
State: closed

closes https://github.com/pydata/pandas/issues/9671

`self.style` can be a list of strings or a string. If it is a string, we make a list containing that string, so that we only have to deal with a list of strings. We check every element in that list and raise an error if it has a color symbol.


Labels:
name: Visualization
description: plotting

Comments:
Thanks for this. A test would be great. You can put it in `pandas/tests/test_graphics.py`.

This will go in .16.1, and we'll start merging PRs for that after .16.0 is released (maybe this weekend).
BUG: support TimedeltaIndex serialization in fixed stores (GH9635)
State: closed

closes #9635

Hop, revival of PR9662, after some bad git moves.


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore
name: Timedelta
description: Timedelta data type

Comments:
whats new is now available for 0.16.1 https://github.com/pydata/pandas/commit/5ebf521eac528f5a4b772e4677f70b689b8b1826

merged via https://github.com/pydata/pandas/commit/a2a5cec32ab63fe18b5840eb8a7712e9bf99c319

thanks!
BUG: support TimedeltaIndex serialization in fixed stores (GH9635)
State: closed

closes https://github.com/pydata/pandas/issues/9635

Small change to fix issue 9635. I'm a beginner contributor, feel free to crush it out.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO HDF5
description: read_hdf, HDFStore
name: Timedelta
description: Timedelta data type

Comments:
I updated your comment with "closes" and a link to the issue. Now Github will automatically close that issue when we merge this.

A couple more things.
1. Could you add a test for this? Something that fails on master (without your patch), and works on your branch
2. Add a release note in `doc/source/whatsnew/v0.16.1` with a sentence describing the bug.

Actually, that whatsnew file may not exist yet. Once we start merging pull requests for .16.1 it will be there.

@JonasAbernot pretty good!

will need a release note (wait till the end of the week after I push the file up after we release 0.16.0)

Woups, this time I may need your help. Are all these commits normal ? I tried the 

```
git fetch upstream
git rebase upstream/master
```

from the wiki.  
Should I come back ? And how ?

[This](https://github.com/pydata/pandas/wiki/Using-Git#fetch-and-then-rebase-interactively-to-squash-reword-fix-otherwise-change-some-commits) is the one you want.

The `--interactive` should open up a window in your editor. [Here's](https://help.github.com/articles/about-git-rebase/) some more info. Basically you'll squash them all down to the first commit. Ping us in [chat](https://gitter.im/pydata/pandas) if you have any trouble.
DOC: organize stringmethods in what's new
State: closed

Added summarized section to the release note based on the discussion in  #9386


Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
@sinhrks Thanks! But, you will have to rebase (and probably solve some conflicts after my PR that reorganized the whatsnew file a bit). But you can put the section under "New features" (eg just before 'other enhancements')

Sure, rebased.

@sinhrks Merged this, thanks! I will look into the docstring linking thing. Ping #9599